2024-04-06T13:52:51.104 INFO:root:teuthology version: 0.0.1.dev265+gd7d6e60 2024-04-06T13:52:51.104 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_teuthology@teuthology --name teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067 --verbose -- /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067/orig.config.yaml 2024-04-06T13:52:51.165 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T13:52:51.259 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067 branch: reef description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7639067' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: 800ec6d64a7fa0648ad53edfc6e50b41fc4cd674 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 3144 sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 sleep_before_teardown: 0 subset: 362/512 suite: fs suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 800ec6d64a7fa0648ad53edfc6e50b41fc4cd674 targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBUzMqv0R5sLiOVPnkaYLbbQGtnOuclmW8fIea0arHXtA/fEe7V0zvFUdlGemM9Ey00z8THJzr3NxzggwIskAR0= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLm54Q5oAVlTy7szXP2arVi+qjf+j/XFM0EQrOPWyeaO39iyb3srDW+IlRv2FfOhae+FDQ1wsR0DIB2kPciUHQU= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: d7d6e60867cadf19fbfd7655a551ae7d58060757 timestamp: 2024-04-03_22:24:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-06T13:52:51.259 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa; will attempt to use it 2024-04-06T13:52:51.260 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks 2024-04-06T13:52:51.260 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-06T13:52:51.262 INFO:teuthology.task.internal:Checking packages... 2024-04-06T13:52:51.286 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '052adcadd0534eddb0d73c5afbc083a84cd9dc46' 2024-04-06T13:52:51.286 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T13:52:51.286 INFO:teuthology.packaging:ref: None 2024-04-06T13:52:51.287 INFO:teuthology.packaging:tag: None 2024-04-06T13:52:51.287 INFO:teuthology.packaging:branch: reef 2024-04-06T13:52:51.287 INFO:teuthology.packaging:sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T13:52:51.287 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef 2024-04-06T13:52:51.783 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1169.g37f5b3f8 2024-04-06T13:52:51.785 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-06T13:52:51.792 INFO:teuthology.task.internal:no buildpackages task found 2024-04-06T13:52:51.793 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-06T13:52:51.849 INFO:teuthology.task.internal:Saving configuration 2024-04-06T13:52:51.869 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-06T13:52:51.911 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T13:52:51.936 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 13:47:45.784755', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBUzMqv0R5sLiOVPnkaYLbbQGtnOuclmW8fIea0arHXtA/fEe7V0zvFUdlGemM9Ey00z8THJzr3NxzggwIskAR0='} 2024-04-06T13:52:51.958 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 13:47:45.786645', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLm54Q5oAVlTy7szXP2arVi+qjf+j/XFM0EQrOPWyeaO39iyb3srDW+IlRv2FfOhae+FDQ1wsR0DIB2kPciUHQU='} 2024-04-06T13:52:51.958 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-06T13:52:51.978 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-06T13:52:51.979 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-06T13:52:51.979 INFO:teuthology.run_tasks:Running task console_log... 2024-04-06T13:52:52.087 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1a778873a0>, signals=[15]) 2024-04-06T13:52:52.088 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-06T13:52:52.094 INFO:teuthology.task.internal:Opening connections... 2024-04-06T13:52:52.094 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T13:52:52.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T13:52:52.171 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-04-06T13:52:52.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T13:52:52.243 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-06T13:52:52.250 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-04-06T13:52:52.273 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-04-06T13:52:52.274 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8" 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8" 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8" 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-06T13:52:52.336 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-04-06T13:52:52.337 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T13:52:52.337 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-06T13:52:52.337 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T13:52:52.337 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-04-06T13:52:52.359 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-04-06T13:52:52.381 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-04-06T13:52:52.381 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-04-06T13:52:52.443 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T13:52:52.444 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-06T13:52:52.444 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T13:52:52.444 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-04-06T13:52:52.467 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-06T13:52:52.477 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-06T13:52:52.546 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-06T13:52:52.546 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-04-06T13:52:52.549 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-04-06T13:52:52.570 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-06T13:52:52.599 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-06T13:52:52.599 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-04-06T13:52:52.611 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-04-06T13:52:52.648 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-06T13:52:52.710 INFO:teuthology.run_tasks:Running task kernel... 2024-04-06T13:52:52.778 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-06T13:52:52.778 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-06T13:52:52.779 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T13:52:52.779 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T13:52:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:52:52.800 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-04-06T13:52:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:52:52.802 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-06T13:52:52.862 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-489.el8.x86_64 2024-04-06T13:52:52.862 INFO:teuthology.task.kernel:Running kernel on smithi086: 4.18.0-489.el8.x86_64 2024-04-06T13:52:52.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-04-06T13:52:52.864 INFO:teuthology.orchestra.run.smithi032.stdout:4.18.0-489.el8.x86_64 2024-04-06T13:52:52.864 INFO:teuthology.task.kernel:Running kernel on smithi032: 4.18.0-489.el8.x86_64 2024-04-06T13:52:52.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-04-06T13:52:58.586 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - AppStream 9.3 MB/s | 28 MB 00:02 2024-04-06T13:52:58.659 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - AppStream 8.7 MB/s | 28 MB 00:03 2024-04-06T13:53:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - BaseOS 8.8 MB/s | 10 MB 00:01 2024-04-06T13:53:04.928 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:00 2024-04-06T13:53:06.635 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras 118 kB/s | 18 kB 00:00 2024-04-06T13:53:06.953 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras 63 kB/s | 18 kB 00:00 2024-04-06T13:53:07.044 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 7.7 kB 00:00 2024-04-06T13:53:07.270 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-04-06T13:53:07.504 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 7.7 kB 00:00 2024-04-06T13:53:07.822 INFO:teuthology.orchestra.run.smithi032.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-04-06T13:53:09.271 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 16 MB 00:01 2024-04-06T13:53:10.231 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 16 MB 00:02 2024-04-06T13:53:13.050 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 332 kB/s | 24 kB 00:00 2024-04-06T13:53:14.256 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 536 kB/s | 24 kB 00:00 2024-04-06T13:53:16.126 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-06T13:53:16.126 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-06T13:53:16.242 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T13:53:16.243 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T13:53:16.243 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-04-06T13:53:16.243 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T13:53:16.243 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T13:53:16.243 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 4.18.0-548.el8 baseos 10 M 2024-04-06T13:53:16.243 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T13:53:16.243 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 4.18.0-548.el8 baseos 44 M 2024-04-06T13:53:16.244 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 4.18.0-548.el8 baseos 36 M 2024-04-06T13:53:16.244 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T13:53:16.244 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T13:53:16.244 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T13:53:16.244 INFO:teuthology.orchestra.run.smithi086.stdout:Install 3 Packages 2024-04-06T13:53:16.244 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T13:53:16.245 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 90 M 2024-04-06T13:53:16.245 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 96 M 2024-04-06T13:53:16.245 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T13:53:17.448 INFO:teuthology.orchestra.run.smithi086.stdout:(1/3): kernel-4.18.0-548.el8.x86_64.rpm 10 MB/s | 10 MB 00:01 2024-04-06T13:53:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-06T13:53:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-06T13:53:17.602 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T13:53:17.602 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 4.18.0-548.el8 baseos 10 M 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 4.18.0-548.el8 baseos 44 M 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 4.18.0-548.el8 baseos 36 M 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout:Install 3 Packages 2024-04-06T13:53:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T13:53:17.604 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 90 M 2024-04-06T13:53:17.604 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 96 M 2024-04-06T13:53:17.604 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T13:53:18.340 INFO:teuthology.orchestra.run.smithi086.stdout:(2/3): kernel-modules-4.18.0-548.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-04-06T13:53:18.808 INFO:teuthology.orchestra.run.smithi032.stdout:(1/3): kernel-4.18.0-548.el8.x86_64.rpm 10 MB/s | 10 MB 00:00 2024-04-06T13:53:18.923 INFO:teuthology.orchestra.run.smithi086.stdout:(3/3): kernel-core-4.18.0-548.el8.x86_64.rpm 18 MB/s | 44 MB 00:02 2024-04-06T13:53:18.923 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T13:53:18.924 INFO:teuthology.orchestra.run.smithi086.stdout:Total 34 MB/s | 90 MB 00:02 2024-04-06T13:53:19.615 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T13:53:19.866 INFO:teuthology.orchestra.run.smithi032.stdout:(2/3): kernel-modules-4.18.0-548.el8.x86_64.rpm 18 MB/s | 36 MB 00:02 2024-04-06T13:53:20.042 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T13:53:20.043 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T13:53:20.267 INFO:teuthology.orchestra.run.smithi032.stdout:(3/3): kernel-core-4.18.0-548.el8.x86_64.rpm 18 MB/s | 44 MB 00:02 2024-04-06T13:53:20.267 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T13:53:20.267 INFO:teuthology.orchestra.run.smithi032.stdout:Total 34 MB/s | 90 MB 00:02 2024-04-06T13:53:20.987 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T13:53:21.284 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T13:53:21.308 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T13:53:21.436 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T13:53:21.436 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T13:53:22.749 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T13:53:22.768 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T13:53:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T13:53:24.567 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-4.18.0-548.el8.x86_64 1/3 2024-04-06T13:53:25.692 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T13:53:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-4.18.0-548.el8.x86_64 1/3 2024-04-06T13:53:28.106 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-548.el8.x86_64 1/3 2024-04-06T13:53:28.483 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-4.18.0-548.el8.x86_64 2/3 2024-04-06T13:53:29.896 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-4.18.0-548.el8.x86_64 1/3 2024-04-06T13:53:30.312 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-4.18.0-548.el8.x86_64 2/3 2024-04-06T13:53:31.924 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-548.el8.x86_64 2/3 2024-04-06T13:53:32.265 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:34.000 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-4.18.0-548.el8.x86_64 2/3 2024-04-06T13:53:34.298 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:54.219 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:54.242 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:54.589 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:54.589 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-4.18.0-548.el8.x86_64 1/3 2024-04-06T13:53:54.589 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-4.18.0-548.el8.x86_64 2/3 2024-04-06T13:53:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T13:53:55.116 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T13:53:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-4.18.0-548.el8.x86_64 kernel-core-4.18.0-548.el8.x86_64 2024-04-06T13:53:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-4.18.0-548.el8.x86_64 2024-04-06T13:53:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T13:53:55.116 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T13:53:55.272 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T13:53:55.488 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-548.el8.x86_64 2024-04-06T13:53:55.489 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-548.el8.x86_64 2024-04-06T13:53:55.489 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-548.el8.x86_64 2024-04-06T13:53:55.489 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T13:53:55.489 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T13:53:55.489 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-06T13:53:55.489 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T13:53:55.490 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T13:53:55.701 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-548.el8.x86_64 2024-04-06T13:53:55.701 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-548.el8.x86_64 2024-04-06T13:53:55.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-04-06T13:53:55.734 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-04-06T13:53:55.734 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-04-06T13:53:55.734 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.02 2024-04-06T13:53:55.734 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 129.el8 2024-04-06T13:53:55.734 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-04-06T13:53:55.734 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-06T13:53:55.734 INFO:teuthology.orchestra.run.smithi086.stdout:Group : System Environment/Base 2024-04-06T13:53:55.734 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 9265586 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:Relocations : (not relocatable) 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : CentOS BuildSystem 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-06T13:53:55.735 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-06T13:53:55.736 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-04-06T13:53:55.736 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T13:53:55.736 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-04-06T13:53:55.736 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-06T13:53:55.737 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 4.18.0-548.el8.x86_64 2024-04-06T13:53:55.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-06T13:53:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-04-06T13:53:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-4.18.0-548.el8.x86_64 1/3 2024-04-06T13:53:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-4.18.0-548.el8.x86_64 2/3 2024-04-06T13:53:58.603 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-4.18.0-548.el8.x86_64 3/3 2024-04-06T13:53:58.603 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T13:53:58.603 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T13:53:58.603 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-4.18.0-548.el8.x86_64 kernel-core-4.18.0-548.el8.x86_64 2024-04-06T13:53:58.603 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-4.18.0-548.el8.x86_64 2024-04-06T13:53:58.603 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T13:53:58.603 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T13:53:58.750 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T13:53:58.979 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-4.18.0-548.el8.x86_64 2024-04-06T13:53:58.982 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-548.el8.x86_64 2024-04-06T13:53:58.982 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-548.el8.x86_64 2024-04-06T13:53:58.982 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T13:53:58.982 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T13:53:58.982 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-06T13:53:58.982 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T13:53:58.982 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T13:53:59.204 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-4.18.0-548.el8.x86_64 2024-04-06T13:53:59.204 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-548.el8.x86_64 2024-04-06T13:53:59.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.02 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 129.el8 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Group : System Environment/Base 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 9265586 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-06T13:53:59.240 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:Relocations : (not relocatable) 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : CentOS BuildSystem 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T13:53:59.241 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-04-06T13:53:59.242 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-06T13:53:59.242 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 4.18.0-548.el8.x86_64 2024-04-06T13:53:59.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-06T13:53:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-04-06T13:53:59.380 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-04-06T13:53:59.413 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.2lYFm6FPwC 2024-04-06T13:53:59.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2lYFm6FPwC 2024-04-06T13:53:59.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.2lYFm6FPwC 2024-04-06T13:53:59.649 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.2lYFm6FPwC is 6KB 2024-04-06T13:53:59.697 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.2lYFm6FPwC 2024-04-06T13:53:59.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-04-06T13:53:59.796 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-06T13:53:59.796 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-06T13:53:59.796 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-06T13:53:59.796 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-548.el8.x86_64.conf 2024-04-06T13:53:59.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-548.el8.x86_64 2024-04-06T13:53:59.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-04-06T13:54:01.073 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-04-06T13:54:02.868 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-04-06T13:54:02.872 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-04-06T13:54:02.910 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.Uf5SsWZ5nu 2024-04-06T13:54:02.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Uf5SsWZ5nu 2024-04-06T13:54:02.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.Uf5SsWZ5nu 2024-04-06T13:54:03.116 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.Uf5SsWZ5nu is 6KB 2024-04-06T13:54:03.165 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.Uf5SsWZ5nu 2024-04-06T13:54:03.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-04-06T13:54:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-06T13:54:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-06T13:54:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-06T13:54:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-548.el8.x86_64.conf 2024-04-06T13:54:03.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-548.el8.x86_64 2024-04-06T13:54:03.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-04-06T13:54:29.956 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-06T13:54:29.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-06T13:54:29.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T13:54:33.414 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-06T13:54:33.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-06T13:54:33.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T13:54:48.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-04-06T13:54:51.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-04-06T13:54:57.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-06T13:54:57.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T13:55:00.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-04-06T13:55:00.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-06T13:55:00.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T13:55:12.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-06T13:55:12.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T13:55:19.202 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-04-06T13:55:30.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-04-06T13:55:31.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-06T13:55:31.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T13:55:31.564 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-06T13:55:31.943 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-06T13:55:31.943 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-548.el8.x86_64"... 2024-04-06T13:55:31.943 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-06T13:55:32.001 INFO:teuthology.orchestra.run.smithi032.stdout:4.18.0-548.el8.x86_64 2024-04-06T13:55:32.001 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-548.el8.x86_64 vs 4.18.0-548.el8.x86_64 2024-04-06T13:55:32.001 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T13:55:32.002 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-06T13:55:33.002 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-06T13:55:33.002 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T13:55:33.092 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-04-06T13:55:33.115 DEBUG:teuthology.parallel:result is None 2024-04-06T13:55:45.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-06T13:55:45.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T13:55:46.325 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-04-06T13:55:46.572 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-06T13:55:46.572 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-548.el8.x86_64"... 2024-04-06T13:55:46.573 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-04-06T13:55:46.589 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-548.el8.x86_64 2024-04-06T13:55:46.589 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-548.el8.x86_64 vs 4.18.0-548.el8.x86_64 2024-04-06T13:55:46.589 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T13:55:46.589 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-06T13:55:47.590 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-06T13:55:47.590 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T13:55:48.380 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-04-06T13:55:48.412 DEBUG:teuthology.parallel:result is None 2024-04-06T13:55:48.413 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-06T13:55:48.420 INFO:teuthology.task.internal:Creating test directory... 2024-04-06T13:55:48.420 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T13:55:48.423 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T13:55:48.680 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-06T13:55:48.687 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-06T13:55:48.705 INFO:teuthology.task.internal:Creating archive directory... 2024-04-06T13:55:48.705 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T13:55:48.708 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T13:55:48.750 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-06T13:55:48.757 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-06T13:55:48.757 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T13:55:48.779 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T13:55:48.820 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T13:55:48.820 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T13:55:48.831 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T13:55:48.832 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T13:55:48.833 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-06T13:55:48.840 INFO:teuthology.task.internal:Configuring sudo... 2024-04-06T13:55:48.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T13:55:48.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T13:55:48.905 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-06T13:55:48.914 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-06T13:55:48.914 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T13:55:48.948 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T13:55:48.966 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T13:55:49.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T13:55:49.120 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T13:55:49.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T13:55:49.208 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T13:55:49.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T13:55:49.277 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T13:55:49.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T13:55:49.371 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T13:55:49.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T13:55:49.493 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T13:55:49.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T13:55:49.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-04-06T13:55:49.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-04-06T13:55:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T13:55:49.644 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T13:55:50.096 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-06T13:55:50.105 INFO:teuthology.task.internal:Starting timer... 2024-04-06T13:55:50.105 INFO:teuthology.run_tasks:Running task pcp... 2024-04-06T13:55:50.150 INFO:teuthology.run_tasks:Running task selinux... 2024-04-06T13:55:50.182 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-06T13:55:50.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-04-06T13:55:50.284 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-04-06T13:55:50.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-04-06T13:55:50.345 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-04-06T13:55:50.347 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-06T13:55:50.347 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-04-06T13:55:50.374 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-04-06T13:55:50.374 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-04-06T13:55:50.409 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-04-06T13:55:50.409 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-04-06T13:55:50.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T13:55:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:55:50.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T13:55:50.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:55:50.473 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-06T13:55:50.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-04-06T13:55:50.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-04-06T13:55:50.540 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-06T13:55:50.550 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-06T13:55:50.556 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-06T13:55:50.691 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-06T13:55:50.706 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-06T13:55:50.708 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-06T14:01:48.192 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-04-06T14:01:48.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-06T14:01:48.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T14:01:48.282 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-06T14:01:48.349 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-06T14:01:48.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-06T14:01:48.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T14:01:48.435 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-04-06T14:01:48.506 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-06T14:01:48.507 INFO:teuthology.run_tasks:Running task clock... 2024-04-06T14:01:48.517 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-06T14:01:48.518 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T14:01:48.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T14:01:48.520 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T14:01:48.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T14:01:48.552 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T14:01:48.566 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T14:01:48.581 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T14:01:48.595 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T14:01:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-04-06T14:01:48.645 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-04-06T14:01:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T14:01:48.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-04-06T14:01:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T14:01:48.675 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-04-06T14:01:48.689 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T14:01:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T14:01:48.750 INFO:teuthology.orchestra.run.smithi032.stderr:bash: ntpq: command not found 2024-04-06T14:01:48.753 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T14:01:48.754 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-06T14:01:48.754 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T14:01:48.754 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T14:01:48.754 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T14:01:48.754 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T14:01:48.791 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2024-04-06T14:01:48.795 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T14:01:48.795 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-04-06T14:01:48.795 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T14:01:48.795 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T14:01:48.795 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T14:01:48.795 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T14:01:48.796 INFO:teuthology.run_tasks:Running task pexec... 2024-04-06T14:01:48.805 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-06T14:01:48.806 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:01:48.806 DEBUG:teuthology.orchestra.run.smithi032:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-06T14:01:48.806 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi086.front.sepia.ceph.com 2024-04-06T14:01:48.807 DEBUG:teuthology.orchestra.run.smithi086:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-06T14:01:49.592 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:32 ago on Sat 06 Apr 2024 02:00:17 PM UTC. 2024-04-06T14:01:49.593 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:24 ago on Sat 06 Apr 2024 02:00:25 PM UTC. 2024-04-06T14:01:51.737 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:01:51.738 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:01:51.738 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-04-06T14:01:51.738 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:01:51.738 INFO:teuthology.orchestra.run.smithi086.stdout:Resetting modules: 2024-04-06T14:01:51.738 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools 2024-04-06T14:01:51.738 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:01:51.738 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:01:51.738 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:01:51.738 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:01:51.879 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:01:51.880 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:01:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-06T14:01:51.880 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:01:51.880 INFO:teuthology.orchestra.run.smithi032.stdout:Resetting modules: 2024-04-06T14:01:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: container-tools 2024-04-06T14:01:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:01:51.881 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:01:51.881 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:01:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:01:51.998 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:01:52.135 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:01:52.808 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:35 ago on Sat 06 Apr 2024 02:00:17 PM UTC. 2024-04-06T14:01:53.005 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:27 ago on Sat 06 Apr 2024 02:00:25 PM UTC. 2024-04-06T14:01:54.885 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:01:54.922 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:01:54.922 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:01:54.922 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:01:54.922 INFO:teuthology.orchestra.run.smithi086.stdout:Installing group/module packages: 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-06T14:01:54.923 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout:Installing module profiles: 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools/common 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling module streams: 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools rhel8 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:01:54.924 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:01:54.925 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-04-06T14:01:54.925 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:01:54.925 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 27 M 2024-04-06T14:01:54.926 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 91 M 2024-04-06T14:01:54.926 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:01:55.311 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:01:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:01:55.352 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:01:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:01:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:Installing group/module packages: 2024-04-06T14:01:55.352 INFO:teuthology.orchestra.run.smithi032.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-06T14:01:55.352 INFO:teuthology.orchestra.run.smithi032.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-06T14:01:55.352 INFO:teuthology.orchestra.run.smithi032.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-06T14:01:55.352 INFO:teuthology.orchestra.run.smithi032.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-06T14:01:55.352 INFO:teuthology.orchestra.run.smithi032.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:Installing module profiles: 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: container-tools/common 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling module streams: 2024-04-06T14:01:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: container-tools rhel8 2024-04-06T14:01:55.354 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:01:55.354 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:01:55.354 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:01:55.354 INFO:teuthology.orchestra.run.smithi032.stdout:Install 13 Packages 2024-04-06T14:01:55.354 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:01:55.354 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 27 M 2024-04-06T14:01:55.355 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 91 M 2024-04-06T14:01:55.355 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:01:55.574 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.1 MB/s | 689 kB 00:00 2024-04-06T14:01:55.641 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.5 MB/s | 1.0 MB 00:00 2024-04-06T14:01:55.833 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 146 kB/s | 38 kB 00:00 2024-04-06T14:01:56.024 INFO:teuthology.orchestra.run.smithi032.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.1 MB/s | 689 kB 00:00 2024-04-06T14:01:56.091 INFO:teuthology.orchestra.run.smithi032.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.5 MB/s | 1.0 MB 00:00 2024-04-06T14:01:56.158 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 499 kB/s | 258 kB 00:00 2024-04-06T14:01:56.291 INFO:teuthology.orchestra.run.smithi032.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 142 kB/s | 38 kB 00:00 2024-04-06T14:01:56.592 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 416 kB/s | 180 kB 00:00 2024-04-06T14:01:56.608 INFO:teuthology.orchestra.run.smithi032.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 499 kB/s | 258 kB 00:00 2024-04-06T14:01:56.908 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): buildah-1.33.6-2.module_el8+976+1136073 5.0 MB/s | 9.6 MB 00:01 2024-04-06T14:01:56.934 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 72 kB/s | 25 kB 00:00 2024-04-06T14:01:56.992 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 3.5 MB/s | 4.1 MB 00:01 2024-04-06T14:01:57.042 INFO:teuthology.orchestra.run.smithi032.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 416 kB/s | 180 kB 00:00 2024-04-06T14:01:57.268 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 262 kB/s | 94 kB 00:00 2024-04-06T14:01:57.284 INFO:teuthology.orchestra.run.smithi032.stdout:(6/13): buildah-1.33.6-2.module_el8+976+1136073 5.1 MB/s | 9.6 MB 00:01 2024-04-06T14:01:57.301 INFO:teuthology.orchestra.run.smithi032.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 96 kB/s | 25 kB 00:00 2024-04-06T14:01:57.551 INFO:teuthology.orchestra.run.smithi032.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 3.3 MB/s | 4.1 MB 00:01 2024-04-06T14:01:57.610 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 154 kB/s | 53 kB 00:00 2024-04-06T14:01:57.626 INFO:teuthology.orchestra.run.smithi032.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 274 kB/s | 94 kB 00:00 2024-04-06T14:01:57.885 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 149 kB/s | 41 kB 00:00 2024-04-06T14:01:57.935 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 2.7 MB/s | 2.5 MB 00:00 2024-04-06T14:01:57.993 INFO:teuthology.orchestra.run.smithi032.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 143 kB/s | 53 kB 00:00 2024-04-06T14:01:58.252 INFO:teuthology.orchestra.run.smithi032.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 158 kB/s | 41 kB 00:00 2024-04-06T14:01:58.502 INFO:teuthology.orchestra.run.smithi032.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 2.7 MB/s | 2.5 MB 00:00 2024-04-06T14:01:58.762 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 4.8 MB/s | 8.8 MB 00:01 2024-04-06T14:01:58.763 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:01:58.763 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.1 MB/s | 27 MB 00:03 2024-04-06T14:01:59.011 INFO:teuthology.orchestra.run.smithi032.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 5.2 MB/s | 8.8 MB 00:01 2024-04-06T14:01:59.011 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:01:59.011 INFO:teuthology.orchestra.run.smithi032.stdout:Total 7.5 MB/s | 27 MB 00:03 2024-04-06T14:01:59.035 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:01:59.057 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:01:59.057 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:01:59.293 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:01:59.315 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:01:59.315 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:01:59.319 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:01:59.322 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:01:59.594 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:01:59.597 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:02:00.021 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:02:00.296 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:02:00.978 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-06T14:02:01.180 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-06T14:02:01.296 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-06T14:02:01.351 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-06T14:02:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-06T14:02:01.534 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-06T14:02:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-06T14:02:01.802 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-06T14:02:02.043 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-06T14:02:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-06T14:02:02.352 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-06T14:02:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-06T14:02:03.322 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-06T14:02:03.567 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-06T14:02:03.787 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-06T14:02:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-06T14:02:04.146 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-06T14:02:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-06T14:02:04.428 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-06T14:02:04.825 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-06T14:02:04.881 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-06T14:02:05.064 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-06T14:02:05.087 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-06T14:02:05.283 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-06T14:02:05.293 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-06T14:02:05.494 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-06T14:02:05.708 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-06T14:02:05.708 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-06T14:02:05.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-06T14:02:05.939 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-06T14:02:05.939 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-06T14:02:05.939 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-06T14:02:05.939 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-06T14:02:05.939 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-06T14:02:05.939 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-06T14:02:05.939 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-06T14:02:05.939 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-06T14:02:05.940 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-06T14:02:05.940 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-06T14:02:05.940 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-06T14:02:05.940 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-06T14:02:05.940 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-06T14:02:06.223 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-06T14:02:06.224 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-06T14:02:06.225 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-06T14:02:06.225 INFO:teuthology.orchestra.run.smithi086.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-06T14:02:06.225 INFO:teuthology.orchestra.run.smithi086.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-06T14:02:06.225 INFO:teuthology.orchestra.run.smithi086.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-06T14:02:06.225 INFO:teuthology.orchestra.run.smithi086.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-06T14:02:06.225 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:02:06.225 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-06T14:02:06.423 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-06T14:02:06.424 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-06T14:02:06.424 INFO:teuthology.orchestra.run.smithi032.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-06T14:02:06.424 INFO:teuthology.orchestra.run.smithi032.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-06T14:02:06.424 INFO:teuthology.orchestra.run.smithi032.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-06T14:02:06.424 INFO:teuthology.orchestra.run.smithi032.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-06T14:02:06.424 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:02:06.424 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:02:06.509 DEBUG:teuthology.parallel:result is None 2024-04-06T14:02:06.687 DEBUG:teuthology.parallel:result is None 2024-04-06T14:02:06.687 INFO:teuthology.run_tasks:Running task install... 2024-04-06T14:02:06.696 DEBUG:teuthology.task.install:project ceph 2024-04-06T14:02:06.696 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'}} 2024-04-06T14:02:06.696 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'} 2024-04-06T14:02:06.696 INFO:teuthology.task.install:Using flavor: default 2024-04-06T14:02:06.706 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-06T14:02:06.706 INFO:teuthology.task.install:extra packages: [] 2024-04-06T14:02:06.706 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'tag': None, 'wait_for_package': False} 2024-04-06T14:02:06.706 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T14:02:06.706 INFO:teuthology.packaging:ref: None 2024-04-06T14:02:06.707 INFO:teuthology.packaging:tag: None 2024-04-06T14:02:06.707 INFO:teuthology.packaging:branch: quincy 2024-04-06T14:02:06.707 INFO:teuthology.packaging:sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:02:06.707 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-04-06T14:02:06.709 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'tag': None, 'wait_for_package': False} 2024-04-06T14:02:06.710 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T14:02:06.710 INFO:teuthology.packaging:ref: None 2024-04-06T14:02:06.710 INFO:teuthology.packaging:tag: None 2024-04-06T14:02:06.710 INFO:teuthology.packaging:branch: quincy 2024-04-06T14:02:06.710 INFO:teuthology.packaging:sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:02:06.710 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-04-06T14:02:07.035 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/6699495ce2af437c1ff648f1736d2457f417a1aa/centos/8/flavors/default/ 2024-04-06T14:02:07.035 INFO:teuthology.task.install.rpm:Package version is 17.2.7-819.g6699495c 2024-04-06T14:02:07.054 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/6699495ce2af437c1ff648f1736d2457f417a1aa/centos/8/flavors/default/ 2024-04-06T14:02:07.055 INFO:teuthology.task.install.rpm:Package version is 17.2.7-819.g6699495c 2024-04-06T14:02:07.173 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6699495ce2af437c1ff648f1736d2457f417a1aa/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6699495ce2af437c1ff648f1736d2457f417a1aa/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6699495ce2af437c1ff648f1736d2457f417a1aa/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-06T14:02:07.173 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:02:07.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-06T14:02:07.207 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6699495ce2af437c1ff648f1736d2457f417a1aa/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6699495ce2af437c1ff648f1736d2457f417a1aa/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6699495ce2af437c1ff648f1736d2457f417a1aa/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-06T14:02:07.207 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:02:07.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-06T14:02:07.208 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-06T14:02:07.208 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T14:02:07.209 INFO:teuthology.packaging:ref: None 2024-04-06T14:02:07.209 INFO:teuthology.packaging:tag: None 2024-04-06T14:02:07.209 INFO:teuthology.packaging:branch: quincy 2024-04-06T14:02:07.209 INFO:teuthology.packaging:sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:02:07.209 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-06T14:02:07.237 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-06T14:02:07.237 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T14:02:07.238 INFO:teuthology.packaging:ref: None 2024-04-06T14:02:07.238 INFO:teuthology.packaging:tag: None 2024-04-06T14:02:07.238 INFO:teuthology.packaging:branch: quincy 2024-04-06T14:02:07.238 INFO:teuthology.packaging:sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:02:07.238 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-06T14:02:07.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-06T14:02:07.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-06T14:02:07.362 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-06T14:02:07.384 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-06T14:02:07.431 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-04-06T14:02:07.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-04-06T14:02:07.450 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-04-06T14:02:07.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-04-06T14:02:07.705 INFO:teuthology.orchestra.run.smithi032.stdout:76 files removed 2024-04-06T14:02:07.723 INFO:teuthology.orchestra.run.smithi086.stdout:76 files removed 2024-04-06T14:02:07.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw 2024-04-06T14:02:07.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-04-06T14:02:08.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 212 kB/s | 79 kB 00:00 2024-04-06T14:02:08.554 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 202 kB/s | 79 kB 00:00 2024-04-06T14:02:08.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 34 kB/s | 9.1 kB 00:00 2024-04-06T14:02:08.913 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 29 kB/s | 9.1 kB 00:00 2024-04-06T14:02:09.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 4.7 kB/s | 1.5 kB 00:00 2024-04-06T14:02:09.290 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 4.2 kB/s | 1.5 kB 00:00 2024-04-06T14:02:12.597 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - AppStream 8.2 MB/s | 28 MB 00:03 2024-04-06T14:02:13.330 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - AppStream 6.9 MB/s | 28 MB 00:04 2024-04-06T14:02:18.689 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - BaseOS 8.8 MB/s | 10 MB 00:01 2024-04-06T14:02:19.831 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - BaseOS 9.5 MB/s | 10 MB 00:01 2024-04-06T14:02:20.431 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras 110 kB/s | 18 kB 00:00 2024-04-06T14:02:20.982 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 7.7 kB 00:00 2024-04-06T14:02:21.675 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - PowerTools 8.1 MB/s | 4.9 MB 00:00 2024-04-06T14:02:21.932 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras 57 kB/s | 18 kB 00:00 2024-04-06T14:02:22.482 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 7.7 kB 00:00 2024-04-06T14:02:22.718 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-04-06T14:02:23.000 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - PowerTools 12 MB/s | 4.9 MB 00:00 2024-04-06T14:02:23.979 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-06T14:02:24.235 INFO:teuthology.orchestra.run.smithi032.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-04-06T14:02:26.119 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 16 MB 00:01 2024-04-06T14:02:27.769 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 388 kB/s | 24 kB 00:00 2024-04-06T14:02:30.186 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 455 kB/s | 24 kB 00:00 2024-04-06T14:02:31.401 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:17.2.7-819.g6699495c.el8 ceph 17 M 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:17.2.7-819.g6699495c.el8 ceph 3.5 M 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:17.2.7-819.g6699495c.el8 ceph 3.6 M 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:17.2.7-819.g6699495c.el8 ceph 5.3 M 2024-04-06T14:02:31.404 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:17.2.7-819.g6699495c.el8 ceph 24 M 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:17.2.7-819.g6699495c.el8 ceph 24 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:17.2.7-819.g6699495c.el8 ceph 798 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:17.2.7-819.g6699495c.el8 ceph 505 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:17.2.7-819.g6699495c.el8 ceph 5.6 M 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:17.2.7-819.g6699495c.el8 ceph 45 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:17.2.7-819.g6699495c.el8 ceph 117 k 2024-04-06T14:02:31.405 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:17.2.7-819.g6699495c.el8 ceph 190 k 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:17.2.7-819.g6699495c.el8 ceph 371 k 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:17.2.7-819.g6699495c.el8 ceph 359 k 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:17.2.7-819.g6699495c.el8 ceph 106 k 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout:Install 21 Packages 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-04-06T14:02:31.406 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:02:31.407 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 64 M 2024-04-06T14:02:31.407 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:02:34.983 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:02:34.986 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:02:34.986 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:02:34.986 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:02:34.986 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:17.2.7-819.g6699495c.el8 ceph 17 M 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:17.2.7-819.g6699495c.el8 ceph 3.5 M 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:17.2.7-819.g6699495c.el8 ceph 3.6 M 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:17.2.7-819.g6699495c.el8 ceph 5.3 M 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:17.2.7-819.g6699495c.el8 ceph 24 M 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:17.2.7-819.g6699495c.el8 ceph 24 k 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:17.2.7-819.g6699495c.el8 ceph 798 k 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:17.2.7-819.g6699495c.el8 ceph 505 k 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-06T14:02:34.987 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:17.2.7-819.g6699495c.el8 ceph 5.6 M 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:17.2.7-819.g6699495c.el8 ceph 45 k 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:17.2.7-819.g6699495c.el8 ceph 117 k 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:17.2.7-819.g6699495c.el8 ceph 190 k 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:17.2.7-819.g6699495c.el8 ceph 371 k 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:17.2.7-819.g6699495c.el8 ceph 359 k 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:17.2.7-819.g6699495c.el8 ceph 106 k 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:02:34.988 INFO:teuthology.orchestra.run.smithi032.stdout:Install 21 Packages 2024-04-06T14:02:34.989 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 2 Packages 2024-04-06T14:02:34.989 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:02:34.989 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 64 M 2024-04-06T14:02:34.989 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:02:39.538 INFO:teuthology.orchestra.run.smithi032.stdout:(1/23): ceph-base-17.2.7-819.g6699495c.el8.x86_ 1.2 MB/s | 5.3 MB 00:04 2024-04-06T14:02:39.630 INFO:teuthology.orchestra.run.smithi032.stdout:(2/23): ceph-selinux-17.2.7-819.g6699495c.el8.x 265 kB/s | 24 kB 00:00 2024-04-06T14:02:39.805 INFO:teuthology.orchestra.run.smithi032.stdout:(3/23): libcephfs2-17.2.7-819.g6699495c.el8.x86 4.5 MB/s | 798 kB 00:00 2024-04-06T14:02:39.913 INFO:teuthology.orchestra.run.smithi032.stdout:(4/23): libradosstriper1-17.2.7-819.g6699495c.e 4.6 MB/s | 505 kB 00:00 2024-04-06T14:02:40.814 INFO:teuthology.orchestra.run.smithi032.stdout:(5/23): ceph-radosgw-17.2.7-819.g6699495c.el8.x 3.0 MB/s | 17 MB 00:05 2024-04-06T14:02:40.898 INFO:teuthology.orchestra.run.smithi032.stdout:(6/23): librgw2-17.2.7-819.g6699495c.el8.x86_64 5.7 MB/s | 5.6 MB 00:00 2024-04-06T14:02:40.923 INFO:teuthology.orchestra.run.smithi032.stdout:(7/23): python3-ceph-argparse-17.2.7-819.g66994 412 kB/s | 45 kB 00:00 2024-04-06T14:02:40.982 INFO:teuthology.orchestra.run.smithi032.stdout:(8/23): python3-ceph-common-17.2.7-819.g6699495 1.4 MB/s | 117 kB 00:00 2024-04-06T14:02:41.007 INFO:teuthology.orchestra.run.smithi032.stdout:(9/23): python3-cephfs-17.2.7-819.g6699495c.el8 2.2 MB/s | 190 kB 00:00 2024-04-06T14:02:41.065 INFO:teuthology.orchestra.run.smithi032.stdout:(10/23): python3-rados-17.2.7-819.g6699495c.el8 4.3 MB/s | 371 kB 00:00 2024-04-06T14:02:41.107 INFO:teuthology.orchestra.run.smithi032.stdout:(11/23): python3-rbd-17.2.7-819.g6699495c.el8.x 3.5 MB/s | 359 kB 00:00 2024-04-06T14:02:41.349 INFO:teuthology.orchestra.run.smithi032.stdout:(12/23): ceph-common-17.2.7-819.g6699495c.el8.x 3.9 MB/s | 24 MB 00:06 2024-04-06T14:02:41.366 INFO:teuthology.orchestra.run.smithi032.stdout:(13/23): python3-rgw-17.2.7-819.g6699495c.el8.x 354 kB/s | 106 kB 00:00 2024-04-06T14:02:41.487 INFO:teuthology.orchestra.run.smithi086.stdout:(1/23): ceph-base-17.2.7-819.g6699495c.el8.x86_ 579 kB/s | 5.3 MB 00:09 2024-04-06T14:02:41.562 INFO:teuthology.orchestra.run.smithi086.stdout:(2/23): ceph-selinux-17.2.7-819.g6699495c.el8.x 327 kB/s | 24 kB 00:00 2024-04-06T14:02:41.708 INFO:teuthology.orchestra.run.smithi032.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 128 kB/s | 44 kB 00:00 2024-04-06T14:02:41.858 INFO:teuthology.orchestra.run.smithi032.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 510 kB/s | 259 kB 00:00 2024-04-06T14:02:41.946 INFO:teuthology.orchestra.run.smithi086.stdout:(3/23): ceph-radosgw-17.2.7-819.g6699495c.el8.x 1.7 MB/s | 17 MB 00:09 2024-04-06T14:02:41.979 INFO:teuthology.orchestra.run.smithi086.stdout:(4/23): libcephfs2-17.2.7-819.g6699495c.el8.x86 1.9 MB/s | 798 kB 00:00 2024-04-06T14:02:42.009 INFO:teuthology.orchestra.run.smithi032.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 157 kB/s | 47 kB 00:00 2024-04-06T14:02:42.029 INFO:teuthology.orchestra.run.smithi086.stdout:(5/23): libradosstriper1-17.2.7-819.g6699495c.e 5.9 MB/s | 505 kB 00:00 2024-04-06T14:02:42.043 INFO:teuthology.orchestra.run.smithi032.stdout:(17/23): librdkafka-1.6.1-1.el8.x86_64.rpm 724 kB/s | 677 kB 00:00 2024-04-06T14:02:42.071 INFO:teuthology.orchestra.run.smithi086.stdout:(6/23): python3-ceph-argparse-17.2.7-819.g66994 1.1 MB/s | 45 kB 00:00 2024-04-06T14:02:42.122 INFO:teuthology.orchestra.run.smithi086.stdout:(7/23): python3-ceph-common-17.2.7-819.g6699495 2.3 MB/s | 117 kB 00:00 2024-04-06T14:02:42.205 INFO:teuthology.orchestra.run.smithi086.stdout:(8/23): python3-cephfs-17.2.7-819.g6699495c.el8 2.2 MB/s | 190 kB 00:00 2024-04-06T14:02:42.297 INFO:teuthology.orchestra.run.smithi086.stdout:(9/23): python3-rados-17.2.7-819.g6699495c.el8. 4.0 MB/s | 371 kB 00:00 2024-04-06T14:02:42.389 INFO:teuthology.orchestra.run.smithi086.stdout:(10/23): python3-rbd-17.2.7-819.g6699495c.el8.x 3.8 MB/s | 359 kB 00:00 2024-04-06T14:02:42.426 INFO:teuthology.orchestra.run.smithi032.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 141 kB/s | 59 kB 00:00 2024-04-06T14:02:42.448 INFO:teuthology.orchestra.run.smithi086.stdout:(11/23): python3-rgw-17.2.7-819.g6699495c.el8.x 1.8 MB/s | 106 kB 00:00 2024-04-06T14:02:42.468 INFO:teuthology.orchestra.run.smithi032.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 503 kB/s | 306 kB 00:00 2024-04-06T14:02:42.493 INFO:teuthology.orchestra.run.smithi032.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 167 kB/s | 75 kB 00:00 2024-04-06T14:02:42.762 INFO:teuthology.orchestra.run.smithi086.stdout:(12/23): ceph-common-17.2.7-819.g6699495c.el8.x 2.2 MB/s | 24 MB 00:10 2024-04-06T14:02:42.869 INFO:teuthology.orchestra.run.smithi032.stdout:(21/23): librados2-17.2.7-819.g6699495c.el8.x86 8.7 MB/s | 3.5 MB 00:00 2024-04-06T14:02:42.919 INFO:teuthology.orchestra.run.smithi032.stdout:(22/23): librbd1-17.2.7-819.g6699495c.el8.x86_6 8.4 MB/s | 3.6 MB 00:00 2024-04-06T14:02:42.961 INFO:teuthology.orchestra.run.smithi032.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 3.2 MB/s | 1.7 MB 00:00 2024-04-06T14:02:42.966 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:02:42.966 INFO:teuthology.orchestra.run.smithi032.stdout:Total 8.0 MB/s | 64 MB 00:07 2024-04-06T14:02:43.008 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:02:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:02:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:02:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 509 kB/s | 259 kB 00:00 2024-04-06T14:02:43.299 INFO:teuthology.orchestra.run.smithi086.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 796 kB/s | 677 kB 00:00 2024-04-06T14:02:43.449 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:02:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:02:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 153 kB/s | 47 kB 00:00 2024-04-06T14:02:43.633 INFO:teuthology.orchestra.run.smithi086.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 122 kB/s | 44 kB 00:00 2024-04-06T14:02:43.975 INFO:teuthology.orchestra.run.smithi086.stdout:(17/23): librgw2-17.2.7-819.g6699495c.el8.x86_6 2.8 MB/s | 5.6 MB 00:01 2024-04-06T14:02:44.100 INFO:teuthology.orchestra.run.smithi086.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 126 kB/s | 59 kB 00:00 2024-04-06T14:02:44.292 INFO:teuthology.orchestra.run.smithi086.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 447 kB/s | 306 kB 00:00 2024-04-06T14:02:44.459 INFO:teuthology.orchestra.run.smithi086.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 155 kB/s | 75 kB 00:00 2024-04-06T14:02:44.510 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:02:44.651 INFO:teuthology.orchestra.run.smithi086.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 3.2 MB/s | 1.7 MB 00:00 2024-04-06T14:02:44.730 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-06T14:02:44.885 INFO:teuthology.orchestra.run.smithi086.stdout:(22/23): librbd1-17.2.7-819.g6699495c.el8.x86_6 8.4 MB/s | 3.6 MB 00:00 2024-04-06T14:02:44.939 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-06T14:02:45.260 INFO:teuthology.orchestra.run.smithi086.stdout:(23/23): librados2-17.2.7-819.g6699495c.el8.x86 3.6 MB/s | 3.5 MB 00:00 2024-04-06T14:02:45.265 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:02:45.265 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.6 MB/s | 64 MB 00:13 2024-04-06T14:02:45.307 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:02:45.325 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-06T14:02:45.353 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:02:45.354 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:02:45.648 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:17.2.7-819.g6699495c.el8.x86_64 3/25 2024-04-06T14:02:45.725 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:02:45.728 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:02:45.770 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:17.2.7-819.g6699495c.el8.x86_64 3/25 2024-04-06T14:02:46.001 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:17.2.7-819.g6699495c.el8.x86_64 4/25 2024-04-06T14:02:46.214 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 5/25 2024-04-06T14:02:46.646 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 5/25 2024-04-06T14:02:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:02:46.831 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:17.2.7-819.g6699495c.el8.x86_64 6/25 2024-04-06T14:02:47.008 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-06T14:02:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:17.2.7-819.g6699495c.el8.x86_64 6/25 2024-04-06T14:02:47.170 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-06T14:02:47.291 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-04-06T14:02:47.519 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-06T14:02:47.552 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-06T14:02:47.736 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:17.2.7-819.g6699495c.el8.x86_64 3/25 2024-04-06T14:02:47.816 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:17.2.7-819.g6699495c.el8.x86_64 3/25 2024-04-06T14:02:48.071 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:17.2.7-819.g6699495c.el8.x86_64 4/25 2024-04-06T14:02:48.253 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 5/25 2024-04-06T14:02:48.257 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-06T14:02:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:17.2.7-819.g6699495c.el8.x86_64 10/25 2024-04-06T14:02:48.480 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:17.2.7-819.g6699495c.el8.x86_64 10/25 2024-04-06T14:02:48.661 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 5/25 2024-04-06T14:02:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:17.2.7-819.g6699495c.el8.x 11/25 2024-04-06T14:02:48.820 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:17.2.7-819.g6699495c.el8.x86_64 6/25 2024-04-06T14:02:48.851 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:17.2.7-819.g6699495c.el8.x86_64 12/25 2024-04-06T14:02:49.043 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:17.2.7-819.g6699495c.el8.x86_64 13/25 2024-04-06T14:02:49.045 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:17.2.7-819.g6699495c.el8.x86_64 6/25 2024-04-06T14:02:49.256 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-04-06T14:02:49.278 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:17.2.7-819.g6699495c.el8.x86_64 14/25 2024-04-06T14:02:49.422 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:17.2.7-819.g6699495c.el8.x86_64 15/25 2024-04-06T14:02:49.437 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-06T14:02:49.501 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:17.2.7-819.g6699495c.el8.x86_64 15/25 2024-04-06T14:02:49.742 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-06T14:02:49.906 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-06T14:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-06T14:02:50.089 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-06T14:02:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:17.2.7-819.g6699495c.el8.x86_64 10/25 2024-04-06T14:02:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:17.2.7-819.g6699495c.el8.x86 19/25 2024-04-06T14:02:50.302 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:17.2.7-819.g6699495c.el8.x86_64 10/25 2024-04-06T14:02:50.463 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:17.2.7-819.g6699495c.el8.x 11/25 2024-04-06T14:02:50.615 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:17.2.7-819.g6699495c.el8.x86_64 12/25 2024-04-06T14:02:50.804 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:17.2.7-819.g6699495c.el8.x86_64 13/25 2024-04-06T14:02:51.031 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:17.2.7-819.g6699495c.el8.x86_64 14/25 2024-04-06T14:02:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:17.2.7-819.g6699495c.el8.x86_64 15/25 2024-04-06T14:02:51.225 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:17.2.7-819.g6699495c.el8.x86_64 15/25 2024-04-06T14:02:51.445 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-06T14:02:51.628 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-06T14:02:52.073 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-06T14:02:52.254 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:17.2.7-819.g6699495c.el8.x86 19/25 2024-04-06T14:02:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 20/25 2024-04-06T14:02:53.547 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 20/25 2024-04-06T14:02:54.179 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 20/25 2024-04-06T14:02:54.435 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 21/25 2024-04-06T14:02:54.579 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 21/25 2024-04-06T14:02:54.763 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:17.2.7-819.g6699495c.el8.x86_64 22/25 2024-04-06T14:02:54.992 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 20/25 2024-04-06T14:02:55.211 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 20/25 2024-04-06T14:02:55.805 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 20/25 2024-04-06T14:02:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 21/25 2024-04-06T14:02:56.227 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 21/25 2024-04-06T14:02:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:17.2.7-819.g6699495c.el8.x86_64 22/25 2024-04-06T14:03:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:17.2.7-819.g6699495c.el8.x86_64 22/25 2024-04-06T14:03:14.973 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-04-06T14:03:14.973 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-04-06T14:03:14.973 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /dev 2024-04-06T14:03:14.973 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /run 2024-04-06T14:03:14.973 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-04-06T14:03:14.974 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-04-06T14:03:14.974 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-04-06T14:03:14.974 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-04-06T14:03:14.974 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /dev 2024-04-06T14:03:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:15.539 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:17.2.7-819.g6699495c.el8.x86_64 22/25 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /run 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2024-04-06T14:03:15.540 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:03:16.799 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:17.2.7-819.g6699495c.el8.x86_64 23/25 2024-04-06T14:03:16.876 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-819.g6699495c.el8.x86_64 23/25 2024-04-06T14:03:16.876 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-06T14:03:17.104 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-06T14:03:17.105 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-06T14:03:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:17.2.7-819.g6699495c.el8.x86_64 23/25 2024-04-06T14:03:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-819.g6699495c.el8.x86_64 23/25 2024-04-06T14:03:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-06T14:03:17.568 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-06T14:03:17.568 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-06T14:03:17.832 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-06T14:03:17.832 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 1/25 2024-04-06T14:03:17.832 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 2/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:17.2.7-819.g6699495c.el8.x86_64 3/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:17.2.7-819.g6699495c.el8.x86_64 4/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 5/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:17.2.7-819.g6699495c.el8.x86_64 6/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:17.2.7-819.g6699495c.el8.x86_64 7/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:17.2.7-819.g6699495c.el8.x 8/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:17.2.7-819.g6699495c.el8.x86 9/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:17.2.7-819.g6699495c.el8.x86_64 10/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:17.2.7-819.g6699495c.el8.x86_64 11/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:17.2.7-819.g6699495c.el8.x86_64 12/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:17.2.7-819.g6699495c.el8.x86_64 13/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-06T14:03:17.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-06T14:03:17.834 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-06T14:03:17.834 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-06T14:03:17.834 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-06T14:03:17.834 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-06T14:03:17.834 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-06T14:03:17.834 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-06T14:03:17.834 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:17.2.7-819.g6699495c.el8.x86_64 22/25 2024-04-06T14:03:17.834 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-06T14:03:17.834 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:17.2.7-819.g6699495c.el8.x86_64 24/25 2024-04-06T14:03:18.274 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-06T14:03:18.274 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 1/25 2024-04-06T14:03:18.274 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 2/25 2024-04-06T14:03:18.274 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:17.2.7-819.g6699495c.el8.x86_64 3/25 2024-04-06T14:03:18.274 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:17.2.7-819.g6699495c.el8.x86_64 4/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 5/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:17.2.7-819.g6699495c.el8.x86_64 6/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:17.2.7-819.g6699495c.el8.x86_64 7/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:17.2.7-819.g6699495c.el8.x 8/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:17.2.7-819.g6699495c.el8.x86 9/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:17.2.7-819.g6699495c.el8.x86_64 10/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:17.2.7-819.g6699495c.el8.x86_64 11/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:17.2.7-819.g6699495c.el8.x86_64 12/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:17.2.7-819.g6699495c.el8.x86_64 13/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-06T14:03:18.275 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-06T14:03:18.276 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-06T14:03:18.276 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-06T14:03:18.276 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-06T14:03:18.276 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:17.2.7-819.g6699495c.el8.x86_64 22/25 2024-04-06T14:03:18.276 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-06T14:03:18.276 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:17.2.7-819.g6699495c.el8.x86_64 24/25 2024-04-06T14:03:18.682 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-06T14:03:18.685 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:18.685 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:03:19.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-test 2024-04-06T14:03:19.152 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-06T14:03:19.153 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:03:19.153 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-04-06T14:03:19.153 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.153 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-06T14:03:19.154 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:03:19.155 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:03:19.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-04-06T14:03:19.787 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:49 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:03:20.167 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:53 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:03:21.520 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:03:21.521 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:03:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:03:21.521 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:03:21.521 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:03:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:17.2.7-819.g6699495c.el8 ceph 53 M 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:17.2.7-819.g6699495c.el8 ceph 179 k 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-04-06T14:03:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:21.523 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 54 M 2024-04-06T14:03:21.523 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 236 M 2024-04-06T14:03:21.523 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:03:21.790 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:03:21.791 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:03:21.791 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:03:21.791 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:03:21.791 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:03:21.791 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:17.2.7-819.g6699495c.el8 ceph 53 M 2024-04-06T14:03:21.791 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:17.2.7-819.g6699495c.el8 ceph 179 k 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-04-06T14:03:21.792 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:03:21.793 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): libcephsqlite-17.2.7-819.g6699495c.el8.x 829 kB/s | 179 kB 00:00 2024-04-06T14:03:21.794 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 54 M 2024-04-06T14:03:21.794 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 236 M 2024-04-06T14:03:21.794 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:03:22.008 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 470 kB/s | 203 kB 00:00 2024-04-06T14:03:22.205 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): libcephsqlite-17.2.7-819.g6699495c.el8.x 825 kB/s | 179 kB 00:00 2024-04-06T14:03:22.225 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 434 kB/s | 188 kB 00:00 2024-04-06T14:03:22.414 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 477 kB/s | 203 kB 00:00 2024-04-06T14:03:22.517 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 635 kB/s | 323 kB 00:00 2024-04-06T14:03:22.567 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 207 kB/s | 71 kB 00:00 2024-04-06T14:03:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 434 kB/s | 188 kB 00:00 2024-04-06T14:03:22.923 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 635 kB/s | 323 kB 00:00 2024-04-06T14:03:22.981 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 207 kB/s | 71 kB 00:00 2024-04-06T14:03:25.901 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): ceph-test-17.2.7-819.g6699495c.el8.x86_6 12 MB/s | 53 MB 00:04 2024-04-06T14:03:25.902 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:03:25.903 INFO:teuthology.orchestra.run.smithi032.stdout:Total 12 MB/s | 54 MB 00:04 2024-04-06T14:03:25.947 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:03:25.969 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:03:25.969 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:03:26.039 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-17.2.7-819.g6699495c.el8.x86_6 13 MB/s | 53 MB 00:04 2024-04-06T14:03:26.041 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:03:26.042 INFO:teuthology.orchestra.run.smithi086.stdout:Total 13 MB/s | 54 MB 00:04 2024-04-06T14:03:26.084 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:03:26.105 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:03:26.105 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:03:26.295 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:03:26.298 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:03:26.420 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:03:26.423 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:03:26.804 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:03:26.926 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:03:27.023 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-06T14:03:27.150 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-06T14:03:27.221 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-06T14:03:27.318 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-06T14:03:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-06T14:03:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-06T14:03:27.440 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-06T14:03:27.527 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-06T14:03:27.634 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-06T14:03:27.731 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-06T14:03:27.842 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:17.2.7-819.g6699495c.el8.x86_64 5/6 2024-04-06T14:03:27.898 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:17.2.7-819.g6699495c.el8.x86_64 5/6 2024-04-06T14:03:32.929 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:17.2.7-819.g6699495c.el8.x86_64 5/6 2024-04-06T14:03:33.171 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:17.2.7-819.g6699495c.el8.x86_64 6/6 2024-04-06T14:03:33.203 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:17.2.7-819.g6699495c.el8.x86_64 5/6 2024-04-06T14:03:33.432 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:17.2.7-819.g6699495c.el8.x86_64 6/6 2024-04-06T14:03:33.456 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:17.2.7-819.g6699495c.el8.x86_64 6/6 2024-04-06T14:03:33.456 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:17.2.7-819.g6699495c.el8.x86_64 1/6 2024-04-06T14:03:33.456 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:17.2.7-819.g6699495c.el8.x86_64 2/6 2024-04-06T14:03:33.456 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-06T14:03:33.456 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-06T14:03:33.456 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-06T14:03:33.725 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-test-2:17.2.7-819.g6699495c.el8.x86_64 6/6 2024-04-06T14:03:33.725 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:17.2.7-819.g6699495c.el8.x86_64 1/6 2024-04-06T14:03:33.725 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:17.2.7-819.g6699495c.el8.x86_64 2/6 2024-04-06T14:03:33.725 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-06T14:03:33.725 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-06T14:03:33.726 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-06T14:03:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-8.el8.x86_64 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:03:33.851 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:03:34.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: jq-1.6-8.el8.x86_64 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:34.174 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:03:34.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph 2024-04-06T14:03:34.845 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:07 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:03:35.173 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:05 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:03:36.482 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:03:36.485 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:03:36.485 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:03:36.485 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:17.2.7-819.g6699495c.el8 ceph 6.4 k 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:17.2.7-819.g6699495c.el8 ceph 2.3 M 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:17.2.7-819.g6699495c.el8 ceph 1.6 M 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 240 k 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:17.2.7-819.g6699495c.el8 ceph 4.0 M 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:17.2.7-819.g6699495c.el8 ceph 17 M 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-06T14:03:36.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-06T14:03:36.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout:Install 32 Packages 2024-04-06T14:03:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:03:36.489 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 29 M 2024-04-06T14:03:36.489 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 113 M 2024-04-06T14:03:36.489 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:03:36.927 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:17.2.7-819.g6699495c.el8 ceph 6.4 k 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:17.2.7-819.g6699495c.el8 ceph 2.3 M 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:17.2.7-819.g6699495c.el8 ceph 1.6 M 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 240 k 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:17.2.7-819.g6699495c.el8 ceph 4.0 M 2024-04-06T14:03:36.931 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:17.2.7-819.g6699495c.el8 ceph 17 M 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-06T14:03:36.932 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:36.933 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:03:36.934 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:03:36.934 INFO:teuthology.orchestra.run.smithi032.stdout:Install 32 Packages 2024-04-06T14:03:36.934 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:36.934 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 29 M 2024-04-06T14:03:36.934 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 113 M 2024-04-06T14:03:36.935 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:03:37.018 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-17.2.7-819.g6699495c.el8.x86_64.rp 57 kB/s | 6.4 kB 00:00 2024-04-06T14:03:37.233 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-mds-17.2.7-819.g6699495c.el8.x86_6 7.0 MB/s | 2.3 MB 00:00 2024-04-06T14:03:37.658 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): ceph-mgr-17.2.7-819.g6699495c.el8.x86_6 2.1 MB/s | 1.6 MB 00:00 2024-04-06T14:03:37.767 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): ceph-mgr-modules-core-17.2.7-819.g66994 2.2 MB/s | 240 kB 00:00 2024-04-06T14:03:37.770 INFO:teuthology.orchestra.run.smithi032.stdout:(1/32): ceph-17.2.7-819.g6699495c.el8.x86_64.rp 23 kB/s | 6.4 kB 00:00 2024-04-06T14:03:37.850 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): ceph-mon-17.2.7-819.g6699495c.el8.x86_6 4.8 MB/s | 4.0 MB 00:00 2024-04-06T14:03:38.209 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): ceph-osd-17.2.7-819.g6699495c.el8.x86_6 18 MB/s | 17 MB 00:00 2024-04-06T14:03:38.234 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 101 kB/s | 39 kB 00:00 2024-04-06T14:03:38.246 INFO:teuthology.orchestra.run.smithi032.stdout:(2/32): ceph-mon-17.2.7-819.g6699495c.el8.x86_6 8.5 MB/s | 4.0 MB 00:00 2024-04-06T14:03:38.259 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 319 kB/s | 157 kB 00:00 2024-04-06T14:03:38.577 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 167 kB/s | 57 kB 00:00 2024-04-06T14:03:38.595 INFO:teuthology.orchestra.run.smithi032.stdout:(3/32): ceph-mgr-17.2.7-819.g6699495c.el8.x86_6 1.4 MB/s | 1.6 MB 00:01 2024-04-06T14:03:38.635 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 243 kB/s | 103 kB 00:00 2024-04-06T14:03:38.762 INFO:teuthology.orchestra.run.smithi032.stdout:(4/32): ceph-mgr-modules-core-17.2.7-819.g66994 1.4 MB/s | 240 kB 00:00 2024-04-06T14:03:38.852 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 772 kB/s | 457 kB 00:00 2024-04-06T14:03:38.938 INFO:teuthology.orchestra.run.smithi032.stdout:(5/32): ceph-mds-17.2.7-819.g6699495c.el8.x86_6 1.6 MB/s | 2.3 MB 00:01 2024-04-06T14:03:39.027 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 552 kB/s | 249 kB 00:00 2024-04-06T14:03:39.122 INFO:teuthology.orchestra.run.smithi032.stdout:(6/32): ceph-osd-17.2.7-819.g6699495c.el8.x86_6 20 MB/s | 17 MB 00:00 2024-04-06T14:03:39.144 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.2 MB/s | 641 kB 00:00 2024-04-06T14:03:39.197 INFO:teuthology.orchestra.run.smithi032.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 362 kB/s | 157 kB 00:00 2024-04-06T14:03:39.220 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 297 kB/s | 109 kB 00:00 2024-04-06T14:03:39.305 INFO:teuthology.orchestra.run.smithi032.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 106 kB/s | 39 kB 00:00 2024-04-06T14:03:39.350 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 902 kB/s | 185 kB 00:00 2024-04-06T14:03:39.454 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.6 MB/s | 384 kB 00:00 2024-04-06T14:03:39.512 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 191 kB/s | 11 kB 00:00 2024-04-06T14:03:39.547 INFO:teuthology.orchestra.run.smithi032.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 163 kB/s | 57 kB 00:00 2024-04-06T14:03:39.571 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 301 kB/s | 18 kB 00:00 2024-04-06T14:03:39.575 INFO:teuthology.orchestra.run.smithi032.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 228 kB/s | 103 kB 00:00 2024-04-06T14:03:39.629 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 836 kB/s | 49 kB 00:00 2024-04-06T14:03:39.688 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2024-04-06T14:03:39.763 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2024-04-06T14:03:39.823 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2024-04-06T14:03:39.881 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 321 kB/s | 19 kB 00:00 2024-04-06T14:03:39.948 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 358 kB/s | 24 kB 00:00 2024-04-06T14:03:39.992 INFO:teuthology.orchestra.run.smithi032.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 667 kB/s | 457 kB 00:00 2024-04-06T14:03:40.006 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 502 kB/s | 29 kB 00:00 2024-04-06T14:03:40.025 INFO:teuthology.orchestra.run.smithi032.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 521 kB/s | 249 kB 00:00 2024-04-06T14:03:40.073 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 413 kB/s | 27 kB 00:00 2024-04-06T14:03:40.148 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-04-06T14:03:40.159 INFO:teuthology.orchestra.run.smithi032.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.1 MB/s | 641 kB 00:00 2024-04-06T14:03:40.215 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.7 MB/s | 251 kB 00:00 2024-04-06T14:03:40.234 INFO:teuthology.orchestra.run.smithi032.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 209 kB/s | 44 kB 00:00 2024-04-06T14:03:40.274 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2024-04-06T14:03:40.333 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 394 kB/s | 23 kB 00:00 2024-04-06T14:03:40.359 INFO:teuthology.orchestra.run.smithi032.stdout:(15/32): python3-pycparser-2.14-14.el8.noarch.r 297 kB/s | 109 kB 00:00 2024-04-06T14:03:40.418 INFO:teuthology.orchestra.run.smithi032.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 716 kB/s | 185 kB 00:00 2024-04-06T14:03:40.493 INFO:teuthology.orchestra.run.smithi032.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 667 kB/s | 173 kB 00:00 2024-04-06T14:03:40.510 INFO:teuthology.orchestra.run.smithi032.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-04-06T14:03:40.569 INFO:teuthology.orchestra.run.smithi032.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 234 kB/s | 18 kB 00:00 2024-04-06T14:03:40.586 INFO:teuthology.orchestra.run.smithi032.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 647 kB/s | 49 kB 00:00 2024-04-06T14:03:40.644 INFO:teuthology.orchestra.run.smithi032.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 781 kB/s | 59 kB 00:00 2024-04-06T14:03:40.678 INFO:teuthology.orchestra.run.smithi032.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2024-04-06T14:03:40.712 INFO:teuthology.orchestra.run.smithi032.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2024-04-06T14:03:40.737 INFO:teuthology.orchestra.run.smithi032.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2024-04-06T14:03:40.762 INFO:teuthology.orchestra.run.smithi032.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 224 kB/s | 19 kB 00:00 2024-04-06T14:03:40.827 INFO:teuthology.orchestra.run.smithi032.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 209 kB/s | 24 kB 00:00 2024-04-06T14:03:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 250 kB/s | 29 kB 00:00 2024-04-06T14:03:40.879 INFO:teuthology.orchestra.run.smithi032.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2024-04-06T14:03:40.955 INFO:teuthology.orchestra.run.smithi032.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2024-04-06T14:03:40.980 INFO:teuthology.orchestra.run.smithi032.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.5 MB/s | 251 kB 00:00 2024-04-06T14:03:41.039 INFO:teuthology.orchestra.run.smithi032.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 275 kB/s | 23 kB 00:00 2024-04-06T14:03:41.155 INFO:teuthology.orchestra.run.smithi032.stdout:(32/32): python3-waitress-1.4.3-1.el8.noarch.rp 778 kB/s | 255 kB 00:00 2024-04-06T14:03:41.157 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:03:41.157 INFO:teuthology.orchestra.run.smithi032.stdout:Total 6.9 MB/s | 29 MB 00:04 2024-04-06T14:03:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:03:41.281 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:03:41.281 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:03:41.605 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:03:41.608 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:03:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:03:43.100 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-06T14:03:43.491 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-06T14:03:43.747 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-06T14:03:44.023 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-06T14:03:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-06T14:03:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-06T14:03:44.660 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-06T14:03:44.880 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-06T14:03:45.124 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-06T14:03:45.370 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-06T14:03:45.633 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-06T14:03:45.891 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-06T14:03:46.286 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-06T14:03:46.299 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 6.0 kB/s | 44 kB 00:07 2024-04-06T14:03:46.535 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-06T14:03:46.715 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-06T14:03:47.012 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-06T14:03:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-06T14:03:47.728 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-06T14:03:47.998 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-06T14:03:48.152 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-06T14:03:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-06T14:03:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-06T14:03:48.896 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-06T14:03:49.133 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-06T14:03:49.596 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-06T14:03:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-06T14:03:51.153 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-819.g6699495c.el8.n 27/32 2024-04-06T14:03:51.369 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 28/32 2024-04-06T14:03:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 28/32 2024-04-06T14:03:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:17.2.7-819.g6699495c.el8.x86_64 29/32 2024-04-06T14:03:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:17.2.7-819.g6699495c.el8.x86_64 29/32 2024-04-06T14:03:53.872 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:17.2.7-819.g6699495c.el8.x86_64 30/32 2024-04-06T14:03:54.102 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:17.2.7-819.g6699495c.el8.x86_64 30/32 2024-04-06T14:03:54.297 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:17.2.7-819.g6699495c.el8.x86_64 31/32 2024-04-06T14:03:54.316 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:17.2.7-819.g6699495c.el8.x86_64 31/32 2024-04-06T14:03:54.481 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:17.2.7-819.g6699495c.el8.x86_64 32/32 2024-04-06T14:03:54.912 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-2:17.2.7-819.g6699495c.el8.x86_64 32/32 2024-04-06T14:03:54.912 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:17.2.7-819.g6699495c.el8.x86_64 1/32 2024-04-06T14:03:54.912 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:17.2.7-819.g6699495c.el8.x86_64 2/32 2024-04-06T14:03:54.912 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 3/32 2024-04-06T14:03:54.912 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:17.2.7-819.g6699495c.el8.x86_64 4/32 2024-04-06T14:03:54.912 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:17.2.7-819.g6699495c.el8.x86_64 5/32 2024-04-06T14:03:54.912 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-819.g6699495c.el8.n 6/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-06T14:03:54.913 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-06T14:03:54.914 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-06T14:03:54.914 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-06T14:03:54.914 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-06T14:03:54.914 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-06T14:03:54.914 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-06T14:03:54.914 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-06T14:03:54.914 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-06T14:03:54.914 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-06T14:03:54.914 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-06T14:03:54.915 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-06T14:03:54.915 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-06T14:03:55.776 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-06T14:03:55.776 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:55.776 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:03:55.776 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-06T14:03:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-2.6-1.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-06T14:03:55.778 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-06T14:03:55.779 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-06T14:03:55.779 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-06T14:03:55.779 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-06T14:03:55.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:03:55.779 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:03:56.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base 2024-04-06T14:03:56.867 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:26 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:03:58.573 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:03:58.627 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:03:58.628 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-06T14:03:58.628 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:03:58.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install cephadm 2024-04-06T14:03:59.553 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:29 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:04:01.297 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:04:01.298 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:04:01.298 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:01.298 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:04:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 82 k 2024-04-06T14:04:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:01.299 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:04:01.299 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:01.299 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-06T14:04:01.299 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:01.299 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 82 k 2024-04-06T14:04:01.299 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 372 k 2024-04-06T14:04:01.299 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:04:01.801 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm-17.2.7-819.g6699495c.el8.noarch.rpm 163 kB/s | 82 kB 00:00 2024-04-06T14:04:01.801 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:01.802 INFO:teuthology.orchestra.run.smithi032.stdout:Total 163 kB/s | 82 kB 00:00 2024-04-06T14:04:01.802 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:04:01.812 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:04:01.812 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:04:01.814 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:04:01.818 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:04:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:04:03.667 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/1 2024-04-06T14:04:03.881 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/1 2024-04-06T14:04:04.127 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/1 2024-04-06T14:04:04.513 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/1 2024-04-06T14:04:04.513 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:04.513 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:04:04.513 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:04.513 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:04.513 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:04:04.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-immutable-object-cache 2024-04-06T14:04:05.535 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:35 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:04:07.278 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:04:07.279 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-06T14:04:07.279 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:07.279 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:04:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-819.g6699495c.el8 ceph 162 k 2024-04-06T14:04:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:07.280 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:04:07.280 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:07.280 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-06T14:04:07.280 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:07.280 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 162 k 2024-04-06T14:04:07.280 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 494 k 2024-04-06T14:04:07.280 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:04:07.479 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-immutable-object-cache-17.2.7-819.g6699495 826 kB/s | 162 kB 00:00 2024-04-06T14:04:07.479 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:07.480 INFO:teuthology.orchestra.run.smithi032.stdout:Total 821 kB/s | 162 kB 00:00 2024-04-06T14:04:07.480 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:04:07.488 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:04:07.488 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:04:07.539 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:04:07.542 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:04:07.655 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:04:07.888 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-819.g6699495c.e 1/1 2024-04-06T14:04:08.321 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-819.g6699495c.e 1/1 2024-04-06T14:04:08.696 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-819.g6699495c.e 1/1 2024-04-06T14:04:08.696 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:08.696 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:04:08.696 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:04:08.696 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:08.696 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:04:08.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr 2024-04-06T14:04:09.707 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:39 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:04:11.164 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 5.4 kB/s | 173 kB 00:31 2024-04-06T14:04:11.165 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:11.165 INFO:teuthology.orchestra.run.smithi086.stdout:Total 855 kB/s | 29 MB 00:34 2024-04-06T14:04:11.243 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:04:11.280 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:04:11.281 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:04:11.422 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:04:11.474 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:04:11.475 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-06T14:04:11.475 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:04:11.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-dashboard 2024-04-06T14:04:11.606 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:04:11.609 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:04:12.374 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:42 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:04:12.800 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:04:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-06T14:04:13.282 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-06T14:04:13.526 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-06T14:04:13.749 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-06T14:04:13.947 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-06T14:04:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:04:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:14.127 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:04:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:04:14.127 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 1.7 M 2024-04-06T14:04:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T14:04:14.127 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 24 k 2024-04-06T14:04:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 14 k 2024-04-06T14:04:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-06T14:04:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-06T14:04:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-06T14:04:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:04:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-04-06T14:04:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:14.129 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 2.0 M 2024-04-06T14:04:14.130 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 8.1 M 2024-04-06T14:04:14.130 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:04:14.162 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-06T14:04:14.346 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-06T14:04:14.515 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-06T14:04:14.538 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): ceph-prometheus-alerts-17.2.7-819.g66994 104 kB/s | 14 kB 00:00 2024-04-06T14:04:14.638 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): ceph-grafana-dashboards-17.2.7-819.g6699 101 kB/s | 24 kB 00:00 2024-04-06T14:04:14.766 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-06T14:04:14.771 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 252 kB/s | 34 kB 00:00 2024-04-06T14:04:14.830 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 146 kB/s | 43 kB 00:00 2024-04-06T14:04:14.914 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2024-04-06T14:04:14.955 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): ceph-mgr-dashboard-17.2.7-819.g6699495c. 3.0 MB/s | 1.7 MB 00:00 2024-04-06T14:04:14.957 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:14.957 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-04-06T14:04:14.969 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:04:14.979 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:04:14.979 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:04:14.985 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-06T14:04:15.070 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:04:15.073 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:04:15.199 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-06T14:04:15.343 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:04:15.465 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-06T14:04:15.586 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-06T14:04:15.815 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-06T14:04:15.831 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-06T14:04:16.047 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-06T14:04:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-06T14:04:16.160 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-819.g6699495c.el8.no 4/6 2024-04-06T14:04:16.233 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-06T14:04:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-06T14:04:16.551 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-819.g6699495c.el8.n 5/6 2024-04-06T14:04:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-06T14:04:16.749 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-819.g6699495c.el8.noarch 6/6 2024-04-06T14:04:17.036 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-06T14:04:17.054 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-819.g6699495c.el8.noarch 6/6 2024-04-06T14:04:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-819.g6699495c.el8.n 1/6 2024-04-06T14:04:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-819.g6699495c.el8.noarch 2/6 2024-04-06T14:04:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-819.g6699495c.el8.no 3/6 2024-04-06T14:04:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-06T14:04:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-06T14:04:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-06T14:04:17.445 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-06T14:04:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-06T14:04:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:17.463 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:04:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:17.464 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-06T14:04:17.464 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-06T14:04:17.464 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-06T14:04:17.464 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:17.464 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:04:17.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-06T14:04:17.721 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-06T14:04:18.163 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-06T14:04:18.378 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:48 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:04:18.380 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-06T14:04:18.580 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-06T14:04:18.907 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-06T14:04:19.196 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-06T14:04:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-819.g6699495c.el8.n 27/32 2024-04-06T14:04:19.837 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 28/32 2024-04-06T14:04:20.121 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:04:20.122 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-04-06T14:04:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:04:20.122 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-04-06T14:04:20.122 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:04:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 7.4 M 2024-04-06T14:04:20.122 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T14:04:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout:Install 8 Packages 2024-04-06T14:04:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:20.124 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 36 M 2024-04-06T14:04:20.124 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 210 M 2024-04-06T14:04:20.124 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:04:21.359 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 28/32 2024-04-06T14:04:21.413 INFO:teuthology.orchestra.run.smithi032.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.7-819 7.1 MB/s | 7.4 MB 00:01 2024-04-06T14:04:21.554 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:17.2.7-819.g6699495c.el8.x86_64 29/32 2024-04-06T14:04:21.597 INFO:teuthology.orchestra.run.smithi032.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.0 MB/s | 4.9 MB 00:01 2024-04-06T14:04:21.667 INFO:teuthology.orchestra.run.smithi032.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 3.7 MB/s | 4.7 MB 00:01 2024-04-06T14:04:21.968 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:17.2.7-819.g6699495c.el8.x86_64 29/32 2024-04-06T14:04:22.098 INFO:teuthology.orchestra.run.smithi032.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 450 kB/s | 225 kB 00:00 2024-04-06T14:04:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:17.2.7-819.g6699495c.el8.x86_64 30/32 2024-04-06T14:04:22.374 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:17.2.7-819.g6699495c.el8.x86_64 30/32 2024-04-06T14:04:22.523 INFO:teuthology.orchestra.run.smithi032.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.4 MB/s | 3.7 MB 00:01 2024-04-06T14:04:22.550 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:17.2.7-819.g6699495c.el8.x86_64 31/32 2024-04-06T14:04:22.567 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:17.2.7-819.g6699495c.el8.x86_64 31/32 2024-04-06T14:04:22.631 INFO:teuthology.orchestra.run.smithi032.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-04-06T14:04:22.707 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:17.2.7-819.g6699495c.el8.x86_64 32/32 2024-04-06T14:04:22.890 INFO:teuthology.orchestra.run.smithi032.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 470 kB/s | 172 kB 00:00 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:17.2.7-819.g6699495c.el8.x86_64 32/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:17.2.7-819.g6699495c.el8.x86_64 1/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:17.2.7-819.g6699495c.el8.x86_64 2/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 3/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:17.2.7-819.g6699495c.el8.x86_64 4/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:17.2.7-819.g6699495c.el8.x86_64 5/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-819.g6699495c.el8.n 6/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-06T14:04:23.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-06T14:04:23.117 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-06T14:04:23.118 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-06T14:04:23.118 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-06T14:04:23.118 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-06T14:04:23.118 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-06T14:04:23.118 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-06T14:04:23.118 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-06T14:04:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-06T14:04:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:23.944 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:04:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:04:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:04:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:04:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:04:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-06T14:04:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-2.6-1.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-06T14:04:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:23.947 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:04:24.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-04-06T14:04:24.555 INFO:teuthology.orchestra.run.smithi032.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 4.9 MB/s | 14 MB 00:02 2024-04-06T14:04:24.556 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:24.556 INFO:teuthology.orchestra.run.smithi032.stdout:Total 8.1 MB/s | 36 MB 00:04 2024-04-06T14:04:24.796 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:04:24.815 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:04:24.815 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:04:25.019 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:58 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:04:25.218 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:04:25.221 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:04:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:04:25.926 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-06T14:04:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-06T14:04:26.251 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-06T14:04:26.595 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:04:26.643 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:04:26.644 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-06T14:04:26.644 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:04:26.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-04-06T14:04:26.883 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-06T14:04:27.085 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-06T14:04:27.498 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:00 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:04:27.719 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-06T14:04:27.859 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-06T14:04:28.609 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-06T14:04:28.976 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-06T14:04:29.131 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:04:29.132 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:04:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:04:29.132 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:04:29.132 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:04:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 82 k 2024-04-06T14:04:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:29.133 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:04:29.133 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:04:29.133 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-06T14:04:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:29.133 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 82 k 2024-04-06T14:04:29.133 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 372 k 2024-04-06T14:04:29.133 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:04:29.201 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-06T14:04:29.395 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-17.2.7-819.g6699495c.el8.noarch.rpm 313 kB/s | 82 kB 00:00 2024-04-06T14:04:29.396 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:29.396 INFO:teuthology.orchestra.run.smithi086.stdout:Total 312 kB/s | 82 kB 00:00 2024-04-06T14:04:29.396 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:04:29.403 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:04:29.404 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:04:29.405 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:04:29.408 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:04:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:04:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/1 2024-04-06T14:04:30.107 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/1 2024-04-06T14:04:30.330 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/1 2024-04-06T14:04:30.726 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/1 2024-04-06T14:04:30.726 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:30.727 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:04:30.727 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:30.727 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:30.727 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:04:30.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-04-06T14:04:31.076 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-06T14:04:31.731 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:04 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:04:32.399 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-06T14:04:32.601 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-819.g6699495c 8/8 2024-04-06T14:04:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-819.g6699495c 8/8 2024-04-06T14:04:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-819.g6699495c 1/8 2024-04-06T14:04:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-06T14:04:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-06T14:04:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-06T14:04:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-06T14:04:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-06T14:04:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-06T14:04:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-06T14:04:33.347 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:33.347 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:04:33.349 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:04:33.350 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:04:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-04-06T14:04:33.350 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:04:33.350 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:04:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-819.g6699495c.el8 ceph 162 k 2024-04-06T14:04:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:33.350 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:04:33.351 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:04:33.351 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-06T14:04:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:33.351 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 162 k 2024-04-06T14:04:33.351 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 494 k 2024-04-06T14:04:33.351 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:04:33.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-rook 2024-04-06T14:04:33.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-17.2.7-819.g6699495 893 kB/s | 162 kB 00:00 2024-04-06T14:04:33.533 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:33.533 INFO:teuthology.orchestra.run.smithi086.stdout:Total 887 kB/s | 162 kB 00:00 2024-04-06T14:04:33.533 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:04:33.543 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:04:33.544 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:04:33.592 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:04:33.594 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:04:33.705 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:04:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-819.g6699495c.e 1/1 2024-04-06T14:04:34.270 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:04 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:04:34.313 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-819.g6699495c.e 1/1 2024-04-06T14:04:34.707 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-819.g6699495c.e 1/1 2024-04-06T14:04:34.707 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:34.708 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:04:34.708 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:04:34.708 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:34.708 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:04:34.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-04-06T14:04:35.684 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:08 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:04:36.025 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:04:36.027 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:36.027 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:04:36.027 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:36.027 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:04:36.028 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 47 k 2024-04-06T14:04:36.028 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T14:04:36.028 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-06T14:04:36.028 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-06T14:04:36.028 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-06T14:04:36.028 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-06T14:04:36.029 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-06T14:04:36.029 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-06T14:04:36.029 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-06T14:04:36.029 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-06T14:04:36.029 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-06T14:04:36.029 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-06T14:04:36.030 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-06T14:04:36.030 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-06T14:04:36.030 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:36.030 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:04:36.030 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:36.031 INFO:teuthology.orchestra.run.smithi032.stdout:Install 13 Packages 2024-04-06T14:04:36.031 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:36.032 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 1.9 M 2024-04-06T14:04:36.032 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 25 M 2024-04-06T14:04:36.032 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:04:36.604 INFO:teuthology.orchestra.run.smithi032.stdout:(1/13): ceph-mgr-rook-17.2.7-819.g6699495c.el8. 280 kB/s | 47 kB 00:00 2024-04-06T14:04:36.696 INFO:teuthology.orchestra.run.smithi032.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 76 kB/s | 20 kB 00:00 2024-04-06T14:04:36.722 INFO:teuthology.orchestra.run.smithi032.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 93 kB/s | 27 kB 00:00 2024-04-06T14:04:37.064 INFO:teuthology.orchestra.run.smithi032.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 275 kB/s | 126 kB 00:00 2024-04-06T14:04:37.089 INFO:teuthology.orchestra.run.smithi032.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 422 kB/s | 155 kB 00:00 2024-04-06T14:04:37.106 INFO:teuthology.orchestra.run.smithi032.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 269 kB/s | 110 kB 00:00 2024-04-06T14:04:37.256 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:04:37.306 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:04:37.307 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-06T14:04:37.307 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:04:37.356 INFO:teuthology.orchestra.run.smithi032.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 123 kB/s | 33 kB 00:00 2024-04-06T14:04:37.381 INFO:teuthology.orchestra.run.smithi032.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 134 kB/s | 43 kB 00:00 2024-04-06T14:04:37.407 INFO:teuthology.orchestra.run.smithi032.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 54 kB/s | 16 kB 00:00 2024-04-06T14:04:37.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-04-06T14:04:37.699 INFO:teuthology.orchestra.run.smithi032.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 237 kB/s | 81 kB 00:00 2024-04-06T14:04:37.749 INFO:teuthology.orchestra.run.smithi032.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 188 kB/s | 64 kB 00:00 2024-04-06T14:04:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 175 kB/s | 61 kB 00:00 2024-04-06T14:04:38.082 INFO:teuthology.orchestra.run.smithi032.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.7 MB/s | 1.2 MB 00:00 2024-04-06T14:04:38.083 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:38.084 INFO:teuthology.orchestra.run.smithi032.stdout:Total 962 kB/s | 1.9 MB 00:02 2024-04-06T14:04:38.145 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:04:38.161 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:04:38.161 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:04:38.165 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:11 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:04:38.255 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:04:38.259 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:04:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:04:39.089 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-06T14:04:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-06T14:04:39.537 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-06T14:04:39.757 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-06T14:04:39.800 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:04:39.801 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:04:39.801 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:04:39.801 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:04:39.801 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:04:39.801 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 1.7 M 2024-04-06T14:04:39.801 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T14:04:39.801 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 24 k 2024-04-06T14:04:39.801 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 14 k 2024-04-06T14:04:39.802 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-06T14:04:39.802 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-06T14:04:39.802 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-06T14:04:39.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:39.802 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:04:39.802 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:04:39.802 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-04-06T14:04:39.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:39.803 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.0 M 2024-04-06T14:04:39.803 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 8.1 M 2024-04-06T14:04:39.803 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:04:40.020 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-06T14:04:40.259 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-06T14:04:40.485 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): ceph-prometheus-alerts-17.2.7-819.g66994 97 kB/s | 14 kB 00:00 2024-04-06T14:04:40.502 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-grafana-dashboards-17.2.7-819.g6699 146 kB/s | 24 kB 00:00 2024-04-06T14:04:40.580 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-06T14:04:40.628 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): ceph-mgr-dashboard-17.2.7-819.g6699495c. 5.8 MB/s | 1.7 MB 00:00 2024-04-06T14:04:40.645 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2024-04-06T14:04:40.804 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 134 kB/s | 43 kB 00:00 2024-04-06T14:04:40.833 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-06T14:04:40.854 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 871 kB/s | 196 kB 00:00 2024-04-06T14:04:40.856 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:40.856 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-04-06T14:04:40.867 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:04:40.877 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:04:40.877 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:04:40.962 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:04:40.965 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:04:41.215 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:04:41.445 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-06T14:04:41.690 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-06T14:04:41.900 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-06T14:04:41.951 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-06T14:04:42.045 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-819.g6699495c.el8.no 4/6 2024-04-06T14:04:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-06T14:04:42.393 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-819.g6699495c.el8.n 5/6 2024-04-06T14:04:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-06T14:04:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-819.g6699495c.el8.noarch 6/6 2024-04-06T14:04:42.686 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-06T14:04:42.849 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:17.2.7-819.g6699495c.el8.noarch 13/13 2024-04-06T14:04:42.961 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-819.g6699495c.el8.noarch 6/6 2024-04-06T14:04:42.962 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-819.g6699495c.el8.n 1/6 2024-04-06T14:04:42.962 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-819.g6699495c.el8.noarch 2/6 2024-04-06T14:04:42.962 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-819.g6699495c.el8.no 3/6 2024-04-06T14:04:42.962 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-06T14:04:42.962 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-06T14:04:43.171 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-819.g6699495c.el8.noarch 13/13 2024-04-06T14:04:43.171 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:17.2.7-819.g6699495c.el8.noarch 1/13 2024-04-06T14:04:43.171 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-06T14:04:43.171 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-06T14:04:43.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-06T14:04:43.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-06T14:04:43.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-06T14:04:43.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-06T14:04:43.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-06T14:04:43.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-06T14:04:43.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-06T14:04:43.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-06T14:04:43.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-06T14:04:43.349 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-06T14:04:43.349 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:43.349 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:04:43.349 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:43.349 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:43.349 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:43.349 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-06T14:04:43.350 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-06T14:04:43.350 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-06T14:04:43.350 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:43.350 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:04:43.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-06T14:04:43.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-06T14:04:43.639 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:43.639 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:04:43.639 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:43.639 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-06T14:04:43.639 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:43.640 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:04:43.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-cephadm 2024-04-06T14:04:44.229 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:17 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:04:44.553 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:14 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:04:45.858 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:04:45.859 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-04-06T14:04:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:04:45.859 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-04-06T14:04:45.859 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:04:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 7.4 M 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:Install 8 Packages 2024-04-06T14:04:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:45.861 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 36 M 2024-04-06T14:04:45.861 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 210 M 2024-04-06T14:04:45.861 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:04:46.343 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:04:46.344 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:46.344 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:04:46.344 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 123 k 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout:Install 5 Packages 2024-04-06T14:04:46.345 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:46.346 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 6.0 M 2024-04-06T14:04:46.347 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 26 M 2024-04-06T14:04:46.347 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:04:46.623 INFO:teuthology.orchestra.run.smithi032.stdout:(1/5): ceph-mgr-cephadm-17.2.7-819.g6699495c.el 634 kB/s | 123 kB 00:00 2024-04-06T14:04:47.002 INFO:teuthology.orchestra.run.smithi086.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.7-819 8.2 MB/s | 7.4 MB 00:00 2024-04-06T14:04:47.023 INFO:teuthology.orchestra.run.smithi032.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 973 kB/s | 578 kB 00:00 2024-04-06T14:04:47.057 INFO:teuthology.orchestra.run.smithi032.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.1 MB/s | 476 kB 00:00 2024-04-06T14:04:47.315 INFO:teuthology.orchestra.run.smithi032.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 206 kB/s | 60 kB 00:00 2024-04-06T14:04:47.344 INFO:teuthology.orchestra.run.smithi086.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 3.8 MB/s | 4.7 MB 00:01 2024-04-06T14:04:47.419 INFO:teuthology.orchestra.run.smithi086.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.7 MB/s | 4.9 MB 00:01 2024-04-06T14:04:47.649 INFO:teuthology.orchestra.run.smithi032.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 3.9 MB/s | 4.8 MB 00:01 2024-04-06T14:04:47.652 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:47.652 INFO:teuthology.orchestra.run.smithi032.stdout:Total 4.6 MB/s | 6.0 MB 00:01 2024-04-06T14:04:47.708 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:04:47.720 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:04:47.720 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:04:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:04:47.834 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:04:47.861 INFO:teuthology.orchestra.run.smithi086.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 435 kB/s | 225 kB 00:00 2024-04-06T14:04:48.097 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:04:48.111 INFO:teuthology.orchestra.run.smithi086.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.4 MB/s | 3.7 MB 00:01 2024-04-06T14:04:48.403 INFO:teuthology.orchestra.run.smithi086.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-04-06T14:04:48.478 INFO:teuthology.orchestra.run.smithi086.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 470 kB/s | 172 kB 00:00 2024-04-06T14:04:48.481 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-06T14:04:49.531 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-06T14:04:49.747 INFO:teuthology.orchestra.run.smithi086.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 6.1 MB/s | 14 MB 00:02 2024-04-06T14:04:49.748 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:49.749 INFO:teuthology.orchestra.run.smithi086.stdout:Total 9.2 MB/s | 36 MB 00:03 2024-04-06T14:04:49.902 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-06T14:04:49.981 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:04:50.000 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:04:50.000 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:04:50.384 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:04:50.387 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:04:50.395 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-06T14:04:50.589 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-819.g6699495c.el8.noarch 5/5 2024-04-06T14:04:50.926 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-819.g6699495c.el8.noarch 5/5 2024-04-06T14:04:50.926 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/5 2024-04-06T14:04:50.926 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-06T14:04:50.926 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-06T14:04:50.926 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-06T14:04:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:04:51.063 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-06T14:04:51.199 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-06T14:04:51.337 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-06T14:04:51.337 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:51.337 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:04:51.337 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:51.338 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-06T14:04:51.338 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-06T14:04:51.338 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-06T14:04:51.338 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-06T14:04:51.338 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:51.338 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:04:51.354 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-06T14:04:51.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-fuse 2024-04-06T14:04:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-06T14:04:52.087 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-06T14:04:52.258 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:22 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:04:52.690 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-06T14:04:52.829 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-06T14:04:53.505 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-06T14:04:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-06T14:04:53.996 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:04:53.997 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:53.997 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-06T14:04:53.997 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:53.997 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:04:53.997 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:17.2.7-819.g6699495c.el8 ceph 876 k 2024-04-06T14:04:53.997 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-06T14:04:53.997 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-06T14:04:53.997 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:53.997 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:04:53.998 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:53.998 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-04-06T14:04:53.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:53.998 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 960 k 2024-04-06T14:04:53.998 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 3.1 M 2024-04-06T14:04:53.998 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:04:54.070 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-06T14:04:54.440 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): ceph-fuse-17.2.7-819.g6699495c.el8.x86_6 3.3 MB/s | 876 kB 00:00 2024-04-06T14:04:54.482 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 276 kB/s | 83 kB 00:00 2024-04-06T14:04:54.482 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:54.483 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.9 MB/s | 960 kB 00:00 2024-04-06T14:04:54.494 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:04:54.516 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:04:54.516 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:04:54.611 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:04:54.614 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:04:54.788 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:04:55.070 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-06T14:04:55.281 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:17.2.7-819.g6699495c.el8.x86_64 2/2 2024-04-06T14:04:55.677 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-fuse-2:17.2.7-819.g6699495c.el8.x86_64 2/2 2024-04-06T14:04:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:17.2.7-819.g6699495c.el8.x86_64 1/2 2024-04-06T14:04:55.820 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-06T14:04:56.142 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-06T14:04:56.142 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:56.143 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:04:56.143 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:17.2.7-819.g6699495c.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-06T14:04:56.143 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:56.143 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:04:56.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados-devel 2024-04-06T14:04:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-06T14:04:57.193 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:27 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:04:57.262 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-819.g6699495c 8/8 2024-04-06T14:04:57.567 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-819.g6699495c 8/8 2024-04-06T14:04:57.567 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-819.g6699495c 1/8 2024-04-06T14:04:57.567 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-06T14:04:57.567 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-06T14:04:57.567 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-06T14:04:57.567 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-06T14:04:57.568 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-06T14:04:57.568 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-06T14:04:58.014 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-06T14:04:58.014 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:58.014 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:04:58.014 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:04:58.014 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-06T14:04:58.014 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-06T14:04:58.014 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-06T14:04:58.014 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-06T14:04:58.015 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-06T14:04:58.015 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-06T14:04:58.015 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-06T14:04:58.015 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:04:58.015 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:04:58.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-04-06T14:04:58.889 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:31 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:04:58.968 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:17.2.7-819.g6699495c.el8 ceph 130 k 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-06T14:04:58.969 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:04:58.970 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 130 k 2024-04-06T14:04:58.970 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 464 k 2024-04-06T14:04:58.970 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:04:59.161 INFO:teuthology.orchestra.run.smithi032.stdout:librados-devel-17.2.7-819.g6699495c.el8.x86_64. 682 kB/s | 130 kB 00:00 2024-04-06T14:04:59.161 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:04:59.161 INFO:teuthology.orchestra.run.smithi032.stdout:Total 678 kB/s | 130 kB 00:00 2024-04-06T14:04:59.161 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:04:59.167 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:04:59.167 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:04:59.220 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:04:59.224 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:04:59.345 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:04:59.630 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:04:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:00.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:00.299 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:00.299 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:05:00.299 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:00.300 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:00.300 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:00.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs2 2024-04-06T14:05:00.518 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 47 k 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-06T14:05:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-04-06T14:05:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:00.522 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.9 M 2024-04-06T14:05:00.522 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 25 M 2024-04-06T14:05:00.522 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:05:01.011 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): ceph-mgr-rook-17.2.7-819.g6699495c.el8. 181 kB/s | 47 kB 00:00 2024-04-06T14:05:01.036 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 92 kB/s | 27 kB 00:00 2024-04-06T14:05:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 64 kB/s | 20 kB 00:00 2024-04-06T14:05:01.306 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:31 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 413 kB/s | 155 kB 00:00 2024-04-06T14:05:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 280 kB/s | 126 kB 00:00 2024-04-06T14:05:01.488 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 258 kB/s | 110 kB 00:00 2024-04-06T14:05:01.530 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 460 kB/s | 43 kB 00:00 2024-04-06T14:05:01.730 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 123 kB/s | 33 kB 00:00 2024-04-06T14:05:01.755 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 61 kB/s | 16 kB 00:00 2024-04-06T14:05:01.880 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 232 kB/s | 81 kB 00:00 2024-04-06T14:05:02.106 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 183 kB/s | 64 kB 00:00 2024-04-06T14:05:02.231 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 175 kB/s | 61 kB 00:00 2024-04-06T14:05:02.439 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.6 MB/s | 1.2 MB 00:00 2024-04-06T14:05:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.0 MB/s | 1.9 MB 00:01 2024-04-06T14:05:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:05:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:05:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:05:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:05:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:05:03.010 INFO:teuthology.orchestra.run.smithi032.stdout:Package libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:03.062 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:03.063 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-06T14:05:03.063 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:03.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs-devel 2024-04-06T14:05:03.175 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:05:03.405 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-06T14:05:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-06T14:05:03.846 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-06T14:05:03.951 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:33 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:04.075 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-06T14:05:04.291 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-06T14:05:04.508 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-06T14:05:04.818 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-06T14:05:05.050 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-06T14:05:05.708 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:05.709 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:05.709 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-06T14:05:05.709 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:05.709 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:05:05.709 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:17.2.7-819.g6699495c.el8 ceph 25 k 2024-04-06T14:05:05.709 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:05.709 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:05:05.709 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:05.709 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-06T14:05:05.710 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:05.710 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 25 k 2024-04-06T14:05:05.710 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 109 k 2024-04-06T14:05:05.710 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:05:05.934 INFO:teuthology.orchestra.run.smithi032.stdout:libcephfs-devel-17.2.7-819.g6699495c.el8.x86_64 111 kB/s | 25 kB 00:00 2024-04-06T14:05:05.934 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:05.935 INFO:teuthology.orchestra.run.smithi032.stdout:Total 110 kB/s | 25 kB 00:00 2024-04-06T14:05:05.935 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:05:05.939 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:05:05.939 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:05:05.941 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:05:05.944 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:05:05.992 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:05:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-06T14:05:06.196 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:06.392 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-06T14:05:06.479 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-06T14:05:06.761 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-06T14:05:06.850 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:06.850 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:06.850 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:05:06.850 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:06.850 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:06.850 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:06.952 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:17.2.7-819.g6699495c.el8.noarch 13/13 2024-04-06T14:05:06.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados2 2024-04-06T14:05:07.249 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-819.g6699495c.el8.noarch 13/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:17.2.7-819.g6699495c.el8.noarch 1/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-06T14:05:07.250 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-06T14:05:07.693 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-06T14:05:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-06T14:05:07.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-06T14:05:07.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-06T14:05:07.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-06T14:05:07.695 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:07.695 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:07.753 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:37 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:07.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-04-06T14:05:08.577 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:41 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:09.439 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:09.491 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:09.492 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-06T14:05:09.492 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:09.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librbd1 2024-04-06T14:05:10.203 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:10.204 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:10.204 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:05:10.205 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:10.205 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:05:10.205 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:17.2.7-819.g6699495c.el8 ceph-noarch 123 k 2024-04-06T14:05:10.205 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T14:05:10.205 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-06T14:05:10.205 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-06T14:05:10.206 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-06T14:05:10.206 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-06T14:05:10.206 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:10.206 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:05:10.206 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:10.206 INFO:teuthology.orchestra.run.smithi086.stdout:Install 5 Packages 2024-04-06T14:05:10.206 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:10.207 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.0 M 2024-04-06T14:05:10.208 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-04-06T14:05:10.208 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:05:10.420 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:40 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:10.591 INFO:teuthology.orchestra.run.smithi086.stdout:(1/5): ceph-mgr-cephadm-17.2.7-819.g6699495c.el 412 kB/s | 123 kB 00:00 2024-04-06T14:05:10.883 INFO:teuthology.orchestra.run.smithi086.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 979 kB/s | 578 kB 00:00 2024-04-06T14:05:11.174 INFO:teuthology.orchestra.run.smithi086.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 815 kB/s | 476 kB 00:00 2024-04-06T14:05:11.216 INFO:teuthology.orchestra.run.smithi086.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 180 kB/s | 60 kB 00:00 2024-04-06T14:05:11.533 INFO:teuthology.orchestra.run.smithi086.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 3.9 MB/s | 4.8 MB 00:01 2024-04-06T14:05:11.534 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:11.535 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.5 MB/s | 6.0 MB 00:01 2024-04-06T14:05:11.589 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:05:11.600 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:05:11.600 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:05:11.701 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:05:11.704 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:05:11.966 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:05:12.127 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:12.179 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:12.180 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-06T14:05:12.180 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:12.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rados 2024-04-06T14:05:12.330 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-06T14:05:13.080 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:43 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:13.256 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-06T14:05:13.629 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-06T14:05:13.862 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-06T14:05:14.076 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-819.g6699495c.el8.noarch 5/5 2024-04-06T14:05:14.345 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-819.g6699495c.el8.noarch 5/5 2024-04-06T14:05:14.345 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-819.g6699495c.el8.noarch 1/5 2024-04-06T14:05:14.345 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-06T14:05:14.345 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-06T14:05:14.346 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-06T14:05:14.735 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-06T14:05:14.735 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:14.735 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:05:14.736 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:17.2.7-819.g6699495c.el8.noarch 2024-04-06T14:05:14.736 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-06T14:05:14.736 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-06T14:05:14.736 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-06T14:05:14.736 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-06T14:05:14.736 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:14.737 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:14.775 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rados-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:14.829 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:14.830 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-06T14:05:14.830 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:14.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-04-06T14:05:14.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rgw 2024-04-06T14:05:15.607 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:48 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:15.736 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:45 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:17.237 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:17.2.7-819.g6699495c.el8 ceph 876 k 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-04-06T14:05:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:17.239 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 960 k 2024-04-06T14:05:17.239 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 3.1 M 2024-04-06T14:05:17.239 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:05:17.415 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rgw-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:17.466 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:17.467 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-06T14:05:17.467 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:17.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-cephfs 2024-04-06T14:05:17.674 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): ceph-fuse-17.2.7-819.g6699495c.el8.x86_6 3.4 MB/s | 876 kB 00:00 2024-04-06T14:05:17.725 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 275 kB/s | 83 kB 00:00 2024-04-06T14:05:17.726 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:17.726 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.9 MB/s | 960 kB 00:00 2024-04-06T14:05:17.736 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:05:17.757 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:05:17.757 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:05:17.843 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:05:17.846 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:05:18.042 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:05:18.303 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-06T14:05:18.361 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:48 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:18.511 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:17.2.7-819.g6699495c.el8.x86_64 2/2 2024-04-06T14:05:18.913 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:17.2.7-819.g6699495c.el8.x86_64 2/2 2024-04-06T14:05:18.913 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:17.2.7-819.g6699495c.el8.x86_64 1/2 2024-04-06T14:05:19.313 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-06T14:05:19.313 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:19.313 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:05:19.313 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:17.2.7-819.g6699495c.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-06T14:05:19.313 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:19.314 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:19.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-04-06T14:05:20.053 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-cephfs-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:20.106 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:20.107 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-06T14:05:20.107 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:20.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rbd 2024-04-06T14:05:20.289 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:53 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:21.013 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:51 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:21.912 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:21.912 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:21.913 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-04-06T14:05:21.913 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:21.913 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:05:21.913 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:17.2.7-819.g6699495c.el8 ceph 130 k 2024-04-06T14:05:21.913 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:21.913 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:05:21.913 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:21.913 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-06T14:05:21.913 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:21.914 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 130 k 2024-04-06T14:05:21.914 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 464 k 2024-04-06T14:05:21.914 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:05:22.204 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-17.2.7-819.g6699495c.el8.x86_64. 447 kB/s | 130 kB 00:00 2024-04-06T14:05:22.205 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:22.205 INFO:teuthology.orchestra.run.smithi086.stdout:Total 445 kB/s | 130 kB 00:00 2024-04-06T14:05:22.205 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:05:22.211 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:05:22.211 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:05:22.260 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:05:22.263 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:05:22.368 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:05:22.607 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:22.724 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rbd-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:22.775 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:22.777 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-06T14:05:22.777 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:22.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-fuse 2024-04-06T14:05:22.896 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:23.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:23.303 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:23.303 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:05:23.303 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:23.303 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:23.303 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:23.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-04-06T14:05:23.683 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:53 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:24.278 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:57 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:25.450 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:25.451 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:25.451 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-06T14:05:25.451 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:25.451 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:05:25.451 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:17.2.7-819.g6699495c.el8 ceph 91 k 2024-04-06T14:05:25.451 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:25.451 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:05:25.452 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:25.452 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-06T14:05:25.452 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:25.452 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 91 k 2024-04-06T14:05:25.452 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 268 k 2024-04-06T14:05:25.452 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:05:25.623 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-fuse-17.2.7-819.g6699495c.el8.x86_64.rpm 531 kB/s | 91 kB 00:00 2024-04-06T14:05:25.624 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:25.624 INFO:teuthology.orchestra.run.smithi032.stdout:Total 527 kB/s | 91 kB 00:00 2024-04-06T14:05:25.624 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:05:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:05:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:05:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:05:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:05:25.794 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:05:25.848 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:25.895 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:25.896 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-06T14:05:25.897 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:26.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-04-06T14:05:26.035 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:26.348 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-fuse-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:26.752 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:59 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:26.763 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:05:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:26.763 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:27.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-mirror 2024-04-06T14:05:27.804 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:57 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:28.386 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:17.2.7-819.g6699495c.el8 ceph 25 k 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 25 k 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 109 k 2024-04-06T14:05:28.387 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:05:28.527 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-17.2.7-819.g6699495c.el8.x86_64 178 kB/s | 25 kB 00:00 2024-04-06T14:05:28.528 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:28.528 INFO:teuthology.orchestra.run.smithi086.stdout:Total 176 kB/s | 25 kB 00:00 2024-04-06T14:05:28.528 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:05:28.532 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:05:28.532 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:05:28.534 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:05:28.537 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:05:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:05:28.790 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:29.393 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:29.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:29.393 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:05:29.393 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:29.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:29.393 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:29.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-04-06T14:05:29.557 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:29.558 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:29.558 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-06T14:05:29.558 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:29.558 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:05:29.558 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:17.2.7-819.g6699495c.el8 ceph 3.5 M 2024-04-06T14:05:29.559 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:29.559 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:05:29.559 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:29.559 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-06T14:05:29.559 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:29.559 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 3.5 M 2024-04-06T14:05:29.559 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 16 M 2024-04-06T14:05:29.559 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:05:30.076 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-mirror-17.2.7-819.g6699495c.el8.x86_64.rpm 6.7 MB/s | 3.5 MB 00:00 2024-04-06T14:05:30.077 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:30.077 INFO:teuthology.orchestra.run.smithi032.stdout:Total 6.7 MB/s | 3.5 MB 00:00 2024-04-06T14:05:30.077 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:05:30.090 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:05:30.090 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:05:30.160 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:05:30.163 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:05:30.256 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:03 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:30.625 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:05:30.852 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:31.282 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:31.660 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:31.660 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:31.660 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:05:31.660 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:31.660 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:31.660 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:31.805 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:31.852 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:31.854 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-06T14:05:31.854 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:31.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd 2024-04-06T14:05:31.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-04-06T14:05:32.701 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:03:02 ago on Sat 06 Apr 2024 02:02:30 PM UTC. 2024-04-06T14:05:32.709 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:05 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:34.291 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:34.339 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:34.340 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-06T14:05:34.340 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:34.442 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-06T14:05:34.442 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-06T14:05:34.443 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:34.443 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-06T14:05:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:17.2.7-819.g6699495c.el8 ceph 175 k 2024-04-06T14:05:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:34.443 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-06T14:05:34.443 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-06T14:05:34.443 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-06T14:05:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:34.444 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 175 k 2024-04-06T14:05:34.444 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 556 k 2024-04-06T14:05:34.444 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-06T14:05:34.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-04-06T14:05:34.745 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-nbd-17.2.7-819.g6699495c.el8.x86_64.rpm 580 kB/s | 175 kB 00:00 2024-04-06T14:05:34.746 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:34.746 INFO:teuthology.orchestra.run.smithi032.stdout:Total 578 kB/s | 175 kB 00:00 2024-04-06T14:05:34.746 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-06T14:05:34.758 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-06T14:05:34.758 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-06T14:05:34.813 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-06T14:05:34.816 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-06T14:05:34.940 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-06T14:05:35.184 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:35.222 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:08 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:35.471 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-nbd-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:35.894 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-06T14:05:35.895 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:35.895 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:05:35.895 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-06T14:05:36.111 DEBUG:teuthology.parallel:result is None 2024-04-06T14:05:36.801 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:36.848 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:36.850 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-06T14:05:36.850 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:36.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-04-06T14:05:37.709 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:10 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:39.382 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:39.429 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:39.430 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-06T14:05:39.430 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:39.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-04-06T14:05:40.288 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:13 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:41.859 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:41.907 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:41.908 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-06T14:05:41.908 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:42.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-04-06T14:05:42.772 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:15 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:44.351 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:17.2.7-819.g6699495c.el8.x86_64 is already installed. 2024-04-06T14:05:44.399 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:44.401 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-06T14:05:44.401 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:44.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-04-06T14:05:45.266 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:18 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:46.907 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:46.908 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:46.908 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-04-06T14:05:46.908 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:46.908 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:05:46.908 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:17.2.7-819.g6699495c.el8 ceph 91 k 2024-04-06T14:05:46.908 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:46.908 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:05:46.908 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:46.908 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-06T14:05:46.909 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:46.909 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 91 k 2024-04-06T14:05:46.909 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 268 k 2024-04-06T14:05:46.909 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:05:47.075 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-17.2.7-819.g6699495c.el8.x86_64.rpm 545 kB/s | 91 kB 00:00 2024-04-06T14:05:47.076 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:47.076 INFO:teuthology.orchestra.run.smithi086.stdout:Total 541 kB/s | 91 kB 00:00 2024-04-06T14:05:47.076 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:05:47.086 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:05:47.086 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:05:47.137 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:05:47.139 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:05:47.244 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:05:47.421 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:47.711 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:48.073 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:48.074 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:48.074 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:05:48.074 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:48.074 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:48.074 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:48.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-04-06T14:05:49.084 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:22 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:50.698 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:50.698 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:50.699 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-04-06T14:05:50.699 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:50.699 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:05:50.699 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:17.2.7-819.g6699495c.el8 ceph 3.5 M 2024-04-06T14:05:50.699 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:50.699 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:05:50.699 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:50.699 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-06T14:05:50.699 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:50.700 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.5 M 2024-04-06T14:05:50.700 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 16 M 2024-04-06T14:05:50.700 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:05:51.229 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-17.2.7-819.g6699495c.el8.x86_64.rpm 6.5 MB/s | 3.5 MB 00:00 2024-04-06T14:05:51.230 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:51.230 INFO:teuthology.orchestra.run.smithi086.stdout:Total 6.5 MB/s | 3.5 MB 00:00 2024-04-06T14:05:51.230 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:05:51.241 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:05:51.241 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:05:51.304 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:05:51.307 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:05:51.750 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:05:51.930 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:52.345 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:52.721 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:52.722 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:52.722 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:05:52.722 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:52.722 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:52.722 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:52.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-04-06T14:05:53.742 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:26 ago on Sat 06 Apr 2024 02:02:27 PM UTC. 2024-04-06T14:05:55.364 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-06T14:05:55.365 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:55.365 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-04-06T14:05:55.365 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:55.365 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-06T14:05:55.365 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:17.2.7-819.g6699495c.el8 ceph 175 k 2024-04-06T14:05:55.365 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:55.365 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-06T14:05:55.365 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-06T14:05:55.365 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-06T14:05:55.366 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:55.366 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 175 k 2024-04-06T14:05:55.366 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 556 k 2024-04-06T14:05:55.366 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-06T14:05:55.666 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-17.2.7-819.g6699495c.el8.x86_64.rpm 584 kB/s | 175 kB 00:00 2024-04-06T14:05:55.666 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:05:55.666 INFO:teuthology.orchestra.run.smithi086.stdout:Total 581 kB/s | 175 kB 00:00 2024-04-06T14:05:55.667 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-06T14:05:55.677 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-06T14:05:55.677 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-06T14:05:55.726 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-06T14:05:55.729 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-06T14:05:55.886 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-06T14:05:56.086 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:17.2.7-819.g6699495c.el8.x86_64 1/1 2024-04-06T14:05:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:56.712 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-06T14:05:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:17.2.7-819.g6699495c.el8.x86_64 2024-04-06T14:05:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:05:56.713 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-06T14:05:56.940 DEBUG:teuthology.parallel:result is None 2024-04-06T14:05:56.941 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T14:05:56.941 INFO:teuthology.packaging:ref: None 2024-04-06T14:05:56.941 INFO:teuthology.packaging:tag: None 2024-04-06T14:05:56.941 INFO:teuthology.packaging:branch: quincy 2024-04-06T14:05:56.941 INFO:teuthology.packaging:sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:05:56.941 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-04-06T14:05:57.179 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-06T14:05:57.208 INFO:teuthology.orchestra.run.smithi032.stdout:17.2.7-819.g6699495c.el8 2024-04-06T14:05:57.209 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-819.g6699495c.el8 2024-04-06T14:05:57.209 INFO:teuthology.task.install:The correct ceph version 17.2.7-819.g6699495c is installed. 2024-04-06T14:05:57.211 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T14:05:57.211 INFO:teuthology.packaging:ref: None 2024-04-06T14:05:57.212 INFO:teuthology.packaging:tag: None 2024-04-06T14:05:57.212 INFO:teuthology.packaging:branch: quincy 2024-04-06T14:05:57.212 INFO:teuthology.packaging:sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:05:57.212 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-04-06T14:05:57.436 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-06T14:05:57.459 INFO:teuthology.orchestra.run.smithi086.stdout:17.2.7-819.g6699495c.el8 2024-04-06T14:05:57.459 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-819.g6699495c.el8 2024-04-06T14:05:57.459 INFO:teuthology.task.install:The correct ceph version 17.2.7-819.g6699495c is installed. 2024-04-06T14:05:57.461 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-06T14:05:57.461 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:05:57.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T14:05:57.496 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:05:57.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T14:05:57.528 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-06T14:05:57.528 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:05:57.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T14:05:57.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T14:05:57.640 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:05:57.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T14:05:57.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T14:05:57.733 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-06T14:05:57.733 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:05:57.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T14:05:57.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T14:05:57.835 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:05:57.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T14:05:57.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T14:05:57.928 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-06T14:05:57.928 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:05:57.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T14:05:57.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T14:05:58.029 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:05:58.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T14:05:58.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T14:05:58.122 INFO:teuthology.run_tasks:Running task print... 2024-04-06T14:05:58.131 INFO:teuthology.task.print:**** done install task... 2024-04-06T14:05:58.131 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-06T14:05:58.268 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'} 2024-04-06T14:05:58.269 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-04-06T14:05:58.269 INFO:tasks.cephadm:Cluster fsid is cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:05:58.269 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-06T14:05:58.269 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-06T14:05:58.269 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi032': '172.21.15.32', 'mon.smithi086': '172.21.15.86'} 2024-04-06T14:05:58.270 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-06T14:05:58.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2024-04-06T14:05:58.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-04-06T14:05:58.330 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-04-06T14:05:58.330 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T14:05:58.608 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-rw-r--. 1 ubuntu ubuntu 375153 Apr 6 14:05 /home/ubuntu/cephtest/cephadm 2024-04-06T14:05:58.609 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T14:05:58.828 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-rw-r--. 1 ubuntu ubuntu 375153 Apr 6 14:05 /home/ubuntu/cephtest/cephadm 2024-04-06T14:05:58.828 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T14:05:58.852 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T14:05:58.881 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-04-06T14:05:58.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-04-06T14:05:58.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-04-06T14:06:00.008 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-06T14:06:00.110 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-06T14:06:29.336 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:06:29.337 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)", 2024-04-06T14:06:29.337 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "ec300eaa768b18bd6cc8d1b248efb446cbfceeec0acea282e973a2c7c0cc62b7", 2024-04-06T14:06:29.337 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2024-04-06T14:06:29.337 INFO:teuthology.orchestra.run.smithi032.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:d049538dc20db76390a0232d25def63f9e4809620d469762c761112812aa14b8" 2024-04-06T14:06:29.337 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-06T14:06:29.337 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:06:31.219 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-06T14:06:31.219 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)", 2024-04-06T14:06:31.220 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "ec300eaa768b18bd6cc8d1b248efb446cbfceeec0acea282e973a2c7c0cc62b7", 2024-04-06T14:06:31.220 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-04-06T14:06:31.220 INFO:teuthology.orchestra.run.smithi086.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:d049538dc20db76390a0232d25def63f9e4809620d469762c761112812aa14b8" 2024-04-06T14:06:31.220 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-06T14:06:31.220 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-06T14:06:31.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2024-04-06T14:06:31.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-04-06T14:06:31.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2024-04-06T14:06:31.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-04-06T14:06:31.374 INFO:tasks.cephadm:Writing seed config... 2024-04-06T14:06:31.375 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-06T14:06:31.375 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-06T14:06:31.375 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-06T14:06:31.375 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-06T14:06:31.375 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-06T14:06:31.376 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-06T14:06:31.377 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-06T14:06:31.378 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-06T14:06:31.378 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-06T14:06:31.378 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-06T14:06:31.378 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-06T14:06:31.378 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-06T14:06:31.378 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:06:31.379 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-06T14:06:31.407 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = cabdc720-f41e-11ee-b649-cb9ed24678a4 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-06T14:06:31.407 DEBUG:teuthology.orchestra.run.smithi032:mon.smithi032> sudo journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi032.service 2024-04-06T14:06:31.449 INFO:tasks.cephadm:Bootstrapping... 2024-04-06T14:06:31.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.32 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-06T14:06:31.478 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:-- Logs begin at Sat 2024-04-06 13:55:15 UTC. -- 2024-04-06T14:06:31.664 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-06T14:06:31.664 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'cabdc720-f41e-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.32', '--skip-admin-label'] 2024-04-06T14:06:31.698 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-06T14:06:31.698 INFO:teuthology.orchestra.run.smithi032.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-06T14:06:31.698 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying podman|docker is present... 2024-04-06T14:06:31.730 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-06T14:06:31.730 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying lvm2 is present... 2024-04-06T14:06:31.730 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying time synchronization is in place... 2024-04-06T14:06:31.737 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-06T14:06:31.737 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T14:06:31.743 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-06T14:06:31.743 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-06T14:06:31.749 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-06T14:06:31.755 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-06T14:06:31.755 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-04-06T14:06:31.755 INFO:teuthology.orchestra.run.smithi032.stdout:Repeating the final host check... 2024-04-06T14:06:31.790 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-06T14:06:31.790 INFO:teuthology.orchestra.run.smithi032.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-06T14:06:31.790 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl is present 2024-04-06T14:06:31.790 INFO:teuthology.orchestra.run.smithi032.stdout:lvcreate is present 2024-04-06T14:06:31.796 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-06T14:06:31.796 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T14:06:31.802 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-06T14:06:31.802 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-06T14:06:31.808 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-06T14:06:31.814 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-06T14:06:31.814 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-04-06T14:06:31.814 INFO:teuthology.orchestra.run.smithi032.stdout:Host looks OK 2024-04-06T14:06:31.814 INFO:teuthology.orchestra.run.smithi032.stdout:Cluster fsid: cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:06:31.814 INFO:teuthology.orchestra.run.smithi032.stdout:Acquiring lock 140623379874704 on /run/cephadm/cabdc720-f41e-11ee-b649-cb9ed24678a4.lock 2024-04-06T14:06:31.814 INFO:teuthology.orchestra.run.smithi032.stdout:Lock 140623379874704 acquired on /run/cephadm/cabdc720-f41e-11ee-b649-cb9ed24678a4.lock 2024-04-06T14:06:31.815 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 3300 ... 2024-04-06T14:06:31.815 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 6789 ... 2024-04-06T14:06:31.815 INFO:teuthology.orchestra.run.smithi032.stdout:Base mon IP(s) is [172.21.15.32:3300, 172.21.15.32:6789], mon addrv is [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-04-06T14:06:31.819 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.32 metric 100 2024-04-06T14:06:31.819 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.32 metric 100 2024-04-06T14:06:31.822 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-06T14:06:31.822 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-04-06T14:06:31.822 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1783sec hoplimit 64 pref medium 2024-04-06T14:06:31.825 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-06T14:06:31.825 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-06T14:06:31.825 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-06T14:06:31.825 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-06T14:06:31.825 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2602/64 scope link 2024-04-06T14:06:31.825 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-06T14:06:31.825 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-04-06T14:06:31.826 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-04-06T14:06:31.826 INFO:teuthology.orchestra.run.smithi032.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-06T14:06:31.826 INFO:teuthology.orchestra.run.smithi032.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-06T14:06:31.827 INFO:teuthology.orchestra.run.smithi032.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-06T14:06:35.204 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout ec300eaa768b18bd6cc8d1b248efb446cbfceeec0acea282e973a2c7c0cc62b7 2024-04-06T14:06:35.204 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-06T14:06:35.204 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Getting image source signatures 2024-04-06T14:06:35.204 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:b91f190a3ce1b82447b54a8039ed3c8de0d076fc686a09d670cfcd1b7134f590 2024-04-06T14:06:35.204 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:77070339c81bcb5c53bbbf3f31a47f7187e5393cb0e096452852a31192d97570 2024-04-06T14:06:35.204 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying config sha256:ec300eaa768b18bd6cc8d1b248efb446cbfceeec0acea282e973a2c7c0cc62b7 2024-04-06T14:06:35.204 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-06T14:06:40.705 INFO:teuthology.orchestra.run.smithi032.stdout:ceph: stdout ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable) 2024-04-06T14:06:40.705 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph version: ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable) 2024-04-06T14:06:40.705 INFO:teuthology.orchestra.run.smithi032.stdout:Extracting ceph user uid/gid from container image... 2024-04-06T14:06:41.744 INFO:teuthology.orchestra.run.smithi032.stdout:stat: stdout 167 167 2024-04-06T14:06:41.744 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial keys... 2024-04-06T14:06:42.918 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQDyVhFmdUJdDBAAnTgAUPWVv+wqbwq0lcAXHw== 2024-04-06T14:06:43.975 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQDzVhFmopsmExAAD/WejRw/57q1P+HiWilSOQ== 2024-04-06T14:06:45.034 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQD0VhFmVNIQGhAAeWIaUgPPaT+KYnsAwejstw== 2024-04-06T14:06:45.035 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial monmap... 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool for smithi032 [v2:172.21.15.32:3300,v1:172.21.15.32:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = octopus 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: set fsid to cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:06:46.174 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mon... 2024-04-06T14:06:47.281 INFO:teuthology.orchestra.run.smithi032.stdout:create mon.smithi032 on 2024-04-06T14:06:47.389 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-06T14:06:47.483 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-06T14:06:47.578 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4.target. 2024-04-06T14:06:47.579 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4.target. 2024-04-06T14:06:47.707 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi032 2024-04-06T14:06:47.707 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi032.service: Unit ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi032.service not loaded. 2024-04-06T14:06:47.803 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4.target.wants/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi032.service → /etc/systemd/system/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@.service. 2024-04-06T14:06:47.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:47 smithi032 systemd[1]: Starting Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:06:48.354 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-06T14:06:48.355 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-06T14:06:48.363 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-06T14:06:48.363 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-06T14:06:48.363 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-06T14:06:48.363 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-06T14:06:48.363 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon to start... 2024-04-06T14:06:48.363 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon... 2024-04-06T14:06:48.645 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:48 smithi032 ceph-mon[39247]: mkfs cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:06:48.646 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:48 smithi032 ceph-mon[39247]: mon.smithi032 is new leader, mons smithi032 in quorum (ranks 0) 2024-04-06T14:06:48.646 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:48 smithi032 bash[39210]: 7e5b4ad1ed7e830655ca3e0cd7f4be1a69be70e79d4c31c3ea26a1eccc02fd68 2024-04-06T14:06:48.646 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:48 smithi032 systemd[1]: Started Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:06:49.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout cluster: 2024-04-06T14:06:49.597 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout id: cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:06:49.597 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-06T14:06:49.597 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:06:49.597 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout services: 2024-04-06T14:06:49.597 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi032 (age 0.679735s) 2024-04-06T14:06:49.597 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout data: 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pgs: 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.929+0000 7f27837a0700 1 Processor -- start 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.930+0000 7f27837a0700 1 -- start start 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.930+0000 7f27837a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.930+0000 7f27837a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c102ed0 con 0x7f277c102580 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.930+0000 7f278153c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:49.598 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.930+0000 7f278153c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46968/0 (socket says 172.21.15.32:46968) 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.930+0000 7f278153c700 1 -- 172.21.15.32:0/78801874 learned_addr learned my addr 172.21.15.32:0/78801874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f278153c700 1 -- 172.21.15.32:0/78801874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277c103010 con 0x7f277c102580 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f278153c700 1 --2- 172.21.15.32:0/78801874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c102990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f27700098e0 tx=0x7f2770009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1cba9893cc02c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f276ffff700 1 -- 172.21.15.32:0/78801874 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f277000b860 con 0x7f277c102580 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f276ffff700 1 -- 172.21.15.32:0/78801874 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277000b9c0 con 0x7f277c102580 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f276ffff700 1 -- 172.21.15.32:0/78801874 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f277000bc90 con 0x7f277c102580 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f27837a0700 1 -- 172.21.15.32:0/78801874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 msgr2=0x7f277c102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f27837a0700 1 --2- 172.21.15.32:0/78801874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c102990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f27700098e0 tx=0x7f2770009250 comp rx=0 tx=0).stop 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f27837a0700 1 -- 172.21.15.32:0/78801874 shutdown_connections 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f27837a0700 1 --2- 172.21.15.32:0/78801874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c102990 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.931+0000 7f27837a0700 1 -- 172.21.15.32:0/78801874 >> 172.21.15.32:0/78801874 conn(0x7f277c0fdd00 msgr2=0x7f277c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.932+0000 7f27837a0700 1 -- 172.21.15.32:0/78801874 shutdown_connections 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.932+0000 7f27837a0700 1 -- 172.21.15.32:0/78801874 wait complete. 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.932+0000 7f27837a0700 1 Processor -- start 2024-04-06T14:06:49.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.932+0000 7f27837a0700 1 -- start start 2024-04-06T14:06:49.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.932+0000 7f27837a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:49.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.932+0000 7f27837a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c111110 con 0x7f277c102580 2024-04-06T14:06:49.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.932+0000 7f278153c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:49.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.932+0000 7f278153c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46982/0 (socket says 172.21.15.32:46982) 2024-04-06T14:06:49.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.932+0000 7f278153c700 1 -- 172.21.15.32:0/1798064857 learned_addr learned my addr 172.21.15.32:0/1798064857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:49.600 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.933+0000 7f278153c700 1 -- 172.21.15.32:0/1798064857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27700094f0 con 0x7f277c102580 2024-04-06T14:06:49.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:49 smithi032 ceph-mon[39247]: mon.smithi032 is new leader, mons smithi032 in quorum (ranks 0) 2024-04-06T14:06:49.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:49 smithi032 ceph-mon[39247]: monmap e1: 1 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T14:06:49.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:49 smithi032 ceph-mon[39247]: fsmap 2024-04-06T14:06:49.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:49 smithi032 ceph-mon[39247]: osdmap e1: 0 total, 0 up, 0 in 2024-04-06T14:06:49.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:49 smithi032 ceph-mon[39247]: mgrmap e1: no daemons active 2024-04-06T14:06:49.601 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:49 smithi032 ceph-mon[39247]: from='client.? 172.21.15.32:0/1798064857' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.933+0000 7f278153c700 1 --2- 172.21.15.32:0/1798064857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c110bd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2770003080 tx=0x7f2770016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.933+0000 7f276dffb700 1 -- 172.21.15.32:0/1798064857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f277001c5c0 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.933+0000 7f276dffb700 1 -- 172.21.15.32:0/1798064857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277001c720 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.933+0000 7f276dffb700 1 -- 172.21.15.32:0/1798064857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f277001ca30 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.933+0000 7f27837a0700 1 -- 172.21.15.32:0/1798064857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277c111310 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.933+0000 7f27837a0700 1 -- 172.21.15.32:0/1798064857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277c111e80 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.933+0000 7f276dffb700 1 -- 172.21.15.32:0/1798064857 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f277001a450 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.933+0000 7f276dffb700 1 -- 172.21.15.32:0/1798064857 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f277002e790 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.934+0000 7f27837a0700 1 -- 172.21.15.32:0/1798064857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277c05f5f0 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:48.936+0000 7f276dffb700 1 -- 172.21.15.32:0/1798064857 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f277001a890 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:49.008+0000 7f27837a0700 1 -- 172.21.15.32:0/1798064857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f277c112890 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:49.008+0000 7f276dffb700 1 -- 172.21.15.32:0/1798064857 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f277001aa30 con 0x7f277c102580 2024-04-06T14:06:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:49.009+0000 7f27837a0700 1 -- 172.21.15.32:0/1798064857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 msgr2=0x7f277c110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:49.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:49.009+0000 7f27837a0700 1 --2- 172.21.15.32:0/1798064857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c110bd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2770003080 tx=0x7f2770016000 comp rx=0 tx=0).stop 2024-04-06T14:06:49.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:49.009+0000 7f27837a0700 1 -- 172.21.15.32:0/1798064857 shutdown_connections 2024-04-06T14:06:49.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:49.009+0000 7f27837a0700 1 --2- 172.21.15.32:0/1798064857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f277c102580 0x7f277c110bd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:49.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:49.009+0000 7f27837a0700 1 -- 172.21.15.32:0/1798064857 >> 172.21.15.32:0/1798064857 conn(0x7f277c0fdd00 msgr2=0x7f277c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:49.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:49.009+0000 7f27837a0700 1 -- 172.21.15.32:0/1798064857 shutdown_connections 2024-04-06T14:06:49.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:49.009+0000 7f27837a0700 1 -- 172.21.15.32:0/1798064857 wait complete. 2024-04-06T14:06:49.604 INFO:teuthology.orchestra.run.smithi032.stdout:mon is available 2024-04-06T14:06:49.604 INFO:teuthology.orchestra.run.smithi032.stdout:Assimilating anything we can from ceph.conf... 2024-04-06T14:06:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:50 smithi032 ceph-mon[39247]: from='client.? 172.21.15.32:0/3251704336' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T14:06:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:50 smithi032 ceph-mon[39247]: from='client.? 172.21.15.32:0/3251704336' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-06T14:06:50.855 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:06:50.855 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-06T14:06:50.855 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:06:50.855 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-06T14:06:50.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.119+0000 7fdc3dc6b700 1 Processor -- start 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.119+0000 7fdc3dc6b700 1 -- start start 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc3dc6b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38105210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc3dc6b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38105750 con 0x7fdc38076c20 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38105210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38105210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46986/0 (socket says 172.21.15.32:46986) 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc377fe700 1 -- 172.21.15.32:0/1946649886 learned_addr learned my addr 172.21.15.32:0/1946649886 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc377fe700 1 -- 172.21.15.32:0/1946649886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc38105890 con 0x7fdc38076c20 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc377fe700 1 --2- 172.21.15.32:0/1946649886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38105210 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdc280098e0 tx=0x7fdc28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb253686c3cf192 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc367fc700 1 -- 172.21.15.32:0/1946649886 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc2800b860 con 0x7fdc38076c20 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc367fc700 1 -- 172.21.15.32:0/1946649886 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2800b9c0 con 0x7fdc38076c20 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.120+0000 7fdc367fc700 1 -- 172.21.15.32:0/1946649886 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc2800bc90 con 0x7fdc38076c20 2024-04-06T14:06:50.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.121+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/1946649886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 msgr2=0x7fdc38105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.121+0000 7fdc3dc6b700 1 --2- 172.21.15.32:0/1946649886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38105210 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdc280098e0 tx=0x7fdc28009250 comp rx=0 tx=0).stop 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.121+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/1946649886 shutdown_connections 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.121+0000 7fdc3dc6b700 1 --2- 172.21.15.32:0/1946649886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38105210 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.121+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/1946649886 >> 172.21.15.32:0/1946649886 conn(0x7fdc380fcc60 msgr2=0x7fdc380ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.121+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/1946649886 shutdown_connections 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.121+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/1946649886 wait complete. 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.122+0000 7fdc3dc6b700 1 Processor -- start 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.122+0000 7fdc3dc6b700 1 -- start start 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.122+0000 7fdc3dc6b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.122+0000 7fdc3dc6b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc381030a0 con 0x7fdc38076c20 2024-04-06T14:06:50.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.122+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.122+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38102b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46998/0 (socket says 172.21.15.32:46998) 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.122+0000 7fdc377fe700 1 -- 172.21.15.32:0/3251704336 learned_addr learned my addr 172.21.15.32:0/3251704336 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.123+0000 7fdc377fe700 1 -- 172.21.15.32:0/3251704336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc280094f0 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.123+0000 7fdc377fe700 1 --2- 172.21.15.32:0/3251704336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38102b60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009290 tx=0x7fdc28016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.123+0000 7fdc1ffff700 1 -- 172.21.15.32:0/3251704336 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc2801c5c0 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.123+0000 7fdc1ffff700 1 -- 172.21.15.32:0/3251704336 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2801c740 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.123+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/3251704336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc38101110 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.123+0000 7fdc1ffff700 1 -- 172.21.15.32:0/3251704336 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc2801ca50 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.123+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/3251704336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc38101600 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.123+0000 7fdc1ffff700 1 -- 172.21.15.32:0/3251704336 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fdc2801a450 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.123+0000 7fdc1ffff700 1 -- 172.21.15.32:0/3251704336 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdc2802e790 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.124+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/3251704336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3804bed0 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.126+0000 7fdc1ffff700 1 -- 172.21.15.32:0/3251704336 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fdc2801ce30 con 0x7fdc38076c20 2024-04-06T14:06:50.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.196+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/3251704336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdc38105b40 con 0x7fdc38076c20 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.199+0000 7fdc1ffff700 1 -- 172.21.15.32:0/3251704336 <== mon.0 v2:172.21.15.32:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdc280243e0 con 0x7fdc38076c20 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.199+0000 7fdc1ffff700 1 -- 172.21.15.32:0/3251704336 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fdc2802e9f0 con 0x7fdc38076c20 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.200+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/3251704336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 msgr2=0x7fdc38102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.200+0000 7fdc3dc6b700 1 --2- 172.21.15.32:0/3251704336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38102b60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009290 tx=0x7fdc28016000 comp rx=0 tx=0).stop 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.200+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/3251704336 shutdown_connections 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.200+0000 7fdc3dc6b700 1 --2- 172.21.15.32:0/3251704336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc38076c20 0x7fdc38102b60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.200+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/3251704336 >> 172.21.15.32:0/3251704336 conn(0x7fdc380fcc60 msgr2=0x7fdc380fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.200+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/3251704336 shutdown_connections 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:50.200+0000 7fdc3dc6b700 1 -- 172.21.15.32:0/3251704336 wait complete. 2024-04-06T14:06:50.860 INFO:teuthology.orchestra.run.smithi032.stdout:Generating new minimal ceph.conf... 2024-04-06T14:06:51.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:51 smithi032 ceph-mon[39247]: from='client.? 172.21.15.32:0/2339790454' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:06:52.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.348+0000 7f2eceaa2700 1 Processor -- start 2024-04-06T14:06:52.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.348+0000 7f2eceaa2700 1 -- start start 2024-04-06T14:06:52.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.348+0000 7f2eceaa2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec81051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:52.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.348+0000 7f2eceaa2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec8105730 con 0x7f2ec8076c40 2024-04-06T14:06:52.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.348+0000 7f2ecc83e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec81051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:52.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.348+0000 7f2ecc83e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec81051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47000/0 (socket says 172.21.15.32:47000) 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.348+0000 7f2ecc83e700 1 -- 172.21.15.32:0/3856312630 learned_addr learned my addr 172.21.15.32:0/3856312630 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.349+0000 7f2ecc83e700 1 -- 172.21.15.32:0/3856312630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec8105870 con 0x7f2ec8076c40 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.349+0000 7f2ecc83e700 1 --2- 172.21.15.32:0/3856312630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec81051f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0098e0 tx=0x7f2ebc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7f18450429ab16 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.349+0000 7f2ec77fe700 1 -- 172.21.15.32:0/3856312630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ebc00b860 con 0x7f2ec8076c40 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.349+0000 7f2ec77fe700 1 -- 172.21.15.32:0/3856312630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2ebc00be30 con 0x7f2ec8076c40 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.349+0000 7f2ec77fe700 1 -- 172.21.15.32:0/3856312630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ebc010380 con 0x7f2ec8076c40 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.350+0000 7f2eceaa2700 1 -- 172.21.15.32:0/3856312630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 msgr2=0x7f2ec81051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.350+0000 7f2eceaa2700 1 --2- 172.21.15.32:0/3856312630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec81051f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0098e0 tx=0x7f2ebc009250 comp rx=0 tx=0).stop 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.350+0000 7f2eceaa2700 1 -- 172.21.15.32:0/3856312630 shutdown_connections 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.350+0000 7f2eceaa2700 1 --2- 172.21.15.32:0/3856312630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec81051f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.350+0000 7f2eceaa2700 1 -- 172.21.15.32:0/3856312630 >> 172.21.15.32:0/3856312630 conn(0x7f2ec80fcc60 msgr2=0x7f2ec80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:52.102 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.350+0000 7f2eceaa2700 1 -- 172.21.15.32:0/3856312630 shutdown_connections 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.350+0000 7f2eceaa2700 1 -- 172.21.15.32:0/3856312630 wait complete. 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.350+0000 7f2eceaa2700 1 Processor -- start 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.350+0000 7f2eceaa2700 1 -- start start 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2eceaa2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec8111320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2eceaa2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec8111860 con 0x7f2ec8076c40 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2ecc83e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec8111320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2ecc83e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec8111320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47016/0 (socket says 172.21.15.32:47016) 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2ecc83e700 1 -- 172.21.15.32:0/2339790454 learned_addr learned my addr 172.21.15.32:0/2339790454 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2ecc83e700 1 -- 172.21.15.32:0/2339790454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0094f0 con 0x7f2ec8076c40 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2ecc83e700 1 --2- 172.21.15.32:0/2339790454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec8111320 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc006d40 tx=0x7f2ebc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2ec57fa700 1 -- 172.21.15.32:0/2339790454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ebc0109b0 con 0x7f2ec8076c40 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2ec57fa700 1 -- 172.21.15.32:0/2339790454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2ebc024440 con 0x7f2ec8076c40 2024-04-06T14:06:52.103 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2eceaa2700 1 -- 172.21.15.32:0/2339790454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec8111ac0 con 0x7f2ec8076c40 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2ec57fa700 1 -- 172.21.15.32:0/2339790454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ebc01a5f0 con 0x7f2ec8076c40 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.351+0000 7f2eceaa2700 1 -- 172.21.15.32:0/2339790454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec81125d0 con 0x7f2ec8076c40 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.352+0000 7f2ec57fa700 1 -- 172.21.15.32:0/2339790454 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2ebc01aaa0 con 0x7f2ec8076c40 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.352+0000 7f2ec57fa700 1 -- 172.21.15.32:0/2339790454 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2ebc02d9f0 con 0x7f2ec8076c40 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.352+0000 7f2eceaa2700 1 -- 172.21.15.32:0/2339790454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec805f5f0 con 0x7f2ec8076c40 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.355+0000 7f2ec57fa700 1 -- 172.21.15.32:0/2339790454 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2ebc024970 con 0x7f2ec8076c40 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.425+0000 7f2eceaa2700 1 -- 172.21.15.32:0/2339790454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2ec804bed0 con 0x7f2ec8076c40 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.425+0000 7f2ec57fa700 1 -- 172.21.15.32:0/2339790454 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f2ebc035090 con 0x7f2ec8076c40 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.426+0000 7f2eceaa2700 1 -- 172.21.15.32:0/2339790454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 msgr2=0x7f2ec8111320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.426+0000 7f2eceaa2700 1 --2- 172.21.15.32:0/2339790454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec8111320 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc006d40 tx=0x7f2ebc009250 comp rx=0 tx=0).stop 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.426+0000 7f2eceaa2700 1 -- 172.21.15.32:0/2339790454 shutdown_connections 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.426+0000 7f2eceaa2700 1 --2- 172.21.15.32:0/2339790454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ec8076c40 0x7f2ec8111320 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.426+0000 7f2eceaa2700 1 -- 172.21.15.32:0/2339790454 >> 172.21.15.32:0/2339790454 conn(0x7f2ec80fcc60 msgr2=0x7f2ec80fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:52.104 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.426+0000 7f2eceaa2700 1 -- 172.21.15.32:0/2339790454 shutdown_connections 2024-04-06T14:06:52.105 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:51.426+0000 7f2eceaa2700 1 -- 172.21.15.32:0/2339790454 wait complete. 2024-04-06T14:06:52.105 INFO:teuthology.orchestra.run.smithi032.stdout:Restarting the monitor... 2024-04-06T14:06:52.370 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:52 smithi032 systemd[1]: Stopping Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:06:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:52 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[39224]: 2024-04-06T14:06:52.369+0000 7f2c4ceb9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi032 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T14:06:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:52 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[39224]: 2024-04-06T14:06:52.369+0000 7f2c4ceb9700 -1 mon.smithi032@0(leader) e1 *** Got Signal Terminated *** 2024-04-06T14:06:53.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 bash[39572]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032 2024-04-06T14:06:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi032.service: Succeeded. 2024-04-06T14:06:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 systemd[1]: Stopped Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:06:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 systemd[1]: Starting Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:06:53.971 INFO:teuthology.orchestra.run.smithi032.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable), process ceph-mon, pid 7 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: pidfile_write: ignore empty --pid-file 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: load: jerasure load: lrc 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: RocksDB version: 6.15.5 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Compile date Apr 5 2024 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: DB SUMMARY 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: DB Session ID: NH153Y3SPE103B7QM8RV 2024-04-06T14:06:54.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: CURRENT file: CURRENT 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi032/store.db dir, Total Num: 1, files: 000008.sst 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi032/store.db: 000010.log size: 96732 ; 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.error_if_exists: 0 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.create_if_missing: 0 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.paranoid_checks: 1 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.env: 0x564b35d9fa80 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.fs: Posix File System 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.info_log: 0x564b372b3c00 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.statistics: (nil) 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.use_fsync: 0 2024-04-06T14:06:54.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_log_file_size: 0 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.allow_fallocate: 1 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.use_direct_reads: 0 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.db_log_dir: 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi032/store.db 2024-04-06T14:06:54.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.write_buffer_manager: 0x564b380289f0 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T14:06:54.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.unordered_write: 0 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.row_cache: None 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.wal_filter: None 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.preserve_deletes: 0 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.two_write_queues: 0 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T14:06:54.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.atomic_flush: 0 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.log_readahead_size: 0 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_background_jobs: 2 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_background_compactions: -1 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_subcompactions: 1 2024-04-06T14:06:54.193 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_open_files: -1 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_background_flushes: -1 2024-04-06T14:06:54.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Compression algorithms supported: 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: kZSTD supported: 0 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: kXpressCompression supported: 0 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: kZlibCompression supported: 1 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: kSnappyCompression supported: 1 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: kLZ4Compression supported: 1 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: kBZip2Compression supported: 0 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000009 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.merge_operator: 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_filter: None 2024-04-06T14:06:54.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564b37308f58) 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks: 1 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_top_level_index_and_filter: 1 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_type: 0 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_index_type: 0 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_shortening: 1 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: hash_index_allow_collision: 1 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: checksum: 1 2024-04-06T14:06:54.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: no_block_cache: 0 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache: 0x564b3733b090 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_name: BinnedLRUCache 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_options: 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: capacity : 536870912 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: num_shard_bits : 4 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: strict_capacity_limit : 0 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: high_pri_pool_ratio: 0.000 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_compressed: (nil) 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: persistent_cache: (nil) 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size: 4096 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size_deviation: 10 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_restart_interval: 16 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_block_restart_interval: 1 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: metadata_block_size: 4096 2024-04-06T14:06:54.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: partition_filters: 0 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: use_delta_encoding: 1 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: whole_key_filtering: 1 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: verify_compression: 0 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: read_amp_bytes_per_bit: 0 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: format_version: 4 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_index_compression: 1 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_align: 0 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compression: NoCompression 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T14:06:54.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.num_levels: 7 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T14:06:54.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T14:06:54.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T14:06:54.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T14:06:54.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T14:06:54.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T14:06:54.200 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T14:06:54.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.arena_block_size: 4194304 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T14:06:54.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.table_properties_collectors: 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.inplace_update_support: 0 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.bloom_locality: 0 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.max_successive_merges: 0 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T14:06:54.203 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.ttl: 2592000 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.enable_blob_files: false 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.min_blob_size: 0 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712412413921481, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-04-06T14:06:54.204 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712412413922280, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92101, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90196, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 82935, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712412413, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "64bbb27e-29c9-44ab-a860-51947a052ba2", "db_session_id": "NH153Y3SPE103B7QM8RV"}} 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712412413922623, "job": 1, "event": "recovery_finished"} 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi032/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x564b372cf180 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: DB pointer 0x564b373a4000 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** DB Stats ** 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T14:06:54.205 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L0 2/0 91.74 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.7 0.00 0.00 1 0.000 0 0 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Sum 2/0 91.74 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.7 0.00 0.00 1 0.000 0 0 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.7 0.00 0.00 1 0.000 0 0 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 175.7 0.00 0.00 1 0.000 0 0 2024-04-06T14:06:54.206 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative compaction: 0.00 GB write, 28.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval compaction: 0.00 GB write, 28.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:06:54.207 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L0 2/0 91.74 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.7 0.00 0.00 1 0.000 0 0 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Sum 2/0 91.74 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.7 0.00 0.00 1 0.000 0 0 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 175.7 0.00 0.00 1 0.000 0 0 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative compaction: 0.00 GB write, 28.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:06:54.208 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: starting mon.smithi032 rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi032 fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mon.smithi032@-1(???) e1 preinit fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mon.smithi032@-1(???).mds e1 new map 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mon.smithi032@-1(???).mds e1 print_map 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: e1 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: legacy client fscid: -1 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: No filesystems configured 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mon.smithi032@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mon.smithi032@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T14:06:54.209 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mon.smithi032@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T14:06:54.210 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mon.smithi032@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T14:06:54.210 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mon.smithi032@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-06T14:06:54.210 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 bash[39692]: 71e43d3091dd983c943b2b52088036ba3f623a8011b8828b4ef7371e5435d21a 2024-04-06T14:06:54.210 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 systemd[1]: Started Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:06:54.210 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mon.smithi032 is new leader, mons smithi032 in quorum (ranks 0) 2024-04-06T14:06:54.210 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: monmap e1: 1 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T14:06:54.210 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: fsmap 2024-04-06T14:06:54.210 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: osdmap e1: 0 total, 0 up, 0 in 2024-04-06T14:06:54.210 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:53 smithi032 ceph-mon[39730]: mgrmap e1: no daemons active 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.522+0000 7fa4afa2b700 1 Processor -- start 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.522+0000 7fa4afa2b700 1 -- start start 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.522+0000 7fa4afa2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8105480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.522+0000 7fa4afa2b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8076d40 con 0x7fa4a81030a0 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.523+0000 7fa4ad7c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8105480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.523+0000 7fa4ad7c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8105480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39326/0 (socket says 172.21.15.32:39326) 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.523+0000 7fa4ad7c7700 1 -- 172.21.15.32:0/4119325343 learned_addr learned my addr 172.21.15.32:0/4119325343 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.523+0000 7fa4ad7c7700 1 -- 172.21.15.32:0/4119325343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a8076e80 con 0x7fa4a81030a0 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.523+0000 7fa4ad7c7700 1 --2- 172.21.15.32:0/4119325343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8105480 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40098e0 tx=0x7fa4a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d30f3ee7cfaf4a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.523+0000 7fa49bfff700 1 -- 172.21.15.32:0/4119325343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a400b860 con 0x7fa4a81030a0 2024-04-06T14:06:55.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.523+0000 7fa49bfff700 1 -- 172.21.15.32:0/4119325343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa4a400be30 con 0x7fa4a81030a0 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.523+0000 7fa49bfff700 1 -- 172.21.15.32:0/4119325343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a4010380 con 0x7fa4a81030a0 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.524+0000 7fa4afa2b700 1 -- 172.21.15.32:0/4119325343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 msgr2=0x7fa4a8105480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.524+0000 7fa4afa2b700 1 --2- 172.21.15.32:0/4119325343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8105480 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40098e0 tx=0x7fa4a4009250 comp rx=0 tx=0).stop 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.524+0000 7fa4afa2b700 1 -- 172.21.15.32:0/4119325343 shutdown_connections 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.524+0000 7fa4afa2b700 1 --2- 172.21.15.32:0/4119325343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8105480 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.524+0000 7fa4afa2b700 1 -- 172.21.15.32:0/4119325343 >> 172.21.15.32:0/4119325343 conn(0x7fa4a80fcb20 msgr2=0x7fa4a80fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.524+0000 7fa4afa2b700 1 -- 172.21.15.32:0/4119325343 shutdown_connections 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.524+0000 7fa4afa2b700 1 -- 172.21.15.32:0/4119325343 wait complete. 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.525+0000 7fa4afa2b700 1 Processor -- start 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.525+0000 7fa4afa2b700 1 -- start start 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.525+0000 7fa4afa2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8111340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.525+0000 7fa4afa2b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8111880 con 0x7fa4a81030a0 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.525+0000 7fa4ad7c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8111340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:55.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.525+0000 7fa4ad7c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8111340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39340/0 (socket says 172.21.15.32:39340) 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.525+0000 7fa4ad7c7700 1 -- 172.21.15.32:0/3772838432 learned_addr learned my addr 172.21.15.32:0/3772838432 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.525+0000 7fa4ad7c7700 1 -- 172.21.15.32:0/3772838432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40094f0 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.525+0000 7fa4ad7c7700 1 --2- 172.21.15.32:0/3772838432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8111340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4006d40 tx=0x7fa4a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.526+0000 7fa499ffb700 1 -- 172.21.15.32:0/3772838432 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a40109b0 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.526+0000 7fa499ffb700 1 -- 172.21.15.32:0/3772838432 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa4a4024440 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.526+0000 7fa499ffb700 1 -- 172.21.15.32:0/3772838432 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a401a5f0 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.526+0000 7fa4afa2b700 1 -- 172.21.15.32:0/3772838432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a8111a80 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.526+0000 7fa4afa2b700 1 -- 172.21.15.32:0/3772838432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a81125f0 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.527+0000 7fa499ffb700 1 -- 172.21.15.32:0/3772838432 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa4a4010d90 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.527+0000 7fa4afa2b700 1 -- 172.21.15.32:0/3772838432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a805f5f0 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.527+0000 7fa499ffb700 1 -- 172.21.15.32:0/3772838432 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa4a401ae40 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.529+0000 7fa499ffb700 1 -- 172.21.15.32:0/3772838432 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa4a4024970 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.597+0000 7fa4afa2b700 1 -- 172.21.15.32:0/3772838432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fa4a804bed0 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.600+0000 7fa499ffb700 1 -- 172.21.15.32:0/3772838432 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa4a401a820 con 0x7fa4a81030a0 2024-04-06T14:06:55.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.600+0000 7fa4afa2b700 1 -- 172.21.15.32:0/3772838432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 msgr2=0x7fa4a8111340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.600+0000 7fa4afa2b700 1 --2- 172.21.15.32:0/3772838432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8111340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4006d40 tx=0x7fa4a4009250 comp rx=0 tx=0).stop 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.601+0000 7fa4afa2b700 1 -- 172.21.15.32:0/3772838432 shutdown_connections 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.601+0000 7fa4afa2b700 1 --2- 172.21.15.32:0/3772838432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a81030a0 0x7fa4a8111340 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.601+0000 7fa4afa2b700 1 -- 172.21.15.32:0/3772838432 >> 172.21.15.32:0/3772838432 conn(0x7fa4a80fcb20 msgr2=0x7fa4a80fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.601+0000 7fa4afa2b700 1 -- 172.21.15.32:0/3772838432 shutdown_connections 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:54.601+0000 7fa4afa2b700 1 -- 172.21.15.32:0/3772838432 wait complete. 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mgr... 2024-04-06T14:06:55.200 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 9283 ... 2024-04-06T14:06:55.340 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mgr.smithi032.peohxu 2024-04-06T14:06:55.340 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mgr.smithi032.peohxu.service: Unit ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mgr.smithi032.peohxu.service not loaded. 2024-04-06T14:06:55.435 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4.target.wants/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mgr.smithi032.peohxu.service → /etc/systemd/system/ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@.service. 2024-04-06T14:06:55.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:55 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3772838432' entity='client.admin' 2024-04-06T14:06:56.929 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-06T14:06:56.929 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-06T14:06:56.939 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-06T14:06:56.939 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-06T14:06:56.939 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-06T14:06:56.939 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-06T14:06:56.948 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-06T14:06:56.948 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-06T14:06:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-06T14:06:56.957 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-06T14:06:56.957 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-06T14:06:56.957 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-06T14:06:56.957 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr to start... 2024-04-06T14:06:56.957 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr... 2024-04-06T14:06:57.978 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:06:57 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3105860850' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T14:06:58.301 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:06:58.301 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-06T14:06:58.301 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "cabdc720-f41e-11ee-b649-cb9ed24678a4", 2024-04-06T14:06:58.301 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T14:06:58.301 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "smithi032" 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-06T14:06:58.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T14:06:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:06:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T14:06:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:06:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T14:06:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T14:06:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T14:06:58.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T14:06:58.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T14:06:48.331052+0000", 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.549+0000 7f7c794fc700 1 Processor -- start 2024-04-06T14:06:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.549+0000 7f7c794fc700 1 -- start start 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.550+0000 7f7c794fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c740772a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.550+0000 7f7c794fc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740790b0 con 0x7f7c74078ca0 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.550+0000 7f7c72ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c740772a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.550+0000 7f7c72ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c740772a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39370/0 (socket says 172.21.15.32:39370) 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.550+0000 7f7c72ffd700 1 -- 172.21.15.32:0/3718486117 learned_addr learned my addr 172.21.15.32:0/3718486117 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.550+0000 7f7c72ffd700 1 -- 172.21.15.32:0/3718486117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c740777e0 con 0x7f7c74078ca0 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.551+0000 7f7c72ffd700 1 --2- 172.21.15.32:0/3718486117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c740772a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7c640098e0 tx=0x7f7c64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9841fc79c747d0fd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.551+0000 7f7c71ffb700 1 -- 172.21.15.32:0/3718486117 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c6400b860 con 0x7f7c74078ca0 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.551+0000 7f7c71ffb700 1 -- 172.21.15.32:0/3718486117 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f7c6400be30 con 0x7f7c74078ca0 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.551+0000 7f7c71ffb700 1 -- 172.21.15.32:0/3718486117 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c64010380 con 0x7f7c74078ca0 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.551+0000 7f7c794fc700 1 -- 172.21.15.32:0/3718486117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 msgr2=0x7f7c740772a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.551+0000 7f7c794fc700 1 --2- 172.21.15.32:0/3718486117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c740772a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7c640098e0 tx=0x7f7c64009250 comp rx=0 tx=0).stop 2024-04-06T14:06:58.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.551+0000 7f7c794fc700 1 -- 172.21.15.32:0/3718486117 shutdown_connections 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.551+0000 7f7c794fc700 1 --2- 172.21.15.32:0/3718486117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c740772a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.551+0000 7f7c794fc700 1 -- 172.21.15.32:0/3718486117 >> 172.21.15.32:0/3718486117 conn(0x7f7c740fdde0 msgr2=0x7f7c741001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.552+0000 7f7c794fc700 1 -- 172.21.15.32:0/3718486117 shutdown_connections 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.552+0000 7f7c794fc700 1 -- 172.21.15.32:0/3718486117 wait complete. 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.552+0000 7f7c794fc700 1 Processor -- start 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.552+0000 7f7c794fc700 1 -- start start 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.552+0000 7f7c794fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c74113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.552+0000 7f7c794fc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c74113b50 con 0x7f7c74078ca0 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c72ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c74113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c72ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c74113610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39386/0 (socket says 172.21.15.32:39386) 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c72ffd700 1 -- 172.21.15.32:0/3105860850 learned_addr learned my addr 172.21.15.32:0/3105860850 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c72ffd700 1 -- 172.21.15.32:0/3105860850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c640094f0 con 0x7f7c74078ca0 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c72ffd700 1 --2- 172.21.15.32:0/3105860850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c74113610 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7c64012fa0 tx=0x7f7c64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:06:58.315 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c5b7fe700 1 -- 172.21.15.32:0/3105860850 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c640107a0 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c5b7fe700 1 -- 172.21.15.32:0/3105860850 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f7c64010d70 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c5b7fe700 1 -- 172.21.15.32:0/3105860850 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c6401ad50 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c794fc700 1 -- 172.21.15.32:0/3105860850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c74113d50 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c794fc700 1 -- 172.21.15.32:0/3105860850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c741148c0 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c5b7fe700 1 -- 172.21.15.32:0/3105860850 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7c6402b6c0 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.553+0000 7f7c5b7fe700 1 -- 172.21.15.32:0/3105860850 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7c6401e070 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.554+0000 7f7c794fc700 1 -- 172.21.15.32:0/3105860850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7405f5f0 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.556+0000 7f7c5b7fe700 1 -- 172.21.15.32:0/3105860850 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f7c6401a530 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.630+0000 7f7c794fc700 1 -- 172.21.15.32:0/3105860850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7c74108820 con 0x7f7c74078ca0 2024-04-06T14:06:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.630+0000 7f7c5b7fe700 1 -- 172.21.15.32:0/3105860850 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f7c64010900 con 0x7f7c74078ca0 2024-04-06T14:06:58.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.631+0000 7f7c794fc700 1 -- 172.21.15.32:0/3105860850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 msgr2=0x7f7c74113610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:06:58.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.631+0000 7f7c794fc700 1 --2- 172.21.15.32:0/3105860850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c74113610 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7c64012fa0 tx=0x7f7c64009250 comp rx=0 tx=0).stop 2024-04-06T14:06:58.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.631+0000 7f7c794fc700 1 -- 172.21.15.32:0/3105860850 shutdown_connections 2024-04-06T14:06:58.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.631+0000 7f7c794fc700 1 --2- 172.21.15.32:0/3105860850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c74078ca0 0x7f7c74113610 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:06:58.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.631+0000 7f7c794fc700 1 -- 172.21.15.32:0/3105860850 >> 172.21.15.32:0/3105860850 conn(0x7f7c740fdde0 msgr2=0x7f7c740fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:06:58.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.631+0000 7f7c794fc700 1 -- 172.21.15.32:0/3105860850 shutdown_connections 2024-04-06T14:06:58.317 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:06:57.631+0000 7f7c794fc700 1 -- 172.21.15.32:0/3105860850 wait complete. 2024-04-06T14:06:58.317 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (1/15)... 2024-04-06T14:07:01.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:01 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1778144994' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T14:07:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "cabdc720-f41e-11ee-b649-cb9ed24678a4", 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T14:07:01.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "smithi032" 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T14:07:01.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T14:07:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T14:07:01.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T14:06:48.331052+0000", 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.887+0000 7fe8224bf700 1 Processor -- start 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.887+0000 7fe8224bf700 1 -- start start 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.887+0000 7fe8224bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c102f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.887+0000 7fe8224bf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c1034c0 con 0x7fe81c102b70 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.887+0000 7fe81bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.887+0000 7fe81bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c102f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39402/0 (socket says 172.21.15.32:39402) 2024-04-06T14:07:01.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.887+0000 7fe81bfff700 1 -- 172.21.15.32:0/754894056 learned_addr learned my addr 172.21.15.32:0/754894056 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.887+0000 7fe81bfff700 1 -- 172.21.15.32:0/754894056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c103600 con 0x7fe81c102b70 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.888+0000 7fe81bfff700 1 --2- 172.21.15.32:0/754894056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c102f80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0098e0 tx=0x7fe80c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e1a8410a563dbd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.888+0000 7fe81affd700 1 -- 172.21.15.32:0/754894056 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe80c00b860 con 0x7fe81c102b70 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.888+0000 7fe81affd700 1 -- 172.21.15.32:0/754894056 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe80c00be30 con 0x7fe81c102b70 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.888+0000 7fe81affd700 1 -- 172.21.15.32:0/754894056 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe80c010380 con 0x7fe81c102b70 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.888+0000 7fe8224bf700 1 -- 172.21.15.32:0/754894056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 msgr2=0x7fe81c102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.888+0000 7fe8224bf700 1 --2- 172.21.15.32:0/754894056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c102f80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0098e0 tx=0x7fe80c009250 comp rx=0 tx=0).stop 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.888+0000 7fe8224bf700 1 -- 172.21.15.32:0/754894056 shutdown_connections 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.888+0000 7fe8224bf700 1 --2- 172.21.15.32:0/754894056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c102f80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.888+0000 7fe8224bf700 1 -- 172.21.15.32:0/754894056 >> 172.21.15.32:0/754894056 conn(0x7fe81c0fc340 msgr2=0x7fe81c0fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.889+0000 7fe8224bf700 1 -- 172.21.15.32:0/754894056 shutdown_connections 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.889+0000 7fe8224bf700 1 -- 172.21.15.32:0/754894056 wait complete. 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.889+0000 7fe8224bf700 1 Processor -- start 2024-04-06T14:07:01.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.889+0000 7fe8224bf700 1 -- start start 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.889+0000 7fe8224bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c113520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.889+0000 7fe8224bf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c113a60 con 0x7fe81c102b70 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe81bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c113520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe81bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c113520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39406/0 (socket says 172.21.15.32:39406) 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe81bfff700 1 -- 172.21.15.32:0/1778144994 learned_addr learned my addr 172.21.15.32:0/1778144994 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe81bfff700 1 -- 172.21.15.32:0/1778144994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0094f0 con 0x7fe81c102b70 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe81bfff700 1 --2- 172.21.15.32:0/1778144994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c113520 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe80c006b00 tx=0x7fe80c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe818ff9700 1 -- 172.21.15.32:0/1778144994 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe80c0109b0 con 0x7fe81c102b70 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe818ff9700 1 -- 172.21.15.32:0/1778144994 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe80c024440 con 0x7fe81c102b70 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe8224bf700 1 -- 172.21.15.32:0/1778144994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81c113c60 con 0x7fe81c102b70 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe818ff9700 1 -- 172.21.15.32:0/1778144994 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe80c01a5f0 con 0x7fe81c102b70 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.890+0000 7fe8224bf700 1 -- 172.21.15.32:0/1778144994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81c1147d0 con 0x7fe81c102b70 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.891+0000 7fe818ff9700 1 -- 172.21.15.32:0/1778144994 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe80c01aae0 con 0x7fe81c102b70 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.891+0000 7fe818ff9700 1 -- 172.21.15.32:0/1778144994 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe80c02db40 con 0x7fe81c102b70 2024-04-06T14:07:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.891+0000 7fe8224bf700 1 -- 172.21.15.32:0/1778144994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c05f5f0 con 0x7fe81c102b70 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.894+0000 7fe818ff9700 1 -- 172.21.15.32:0/1778144994 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe80c02dda0 con 0x7fe81c102b70 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.967+0000 7fe8224bf700 1 -- 172.21.15.32:0/1778144994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe81c1087b0 con 0x7fe81c102b70 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.967+0000 7fe818ff9700 1 -- 172.21.15.32:0/1778144994 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fe80c032030 con 0x7fe81c102b70 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.968+0000 7fe8224bf700 1 -- 172.21.15.32:0/1778144994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 msgr2=0x7fe81c113520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.968+0000 7fe8224bf700 1 --2- 172.21.15.32:0/1778144994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c113520 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe80c006b00 tx=0x7fe80c009250 comp rx=0 tx=0).stop 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.968+0000 7fe8224bf700 1 -- 172.21.15.32:0/1778144994 shutdown_connections 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.968+0000 7fe8224bf700 1 --2- 172.21.15.32:0/1778144994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe81c102b70 0x7fe81c113520 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.968+0000 7fe8224bf700 1 -- 172.21.15.32:0/1778144994 >> 172.21.15.32:0/1778144994 conn(0x7fe81c0fc340 msgr2=0x7fe81c0fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.968+0000 7fe8224bf700 1 -- 172.21.15.32:0/1778144994 shutdown_connections 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:00.968+0000 7fe8224bf700 1 -- 172.21.15.32:0/1778144994 wait complete. 2024-04-06T14:07:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (2/15)... 2024-04-06T14:07:03.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: Activating manager daemon smithi032.peohxu 2024-04-06T14:07:03.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: mgrmap e2: smithi032.peohxu(active, starting, since 0.00278232s) 2024-04-06T14:07:03.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:07:03.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:07:03.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:07:03.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:07:03.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:07:03.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: Manager daemon smithi032.peohxu is now available 2024-04-06T14:07:03.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:07:03.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:03.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/trash_purge_schedule"}]: dispatch 2024-04-06T14:07:03.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:03.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:03 smithi032 ceph-mon[39730]: from='mgr.14100 172.21.15.32:0/3989952307' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "cabdc720-f41e-11ee-b649-cb9ed24678a4", 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T14:07:04.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "smithi032" 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T14:07:04.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T14:07:04.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T14:07:04.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T14:06:48.331052+0000", 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.209+0000 7f9742346700 1 Processor -- start 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.209+0000 7f9742346700 1 -- start start 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.209+0000 7f9742346700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c101900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.209+0000 7f9742346700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c101e40 con 0x7f973c1014f0 2024-04-06T14:07:04.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.209+0000 7f973bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.209+0000 7f973bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c101900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39480/0 (socket says 172.21.15.32:39480) 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.209+0000 7f973bfff700 1 -- 172.21.15.32:0/1465412282 learned_addr learned my addr 172.21.15.32:0/1465412282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.209+0000 7f973bfff700 1 -- 172.21.15.32:0/1465412282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c101f80 con 0x7f973c1014f0 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.209+0000 7f973bfff700 1 --2- 172.21.15.32:0/1465412282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c101900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f972c00bb80 tx=0x7f972c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af000d151cb847c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f973affd700 1 -- 172.21.15.32:0/1465412282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f972c00dd00 con 0x7f973c1014f0 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f973affd700 1 -- 172.21.15.32:0/1465412282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f972c00de60 con 0x7f973c1014f0 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f973affd700 1 -- 172.21.15.32:0/1465412282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f972c018840 con 0x7f973c1014f0 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f9742346700 1 -- 172.21.15.32:0/1465412282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 msgr2=0x7f973c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f9742346700 1 --2- 172.21.15.32:0/1465412282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c101900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f972c00bb80 tx=0x7f972c00b4f0 comp rx=0 tx=0).stop 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f9742346700 1 -- 172.21.15.32:0/1465412282 shutdown_connections 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f9742346700 1 --2- 172.21.15.32:0/1465412282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c101900 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f9742346700 1 -- 172.21.15.32:0/1465412282 >> 172.21.15.32:0/1465412282 conn(0x7f973c0fcc70 msgr2=0x7f973c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f9742346700 1 -- 172.21.15.32:0/1465412282 shutdown_connections 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.210+0000 7f9742346700 1 -- 172.21.15.32:0/1465412282 wait complete. 2024-04-06T14:07:04.848 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.211+0000 7f9742346700 1 Processor -- start 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.211+0000 7f9742346700 1 -- start start 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.211+0000 7f9742346700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c10f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.211+0000 7f9742346700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c10f640 con 0x7f973c1014f0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.211+0000 7f973bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c10f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.211+0000 7f973bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c10f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39486/0 (socket says 172.21.15.32:39486) 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.211+0000 7f973bfff700 1 -- 172.21.15.32:0/3722799424 learned_addr learned my addr 172.21.15.32:0/3722799424 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.211+0000 7f973bfff700 1 -- 172.21.15.32:0/3722799424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c00b790 con 0x7f973c1014f0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.211+0000 7f973bfff700 1 --2- 172.21.15.32:0/3722799424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c10f100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f972c006d40 tx=0x7f972c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.212+0000 7f9738ff9700 1 -- 172.21.15.32:0/3722799424 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f972c010a00 con 0x7f973c1014f0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.212+0000 7f9738ff9700 1 -- 172.21.15.32:0/3722799424 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f972c01d050 con 0x7f973c1014f0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.212+0000 7f9742346700 1 -- 172.21.15.32:0/3722799424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c10f840 con 0x7f973c1014f0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.212+0000 7f9738ff9700 1 -- 172.21.15.32:0/3722799424 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f972c022590 con 0x7f973c1014f0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.212+0000 7f9742346700 1 -- 172.21.15.32:0/3722799424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c1103b0 con 0x7f973c1014f0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.213+0000 7f9738ff9700 1 -- 172.21.15.32:0/3722799424 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7f972c010b60 con 0x7f973c1014f0 2024-04-06T14:07:04.849 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.213+0000 7f9742346700 1 -- 172.21.15.32:0/3722799424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973c05f5f0 con 0x7f973c1014f0 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.213+0000 7f9738ff9700 1 -- 172.21.15.32:0/3722799424 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f972c014070 con 0x7f973c1014f0 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.216+0000 7f9738ff9700 1 -- 172.21.15.32:0/3722799424 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f972c022b10 con 0x7f973c1014f0 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.288+0000 7f9742346700 1 -- 172.21.15.32:0/3722799424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f973c04bed0 con 0x7f973c1014f0 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.288+0000 7f9738ff9700 1 -- 172.21.15.32:0/3722799424 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f972c00e360 con 0x7f973c1014f0 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.289+0000 7f9742346700 1 -- 172.21.15.32:0/3722799424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 msgr2=0x7f973c10f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.289+0000 7f9742346700 1 --2- 172.21.15.32:0/3722799424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c10f100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f972c006d40 tx=0x7f972c00b4f0 comp rx=0 tx=0).stop 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.289+0000 7f9742346700 1 -- 172.21.15.32:0/3722799424 shutdown_connections 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.289+0000 7f9742346700 1 --2- 172.21.15.32:0/3722799424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c1014f0 0x7f973c10f100 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.289+0000 7f9742346700 1 -- 172.21.15.32:0/3722799424 >> 172.21.15.32:0/3722799424 conn(0x7f973c0fcc70 msgr2=0x7f973c105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.289+0000 7f9742346700 1 -- 172.21.15.32:0/3722799424 shutdown_connections 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:04.289+0000 7f9742346700 1 -- 172.21.15.32:0/3722799424 wait complete. 2024-04-06T14:07:04.850 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (3/15)... 2024-04-06T14:07:04.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:04 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3722799424' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T14:07:04.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:04 smithi032 ceph-mon[39730]: mgrmap e3: smithi032.peohxu(active, since 1.00612s) 2024-04-06T14:07:06.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:06 smithi032 ceph-mon[39730]: mgrmap e4: smithi032.peohxu(active, since 2s) 2024-04-06T14:07:08.071 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:07 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2568654240' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T14:07:08.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:07:08.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-06T14:07:08.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "cabdc720-f41e-11ee-b649-cb9ed24678a4", 2024-04-06T14:07:08.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T14:07:08.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T14:07:08.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T14:07:08.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T14:07:08.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:08.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T14:07:08.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T14:07:08.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-06T14:07:08.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "smithi032" 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T14:07:08.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T14:07:08.355 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T14:07:08.355 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T14:07:08.356 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T14:07:08.356 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:08.356 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T14:07:08.356 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T14:07:08.356 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T14:07:08.356 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T14:07:08.356 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T14:07:08.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T14:07:08.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T14:07:08.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T14:07:08.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T14:07:08.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:08.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T14:07:08.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:08.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T14:07:08.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T14:07:08.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:08.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T14:07:08.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-06T14:07:08.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T14:07:08.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T14:07:08.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T14:07:08.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T14:07:08.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T14:07:08.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-06T14:07:08.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T14:07:08.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:08.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T14:07:08.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T14:07:08.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T14:06:48.331052+0000", 2024-04-06T14:07:08.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T14:07:08.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.429+0000 7faf5fc81700 1 Processor -- start 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.430+0000 7faf5fc81700 1 -- start start 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.430+0000 7faf5fc81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf581051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.430+0000 7faf5fc81700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf581056f0 con 0x7faf58076c80 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.430+0000 7faf5da1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf581051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.430+0000 7faf5da1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf581051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50262/0 (socket says 172.21.15.32:50262) 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.430+0000 7faf5da1d700 1 -- 172.21.15.32:0/2633834822 learned_addr learned my addr 172.21.15.32:0/2633834822 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.430+0000 7faf5da1d700 1 -- 172.21.15.32:0/2633834822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf58105830 con 0x7faf58076c80 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5da1d700 1 --2- 172.21.15.32:0/2633834822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf581051b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faf540098e0 tx=0x7faf54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e300903aeb382c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5ca1b700 1 -- 172.21.15.32:0/2633834822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf5400b860 con 0x7faf58076c80 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5ca1b700 1 -- 172.21.15.32:0/2633834822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7faf5400be30 con 0x7faf58076c80 2024-04-06T14:07:08.361 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5ca1b700 1 -- 172.21.15.32:0/2633834822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf54010380 con 0x7faf58076c80 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5fc81700 1 -- 172.21.15.32:0/2633834822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 msgr2=0x7faf581051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5fc81700 1 --2- 172.21.15.32:0/2633834822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf581051b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faf540098e0 tx=0x7faf54009250 comp rx=0 tx=0).stop 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5fc81700 1 -- 172.21.15.32:0/2633834822 shutdown_connections 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5fc81700 1 --2- 172.21.15.32:0/2633834822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf581051b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5fc81700 1 -- 172.21.15.32:0/2633834822 >> 172.21.15.32:0/2633834822 conn(0x7faf580fcc60 msgr2=0x7faf580ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5fc81700 1 -- 172.21.15.32:0/2633834822 shutdown_connections 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.431+0000 7faf5fc81700 1 -- 172.21.15.32:0/2633834822 wait complete. 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.432+0000 7faf5fc81700 1 Processor -- start 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.432+0000 7faf5fc81700 1 -- start start 2024-04-06T14:07:08.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.432+0000 7faf5fc81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf58111370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.432+0000 7faf5fc81700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf581118b0 con 0x7faf58076c80 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf5da1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf58111370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf5da1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf58111370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50266/0 (socket says 172.21.15.32:50266) 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf5da1d700 1 -- 172.21.15.32:0/2568654240 learned_addr learned my addr 172.21.15.32:0/2568654240 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf5da1d700 1 -- 172.21.15.32:0/2568654240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf540094f0 con 0x7faf58076c80 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf5da1d700 1 --2- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf58111370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faf54006d40 tx=0x7faf54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf4a7fc700 1 -- 172.21.15.32:0/2568654240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf540109b0 con 0x7faf58076c80 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf4a7fc700 1 -- 172.21.15.32:0/2568654240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7faf54024440 con 0x7faf58076c80 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf4a7fc700 1 -- 172.21.15.32:0/2568654240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf5401a5f0 con 0x7faf58076c80 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf58111ab0 con 0x7faf58076c80 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.433+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf58112620 con 0x7faf58076c80 2024-04-06T14:07:08.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.434+0000 7faf4a7fc700 1 -- 172.21.15.32:0/2568654240 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7faf54010b10 con 0x7faf58076c80 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.434+0000 7faf4a7fc700 1 --2- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7faf44036740 0x7faf44038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.434+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5805f5f0 con 0x7faf58076c80 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.434+0000 7faf4a7fc700 1 -- 172.21.15.32:0/2568654240 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7faf5404a620 con 0x7faf58076c80 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.434+0000 7faf5d21c700 1 --2- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7faf44036740 0x7faf44038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.435+0000 7faf5d21c700 1 --2- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7faf44036740 0x7faf44038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009990 tx=0x7faf4c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.439+0000 7faf4a7fc700 1 -- 172.21.15.32:0/2568654240 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7faf54024970 con 0x7faf58076c80 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.711+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7faf5804bed0 con 0x7faf58076c80 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.712+0000 7faf4a7fc700 1 -- 172.21.15.32:0/2568654240 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7faf5403d090 con 0x7faf58076c80 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.713+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7faf44036740 msgr2=0x7faf44038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.713+0000 7faf5fc81700 1 --2- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7faf44036740 0x7faf44038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009990 tx=0x7faf4c006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.713+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 msgr2=0x7faf58111370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.713+0000 7faf5fc81700 1 --2- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf58111370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faf54006d40 tx=0x7faf54009250 comp rx=0 tx=0).stop 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.713+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 shutdown_connections 2024-04-06T14:07:08.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.713+0000 7faf5fc81700 1 --2- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7faf44036740 0x7faf44038bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:08.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.714+0000 7faf5fc81700 1 --2- 172.21.15.32:0/2568654240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf58076c80 0x7faf58111370 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:08.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.714+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 >> 172.21.15.32:0/2568654240 conn(0x7faf580fcc60 msgr2=0x7faf580fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:08.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.714+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 shutdown_connections 2024-04-06T14:07:08.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:07.714+0000 7faf5fc81700 1 -- 172.21.15.32:0/2568654240 wait complete. 2024-04-06T14:07:08.365 INFO:teuthology.orchestra.run.smithi032.stdout:mgr is available 2024-04-06T14:07:09.628 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:09 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2870185015' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T14:07:09.889 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:07:09.889 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-06T14:07:09.889 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:07:09.889 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-06T14:07:09.889 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-06T14:07:09.889 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.042+0000 7f8c5fe8c700 1 Processor -- start 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.043+0000 7f8c5fe8c700 1 -- start start 2024-04-06T14:07:09.890 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.043+0000 7f8c5fe8c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c581051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.043+0000 7f8c5fe8c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c581056f0 con 0x7f8c58076c80 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.043+0000 7f8c5dc28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c581051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.043+0000 7f8c5dc28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c581051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50282/0 (socket says 172.21.15.32:50282) 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.043+0000 7f8c5dc28700 1 -- 172.21.15.32:0/198334115 learned_addr learned my addr 172.21.15.32:0/198334115 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.043+0000 7f8c5dc28700 1 -- 172.21.15.32:0/198334115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c58105830 con 0x7f8c58076c80 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.043+0000 7f8c5dc28700 1 --2- 172.21.15.32:0/198334115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c581051b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8c540098e0 tx=0x7f8c54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0e825d88e7e5f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5cc26700 1 -- 172.21.15.32:0/198334115 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c5400b860 con 0x7f8c58076c80 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5cc26700 1 -- 172.21.15.32:0/198334115 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f8c5400be30 con 0x7f8c58076c80 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5cc26700 1 -- 172.21.15.32:0/198334115 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c54010380 con 0x7f8c58076c80 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/198334115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 msgr2=0x7f8c581051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5fe8c700 1 --2- 172.21.15.32:0/198334115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c581051b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8c540098e0 tx=0x7f8c54009250 comp rx=0 tx=0).stop 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/198334115 shutdown_connections 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5fe8c700 1 --2- 172.21.15.32:0/198334115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c581051b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/198334115 >> 172.21.15.32:0/198334115 conn(0x7f8c580fcc60 msgr2=0x7f8c580ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/198334115 shutdown_connections 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.044+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/198334115 wait complete. 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.045+0000 7f8c5fe8c700 1 Processor -- start 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.045+0000 7f8c5fe8c700 1 -- start start 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.045+0000 7f8c5fe8c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c58102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.045+0000 7f8c5fe8c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c58103040 con 0x7f8c58076c80 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.045+0000 7f8c5dc28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c58102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.045+0000 7f8c5dc28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c58102b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50288/0 (socket says 172.21.15.32:50288) 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.045+0000 7f8c5dc28700 1 -- 172.21.15.32:0/2870185015 learned_addr learned my addr 172.21.15.32:0/2870185015 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.046+0000 7f8c5dc28700 1 -- 172.21.15.32:0/2870185015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c540094f0 con 0x7f8c58076c80 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.046+0000 7f8c5dc28700 1 --2- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c58102b00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8c54006d40 tx=0x7f8c54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.046+0000 7f8c4e7fc700 1 -- 172.21.15.32:0/2870185015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c540109b0 con 0x7f8c58076c80 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.046+0000 7f8c4e7fc700 1 -- 172.21.15.32:0/2870185015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f8c54024440 con 0x7f8c58076c80 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.046+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c581010b0 con 0x7f8c58076c80 2024-04-06T14:07:09.892 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.046+0000 7f8c4e7fc700 1 -- 172.21.15.32:0/2870185015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c5401a5f0 con 0x7f8c58076c80 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.046+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c581015a0 con 0x7f8c58076c80 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.047+0000 7f8c4e7fc700 1 -- 172.21.15.32:0/2870185015 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f8c5401a870 con 0x7f8c58076c80 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.047+0000 7f8c4e7fc700 1 --2- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f8c44036740 0x7f8c44038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.047+0000 7f8c4e7fc700 1 -- 172.21.15.32:0/2870185015 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8c5404a5e0 con 0x7f8c58076c80 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.047+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5804bed0 con 0x7f8c58076c80 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.047+0000 7f8c5d427700 1 --2- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f8c44036740 0x7f8c44038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.047+0000 7f8c5d427700 1 --2- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f8c44036740 0x7f8c44038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009990 tx=0x7f8c48006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.052+0000 7f8c4e7fc700 1 -- 172.21.15.32:0/2870185015 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8c54018db0 con 0x7f8c58076c80 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.245+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f8c58101a20 con 0x7f8c58076c80 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.246+0000 7f8c4e7fc700 1 -- 172.21.15.32:0/2870185015 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f8c54048020 con 0x7f8c58076c80 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f8c44036740 msgr2=0x7f8c44038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 --2- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f8c44036740 0x7f8c44038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009990 tx=0x7f8c48006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 msgr2=0x7f8c58102b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:09.893 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 --2- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c58102b00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8c54006d40 tx=0x7f8c54009250 comp rx=0 tx=0).stop 2024-04-06T14:07:09.894 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 shutdown_connections 2024-04-06T14:07:09.894 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 --2- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f8c44036740 0x7f8c44038bf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:09.894 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 --2- 172.21.15.32:0/2870185015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c58076c80 0x7f8c58102b00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:09.894 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 >> 172.21.15.32:0/2870185015 conn(0x7f8c580fcc60 msgr2=0x7f8c580fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:09.894 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 shutdown_connections 2024-04-06T14:07:09.894 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:09.248+0000 7f8c5fe8c700 1 -- 172.21.15.32:0/2870185015 wait complete. 2024-04-06T14:07:09.894 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling cephadm module... 2024-04-06T14:07:11.133 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:10 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/438813393' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-06T14:07:11.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.485+0000 7ff9cf770700 1 Processor -- start 2024-04-06T14:07:11.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.485+0000 7ff9cf770700 1 -- start start 2024-04-06T14:07:11.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.485+0000 7ff9cf770700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8105480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:11.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.485+0000 7ff9cf770700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8076d40 con 0x7ff9c81030a0 2024-04-06T14:07:11.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.485+0000 7ff9cd50c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8105480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:11.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.486+0000 7ff9cd50c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8105480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50294/0 (socket says 172.21.15.32:50294) 2024-04-06T14:07:11.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.486+0000 7ff9cd50c700 1 -- 172.21.15.32:0/3339686204 learned_addr learned my addr 172.21.15.32:0/3339686204 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:11.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.486+0000 7ff9cd50c700 1 -- 172.21.15.32:0/3339686204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8076e80 con 0x7ff9c81030a0 2024-04-06T14:07:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.486+0000 7ff9cd50c700 1 --2- 172.21.15.32:0/3339686204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8105480 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40098e0 tx=0x7ff9c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0e630084d394ad server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.486+0000 7ff9bffff700 1 -- 172.21.15.32:0/3339686204 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c400b860 con 0x7ff9c81030a0 2024-04-06T14:07:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.486+0000 7ff9bffff700 1 -- 172.21.15.32:0/3339686204 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff9c400be30 con 0x7ff9c81030a0 2024-04-06T14:07:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.486+0000 7ff9bffff700 1 -- 172.21.15.32:0/3339686204 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c4010380 con 0x7ff9c81030a0 2024-04-06T14:07:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.487+0000 7ff9cf770700 1 -- 172.21.15.32:0/3339686204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 msgr2=0x7ff9c8105480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.487+0000 7ff9cf770700 1 --2- 172.21.15.32:0/3339686204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8105480 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40098e0 tx=0x7ff9c4009250 comp rx=0 tx=0).stop 2024-04-06T14:07:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.487+0000 7ff9cf770700 1 -- 172.21.15.32:0/3339686204 shutdown_connections 2024-04-06T14:07:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.487+0000 7ff9cf770700 1 --2- 172.21.15.32:0/3339686204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8105480 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:11.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.487+0000 7ff9cf770700 1 -- 172.21.15.32:0/3339686204 >> 172.21.15.32:0/3339686204 conn(0x7ff9c80fcb20 msgr2=0x7ff9c80fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.487+0000 7ff9cf770700 1 -- 172.21.15.32:0/3339686204 shutdown_connections 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.487+0000 7ff9cf770700 1 -- 172.21.15.32:0/3339686204 wait complete. 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.487+0000 7ff9cf770700 1 Processor -- start 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9cf770700 1 -- start start 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9cf770700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8102a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9cf770700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8102f40 con 0x7ff9c81030a0 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9cd50c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8102a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9cd50c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8102a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50308/0 (socket says 172.21.15.32:50308) 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9cd50c700 1 -- 172.21.15.32:0/438813393 learned_addr learned my addr 172.21.15.32:0/438813393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9cd50c700 1 -- 172.21.15.32:0/438813393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c40094f0 con 0x7ff9c81030a0 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9cd50c700 1 --2- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8102a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4006b00 tx=0x7ff9c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:11.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9bdffb700 1 -- 172.21.15.32:0/438813393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c4010a80 con 0x7ff9c81030a0 2024-04-06T14:07:11.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.488+0000 7ff9bdffb700 1 -- 172.21.15.32:0/438813393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff9c401a430 con 0x7ff9c81030a0 2024-04-06T14:07:11.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.489+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c8101010 con 0x7ff9c81030a0 2024-04-06T14:07:11.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.489+0000 7ff9bdffb700 1 -- 172.21.15.32:0/438813393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c40245b0 con 0x7ff9c81030a0 2024-04-06T14:07:11.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.489+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c8101500 con 0x7ff9c81030a0 2024-04-06T14:07:11.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.489+0000 7ff9bdffb700 1 -- 172.21.15.32:0/438813393 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7ff9c40247e0 con 0x7ff9c81030a0 2024-04-06T14:07:11.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.490+0000 7ff9bdffb700 1 --2- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7ff9b4036790 0x7ff9b4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:11.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.490+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c804bed0 con 0x7ff9c81030a0 2024-04-06T14:07:11.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.490+0000 7ff9bdffb700 1 -- 172.21.15.32:0/438813393 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff9c404ac60 con 0x7ff9c81030a0 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.490+0000 7ff9ccd0b700 1 --2- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7ff9b4036790 0x7ff9b4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.490+0000 7ff9ccd0b700 1 --2- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7ff9b4036790 0x7ff9b4038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009990 tx=0x7ff9b8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.495+0000 7ff9bdffb700 1 -- 172.21.15.32:0/438813393 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff9c4018350 con 0x7ff9c81030a0 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.745+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff9c8106520 con 0x7ff9c81030a0 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.798+0000 7ff9bdffb700 1 -- 172.21.15.32:0/438813393 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff9c4024cc0 con 0x7ff9c81030a0 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.799+0000 7ff9bdffb700 1 -- 172.21.15.32:0/438813393 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff9c4030d60 con 0x7ff9c81030a0 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7ff9b4036790 msgr2=0x7ff9b4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 --2- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7ff9b4036790 0x7ff9b4038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009990 tx=0x7ff9b8006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 msgr2=0x7ff9c8102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 --2- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8102a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4006b00 tx=0x7ff9c4009250 comp rx=0 tx=0).stop 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 shutdown_connections 2024-04-06T14:07:11.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 --2- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7ff9b4036790 0x7ff9b4038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:11.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 --2- 172.21.15.32:0/438813393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9c81030a0 0x7ff9c8102a00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:11.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 >> 172.21.15.32:0/438813393 conn(0x7ff9c80fcb20 msgr2=0x7ff9c80fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:11.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 shutdown_connections 2024-04-06T14:07:11.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:10.802+0000 7ff9cf770700 1 -- 172.21.15.32:0/438813393 wait complete. 2024-04-06T14:07:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:11 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/438813393' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-06T14:07:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:11 smithi032 ceph-mon[39730]: mgrmap e5: smithi032.peohxu(active, since 7s) 2024-04-06T14:07:12.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-06T14:07:12.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-06T14:07:12.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-06T14:07:12.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "smithi032.peohxu", 2024-04-06T14:07:12.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-06T14:07:12.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-06T14:07:12.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.989+0000 7f78df378700 1 Processor -- start 2024-04-06T14:07:12.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.990+0000 7f78df378700 1 -- start start 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.990+0000 7f78df378700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8105210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.990+0000 7f78df378700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8105750 con 0x7f78d8076c20 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.990+0000 7f78dd114700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8105210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.990+0000 7f78dd114700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8105210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50344/0 (socket says 172.21.15.32:50344) 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.990+0000 7f78dd114700 1 -- 172.21.15.32:0/3991114857 learned_addr learned my addr 172.21.15.32:0/3991114857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.990+0000 7f78dd114700 1 -- 172.21.15.32:0/3991114857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d8105890 con 0x7f78d8076c20 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.990+0000 7f78dd114700 1 --2- 172.21.15.32:0/3991114857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8105210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f78d40098e0 tx=0x7f78d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ed6cc8554c639a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78cffff700 1 -- 172.21.15.32:0/3991114857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d400b860 con 0x7f78d8076c20 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78cffff700 1 -- 172.21.15.32:0/3991114857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f78d400be30 con 0x7f78d8076c20 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78cffff700 1 -- 172.21.15.32:0/3991114857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d4010380 con 0x7f78d8076c20 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78df378700 1 -- 172.21.15.32:0/3991114857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 msgr2=0x7f78d8105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78df378700 1 --2- 172.21.15.32:0/3991114857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8105210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f78d40098e0 tx=0x7f78d4009250 comp rx=0 tx=0).stop 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78df378700 1 -- 172.21.15.32:0/3991114857 shutdown_connections 2024-04-06T14:07:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78df378700 1 --2- 172.21.15.32:0/3991114857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8105210 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78df378700 1 -- 172.21.15.32:0/3991114857 >> 172.21.15.32:0/3991114857 conn(0x7f78d80fcc60 msgr2=0x7f78d80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78df378700 1 -- 172.21.15.32:0/3991114857 shutdown_connections 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.991+0000 7f78df378700 1 -- 172.21.15.32:0/3991114857 wait complete. 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.992+0000 7f78df378700 1 Processor -- start 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.992+0000 7f78df378700 1 -- start start 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.992+0000 7f78df378700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.992+0000 7f78df378700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d81030a0 con 0x7f78d8076c20 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.992+0000 7f78dd114700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.992+0000 7f78dd114700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8102b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50360/0 (socket says 172.21.15.32:50360) 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.992+0000 7f78dd114700 1 -- 172.21.15.32:0/2361831484 learned_addr learned my addr 172.21.15.32:0/2361831484 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.992+0000 7f78dd114700 1 -- 172.21.15.32:0/2361831484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d40094f0 con 0x7f78d8076c20 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.993+0000 7f78dd114700 1 --2- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8102b60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f78d4006b00 tx=0x7f78d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.993+0000 7f78cdffb700 1 -- 172.21.15.32:0/2361831484 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d4010a80 con 0x7f78d8076c20 2024-04-06T14:07:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.993+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d8101110 con 0x7f78d8076c20 2024-04-06T14:07:12.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.993+0000 7f78cdffb700 1 -- 172.21.15.32:0/2361831484 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f78d401a430 con 0x7f78d8076c20 2024-04-06T14:07:12.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:12 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2361831484' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T14:07:12.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.993+0000 7f78cdffb700 1 -- 172.21.15.32:0/2361831484 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d40245b0 con 0x7f78d8076c20 2024-04-06T14:07:12.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.993+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d8101600 con 0x7f78d8076c20 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.994+0000 7f78cdffb700 1 -- 172.21.15.32:0/2361831484 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f78d4024830 con 0x7f78d8076c20 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.994+0000 7f78cdffb700 1 --2- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f78c4036790 0x7f78c4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.994+0000 7f78cdffb700 1 -- 172.21.15.32:0/2361831484 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f78d404ad80 con 0x7f78d8076c20 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.994+0000 7f78dc913700 1 -- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f78c4036790 msgr2=0x7f78c4038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/818700653 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.994+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d804bed0 con 0x7f78d8076c20 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.994+0000 7f78dc913700 1 --2- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f78c4036790 0x7f78c4038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:11.999+0000 7f78cdffb700 1 -- 172.21.15.32:0/2361831484 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f78d4018350 con 0x7f78d8076c20 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.195+0000 7f78dc913700 1 -- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f78c4036790 msgr2=0x7f78c4038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/818700653 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.195+0000 7f78dc913700 1 --2- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f78c4036790 0x7f78c4038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T14:07:12.954 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.249+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f78d8101d60 con 0x7f78d8076c20 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.249+0000 7f78cdffb700 1 -- 172.21.15.32:0/2361831484 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f78d4049030 con 0x7f78d8076c20 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f78c4036790 msgr2=0x7f78c4038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 --2- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f78c4036790 0x7f78c4038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 msgr2=0x7f78d8102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 --2- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8102b60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f78d4006b00 tx=0x7f78d4009250 comp rx=0 tx=0).stop 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 shutdown_connections 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 --2- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f78c4036790 0x7f78c4038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 --2- 172.21.15.32:0/2361831484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d8076c20 0x7f78d8102b60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 >> 172.21.15.32:0/2361831484 conn(0x7f78d80fcc60 msgr2=0x7f78d80fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 shutdown_connections 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:12.251+0000 7f78df378700 1 -- 172.21.15.32:0/2361831484 wait complete. 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-06T14:07:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 5... 2024-04-06T14:07:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: Active manager daemon smithi032.peohxu restarted 2024-04-06T14:07:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: Activating manager daemon smithi032.peohxu 2024-04-06T14:07:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: osdmap e2: 0 total, 0 up, 0 in 2024-04-06T14:07:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: mgrmap e6: smithi032.peohxu(active, starting, since 0.00297123s) 2024-04-06T14:07:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:07:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:07:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:07:17.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:07:17.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:07:17.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: Manager daemon smithi032.peohxu is now available 2024-04-06T14:07:17.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:17.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:17.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:07:17.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:17 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:07:18.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:18 smithi032 ceph-mon[39730]: Found migration_current of "None". Setting to last migration. 2024-04-06T14:07:18.974 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:18 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:07:18.974 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:18 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:07:18.974 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:18 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/trash_purge_schedule"}]: dispatch 2024-04-06T14:07:18.974 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:18 smithi032 ceph-mon[39730]: mgrmap e7: smithi032.peohxu(active, since 1.00536s) 2024-04-06T14:07:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-06T14:07:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-06T14:07:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-06T14:07:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-06T14:07:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.532+0000 7f47ba63c700 1 Processor -- start 2024-04-06T14:07:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.532+0000 7f47ba63c700 1 -- start start 2024-04-06T14:07:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.533+0000 7f47ba63c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.533+0000 7f47ba63c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b4102ed0 con 0x7f47b4102580 2024-04-06T14:07:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.533+0000 7f47b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.533+0000 7f47b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50364/0 (socket says 172.21.15.32:50364) 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.533+0000 7f47b3fff700 1 -- 172.21.15.32:0/3352003030 learned_addr learned my addr 172.21.15.32:0/3352003030 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.533+0000 7f47b3fff700 1 -- 172.21.15.32:0/3352003030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b4103010 con 0x7f47b4102580 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.533+0000 7f47b3fff700 1 --2- 172.21.15.32:0/3352003030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4102990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f47a40098e0 tx=0x7f47a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d0056b40eaff62 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.533+0000 7f47b2ffd700 1 -- 172.21.15.32:0/3352003030 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a400b860 con 0x7f47b4102580 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.534+0000 7f47b2ffd700 1 -- 172.21.15.32:0/3352003030 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f47a400be30 con 0x7f47b4102580 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.534+0000 7f47b2ffd700 1 -- 172.21.15.32:0/3352003030 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a4010380 con 0x7f47b4102580 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.534+0000 7f47ba63c700 1 -- 172.21.15.32:0/3352003030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 msgr2=0x7f47b4102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.534+0000 7f47ba63c700 1 --2- 172.21.15.32:0/3352003030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4102990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f47a40098e0 tx=0x7f47a4009250 comp rx=0 tx=0).stop 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.534+0000 7f47ba63c700 1 -- 172.21.15.32:0/3352003030 shutdown_connections 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.534+0000 7f47ba63c700 1 --2- 172.21.15.32:0/3352003030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4102990 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.534+0000 7f47ba63c700 1 -- 172.21.15.32:0/3352003030 >> 172.21.15.32:0/3352003030 conn(0x7f47b40fdd00 msgr2=0x7f47b4100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.534+0000 7f47ba63c700 1 -- 172.21.15.32:0/3352003030 shutdown_connections 2024-04-06T14:07:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.534+0000 7f47ba63c700 1 -- 172.21.15.32:0/3352003030 wait complete. 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.535+0000 7f47ba63c700 1 Processor -- start 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.535+0000 7f47ba63c700 1 -- start start 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.535+0000 7f47ba63c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.535+0000 7f47ba63c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b4079190 con 0x7f47b4102580 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.535+0000 7f47b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.535+0000 7f47b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50380/0 (socket says 172.21.15.32:50380) 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.535+0000 7f47b3fff700 1 -- 172.21.15.32:0/1988192787 learned_addr learned my addr 172.21.15.32:0/1988192787 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.536+0000 7f47b3fff700 1 -- 172.21.15.32:0/1988192787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a40094f0 con 0x7f47b4102580 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.536+0000 7f47b3fff700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4078c50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f47a4006b00 tx=0x7f47a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.536+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a40109b0 con 0x7f47b4102580 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.536+0000 7f47ba63c700 1 -- 172.21.15.32:0/1988192787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b4077200 con 0x7f47b4102580 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.536+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f47a4024440 con 0x7f47b4102580 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.536+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a401a5f0 con 0x7f47b4102580 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.536+0000 7f47ba63c700 1 -- 172.21.15.32:0/1988192787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b40776f0 con 0x7f47b4102580 2024-04-06T14:07:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.537+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f47a401a8b0 con 0x7f47b4102580 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.537+0000 7f47b0ff9700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 0x7f479c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.537+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 --> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f479c039350 con 0x7f479c036790 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.537+0000 7f47b37fe700 1 -- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 msgr2=0x7f479c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/818700653 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.537+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f47a404a7c0 con 0x7f47b4102580 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.537+0000 7f47b37fe700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 0x7f479c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.738+0000 7f47b37fe700 1 -- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 msgr2=0x7f479c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/818700653 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:13.738+0000 7f47b37fe700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 0x7f479c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:14.139+0000 7f47b37fe700 1 -- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 msgr2=0x7f479c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/818700653 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:14.139+0000 7f47b37fe700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 0x7f479c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:14.940+0000 7f47b37fe700 1 -- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 msgr2=0x7f479c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/818700653 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:14.940+0000 7f47b37fe700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 0x7f479c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:16.541+0000 7f47b37fe700 1 -- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 msgr2=0x7f479c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/818700653 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:16.541+0000 7f47b37fe700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 0x7f479c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:17.561+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f47a400fdc0 con 0x7f47b4102580 2024-04-06T14:07:19.312 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:17.561+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 msgr2=0x7f479c038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:17.561+0000 7f47b0ff9700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 0x7f479c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.562+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f47a4024990 con 0x7f47b4102580 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.562+0000 7f47b0ff9700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f479c039fc0 0x7f479c03c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.562+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f479c039350 con 0x7f479c039fc0 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.563+0000 7f47b37fe700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f479c039fc0 0x7f479c03c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.563+0000 7f47b37fe700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f479c039fc0 0x7f479c03c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f47a80039d0 tx=0x7f47a8009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.564+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f479c039350 con 0x7f479c039fc0 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47ba63c700 1 -- 172.21.15.32:0/1988192787 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f47b404bed0 con 0x7f479c039fc0 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47b0ff9700 1 -- 172.21.15.32:0/1988192787 <== mgr.14120 v2:172.21.15.32:6800/1833723736 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f47b404bed0 con 0x7f479c039fc0 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47ba63c700 1 -- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f479c039fc0 msgr2=0x7f479c03c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47ba63c700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f479c039fc0 0x7f479c03c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f47a80039d0 tx=0x7f47a8009350 comp rx=0 tx=0).stop 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47ba63c700 1 -- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 msgr2=0x7f47b4078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47ba63c700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4078c50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f47a4006b00 tx=0x7f47a4009250 comp rx=0 tx=0).stop 2024-04-06T14:07:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47ba63c700 1 -- 172.21.15.32:0/1988192787 shutdown_connections 2024-04-06T14:07:19.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47ba63c700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f479c039fc0 0x7f479c03c3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:19.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47ba63c700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47b4102580 0x7f47b4078c50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:19.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.570+0000 7f47ba63c700 1 --2- 172.21.15.32:0/1988192787 >> [v2:172.21.15.32:6800/818700653,v1:172.21.15.32:6801/818700653] conn(0x7f479c036790 0x7f479c038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:19.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.571+0000 7f47ba63c700 1 -- 172.21.15.32:0/1988192787 >> 172.21.15.32:0/1988192787 conn(0x7f47b40fdd00 msgr2=0x7f47b40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:19.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.571+0000 7f47ba63c700 1 -- 172.21.15.32:0/1988192787 shutdown_connections 2024-04-06T14:07:19.314 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:18.571+0000 7f47ba63c700 1 -- 172.21.15.32:0/1988192787 wait complete. 2024-04-06T14:07:19.314 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 5 is available 2024-04-06T14:07:19.314 INFO:teuthology.orchestra.run.smithi032.stdout:Setting orchestrator backend to cephadm... 2024-04-06T14:07:19.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:19 smithi032 ceph-mon[39730]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-06T14:07:19.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:19 smithi032 ceph-mon[39730]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-06T14:07:19.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:19 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:19.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:19 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:07:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:20 smithi032 ceph-mon[39730]: [06/Apr/2024:14:07:18] ENGINE Bus STARTING 2024-04-06T14:07:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:20 smithi032 ceph-mon[39730]: [06/Apr/2024:14:07:18] ENGINE Serving on https://172.21.15.32:7150 2024-04-06T14:07:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:20 smithi032 ceph-mon[39730]: [06/Apr/2024:14:07:18] ENGINE Bus STARTED 2024-04-06T14:07:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:20 smithi032 ceph-mon[39730]: mgrmap e8: smithi032.peohxu(active, since 2s) 2024-04-06T14:07:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:20 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:20 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:07:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.881+0000 7fdd599ce700 1 Processor -- start 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.881+0000 7fdd599ce700 1 -- start start 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.881+0000 7fdd599ce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54102b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.881+0000 7fdd599ce700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54103040 con 0x7fdd541026f0 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.881+0000 7fdd52ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.881+0000 7fdd52ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54102b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43060/0 (socket says 172.21.15.32:43060) 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.881+0000 7fdd52ffd700 1 -- 172.21.15.32:0/4285954455 learned_addr learned my addr 172.21.15.32:0/4285954455 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.882+0000 7fdd52ffd700 1 -- 172.21.15.32:0/4285954455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd54103180 con 0x7fdd541026f0 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.882+0000 7fdd52ffd700 1 --2- 172.21.15.32:0/4285954455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54102b00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd440098e0 tx=0x7fdd44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509dc24b8808259d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.882+0000 7fdd51ffb700 1 -- 172.21.15.32:0/4285954455 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd4400b860 con 0x7fdd541026f0 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.882+0000 7fdd51ffb700 1 -- 172.21.15.32:0/4285954455 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdd4400be30 con 0x7fdd541026f0 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.882+0000 7fdd51ffb700 1 -- 172.21.15.32:0/4285954455 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd44010380 con 0x7fdd541026f0 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.882+0000 7fdd599ce700 1 -- 172.21.15.32:0/4285954455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 msgr2=0x7fdd54102b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.882+0000 7fdd599ce700 1 --2- 172.21.15.32:0/4285954455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54102b00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd440098e0 tx=0x7fdd44009250 comp rx=0 tx=0).stop 2024-04-06T14:07:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.883+0000 7fdd599ce700 1 -- 172.21.15.32:0/4285954455 shutdown_connections 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.883+0000 7fdd599ce700 1 --2- 172.21.15.32:0/4285954455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54102b00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.883+0000 7fdd599ce700 1 -- 172.21.15.32:0/4285954455 >> 172.21.15.32:0/4285954455 conn(0x7fdd540fde50 msgr2=0x7fdd54100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.883+0000 7fdd599ce700 1 -- 172.21.15.32:0/4285954455 shutdown_connections 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.883+0000 7fdd599ce700 1 -- 172.21.15.32:0/4285954455 wait complete. 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.883+0000 7fdd599ce700 1 Processor -- start 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.883+0000 7fdd599ce700 1 -- start start 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd599ce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54111350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd599ce700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54111890 con 0x7fdd541026f0 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd52ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54111350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd52ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54111350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43062/0 (socket says 172.21.15.32:43062) 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd52ffd700 1 -- 172.21.15.32:0/3593214281 learned_addr learned my addr 172.21.15.32:0/3593214281 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd52ffd700 1 -- 172.21.15.32:0/3593214281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd440094f0 con 0x7fdd541026f0 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd52ffd700 1 --2- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54111350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdd44006b00 tx=0x7fdd44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd3b7fe700 1 -- 172.21.15.32:0/3593214281 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd440109b0 con 0x7fdd541026f0 2024-04-06T14:07:20.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd3b7fe700 1 -- 172.21.15.32:0/3593214281 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdd44024440 con 0x7fdd541026f0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd54111a90 con 0x7fdd541026f0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd3b7fe700 1 -- 172.21.15.32:0/3593214281 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd4401a5f0 con 0x7fdd541026f0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.884+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd54112600 con 0x7fdd541026f0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.885+0000 7fdd3b7fe700 1 -- 172.21.15.32:0/3593214281 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fdd4401a8b0 con 0x7fdd541026f0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.885+0000 7fdd3b7fe700 1 --2- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd3c036660 0x7fdd3c038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.885+0000 7fdd3b7fe700 1 -- 172.21.15.32:0/3593214281 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdd4404a730 con 0x7fdd541026f0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.885+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5405f5f0 con 0x7fdd541026f0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.885+0000 7fdd527fc700 1 --2- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd3c036660 0x7fdd3c038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.886+0000 7fdd527fc700 1 --2- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd3c036660 0x7fdd3c038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdd48009990 tx=0x7fdd48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.890+0000 7fdd3b7fe700 1 -- 172.21.15.32:0/3593214281 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdd44018b40 con 0x7fdd541026f0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:19.996+0000 7fdd3b7fe700 1 -- 172.21.15.32:0/3593214281 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fdd44024990 con 0x7fdd541026f0 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.101+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd540ff010 con 0x7fdd3c036660 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.108+0000 7fdd3b7fe700 1 -- 172.21.15.32:0/3593214281 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdd540ff010 con 0x7fdd3c036660 2024-04-06T14:07:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.110+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd3c036660 msgr2=0x7fdd3c038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.110+0000 7fdd599ce700 1 --2- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd3c036660 0x7fdd3c038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdd48009990 tx=0x7fdd48006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.110+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 msgr2=0x7fdd54111350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.110+0000 7fdd599ce700 1 --2- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54111350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdd44006b00 tx=0x7fdd44009250 comp rx=0 tx=0).stop 2024-04-06T14:07:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.110+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 shutdown_connections 2024-04-06T14:07:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.110+0000 7fdd599ce700 1 --2- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd3c036660 0x7fdd3c038b10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.110+0000 7fdd599ce700 1 --2- 172.21.15.32:0/3593214281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd541026f0 0x7fdd54111350 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.110+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 >> 172.21.15.32:0/3593214281 conn(0x7fdd540fde50 msgr2=0x7fdd540fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.110+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 shutdown_connections 2024-04-06T14:07:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:20.111+0000 7fdd599ce700 1 -- 172.21.15.32:0/3593214281 wait complete. 2024-04-06T14:07:21.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:21 smithi032 ceph-mon[39730]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:22.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-06T14:07:22.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.268+0000 7f529239d700 1 Processor -- start 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.268+0000 7f529239d700 1 -- start start 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.268+0000 7f529239d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c104ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.268+0000 7f529239d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c105530 con 0x7f528c076c90 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.268+0000 7f529139b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c104ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.268+0000 7f529139b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c104ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43078/0 (socket says 172.21.15.32:43078) 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.268+0000 7f529139b700 1 -- 172.21.15.32:0/4213484259 learned_addr learned my addr 172.21.15.32:0/4213484259 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.268+0000 7f529139b700 1 -- 172.21.15.32:0/4213484259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c105670 con 0x7f528c076c90 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.269+0000 7f529139b700 1 --2- 172.21.15.32:0/4213484259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c104ff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f52880098e0 tx=0x7f5288009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37bce408454b006e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.269+0000 7f5283fff700 1 -- 172.21.15.32:0/4213484259 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f528800b860 con 0x7f528c076c90 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.269+0000 7f5283fff700 1 -- 172.21.15.32:0/4213484259 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f528800be30 con 0x7f528c076c90 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.269+0000 7f5283fff700 1 -- 172.21.15.32:0/4213484259 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5288010380 con 0x7f528c076c90 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.269+0000 7f529239d700 1 -- 172.21.15.32:0/4213484259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 msgr2=0x7f528c104ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.269+0000 7f529239d700 1 --2- 172.21.15.32:0/4213484259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c104ff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f52880098e0 tx=0x7f5288009250 comp rx=0 tx=0).stop 2024-04-06T14:07:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.269+0000 7f529239d700 1 -- 172.21.15.32:0/4213484259 shutdown_connections 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.269+0000 7f529239d700 1 --2- 172.21.15.32:0/4213484259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c104ff0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.269+0000 7f529239d700 1 -- 172.21.15.32:0/4213484259 >> 172.21.15.32:0/4213484259 conn(0x7f528c0fca60 msgr2=0x7f528c0fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.270+0000 7f529239d700 1 -- 172.21.15.32:0/4213484259 shutdown_connections 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.270+0000 7f529239d700 1 -- 172.21.15.32:0/4213484259 wait complete. 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.270+0000 7f529239d700 1 Processor -- start 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.270+0000 7f529239d700 1 -- start start 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.270+0000 7f529239d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c102940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.270+0000 7f529239d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c102e80 con 0x7f528c076c90 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.270+0000 7f529139b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.270+0000 7f529139b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43094/0 (socket says 172.21.15.32:43094) 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.270+0000 7f529139b700 1 -- 172.21.15.32:0/2037509280 learned_addr learned my addr 172.21.15.32:0/2037509280 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.271+0000 7f529139b700 1 -- 172.21.15.32:0/2037509280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52880094f0 con 0x7f528c076c90 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.271+0000 7f529139b700 1 --2- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c102940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f528800b440 tx=0x7f5288009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.271+0000 7f5281ffb700 1 -- 172.21.15.32:0/2037509280 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5288010960 con 0x7f528c076c90 2024-04-06T14:07:22.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.271+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c100ef0 con 0x7f528c076c90 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.271+0000 7f5281ffb700 1 -- 172.21.15.32:0/2037509280 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f528801a430 con 0x7f528c076c90 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.271+0000 7f5281ffb700 1 -- 172.21.15.32:0/2037509280 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52880245b0 con 0x7f528c076c90 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.271+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528c1013e0 con 0x7f528c076c90 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.272+0000 7f5281ffb700 1 -- 172.21.15.32:0/2037509280 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f52880247f0 con 0x7f528c076c90 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.272+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528c04bed0 con 0x7f528c076c90 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.272+0000 7f5281ffb700 1 --2- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f5278036790 0x7f5278038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.272+0000 7f5281ffb700 1 -- 172.21.15.32:0/2037509280 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f528804a940 con 0x7f528c076c90 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.272+0000 7f5290b9a700 1 --2- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f5278036790 0x7f5278038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.273+0000 7f5290b9a700 1 --2- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f5278036790 0x7f5278038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f527c009990 tx=0x7f527c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.277+0000 7f5281ffb700 1 -- 172.21.15.32:0/2037509280 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f528802c340 con 0x7f528c076c90 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.472+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f528c101a70 con 0x7f5278036790 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.472+0000 7f5281ffb700 1 -- 172.21.15.32:0/2037509280 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f528c101a70 con 0x7f5278036790 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f5278036790 msgr2=0x7f5278038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 --2- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f5278036790 0x7f5278038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f527c009990 tx=0x7f527c006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 msgr2=0x7f528c102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 --2- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c102940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f528800b440 tx=0x7f5288009250 comp rx=0 tx=0).stop 2024-04-06T14:07:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 shutdown_connections 2024-04-06T14:07:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 --2- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f5278036790 0x7f5278038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 --2- 172.21.15.32:0/2037509280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f528c076c90 0x7f528c102940 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 >> 172.21.15.32:0/2037509280 conn(0x7f528c0fca60 msgr2=0x7f528c0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 shutdown_connections 2024-04-06T14:07:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:21.474+0000 7f529239d700 1 -- 172.21.15.32:0/2037509280 wait complete. 2024-04-06T14:07:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:Generating ssh key... 2024-04-06T14:07:22.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:22 smithi032 ceph-mon[39730]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:23.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.671+0000 7f009d4a0700 1 Processor -- start 2024-04-06T14:07:23.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.671+0000 7f009d4a0700 1 -- start start 2024-04-06T14:07:23.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.671+0000 7f009d4a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f0098100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:23.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.671+0000 7f009d4a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0098101260 con 0x7f0098100910 2024-04-06T14:07:23.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.671+0000 7f0096ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f0098100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:23.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.671+0000 7f0096ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f0098100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43102/0 (socket says 172.21.15.32:43102) 2024-04-06T14:07:23.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.671+0000 7f0096ffd700 1 -- 172.21.15.32:0/1031221240 learned_addr learned my addr 172.21.15.32:0/1031221240 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:23.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.671+0000 7f0096ffd700 1 -- 172.21.15.32:0/1031221240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00981013a0 con 0x7f0098100910 2024-04-06T14:07:23.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.671+0000 7f0096ffd700 1 --2- 172.21.15.32:0/1031221240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f0098100d20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f00880098e0 tx=0x7f0088009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44a8041fd2e4353 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:23.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f0095ffb700 1 -- 172.21.15.32:0/1031221240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f008800b860 con 0x7f0098100910 2024-04-06T14:07:23.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f0095ffb700 1 -- 172.21.15.32:0/1031221240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f008800be30 con 0x7f0098100910 2024-04-06T14:07:23.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f0095ffb700 1 -- 172.21.15.32:0/1031221240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0088010380 con 0x7f0098100910 2024-04-06T14:07:23.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f009d4a0700 1 -- 172.21.15.32:0/1031221240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 msgr2=0x7f0098100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:23.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f009d4a0700 1 --2- 172.21.15.32:0/1031221240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f0098100d20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f00880098e0 tx=0x7f0088009250 comp rx=0 tx=0).stop 2024-04-06T14:07:23.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f009d4a0700 1 -- 172.21.15.32:0/1031221240 shutdown_connections 2024-04-06T14:07:23.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f009d4a0700 1 --2- 172.21.15.32:0/1031221240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f0098100d20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:23.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f009d4a0700 1 -- 172.21.15.32:0/1031221240 >> 172.21.15.32:0/1031221240 conn(0x7f00980fc400 msgr2=0x7f00980fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:23.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f009d4a0700 1 -- 172.21.15.32:0/1031221240 shutdown_connections 2024-04-06T14:07:23.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.672+0000 7f009d4a0700 1 -- 172.21.15.32:0/1031221240 wait complete. 2024-04-06T14:07:23.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.673+0000 7f009d4a0700 1 Processor -- start 2024-04-06T14:07:23.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.673+0000 7f009d4a0700 1 -- start start 2024-04-06T14:07:23.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.673+0000 7f009d4a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f00981115d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:23.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.673+0000 7f009d4a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0098111b10 con 0x7f0098100910 2024-04-06T14:07:23.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.673+0000 7f0096ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f00981115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:23.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.673+0000 7f0096ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f00981115d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43116/0 (socket says 172.21.15.32:43116) 2024-04-06T14:07:23.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.673+0000 7f0096ffd700 1 -- 172.21.15.32:0/3706743986 learned_addr learned my addr 172.21.15.32:0/3706743986 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.674+0000 7f0096ffd700 1 -- 172.21.15.32:0/3706743986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00880094f0 con 0x7f0098100910 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.674+0000 7f0096ffd700 1 --2- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f00981115d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0088000c00 tx=0x7f0088009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.674+0000 7f007f7fe700 1 -- 172.21.15.32:0/3706743986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0088010960 con 0x7f0098100910 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.674+0000 7f007f7fe700 1 -- 172.21.15.32:0/3706743986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0088024440 con 0x7f0098100910 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.674+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0098111d10 con 0x7f0098100910 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.674+0000 7f007f7fe700 1 -- 172.21.15.32:0/3706743986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f008801a5f0 con 0x7f0098100910 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.674+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0098112880 con 0x7f0098100910 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.675+0000 7f007f7fe700 1 -- 172.21.15.32:0/3706743986 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f0088010ac0 con 0x7f0098100910 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.675+0000 7f007f7fe700 1 --2- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f00800367e0 0x7f0080038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:23.674 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.675+0000 7f007f7fe700 1 -- 172.21.15.32:0/3706743986 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f008804a630 con 0x7f0098100910 2024-04-06T14:07:23.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.675+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009805f5f0 con 0x7f0098100910 2024-04-06T14:07:23.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.675+0000 7f00967fc700 1 --2- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f00800367e0 0x7f0080038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:23.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.676+0000 7f00967fc700 1 --2- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f00800367e0 0x7f0080038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f008c009990 tx=0x7f008c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:23.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.680+0000 7f007f7fe700 1 -- 172.21.15.32:0/3706743986 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0088010d70 con 0x7f0098100910 2024-04-06T14:07:23.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.877+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f00981046c0 con 0x7f00800367e0 2024-04-06T14:07:23.675 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.988+0000 7f007f7fe700 1 -- 172.21.15.32:0/3706743986 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f00981046c0 con 0x7f00800367e0 2024-04-06T14:07:23.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f00800367e0 msgr2=0x7f0080038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:23.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 --2- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f00800367e0 0x7f0080038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f008c009990 tx=0x7f008c006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:23.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 msgr2=0x7f00981115d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:23.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 --2- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f00981115d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0088000c00 tx=0x7f0088009250 comp rx=0 tx=0).stop 2024-04-06T14:07:23.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 shutdown_connections 2024-04-06T14:07:23.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 --2- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f00800367e0 0x7f0080038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:23.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 --2- 172.21.15.32:0/3706743986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0098100910 0x7f00981115d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:23.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 >> 172.21.15.32:0/3706743986 conn(0x7f00980fc400 msgr2=0x7f00980fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:23.676 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 shutdown_connections 2024-04-06T14:07:23.677 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:22.990+0000 7f009d4a0700 1 -- 172.21.15.32:0/3706743986 wait complete. 2024-04-06T14:07:24.423 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:23 smithi032 ceph-mon[39730]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:24.423 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:23 smithi032 ceph-mon[39730]: Generating ssh key... 2024-04-06T14:07:24.423 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:23 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:24.423 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:23 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPeCi1pyfhH1qnE/PcnFIdg25tUsYGZMRowYE/EclDHwlT101ohI3EZxuYD1b78o2g6tcj4rc6okVdeag5IxGVKhcQtP9fN+Pe3OQNJZDRmnl6O2MShnAcYkmi7wCIcXZoL8h8/MGEU75LY9ff5DB9PwcrXTo1CQfOlvyxhqW4r/yeMsdHr/VFfRGZ64RG1vutQP02iaK/6SbPkb0d9OyR8Ob/E0V2oXZ65CSIgcJNkhUEhtohFARB+rqgzEqTyioOxqki0/GnerikgkmNxJlJ7D7icy01yZ7p7DE1ZJvTRYsW4sHg0jiXlZUjBkoodYWQ3XlByVViLk3EuJ0dlSgyY7ca1dGTPCmXRqOJI1EZSwoxQ2e/8bQUUmn5c5Ccxh3srWKXGmeCUbn2pRJo6+eJznUrYiycjxWqb4YiD0/B/vUz5mojqPlJYUQmiP1zugAfPwkonzWoHxYWTFA093QfYxB7FjcbSPYMf6aBcqUf+SZrnwSG+lsVeJxW3F2Xus= ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.188+0000 7f1aaa73c700 1 Processor -- start 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.188+0000 7f1aaa73c700 1 -- start start 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.189+0000 7f1aaa73c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4107630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.189+0000 7f1aaa73c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa4103020 con 0x7f1aa4076c00 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.189+0000 7f1aa3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4107630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.189+0000 7f1aa3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4107630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43128/0 (socket says 172.21.15.32:43128) 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.189+0000 7f1aa3fff700 1 -- 172.21.15.32:0/2794180979 learned_addr learned my addr 172.21.15.32:0/2794180979 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.189+0000 7f1aa3fff700 1 -- 172.21.15.32:0/2794180979 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa4107b70 con 0x7f1aa4076c00 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.189+0000 7f1aa3fff700 1 --2- 172.21.15.32:0/2794180979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4107630 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a940098e0 tx=0x7f1a94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459e87a43f3e761 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:25.031 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.189+0000 7f1aa2ffd700 1 -- 172.21.15.32:0/2794180979 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9400b860 con 0x7f1aa4076c00 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.190+0000 7f1aa2ffd700 1 -- 172.21.15.32:0/2794180979 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1a9400be30 con 0x7f1aa4076c00 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.190+0000 7f1aa2ffd700 1 -- 172.21.15.32:0/2794180979 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a94010380 con 0x7f1aa4076c00 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.190+0000 7f1aaa73c700 1 -- 172.21.15.32:0/2794180979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 msgr2=0x7f1aa4107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.190+0000 7f1aaa73c700 1 --2- 172.21.15.32:0/2794180979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4107630 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a940098e0 tx=0x7f1a94009250 comp rx=0 tx=0).stop 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.190+0000 7f1aaa73c700 1 -- 172.21.15.32:0/2794180979 shutdown_connections 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.190+0000 7f1aaa73c700 1 --2- 172.21.15.32:0/2794180979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4107630 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.190+0000 7f1aaa73c700 1 -- 172.21.15.32:0/2794180979 >> 172.21.15.32:0/2794180979 conn(0x7f1aa40fcc60 msgr2=0x7f1aa40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.190+0000 7f1aaa73c700 1 -- 172.21.15.32:0/2794180979 shutdown_connections 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.190+0000 7f1aaa73c700 1 -- 172.21.15.32:0/2794180979 wait complete. 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.191+0000 7f1aaa73c700 1 Processor -- start 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.191+0000 7f1aaa73c700 1 -- start start 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.191+0000 7f1aaa73c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.191+0000 7f1aaa73c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa4113ab0 con 0x7f1aa4076c00 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.191+0000 7f1aa3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:25.032 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.191+0000 7f1aa3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43136/0 (socket says 172.21.15.32:43136) 2024-04-06T14:07:25.033 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.191+0000 7f1aa3fff700 1 -- 172.21.15.32:0/1378066910 learned_addr learned my addr 172.21.15.32:0/1378066910 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:25.033 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.191+0000 7f1aa3fff700 1 -- 172.21.15.32:0/1378066910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a940094f0 con 0x7f1aa4076c00 2024-04-06T14:07:25.033 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.191+0000 7f1aa3fff700 1 --2- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4113570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1a94006d40 tx=0x7f1a94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:25.033 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.192+0000 7f1aa0ff9700 1 -- 172.21.15.32:0/1378066910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a940109b0 con 0x7f1aa4076c00 2024-04-06T14:07:25.033 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.192+0000 7f1aa0ff9700 1 -- 172.21.15.32:0/1378066910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1a94024440 con 0x7f1aa4076c00 2024-04-06T14:07:25.033 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.192+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa4113cb0 con 0x7f1aa4076c00 2024-04-06T14:07:25.033 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.192+0000 7f1aa0ff9700 1 -- 172.21.15.32:0/1378066910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9401a5f0 con 0x7f1aa4076c00 2024-04-06T14:07:25.033 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.192+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa4114820 con 0x7f1aa4076c00 2024-04-06T14:07:25.033 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.193+0000 7f1aa0ff9700 1 -- 172.21.15.32:0/1378066910 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f1a9401a820 con 0x7f1aa4076c00 2024-04-06T14:07:25.033 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:24 smithi032 ceph-mon[39730]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:25.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.193+0000 7f1aa0ff9700 1 --2- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f1a8c0367e0 0x7f1a8c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:25.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.193+0000 7f1aa0ff9700 1 -- 172.21.15.32:0/1378066910 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1a9404a720 con 0x7f1aa4076c00 2024-04-06T14:07:25.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.193+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa405f5f0 con 0x7f1aa4076c00 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.193+0000 7f1aa37fe700 1 --2- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f1a8c0367e0 0x7f1a8c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.193+0000 7f1aa37fe700 1 --2- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f1a8c0367e0 0x7f1a8c038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009990 tx=0x7f1a98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.198+0000 7f1aa0ff9700 1 -- 172.21.15.32:0/1378066910 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1a9401aad0 con 0x7f1aa4076c00 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.393+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1aa40fde70 con 0x7f1a8c0367e0 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.394+0000 7f1aa0ff9700 1 -- 172.21.15.32:0/1378066910 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f1aa40fde70 con 0x7f1a8c0367e0 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f1a8c0367e0 msgr2=0x7f1a8c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 --2- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f1a8c0367e0 0x7f1a8c038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009990 tx=0x7f1a98006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 msgr2=0x7f1aa4113570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 --2- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4113570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1a94006d40 tx=0x7f1a94009250 comp rx=0 tx=0).stop 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 shutdown_connections 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 --2- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f1a8c0367e0 0x7f1a8c038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 --2- 172.21.15.32:0/1378066910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1aa4076c00 0x7f1aa4113570 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 >> 172.21.15.32:0/1378066910 conn(0x7f1aa40fcc60 msgr2=0x7f1aa40fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:25.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 shutdown_connections 2024-04-06T14:07:25.037 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:24.396+0000 7f1aaa73c700 1 -- 172.21.15.32:0/1378066910 wait complete. 2024-04-06T14:07:25.037 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-06T14:07:25.037 INFO:teuthology.orchestra.run.smithi032.stdout:Adding key to root@localhost authorized_keys... 2024-04-06T14:07:25.037 INFO:teuthology.orchestra.run.smithi032.stdout:Adding host smithi032... 2024-04-06T14:07:26.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:25 smithi032 ceph-mon[39730]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:27.263 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:26 smithi032 ceph-mon[39730]: Deploying cephadm binary to smithi032 2024-04-06T14:07:28.282 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Added host 'smithi032' with addr '172.21.15.32' 2024-04-06T14:07:28.282 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.609+0000 7fe8517c2700 1 Processor -- start 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.609+0000 7fe8517c2700 1 -- start start 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe8517c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe8517c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c071500 con 0x7fe84c102be0 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe84affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe84affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c102ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35814/0 (socket says 172.21.15.32:35814) 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe84affd700 1 -- 172.21.15.32:0/3261443033 learned_addr learned my addr 172.21.15.32:0/3261443033 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe84affd700 1 -- 172.21.15.32:0/3261443033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c103530 con 0x7fe84c102be0 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe84affd700 1 --2- 172.21.15.32:0/3261443033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c102ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0098e0 tx=0x7fe83c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89a85e7cf35347 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe849ffb700 1 -- 172.21.15.32:0/3261443033 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83c00b860 con 0x7fe84c102be0 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe849ffb700 1 -- 172.21.15.32:0/3261443033 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe83c00be30 con 0x7fe84c102be0 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.610+0000 7fe849ffb700 1 -- 172.21.15.32:0/3261443033 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83c010380 con 0x7fe84c102be0 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.611+0000 7fe8517c2700 1 -- 172.21.15.32:0/3261443033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 msgr2=0x7fe84c102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.611+0000 7fe8517c2700 1 --2- 172.21.15.32:0/3261443033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c102ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0098e0 tx=0x7fe83c009250 comp rx=0 tx=0).stop 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.611+0000 7fe8517c2700 1 -- 172.21.15.32:0/3261443033 shutdown_connections 2024-04-06T14:07:28.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.611+0000 7fe8517c2700 1 --2- 172.21.15.32:0/3261443033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c102ff0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.611+0000 7fe8517c2700 1 -- 172.21.15.32:0/3261443033 >> 172.21.15.32:0/3261443033 conn(0x7fe84c0fc3d0 msgr2=0x7fe84c0fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.611+0000 7fe8517c2700 1 -- 172.21.15.32:0/3261443033 shutdown_connections 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.611+0000 7fe8517c2700 1 -- 172.21.15.32:0/3261443033 wait complete. 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.612+0000 7fe8517c2700 1 Processor -- start 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.612+0000 7fe8517c2700 1 -- start start 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.612+0000 7fe8517c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.612+0000 7fe8517c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c113ac0 con 0x7fe84c102be0 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.612+0000 7fe84affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.612+0000 7fe84affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c113580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35822/0 (socket says 172.21.15.32:35822) 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.612+0000 7fe84affd700 1 -- 172.21.15.32:0/985808478 learned_addr learned my addr 172.21.15.32:0/985808478 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.612+0000 7fe84affd700 1 -- 172.21.15.32:0/985808478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c0094f0 con 0x7fe84c102be0 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.613+0000 7fe84affd700 1 --2- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c113580 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe83c003080 tx=0x7fe83c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.613+0000 7fe8437fe700 1 -- 172.21.15.32:0/985808478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83c0109b0 con 0x7fe84c102be0 2024-04-06T14:07:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.613+0000 7fe8437fe700 1 -- 172.21.15.32:0/985808478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe83c01a430 con 0x7fe84c102be0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.613+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84c113cc0 con 0x7fe84c102be0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.613+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84c114830 con 0x7fe84c102be0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.613+0000 7fe8437fe700 1 -- 172.21.15.32:0/985808478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83c0245b0 con 0x7fe84c102be0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.614+0000 7fe8437fe700 1 -- 172.21.15.32:0/985808478 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe83c0247f0 con 0x7fe84c102be0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.614+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82c0052e0 con 0x7fe84c102be0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.614+0000 7fe8437fe700 1 --2- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fe838036790 0x7fe838038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.614+0000 7fe8437fe700 1 -- 172.21.15.32:0/985808478 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe83c04a710 con 0x7fe84c102be0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.614+0000 7fe84a7fc700 1 --2- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fe838036790 0x7fe838038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.614+0000 7fe84a7fc700 1 --2- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fe838036790 0x7fe838038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe834009990 tx=0x7fe834006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.619+0000 7fe8437fe700 1 -- 172.21.15.32:0/985808478 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe83c02c990 con 0x7fe84c102be0 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:25.817+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}) v1 -- 0x7fe82c000bf0 con 0x7fe838036790 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.609+0000 7fe8437fe700 1 -- 172.21.15.32:0/985808478 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fe82c000bf0 con 0x7fe838036790 2024-04-06T14:07:28.285 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.611+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fe838036790 msgr2=0x7fe838038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.611+0000 7fe8517c2700 1 --2- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fe838036790 0x7fe838038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe834009990 tx=0x7fe834006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.611+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 msgr2=0x7fe84c113580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.611+0000 7fe8517c2700 1 --2- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c113580 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe83c003080 tx=0x7fe83c009250 comp rx=0 tx=0).stop 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.611+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 shutdown_connections 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.611+0000 7fe8517c2700 1 --2- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fe838036790 0x7fe838038c40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.611+0000 7fe8517c2700 1 --2- 172.21.15.32:0/985808478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c102be0 0x7fe84c113580 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.611+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 >> 172.21.15.32:0/985808478 conn(0x7fe84c0fc3d0 msgr2=0x7fe84c0fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.612+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 shutdown_connections 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:27.612+0000 7fe8517c2700 1 -- 172.21.15.32:0/985808478 wait complete. 2024-04-06T14:07:28.286 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying mon service with default placement... 2024-04-06T14:07:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:28 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:28 smithi032 ceph-mon[39730]: Added host smithi032 2024-04-06T14:07:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:28 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:07:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:30 smithi032 ceph-mon[39730]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:30 smithi032 ceph-mon[39730]: Saving service mon spec with placement count:5 2024-04-06T14:07:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:30 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:30.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-06T14:07:30.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.836+0000 7fecfee9d700 1 Processor -- start 2024-04-06T14:07:30.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.836+0000 7fecfee9d700 1 -- start start 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.836+0000 7fecfee9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf81051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.836+0000 7fecfee9d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf8105710 con 0x7fecf8076c60 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.837+0000 7fecfcc39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf81051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.837+0000 7fecfcc39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf81051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35826/0 (socket says 172.21.15.32:35826) 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.837+0000 7fecfcc39700 1 -- 172.21.15.32:0/2959255437 learned_addr learned my addr 172.21.15.32:0/2959255437 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.837+0000 7fecfcc39700 1 -- 172.21.15.32:0/2959255437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf8105850 con 0x7fecf8076c60 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.837+0000 7fecfcc39700 1 --2- 172.21.15.32:0/2959255437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf81051d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fecec0098e0 tx=0x7fecec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1078d5689a9ceb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.837+0000 7fecf77fe700 1 -- 172.21.15.32:0/2959255437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecec00b860 con 0x7fecf8076c60 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.837+0000 7fecf77fe700 1 -- 172.21.15.32:0/2959255437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fecec00be30 con 0x7fecf8076c60 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.837+0000 7fecf77fe700 1 -- 172.21.15.32:0/2959255437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecec010380 con 0x7fecf8076c60 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.838+0000 7fecfee9d700 1 -- 172.21.15.32:0/2959255437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 msgr2=0x7fecf81051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.838+0000 7fecfee9d700 1 --2- 172.21.15.32:0/2959255437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf81051d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fecec0098e0 tx=0x7fecec009250 comp rx=0 tx=0).stop 2024-04-06T14:07:30.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.838+0000 7fecfee9d700 1 -- 172.21.15.32:0/2959255437 shutdown_connections 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.838+0000 7fecfee9d700 1 --2- 172.21.15.32:0/2959255437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf81051d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.838+0000 7fecfee9d700 1 -- 172.21.15.32:0/2959255437 >> 172.21.15.32:0/2959255437 conn(0x7fecf80fcc60 msgr2=0x7fecf80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.838+0000 7fecfee9d700 1 -- 172.21.15.32:0/2959255437 shutdown_connections 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.838+0000 7fecfee9d700 1 -- 172.21.15.32:0/2959255437 wait complete. 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.838+0000 7fecfee9d700 1 Processor -- start 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.838+0000 7fecfee9d700 1 -- start start 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecfee9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf8111370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecfee9d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf81118b0 con 0x7fecf8076c60 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecfcc39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf8111370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecfcc39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf8111370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35836/0 (socket says 172.21.15.32:35836) 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecfcc39700 1 -- 172.21.15.32:0/1904702634 learned_addr learned my addr 172.21.15.32:0/1904702634 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecfcc39700 1 -- 172.21.15.32:0/1904702634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecec0094f0 con 0x7fecf8076c60 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecfcc39700 1 --2- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf8111370 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fecec006b00 tx=0x7fecec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecf57fa700 1 -- 172.21.15.32:0/1904702634 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecec010a80 con 0x7fecf8076c60 2024-04-06T14:07:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecf57fa700 1 -- 172.21.15.32:0/1904702634 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fecec01a430 con 0x7fecf8076c60 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf8111ab0 con 0x7fecf8076c60 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecf57fa700 1 -- 172.21.15.32:0/1904702634 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecec0245b0 con 0x7fecf8076c60 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.839+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf8112620 con 0x7fecf8076c60 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.840+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf805f5f0 con 0x7fecf8076c60 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.841+0000 7fecf57fa700 1 -- 172.21.15.32:0/1904702634 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fecec010be0 con 0x7fecf8076c60 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.841+0000 7fecf57fa700 1 --2- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fece80367e0 0x7fece8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.841+0000 7fecf57fa700 1 -- 172.21.15.32:0/1904702634 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fecec04b180 con 0x7fecf8076c60 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.841+0000 7fecf7fff700 1 --2- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fece80367e0 0x7fece8038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.841+0000 7fecf7fff700 1 --2- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fece80367e0 0x7fece8038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fece4009990 tx=0x7fece4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:28.845+0000 7fecf57fa700 1 -- 172.21.15.32:0/1904702634 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fecec010e90 con 0x7fecf8076c60 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.048+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fecf80fde50 con 0x7fece80367e0 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.052+0000 7fecf57fa700 1 -- 172.21.15.32:0/1904702634 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fecf80fde50 con 0x7fece80367e0 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fece80367e0 msgr2=0x7fece8038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 --2- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fece80367e0 0x7fece8038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fece4009990 tx=0x7fece4006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 msgr2=0x7fecf8111370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:30.277 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 --2- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf8111370 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fecec006b00 tx=0x7fecec009250 comp rx=0 tx=0).stop 2024-04-06T14:07:30.277 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 shutdown_connections 2024-04-06T14:07:30.277 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 --2- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fece80367e0 0x7fece8038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:30.277 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 --2- 172.21.15.32:0/1904702634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8076c60 0x7fecf8111370 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:30.277 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 >> 172.21.15.32:0/1904702634 conn(0x7fecf80fcc60 msgr2=0x7fecf80fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:30.277 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 shutdown_connections 2024-04-06T14:07:30.277 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:29.054+0000 7fecfee9d700 1 -- 172.21.15.32:0/1904702634 wait complete. 2024-04-06T14:07:30.277 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying mgr service with default placement... 2024-04-06T14:07:32.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-06T14:07:32.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.066+0000 7f794b6c0700 1 Processor -- start 2024-04-06T14:07:32.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.066+0000 7f794b6c0700 1 -- start start 2024-04-06T14:07:32.188 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.066+0000 7f794b6c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.066+0000 7f794b6c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7944102ed0 con 0x7f7944102580 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.067+0000 7f794945c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.067+0000 7f794945c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35852/0 (socket says 172.21.15.32:35852) 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.067+0000 7f794945c700 1 -- 172.21.15.32:0/102013541 learned_addr learned my addr 172.21.15.32:0/102013541 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.067+0000 7f794945c700 1 -- 172.21.15.32:0/102013541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7944103010 con 0x7f7944102580 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.067+0000 7f794945c700 1 --2- 172.21.15.32:0/102013541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944102990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79340098e0 tx=0x7f7934009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c24588c3bd3e9d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.067+0000 7f793bfff700 1 -- 172.21.15.32:0/102013541 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f793400b860 con 0x7f7944102580 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.067+0000 7f793bfff700 1 -- 172.21.15.32:0/102013541 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f793400be30 con 0x7f7944102580 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.067+0000 7f793bfff700 1 -- 172.21.15.32:0/102013541 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7934010380 con 0x7f7944102580 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.068+0000 7f794b6c0700 1 -- 172.21.15.32:0/102013541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 msgr2=0x7f7944102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.068+0000 7f794b6c0700 1 --2- 172.21.15.32:0/102013541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944102990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79340098e0 tx=0x7f7934009250 comp rx=0 tx=0).stop 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.068+0000 7f794b6c0700 1 -- 172.21.15.32:0/102013541 shutdown_connections 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.068+0000 7f794b6c0700 1 --2- 172.21.15.32:0/102013541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944102990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.068+0000 7f794b6c0700 1 -- 172.21.15.32:0/102013541 >> 172.21.15.32:0/102013541 conn(0x7f79440fdd00 msgr2=0x7f7944100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:32.189 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.068+0000 7f794b6c0700 1 -- 172.21.15.32:0/102013541 shutdown_connections 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.068+0000 7f794b6c0700 1 -- 172.21.15.32:0/102013541 wait complete. 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.068+0000 7f794b6c0700 1 Processor -- start 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.068+0000 7f794b6c0700 1 -- start start 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f794b6c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944111340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f794b6c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7944111880 con 0x7f7944102580 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f794945c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944111340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f794945c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944111340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35858/0 (socket says 172.21.15.32:35858) 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f794945c700 1 -- 172.21.15.32:0/4256160771 learned_addr learned my addr 172.21.15.32:0/4256160771 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f794945c700 1 -- 172.21.15.32:0/4256160771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79340094f0 con 0x7f7944102580 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f794945c700 1 --2- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944111340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7934000c00 tx=0x7f7934009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f7939ffb700 1 -- 172.21.15.32:0/4256160771 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7934010a80 con 0x7f7944102580 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f7939ffb700 1 -- 172.21.15.32:0/4256160771 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f793401a430 con 0x7f7944102580 2024-04-06T14:07:32.190 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7944111a80 con 0x7f7944102580 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.069+0000 7f7939ffb700 1 -- 172.21.15.32:0/4256160771 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79340245b0 con 0x7f7944102580 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.070+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79441125f0 con 0x7f7944102580 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.070+0000 7f7939ffb700 1 -- 172.21.15.32:0/4256160771 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f79340247a0 con 0x7f7944102580 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.070+0000 7f7939ffb700 1 --2- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f79300367e0 0x7f7930038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.070+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794405f5f0 con 0x7f7944102580 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.071+0000 7f7939ffb700 1 -- 172.21.15.32:0/4256160771 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f793404ad00 con 0x7f7944102580 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.071+0000 7f7948c5b700 1 --2- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f79300367e0 0x7f7930038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.071+0000 7f7948c5b700 1 --2- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f79300367e0 0x7f7930038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7940009990 tx=0x7f7940006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.075+0000 7f7939ffb700 1 -- 172.21.15.32:0/4256160771 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7934023750 con 0x7f7944102580 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.283+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f79440fef40 con 0x7f79300367e0 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.288+0000 7f7939ffb700 1 -- 172.21.15.32:0/4256160771 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f79440fef40 con 0x7f79300367e0 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f79300367e0 msgr2=0x7f7930038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 --2- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f79300367e0 0x7f7930038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7940009990 tx=0x7f7940006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 msgr2=0x7f7944111340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:32.191 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 --2- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944111340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7934000c00 tx=0x7f7934009250 comp rx=0 tx=0).stop 2024-04-06T14:07:32.192 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 shutdown_connections 2024-04-06T14:07:32.192 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 --2- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f79300367e0 0x7f7930038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:32.192 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 --2- 172.21.15.32:0/4256160771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7944102580 0x7f7944111340 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:32.192 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 >> 172.21.15.32:0/4256160771 conn(0x7f79440fdd00 msgr2=0x7f79440fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:32.192 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 shutdown_connections 2024-04-06T14:07:32.192 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:31.290+0000 7f794b6c0700 1 -- 172.21.15.32:0/4256160771 wait complete. 2024-04-06T14:07:32.192 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying crash service with default placement... 2024-04-06T14:07:32.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:32 smithi032 ceph-mon[39730]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:32.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:32 smithi032 ceph-mon[39730]: Saving service mgr spec with placement count:2 2024-04-06T14:07:32.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:32 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:32.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:32 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:32.449 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:32 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:33.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-06T14:07:33.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.693+0000 7fdcdce4b700 1 Processor -- start 2024-04-06T14:07:33.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.693+0000 7fdcdce4b700 1 -- start start 2024-04-06T14:07:33.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.693+0000 7fdcdce4b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:33.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.694+0000 7fdcdce4b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd8101e30 con 0x7fdcd81014e0 2024-04-06T14:07:33.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.694+0000 7fdcd659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:33.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.694+0000 7fdcd659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35864/0 (socket says 172.21.15.32:35864) 2024-04-06T14:07:33.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.694+0000 7fdcd659c700 1 -- 172.21.15.32:0/1237311923 learned_addr learned my addr 172.21.15.32:0/1237311923 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.694+0000 7fdcd659c700 1 -- 172.21.15.32:0/1237311923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd8101f70 con 0x7fdcd81014e0 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.694+0000 7fdcd659c700 1 --2- 172.21.15.32:0/1237311923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81018f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80098e0 tx=0x7fdcc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3804709276e43e9b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.694+0000 7fdcd559a700 1 -- 172.21.15.32:0/1237311923 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc800b860 con 0x7fdcd81014e0 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.694+0000 7fdcd559a700 1 -- 172.21.15.32:0/1237311923 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcc800be70 con 0x7fdcd81014e0 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.694+0000 7fdcd559a700 1 -- 172.21.15.32:0/1237311923 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc8010380 con 0x7fdcd81014e0 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.695+0000 7fdcdce4b700 1 -- 172.21.15.32:0/1237311923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 msgr2=0x7fdcd81018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.695+0000 7fdcdce4b700 1 --2- 172.21.15.32:0/1237311923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81018f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80098e0 tx=0x7fdcc8009250 comp rx=0 tx=0).stop 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.695+0000 7fdcdce4b700 1 -- 172.21.15.32:0/1237311923 shutdown_connections 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.695+0000 7fdcdce4b700 1 --2- 172.21.15.32:0/1237311923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81018f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.695+0000 7fdcdce4b700 1 -- 172.21.15.32:0/1237311923 >> 172.21.15.32:0/1237311923 conn(0x7fdcd80fcc60 msgr2=0x7fdcd80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.695+0000 7fdcdce4b700 1 -- 172.21.15.32:0/1237311923 shutdown_connections 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.695+0000 7fdcdce4b700 1 -- 172.21.15.32:0/1237311923 wait complete. 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.695+0000 7fdcdce4b700 1 Processor -- start 2024-04-06T14:07:33.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.695+0000 7fdcdce4b700 1 -- start start 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcdce4b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81076f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcdce4b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd8107c30 con 0x7fdcd81014e0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcd659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcd659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81076f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35866/0 (socket says 172.21.15.32:35866) 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcd659c700 1 -- 172.21.15.32:0/3603730810 learned_addr learned my addr 172.21.15.32:0/3603730810 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcd659c700 1 -- 172.21.15.32:0/3603730810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc80094f0 con 0x7fdcd81014e0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcd659c700 1 --2- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81076f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c010 tx=0x7fdcc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcceffd700 1 -- 172.21.15.32:0/3603730810 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc8010960 con 0x7fdcd81014e0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcceffd700 1 -- 172.21.15.32:0/3603730810 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcc801a430 con 0x7fdcd81014e0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd8105cf0 con 0x7fdcd81014e0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcceffd700 1 -- 172.21.15.32:0/3603730810 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc8019580 con 0x7fdcd81014e0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.696+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd81061e0 con 0x7fdcd81014e0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.697+0000 7fdcceffd700 1 -- 172.21.15.32:0/3603730810 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fdcc8019800 con 0x7fdcd81014e0 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.697+0000 7fdcceffd700 1 --2- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdcc4036790 0x7fdcc4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.697+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd804bed0 con 0x7fdcd81014e0 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.697+0000 7fdcceffd700 1 -- 172.21.15.32:0/3603730810 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdcc804a760 con 0x7fdcd81014e0 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.697+0000 7fdcd5d9b700 1 --2- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdcc4036790 0x7fdcc4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.698+0000 7fdcd5d9b700 1 --2- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdcc4036790 0x7fdcc4038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009990 tx=0x7fdcc0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.702+0000 7fdcceffd700 1 -- 172.21.15.32:0/3603730810 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdcc802b4e0 con 0x7fdcd81014e0 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.914+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcd810b3d0 con 0x7fdcc4036790 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.918+0000 7fdcceffd700 1 -- 172.21.15.32:0/3603730810 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fdcd810b3d0 con 0x7fdcc4036790 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdcc4036790 msgr2=0x7fdcc4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 --2- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdcc4036790 0x7fdcc4038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009990 tx=0x7fdcc0006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 msgr2=0x7fdcd81076f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 --2- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81076f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c010 tx=0x7fdcc8009250 comp rx=0 tx=0).stop 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 shutdown_connections 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 --2- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdcc4036790 0x7fdcc4038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 --2- 172.21.15.32:0/3603730810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcd81014e0 0x7fdcd81076f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 >> 172.21.15.32:0/3603730810 conn(0x7fdcd80fcc60 msgr2=0x7fdcd80fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:33.967 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 shutdown_connections 2024-04-06T14:07:33.967 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:32.920+0000 7fdcdce4b700 1 -- 172.21.15.32:0/3603730810 wait complete. 2024-04-06T14:07:33.967 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying prometheus service with default placement... 2024-04-06T14:07:34.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:33 smithi032 ceph-mon[39730]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:34.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:33 smithi032 ceph-mon[39730]: Saving service crash spec with placement * 2024-04-06T14:07:34.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:33 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:35.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:35 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:35.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:35 smithi032 ceph-mon[39730]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:35.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:35 smithi032 ceph-mon[39730]: Saving service prometheus spec with placement count:1 2024-04-06T14:07:35.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:35 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:35.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:35 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:35.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-06T14:07:35.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.481+0000 7ff7d6e22700 1 Processor -- start 2024-04-06T14:07:35.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.481+0000 7ff7d6e22700 1 -- start start 2024-04-06T14:07:35.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.481+0000 7ff7d6e22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:35.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.481+0000 7ff7d6e22700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d01056f0 con 0x7ff7d0076c80 2024-04-06T14:07:35.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.481+0000 7ff7d4bbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:35.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.481+0000 7ff7d4bbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59076/0 (socket says 172.21.15.32:59076) 2024-04-06T14:07:35.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.481+0000 7ff7d4bbe700 1 -- 172.21.15.32:0/3176266808 learned_addr learned my addr 172.21.15.32:0/3176266808 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:35.506 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.481+0000 7ff7d4bbe700 1 -- 172.21.15.32:0/3176266808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d0105830 con 0x7ff7d0076c80 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.482+0000 7ff7d4bbe700 1 --2- 172.21.15.32:0/3176266808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01051b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40098e0 tx=0x7ff7c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b58fea2f2381c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.482+0000 7ff7cf7fe700 1 -- 172.21.15.32:0/3176266808 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c400b860 con 0x7ff7d0076c80 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.482+0000 7ff7cf7fe700 1 -- 172.21.15.32:0/3176266808 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7c400be70 con 0x7ff7d0076c80 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.482+0000 7ff7cf7fe700 1 -- 172.21.15.32:0/3176266808 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c4010380 con 0x7ff7d0076c80 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.482+0000 7ff7d6e22700 1 -- 172.21.15.32:0/3176266808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 msgr2=0x7ff7d01051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.482+0000 7ff7d6e22700 1 --2- 172.21.15.32:0/3176266808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01051b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40098e0 tx=0x7ff7c4009250 comp rx=0 tx=0).stop 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.482+0000 7ff7d6e22700 1 -- 172.21.15.32:0/3176266808 shutdown_connections 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.482+0000 7ff7d6e22700 1 --2- 172.21.15.32:0/3176266808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01051b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.482+0000 7ff7d6e22700 1 -- 172.21.15.32:0/3176266808 >> 172.21.15.32:0/3176266808 conn(0x7ff7d00fcc60 msgr2=0x7ff7d00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.483+0000 7ff7d6e22700 1 -- 172.21.15.32:0/3176266808 shutdown_connections 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.483+0000 7ff7d6e22700 1 -- 172.21.15.32:0/3176266808 wait complete. 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.483+0000 7ff7d6e22700 1 Processor -- start 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.483+0000 7ff7d6e22700 1 -- start start 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.483+0000 7ff7d6e22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01113b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.483+0000 7ff7d6e22700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d01118f0 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.483+0000 7ff7d4bbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.484+0000 7ff7d4bbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59082/0 (socket says 172.21.15.32:59082) 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.484+0000 7ff7d4bbe700 1 -- 172.21.15.32:0/2701008805 learned_addr learned my addr 172.21.15.32:0/2701008805 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.484+0000 7ff7d4bbe700 1 -- 172.21.15.32:0/2701008805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c40094f0 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.484+0000 7ff7d4bbe700 1 --2- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01113b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4006d40 tx=0x7ff7c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.484+0000 7ff7cd7fa700 1 -- 172.21.15.32:0/2701008805 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c4010960 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.484+0000 7ff7cd7fa700 1 -- 172.21.15.32:0/2701008805 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7c401a430 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.484+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d0111af0 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.484+0000 7ff7cd7fa700 1 -- 172.21.15.32:0/2701008805 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c40245b0 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.484+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d0112660 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.485+0000 7ff7cd7fa700 1 -- 172.21.15.32:0/2701008805 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff7c402d3f0 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.485+0000 7ff7cd7fa700 1 --2- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7ff7c00367e0 0x7ff7c0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.485+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d005f5f0 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.485+0000 7ff7cd7fa700 1 -- 172.21.15.32:0/2701008805 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff7c404a2b0 con 0x7ff7d0076c80 2024-04-06T14:07:35.508 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.485+0000 7ff7cffff700 1 --2- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7ff7c00367e0 0x7ff7c0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.486+0000 7ff7cffff700 1 --2- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7ff7c00367e0 0x7ff7c0038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009990 tx=0x7ff7bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.490+0000 7ff7cd7fa700 1 -- 172.21.15.32:0/2701008805 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff7c402b8e0 con 0x7ff7d0076c80 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.700+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7d00fde50 con 0x7ff7c00367e0 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.705+0000 7ff7cd7fa700 1 -- 172.21.15.32:0/2701008805 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7ff7d00fde50 con 0x7ff7c00367e0 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7ff7c00367e0 msgr2=0x7ff7c0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 --2- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7ff7c00367e0 0x7ff7c0038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009990 tx=0x7ff7bc006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 msgr2=0x7ff7d01113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 --2- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01113b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4006d40 tx=0x7ff7c4009250 comp rx=0 tx=0).stop 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 shutdown_connections 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 --2- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7ff7c00367e0 0x7ff7c0038c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 --2- 172.21.15.32:0/2701008805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7d0076c80 0x7ff7d01113b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 >> 172.21.15.32:0/2701008805 conn(0x7ff7d00fcc60 msgr2=0x7ff7d00fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 shutdown_connections 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:34.707+0000 7ff7d6e22700 1 -- 172.21.15.32:0/2701008805 wait complete. 2024-04-06T14:07:35.509 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying grafana service with default placement... 2024-04-06T14:07:37.414 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-06T14:07:37.414 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.076+0000 7f20cc5c6700 1 Processor -- start 2024-04-06T14:07:37.414 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.076+0000 7f20cc5c6700 1 -- start start 2024-04-06T14:07:37.414 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.077+0000 7f20cc5c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4103000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:37.414 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.077+0000 7f20cc5c6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c4070c70 con 0x7f20c4102bf0 2024-04-06T14:07:37.414 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.077+0000 7f20ca362700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4103000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:37.414 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.077+0000 7f20ca362700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4103000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59094/0 (socket says 172.21.15.32:59094) 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.077+0000 7f20ca362700 1 -- 172.21.15.32:0/4162581460 learned_addr learned my addr 172.21.15.32:0/4162581460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.077+0000 7f20ca362700 1 -- 172.21.15.32:0/4162581460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4103540 con 0x7f20c4102bf0 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.077+0000 7f20ca362700 1 --2- 172.21.15.32:0/4162581460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4103000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f20c00098e0 tx=0x7f20c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffb1e5e9f23820c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.077+0000 7f20c9360700 1 -- 172.21.15.32:0/4162581460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20c000b860 con 0x7f20c4102bf0 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.078+0000 7f20c9360700 1 -- 172.21.15.32:0/4162581460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20c000be70 con 0x7f20c4102bf0 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.078+0000 7f20c9360700 1 -- 172.21.15.32:0/4162581460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20c0010380 con 0x7f20c4102bf0 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.078+0000 7f20cc5c6700 1 -- 172.21.15.32:0/4162581460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 msgr2=0x7f20c4103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.078+0000 7f20cc5c6700 1 --2- 172.21.15.32:0/4162581460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4103000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f20c00098e0 tx=0x7f20c0009250 comp rx=0 tx=0).stop 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.078+0000 7f20cc5c6700 1 -- 172.21.15.32:0/4162581460 shutdown_connections 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.078+0000 7f20cc5c6700 1 --2- 172.21.15.32:0/4162581460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4103000 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.078+0000 7f20cc5c6700 1 -- 172.21.15.32:0/4162581460 >> 172.21.15.32:0/4162581460 conn(0x7f20c40fc400 msgr2=0x7f20c40fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.078+0000 7f20cc5c6700 1 -- 172.21.15.32:0/4162581460 shutdown_connections 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.078+0000 7f20cc5c6700 1 -- 172.21.15.32:0/4162581460 wait complete. 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.079+0000 7f20cc5c6700 1 Processor -- start 2024-04-06T14:07:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.079+0000 7f20cc5c6700 1 -- start start 2024-04-06T14:07:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.079+0000 7f20cc5c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4113550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.079+0000 7f20cc5c6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c4113a90 con 0x7f20c4102bf0 2024-04-06T14:07:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.079+0000 7f20ca362700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4113550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.079+0000 7f20ca362700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4113550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59108/0 (socket says 172.21.15.32:59108) 2024-04-06T14:07:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.079+0000 7f20ca362700 1 -- 172.21.15.32:0/480930104 learned_addr learned my addr 172.21.15.32:0/480930104 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.080+0000 7f20ca362700 1 -- 172.21.15.32:0/480930104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c00094f0 con 0x7f20c4102bf0 2024-04-06T14:07:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.080+0000 7f20ca362700 1 --2- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4113550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20c0011ca0 tx=0x7f20c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.080+0000 7f20baffd700 1 -- 172.21.15.32:0/480930104 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20c0010850 con 0x7f20c4102bf0 2024-04-06T14:07:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.080+0000 7f20baffd700 1 -- 172.21.15.32:0/480930104 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20c0010e60 con 0x7f20c4102bf0 2024-04-06T14:07:37.416 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:37 smithi032 ceph-mon[39730]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:37.416 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:37 smithi032 ceph-mon[39730]: Saving service grafana spec with placement count:1 2024-04-06T14:07:37.417 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:37 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:37.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.080+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c4113c90 con 0x7f20c4102bf0 2024-04-06T14:07:37.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.080+0000 7f20baffd700 1 -- 172.21.15.32:0/480930104 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20c001ae20 con 0x7f20c4102bf0 2024-04-06T14:07:37.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.080+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c4114800 con 0x7f20c4102bf0 2024-04-06T14:07:37.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.081+0000 7f20baffd700 1 -- 172.21.15.32:0/480930104 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f20c002d3f0 con 0x7f20c4102bf0 2024-04-06T14:07:37.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.081+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c405f5f0 con 0x7f20c4102bf0 2024-04-06T14:07:37.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.081+0000 7f20baffd700 1 --2- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f20b00367e0 0x7f20b0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:37.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.081+0000 7f20baffd700 1 -- 172.21.15.32:0/480930104 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f20c004a4a0 con 0x7f20c4102bf0 2024-04-06T14:07:37.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.081+0000 7f20c9b61700 1 --2- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f20b00367e0 0x7f20b0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:37.419 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.082+0000 7f20c9b61700 1 --2- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f20b00367e0 0x7f20b0038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009990 tx=0x7f20b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.086+0000 7f20baffd700 1 -- 172.21.15.32:0/480930104 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f20c002b8e0 con 0x7f20c4102bf0 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.304+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f20c40fd5e0 con 0x7f20b00367e0 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.308+0000 7f20baffd700 1 -- 172.21.15.32:0/480930104 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f20c40fd5e0 con 0x7f20b00367e0 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f20b00367e0 msgr2=0x7f20b0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 --2- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f20b00367e0 0x7f20b0038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009990 tx=0x7f20b4006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 msgr2=0x7f20c4113550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 --2- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4113550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20c0011ca0 tx=0x7f20c0009250 comp rx=0 tx=0).stop 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 shutdown_connections 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 --2- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f20b00367e0 0x7f20b0038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 --2- 172.21.15.32:0/480930104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20c4102bf0 0x7f20c4113550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 >> 172.21.15.32:0/480930104 conn(0x7f20c40fc400 msgr2=0x7f20c40fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 shutdown_connections 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:36.310+0000 7f20cc5c6700 1 -- 172.21.15.32:0/480930104 wait complete. 2024-04-06T14:07:37.420 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying node-exporter service with default placement... 2024-04-06T14:07:38.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:38 smithi032 ceph-mon[39730]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:07:38.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:38 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.268+0000 7f0424882700 1 Processor -- start 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.268+0000 7f0424882700 1 -- start start 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.268+0000 7f0424882700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f04201051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.268+0000 7f0424882700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0420105730 con 0x7f0420076c40 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.269+0000 7f041e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f04201051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.269+0000 7f041e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f04201051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59124/0 (socket says 172.21.15.32:59124) 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.269+0000 7f041e59c700 1 -- 172.21.15.32:0/2236601265 learned_addr learned my addr 172.21.15.32:0/2236601265 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.269+0000 7f041e59c700 1 -- 172.21.15.32:0/2236601265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0420105870 con 0x7f0420076c40 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.269+0000 7f041e59c700 1 --2- 172.21.15.32:0/2236601265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f04201051f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f04100098e0 tx=0x7f0410009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b696dcf7fb59ca server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.269+0000 7f041dd9b700 1 -- 172.21.15.32:0/2236601265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f041000b860 con 0x7f0420076c40 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.269+0000 7f041dd9b700 1 -- 172.21.15.32:0/2236601265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f041000be70 con 0x7f0420076c40 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.269+0000 7f041dd9b700 1 -- 172.21.15.32:0/2236601265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0410010380 con 0x7f0420076c40 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.270+0000 7f0424882700 1 -- 172.21.15.32:0/2236601265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 msgr2=0x7f04201051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.270+0000 7f0424882700 1 --2- 172.21.15.32:0/2236601265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f04201051f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f04100098e0 tx=0x7f0410009250 comp rx=0 tx=0).stop 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.270+0000 7f0424882700 1 -- 172.21.15.32:0/2236601265 shutdown_connections 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.270+0000 7f0424882700 1 --2- 172.21.15.32:0/2236601265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f04201051f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.270+0000 7f0424882700 1 -- 172.21.15.32:0/2236601265 >> 172.21.15.32:0/2236601265 conn(0x7f04200fcc60 msgr2=0x7f04200ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.270+0000 7f0424882700 1 -- 172.21.15.32:0/2236601265 shutdown_connections 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.270+0000 7f0424882700 1 -- 172.21.15.32:0/2236601265 wait complete. 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f0424882700 1 Processor -- start 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f0424882700 1 -- start start 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f0424882700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f0420111360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f0424882700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04201118a0 con 0x7f0420076c40 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f041e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f0420111360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f041e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f0420111360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59140/0 (socket says 172.21.15.32:59140) 2024-04-06T14:07:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f041e59c700 1 -- 172.21.15.32:0/2566947181 learned_addr learned my addr 172.21.15.32:0/2566947181 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f041e59c700 1 -- 172.21.15.32:0/2566947181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04100094f0 con 0x7f0420076c40 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f041e59c700 1 --2- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f0420111360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04100094c0 tx=0x7f0410009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.271+0000 7f0416ffd700 1 -- 172.21.15.32:0/2566947181 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0410010850 con 0x7f0420076c40 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.272+0000 7f0416ffd700 1 -- 172.21.15.32:0/2566947181 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0410010e60 con 0x7f0420076c40 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.272+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0420111b00 con 0x7f0420076c40 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.272+0000 7f0416ffd700 1 -- 172.21.15.32:0/2566947181 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0410024e90 con 0x7f0420076c40 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.272+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0420112610 con 0x7f0420076c40 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.273+0000 7f0416ffd700 1 -- 172.21.15.32:0/2566947181 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f04100109b0 con 0x7f0420076c40 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.273+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042005f5f0 con 0x7f0420076c40 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.273+0000 7f0416ffd700 1 --2- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f040c0367e0 0x7f040c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.273+0000 7f0416ffd700 1 -- 172.21.15.32:0/2566947181 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f041004a4c0 con 0x7f0420076c40 2024-04-06T14:07:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.273+0000 7f0417fff700 1 --2- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f040c0367e0 0x7f040c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:39.145 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.273+0000 7f0417fff700 1 --2- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f040c0367e0 0x7f040c038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0408009990 tx=0x7f0408006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:39.145 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.278+0000 7f0416ffd700 1 -- 172.21.15.32:0/2566947181 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0410010c60 con 0x7f0420076c40 2024-04-06T14:07:39.145 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.484+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f04200fde50 con 0x7f040c0367e0 2024-04-06T14:07:39.145 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.488+0000 7f0416ffd700 1 -- 172.21.15.32:0/2566947181 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f04200fde50 con 0x7f040c0367e0 2024-04-06T14:07:39.145 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f040c0367e0 msgr2=0x7f040c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:39.145 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 --2- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f040c0367e0 0x7f040c038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0408009990 tx=0x7f0408006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:39.146 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 msgr2=0x7f0420111360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:39.146 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 --2- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f0420111360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04100094c0 tx=0x7f0410009250 comp rx=0 tx=0).stop 2024-04-06T14:07:39.146 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 shutdown_connections 2024-04-06T14:07:39.146 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 --2- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f040c0367e0 0x7f040c038c90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:39.146 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 --2- 172.21.15.32:0/2566947181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0420076c40 0x7f0420111360 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:39.146 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 >> 172.21.15.32:0/2566947181 conn(0x7f04200fcc60 msgr2=0x7f04200fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:39.146 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 shutdown_connections 2024-04-06T14:07:39.147 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:38.490+0000 7f0424882700 1 -- 172.21.15.32:0/2566947181 wait complete. 2024-04-06T14:07:39.147 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying alertmanager service with default placement... 2024-04-06T14:07:39.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:39 smithi032 ceph-mon[39730]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:39.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:39 smithi032 ceph-mon[39730]: Saving service node-exporter spec with placement * 2024-04-06T14:07:40.559 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-06T14:07:40.559 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.687+0000 7fdd39ca2700 1 Processor -- start 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.687+0000 7fdd39ca2700 1 -- start start 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.687+0000 7fdd39ca2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd341051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.687+0000 7fdd39ca2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd34105730 con 0x7fdd34076c40 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.688+0000 7fdd337fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd341051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.688+0000 7fdd337fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd341051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59142/0 (socket says 172.21.15.32:59142) 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.688+0000 7fdd337fe700 1 -- 172.21.15.32:0/2984500070 learned_addr learned my addr 172.21.15.32:0/2984500070 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.688+0000 7fdd337fe700 1 -- 172.21.15.32:0/2984500070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34105870 con 0x7fdd34076c40 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.688+0000 7fdd337fe700 1 --2- 172.21.15.32:0/2984500070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd341051f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd240098e0 tx=0x7fdd24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81e86ab97c01e3f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.688+0000 7fdd327fc700 1 -- 172.21.15.32:0/2984500070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd2400b860 con 0x7fdd34076c40 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.688+0000 7fdd327fc700 1 -- 172.21.15.32:0/2984500070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd2400be70 con 0x7fdd34076c40 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.688+0000 7fdd327fc700 1 -- 172.21.15.32:0/2984500070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd24010380 con 0x7fdd34076c40 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.689+0000 7fdd39ca2700 1 -- 172.21.15.32:0/2984500070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 msgr2=0x7fdd341051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.689+0000 7fdd39ca2700 1 --2- 172.21.15.32:0/2984500070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd341051f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd240098e0 tx=0x7fdd24009250 comp rx=0 tx=0).stop 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.689+0000 7fdd39ca2700 1 -- 172.21.15.32:0/2984500070 shutdown_connections 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.689+0000 7fdd39ca2700 1 --2- 172.21.15.32:0/2984500070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd341051f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.689+0000 7fdd39ca2700 1 -- 172.21.15.32:0/2984500070 >> 172.21.15.32:0/2984500070 conn(0x7fdd340fcc60 msgr2=0x7fdd340ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.689+0000 7fdd39ca2700 1 -- 172.21.15.32:0/2984500070 shutdown_connections 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.689+0000 7fdd39ca2700 1 -- 172.21.15.32:0/2984500070 wait complete. 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.689+0000 7fdd39ca2700 1 Processor -- start 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.690+0000 7fdd39ca2700 1 -- start start 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.690+0000 7fdd39ca2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd34111360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.690+0000 7fdd39ca2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd341118a0 con 0x7fdd34076c40 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.690+0000 7fdd337fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd34111360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.690+0000 7fdd337fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd34111360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59158/0 (socket says 172.21.15.32:59158) 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.690+0000 7fdd337fe700 1 -- 172.21.15.32:0/4114976723 learned_addr learned my addr 172.21.15.32:0/4114976723 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.690+0000 7fdd337fe700 1 -- 172.21.15.32:0/4114976723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd240094f0 con 0x7fdd34076c40 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.690+0000 7fdd337fe700 1 --2- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd34111360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd24006d40 tx=0x7fdd24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.691+0000 7fdd2bfff700 1 -- 172.21.15.32:0/4114976723 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd24010960 con 0x7fdd34076c40 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.691+0000 7fdd2bfff700 1 -- 172.21.15.32:0/4114976723 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd2401a430 con 0x7fdd34076c40 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.691+0000 7fdd2bfff700 1 -- 172.21.15.32:0/4114976723 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd240245b0 con 0x7fdd34076c40 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.691+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd34111b00 con 0x7fdd34076c40 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.691+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd34112610 con 0x7fdd34076c40 2024-04-06T14:07:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.692+0000 7fdd2bfff700 1 -- 172.21.15.32:0/4114976723 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fdd24010ac0 con 0x7fdd34076c40 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.692+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3405f5f0 con 0x7fdd34076c40 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.692+0000 7fdd2bfff700 1 --2- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd200367e0 0x7fdd20038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.692+0000 7fdd2bfff700 1 -- 172.21.15.32:0/4114976723 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdd2404a240 con 0x7fdd34076c40 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.692+0000 7fdd32ffd700 1 --2- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd200367e0 0x7fdd20038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.692+0000 7fdd32ffd700 1 --2- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd200367e0 0x7fdd20038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009990 tx=0x7fdd1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.697+0000 7fdd2bfff700 1 -- 172.21.15.32:0/4114976723 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdd2402b4d0 con 0x7fdd34076c40 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.900+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd340fde50 con 0x7fdd200367e0 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.904+0000 7fdd2bfff700 1 -- 172.21.15.32:0/4114976723 <== mgr.14120 v2:172.21.15.32:6800/1833723736 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fdd340fde50 con 0x7fdd200367e0 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd200367e0 msgr2=0x7fdd20038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 --2- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd200367e0 0x7fdd20038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009990 tx=0x7fdd1c006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 msgr2=0x7fdd34111360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 --2- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd34111360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd24006d40 tx=0x7fdd24009250 comp rx=0 tx=0).stop 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 shutdown_connections 2024-04-06T14:07:40.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 --2- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7fdd200367e0 0x7fdd20038c90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:40.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 --2- 172.21.15.32:0/4114976723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd34076c40 0x7fdd34111360 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:40.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 >> 172.21.15.32:0/4114976723 conn(0x7fdd340fcc60 msgr2=0x7fdd340fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:40.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 shutdown_connections 2024-04-06T14:07:40.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:39.906+0000 7fdd39ca2700 1 -- 172.21.15.32:0/4114976723 wait complete. 2024-04-06T14:07:40.822 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:40 smithi032 ceph-mon[39730]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:07:40.822 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:40 smithi032 ceph-mon[39730]: from='mgr.14120 172.21.15.32:0/98613908' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:41.662 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:41 smithi032 ceph-mon[39730]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:41.662 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:41 smithi032 ceph-mon[39730]: Saving service alertmanager spec with placement count:1 2024-04-06T14:07:41.662 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:41 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1279714679' entity='client.admin' 2024-04-06T14:07:41.970 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.091+0000 7f9c1ea12700 1 Processor -- start 2024-04-06T14:07:41.970 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.091+0000 7f9c1ea12700 1 -- start start 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.091+0000 7f9c1ea12700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c181018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.091+0000 7f9c1ea12700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c18101e30 con 0x7f9c181014e0 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.091+0000 7f9c17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c181018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.091+0000 7f9c17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c181018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59168/0 (socket says 172.21.15.32:59168) 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.091+0000 7f9c17fff700 1 -- 172.21.15.32:0/2026051932 learned_addr learned my addr 172.21.15.32:0/2026051932 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.091+0000 7f9c17fff700 1 -- 172.21.15.32:0/2026051932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c18101f70 con 0x7f9c181014e0 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c17fff700 1 --2- 172.21.15.32:0/2026051932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c181018f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c080098e0 tx=0x7f9c08009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d481c65bd62824d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c16ffd700 1 -- 172.21.15.32:0/2026051932 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c0800b860 con 0x7f9c181014e0 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c16ffd700 1 -- 172.21.15.32:0/2026051932 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c0800be70 con 0x7f9c181014e0 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c16ffd700 1 -- 172.21.15.32:0/2026051932 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c08010380 con 0x7f9c181014e0 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c1ea12700 1 -- 172.21.15.32:0/2026051932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 msgr2=0x7f9c181018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c1ea12700 1 --2- 172.21.15.32:0/2026051932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c181018f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c080098e0 tx=0x7f9c08009250 comp rx=0 tx=0).stop 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c1ea12700 1 -- 172.21.15.32:0/2026051932 shutdown_connections 2024-04-06T14:07:41.971 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c1ea12700 1 --2- 172.21.15.32:0/2026051932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c181018f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c1ea12700 1 -- 172.21.15.32:0/2026051932 >> 172.21.15.32:0/2026051932 conn(0x7f9c180fcc60 msgr2=0x7f9c180ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c1ea12700 1 -- 172.21.15.32:0/2026051932 shutdown_connections 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.092+0000 7f9c1ea12700 1 -- 172.21.15.32:0/2026051932 wait complete. 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.093+0000 7f9c1ea12700 1 Processor -- start 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.093+0000 7f9c1ea12700 1 -- start start 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.093+0000 7f9c1ea12700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c18076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.093+0000 7f9c1ea12700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c18114080 con 0x7f9c181014e0 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c18076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c18076b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59172/0 (socket says 172.21.15.32:59172) 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c17fff700 1 -- 172.21.15.32:0/1279714679 learned_addr learned my addr 172.21.15.32:0/1279714679 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c17fff700 1 -- 172.21.15.32:0/1279714679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c080094f0 con 0x7f9c181014e0 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c17fff700 1 --2- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c18076b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9c08011ca0 tx=0x7f9c08009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c14ff9700 1 -- 172.21.15.32:0/1279714679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c08010850 con 0x7f9c181014e0 2024-04-06T14:07:41.972 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c14ff9700 1 -- 172.21.15.32:0/1279714679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c08010e60 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c181141c0 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c14ff9700 1 -- 172.21.15.32:0/1279714679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c0801ae20 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.094+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c18114650 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.095+0000 7f9c14ff9700 1 -- 172.21.15.32:0/1279714679 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f9c0802d3f0 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.095+0000 7f9c14ff9700 1 --2- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9c000367e0 0x7f9c00038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.095+0000 7f9c14ff9700 1 -- 172.21.15.32:0/1279714679 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9c0804a480 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.095+0000 7f9c177fe700 1 --2- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9c000367e0 0x7f9c00038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.095+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1804bed0 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.096+0000 7f9c177fe700 1 --2- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9c000367e0 0x7f9c00038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009990 tx=0x7f9c0c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.100+0000 7f9c14ff9700 1 -- 172.21.15.32:0/1279714679 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9c0802b8e0 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.294+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f9c18103270 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.297+0000 7f9c14ff9700 1 -- 172.21.15.32:0/1279714679 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9c0802b2f0 con 0x7f9c181014e0 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9c000367e0 msgr2=0x7f9c00038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:41.973 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 --2- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9c000367e0 0x7f9c00038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009990 tx=0x7f9c0c006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:41.974 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 msgr2=0x7f9c18076b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:41.974 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 --2- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c18076b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9c08011ca0 tx=0x7f9c08009250 comp rx=0 tx=0).stop 2024-04-06T14:07:41.974 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 shutdown_connections 2024-04-06T14:07:41.974 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 --2- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9c000367e0 0x7f9c00038c90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:41.974 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 --2- 172.21.15.32:0/1279714679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c181014e0 0x7f9c18076b30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:41.974 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 >> 172.21.15.32:0/1279714679 conn(0x7f9c180fcc60 msgr2=0x7f9c180fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:41.974 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 shutdown_connections 2024-04-06T14:07:41.974 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:41.300+0000 7f9c1ea12700 1 -- 172.21.15.32:0/1279714679 wait complete. 2024-04-06T14:07:42.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:42 smithi032 ceph-mon[39730]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.481+0000 7f9fd18d6700 1 Processor -- start 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.481+0000 7f9fd18d6700 1 -- start start 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.481+0000 7f9fd18d6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc1018c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.481+0000 7f9fd18d6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc101e00 con 0x7f9fcc1014b0 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.482+0000 7f9fcaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc1018c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.482+0000 7f9fcaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc1018c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59174/0 (socket says 172.21.15.32:59174) 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.482+0000 7f9fcaffd700 1 -- 172.21.15.32:0/1096636753 learned_addr learned my addr 172.21.15.32:0/1096636753 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.482+0000 7f9fcaffd700 1 -- 172.21.15.32:0/1096636753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc101f40 con 0x7f9fcc1014b0 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.482+0000 7f9fcaffd700 1 --2- 172.21.15.32:0/1096636753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc1018c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00098e0 tx=0x7f9fc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3f356ff8a08fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.482+0000 7f9fc9ffb700 1 -- 172.21.15.32:0/1096636753 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fc000b860 con 0x7f9fcc1014b0 2024-04-06T14:07:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.482+0000 7f9fc9ffb700 1 -- 172.21.15.32:0/1096636753 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fc000be70 con 0x7f9fcc1014b0 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.482+0000 7f9fc9ffb700 1 -- 172.21.15.32:0/1096636753 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fc0010380 con 0x7f9fcc1014b0 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.483+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1096636753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 msgr2=0x7f9fcc1018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.483+0000 7f9fd18d6700 1 --2- 172.21.15.32:0/1096636753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc1018c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00098e0 tx=0x7f9fc0009250 comp rx=0 tx=0).stop 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.483+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1096636753 shutdown_connections 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.483+0000 7f9fd18d6700 1 --2- 172.21.15.32:0/1096636753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc1018c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.483+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1096636753 >> 172.21.15.32:0/1096636753 conn(0x7f9fcc0fcc30 msgr2=0x7f9fcc0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.483+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1096636753 shutdown_connections 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.483+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1096636753 wait complete. 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fd18d6700 1 Processor -- start 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fd18d6700 1 -- start start 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fd18d6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc107710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fd18d6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc107c50 con 0x7f9fcc1014b0 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fcaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fcaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc107710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59190/0 (socket says 172.21.15.32:59190) 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fcaffd700 1 -- 172.21.15.32:0/1080196217 learned_addr learned my addr 172.21.15.32:0/1080196217 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:43.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fcaffd700 1 -- 172.21.15.32:0/1080196217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc00094f0 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fcaffd700 1 --2- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc107710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0011ca0 tx=0x7f9fc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fb37fe700 1 -- 172.21.15.32:0/1080196217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fc0010850 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fb37fe700 1 -- 172.21.15.32:0/1080196217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fc0010e60 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fb37fe700 1 -- 172.21.15.32:0/1080196217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fc001ae20 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.484+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc105cc0 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.485+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fcc1061b0 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.485+0000 7f9fb37fe700 1 -- 172.21.15.32:0/1080196217 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f9fc00109b0 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.485+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fcc04bed0 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.486+0000 7f9fb37fe700 1 --2- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb40367e0 0x7f9fb4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.486+0000 7f9fb37fe700 1 -- 172.21.15.32:0/1080196217 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9fc004a460 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.486+0000 7f9fca7fc700 1 --2- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb40367e0 0x7f9fb4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.486+0000 7f9fca7fc700 1 --2- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb40367e0 0x7f9fb4038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009990 tx=0x7f9fbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.490+0000 7f9fb37fe700 1 -- 172.21.15.32:0/1080196217 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9fc001a6a0 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.688+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f9fcc103240 con 0x7f9fcc1014b0 2024-04-06T14:07:43.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.690+0000 7f9fb37fe700 1 -- 172.21.15.32:0/1080196217 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f9fc002b840 con 0x7f9fcc1014b0 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb40367e0 msgr2=0x7f9fb4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 --2- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb40367e0 0x7f9fb4038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009990 tx=0x7f9fbc006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 msgr2=0x7f9fcc107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 --2- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc107710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0011ca0 tx=0x7f9fc0009250 comp rx=0 tx=0).stop 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 shutdown_connections 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 --2- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb40367e0 0x7f9fb4038c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 --2- 172.21.15.32:0/1080196217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fcc1014b0 0x7f9fcc107710 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 >> 172.21.15.32:0/1080196217 conn(0x7f9fcc0fcc30 msgr2=0x7f9fcc0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 shutdown_connections 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:42.693+0000 7f9fd18d6700 1 -- 172.21.15.32:0/1080196217 wait complete. 2024-04-06T14:07:43.239 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling the dashboard module... 2024-04-06T14:07:43.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:43 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1080196217' entity='client.admin' 2024-04-06T14:07:44.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:44 smithi032 ceph-mon[39730]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:07:44.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:44 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1872018497' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-06T14:07:45.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.807+0000 7f23234fa700 1 Processor -- start 2024-04-06T14:07:45.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.807+0000 7f23234fa700 1 -- start start 2024-04-06T14:07:45.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.807+0000 7f23234fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c105480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:45.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.807+0000 7f23234fa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c076d40 con 0x7f231c1030a0 2024-04-06T14:07:45.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.807+0000 7f2321296700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c105480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:45.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.807+0000 7f2321296700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c105480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59196/0 (socket says 172.21.15.32:59196) 2024-04-06T14:07:45.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.807+0000 7f2321296700 1 -- 172.21.15.32:0/1169430529 learned_addr learned my addr 172.21.15.32:0/1169430529 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:45.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.807+0000 7f2321296700 1 -- 172.21.15.32:0/1169430529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c076e80 con 0x7f231c1030a0 2024-04-06T14:07:45.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f2321296700 1 --2- 172.21.15.32:0/1169430529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c105480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f230c0098e0 tx=0x7f230c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a19a9b84d44e5ab server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f2313fff700 1 -- 172.21.15.32:0/1169430529 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230c00b860 con 0x7f231c1030a0 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f2313fff700 1 -- 172.21.15.32:0/1169430529 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f230c00be70 con 0x7f231c1030a0 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f2313fff700 1 -- 172.21.15.32:0/1169430529 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230c010380 con 0x7f231c1030a0 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f23234fa700 1 -- 172.21.15.32:0/1169430529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 msgr2=0x7f231c105480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f23234fa700 1 --2- 172.21.15.32:0/1169430529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c105480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f230c0098e0 tx=0x7f230c009250 comp rx=0 tx=0).stop 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f23234fa700 1 -- 172.21.15.32:0/1169430529 shutdown_connections 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f23234fa700 1 --2- 172.21.15.32:0/1169430529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c105480 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f23234fa700 1 -- 172.21.15.32:0/1169430529 >> 172.21.15.32:0/1169430529 conn(0x7f231c0fcb20 msgr2=0x7f231c0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.808+0000 7f23234fa700 1 -- 172.21.15.32:0/1169430529 shutdown_connections 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.809+0000 7f23234fa700 1 -- 172.21.15.32:0/1169430529 wait complete. 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.809+0000 7f23234fa700 1 Processor -- start 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.809+0000 7f23234fa700 1 -- start start 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.809+0000 7f23234fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c111380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:45.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.809+0000 7f23234fa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c1118c0 con 0x7f231c1030a0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.809+0000 7f2321296700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c111380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.809+0000 7f2321296700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c111380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59198/0 (socket says 172.21.15.32:59198) 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.809+0000 7f2321296700 1 -- 172.21.15.32:0/1872018497 learned_addr learned my addr 172.21.15.32:0/1872018497 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.810+0000 7f2321296700 1 -- 172.21.15.32:0/1872018497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c0094f0 con 0x7f231c1030a0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.810+0000 7f2321296700 1 --2- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c111380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f230c0094c0 tx=0x7f230c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.810+0000 7f2311ffb700 1 -- 172.21.15.32:0/1872018497 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230c010850 con 0x7f231c1030a0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.810+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c111b20 con 0x7f231c1030a0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.810+0000 7f2311ffb700 1 -- 172.21.15.32:0/1872018497 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f230c010e60 con 0x7f231c1030a0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.810+0000 7f2311ffb700 1 -- 172.21.15.32:0/1872018497 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230c024e90 con 0x7f231c1030a0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.810+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c112630 con 0x7f231c1030a0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.811+0000 7f2311ffb700 1 -- 172.21.15.32:0/1872018497 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f230c01a4c0 con 0x7f231c1030a0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.811+0000 7f2311ffb700 1 --2- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f2308036770 0x7f2308038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.811+0000 7f2311ffb700 1 -- 172.21.15.32:0/1872018497 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f230c04a530 con 0x7f231c1030a0 2024-04-06T14:07:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.811+0000 7f2320a95700 1 --2- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f2308036770 0x7f2308038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.811+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231c05f5f0 con 0x7f231c1030a0 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.812+0000 7f2320a95700 1 --2- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f2308036770 0x7f2308038c20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2318009990 tx=0x7f2318006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:43.816+0000 7f2311ffb700 1 -- 172.21.15.32:0/1872018497 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f230c018940 con 0x7f231c1030a0 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.065+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f231c112970 con 0x7f231c1030a0 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.694+0000 7f2311ffb700 1 -- 172.21.15.32:0/1872018497 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f230c02ad60 con 0x7f231c1030a0 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.695+0000 7f2311ffb700 1 -- 172.21.15.32:0/1872018497 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f230c00f2e0 con 0x7f231c1030a0 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f2308036770 msgr2=0x7f2308038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 --2- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f2308036770 0x7f2308038c20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2318009990 tx=0x7f2318006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 msgr2=0x7f231c111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 --2- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c111380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f230c0094c0 tx=0x7f230c009250 comp rx=0 tx=0).stop 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 shutdown_connections 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 --2- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f2308036770 0x7f2308038c20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 --2- 172.21.15.32:0/1872018497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c1030a0 0x7f231c111380 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 >> 172.21.15.32:0/1872018497 conn(0x7f231c0fcb20 msgr2=0x7f231c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:45.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 shutdown_connections 2024-04-06T14:07:45.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:44.699+0000 7f23234fa700 1 -- 172.21.15.32:0/1872018497 wait complete. 2024-04-06T14:07:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:45 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1872018497' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-06T14:07:45.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:45 smithi032 ceph-mon[39730]: mgrmap e9: smithi032.peohxu(active, since 27s) 2024-04-06T14:07:46.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:46 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3377880056' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T14:07:46.985 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "smithi032.peohxu", 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.049+0000 7f9fd5348700 1 Processor -- start 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.049+0000 7f9fd5348700 1 -- start start 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.049+0000 7f9fd5348700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd0077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.049+0000 7f9fd5348700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd0079160 con 0x7f9fd0078d50 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.049+0000 7f9fceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd0077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:46.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.050+0000 7f9fceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd0077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36108/0 (socket says 172.21.15.32:36108) 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.050+0000 7f9fceffd700 1 -- 172.21.15.32:0/1384603040 learned_addr learned my addr 172.21.15.32:0/1384603040 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.050+0000 7f9fceffd700 1 -- 172.21.15.32:0/1384603040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd00777a0 con 0x7f9fd0078d50 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.050+0000 7f9fceffd700 1 --2- 172.21.15.32:0/1384603040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd0077260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00098e0 tx=0x7f9fc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584dff187016a7c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.050+0000 7f9fcdffb700 1 -- 172.21.15.32:0/1384603040 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fc000b860 con 0x7f9fd0078d50 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.050+0000 7f9fcdffb700 1 -- 172.21.15.32:0/1384603040 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fc000be70 con 0x7f9fd0078d50 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.050+0000 7f9fcdffb700 1 -- 172.21.15.32:0/1384603040 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fc0010380 con 0x7f9fd0078d50 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.051+0000 7f9fd5348700 1 -- 172.21.15.32:0/1384603040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 msgr2=0x7f9fd0077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.051+0000 7f9fd5348700 1 --2- 172.21.15.32:0/1384603040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd0077260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00098e0 tx=0x7f9fc0009250 comp rx=0 tx=0).stop 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.051+0000 7f9fd5348700 1 -- 172.21.15.32:0/1384603040 shutdown_connections 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.051+0000 7f9fd5348700 1 --2- 172.21.15.32:0/1384603040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd0077260 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.051+0000 7f9fd5348700 1 -- 172.21.15.32:0/1384603040 >> 172.21.15.32:0/1384603040 conn(0x7f9fd00fdd00 msgr2=0x7f9fd0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.051+0000 7f9fd5348700 1 -- 172.21.15.32:0/1384603040 shutdown_connections 2024-04-06T14:07:46.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.051+0000 7f9fd5348700 1 -- 172.21.15.32:0/1384603040 wait complete. 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fd5348700 1 Processor -- start 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fd5348700 1 -- start start 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fd5348700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd01135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fd5348700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd0113ae0 con 0x7f9fd0078d50 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd01135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd01135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36116/0 (socket says 172.21.15.32:36116) 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fceffd700 1 -- 172.21.15.32:0/3377880056 learned_addr learned my addr 172.21.15.32:0/3377880056 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fceffd700 1 -- 172.21.15.32:0/3377880056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc00094f0 con 0x7f9fd0078d50 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fceffd700 1 --2- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd01135a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00094c0 tx=0x7f9fc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.052+0000 7f9fb77fe700 1 -- 172.21.15.32:0/3377880056 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fc0010850 con 0x7f9fd0078d50 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.053+0000 7f9fb77fe700 1 -- 172.21.15.32:0/3377880056 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fc0010e60 con 0x7f9fd0078d50 2024-04-06T14:07:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.053+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd0113ce0 con 0x7f9fd0078d50 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.053+0000 7f9fb77fe700 1 -- 172.21.15.32:0/3377880056 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fc001a430 con 0x7f9fd0078d50 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.053+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd0114850 con 0x7f9fd0078d50 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.053+0000 7f9fb77fe700 1 -- 172.21.15.32:0/3377880056 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f9fc001a660 con 0x7f9fd0078d50 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.054+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd005f5f0 con 0x7f9fd0078d50 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.054+0000 7f9fb77fe700 1 --2- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb8036830 0x7f9fb8038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.054+0000 7f9fb77fe700 1 -- 172.21.15.32:0/3377880056 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9fc004a680 con 0x7f9fd0078d50 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.054+0000 7f9fce7fc700 1 -- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb8036830 msgr2=0x7f9fb8038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1833723736 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.054+0000 7f9fce7fc700 1 --2- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb8036830 0x7f9fb8038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.061+0000 7f9fb77fe700 1 -- 172.21.15.32:0/3377880056 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9fc0022340 con 0x7f9fd0078d50 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.255+0000 7f9fce7fc700 1 -- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb8036830 msgr2=0x7f9fb8038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1833723736 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.255+0000 7f9fce7fc700 1 --2- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb8036830 0x7f9fb8038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.350+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9fd0114b90 con 0x7f9fd0078d50 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.350+0000 7f9fb77fe700 1 -- 172.21.15.32:0/3377880056 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f9fc0018b40 con 0x7f9fd0078d50 2024-04-06T14:07:46.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.352+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb8036830 msgr2=0x7f9fb8038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.352+0000 7f9fd5348700 1 --2- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb8036830 0x7f9fb8038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.352+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 msgr2=0x7f9fd01135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.352+0000 7f9fd5348700 1 --2- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd01135a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00094c0 tx=0x7f9fc0009250 comp rx=0 tx=0).stop 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.352+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 shutdown_connections 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.353+0000 7f9fd5348700 1 --2- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7f9fb8036830 0x7f9fb8038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.353+0000 7f9fd5348700 1 --2- 172.21.15.32:0/3377880056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd0078d50 0x7f9fd01135a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.353+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 >> 172.21.15.32:0/3377880056 conn(0x7f9fd00fdd00 msgr2=0x7f9fd00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.353+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 shutdown_connections 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:46.353+0000 7f9fd5348700 1 -- 172.21.15.32:0/3377880056 wait complete. 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-06T14:07:46.996 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 9... 2024-04-06T14:07:51.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: Active manager daemon smithi032.peohxu restarted 2024-04-06T14:07:51.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: Activating manager daemon smithi032.peohxu 2024-04-06T14:07:51.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: osdmap e3: 0 total, 0 up, 0 in 2024-04-06T14:07:51.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: mgrmap e10: smithi032.peohxu(active, starting, since 0.00297392s) 2024-04-06T14:07:51.641 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:07:51.642 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:07:51.642 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:07:51.642 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:07:51.642 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:07:51.642 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: Manager daemon smithi032.peohxu is now available 2024-04-06T14:07:51.642 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:07:51.642 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:51 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:07:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:52 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:07:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:52 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/trash_purge_schedule"}]: dispatch 2024-04-06T14:07:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:52 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:52 smithi032 ceph-mon[39730]: mgrmap e11: smithi032.peohxu(active, since 1.00552s) 2024-04-06T14:07:53.162 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-06T14:07:53.162 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-06T14:07:53.162 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-06T14:07:53.162 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-06T14:07:53.162 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.583+0000 7faf31be3700 1 Processor -- start 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.583+0000 7faf31be3700 1 -- start start 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.583+0000 7faf31be3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c1051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.583+0000 7faf31be3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c105730 con 0x7faf2c076c40 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.583+0000 7faf2b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c1051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.583+0000 7faf2b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c1051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36124/0 (socket says 172.21.15.32:36124) 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.583+0000 7faf2b7fe700 1 -- 172.21.15.32:0/1504340810 learned_addr learned my addr 172.21.15.32:0/1504340810 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.584+0000 7faf2b7fe700 1 -- 172.21.15.32:0/1504340810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c105870 con 0x7faf2c076c40 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.584+0000 7faf2b7fe700 1 --2- 172.21.15.32:0/1504340810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c1051f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0098e0 tx=0x7faf1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb6fa9c6d264ec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.584+0000 7faf2a7fc700 1 -- 172.21.15.32:0/1504340810 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf1c00b860 con 0x7faf2c076c40 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.584+0000 7faf2a7fc700 1 -- 172.21.15.32:0/1504340810 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf1c00be70 con 0x7faf2c076c40 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.584+0000 7faf2a7fc700 1 -- 172.21.15.32:0/1504340810 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf1c010380 con 0x7faf2c076c40 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.585+0000 7faf31be3700 1 -- 172.21.15.32:0/1504340810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 msgr2=0x7faf2c1051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.585+0000 7faf31be3700 1 --2- 172.21.15.32:0/1504340810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c1051f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0098e0 tx=0x7faf1c009250 comp rx=0 tx=0).stop 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.585+0000 7faf31be3700 1 -- 172.21.15.32:0/1504340810 shutdown_connections 2024-04-06T14:07:53.163 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.585+0000 7faf31be3700 1 --2- 172.21.15.32:0/1504340810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c1051f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.585+0000 7faf31be3700 1 -- 172.21.15.32:0/1504340810 >> 172.21.15.32:0/1504340810 conn(0x7faf2c0fcc60 msgr2=0x7faf2c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.585+0000 7faf31be3700 1 -- 172.21.15.32:0/1504340810 shutdown_connections 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.585+0000 7faf31be3700 1 -- 172.21.15.32:0/1504340810 wait complete. 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.585+0000 7faf31be3700 1 Processor -- start 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.585+0000 7faf31be3700 1 -- start start 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf31be3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c111360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf31be3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1118a0 con 0x7faf2c076c40 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf2b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c111360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf2b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c111360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36138/0 (socket says 172.21.15.32:36138) 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf2b7fe700 1 -- 172.21.15.32:0/24360255 learned_addr learned my addr 172.21.15.32:0/24360255 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:53.164 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf2b7fe700 1 -- 172.21.15.32:0/24360255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1c0094f0 con 0x7faf2c076c40 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf2b7fe700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c111360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0094c0 tx=0x7faf1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf1c010850 con 0x7faf2c076c40 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf1c010e60 con 0x7faf2c076c40 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf1c024e90 con 0x7faf2c076c40 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2c111b00 con 0x7faf2c076c40 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.586+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2c112610 con 0x7faf2c076c40 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.587+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7faf1c01a4c0 con 0x7faf2c076c40 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.587+0000 7faf13fff700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 0x7faf14038c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.587+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 --> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf2c05f5f0 con 0x7faf140367b0 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.587+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7faf1c04a6f0 con 0x7faf2c076c40 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.587+0000 7faf2affd700 1 -- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 msgr2=0x7faf14038c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1833723736 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.587+0000 7faf2affd700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 0x7faf14038c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.788+0000 7faf2affd700 1 -- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 msgr2=0x7faf14038c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1833723736 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:47.788+0000 7faf2affd700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 0x7faf14038c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T14:07:53.165 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:48.189+0000 7faf2affd700 1 -- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 msgr2=0x7faf14038c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1833723736 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:48.189+0000 7faf2affd700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 0x7faf14038c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:48.990+0000 7faf2affd700 1 -- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 msgr2=0x7faf14038c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1833723736 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:48.990+0000 7faf2affd700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 0x7faf14038c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:50.591+0000 7faf2affd700 1 -- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 msgr2=0x7faf14038c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1833723736 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:50.591+0000 7faf2affd700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 0x7faf14038c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:51.524+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7faf1c024740 con 0x7faf2c076c40 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:51.524+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 msgr2=0x7faf14038c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:51.524+0000 7faf13fff700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 0x7faf14038c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.525+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7faf1c02d770 con 0x7faf2c076c40 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.525+0000 7faf13fff700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7faf14039fc0 0x7faf1403c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.525+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 --> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf2c05f5f0 con 0x7faf14039fc0 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.526+0000 7faf2affd700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7faf14039fc0 0x7faf1403c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.526+0000 7faf2affd700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7faf14039fc0 0x7faf1403c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7faf200039d0 tx=0x7faf200092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:53.166 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.527+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 <== mgr.14162 v2:172.21.15.32:6800/3947155315 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7faf2c05f5f0 con 0x7faf14039fc0 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.533+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 --> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7faf2c04bed0 con 0x7faf14039fc0 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.533+0000 7faf13fff700 1 -- 172.21.15.32:0/24360255 <== mgr.14162 v2:172.21.15.32:6800/3947155315 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7faf2c04bed0 con 0x7faf14039fc0 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.533+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7faf14039fc0 msgr2=0x7faf1403c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.533+0000 7faf31be3700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7faf14039fc0 0x7faf1403c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7faf200039d0 tx=0x7faf200092f0 comp rx=0 tx=0).stop 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.533+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 msgr2=0x7faf2c111360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.533+0000 7faf31be3700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c111360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0094c0 tx=0x7faf1c009250 comp rx=0 tx=0).stop 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.534+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 shutdown_connections 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.534+0000 7faf31be3700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7faf14039fc0 0x7faf1403c3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.534+0000 7faf31be3700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c076c40 0x7faf2c111360 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.534+0000 7faf31be3700 1 --2- 172.21.15.32:0/24360255 >> [v2:172.21.15.32:6800/1833723736,v1:172.21.15.32:6801/1833723736] conn(0x7faf140367b0 0x7faf14038c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.534+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 >> 172.21.15.32:0/24360255 conn(0x7faf2c0fcc60 msgr2=0x7faf2c0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.534+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 shutdown_connections 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:52.534+0000 7faf31be3700 1 -- 172.21.15.32:0/24360255 wait complete. 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 9 is available 2024-04-06T14:07:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:Generating a dashboard self-signed certificate... 2024-04-06T14:07:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:53 smithi032 ceph-mon[39730]: [06/Apr/2024:14:07:52] ENGINE Bus STARTING 2024-04-06T14:07:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:53 smithi032 ceph-mon[39730]: [06/Apr/2024:14:07:52] ENGINE Serving on https://172.21.15.32:7150 2024-04-06T14:07:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:53 smithi032 ceph-mon[39730]: [06/Apr/2024:14:07:52] ENGINE Bus STARTED 2024-04-06T14:07:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:53 smithi032 ceph-mon[39730]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-06T14:07:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:53 smithi032 ceph-mon[39730]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-06T14:07:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:53 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:54.700 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-06T14:07:54.700 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.803+0000 7fa795c07700 1 Processor -- start 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.803+0000 7fa795c07700 1 -- start start 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.803+0000 7fa795c07700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.803+0000 7fa795c07700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901056f0 con 0x7fa790076c80 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.803+0000 7fa78f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.803+0000 7fa78f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36192/0 (socket says 172.21.15.32:36192) 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.803+0000 7fa78f7fe700 1 -- 172.21.15.32:0/3758351676 learned_addr learned my addr 172.21.15.32:0/3758351676 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.804+0000 7fa78f7fe700 1 -- 172.21.15.32:0/3758351676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790105830 con 0x7fa790076c80 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.804+0000 7fa78f7fe700 1 --2- 172.21.15.32:0/3758351676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901051b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7800098e0 tx=0x7fa780009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638551d97d6da3b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.804+0000 7fa78e7fc700 1 -- 172.21.15.32:0/3758351676 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa78000b860 con 0x7fa790076c80 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.804+0000 7fa78e7fc700 1 -- 172.21.15.32:0/3758351676 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa78000be70 con 0x7fa790076c80 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.804+0000 7fa78e7fc700 1 -- 172.21.15.32:0/3758351676 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa780010380 con 0x7fa790076c80 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.804+0000 7fa795c07700 1 -- 172.21.15.32:0/3758351676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 msgr2=0x7fa7901051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.804+0000 7fa795c07700 1 --2- 172.21.15.32:0/3758351676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901051b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7800098e0 tx=0x7fa780009250 comp rx=0 tx=0).stop 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.805+0000 7fa795c07700 1 -- 172.21.15.32:0/3758351676 shutdown_connections 2024-04-06T14:07:54.701 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.805+0000 7fa795c07700 1 --2- 172.21.15.32:0/3758351676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901051b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.805+0000 7fa795c07700 1 -- 172.21.15.32:0/3758351676 >> 172.21.15.32:0/3758351676 conn(0x7fa7900fcc60 msgr2=0x7fa7900ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.805+0000 7fa795c07700 1 -- 172.21.15.32:0/3758351676 shutdown_connections 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.805+0000 7fa795c07700 1 -- 172.21.15.32:0/3758351676 wait complete. 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.805+0000 7fa795c07700 1 Processor -- start 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.805+0000 7fa795c07700 1 -- start start 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.805+0000 7fa795c07700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901113b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.805+0000 7fa795c07700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901118f0 con 0x7fa790076c80 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa78f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa78f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36202/0 (socket says 172.21.15.32:36202) 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa78f7fe700 1 -- 172.21.15.32:0/1716713818 learned_addr learned my addr 172.21.15.32:0/1716713818 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa78f7fe700 1 -- 172.21.15.32:0/1716713818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800094f0 con 0x7fa790076c80 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa78f7fe700 1 --2- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901113b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa780006d40 tx=0x7fa780009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa787fff700 1 -- 172.21.15.32:0/1716713818 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa780010960 con 0x7fa790076c80 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa790111af0 con 0x7fa790076c80 2024-04-06T14:07:54.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa787fff700 1 -- 172.21.15.32:0/1716713818 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa78001a430 con 0x7fa790076c80 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa787fff700 1 -- 172.21.15.32:0/1716713818 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7800245b0 con 0x7fa790076c80 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.806+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa790112660 con 0x7fa790076c80 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.807+0000 7fa787fff700 1 -- 172.21.15.32:0/1716713818 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fa78002d3f0 con 0x7fa790076c80 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.807+0000 7fa787fff700 1 --2- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa77c0366b0 0x7fa77c038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.807+0000 7fa787fff700 1 -- 172.21.15.32:0/1716713818 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa78004a270 con 0x7fa790076c80 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.807+0000 7fa78effd700 1 --2- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa77c0366b0 0x7fa77c038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.808+0000 7fa78effd700 1 --2- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa77c0366b0 0x7fa77c038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa778009990 tx=0x7fa778006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.808+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79005f5f0 con 0x7fa790076c80 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:53.813+0000 7fa787fff700 1 -- 172.21.15.32:0/1716713818 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa78002b8e0 con 0x7fa790076c80 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.022+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 --> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7900fde50 con 0x7fa77c0366b0 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.096+0000 7fa787fff700 1 -- 172.21.15.32:0/1716713818 <== mgr.14162 v2:172.21.15.32:6800/3947155315 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa7900fde50 con 0x7fa77c0366b0 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa77c0366b0 msgr2=0x7fa77c038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 --2- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa77c0366b0 0x7fa77c038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa778009990 tx=0x7fa778006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 msgr2=0x7fa7901113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:54.703 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 --2- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901113b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa780006d40 tx=0x7fa780009250 comp rx=0 tx=0).stop 2024-04-06T14:07:54.704 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 shutdown_connections 2024-04-06T14:07:54.704 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 --2- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa77c0366b0 0x7fa77c038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:54.704 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 --2- 172.21.15.32:0/1716713818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa790076c80 0x7fa7901113b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:54.704 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 >> 172.21.15.32:0/1716713818 conn(0x7fa7900fcc60 msgr2=0x7fa7900fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:54.704 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 shutdown_connections 2024-04-06T14:07:54.704 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:54.098+0000 7fa795c07700 1 -- 172.21.15.32:0/1716713818 wait complete. 2024-04-06T14:07:54.704 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial admin user... 2024-04-06T14:07:55.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:54 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:55.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:54 smithi032 ceph-mon[39730]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:55.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:54 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:55.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:54 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:55.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:54 smithi032 ceph-mon[39730]: mgrmap e12: smithi032.peohxu(active, since 2s) 2024-04-06T14:07:56.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:55 smithi032 ceph-mon[39730]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:07:56.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:55 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:07:57.136 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$IAASyPY5K7nYhDyyZ3RiKuK.WEzo6QREwD26Wy.bHFJDgc4SQHoVW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712412475, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-06T14:07:57.136 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.325+0000 7f7536e8d700 1 Processor -- start 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.326+0000 7f7536e8d700 1 -- start start 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.326+0000 7f7536e8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f75301018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.326+0000 7f7536e8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530101e30 con 0x7f75301014e0 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.326+0000 7f7534c29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f75301018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.326+0000 7f7534c29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f75301018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45740/0 (socket says 172.21.15.32:45740) 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.326+0000 7f7534c29700 1 -- 172.21.15.32:0/3638878604 learned_addr learned my addr 172.21.15.32:0/3638878604 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.326+0000 7f7534c29700 1 -- 172.21.15.32:0/3638878604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7530101f70 con 0x7f75301014e0 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f7534c29700 1 --2- 172.21.15.32:0/3638878604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f75301018f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f75240098e0 tx=0x7f7524009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbec2546a982b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f752f7fe700 1 -- 172.21.15.32:0/3638878604 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752400b860 con 0x7f75301014e0 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f752f7fe700 1 -- 172.21.15.32:0/3638878604 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f752400be70 con 0x7f75301014e0 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f752f7fe700 1 -- 172.21.15.32:0/3638878604 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7524010380 con 0x7f75301014e0 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f7536e8d700 1 -- 172.21.15.32:0/3638878604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 msgr2=0x7f75301018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f7536e8d700 1 --2- 172.21.15.32:0/3638878604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f75301018f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f75240098e0 tx=0x7f7524009250 comp rx=0 tx=0).stop 2024-04-06T14:07:57.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f7536e8d700 1 -- 172.21.15.32:0/3638878604 shutdown_connections 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f7536e8d700 1 --2- 172.21.15.32:0/3638878604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f75301018f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f7536e8d700 1 -- 172.21.15.32:0/3638878604 >> 172.21.15.32:0/3638878604 conn(0x7f75300fcc60 msgr2=0x7f75300ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f7536e8d700 1 -- 172.21.15.32:0/3638878604 shutdown_connections 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.327+0000 7f7536e8d700 1 -- 172.21.15.32:0/3638878604 wait complete. 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.328+0000 7f7536e8d700 1 Processor -- start 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.328+0000 7f7536e8d700 1 -- start start 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.328+0000 7f7536e8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f7530076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.328+0000 7f7536e8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530114080 con 0x7f75301014e0 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.328+0000 7f7534c29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f7530076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.328+0000 7f7534c29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f7530076b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45750/0 (socket says 172.21.15.32:45750) 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.328+0000 7f7534c29700 1 -- 172.21.15.32:0/1583160940 learned_addr learned my addr 172.21.15.32:0/1583160940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.329+0000 7f7534c29700 1 -- 172.21.15.32:0/1583160940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75240094f0 con 0x7f75301014e0 2024-04-06T14:07:57.138 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.329+0000 7f7534c29700 1 --2- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f7530076b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7524011ca0 tx=0x7f7524009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:57.139 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.329+0000 7f752d7fa700 1 -- 172.21.15.32:0/1583160940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7524010850 con 0x7f75301014e0 2024-04-06T14:07:57.139 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.329+0000 7f752d7fa700 1 -- 172.21.15.32:0/1583160940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7524010e60 con 0x7f75301014e0 2024-04-06T14:07:57.139 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.329+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75301141c0 con 0x7f75301014e0 2024-04-06T14:07:57.139 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.329+0000 7f752d7fa700 1 -- 172.21.15.32:0/1583160940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752401ae20 con 0x7f75301014e0 2024-04-06T14:07:57.139 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.329+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7530114650 con 0x7f75301014e0 2024-04-06T14:07:57.141 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.330+0000 7f752d7fa700 1 -- 172.21.15.32:0/1583160940 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f752402d3f0 con 0x7f75301014e0 2024-04-06T14:07:57.141 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.330+0000 7f752d7fa700 1 --2- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f75200367e0 0x7f7520038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.330+0000 7f752d7fa700 1 -- 172.21.15.32:0/1583160940 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f752404a560 con 0x7f75301014e0 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.330+0000 7f752ffff700 1 --2- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f75200367e0 0x7f7520038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.330+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753004bed0 con 0x7f75301014e0 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.330+0000 7f752ffff700 1 --2- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f75200367e0 0x7f7520038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f751c009990 tx=0x7f751c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.335+0000 7f752d7fa700 1 -- 172.21.15.32:0/1583160940 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f752402b8e0 con 0x7f75301014e0 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.546+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 --> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f753005f410 con 0x7f75200367e0 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.793+0000 7f752d7fa700 1 -- 172.21.15.32:0/1583160940 <== mgr.14162 v2:172.21.15.32:6800/3947155315 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f753005f410 con 0x7f75200367e0 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.795+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f75200367e0 msgr2=0x7f7520038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.795+0000 7f7536e8d700 1 --2- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f75200367e0 0x7f7520038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f751c009990 tx=0x7f751c006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.796+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 msgr2=0x7f7530076b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.796+0000 7f7536e8d700 1 --2- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f7530076b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7524011ca0 tx=0x7f7524009250 comp rx=0 tx=0).stop 2024-04-06T14:07:57.142 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.796+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 shutdown_connections 2024-04-06T14:07:57.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.796+0000 7f7536e8d700 1 --2- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f75200367e0 0x7f7520038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:57.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.796+0000 7f7536e8d700 1 --2- 172.21.15.32:0/1583160940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75301014e0 0x7f7530076b30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:57.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.796+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 >> 172.21.15.32:0/1583160940 conn(0x7f75300fcc60 msgr2=0x7f75300fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:57.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.796+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 shutdown_connections 2024-04-06T14:07:57.143 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:55.796+0000 7f7536e8d700 1 -- 172.21.15.32:0/1583160940 wait complete. 2024-04-06T14:07:57.143 INFO:teuthology.orchestra.run.smithi032.stdout:Fetching dashboard port number... 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 8443 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.635+0000 7fdc03f59700 1 Processor -- start 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.635+0000 7fdc03f59700 1 -- start start 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.635+0000 7fdc03f59700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc076d00 0x7fdbfc1074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.635+0000 7fdc03f59700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc102ee0 con 0x7fdbfc076d00 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.636+0000 7fdc01cf5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc076d00 0x7fdbfc1074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.636+0000 7fdc01cf5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc076d00 0x7fdbfc1074f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45766/0 (socket says 172.21.15.32:45766) 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.636+0000 7fdc01cf5700 1 -- 172.21.15.32:0/3643922152 learned_addr learned my addr 172.21.15.32:0/3643922152 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.636+0000 7fdc01cf5700 1 -- 172.21.15.32:0/3643922152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc107a30 con 0x7fdbfc076d00 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.636+0000 7fdc01cf5700 1 --2- 172.21.15.32:0/3643922152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc076d00 0x7fdbfc1074f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0098e0 tx=0x7fdbec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb61e915c49acbd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.636+0000 7fdc00cf3700 1 -- 172.21.15.32:0/3643922152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbec00b860 con 0x7fdbfc076d00 2024-04-06T14:07:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.636+0000 7fdc00cf3700 1 -- 172.21.15.32:0/3643922152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbec00be70 con 0x7fdbfc076d00 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.636+0000 7fdc00cf3700 1 -- 172.21.15.32:0/3643922152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbec010380 con 0x7fdbfc076d00 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.637+0000 7fdc03f59700 1 -- 172.21.15.32:0/3643922152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc076d00 msgr2=0x7fdbfc1074f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.637+0000 7fdc03f59700 1 --2- 172.21.15.32:0/3643922152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc076d00 0x7fdbfc1074f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0098e0 tx=0x7fdbec009250 comp rx=0 tx=0).stop 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.637+0000 7fdc03f59700 1 -- 172.21.15.32:0/3643922152 shutdown_connections 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.637+0000 7fdc03f59700 1 --2- 172.21.15.32:0/3643922152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc076d00 0x7fdbfc1074f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.637+0000 7fdc03f59700 1 -- 172.21.15.32:0/3643922152 >> 172.21.15.32:0/3643922152 conn(0x7fdbfc0fcb20 msgr2=0x7fdbfc0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.637+0000 7fdc03f59700 1 -- 172.21.15.32:0/3643922152 shutdown_connections 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.637+0000 7fdc03f59700 1 -- 172.21.15.32:0/3643922152 wait complete. 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc03f59700 1 Processor -- start 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc03f59700 1 -- start start 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc03f59700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc1135a0 0x7fdbfc1139b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc03f59700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc113ef0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc01cf5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc1135a0 0x7fdbfc1139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc01cf5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc1135a0 0x7fdbfc1139b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45780/0 (socket says 172.21.15.32:45780) 2024-04-06T14:07:58.453 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc01cf5700 1 -- 172.21.15.32:0/3491580104 learned_addr learned my addr 172.21.15.32:0/3491580104 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc01cf5700 1 -- 172.21.15.32:0/3491580104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec0094f0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc01cf5700 1 --2- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc1135a0 0x7fdbfc1139b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdbec006d40 tx=0x7fdbec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdbf27fc700 1 -- 172.21.15.32:0/3491580104 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbec010960 con 0x7fdbfc1135a0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdbf27fc700 1 -- 172.21.15.32:0/3491580104 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbec01a430 con 0x7fdbfc1135a0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc1140f0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdbf27fc700 1 -- 172.21.15.32:0/3491580104 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbec0245b0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.638+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc114ad0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.639+0000 7fdbf27fc700 1 -- 172.21.15.32:0/3491580104 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fdbec010ac0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.640+0000 7fdbf27fc700 1 --2- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdbe8036880 0x7fdbe8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.640+0000 7fdbf27fc700 1 -- 172.21.15.32:0/3491580104 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdbec04a4e0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.640+0000 7fdc014f4700 1 --2- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdbe8036880 0x7fdbe8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.640+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc05f5f0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.640+0000 7fdc014f4700 1 --2- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdbe8036880 0x7fdbe8038d30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009990 tx=0x7fdbf8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.645+0000 7fdbf27fc700 1 -- 172.21.15.32:0/3491580104 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdbec01a5a0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.838+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fdbfc04bed0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.838+0000 7fdbf27fc700 1 -- 172.21.15.32:0/3491580104 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fdbec02b2f0 con 0x7fdbfc1135a0 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdbe8036880 msgr2=0x7fdbe8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 --2- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdbe8036880 0x7fdbe8038d30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009990 tx=0x7fdbf8006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc1135a0 msgr2=0x7fdbfc1139b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 --2- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc1135a0 0x7fdbfc1139b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdbec006d40 tx=0x7fdbec009250 comp rx=0 tx=0).stop 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 shutdown_connections 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 --2- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdbe8036880 0x7fdbe8038d30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 --2- 172.21.15.32:0/3491580104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbfc1135a0 0x7fdbfc1139b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 >> 172.21.15.32:0/3491580104 conn(0x7fdbfc0fcb20 msgr2=0x7fdbfc0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 shutdown_connections 2024-04-06T14:07:58.455 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:57.840+0000 7fdc03f59700 1 -- 172.21.15.32:0/3491580104 wait complete. 2024-04-06T14:07:58.461 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-06T14:07:58.461 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-06T14:07:58.469 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-06T14:07:58.469 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-06T14:07:58.470 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-06T14:07:58.470 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-06T14:07:58.472 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph Dashboard is now available at: 2024-04-06T14:07:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:07:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: URL: https://smithi032.front.sepia.ceph.com:8443/ 2024-04-06T14:07:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: User: admin 2024-04-06T14:07:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: Password: ji7v9rnfve 2024-04-06T14:07:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:07:58.473 INFO:teuthology.orchestra.run.smithi032.stdout:Saving cluster configuration to /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config directory 2024-04-06T14:07:58.473 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling autotune for osd_memory_target 2024-04-06T14:07:58.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:58 smithi032 ceph-mon[39730]: mgrmap e13: smithi032.peohxu(active, since 6s) 2024-04-06T14:07:58.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:07:58 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3491580104' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-06T14:07:59.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.006+0000 7fdd44ce2700 1 Processor -- start 2024-04-06T14:07:59.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.006+0000 7fdd44ce2700 1 -- start start 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.006+0000 7fdd44ce2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40102960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.007+0000 7fdd44ce2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd40102ea0 con 0x7fdd40102550 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.007+0000 7fdd3e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.007+0000 7fdd3e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40102960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45790/0 (socket says 172.21.15.32:45790) 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.007+0000 7fdd3e59c700 1 -- 172.21.15.32:0/2196976898 learned_addr learned my addr 172.21.15.32:0/2196976898 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.007+0000 7fdd3e59c700 1 -- 172.21.15.32:0/2196976898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd40102fe0 con 0x7fdd40102550 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.007+0000 7fdd3e59c700 1 --2- 172.21.15.32:0/2196976898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40102960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdd300098e0 tx=0x7fdd30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48872af1220d0da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.007+0000 7fdd3d59a700 1 -- 172.21.15.32:0/2196976898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3000b860 con 0x7fdd40102550 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.008+0000 7fdd3d59a700 1 -- 172.21.15.32:0/2196976898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd3000be70 con 0x7fdd40102550 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.008+0000 7fdd3d59a700 1 -- 172.21.15.32:0/2196976898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd30010380 con 0x7fdd40102550 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.008+0000 7fdd44ce2700 1 -- 172.21.15.32:0/2196976898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 msgr2=0x7fdd40102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:59.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.008+0000 7fdd44ce2700 1 --2- 172.21.15.32:0/2196976898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40102960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdd300098e0 tx=0x7fdd30009250 comp rx=0 tx=0).stop 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.008+0000 7fdd44ce2700 1 -- 172.21.15.32:0/2196976898 shutdown_connections 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.008+0000 7fdd44ce2700 1 --2- 172.21.15.32:0/2196976898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40102960 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.008+0000 7fdd44ce2700 1 -- 172.21.15.32:0/2196976898 >> 172.21.15.32:0/2196976898 conn(0x7fdd400fdd10 msgr2=0x7fdd40100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.008+0000 7fdd44ce2700 1 -- 172.21.15.32:0/2196976898 shutdown_connections 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.008+0000 7fdd44ce2700 1 -- 172.21.15.32:0/2196976898 wait complete. 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.009+0000 7fdd44ce2700 1 Processor -- start 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.009+0000 7fdd44ce2700 1 -- start start 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.009+0000 7fdd44ce2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.009+0000 7fdd44ce2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd40079190 con 0x7fdd40102550 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.009+0000 7fdd3e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.009+0000 7fdd3e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45794/0 (socket says 172.21.15.32:45794) 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.009+0000 7fdd3e59c700 1 -- 172.21.15.32:0/729881609 learned_addr learned my addr 172.21.15.32:0/729881609 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.009+0000 7fdd3e59c700 1 -- 172.21.15.32:0/729881609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd300094f0 con 0x7fdd40102550 2024-04-06T14:07:59.785 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.009+0000 7fdd3e59c700 1 --2- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40078c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd30000c00 tx=0x7fdd30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.010+0000 7fdd2effd700 1 -- 172.21.15.32:0/729881609 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd300105f0 con 0x7fdd40102550 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.010+0000 7fdd2effd700 1 -- 172.21.15.32:0/729881609 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd30010c00 con 0x7fdd40102550 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.010+0000 7fdd2effd700 1 -- 172.21.15.32:0/729881609 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3001ac70 con 0x7fdd40102550 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.010+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd40077200 con 0x7fdd40102550 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.010+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd400776f0 con 0x7fdd40102550 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.010+0000 7fdd2effd700 1 -- 172.21.15.32:0/729881609 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fdd30025400 con 0x7fdd40102550 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.011+0000 7fdd2effd700 1 --2- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdd280368d0 0x7fdd28038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.011+0000 7fdd2effd700 1 -- 172.21.15.32:0/729881609 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdd3004a440 con 0x7fdd40102550 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.011+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4004bed0 con 0x7fdd40102550 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.011+0000 7fdd3dd9b700 1 --2- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdd280368d0 0x7fdd28038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.011+0000 7fdd3dd9b700 1 --2- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdd280368d0 0x7fdd28038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009990 tx=0x7fdd34006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.016+0000 7fdd2effd700 1 -- 172.21.15.32:0/729881609 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdd30020070 con 0x7fdd40102550 2024-04-06T14:07:59.786 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.210+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fdd40077b70 con 0x7fdd40102550 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.210+0000 7fdd2effd700 1 -- 172.21.15.32:0/729881609 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fdd3001ae60 con 0x7fdd40102550 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdd280368d0 msgr2=0x7fdd28038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 --2- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdd280368d0 0x7fdd28038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009990 tx=0x7fdd34006e40 comp rx=0 tx=0).stop 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 msgr2=0x7fdd40078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 --2- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40078c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd30000c00 tx=0x7fdd30009250 comp rx=0 tx=0).stop 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 shutdown_connections 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 --2- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fdd280368d0 0x7fdd28038d80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 --2- 172.21.15.32:0/729881609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd40102550 0x7fdd40078c50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 >> 172.21.15.32:0/729881609 conn(0x7fdd400fdd10 msgr2=0x7fdd400fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:07:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 shutdown_connections 2024-04-06T14:07:59.788 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:07:59.212+0000 7fdd44ce2700 1 -- 172.21.15.32:0/729881609 wait complete. 2024-04-06T14:08:01.306 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.396+0000 7ffb2bef6700 1 Processor -- start 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.397+0000 7ffb2bef6700 1 -- start start 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.397+0000 7ffb2bef6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb241051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.397+0000 7ffb2bef6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb24105700 con 0x7ffb24076c40 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.397+0000 7ffb29c92700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb241051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.397+0000 7ffb29c92700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb241051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45804/0 (socket says 172.21.15.32:45804) 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.397+0000 7ffb29c92700 1 -- 172.21.15.32:0/636460331 learned_addr learned my addr 172.21.15.32:0/636460331 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.397+0000 7ffb29c92700 1 -- 172.21.15.32:0/636460331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24105840 con 0x7ffb24076c40 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.397+0000 7ffb29c92700 1 --2- 172.21.15.32:0/636460331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb241051c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffb200098e0 tx=0x7ffb20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d561a3fd2527497f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb28c90700 1 -- 172.21.15.32:0/636460331 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2000b860 con 0x7ffb24076c40 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb28c90700 1 -- 172.21.15.32:0/636460331 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb2000be70 con 0x7ffb24076c40 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb28c90700 1 -- 172.21.15.32:0/636460331 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb20010380 con 0x7ffb24076c40 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb2bef6700 1 -- 172.21.15.32:0/636460331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 msgr2=0x7ffb241051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb2bef6700 1 --2- 172.21.15.32:0/636460331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb241051c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffb200098e0 tx=0x7ffb20009250 comp rx=0 tx=0).stop 2024-04-06T14:08:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb2bef6700 1 -- 172.21.15.32:0/636460331 shutdown_connections 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb2bef6700 1 --2- 172.21.15.32:0/636460331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb241051c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb2bef6700 1 -- 172.21.15.32:0/636460331 >> 172.21.15.32:0/636460331 conn(0x7ffb240fcc30 msgr2=0x7ffb240ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb2bef6700 1 -- 172.21.15.32:0/636460331 shutdown_connections 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.398+0000 7ffb2bef6700 1 -- 172.21.15.32:0/636460331 wait complete. 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.399+0000 7ffb2bef6700 1 Processor -- start 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.399+0000 7ffb2bef6700 1 -- start start 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.399+0000 7ffb2bef6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb24102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.399+0000 7ffb2bef6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb24103050 con 0x7ffb24076c40 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.399+0000 7ffb29c92700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb24102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.399+0000 7ffb29c92700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb24102b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45814/0 (socket says 172.21.15.32:45814) 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.399+0000 7ffb29c92700 1 -- 172.21.15.32:0/4109337260 learned_addr learned my addr 172.21.15.32:0/4109337260 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.399+0000 7ffb29c92700 1 -- 172.21.15.32:0/4109337260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb200094f0 con 0x7ffb24076c40 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.400+0000 7ffb29c92700 1 --2- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb24102b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffb200094c0 tx=0x7ffb20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.400+0000 7ffb1a7fc700 1 -- 172.21.15.32:0/4109337260 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb20010850 con 0x7ffb24076c40 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.400+0000 7ffb1a7fc700 1 -- 172.21.15.32:0/4109337260 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb20010e60 con 0x7ffb24076c40 2024-04-06T14:08:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.400+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb24101120 con 0x7ffb24076c40 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.400+0000 7ffb1a7fc700 1 -- 172.21.15.32:0/4109337260 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2001a430 con 0x7ffb24076c40 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.400+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb24101610 con 0x7ffb24076c40 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.401+0000 7ffb1a7fc700 1 -- 172.21.15.32:0/4109337260 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ffb2001a660 con 0x7ffb24076c40 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.401+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2404bed0 con 0x7ffb24076c40 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.401+0000 7ffb1a7fc700 1 --2- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ffb100368d0 0x7ffb10038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.401+0000 7ffb1a7fc700 1 -- 172.21.15.32:0/4109337260 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ffb2004a7d0 con 0x7ffb24076c40 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.401+0000 7ffb29491700 1 --2- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ffb100368d0 0x7ffb10038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.401+0000 7ffb29491700 1 --2- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ffb100368d0 0x7ffb10038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009990 tx=0x7ffb14006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.406+0000 7ffb1a7fc700 1 -- 172.21.15.32:0/4109337260 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ffb20018350 con 0x7ffb24076c40 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.708+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7ffb24101af0 con 0x7ffb24076c40 2024-04-06T14:08:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.710+0000 7ffb1a7fc700 1 -- 172.21.15.32:0/4109337260 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7ffb20018350 con 0x7ffb24076c40 2024-04-06T14:08:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ffb100368d0 msgr2=0x7ffb10038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 --2- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ffb100368d0 0x7ffb10038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009990 tx=0x7ffb14006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 msgr2=0x7ffb24102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 --2- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb24102b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffb200094c0 tx=0x7ffb20009250 comp rx=0 tx=0).stop 2024-04-06T14:08:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 shutdown_connections 2024-04-06T14:08:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 --2- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ffb100368d0 0x7ffb10038d80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 --2- 172.21.15.32:0/4109337260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb24076c40 0x7ffb24102b10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 >> 172.21.15.32:0/4109337260 conn(0x7ffb240fcc30 msgr2=0x7ffb240fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 shutdown_connections 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-06T14:08:00.712+0000 7ffb2bef6700 1 -- 172.21.15.32:0/4109337260 wait complete. 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout:Or, if you are only running a single cluster on this host: 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:08:01.311 INFO:teuthology.orchestra.run.smithi032.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-06T14:08:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:08:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: ceph telemetry on 2024-04-06T14:08:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:08:01.312 INFO:teuthology.orchestra.run.smithi032.stdout:For more information see: 2024-04-06T14:08:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:08:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-06T14:08:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:08:01.312 INFO:teuthology.orchestra.run.smithi032.stdout:Bootstrap complete. 2024-04-06T14:08:01.342 INFO:tasks.cephadm:Fetching config... 2024-04-06T14:08:01.342 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:08:01.342 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-06T14:08:01.401 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-06T14:08:01.401 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:08:01.401 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-06T14:08:01.460 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-06T14:08:01.460 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:08:01.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/keyring of=/dev/stdout 2024-04-06T14:08:01.528 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-06T14:08:01.528 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:08:01.529 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-06T14:08:01.587 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-06T14:08:01.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPeCi1pyfhH1qnE/PcnFIdg25tUsYGZMRowYE/EclDHwlT101ohI3EZxuYD1b78o2g6tcj4rc6okVdeag5IxGVKhcQtP9fN+Pe3OQNJZDRmnl6O2MShnAcYkmi7wCIcXZoL8h8/MGEU75LY9ff5DB9PwcrXTo1CQfOlvyxhqW4r/yeMsdHr/VFfRGZ64RG1vutQP02iaK/6SbPkb0d9OyR8Ob/E0V2oXZ65CSIgcJNkhUEhtohFARB+rqgzEqTyioOxqki0/GnerikgkmNxJlJ7D7icy01yZ7p7DE1ZJvTRYsW4sHg0jiXlZUjBkoodYWQ3XlByVViLk3EuJ0dlSgyY7ca1dGTPCmXRqOJI1EZSwoxQ2e/8bQUUmn5c5Ccxh3srWKXGmeCUbn2pRJo6+eJznUrYiycjxWqb4YiD0/B/vUz5mojqPlJYUQmiP1zugAfPwkonzWoHxYWTFA093QfYxB7FjcbSPYMf6aBcqUf+SZrnwSG+lsVeJxW3F2Xus= ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T14:08:01.669 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPeCi1pyfhH1qnE/PcnFIdg25tUsYGZMRowYE/EclDHwlT101ohI3EZxuYD1b78o2g6tcj4rc6okVdeag5IxGVKhcQtP9fN+Pe3OQNJZDRmnl6O2MShnAcYkmi7wCIcXZoL8h8/MGEU75LY9ff5DB9PwcrXTo1CQfOlvyxhqW4r/yeMsdHr/VFfRGZ64RG1vutQP02iaK/6SbPkb0d9OyR8Ob/E0V2oXZ65CSIgcJNkhUEhtohFARB+rqgzEqTyioOxqki0/GnerikgkmNxJlJ7D7icy01yZ7p7DE1ZJvTRYsW4sHg0jiXlZUjBkoodYWQ3XlByVViLk3EuJ0dlSgyY7ca1dGTPCmXRqOJI1EZSwoxQ2e/8bQUUmn5c5Ccxh3srWKXGmeCUbn2pRJo6+eJznUrYiycjxWqb4YiD0/B/vUz5mojqPlJYUQmiP1zugAfPwkonzWoHxYWTFA093QfYxB7FjcbSPYMf6aBcqUf+SZrnwSG+lsVeJxW3F2Xus= ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:08:01.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPeCi1pyfhH1qnE/PcnFIdg25tUsYGZMRowYE/EclDHwlT101ohI3EZxuYD1b78o2g6tcj4rc6okVdeag5IxGVKhcQtP9fN+Pe3OQNJZDRmnl6O2MShnAcYkmi7wCIcXZoL8h8/MGEU75LY9ff5DB9PwcrXTo1CQfOlvyxhqW4r/yeMsdHr/VFfRGZ64RG1vutQP02iaK/6SbPkb0d9OyR8Ob/E0V2oXZ65CSIgcJNkhUEhtohFARB+rqgzEqTyioOxqki0/GnerikgkmNxJlJ7D7icy01yZ7p7DE1ZJvTRYsW4sHg0jiXlZUjBkoodYWQ3XlByVViLk3EuJ0dlSgyY7ca1dGTPCmXRqOJI1EZSwoxQ2e/8bQUUmn5c5Ccxh3srWKXGmeCUbn2pRJo6+eJznUrYiycjxWqb4YiD0/B/vUz5mojqPlJYUQmiP1zugAfPwkonzWoHxYWTFA093QfYxB7FjcbSPYMf6aBcqUf+SZrnwSG+lsVeJxW3F2Xus= ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T14:08:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPeCi1pyfhH1qnE/PcnFIdg25tUsYGZMRowYE/EclDHwlT101ohI3EZxuYD1b78o2g6tcj4rc6okVdeag5IxGVKhcQtP9fN+Pe3OQNJZDRmnl6O2MShnAcYkmi7wCIcXZoL8h8/MGEU75LY9ff5DB9PwcrXTo1CQfOlvyxhqW4r/yeMsdHr/VFfRGZ64RG1vutQP02iaK/6SbPkb0d9OyR8Ob/E0V2oXZ65CSIgcJNkhUEhtohFARB+rqgzEqTyioOxqki0/GnerikgkmNxJlJ7D7icy01yZ7p7DE1ZJvTRYsW4sHg0jiXlZUjBkoodYWQ3XlByVViLk3EuJ0dlSgyY7ca1dGTPCmXRqOJI1EZSwoxQ2e/8bQUUmn5c5Ccxh3srWKXGmeCUbn2pRJo6+eJznUrYiycjxWqb4YiD0/B/vUz5mojqPlJYUQmiP1zugAfPwkonzWoHxYWTFA093QfYxB7FjcbSPYMf6aBcqUf+SZrnwSG+lsVeJxW3F2Xus= ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:08:01.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-06T14:08:01.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:01 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/4109337260' entity='client.admin' 2024-04-06T14:08:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:08:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.289+0000 7fc21acc9700 1 -- 172.21.15.32:0/1323632253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 msgr2=0x7fc2140fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.289+0000 7fc21acc9700 1 --2- 172.21.15.32:0/1323632253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 0x7fc2140fd460 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2080048c0 tx=0x7fc208009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.289+0000 7fc21acc9700 1 -- 172.21.15.32:0/1323632253 shutdown_connections 2024-04-06T14:08:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.289+0000 7fc21acc9700 1 --2- 172.21.15.32:0/1323632253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 0x7fc2140fd460 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.289+0000 7fc21acc9700 1 -- 172.21.15.32:0/1323632253 >> 172.21.15.32:0/1323632253 conn(0x7fc2140f70d0 msgr2=0x7fc2140f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.289+0000 7fc21acc9700 1 -- 172.21.15.32:0/1323632253 shutdown_connections 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.289+0000 7fc21acc9700 1 -- 172.21.15.32:0/1323632253 wait complete. 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.290+0000 7fc21acc9700 1 Processor -- start 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.290+0000 7fc21acc9700 1 -- start start 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.290+0000 7fc21acc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 0x7fc21410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.290+0000 7fc21acc9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21410d510 con 0x7fc2140fd050 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.290+0000 7fc218a65700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 0x7fc21410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.290+0000 7fc218a65700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 0x7fc21410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52752/0 (socket says 172.21.15.32:52752) 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.290+0000 7fc218a65700 1 -- 172.21.15.32:0/515820033 learned_addr learned my addr 172.21.15.32:0/515820033 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.290+0000 7fc218a65700 1 -- 172.21.15.32:0/515820033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc208005040 con 0x7fc2140fd050 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.291+0000 7fc218a65700 1 --2- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 0x7fc21410cfd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc208009300 tx=0x7fc208009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.291+0000 7fc2117fa700 1 -- 172.21.15.32:0/515820033 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc208009600 con 0x7fc2140fd050 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.291+0000 7fc2117fa700 1 -- 172.21.15.32:0/515820033 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc208009760 con 0x7fc2140fd050 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.291+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21410d710 con 0x7fc2140fd050 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.291+0000 7fc2117fa700 1 -- 172.21.15.32:0/515820033 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc20801c5b0 con 0x7fc2140fd050 2024-04-06T14:08:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.291+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21410e280 con 0x7fc2140fd050 2024-04-06T14:08:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.292+0000 7fc2117fa700 1 -- 172.21.15.32:0/515820033 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc20801c870 con 0x7fc2140fd050 2024-04-06T14:08:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.292+0000 7fc2117fa700 1 --2- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fc204036880 0x7fc204038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.292+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc214101d30 con 0x7fc2140fd050 2024-04-06T14:08:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.292+0000 7fc2117fa700 1 -- 172.21.15.32:0/515820033 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc20804a990 con 0x7fc2140fd050 2024-04-06T14:08:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.292+0000 7fc213fff700 1 --2- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fc204036880 0x7fc204038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.292+0000 7fc213fff700 1 --2- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fc204036880 0x7fc204038d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc200009990 tx=0x7fc200006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.297+0000 7fc2117fa700 1 -- 172.21.15.32:0/515820033 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc208011db0 con 0x7fc2140fd050 2024-04-06T14:08:05.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.492+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc21405f5f0 con 0x7fc2140fd050 2024-04-06T14:08:05.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.495+0000 7fc2117fa700 1 -- 172.21.15.32:0/515820033 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc208023990 con 0x7fc2140fd050 2024-04-06T14:08:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fc204036880 msgr2=0x7fc204038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 --2- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fc204036880 0x7fc204038d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc200009990 tx=0x7fc200006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 msgr2=0x7fc21410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 --2- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 0x7fc21410cfd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc208009300 tx=0x7fc208009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 shutdown_connections 2024-04-06T14:08:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 --2- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fc204036880 0x7fc204038d30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 --2- 172.21.15.32:0/515820033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2140fd050 0x7fc21410cfd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 >> 172.21.15.32:0/515820033 conn(0x7fc2140f70d0 msgr2=0x7fc214065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 shutdown_connections 2024-04-06T14:08:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:05.498+0000 7fc21acc9700 1 -- 172.21.15.32:0/515820033 wait complete. 2024-04-06T14:08:06.569 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-06T14:08:06.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-06T14:08:06.780 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:06 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/515820033' entity='client.admin' 2024-04-06T14:08:06.780 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:06 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:06.780 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:06 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:06.780 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:06 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:08:06.780 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:06 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:06.780 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:06 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:08:06.780 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:06 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-06T14:08:06.780 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:06 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:08:07.696 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:08:07.699 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:07 smithi032 ceph-mon[39730]: Deploying daemon crash.smithi032 on smithi032 2024-04-06T14:08:09.893 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:09 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:09.893 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:09 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:09.893 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:09 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:09.893 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:09 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:09.893 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:09 smithi032 ceph-mon[39730]: Deploying daemon node-exporter.smithi032 on smithi032 2024-04-06T14:08:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.033+0000 7f6283b70700 1 -- 172.21.15.32:0/250402454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 msgr2=0x7f627c0fb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.033+0000 7f6283b70700 1 --2- 172.21.15.32:0/250402454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 0x7f627c0fb500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f62780048c0 tx=0x7f6278009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.033+0000 7f6283b70700 1 -- 172.21.15.32:0/250402454 shutdown_connections 2024-04-06T14:08:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.033+0000 7f6283b70700 1 --2- 172.21.15.32:0/250402454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 0x7f627c0fb500 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.033+0000 7f6283b70700 1 -- 172.21.15.32:0/250402454 >> 172.21.15.32:0/250402454 conn(0x7f627c0f6850 msgr2=0x7f627c0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.033+0000 7f6283b70700 1 -- 172.21.15.32:0/250402454 shutdown_connections 2024-04-06T14:08:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.033+0000 7f6283b70700 1 -- 172.21.15.32:0/250402454 wait complete. 2024-04-06T14:08:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.033+0000 7f6283b70700 1 Processor -- start 2024-04-06T14:08:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.034+0000 7f6283b70700 1 -- start start 2024-04-06T14:08:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.034+0000 7f6283b70700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 0x7f627c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.034+0000 7f6283b70700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c10d510 con 0x7f627c0fb0f0 2024-04-06T14:08:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.034+0000 7f628190c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 0x7f627c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.034+0000 7f628190c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 0x7f627c10cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52792/0 (socket says 172.21.15.32:52792) 2024-04-06T14:08:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.034+0000 7f628190c700 1 -- 172.21.15.32:0/381160453 learned_addr learned my addr 172.21.15.32:0/381160453 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:08:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.034+0000 7f628190c700 1 -- 172.21.15.32:0/381160453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278005040 con 0x7f627c0fb0f0 2024-04-06T14:08:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.034+0000 7f628190c700 1 --2- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 0x7f627c10cfd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6278009300 tx=0x7f6278009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.035+0000 7f62727fc700 1 -- 172.21.15.32:0/381160453 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6278009600 con 0x7f627c0fb0f0 2024-04-06T14:08:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.035+0000 7f62727fc700 1 -- 172.21.15.32:0/381160453 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6278009760 con 0x7f627c0fb0f0 2024-04-06T14:08:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.035+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627c10d710 con 0x7f627c0fb0f0 2024-04-06T14:08:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.035+0000 7f62727fc700 1 -- 172.21.15.32:0/381160453 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627801c5b0 con 0x7f627c0fb0f0 2024-04-06T14:08:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.035+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627c10e280 con 0x7f627c0fb0f0 2024-04-06T14:08:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.036+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f627c100c40 con 0x7f627c0fb0f0 2024-04-06T14:08:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.037+0000 7f62727fc700 1 -- 172.21.15.32:0/381160453 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f6278013960 con 0x7f627c0fb0f0 2024-04-06T14:08:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.037+0000 7f62727fc700 1 --2- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6268036910 0x7f6268038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.037+0000 7f62727fc700 1 -- 172.21.15.32:0/381160453 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6278049ae0 con 0x7f627c0fb0f0 2024-04-06T14:08:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.037+0000 7f628110b700 1 --2- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6268036910 0x7f6268038dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.037+0000 7f628110b700 1 --2- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6268036910 0x7f6268038dc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f626c009990 tx=0x7f626c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:10.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.042+0000 7f62727fc700 1 -- 172.21.15.32:0/381160453 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f62780137a0 con 0x7f627c0fb0f0 2024-04-06T14:08:10.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.250+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 --> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f627c05f690 con 0x7f6268036910 2024-04-06T14:08:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.253+0000 7f62727fc700 1 -- 172.21.15.32:0/381160453 <== mgr.14162 v2:172.21.15.32:6800/3947155315 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f627c05f690 con 0x7f6268036910 2024-04-06T14:08:10.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6268036910 msgr2=0x7f6268038dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:10.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 --2- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6268036910 0x7f6268038dc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f626c009990 tx=0x7f626c006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 msgr2=0x7f627c10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 --2- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 0x7f627c10cfd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6278009300 tx=0x7f6278009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 shutdown_connections 2024-04-06T14:08:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 --2- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6268036910 0x7f6268038dc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 --2- 172.21.15.32:0/381160453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f627c0fb0f0 0x7f627c10cfd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 >> 172.21.15.32:0/381160453 conn(0x7f627c0f6850 msgr2=0x7f627c0f70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 shutdown_connections 2024-04-06T14:08:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:10.255+0000 7f6283b70700 1 -- 172.21.15.32:0/381160453 wait complete. 2024-04-06T14:08:11.145 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi086 2024-04-06T14:08:11.145 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:08:11.145 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.conf 2024-04-06T14:08:11.167 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:08:11.167 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:08:11.227 INFO:tasks.cephadm:Adding host smithi086 to orchestrator... 2024-04-06T14:08:11.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi086 2024-04-06T14:08:11.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:11 smithi032 ceph-mon[39730]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:08:11.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:11 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:11.497 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:08:12.949 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:12 smithi032 ceph-mon[39730]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:12.950 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:12 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:14.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.055+0000 7fea5359a700 1 -- 172.21.15.32:0/3430868793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 msgr2=0x7fea4c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:14.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.055+0000 7fea5359a700 1 --2- 172.21.15.32:0/3430868793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 0x7fea4c0fd460 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fea480048c0 tx=0x7fea48009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:14.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.055+0000 7fea5359a700 1 -- 172.21.15.32:0/3430868793 shutdown_connections 2024-04-06T14:08:14.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.055+0000 7fea5359a700 1 --2- 172.21.15.32:0/3430868793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 0x7fea4c0fd460 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.055+0000 7fea5359a700 1 -- 172.21.15.32:0/3430868793 >> 172.21.15.32:0/3430868793 conn(0x7fea4c0f70d0 msgr2=0x7fea4c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.055+0000 7fea5359a700 1 -- 172.21.15.32:0/3430868793 shutdown_connections 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.055+0000 7fea5359a700 1 -- 172.21.15.32:0/3430868793 wait complete. 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.056+0000 7fea5359a700 1 Processor -- start 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.056+0000 7fea5359a700 1 -- start start 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.056+0000 7fea5359a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 0x7fea4c10f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.056+0000 7fea5359a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c10f6f0 con 0x7fea4c0fd050 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.056+0000 7fea51336700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 0x7fea4c10f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.056+0000 7fea51336700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 0x7fea4c10f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52806/0 (socket says 172.21.15.32:52806) 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.056+0000 7fea51336700 1 -- 172.21.15.32:0/451009848 learned_addr learned my addr 172.21.15.32:0/451009848 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.056+0000 7fea51336700 1 -- 172.21.15.32:0/451009848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea48005040 con 0x7fea4c0fd050 2024-04-06T14:08:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.056+0000 7fea51336700 1 --2- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 0x7fea4c10f1b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fea48009300 tx=0x7fea48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.057+0000 7fea41ffb700 1 -- 172.21.15.32:0/451009848 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea48009600 con 0x7fea4c0fd050 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.057+0000 7fea41ffb700 1 -- 172.21.15.32:0/451009848 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea48009760 con 0x7fea4c0fd050 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.057+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c10f8f0 con 0x7fea4c0fd050 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.057+0000 7fea41ffb700 1 -- 172.21.15.32:0/451009848 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea4801c5b0 con 0x7fea4c0fd050 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.057+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c110460 con 0x7fea4c0fd050 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.058+0000 7fea41ffb700 1 -- 172.21.15.32:0/451009848 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fea4801c870 con 0x7fea4c0fd050 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.058+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4c101d30 con 0x7fea4c0fd050 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.058+0000 7fea41ffb700 1 --2- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea38036880 0x7fea38038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.058+0000 7fea41ffb700 1 -- 172.21.15.32:0/451009848 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fea48049a90 con 0x7fea4c0fd050 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.058+0000 7fea50b35700 1 --2- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea38036880 0x7fea38038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.058+0000 7fea50b35700 1 --2- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea38036880 0x7fea38038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009990 tx=0x7fea3c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:14.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.063+0000 7fea41ffb700 1 -- 172.21.15.32:0/451009848 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fea48011db0 con 0x7fea4c0fd050 2024-04-06T14:08:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:14.263+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 --> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}) v1 -- 0x7fea4c05f690 con 0x7fea38036880 2024-04-06T14:08:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:14 smithi032 ceph-mon[39730]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:14 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:14 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:14 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:14 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:15.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:15 smithi032 ceph-mon[39730]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:08:15.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:15 smithi032 ceph-mon[39730]: Deploying daemon alertmanager.smithi032 on smithi032 2024-04-06T14:08:16.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.051+0000 7fea41ffb700 1 -- 172.21.15.32:0/451009848 <== mgr.14162 v2:172.21.15.32:6800/3947155315 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fea4c05f690 con 0x7fea38036880 2024-04-06T14:08:16.052 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi086' with addr '172.21.15.86' 2024-04-06T14:08:16.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.054+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea38036880 msgr2=0x7fea38038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:16.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.054+0000 7fea5359a700 1 --2- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea38036880 0x7fea38038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009990 tx=0x7fea3c006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.054+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 msgr2=0x7fea4c10f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.054+0000 7fea5359a700 1 --2- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 0x7fea4c10f1b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fea48009300 tx=0x7fea48009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.054+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 shutdown_connections 2024-04-06T14:08:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.054+0000 7fea5359a700 1 --2- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea38036880 0x7fea38038d30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.054+0000 7fea5359a700 1 --2- 172.21.15.32:0/451009848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea4c0fd050 0x7fea4c10f1b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.054+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 >> 172.21.15.32:0/451009848 conn(0x7fea4c0f70d0 msgr2=0x7fea4c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.055+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 shutdown_connections 2024-04-06T14:08:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:16.055+0000 7fea5359a700 1 -- 172.21.15.32:0/451009848 wait complete. 2024-04-06T14:08:16.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:16 smithi032 ceph-mon[39730]: Deploying cephadm binary to smithi086 2024-04-06T14:08:16.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:16 smithi032 ceph-mon[39730]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:16.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:16 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:17.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-06T14:08:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:08:17.949 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:17 smithi032 ceph-mon[39730]: Added host smithi086 2024-04-06T14:08:17.949 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:17 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:18.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:18 smithi032 ceph-mon[39730]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:20.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:20 smithi032 ceph-mon[39730]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:21.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.764+0000 7f0647640700 1 -- 172.21.15.32:0/2556803856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 msgr2=0x7f06400fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.764+0000 7f0647640700 1 --2- 172.21.15.32:0/2556803856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 0x7f06400fbd60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f063c0048c0 tx=0x7f063c009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.764+0000 7f0647640700 1 -- 172.21.15.32:0/2556803856 shutdown_connections 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.764+0000 7f0647640700 1 --2- 172.21.15.32:0/2556803856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 0x7f06400fbd60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.764+0000 7f0647640700 1 -- 172.21.15.32:0/2556803856 >> 172.21.15.32:0/2556803856 conn(0x7f06400f7090 msgr2=0x7f06400f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.764+0000 7f0647640700 1 -- 172.21.15.32:0/2556803856 shutdown_connections 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.764+0000 7f0647640700 1 -- 172.21.15.32:0/2556803856 wait complete. 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.764+0000 7f0647640700 1 Processor -- start 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f0647640700 1 -- start start 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f0647640700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 0x7f064010cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f0647640700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064010d530 con 0x7f06400fb950 2024-04-06T14:08:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f06453dc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 0x7f064010cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f06453dc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 0x7f064010cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51376/0 (socket says 172.21.15.32:51376) 2024-04-06T14:08:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f06453dc700 1 -- 172.21.15.32:0/1913778795 learned_addr learned my addr 172.21.15.32:0/1913778795 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:08:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f06453dc700 1 -- 172.21.15.32:0/1913778795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c005040 con 0x7f06400fb950 2024-04-06T14:08:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f06453dc700 1 --2- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 0x7f064010cff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f063c000c00 tx=0x7f063c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f0635ffb700 1 -- 172.21.15.32:0/1913778795 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063c009620 con 0x7f06400fb950 2024-04-06T14:08:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f0635ffb700 1 -- 172.21.15.32:0/1913778795 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f063c009780 con 0x7f06400fb950 2024-04-06T14:08:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064010d730 con 0x7f06400fb950 2024-04-06T14:08:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.765+0000 7f0635ffb700 1 -- 172.21.15.32:0/1913778795 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063c01c5b0 con 0x7f06400fb950 2024-04-06T14:08:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.766+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064010e250 con 0x7f06400fb950 2024-04-06T14:08:21.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.767+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0640101d20 con 0x7f06400fb950 2024-04-06T14:08:21.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.767+0000 7f0635ffb700 1 -- 172.21.15.32:0/1913778795 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f063c013a60 con 0x7f06400fb950 2024-04-06T14:08:21.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.767+0000 7f0635ffb700 1 --2- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f062c036880 0x7f062c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:21.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.767+0000 7f0635ffb700 1 -- 172.21.15.32:0/1913778795 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f063c049a30 con 0x7f06400fb950 2024-04-06T14:08:21.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.767+0000 7f0644bdb700 1 --2- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f062c036880 0x7f062c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:21.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.768+0000 7f0644bdb700 1 --2- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f062c036880 0x7f062c038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0630009990 tx=0x7f0630006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:21.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.772+0000 7f0635ffb700 1 -- 172.21.15.32:0/1913778795 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f063c025400 con 0x7f06400fb950 2024-04-06T14:08:21.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.974+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 --> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06400663f0 con 0x7f062c036880 2024-04-06T14:08:21.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.975+0000 7f0635ffb700 1 -- 172.21.15.32:0/1913778795 <== mgr.14162 v2:172.21.15.32:6800/3947155315 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f06400663f0 con 0x7f062c036880 2024-04-06T14:08:21.975 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:08:21.975 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}] 2024-04-06T14:08:21.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f062c036880 msgr2=0x7f062c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:21.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 --2- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f062c036880 0x7f062c038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0630009990 tx=0x7f0630006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:21.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 msgr2=0x7f064010cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:21.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 --2- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 0x7f064010cff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f063c000c00 tx=0x7f063c009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:21.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 shutdown_connections 2024-04-06T14:08:21.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 --2- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f062c036880 0x7f062c038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:21.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 --2- 172.21.15.32:0/1913778795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06400fb950 0x7f064010cff0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:21.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 >> 172.21.15.32:0/1913778795 conn(0x7f06400f7090 msgr2=0x7f0640065ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:21.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 shutdown_connections 2024-04-06T14:08:21.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:21.977+0000 7f0647640700 1 -- 172.21.15.32:0/1913778795 wait complete. 2024-04-06T14:08:22.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:22 smithi032 ceph-mon[39730]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:23.312 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-06T14:08:23.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-06T14:08:23.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:23 smithi032 ceph-mon[39730]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T14:08:24.154 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:08:24.687 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:24 smithi032 ceph-mon[39730]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:24.688 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:24 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:24.688 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:24 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:24.688 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:24 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:24.688 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:24 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:24.688 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:24 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:24.688 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:24 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:24.688 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:24 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T14:08:24.688 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:24 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:25 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T14:08:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:25 smithi032 ceph-mon[39730]: Deploying daemon grafana.smithi032 on smithi032 2024-04-06T14:08:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.835+0000 7f1843727700 1 -- 172.21.15.32:0/4289031526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 msgr2=0x7f183c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.835+0000 7f1843727700 1 --2- 172.21.15.32:0/4289031526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 0x7f183c0fd460 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18380048c0 tx=0x7f1838009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.835+0000 7f1843727700 1 -- 172.21.15.32:0/4289031526 shutdown_connections 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.835+0000 7f1843727700 1 --2- 172.21.15.32:0/4289031526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 0x7f183c0fd460 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.835+0000 7f1843727700 1 -- 172.21.15.32:0/4289031526 >> 172.21.15.32:0/4289031526 conn(0x7f183c0f70d0 msgr2=0x7f183c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.835+0000 7f1843727700 1 -- 172.21.15.32:0/4289031526 shutdown_connections 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.835+0000 7f1843727700 1 -- 172.21.15.32:0/4289031526 wait complete. 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f1843727700 1 Processor -- start 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f1843727700 1 -- start start 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f1843727700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 0x7f183c065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f1843727700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c10fcd0 con 0x7f183c0fd050 2024-04-06T14:08:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f18414c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 0x7f183c065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f18414c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 0x7f183c065bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53054/0 (socket says 172.21.15.32:53054) 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f18414c3700 1 -- 172.21.15.32:0/2186567587 learned_addr learned my addr 172.21.15.32:0/2186567587 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f18414c3700 1 -- 172.21.15.32:0/2186567587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1838005040 con 0x7f183c0fd050 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f18414c3700 1 --2- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 0x7f183c065bd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1838009300 tx=0x7f1838009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.836+0000 7f1831ffb700 1 -- 172.21.15.32:0/2186567587 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1838009600 con 0x7f183c0fd050 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.837+0000 7f1831ffb700 1 -- 172.21.15.32:0/2186567587 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1838009760 con 0x7f183c0fd050 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.837+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c10fe10 con 0x7f183c0fd050 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.837+0000 7f1831ffb700 1 -- 172.21.15.32:0/2186567587 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f183801c5b0 con 0x7f183c0fd050 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.837+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c110300 con 0x7f183c0fd050 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.837+0000 7f1831ffb700 1 -- 172.21.15.32:0/2186567587 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f1838013420 con 0x7f183c0fd050 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.838+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c106060 con 0x7f183c0fd050 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.838+0000 7f1831ffb700 1 --2- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f18280368d0 0x7f1828038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.838+0000 7f1831ffb700 1 -- 172.21.15.32:0/2186567587 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f18380499c0 con 0x7f183c0fd050 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.838+0000 7f1840cc2700 1 --2- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f18280368d0 0x7f1828038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.838+0000 7f1840cc2700 1 --2- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f18280368d0 0x7f1828038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f182c009990 tx=0x7f182c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:26.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:25.843+0000 7f1831ffb700 1 -- 172.21.15.32:0/2186567587 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1838013960 con 0x7f183c0fd050 2024-04-06T14:08:26.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.045+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f183c101d30 con 0x7f183c0fd050 2024-04-06T14:08:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.596+0000 7f1831ffb700 1 -- 172.21.15.32:0/2186567587 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1838011370 con 0x7f183c0fd050 2024-04-06T14:08:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f18280368d0 msgr2=0x7f1828038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 --2- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f18280368d0 0x7f1828038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f182c009990 tx=0x7f182c006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 msgr2=0x7f183c065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 --2- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 0x7f183c065bd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1838009300 tx=0x7f1838009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 shutdown_connections 2024-04-06T14:08:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 --2- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f18280368d0 0x7f1828038d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 --2- 172.21.15.32:0/2186567587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f183c0fd050 0x7f183c065bd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 >> 172.21.15.32:0/2186567587 conn(0x7f183c0f70d0 msgr2=0x7f183c0f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 shutdown_connections 2024-04-06T14:08:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:08:26.599+0000 7f1843727700 1 -- 172.21.15.32:0/2186567587 wait complete. 2024-04-06T14:08:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-04-06T14:08:26.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:26 smithi032 ceph-mon[39730]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:26.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:26 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2186567587' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-06T14:08:26.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:26 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:27.359 INFO:tasks.cephadm:Adding mon.smithi032 on smithi032 2024-04-06T14:08:27.359 INFO:tasks.cephadm:Adding mon.smithi086 on smithi086 2024-04-06T14:08:27.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '2;smithi032:172.21.15.32=smithi032;smithi086:172.21.15.86=smithi086' 2024-04-06T14:08:27.647 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:27.708 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:27.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:27 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2186567587' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-06T14:08:27.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:27 smithi032 ceph-mon[39730]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T14:08:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:28 smithi032 ceph-mon[39730]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:29.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.271+0000 7f69201c2700 1 -- 172.21.15.86:0/2003642702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 msgr2=0x7f69180fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:29.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.271+0000 7f69201c2700 1 --2- 172.21.15.86:0/2003642702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 0x7f69180fd460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f69080048c0 tx=0x7f6908009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:29.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.271+0000 7f69201c2700 1 -- 172.21.15.86:0/2003642702 shutdown_connections 2024-04-06T14:08:29.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.271+0000 7f69201c2700 1 --2- 172.21.15.86:0/2003642702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 0x7f69180fd460 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:29.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.271+0000 7f69201c2700 1 -- 172.21.15.86:0/2003642702 >> 172.21.15.86:0/2003642702 conn(0x7f69180f70d0 msgr2=0x7f69180f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:29.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.271+0000 7f69201c2700 1 -- 172.21.15.86:0/2003642702 shutdown_connections 2024-04-06T14:08:29.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.271+0000 7f69201c2700 1 -- 172.21.15.86:0/2003642702 wait complete. 2024-04-06T14:08:29.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.271+0000 7f69201c2700 1 Processor -- start 2024-04-06T14:08:29.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.271+0000 7f69201c2700 1 -- start start 2024-04-06T14:08:29.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f69201c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 0x7f6918065bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:29.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f69201c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691810fcb0 con 0x7f69180fd050 2024-04-06T14:08:29.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f691df5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 0x7f6918065bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:29.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f691df5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 0x7f6918065bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:41002/0 (socket says 172.21.15.86:41002) 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f691df5e700 1 -- 172.21.15.86:0/2307164347 learned_addr learned my addr 172.21.15.86:0/2307164347 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f691df5e700 1 -- 172.21.15.86:0/2307164347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6908005040 con 0x7f69180fd050 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f691df5e700 1 --2- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 0x7f6918065bf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6908009300 tx=0x7f6908009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f690e7fc700 1 -- 172.21.15.86:0/2307164347 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6908009600 con 0x7f69180fd050 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f690e7fc700 1 -- 172.21.15.86:0/2307164347 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6908009760 con 0x7f69180fd050 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691810fdf0 con 0x7f69180fd050 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f690e7fc700 1 -- 172.21.15.86:0/2307164347 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f690801c5b0 con 0x7f69180fd050 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.272+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69181102e0 con 0x7f69180fd050 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.273+0000 7f690e7fc700 1 -- 172.21.15.86:0/2307164347 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f690801c870 con 0x7f69180fd050 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.273+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6918106060 con 0x7f69180fd050 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.273+0000 7f690e7fc700 1 --2- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6904036880 0x7f6904038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.273+0000 7f690e7fc700 1 -- 172.21.15.86:0/2307164347 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6908049c10 con 0x7f69180fd050 2024-04-06T14:08:29.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.273+0000 7f691d75d700 1 --2- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6904036880 0x7f6904038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.274+0000 7f691d75d700 1 --2- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6904036880 0x7f6904038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6914009990 tx=0x7f6914006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.278+0000 7f690e7fc700 1 -- 172.21.15.86:0/2307164347 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6908011db0 con 0x7f69180fd050 2024-04-06T14:08:29.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.470+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 --> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi032:172.21.15.32=smithi032;smithi086:172.21.15.86=smithi086", "target": ["mon-mgr", ""]}) v1 -- 0x7f691805e400 con 0x7f6904036880 2024-04-06T14:08:29.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.477+0000 7f690e7fc700 1 -- 172.21.15.86:0/2307164347 <== mgr.14162 v2:172.21.15.32:6800/3947155315 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f691805e400 con 0x7f6904036880 2024-04-06T14:08:29.477 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mon update... 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6904036880 msgr2=0x7f6904038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 --2- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6904036880 0x7f6904038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6914009990 tx=0x7f6914006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 msgr2=0x7f6918065bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 --2- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 0x7f6918065bf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6908009300 tx=0x7f6908009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 shutdown_connections 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 --2- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f6904036880 0x7f6904038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 --2- 172.21.15.86:0/2307164347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69180fd050 0x7f6918065bf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 >> 172.21.15.86:0/2307164347 conn(0x7f69180f70d0 msgr2=0x7f69180f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 shutdown_connections 2024-04-06T14:08:29.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:29.478+0000 7f69201c2700 1 -- 172.21.15.86:0/2307164347 wait complete. 2024-04-06T14:08:30.029 DEBUG:teuthology.orchestra.run.smithi086:mon.smithi086> sudo journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi086.service 2024-04-06T14:08:30.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:08:30.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:08:30.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:-- Logs begin at Sat 2024-04-06 13:55:25 UTC. -- 2024-04-06T14:08:30.333 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:30.397 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:30 smithi032 ceph-mon[39730]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi032:172.21.15.32=smithi032;smithi086:172.21.15.86=smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:08:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:30 smithi032 ceph-mon[39730]: Saving service mon spec with placement smithi032:172.21.15.32=smithi032;smithi086:172.21.15.86=smithi086;count:2 2024-04-06T14:08:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:30 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:30 smithi032 ceph-mon[39730]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:31.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.941+0000 7feaa852b700 1 -- 172.21.15.86:0/2349550642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 msgr2=0x7feaa00fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:31.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.941+0000 7feaa852b700 1 --2- 172.21.15.86:0/2349550642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 0x7feaa00fd460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fea940048c0 tx=0x7fea94009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.941+0000 7feaa852b700 1 -- 172.21.15.86:0/2349550642 shutdown_connections 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.941+0000 7feaa852b700 1 --2- 172.21.15.86:0/2349550642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 0x7feaa00fd460 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.941+0000 7feaa852b700 1 -- 172.21.15.86:0/2349550642 >> 172.21.15.86:0/2349550642 conn(0x7feaa00f70d0 msgr2=0x7feaa00f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.941+0000 7feaa852b700 1 -- 172.21.15.86:0/2349550642 shutdown_connections 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.941+0000 7feaa852b700 1 -- 172.21.15.86:0/2349550642 wait complete. 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.942+0000 7feaa852b700 1 Processor -- start 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.942+0000 7feaa852b700 1 -- start start 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.942+0000 7feaa852b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 0x7feaa0065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.942+0000 7feaa852b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa010dac0 con 0x7feaa00fd050 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.942+0000 7feaa62c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 0x7feaa0065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.942+0000 7feaa62c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 0x7feaa0065bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:41024/0 (socket says 172.21.15.86:41024) 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.942+0000 7feaa62c7700 1 -- 172.21.15.86:0/3425539648 learned_addr learned my addr 172.21.15.86:0/3425539648 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.942+0000 7feaa62c7700 1 -- 172.21.15.86:0/3425539648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea94005040 con 0x7feaa00fd050 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.943+0000 7feaa62c7700 1 --2- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 0x7feaa0065bd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fea94000c00 tx=0x7fea94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.943+0000 7fea92ffd700 1 -- 172.21.15.86:0/3425539648 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea94009620 con 0x7feaa00fd050 2024-04-06T14:08:31.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.943+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa01120d0 con 0x7feaa00fd050 2024-04-06T14:08:31.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.943+0000 7fea92ffd700 1 -- 172.21.15.86:0/3425539648 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea94009780 con 0x7feaa00fd050 2024-04-06T14:08:31.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.943+0000 7fea92ffd700 1 -- 172.21.15.86:0/3425539648 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea9401c5b0 con 0x7feaa00fd050 2024-04-06T14:08:31.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.943+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa0112500 con 0x7feaa00fd050 2024-04-06T14:08:31.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.944+0000 7fea92ffd700 1 -- 172.21.15.86:0/3425539648 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fea9401c7f0 con 0x7feaa00fd050 2024-04-06T14:08:31.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.944+0000 7fea92ffd700 1 --2- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea8c036880 0x7fea8c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:31.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.944+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa0106060 con 0x7feaa00fd050 2024-04-06T14:08:31.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.944+0000 7fea92ffd700 1 -- 172.21.15.86:0/3425539648 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fea94049b00 con 0x7feaa00fd050 2024-04-06T14:08:31.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.944+0000 7feaa5ac6700 1 --2- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea8c036880 0x7fea8c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:31.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.944+0000 7feaa5ac6700 1 --2- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea8c036880 0x7fea8c038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fea9c009990 tx=0x7fea9c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:31.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:31.953+0000 7fea92ffd700 1 -- 172.21.15.86:0/3425539648 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fea94011db0 con 0x7feaa00fd050 2024-04-06T14:08:32.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.216+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feaa005f5f0 con 0x7feaa00fd050 2024-04-06T14:08:32.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.216+0000 7fea92ffd700 1 -- 172.21.15.86:0/3425539648 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fea94023990 con 0x7feaa00fd050 2024-04-06T14:08:32.217 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:08:32.217 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:08:32.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea8c036880 msgr2=0x7fea8c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:32.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 --2- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea8c036880 0x7fea8c038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fea9c009990 tx=0x7fea9c006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:32.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 msgr2=0x7feaa0065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:32.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 --2- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 0x7feaa0065bd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fea94000c00 tx=0x7fea94009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:32.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 shutdown_connections 2024-04-06T14:08:32.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 --2- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fea8c036880 0x7fea8c038d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:32.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 --2- 172.21.15.86:0/3425539648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa00fd050 0x7feaa0065bd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:32.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 >> 172.21.15.86:0/3425539648 conn(0x7feaa00f70d0 msgr2=0x7feaa00f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:32.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 shutdown_connections 2024-04-06T14:08:32.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:32.218+0000 7feaa852b700 1 -- 172.21.15.86:0/3425539648 wait complete. 2024-04-06T14:08:32.220 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:08:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:32 smithi032 ceph-mon[39730]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:32 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/3425539648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:08:33.997 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:08:33.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:08:34.297 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:34.362 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:34.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:34 smithi032 ceph-mon[39730]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.037+0000 7f18240d8700 1 -- 172.21.15.86:0/3639478797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 msgr2=0x7f181c0fe580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.037+0000 7f18240d8700 1 --2- 172.21.15.86:0/3639478797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 0x7f181c0fe580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f180c0048c0 tx=0x7f180c009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.037+0000 7f18240d8700 1 -- 172.21.15.86:0/3639478797 shutdown_connections 2024-04-06T14:08:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.037+0000 7f18240d8700 1 --2- 172.21.15.86:0/3639478797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 0x7f181c0fe580 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.037+0000 7f18240d8700 1 -- 172.21.15.86:0/3639478797 >> 172.21.15.86:0/3639478797 conn(0x7f181c0f6850 msgr2=0x7f181c0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.037+0000 7f18240d8700 1 -- 172.21.15.86:0/3639478797 shutdown_connections 2024-04-06T14:08:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.037+0000 7f18240d8700 1 -- 172.21.15.86:0/3639478797 wait complete. 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.038+0000 7f18240d8700 1 Processor -- start 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.038+0000 7f18240d8700 1 -- start start 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.038+0000 7f18240d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 0x7f181c0fd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.038+0000 7f18240d8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0fddc0 con 0x7f181c0fe170 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.038+0000 7f1821e74700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 0x7f181c0fd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.038+0000 7f1821e74700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 0x7f181c0fd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:42774/0 (socket says 172.21.15.86:42774) 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.038+0000 7f1821e74700 1 -- 172.21.15.86:0/962924365 learned_addr learned my addr 172.21.15.86:0/962924365 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.038+0000 7f1821e74700 1 -- 172.21.15.86:0/962924365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c005040 con 0x7f181c0fe170 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.039+0000 7f1821e74700 1 --2- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 0x7f181c0fd880 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f180c009300 tx=0x7f180c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.039+0000 7f18127fc700 1 -- 172.21.15.86:0/962924365 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f180c009600 con 0x7f181c0fe170 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.039+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c0fbe90 con 0x7f181c0fe170 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.039+0000 7f18127fc700 1 -- 172.21.15.86:0/962924365 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f180c009760 con 0x7f181c0fe170 2024-04-06T14:08:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.039+0000 7f18127fc700 1 -- 172.21.15.86:0/962924365 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f180c01c5b0 con 0x7f181c0fe170 2024-04-06T14:08:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.039+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c0fc380 con 0x7f181c0fe170 2024-04-06T14:08:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.040+0000 7f18127fc700 1 -- 172.21.15.86:0/962924365 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f180c013960 con 0x7f181c0fe170 2024-04-06T14:08:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.040+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c05f5f0 con 0x7f181c0fe170 2024-04-06T14:08:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.040+0000 7f18127fc700 1 --2- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1808036880 0x7f1808038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.040+0000 7f18127fc700 1 -- 172.21.15.86:0/962924365 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f180c049a80 con 0x7f181c0fe170 2024-04-06T14:08:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.040+0000 7f1821673700 1 --2- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1808036880 0x7f1808038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.040+0000 7f1821673700 1 --2- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1808036880 0x7f1808038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1818009990 tx=0x7f1818006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:36.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.045+0000 7f18127fc700 1 -- 172.21.15.86:0/962924365 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f180c0098d0 con 0x7f181c0fe170 2024-04-06T14:08:36.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.306+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f181c106b90 con 0x7f181c0fe170 2024-04-06T14:08:36.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.307+0000 7f18127fc700 1 -- 172.21.15.86:0/962924365 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f180c011370 con 0x7f181c0fe170 2024-04-06T14:08:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:08:36.307 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1808036880 msgr2=0x7f1808038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 --2- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1808036880 0x7f1808038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1818009990 tx=0x7f1818006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 msgr2=0x7f181c0fd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 --2- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 0x7f181c0fd880 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f180c009300 tx=0x7f180c009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 shutdown_connections 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 --2- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1808036880 0x7f1808038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 --2- 172.21.15.86:0/962924365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f181c0fe170 0x7f181c0fd880 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 >> 172.21.15.86:0/962924365 conn(0x7f181c0f6850 msgr2=0x7f181c0f7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 shutdown_connections 2024-04-06T14:08:36.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:36.309+0000 7f18240d8700 1 -- 172.21.15.86:0/962924365 wait complete. 2024-04-06T14:08:36.310 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:08:36.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:36 smithi032 ceph-mon[39730]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:36.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:36 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/962924365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:08:37.839 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:08:37.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:08:38.125 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:38.191 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:38.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:38 smithi032 ceph-mon[39730]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:38.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:38 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:38.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:38 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:38.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:38 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:38.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:38 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:38.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:38 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:38.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:38 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.871+0000 7ff837f9e700 1 -- 172.21.15.86:0/1730829967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 msgr2=0x7ff8300fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.871+0000 7ff837f9e700 1 --2- 172.21.15.86:0/1730829967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 0x7ff8300fd460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff8240048c0 tx=0x7ff824009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff837f9e700 1 -- 172.21.15.86:0/1730829967 shutdown_connections 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff837f9e700 1 --2- 172.21.15.86:0/1730829967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 0x7ff8300fd460 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff837f9e700 1 -- 172.21.15.86:0/1730829967 >> 172.21.15.86:0/1730829967 conn(0x7ff8300f70d0 msgr2=0x7ff8300f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff837f9e700 1 -- 172.21.15.86:0/1730829967 shutdown_connections 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff837f9e700 1 -- 172.21.15.86:0/1730829967 wait complete. 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff837f9e700 1 Processor -- start 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff837f9e700 1 -- start start 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff837f9e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 0x7ff830065bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff837f9e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83010fcb0 con 0x7ff8300fd050 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff835d3a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 0x7ff830065bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff835d3a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 0x7ff830065bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:42782/0 (socket says 172.21.15.86:42782) 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.872+0000 7ff835d3a700 1 -- 172.21.15.86:0/3463972218 learned_addr learned my addr 172.21.15.86:0/3463972218 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.873+0000 7ff835d3a700 1 -- 172.21.15.86:0/3463972218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff824005040 con 0x7ff8300fd050 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.873+0000 7ff835d3a700 1 --2- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 0x7ff830065bf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff824000c00 tx=0x7ff824009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.873+0000 7ff8227fc700 1 -- 172.21.15.86:0/3463972218 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff824009620 con 0x7ff8300fd050 2024-04-06T14:08:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.873+0000 7ff8227fc700 1 -- 172.21.15.86:0/3463972218 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff824009780 con 0x7ff8300fd050 2024-04-06T14:08:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.873+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83010fdf0 con 0x7ff8300fd050 2024-04-06T14:08:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.873+0000 7ff8227fc700 1 -- 172.21.15.86:0/3463972218 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff82401c5b0 con 0x7ff8300fd050 2024-04-06T14:08:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.874+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8301102e0 con 0x7ff8300fd050 2024-04-06T14:08:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.874+0000 7ff8227fc700 1 -- 172.21.15.86:0/3463972218 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ff824013a60 con 0x7ff8300fd050 2024-04-06T14:08:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.874+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff830106060 con 0x7ff8300fd050 2024-04-06T14:08:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.874+0000 7ff8227fc700 1 --2- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ff81c036880 0x7ff81c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.875+0000 7ff8227fc700 1 -- 172.21.15.86:0/3463972218 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff824049690 con 0x7ff8300fd050 2024-04-06T14:08:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.875+0000 7ff835539700 1 --2- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ff81c036880 0x7ff81c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.875+0000 7ff835539700 1 --2- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ff81c036880 0x7ff81c038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009990 tx=0x7ff82c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:39.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:39.880+0000 7ff8227fc700 1 -- 172.21.15.86:0/3463972218 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff824024d30 con 0x7ff8300fd050 2024-04-06T14:08:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:39 smithi032 ceph-mon[39730]: Deploying daemon prometheus.smithi032 on smithi032 2024-04-06T14:08:40.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.139+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff830101d30 con 0x7ff8300fd050 2024-04-06T14:08:40.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.139+0000 7ff8227fc700 1 -- 172.21.15.86:0/3463972218 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff824025cb0 con 0x7ff8300fd050 2024-04-06T14:08:40.139 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:08:40.139 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:08:40.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ff81c036880 msgr2=0x7ff81c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:40.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 --2- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ff81c036880 0x7ff81c038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009990 tx=0x7ff82c006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:40.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 msgr2=0x7ff830065bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:40.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 --2- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 0x7ff830065bf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff824000c00 tx=0x7ff824009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:40.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 shutdown_connections 2024-04-06T14:08:40.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 --2- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7ff81c036880 0x7ff81c038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:40.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 --2- 172.21.15.86:0/3463972218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8300fd050 0x7ff830065bf0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:40.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 >> 172.21.15.86:0/3463972218 conn(0x7ff8300f70d0 msgr2=0x7ff8300f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:40.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 shutdown_connections 2024-04-06T14:08:40.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:40.141+0000 7ff837f9e700 1 -- 172.21.15.86:0/3463972218 wait complete. 2024-04-06T14:08:40.142 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:08:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:40 smithi032 ceph-mon[39730]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:40 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/3463972218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:08:41.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:08:41.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:08:42.103 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:42.165 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:42.979 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:42 smithi032 ceph-mon[39730]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:42 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:43.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.764+0000 7fa5311fb700 1 -- 172.21.15.86:0/1223374518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 msgr2=0x7fa52c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:43.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.764+0000 7fa5311fb700 1 --2- 172.21.15.86:0/1223374518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 0x7fa52c0fd460 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0048c0 tx=0x7fa51c009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:43.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.764+0000 7fa5311fb700 1 -- 172.21.15.86:0/1223374518 shutdown_connections 2024-04-06T14:08:43.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.764+0000 7fa5311fb700 1 --2- 172.21.15.86:0/1223374518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 0x7fa52c0fd460 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:43.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.764+0000 7fa5311fb700 1 -- 172.21.15.86:0/1223374518 >> 172.21.15.86:0/1223374518 conn(0x7fa52c0f70d0 msgr2=0x7fa52c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:43.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.764+0000 7fa5311fb700 1 -- 172.21.15.86:0/1223374518 shutdown_connections 2024-04-06T14:08:43.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.764+0000 7fa5311fb700 1 -- 172.21.15.86:0/1223374518 wait complete. 2024-04-06T14:08:43.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.765+0000 7fa5311fb700 1 Processor -- start 2024-04-06T14:08:43.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.765+0000 7fa5311fb700 1 -- start start 2024-04-06T14:08:43.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.765+0000 7fa5311fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 0x7fa52c10bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:43.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.765+0000 7fa5311fb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c10c460 con 0x7fa52c0fd050 2024-04-06T14:08:43.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.765+0000 7fa52ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 0x7fa52c10bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:43.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.765+0000 7fa52ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 0x7fa52c10bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:42808/0 (socket says 172.21.15.86:42808) 2024-04-06T14:08:43.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.765+0000 7fa52ad9d700 1 -- 172.21.15.86:0/178194230 learned_addr learned my addr 172.21.15.86:0/178194230 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:08:43.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.766+0000 7fa52ad9d700 1 -- 172.21.15.86:0/178194230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c005040 con 0x7fa52c0fd050 2024-04-06T14:08:43.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.766+0000 7fa52ad9d700 1 --2- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 0x7fa52c10bf20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009300 tx=0x7fa51c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:43.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.766+0000 7fa5237fe700 1 -- 172.21.15.86:0/178194230 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa51c009600 con 0x7fa52c0fd050 2024-04-06T14:08:43.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.766+0000 7fa5237fe700 1 -- 172.21.15.86:0/178194230 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa51c009760 con 0x7fa52c0fd050 2024-04-06T14:08:43.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.766+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c10c660 con 0x7fa52c0fd050 2024-04-06T14:08:43.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.766+0000 7fa5237fe700 1 -- 172.21.15.86:0/178194230 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa51c01c5b0 con 0x7fa52c0fd050 2024-04-06T14:08:43.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.766+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c10d0f0 con 0x7fa52c0fd050 2024-04-06T14:08:43.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.767+0000 7fa5237fe700 1 -- 172.21.15.86:0/178194230 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fa51c01c870 con 0x7fa52c0fd050 2024-04-06T14:08:43.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.767+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c101d30 con 0x7fa52c0fd050 2024-04-06T14:08:43.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.767+0000 7fa5237fe700 1 --2- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa518036880 0x7fa518038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:43.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.767+0000 7fa5237fe700 1 -- 172.21.15.86:0/178194230 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa51c049c90 con 0x7fa52c0fd050 2024-04-06T14:08:43.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.767+0000 7fa52a59c700 1 --2- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa518036880 0x7fa518038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:43.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.767+0000 7fa52a59c700 1 --2- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa518036880 0x7fa518038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa514009990 tx=0x7fa514006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:43.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:43.772+0000 7fa5237fe700 1 -- 172.21.15.86:0/178194230 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa51c011db0 con 0x7fa52c0fd050 2024-04-06T14:08:44.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.031+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa52c05f5f0 con 0x7fa52c0fd050 2024-04-06T14:08:44.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.032+0000 7fa5237fe700 1 -- 172.21.15.86:0/178194230 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa51c023990 con 0x7fa52c0fd050 2024-04-06T14:08:44.032 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:08:44.032 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa518036880 msgr2=0x7fa518038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 --2- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa518036880 0x7fa518038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa514009990 tx=0x7fa514006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 msgr2=0x7fa52c10bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 --2- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 0x7fa52c10bf20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009300 tx=0x7fa51c009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 shutdown_connections 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 --2- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fa518036880 0x7fa518038d30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 --2- 172.21.15.86:0/178194230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52c0fd050 0x7fa52c10bf20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 >> 172.21.15.86:0/178194230 conn(0x7fa52c0f70d0 msgr2=0x7fa52c0f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 shutdown_connections 2024-04-06T14:08:44.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:44.034+0000 7fa5311fb700 1 -- 172.21.15.86:0/178194230 wait complete. 2024-04-06T14:08:44.035 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:08:44.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:44 smithi032 ceph-mon[39730]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:44.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:44 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/178194230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:08:45.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:08:45.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:08:45.846 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:45.907 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:46 smithi032 ceph-mon[39730]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.424+0000 7fb425f88700 1 -- 172.21.15.86:0/566262118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 msgr2=0x7fb4200fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.424+0000 7fb425f88700 1 --2- 172.21.15.86:0/566262118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 0x7fb4200fce60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb4080048c0 tx=0x7fb408009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.424+0000 7fb425f88700 1 -- 172.21.15.86:0/566262118 shutdown_connections 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.424+0000 7fb425f88700 1 --2- 172.21.15.86:0/566262118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 0x7fb4200fce60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.424+0000 7fb425f88700 1 -- 172.21.15.86:0/566262118 >> 172.21.15.86:0/566262118 conn(0x7fb4200f8190 msgr2=0x7fb4200fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.424+0000 7fb425f88700 1 -- 172.21.15.86:0/566262118 shutdown_connections 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.424+0000 7fb425f88700 1 -- 172.21.15.86:0/566262118 wait complete. 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.424+0000 7fb425f88700 1 Processor -- start 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.424+0000 7fb425f88700 1 -- start start 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.425+0000 7fb425f88700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 0x7fb42010cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.425+0000 7fb425f88700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42010d4e0 con 0x7fb4200fca50 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.425+0000 7fb41f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 0x7fb42010cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.425+0000 7fb41f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 0x7fb42010cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:38244/0 (socket says 172.21.15.86:38244) 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.425+0000 7fb41f7fe700 1 -- 172.21.15.86:0/2878069005 learned_addr learned my addr 172.21.15.86:0/2878069005 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.425+0000 7fb41f7fe700 1 -- 172.21.15.86:0/2878069005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb408005040 con 0x7fb4200fca50 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.425+0000 7fb41f7fe700 1 --2- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 0x7fb42010cfa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb408009300 tx=0x7fb408009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.426+0000 7fb417fff700 1 -- 172.21.15.86:0/2878069005 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb408009600 con 0x7fb4200fca50 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.426+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42010d6e0 con 0x7fb4200fca50 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.426+0000 7fb417fff700 1 -- 172.21.15.86:0/2878069005 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb408009760 con 0x7fb4200fca50 2024-04-06T14:08:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.426+0000 7fb417fff700 1 -- 172.21.15.86:0/2878069005 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb408025560 con 0x7fb4200fca50 2024-04-06T14:08:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.426+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42010e250 con 0x7fb4200fca50 2024-04-06T14:08:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.426+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb420102e20 con 0x7fb4200fca50 2024-04-06T14:08:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.426+0000 7fb417fff700 1 -- 172.21.15.86:0/2878069005 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fb408013960 con 0x7fb4200fca50 2024-04-06T14:08:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.427+0000 7fb417fff700 1 --2- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fb40c036880 0x7fb40c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.427+0000 7fb417fff700 1 -- 172.21.15.86:0/2878069005 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb408049a40 con 0x7fb4200fca50 2024-04-06T14:08:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.427+0000 7fb41effd700 1 --2- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fb40c036880 0x7fb40c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.427+0000 7fb41effd700 1 --2- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fb40c036880 0x7fb40c038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb410009990 tx=0x7fb410006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.433+0000 7fb417fff700 1 -- 172.21.15.86:0/2878069005 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb4080098d0 con 0x7fb4200fca50 2024-04-06T14:08:47.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.690+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb42010ec60 con 0x7fb4200fca50 2024-04-06T14:08:47.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.691+0000 7fb417fff700 1 -- 172.21.15.86:0/2878069005 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb4080117e0 con 0x7fb4200fca50 2024-04-06T14:08:47.691 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:08:47.691 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fb40c036880 msgr2=0x7fb40c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 --2- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fb40c036880 0x7fb40c038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb410009990 tx=0x7fb410006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 msgr2=0x7fb42010cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 --2- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 0x7fb42010cfa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb408009300 tx=0x7fb408009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 shutdown_connections 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 --2- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7fb40c036880 0x7fb40c038d30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 --2- 172.21.15.86:0/2878069005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4200fca50 0x7fb42010cfa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 >> 172.21.15.86:0/2878069005 conn(0x7fb4200f8190 msgr2=0x7fb4200f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 shutdown_connections 2024-04-06T14:08:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:47.693+0000 7fb425f88700 1 -- 172.21.15.86:0/2878069005 wait complete. 2024-04-06T14:08:47.694 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:08:48.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:48 smithi032 ceph-mon[39730]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:48.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:48 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/2878069005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:08:49.320 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:08:49.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:08:49.616 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:49.683 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:50.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:50 smithi032 ceph-mon[39730]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:08:50.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:50 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:50.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:50 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:50.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:50 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:50.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:50 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T14:08:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.283+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/2610244890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 msgr2=0x7f1b380fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.283+0000 7f1b3e6bb700 1 --2- 172.21.15.86:0/2610244890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 0x7f1b380fd460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1b200048c0 tx=0x7f1b20009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.283+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/2610244890 shutdown_connections 2024-04-06T14:08:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.283+0000 7f1b3e6bb700 1 --2- 172.21.15.86:0/2610244890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 0x7f1b380fd460 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.283+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/2610244890 >> 172.21.15.86:0/2610244890 conn(0x7f1b380f70d0 msgr2=0x7f1b380f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.283+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/2610244890 shutdown_connections 2024-04-06T14:08:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.283+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/2610244890 wait complete. 2024-04-06T14:08:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.284+0000 7f1b3e6bb700 1 Processor -- start 2024-04-06T14:08:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.284+0000 7f1b3e6bb700 1 -- start start 2024-04-06T14:08:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.284+0000 7f1b3e6bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 0x7f1b38065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.284+0000 7f1b3e6bb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3810fcd0 con 0x7f1b380fd050 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.284+0000 7f1b37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 0x7f1b38065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.284+0000 7f1b37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 0x7f1b38065bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:38272/0 (socket says 172.21.15.86:38272) 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.284+0000 7f1b37fff700 1 -- 172.21.15.86:0/612378315 learned_addr learned my addr 172.21.15.86:0/612378315 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.284+0000 7f1b37fff700 1 -- 172.21.15.86:0/612378315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b20005040 con 0x7f1b380fd050 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.284+0000 7f1b37fff700 1 --2- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 0x7f1b38065bd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1b20000c00 tx=0x7f1b20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.285+0000 7f1b34ff9700 1 -- 172.21.15.86:0/612378315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b20009620 con 0x7f1b380fd050 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.285+0000 7f1b34ff9700 1 -- 172.21.15.86:0/612378315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b20009780 con 0x7f1b380fd050 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.285+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3810fe10 con 0x7f1b380fd050 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.285+0000 7f1b34ff9700 1 -- 172.21.15.86:0/612378315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2001c5b0 con 0x7f1b380fd050 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.285+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b38110300 con 0x7f1b380fd050 2024-04-06T14:08:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.285+0000 7f1b34ff9700 1 -- 172.21.15.86:0/612378315 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f1b2001c7f0 con 0x7f1b380fd050 2024-04-06T14:08:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.286+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b38106060 con 0x7f1b380fd050 2024-04-06T14:08:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.286+0000 7f1b34ff9700 1 --2- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1b240368d0 0x7f1b24038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.286+0000 7f1b34ff9700 1 -- 172.21.15.86:0/612378315 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1b20049c10 con 0x7f1b380fd050 2024-04-06T14:08:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.286+0000 7f1b377fe700 1 -- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1b240368d0 msgr2=0x7f1b24038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3947155315 2024-04-06T14:08:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.286+0000 7f1b377fe700 1 --2- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1b240368d0 0x7f1b24038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:08:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.291+0000 7f1b34ff9700 1 -- 172.21.15.86:0/612378315 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1b20025920 con 0x7f1b380fd050 2024-04-06T14:08:51.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.487+0000 7f1b377fe700 1 -- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1b240368d0 msgr2=0x7f1b24038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3947155315 2024-04-06T14:08:51.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.487+0000 7f1b377fe700 1 --2- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1b240368d0 0x7f1b24038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T14:08:51.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.553+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b3805f5f0 con 0x7f1b380fd050 2024-04-06T14:08:51.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.553+0000 7f1b34ff9700 1 -- 172.21.15.86:0/612378315 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1b20011370 con 0x7f1b380fd050 2024-04-06T14:08:51.553 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:08:51.554 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.555+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1b240368d0 msgr2=0x7f1b24038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.555+0000 7f1b3e6bb700 1 --2- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1b240368d0 0x7f1b24038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.555+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 msgr2=0x7f1b38065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.555+0000 7f1b3e6bb700 1 --2- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 0x7f1b38065bd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1b20000c00 tx=0x7f1b20009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.555+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 shutdown_connections 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.555+0000 7f1b3e6bb700 1 --2- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f1b240368d0 0x7f1b24038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.555+0000 7f1b3e6bb700 1 --2- 172.21.15.86:0/612378315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b380fd050 0x7f1b38065bd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.555+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 >> 172.21.15.86:0/612378315 conn(0x7f1b380f70d0 msgr2=0x7f1b380f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.555+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 shutdown_connections 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:51.556+0000 7f1b3e6bb700 1 -- 172.21.15.86:0/612378315 wait complete. 2024-04-06T14:08:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:08:52.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:51 smithi032 ceph-mon[39730]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T14:08:52.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:51 smithi032 ceph-mon[39730]: mgrmap e14: smithi032.peohxu(active, since 59s) 2024-04-06T14:08:52.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:51 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/612378315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:08:53.246 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:08:53.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:08:53.530 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:55.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.242+0000 7f5bc1af1700 1 -- 172.21.15.86:0/3202357844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 msgr2=0x7f5bbc0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:55.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.242+0000 7f5bc1af1700 1 --2- 172.21.15.86:0/3202357844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 0x7f5bbc0ffa30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0048c0 tx=0x7f5bac009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:55.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.242+0000 7f5bc1af1700 1 -- 172.21.15.86:0/3202357844 shutdown_connections 2024-04-06T14:08:55.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.242+0000 7f5bc1af1700 1 --2- 172.21.15.86:0/3202357844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 0x7f5bbc0ffa30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:55.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.242+0000 7f5bc1af1700 1 -- 172.21.15.86:0/3202357844 >> 172.21.15.86:0/3202357844 conn(0x7f5bbc0f70d0 msgr2=0x7f5bbc0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:55.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.242+0000 7f5bc1af1700 1 -- 172.21.15.86:0/3202357844 shutdown_connections 2024-04-06T14:08:55.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.242+0000 7f5bc1af1700 1 -- 172.21.15.86:0/3202357844 wait complete. 2024-04-06T14:08:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.243+0000 7f5bc1af1700 1 Processor -- start 2024-04-06T14:08:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.243+0000 7f5bc1af1700 1 -- start start 2024-04-06T14:08:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.243+0000 7f5bc1af1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 0x7f5bbc10cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.243+0000 7f5bc1af1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc10d4e0 con 0x7f5bbc0fd650 2024-04-06T14:08:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.243+0000 7f5bbb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 0x7f5bbc10cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.243+0000 7f5bbb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 0x7f5bbc10cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:51422/0 (socket says 172.21.15.86:51422) 2024-04-06T14:08:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.243+0000 7f5bbb7fe700 1 -- 172.21.15.86:0/4173331106 learned_addr learned my addr 172.21.15.86:0/4173331106 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:08:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.244+0000 7f5bbb7fe700 1 -- 172.21.15.86:0/4173331106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bac005040 con 0x7f5bbc0fd650 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.244+0000 7f5bbb7fe700 1 --2- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 0x7f5bbc10cfa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00b3b0 tx=0x7f5bac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.244+0000 7f5bb3fff700 1 -- 172.21.15.86:0/4173331106 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bac0095d0 con 0x7f5bbc0fd650 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.244+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bbc10d6e0 con 0x7f5bbc0fd650 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.244+0000 7f5bb3fff700 1 -- 172.21.15.86:0/4173331106 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bac009730 con 0x7f5bbc0fd650 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.244+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bbc10e250 con 0x7f5bbc0fd650 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.244+0000 7f5bb3fff700 1 -- 172.21.15.86:0/4173331106 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bac025560 con 0x7f5bbc0fd650 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.245+0000 7f5bb3fff700 1 -- 172.21.15.86:0/4173331106 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f5bac0256c0 con 0x7f5bbc0fd650 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.245+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bbc102e20 con 0x7f5bbc0fd650 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.245+0000 7f5bb3fff700 1 --2- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f5ba80368d0 0x7f5ba8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.245+0000 7f5bb3fff700 1 -- 172.21.15.86:0/4173331106 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5bac049c60 con 0x7f5bbc0fd650 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.245+0000 7f5bbaffd700 1 -- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f5ba80368d0 msgr2=0x7f5ba8038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3947155315 2024-04-06T14:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.245+0000 7f5bbaffd700 1 --2- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f5ba80368d0 0x7f5ba8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:08:55.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.250+0000 7f5bb3fff700 1 -- 172.21.15.86:0/4173331106 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5bac01c3f0 con 0x7f5bbc0fd650 2024-04-06T14:08:55.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.446+0000 7f5bbaffd700 1 -- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f5ba80368d0 msgr2=0x7f5ba8038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3947155315 2024-04-06T14:08:55.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.446+0000 7f5bbaffd700 1 --2- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f5ba80368d0 0x7f5ba8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T14:08:55.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.510+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bbc10ec60 con 0x7f5bbc0fd650 2024-04-06T14:08:55.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.511+0000 7f5bb3fff700 1 -- 172.21.15.86:0/4173331106 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5bac011b60 con 0x7f5bbc0fd650 2024-04-06T14:08:55.511 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:08:55.511 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:08:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f5ba80368d0 msgr2=0x7f5ba8038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:08:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 --2- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f5ba80368d0 0x7f5ba8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 msgr2=0x7f5bbc10cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 --2- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 0x7f5bbc10cfa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00b3b0 tx=0x7f5bac009e30 comp rx=0 tx=0).stop 2024-04-06T14:08:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 shutdown_connections 2024-04-06T14:08:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 --2- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:6800/3947155315,v1:172.21.15.32:6801/3947155315] conn(0x7f5ba80368d0 0x7f5ba8038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 --2- 172.21.15.86:0/4173331106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bbc0fd650 0x7f5bbc10cfa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 >> 172.21.15.86:0/4173331106 conn(0x7f5bbc0f70d0 msgr2=0x7f5bbc1039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 shutdown_connections 2024-04-06T14:08:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:55.513+0000 7f5bc1af1700 1 -- 172.21.15.86:0/4173331106 wait complete. 2024-04-06T14:08:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:08:55.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:55 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/4173331106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:08:57.163 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:08:57.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:08:57.451 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:08:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: Active manager daemon smithi032.peohxu restarted 2024-04-06T14:08:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: Activating manager daemon smithi032.peohxu 2024-04-06T14:08:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: osdmap e5: 0 total, 0 up, 0 in 2024-04-06T14:08:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: mgrmap e15: smithi032.peohxu(active, starting, since 0.00329211s) 2024-04-06T14:08:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:08:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:08:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:08:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:08:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:08:57.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: Manager daemon smithi032.peohxu is now available 2024-04-06T14:08:57.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:08:57.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:08:57.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:57.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:08:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:58 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:08:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:58 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/trash_purge_schedule"}]: dispatch 2024-04-06T14:08:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:58 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:08:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:58 smithi032 ceph-mon[39730]: mgrmap e16: smithi032.peohxu(active, since 1.0054s) 2024-04-06T14:08:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.177+0000 7f98afeae700 1 -- 172.21.15.86:0/2188351952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 msgr2=0x7f98a80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.177+0000 7f98afeae700 1 --2- 172.21.15.86:0/2188351952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 0x7f98a80fd460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9898009c60 tx=0x7f9898004d10 comp rx=0 tx=0).stop 2024-04-06T14:08:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.177+0000 7f98afeae700 1 -- 172.21.15.86:0/2188351952 shutdown_connections 2024-04-06T14:08:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.177+0000 7f98afeae700 1 --2- 172.21.15.86:0/2188351952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 0x7f98a80fd460 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.177+0000 7f98afeae700 1 -- 172.21.15.86:0/2188351952 >> 172.21.15.86:0/2188351952 conn(0x7f98a80f70d0 msgr2=0x7f98a80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.177+0000 7f98afeae700 1 -- 172.21.15.86:0/2188351952 shutdown_connections 2024-04-06T14:08:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.178+0000 7f98afeae700 1 -- 172.21.15.86:0/2188351952 wait complete. 2024-04-06T14:08:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.178+0000 7f98afeae700 1 Processor -- start 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.178+0000 7f98afeae700 1 -- start start 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.178+0000 7f98afeae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 0x7f98a8065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.178+0000 7f98afeae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a810fde0 con 0x7f98a80fd050 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.178+0000 7f98adc4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 0x7f98a8065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.178+0000 7f98adc4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 0x7f98a8065bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:51436/0 (socket says 172.21.15.86:51436) 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.178+0000 7f98adc4a700 1 -- 172.21.15.86:0/2600681450 learned_addr learned my addr 172.21.15.86:0/2600681450 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.179+0000 7f98adc4a700 1 -- 172.21.15.86:0/2600681450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9898009940 con 0x7f98a80fd050 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.179+0000 7f98adc4a700 1 --2- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 0x7f98a8065bd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9898009490 tx=0x7f9898004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.179+0000 7f98a67fc700 1 -- 172.21.15.86:0/2600681450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f989800be40 con 0x7f98a80fd050 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.179+0000 7f98a67fc700 1 -- 172.21.15.86:0/2600681450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f989801d8f0 con 0x7f98a80fd050 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.179+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a810ff20 con 0x7f98a80fd050 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.179+0000 7f98a67fc700 1 -- 172.21.15.86:0/2600681450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9898012850 con 0x7f98a80fd050 2024-04-06T14:08:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.179+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a8110410 con 0x7f98a80fd050 2024-04-06T14:08:59.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.180+0000 7f98a67fc700 1 -- 172.21.15.86:0/2600681450 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f9898019040 con 0x7f98a80fd050 2024-04-06T14:08:59.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.180+0000 7f98a67fc700 1 --2- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9890036710 0x7f9890038bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:08:59.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.180+0000 7f98a67fc700 1 -- 172.21.15.86:0/2600681450 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f989804b0a0 con 0x7f98a80fd050 2024-04-06T14:08:59.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.180+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a8106060 con 0x7f98a80fd050 2024-04-06T14:08:59.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.180+0000 7f98ad449700 1 --2- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9890036710 0x7f9890038bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:08:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.181+0000 7f98ad449700 1 --2- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9890036710 0x7f9890038bc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f989c009990 tx=0x7f989c006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:08:59.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.186+0000 7f98a67fc700 1 -- 172.21.15.86:0/2600681450 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9898021080 con 0x7f98a80fd050 2024-04-06T14:08:59.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.458+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98a804bf00 con 0x7f98a80fd050 2024-04-06T14:08:59.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.459+0000 7f98a67fc700 1 -- 172.21.15.86:0/2600681450 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9898017020 con 0x7f98a80fd050 2024-04-06T14:08:59.459 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:08:59.459 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:08:59.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9890036710 msgr2=0x7f9890038bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:59.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 --2- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9890036710 0x7f9890038bc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f989c009990 tx=0x7f989c006e40 comp rx=0 tx=0).stop 2024-04-06T14:08:59.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 msgr2=0x7f98a8065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:08:59.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 --2- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 0x7f98a8065bd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9898009490 tx=0x7f9898004d10 comp rx=0 tx=0).stop 2024-04-06T14:08:59.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 shutdown_connections 2024-04-06T14:08:59.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 --2- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9890036710 0x7f9890038bc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:59.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 --2- 172.21.15.86:0/2600681450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98a80fd050 0x7f98a8065bd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:08:59.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 >> 172.21.15.86:0/2600681450 conn(0x7f98a80f70d0 msgr2=0x7f98a80f7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:08:59.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 shutdown_connections 2024-04-06T14:08:59.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:08:59.461+0000 7f98afeae700 1 -- 172.21.15.86:0/2600681450 wait complete. 2024-04-06T14:08:59.461 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:09:00.120 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:59 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:00.121 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:59 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:00.121 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:59 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/2600681450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:00.121 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:08:59 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:01.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:09:01.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:09:01.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:00 smithi032 ceph-mon[39730]: [06/Apr/2024:14:08:59] ENGINE Bus STARTING 2024-04-06T14:09:01.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:00 smithi032 ceph-mon[39730]: [06/Apr/2024:14:08:59] ENGINE Serving on https://172.21.15.32:7150 2024-04-06T14:09:01.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:00 smithi032 ceph-mon[39730]: [06/Apr/2024:14:08:59] ENGINE Bus STARTED 2024-04-06T14:09:01.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:00 smithi032 ceph-mon[39730]: mgrmap e17: smithi032.peohxu(active, since 2s) 2024-04-06T14:09:01.421 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:09:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T14:09:03.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:03.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:03.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:03.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:03.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:09:04.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.280+0000 7fe359a2e700 1 -- 172.21.15.86:0/4082420641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 msgr2=0x7fe354101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:04.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.280+0000 7fe359a2e700 1 --2- 172.21.15.86:0/4082420641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 0x7fe354101d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe3440048c0 tx=0x7fe344009e30 comp rx=0 tx=0).stop 2024-04-06T14:09:04.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.280+0000 7fe359a2e700 1 -- 172.21.15.86:0/4082420641 shutdown_connections 2024-04-06T14:09:04.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.280+0000 7fe359a2e700 1 --2- 172.21.15.86:0/4082420641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 0x7fe354101d80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:04.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.280+0000 7fe359a2e700 1 -- 172.21.15.86:0/4082420641 >> 172.21.15.86:0/4082420641 conn(0x7fe3540f7510 msgr2=0x7fe3540f9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:04.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.280+0000 7fe359a2e700 1 -- 172.21.15.86:0/4082420641 shutdown_connections 2024-04-06T14:09:04.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.280+0000 7fe359a2e700 1 -- 172.21.15.86:0/4082420641 wait complete. 2024-04-06T14:09:04.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.280+0000 7fe359a2e700 1 Processor -- start 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.280+0000 7fe359a2e700 1 -- start start 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.281+0000 7fe359a2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 0x7fe354111fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.281+0000 7fe359a2e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe354112520 con 0x7fe3541019b0 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.281+0000 7fe352ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 0x7fe354111fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.281+0000 7fe352ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 0x7fe354111fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:51446/0 (socket says 172.21.15.86:51446) 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.281+0000 7fe352ffd700 1 -- 172.21.15.86:0/3149930154 learned_addr learned my addr 172.21.15.86:0/3149930154 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.281+0000 7fe352ffd700 1 -- 172.21.15.86:0/3149930154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe344005040 con 0x7fe3541019b0 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.281+0000 7fe352ffd700 1 --2- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 0x7fe354111fe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe344007e80 tx=0x7fe344009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.281+0000 7fe33b7fe700 1 -- 172.21.15.86:0/3149930154 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3440093c0 con 0x7fe3541019b0 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.282+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe354112720 con 0x7fe3541019b0 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.282+0000 7fe33b7fe700 1 -- 172.21.15.86:0/3149930154 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe344013420 con 0x7fe3541019b0 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.282+0000 7fe33b7fe700 1 -- 172.21.15.86:0/3149930154 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe34401c5b0 con 0x7fe3541019b0 2024-04-06T14:09:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.282+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe354112b90 con 0x7fe3541019b0 2024-04-06T14:09:04.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.282+0000 7fe33b7fe700 1 -- 172.21.15.86:0/3149930154 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7fe34401c830 con 0x7fe3541019b0 2024-04-06T14:09:04.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.282+0000 7fe33b7fe700 1 --2- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe33c036970 0x7fe33c038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:04.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.283+0000 7fe33b7fe700 1 -- 172.21.15.86:0/3149930154 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fe344009670 con 0x7fe3541019b0 2024-04-06T14:09:04.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.283+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3540fd0b0 con 0x7fe3541019b0 2024-04-06T14:09:04.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.283+0000 7fe3527fc700 1 --2- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe33c036970 0x7fe33c038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:04.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.283+0000 7fe3527fc700 1 --2- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe33c036970 0x7fe33c038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe348009990 tx=0x7fe348006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:04.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.288+0000 7fe33b7fe700 1 -- 172.21.15.86:0/3149930154 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe344023300 con 0x7fe3541019b0 2024-04-06T14:09:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:09:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:09:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:09:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:09:04.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:04.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:04.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:09:04.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:04.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:04.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.557+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe354105b90 con 0x7fe3541019b0 2024-04-06T14:09:04.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.558+0000 7fe33b7fe700 1 -- 172.21.15.86:0/3149930154 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe344011b60 con 0x7fe3541019b0 2024-04-06T14:09:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:09:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:09:04.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe33c036970 msgr2=0x7fe33c038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:04.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 --2- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe33c036970 0x7fe33c038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe348009990 tx=0x7fe348006e40 comp rx=0 tx=0).stop 2024-04-06T14:09:04.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 msgr2=0x7fe354111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:04.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 --2- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 0x7fe354111fe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe344007e80 tx=0x7fe344009e30 comp rx=0 tx=0).stop 2024-04-06T14:09:04.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 shutdown_connections 2024-04-06T14:09:04.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 --2- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe33c036970 0x7fe33c038e20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:04.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 --2- 172.21.15.86:0/3149930154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3541019b0 0x7fe354111fe0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:04.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 >> 172.21.15.86:0/3149930154 conn(0x7fe3540f7510 msgr2=0x7fe3540f8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:04.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 shutdown_connections 2024-04-06T14:09:04.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:04.560+0000 7fe359a2e700 1 -- 172.21.15.86:0/3149930154 wait complete. 2024-04-06T14:09:04.561 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: Deploying daemon crash.smithi086 on smithi086 2024-04-06T14:09:05.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:05 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/3149930154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:06.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:09:06.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:09:06.504 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:08.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.584+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/3182106359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 msgr2=0x7fb0b0101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:08.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.584+0000 7fb0b5d0f700 1 --2- 172.21.15.86:0/3182106359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 0x7fb0b0101db0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00048c0 tx=0x7fb0a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:09:08.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.584+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/3182106359 shutdown_connections 2024-04-06T14:09:08.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.584+0000 7fb0b5d0f700 1 --2- 172.21.15.86:0/3182106359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 0x7fb0b0101db0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:08.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.584+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/3182106359 >> 172.21.15.86:0/3182106359 conn(0x7fb0b00f6c90 msgr2=0x7fb0b00f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:08.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.584+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/3182106359 shutdown_connections 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.584+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/3182106359 wait complete. 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.584+0000 7fb0b5d0f700 1 Processor -- start 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.585+0000 7fb0b5d0f700 1 -- start start 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.585+0000 7fb0b5d0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 0x7fb0b010e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.585+0000 7fb0b5d0f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00fbef0 con 0x7fb0b01019e0 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.585+0000 7fb0af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 0x7fb0b010e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.585+0000 7fb0af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 0x7fb0b010e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:43048/0 (socket says 172.21.15.86:43048) 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.585+0000 7fb0af7fe700 1 -- 172.21.15.86:0/2876340741 learned_addr learned my addr 172.21.15.86:0/2876340741 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.585+0000 7fb0af7fe700 1 -- 172.21.15.86:0/2876340741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a0005040 con 0x7fb0b01019e0 2024-04-06T14:09:08.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.585+0000 7fb0af7fe700 1 --2- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 0x7fb0b010e140 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0007e80 tx=0x7fb0a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:08.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.586+0000 7fb097fff700 1 -- 172.21.15.86:0/2876340741 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0a00093c0 con 0x7fb0b01019e0 2024-04-06T14:09:08.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.586+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b010e6e0 con 0x7fb0b01019e0 2024-04-06T14:09:08.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.586+0000 7fb097fff700 1 -- 172.21.15.86:0/2876340741 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0a0013420 con 0x7fb0b01019e0 2024-04-06T14:09:08.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.586+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b010ebd0 con 0x7fb0b01019e0 2024-04-06T14:09:08.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.586+0000 7fb097fff700 1 -- 172.21.15.86:0/2876340741 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0a001c5b0 con 0x7fb0b01019e0 2024-04-06T14:09:08.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.587+0000 7fb097fff700 1 -- 172.21.15.86:0/2876340741 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7fb0a0009520 con 0x7fb0b01019e0 2024-04-06T14:09:08.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.587+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b00f7360 con 0x7fb0b01019e0 2024-04-06T14:09:08.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.587+0000 7fb097fff700 1 --2- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0980369c0 0x7fb098038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:08.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.587+0000 7fb097fff700 1 -- 172.21.15.86:0/2876340741 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb0a0049a00 con 0x7fb0b01019e0 2024-04-06T14:09:08.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.587+0000 7fb0aeffd700 1 --2- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0980369c0 0x7fb098038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:08.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.587+0000 7fb0aeffd700 1 --2- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0980369c0 0x7fb098038e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009990 tx=0x7fb0a4006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:08.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.592+0000 7fb097fff700 1 -- 172.21.15.86:0/2876340741 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb0a0013590 con 0x7fb0b01019e0 2024-04-06T14:09:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:09:08.921 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:09:08.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.920+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0b004bed0 con 0x7fb0b01019e0 2024-04-06T14:09:08.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.920+0000 7fb097fff700 1 -- 172.21.15.86:0/2876340741 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb0a0049210 con 0x7fb0b01019e0 2024-04-06T14:09:08.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0980369c0 msgr2=0x7fb098038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:08.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 --2- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0980369c0 0x7fb098038e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009990 tx=0x7fb0a4006e40 comp rx=0 tx=0).stop 2024-04-06T14:09:08.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 msgr2=0x7fb0b010e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:08.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 --2- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 0x7fb0b010e140 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0007e80 tx=0x7fb0a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:09:08.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 shutdown_connections 2024-04-06T14:09:08.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 --2- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0980369c0 0x7fb098038e70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:08.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 --2- 172.21.15.86:0/2876340741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b01019e0 0x7fb0b010e140 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:08.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 >> 172.21.15.86:0/2876340741 conn(0x7fb0b00f6c90 msgr2=0x7fb0b00f8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:08.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 shutdown_connections 2024-04-06T14:09:08.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:08.922+0000 7fb0b5d0f700 1 -- 172.21.15.86:0/2876340741 wait complete. 2024-04-06T14:09:08.923 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:09:09.479 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:09 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:09 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:09 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:09 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:09 smithi032 ceph-mon[39730]: Deploying daemon node-exporter.smithi086 on smithi086 2024-04-06T14:09:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:09 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/2876340741' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:10.479 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:09:10.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:09:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:13.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.345+0000 7fdccbba4700 1 -- 172.21.15.86:0/1683867328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 msgr2=0x7fdcc40fdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.345+0000 7fdccbba4700 1 --2- 172.21.15.86:0/1683867328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 0x7fdcc40fdea0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00048c0 tx=0x7fdcc0009e30 comp rx=0 tx=0).stop 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.345+0000 7fdccbba4700 1 -- 172.21.15.86:0/1683867328 shutdown_connections 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.345+0000 7fdccbba4700 1 --2- 172.21.15.86:0/1683867328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 0x7fdcc40fdea0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.345+0000 7fdccbba4700 1 -- 172.21.15.86:0/1683867328 >> 172.21.15.86:0/1683867328 conn(0x7fdcc40f74e0 msgr2=0x7fdcc40f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.345+0000 7fdccbba4700 1 -- 172.21.15.86:0/1683867328 shutdown_connections 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.345+0000 7fdccbba4700 1 -- 172.21.15.86:0/1683867328 wait complete. 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.346+0000 7fdccbba4700 1 Processor -- start 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.346+0000 7fdccbba4700 1 -- start start 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.346+0000 7fdccbba4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 0x7fdcc4113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.346+0000 7fdccbba4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc4113570 con 0x7fdcc40fdad0 2024-04-06T14:09:13.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.346+0000 7fdcc9940700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 0x7fdcc4113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.346+0000 7fdcc9940700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 0x7fdcc4113030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:43058/0 (socket says 172.21.15.86:43058) 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.346+0000 7fdcc9940700 1 -- 172.21.15.86:0/2532954070 learned_addr learned my addr 172.21.15.86:0/2532954070 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.347+0000 7fdcc9940700 1 -- 172.21.15.86:0/2532954070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc0005040 con 0x7fdcc40fdad0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.347+0000 7fdcc9940700 1 --2- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 0x7fdcc4113030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0007e80 tx=0x7fdcc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.347+0000 7fdcba7fc700 1 -- 172.21.15.86:0/2532954070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc00093c0 con 0x7fdcc40fdad0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.347+0000 7fdcba7fc700 1 -- 172.21.15.86:0/2532954070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcc0013420 con 0x7fdcc40fdad0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.347+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc4113770 con 0x7fdcc40fdad0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.347+0000 7fdcba7fc700 1 -- 172.21.15.86:0/2532954070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc001c5b0 con 0x7fdcc40fdad0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.347+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc4113be0 con 0x7fdcc40fdad0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.348+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc4066180 con 0x7fdcc40fdad0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.348+0000 7fdcba7fc700 1 -- 172.21.15.86:0/2532954070 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7fdcc0013a00 con 0x7fdcc40fdad0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.348+0000 7fdcba7fc700 1 --2- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcb0036970 0x7fdcb0038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.348+0000 7fdcba7fc700 1 -- 172.21.15.86:0/2532954070 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fdcc0049aa0 con 0x7fdcc40fdad0 2024-04-06T14:09:13.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.348+0000 7fdcc913f700 1 --2- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcb0036970 0x7fdcb0038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:13.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.349+0000 7fdcc913f700 1 --2- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcb0036970 0x7fdcb0038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009990 tx=0x7fdcb4006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:13.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.353+0000 7fdcba7fc700 1 -- 172.21.15.86:0/2532954070 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdcc0011940 con 0x7fdcc40fdad0 2024-04-06T14:09:13.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.634+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcc404bed0 con 0x7fdcc40fdad0 2024-04-06T14:09:13.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.635+0000 7fdcba7fc700 1 -- 172.21.15.86:0/2532954070 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fdcc0011370 con 0x7fdcc40fdad0 2024-04-06T14:09:13.636 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:09:13.636 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:09:13.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.639+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcb0036970 msgr2=0x7fdcb0038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:13.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.639+0000 7fdccbba4700 1 --2- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcb0036970 0x7fdcb0038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009990 tx=0x7fdcb4006e40 comp rx=0 tx=0).stop 2024-04-06T14:09:13.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.639+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 msgr2=0x7fdcc4113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:13.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.639+0000 7fdccbba4700 1 --2- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 0x7fdcc4113030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0007e80 tx=0x7fdcc0009e30 comp rx=0 tx=0).stop 2024-04-06T14:09:13.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.640+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 shutdown_connections 2024-04-06T14:09:13.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.640+0000 7fdccbba4700 1 --2- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcb0036970 0x7fdcb0038e20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:13.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.640+0000 7fdccbba4700 1 --2- 172.21.15.86:0/2532954070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc40fdad0 0x7fdcc4113030 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:13.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.640+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 >> 172.21.15.86:0/2532954070 conn(0x7fdcc40f74e0 msgr2=0x7fdcc4103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:13.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.640+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 shutdown_connections 2024-04-06T14:09:13.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:13.640+0000 7fdccbba4700 1 -- 172.21.15.86:0/2532954070 wait complete. 2024-04-06T14:09:13.641 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:09:13.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:13.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:09:13.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:13.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:13.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:13.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:13.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:09:13.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T14:09:13.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:09:13.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:13.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:13 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/2532954070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:14 smithi032 ceph-mon[39730]: Deploying daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:09:16.190 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:09:16.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:09:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:16 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:16 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:16 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:16 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:16 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:09:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:16 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:17 smithi032 ceph-mon[39730]: Deploying daemon mon.smithi086 on smithi086 2024-04-06T14:09:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:18 smithi032 ceph-mon[39730]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:19.677 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:19 smithi086 systemd[1]: Starting Ceph mon.smithi086 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:09:20.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.277+0000 7f9d6adcb700 1 -- 172.21.15.86:0/376291549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 msgr2=0x7f9d64102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:20.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.277+0000 7f9d6adcb700 1 --2- 172.21.15.86:0/376291549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 0x7f9d64102ed0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9d580048c0 tx=0x7f9d58009e30 comp rx=0 tx=0).stop 2024-04-06T14:09:20.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.277+0000 7f9d6adcb700 1 -- 172.21.15.86:0/376291549 shutdown_connections 2024-04-06T14:09:20.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.277+0000 7f9d6adcb700 1 --2- 172.21.15.86:0/376291549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 0x7f9d64102ed0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:20.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.277+0000 7f9d6adcb700 1 -- 172.21.15.86:0/376291549 >> 172.21.15.86:0/376291549 conn(0x7f9d640f8620 msgr2=0x7f9d640faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.277+0000 7f9d6adcb700 1 -- 172.21.15.86:0/376291549 shutdown_connections 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.277+0000 7f9d6adcb700 1 -- 172.21.15.86:0/376291549 wait complete. 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.277+0000 7f9d6adcb700 1 Processor -- start 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.277+0000 7f9d6adcb700 1 -- start start 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.278+0000 7f9d6adcb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 0x7f9d64113050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.278+0000 7f9d6adcb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64113590 con 0x7f9d64102b00 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.278+0000 7f9d68b67700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 0x7f9d64113050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.278+0000 7f9d68b67700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 0x7f9d64113050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.86:37464/0 (socket says 172.21.15.86:37464) 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.278+0000 7f9d68b67700 1 -- 172.21.15.86:0/1214101167 learned_addr learned my addr 172.21.15.86:0/1214101167 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.278+0000 7f9d68b67700 1 -- 172.21.15.86:0/1214101167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d58005040 con 0x7f9d64102b00 2024-04-06T14:09:20.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.279+0000 7f9d68b67700 1 --2- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 0x7f9d64113050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9d58007e80 tx=0x7f9d58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:20.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.279+0000 7f9d617fa700 1 -- 172.21.15.86:0/1214101167 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d580093c0 con 0x7f9d64102b00 2024-04-06T14:09:20.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.279+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d64113790 con 0x7f9d64102b00 2024-04-06T14:09:20.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.279+0000 7f9d617fa700 1 -- 172.21.15.86:0/1214101167 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d58013420 con 0x7f9d64102b00 2024-04-06T14:09:20.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.279+0000 7f9d617fa700 1 -- 172.21.15.86:0/1214101167 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d5801c5b0 con 0x7f9d64102b00 2024-04-06T14:09:20.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.279+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d64113c80 con 0x7f9d64102b00 2024-04-06T14:09:20.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.280+0000 7f9d617fa700 1 -- 172.21.15.86:0/1214101167 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f9d5801c830 con 0x7f9d64102b00 2024-04-06T14:09:20.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.280+0000 7f9d617fa700 1 --2- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9d54036970 0x7f9d54038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:20.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.280+0000 7f9d617fa700 1 -- 172.21.15.86:0/1214101167 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9d5804abb0 con 0x7f9d64102b00 2024-04-06T14:09:20.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.280+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d640fe200 con 0x7f9d64102b00 2024-04-06T14:09:20.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.280+0000 7f9d63fff700 1 --2- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9d54036970 0x7f9d54038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:20.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.281+0000 7f9d63fff700 1 --2- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9d54036970 0x7f9d54038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009990 tx=0x7f9d50006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:20.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.286+0000 7f9d617fa700 1 -- 172.21.15.86:0/1214101167 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9d58023300 con 0x7f9d64102b00 2024-04-06T14:09:20.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.557+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d6405f5f0 con 0x7f9d64102b00 2024-04-06T14:09:20.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.558+0000 7f9d617fa700 1 -- 172.21.15.86:0/1214101167 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9d58034090 con 0x7f9d64102b00 2024-04-06T14:09:20.558 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:09:20.558 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:06:45.491491Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.561+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9d54036970 msgr2=0x7f9d54038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.561+0000 7f9d6adcb700 1 --2- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9d54036970 0x7f9d54038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009990 tx=0x7f9d50006e40 comp rx=0 tx=0).stop 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.562+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 msgr2=0x7f9d64113050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.562+0000 7f9d6adcb700 1 --2- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 0x7f9d64113050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9d58007e80 tx=0x7f9d58009e30 comp rx=0 tx=0).stop 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.562+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 shutdown_connections 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.562+0000 7f9d6adcb700 1 --2- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9d54036970 0x7f9d54038e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.562+0000 7f9d6adcb700 1 --2- 172.21.15.86:0/1214101167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64102b00 0x7f9d64113050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.562+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 >> 172.21.15.86:0/1214101167 conn(0x7f9d640f8620 msgr2=0x7f9d640f9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.562+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 shutdown_connections 2024-04-06T14:09:20.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:20.562+0000 7f9d6adcb700 1 -- 172.21.15.86:0/1214101167 wait complete. 2024-04-06T14:09:20.563 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable), process ceph-mon, pid 7 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pidfile_write: ignore empty --pid-file 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: load: jerasure load: lrc 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: RocksDB version: 6.15.5 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Compile date Apr 5 2024 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: DB SUMMARY 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: DB Session ID: DHW0Z1XQNFPI8XE5ZPCZ 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: CURRENT file: CURRENT 2024-04-06T14:09:20.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi086/store.db dir, Total Num: 0, files: 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi086/store.db: 000004.log size: 511 ; 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.error_if_exists: 0 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.create_if_missing: 0 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.paranoid_checks: 1 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.env: 0x556b70c61a80 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.fs: Posix File System 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.info_log: 0x556b72f75c00 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.statistics: (nil) 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.use_fsync: 0 2024-04-06T14:09:20.881 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_log_file_size: 0 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.allow_fallocate: 1 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.use_direct_reads: 0 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.db_log_dir: 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi086/store.db 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T14:09:20.882 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.write_buffer_manager: 0x556b73cea9f0 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T14:09:20.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.unordered_write: 0 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.row_cache: None 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.wal_filter: None 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.preserve_deletes: 0 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.two_write_queues: 0 2024-04-06T14:09:20.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.atomic_flush: 0 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.log_readahead_size: 0 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_background_jobs: 2 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_background_compactions: -1 2024-04-06T14:09:20.885 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_subcompactions: 1 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_open_files: -1 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T14:09:20.886 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_background_flushes: -1 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Compression algorithms supported: 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: kZSTD supported: 0 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: kXpressCompression supported: 0 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: kZlibCompression supported: 1 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: kSnappyCompression supported: 1 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: kLZ4Compression supported: 1 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: kBZip2Compression supported: 0 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000003 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.merge_operator: 2024-04-06T14:09:20.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_filter: None 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556b72fcaf58) 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_type: 0 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_block_index_type: 0 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_shortening: 1 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: hash_index_allow_collision: 1 2024-04-06T14:09:20.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: checksum: 1 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: no_block_cache: 0 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache: 0x556b72ffd090 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_options: 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: capacity : 536870912 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: num_shard_bits : 4 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: strict_capacity_limit : 0 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_compressed: (nil) 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: persistent_cache: (nil) 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_size: 4096 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_size_deviation: 10 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_restart_interval: 16 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_block_restart_interval: 1 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: metadata_block_size: 4096 2024-04-06T14:09:20.889 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: partition_filters: 0 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: use_delta_encoding: 1 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: whole_key_filtering: 1 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: verify_compression: 0 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: format_version: 4 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: enable_index_compression: 1 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_align: 0 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compression: NoCompression 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T14:09:20.890 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.num_levels: 7 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T14:09:20.891 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T14:09:20.892 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.arena_block_size: 4194304 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T14:09:20.893 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.table_properties_collectors: 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.inplace_update_support: 0 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.bloom_locality: 0 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.max_successive_merges: 0 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T14:09:20.894 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.ttl: 2592000 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.enable_blob_files: false 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.min_blob_size: 0 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712412560688577, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-06T14:09:20.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712412560688852, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712412560, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "086c9831-528e-4fb2-9455-8c93a6754a25", "db_session_id": "DHW0Z1XQNFPI8XE5ZPCZ"}} 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712412560689162, "job": 1, "event": "recovery_finished"} 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi086/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x556b72f91180 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: DB pointer 0x556b73066000 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** DB Stats ** 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T14:09:20.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:09:20.897 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-04-06T14:09:20.898 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T14:09:20.899 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086 does not exist in monmap, will attempt to join an existing cluster 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: using public_addr v2:172.21.15.86:0/0 -> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: starting mon.smithi086 rank -1 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi086 fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(???) e0 preinit fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).mds e1 new map 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).mds e1 print_map 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: e1 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:09:20.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: legacy client fscid: -1 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: No filesystems configured 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Deploying daemon node-exporter.smithi032 on smithi032 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:09:20.902 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Deploying daemon alertmanager.smithi032 on smithi032 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Deploying cephadm binary to smithi086 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Added host smithi086 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.903 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Deploying daemon grafana.smithi032 on smithi032 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2186567587' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2186567587' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi032:172.21.15.32=smithi032;smithi086:172.21.15.86=smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Saving service mon spec with placement smithi032:172.21.15.32=smithi032;smithi086:172.21.15.86=smithi086;count:2 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.904 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/3425539648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/962924365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Deploying daemon prometheus.smithi032 on smithi032 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/3463972218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/178194230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/2878069005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14162 172.21.15.32:0/929169365' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mgrmap e14: smithi032.peohxu(active, since 59s) 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/612378315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.906 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/4173331106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Active manager daemon smithi032.peohxu restarted 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Activating manager daemon smithi032.peohxu 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: osdmap e5: 0 total, 0 up, 0 in 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mgrmap e15: smithi032.peohxu(active, starting, since 0.00329211s) 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Manager daemon smithi032.peohxu is now available 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:09:20.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/trash_purge_schedule"}]: dispatch 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mgrmap e16: smithi032.peohxu(active, since 1.0054s) 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/2600681450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: [06/Apr/2024:14:08:59] ENGINE Bus STARTING 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: [06/Apr/2024:14:08:59] ENGINE Serving on https://172.21.15.32:7150 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: [06/Apr/2024:14:08:59] ENGINE Bus STARTED 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mgrmap e17: smithi032.peohxu(active, since 2s) 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.908 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:20.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:09:20.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Deploying daemon crash.smithi086 on smithi086 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/3149930154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Deploying daemon node-exporter.smithi086 on smithi086 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/2876340741' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/2532954070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Deploying daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: Deploying daemon mon.smithi086 on smithi086 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:20.912 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:20 smithi086 ceph-mon[47925]: mon.smithi086@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-06T14:09:21.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:21 smithi086 bash[47850]: 8dd9df00b36721ee20804b4dbddf8930df9fe688e5a3091e749c983e3587d1f2 2024-04-06T14:09:21.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:21 smithi086 systemd[1]: Started Ceph mon.smithi086 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:09:22.286 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T14:09:22.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T14:09:22.550 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:09:26.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:09:26.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: mon.smithi032 calling monitor election 2024-04-06T14:09:26.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:26.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/688834683' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:09:26.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: mon.smithi086 calling monitor election 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: mon.smithi032 is new leader, mons smithi032,smithi086 in quorum (ranks 0,1) 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: fsmap 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: osdmap e5: 0 total, 0 up, 0 in 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: mgrmap e17: smithi032.peohxu(active, since 28s) 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: overall HEALTH_OK 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: Standby manager daemon smithi086.fyyhxm started 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/688834683' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/688834683' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:09:26.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/688834683' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:09:26.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:26.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:09:26.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: mon.smithi032 calling monitor election 2024-04-06T14:09:26.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:26.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/688834683' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: mon.smithi086 calling monitor election 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: mon.smithi032 is new leader, mons smithi032,smithi086 in quorum (ranks 0,1) 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: fsmap 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: osdmap e5: 0 total, 0 up, 0 in 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: mgrmap e17: smithi032.peohxu(active, since 28s) 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: overall HEALTH_OK 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: Standby manager daemon smithi086.fyyhxm started 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/688834683' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:09:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:26.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/688834683' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:09:26.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/688834683' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:09:26.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:27.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:26 smithi086 ceph-mon[47925]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:27.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:26 smithi086 ceph-mon[47925]: mgrmap e18: smithi032.peohxu(active, since 28s), standbys: smithi086.fyyhxm 2024-04-06T14:09:27.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi086.fyyhxm", "id": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:09:27.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:27.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:27.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:27.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:09:27.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:26 smithi032 ceph-mon[39730]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:26 smithi032 ceph-mon[39730]: mgrmap e18: smithi032.peohxu(active, since 28s), standbys: smithi086.fyyhxm 2024-04-06T14:09:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi086.fyyhxm", "id": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:09:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:09:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:09:27.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.242+0000 7feb6f151700 1 -- 172.21.15.86:0/764363901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb58003580 msgr2=0x7feb58005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:27.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.242+0000 7feb6f151700 1 --2- 172.21.15.86:0/764363901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb58003580 0x7feb58005a00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feb68065630 tx=0x7feb5c00b040 comp rx=0 tx=0).stop 2024-04-06T14:09:27.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.242+0000 7feb6f151700 1 -- 172.21.15.86:0/764363901 shutdown_connections 2024-04-06T14:09:27.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.242+0000 7feb6f151700 1 --2- 172.21.15.86:0/764363901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb58003580 0x7feb58005a00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:27.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.242+0000 7feb6f151700 1 -- 172.21.15.86:0/764363901 >> 172.21.15.86:0/764363901 conn(0x7feb680719d0 msgr2=0x7feb68071dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:27.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.242+0000 7feb6f151700 1 -- 172.21.15.86:0/764363901 shutdown_connections 2024-04-06T14:09:27.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.242+0000 7feb6f151700 1 -- 172.21.15.86:0/764363901 wait complete. 2024-04-06T14:09:27.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb6f151700 1 Processor -- start 2024-04-06T14:09:27.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb6f151700 1 -- start start 2024-04-06T14:09:27.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb6f151700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb58003580 0x7feb68113310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:27.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb6f151700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 0x7feb6810d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:27.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb6f151700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb68113e60 con 0x7feb58003580 2024-04-06T14:09:27.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb6f151700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6810d7b0 con 0x7feb68113850 2024-04-06T14:09:27.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb67fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 0x7feb6810d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb67fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 0x7feb6810d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60756/0 (socket says 172.21.15.86:60756) 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb67fff700 1 -- 172.21.15.86:0/1860769447 learned_addr learned my addr 172.21.15.86:0/1860769447 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb6ceed700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb58003580 0x7feb68113310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb67fff700 1 -- 172.21.15.86:0/1860769447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 msgr2=0x7feb6810d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb67fff700 1 -- 172.21.15.86:0/1860769447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 msgr2=0x7feb6810d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.243+0000 7feb67fff700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 0x7feb6810d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb67fff700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 0x7feb6810d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb6ceed700 1 -- 172.21.15.86:0/1860769447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 msgr2=0x7feb6810d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb6ceed700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 0x7feb6810d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb6ceed700 1 -- 172.21.15.86:0/1860769447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c005040 con 0x7feb58003580 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb6ceed700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb58003580 0x7feb68113310 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7feb5c007c90 tx=0x7feb5c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb657fa700 1 -- 172.21.15.86:0/1860769447 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb5c01b070 con 0x7feb58003580 2024-04-06T14:09:27.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb6810da30 con 0x7feb58003580 2024-04-06T14:09:27.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb657fa700 1 -- 172.21.15.86:0/1860769447 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb5c0208f0 con 0x7feb58003580 2024-04-06T14:09:27.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb657fa700 1 -- 172.21.15.86:0/1860769447 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb5c01f6f0 con 0x7feb58003580 2024-04-06T14:09:27.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.244+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6810df70 con 0x7feb58003580 2024-04-06T14:09:27.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.245+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb6805f5f0 con 0x7feb58003580 2024-04-06T14:09:27.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.246+0000 7feb657fa700 1 -- 172.21.15.86:0/1860769447 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7feb5c020440 con 0x7feb58003580 2024-04-06T14:09:27.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.246+0000 7feb657fa700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7feb4c069360 0x7feb4c06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:27.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.246+0000 7feb657fa700 1 -- 172.21.15.86:0/1860769447 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7feb5c087dc0 con 0x7feb58003580 2024-04-06T14:09:27.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.246+0000 7feb67fff700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7feb4c069360 0x7feb4c06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:27.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.247+0000 7feb67fff700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7feb4c069360 0x7feb4c06b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feb54006fd0 tx=0x7feb54008040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:27.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.251+0000 7feb657fa700 1 -- 172.21.15.86:0/1860769447 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7feb5c058be0 con 0x7feb58003580 2024-04-06T14:09:27.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.522+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb6804bf30 con 0x7feb58003580 2024-04-06T14:09:27.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.523+0000 7feb657fa700 1 -- 172.21.15.86:0/1860769447 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1013 (secure 0 0 0) 0x7feb5c087570 con 0x7feb58003580 2024-04-06T14:09:27.523 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-06T14:09:27.524 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":2,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","modified":"2024-04-06T14:09:20.717899Z","created":"2024-04-06T14:06:45.491491Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-06T14:09:27.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.525+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7feb4c069360 msgr2=0x7feb4c06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:27.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.525+0000 7feb6f151700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7feb4c069360 0x7feb4c06b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feb54006fd0 tx=0x7feb54008040 comp rx=0 tx=0).stop 2024-04-06T14:09:27.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.525+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb58003580 msgr2=0x7feb68113310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:27.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.525+0000 7feb6f151700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb58003580 0x7feb68113310 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7feb5c007c90 tx=0x7feb5c00b040 comp rx=0 tx=0).stop 2024-04-06T14:09:27.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.526+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 shutdown_connections 2024-04-06T14:09:27.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.526+0000 7feb6f151700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7feb4c069360 0x7feb4c06b810 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:27.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.526+0000 7feb6f151700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb58003580 0x7feb68113310 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:27.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.526+0000 7feb6f151700 1 --2- 172.21.15.86:0/1860769447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb68113850 0x7feb6810d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:27.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.526+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 >> 172.21.15.86:0/1860769447 conn(0x7feb680719d0 msgr2=0x7feb68106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:27.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.526+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 shutdown_connections 2024-04-06T14:09:27.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:09:27.526+0000 7feb6f151700 1 -- 172.21.15.86:0/1860769447 wait complete. 2024-04-06T14:09:27.527 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 2 2024-04-06T14:09:28.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:27 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/1860769447' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:28.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:27 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:09:28.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:27 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:27 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/1860769447' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T14:09:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:27 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:09:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:27 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:28.437 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-06T14:09:28.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-06T14:09:28.741 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:09:29.085 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:28 smithi032 ceph-mon[39730]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:29.085 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:29.085 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:29.085 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:29.085 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:09:29.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:28 smithi086 ceph-mon[47925]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:29.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:29.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:29.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:29.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:09:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:09:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:09:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:09:30.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:09:30.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:29 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:09:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:09:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:09:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:30.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:09:30.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:09:30.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:29 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:31.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.102+0000 7fbf9c565700 1 -- 172.21.15.32:0/4114050100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 msgr2=0x7fbf94102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:31.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.102+0000 7fbf9c565700 1 --2- 172.21.15.32:0/4114050100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 0x7fbf94102fa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009c60 tx=0x7fbf8c004d10 comp rx=0 tx=0).stop 2024-04-06T14:09:31.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.102+0000 7fbf9c565700 1 -- 172.21.15.32:0/4114050100 shutdown_connections 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.102+0000 7fbf9c565700 1 --2- 172.21.15.32:0/4114050100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 0x7fbf94102fa0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.102+0000 7fbf9c565700 1 -- 172.21.15.32:0/4114050100 >> 172.21.15.32:0/4114050100 conn(0x7fbf940f85e0 msgr2=0x7fbf940fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.103+0000 7fbf9c565700 1 -- 172.21.15.32:0/4114050100 shutdown_connections 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.103+0000 7fbf9c565700 1 -- 172.21.15.32:0/4114050100 wait complete. 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.103+0000 7fbf9c565700 1 Processor -- start 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.103+0000 7fbf9c565700 1 -- start start 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.103+0000 7fbf9c565700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 0x7fbf94113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.103+0000 7fbf9c565700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf94113870 0x7fbf9410d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.103+0000 7fbf9c565700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94113e80 con 0x7fbf94102bd0 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.103+0000 7fbf9c565700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9410d7d0 con 0x7fbf94113870 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.103+0000 7fbf9a301700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 0x7fbf94113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf9a301700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 0x7fbf94113330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59838/0 (socket says 172.21.15.32:59838) 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf9a301700 1 -- 172.21.15.32:0/2726756207 learned_addr learned my addr 172.21.15.32:0/2726756207 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:09:31.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf9a301700 1 -- 172.21.15.32:0/2726756207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf94113870 msgr2=0x7fbf9410d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf9a301700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf94113870 0x7fbf9410d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf9a301700 1 -- 172.21.15.32:0/2726756207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c009940 con 0x7fbf94102bd0 2024-04-06T14:09:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf9a301700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 0x7fbf94113330 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00be10 tx=0x7fbf8c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf92ffd700 1 -- 172.21.15.32:0/2726756207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf8c00e040 con 0x7fbf94102bd0 2024-04-06T14:09:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf92ffd700 1 -- 172.21.15.32:0/2726756207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf8c008ef0 con 0x7fbf94102bd0 2024-04-06T14:09:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9410da50 con 0x7fbf94102bd0 2024-04-06T14:09:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf92ffd700 1 -- 172.21.15.32:0/2726756207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf8c01c6d0 con 0x7fbf94102bd0 2024-04-06T14:09:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.104+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9410e010 con 0x7fbf94102bd0 2024-04-06T14:09:31.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.105+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf94101e40 con 0x7fbf94102bd0 2024-04-06T14:09:31.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.106+0000 7fbf92ffd700 1 -- 172.21.15.32:0/2726756207 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fbf8c012ac0 con 0x7fbf94102bd0 2024-04-06T14:09:31.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.106+0000 7fbf92ffd700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf7c069370 0x7fbf7c06b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:31.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.106+0000 7fbf92ffd700 1 -- 172.21.15.32:0/2726756207 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fbf8c087d60 con 0x7fbf94102bd0 2024-04-06T14:09:31.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.106+0000 7fbf99b00700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf7c069370 0x7fbf7c06b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:31.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.106+0000 7fbf99b00700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf7c069370 0x7fbf7c06b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009930 tx=0x7fbf88006de0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:31.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.111+0000 7fbf92ffd700 1 -- 172.21.15.32:0/2726756207 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbf8c058c00 con 0x7fbf94102bd0 2024-04-06T14:09:31.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:30 smithi086 ceph-mon[47925]: Reconfiguring mon.smithi032 (unknown last config time)... 2024-04-06T14:09:31.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:30 smithi086 ceph-mon[47925]: Reconfiguring daemon mon.smithi032 on smithi032 2024-04-06T14:09:31.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:30 smithi086 ceph-mon[47925]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:30 smithi032 ceph-mon[39730]: Reconfiguring mon.smithi032 (unknown last config time)... 2024-04-06T14:09:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:30 smithi032 ceph-mon[39730]: Reconfiguring daemon mon.smithi032 on smithi032 2024-04-06T14:09:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:30 smithi032 ceph-mon[39730]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:31.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.306+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fbf94106d80 con 0x7fbf94102bd0 2024-04-06T14:09:31.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.307+0000 7fbf92ffd700 1 -- 172.21.15.32:0/2726756207 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7fbf8c0200f0 con 0x7fbf94102bd0 2024-04-06T14:09:31.307 INFO:teuthology.orchestra.run.smithi032.stdout:# minimal ceph.conf for cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:09:31.307 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-04-06T14:09:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:09:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: mon_host = [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.308+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf7c069370 msgr2=0x7fbf7c06b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf7c069370 0x7fbf7c06b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009930 tx=0x7fbf88006de0 comp rx=0 tx=0).stop 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 msgr2=0x7fbf94113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 0x7fbf94113330 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00be10 tx=0x7fbf8c004d10 comp rx=0 tx=0).stop 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 shutdown_connections 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf7c069370 0x7fbf7c06b820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf94102bd0 0x7fbf94113330 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 --2- 172.21.15.32:0/2726756207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf94113870 0x7fbf9410d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 >> 172.21.15.32:0/2726756207 conn(0x7fbf940f85e0 msgr2=0x7fbf940fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 shutdown_connections 2024-04-06T14:09:31.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:31.309+0000 7fbf9c565700 1 -- 172.21.15.32:0/2726756207 wait complete. 2024-04-06T14:09:32.025 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:31 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2726756207' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:32.047 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-06T14:09:32.047 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:09:32.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T14:09:32.081 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:09:32.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:09:32.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:31 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2726756207' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:32.152 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:09:32.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T14:09:32.188 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:09:32.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:09:32.258 INFO:tasks.cephadm:Deploying OSDs... 2024-04-06T14:09:32.259 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:09:32.259 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T14:09:32.278 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T14:09:32.278 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-04-06T14:09:32.338 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T14:09:32.339 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T14:09:32.339 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 161548 Links: 1 2024-04-06T14:09:32.339 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:09:32.339 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T14:09:32.339 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-06 14:08:49.057966847 +0000 2024-04-06T14:09:32.339 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-06 14:01:16.191408070 +0000 2024-04-06T14:09:32.339 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-06 14:01:16.191408070 +0000 2024-04-06T14:09:32.339 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-06T14:09:32.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T14:09:32.406 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-06T14:09:32.406 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-06T14:09:32.406 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000102087 s, 5.0 MB/s 2024-04-06T14:09:32.407 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T14:09:32.468 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-04-06T14:09:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T14:09:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T14:09:32.527 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 161600 Links: 1 2024-04-06T14:09:32.527 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:09:32.527 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T14:09:32.527 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-06 14:08:49.057966847 +0000 2024-04-06T14:09:32.527 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-06 14:01:16.524401434 +0000 2024-04-06T14:09:32.527 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-06 14:01:16.524401434 +0000 2024-04-06T14:09:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-06T14:09:32.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T14:09:32.595 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-06T14:09:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-06T14:09:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000108152 s, 4.7 MB/s 2024-04-06T14:09:32.597 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T14:09:32.658 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-04-06T14:09:32.716 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T14:09:32.716 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T14:09:32.716 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 163777 Links: 1 2024-04-06T14:09:32.716 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:09:32.716 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T14:09:32.716 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-06 14:08:49.057966847 +0000 2024-04-06T14:09:32.716 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-06 14:01:16.875394438 +0000 2024-04-06T14:09:32.716 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-06 14:01:16.875394438 +0000 2024-04-06T14:09:32.716 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-06T14:09:32.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T14:09:32.783 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-06T14:09:32.784 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-06T14:09:32.784 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.0402e-05 s, 5.7 MB/s 2024-04-06T14:09:32.785 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T14:09:32.844 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-04-06T14:09:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T14:09:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T14:09:32.904 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 171130 Links: 1 2024-04-06T14:09:32.904 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:09:32.904 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T14:09:32.904 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-06 14:08:49.057966847 +0000 2024-04-06T14:09:32.905 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-06 14:01:17.206387842 +0000 2024-04-06T14:09:32.905 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-06 14:01:17.206387842 +0000 2024-04-06T14:09:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-06T14:09:32.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T14:09:32.971 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-06T14:09:32.971 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-06T14:09:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000128846 s, 4.0 MB/s 2024-04-06T14:09:32.972 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T14:09:33.032 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:09:33.032 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T14:09:33.054 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T14:09:33.054 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-04-06T14:09:33.119 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T14:09:33.119 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T14:09:33.119 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 168233 Links: 1 2024-04-06T14:09:33.119 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:09:33.119 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T14:09:33.119 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-06 14:09:19.350805194 +0000 2024-04-06T14:09:33.119 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-06 14:01:08.215175251 +0000 2024-04-06T14:09:33.119 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-06 14:01:08.215175251 +0000 2024-04-06T14:09:33.119 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-04-06T14:09:33.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T14:09:33.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:32 smithi086 ceph-mon[47925]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:33.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:32 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:33.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:32 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:33.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:32 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.peohxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:09:33.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:32 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:09:33.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:32 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:33.157 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-06T14:09:33.157 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-06T14:09:33.157 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000334638 s, 1.5 MB/s 2024-04-06T14:09:33.158 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T14:09:33.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:32 smithi032 ceph-mon[39730]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:33.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:32 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:33.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:32 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:33.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:32 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.peohxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:09:33.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:32 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:09:33.188 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:32 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:33.219 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-04-06T14:09:33.279 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T14:09:33.279 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T14:09:33.279 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 160583 Links: 1 2024-04-06T14:09:33.279 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:09:33.279 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T14:09:33.279 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-06 14:09:19.350805194 +0000 2024-04-06T14:09:33.279 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-06 14:01:08.555167712 +0000 2024-04-06T14:09:33.279 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-06 14:01:08.555167712 +0000 2024-04-06T14:09:33.279 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-04-06T14:09:33.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T14:09:33.346 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-06T14:09:33.346 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-06T14:09:33.346 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000216436 s, 2.4 MB/s 2024-04-06T14:09:33.347 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T14:09:33.407 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-04-06T14:09:33.467 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T14:09:33.467 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T14:09:33.467 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 160628 Links: 1 2024-04-06T14:09:33.467 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:09:33.467 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T14:09:33.467 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-06 14:09:19.350805194 +0000 2024-04-06T14:09:33.467 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-06 14:01:08.900160062 +0000 2024-04-06T14:09:33.467 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-06 14:01:08.900160062 +0000 2024-04-06T14:09:33.467 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-04-06T14:09:33.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T14:09:33.533 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-06T14:09:33.534 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-06T14:09:33.534 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000163528 s, 3.1 MB/s 2024-04-06T14:09:33.535 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T14:09:33.593 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-04-06T14:09:33.651 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T14:09:33.651 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T14:09:33.651 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 160698 Links: 1 2024-04-06T14:09:33.651 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:09:33.651 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T14:09:33.652 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-06 14:09:19.351805172 +0000 2024-04-06T14:09:33.652 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-06 14:01:09.246152391 +0000 2024-04-06T14:09:33.652 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-06 14:01:09.246152391 +0000 2024-04-06T14:09:33.652 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-04-06T14:09:33.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T14:09:33.718 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-06T14:09:33.718 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-06T14:09:33.718 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000232849 s, 2.2 MB/s 2024-04-06T14:09:33.718 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T14:09:33.777 INFO:tasks.cephadm:Deploying osd.0 on smithi032 with /dev/vg_nvme/lv_4... 2024-04-06T14:09:33.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T14:09:34.043 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:33 smithi032 ceph-mon[39730]: Reconfiguring mgr.smithi032.peohxu (unknown last config time)... 2024-04-06T14:09:34.044 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:33 smithi032 ceph-mon[39730]: Reconfiguring daemon mgr.smithi032.peohxu on smithi032 2024-04-06T14:09:34.116 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:09:34.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:33 smithi086 ceph-mon[47925]: Reconfiguring mgr.smithi032.peohxu (unknown last config time)... 2024-04-06T14:09:34.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:33 smithi086 ceph-mon[47925]: Reconfiguring daemon mgr.smithi032.peohxu on smithi032 2024-04-06T14:09:34.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:34 smithi032 ceph-mon[39730]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:34.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:34 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:34.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:34 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:34.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:34 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:09:34.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:34 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:34 smithi086 ceph-mon[47925]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:34 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:34 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:34 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:09:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:34 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:36.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:35 smithi086 ceph-mon[47925]: Reconfiguring crash.smithi032 (monmap changed)... 2024-04-06T14:09:36.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:35 smithi086 ceph-mon[47925]: Reconfiguring daemon crash.smithi032 on smithi032 2024-04-06T14:09:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:35 smithi032 ceph-mon[39730]: Reconfiguring crash.smithi032 (monmap changed)... 2024-04-06T14:09:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:35 smithi032 ceph-mon[39730]: Reconfiguring daemon crash.smithi032 on smithi032 2024-04-06T14:09:37.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:36 smithi086 ceph-mon[47925]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:37.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:37.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T14:09:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T14:09:37.161 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-06T14:09:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-06T14:09:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00830281 s, 1.3 GB/s 2024-04-06T14:09:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-06T14:09:37.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:36 smithi032 ceph-mon[39730]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:37.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:37.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:37.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:37 smithi032 ceph-mon[39730]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-04-06T14:09:37.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:37 smithi032 ceph-mon[39730]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-04-06T14:09:38.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:37 smithi086 ceph-mon[47925]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-04-06T14:09:38.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:37 smithi086 ceph-mon[47925]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-04-06T14:09:39.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:38 smithi086 ceph-mon[47925]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:39.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:38 smithi032 ceph-mon[39730]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:39.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi032:vg_nvme/lv_4 2024-04-06T14:09:40.410 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:09:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:40 smithi032 ceph-mon[39730]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:40 smithi086 ceph-mon[47925]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.872+0000 7f0024ebf700 1 -- 172.21.15.32:0/2757022324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200fe450 msgr2=0x7f00200fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.872+0000 7f0024ebf700 1 --2- 172.21.15.32:0/2757022324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200fe450 0x7f00200fe860 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0014004940 tx=0x7f0014009e30 comp rx=0 tx=0).stop 2024-04-06T14:09:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.872+0000 7f0024ebf700 1 -- 172.21.15.32:0/2757022324 shutdown_connections 2024-04-06T14:09:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.872+0000 7f0024ebf700 1 --2- 172.21.15.32:0/2757022324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00200feda0 0x7f00200faee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.872+0000 7f0024ebf700 1 --2- 172.21.15.32:0/2757022324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200fe450 0x7f00200fe860 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.872+0000 7f0024ebf700 1 -- 172.21.15.32:0/2757022324 >> 172.21.15.32:0/2757022324 conn(0x7f00200f6c10 msgr2=0x7f00200f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:09:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.872+0000 7f0024ebf700 1 -- 172.21.15.32:0/2757022324 shutdown_connections 2024-04-06T14:09:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.872+0000 7f0024ebf700 1 -- 172.21.15.32:0/2757022324 wait complete. 2024-04-06T14:09:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.872+0000 7f0024ebf700 1 Processor -- start 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f0024ebf700 1 -- start start 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f0024ebf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00200fe450 0x7f002010d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f0024ebf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200feda0 0x7f002010dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f0024ebf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002010e390 con 0x7f00200feda0 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f0024ebf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002010e500 con 0x7f00200fe450 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f001dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200feda0 0x7f002010dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f001dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200feda0 0x7f002010dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45946/0 (socket says 172.21.15.32:45946) 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f001dd9b700 1 -- 172.21.15.32:0/1914025028 learned_addr learned my addr 172.21.15.32:0/1914025028 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f001e59c700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00200fe450 0x7f002010d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f001dd9b700 1 -- 172.21.15.32:0/1914025028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00200fe450 msgr2=0x7f002010d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f001dd9b700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00200fe450 0x7f002010d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f001dd9b700 1 -- 172.21.15.32:0/1914025028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0014005040 con 0x7f00200feda0 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f001dd9b700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200feda0 0x7f002010dd90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f001000c830 tx=0x7f0010004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:42.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.873+0000 7f000effd700 1 -- 172.21.15.32:0/1914025028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00100106f0 con 0x7f00200feda0 2024-04-06T14:09:42.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.874+0000 7f000effd700 1 -- 172.21.15.32:0/1914025028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0010010d00 con 0x7f00200feda0 2024-04-06T14:09:42.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.874+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0020112f60 con 0x7f00200feda0 2024-04-06T14:09:42.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.874+0000 7f000effd700 1 -- 172.21.15.32:0/1914025028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0010012e90 con 0x7f00200feda0 2024-04-06T14:09:42.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.874+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00201134f0 con 0x7f00200feda0 2024-04-06T14:09:42.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.875+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002004c710 con 0x7f00200feda0 2024-04-06T14:09:42.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.875+0000 7f000effd700 1 -- 172.21.15.32:0/1914025028 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f0010010850 con 0x7f00200feda0 2024-04-06T14:09:42.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.875+0000 7f000effd700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0008069210 0x7f000806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:09:42.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.876+0000 7f000effd700 1 -- 172.21.15.32:0/1914025028 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0010085fa0 con 0x7f00200feda0 2024-04-06T14:09:42.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.876+0000 7f001e59c700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0008069210 0x7f000806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:09:42.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.876+0000 7f001e59c700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0008069210 0x7f000806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0014009fd0 tx=0x7f0014009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:09:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:42.880+0000 7f000effd700 1 -- 172.21.15.32:0/1914025028 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0010056ee0 con 0x7f00200feda0 2024-04-06T14:09:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:42 smithi032 ceph-mon[39730]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:09:43.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:09:43.096+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0020102610 con 0x7f0008069210 2024-04-06T14:09:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:42 smithi086 ceph-mon[47925]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:09:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:09:44.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:09:44.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:44.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:09:44.157 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:09:44.157 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:09:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:44 smithi032 ceph-mon[39730]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:09:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:44 smithi032 ceph-mon[39730]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:44 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:44 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:45.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:44 smithi086 ceph-mon[47925]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:09:45.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:44 smithi086 ceph-mon[47925]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:45.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:44 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:45.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:44 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:46.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:45 smithi032 ceph-mon[39730]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-04-06T14:09:46.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:45 smithi032 ceph-mon[39730]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-04-06T14:09:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:45 smithi086 ceph-mon[47925]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-04-06T14:09:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:45 smithi086 ceph-mon[47925]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-04-06T14:09:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:46 smithi032 ceph-mon[39730]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:47.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:46 smithi086 ceph-mon[47925]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:47 smithi032 ceph-mon[39730]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:48.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:47 smithi086 ceph-mon[47925]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:50 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:50 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:50 smithi032 ceph-mon[39730]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-06T14:09:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:50 smithi032 ceph-mon[39730]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-06T14:09:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:50 smithi032 ceph-mon[39730]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:50.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:50 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:50.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:50 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:09:50.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:50 smithi086 ceph-mon[47925]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-06T14:09:50.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:50 smithi086 ceph-mon[47925]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-06T14:09:50.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:50 smithi086 ceph-mon[47925]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:51 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/95856247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f"}]: dispatch 2024-04-06T14:09:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:51 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/95856247' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f"}]': finished 2024-04-06T14:09:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:51 smithi032 ceph-mon[39730]: osdmap e6: 1 total, 0 up, 1 in 2024-04-06T14:09:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:51 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:09:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:51 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2914099926' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:09:51.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:51 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/95856247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f"}]: dispatch 2024-04-06T14:09:51.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:51 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/95856247' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f"}]': finished 2024-04-06T14:09:51.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:51 smithi086 ceph-mon[47925]: osdmap e6: 1 total, 0 up, 1 in 2024-04-06T14:09:51.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:51 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:09:51.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:51 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2914099926' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:09:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:52 smithi032 ceph-mon[39730]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:52.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:52 smithi086 ceph-mon[47925]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:54.806 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:54 smithi032 ceph-mon[39730]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:55.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:54 smithi086 ceph-mon[47925]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:56.955 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:56 smithi032 ceph-mon[39730]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:57.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:56 smithi086 ceph-mon[47925]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:57.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:09:58.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:57 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:09:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:09:58 smithi032 ceph-mon[39730]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:09:59.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:09:58 smithi086 ceph-mon[47925]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:00.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:00 smithi086 ceph-mon[47925]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:00.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:00 smithi086 ceph-mon[47925]: overall HEALTH_OK 2024-04-06T14:10:00.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:00 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:00.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:00 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:00.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:00 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:10:00.907 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:00 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:00.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:00 smithi032 ceph-mon[39730]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:00.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:00 smithi032 ceph-mon[39730]: overall HEALTH_OK 2024-04-06T14:10:00.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:00 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:00.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:00 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:00.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:00 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:10:00.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:00 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:01.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:01 smithi086 ceph-mon[47925]: Reconfiguring crash.smithi086 (monmap changed)... 2024-04-06T14:10:01.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:01 smithi086 ceph-mon[47925]: Reconfiguring daemon crash.smithi086 on smithi086 2024-04-06T14:10:01.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:01 smithi032 ceph-mon[39730]: Reconfiguring crash.smithi086 (monmap changed)... 2024-04-06T14:10:01.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:01 smithi032 ceph-mon[39730]: Reconfiguring daemon crash.smithi086 on smithi086 2024-04-06T14:10:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: Reconfiguring mgr.smithi086.fyyhxm (monmap changed)... 2024-04-06T14:10:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:10:02.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:10:02.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:02.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: Reconfiguring daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:10:02.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T14:10:02.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: Reconfiguring mgr.smithi086.fyyhxm (monmap changed)... 2024-04-06T14:10:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:10:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:10:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:03.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: Reconfiguring daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:10:03.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T14:10:03.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:03.883 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:03 smithi032 ceph-mon[39730]: Deploying daemon osd.0 on smithi032 2024-04-06T14:10:03.883 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:03.883 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:03.883 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:10:03.883 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:10:03.883 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:03 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:04.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:03 smithi086 ceph-mon[47925]: Deploying daemon osd.0 on smithi032 2024-04-06T14:10:04.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:03 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:04.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:03 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:04.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:03 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:10:04.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:03 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:10:04.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:03 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:04.762 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:04 smithi086 ceph-mon[47925]: Reconfiguring mon.smithi086 (monmap changed)... 2024-04-06T14:10:04.762 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:04 smithi086 ceph-mon[47925]: Reconfiguring daemon mon.smithi086 on smithi086 2024-04-06T14:10:04.762 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:04 smithi086 ceph-mon[47925]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:04.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:04 smithi032 ceph-mon[39730]: Reconfiguring mon.smithi086 (monmap changed)... 2024-04-06T14:10:04.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:04 smithi032 ceph-mon[39730]: Reconfiguring daemon mon.smithi086 on smithi086 2024-04-06T14:10:04.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:04 smithi032 ceph-mon[39730]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:05.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:10:05.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:10:05.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T14:10:05.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T14:10:05.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T14:10:05.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:10:05.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:10:05.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T14:10:05.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T14:10:05.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:05.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:10:07.074 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:06 smithi032 ceph-mon[39730]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:07.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:06 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:07.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:06 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:07.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:06 smithi086 ceph-mon[47925]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:07.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:06 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:07.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:06 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:09.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:08 smithi086 ceph-mon[47925]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:09.158 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:08 smithi032 ceph-mon[39730]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:09 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:09 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:09 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:09 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:10.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:10 smithi032 ceph-mon[39730]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:11.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:10 smithi086 ceph-mon[47925]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:12.892 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 0 on host 'smithi032' 2024-04-06T14:10:12.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.891+0000 7f000effd700 1 -- 172.21.15.32:0/1914025028 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0020102610 con 0x7f0008069210 2024-04-06T14:10:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.893+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0008069210 msgr2=0x7f000806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:10:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.893+0000 7f0024ebf700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0008069210 0x7f000806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0014009fd0 tx=0x7f0014009e30 comp rx=0 tx=0).stop 2024-04-06T14:10:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.893+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200feda0 msgr2=0x7f002010dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:10:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.893+0000 7f0024ebf700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200feda0 0x7f002010dd90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f001000c830 tx=0x7f0010004470 comp rx=0 tx=0).stop 2024-04-06T14:10:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.894+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 shutdown_connections 2024-04-06T14:10:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.894+0000 7f0024ebf700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0008069210 0x7f000806b6c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.894+0000 7f0024ebf700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00200fe450 0x7f002010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.894+0000 7f0024ebf700 1 --2- 172.21.15.32:0/1914025028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00200feda0 0x7f002010dd90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.894+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 >> 172.21.15.32:0/1914025028 conn(0x7f00200f6c10 msgr2=0x7f00200f8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:10:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.894+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 shutdown_connections 2024-04-06T14:10:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:12.894+0000 7f0024ebf700 1 -- 172.21.15.32:0/1914025028 wait complete. 2024-04-06T14:10:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:12 smithi032 ceph-mon[39730]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:10:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:12 smithi032 ceph-mon[39730]: from='osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T14:10:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:13.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:12 smithi086 ceph-mon[47925]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:10:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:12 smithi086 ceph-mon[47925]: from='osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T14:10:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:13.570 DEBUG:teuthology.orchestra.run.smithi032:osd.0> sudo journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-06T14:10:13.572 INFO:tasks.cephadm:Deploying osd.1 on smithi032 with /dev/vg_nvme/lv_3... 2024-04-06T14:10:13.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T14:10:13.605 INFO:journalctl@ceph.osd.0.smithi032.stdout:-- Logs begin at Sat 2024-04-06 13:55:15 UTC. -- 2024-04-06T14:10:14.407 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:10:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:14 smithi032 ceph-mon[39730]: from='osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T14:10:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:14 smithi032 ceph-mon[39730]: osdmap e7: 1 total, 0 up, 1 in 2024-04-06T14:10:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:14 smithi032 ceph-mon[39730]: from='osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:10:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:14 smithi032 ceph-mon[39730]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:14.731 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:10:14 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[63509]: 2024-04-06T14:10:14.418+0000 7f8dd37a2700 -1 osd.0 0 waiting for initial osdmap 2024-04-06T14:10:14.731 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:10:14 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[63509]: 2024-04-06T14:10:14.425+0000 7f8dcbd98700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:10:14.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:14 smithi086 ceph-mon[47925]: from='osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T14:10:14.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:14 smithi086 ceph-mon[47925]: osdmap e7: 1 total, 0 up, 1 in 2024-04-06T14:10:14.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:14.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:14 smithi086 ceph-mon[47925]: from='osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:10:14.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:14 smithi086 ceph-mon[47925]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:15.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:15 smithi032 ceph-mon[39730]: from='osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-06T14:10:15.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:15 smithi032 ceph-mon[39730]: osdmap e8: 1 total, 0 up, 1 in 2024-04-06T14:10:15.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:15.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:15.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:15 smithi086 ceph-mon[47925]: from='osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-06T14:10:15.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:15 smithi086 ceph-mon[47925]: osdmap e8: 1 total, 0 up, 1 in 2024-04-06T14:10:15.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:15.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:16 smithi032 ceph-mon[39730]: purged_snaps scrub starts 2024-04-06T14:10:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:16 smithi032 ceph-mon[39730]: purged_snaps scrub ok 2024-04-06T14:10:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:16 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:16 smithi032 ceph-mon[39730]: osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] boot 2024-04-06T14:10:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:16 smithi032 ceph-mon[39730]: osdmap e9: 1 total, 1 up, 1 in 2024-04-06T14:10:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:16 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:16 smithi032 ceph-mon[39730]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:16.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:16 smithi086 ceph-mon[47925]: purged_snaps scrub starts 2024-04-06T14:10:16.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:16 smithi086 ceph-mon[47925]: purged_snaps scrub ok 2024-04-06T14:10:16.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:16 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:16.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:16 smithi086 ceph-mon[47925]: osd.0 [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] boot 2024-04-06T14:10:16.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:16 smithi086 ceph-mon[47925]: osdmap e9: 1 total, 1 up, 1 in 2024-04-06T14:10:16.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:16 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:10:16.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:16 smithi086 ceph-mon[47925]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T14:10:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:17 smithi032 ceph-mon[39730]: osdmap e10: 1 total, 1 up, 1 in 2024-04-06T14:10:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:10:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:10:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:17.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:10:17.843 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:17 smithi086 ceph-mon[47925]: osdmap e10: 1 total, 1 up, 1 in 2024-04-06T14:10:17.843 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:17.843 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:17.843 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:10:17.843 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:17.843 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:17.843 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:10:17.843 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:17.843 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:10:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T14:10:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T14:10:18.501 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-06T14:10:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-06T14:10:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00862926 s, 1.2 GB/s 2024-04-06T14:10:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-06T14:10:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:18 smithi032 ceph-mon[39730]: Detected new or changed devices on smithi032 2024-04-06T14:10:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:18 smithi032 ceph-mon[39730]: Adjusting osd_memory_target on smithi032 to 12902M 2024-04-06T14:10:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:18 smithi032 ceph-mon[39730]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:18.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:18 smithi086 ceph-mon[47925]: Detected new or changed devices on smithi032 2024-04-06T14:10:18.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:18 smithi086 ceph-mon[47925]: Adjusting osd_memory_target on smithi032 to 12902M 2024-04-06T14:10:18.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:18 smithi086 ceph-mon[47925]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:19.582 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi032:vg_nvme/lv_3 2024-04-06T14:10:20.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:19 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:20.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:19 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:19 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:19 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:10:20.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:20 smithi032 ceph-mon[39730]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:21.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:20 smithi086 ceph-mon[47925]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.043+0000 7fd0bfef3700 1 -- 172.21.15.32:0/2202538549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b80fff70 msgr2=0x7fd0b8102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:10:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.043+0000 7fd0bfef3700 1 --2- 172.21.15.32:0/2202538549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b80fff70 0x7fd0b8102350 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac004940 tx=0x7fd0ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:10:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 -- 172.21.15.32:0/2202538549 shutdown_connections 2024-04-06T14:10:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 --2- 172.21.15.32:0/2202538549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b80fff70 0x7fd0b8102350 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 --2- 172.21.15.32:0/2202538549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0b8066b10 0x7fd0b80ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 -- 172.21.15.32:0/2202538549 >> 172.21.15.32:0/2202538549 conn(0x7fd0b80f7440 msgr2=0x7fd0b80f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 -- 172.21.15.32:0/2202538549 shutdown_connections 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 -- 172.21.15.32:0/2202538549 wait complete. 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 Processor -- start 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 -- start start 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b8066b10 0x7fd0b810d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0b80fff70 0x7fd0b810def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b810e4f0 con 0x7fd0b8066b10 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bfef3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b810ed40 con 0x7fd0b80fff70 2024-04-06T14:10:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bdc8f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b8066b10 0x7fd0b810d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.044+0000 7fd0bdc8f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b8066b10 0x7fd0b810d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38110/0 (socket says 172.21.15.32:38110) 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0bdc8f700 1 -- 172.21.15.32:0/828180231 learned_addr learned my addr 172.21.15.32:0/828180231 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0bd48e700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0b80fff70 0x7fd0b810def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0bdc8f700 1 -- 172.21.15.32:0/828180231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0b80fff70 msgr2=0x7fd0b810def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0bdc8f700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0b80fff70 0x7fd0b810def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0bdc8f700 1 -- 172.21.15.32:0/828180231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac005040 con 0x7fd0b8066b10 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0bd48e700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0b80fff70 0x7fd0b810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0bdc8f700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b8066b10 0x7fd0b810d9b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400c800 tx=0x7fd0b4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0aa7fc700 1 -- 172.21.15.32:0/828180231 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0b400a760 con 0x7fd0b8066b10 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0aa7fc700 1 -- 172.21.15.32:0/828180231 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0b400ad70 con 0x7fd0b8066b10 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0aa7fc700 1 -- 172.21.15.32:0/828180231 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0b40113a0 con 0x7fd0b8066b10 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b810f020 con 0x7fd0b8066b10 2024-04-06T14:10:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.045+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b810f560 con 0x7fd0b8066b10 2024-04-06T14:10:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.046+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b805f5f0 con 0x7fd0b8066b10 2024-04-06T14:10:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.046+0000 7fd0aa7fc700 1 -- 172.21.15.32:0/828180231 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fd0b4012400 con 0x7fd0b8066b10 2024-04-06T14:10:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.047+0000 7fd0aa7fc700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd0a4069210 0x7fd0a406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:10:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.047+0000 7fd0aa7fc700 1 -- 172.21.15.32:0/828180231 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fd0b4086240 con 0x7fd0b8066b10 2024-04-06T14:10:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.047+0000 7fd0bd48e700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd0a4069210 0x7fd0a406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:10:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.047+0000 7fd0bd48e700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd0a4069210 0x7fd0a406b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac004f90 tx=0x7fd0ac009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:10:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.051+0000 7fd0aa7fc700 1 -- 172.21.15.32:0/828180231 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd0b4056f20 con 0x7fd0b8066b10 2024-04-06T14:10:23.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:22 smithi032 ceph-mon[39730]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:23.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:23.258+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0b810f9a0 con 0x7fd0a4069210 2024-04-06T14:10:23.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:22 smithi086 ceph-mon[47925]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:24.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:10:24.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:10:24.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:24.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:10:24.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:10:24.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:24 smithi032 ceph-mon[39730]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:10:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:24 smithi032 ceph-mon[39730]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:10:25.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:25.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:24 smithi086 ceph-mon[47925]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:10:25.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:24 smithi086 ceph-mon[47925]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:25.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:25.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:25.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:25.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:10:25.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:26 smithi032 ceph-mon[39730]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:26 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3229825635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9185a389-ef69-41bc-9e10-46c6a2e02a6e"}]: dispatch 2024-04-06T14:10:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:26 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3229825635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9185a389-ef69-41bc-9e10-46c6a2e02a6e"}]': finished 2024-04-06T14:10:27.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:26 smithi032 ceph-mon[39730]: osdmap e11: 2 total, 1 up, 2 in 2024-04-06T14:10:27.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:27.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:26 smithi086 ceph-mon[47925]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:27.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:26 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3229825635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9185a389-ef69-41bc-9e10-46c6a2e02a6e"}]: dispatch 2024-04-06T14:10:27.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:26 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3229825635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9185a389-ef69-41bc-9e10-46c6a2e02a6e"}]': finished 2024-04-06T14:10:27.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:26 smithi086 ceph-mon[47925]: osdmap e11: 2 total, 1 up, 2 in 2024-04-06T14:10:27.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:27 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1567599789' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:10:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:27 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:10:28.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:27 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1567599789' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:10:28.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:27 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:10:29.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:28 smithi032 ceph-mon[39730]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:29.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:28 smithi086 ceph-mon[47925]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:30 smithi032 ceph-mon[39730]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:31.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:30 smithi086 ceph-mon[47925]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:32 smithi032 ceph-mon[39730]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:33.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:32 smithi086 ceph-mon[47925]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:34.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:34 smithi032 ceph-mon[39730]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:34.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:34 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T14:10:34.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:34 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:35.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:34 smithi086 ceph-mon[47925]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:35.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:34 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T14:10:35.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:34 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:35.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:35 smithi032 ceph-mon[39730]: Deploying daemon osd.1 on smithi032 2024-04-06T14:10:36.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:35 smithi086 ceph-mon[47925]: Deploying daemon osd.1 on smithi032 2024-04-06T14:10:37.074 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:36 smithi032 ceph-mon[39730]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:37.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:36 smithi086 ceph-mon[47925]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:38.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:38 smithi032 ceph-mon[39730]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:38 smithi086 ceph-mon[47925]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:40 smithi032 ceph-mon[39730]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:10:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:40 smithi086 ceph-mon[47925]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:10:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:42 smithi032 ceph-mon[39730]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:42 smithi086 ceph-mon[47925]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:43.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.466+0000 7fd0aa7fc700 1 -- 172.21.15.32:0/828180231 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd0b810f9a0 con 0x7fd0a4069210 2024-04-06T14:10:43.467 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 1 on host 'smithi032' 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.468+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd0a4069210 msgr2=0x7fd0a406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.468+0000 7fd0bfef3700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd0a4069210 0x7fd0a406b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac004f90 tx=0x7fd0ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.468+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b8066b10 msgr2=0x7fd0b810d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.468+0000 7fd0bfef3700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b8066b10 0x7fd0b810d9b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400c800 tx=0x7fd0b4004470 comp rx=0 tx=0).stop 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.468+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 shutdown_connections 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.468+0000 7fd0bfef3700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd0a4069210 0x7fd0a406b6c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.468+0000 7fd0bfef3700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0b8066b10 0x7fd0b810d9b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.468+0000 7fd0bfef3700 1 --2- 172.21.15.32:0/828180231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0b80fff70 0x7fd0b810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.468+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 >> 172.21.15.32:0/828180231 conn(0x7fd0b80f7440 msgr2=0x7fd0b80f7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.469+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 shutdown_connections 2024-04-06T14:10:43.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:43.469+0000 7fd0bfef3700 1 -- 172.21.15.32:0/828180231 wait complete. 2024-04-06T14:10:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:10:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:43 smithi032 ceph-mon[39730]: from='osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T14:10:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:10:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:44.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:44.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:43 smithi086 ceph-mon[47925]: from='osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T14:10:44.523 DEBUG:teuthology.orchestra.run.smithi032:osd.1> sudo journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-06T14:10:44.526 INFO:tasks.cephadm:Deploying osd.2 on smithi032 with /dev/vg_nvme/lv_2... 2024-04-06T14:10:44.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T14:10:44.561 INFO:journalctl@ceph.osd.1.smithi032.stdout:-- Logs begin at Sat 2024-04-06 13:55:15 UTC. -- 2024-04-06T14:10:44.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:44 smithi032 ceph-mon[39730]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:44.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:44 smithi032 ceph-mon[39730]: from='osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T14:10:44.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:44 smithi032 ceph-mon[39730]: osdmap e12: 2 total, 1 up, 2 in 2024-04-06T14:10:44.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:44 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:44.881 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:44 smithi032 ceph-mon[39730]: from='osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:10:45.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:44 smithi086 ceph-mon[47925]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:45.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:44 smithi086 ceph-mon[47925]: from='osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T14:10:45.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:44 smithi086 ceph-mon[47925]: osdmap e12: 2 total, 1 up, 2 in 2024-04-06T14:10:45.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:44 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:45.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:44 smithi086 ceph-mon[47925]: from='osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:10:45.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:10:45 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[72327]: 2024-04-06T14:10:45.448+0000 7f0b79910700 -1 osd.1 0 waiting for initial osdmap 2024-04-06T14:10:45.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:10:45 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[72327]: 2024-04-06T14:10:45.452+0000 7f0b73709700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:10:45.813 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:10:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:46 smithi032 ceph-mon[39730]: from='osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-06T14:10:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:46 smithi032 ceph-mon[39730]: osdmap e13: 2 total, 1 up, 2 in 2024-04-06T14:10:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:46 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:46 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:46 smithi032 ceph-mon[39730]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:46.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:46 smithi086 ceph-mon[47925]: from='osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-06T14:10:46.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:46 smithi086 ceph-mon[47925]: osdmap e13: 2 total, 1 up, 2 in 2024-04-06T14:10:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:46 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:46 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:46 smithi086 ceph-mon[47925]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-06T14:10:47.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:47 smithi032 ceph-mon[39730]: purged_snaps scrub starts 2024-04-06T14:10:47.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:47 smithi032 ceph-mon[39730]: purged_snaps scrub ok 2024-04-06T14:10:47.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:47 smithi032 ceph-mon[39730]: osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] boot 2024-04-06T14:10:47.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:47 smithi032 ceph-mon[39730]: osdmap e14: 2 total, 2 up, 2 in 2024-04-06T14:10:47.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:47 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:47.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:47 smithi086 ceph-mon[47925]: purged_snaps scrub starts 2024-04-06T14:10:47.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:47 smithi086 ceph-mon[47925]: purged_snaps scrub ok 2024-04-06T14:10:47.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:47 smithi086 ceph-mon[47925]: osd.1 [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] boot 2024-04-06T14:10:47.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:47 smithi086 ceph-mon[47925]: osdmap e14: 2 total, 2 up, 2 in 2024-04-06T14:10:47.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:47 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:10:48.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:48 smithi032 ceph-mon[39730]: osdmap e15: 2 total, 2 up, 2 in 2024-04-06T14:10:48.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:48 smithi032 ceph-mon[39730]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:48.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:48 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:48.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:48 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:48.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:48 smithi086 ceph-mon[47925]: osdmap e15: 2 total, 2 up, 2 in 2024-04-06T14:10:48.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:48 smithi086 ceph-mon[47925]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:48.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:48 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:48.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:48 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:49.009 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T14:10:49.009 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T14:10:49.009 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-06T14:10:49.009 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-06T14:10:49.009 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00820459 s, 1.3 GB/s 2024-04-06T14:10:49.009 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 2024-04-06T14:10:49.009 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-06T14:10:50.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi032:vg_nvme/lv_2 2024-04-06T14:10:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:10:50.939 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:50 smithi032 ceph-mon[39730]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:51.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:50 smithi086 ceph-mon[47925]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:52.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:52 smithi032 ceph-mon[39730]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:53.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:52 smithi086 ceph-mon[47925]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.571+0000 7f67b26a8700 1 -- 172.21.15.32:0/3516787675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 msgr2=0x7f67ac0fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.571+0000 7f67b26a8700 1 --2- 172.21.15.32:0/3516787675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 0x7f67ac0fe4b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6794004940 tx=0x7f6794009e30 comp rx=0 tx=0).stop 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.571+0000 7f67b26a8700 1 -- 172.21.15.32:0/3516787675 shutdown_connections 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.571+0000 7f67b26a8700 1 --2- 172.21.15.32:0/3516787675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 0x7f67ac0fe4b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.571+0000 7f67b26a8700 1 --2- 172.21.15.32:0/3516787675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67ac0fce40 0x7f67ac0fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.571+0000 7f67b26a8700 1 -- 172.21.15.32:0/3516787675 >> 172.21.15.32:0/3516787675 conn(0x7f67ac0f8550 msgr2=0x7f67ac0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.571+0000 7f67b26a8700 1 -- 172.21.15.32:0/3516787675 shutdown_connections 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.571+0000 7f67b26a8700 1 -- 172.21.15.32:0/3516787675 wait complete. 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.571+0000 7f67b26a8700 1 Processor -- start 2024-04-06T14:10:53.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67b26a8700 1 -- start start 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67b26a8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67ac0fce40 0x7f67ac071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67b26a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 0x7f67ac070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67b26a8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac0708c0 con 0x7f67ac0fe040 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67b26a8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac070a30 con 0x7f67ac0fce40 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67ab7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 0x7f67ac070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67ab7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 0x7f67ac070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44148/0 (socket says 172.21.15.32:44148) 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67ab7fe700 1 -- 172.21.15.32:0/4280108511 learned_addr learned my addr 172.21.15.32:0/4280108511 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67abfff700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67ac0fce40 0x7f67ac071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67ab7fe700 1 -- 172.21.15.32:0/4280108511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67ac0fce40 msgr2=0x7f67ac071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67ab7fe700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67ac0fce40 0x7f67ac071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67ab7fe700 1 -- 172.21.15.32:0/4280108511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6794005040 con 0x7f67ac0fe040 2024-04-06T14:10:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67ab7fe700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 0x7f67ac070310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f679400b3b0 tx=0x7f6794009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:10:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67a8ff9700 1 -- 172.21.15.32:0/4280108511 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f679400e040 con 0x7f67ac0fe040 2024-04-06T14:10:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.572+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ac070cb0 con 0x7f67ac0fe040 2024-04-06T14:10:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.573+0000 7f67a8ff9700 1 -- 172.21.15.32:0/4280108511 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f679400bd20 con 0x7f67ac0fe040 2024-04-06T14:10:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.573+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ac115990 con 0x7f67ac0fe040 2024-04-06T14:10:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.573+0000 7f67a8ff9700 1 -- 172.21.15.32:0/4280108511 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f679401c630 con 0x7f67ac0fe040 2024-04-06T14:10:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.573+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ac05f5f0 con 0x7f67ac0fe040 2024-04-06T14:10:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.574+0000 7f67a8ff9700 1 -- 172.21.15.32:0/4280108511 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f6794013420 con 0x7f67ac0fe040 2024-04-06T14:10:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.574+0000 7f67a8ff9700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f6798069210 0x7f679806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:10:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.574+0000 7f67a8ff9700 1 -- 172.21.15.32:0/4280108511 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f6794087250 con 0x7f67ac0fe040 2024-04-06T14:10:53.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.574+0000 7f67abfff700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f6798069210 0x7f679806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:10:53.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.575+0000 7f67abfff700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f6798069210 0x7f679806b6c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f67ac071bd0 tx=0x7f679c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:10:53.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.579+0000 7f67a8ff9700 1 -- 172.21.15.32:0/4280108511 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f67940541d0 con 0x7f67ac0fe040 2024-04-06T14:10:53.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:10:53.798+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f67ac102460 con 0x7f6798069210 2024-04-06T14:10:54.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: Detected new or changed devices on smithi032 2024-04-06T14:10:54.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:54.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:54.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:10:54.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:10:54.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: Adjusting osd_memory_target on smithi032 to 6451M 2024-04-06T14:10:54.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:54.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:54.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:10:54.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:54.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:10:54.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:54.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:10:54.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:10:54.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:10:54.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:54 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:54.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: Detected new or changed devices on smithi032 2024-04-06T14:10:54.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:54.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: Adjusting osd_memory_target on smithi032 to 6451M 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:10:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:10:54.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:54 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:10:56.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:56 smithi032 ceph-mon[39730]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:57.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:56 smithi086 ceph-mon[47925]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:57.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:57 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3560109610' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af4793e1-b5bb-4919-838e-67703c83296a"}]: dispatch 2024-04-06T14:10:57.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:57 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3560109610' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af4793e1-b5bb-4919-838e-67703c83296a"}]': finished 2024-04-06T14:10:57.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:57 smithi032 ceph-mon[39730]: osdmap e16: 3 total, 2 up, 3 in 2024-04-06T14:10:57.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:10:57.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:10:58.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:57 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3560109610' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af4793e1-b5bb-4919-838e-67703c83296a"}]: dispatch 2024-04-06T14:10:58.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:57 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3560109610' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af4793e1-b5bb-4919-838e-67703c83296a"}]': finished 2024-04-06T14:10:58.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:57 smithi086 ceph-mon[47925]: osdmap e16: 3 total, 2 up, 3 in 2024-04-06T14:10:58.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:57 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:10:58.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:57 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:10:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:58 smithi032 ceph-mon[39730]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:10:58 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1449913538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:10:59.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:58 smithi086 ceph-mon[47925]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:10:59.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:10:58 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1449913538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:11:00.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:00 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:00.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:00 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:00.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:00 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:00.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:00 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:00.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:00 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:00.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:00 smithi086 ceph-mon[47925]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:00 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:00 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:00 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:00 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:00 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:00 smithi032 ceph-mon[39730]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:02.929 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:02 smithi032 ceph-mon[39730]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:03.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:02 smithi086 ceph-mon[47925]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:04.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:04 smithi032 ceph-mon[39730]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:05.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:04 smithi086 ceph-mon[47925]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:05.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T14:11:05.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:05 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:06.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T14:11:06.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:05 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:06.978 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:06 smithi032 ceph-mon[39730]: Deploying daemon osd.2 on smithi032 2024-04-06T14:11:06.978 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:06 smithi032 ceph-mon[39730]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:07.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:06 smithi086 ceph-mon[47925]: Deploying daemon osd.2 on smithi032 2024-04-06T14:11:07.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:06 smithi086 ceph-mon[47925]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:08.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:08 smithi032 ceph-mon[39730]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:09.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:08 smithi086 ceph-mon[47925]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:10.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:10 smithi032 ceph-mon[39730]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:11.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:10 smithi086 ceph-mon[47925]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:11:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:12 smithi086 ceph-mon[47925]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:12.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:12.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:12.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:11:12.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:12 smithi032 ceph-mon[39730]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:11:13.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:13 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:11:14.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:14 smithi086 ceph-mon[47925]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:14 smithi032 ceph-mon[39730]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:14.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.966+0000 7f67a8ff9700 1 -- 172.21.15.32:0/4280108511 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f67ac102460 con 0x7f6798069210 2024-04-06T14:11:14.965 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 2 on host 'smithi032' 2024-04-06T14:11:14.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f6798069210 msgr2=0x7f679806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:11:14.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f6798069210 0x7f679806b6c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f67ac071bd0 tx=0x7f679c00b040 comp rx=0 tx=0).stop 2024-04-06T14:11:14.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 msgr2=0x7f67ac070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:11:14.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 0x7f67ac070310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f679400b3b0 tx=0x7f6794009e30 comp rx=0 tx=0).stop 2024-04-06T14:11:14.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 shutdown_connections 2024-04-06T14:11:14.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f6798069210 0x7f679806b6c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:14.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67ac0fce40 0x7f67ac071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:14.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 --2- 172.21.15.32:0/4280108511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67ac0fe040 0x7f67ac070310 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:14.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 >> 172.21.15.32:0/4280108511 conn(0x7f67ac0f8550 msgr2=0x7f67ac0f9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:11:14.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 shutdown_connections 2024-04-06T14:11:14.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:11:14.968+0000 7f67b26a8700 1 -- 172.21.15.32:0/4280108511 wait complete. 2024-04-06T14:11:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:15 smithi032 ceph-mon[39730]: from='osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T14:11:16.260 DEBUG:teuthology.orchestra.run.smithi032:osd.2> sudo journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-06T14:11:16.262 INFO:tasks.cephadm:Deploying osd.3 on smithi086 with /dev/vg_nvme/lv_4... 2024-04-06T14:11:16.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T14:11:16.295 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:16.295 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:16.296 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:15 smithi086 ceph-mon[47925]: from='osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T14:11:16.296 INFO:journalctl@ceph.osd.2.smithi032.stdout:-- Logs begin at Sat 2024-04-06 13:55:15 UTC. -- 2024-04-06T14:11:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:11:17.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:16 smithi086 ceph-mon[47925]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:17.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:16 smithi086 ceph-mon[47925]: from='osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T14:11:17.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:16 smithi086 ceph-mon[47925]: osdmap e17: 3 total, 2 up, 3 in 2024-04-06T14:11:17.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:16 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:11:17.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:16 smithi086 ceph-mon[47925]: from='osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:11:17.230 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:16 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[82087]: 2024-04-06T14:11:16.924+0000 7fbd2b804700 -1 osd.2 0 waiting for initial osdmap 2024-04-06T14:11:17.231 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:16 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[82087]: 2024-04-06T14:11:16.929+0000 7fbd26e00700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:11:17.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:16 smithi032 ceph-mon[39730]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:17.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:16 smithi032 ceph-mon[39730]: from='osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T14:11:17.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:16 smithi032 ceph-mon[39730]: osdmap e17: 3 total, 2 up, 3 in 2024-04-06T14:11:17.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:16 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:11:17.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:16 smithi032 ceph-mon[39730]: from='osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:11:18.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:17 smithi086 ceph-mon[47925]: purged_snaps scrub starts 2024-04-06T14:11:18.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:17 smithi086 ceph-mon[47925]: purged_snaps scrub ok 2024-04-06T14:11:18.041 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:17 smithi086 ceph-mon[47925]: from='osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-06T14:11:18.041 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:17 smithi086 ceph-mon[47925]: osdmap e18: 3 total, 2 up, 3 in 2024-04-06T14:11:18.041 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:11:18.041 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:11:18.041 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:17 smithi086 ceph-mon[47925]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:17 smithi032 ceph-mon[39730]: purged_snaps scrub starts 2024-04-06T14:11:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:17 smithi032 ceph-mon[39730]: purged_snaps scrub ok 2024-04-06T14:11:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:17 smithi032 ceph-mon[39730]: from='osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-06T14:11:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:17 smithi032 ceph-mon[39730]: osdmap e18: 3 total, 2 up, 3 in 2024-04-06T14:11:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:11:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:11:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:17 smithi032 ceph-mon[39730]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-06T14:11:18.570 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T14:11:18.571 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T14:11:18.571 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-04-06T14:11:18.571 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-04-06T14:11:18.571 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110746 s, 947 MB/s 2024-04-06T14:11:18.571 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-04-06T14:11:19.196 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:18 smithi086 ceph-mon[47925]: osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] boot 2024-04-06T14:11:19.196 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:18 smithi086 ceph-mon[47925]: osdmap e19: 3 total, 3 up, 3 in 2024-04-06T14:11:19.196 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:18 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:11:19.196 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:18 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:19.196 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:18 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:19.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-04-06T14:11:19.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:18 smithi032 ceph-mon[39730]: osd.2 [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] boot 2024-04-06T14:11:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:18 smithi032 ceph-mon[39730]: osdmap e19: 3 total, 3 up, 3 in 2024-04-06T14:11:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:18 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:11:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:18 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:18 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:19.517 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:11:20.685 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:20 smithi086 ceph-mon[47925]: osdmap e20: 3 total, 3 up, 3 in 2024-04-06T14:11:20.686 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:20 smithi086 ceph-mon[47925]: pgmap v79: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:20.686 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:20 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-06T14:11:20.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:20 smithi032 ceph-mon[39730]: osdmap e20: 3 total, 3 up, 3 in 2024-04-06T14:11:20.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:20 smithi032 ceph-mon[39730]: pgmap v79: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:20.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:20 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-06T14:11:21.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.227+0000 7f8eae260700 1 -- 172.21.15.86:0/2029818213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 msgr2=0x7f8ea80fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:11:21.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.227+0000 7f8eae260700 1 --2- 172.21.15.86:0/2029818213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 0x7f8ea80fd250 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c004940 tx=0x7f8e9c009e30 comp rx=0 tx=0).stop 2024-04-06T14:11:21.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.227+0000 7f8eae260700 1 -- 172.21.15.86:0/2029818213 shutdown_connections 2024-04-06T14:11:21.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.227+0000 7f8eae260700 1 --2- 172.21.15.86:0/2029818213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea80fe040 0x7f8ea80fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:21.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.227+0000 7f8eae260700 1 --2- 172.21.15.86:0/2029818213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 0x7f8ea80fd250 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:21.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.227+0000 7f8eae260700 1 -- 172.21.15.86:0/2029818213 >> 172.21.15.86:0/2029818213 conn(0x7f8ea80f8510 msgr2=0x7f8ea80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:11:21.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.227+0000 7f8eae260700 1 -- 172.21.15.86:0/2029818213 shutdown_connections 2024-04-06T14:11:21.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.227+0000 7f8eae260700 1 -- 172.21.15.86:0/2029818213 wait complete. 2024-04-06T14:11:21.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8eae260700 1 Processor -- start 2024-04-06T14:11:21.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8eae260700 1 -- start start 2024-04-06T14:11:21.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8eae260700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 0x7f8ea810d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8eae260700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea80fe040 0x7f8ea810de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8eae260700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea810e480 con 0x7f8ea80fe040 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8eae260700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea810ecd0 con 0x7f8ea80fce40 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8ea77fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 0x7f8ea810d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8ea77fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 0x7f8ea810d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60204/0 (socket says 172.21.15.86:60204) 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8ea77fe700 1 -- 172.21.15.86:0/1163881019 learned_addr learned my addr 172.21.15.86:0/1163881019 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.228+0000 7f8ea6ffd700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea80fe040 0x7f8ea810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.229+0000 7f8ea77fe700 1 -- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea80fe040 msgr2=0x7f8ea810de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.229+0000 7f8ea77fe700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea80fe040 0x7f8ea810de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.229+0000 7f8ea77fe700 1 -- 172.21.15.86:0/1163881019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c005040 con 0x7f8ea80fce40 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.229+0000 7f8ea77fe700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 0x7f8ea810d940 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c000c00 tx=0x7f8e9c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.229+0000 7f8e8ffff700 1 -- 172.21.15.86:0/1163881019 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9c00e040 con 0x7f8ea80fce40 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.229+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea810ef50 con 0x7f8ea80fce40 2024-04-06T14:11:21.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.229+0000 7f8e8ffff700 1 -- 172.21.15.86:0/1163881019 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e9c00bc50 con 0x7f8ea80fce40 2024-04-06T14:11:21.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.229+0000 7f8e8ffff700 1 -- 172.21.15.86:0/1163881019 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9c01c630 con 0x7f8ea80fce40 2024-04-06T14:11:21.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.229+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea810f490 con 0x7f8ea80fce40 2024-04-06T14:11:21.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.230+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea805f5f0 con 0x7f8ea80fce40 2024-04-06T14:11:21.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.230+0000 7f8e8ffff700 1 -- 172.21.15.86:0/1163881019 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f8e9c01c790 con 0x7f8ea80fce40 2024-04-06T14:11:21.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.231+0000 7f8e8ffff700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f8e90069210 0x7f8e9006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:11:21.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.231+0000 7f8e8ffff700 1 -- 172.21.15.86:0/1163881019 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3217+0+0 (secure 0 0 0) 0x7f8e9c087b10 con 0x7f8ea80fce40 2024-04-06T14:11:21.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.231+0000 7f8ea6ffd700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f8e90069210 0x7f8e9006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:11:21.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.231+0000 7f8ea6ffd700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f8e90069210 0x7f8e9006b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e98004f40 tx=0x7f8e9800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:11:21.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.235+0000 7f8e8ffff700 1 -- 172.21.15.86:0/1163881019 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8e9c0582e0 con 0x7f8ea80fce40 2024-04-06T14:11:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:21.428+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ea810f810 con 0x7f8e90069210 2024-04-06T14:11:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:21 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-06T14:11:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:21 smithi032 ceph-mon[39730]: osdmap e21: 3 total, 3 up, 3 in 2024-04-06T14:11:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:21 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T14:11:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:21 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:11:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:21 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:11:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:21 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:21.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:21 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-06T14:11:21.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:21 smithi086 ceph-mon[47925]: osdmap e21: 3 total, 3 up, 3 in 2024-04-06T14:11:21.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:21 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T14:11:21.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:21 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:11:21.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:21 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:11:21.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:21 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:22.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:22.463+0000 7f8e8ffff700 1 -- 172.21.15.86:0/1163881019 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8e9c054c20 con 0x7f8ea80fce40 2024-04-06T14:11:22.480 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:11:22 smithi032 sudo[88190]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T14:11:22.480 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:11:22 smithi032 sudo[88190]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:22.480 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:11:22 smithi032 sudo[88190]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:22.480 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:11:22 smithi032 sudo[88190]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:22.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:22 smithi032 ceph-mon[39730]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:11:22.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:22 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T14:11:22.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:22 smithi032 ceph-mon[39730]: osdmap e22: 3 total, 3 up, 3 in 2024-04-06T14:11:22.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:22 smithi032 ceph-mon[39730]: pgmap v82: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:22.589 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:22 smithi086 ceph-mon[47925]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:11:22.589 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:22 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T14:11:22.589 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:22 smithi086 ceph-mon[47925]: osdmap e22: 3 total, 3 up, 3 in 2024-04-06T14:11:22.589 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:22 smithi086 ceph-mon[47925]: pgmap v82: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:23.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:11:22 smithi032 sudo[88290]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T14:11:23.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:11:22 smithi032 sudo[88290]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:23.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:11:22 smithi032 sudo[88290]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:23.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:11:22 smithi032 sudo[88290]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: osdmap e23: 3 total, 3 up, 3 in 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: mgrmap e19: smithi032.peohxu(active, since 2m), standbys: smithi086.fyyhxm 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: Detected new or changed devices on smithi032 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: Adjusting osd_memory_target on smithi032 to 4300M 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:23.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:23.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:11:23.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:23 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:23.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:11:23 smithi032 sudo[88392]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T14:11:23.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:11:23 smithi032 sudo[88392]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:23.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:11:23 smithi032 sudo[88392]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:23.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:11:23 smithi032 sudo[88392]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: osdmap e23: 3 total, 3 up, 3 in 2024-04-06T14:11:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: mgrmap e19: smithi032.peohxu(active, since 2m), standbys: smithi086.fyyhxm 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: Detected new or changed devices on smithi032 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: Adjusting osd_memory_target on smithi032 to 4300M 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:11:23.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:23 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:24.228 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:11:23 smithi032 sudo[88427]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T14:11:24.228 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:11:23 smithi032 sudo[88427]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:24.228 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:11:23 smithi032 sudo[88427]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:24.228 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:11:23 smithi032 sudo[88427]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:24.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:24 smithi032 sudo[88513]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T14:11:24.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:24 smithi032 sudo[88513]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:24.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:24 smithi032 sudo[88513]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:24.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:24 smithi032 sudo[88513]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:24.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:24 smithi032 ceph-mon[39730]: pgmap v84: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:24.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:24 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/2927273883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70c8b884-5214-44f2-9a31-55556a11c20a"}]: dispatch 2024-04-06T14:11:24.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:24 smithi032 ceph-mon[39730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70c8b884-5214-44f2-9a31-55556a11c20a"}]: dispatch 2024-04-06T14:11:24.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:24 smithi032 ceph-mon[39730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70c8b884-5214-44f2-9a31-55556a11c20a"}]': finished 2024-04-06T14:11:24.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:24 smithi032 ceph-mon[39730]: osdmap e24: 4 total, 3 up, 4 in 2024-04-06T14:11:24.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:24.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:24 smithi086 ceph-mon[47925]: pgmap v84: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:24 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/2927273883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70c8b884-5214-44f2-9a31-55556a11c20a"}]: dispatch 2024-04-06T14:11:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:24 smithi086 ceph-mon[47925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70c8b884-5214-44f2-9a31-55556a11c20a"}]: dispatch 2024-04-06T14:11:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:24 smithi086 ceph-mon[47925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70c8b884-5214-44f2-9a31-55556a11c20a"}]': finished 2024-04-06T14:11:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:24 smithi086 ceph-mon[47925]: osdmap e24: 4 total, 3 up, 4 in 2024-04-06T14:11:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:24.980 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:24 smithi032 sudo[88550]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T14:11:24.980 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:24 smithi032 sudo[88550]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:24.980 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:24 smithi032 sudo[88550]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:24.980 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:11:24 smithi032 sudo[88550]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:25.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 sudo[88599]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T14:11:25.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 sudo[88599]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:25.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 sudo[88599]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:25.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 sudo[88599]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:25.848 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:25 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/2342726162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:11:25.848 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:25 smithi086 ceph-mon[47925]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T14:11:25.848 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:25 smithi086 sudo[52546]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T14:11:25.849 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:25 smithi086 sudo[52546]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:25.849 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:25 smithi086 sudo[52546]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:25.849 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:25 smithi086 sudo[52546]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 sudo[88643]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T14:11:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 sudo[88643]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 sudo[88643]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 sudo[88643]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/2342726162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:11:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:25 smithi032 ceph-mon[39730]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T14:11:26.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 sudo[52551]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T14:11:26.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 sudo[52551]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T14:11:26.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 sudo[52551]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T14:11:26.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 sudo[52551]: pam_unix(sudo:session): session closed for user root 2024-04-06T14:11:26.870 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:26 smithi032 ceph-mon[39730]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T14:11:26.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:11:26.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:11:26.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:26 smithi032 ceph-mon[39730]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T14:11:26.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:11:26.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:11:26.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:26 smithi032 ceph-mon[39730]: pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:26.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:26 smithi032 ceph-mon[39730]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T14:11:26.871 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 ceph-mon[47925]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T14:11:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:11:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:11:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 ceph-mon[47925]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T14:11:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:11:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:11:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 ceph-mon[47925]: pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:26.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 ceph-mon[47925]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T14:11:26.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:28.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:28.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:28.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:28.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:28.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:28.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:28 smithi086 ceph-mon[47925]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:28.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:11:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:28 smithi032 ceph-mon[39730]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:28.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:11:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:30 smithi032 ceph-mon[39730]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:31.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:30 smithi086 ceph-mon[47925]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:31.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:31 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T14:11:31.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:31 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:31 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T14:11:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:31 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:32 smithi032 ceph-mon[39730]: Deploying daemon osd.3 on smithi086 2024-04-06T14:11:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:32 smithi032 ceph-mon[39730]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:33.003 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:32 smithi086 ceph-mon[47925]: Deploying daemon osd.3 on smithi086 2024-04-06T14:11:33.003 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:32 smithi086 ceph-mon[47925]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:34.970 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:34 smithi086 ceph-mon[47925]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:34.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:34 smithi032 ceph-mon[39730]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:36.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:36 smithi086 ceph-mon[47925]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:36.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:36.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:36 smithi032 ceph-mon[39730]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:37.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:37 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:11:38.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:37 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:11:38.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:38 smithi086 ceph-mon[47925]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:38.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:38 smithi032 ceph-mon[39730]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:40.447 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 3 on host 'smithi086' 2024-04-06T14:11:40.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.446+0000 7f8e8ffff700 1 -- 172.21.15.86:0/1163881019 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8ea810f810 con 0x7f8e90069210 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f8e90069210 msgr2=0x7f8e9006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f8e90069210 0x7f8e9006b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e98004f40 tx=0x7f8e9800b040 comp rx=0 tx=0).stop 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 msgr2=0x7f8ea810d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 0x7f8ea810d940 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c000c00 tx=0x7f8e9c009e30 comp rx=0 tx=0).stop 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 shutdown_connections 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f8e90069210 0x7f8e9006b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ea80fce40 0x7f8ea810d940 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 --2- 172.21.15.86:0/1163881019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea80fe040 0x7f8ea810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 >> 172.21.15.86:0/1163881019 conn(0x7f8ea80f8510 msgr2=0x7f8ea80f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 shutdown_connections 2024-04-06T14:11:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:40.449+0000 7f8eae260700 1 -- 172.21.15.86:0/1163881019 wait complete. 2024-04-06T14:11:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:40 smithi032 ceph-mon[39730]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:40 smithi032 ceph-mon[39730]: from='osd.3 [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T14:11:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:40 smithi032 ceph-mon[39730]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T14:11:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:41.072 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:40 smithi086 ceph-mon[47925]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:41.072 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:41.072 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:41.072 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:40 smithi086 ceph-mon[47925]: from='osd.3 [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T14:11:41.072 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:40 smithi086 ceph-mon[47925]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T14:11:41.072 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:41.072 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:41.093 DEBUG:teuthology.orchestra.run.smithi086:osd.3> sudo journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-06T14:11:41.095 INFO:tasks.cephadm:Deploying osd.4 on smithi086 with /dev/vg_nvme/lv_3... 2024-04-06T14:11:41.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T14:11:41.125 INFO:journalctl@ceph.osd.3.smithi086.stdout:-- Logs begin at Sat 2024-04-06 13:55:25 UTC. -- 2024-04-06T14:11:42.423 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:11:42.704 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:11:42 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[54402]: 2024-04-06T14:11:42.428+0000 7f4696e86700 -1 osd.3 0 waiting for initial osdmap 2024-04-06T14:11:42.704 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:11:42 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[54402]: 2024-04-06T14:11:42.438+0000 7f468fc7d700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:11:42.704 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:42 smithi086 ceph-mon[47925]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T14:11:42.704 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:42 smithi086 ceph-mon[47925]: osdmap e25: 4 total, 3 up, 4 in 2024-04-06T14:11:42.704 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:42.704 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:42 smithi086 ceph-mon[47925]: from='osd.3 [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:11:42.704 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:42 smithi086 ceph-mon[47925]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:11:42.705 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:42 smithi086 ceph-mon[47925]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:42 smithi032 ceph-mon[39730]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T14:11:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:42 smithi032 ceph-mon[39730]: osdmap e25: 4 total, 3 up, 4 in 2024-04-06T14:11:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:42 smithi032 ceph-mon[39730]: from='osd.3 [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:11:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:42 smithi032 ceph-mon[39730]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:11:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:42 smithi032 ceph-mon[39730]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-06T14:11:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:43 smithi032 ceph-mon[39730]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-04-06T14:11:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:43 smithi032 ceph-mon[39730]: osdmap e26: 4 total, 3 up, 4 in 2024-04-06T14:11:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:11:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:43 smithi032 ceph-mon[39730]: osd.3 [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] boot 2024-04-06T14:11:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:43 smithi032 ceph-mon[39730]: osdmap e27: 4 total, 4 up, 4 in 2024-04-06T14:11:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:43 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:43 smithi086 ceph-mon[47925]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-04-06T14:11:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:43 smithi086 ceph-mon[47925]: osdmap e26: 4 total, 3 up, 4 in 2024-04-06T14:11:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:11:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:43 smithi086 ceph-mon[47925]: osd.3 [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] boot 2024-04-06T14:11:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:43 smithi086 ceph-mon[47925]: osdmap e27: 4 total, 4 up, 4 in 2024-04-06T14:11:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:43 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:11:44.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:44 smithi086 ceph-mon[47925]: purged_snaps scrub starts 2024-04-06T14:11:44.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:44 smithi086 ceph-mon[47925]: purged_snaps scrub ok 2024-04-06T14:11:44.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:44 smithi086 ceph-mon[47925]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:11:44.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:44 smithi086 ceph-mon[47925]: osdmap e28: 4 total, 4 up, 4 in 2024-04-06T14:11:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:44 smithi032 ceph-mon[39730]: purged_snaps scrub starts 2024-04-06T14:11:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:44 smithi032 ceph-mon[39730]: purged_snaps scrub ok 2024-04-06T14:11:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:44 smithi032 ceph-mon[39730]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:11:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:44 smithi032 ceph-mon[39730]: osdmap e28: 4 total, 4 up, 4 in 2024-04-06T14:11:45.632 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T14:11:45.633 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T14:11:45.633 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-04-06T14:11:45.633 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-04-06T14:11:45.633 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0152199 s, 689 MB/s 2024-04-06T14:11:45.633 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-04-06T14:11:45.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: Detected new or changed devices on smithi086 2024-04-06T14:11:45.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:45.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: Adjusting osd_memory_target on smithi086 to 15970M 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: osdmap e29: 4 total, 4 up, 4 in 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:45.969 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:45 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:46.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: Detected new or changed devices on smithi086 2024-04-06T14:11:46.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:46.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:46.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: Adjusting osd_memory_target on smithi086 to 15970M 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: osdmap e29: 4 total, 4 up, 4 in 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:11:46.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:45 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:11:46.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-04-06T14:11:46.512 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:11:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:46 smithi086 ceph-mon[47925]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:11:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:46 smithi086 ceph-mon[47925]: osdmap e30: 4 total, 4 up, 4 in 2024-04-06T14:11:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:46 smithi032 ceph-mon[39730]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:11:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:46 smithi032 ceph-mon[39730]: osdmap e30: 4 total, 4 up, 4 in 2024-04-06T14:11:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.082+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2985626655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 msgr2=0x7fba080fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:11:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.082+0000 7fba0f3c8700 1 --2- 172.21.15.86:0/2985626655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 0x7fba080fe4d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8004940 tx=0x7fb9f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:11:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.082+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2985626655 shutdown_connections 2024-04-06T14:11:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.082+0000 7fba0f3c8700 1 --2- 172.21.15.86:0/2985626655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 0x7fba080fe4d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.082+0000 7fba0f3c8700 1 --2- 172.21.15.86:0/2985626655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba080fcef0 0x7fba080fd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.082+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2985626655 >> 172.21.15.86:0/2985626655 conn(0x7fba080f85b0 msgr2=0x7fba080fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2985626655 shutdown_connections 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2985626655 wait complete. 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0f3c8700 1 Processor -- start 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0f3c8700 1 -- start start 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0f3c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba080fcef0 0x7fba0810d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0f3c8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 0x7fba0810de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0f3c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0810e480 con 0x7fba080fcef0 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0f3c8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0810ecd0 con 0x7fba080fe060 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0c963700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 0x7fba0810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0c963700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 0x7fba0810de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51652/0 (socket says 172.21.15.86:51652) 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.083+0000 7fba0c963700 1 -- 172.21.15.86:0/2826339859 learned_addr learned my addr 172.21.15.86:0/2826339859 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fba0d164700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba080fcef0 0x7fba0810d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fba0c963700 1 -- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba080fcef0 msgr2=0x7fba0810d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:11:48.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fba0c963700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba080fcef0 0x7fba0810d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:11:48.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fba0c963700 1 -- 172.21.15.86:0/2826339859 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f8005040 con 0x7fba080fe060 2024-04-06T14:11:48.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fba0c963700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 0x7fba0810de80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800b3b0 tx=0x7fb9f8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:11:48.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fb9fdffb700 1 -- 172.21.15.86:0/2826339859 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f800e040 con 0x7fba080fe060 2024-04-06T14:11:48.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fb9fdffb700 1 -- 172.21.15.86:0/2826339859 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9f800bd20 con 0x7fba080fe060 2024-04-06T14:11:48.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0810ef50 con 0x7fba080fe060 2024-04-06T14:11:48.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fb9fdffb700 1 -- 172.21.15.86:0/2826339859 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f801c630 con 0x7fba080fe060 2024-04-06T14:11:48.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.084+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0810f490 con 0x7fba080fe060 2024-04-06T14:11:48.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.085+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0805f5f0 con 0x7fba080fe060 2024-04-06T14:11:48.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.085+0000 7fb9fdffb700 1 -- 172.21.15.86:0/2826339859 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb9f8013420 con 0x7fba080fe060 2024-04-06T14:11:48.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.086+0000 7fb9fdffb700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb9f40692a0 0x7fb9f406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:11:48.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.086+0000 7fb9fdffb700 1 -- 172.21.15.86:0/2826339859 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fb9f8087800 con 0x7fba080fe060 2024-04-06T14:11:48.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.086+0000 7fba0d164700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb9f40692a0 0x7fb9f406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:11:48.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.087+0000 7fba0d164700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb9f40692a0 0x7fb9f406b750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba04004d60 tx=0x7fba0400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:11:48.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.091+0000 7fb9fdffb700 1 -- 172.21.15.86:0/2826339859 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb9f8057c60 con 0x7fba080fe060 2024-04-06T14:11:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:47 smithi086 ceph-mon[47925]: osdmap e31: 4 total, 4 up, 4 in 2024-04-06T14:11:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:47 smithi086 ceph-mon[47925]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-06T14:11:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:47 smithi032 ceph-mon[39730]: osdmap e31: 4 total, 4 up, 4 in 2024-04-06T14:11:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:47 smithi032 ceph-mon[39730]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-06T14:11:48.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:11:48.281+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fba0810f7c0 con 0x7fb9f40692a0 2024-04-06T14:11:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:48 smithi032 ceph-mon[39730]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:11:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:48 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:11:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:48 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:11:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:48 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:49.360 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:48 smithi086 ceph-mon[47925]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:11:49.360 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:48 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:11:49.360 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:48 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:11:49.360 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:48 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:50.252 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:49 smithi086 ceph-mon[47925]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-06T14:11:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:49 smithi032 ceph-mon[39730]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-06T14:11:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:51 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/2335389517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f548ac-4f42-4083-a027-295b829646a3"}]: dispatch 2024-04-06T14:11:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:51 smithi032 ceph-mon[39730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f548ac-4f42-4083-a027-295b829646a3"}]: dispatch 2024-04-06T14:11:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:51 smithi032 ceph-mon[39730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f548ac-4f42-4083-a027-295b829646a3"}]': finished 2024-04-06T14:11:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:51 smithi032 ceph-mon[39730]: osdmap e32: 5 total, 4 up, 5 in 2024-04-06T14:11:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:51 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:11:51.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:51 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/2335389517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f548ac-4f42-4083-a027-295b829646a3"}]: dispatch 2024-04-06T14:11:51.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:51 smithi086 ceph-mon[47925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f548ac-4f42-4083-a027-295b829646a3"}]: dispatch 2024-04-06T14:11:51.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:51 smithi086 ceph-mon[47925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f548ac-4f42-4083-a027-295b829646a3"}]': finished 2024-04-06T14:11:51.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:51 smithi086 ceph-mon[47925]: osdmap e32: 5 total, 4 up, 5 in 2024-04-06T14:11:51.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:51 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:11:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:52 smithi032 ceph-mon[39730]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-06T14:11:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:52 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/1151891939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:11:52.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:52 smithi086 ceph-mon[47925]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-06T14:11:52.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:52 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/1151891939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:11:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:54 smithi032 ceph-mon[39730]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-06T14:11:55.019 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:54 smithi086 ceph-mon[47925]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-06T14:11:56.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:56 smithi032 ceph-mon[39730]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:11:57.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:56 smithi086 ceph-mon[47925]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:11:57.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:11:58.096 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:57 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:11:58.953 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:58 smithi086 ceph-mon[47925]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:11:58.954 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:58 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T14:11:58.954 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:58 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:58 smithi032 ceph-mon[39730]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:11:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:58 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T14:11:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:58 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:11:59.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:11:59 smithi086 ceph-mon[47925]: Deploying daemon osd.4 on smithi086 2024-04-06T14:11:59.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:11:59 smithi032 ceph-mon[39730]: Deploying daemon osd.4 on smithi086 2024-04-06T14:12:00.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:00 smithi086 ceph-mon[47925]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:00 smithi032 ceph-mon[39730]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:02 smithi032 ceph-mon[39730]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:02 smithi086 ceph-mon[47925]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:04.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:04 smithi032 ceph-mon[39730]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:04.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:04.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:04.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:04 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:12:05.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:04 smithi086 ceph-mon[47925]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:05.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:04 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:05.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:04 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:05.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:04 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:12:06.852 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:06 smithi086 ceph-mon[47925]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:06.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:06 smithi032 ceph-mon[39730]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.250+0000 7fb9fdffb700 1 -- 172.21.15.86:0/2826339859 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fba0810f7c0 con 0x7fb9f40692a0 2024-04-06T14:12:07.250 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 4 on host 'smithi086' 2024-04-06T14:12:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb9f40692a0 msgr2=0x7fb9f406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb9f40692a0 0x7fb9f406b750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba04004d60 tx=0x7fba0400b040 comp rx=0 tx=0).stop 2024-04-06T14:12:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 msgr2=0x7fba0810de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 0x7fba0810de80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800b3b0 tx=0x7fb9f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 shutdown_connections 2024-04-06T14:12:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb9f40692a0 0x7fb9f406b750 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba080fcef0 0x7fba0810d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 --2- 172.21.15.86:0/2826339859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba080fe060 0x7fba0810de80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 >> 172.21.15.86:0/2826339859 conn(0x7fba080f85b0 msgr2=0x7fba080f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 shutdown_connections 2024-04-06T14:12:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:07.252+0000 7fba0f3c8700 1 -- 172.21.15.86:0/2826339859 wait complete. 2024-04-06T14:12:08.322 DEBUG:teuthology.orchestra.run.smithi086:osd.4> sudo journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-06T14:12:08.325 INFO:tasks.cephadm:Deploying osd.5 on smithi086 with /dev/vg_nvme/lv_2... 2024-04-06T14:12:08.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T14:12:08.356 INFO:journalctl@ceph.osd.4.smithi086.stdout:-- Logs begin at Sat 2024-04-06 13:55:25 UTC. -- 2024-04-06T14:12:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:08 smithi032 ceph-mon[39730]: from='osd.4 [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T14:12:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:08 smithi032 ceph-mon[39730]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T14:12:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:08 smithi032 ceph-mon[39730]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:08 smithi086 ceph-mon[47925]: from='osd.4 [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T14:12:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:08 smithi086 ceph-mon[47925]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T14:12:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:08 smithi086 ceph-mon[47925]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:08.629 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:12:09.627 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:12:09 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[63137]: 2024-04-06T14:12:09.242+0000 7fdfd41a3700 -1 osd.4 0 waiting for initial osdmap 2024-04-06T14:12:09.627 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:12:09 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[63137]: 2024-04-06T14:12:09.249+0000 7fdfcffa0700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:12:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:09 smithi086 ceph-mon[47925]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T14:12:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:09 smithi086 ceph-mon[47925]: osdmap e33: 5 total, 4 up, 5 in 2024-04-06T14:12:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:09 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:09 smithi086 ceph-mon[47925]: from='osd.4 [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:12:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:09 smithi086 ceph-mon[47925]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:12:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:09 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:09 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:09 smithi032 ceph-mon[39730]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T14:12:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:09 smithi032 ceph-mon[39730]: osdmap e33: 5 total, 4 up, 5 in 2024-04-06T14:12:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:09 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:09 smithi032 ceph-mon[39730]: from='osd.4 [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:12:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:09 smithi032 ceph-mon[39730]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:12:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:09 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:09 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:10.303 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:10 smithi086 ceph-mon[47925]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-04-06T14:12:10.303 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:10 smithi086 ceph-mon[47925]: osdmap e34: 5 total, 4 up, 5 in 2024-04-06T14:12:10.303 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:10 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:10.303 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:10 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:10.303 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:10 smithi086 ceph-mon[47925]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:10 smithi032 ceph-mon[39730]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-04-06T14:12:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:10 smithi032 ceph-mon[39730]: osdmap e34: 5 total, 4 up, 5 in 2024-04-06T14:12:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:10 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:10 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:10 smithi032 ceph-mon[39730]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-06T14:12:11.016 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T14:12:11.017 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T14:12:11.017 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-04-06T14:12:11.017 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-04-06T14:12:11.017 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111172 s, 943 MB/s 2024-04-06T14:12:11.017 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-04-06T14:12:11.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:11 smithi086 ceph-mon[47925]: purged_snaps scrub starts 2024-04-06T14:12:11.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:11 smithi086 ceph-mon[47925]: purged_snaps scrub ok 2024-04-06T14:12:11.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:11 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:11.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:11 smithi086 ceph-mon[47925]: osd.4 [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] boot 2024-04-06T14:12:11.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:11 smithi086 ceph-mon[47925]: osdmap e35: 5 total, 5 up, 5 in 2024-04-06T14:12:11.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:11 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:11.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:11 smithi032 ceph-mon[39730]: purged_snaps scrub starts 2024-04-06T14:12:11.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:11 smithi032 ceph-mon[39730]: purged_snaps scrub ok 2024-04-06T14:12:11.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:11 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:11.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:11 smithi032 ceph-mon[39730]: osd.4 [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] boot 2024-04-06T14:12:11.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:11 smithi032 ceph-mon[39730]: osdmap e35: 5 total, 5 up, 5 in 2024-04-06T14:12:11.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:11 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:12:12.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-04-06T14:12:12.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:12 smithi086 ceph-mon[47925]: osdmap e36: 5 total, 5 up, 5 in 2024-04-06T14:12:12.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:12 smithi086 ceph-mon[47925]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:12:12.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:12 smithi032 ceph-mon[39730]: osdmap e36: 5 total, 5 up, 5 in 2024-04-06T14:12:12.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:12 smithi032 ceph-mon[39730]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:13.308 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:13 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:12:13.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:12:14.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:14.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:12:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:12:15.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.040+0000 7f727f8ca700 1 -- 172.21.15.86:0/754032571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 msgr2=0x7f72780faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:15.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.040+0000 7f727f8ca700 1 --2- 172.21.15.86:0/754032571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 0x7f72780faed0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7274004940 tx=0x7f7274009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:15.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.040+0000 7f727f8ca700 1 -- 172.21.15.86:0/754032571 shutdown_connections 2024-04-06T14:12:15.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.040+0000 7f727f8ca700 1 --2- 172.21.15.86:0/754032571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 0x7f72780faed0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.040+0000 7f727f8ca700 1 --2- 172.21.15.86:0/754032571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72780fe440 0x7f72780fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.040+0000 7f727f8ca700 1 -- 172.21.15.86:0/754032571 >> 172.21.15.86:0/754032571 conn(0x7f72780f6be0 msgr2=0x7f72780f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.040+0000 7f727f8ca700 1 -- 172.21.15.86:0/754032571 shutdown_connections 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.040+0000 7f727f8ca700 1 -- 172.21.15.86:0/754032571 wait complete. 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.040+0000 7f727f8ca700 1 Processor -- start 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727f8ca700 1 -- start start 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727f8ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72780fe440 0x7f727810d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727f8ca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 0x7f727810dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727f8ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727810e420 con 0x7f72780fe440 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727f8ca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7278112dc0 con 0x7f72780fed90 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727ce65700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 0x7f727810dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727ce65700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 0x7f727810dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49384/0 (socket says 172.21.15.86:49384) 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727ce65700 1 -- 172.21.15.86:0/2576875263 learned_addr learned my addr 172.21.15.86:0/2576875263 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727ce65700 1 -- 172.21.15.86:0/2576875263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72780fe440 msgr2=0x7f727810d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:15.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727ce65700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72780fe440 0x7f727810d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:15.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727ce65700 1 -- 172.21.15.86:0/2576875263 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7274005040 con 0x7f72780fed90 2024-04-06T14:12:15.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727ce65700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 0x7f727810dd90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7274009fd0 tx=0x7f7274009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:15.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f726dffb700 1 -- 172.21.15.86:0/2576875263 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f727400e040 con 0x7f72780fed90 2024-04-06T14:12:15.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f726dffb700 1 -- 172.21.15.86:0/2576875263 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7274004290 con 0x7f72780fed90 2024-04-06T14:12:15.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7278112fc0 con 0x7f72780fed90 2024-04-06T14:12:15.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.041+0000 7f726dffb700 1 -- 172.21.15.86:0/2576875263 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f727401c3f0 con 0x7f72780fed90 2024-04-06T14:12:15.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.042+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7278113500 con 0x7f72780fed90 2024-04-06T14:12:15.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.043+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727804c710 con 0x7f72780fed90 2024-04-06T14:12:15.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.043+0000 7f726dffb700 1 -- 172.21.15.86:0/2576875263 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f727400bc50 con 0x7f72780fed90 2024-04-06T14:12:15.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.043+0000 7f726dffb700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72640692a0 0x7f726406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:15.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.043+0000 7f726dffb700 1 -- 172.21.15.86:0/2576875263 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f72740881d0 con 0x7f72780fed90 2024-04-06T14:12:15.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.044+0000 7f727d666700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72640692a0 0x7f726406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:15.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.044+0000 7f727d666700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72640692a0 0x7f726406b750 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7268006fd0 tx=0x7f7268006db0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:15.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.048+0000 7f726dffb700 1 -- 172.21.15.86:0/2576875263 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7274058420 con 0x7f72780fed90 2024-04-06T14:12:15.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:15.240+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7278102560 con 0x7f72640692a0 2024-04-06T14:12:15.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:15 smithi086 ceph-mon[47925]: Detected new or changed devices on smithi086 2024-04-06T14:12:15.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:15 smithi086 ceph-mon[47925]: Adjusting osd_memory_target on smithi086 to 7985M 2024-04-06T14:12:15.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:12:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:12:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:12:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:15.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:15 smithi032 ceph-mon[39730]: Detected new or changed devices on smithi086 2024-04-06T14:12:15.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:15 smithi032 ceph-mon[39730]: Adjusting osd_memory_target on smithi086 to 7985M 2024-04-06T14:12:15.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:15.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:12:15.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:15.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T14:12:15.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T14:12:15.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:16.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:16 smithi086 ceph-mon[47925]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:12:16.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:16 smithi086 ceph-mon[47925]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:16.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:16 smithi032 ceph-mon[39730]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:12:16.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:16 smithi032 ceph-mon[39730]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:18.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:18 smithi032 ceph-mon[39730]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:18.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:18 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/2030082980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7037f613-ffd9-4b0e-84a1-0c04d4181ff1"}]: dispatch 2024-04-06T14:12:18.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:18 smithi032 ceph-mon[39730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7037f613-ffd9-4b0e-84a1-0c04d4181ff1"}]: dispatch 2024-04-06T14:12:18.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:18 smithi032 ceph-mon[39730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7037f613-ffd9-4b0e-84a1-0c04d4181ff1"}]': finished 2024-04-06T14:12:18.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:18 smithi032 ceph-mon[39730]: osdmap e37: 6 total, 5 up, 6 in 2024-04-06T14:12:18.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:18 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:18.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:18 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/813678052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:12:19.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:18 smithi086 ceph-mon[47925]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:19.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:18 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/2030082980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7037f613-ffd9-4b0e-84a1-0c04d4181ff1"}]: dispatch 2024-04-06T14:12:19.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:18 smithi086 ceph-mon[47925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7037f613-ffd9-4b0e-84a1-0c04d4181ff1"}]: dispatch 2024-04-06T14:12:19.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:18 smithi086 ceph-mon[47925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7037f613-ffd9-4b0e-84a1-0c04d4181ff1"}]': finished 2024-04-06T14:12:19.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:18 smithi086 ceph-mon[47925]: osdmap e37: 6 total, 5 up, 6 in 2024-04-06T14:12:19.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:18 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:19.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:18 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/813678052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T14:12:20.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:20 smithi032 ceph-mon[39730]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:21.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:20 smithi086 ceph-mon[47925]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:22.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:22 smithi032 ceph-mon[39730]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:23.095 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:22 smithi086 ceph-mon[47925]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:24 smithi086 ceph-mon[47925]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:24.979 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:24 smithi032 ceph-mon[39730]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T14:12:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:25 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:26.027 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T14:12:26.027 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:25 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:26.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:26 smithi032 ceph-mon[39730]: Deploying daemon osd.5 on smithi086 2024-04-06T14:12:26.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:26 smithi032 ceph-mon[39730]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:26.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:26 smithi086 ceph-mon[47925]: Deploying daemon osd.5 on smithi086 2024-04-06T14:12:26.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:26 smithi086 ceph-mon[47925]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:27.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:27 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:12:28.000 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:27 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:12:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:28 smithi032 ceph-mon[39730]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:29.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:28 smithi086 ceph-mon[47925]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:31.026 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:30 smithi086 ceph-mon[47925]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:31.026 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:31.026 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:12:31.026 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:30 smithi032 ceph-mon[39730]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:12:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:32.999 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:32 smithi086 ceph-mon[47925]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:33.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:32 smithi032 ceph-mon[39730]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:34.510 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 5 on host 'smithi086' 2024-04-06T14:12:34.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.510+0000 7f726dffb700 1 -- 172.21.15.86:0/2576875263 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7278102560 con 0x7f72640692a0 2024-04-06T14:12:34.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72640692a0 msgr2=0x7f726406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:34.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72640692a0 0x7f726406b750 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7268006fd0 tx=0x7f7268006db0 comp rx=0 tx=0).stop 2024-04-06T14:12:34.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 msgr2=0x7f727810dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:34.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 0x7f727810dd90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7274009fd0 tx=0x7f7274009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:34.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 shutdown_connections 2024-04-06T14:12:34.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72640692a0 0x7f726406b750 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:34.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72780fe440 0x7f727810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:34.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 --2- 172.21.15.86:0/2576875263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72780fed90 0x7f727810dd90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:34.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 >> 172.21.15.86:0/2576875263 conn(0x7f72780f6be0 msgr2=0x7f72780f8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:34.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 shutdown_connections 2024-04-06T14:12:34.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:34.512+0000 7f727f8ca700 1 -- 172.21.15.86:0/2576875263 wait complete. 2024-04-06T14:12:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:34 smithi086 ceph-mon[47925]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:34 smithi086 ceph-mon[47925]: from='osd.5 [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T14:12:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:34 smithi086 ceph-mon[47925]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T14:12:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:34 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:34 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:35.191 DEBUG:teuthology.orchestra.run.smithi086:osd.5> sudo journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-06T14:12:35.193 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-06T14:12:35.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-06T14:12:35.223 INFO:journalctl@ceph.osd.5.smithi086.stdout:-- Logs begin at Sat 2024-04-06 13:55:25 UTC. -- 2024-04-06T14:12:35.228 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:34 smithi032 ceph-mon[39730]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:35.228 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:34 smithi032 ceph-mon[39730]: from='osd.5 [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T14:12:35.228 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:34 smithi032 ceph-mon[39730]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T14:12:35.228 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:34 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:35.228 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:34 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:35.487 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:12:36.100 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:12:35 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[71241]: 2024-04-06T14:12:35.747+0000 7f6d4b6d4700 -1 osd.5 0 waiting for initial osdmap 2024-04-06T14:12:36.100 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:12:35 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[71241]: 2024-04-06T14:12:35.754+0000 7f6d45cce700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:12:36.100 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:35 smithi086 ceph-mon[47925]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T14:12:36.101 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:35 smithi086 ceph-mon[47925]: osdmap e38: 6 total, 5 up, 6 in 2024-04-06T14:12:36.101 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:35 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:36.101 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:35 smithi086 ceph-mon[47925]: from='osd.5 [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:12:36.101 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:35 smithi086 ceph-mon[47925]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:12:36.101 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:35 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:36.101 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:35 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:35 smithi032 ceph-mon[39730]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T14:12:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:35 smithi032 ceph-mon[39730]: osdmap e38: 6 total, 5 up, 6 in 2024-04-06T14:12:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:35 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:35 smithi032 ceph-mon[39730]: from='osd.5 [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:12:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:35 smithi032 ceph-mon[39730]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:12:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:35 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:35 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:37.038 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:36 smithi032 ceph-mon[39730]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:37.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:36 smithi032 ceph-mon[39730]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-04-06T14:12:37.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:36 smithi032 ceph-mon[39730]: osdmap e39: 6 total, 5 up, 6 in 2024-04-06T14:12:37.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:37.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:37.067 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:36 smithi086 ceph-mon[47925]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:37.067 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:36 smithi086 ceph-mon[47925]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-04-06T14:12:37.067 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:36 smithi086 ceph-mon[47925]: osdmap e39: 6 total, 5 up, 6 in 2024-04-06T14:12:37.067 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:37.068 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:37.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.295+0000 7f9c57a37700 1 -- 172.21.15.32:0/3113626784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500fe400 msgr2=0x7f9c500fe810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:37.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.295+0000 7f9c57a37700 1 --2- 172.21.15.32:0/3113626784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500fe400 0x7f9c500fe810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c40004940 tx=0x7f9c40009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:37.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.295+0000 7f9c57a37700 1 -- 172.21.15.32:0/3113626784 shutdown_connections 2024-04-06T14:12:37.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.295+0000 7f9c57a37700 1 --2- 172.21.15.32:0/3113626784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c500fed50 0x7f9c500fae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:37.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.295+0000 7f9c57a37700 1 --2- 172.21.15.32:0/3113626784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500fe400 0x7f9c500fe810 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:37.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.295+0000 7f9c57a37700 1 -- 172.21.15.32:0/3113626784 >> 172.21.15.32:0/3113626784 conn(0x7f9c500f6ba0 msgr2=0x7f9c500f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:37.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.295+0000 7f9c57a37700 1 -- 172.21.15.32:0/3113626784 shutdown_connections 2024-04-06T14:12:37.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.295+0000 7f9c57a37700 1 -- 172.21.15.32:0/3113626784 wait complete. 2024-04-06T14:12:37.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c57a37700 1 Processor -- start 2024-04-06T14:12:37.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c57a37700 1 -- start start 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c57a37700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c500fe400 0x7f9c5010d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c57a37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500fed50 0x7f9c5010de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c57a37700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5010e400 con 0x7f9c500fed50 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c57a37700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5010ec50 con 0x7f9c500fe400 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c54fd2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500fed50 0x7f9c5010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c557d3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c500fe400 0x7f9c5010d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c557d3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c500fe400 0x7f9c5010d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:46456/0 (socket says 172.21.15.32:46456) 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c54fd2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500fed50 0x7f9c5010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40504/0 (socket says 172.21.15.32:40504) 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.296+0000 7f9c557d3700 1 -- 172.21.15.32:0/3834918736 learned_addr learned my addr 172.21.15.32:0/3834918736 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.297+0000 7f9c557d3700 1 -- 172.21.15.32:0/3834918736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500fed50 msgr2=0x7f9c5010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.297+0000 7f9c557d3700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500fed50 0x7f9c5010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.297+0000 7f9c557d3700 1 -- 172.21.15.32:0/3834918736 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c40005040 con 0x7f9c500fe400 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.297+0000 7f9c557d3700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c500fe400 0x7f9c5010d8c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9c400137b0 tx=0x7f9c40009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.297+0000 7f9c45ffb700 1 -- 172.21.15.32:0/3834918736 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c4000e070 con 0x7f9c500fe400 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.297+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5010eed0 con 0x7f9c500fe400 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.297+0000 7f9c45ffb700 1 -- 172.21.15.32:0/3834918736 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c4001c400 con 0x7f9c500fe400 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.297+0000 7f9c45ffb700 1 -- 172.21.15.32:0/3834918736 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c4000b650 con 0x7f9c500fe400 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.297+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5010f3b0 con 0x7f9c500fe400 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.298+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5005f5f0 con 0x7f9c500fe400 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.299+0000 7f9c45ffb700 1 -- 172.21.15.32:0/3834918736 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9c4001c9e0 con 0x7f9c500fe400 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.299+0000 7f9c45ffb700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9c3c0692a0 0x7f9c3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.299+0000 7f9c45ffb700 1 -- 172.21.15.32:0/3834918736 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9c40088470 con 0x7f9c500fe400 2024-04-06T14:12:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.299+0000 7f9c54fd2700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9c3c0692a0 0x7f9c3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:37.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.300+0000 7f9c54fd2700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9c3c0692a0 0x7f9c3c06b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c009b80 tx=0x7f9c4c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:37.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.304+0000 7f9c45ffb700 1 -- 172.21.15.32:0/3834918736 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9c40058550 con 0x7f9c500fe400 2024-04-06T14:12:37.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.499+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f9c5010f780 con 0x7f9c500fe400 2024-04-06T14:12:37.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.499+0000 7f9c45ffb700 1 -- 172.21.15.32:0/3834918736 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f9c40004490 con 0x7f9c500fe400 2024-04-06T14:12:37.500 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9c3c0692a0 msgr2=0x7f9c3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9c3c0692a0 0x7f9c3c06b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c009b80 tx=0x7f9c4c00b040 comp rx=0 tx=0).stop 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c500fe400 msgr2=0x7f9c5010d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c500fe400 0x7f9c5010d8c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9c400137b0 tx=0x7f9c40009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 shutdown_connections 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9c3c0692a0 0x7f9c3c06b750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c500fe400 0x7f9c5010d8c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 --2- 172.21.15.32:0/3834918736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c500fed50 0x7f9c5010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 >> 172.21.15.32:0/3834918736 conn(0x7f9c500f6ba0 msgr2=0x7f9c500f8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 shutdown_connections 2024-04-06T14:12:37.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:37.502+0000 7f9c57a37700 1 -- 172.21.15.32:0/3834918736 wait complete. 2024-04-06T14:12:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:37 smithi032 ceph-mon[39730]: purged_snaps scrub starts 2024-04-06T14:12:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:37 smithi032 ceph-mon[39730]: purged_snaps scrub ok 2024-04-06T14:12:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:37 smithi032 ceph-mon[39730]: osd.5 [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] boot 2024-04-06T14:12:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:37 smithi032 ceph-mon[39730]: osdmap e40: 6 total, 6 up, 6 in 2024-04-06T14:12:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:37 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:37 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3834918736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T14:12:38.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:37 smithi086 ceph-mon[47925]: purged_snaps scrub starts 2024-04-06T14:12:38.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:37 smithi086 ceph-mon[47925]: purged_snaps scrub ok 2024-04-06T14:12:38.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:37 smithi086 ceph-mon[47925]: osd.5 [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] boot 2024-04-06T14:12:38.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:37 smithi086 ceph-mon[47925]: osdmap e40: 6 total, 6 up, 6 in 2024-04-06T14:12:38.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:37 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:12:38.110 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:37 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3834918736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T14:12:38.193 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1712412756,"num_in_osds":6,"osd_in_since":1712412737,"num_remapped_pgs":0} 2024-04-06T14:12:38.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T14:12:38.521 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: osdmap e41: 6 total, 6 up, 6 in 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:39.051 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:12:39.052 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:38 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:39.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: osdmap e41: 6 total, 6 up, 6 in 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:12:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:38 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:40.111 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:39 smithi032 ceph-mon[39730]: Detected new or changed devices on smithi086 2024-04-06T14:12:40.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:39 smithi032 ceph-mon[39730]: Adjusting osd_memory_target on smithi086 to 5323M 2024-04-06T14:12:40.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:39 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:12:40.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:39 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:40.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:39 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:12:40.112 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:39 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:39 smithi086 ceph-mon[47925]: Detected new or changed devices on smithi086 2024-04-06T14:12:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:39 smithi086 ceph-mon[47925]: Adjusting osd_memory_target on smithi086 to 5323M 2024-04-06T14:12:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:39 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:12:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:39 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:12:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:39 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:12:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:39 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:12:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.371+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/185335395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c781037b0 msgr2=0x7f2c780fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.371+0000 7f2c7f8e3700 1 --2- 172.21.15.32:0/185335395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c781037b0 0x7f2c780fce20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2c74004940 tx=0x7f2c74009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.371+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/185335395 shutdown_connections 2024-04-06T14:12:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.371+0000 7f2c7f8e3700 1 --2- 172.21.15.32:0/185335395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c781037b0 0x7f2c780fce20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.371+0000 7f2c7f8e3700 1 --2- 172.21.15.32:0/185335395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c78102e10 0x7f2c781031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.371+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/185335395 >> 172.21.15.32:0/185335395 conn(0x7f2c780f89a0 msgr2=0x7f2c780fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.371+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/185335395 shutdown_connections 2024-04-06T14:12:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.371+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/185335395 wait complete. 2024-04-06T14:12:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.372+0000 7f2c7f8e3700 1 Processor -- start 2024-04-06T14:12:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.372+0000 7f2c7f8e3700 1 -- start start 2024-04-06T14:12:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.372+0000 7f2c7f8e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c78102e10 0x7f2c78113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.372+0000 7f2c7f8e3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781037b0 0x7f2c78114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.372+0000 7f2c7f8e3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78114660 con 0x7f2c78102e10 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.372+0000 7f2c7f8e3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7810da50 con 0x7f2c781037b0 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.372+0000 7f2c7d67f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c78102e10 0x7f2c78113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.372+0000 7f2c7d67f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c78102e10 0x7f2c78113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40524/0 (socket says 172.21.15.32:40524) 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.372+0000 7f2c7d67f700 1 -- 172.21.15.32:0/2212444101 learned_addr learned my addr 172.21.15.32:0/2212444101 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c7ce7e700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781037b0 0x7f2c78114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c7d67f700 1 -- 172.21.15.32:0/2212444101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781037b0 msgr2=0x7f2c78114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c7d67f700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781037b0 0x7f2c78114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c7d67f700 1 -- 172.21.15.32:0/2212444101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c74005040 con 0x7f2c78102e10 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c7d67f700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c78102e10 0x7f2c78113ad0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00c810 tx=0x7f2c6c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c69ffb700 1 -- 172.21.15.32:0/2212444101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6c012810 con 0x7f2c78102e10 2024-04-06T14:12:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c69ffb700 1 -- 172.21.15.32:0/2212444101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c6c012e20 con 0x7f2c78102e10 2024-04-06T14:12:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7810dd30 con 0x7f2c78102e10 2024-04-06T14:12:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c69ffb700 1 -- 172.21.15.32:0/2212444101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6c0102e0 con 0x7f2c78102e10 2024-04-06T14:12:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.373+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7810e2c0 con 0x7f2c78102e10 2024-04-06T14:12:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.374+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c780fe560 con 0x7f2c78102e10 2024-04-06T14:12:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.375+0000 7f2c69ffb700 1 -- 172.21.15.32:0/2212444101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2c6c011380 con 0x7f2c78102e10 2024-04-06T14:12:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.375+0000 7f2c69ffb700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2c640692b0 0x7f2c6406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.375+0000 7f2c69ffb700 1 -- 172.21.15.32:0/2212444101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2c6c0871f0 con 0x7f2c78102e10 2024-04-06T14:12:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.375+0000 7f2c7ce7e700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2c640692b0 0x7f2c6406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.376+0000 7f2c7ce7e700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2c640692b0 0x7f2c6406b760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c7810ed30 tx=0x7f2c74009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:40.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.380+0000 7f2c69ffb700 1 -- 172.21.15.32:0/2212444101 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2c6c057270 con 0x7f2c78102e10 2024-04-06T14:12:40.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.574+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c78109dd0 con 0x7f2c78102e10 2024-04-06T14:12:40.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.575+0000 7f2c69ffb700 1 -- 172.21.15.32:0/2212444101 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11043 (secure 0 0 0) 0x7f2c6c056e00 con 0x7f2c78102e10 2024-04-06T14:12:40.575 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:12:40.575 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":41,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","created":"2024-04-06T14:06:48.330591+0000","modified":"2024-04-06T14:12:37.750341+0000","last_up_change":"2024-04-06T14:12:36.744888+0000","last_in_change":"2024-04-06T14:12:17.748510+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T14:11:20.110423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6803","nonce":1233836109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6805","nonce":1233836109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6809","nonce":1233836109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6807","nonce":1233836109}]},"public_addr":"172.21.15.32:6803/1233836109","cluster_addr":"172.21.15.32:6805/1233836109","heartbeat_back_addr":"172.21.15.32:6809/1233836109","heartbeat_front_addr":"172.21.15.32:6807/1233836109","state":["exists","up"]},{"osd":1,"uuid":"9185a389-ef69-41bc-9e10-46c6a2e02a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6811","nonce":104495793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6813","nonce":104495793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6817","nonce":104495793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6815","nonce":104495793}]},"public_addr":"172.21.15.32:6811/104495793","cluster_addr":"172.21.15.32:6813/104495793","heartbeat_back_addr":"172.21.15.32:6817/104495793","heartbeat_front_addr":"172.21.15.32:6815/104495793","state":["exists","up"]},{"osd":2,"uuid":"af4793e1-b5bb-4919-838e-67703c83296a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6819","nonce":2571669324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6821","nonce":2571669324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6825","nonce":2571669324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6823","nonce":2571669324}]},"public_addr":"172.21.15.32:6819/2571669324","cluster_addr":"172.21.15.32:6821/2571669324","heartbeat_back_addr":"172.21.15.32:6825/2571669324","heartbeat_front_addr":"172.21.15.32:6823/2571669324","state":["exists","up"]},{"osd":3,"uuid":"70c8b884-5214-44f2-9a31-55556a11c20a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6801","nonce":3912040245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6803","nonce":3912040245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6807","nonce":3912040245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6805","nonce":3912040245}]},"public_addr":"172.21.15.86:6801/3912040245","cluster_addr":"172.21.15.86:6803/3912040245","heartbeat_back_addr":"172.21.15.86:6807/3912040245","heartbeat_front_addr":"172.21.15.86:6805/3912040245","state":["exists","up"]},{"osd":4,"uuid":"91f548ac-4f42-4083-a027-295b829646a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6809","nonce":3794387444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6811","nonce":3794387444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6815","nonce":3794387444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6813","nonce":3794387444}]},"public_addr":"172.21.15.86:6809/3794387444","cluster_addr":"172.21.15.86:6811/3794387444","heartbeat_back_addr":"172.21.15.86:6815/3794387444","heartbeat_front_addr":"172.21.15.86:6813/3794387444","state":["exists","up"]},{"osd":5,"uuid":"7037f613-ffd9-4b0e-84a1-0c04d4181ff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6817","nonce":2011390682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6819","nonce":2011390682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6823","nonce":2011390682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6821","nonce":2011390682}]},"public_addr":"172.21.15.86:6817/2011390682","cluster_addr":"172.21.15.86:6819/2011390682","heartbeat_back_addr":"172.21.15.86:6823/2011390682","heartbeat_front_addr":"172.21.15.86:6821/2011390682","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:13.684938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:44.549488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:15.893369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:41.588409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:08.612078+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:35.380707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6800/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/711512148":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/3330131734":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/2227455350":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/1187866165":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6800/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/1877817963":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6801/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:6800/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/68274703":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/3548398153":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/935909372":"2024-04-07T14:07:51.523157+0000","172.21.15.32:6801/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6801/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/3250004174":"2024-04-07T14:07:17.560285+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2c640692b0 msgr2=0x7f2c6406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2c640692b0 0x7f2c6406b760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c7810ed30 tx=0x7f2c74009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c78102e10 msgr2=0x7f2c78113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c78102e10 0x7f2c78113ad0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00c810 tx=0x7f2c6c004470 comp rx=0 tx=0).stop 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 shutdown_connections 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2c640692b0 0x7f2c6406b760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c78102e10 0x7f2c78113ad0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 --2- 172.21.15.32:0/2212444101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781037b0 0x7f2c78114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 >> 172.21.15.32:0/2212444101 conn(0x7f2c780f89a0 msgr2=0x7f2c780f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 shutdown_connections 2024-04-06T14:12:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:40.577+0000 7f2c7f8e3700 1 -- 172.21.15.32:0/2212444101 wait complete. 2024-04-06T14:12:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:40 smithi086 ceph-mon[47925]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:40 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2212444101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:12:41.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:40 smithi032 ceph-mon[39730]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:41.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:40 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2212444101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:12:41.400 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T14:11:20.110423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-06T14:12:41.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-06T14:12:41.682 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:12:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:42 smithi032 ceph-mon[39730]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:12:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:42 smithi086 ceph-mon[47925]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:12:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.638+0000 7f150ea4a700 1 -- 172.21.15.32:0/1830442006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 msgr2=0x7f15080fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.638+0000 7f150ea4a700 1 --2- 172.21.15.32:0/1830442006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 0x7f15080fce20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f14fc004940 tx=0x7f14fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.638+0000 7f150ea4a700 1 -- 172.21.15.32:0/1830442006 shutdown_connections 2024-04-06T14:12:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.638+0000 7f150ea4a700 1 --2- 172.21.15.32:0/1830442006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 0x7f15080fce20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.638+0000 7f150ea4a700 1 --2- 172.21.15.32:0/1830442006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1508102e10 0x7f15081031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.638+0000 7f150ea4a700 1 -- 172.21.15.32:0/1830442006 >> 172.21.15.32:0/1830442006 conn(0x7f15080f89a0 msgr2=0x7f15080fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.638+0000 7f150ea4a700 1 -- 172.21.15.32:0/1830442006 shutdown_connections 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.638+0000 7f150ea4a700 1 -- 172.21.15.32:0/1830442006 wait complete. 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.638+0000 7f150ea4a700 1 Processor -- start 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f150ea4a700 1 -- start start 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f150ea4a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1508102e10 0x7f1508113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f150ea4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 0x7f1508113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f150ea4a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081145c0 con 0x7f15081037b0 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f150ea4a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150810d9b0 con 0x7f1508102e10 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f15077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 0x7f1508113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f15077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 0x7f1508113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40532/0 (socket says 172.21.15.32:40532) 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f15077fe700 1 -- 172.21.15.32:0/3084080689 learned_addr learned my addr 172.21.15.32:0/3084080689 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:12:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f1507fff700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1508102e10 0x7f1508113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f15077fe700 1 -- 172.21.15.32:0/3084080689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1508102e10 msgr2=0x7f1508113a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f15077fe700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1508102e10 0x7f1508113a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f15077fe700 1 -- 172.21.15.32:0/3084080689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc005040 con 0x7f15081037b0 2024-04-06T14:12:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.639+0000 7f15077fe700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 0x7f1508113f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0093b0 tx=0x7f14fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.640+0000 7f1504ff9700 1 -- 172.21.15.32:0/3084080689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14fc00e040 con 0x7f15081037b0 2024-04-06T14:12:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.640+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150810dc30 con 0x7f15081037b0 2024-04-06T14:12:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.640+0000 7f1504ff9700 1 -- 172.21.15.32:0/3084080689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14fc013420 con 0x7f15081037b0 2024-04-06T14:12:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.640+0000 7f1504ff9700 1 -- 172.21.15.32:0/3084080689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14fc01c630 con 0x7f15081037b0 2024-04-06T14:12:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.640+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150810e1f0 con 0x7f15081037b0 2024-04-06T14:12:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.641+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15080fe560 con 0x7f15081037b0 2024-04-06T14:12:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.641+0000 7f1504ff9700 1 -- 172.21.15.32:0/3084080689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f14fc0139e0 con 0x7f15081037b0 2024-04-06T14:12:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.642+0000 7f1504ff9700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f14f00692a0 0x7f14f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.642+0000 7f1504ff9700 1 -- 172.21.15.32:0/3084080689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f14fc087fe0 con 0x7f15081037b0 2024-04-06T14:12:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.642+0000 7f1507fff700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f14f00692a0 0x7f14f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.642+0000 7f1507fff700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f14f00692a0 0x7f14f006b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009fd0 tx=0x7f14f800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.646+0000 7f1504ff9700 1 -- 172.21.15.32:0/3084080689 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f14fc0ba190 con 0x7f15081037b0 2024-04-06T14:12:43.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.839+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1508109db0 con 0x7f15081037b0 2024-04-06T14:12:43.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.840+0000 7f1504ff9700 1 -- 172.21.15.32:0/3084080689 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14fc058040 con 0x7f15081037b0 2024-04-06T14:12:43.840 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-06T14:12:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f14f00692a0 msgr2=0x7f14f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f14f00692a0 0x7f14f006b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009fd0 tx=0x7f14f800b040 comp rx=0 tx=0).stop 2024-04-06T14:12:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 msgr2=0x7f1508113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 0x7f1508113f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0093b0 tx=0x7f14fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 shutdown_connections 2024-04-06T14:12:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f14f00692a0 0x7f14f006b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1508102e10 0x7f1508113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 --2- 172.21.15.32:0/3084080689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15081037b0 0x7f1508113f70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 >> 172.21.15.32:0/3084080689 conn(0x7f15080f89a0 msgr2=0x7f15080f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 shutdown_connections 2024-04-06T14:12:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:43.842+0000 7f150ea4a700 1 -- 172.21.15.32:0/3084080689 wait complete. 2024-04-06T14:12:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:43 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3084080689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:12:44.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:43 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3084080689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:12:44.605 INFO:tasks.cephadm:Setting up client nodes... 2024-04-06T14:12:44.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-06T14:12:44.888 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:12:45.118 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:44 smithi032 ceph-mon[39730]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:45.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:44 smithi086 ceph-mon[47925]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.813+0000 7fa3c8f61700 1 -- 172.21.15.32:0/1758762192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fbd20 msgr2=0x7fa3c40fc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.813+0000 7fa3c8f61700 1 --2- 172.21.15.32:0/1758762192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fbd20 0x7fa3c40fc130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8004940 tx=0x7fa3b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.813+0000 7fa3c8f61700 1 -- 172.21.15.32:0/1758762192 shutdown_connections 2024-04-06T14:12:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.813+0000 7fa3c8f61700 1 --2- 172.21.15.32:0/1758762192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3c40fcfd0 0x7fa3c40fd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.813+0000 7fa3c8f61700 1 --2- 172.21.15.32:0/1758762192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fbd20 0x7fa3c40fc130 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.813+0000 7fa3c8f61700 1 -- 172.21.15.32:0/1758762192 >> 172.21.15.32:0/1758762192 conn(0x7fa3c40f7460 msgr2=0x7fa3c40f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.813+0000 7fa3c8f61700 1 -- 172.21.15.32:0/1758762192 shutdown_connections 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.813+0000 7fa3c8f61700 1 -- 172.21.15.32:0/1758762192 wait complete. 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c8f61700 1 Processor -- start 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c8f61700 1 -- start start 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c8f61700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3c40fbd20 0x7fa3c4066000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c8f61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fcfd0 0x7fa3c41125d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c8f61700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c4112b10 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c8f61700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c4112c50 con 0x7fa3c40fbd20 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fcfd0 0x7fa3c41125d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fcfd0 0x7fa3c41125d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39770/0 (socket says 172.21.15.32:39770) 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c1d9b700 1 -- 172.21.15.32:0/894476464 learned_addr learned my addr 172.21.15.32:0/894476464 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:12:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c259c700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3c40fbd20 0x7fa3c4066000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c1d9b700 1 -- 172.21.15.32:0/894476464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3c40fbd20 msgr2=0x7fa3c4066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c1d9b700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3c40fbd20 0x7fa3c4066000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c1d9b700 1 -- 172.21.15.32:0/894476464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8005040 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.814+0000 7fa3c1d9b700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fcfd0 0x7fa3c41125d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400eb40 tx=0x7fa3b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.815+0000 7fa3b2ffd700 1 -- 172.21.15.32:0/894476464 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3b4007cc0 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.815+0000 7fa3b2ffd700 1 -- 172.21.15.32:0/894476464 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3b400f040 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.815+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c4112e50 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.815+0000 7fa3b2ffd700 1 -- 172.21.15.32:0/894476464 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3b4009570 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.815+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c41133e0 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.816+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c404c710 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.816+0000 7fa3b2ffd700 1 -- 172.21.15.32:0/894476464 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa3b40096d0 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.817+0000 7fa3b2ffd700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fa3ac0692a0 0x7fa3ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.817+0000 7fa3b2ffd700 1 -- 172.21.15.32:0/894476464 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa3b4086cd0 con 0x7fa3c40fcfd0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.817+0000 7fa3c259c700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fa3ac0692a0 0x7fa3ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.817+0000 7fa3c259c700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fa3ac0692a0 0x7fa3ac06b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80137b0 tx=0x7fa3b8009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:46.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:46.821+0000 7fa3b2ffd700 1 -- 172.21.15.32:0/894476464 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa3b4056e80 con 0x7fa3c40fcfd0 2024-04-06T14:12:47.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.092+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa3c41136e0 con 0x7fa3c40fcfd0 2024-04-06T14:12:47.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.095+0000 7fa3b2ffd700 1 -- 172.21.15.32:0/894476464 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa3b4056a10 con 0x7fa3c40fcfd0 2024-04-06T14:12:47.095 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-04-06T14:12:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBfWBFmhWKUBRAAfAGubGIR9uluOzeni6vh2Q== 2024-04-06T14:12:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.097+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fa3ac0692a0 msgr2=0x7fa3ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.097+0000 7fa3c8f61700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fa3ac0692a0 0x7fa3ac06b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80137b0 tx=0x7fa3b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.097+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fcfd0 msgr2=0x7fa3c41125d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.097+0000 7fa3c8f61700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fcfd0 0x7fa3c41125d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400eb40 tx=0x7fa3b400b040 comp rx=0 tx=0).stop 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.097+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 shutdown_connections 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.097+0000 7fa3c8f61700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fa3ac0692a0 0x7fa3ac06b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.097+0000 7fa3c8f61700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3c40fbd20 0x7fa3c4066000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.097+0000 7fa3c8f61700 1 --2- 172.21.15.32:0/894476464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3c40fcfd0 0x7fa3c41125d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.097+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 >> 172.21.15.32:0/894476464 conn(0x7fa3c40f7460 msgr2=0x7fa3c40f8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.098+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 shutdown_connections 2024-04-06T14:12:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:47.098+0000 7fa3c8f61700 1 -- 172.21.15.32:0/894476464 wait complete. 2024-04-06T14:12:47.126 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:46 smithi032 ceph-mon[39730]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:46 smithi086 ceph-mon[47925]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:47.877 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:12:47.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-06T14:12:47.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-06T14:12:47.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-06T14:12:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:47 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/894476464' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T14:12:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:47 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/894476464' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T14:12:48.199 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:12:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:47 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/894476464' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T14:12:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:47 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/894476464' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T14:12:49.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:48 smithi086 ceph-mon[47925]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:48 smithi032 ceph-mon[39730]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 -- 172.21.15.86:0/3042453422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 msgr2=0x7f9fd80fd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 --2- 172.21.15.86:0/3042453422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 0x7f9fd80fd440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4004940 tx=0x7f9fd4009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 -- 172.21.15.86:0/3042453422 shutdown_connections 2024-04-06T14:12:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 --2- 172.21.15.86:0/3042453422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 0x7f9fd80fd440 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 --2- 172.21.15.86:0/3042453422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd80fbd20 0x7f9fd80fc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 -- 172.21.15.86:0/3042453422 >> 172.21.15.86:0/3042453422 conn(0x7f9fd80f7480 msgr2=0x7f9fd80f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 -- 172.21.15.86:0/3042453422 shutdown_connections 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 -- 172.21.15.86:0/3042453422 wait complete. 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 Processor -- start 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.889+0000 7f9fe004a700 1 -- start start 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fe004a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd80fbd20 0x7f9fd80ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fe004a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 0x7f9fd80fe310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fe004a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8100210 con 0x7f9fd80fbd20 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fe004a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd80fe850 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fdd5e5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 0x7f9fd80fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fdd5e5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 0x7f9fd80fe310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46684/0 (socket says 172.21.15.86:46684) 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fdd5e5700 1 -- 172.21.15.86:0/223351706 learned_addr learned my addr 172.21.15.86:0/223351706 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fdd5e5700 1 -- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd80fbd20 msgr2=0x7f9fd80ffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fddde6700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd80fbd20 0x7f9fd80ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fdd5e5700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd80fbd20 0x7f9fd80ffc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fdd5e5700 1 -- 172.21.15.86:0/223351706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd4005040 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fdd5e5700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 0x7f9fd80fe310 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009380 tx=0x7f9fd4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fce7fc700 1 -- 172.21.15.86:0/223351706 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fd400e040 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fce7fc700 1 -- 172.21.15.86:0/223351706 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fd400bd20 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd80fead0 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fce7fc700 1 -- 172.21.15.86:0/223351706 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fd401c630 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.890+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd80ff010 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.891+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd8101b60 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.892+0000 7f9fce7fc700 1 -- 172.21.15.86:0/223351706 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9fd4013420 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.892+0000 7f9fce7fc700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9fc40692a0 0x7f9fc406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.892+0000 7f9fce7fc700 1 -- 172.21.15.86:0/223351706 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9fd4087b00 con 0x7f9fd80fcfd0 2024-04-06T14:12:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.892+0000 7f9fddde6700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9fc40692a0 0x7f9fc406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.893+0000 7f9fddde6700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9fc40692a0 0x7f9fc406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8004d60 tx=0x7f9fc800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:49.896+0000 7f9fce7fc700 1 -- 172.21.15.86:0/223351706 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9fd4057ca0 con 0x7f9fd80fcfd0 2024-04-06T14:12:50.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.153+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f9fd804c710 con 0x7f9fd80fcfd0 2024-04-06T14:12:50.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.158+0000 7f9fce7fc700 1 -- 172.21.15.86:0/223351706 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f9fd405be60 con 0x7f9fd80fcfd0 2024-04-06T14:12:50.158 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2024-04-06T14:12:50.158 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBiWBFmB4xFCRAA/K6EGNVdLFMqZj69R6r2rw== 2024-04-06T14:12:50.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9fc40692a0 msgr2=0x7f9fc406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:50.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9fc40692a0 0x7f9fc406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8004d60 tx=0x7f9fc800b040 comp rx=0 tx=0).stop 2024-04-06T14:12:50.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 msgr2=0x7f9fd80fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:50.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 0x7f9fd80fe310 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009380 tx=0x7f9fd4009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:50.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 shutdown_connections 2024-04-06T14:12:50.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9fc40692a0 0x7f9fc406b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:50.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd80fbd20 0x7f9fd80ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:50.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 --2- 172.21.15.86:0/223351706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fd80fcfd0 0x7f9fd80fe310 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:50.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 >> 172.21.15.86:0/223351706 conn(0x7f9fd80f7480 msgr2=0x7f9fd80664e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:50.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 shutdown_connections 2024-04-06T14:12:50.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-06T14:12:50.160+0000 7f9fe004a700 1 -- 172.21.15.86:0/223351706 wait complete. 2024-04-06T14:12:50.972 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:50 smithi086 ceph-mon[47925]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:50.972 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:50 smithi086 ceph-mon[47925]: from='client.? 172.21.15.86:0/223351706' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T14:12:50.972 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:50 smithi086 ceph-mon[47925]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T14:12:50.972 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:50 smithi086 ceph-mon[47925]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T14:12:50.995 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:12:50.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-06T14:12:50.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-06T14:12:51.038 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-06T14:12:51.039 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-06T14:12:51.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-06T14:12:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:50 smithi032 ceph-mon[39730]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:50 smithi032 ceph-mon[39730]: from='client.? 172.21.15.86:0/223351706' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T14:12:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:50 smithi032 ceph-mon[39730]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T14:12:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:50 smithi032 ceph-mon[39730]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T14:12:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:12:52.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:52 smithi032 ceph-mon[39730]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:53.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:52 smithi086 ceph-mon[47925]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:53.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.253+0000 7facb2b8d700 1 -- 172.21.15.32:0/3785738397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 msgr2=0x7facac1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:53.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.253+0000 7facb2b8d700 1 --2- 172.21.15.32:0/3785738397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 0x7facac1031d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faca0004940 tx=0x7faca0009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:53.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.254+0000 7facb2b8d700 1 -- 172.21.15.32:0/3785738397 shutdown_connections 2024-04-06T14:12:53.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.254+0000 7facb2b8d700 1 --2- 172.21.15.32:0/3785738397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facac1037a0 0x7facac0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:53.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.254+0000 7facb2b8d700 1 --2- 172.21.15.32:0/3785738397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 0x7facac1031d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:53.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.254+0000 7facb2b8d700 1 -- 172.21.15.32:0/3785738397 >> 172.21.15.32:0/3785738397 conn(0x7facac0f89b0 msgr2=0x7facac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:53.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.254+0000 7facb2b8d700 1 -- 172.21.15.32:0/3785738397 shutdown_connections 2024-04-06T14:12:53.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.254+0000 7facb2b8d700 1 -- 172.21.15.32:0/3785738397 wait complete. 2024-04-06T14:12:53.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.254+0000 7facb2b8d700 1 Processor -- start 2024-04-06T14:12:53.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.254+0000 7facb2b8d700 1 -- start start 2024-04-06T14:12:53.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb2b8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 0x7facac1138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:53.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb2b8d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facac1037a0 0x7facac113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:53.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb2b8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac114440 con 0x7facac102e00 2024-04-06T14:12:53.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb2b8d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac10da40 con 0x7facac1037a0 2024-04-06T14:12:53.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb0929700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 0x7facac1138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:53.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb0929700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 0x7facac1138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39788/0 (socket says 172.21.15.32:39788) 2024-04-06T14:12:53.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb0929700 1 -- 172.21.15.32:0/2408635832 learned_addr learned my addr 172.21.15.32:0/2408635832 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:12:53.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facabfff700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facac1037a0 0x7facac113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:53.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb0929700 1 -- 172.21.15.32:0/2408635832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facac1037a0 msgr2=0x7facac113df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:53.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb0929700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facac1037a0 0x7facac113df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:53.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb0929700 1 -- 172.21.15.32:0/2408635832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca0005040 con 0x7facac102e00 2024-04-06T14:12:53.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb0929700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 0x7facac1138b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faca000b3b0 tx=0x7faca0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:53.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7faca97fa700 1 -- 172.21.15.32:0/2408635832 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faca000e040 con 0x7facac102e00 2024-04-06T14:12:53.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac10dcc0 con 0x7facac102e00 2024-04-06T14:12:53.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7faca97fa700 1 -- 172.21.15.32:0/2408635832 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faca0013420 con 0x7facac102e00 2024-04-06T14:12:53.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7faca97fa700 1 -- 172.21.15.32:0/2408635832 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faca001c630 con 0x7facac102e00 2024-04-06T14:12:53.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.255+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac10e280 con 0x7facac102e00 2024-04-06T14:12:53.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.257+0000 7faca97fa700 1 -- 172.21.15.32:0/2408635832 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7faca0013970 con 0x7facac102e00 2024-04-06T14:12:53.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.257+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac0fe550 con 0x7facac102e00 2024-04-06T14:12:53.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.257+0000 7faca97fa700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fac9c0692a0 0x7fac9c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:53.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.257+0000 7facabfff700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fac9c0692a0 0x7fac9c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:53.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.258+0000 7faca97fa700 1 -- 172.21.15.32:0/2408635832 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faca0088090 con 0x7facac102e00 2024-04-06T14:12:53.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.258+0000 7facabfff700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fac9c0692a0 0x7fac9c06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fac98007fd0 tx=0x7fac98009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:53.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.262+0000 7faca97fa700 1 -- 172.21.15.32:0/2408635832 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7faca0058190 con 0x7facac102e00 2024-04-06T14:12:53.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.509+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7facac109dc0 con 0x7facac102e00 2024-04-06T14:12:53.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.512+0000 7faca97fa700 1 -- 172.21.15.32:0/2408635832 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167169 (secure 0 0 0) 0x7faca0057d20 con 0x7facac102e00 2024-04-06T14:12:53.514 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:12:53.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fac9c0692a0 msgr2=0x7fac9c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:53.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fac9c0692a0 0x7fac9c06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fac98007fd0 tx=0x7fac98009040 comp rx=0 tx=0).stop 2024-04-06T14:12:53.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 msgr2=0x7facac1138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:53.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 0x7facac1138b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faca000b3b0 tx=0x7faca0009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:53.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 shutdown_connections 2024-04-06T14:12:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fac9c0692a0 0x7fac9c06b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facac102e00 0x7facac1138b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 --2- 172.21.15.32:0/2408635832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facac1037a0 0x7facac113df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 >> 172.21.15.32:0/2408635832 conn(0x7facac0f89b0 msgr2=0x7facac0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 shutdown_connections 2024-04-06T14:12:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:53.515+0000 7facb2b8d700 1 -- 172.21.15.32:0/2408635832 wait complete. 2024-04-06T14:12:54.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:53 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2408635832' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T14:12:54.341 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":19,"active_gid":14206,"active_name":"smithi032.peohxu","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":3067649142},{"type":"v1","addr":"172.21.15.32:6801","nonce":3067649142}]},"active_addr":"172.21.15.32:6801/3067649142","active_change":"2024-04-06T14:08:57.599195+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14226,"name":"smithi086.fyyhxm","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.32:8443/","prometheus":"http://172.21.15.32:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":2574238932}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":3796020247}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":1664607864}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":3226572738}]}]} 2024-04-06T14:12:54.347 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-06T14:12:54.347 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-06T14:12:54.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T14:12:54.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:53 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2408635832' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T14:12:54.630 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:12:55.160 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:54 smithi032 ceph-mon[39730]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:55.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:54 smithi086 ceph-mon[47925]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.510+0000 7fe931e0a700 1 -- 172.21.15.32:0/1704774263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 msgr2=0x7fe92c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.510+0000 7fe931e0a700 1 --2- 172.21.15.32:0/1704774263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 0x7fe92c0fbd80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe920004940 tx=0x7fe920009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.510+0000 7fe931e0a700 1 -- 172.21.15.32:0/1704774263 shutdown_connections 2024-04-06T14:12:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.510+0000 7fe931e0a700 1 --2- 172.21.15.32:0/1704774263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 0x7fe92c0fbd80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.510+0000 7fe931e0a700 1 --2- 172.21.15.32:0/1704774263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe92c101d70 0x7fe92c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.510+0000 7fe931e0a700 1 -- 172.21.15.32:0/1704774263 >> 172.21.15.32:0/1704774263 conn(0x7fe92c072bc0 msgr2=0x7fe92c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.510+0000 7fe931e0a700 1 -- 172.21.15.32:0/1704774263 shutdown_connections 2024-04-06T14:12:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.510+0000 7fe931e0a700 1 -- 172.21.15.32:0/1704774263 wait complete. 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe931e0a700 1 Processor -- start 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe931e0a700 1 -- start start 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe931e0a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe92c101d70 0x7fe92c115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe931e0a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 0x7fe92c116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe931e0a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c116850 con 0x7fe92c102710 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe931e0a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c10fc20 con 0x7fe92c101d70 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe92affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 0x7fe92c116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe92affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 0x7fe92c116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57420/0 (socket says 172.21.15.32:57420) 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe92affd700 1 -- 172.21.15.32:0/4288738202 learned_addr learned my addr 172.21.15.32:0/4288738202 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.511+0000 7fe92b7fe700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe92c101d70 0x7fe92c115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.512+0000 7fe92affd700 1 -- 172.21.15.32:0/4288738202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe92c101d70 msgr2=0x7fe92c115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.512+0000 7fe92affd700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe92c101d70 0x7fe92c115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.512+0000 7fe92affd700 1 -- 172.21.15.32:0/4288738202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe920005040 con 0x7fe92c102710 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.512+0000 7fe92affd700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 0x7fe92c116200 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe920009380 tx=0x7fe920009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.512+0000 7fe913fff700 1 -- 172.21.15.32:0/4288738202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe92000e040 con 0x7fe92c102710 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.512+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c10fea0 con 0x7fe92c102710 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.512+0000 7fe913fff700 1 -- 172.21.15.32:0/4288738202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe920013420 con 0x7fe92c102710 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.512+0000 7fe913fff700 1 -- 172.21.15.32:0/4288738202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe92001c630 con 0x7fe92c102710 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.512+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c110460 con 0x7fe92c102710 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.513+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c0fd4c0 con 0x7fe92c102710 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.514+0000 7fe913fff700 1 -- 172.21.15.32:0/4288738202 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe9200139b0 con 0x7fe92c102710 2024-04-06T14:12:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.514+0000 7fe913fff700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe9140692a0 0x7fe91406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:56.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.514+0000 7fe913fff700 1 -- 172.21.15.32:0/4288738202 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe920088aa0 con 0x7fe92c102710 2024-04-06T14:12:56.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.514+0000 7fe92b7fe700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe9140692a0 0x7fe91406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:56.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.515+0000 7fe92b7fe700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe9140692a0 0x7fe91406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe91c004f40 tx=0x7fe91c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:56.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.518+0000 7fe913fff700 1 -- 172.21.15.32:0/4288738202 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe920058b80 con 0x7fe92c102710 2024-04-06T14:12:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.711+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe92c05f5f0 con 0x7fe92c102710 2024-04-06T14:12:56.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.712+0000 7fe913fff700 1 -- 172.21.15.32:0/4288738202 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11043 (secure 0 0 0) 0x7fe920058710 con 0x7fe92c102710 2024-04-06T14:12:56.712 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:12:56.712 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":41,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","created":"2024-04-06T14:06:48.330591+0000","modified":"2024-04-06T14:12:37.750341+0000","last_up_change":"2024-04-06T14:12:36.744888+0000","last_in_change":"2024-04-06T14:12:17.748510+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T14:11:20.110423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6803","nonce":1233836109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6805","nonce":1233836109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6809","nonce":1233836109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6807","nonce":1233836109}]},"public_addr":"172.21.15.32:6803/1233836109","cluster_addr":"172.21.15.32:6805/1233836109","heartbeat_back_addr":"172.21.15.32:6809/1233836109","heartbeat_front_addr":"172.21.15.32:6807/1233836109","state":["exists","up"]},{"osd":1,"uuid":"9185a389-ef69-41bc-9e10-46c6a2e02a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6811","nonce":104495793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6813","nonce":104495793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6817","nonce":104495793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6815","nonce":104495793}]},"public_addr":"172.21.15.32:6811/104495793","cluster_addr":"172.21.15.32:6813/104495793","heartbeat_back_addr":"172.21.15.32:6817/104495793","heartbeat_front_addr":"172.21.15.32:6815/104495793","state":["exists","up"]},{"osd":2,"uuid":"af4793e1-b5bb-4919-838e-67703c83296a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6819","nonce":2571669324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6821","nonce":2571669324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6825","nonce":2571669324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6823","nonce":2571669324}]},"public_addr":"172.21.15.32:6819/2571669324","cluster_addr":"172.21.15.32:6821/2571669324","heartbeat_back_addr":"172.21.15.32:6825/2571669324","heartbeat_front_addr":"172.21.15.32:6823/2571669324","state":["exists","up"]},{"osd":3,"uuid":"70c8b884-5214-44f2-9a31-55556a11c20a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6801","nonce":3912040245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6803","nonce":3912040245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6807","nonce":3912040245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6805","nonce":3912040245}]},"public_addr":"172.21.15.86:6801/3912040245","cluster_addr":"172.21.15.86:6803/3912040245","heartbeat_back_addr":"172.21.15.86:6807/3912040245","heartbeat_front_addr":"172.21.15.86:6805/3912040245","state":["exists","up"]},{"osd":4,"uuid":"91f548ac-4f42-4083-a027-295b829646a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6809","nonce":3794387444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6811","nonce":3794387444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6815","nonce":3794387444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6813","nonce":3794387444}]},"public_addr":"172.21.15.86:6809/3794387444","cluster_addr":"172.21.15.86:6811/3794387444","heartbeat_back_addr":"172.21.15.86:6815/3794387444","heartbeat_front_addr":"172.21.15.86:6813/3794387444","state":["exists","up"]},{"osd":5,"uuid":"7037f613-ffd9-4b0e-84a1-0c04d4181ff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6817","nonce":2011390682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6819","nonce":2011390682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6823","nonce":2011390682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6821","nonce":2011390682}]},"public_addr":"172.21.15.86:6817/2011390682","cluster_addr":"172.21.15.86:6819/2011390682","heartbeat_back_addr":"172.21.15.86:6823/2011390682","heartbeat_front_addr":"172.21.15.86:6821/2011390682","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:13.684938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:44.549488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:15.893369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:41.588409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:08.612078+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:35.380707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6800/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/711512148":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/3330131734":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/2227455350":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/1187866165":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6800/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/1877817963":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6801/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:6800/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/68274703":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/3548398153":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/935909372":"2024-04-07T14:07:51.523157+0000","172.21.15.32:6801/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6801/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/3250004174":"2024-04-07T14:07:17.560285+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T14:12:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe9140692a0 msgr2=0x7fe91406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe9140692a0 0x7fe91406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe91c004f40 tx=0x7fe91c00b040 comp rx=0 tx=0).stop 2024-04-06T14:12:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 msgr2=0x7fe92c116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 0x7fe92c116200 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe920009380 tx=0x7fe920009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 shutdown_connections 2024-04-06T14:12:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe9140692a0 0x7fe91406b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe92c101d70 0x7fe92c115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 --2- 172.21.15.32:0/4288738202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe92c102710 0x7fe92c116200 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:56.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 >> 172.21.15.32:0/4288738202 conn(0x7fe92c072bc0 msgr2=0x7fe92c0659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:56.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 shutdown_connections 2024-04-06T14:12:56.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:56.714+0000 7fe931e0a700 1 -- 172.21.15.32:0/4288738202 wait complete. 2024-04-06T14:12:57.073 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:56 smithi032 ceph-mon[39730]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:57.073 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:56 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/4288738202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:12:57.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:56 smithi086 ceph-mon[47925]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:57.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:56 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/4288738202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:12:57.538 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-06T14:12:57.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T14:12:57.822 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:12:58.156 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:12:58.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:57 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:12:59.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:12:58 smithi032 ceph-mon[39730]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:59.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:12:58 smithi086 ceph-mon[47925]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:12:59.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.845+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/3118212180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fe830 msgr2=0x7fbb58106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:59.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.845+0000 7fbb5f4a4700 1 --2- 172.21.15.32:0/3118212180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fe830 0x7fbb58106f00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbb48004940 tx=0x7fbb48009e30 comp rx=0 tx=0).stop 2024-04-06T14:12:59.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.845+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/3118212180 shutdown_connections 2024-04-06T14:12:59.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.845+0000 7fbb5f4a4700 1 --2- 172.21.15.32:0/3118212180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fe830 0x7fbb58106f00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.845+0000 7fbb5f4a4700 1 --2- 172.21.15.32:0/3118212180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb580fde90 0x7fbb580fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.845+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/3118212180 >> 172.21.15.32:0/3118212180 conn(0x7fbb58072bc0 msgr2=0x7fbb58072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.845+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/3118212180 shutdown_connections 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.845+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/3118212180 wait complete. 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5f4a4700 1 Processor -- start 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5f4a4700 1 -- start start 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5f4a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fde90 0x7fbb580fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5f4a4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb580fe830 0x7fbb580fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5f4a4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580fc8b0 con 0x7fbb580fde90 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5f4a4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580fc9f0 con 0x7fbb580fe830 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5d240700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fde90 0x7fbb580fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5d240700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fde90 0x7fbb580fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57434/0 (socket says 172.21.15.32:57434) 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5d240700 1 -- 172.21.15.32:0/4024627203 learned_addr learned my addr 172.21.15.32:0/4024627203 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.846+0000 7fbb5ca3f700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb580fe830 0x7fbb580fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.847+0000 7fbb5d240700 1 -- 172.21.15.32:0/4024627203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb580fe830 msgr2=0x7fbb580fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:12:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.847+0000 7fbb5d240700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb580fe830 0x7fbb580fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:12:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.847+0000 7fbb5d240700 1 -- 172.21.15.32:0/4024627203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb48005040 con 0x7fbb580fde90 2024-04-06T14:12:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.847+0000 7fbb5d240700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fde90 0x7fbb580fbd50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbb5400b790 tx=0x7fbb54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.847+0000 7fbb4dffb700 1 -- 172.21.15.32:0/4024627203 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb5400f6f0 con 0x7fbb580fde90 2024-04-06T14:12:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.847+0000 7fbb4dffb700 1 -- 172.21.15.32:0/4024627203 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb5400fd00 con 0x7fbb580fde90 2024-04-06T14:12:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.847+0000 7fbb4dffb700 1 -- 172.21.15.32:0/4024627203 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb540103a0 con 0x7fbb580fde90 2024-04-06T14:12:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.847+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb58119db0 con 0x7fbb580fde90 2024-04-06T14:12:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.847+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5811a240 con 0x7fbb580fde90 2024-04-06T14:12:59.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.848+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb58065e30 con 0x7fbb580fde90 2024-04-06T14:12:59.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.848+0000 7fbb4dffb700 1 -- 172.21.15.32:0/4024627203 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fbb5400f850 con 0x7fbb580fde90 2024-04-06T14:12:59.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.849+0000 7fbb4dffb700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbb440692a0 0x7fbb4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:12:59.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.849+0000 7fbb4dffb700 1 -- 172.21.15.32:0/4024627203 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbb54086c20 con 0x7fbb580fde90 2024-04-06T14:12:59.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.849+0000 7fbb5ca3f700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbb440692a0 0x7fbb4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:12:59.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.849+0000 7fbb5ca3f700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbb440692a0 0x7fbb4406b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800b7e0 tx=0x7fbb48009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:12:59.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:12:59.853+0000 7fbb4dffb700 1 -- 172.21.15.32:0/4024627203 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbb54053290 con 0x7fbb580fde90 2024-04-06T14:13:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.047+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb5805f5f0 con 0x7fbb580fde90 2024-04-06T14:13:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.048+0000 7fbb4dffb700 1 -- 172.21.15.32:0/4024627203 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11043 (secure 0 0 0) 0x7fbb540568b0 con 0x7fbb580fde90 2024-04-06T14:13:00.048 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:13:00.048 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":41,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","created":"2024-04-06T14:06:48.330591+0000","modified":"2024-04-06T14:12:37.750341+0000","last_up_change":"2024-04-06T14:12:36.744888+0000","last_in_change":"2024-04-06T14:12:17.748510+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T14:11:20.110423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6803","nonce":1233836109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6805","nonce":1233836109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6809","nonce":1233836109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6807","nonce":1233836109}]},"public_addr":"172.21.15.32:6803/1233836109","cluster_addr":"172.21.15.32:6805/1233836109","heartbeat_back_addr":"172.21.15.32:6809/1233836109","heartbeat_front_addr":"172.21.15.32:6807/1233836109","state":["exists","up"]},{"osd":1,"uuid":"9185a389-ef69-41bc-9e10-46c6a2e02a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6811","nonce":104495793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6813","nonce":104495793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6817","nonce":104495793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6815","nonce":104495793}]},"public_addr":"172.21.15.32:6811/104495793","cluster_addr":"172.21.15.32:6813/104495793","heartbeat_back_addr":"172.21.15.32:6817/104495793","heartbeat_front_addr":"172.21.15.32:6815/104495793","state":["exists","up"]},{"osd":2,"uuid":"af4793e1-b5bb-4919-838e-67703c83296a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6819","nonce":2571669324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6821","nonce":2571669324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6825","nonce":2571669324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6823","nonce":2571669324}]},"public_addr":"172.21.15.32:6819/2571669324","cluster_addr":"172.21.15.32:6821/2571669324","heartbeat_back_addr":"172.21.15.32:6825/2571669324","heartbeat_front_addr":"172.21.15.32:6823/2571669324","state":["exists","up"]},{"osd":3,"uuid":"70c8b884-5214-44f2-9a31-55556a11c20a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6801","nonce":3912040245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6803","nonce":3912040245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6807","nonce":3912040245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6805","nonce":3912040245}]},"public_addr":"172.21.15.86:6801/3912040245","cluster_addr":"172.21.15.86:6803/3912040245","heartbeat_back_addr":"172.21.15.86:6807/3912040245","heartbeat_front_addr":"172.21.15.86:6805/3912040245","state":["exists","up"]},{"osd":4,"uuid":"91f548ac-4f42-4083-a027-295b829646a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6809","nonce":3794387444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6811","nonce":3794387444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6815","nonce":3794387444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6813","nonce":3794387444}]},"public_addr":"172.21.15.86:6809/3794387444","cluster_addr":"172.21.15.86:6811/3794387444","heartbeat_back_addr":"172.21.15.86:6815/3794387444","heartbeat_front_addr":"172.21.15.86:6813/3794387444","state":["exists","up"]},{"osd":5,"uuid":"7037f613-ffd9-4b0e-84a1-0c04d4181ff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6817","nonce":2011390682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6819","nonce":2011390682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6823","nonce":2011390682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6821","nonce":2011390682}]},"public_addr":"172.21.15.86:6817/2011390682","cluster_addr":"172.21.15.86:6819/2011390682","heartbeat_back_addr":"172.21.15.86:6823/2011390682","heartbeat_front_addr":"172.21.15.86:6821/2011390682","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:13.684938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:44.549488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:15.893369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:41.588409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:08.612078+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:35.380707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6800/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/711512148":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/3330131734":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/2227455350":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/1187866165":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6800/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/1877817963":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6801/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:6800/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/68274703":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/3548398153":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/935909372":"2024-04-07T14:07:51.523157+0000","172.21.15.32:6801/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6801/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/3250004174":"2024-04-07T14:07:17.560285+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbb440692a0 msgr2=0x7fbb4406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbb440692a0 0x7fbb4406b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800b7e0 tx=0x7fbb48009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fde90 msgr2=0x7fbb580fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fde90 0x7fbb580fbd50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbb5400b790 tx=0x7fbb54009250 comp rx=0 tx=0).stop 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 shutdown_connections 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbb440692a0 0x7fbb4406b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb580fde90 0x7fbb580fbd50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 --2- 172.21.15.32:0/4024627203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb580fe830 0x7fbb580fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 >> 172.21.15.32:0/4024627203 conn(0x7fbb58072bc0 msgr2=0x7fbb581037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 shutdown_connections 2024-04-06T14:13:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:00.050+0000 7fbb5f4a4700 1 -- 172.21.15.32:0/4024627203 wait complete. 2024-04-06T14:13:00.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-06T14:13:00.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-06T14:13:00.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-06T14:13:00.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-06T14:13:00.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-06T14:13:00.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-06T14:13:01.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:00 smithi032 ceph-mon[39730]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:01.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:00 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/4024627203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:13:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:01.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:00 smithi086 ceph-mon[47925]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:01.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:00 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/4024627203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:13:02.176 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:03.147 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:02 smithi032 ceph-mon[39730]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:03.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:02 smithi086 ceph-mon[47925]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:05.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:04 smithi032 ceph-mon[39730]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:05.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:04 smithi086 ceph-mon[47925]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 -- 172.21.15.32:0/2423077379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 msgr2=0x7f730c0706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2423077379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 0x7f730c0706b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f72fc004940 tx=0x7f72fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 -- 172.21.15.32:0/2423077379 shutdown_connections 2024-04-06T14:13:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2423077379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c070bf0 0x7f730c1091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2423077379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 0x7f730c0706b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 -- 172.21.15.32:0/2423077379 >> 172.21.15.32:0/2423077379 conn(0x7f730c0f89a0 msgr2=0x7f730c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 -- 172.21.15.32:0/2423077379 shutdown_connections 2024-04-06T14:13:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 -- 172.21.15.32:0/2423077379 wait complete. 2024-04-06T14:13:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 Processor -- start 2024-04-06T14:13:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.498+0000 7f7311e0c700 1 -- start start 2024-04-06T14:13:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f7311e0c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 0x7f730c115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f7311e0c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c070bf0 0x7f730c116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f7311e0c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c116870 con 0x7f730c0702e0 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f7311e0c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c10fc40 con 0x7f730c070bf0 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f730b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 0x7f730c115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f730b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 0x7f730c115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58826/0 (socket says 172.21.15.32:58826) 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f730b7fe700 1 -- 172.21.15.32:0/2376702663 learned_addr learned my addr 172.21.15.32:0/2376702663 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f730affd700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c070bf0 0x7f730c116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f730b7fe700 1 -- 172.21.15.32:0/2376702663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c070bf0 msgr2=0x7f730c116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f730b7fe700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c070bf0 0x7f730c116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f730b7fe700 1 -- 172.21.15.32:0/2376702663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc005040 con 0x7f730c0702e0 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f730b7fe700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 0x7f730c115ce0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00b3b0 tx=0x7f72fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.499+0000 7f72f3fff700 1 -- 172.21.15.32:0/2376702663 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72fc00e040 con 0x7f730c0702e0 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.500+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c10fec0 con 0x7f730c0702e0 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.500+0000 7f72f3fff700 1 -- 172.21.15.32:0/2376702663 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72fc013420 con 0x7f730c0702e0 2024-04-06T14:13:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.500+0000 7f72f3fff700 1 -- 172.21.15.32:0/2376702663 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72fc01c610 con 0x7f730c0702e0 2024-04-06T14:13:05.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.500+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c110400 con 0x7f730c0702e0 2024-04-06T14:13:05.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.500+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f730c1108e0 con 0x7f730c0702e0 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.501+0000 7f72f3fff700 1 -- 172.21.15.32:0/2376702663 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f72fc013a00 con 0x7f730c0702e0 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.501+0000 7f72f3fff700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72f40692a0 0x7f72f406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.502+0000 7f72f3fff700 1 -- 172.21.15.32:0/2376702663 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f72fc087ff0 con 0x7f730c0702e0 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.502+0000 7f730affd700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72f40692a0 0x7f72f406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.502+0000 7f72f3fff700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] conn(0x7f72f406ef70 0x7f72f4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.502+0000 7f72f3fff700 1 -- 172.21.15.32:0/2376702663 --> [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72f4071a30 con 0x7f72f406ef70 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.502+0000 7f72f3fff700 1 -- 172.21.15.32:0/2376702663 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72fc08a5e0 con 0x7f730c0702e0 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.502+0000 7f730bfff700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] conn(0x7f72f406ef70 0x7f72f4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.502+0000 7f730affd700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72f40692a0 0x7f72f406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f730c111580 tx=0x7f730000b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.502+0000 7f730bfff700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] conn(0x7f72f406ef70 0x7f72f4071380 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:05.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.503+0000 7f72f3fff700 1 -- 172.21.15.32:0/2376702663 <== osd.1 v2:172.21.15.32:6810/104495793 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f72f4071a30 con 0x7f72f406ef70 2024-04-06T14:13:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.518+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 --> [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f730c110aa0 con 0x7f72f406ef70 2024-04-06T14:13:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.518+0000 7f72f3fff700 1 -- 172.21.15.32:0/2376702663 <== osd.1 v2:172.21.15.32:6810/104495793 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f730c110aa0 con 0x7f72f406ef70 2024-04-06T14:13:05.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.518+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] conn(0x7f72f406ef70 msgr2=0x7f72f4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:05.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.518+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] conn(0x7f72f406ef70 0x7f72f4071380 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:05.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.518+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72f40692a0 msgr2=0x7f72f406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:05.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.518+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72f40692a0 0x7f72f406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f730c111580 tx=0x7f730000b040 comp rx=0 tx=0).stop 2024-04-06T14:13:05.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.518+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 msgr2=0x7f730c115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:05.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.518+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 0x7f730c115ce0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00b3b0 tx=0x7f72fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:05.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.519+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 shutdown_connections 2024-04-06T14:13:05.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.519+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6810/104495793,v1:172.21.15.32:6811/104495793] conn(0x7f72f406ef70 0x7f72f4071380 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:05.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.519+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f72f40692a0 0x7f72f406b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:05.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.519+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730c0702e0 0x7f730c115ce0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:05.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.519+0000 7f7311e0c700 1 --2- 172.21.15.32:0/2376702663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c070bf0 0x7f730c116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:05.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.519+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 >> 172.21.15.32:0/2376702663 conn(0x7f730c0f89a0 msgr2=0x7f730c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:05.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.519+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 shutdown_connections 2024-04-06T14:13:05.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:05.519+0000 7f7311e0c700 1 -- 172.21.15.32:0/2376702663 wait complete. 2024-04-06T14:13:05.585 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:07.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:06 smithi032 ceph-mon[39730]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:07.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:06 smithi086 ceph-mon[47925]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:07.885 INFO:teuthology.orchestra.run.smithi032.stdout:60129542174 2024-04-06T14:13:07.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.529+0000 7fb65b543700 1 -- 172.21.15.32:0/1649665475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 msgr2=0x7fb6540fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.529+0000 7fb65b543700 1 --2- 172.21.15.32:0/1649665475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 0x7fb6540fe1e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb650004940 tx=0x7fb650009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 -- 172.21.15.32:0/1649665475 shutdown_connections 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 --2- 172.21.15.32:0/1649665475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6540fe7b0 0x7fb654106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 --2- 172.21.15.32:0/1649665475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 0x7fb6540fe1e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 -- 172.21.15.32:0/1649665475 >> 172.21.15.32:0/1649665475 conn(0x7fb6540f7880 msgr2=0x7fb6540f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 -- 172.21.15.32:0/1649665475 shutdown_connections 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 -- 172.21.15.32:0/1649665475 wait complete. 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 Processor -- start 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 -- start start 2024-04-06T14:13:08.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 0x7fb654113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6540fe7b0 0x7fb654114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb654114970 con 0x7fb6540fde10 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.530+0000 7fb65b543700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65410dd40 con 0x7fb6540fe7b0 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb6592df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 0x7fb654113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb6592df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 0x7fb654113de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58858/0 (socket says 172.21.15.32:58858) 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb6592df700 1 -- 172.21.15.32:0/1770279541 learned_addr learned my addr 172.21.15.32:0/1770279541 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb658ade700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6540fe7b0 0x7fb654114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb6592df700 1 -- 172.21.15.32:0/1770279541 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6540fe7b0 msgr2=0x7fb654114320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb6592df700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6540fe7b0 0x7fb654114320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb6592df700 1 -- 172.21.15.32:0/1770279541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb650005040 con 0x7fb6540fde10 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb6592df700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 0x7fb654113de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb650009380 tx=0x7fb650009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb649ffb700 1 -- 172.21.15.32:0/1770279541 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb65000e040 con 0x7fb6540fde10 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb649ffb700 1 -- 172.21.15.32:0/1770279541 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb650004290 con 0x7fb6540fde10 2024-04-06T14:13:08.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65410dfc0 con 0x7fb6540fde10 2024-04-06T14:13:08.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb649ffb700 1 -- 172.21.15.32:0/1770279541 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb65001c3f0 con 0x7fb6540fde10 2024-04-06T14:13:08.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.531+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65410e500 con 0x7fb6540fde10 2024-04-06T14:13:08.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.532+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb654066770 con 0x7fb6540fde10 2024-04-06T14:13:08.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.533+0000 7fb649ffb700 1 -- 172.21.15.32:0/1770279541 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb650013790 con 0x7fb6540fde10 2024-04-06T14:13:08.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.533+0000 7fb649ffb700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb6400692a0 0x7fb64006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:08.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.533+0000 7fb649ffb700 1 -- 172.21.15.32:0/1770279541 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb650088140 con 0x7fb6540fde10 2024-04-06T14:13:08.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.533+0000 7fb658ade700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb6400692a0 0x7fb64006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:08.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.533+0000 7fb649ffb700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] conn(0x7fb64006ef70 0x7fb640071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:08.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.533+0000 7fb649ffb700 1 -- 172.21.15.32:0/1770279541 --> [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb640071a30 con 0x7fb64006ef70 2024-04-06T14:13:08.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.533+0000 7fb649ffb700 1 -- 172.21.15.32:0/1770279541 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb65008a730 con 0x7fb6540fde10 2024-04-06T14:13:08.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.533+0000 7fb659ae0700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] conn(0x7fb64006ef70 0x7fb640071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:08.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.533+0000 7fb658ade700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb6400692a0 0x7fb64006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb644009fd0 tx=0x7fb644009f20 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:08.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.534+0000 7fb659ae0700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] conn(0x7fb64006ef70 0x7fb640071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:08.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.534+0000 7fb649ffb700 1 -- 172.21.15.32:0/1770279541 <== osd.0 v2:172.21.15.32:6802/1233836109 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb640071a30 con 0x7fb64006ef70 2024-04-06T14:13:08.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.550+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 --> [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb65410e150 con 0x7fb64006ef70 2024-04-06T14:13:08.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.550+0000 7fb649ffb700 1 -- 172.21.15.32:0/1770279541 <== osd.0 v2:172.21.15.32:6802/1233836109 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb65410e150 con 0x7fb64006ef70 2024-04-06T14:13:08.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.550+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] conn(0x7fb64006ef70 msgr2=0x7fb640071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:08.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.550+0000 7fb65b543700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] conn(0x7fb64006ef70 0x7fb640071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:08.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb6400692a0 msgr2=0x7fb64006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:08.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb6400692a0 0x7fb64006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb644009fd0 tx=0x7fb644009f20 comp rx=0 tx=0).stop 2024-04-06T14:13:08.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 msgr2=0x7fb654113de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:08.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 0x7fb654113de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb650009380 tx=0x7fb650009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:08.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 shutdown_connections 2024-04-06T14:13:08.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6802/1233836109,v1:172.21.15.32:6803/1233836109] conn(0x7fb64006ef70 0x7fb640071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:08.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb6400692a0 0x7fb64006b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:08.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6540fde10 0x7fb654113de0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:08.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 --2- 172.21.15.32:0/1770279541 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6540fe7b0 0x7fb654114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:08.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 >> 172.21.15.32:0/1770279541 conn(0x7fb6540f7880 msgr2=0x7fb6540f9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:08.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 shutdown_connections 2024-04-06T14:13:08.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:08.551+0000 7fb65b543700 1 -- 172.21.15.32:0/1770279541 wait complete. 2024-04-06T14:13:09.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:08 smithi032 ceph-mon[39730]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:09.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:08 smithi086 ceph-mon[47925]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:10.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:10 smithi032 ceph-mon[39730]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.925+0000 7f5e3ea87700 1 -- 172.21.15.32:0/1656162231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38102710 msgr2=0x7f5e380fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.925+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/1656162231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38102710 0x7f5e380fbd80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5e20004940 tx=0x7f5e20009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.925+0000 7f5e3ea87700 1 -- 172.21.15.32:0/1656162231 shutdown_connections 2024-04-06T14:13:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.925+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/1656162231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38102710 0x7f5e380fbd80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.925+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/1656162231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e38101d70 0x7f5e38102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.925+0000 7f5e3ea87700 1 -- 172.21.15.32:0/1656162231 >> 172.21.15.32:0/1656162231 conn(0x7f5e38072bc0 msgr2=0x7f5e38072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.925+0000 7f5e3ea87700 1 -- 172.21.15.32:0/1656162231 shutdown_connections 2024-04-06T14:13:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.925+0000 7f5e3ea87700 1 -- 172.21.15.32:0/1656162231 wait complete. 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3ea87700 1 Processor -- start 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3ea87700 1 -- start start 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3ea87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38101d70 0x7f5e38113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3ea87700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e38102710 0x7f5e38113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3ea87700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e38114590 con 0x7f5e38101d70 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3ea87700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3810d980 con 0x7f5e38102710 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3c823700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38101d70 0x7f5e38113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3c823700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38101d70 0x7f5e38113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58874/0 (socket says 172.21.15.32:58874) 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3c823700 1 -- 172.21.15.32:0/2552439943 learned_addr learned my addr 172.21.15.32:0/2552439943 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e37fff700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e38102710 0x7f5e38113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3c823700 1 -- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e38102710 msgr2=0x7f5e38113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3c823700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e38102710 0x7f5e38113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.926+0000 7f5e3c823700 1 -- 172.21.15.32:0/2552439943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e20005040 con 0x7f5e38101d70 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.927+0000 7f5e3c823700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38101d70 0x7f5e38113a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800eb40 tx=0x7f5e2800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.927+0000 7f5e357fa700 1 -- 172.21.15.32:0/2552439943 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e28007cc0 con 0x7f5e38101d70 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.927+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3810dc00 con 0x7f5e38101d70 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.927+0000 7f5e357fa700 1 -- 172.21.15.32:0/2552439943 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e2800f040 con 0x7f5e38101d70 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.927+0000 7f5e357fa700 1 -- 172.21.15.32:0/2552439943 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e28009570 con 0x7f5e38101d70 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.927+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3810e190 con 0x7f5e38101d70 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.927+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e3810e670 con 0x7f5e38101d70 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.928+0000 7f5e357fa700 1 -- 172.21.15.32:0/2552439943 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5e280096d0 con 0x7f5e38101d70 2024-04-06T14:13:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.929+0000 7f5e357fa700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5e240692a0 0x7f5e2406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.929+0000 7f5e357fa700 1 -- 172.21.15.32:0/2552439943 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5e28086cc0 con 0x7f5e38101d70 2024-04-06T14:13:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.929+0000 7f5e37fff700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5e240692a0 0x7f5e2406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.929+0000 7f5e357fa700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] conn(0x7f5e2406ef70 0x7f5e24071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.929+0000 7f5e357fa700 1 -- 172.21.15.32:0/2552439943 --> [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e24071a30 con 0x7f5e2406ef70 2024-04-06T14:13:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.929+0000 7f5e357fa700 1 -- 172.21.15.32:0/2552439943 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e280870a0 con 0x7f5e38101d70 2024-04-06T14:13:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.929+0000 7f5e37fff700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5e240692a0 0x7f5e2406b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5e3810ec60 tx=0x7f5e20009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.929+0000 7f5e3d024700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] conn(0x7f5e2406ef70 0x7f5e24071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.930+0000 7f5e3d024700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] conn(0x7f5e2406ef70 0x7f5e24071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.930+0000 7f5e357fa700 1 -- 172.21.15.32:0/2552439943 <== osd.5 v2:172.21.15.86:6816/2011390682 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5e24071a30 con 0x7f5e2406ef70 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.945+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 --> [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e38105fe0 con 0x7f5e2406ef70 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e357fa700 1 -- 172.21.15.32:0/2552439943 <== osd.5 v2:172.21.15.86:6816/2011390682 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5e38105fe0 con 0x7f5e2406ef70 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] conn(0x7f5e2406ef70 msgr2=0x7f5e24071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] conn(0x7f5e2406ef70 0x7f5e24071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5e240692a0 msgr2=0x7f5e2406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5e240692a0 0x7f5e2406b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5e3810ec60 tx=0x7f5e20009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38101d70 msgr2=0x7f5e38113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38101d70 0x7f5e38113a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800eb40 tx=0x7f5e2800b040 comp rx=0 tx=0).stop 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 shutdown_connections 2024-04-06T14:13:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:6816/2011390682,v1:172.21.15.86:6817/2011390682] conn(0x7f5e2406ef70 0x7f5e24071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5e240692a0 0x7f5e2406b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e38101d70 0x7f5e38113a00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 --2- 172.21.15.32:0/2552439943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e38102710 0x7f5e38113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 >> 172.21.15.32:0/2552439943 conn(0x7f5e38072bc0 msgr2=0x7f5e38065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 shutdown_connections 2024-04-06T14:13:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:10.946+0000 7f5e3ea87700 1 -- 172.21.15.32:0/2552439943 wait complete. 2024-04-06T14:13:11.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:10 smithi086 ceph-mon[47925]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:11.862 INFO:teuthology.orchestra.run.smithi032.stdout:38654705701 2024-04-06T14:13:11.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-06T14:13:12.491 INFO:teuthology.orchestra.run.smithi032.stdout:171798691849 2024-04-06T14:13:12.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-06T14:13:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.727+0000 7f80b1b8b700 1 -- 172.21.15.32:0/1138759894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 msgr2=0x7f80ac0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.727+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/1138759894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 0x7f80ac0fbd80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f80a0004940 tx=0x7f80a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:12.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.727+0000 7f80b1b8b700 1 -- 172.21.15.32:0/1138759894 shutdown_connections 2024-04-06T14:13:12.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.727+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/1138759894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 0x7f80ac0fbd80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.727+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/1138759894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac101d70 0x7f80ac102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.727+0000 7f80b1b8b700 1 -- 172.21.15.32:0/1138759894 >> 172.21.15.32:0/1138759894 conn(0x7f80ac072bc0 msgr2=0x7f80ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:12.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.727+0000 7f80b1b8b700 1 -- 172.21.15.32:0/1138759894 shutdown_connections 2024-04-06T14:13:12.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.727+0000 7f80b1b8b700 1 -- 172.21.15.32:0/1138759894 wait complete. 2024-04-06T14:13:12.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.728+0000 7f80b1b8b700 1 Processor -- start 2024-04-06T14:13:12.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.728+0000 7f80b1b8b700 1 -- start start 2024-04-06T14:13:12.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.728+0000 7f80b1b8b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac101d70 0x7f80ac102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.728+0000 7f80b1b8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 0x7f80ac103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.728+0000 7f80b1b8b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac1036a0 con 0x7f80ac102710 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.728+0000 7f80b1b8b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac1037e0 con 0x7f80ac101d70 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 0x7f80ac103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 0x7f80ac103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58888/0 (socket says 172.21.15.32:58888) 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80aaffd700 1 -- 172.21.15.32:0/3960747096 learned_addr learned my addr 172.21.15.32:0/3960747096 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80ab7fe700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac101d70 0x7f80ac102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80aaffd700 1 -- 172.21.15.32:0/3960747096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac101d70 msgr2=0x7f80ac102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80aaffd700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac101d70 0x7f80ac102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80aaffd700 1 -- 172.21.15.32:0/3960747096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a0005040 con 0x7f80ac102710 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80aaffd700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 0x7f80ac103110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009fd0 tx=0x7f80a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f8093fff700 1 -- 172.21.15.32:0/3960747096 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80a000e030 con 0x7f80ac102710 2024-04-06T14:13:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f8093fff700 1 -- 172.21.15.32:0/3960747096 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80a00093b0 con 0x7f80ac102710 2024-04-06T14:13:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ac103a30 con 0x7f80ac102710 2024-04-06T14:13:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f8093fff700 1 -- 172.21.15.32:0/3960747096 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80a001cc80 con 0x7f80ac102710 2024-04-06T14:13:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.729+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ac11a130 con 0x7f80ac102710 2024-04-06T14:13:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.730+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f80ac105fe0 con 0x7f80ac102710 2024-04-06T14:13:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.731+0000 7f8093fff700 1 -- 172.21.15.32:0/3960747096 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f80a001c400 con 0x7f80ac102710 2024-04-06T14:13:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.732+0000 7f8093fff700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f80940692b0 0x7f809406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.732+0000 7f8093fff700 1 -- 172.21.15.32:0/3960747096 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f80a0088ec0 con 0x7f80ac102710 2024-04-06T14:13:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.732+0000 7f80ab7fe700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f80940692b0 0x7f809406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.732+0000 7f8093fff700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] conn(0x7f809406ef80 0x7f8094071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.732+0000 7f8093fff700 1 -- 172.21.15.32:0/3960747096 --> [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8094071a40 con 0x7f809406ef80 2024-04-06T14:13:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.732+0000 7f8093fff700 1 -- 172.21.15.32:0/3960747096 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80a00892a0 con 0x7f80ac102710 2024-04-06T14:13:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.732+0000 7f80abfff700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] conn(0x7f809406ef80 0x7f8094071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.732+0000 7f80ab7fe700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f80940692b0 0x7f809406b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f809c009b80 tx=0x7f809c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.732+0000 7f80abfff700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] conn(0x7f809406ef80 0x7f8094071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.733+0000 7f8093fff700 1 -- 172.21.15.32:0/3960747096 <== osd.2 v2:172.21.15.32:6818/2571669324 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f8094071a40 con 0x7f809406ef80 2024-04-06T14:13:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.753+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 --> [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f80ac119d00 con 0x7f809406ef80 2024-04-06T14:13:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.753+0000 7f8093fff700 1 -- 172.21.15.32:0/3960747096 <== osd.2 v2:172.21.15.32:6818/2571669324 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f80ac119d00 con 0x7f809406ef80 2024-04-06T14:13:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] conn(0x7f809406ef80 msgr2=0x7f8094071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] conn(0x7f809406ef80 0x7f8094071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f80940692b0 msgr2=0x7f809406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f80940692b0 0x7f809406b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f809c009b80 tx=0x7f809c00b040 comp rx=0 tx=0).stop 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 msgr2=0x7f80ac103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 0x7f80ac103110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009fd0 tx=0x7f80a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 shutdown_connections 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f80940692b0 0x7f809406b760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac101d70 0x7f80ac102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:6818/2571669324,v1:172.21.15.32:6819/2571669324] conn(0x7f809406ef80 0x7f8094071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 --2- 172.21.15.32:0/3960747096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac102710 0x7f80ac103110 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 >> 172.21.15.32:0/3960747096 conn(0x7f80ac072bc0 msgr2=0x7f80ac065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 shutdown_connections 2024-04-06T14:13:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.754+0000 7f80b1b8b700 1 -- 172.21.15.32:0/3960747096 wait complete. 2024-04-06T14:13:12.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.971+0000 7fb10af60700 1 -- 172.21.15.32:0/3703621430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 msgr2=0x7fb104106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.971+0000 7fb10af60700 1 --2- 172.21.15.32:0/3703621430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 0x7fb104106f00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0004940 tx=0x7fb0f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:12.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.972+0000 7fb10af60700 1 -- 172.21.15.32:0/3703621430 shutdown_connections 2024-04-06T14:13:12.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.972+0000 7fb10af60700 1 --2- 172.21.15.32:0/3703621430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 0x7fb104106f00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.972+0000 7fb10af60700 1 --2- 172.21.15.32:0/3703621430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1040fde90 0x7fb1040fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.972+0000 7fb10af60700 1 -- 172.21.15.32:0/3703621430 >> 172.21.15.32:0/3703621430 conn(0x7fb104072bc0 msgr2=0x7fb104072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:12.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.972+0000 7fb10af60700 1 -- 172.21.15.32:0/3703621430 shutdown_connections 2024-04-06T14:13:12.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.972+0000 7fb10af60700 1 -- 172.21.15.32:0/3703621430 wait complete. 2024-04-06T14:13:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.972+0000 7fb10af60700 1 Processor -- start 2024-04-06T14:13:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.972+0000 7fb10af60700 1 -- start start 2024-04-06T14:13:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb10af60700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1040fde90 0x7fb104113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb10af60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 0x7fb104114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb10af60700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb104114670 con 0x7fb1040fe830 2024-04-06T14:13:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb10af60700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10410da60 con 0x7fb1040fde90 2024-04-06T14:13:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb103fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 0x7fb104114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb103fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 0x7fb104114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58904/0 (socket says 172.21.15.32:58904) 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb108cfc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1040fde90 0x7fb104113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb103fff700 1 -- 172.21.15.32:0/2941940441 learned_addr learned my addr 172.21.15.32:0/2941940441 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb103fff700 1 -- 172.21.15.32:0/2941940441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1040fde90 msgr2=0x7fb104113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb103fff700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1040fde90 0x7fb104113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb103fff700 1 -- 172.21.15.32:0/2941940441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f0005040 con 0x7fb1040fe830 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb103fff700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 0x7fb104114020 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000b3b0 tx=0x7fb0f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb1017fa700 1 -- 172.21.15.32:0/2941940441 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0f000e040 con 0x7fb1040fe830 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10410dce0 con 0x7fb1040fe830 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb1017fa700 1 -- 172.21.15.32:0/2941940441 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0f0013420 con 0x7fb1040fe830 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.973+0000 7fb1017fa700 1 -- 172.21.15.32:0/2941940441 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0f001c5f0 con 0x7fb1040fe830 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.974+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10410e220 con 0x7fb1040fe830 2024-04-06T14:13:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.974+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb10410e740 con 0x7fb1040fe830 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.975+0000 7fb1017fa700 1 -- 172.21.15.32:0/2941940441 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb0f0013590 con 0x7fb1040fe830 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.975+0000 7fb1017fa700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0f40692a0 0x7fb0f406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.975+0000 7fb1017fa700 1 -- 172.21.15.32:0/2941940441 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb0f0088140 con 0x7fb1040fe830 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.975+0000 7fb108cfc700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0f40692a0 0x7fb0f406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.975+0000 7fb1017fa700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] conn(0x7fb0f406ef70 0x7fb0f4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.975+0000 7fb1017fa700 1 -- 172.21.15.32:0/2941940441 --> [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0f4071a30 con 0x7fb0f406ef70 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.975+0000 7fb1017fa700 1 -- 172.21.15.32:0/2941940441 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0f008a790 con 0x7fb1040fe830 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.976+0000 7fb1094fd700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] conn(0x7fb0f406ef70 0x7fb0f4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.976+0000 7fb108cfc700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0f40692a0 0x7fb0f406b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8004f40 tx=0x7fb0f800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.976+0000 7fb1094fd700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] conn(0x7fb0f406ef70 0x7fb0f4071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.977+0000 7fb1017fa700 1 -- 172.21.15.32:0/2941940441 <== osd.3 v2:172.21.15.86:6800/3912040245 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb0f4071a30 con 0x7fb0f406ef70 2024-04-06T14:13:12.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.992+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 --> [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb104106690 con 0x7fb0f406ef70 2024-04-06T14:13:12.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.992+0000 7fb1017fa700 1 -- 172.21.15.32:0/2941940441 <== osd.3 v2:172.21.15.86:6800/3912040245 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb104106690 con 0x7fb0f406ef70 2024-04-06T14:13:12.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 >> [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] conn(0x7fb0f406ef70 msgr2=0x7fb0f4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] conn(0x7fb0f406ef70 0x7fb0f4071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0f40692a0 msgr2=0x7fb0f406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0f40692a0 0x7fb0f406b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8004f40 tx=0x7fb0f800b040 comp rx=0 tx=0).stop 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 msgr2=0x7fb104114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 0x7fb104114020 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000b3b0 tx=0x7fb0f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 shutdown_connections 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fb0f40692a0 0x7fb0f406b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1040fde90 0x7fb104113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.86:6800/3912040245,v1:172.21.15.86:6801/3912040245] conn(0x7fb0f406ef70 0x7fb0f4071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 --2- 172.21.15.32:0/2941940441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1040fe830 0x7fb104114020 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 >> 172.21.15.32:0/2941940441 conn(0x7fb104072bc0 msgr2=0x7fb104103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 shutdown_connections 2024-04-06T14:13:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:12.993+0000 7fb10af60700 1 -- 172.21.15.32:0/2941940441 wait complete. 2024-04-06T14:13:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:12 smithi032 ceph-mon[39730]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:13:13.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:12 smithi086 ceph-mon[47925]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:13.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:13:13.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.612+0000 7efcf91f2700 1 -- 172.21.15.32:0/4157092774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fe830 msgr2=0x7efcf4106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:13.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.612+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4157092774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fe830 0x7efcf4106f00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efce4004940 tx=0x7efce4009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:13.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.613+0000 7efcf91f2700 1 -- 172.21.15.32:0/4157092774 shutdown_connections 2024-04-06T14:13:13.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.613+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4157092774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fe830 0x7efcf4106f00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:13.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.613+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4157092774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40fde90 0x7efcf40fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:13.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.613+0000 7efcf91f2700 1 -- 172.21.15.32:0/4157092774 >> 172.21.15.32:0/4157092774 conn(0x7efcf4072bc0 msgr2=0x7efcf4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:13.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.613+0000 7efcf91f2700 1 -- 172.21.15.32:0/4157092774 shutdown_connections 2024-04-06T14:13:13.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.613+0000 7efcf91f2700 1 -- 172.21.15.32:0/4157092774 wait complete. 2024-04-06T14:13:13.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.613+0000 7efcf91f2700 1 Processor -- start 2024-04-06T14:13:13.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.613+0000 7efcf91f2700 1 -- start start 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf91f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fde90 0x7efcf40fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf91f2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40fe830 0x7efcf40fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf91f2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf40fc820 con 0x7efcf40fde90 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf91f2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf40fc960 con 0x7efcf40fe830 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fde90 0x7efcf40fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fde90 0x7efcf40fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58916/0 (socket says 172.21.15.32:58916) 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf2d9d700 1 -- 172.21.15.32:0/4034182397 learned_addr learned my addr 172.21.15.32:0/4034182397 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf259c700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40fe830 0x7efcf40fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf2d9d700 1 -- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40fe830 msgr2=0x7efcf40fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf2d9d700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40fe830 0x7efcf40fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf2d9d700 1 -- 172.21.15.32:0/4034182397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce4005040 con 0x7efcf40fde90 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf2d9d700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fde90 0x7efcf40fbd50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efce8002a50 tx=0x7efce80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcdb7fe700 1 -- 172.21.15.32:0/4034182397 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce8007cc0 con 0x7efcf40fde90 2024-04-06T14:13:13.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf4119de0 con 0x7efcf40fde90 2024-04-06T14:13:13.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.614+0000 7efcdb7fe700 1 -- 172.21.15.32:0/4034182397 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efce800f040 con 0x7efcf40fde90 2024-04-06T14:13:13.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.615+0000 7efcdb7fe700 1 -- 172.21.15.32:0/4034182397 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce8009540 con 0x7efcf40fde90 2024-04-06T14:13:13.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.615+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf411a1e0 con 0x7efcf40fde90 2024-04-06T14:13:13.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.615+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efcf4065e30 con 0x7efcf40fde90 2024-04-06T14:13:13.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.617+0000 7efcdb7fe700 1 -- 172.21.15.32:0/4034182397 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7efce80096a0 con 0x7efcf40fde90 2024-04-06T14:13:13.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.617+0000 7efcdb7fe700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7efcdc0692a0 0x7efcdc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:13.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.617+0000 7efcdb7fe700 1 -- 172.21.15.32:0/4034182397 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efce8086c80 con 0x7efcf40fde90 2024-04-06T14:13:13.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.617+0000 7efcf259c700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7efcdc0692a0 0x7efcdc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.617+0000 7efcdb7fe700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] conn(0x7efcdc06ef70 0x7efcdc071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.617+0000 7efcdb7fe700 1 -- 172.21.15.32:0/4034182397 --> [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efcdc071a30 con 0x7efcdc06ef70 2024-04-06T14:13:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.617+0000 7efcdb7fe700 1 -- 172.21.15.32:0/4034182397 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7efce808c280 con 0x7efcf40fde90 2024-04-06T14:13:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.618+0000 7efcf259c700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7efcdc0692a0 0x7efcdc06b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efcf40fd690 tx=0x7efce4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.618+0000 7efcf359e700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] conn(0x7efcdc06ef70 0x7efcdc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.618+0000 7efcf359e700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] conn(0x7efcdc06ef70 0x7efcdc071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.619+0000 7efcdb7fe700 1 -- 172.21.15.32:0/4034182397 <== osd.4 v2:172.21.15.86:6808/3794387444 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7efcdc071a30 con 0x7efcdc06ef70 2024-04-06T14:13:13.632 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:13.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.637+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 --> [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efcf4106610 con 0x7efcdc06ef70 2024-04-06T14:13:13.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.637+0000 7efcdb7fe700 1 -- 172.21.15.32:0/4034182397 <== osd.4 v2:172.21.15.86:6808/3794387444 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efcf4106610 con 0x7efcdc06ef70 2024-04-06T14:13:13.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] conn(0x7efcdc06ef70 msgr2=0x7efcdc071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:13.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] conn(0x7efcdc06ef70 0x7efcdc071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:13.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7efcdc0692a0 msgr2=0x7efcdc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:13.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7efcdc0692a0 0x7efcdc06b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efcf40fd690 tx=0x7efce4009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:13.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fde90 msgr2=0x7efcf40fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:13.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fde90 0x7efcf40fbd50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efce8002a50 tx=0x7efce80046b0 comp rx=0 tx=0).stop 2024-04-06T14:13:13.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 shutdown_connections 2024-04-06T14:13:13.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:6808/3794387444,v1:172.21.15.86:6809/3794387444] conn(0x7efcdc06ef70 0x7efcdc071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:13.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7efcdc0692a0 0x7efcdc06b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:13.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcf40fde90 0x7efcf40fbd50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:13.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 --2- 172.21.15.32:0/4034182397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40fe830 0x7efcf40fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:13.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 >> 172.21.15.32:0/4034182397 conn(0x7efcf4072bc0 msgr2=0x7efcf4103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:13.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 shutdown_connections 2024-04-06T14:13:13.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:13.638+0000 7efcf91f2700 1 -- 172.21.15.32:0/4034182397 wait complete. 2024-04-06T14:13:13.659 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:14.867 INFO:teuthology.orchestra.run.smithi032.stdout:81604378649 2024-04-06T14:13:14.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-06T14:13:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:14 smithi032 ceph-mon[39730]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:15.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:14 smithi086 ceph-mon[47925]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.224+0000 7f44418d5700 1 -- 172.21.15.32:0/421186969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c101f00 msgr2=0x7f443c1022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.224+0000 7f44418d5700 1 --2- 172.21.15.32:0/421186969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c101f00 0x7f443c1022d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f442c004940 tx=0x7f442c009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.224+0000 7f44418d5700 1 -- 172.21.15.32:0/421186969 shutdown_connections 2024-04-06T14:13:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.224+0000 7f44418d5700 1 --2- 172.21.15.32:0/421186969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f443c0fc5e0 0x7f443c0fca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.224+0000 7f44418d5700 1 --2- 172.21.15.32:0/421186969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c101f00 0x7f443c1022d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.224+0000 7f44418d5700 1 -- 172.21.15.32:0/421186969 >> 172.21.15.32:0/421186969 conn(0x7f443c0714c0 msgr2=0x7f443c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.224+0000 7f44418d5700 1 -- 172.21.15.32:0/421186969 shutdown_connections 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.224+0000 7f44418d5700 1 -- 172.21.15.32:0/421186969 wait complete. 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.225+0000 7f44418d5700 1 Processor -- start 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.225+0000 7f44418d5700 1 -- start start 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.225+0000 7f44418d5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c0fc5e0 0x7f443c10c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.225+0000 7f44418d5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f443c101f00 0x7f443c10cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.225+0000 7f44418d5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0fc360 con 0x7f443c0fc5e0 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.225+0000 7f44418d5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c10d330 con 0x7f443c101f00 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.225+0000 7f443affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c0fc5e0 0x7f443c10c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.225+0000 7f443affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c0fc5e0 0x7f443c10c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60534/0 (socket says 172.21.15.32:60534) 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.225+0000 7f443affd700 1 -- 172.21.15.32:0/2322535016 learned_addr learned my addr 172.21.15.32:0/2322535016 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f443a7fc700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f443c101f00 0x7f443c10cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f443affd700 1 -- 172.21.15.32:0/2322535016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f443c101f00 msgr2=0x7f443c10cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f443affd700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f443c101f00 0x7f443c10cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f443affd700 1 -- 172.21.15.32:0/2322535016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c005040 con 0x7f443c0fc5e0 2024-04-06T14:13:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f443affd700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c0fc5e0 0x7f443c10c8b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f442c009150 tx=0x7f442c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f44337fe700 1 -- 172.21.15.32:0/2322535016 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f442c00e040 con 0x7f443c0fc5e0 2024-04-06T14:13:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f44337fe700 1 -- 172.21.15.32:0/2322535016 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f442c013420 con 0x7f443c0fc5e0 2024-04-06T14:13:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c10d5b0 con 0x7f443c0fc5e0 2024-04-06T14:13:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f44337fe700 1 -- 172.21.15.32:0/2322535016 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f442c01c630 con 0x7f443c0fc5e0 2024-04-06T14:13:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.226+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c11a0e0 con 0x7f443c0fc5e0 2024-04-06T14:13:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.227+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c104340 con 0x7f443c0fc5e0 2024-04-06T14:13:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.228+0000 7f44337fe700 1 -- 172.21.15.32:0/2322535016 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f442c013970 con 0x7f443c0fc5e0 2024-04-06T14:13:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.228+0000 7f44337fe700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f44280692a0 0x7f442806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.228+0000 7f443a7fc700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f44280692a0 0x7f442806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.228+0000 7f44337fe700 1 -- 172.21.15.32:0/2322535016 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f442c0880a0 con 0x7f443c0fc5e0 2024-04-06T14:13:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.229+0000 7f443a7fc700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f44280692a0 0x7f442806b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f443c0fbcc0 tx=0x7f44240043c0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.233+0000 7f44337fe700 1 -- 172.21.15.32:0/2322535016 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f442c0581a0 con 0x7f443c0fc5e0 2024-04-06T14:13:16.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.428+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f443c0fba70 con 0x7f443c0fc5e0 2024-04-06T14:13:16.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.429+0000 7f44337fe700 1 -- 172.21.15.32:0/2322535016 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f442c057d30 con 0x7f443c0fc5e0 2024-04-06T14:13:16.429 INFO:teuthology.orchestra.run.smithi032.stdout:60129542176 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f44280692a0 msgr2=0x7f442806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f44280692a0 0x7f442806b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f443c0fbcc0 tx=0x7f44240043c0 comp rx=0 tx=0).stop 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c0fc5e0 msgr2=0x7f443c10c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c0fc5e0 0x7f443c10c8b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f442c009150 tx=0x7f442c009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 shutdown_connections 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f44280692a0 0x7f442806b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f443c0fc5e0 0x7f443c10c8b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 --2- 172.21.15.32:0/2322535016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f443c101f00 0x7f443c10cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 >> 172.21.15.32:0/2322535016 conn(0x7f443c0714c0 msgr2=0x7f443c0f98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 shutdown_connections 2024-04-06T14:13:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:16.431+0000 7f44418d5700 1 -- 172.21.15.32:0/2322535016 wait complete. 2024-04-06T14:13:17.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:16 smithi032 ceph-mon[39730]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:17.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:16 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2322535016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T14:13:17.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:16 smithi086 ceph-mon[47925]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:17.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:16 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2322535016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T14:13:17.757 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-04-06T14:13:17.758 DEBUG:teuthology.parallel:result is None 2024-04-06T14:13:18.616 INFO:teuthology.orchestra.run.smithi032.stdout:150323855376 2024-04-06T14:13:18.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-06T14:13:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:18 smithi032 ceph-mon[39730]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:19.261 INFO:teuthology.orchestra.run.smithi032.stdout:115964117012 2024-04-06T14:13:19.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-06T14:13:19.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:18 smithi086 ceph-mon[47925]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.533+0000 7ffa43616700 1 -- 172.21.15.32:0/2033199824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 msgr2=0x7ffa3c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.533+0000 7ffa43616700 1 --2- 172.21.15.32:0/2033199824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 0x7ffa3c103240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa38004940 tx=0x7ffa38009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.533+0000 7ffa43616700 1 -- 172.21.15.32:0/2033199824 shutdown_connections 2024-04-06T14:13:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.533+0000 7ffa43616700 1 --2- 172.21.15.32:0/2033199824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0fb4a0 0x7ffa3c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.533+0000 7ffa43616700 1 --2- 172.21.15.32:0/2033199824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 0x7ffa3c103240 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.533+0000 7ffa43616700 1 -- 172.21.15.32:0/2033199824 >> 172.21.15.32:0/2033199824 conn(0x7ffa3c0714c0 msgr2=0x7ffa3c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa43616700 1 -- 172.21.15.32:0/2033199824 shutdown_connections 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa43616700 1 -- 172.21.15.32:0/2033199824 wait complete. 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa43616700 1 Processor -- start 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa43616700 1 -- start start 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa43616700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0fb4a0 0x7ffa3c0fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa43616700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 0x7ffa3c0fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa43616700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c0fd150 con 0x7ffa3c102e70 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa43616700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c0fd290 con 0x7ffa3c0fb4a0 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa40bb1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 0x7ffa3c0fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa40bb1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 0x7ffa3c0fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60556/0 (socket says 172.21.15.32:60556) 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa40bb1700 1 -- 172.21.15.32:0/1741969153 learned_addr learned my addr 172.21.15.32:0/1741969153 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.534+0000 7ffa413b2700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0fb4a0 0x7ffa3c0fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.535+0000 7ffa40bb1700 1 -- 172.21.15.32:0/1741969153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0fb4a0 msgr2=0x7ffa3c0fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.535+0000 7ffa40bb1700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0fb4a0 0x7ffa3c0fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.535+0000 7ffa40bb1700 1 -- 172.21.15.32:0/1741969153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa38005040 con 0x7ffa3c102e70 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.535+0000 7ffa40bb1700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 0x7ffa3c0fcbc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c830 tx=0x7ffa30004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.535+0000 7ffa2dffb700 1 -- 172.21.15.32:0/1741969153 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa300106f0 con 0x7ffa3c102e70 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.535+0000 7ffa2dffb700 1 -- 172.21.15.32:0/1741969153 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa30010d00 con 0x7ffa3c102e70 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.535+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3c119d70 con 0x7ffa3c102e70 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.535+0000 7ffa2dffb700 1 -- 172.21.15.32:0/1741969153 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa30012e90 con 0x7ffa3c102e70 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.535+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3c11a1f0 con 0x7ffa3c102e70 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.536+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3c100110 con 0x7ffa3c102e70 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.536+0000 7ffa2dffb700 1 -- 172.21.15.32:0/1741969153 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ffa30010850 con 0x7ffa3c102e70 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.537+0000 7ffa2dffb700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7ffa280692a0 0x7ffa2806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.537+0000 7ffa2dffb700 1 -- 172.21.15.32:0/1741969153 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ffa30086e40 con 0x7ffa3c102e70 2024-04-06T14:13:19.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.537+0000 7ffa413b2700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7ffa280692a0 0x7ffa2806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:19.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.537+0000 7ffa413b2700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7ffa280692a0 0x7ffa2806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffa38004f90 tx=0x7ffa38009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:19.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.541+0000 7ffa2dffb700 1 -- 172.21.15.32:0/1741969153 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ffa300534b0 con 0x7ffa3c102e70 2024-04-06T14:13:19.565 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:19.615 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:19.615 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:19.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.760+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ffa3c065380 con 0x7ffa3c102e70 2024-04-06T14:13:19.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.761+0000 7ffa2dffb700 1 -- 172.21.15.32:0/1741969153 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffa30056ad0 con 0x7ffa3c102e70 2024-04-06T14:13:19.761 INFO:teuthology.orchestra.run.smithi032.stdout:38654705703 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7ffa280692a0 msgr2=0x7ffa2806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7ffa280692a0 0x7ffa2806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffa38004f90 tx=0x7ffa38009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 msgr2=0x7ffa3c0fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 0x7ffa3c0fcbc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c830 tx=0x7ffa30004470 comp rx=0 tx=0).stop 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 shutdown_connections 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7ffa280692a0 0x7ffa2806b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0fb4a0 0x7ffa3c0fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 --2- 172.21.15.32:0/1741969153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa3c102e70 0x7ffa3c0fcbc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 >> 172.21.15.32:0/1741969153 conn(0x7ffa3c0714c0 msgr2=0x7ffa3c0fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 shutdown_connections 2024-04-06T14:13:19.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:19.763+0000 7ffa43616700 1 -- 172.21.15.32:0/1741969153 wait complete. 2024-04-06T14:13:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:19 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1741969153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T14:13:20.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:19 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1741969153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T14:13:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:20 smithi032 ceph-mon[39730]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:20 smithi086 ceph-mon[47925]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:23.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:22 smithi032 ceph-mon[39730]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:23.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:22 smithi086 ceph-mon[47925]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:23.417 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705701 got 38654705703 for osd.0 2024-04-06T14:13:23.417 DEBUG:teuthology.parallel:result is None 2024-04-06T14:13:24.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.029+0000 7f0f9012d700 1 -- 172.21.15.32:0/4055300145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 msgr2=0x7f0f880fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:24.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.029+0000 7f0f9012d700 1 --2- 172.21.15.32:0/4055300145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 0x7f0f880fbd80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0f84004940 tx=0x7f0f84009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:24.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.029+0000 7f0f9012d700 1 -- 172.21.15.32:0/4055300145 shutdown_connections 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.029+0000 7f0f9012d700 1 --2- 172.21.15.32:0/4055300145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 0x7f0f880fbd80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.029+0000 7f0f9012d700 1 --2- 172.21.15.32:0/4055300145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88101d70 0x7f0f88102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.029+0000 7f0f9012d700 1 -- 172.21.15.32:0/4055300145 >> 172.21.15.32:0/4055300145 conn(0x7f0f88072bc0 msgr2=0x7f0f88072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.029+0000 7f0f9012d700 1 -- 172.21.15.32:0/4055300145 shutdown_connections 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.029+0000 7f0f9012d700 1 -- 172.21.15.32:0/4055300145 wait complete. 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.030+0000 7f0f9012d700 1 Processor -- start 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.030+0000 7f0f9012d700 1 -- start start 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.030+0000 7f0f9012d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88101d70 0x7f0f88102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f9012d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 0x7f0f88103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f9012d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f881036a0 con 0x7f0f88102710 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f9012d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f881037e0 con 0x7f0f88101d70 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f8d6c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 0x7f0f88103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f8d6c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 0x7f0f88103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60566/0 (socket says 172.21.15.32:60566) 2024-04-06T14:13:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f8d6c8700 1 -- 172.21.15.32:0/1225364499 learned_addr learned my addr 172.21.15.32:0/1225364499 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f8dec9700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88101d70 0x7f0f88102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f8d6c8700 1 -- 172.21.15.32:0/1225364499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88101d70 msgr2=0x7f0f88102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f8d6c8700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88101d70 0x7f0f88102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f8d6c8700 1 -- 172.21.15.32:0/1225364499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f84005040 con 0x7f0f88102710 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f8d6c8700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 0x7f0f88103110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009380 tx=0x7f0f84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f7e7fc700 1 -- 172.21.15.32:0/1225364499 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f8400e040 con 0x7f0f88102710 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f7e7fc700 1 -- 172.21.15.32:0/1225364499 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f84013420 con 0x7f0f88102710 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f88103a30 con 0x7f0f88102710 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f7e7fc700 1 -- 172.21.15.32:0/1225364499 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f8401c630 con 0x7f0f88102710 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.031+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8811a1f0 con 0x7f0f88102710 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.032+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f880fd4c0 con 0x7f0f88102710 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.033+0000 7f0f7e7fc700 1 -- 172.21.15.32:0/1225364499 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0f840139b0 con 0x7f0f88102710 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.033+0000 7f0f7e7fc700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0f740692a0 0x7f0f7406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.033+0000 7f0f7e7fc700 1 -- 172.21.15.32:0/1225364499 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0f84087e80 con 0x7f0f88102710 2024-04-06T14:13:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.033+0000 7f0f8dec9700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0f740692a0 0x7f0f7406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.034+0000 7f0f8dec9700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0f740692a0 0x7f0f7406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f78004f40 tx=0x7f0f7800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.038+0000 7f0f7e7fc700 1 -- 172.21.15.32:0/1225364499 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0f84054590 con 0x7f0f88102710 2024-04-06T14:13:24.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.232+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f0f88108d40 con 0x7f0f88102710 2024-04-06T14:13:24.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.232+0000 7f0f7e7fc700 1 -- 172.21.15.32:0/1225364499 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0f84052e60 con 0x7f0f88102710 2024-04-06T14:13:24.234 INFO:teuthology.orchestra.run.smithi032.stdout:81604378651 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0f740692a0 msgr2=0x7f0f7406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0f740692a0 0x7f0f7406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f78004f40 tx=0x7f0f7800b040 comp rx=0 tx=0).stop 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 msgr2=0x7f0f88103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 0x7f0f88103110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009380 tx=0x7f0f84009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 shutdown_connections 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0f740692a0 0x7f0f7406b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88101d70 0x7f0f88102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 --2- 172.21.15.32:0/1225364499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f88102710 0x7f0f88103110 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 >> 172.21.15.32:0/1225364499 conn(0x7f0f88072bc0 msgr2=0x7f0f88065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 shutdown_connections 2024-04-06T14:13:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:24.234+0000 7f0f9012d700 1 -- 172.21.15.32:0/1225364499 wait complete. 2024-04-06T14:13:24.805 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:24 smithi032 ceph-mon[39730]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:24 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1225364499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T14:13:25.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:24 smithi086 ceph-mon[47925]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:25.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:24 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1225364499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T14:13:25.559 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-04-06T14:13:25.559 DEBUG:teuthology.parallel:result is None 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.747+0000 7f16e94a1700 1 -- 172.21.15.32:0/1416992793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fc590 msgr2=0x7f16e41092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.747+0000 7f16e94a1700 1 --2- 172.21.15.32:0/1416992793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fc590 0x7f16e41092f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f16d8004940 tx=0x7f16d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.747+0000 7f16e94a1700 1 -- 172.21.15.32:0/1416992793 shutdown_connections 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.747+0000 7f16e94a1700 1 --2- 172.21.15.32:0/1416992793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fc590 0x7f16e41092f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.747+0000 7f16e94a1700 1 --2- 172.21.15.32:0/1416992793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16e40fbc80 0x7f16e40fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.747+0000 7f16e94a1700 1 -- 172.21.15.32:0/1416992793 >> 172.21.15.32:0/1416992793 conn(0x7f16e4072bc0 msgr2=0x7f16e4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.747+0000 7f16e94a1700 1 -- 172.21.15.32:0/1416992793 shutdown_connections 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.747+0000 7f16e94a1700 1 -- 172.21.15.32:0/1416992793 wait complete. 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.747+0000 7f16e94a1700 1 Processor -- start 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e94a1700 1 -- start start 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e94a1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fbc80 0x7f16e4115ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e94a1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16e40fc590 0x7f16e41160e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:26.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e94a1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4116730 con 0x7f16e40fbc80 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e94a1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e410fb90 con 0x7f16e40fc590 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fbc80 0x7f16e4115ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fbc80 0x7f16e4115ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55798/0 (socket says 172.21.15.32:55798) 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e2ffd700 1 -- 172.21.15.32:0/1365776106 learned_addr learned my addr 172.21.15.32:0/1365776106 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e27fc700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16e40fc590 0x7f16e41160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e2ffd700 1 -- 172.21.15.32:0/1365776106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16e40fc590 msgr2=0x7f16e41160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e2ffd700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16e40fc590 0x7f16e41160e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e2ffd700 1 -- 172.21.15.32:0/1365776106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8005040 con 0x7f16e40fbc80 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e27fc700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16e40fc590 0x7f16e41160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e2ffd700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fbc80 0x7f16e4115ba0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f16d400eb40 tx=0x7f16d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16cb7fe700 1 -- 172.21.15.32:0/1365776106 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16d4007cc0 con 0x7f16e40fbc80 2024-04-06T14:13:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16cb7fe700 1 -- 172.21.15.32:0/1365776106 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16d400f040 con 0x7f16e40fbc80 2024-04-06T14:13:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.748+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e410fe10 con 0x7f16e40fbc80 2024-04-06T14:13:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.749+0000 7f16cb7fe700 1 -- 172.21.15.32:0/1365776106 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16d4009570 con 0x7f16e40fbc80 2024-04-06T14:13:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.749+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e41103d0 con 0x7f16e40fbc80 2024-04-06T14:13:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.750+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e4065e30 con 0x7f16e40fbc80 2024-04-06T14:13:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.750+0000 7f16cb7fe700 1 -- 172.21.15.32:0/1365776106 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f16d40096d0 con 0x7f16e40fbc80 2024-04-06T14:13:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.750+0000 7f16cb7fe700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f16cc0692a0 0x7f16cc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.750+0000 7f16cb7fe700 1 -- 172.21.15.32:0/1365776106 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f16d4086af0 con 0x7f16e40fbc80 2024-04-06T14:13:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.750+0000 7f16e27fc700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f16cc0692a0 0x7f16cc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.751+0000 7f16e27fc700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f16cc0692a0 0x7f16cc06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f16d800b0d0 tx=0x7f16d8009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.755+0000 7f16cb7fe700 1 -- 172.21.15.32:0/1365776106 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f16d404e190 con 0x7f16e40fbc80 2024-04-06T14:13:26.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.645+0000 7f64f59de700 1 -- 172.21.15.32:0/2952174562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 msgr2=0x7f64f01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:26.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.645+0000 7f64f59de700 1 --2- 172.21.15.32:0/2952174562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 0x7f64f01031d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f64e0004940 tx=0x7f64e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.645+0000 7f64f59de700 1 -- 172.21.15.32:0/2952174562 shutdown_connections 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.645+0000 7f64f59de700 1 --2- 172.21.15.32:0/2952174562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f01037a0 0x7f64f00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.645+0000 7f64f59de700 1 --2- 172.21.15.32:0/2952174562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 0x7f64f01031d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.645+0000 7f64f59de700 1 -- 172.21.15.32:0/2952174562 >> 172.21.15.32:0/2952174562 conn(0x7f64f00f89b0 msgr2=0x7f64f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.645+0000 7f64f59de700 1 -- 172.21.15.32:0/2952174562 shutdown_connections 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.645+0000 7f64f59de700 1 -- 172.21.15.32:0/2952174562 wait complete. 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64f59de700 1 Processor -- start 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64f59de700 1 -- start start 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64f59de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 0x7f64f0113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64f59de700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f01037a0 0x7f64f0114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64f59de700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f01146b0 con 0x7f64f0102e00 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64f59de700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f010daa0 con 0x7f64f01037a0 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64eeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 0x7f64f0113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64eeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 0x7f64f0113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55778/0 (socket says 172.21.15.32:55778) 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64eeffd700 1 -- 172.21.15.32:0/1921527724 learned_addr learned my addr 172.21.15.32:0/1921527724 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.646+0000 7f64ee7fc700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f01037a0 0x7f64f0114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.647+0000 7f64eeffd700 1 -- 172.21.15.32:0/1921527724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f01037a0 msgr2=0x7f64f0114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.647+0000 7f64eeffd700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f01037a0 0x7f64f0114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.647+0000 7f64eeffd700 1 -- 172.21.15.32:0/1921527724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e0005040 con 0x7f64f0102e00 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.647+0000 7f64eeffd700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 0x7f64f0113b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f64e000b3b0 tx=0x7f64e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.647+0000 7f64d77fe700 1 -- 172.21.15.32:0/1921527724 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64e000e040 con 0x7f64f0102e00 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.647+0000 7f64d77fe700 1 -- 172.21.15.32:0/1921527724 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64e0013420 con 0x7f64f0102e00 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.647+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f010dd20 con 0x7f64f0102e00 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.647+0000 7f64d77fe700 1 -- 172.21.15.32:0/1921527724 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64e001c630 con 0x7f64f0102e00 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.647+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f010e2e0 con 0x7f64f0102e00 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.648+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f00fe550 con 0x7f64f0102e00 2024-04-06T14:13:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.648+0000 7f64d77fe700 1 -- 172.21.15.32:0/1921527724 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f64e0013970 con 0x7f64f0102e00 2024-04-06T14:13:26.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.649+0000 7f64d77fe700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f64d80692a0 0x7f64d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:26.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.649+0000 7f64d77fe700 1 -- 172.21.15.32:0/1921527724 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f64e0088060 con 0x7f64f0102e00 2024-04-06T14:13:26.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.649+0000 7f64ee7fc700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f64d80692a0 0x7f64d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:26.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.649+0000 7f64ee7fc700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f64d80692a0 0x7f64d806b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f64f010ec60 tx=0x7f64e4009ee0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:26.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.653+0000 7f64d77fe700 1 -- 172.21.15.32:0/1921527724 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f64e0058160 con 0x7f64f0102e00 2024-04-06T14:13:26.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.870+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f64f004bed0 con 0x7f64f0102e00 2024-04-06T14:13:26.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.870+0000 7f64d77fe700 1 -- 172.21.15.32:0/1921527724 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f64e0057cf0 con 0x7f64f0102e00 2024-04-06T14:13:26.870 INFO:teuthology.orchestra.run.smithi032.stdout:171798691852 2024-04-06T14:13:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f64d80692a0 msgr2=0x7f64d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f64d80692a0 0x7f64d806b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f64f010ec60 tx=0x7f64e4009ee0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 msgr2=0x7f64f0113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 0x7f64f0113b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f64e000b3b0 tx=0x7f64e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 shutdown_connections 2024-04-06T14:13:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f64d80692a0 0x7f64d806b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64f0102e00 0x7f64f0113b20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 --2- 172.21.15.32:0/1921527724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f01037a0 0x7f64f0114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 >> 172.21.15.32:0/1921527724 conn(0x7f64f00f89b0 msgr2=0x7f64f00f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 shutdown_connections 2024-04-06T14:13:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.872+0000 7f64f59de700 1 -- 172.21.15.32:0/1921527724 wait complete. 2024-04-06T14:13:26.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.951+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f16e405f5f0 con 0x7f16e40fbc80 2024-04-06T14:13:26.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.952+0000 7f16cb7fe700 1 -- 172.21.15.32:0/1365776106 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f16d4053190 con 0x7f16e40fbc80 2024-04-06T14:13:26.952 INFO:teuthology.orchestra.run.smithi032.stdout:150323855378 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f16cc0692a0 msgr2=0x7f16cc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f16cc0692a0 0x7f16cc06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f16d800b0d0 tx=0x7f16d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fbc80 msgr2=0x7f16e4115ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fbc80 0x7f16e4115ba0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f16d400eb40 tx=0x7f16d400b040 comp rx=0 tx=0).stop 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 shutdown_connections 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f16cc0692a0 0x7f16cc06b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e40fbc80 0x7f16e4115ba0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 --2- 172.21.15.32:0/1365776106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16e40fc590 0x7f16e41160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 >> 172.21.15.32:0/1365776106 conn(0x7f16e4072bc0 msgr2=0x7f16e4105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 shutdown_connections 2024-04-06T14:13:26.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:26.954+0000 7f16e94a1700 1 -- 172.21.15.32:0/1365776106 wait complete. 2024-04-06T14:13:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:26 smithi032 ceph-mon[39730]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:26 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1921527724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T14:13:27.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:26 smithi086 ceph-mon[47925]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:27.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:26 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1921527724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T14:13:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:27 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1365776106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T14:13:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:27 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:13:28.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:27 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1365776106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T14:13:28.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:27 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:13:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 -- 172.21.15.32:0/1934133858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c102710 msgr2=0x7fd54c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 --2- 172.21.15.32:0/1934133858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c102710 0x7fd54c0fbd80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd53c004940 tx=0x7fd53c009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 -- 172.21.15.32:0/1934133858 shutdown_connections 2024-04-06T14:13:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 --2- 172.21.15.32:0/1934133858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c102710 0x7fd54c0fbd80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 --2- 172.21.15.32:0/1934133858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c101d70 0x7fd54c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 -- 172.21.15.32:0/1934133858 >> 172.21.15.32:0/1934133858 conn(0x7fd54c072bc0 msgr2=0x7fd54c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 -- 172.21.15.32:0/1934133858 shutdown_connections 2024-04-06T14:13:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 -- 172.21.15.32:0/1934133858 wait complete. 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 Processor -- start 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.677+0000 7fd5539b2700 1 -- start start 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd5539b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c101d70 0x7fd54c0fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd5539b2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c102710 0x7fd54c0fdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd5539b2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0fe4c0 con 0x7fd54c101d70 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd5539b2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0fe600 con 0x7fd54c102710 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd55174e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c101d70 0x7fd54c0fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd55174e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c101d70 0x7fd54c0fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55814/0 (socket says 172.21.15.32:55814) 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd55174e700 1 -- 172.21.15.32:0/3073482900 learned_addr learned my addr 172.21.15.32:0/3073482900 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd550f4d700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c102710 0x7fd54c0fdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd55174e700 1 -- 172.21.15.32:0/3073482900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c102710 msgr2=0x7fd54c0fdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd55174e700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c102710 0x7fd54c0fdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd55174e700 1 -- 172.21.15.32:0/3073482900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c005040 con 0x7fd54c101d70 2024-04-06T14:13:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd55174e700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c101d70 0x7fd54c0fd9f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd54800eb70 tx=0x7fd54800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd541ffb700 1 -- 172.21.15.32:0/3073482900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd548007cc0 con 0x7fd54c101d70 2024-04-06T14:13:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd541ffb700 1 -- 172.21.15.32:0/3073482900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd54800f040 con 0x7fd54c101d70 2024-04-06T14:13:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c0fe850 con 0x7fd54c101d70 2024-04-06T14:13:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.678+0000 7fd541ffb700 1 -- 172.21.15.32:0/3073482900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5480095f0 con 0x7fd54c101d70 2024-04-06T14:13:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.679+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c11a1d0 con 0x7fd54c101d70 2024-04-06T14:13:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.679+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c0fd4c0 con 0x7fd54c101d70 2024-04-06T14:13:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.680+0000 7fd541ffb700 1 -- 172.21.15.32:0/3073482900 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd548009750 con 0x7fd54c101d70 2024-04-06T14:13:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.680+0000 7fd541ffb700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd5380692b0 0x7fd53806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.680+0000 7fd541ffb700 1 -- 172.21.15.32:0/3073482900 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd5480869e0 con 0x7fd54c101d70 2024-04-06T14:13:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.680+0000 7fd550f4d700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd5380692b0 0x7fd53806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.681+0000 7fd550f4d700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd5380692b0 0x7fd53806b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd54c0ff330 tx=0x7fd53c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:28.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.685+0000 7fd541ffb700 1 -- 172.21.15.32:0/3073482900 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd548053050 con 0x7fd54c101d70 2024-04-06T14:13:28.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.882+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd54c04bf00 con 0x7fd54c101d70 2024-04-06T14:13:28.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.883+0000 7fd541ffb700 1 -- 172.21.15.32:0/3073482900 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd548056670 con 0x7fd54c101d70 2024-04-06T14:13:28.883 INFO:teuthology.orchestra.run.smithi032.stdout:115964117015 2024-04-06T14:13:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd5380692b0 msgr2=0x7fd53806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd5380692b0 0x7fd53806b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd54c0ff330 tx=0x7fd53c009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c101d70 msgr2=0x7fd54c0fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c101d70 0x7fd54c0fd9f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd54800eb70 tx=0x7fd54800b040 comp rx=0 tx=0).stop 2024-04-06T14:13:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 shutdown_connections 2024-04-06T14:13:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd5380692b0 0x7fd53806b760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c101d70 0x7fd54c0fd9f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 --2- 172.21.15.32:0/3073482900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c102710 0x7fd54c0fdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 >> 172.21.15.32:0/3073482900 conn(0x7fd54c072bc0 msgr2=0x7fd54c065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 shutdown_connections 2024-04-06T14:13:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:28.885+0000 7fd5539b2700 1 -- 172.21.15.32:0/3073482900 wait complete. 2024-04-06T14:13:29.021 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691849 got 171798691852 for osd.5 2024-04-06T14:13:29.022 DEBUG:teuthology.parallel:result is None 2024-04-06T14:13:29.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:28 smithi032 ceph-mon[39730]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:29.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:28 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3073482900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T14:13:29.291 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855376 got 150323855378 for osd.4 2024-04-06T14:13:29.291 DEBUG:teuthology.parallel:result is None 2024-04-06T14:13:29.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:28 smithi086 ceph-mon[47925]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:29.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:28 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3073482900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T14:13:29.942 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117012 got 115964117015 for osd.3 2024-04-06T14:13:29.943 DEBUG:teuthology.parallel:result is None 2024-04-06T14:13:29.943 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-06T14:13:29.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T14:13:30.224 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:30.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:29 smithi032 ceph-mon[39730]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:30.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:29 smithi086 ceph-mon[47925]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:32.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 -- 172.21.15.32:0/2912044365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 msgr2=0x7f5268102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:32.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 --2- 172.21.15.32:0/2912044365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 0x7f5268102150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5258004940 tx=0x7f5258009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:32.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 -- 172.21.15.32:0/2912044365 shutdown_connections 2024-04-06T14:13:32.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 --2- 172.21.15.32:0/2912044365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5268102720 0x7f52680fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:32.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 --2- 172.21.15.32:0/2912044365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 0x7f5268102150 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:32.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 -- 172.21.15.32:0/2912044365 >> 172.21.15.32:0/2912044365 conn(0x7f52680f78f0 msgr2=0x7f52680f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:32.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 -- 172.21.15.32:0/2912044365 shutdown_connections 2024-04-06T14:13:32.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 -- 172.21.15.32:0/2912044365 wait complete. 2024-04-06T14:13:32.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 Processor -- start 2024-04-06T14:13:32.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.121+0000 7f526ca0f700 1 -- start start 2024-04-06T14:13:32.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526ca0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 0x7f5268102be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:32.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526ca0f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5268102720 0x7f5268103120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:32.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526ca0f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5268104b80 con 0x7f5268101d80 2024-04-06T14:13:32.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526ca0f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5268103660 con 0x7f5268102720 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 0x7f5268102be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 0x7f5268102be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55826/0 (socket says 172.21.15.32:55826) 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526659c700 1 -- 172.21.15.32:0/2652254381 learned_addr learned my addr 172.21.15.32:0/2652254381 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f5265d9b700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5268102720 0x7f5268103120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526659c700 1 -- 172.21.15.32:0/2652254381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5268102720 msgr2=0x7f5268103120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526659c700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5268102720 0x7f5268103120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526659c700 1 -- 172.21.15.32:0/2652254381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5258005040 con 0x7f5268101d80 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526659c700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 0x7f5268102be0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f525800b3b0 tx=0x7f5258009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f5256ffd700 1 -- 172.21.15.32:0/2652254381 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f525800e040 con 0x7f5268101d80 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f5256ffd700 1 -- 172.21.15.32:0/2652254381 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5258013420 con 0x7f5268101d80 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.122+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52681038b0 con 0x7f5268101d80 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.123+0000 7f5256ffd700 1 -- 172.21.15.32:0/2652254381 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f525801c630 con 0x7f5268101d80 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.123+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5268119f80 con 0x7f5268101d80 2024-04-06T14:13:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.123+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52680fd4d0 con 0x7f5268101d80 2024-04-06T14:13:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.124+0000 7f5256ffd700 1 -- 172.21.15.32:0/2652254381 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5258013970 con 0x7f5268101d80 2024-04-06T14:13:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.124+0000 7f5256ffd700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f52500692a0 0x7f525006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.124+0000 7f5256ffd700 1 -- 172.21.15.32:0/2652254381 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5258087fa0 con 0x7f5268101d80 2024-04-06T14:13:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.124+0000 7f5265d9b700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f52500692a0 0x7f525006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.125+0000 7f5265d9b700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f52500692a0 0x7f525006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5268104310 tx=0x7f525c009ee0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.129+0000 7f5256ffd700 1 -- 172.21.15.32:0/2652254381 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f52580546b0 con 0x7f5268101d80 2024-04-06T14:13:32.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.320+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5268104160 con 0x7f52500692a0 2024-04-06T14:13:32.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.321+0000 7f5256ffd700 1 -- 172.21.15.32:0/2652254381 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27887 (secure 0 0 0) 0x7f5268104160 con 0x7f52500692a0 2024-04-06T14:13:32.322 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:13:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f52500692a0 msgr2=0x7f525006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f52500692a0 0x7f525006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5268104310 tx=0x7f525c009ee0 comp rx=0 tx=0).stop 2024-04-06T14:13:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 msgr2=0x7f5268102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 0x7f5268102be0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f525800b3b0 tx=0x7f5258009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 shutdown_connections 2024-04-06T14:13:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f52500692a0 0x7f525006b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5268101d80 0x7f5268102be0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 --2- 172.21.15.32:0/2652254381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5268102720 0x7f5268103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 >> 172.21.15.32:0/2652254381 conn(0x7f52680f78f0 msgr2=0x7f52680665b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 shutdown_connections 2024-04-06T14:13:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:32.323+0000 7f526ca0f700 1 -- 172.21.15.32:0/2652254381 wait complete. 2024-04-06T14:13:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-06T14:13:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:32 smithi032 ceph-mon[39730]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:33.027 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-04-06T14:13:31.654019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788072,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682840,"statfs":{"total":575970213888,"available":574139228160,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003842"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-06T14:12:36.748319+0000","last_change":"2024-04-06T14:12:10.246249+0000","last_active":"2024-04-06T14:12:36.748319+0000","last_peered":"2024-04-06T14:12:36.748319+0000","last_clean":"2024-04-06T14:12:36.748319+0000","last_became_active":"2024-04-06T14:12:10.245859+0000","last_became_peered":"2024-04-06T14:12:10.245859+0000","last_unstale":"2024-04-06T14:12:36.748319+0000","last_undegraded":"2024-04-06T14:12:36.748319+0000","last_fullsized":"2024-04-06T14:12:36.748319+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T14:11:20.445098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T14:11:20.445098+0000","last_clean_scrub_stamp":"2024-04-06T14:11:20.445098+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:07:18.079441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]},{"osd":4,"up_from":35,"seq":150323855380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 14:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.627}]},{"osd":1,"last update":"Sat Apr 6 14:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.61299999999999999}]},{"osd":2,"last update":"Sat Apr 6 14:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59799999999999998}]},{"osd":3,"last update":"Sat Apr 6 14:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.53200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 14:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Apr 6 14:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.42299999999999999}]},{"osd":2,"last update":"Sat Apr 6 14:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.47199999999999998}]},{"osd":4,"last update":"Sat Apr 6 14:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.379}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 14:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.496,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.47699999999999998}]},{"osd":1,"last update":"Sat Apr 6 14:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.51200000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.44},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.36699999999999999,"15min":0.372},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.48299999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.496}]},{"osd":3,"last update":"Sat Apr 6 14:12:45 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.70499999999999996}]},{"osd":4,"last update":"Sat Apr 6 14:13:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.84399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705706,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 14:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.35799999999999998,"15min":0.35599999999999998},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.36899999999999999,"15min":0.36799999999999999},"min":{"1min":0.26800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.39100000000000001}]},{"osd":2,"last update":"Sat Apr 6 14:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.39900000000000002,"15min":0.40600000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55300000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.40899999999999997,"15min":0.41499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.45500000000000002}]},{"osd":3,"last update":"Sat Apr 6 14:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.69299999999999995}]},{"osd":4,"last update":"Sat Apr 6 14:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 14:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.32900000000000001,"15min":0.32700000000000001},"min":{"1min":0.27200000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.46100000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.34699999999999998,"15min":0.34200000000000003},"min":{"1min":0.25,"5min":0.218,"15min":0.218},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Apr 6 14:13:20 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.39000000000000001,"15min":0.39600000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.318},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41499999999999998,"15min":0.42099999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.30599999999999999}]},{"osd":3,"last update":"Sat Apr 6 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.68600000000000005}]},{"osd":4,"last update":"Sat Apr 6 14:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.58599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T14:13:33.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T14:13:33.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:32 smithi086 ceph-mon[47925]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:33.305 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:33.808 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:33 smithi032 ceph-mon[39730]: from='client.14404 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T14:13:34.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:33 smithi086 ceph-mon[47925]: from='client.14404 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T14:13:34.827 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:34 smithi032 ceph-mon[39730]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.070+0000 7f40422ad700 1 -- 172.21.15.32:0/1520629324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fc590 msgr2=0x7f403c1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.070+0000 7f40422ad700 1 --2- 172.21.15.32:0/1520629324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fc590 0x7f403c1092f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4030004940 tx=0x7f4030009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.070+0000 7f40422ad700 1 -- 172.21.15.32:0/1520629324 shutdown_connections 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.070+0000 7f40422ad700 1 --2- 172.21.15.32:0/1520629324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fc590 0x7f403c1092f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.070+0000 7f40422ad700 1 --2- 172.21.15.32:0/1520629324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f403c0fbc80 0x7f403c0fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.070+0000 7f40422ad700 1 -- 172.21.15.32:0/1520629324 >> 172.21.15.32:0/1520629324 conn(0x7f403c072bc0 msgr2=0x7f403c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.071+0000 7f40422ad700 1 -- 172.21.15.32:0/1520629324 shutdown_connections 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.071+0000 7f40422ad700 1 -- 172.21.15.32:0/1520629324 wait complete. 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.071+0000 7f40422ad700 1 Processor -- start 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.071+0000 7f40422ad700 1 -- start start 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f40422ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fbc80 0x7f403c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f40422ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f403c0fc590 0x7f403c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f40422ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c116800 con 0x7f403c0fbc80 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f40422ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c10fbd0 con 0x7f403c0fc590 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f403bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fbc80 0x7f403c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f403bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fbc80 0x7f403c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44846/0 (socket says 172.21.15.32:44846) 2024-04-06T14:13:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f403bfff700 1 -- 172.21.15.32:0/2327670055 learned_addr learned my addr 172.21.15.32:0/2327670055 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f403b7fe700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f403c0fc590 0x7f403c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f403bfff700 1 -- 172.21.15.32:0/2327670055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f403c0fc590 msgr2=0x7f403c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f403bfff700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f403c0fc590 0x7f403c1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f403bfff700 1 -- 172.21.15.32:0/2327670055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4030005040 con 0x7f403c0fbc80 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f403bfff700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fbc80 0x7f403c115c70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f402c002a50 tx=0x7f402c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f4038ff9700 1 -- 172.21.15.32:0/2327670055 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f402c007cc0 con 0x7f403c0fbc80 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403c10feb0 con 0x7f403c0fbc80 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f4038ff9700 1 -- 172.21.15.32:0/2327670055 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f402c00f040 con 0x7f403c0fbc80 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f4038ff9700 1 -- 172.21.15.32:0/2327670055 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f402c00d9c0 con 0x7f403c0fbc80 2024-04-06T14:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.072+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c110470 con 0x7f403c0fbc80 2024-04-06T14:13:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.073+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403c065e30 con 0x7f403c0fbc80 2024-04-06T14:13:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.074+0000 7f4038ff9700 1 -- 172.21.15.32:0/2327670055 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f402c00db20 con 0x7f403c0fbc80 2024-04-06T14:13:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.074+0000 7f4038ff9700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f40240692a0 0x7f402406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.074+0000 7f403b7fe700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f40240692a0 0x7f402406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.074+0000 7f4038ff9700 1 -- 172.21.15.32:0/2327670055 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f402c086b70 con 0x7f403c0fbc80 2024-04-06T14:13:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.075+0000 7f403b7fe700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f40240692a0 0x7f402406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4030004f90 tx=0x7f4030009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:35.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.079+0000 7f4038ff9700 1 -- 172.21.15.32:0/2327670055 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f402c056bf0 con 0x7f403c0fbc80 2024-04-06T14:13:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:34 smithi086 ceph-mon[47925]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:35.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.269+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f403c04bed0 con 0x7f40240692a0 2024-04-06T14:13:35.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.270+0000 7f4038ff9700 1 -- 172.21.15.32:0/2327670055 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27888 (secure 0 0 0) 0x7f403c04bed0 con 0x7f40240692a0 2024-04-06T14:13:35.271 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:13:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.272+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f40240692a0 msgr2=0x7f402406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.272+0000 7f40422ad700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f40240692a0 0x7f402406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4030004f90 tx=0x7f4030009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.272+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fbc80 msgr2=0x7f403c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.272+0000 7f40422ad700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fbc80 0x7f403c115c70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f402c002a50 tx=0x7f402c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.272+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 shutdown_connections 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.272+0000 7f40422ad700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f40240692a0 0x7f402406b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.272+0000 7f40422ad700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f403c0fbc80 0x7f403c115c70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.272+0000 7f40422ad700 1 --2- 172.21.15.32:0/2327670055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f403c0fc590 0x7f403c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.272+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 >> 172.21.15.32:0/2327670055 conn(0x7f403c072bc0 msgr2=0x7f403c105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.273+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 shutdown_connections 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:35.273+0000 7f40422ad700 1 -- 172.21.15.32:0/2327670055 wait complete. 2024-04-06T14:13:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-06T14:13:36.034 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-06T14:13:33.654398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788072,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682840,"statfs":{"total":575970213888,"available":574139228160,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003934"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-06T14:12:36.748319+0000","last_change":"2024-04-06T14:12:10.246249+0000","last_active":"2024-04-06T14:12:36.748319+0000","last_peered":"2024-04-06T14:12:36.748319+0000","last_clean":"2024-04-06T14:12:36.748319+0000","last_became_active":"2024-04-06T14:12:10.245859+0000","last_became_peered":"2024-04-06T14:12:10.245859+0000","last_unstale":"2024-04-06T14:12:36.748319+0000","last_undegraded":"2024-04-06T14:12:36.748319+0000","last_fullsized":"2024-04-06T14:12:36.748319+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T14:11:20.445098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T14:11:20.445098+0000","last_clean_scrub_stamp":"2024-04-06T14:11:20.445098+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:07:18.079441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 14:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.627}]},{"osd":1,"last update":"Sat Apr 6 14:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.61299999999999999}]},{"osd":2,"last update":"Sat Apr 6 14:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59799999999999998}]},{"osd":3,"last update":"Sat Apr 6 14:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.53200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 14:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.58999999999999997}]},{"osd":1,"last update":"Sat Apr 6 14:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.48699999999999999}]},{"osd":2,"last update":"Sat Apr 6 14:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.56299999999999994}]},{"osd":4,"last update":"Sat Apr 6 14:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.311},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.35699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 14:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.496,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.47699999999999998}]},{"osd":1,"last update":"Sat Apr 6 14:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.51200000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.44},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.36699999999999999,"15min":0.372},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.48299999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.496}]},{"osd":3,"last update":"Sat Apr 6 14:12:45 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.70499999999999996}]},{"osd":4,"last update":"Sat Apr 6 14:13:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.84399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705706,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 14:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.35799999999999998,"15min":0.35599999999999998},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.36899999999999999,"15min":0.36799999999999999},"min":{"1min":0.26800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.39100000000000001}]},{"osd":2,"last update":"Sat Apr 6 14:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.39900000000000002,"15min":0.40600000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55300000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.40899999999999997,"15min":0.41499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.45500000000000002}]},{"osd":3,"last update":"Sat Apr 6 14:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.69299999999999995}]},{"osd":4,"last update":"Sat Apr 6 14:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 14:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.32900000000000001,"15min":0.32700000000000001},"min":{"1min":0.27200000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.46100000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.34699999999999998,"15min":0.34200000000000003},"min":{"1min":0.25,"5min":0.218,"15min":0.218},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Apr 6 14:13:20 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.39000000000000001,"15min":0.39600000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.318},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41499999999999998,"15min":0.42099999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.30599999999999999}]},{"osd":3,"last update":"Sat Apr 6 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.68600000000000005}]},{"osd":4,"last update":"Sat Apr 6 14:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.58599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T14:13:36.035 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-06T14:13:36.035 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-06T14:13:36.035 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-06T14:13:36.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-06T14:13:36.316 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:36.834 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:36 smithi032 ceph-mon[39730]: from='client.14408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T14:13:36.834 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:36 smithi032 ceph-mon[39730]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:37.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:36 smithi086 ceph-mon[47925]: from='client.14408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T14:13:37.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:36 smithi086 ceph-mon[47925]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:38.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.339+0000 7fbf39f5b700 1 -- 172.21.15.32:0/1886206812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fde90 msgr2=0x7fbf340fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:38.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.339+0000 7fbf39f5b700 1 --2- 172.21.15.32:0/1886206812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fde90 0x7fbf340fe260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbf24004940 tx=0x7fbf24009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:38.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.340+0000 7fbf39f5b700 1 -- 172.21.15.32:0/1886206812 shutdown_connections 2024-04-06T14:13:38.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.340+0000 7fbf39f5b700 1 --2- 172.21.15.32:0/1886206812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf340fe830 0x7fbf34106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:38.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.340+0000 7fbf39f5b700 1 --2- 172.21.15.32:0/1886206812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fde90 0x7fbf340fe260 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:38.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.340+0000 7fbf39f5b700 1 -- 172.21.15.32:0/1886206812 >> 172.21.15.32:0/1886206812 conn(0x7fbf34072bc0 msgr2=0x7fbf34072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:38.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.340+0000 7fbf39f5b700 1 -- 172.21.15.32:0/1886206812 shutdown_connections 2024-04-06T14:13:38.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.340+0000 7fbf39f5b700 1 -- 172.21.15.32:0/1886206812 wait complete. 2024-04-06T14:13:38.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.340+0000 7fbf39f5b700 1 Processor -- start 2024-04-06T14:13:38.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.341+0000 7fbf39f5b700 1 -- start start 2024-04-06T14:13:38.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.341+0000 7fbf39f5b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf340fde90 0x7fbf3411a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:38.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.341+0000 7fbf39f5b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fe830 0x7fbf3411a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:38.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.341+0000 7fbf39f5b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3411ac90 con 0x7fbf340fe830 2024-04-06T14:13:38.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.341+0000 7fbf39f5b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf34114060 con 0x7fbf340fde90 2024-04-06T14:13:38.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.341+0000 7fbf32ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fe830 0x7fbf3411a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:38.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.341+0000 7fbf32ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fe830 0x7fbf3411a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44864/0 (socket says 172.21.15.32:44864) 2024-04-06T14:13:38.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.341+0000 7fbf32ffd700 1 -- 172.21.15.32:0/192376873 learned_addr learned my addr 172.21.15.32:0/192376873 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:38.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.341+0000 7fbf337fe700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf340fde90 0x7fbf3411a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:38.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf32ffd700 1 -- 172.21.15.32:0/192376873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf340fde90 msgr2=0x7fbf3411a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:38.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf32ffd700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf340fde90 0x7fbf3411a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:38.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf32ffd700 1 -- 172.21.15.32:0/192376873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24005040 con 0x7fbf340fe830 2024-04-06T14:13:38.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf337fe700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf340fde90 0x7fbf3411a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:13:38.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf32ffd700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fe830 0x7fbf3411a640 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800c830 tx=0x7fbf28004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf1bfff700 1 -- 172.21.15.32:0/192376873 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf2800a760 con 0x7fbf340fe830 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf34114340 con 0x7fbf340fe830 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf1bfff700 1 -- 172.21.15.32:0/192376873 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf2800ad70 con 0x7fbf340fe830 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf1bfff700 1 -- 172.21.15.32:0/192376873 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf280113a0 con 0x7fbf340fe830 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.342+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf34114880 con 0x7fbf340fe830 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.343+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf34102610 con 0x7fbf340fe830 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.344+0000 7fbf1bfff700 1 -- 172.21.15.32:0/192376873 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fbf28012400 con 0x7fbf340fe830 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.344+0000 7fbf1bfff700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf1c0692b0 0x7fbf1c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.344+0000 7fbf1bfff700 1 -- 172.21.15.32:0/192376873 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbf28086ee0 con 0x7fbf340fe830 2024-04-06T14:13:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.344+0000 7fbf337fe700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf1c0692b0 0x7fbf1c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:38.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.344+0000 7fbf337fe700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf1c0692b0 0x7fbf1c06b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf2400b3b0 tx=0x7fbf24009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:38.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.348+0000 7fbf1bfff700 1 -- 172.21.15.32:0/192376873 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbf28056fe0 con 0x7fbf340fe830 2024-04-06T14:13:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.587+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbf34115750 con 0x7fbf340fe830 2024-04-06T14:13:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.588+0000 7fbf1bfff700 1 -- 172.21.15.32:0/192376873 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbf28056b70 con 0x7fbf340fe830 2024-04-06T14:13:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:13:38.588 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T14:13:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.589+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf1c0692b0 msgr2=0x7fbf1c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf1c0692b0 0x7fbf1c06b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf2400b3b0 tx=0x7fbf24009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fe830 msgr2=0x7fbf3411a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fe830 0x7fbf3411a640 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800c830 tx=0x7fbf28004470 comp rx=0 tx=0).stop 2024-04-06T14:13:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 shutdown_connections 2024-04-06T14:13:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fbf1c0692b0 0x7fbf1c06b760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf340fde90 0x7fbf3411a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 --2- 172.21.15.32:0/192376873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf340fe830 0x7fbf3411a640 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 >> 172.21.15.32:0/192376873 conn(0x7fbf34072bc0 msgr2=0x7fbf340fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:38.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 shutdown_connections 2024-04-06T14:13:38.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:38.590+0000 7fbf39f5b700 1 -- 172.21.15.32:0/192376873 wait complete. 2024-04-06T14:13:38.943 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:38 smithi032 ceph-mon[39730]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:38.944 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:38 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/192376873' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T14:13:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:38 smithi086 ceph-mon[47925]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:38 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/192376873' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T14:13:39.295 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-06T14:13:39.295 INFO:tasks.cephadm:Setup complete, yielding 2024-04-06T14:13:39.296 INFO:teuthology.run_tasks:Running task print... 2024-04-06T14:13:39.305 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-04-06T14:13:39.305 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T14:13:39.339 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:13:39.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-06T14:13:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:39 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:13:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:39 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:13:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:39 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:13:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:39 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:13:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:39 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:13:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:39 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:13:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:39 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:13:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:39 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:13:40.890 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:40 smithi032 ceph-mon[39730]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:40 smithi086 ceph-mon[47925]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:41.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.467+0000 7f9086f12700 1 -- 172.21.15.32:0/3886283833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 msgr2=0x7f90800fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.467+0000 7f9086f12700 1 --2- 172.21.15.32:0/3886283833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 0x7f90800fd3f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f906c004940 tx=0x7f906c009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.467+0000 7f9086f12700 1 -- 172.21.15.32:0/3886283833 shutdown_connections 2024-04-06T14:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.467+0000 7f9086f12700 1 --2- 172.21.15.32:0/3886283833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 0x7f90800fd3f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.467+0000 7f9086f12700 1 --2- 172.21.15.32:0/3886283833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90800fbcd0 0x7f90800fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.467+0000 7f9086f12700 1 -- 172.21.15.32:0/3886283833 >> 172.21.15.32:0/3886283833 conn(0x7f90800f7430 msgr2=0x7f90800f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.467+0000 7f9086f12700 1 -- 172.21.15.32:0/3886283833 shutdown_connections 2024-04-06T14:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.467+0000 7f9086f12700 1 -- 172.21.15.32:0/3886283833 wait complete. 2024-04-06T14:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f9086f12700 1 Processor -- start 2024-04-06T14:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f9086f12700 1 -- start start 2024-04-06T14:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f9086f12700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90800fbcd0 0x7f908010d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f9086f12700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 0x7f908010df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f9086f12700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908010e500 con 0x7f90800fcf80 2024-04-06T14:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f9086f12700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908010ed00 con 0x7f90800fbcd0 2024-04-06T14:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f907ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 0x7f908010df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f907ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 0x7f908010df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44888/0 (socket says 172.21.15.32:44888) 2024-04-06T14:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f907ffff700 1 -- 172.21.15.32:0/3716158448 learned_addr learned my addr 172.21.15.32:0/3716158448 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f907ffff700 1 -- 172.21.15.32:0/3716158448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90800fbcd0 msgr2=0x7f908010d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f907ffff700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90800fbcd0 0x7f908010d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f907ffff700 1 -- 172.21.15.32:0/3716158448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c005040 con 0x7f90800fcf80 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.468+0000 7f907ffff700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 0x7f908010df00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f906c009fd0 tx=0x7f906c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.469+0000 7f907d7fa700 1 -- 172.21.15.32:0/3716158448 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f906c00e040 con 0x7f90800fcf80 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.469+0000 7f907d7fa700 1 -- 172.21.15.32:0/3716158448 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f906c004290 con 0x7f90800fcf80 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.469+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908010ef80 con 0x7f90800fcf80 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.469+0000 7f907d7fa700 1 -- 172.21.15.32:0/3716158448 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f906c01c3f0 con 0x7f90800fcf80 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.469+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908010f4c0 con 0x7f90800fcf80 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.470+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9080101b10 con 0x7f90800fcf80 2024-04-06T14:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.470+0000 7f907d7fa700 1 -- 172.21.15.32:0/3716158448 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f906c00bc50 con 0x7f90800fcf80 2024-04-06T14:13:41.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.470+0000 7f907d7fa700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f90700692b0 0x7f907006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:41.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.471+0000 7f907d7fa700 1 -- 172.21.15.32:0/3716158448 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f906c088270 con 0x7f90800fcf80 2024-04-06T14:13:41.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.471+0000 7f9084cae700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f90700692b0 0x7f907006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:41.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.471+0000 7f9084cae700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f90700692b0 0x7f907006b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9074004ad0 tx=0x7f9074007d40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:41.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.475+0000 7f907d7fa700 1 -- 172.21.15.32:0/3716158448 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f906c058410 con 0x7f90800fcf80 2024-04-06T14:13:41.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.667+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f908005f5f0 con 0x7f90800fcf80 2024-04-06T14:13:41.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.671+0000 7f907d7fa700 1 -- 172.21.15.32:0/3716158448 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f906c057fa0 con 0x7f90800fcf80 2024-04-06T14:13:41.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f90700692b0 msgr2=0x7f907006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:41.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f90700692b0 0x7f907006b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9074004ad0 tx=0x7f9074007d40 comp rx=0 tx=0).stop 2024-04-06T14:13:41.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 msgr2=0x7f908010df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:41.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 0x7f908010df00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f906c009fd0 tx=0x7f906c009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:41.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 shutdown_connections 2024-04-06T14:13:41.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f90700692b0 0x7f907006b760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:41.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90800fbcd0 0x7f908010d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:41.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 --2- 172.21.15.32:0/3716158448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90800fcf80 0x7f908010df00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:41.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 >> 172.21.15.32:0/3716158448 conn(0x7f90800f7430 msgr2=0x7f9080065c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:41.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 shutdown_connections 2024-04-06T14:13:41.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:41.674+0000 7f9086f12700 1 -- 172.21.15.32:0/3716158448 wait complete. 2024-04-06T14:13:42.423 INFO:teuthology.run_tasks:Running task print... 2024-04-06T14:13:42.432 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-06T14:13:42.432 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T14:13:42.460 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:13:42.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-06T14:13:42.742 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:42 smithi032 ceph-mon[39730]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:42 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3716158448' entity='client.admin' 2024-04-06T14:13:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:13:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:13:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:13:42.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:13:42.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:13:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:42 smithi086 ceph-mon[47925]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:43.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:42 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3716158448' entity='client.admin' 2024-04-06T14:13:43.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:13:43.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:13:43.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:13:43.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:13:43.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:13:44.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.652+0000 7f3fe983c700 1 -- 172.21.15.32:0/4102334520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fbcd0 msgr2=0x7f3fe40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:44.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.652+0000 7f3fe983c700 1 --2- 172.21.15.32:0/4102334520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fbcd0 0x7f3fe40fc0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4004940 tx=0x7f3fd4009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:44.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.652+0000 7f3fe983c700 1 -- 172.21.15.32:0/4102334520 shutdown_connections 2024-04-06T14:13:44.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.652+0000 7f3fe983c700 1 --2- 172.21.15.32:0/4102334520 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fe40fcf80 0x7f3fe40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:44.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.652+0000 7f3fe983c700 1 --2- 172.21.15.32:0/4102334520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fbcd0 0x7f3fe40fc0e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:44.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.652+0000 7f3fe983c700 1 -- 172.21.15.32:0/4102334520 >> 172.21.15.32:0/4102334520 conn(0x7f3fe40f7450 msgr2=0x7f3fe40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:44.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.652+0000 7f3fe983c700 1 -- 172.21.15.32:0/4102334520 shutdown_connections 2024-04-06T14:13:44.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.652+0000 7f3fe983c700 1 -- 172.21.15.32:0/4102334520 wait complete. 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe983c700 1 Processor -- start 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe983c700 1 -- start start 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe983c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fe40fbcd0 0x7f3fe410f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe983c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fcf80 0x7f3fe410ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe983c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4110500 con 0x7f3fe40fcf80 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe983c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4110670 con 0x7f3fe40fbcd0 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fcf80 0x7f3fe410ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fcf80 0x7f3fe410ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45218/0 (socket says 172.21.15.32:45218) 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe27fc700 1 -- 172.21.15.32:0/3680756255 learned_addr learned my addr 172.21.15.32:0/3680756255 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe2ffd700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fe40fbcd0 0x7f3fe410f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe27fc700 1 -- 172.21.15.32:0/3680756255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fe40fbcd0 msgr2=0x7f3fe410f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:44.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe27fc700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fe40fbcd0 0x7f3fe410f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:44.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.653+0000 7f3fe27fc700 1 -- 172.21.15.32:0/3680756255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd4005040 con 0x7f3fe40fcf80 2024-04-06T14:13:44.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.654+0000 7f3fe27fc700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fcf80 0x7f3fe410ff00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00eb40 tx=0x7f3fcc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:44.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.654+0000 7f3fdb7fe700 1 -- 172.21.15.32:0/3680756255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fcc007cc0 con 0x7f3fe40fcf80 2024-04-06T14:13:44.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.654+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe4115070 con 0x7f3fe40fcf80 2024-04-06T14:13:44.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.654+0000 7f3fdb7fe700 1 -- 172.21.15.32:0/3680756255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fcc00f040 con 0x7f3fe40fcf80 2024-04-06T14:13:44.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.654+0000 7f3fdb7fe700 1 -- 172.21.15.32:0/3680756255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fcc00c990 con 0x7f3fe40fcf80 2024-04-06T14:13:44.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.654+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe4115630 con 0x7f3fe40fcf80 2024-04-06T14:13:44.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.655+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe4101910 con 0x7f3fe40fcf80 2024-04-06T14:13:44.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.655+0000 7f3fdb7fe700 1 -- 172.21.15.32:0/3680756255 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3fcc00cc00 con 0x7f3fe40fcf80 2024-04-06T14:13:44.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.656+0000 7f3fdb7fe700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3fd00692a0 0x7f3fd006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:44.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.656+0000 7f3fe2ffd700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3fd00692a0 0x7f3fd006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:44.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.656+0000 7f3fdb7fe700 1 -- 172.21.15.32:0/3680756255 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3fcc086a10 con 0x7f3fe40fcf80 2024-04-06T14:13:44.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.656+0000 7f3fe2ffd700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3fd00692a0 0x7f3fd006b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4004890 tx=0x7f3fd4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:44.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.660+0000 7f3fdb7fe700 1 -- 172.21.15.32:0/3680756255 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3fcc04e190 con 0x7f3fe40fcf80 2024-04-06T14:13:44.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.871+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fe402b340 con 0x7f3fd00692a0 2024-04-06T14:13:44.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.872+0000 7f3fdb7fe700 1 -- 172.21.15.32:0/3680756255 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3fe402b340 con 0x7f3fd00692a0 2024-04-06T14:13:44.873 INFO:teuthology.orchestra.run.smithi032.stdout:Backend: cephadm 2024-04-06T14:13:44.873 INFO:teuthology.orchestra.run.smithi032.stdout:Available: Yes 2024-04-06T14:13:44.873 INFO:teuthology.orchestra.run.smithi032.stdout:Paused: No 2024-04-06T14:13:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.874+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3fd00692a0 msgr2=0x7f3fd006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.874+0000 7f3fe983c700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3fd00692a0 0x7f3fd006b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4004890 tx=0x7f3fd4009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.874+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fcf80 msgr2=0x7f3fe410ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.874+0000 7f3fe983c700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fcf80 0x7f3fe410ff00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00eb40 tx=0x7f3fcc00b040 comp rx=0 tx=0).stop 2024-04-06T14:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.874+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 shutdown_connections 2024-04-06T14:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.874+0000 7f3fe983c700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3fd00692a0 0x7f3fd006b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.874+0000 7f3fe983c700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fe40fbcd0 0x7f3fe410f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.874+0000 7f3fe983c700 1 --2- 172.21.15.32:0/3680756255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe40fcf80 0x7f3fe410ff00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.874+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 >> 172.21.15.32:0/3680756255 conn(0x7f3fe40f7450 msgr2=0x7f3fe4066850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.875+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 shutdown_connections 2024-04-06T14:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:44.875+0000 7f3fe983c700 1 -- 172.21.15.32:0/3680756255 wait complete. 2024-04-06T14:13:44.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:44 smithi032 ceph-mon[39730]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:45.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:44 smithi086 ceph-mon[47925]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:45.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-06T14:13:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:45.949 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:45 smithi086 ceph-mon[47925]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:45 smithi032 ceph-mon[39730]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:46.952 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:46 smithi032 ceph-mon[39730]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:46 smithi086 ceph-mon[47925]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:47.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.804+0000 7f33d5767700 1 -- 172.21.15.32:0/1325516835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d0066e70 msgr2=0x7f33d00ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:47.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.804+0000 7f33d5767700 1 --2- 172.21.15.32:0/1325516835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d0066e70 0x7f33d00ffa40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f33c4004940 tx=0x7f33c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:47.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.804+0000 7f33d5767700 1 -- 172.21.15.32:0/1325516835 shutdown_connections 2024-04-06T14:13:47.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.804+0000 7f33d5767700 1 --2- 172.21.15.32:0/1325516835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d00fff80 0x7f33d0102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:47.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.804+0000 7f33d5767700 1 --2- 172.21.15.32:0/1325516835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d0066e70 0x7f33d00ffa40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:47.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.804+0000 7f33d5767700 1 -- 172.21.15.32:0/1325516835 >> 172.21.15.32:0/1325516835 conn(0x7f33d00f7430 msgr2=0x7f33d00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:47.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.804+0000 7f33d5767700 1 -- 172.21.15.32:0/1325516835 shutdown_connections 2024-04-06T14:13:47.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.804+0000 7f33d5767700 1 -- 172.21.15.32:0/1325516835 wait complete. 2024-04-06T14:13:47.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.804+0000 7f33d5767700 1 Processor -- start 2024-04-06T14:13:47.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33d5767700 1 -- start start 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33d5767700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d0066e70 0x7f33d010d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33d5767700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d00fff80 0x7f33d010dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33d5767700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d010e300 con 0x7f33d00fff80 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33d5767700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d010e470 con 0x7f33d0066e70 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33ce7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d00fff80 0x7f33d010dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33ce7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d00fff80 0x7f33d010dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45234/0 (socket says 172.21.15.32:45234) 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33ce7fc700 1 -- 172.21.15.32:0/1827585610 learned_addr learned my addr 172.21.15.32:0/1827585610 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33ceffd700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d0066e70 0x7f33d010d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33ce7fc700 1 -- 172.21.15.32:0/1827585610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d0066e70 msgr2=0x7f33d010d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33ce7fc700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d0066e70 0x7f33d010d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33ce7fc700 1 -- 172.21.15.32:0/1827585610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c4005040 con 0x7f33d00fff80 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33ce7fc700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d00fff80 0x7f33d010dd00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f33c000eb40 tx=0x7f33c000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.805+0000 7f33b77fe700 1 -- 172.21.15.32:0/1827585610 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33c0007cc0 con 0x7f33d00fff80 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.806+0000 7f33b77fe700 1 -- 172.21.15.32:0/1827585610 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33c000f040 con 0x7f33d00fff80 2024-04-06T14:13:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.806+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d0112e70 con 0x7f33d00fff80 2024-04-06T14:13:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.806+0000 7f33b77fe700 1 -- 172.21.15.32:0/1827585610 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33c000c990 con 0x7f33d00fff80 2024-04-06T14:13:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.806+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d0113400 con 0x7f33d00fff80 2024-04-06T14:13:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.807+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d004c710 con 0x7f33d00fff80 2024-04-06T14:13:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.807+0000 7f33b77fe700 1 -- 172.21.15.32:0/1827585610 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f33c000caf0 con 0x7f33d00fff80 2024-04-06T14:13:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.807+0000 7f33b77fe700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f33b80692a0 0x7f33b806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.808+0000 7f33b77fe700 1 -- 172.21.15.32:0/1827585610 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f33c0086960 con 0x7f33d00fff80 2024-04-06T14:13:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.808+0000 7f33ceffd700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f33b80692a0 0x7f33b806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.808+0000 7f33ceffd700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f33b80692a0 0x7f33b806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33c4013680 tx=0x7f33c4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:47.812+0000 7f33b77fe700 1 -- 172.21.15.32:0/1827585610 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f33c004e190 con 0x7f33d00fff80 2024-04-06T14:13:48.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.015+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f33d00f8f70 con 0x7f33b80692a0 2024-04-06T14:13:48.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.024+0000 7f33b77fe700 1 -- 172.21.15.32:0/1827585610 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7f33d00f8f70 con 0x7f33b80692a0 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (4m) 2m ago 5m 26.8M - 0.25.0 c8568f914cd2 d6f57d3d10f9 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (5m) 2m ago 5m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (4m) 72s ago 4m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (3m) 2m ago 5m 80.1M - 9.4.7 954c08fa6188 1cc1d256b791 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:9283 running (6m) 2m ago 6m 473M - 17.2.7-819-g6699495c ec300eaa768b 7737c4037eba 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283 running (4m) 72s ago 4m 431M - 17.2.7-819-g6699495c ec300eaa768b b0b7b2c2f6da 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (6m) 2m ago 6m 43.6M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (4m) 72s ago 4m 38.6M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (5m) 2m ago 5m 15.7M - 1.5.0 0da6a335fe13 41a44c41cd57 2024-04-06T14:13:48.024 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (4m) 72s ago 4m 18.0M - 1.5.0 0da6a335fe13 21ac74498650 2024-04-06T14:13:48.025 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (3m) 2m ago 3m 37.8M 4300M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:13:48.025 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (3m) 2m ago 3m 39.5M 4300M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:13:48.025 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (2m) 2m ago 2m 42.4M 4300M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:13:48.025 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (2m) 72s ago 2m 41.8M 5323M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:13:48.025 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (104s) 72s ago 104s 39.3M 5323M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:13:48.025 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (78s) 72s ago 77s 10.7M 5323M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:13:48.025 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (3m) 2m ago 4m 35.9M - 2.43.0 a07b618ecd1d 6742cb6c38a9 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.025+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f33b80692a0 msgr2=0x7f33b806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.025+0000 7f33d5767700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f33b80692a0 0x7f33b806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33c4013680 tx=0x7f33c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.025+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d00fff80 msgr2=0x7f33d010dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.025+0000 7f33d5767700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d00fff80 0x7f33d010dd00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f33c000eb40 tx=0x7f33c000b040 comp rx=0 tx=0).stop 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.026+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 shutdown_connections 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.026+0000 7f33d5767700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f33b80692a0 0x7f33b806b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.026+0000 7f33d5767700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d0066e70 0x7f33d010d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.026+0000 7f33d5767700 1 --2- 172.21.15.32:0/1827585610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33d00fff80 0x7f33d010dd00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.026+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 >> 172.21.15.32:0/1827585610 conn(0x7f33d00f7430 msgr2=0x7f33d00f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.026+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 shutdown_connections 2024-04-06T14:13:48.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:48.026+0000 7f33d5767700 1 -- 172.21.15.32:0/1827585610 wait complete. 2024-04-06T14:13:48.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-06T14:13:48.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:48 smithi032 ceph-mon[39730]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:49.087 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:49.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:48 smithi086 ceph-mon[47925]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:49.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:49 smithi032 ceph-mon[39730]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:50.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:49 smithi086 ceph-mon[47925]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:50.705 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:50 smithi032 ceph-mon[39730]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.998+0000 7fd9c4148700 1 -- 172.21.15.32:0/2019923347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 msgr2=0x7fd9bc0fd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.998+0000 7fd9c4148700 1 --2- 172.21.15.32:0/2019923347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 0x7fd9bc0fd300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8004940 tx=0x7fd9b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.998+0000 7fd9c4148700 1 -- 172.21.15.32:0/2019923347 shutdown_connections 2024-04-06T14:13:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.998+0000 7fd9c4148700 1 --2- 172.21.15.32:0/2019923347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9bc0fe060 0x7fd9bc0fe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.998+0000 7fd9c4148700 1 --2- 172.21.15.32:0/2019923347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 0x7fd9bc0fd300 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.998+0000 7fd9c4148700 1 -- 172.21.15.32:0/2019923347 >> 172.21.15.32:0/2019923347 conn(0x7fd9bc0f85b0 msgr2=0x7fd9bc0fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.999+0000 7fd9c4148700 1 -- 172.21.15.32:0/2019923347 shutdown_connections 2024-04-06T14:13:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.999+0000 7fd9c4148700 1 -- 172.21.15.32:0/2019923347 wait complete. 2024-04-06T14:13:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.999+0000 7fd9c4148700 1 Processor -- start 2024-04-06T14:13:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.999+0000 7fd9c4148700 1 -- start start 2024-04-06T14:13:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.999+0000 7fd9c4148700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 0x7fd9bc10d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:50.999+0000 7fd9c4148700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9bc0fe060 0x7fd9bc10de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c4148700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc10e450 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c4148700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc10eca0 con 0x7fd9bc0fe060 2024-04-06T14:13:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c1ee4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 0x7fd9bc10d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c1ee4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 0x7fd9bc10d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45250/0 (socket says 172.21.15.32:45250) 2024-04-06T14:13:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c1ee4700 1 -- 172.21.15.32:0/803210244 learned_addr learned my addr 172.21.15.32:0/803210244 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c16e3700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9bc0fe060 0x7fd9bc10de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c1ee4700 1 -- 172.21.15.32:0/803210244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9bc0fe060 msgr2=0x7fd9bc10de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c1ee4700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9bc0fe060 0x7fd9bc10de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c1ee4700 1 -- 172.21.15.32:0/803210244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b8005040 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c1ee4700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 0x7fd9bc10d910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009380 tx=0x7fd9b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9b27fc700 1 -- 172.21.15.32:0/803210244 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9b800e040 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9b27fc700 1 -- 172.21.15.32:0/803210244 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9b800bd20 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc10ef20 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9b27fc700 1 -- 172.21.15.32:0/803210244 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9b801c630 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.000+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc10f460 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.001+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc05f5f0 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.002+0000 7fd9b27fc700 1 -- 172.21.15.32:0/803210244 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd9b8013420 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.002+0000 7fd9b27fc700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd9a80692a0 0x7fd9a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.003+0000 7fd9b27fc700 1 -- 172.21.15.32:0/803210244 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd9b8088210 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.003+0000 7fd9c16e3700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd9a80692a0 0x7fd9a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.003+0000 7fd9c16e3700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd9a80692a0 0x7fd9a806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac007fd0 tx=0x7fd9ac009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:51.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.006+0000 7fd9b27fc700 1 -- 172.21.15.32:0/803210244 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd9b80583b0 con 0x7fd9bc0fcef0 2024-04-06T14:13:51.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:50 smithi086 ceph-mon[47925]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:51.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.206+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9bc10f790 con 0x7fd9a80692a0 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.210+0000 7fd9b27fc700 1 -- 172.21.15.32:0/803210244 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1161 (secure 0 0 0) 0x7fd9bc10f790 con 0x7fd9a80692a0 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:crash 2/2 2m ago 6m * 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:mgr 2/2 2m ago 6m count:2 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:mon 2/2 2m ago 5m smithi032:172.21.15.32=smithi032;smithi086:172.21.15.86=smithi086;count:2 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:osd 6 2m ago - 2024-04-06T14:13:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-06T14:13:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd9a80692a0 msgr2=0x7fd9a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd9a80692a0 0x7fd9a806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac007fd0 tx=0x7fd9ac009040 comp rx=0 tx=0).stop 2024-04-06T14:13:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 msgr2=0x7fd9bc10d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 0x7fd9bc10d910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009380 tx=0x7fd9b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 shutdown_connections 2024-04-06T14:13:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd9a80692a0 0x7fd9a806b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9bc0fcef0 0x7fd9bc10d910 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 --2- 172.21.15.32:0/803210244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9bc0fe060 0x7fd9bc10de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 >> 172.21.15.32:0/803210244 conn(0x7fd9bc0f85b0 msgr2=0x7fd9bc0f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 shutdown_connections 2024-04-06T14:13:51.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:51.212+0000 7fd9c4148700 1 -- 172.21.15.32:0/803210244 wait complete. 2024-04-06T14:13:51.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-06T14:13:52.231 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:52 smithi032 ceph-mon[39730]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:52 smithi032 ceph-mon[39730]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:53.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:52 smithi086 ceph-mon[47925]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:53.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:52 smithi086 ceph-mon[47925]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:54.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.214+0000 7f97415ce700 1 -- 172.21.15.32:0/1730955123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 msgr2=0x7f973c0fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:54.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.214+0000 7f97415ce700 1 --2- 172.21.15.32:0/1730955123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 0x7f973c0fd280 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f972c004940 tx=0x7f972c009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.214+0000 7f97415ce700 1 -- 172.21.15.32:0/1730955123 shutdown_connections 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.214+0000 7f97415ce700 1 --2- 172.21.15.32:0/1730955123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f973c0fdfe0 0x7f973c0fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.214+0000 7f97415ce700 1 --2- 172.21.15.32:0/1730955123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 0x7f973c0fd280 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.214+0000 7f97415ce700 1 -- 172.21.15.32:0/1730955123 >> 172.21.15.32:0/1730955123 conn(0x7f973c0f8550 msgr2=0x7f973c0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.214+0000 7f97415ce700 1 -- 172.21.15.32:0/1730955123 shutdown_connections 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.214+0000 7f97415ce700 1 -- 172.21.15.32:0/1730955123 wait complete. 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.215+0000 7f97415ce700 1 Processor -- start 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.215+0000 7f97415ce700 1 -- start start 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.215+0000 7f97415ce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 0x7f973c10d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.215+0000 7f97415ce700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f973c0fdfe0 0x7f973c10ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.215+0000 7f97415ce700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c10e3c0 con 0x7f973c0fce70 2024-04-06T14:13:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.215+0000 7f97415ce700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c10e530 con 0x7f973c0fdfe0 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f973affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 0x7f973c10d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f973affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 0x7f973c10d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45274/0 (socket says 172.21.15.32:45274) 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f973affd700 1 -- 172.21.15.32:0/4049996193 learned_addr learned my addr 172.21.15.32:0/4049996193 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f973a7fc700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f973c0fdfe0 0x7f973c10ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f973affd700 1 -- 172.21.15.32:0/4049996193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f973c0fdfe0 msgr2=0x7f973c10ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f973affd700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f973c0fdfe0 0x7f973c10ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f973affd700 1 -- 172.21.15.32:0/4049996193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c005040 con 0x7f973c0fce70 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f973affd700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 0x7f973c10d880 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f972c00b3b0 tx=0x7f972c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f97337fe700 1 -- 172.21.15.32:0/4049996193 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f972c00e040 con 0x7f973c0fce70 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f97337fe700 1 -- 172.21.15.32:0/4049996193 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f972c004290 con 0x7f973c0fce70 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f97337fe700 1 -- 172.21.15.32:0/4049996193 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f972c01c3f0 con 0x7f973c0fce70 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c112ee0 con 0x7f973c0fce70 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.216+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c1133c0 con 0x7f973c0fce70 2024-04-06T14:13:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.217+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973c04c710 con 0x7f973c0fce70 2024-04-06T14:13:54.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.218+0000 7f97337fe700 1 -- 172.21.15.32:0/4049996193 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f972c013420 con 0x7f973c0fce70 2024-04-06T14:13:54.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.218+0000 7f97337fe700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f97280692a0 0x7f972806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:54.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.218+0000 7f97337fe700 1 -- 172.21.15.32:0/4049996193 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f972c0881c0 con 0x7f973c0fce70 2024-04-06T14:13:54.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.218+0000 7f973a7fc700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f97280692a0 0x7f972806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:54.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.219+0000 7f973a7fc700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f97280692a0 0x7f972806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9724004990 tx=0x7f9724009ee0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:54.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.223+0000 7f97337fe700 1 -- 172.21.15.32:0/4049996193 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f972c0582c0 con 0x7f973c0fce70 2024-04-06T14:13:54.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.424+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f973c113630 con 0x7f97280692a0 2024-04-06T14:13:54.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.426+0000 7f97337fe700 1 -- 172.21.15.32:0/4049996193 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f973c113630 con 0x7f97280692a0 2024-04-06T14:13:54.426 INFO:teuthology.orchestra.run.smithi032.stdout:HOST ADDR LABELS STATUS 2024-04-06T14:13:54.426 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 172.21.15.32 2024-04-06T14:13:54.426 INFO:teuthology.orchestra.run.smithi032.stdout:smithi086 172.21.15.86 2024-04-06T14:13:54.426 INFO:teuthology.orchestra.run.smithi032.stdout:2 hosts in cluster 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f97280692a0 msgr2=0x7f972806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f97280692a0 0x7f972806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9724004990 tx=0x7f9724009ee0 comp rx=0 tx=0).stop 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 msgr2=0x7f973c10d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 0x7f973c10d880 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f972c00b3b0 tx=0x7f972c009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 shutdown_connections 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f97280692a0 0x7f972806b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f973c0fce70 0x7f973c10d880 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 --2- 172.21.15.32:0/4049996193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f973c0fdfe0 0x7f973c10ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 >> 172.21.15.32:0/4049996193 conn(0x7f973c0f8550 msgr2=0x7f973c0f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 shutdown_connections 2024-04-06T14:13:54.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:54.428+0000 7f97415ce700 1 -- 172.21.15.32:0/4049996193 wait complete. 2024-04-06T14:13:54.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:54 smithi032 ceph-mon[39730]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:55.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-06T14:13:55.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:54 smithi086 ceph-mon[47925]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:55.856 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:55 smithi032 ceph-mon[39730]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:56.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:55 smithi086 ceph-mon[47925]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:56.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:56 smithi032 ceph-mon[39730]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:57.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:56 smithi086 ceph-mon[47925]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 -- 172.21.15.32:0/3191342414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fe060 msgr2=0x7f70840fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 --2- 172.21.15.32:0/3191342414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fe060 0x7f70840fe4d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7078004940 tx=0x7f7078009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 -- 172.21.15.32:0/3191342414 shutdown_connections 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 --2- 172.21.15.32:0/3191342414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fe060 0x7f70840fe4d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 --2- 172.21.15.32:0/3191342414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70840fcef0 0x7f70840fd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 -- 172.21.15.32:0/3191342414 >> 172.21.15.32:0/3191342414 conn(0x7f70840f85b0 msgr2=0x7f70840fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 -- 172.21.15.32:0/3191342414 shutdown_connections 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 -- 172.21.15.32:0/3191342414 wait complete. 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 Processor -- start 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.152+0000 7f7088d95700 1 -- start start 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7088d95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fcef0 0x7f7084071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7088d95700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70840fe060 0x7f7084070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7088d95700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70840708c0 con 0x7f70840fcef0 2024-04-06T14:13:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7088d95700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7084070a30 con 0x7f70840fe060 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f708259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fcef0 0x7f7084071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f708259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fcef0 0x7f7084071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42294/0 (socket says 172.21.15.32:42294) 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f708259c700 1 -- 172.21.15.32:0/1355202390 learned_addr learned my addr 172.21.15.32:0/1355202390 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7081d9b700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70840fe060 0x7f7084070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f708259c700 1 -- 172.21.15.32:0/1355202390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70840fe060 msgr2=0x7f7084070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f708259c700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70840fe060 0x7f7084070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f708259c700 1 -- 172.21.15.32:0/1355202390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078005040 con 0x7f70840fcef0 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f708259c700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fcef0 0x7f7084071e40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f707400c830 tx=0x7f7074004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7072ffd700 1 -- 172.21.15.32:0/1355202390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70740106f0 con 0x7f70840fcef0 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7084070d10 con 0x7f70840fcef0 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7072ffd700 1 -- 172.21.15.32:0/1355202390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7074010d00 con 0x7f70840fcef0 2024-04-06T14:13:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7072ffd700 1 -- 172.21.15.32:0/1355202390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7074012e90 con 0x7f70840fcef0 2024-04-06T14:13:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.153+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7084115970 con 0x7f70840fcef0 2024-04-06T14:13:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.154+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708405f5f0 con 0x7f70840fcef0 2024-04-06T14:13:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.155+0000 7f7072ffd700 1 -- 172.21.15.32:0/1355202390 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7074010850 con 0x7f70840fcef0 2024-04-06T14:13:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.155+0000 7f7072ffd700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f706c0692b0 0x7f706c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:13:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.155+0000 7f7072ffd700 1 -- 172.21.15.32:0/1355202390 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7074086e50 con 0x7f70840fcef0 2024-04-06T14:13:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.155+0000 7f7081d9b700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f706c0692b0 0x7f706c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:13:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.156+0000 7f7081d9b700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f706c0692b0 0x7f706c06b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7078004f90 tx=0x7f7078009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:13:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.160+0000 7f7072ffd700 1 -- 172.21.15.32:0/1355202390 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f70740534c0 con 0x7f70840fcef0 2024-04-06T14:13:57.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.363+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7084102440 con 0x7f706c0692b0 2024-04-06T14:13:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.365+0000 7f7072ffd700 1 -- 172.21.15.32:0/1355202390 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f7084102440 con 0x7f706c0692b0 2024-04-06T14:13:57.365 INFO:teuthology.orchestra.run.smithi032.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-06T14:13:57.365 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747402DN480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-04-06T14:13:57.365 INFO:teuthology.orchestra.run.smithi032.stdout:smithi086 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UL400BGN 372G No 78s ago Has a FileSystem, LVM detected 2024-04-06T14:13:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.366+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f706c0692b0 msgr2=0x7f706c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.366+0000 7f7088d95700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f706c0692b0 0x7f706c06b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7078004f90 tx=0x7f7078009e30 comp rx=0 tx=0).stop 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.366+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fcef0 msgr2=0x7f7084071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.366+0000 7f7088d95700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fcef0 0x7f7084071e40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f707400c830 tx=0x7f7074004470 comp rx=0 tx=0).stop 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.367+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 shutdown_connections 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.367+0000 7f7088d95700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f706c0692b0 0x7f706c06b760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.367+0000 7f7088d95700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70840fcef0 0x7f7084071e40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.367+0000 7f7088d95700 1 --2- 172.21.15.32:0/1355202390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70840fe060 0x7f7084070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.367+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 >> 172.21.15.32:0/1355202390 conn(0x7f70840f85b0 msgr2=0x7f70840f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.367+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 shutdown_connections 2024-04-06T14:13:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:13:57.367+0000 7f7088d95700 1 -- 172.21.15.32:0/1355202390 wait complete. 2024-04-06T14:13:57.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:13:58.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:57 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:13:58.171 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T14:13:58.181 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:13:58.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-06T14:13:58.464 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:13:58.978 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:58 smithi032 ceph-mon[39730]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:58.978 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:13:58 smithi032 ceph-mon[39730]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:13:59.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:58 smithi086 ceph-mon[47925]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:13:59.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:13:58 smithi086 ceph-mon[47925]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.315+0000 7fe2932fc700 1 -- 172.21.15.32:0/3346308989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0662e0 msgr2=0x7fe28c0ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.315+0000 7fe2932fc700 1 --2- 172.21.15.32:0/3346308989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0662e0 0x7fe28c0ff9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe27c004940 tx=0x7fe27c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.315+0000 7fe2932fc700 1 -- 172.21.15.32:0/3346308989 shutdown_connections 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.315+0000 7fe2932fc700 1 --2- 172.21.15.32:0/3346308989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe28c0fff00 0x7fe28c102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.315+0000 7fe2932fc700 1 --2- 172.21.15.32:0/3346308989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0662e0 0x7fe28c0ff9c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.315+0000 7fe2932fc700 1 -- 172.21.15.32:0/3346308989 >> 172.21.15.32:0/3346308989 conn(0x7fe28c0f7430 msgr2=0x7fe28c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.315+0000 7fe2932fc700 1 -- 172.21.15.32:0/3346308989 shutdown_connections 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.315+0000 7fe2932fc700 1 -- 172.21.15.32:0/3346308989 wait complete. 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe2932fc700 1 Processor -- start 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe2932fc700 1 -- start start 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe2932fc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe28c0662e0 0x7fe28c10d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe2932fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0fff00 0x7fe28c10de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe2932fc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c10e470 con 0x7fe28c0fff00 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe2932fc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c10ecc0 con 0x7fe28c0662e0 2024-04-06T14:14:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe290897700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0fff00 0x7fe28c10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe290897700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0fff00 0x7fe28c10de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42312/0 (socket says 172.21.15.32:42312) 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe290897700 1 -- 172.21.15.32:0/3715612795 learned_addr learned my addr 172.21.15.32:0/3715612795 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe291098700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe28c0662e0 0x7fe28c10d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe290897700 1 -- 172.21.15.32:0/3715612795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe28c0662e0 msgr2=0x7fe28c10d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe290897700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe28c0662e0 0x7fe28c10d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe290897700 1 -- 172.21.15.32:0/3715612795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c005040 con 0x7fe28c0fff00 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.316+0000 7fe290897700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0fff00 0x7fe28c10de70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe288002a50 tx=0x7fe2880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.317+0000 7fe281ffb700 1 -- 172.21.15.32:0/3715612795 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe288007cc0 con 0x7fe28c0fff00 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.317+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c10efa0 con 0x7fe28c0fff00 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.317+0000 7fe281ffb700 1 -- 172.21.15.32:0/3715612795 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe28800f040 con 0x7fe28c0fff00 2024-04-06T14:14:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.317+0000 7fe281ffb700 1 -- 172.21.15.32:0/3715612795 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe28800d9c0 con 0x7fe28c0fff00 2024-04-06T14:14:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.317+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c10f4e0 con 0x7fe28c0fff00 2024-04-06T14:14:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.318+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c05f5f0 con 0x7fe28c0fff00 2024-04-06T14:14:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.318+0000 7fe281ffb700 1 -- 172.21.15.32:0/3715612795 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe28800db20 con 0x7fe28c0fff00 2024-04-06T14:14:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.319+0000 7fe281ffb700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe2780692a0 0x7fe27806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:00.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.319+0000 7fe281ffb700 1 -- 172.21.15.32:0/3715612795 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe288086b70 con 0x7fe28c0fff00 2024-04-06T14:14:00.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.319+0000 7fe291098700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe2780692a0 0x7fe27806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:00.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.319+0000 7fe291098700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe2780692a0 0x7fe27806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00b900 tx=0x7fe27c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:00.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.323+0000 7fe281ffb700 1 -- 172.21.15.32:0/3715612795 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe288056bf0 con 0x7fe28c0fff00 2024-04-06T14:14:00.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:00.548+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe28c10f810 con 0x7fe2780692a0 2024-04-06T14:14:01.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:00 smithi032 ceph-mon[39730]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:01.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:00 smithi086 ceph-mon[47925]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:02.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.076+0000 7fe281ffb700 1 -- 172.21.15.32:0/3715612795 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe28c10f810 con 0x7fe2780692a0 2024-04-06T14:14:02.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe2780692a0 msgr2=0x7fe27806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:02.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe2780692a0 0x7fe27806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00b900 tx=0x7fe27c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:02.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0fff00 msgr2=0x7fe28c10de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:02.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0fff00 0x7fe28c10de70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe288002a50 tx=0x7fe2880046b0 comp rx=0 tx=0).stop 2024-04-06T14:14:02.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 shutdown_connections 2024-04-06T14:14:02.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fe2780692a0 0x7fe27806b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:02.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe28c0662e0 0x7fe28c10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:02.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 --2- 172.21.15.32:0/3715612795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe28c0fff00 0x7fe28c10de70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:02.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 >> 172.21.15.32:0/3715612795 conn(0x7fe28c0f7430 msgr2=0x7fe28c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:02.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 shutdown_connections 2024-04-06T14:14:02.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:02.079+0000 7fe2932fc700 1 -- 172.21.15.32:0/3715612795 wait complete. 2024-04-06T14:14:02.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:01 smithi032 ceph-mon[39730]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:14:02.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:01 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-06T14:14:02.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:01 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-06T14:14:02.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:01 smithi032 ceph-mon[39730]: osdmap e42: 6 total, 6 up, 6 in 2024-04-06T14:14:02.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:01 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-06T14:14:02.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:01 smithi032 ceph-mon[39730]: pgmap v182: 33 pgs: 13 creating+peering, 1 active+clean, 19 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:02.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[39707]: 2024-04-06T14:14:02.043+0000 7fb0fe253700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T14:14:02.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:01 smithi086 ceph-mon[47925]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:14:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:01 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-06T14:14:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:01 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-06T14:14:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:01 smithi086 ceph-mon[47925]: osdmap e42: 6 total, 6 up, 6 in 2024-04-06T14:14:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:01 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-06T14:14:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:01 smithi086 ceph-mon[47925]: pgmap v182: 33 pgs: 13 creating+peering, 1 active+clean, 19 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:02.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-06T14:14:03.149 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T14:14:03.149 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: osdmap e43: 6 total, 6 up, 6 in 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: osdmap e44: 6 total, 6 up, 6 in 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: fsmap cephfs:0 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: Saving service mds.cephfs spec with placement count:4 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:03.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.gcjdjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:14:03.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.gcjdjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T14:14:03.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:03.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:02 smithi032 ceph-mon[39730]: Deploying daemon mds.cephfs.smithi032.gcjdjo on smithi032 2024-04-06T14:14:03.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: osdmap e43: 6 total, 6 up, 6 in 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: osdmap e44: 6 total, 6 up, 6 in 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: fsmap cephfs:0 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: Saving service mds.cephfs spec with placement count:4 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:14:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:03.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:14:03.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:03.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.gcjdjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:14:03.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.gcjdjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T14:14:03.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:03.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:02 smithi086 ceph-mon[47925]: Deploying daemon mds.cephfs.smithi032.gcjdjo on smithi032 2024-04-06T14:14:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:04.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:04 smithi032 ceph-mon[39730]: osdmap e45: 6 total, 6 up, 6 in 2024-04-06T14:14:04.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:04 smithi032 ceph-mon[39730]: pgmap v186: 65 pgs: 13 creating+peering, 29 active+clean, 23 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:04.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:04 smithi086 ceph-mon[47925]: osdmap e45: 6 total, 6 up, 6 in 2024-04-06T14:14:04.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:04 smithi086 ceph-mon[47925]: pgmap v186: 65 pgs: 13 creating+peering, 29 active+clean, 23 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:05.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:05 smithi032 ceph-mon[39730]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T14:14:05.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:05 smithi086 ceph-mon[47925]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T14:14:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.255+0000 7f835e848700 1 -- 172.21.15.32:0/2214549148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83580fee10 msgr2=0x7f83580faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.255+0000 7f835e848700 1 --2- 172.21.15.32:0/2214549148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83580fee10 0x7f83580faf50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f834c004940 tx=0x7f834c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.255+0000 7f835e848700 1 -- 172.21.15.32:0/2214549148 shutdown_connections 2024-04-06T14:14:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.255+0000 7f835e848700 1 --2- 172.21.15.32:0/2214549148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83580fee10 0x7f83580faf50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.255+0000 7f835e848700 1 --2- 172.21.15.32:0/2214549148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83580fe4c0 0x7f83580fe8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.255+0000 7f835e848700 1 -- 172.21.15.32:0/2214549148 >> 172.21.15.32:0/2214549148 conn(0x7f83580f6ba0 msgr2=0x7f83580f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.255+0000 7f835e848700 1 -- 172.21.15.32:0/2214549148 shutdown_connections 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.255+0000 7f835e848700 1 -- 172.21.15.32:0/2214549148 wait complete. 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.255+0000 7f835e848700 1 Processor -- start 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f835e848700 1 -- start start 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f835e848700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83580fe4c0 0x7f835810faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f835e848700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83580fee10 0x7f835810ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f835e848700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83581105e0 con 0x7f83580fe4c0 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f835e848700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8358110e30 con 0x7f83580fee10 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f8357fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83580fe4c0 0x7f835810faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f8357fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83580fe4c0 0x7f835810faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50424/0 (socket says 172.21.15.32:50424) 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f8357fff700 1 -- 172.21.15.32:0/2581359255 learned_addr learned my addr 172.21.15.32:0/2581359255 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f83577fe700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83580fee10 0x7f835810ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f8357fff700 1 -- 172.21.15.32:0/2581359255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83580fee10 msgr2=0x7f835810ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f8357fff700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83580fee10 0x7f835810ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f8357fff700 1 -- 172.21.15.32:0/2581359255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c005040 con 0x7f83580fe4c0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f8357fff700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83580fe4c0 0x7f835810faa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f834800eb40 tx=0x7f834800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.256+0000 7f8354ff9700 1 -- 172.21.15.32:0/2581359255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8348007cc0 con 0x7f83580fe4c0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.257+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83581110b0 con 0x7f83580fe4c0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.257+0000 7f8354ff9700 1 -- 172.21.15.32:0/2581359255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f834800f040 con 0x7f83580fe4c0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.257+0000 7f8354ff9700 1 -- 172.21.15.32:0/2581359255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f834800c990 con 0x7f83580fe4c0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.257+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83581115f0 con 0x7f83580fe4c0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.258+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835805f5f0 con 0x7f83580fe4c0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.258+0000 7f8354ff9700 1 -- 172.21.15.32:0/2581359255 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f834800cc00 con 0x7f83580fe4c0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.258+0000 7f8354ff9700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f83400692a0 0x7f834006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.259+0000 7f83577fe700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f83400692a0 0x7f834006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.259+0000 7f8354ff9700 1 -- 172.21.15.32:0/2581359255 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8348086de0 con 0x7f83580fe4c0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.259+0000 7f83577fe700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f83400692a0 0x7f834006b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f834c004890 tx=0x7f834c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.263+0000 7f8354ff9700 1 -- 172.21.15.32:0/2581359255 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f834804e190 con 0x7f83580fe4c0 2024-04-06T14:14:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.504+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8358111970 con 0x7f83580fe4c0 2024-04-06T14:14:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.505+0000 7f8354ff9700 1 -- 172.21.15.32:0/2581359255 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1247 (secure 0 0 0) 0x7f8348056bd0 con 0x7f83580fe4c0 2024-04-06T14:14:06.505 INFO:teuthology.orchestra.run.smithi032.stdout:e4 2024-04-06T14:14:06.505 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:14:06.505 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:14:06.505 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:14:06.505 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:14:06.505 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:14:06.505 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 4 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:14:06.062510+0000 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 0 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 1 2024-04-06T14:14:06.506 INFO:teuthology.orchestra.run.smithi032.stdout:in 0 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14444} 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 0 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{0:14444} state up:creating seq 1 addr [v2:172.21.15.32:6826/4235185528,v1:172.21.15.32:6827/4235185528] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:14:06.507 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:14:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f83400692a0 msgr2=0x7f834006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f83400692a0 0x7f834006b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f834c004890 tx=0x7f834c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83580fe4c0 msgr2=0x7f835810faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83580fe4c0 0x7f835810faa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f834800eb40 tx=0x7f834800b040 comp rx=0 tx=0).stop 2024-04-06T14:14:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 shutdown_connections 2024-04-06T14:14:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f83400692a0 0x7f834006b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83580fe4c0 0x7f835810faa0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 --2- 172.21.15.32:0/2581359255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83580fee10 0x7f835810ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 >> 172.21.15.32:0/2581359255 conn(0x7f83580f6ba0 msgr2=0x7f83580f82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 shutdown_connections 2024-04-06T14:14:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:06.507+0000 7f835e848700 1 -- 172.21.15.32:0/2581359255 wait complete. 2024-04-06T14:14:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 4 2024-04-06T14:14:06.609 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:06.609 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:06.609 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:06.609 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.nikfjj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:14:06.609 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.nikfjj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T14:14:06.609 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:06.609 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: Deploying daemon mds.cephfs.smithi086.nikfjj on smithi086 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: pgmap v187: 65 pgs: 13 creating+peering, 38 active+clean, 14 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.32:6826/4235185528,v1:172.21.15.32:6827/4235185528] up:boot 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: daemon mds.cephfs.smithi032.gcjdjo assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: Cluster is now healthy 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: fsmap cephfs:0 1 up:standby 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:creating} 2024-04-06T14:14:06.610 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:06 smithi086 ceph-mon[47925]: daemon mds.cephfs.smithi032.gcjdjo is now active in filesystem cephfs as rank 0 2024-04-06T14:14:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.nikfjj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:14:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.nikfjj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T14:14:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: Deploying daemon mds.cephfs.smithi086.nikfjj on smithi086 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: pgmap v187: 65 pgs: 13 creating+peering, 38 active+clean, 14 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.32:6826/4235185528,v1:172.21.15.32:6827/4235185528] up:boot 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: daemon mds.cephfs.smithi032.gcjdjo assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: Cluster is now healthy 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: fsmap cephfs:0 1 up:standby 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:creating} 2024-04-06T14:14:06.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:06 smithi032 ceph-mon[39730]: daemon mds.cephfs.smithi032.gcjdjo is now active in filesystem cephfs as rank 0 2024-04-06T14:14:07.216 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T14:14:07.226 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:14:07.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-06T14:14:07.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:07 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2581359255' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:14:07.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:07 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.32:6826/4235185528,v1:172.21.15.32:6827/4235185528] up:active 2024-04-06T14:14:07.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:07 smithi032 ceph-mon[39730]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:active} 2024-04-06T14:14:07.516 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:07.810 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:07 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2581359255' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:14:07.810 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:07 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.32:6826/4235185528,v1:172.21.15.32:6827/4235185528] up:active 2024-04-06T14:14:07.810 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:07 smithi086 ceph-mon[47925]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:active} 2024-04-06T14:14:09.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: pgmap v188: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-06T14:14:09.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:09.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:09.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:09.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.mqoaof", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:14:09.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.mqoaof", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T14:14:09.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:09.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] up:boot 2024-04-06T14:14:09.150 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:active} 1 up:standby 2024-04-06T14:14:09.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:14:09.151 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:08 smithi032 ceph-mon[39730]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:active} 1 up:standby 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: pgmap v188: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.mqoaof", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.mqoaof", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] up:boot 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:active} 1 up:standby 2024-04-06T14:14:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:14:09.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:08 smithi086 ceph-mon[47925]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:active} 1 up:standby 2024-04-06T14:14:09.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 -- 172.21.15.32:0/3529488730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fe070 msgr2=0x7fdca40fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:09.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 --2- 172.21.15.32:0/3529488730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fe070 0x7fdca40fe4e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c004940 tx=0x7fdc8c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:09.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 -- 172.21.15.32:0/3529488730 shutdown_connections 2024-04-06T14:14:09.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 --2- 172.21.15.32:0/3529488730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fe070 0x7fdca40fe4e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:09.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 --2- 172.21.15.32:0/3529488730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca40fcf00 0x7fdca40fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:09.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 -- 172.21.15.32:0/3529488730 >> 172.21.15.32:0/3529488730 conn(0x7fdca40f85a0 msgr2=0x7fdca40fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 -- 172.21.15.32:0/3529488730 shutdown_connections 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 -- 172.21.15.32:0/3529488730 wait complete. 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 Processor -- start 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 -- start start 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fcf00 0x7fdca4071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca40fe070 0x7fdca4070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca40708c0 con 0x7fdca40fcf00 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.750+0000 7fdca8da0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4070a30 con 0x7fdca40fe070 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fcf00 0x7fdca4071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fcf00 0x7fdca4071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50436/0 (socket says 172.21.15.32:50436) 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca259c700 1 -- 172.21.15.32:0/2718194368 learned_addr learned my addr 172.21.15.32:0/2718194368 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca1d9b700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca40fe070 0x7fdca4070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:09.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca259c700 1 -- 172.21.15.32:0/2718194368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca40fe070 msgr2=0x7fdca4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca259c700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca40fe070 0x7fdca4070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca259c700 1 -- 172.21.15.32:0/2718194368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c005040 con 0x7fdca40fcf00 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca259c700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fcf00 0x7fdca4071e40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc94002a50 tx=0x7fdc940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdc9affd700 1 -- 172.21.15.32:0/2718194368 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc94007cc0 con 0x7fdca40fcf00 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdc9affd700 1 -- 172.21.15.32:0/2718194368 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc9400f040 con 0x7fdca40fcf00 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca4070d10 con 0x7fdca40fcf00 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdc9affd700 1 -- 172.21.15.32:0/2718194368 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc9400d9c0 con 0x7fdca40fcf00 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.751+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca4115980 con 0x7fdca40fcf00 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.752+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca405f5f0 con 0x7fdca40fcf00 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.753+0000 7fdc9affd700 1 -- 172.21.15.32:0/2718194368 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fdc9400db40 con 0x7fdca40fcf00 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.753+0000 7fdc9affd700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdc900692a0 0x7fdc9006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.753+0000 7fdca1d9b700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdc900692a0 0x7fdc9006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.753+0000 7fdc9affd700 1 -- 172.21.15.32:0/2718194368 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fdc94086ea0 con 0x7fdca40fcf00 2024-04-06T14:14:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.754+0000 7fdca1d9b700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdc900692a0 0x7fdc9006b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00b900 tx=0x7fdc8c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:09.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:09.758+0000 7fdc9affd700 1 -- 172.21.15.32:0/2718194368 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdc94053200 con 0x7fdca40fcf00 2024-04-06T14:14:10.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.004+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fdca41057b0 con 0x7fdca40fcf00 2024-04-06T14:14:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:09 smithi032 ceph-mon[39730]: Deploying daemon mds.cephfs.smithi032.mqoaof on smithi032 2024-04-06T14:14:10.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:09 smithi086 ceph-mon[47925]: Deploying daemon mds.cephfs.smithi032.mqoaof on smithi032 2024-04-06T14:14:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.925+0000 7fdc9affd700 1 -- 172.21.15.32:0/2718194368 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7fdc9400ddf0 con 0x7fdca40fcf00 2024-04-06T14:14:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdc900692a0 msgr2=0x7fdc9006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdc900692a0 0x7fdc9006b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00b900 tx=0x7fdc8c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fcf00 msgr2=0x7fdca4071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fcf00 0x7fdca4071e40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc94002a50 tx=0x7fdc940046b0 comp rx=0 tx=0).stop 2024-04-06T14:14:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 shutdown_connections 2024-04-06T14:14:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdc900692a0 0x7fdc9006b750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdca40fcf00 0x7fdca4071e40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 --2- 172.21.15.32:0/2718194368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca40fe070 0x7fdca4070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 >> 172.21.15.32:0/2718194368 conn(0x7fdca40f85a0 msgr2=0x7fdca40f9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 shutdown_connections 2024-04-06T14:14:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:10.928+0000 7fdca8da0700 1 -- 172.21.15.32:0/2718194368 wait complete. 2024-04-06T14:14:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:10 smithi032 ceph-mon[39730]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-06T14:14:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:10 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2718194368' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-06T14:14:11.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:10 smithi086 ceph-mon[47925]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-06T14:14:11.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:10 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2718194368' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-06T14:14:11.638 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T14:14:11.648 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:14:11.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-04-06T14:14:12.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:11 smithi032 ceph-mon[39730]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T14:14:12.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:11 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2718194368' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-06T14:14:12.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:11 smithi032 ceph-mon[39730]: daemon mds.cephfs.smithi086.nikfjj assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-06T14:14:12.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:11 smithi032 ceph-mon[39730]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-06T14:14:12.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:11 smithi032 ceph-mon[39730]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T14:14:12.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:11 smithi032 ceph-mon[39730]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:active} 1 up:standby 2024-04-06T14:14:12.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:11 smithi032 ceph-mon[39730]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:creating} 2024-04-06T14:14:12.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:11 smithi032 ceph-mon[39730]: daemon mds.cephfs.smithi086.nikfjj is now active in filesystem cephfs as rank 1 2024-04-06T14:14:12.284 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:11 smithi086 ceph-mon[47925]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T14:14:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:11 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2718194368' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-06T14:14:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:11 smithi086 ceph-mon[47925]: daemon mds.cephfs.smithi086.nikfjj assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-06T14:14:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:11 smithi086 ceph-mon[47925]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-06T14:14:12.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:11 smithi086 ceph-mon[47925]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T14:14:12.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:11 smithi086 ceph-mon[47925]: fsmap cephfs:1 {0=cephfs.smithi032.gcjdjo=up:active} 1 up:standby 2024-04-06T14:14:12.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:11 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:creating} 2024-04-06T14:14:12.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:11 smithi086 ceph-mon[47925]: daemon mds.cephfs.smithi086.nikfjj is now active in filesystem cephfs as rank 1 2024-04-06T14:14:13.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-06T14:14:13.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] up:active 2024-04-06T14:14:13.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:active} 2024-04-06T14:14:13.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:13.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:13.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:13.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.ojmchr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:14:13.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.ojmchr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T14:14:13.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:13.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:13.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:14:13.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:13.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-06T14:14:13.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] up:active 2024-04-06T14:14:13.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:active} 2024-04-06T14:14:13.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:13.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:13.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:13.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.ojmchr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:14:13.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.ojmchr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T14:14:13.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:13.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:13.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:14:13.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.056+0000 7f32dab22700 1 -- 172.21.15.32:0/1783489055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 msgr2=0x7f32d40fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.056+0000 7f32dab22700 1 --2- 172.21.15.32:0/1783489055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 0x7f32d40fd310 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f32c8004940 tx=0x7f32c8009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.056+0000 7f32dab22700 1 -- 172.21.15.32:0/1783489055 shutdown_connections 2024-04-06T14:14:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.056+0000 7f32dab22700 1 --2- 172.21.15.32:0/1783489055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32d40fe070 0x7f32d40fe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.056+0000 7f32dab22700 1 --2- 172.21.15.32:0/1783489055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 0x7f32d40fd310 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.056+0000 7f32dab22700 1 -- 172.21.15.32:0/1783489055 >> 172.21.15.32:0/1783489055 conn(0x7f32d40f85a0 msgr2=0x7f32d40fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.056+0000 7f32dab22700 1 -- 172.21.15.32:0/1783489055 shutdown_connections 2024-04-06T14:14:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.056+0000 7f32dab22700 1 -- 172.21.15.32:0/1783489055 wait complete. 2024-04-06T14:14:14.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.057+0000 7f32dab22700 1 Processor -- start 2024-04-06T14:14:14.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.057+0000 7f32dab22700 1 -- start start 2024-04-06T14:14:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.057+0000 7f32dab22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 0x7f32d410d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.057+0000 7f32dab22700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32d40fe070 0x7f32d410de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.057+0000 7f32dab22700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d410e400 con 0x7f32d40fcf00 2024-04-06T14:14:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.057+0000 7f32dab22700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d410ec50 con 0x7f32d40fe070 2024-04-06T14:14:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.057+0000 7f32d88be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 0x7f32d410d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.057+0000 7f32d88be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 0x7f32d410d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50466/0 (socket says 172.21.15.32:50466) 2024-04-06T14:14:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.057+0000 7f32d88be700 1 -- 172.21.15.32:0/2852822529 learned_addr learned my addr 172.21.15.32:0/2852822529 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32d3fff700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32d40fe070 0x7f32d410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:14.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32d88be700 1 -- 172.21.15.32:0/2852822529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32d40fe070 msgr2=0x7f32d410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32d88be700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32d40fe070 0x7f32d410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32d88be700 1 -- 172.21.15.32:0/2852822529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c8005040 con 0x7f32d40fcf00 2024-04-06T14:14:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32d88be700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 0x7f32d410d8c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f32c80137b0 tx=0x7f32c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32d17fa700 1 -- 172.21.15.32:0/2852822529 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32c800e070 con 0x7f32d40fcf00 2024-04-06T14:14:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32d17fa700 1 -- 172.21.15.32:0/2852822529 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32c801c400 con 0x7f32d40fcf00 2024-04-06T14:14:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d410eed0 con 0x7f32d40fcf00 2024-04-06T14:14:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32d17fa700 1 -- 172.21.15.32:0/2852822529 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32c800b650 con 0x7f32d40fcf00 2024-04-06T14:14:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.058+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d410f3b0 con 0x7f32d40fcf00 2024-04-06T14:14:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.059+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d405f5f0 con 0x7f32d40fcf00 2024-04-06T14:14:14.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.060+0000 7f32d17fa700 1 -- 172.21.15.32:0/2852822529 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f32c801c9e0 con 0x7f32d40fcf00 2024-04-06T14:14:14.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.060+0000 7f32d17fa700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f32c40692a0 0x7f32c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:14.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.060+0000 7f32d17fa700 1 -- 172.21.15.32:0/2852822529 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f32c8088790 con 0x7f32d40fcf00 2024-04-06T14:14:14.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.060+0000 7f32d3fff700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f32c40692a0 0x7f32c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:14.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.060+0000 7f32d3fff700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f32c40692a0 0x7f32c406b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009b80 tx=0x7f32c000b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:14.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.065+0000 7f32d17fa700 1 -- 172.21.15.32:0/2852822529 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f32c8058550 con 0x7f32d40fcf00 2024-04-06T14:14:14.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:13 smithi086 ceph-mon[47925]: Deploying daemon mds.cephfs.smithi086.ojmchr on smithi086 2024-04-06T14:14:14.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:13 smithi086 ceph-mon[47925]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-06T14:14:14.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:13 smithi086 ceph-mon[47925]: Cluster is now healthy 2024-04-06T14:14:14.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:13 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:boot 2024-04-06T14:14:14.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:13 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:active} 1 up:standby 2024-04-06T14:14:14.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:13 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:14:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:13 smithi032 ceph-mon[39730]: Deploying daemon mds.cephfs.smithi086.ojmchr on smithi086 2024-04-06T14:14:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:13 smithi032 ceph-mon[39730]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-06T14:14:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:13 smithi032 ceph-mon[39730]: Cluster is now healthy 2024-04-06T14:14:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:13 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:boot 2024-04-06T14:14:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:13 smithi032 ceph-mon[39730]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:active} 1 up:standby 2024-04-06T14:14:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:13 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:14:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.306+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f32d404bed0 con 0x7f32d40fcf00 2024-04-06T14:14:14.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.944+0000 7f32d17fa700 1 -- 172.21.15.32:0/2852822529 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f32c8004490 con 0x7f32d40fcf00 2024-04-06T14:14:14.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f32c40692a0 msgr2=0x7f32c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:14.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f32c40692a0 0x7f32c406b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009b80 tx=0x7f32c000b040 comp rx=0 tx=0).stop 2024-04-06T14:14:14.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 msgr2=0x7f32d410d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:14.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 0x7f32d410d8c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f32c80137b0 tx=0x7f32c8009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:14.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 shutdown_connections 2024-04-06T14:14:14.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f32c40692a0 0x7f32c406b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:14.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32d40fcf00 0x7f32d410d8c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:14.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 --2- 172.21.15.32:0/2852822529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32d40fe070 0x7f32d410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:14.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 >> 172.21.15.32:0/2852822529 conn(0x7f32d40f85a0 msgr2=0x7f32d40f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:14.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 shutdown_connections 2024-04-06T14:14:14.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:14.946+0000 7f32dab22700 1 -- 172.21.15.32:0/2852822529 wait complete. 2024-04-06T14:14:15.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:14 smithi086 ceph-mon[47925]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-04-06T14:14:15.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:14 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2852822529' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-06T14:14:15.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:14 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:14 smithi032 ceph-mon[39730]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-04-06T14:14:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:14 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2852822529' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-06T14:14:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:14 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:15.659 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T14:14:15.669 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:14:15.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-06T14:14:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:15 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2852822529' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-06T14:14:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:15 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] up:boot 2024-04-06T14:14:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:15 smithi032 ceph-mon[39730]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:14:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:15 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:14:16.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:15 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2852822529' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-06T14:14:16.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:15 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] up:boot 2024-04-06T14:14:16.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:15 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:16.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:14:16.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:15 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:14:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:16 smithi032 ceph-mon[39730]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-06T14:14:17.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:16 smithi086 ceph-mon[47925]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-06T14:14:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:standby 2024-04-06T14:14:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: Dropping low affinity active daemon mds.cephfs.smithi032.gcjdjo in favor of higher affinity standby. 2024-04-06T14:14:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: Replacing daemon mds.cephfs.smithi032.gcjdjo as rank 0 with standby daemon mds.cephfs.smithi032.mqoaof 2024-04-06T14:14:18.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T14:14:18.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:18.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: osdmap e46: 6 total, 6 up, 6 in 2024-04-06T14:14:18.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: fsmap cephfs:2/2 {0=cephfs.smithi032.mqoaof=up:replay,1=cephfs.smithi086.nikfjj=up:active} 1 up:standby 2024-04-06T14:14:18.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:18.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:18.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 6 op/s 2024-04-06T14:14:18.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:17 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:standby 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: Dropping low affinity active daemon mds.cephfs.smithi032.gcjdjo in favor of higher affinity standby. 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: Replacing daemon mds.cephfs.smithi032.gcjdjo as rank 0 with standby daemon mds.cephfs.smithi032.mqoaof 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: osdmap e46: 6 total, 6 up, 6 in 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: fsmap cephfs:2/2 {0=cephfs.smithi032.mqoaof=up:replay,1=cephfs.smithi086.nikfjj=up:active} 1 up:standby 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 6 op/s 2024-04-06T14:14:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:17 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.169+0000 7f9551205700 1 -- 172.21.15.32:0/2040222575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fbce0 msgr2=0x7f954c0fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.169+0000 7f9551205700 1 --2- 172.21.15.32:0/2040222575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fbce0 0x7f954c0fc0f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9540004940 tx=0x7f9540009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.169+0000 7f9551205700 1 -- 172.21.15.32:0/2040222575 shutdown_connections 2024-04-06T14:14:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.169+0000 7f9551205700 1 --2- 172.21.15.32:0/2040222575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f954c0fcf90 0x7f954c0fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.169+0000 7f9551205700 1 --2- 172.21.15.32:0/2040222575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fbce0 0x7f954c0fc0f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.169+0000 7f9551205700 1 -- 172.21.15.32:0/2040222575 >> 172.21.15.32:0/2040222575 conn(0x7f954c0f7440 msgr2=0x7f954c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.169+0000 7f9551205700 1 -- 172.21.15.32:0/2040222575 shutdown_connections 2024-04-06T14:14:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.169+0000 7f9551205700 1 -- 172.21.15.32:0/2040222575 wait complete. 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.170+0000 7f9551205700 1 Processor -- start 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.170+0000 7f9551205700 1 -- start start 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.170+0000 7f9551205700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f954c0fbce0 0x7f954c10fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.170+0000 7f9551205700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fcf90 0x7f954c1100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.170+0000 7f9551205700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c1106d0 con 0x7f954c0fcf90 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.170+0000 7f9551205700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c110f20 con 0x7f954c0fbce0 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.170+0000 7f954a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fcf90 0x7f954c1100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.170+0000 7f954a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fcf90 0x7f954c1100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33102/0 (socket says 172.21.15.32:33102) 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.170+0000 7f954a59c700 1 -- 172.21.15.32:0/3385297787 learned_addr learned my addr 172.21.15.32:0/3385297787 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f954ad9d700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f954c0fbce0 0x7f954c10fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f954a59c700 1 -- 172.21.15.32:0/3385297787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f954c0fbce0 msgr2=0x7f954c10fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f954a59c700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f954c0fbce0 0x7f954c10fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f954a59c700 1 -- 172.21.15.32:0/3385297787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9540005040 con 0x7f954c0fcf90 2024-04-06T14:14:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f954a59c700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fcf90 0x7f954c1100d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f953c002a50 tx=0x7f953c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f95337fe700 1 -- 172.21.15.32:0/3385297787 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f953c007cc0 con 0x7f954c0fcf90 2024-04-06T14:14:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f95337fe700 1 -- 172.21.15.32:0/3385297787 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f953c00f040 con 0x7f954c0fcf90 2024-04-06T14:14:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f95337fe700 1 -- 172.21.15.32:0/3385297787 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f953c00d9c0 con 0x7f954c0fcf90 2024-04-06T14:14:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954c111200 con 0x7f954c0fcf90 2024-04-06T14:14:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.171+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954c111740 con 0x7f954c0fcf90 2024-04-06T14:14:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.172+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954c101920 con 0x7f954c0fcf90 2024-04-06T14:14:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.173+0000 7f95337fe700 1 -- 172.21.15.32:0/3385297787 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f953c00db20 con 0x7f954c0fcf90 2024-04-06T14:14:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.173+0000 7f95337fe700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f95340692a0 0x7f953406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.173+0000 7f95337fe700 1 -- 172.21.15.32:0/3385297787 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f953c086e30 con 0x7f954c0fcf90 2024-04-06T14:14:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.173+0000 7f954ad9d700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f95340692a0 0x7f953406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.174+0000 7f954ad9d700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f95340692a0 0x7f953406b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f954000b900 tx=0x7f9540009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.178+0000 7f95337fe700 1 -- 172.21.15.32:0/3385297787 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f953c056ba0 con 0x7f954c0fcf90 2024-04-06T14:14:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:18 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:resolve 2024-04-06T14:14:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:18 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] up:boot 2024-04-06T14:14:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:18 smithi032 ceph-mon[39730]: fsmap cephfs:2/2 {0=cephfs.smithi032.mqoaof=up:resolve,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:18 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:14:19.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:18 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:resolve 2024-04-06T14:14:19.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:18 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] up:boot 2024-04-06T14:14:19.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:18 smithi086 ceph-mon[47925]: fsmap cephfs:2/2 {0=cephfs.smithi032.mqoaof=up:resolve,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:19.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:18 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:14:19.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.433+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f954c1119b0 con 0x7f954c0fcf90 2024-04-06T14:14:19.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.972+0000 7f95337fe700 1 -- 172.21.15.32:0/3385297787 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v17) v1 ==== 168+0+0 (secure 0 0 0) 0x7f953c056730 con 0x7f954c0fcf90 2024-04-06T14:14:19.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f95340692a0 msgr2=0x7f953406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:19.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f95340692a0 0x7f953406b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f954000b900 tx=0x7f9540009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:19.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fcf90 msgr2=0x7f954c1100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:19.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fcf90 0x7f954c1100d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f953c002a50 tx=0x7f953c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:14:19.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 shutdown_connections 2024-04-06T14:14:19.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f95340692a0 0x7f953406b750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:19.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f954c0fbce0 0x7f954c10fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:19.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 --2- 172.21.15.32:0/3385297787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f954c0fcf90 0x7f954c1100d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:19.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 >> 172.21.15.32:0/3385297787 conn(0x7f954c0f7440 msgr2=0x7f954c0664e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:19.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 shutdown_connections 2024-04-06T14:14:19.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:19.974+0000 7f9551205700 1 -- 172.21.15.32:0/3385297787 wait complete. 2024-04-06T14:14:19.975 INFO:teuthology.orchestra.run.smithi032.stderr:inline data enabled 2024-04-06T14:14:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:19 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:reconnect 2024-04-06T14:14:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:19 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] up:standby 2024-04-06T14:14:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:19 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] up:active 2024-04-06T14:14:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:19 smithi032 ceph-mon[39730]: fsmap cephfs:2/2 {0=cephfs.smithi032.mqoaof=up:reconnect,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:19 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3385297787' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T14:14:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:19 smithi032 ceph-mon[39730]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-06T14:14:20.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:19 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:reconnect 2024-04-06T14:14:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:19 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] up:standby 2024-04-06T14:14:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:19 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] up:active 2024-04-06T14:14:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:19 smithi086 ceph-mon[47925]: fsmap cephfs:2/2 {0=cephfs.smithi032.mqoaof=up:reconnect,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:19 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3385297787' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T14:14:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:19 smithi086 ceph-mon[47925]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-06T14:14:20.630 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T14:14:20.643 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:14:20.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-06T14:14:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:20 smithi032 ceph-mon[39730]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-06T14:14:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:20 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:rejoin 2024-04-06T14:14:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:20 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3385297787' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T14:14:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:20 smithi032 ceph-mon[39730]: fsmap cephfs:2/2 {0=cephfs.smithi032.mqoaof=up:rejoin,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:20 smithi032 ceph-mon[39730]: daemon mds.cephfs.smithi032.mqoaof is now active in filesystem cephfs as rank 0 2024-04-06T14:14:21.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:20 smithi086 ceph-mon[47925]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-06T14:14:21.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:20 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:rejoin 2024-04-06T14:14:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3385297787' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T14:14:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:20 smithi086 ceph-mon[47925]: fsmap cephfs:2/2 {0=cephfs.smithi032.mqoaof=up:rejoin,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:20 smithi086 ceph-mon[47925]: daemon mds.cephfs.smithi032.mqoaof is now active in filesystem cephfs as rank 0 2024-04-06T14:14:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:21 smithi032 ceph-mon[39730]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T14:14:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:21 smithi032 ceph-mon[39730]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:active 2024-04-06T14:14:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:21 smithi032 ceph-mon[39730]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:21 smithi032 ceph-mon[39730]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 716 B/s wr, 5 op/s 2024-04-06T14:14:22.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:21 smithi086 ceph-mon[47925]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T14:14:22.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:21 smithi086 ceph-mon[47925]: mds.? [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] up:active 2024-04-06T14:14:22.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:21 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:14:22.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:21 smithi086 ceph-mon[47925]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 716 B/s wr, 5 op/s 2024-04-06T14:14:23.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.695+0000 7f4ad7026700 1 -- 172.21.15.32:0/1903458507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fdbe0 msgr2=0x7f4ad00fc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:23.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.695+0000 7f4ad7026700 1 --2- 172.21.15.32:0/1903458507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fdbe0 0x7f4ad00fc000 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4abc004940 tx=0x7f4abc009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:23.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.695+0000 7f4ad7026700 1 -- 172.21.15.32:0/1903458507 shutdown_connections 2024-04-06T14:14:23.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.695+0000 7f4ad7026700 1 --2- 172.21.15.32:0/1903458507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fdbe0 0x7f4ad00fc000 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:23.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.695+0000 7f4ad7026700 1 --2- 172.21.15.32:0/1903458507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ad00fd290 0x7f4ad00fd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.695+0000 7f4ad7026700 1 -- 172.21.15.32:0/1903458507 >> 172.21.15.32:0/1903458507 conn(0x7f4ad00f6ba0 msgr2=0x7f4ad00f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.695+0000 7f4ad7026700 1 -- 172.21.15.32:0/1903458507 shutdown_connections 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.696+0000 7f4ad7026700 1 -- 172.21.15.32:0/1903458507 wait complete. 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.696+0000 7f4ad7026700 1 Processor -- start 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.696+0000 7f4ad7026700 1 -- start start 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.696+0000 7f4ad7026700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ad00fd290 0x7f4ad01106a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.696+0000 7f4ad7026700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fbad0 0x7f4ad0112bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.696+0000 7f4ad7026700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad0113130 con 0x7f4ad00fbad0 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.696+0000 7f4ad7026700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad01132a0 con 0x7f4ad00fd290 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fbad0 0x7f4ad0112bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fbad0 0x7f4ad0112bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33114/0 (socket says 172.21.15.32:33114) 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acffff700 1 -- 172.21.15.32:0/4199192244 learned_addr learned my addr 172.21.15.32:0/4199192244 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4ad4dc2700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ad00fd290 0x7f4ad01106a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acffff700 1 -- 172.21.15.32:0/4199192244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ad00fd290 msgr2=0x7f4ad01106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acffff700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ad00fd290 0x7f4ad01106a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acffff700 1 -- 172.21.15.32:0/4199192244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc005040 con 0x7f4ad00fbad0 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acffff700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fbad0 0x7f4ad0112bf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009fd0 tx=0x7f4abc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acd7fa700 1 -- 172.21.15.32:0/4199192244 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4abc00e040 con 0x7f4ad00fbad0 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad0113520 con 0x7f4ad00fbad0 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acd7fa700 1 -- 172.21.15.32:0/4199192244 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4abc004290 con 0x7f4ad00fbad0 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4acd7fa700 1 -- 172.21.15.32:0/4199192244 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4abc01c3f0 con 0x7f4ad00fbad0 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.697+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad0113ae0 con 0x7f4ad00fbad0 2024-04-06T14:14:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.698+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad005f5f0 con 0x7f4ad00fbad0 2024-04-06T14:14:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.699+0000 7f4acd7fa700 1 -- 172.21.15.32:0/4199192244 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4abc013790 con 0x7f4ad00fbad0 2024-04-06T14:14:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.699+0000 7f4acd7fa700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f4ac00692b0 0x7f4ac006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.699+0000 7f4ad4dc2700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f4ac00692b0 0x7f4ac006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.699+0000 7f4acd7fa700 1 -- 172.21.15.32:0/4199192244 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4abc0885b0 con 0x7f4ad00fbad0 2024-04-06T14:14:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.700+0000 7f4ad4dc2700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f4ac00692b0 0x7f4ac006b760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4ad00fb860 tx=0x7f4ac4009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.704+0000 7f4acd7fa700 1 -- 172.21.15.32:0/4199192244 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4abc054890 con 0x7f4ad00fbad0 2024-04-06T14:14:23.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.950+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4ad00fdc80 con 0x7f4ad00fbad0 2024-04-06T14:14:23.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.950+0000 7f4acd7fa700 1 -- 172.21.15.32:0/4199192244 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1813 (secure 0 0 0) 0x7f4abc013420 con 0x7f4ad00fbad0 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:e18 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 18 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:14:20.970955+0000 2024-04-06T14:14:23.951 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:14:23.952 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:14:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:14:23.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f4ac00692b0 msgr2=0x7f4ac006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:23.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f4ac00692b0 0x7f4ac006b760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4ad00fb860 tx=0x7f4ac4009040 comp rx=0 tx=0).stop 2024-04-06T14:14:23.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fbad0 msgr2=0x7f4ad0112bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:23.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fbad0 0x7f4ad0112bf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009fd0 tx=0x7f4abc009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:23.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 shutdown_connections 2024-04-06T14:14:23.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f4ac00692b0 0x7f4ac006b760 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:23.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ad00fd290 0x7f4ad01106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:23.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 --2- 172.21.15.32:0/4199192244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ad00fbad0 0x7f4ad0112bf0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:23.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 >> 172.21.15.32:0/4199192244 conn(0x7f4ad00f6ba0 msgr2=0x7f4ad00f83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:23.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 shutdown_connections 2024-04-06T14:14:23.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:23.953+0000 7f4ad7026700 1 -- 172.21.15.32:0/4199192244 wait complete. 2024-04-06T14:14:23.956 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2024-04-06T14:14:24.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-06T14:14:24.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:24 smithi032 ceph-mon[39730]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-06T14:14:24.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:24.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:24.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:24.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:14:24.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:24.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:24 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/4199192244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:14:24.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:24 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:14:25.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:24 smithi086 ceph-mon[47925]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-06T14:14:25.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:25.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:25.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:25.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:14:25.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:25.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:24 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/4199192244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:14:25.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:24 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:14:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:26.963 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:26 smithi032 ceph-mon[39730]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T14:14:26.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:26.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:27.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:26 smithi086 ceph-mon[47925]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T14:14:27.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:27.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.841+0000 7f4988bc8700 1 -- 172.21.15.32:0/659897685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fe4c0 msgr2=0x7f49840fe8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.841+0000 7f4988bc8700 1 --2- 172.21.15.32:0/659897685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fe4c0 0x7f49840fe8d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4974004940 tx=0x7f4974009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.841+0000 7f4988bc8700 1 -- 172.21.15.32:0/659897685 shutdown_connections 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.841+0000 7f4988bc8700 1 --2- 172.21.15.32:0/659897685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49840fee10 0x7f49840faf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.841+0000 7f4988bc8700 1 --2- 172.21.15.32:0/659897685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fe4c0 0x7f49840fe8d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.841+0000 7f4988bc8700 1 -- 172.21.15.32:0/659897685 >> 172.21.15.32:0/659897685 conn(0x7f49840f6ba0 msgr2=0x7f49840f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.841+0000 7f4988bc8700 1 -- 172.21.15.32:0/659897685 shutdown_connections 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.841+0000 7f4988bc8700 1 -- 172.21.15.32:0/659897685 wait complete. 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4988bc8700 1 Processor -- start 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4988bc8700 1 -- start start 2024-04-06T14:14:27.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4988bc8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49840fe4c0 0x7f498410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4988bc8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fee10 0x7f498410de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4988bc8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498410e470 con 0x7f49840fee10 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4988bc8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498410ecc0 con 0x7f49840fe4c0 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4981d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fee10 0x7f498410de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4981d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fee10 0x7f498410de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43474/0 (socket says 172.21.15.32:43474) 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4981d9b700 1 -- 172.21.15.32:0/765684470 learned_addr learned my addr 172.21.15.32:0/765684470 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f498259c700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49840fe4c0 0x7f498410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4981d9b700 1 -- 172.21.15.32:0/765684470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49840fe4c0 msgr2=0x7f498410d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4981d9b700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49840fe4c0 0x7f498410d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4981d9b700 1 -- 172.21.15.32:0/765684470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4974005040 con 0x7f49840fee10 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.842+0000 7f4981d9b700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fee10 0x7f498410de70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f496c002a50 tx=0x7f496c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.843+0000 7f497affd700 1 -- 172.21.15.32:0/765684470 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f496c007cc0 con 0x7f49840fee10 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.843+0000 7f497affd700 1 -- 172.21.15.32:0/765684470 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f496c00f040 con 0x7f49840fee10 2024-04-06T14:14:27.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.843+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498410efa0 con 0x7f49840fee10 2024-04-06T14:14:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.843+0000 7f497affd700 1 -- 172.21.15.32:0/765684470 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f496c009540 con 0x7f49840fee10 2024-04-06T14:14:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.843+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498410f4e0 con 0x7f49840fee10 2024-04-06T14:14:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.843+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498405f5f0 con 0x7f49840fee10 2024-04-06T14:14:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.844+0000 7f497affd700 1 -- 172.21.15.32:0/765684470 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f496c0096a0 con 0x7f49840fee10 2024-04-06T14:14:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.844+0000 7f497affd700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f49700692f0 0x7f497006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.844+0000 7f497affd700 1 -- 172.21.15.32:0/765684470 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f496c087090 con 0x7f49840fee10 2024-04-06T14:14:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.844+0000 7f498259c700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f49700692f0 0x7f497006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.845+0000 7f498259c700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f49700692f0 0x7f497006b7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4974000c00 tx=0x7f4974009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:27.849+0000 7f497affd700 1 -- 172.21.15.32:0/765684470 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f496c056ce0 con 0x7f49840fee10 2024-04-06T14:14:28.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.087+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f498404bed0 con 0x7f49840fee10 2024-04-06T14:14:28.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.087+0000 7f497affd700 1 -- 172.21.15.32:0/765684470 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4700 (secure 0 0 0) 0x7f496c056870 con 0x7f49840fee10 2024-04-06T14:14:28.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.089+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f49700692f0 msgr2=0x7f497006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.089+0000 7f4988bc8700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f49700692f0 0x7f497006b7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4974000c00 tx=0x7f4974009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.089+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fee10 msgr2=0x7f498410de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.089+0000 7f4988bc8700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fee10 0x7f498410de70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f496c002a50 tx=0x7f496c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.089+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 shutdown_connections 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.089+0000 7f4988bc8700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f49700692f0 0x7f497006b7a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.089+0000 7f4988bc8700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49840fe4c0 0x7f498410d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.089+0000 7f4988bc8700 1 --2- 172.21.15.32:0/765684470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49840fee10 0x7f498410de70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.089+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 >> 172.21.15.32:0/765684470 conn(0x7f49840f6ba0 msgr2=0x7f49840f8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.090+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 shutdown_connections 2024-04-06T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:28.090+0000 7f4988bc8700 1 -- 172.21.15.32:0/765684470 wait complete. 2024-04-06T14:14:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2024-04-06T14:14:28.098 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:14:28.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:28.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:28 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:14:28.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:28 smithi032 ceph-mon[39730]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T14:14:28.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:28 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/765684470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T14:14:28.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-06T14:14:29.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:29.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:28 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:14:29.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:28 smithi086 ceph-mon[47925]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T14:14:29.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:28 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/765684470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T14:14:30.095 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:30 smithi032 ceph-mon[39730]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T14:14:31.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:30 smithi086 ceph-mon[47925]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T14:14:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.511+0000 7f2f09565700 1 -- 172.21.15.32:0/4036418141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fce70 msgr2=0x7f2f040fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.511+0000 7f2f09565700 1 --2- 172.21.15.32:0/4036418141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fce70 0x7f2f040fd280 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2eec004940 tx=0x7f2eec009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.511+0000 7f2f09565700 1 -- 172.21.15.32:0/4036418141 shutdown_connections 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.511+0000 7f2f09565700 1 --2- 172.21.15.32:0/4036418141 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f040fdfe0 0x7f2f040fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.511+0000 7f2f09565700 1 --2- 172.21.15.32:0/4036418141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fce70 0x7f2f040fd280 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.511+0000 7f2f09565700 1 -- 172.21.15.32:0/4036418141 >> 172.21.15.32:0/4036418141 conn(0x7f2f040f8550 msgr2=0x7f2f040fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.511+0000 7f2f09565700 1 -- 172.21.15.32:0/4036418141 shutdown_connections 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.511+0000 7f2f09565700 1 -- 172.21.15.32:0/4036418141 wait complete. 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f09565700 1 Processor -- start 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f09565700 1 -- start start 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f09565700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f040fce70 0x7f2f0410d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f09565700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fdfe0 0x7f2f0410ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f09565700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0410e3c0 con 0x7f2f040fdfe0 2024-04-06T14:14:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f09565700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0410e530 con 0x7f2f040fce70 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f027fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fdfe0 0x7f2f0410ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f027fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fdfe0 0x7f2f0410ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43490/0 (socket says 172.21.15.32:43490) 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f027fc700 1 -- 172.21.15.32:0/4078800483 learned_addr learned my addr 172.21.15.32:0/4078800483 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.512+0000 7f2f02ffd700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f040fce70 0x7f2f0410d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.513+0000 7f2f027fc700 1 -- 172.21.15.32:0/4078800483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f040fce70 msgr2=0x7f2f0410d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.513+0000 7f2f027fc700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f040fce70 0x7f2f0410d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.513+0000 7f2f027fc700 1 -- 172.21.15.32:0/4078800483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec005040 con 0x7f2f040fdfe0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.513+0000 7f2f027fc700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fdfe0 0x7f2f0410ddc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c830 tx=0x7f2ef4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.513+0000 7f2efb7fe700 1 -- 172.21.15.32:0/4078800483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef40106f0 con 0x7f2f040fdfe0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.513+0000 7f2efb7fe700 1 -- 172.21.15.32:0/4078800483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ef4010d00 con 0x7f2f040fdfe0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.513+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f04112f40 con 0x7f2f040fdfe0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.513+0000 7f2efb7fe700 1 -- 172.21.15.32:0/4078800483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef4012e90 con 0x7f2f040fdfe0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.513+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f041134d0 con 0x7f2f040fdfe0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.514+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0404c710 con 0x7f2f040fdfe0 2024-04-06T14:14:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.514+0000 7f2efb7fe700 1 -- 172.21.15.32:0/4078800483 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2ef4010850 con 0x7f2f040fdfe0 2024-04-06T14:14:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.515+0000 7f2efb7fe700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2ef00692a0 0x7f2ef006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.515+0000 7f2efb7fe700 1 -- 172.21.15.32:0/4078800483 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2ef4087120 con 0x7f2f040fdfe0 2024-04-06T14:14:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.515+0000 7f2f02ffd700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2ef00692a0 0x7f2ef006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.515+0000 7f2f02ffd700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2ef00692a0 0x7f2ef006b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00b900 tx=0x7f2eec009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.519+0000 7f2efb7fe700 1 -- 172.21.15.32:0/4078800483 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2ef4056f30 con 0x7f2f040fdfe0 2024-04-06T14:14:32.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.850+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f2f04105790 con 0x7f2f040fdfe0 2024-04-06T14:14:32.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.850+0000 7f2efb7fe700 1 -- 172.21.15.32:0/4078800483 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+99 (secure 0 0 0) 0x7f2ef4056ac0 con 0x7f2f040fdfe0 2024-04-06T14:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2ef00692a0 msgr2=0x7f2ef006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2ef00692a0 0x7f2ef006b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00b900 tx=0x7f2eec009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:32.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fdfe0 msgr2=0x7f2f0410ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fdfe0 0x7f2f0410ddc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c830 tx=0x7f2ef4004470 comp rx=0 tx=0).stop 2024-04-06T14:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 shutdown_connections 2024-04-06T14:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f2ef00692a0 0x7f2ef006b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f040fce70 0x7f2f0410d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 --2- 172.21.15.32:0/4078800483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f040fdfe0 0x7f2f0410ddc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 >> 172.21.15.32:0/4078800483 conn(0x7f2f040f8550 msgr2=0x7f2f040f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 shutdown_connections 2024-04-06T14:14:32.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:32.852+0000 7f2f09565700 1 -- 172.21.15.32:0/4078800483 wait complete. 2024-04-06T14:14:32.866 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:14:32.979 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:32 smithi032 ceph-mon[39730]: pgmap v201: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-06T14:14:33.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:32 smithi086 ceph-mon[47925]: pgmap v201: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-06T14:14:33.668 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-06T14:14:33.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T14:14:33.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.817+0000 7f5ba92b7700 1 -- 172.21.15.32:0/3349166249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 msgr2=0x7f5ba4104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:33.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.817+0000 7f5ba92b7700 1 --2- 172.21.15.32:0/3349166249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 0x7f5ba4104640 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5b94004940 tx=0x7f5b94009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.817+0000 7f5ba92b7700 1 -- 172.21.15.32:0/3349166249 shutdown_connections 2024-04-06T14:14:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.817+0000 7f5ba92b7700 1 --2- 172.21.15.32:0/3349166249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 0x7f5ba4104640 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.817+0000 7f5ba92b7700 1 --2- 172.21.15.32:0/3349166249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ba4065a40 0x7f5ba4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.817+0000 7f5ba92b7700 1 -- 172.21.15.32:0/3349166249 >> 172.21.15.32:0/3349166249 conn(0x7f5ba40f7310 msgr2=0x7f5ba40f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.817+0000 7f5ba92b7700 1 -- 172.21.15.32:0/3349166249 shutdown_connections 2024-04-06T14:14:33.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.817+0000 7f5ba92b7700 1 -- 172.21.15.32:0/3349166249 wait complete. 2024-04-06T14:14:33.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba92b7700 1 Processor -- start 2024-04-06T14:14:33.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba92b7700 1 -- start start 2024-04-06T14:14:33.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba92b7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ba4065a40 0x7f5ba410f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:33.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba92b7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 0x7f5ba410fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:33.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba92b7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba4110550 con 0x7f5ba4102260 2024-04-06T14:14:33.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba92b7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba4114ef0 con 0x7f5ba4065a40 2024-04-06T14:14:33.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 0x7f5ba410fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:33.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 0x7f5ba410fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43510/0 (socket says 172.21.15.32:43510) 2024-04-06T14:14:33.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba27fc700 1 -- 172.21.15.32:0/748864777 learned_addr learned my addr 172.21.15.32:0/748864777 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:33.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.818+0000 7f5ba2ffd700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ba4065a40 0x7f5ba410f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:33.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.819+0000 7f5ba27fc700 1 -- 172.21.15.32:0/748864777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ba4065a40 msgr2=0x7f5ba410f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:33.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.819+0000 7f5ba27fc700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ba4065a40 0x7f5ba410f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:33.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.819+0000 7f5ba27fc700 1 -- 172.21.15.32:0/748864777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b94005040 con 0x7f5ba4102260 2024-04-06T14:14:33.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.819+0000 7f5ba27fc700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 0x7f5ba410fec0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400b3b0 tx=0x7f5b94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:33.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.819+0000 7f5b8b7fe700 1 -- 172.21.15.32:0/748864777 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b9400e040 con 0x7f5ba4102260 2024-04-06T14:14:33.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.819+0000 7f5b8b7fe700 1 -- 172.21.15.32:0/748864777 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b94004290 con 0x7f5ba4102260 2024-04-06T14:14:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.819+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba41150f0 con 0x7f5ba4102260 2024-04-06T14:14:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.819+0000 7f5b8b7fe700 1 -- 172.21.15.32:0/748864777 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b9401c3f0 con 0x7f5ba4102260 2024-04-06T14:14:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.819+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba4115660 con 0x7f5ba4102260 2024-04-06T14:14:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.820+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba4064970 con 0x7f5ba4102260 2024-04-06T14:14:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.821+0000 7f5b8b7fe700 1 -- 172.21.15.32:0/748864777 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5b94013420 con 0x7f5ba4102260 2024-04-06T14:14:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.821+0000 7f5b8b7fe700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5b8c0692a0 0x7f5b8c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.821+0000 7f5b8b7fe700 1 -- 172.21.15.32:0/748864777 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5b94088fd0 con 0x7f5ba4102260 2024-04-06T14:14:33.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.821+0000 7f5ba2ffd700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5b8c0692a0 0x7f5b8c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:33.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.822+0000 7f5ba2ffd700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5b8c0692a0 0x7f5b8c06b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5b98006fd0 tx=0x7f5b98008040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:33.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:33.825+0000 7f5b8b7fe700 1 -- 172.21.15.32:0/748864777 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5b94058de0 con 0x7f5ba4102260 2024-04-06T14:14:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:33 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:33 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:33 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:33 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:14:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:33 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:33 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/4078800483' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-06T14:14:34.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.023+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ba4107910 con 0x7f5ba4102260 2024-04-06T14:14:34.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.023+0000 7f5b8b7fe700 1 -- 172.21.15.32:0/748864777 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14358 (secure 0 0 0) 0x7f5b9400bc50 con 0x7f5ba4102260 2024-04-06T14:14:34.024 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:14:34.024 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":46,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","created":"2024-04-06T14:06:48.330591+0000","modified":"2024-04-06T14:14:16.952771+0000","last_up_change":"2024-04-06T14:12:36.744888+0000","last_in_change":"2024-04-06T14:12:17.748510+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T14:11:20.110423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-06T14:14:00.974689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-06T14:14:01.457260+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6803","nonce":1233836109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6805","nonce":1233836109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6809","nonce":1233836109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1233836109},{"type":"v1","addr":"172.21.15.32:6807","nonce":1233836109}]},"public_addr":"172.21.15.32:6803/1233836109","cluster_addr":"172.21.15.32:6805/1233836109","heartbeat_back_addr":"172.21.15.32:6809/1233836109","heartbeat_front_addr":"172.21.15.32:6807/1233836109","state":["exists","up"]},{"osd":1,"uuid":"9185a389-ef69-41bc-9e10-46c6a2e02a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6811","nonce":104495793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6813","nonce":104495793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6817","nonce":104495793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":104495793},{"type":"v1","addr":"172.21.15.32:6815","nonce":104495793}]},"public_addr":"172.21.15.32:6811/104495793","cluster_addr":"172.21.15.32:6813/104495793","heartbeat_back_addr":"172.21.15.32:6817/104495793","heartbeat_front_addr":"172.21.15.32:6815/104495793","state":["exists","up"]},{"osd":2,"uuid":"af4793e1-b5bb-4919-838e-67703c83296a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6819","nonce":2571669324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6821","nonce":2571669324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6825","nonce":2571669324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":2571669324},{"type":"v1","addr":"172.21.15.32:6823","nonce":2571669324}]},"public_addr":"172.21.15.32:6819/2571669324","cluster_addr":"172.21.15.32:6821/2571669324","heartbeat_back_addr":"172.21.15.32:6825/2571669324","heartbeat_front_addr":"172.21.15.32:6823/2571669324","state":["exists","up"]},{"osd":3,"uuid":"70c8b884-5214-44f2-9a31-55556a11c20a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6801","nonce":3912040245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6803","nonce":3912040245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6807","nonce":3912040245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3912040245},{"type":"v1","addr":"172.21.15.86:6805","nonce":3912040245}]},"public_addr":"172.21.15.86:6801/3912040245","cluster_addr":"172.21.15.86:6803/3912040245","heartbeat_back_addr":"172.21.15.86:6807/3912040245","heartbeat_front_addr":"172.21.15.86:6805/3912040245","state":["exists","up"]},{"osd":4,"uuid":"91f548ac-4f42-4083-a027-295b829646a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6809","nonce":3794387444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6811","nonce":3794387444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6815","nonce":3794387444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3794387444},{"type":"v1","addr":"172.21.15.86:6813","nonce":3794387444}]},"public_addr":"172.21.15.86:6809/3794387444","cluster_addr":"172.21.15.86:6811/3794387444","heartbeat_back_addr":"172.21.15.86:6815/3794387444","heartbeat_front_addr":"172.21.15.86:6813/3794387444","state":["exists","up"]},{"osd":5,"uuid":"7037f613-ffd9-4b0e-84a1-0c04d4181ff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6817","nonce":2011390682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6819","nonce":2011390682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6823","nonce":2011390682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":2011390682},{"type":"v1","addr":"172.21.15.86:6821","nonce":2011390682}]},"public_addr":"172.21.15.86:6817/2011390682","cluster_addr":"172.21.15.86:6819/2011390682","heartbeat_back_addr":"172.21.15.86:6823/2011390682","heartbeat_front_addr":"172.21.15.86:6821/2011390682","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:13.684938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:44.549488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:15.893369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:41.588409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:08.612078+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:35.380707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/3250004174":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6801/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6801/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/3548398153":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/68274703":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6801/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/935909372":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/1877817963":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/1187866165":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6800/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/2227455350":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/3330131734":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/711512148":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6827/4235185528":"2024-04-07T14:14:16.952740+0000","172.21.15.32:6800/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6800/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6826/4235185528":"2024-04-07T14:14:16.952740+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5b8c0692a0 msgr2=0x7f5b8c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5b8c0692a0 0x7f5b8c06b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5b98006fd0 tx=0x7f5b98008040 comp rx=0 tx=0).stop 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 msgr2=0x7f5ba410fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 0x7f5ba410fec0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400b3b0 tx=0x7f5b94009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 shutdown_connections 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f5b8c0692a0 0x7f5b8c06b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ba4065a40 0x7f5ba410f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 --2- 172.21.15.32:0/748864777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ba4102260 0x7f5ba410fec0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 >> 172.21.15.32:0/748864777 conn(0x7f5ba40f7310 msgr2=0x7f5ba40f7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 shutdown_connections 2024-04-06T14:14:34.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.026+0000 7f5ba92b7700 1 -- 172.21.15.32:0/748864777 wait complete. 2024-04-06T14:14:34.039 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T14:11:20.110423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-06T14:14:00.974689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-06T14:14:01.457260+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T14:14:34.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T14:14:34.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:33 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:33 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:33 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:14:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:33 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:14:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:33 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:14:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:33 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/4078800483' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-06T14:14:34.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.166+0000 7fc43c9eb700 1 -- 172.21.15.32:0/799385463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 msgr2=0x7fc4380fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.166+0000 7fc43c9eb700 1 --2- 172.21.15.32:0/799385463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 0x7fc4380fbfb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc42c004940 tx=0x7fc42c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:34.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 -- 172.21.15.32:0/799385463 shutdown_connections 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 --2- 172.21.15.32:0/799385463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4380fce50 0x7fc4380fd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 --2- 172.21.15.32:0/799385463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 0x7fc4380fbfb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 -- 172.21.15.32:0/799385463 >> 172.21.15.32:0/799385463 conn(0x7fc4380f7300 msgr2=0x7fc4380f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 -- 172.21.15.32:0/799385463 shutdown_connections 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 -- 172.21.15.32:0/799385463 wait complete. 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 Processor -- start 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 -- start start 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 0x7fc438065820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4380fce50 0x7fc438112550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438112a90 con 0x7fc4380fbba0 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43c9eb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438112bd0 con 0x7fc4380fce50 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 0x7fc438065820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 0x7fc438065820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43534/0 (socket says 172.21.15.32:43534) 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.167+0000 7fc43659c700 1 -- 172.21.15.32:0/1181361263 learned_addr learned my addr 172.21.15.32:0/1181361263 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:34.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc435d9b700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4380fce50 0x7fc438112550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc43659c700 1 -- 172.21.15.32:0/1181361263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4380fce50 msgr2=0x7fc438112550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc43659c700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4380fce50 0x7fc438112550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc43659c700 1 -- 172.21.15.32:0/1181361263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c005040 con 0x7fc4380fbba0 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc435d9b700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4380fce50 0x7fc438112550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc43659c700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 0x7fc438065820 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009fd0 tx=0x7fc42c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc426ffd700 1 -- 172.21.15.32:0/1181361263 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc42c00e040 con 0x7fc4380fbba0 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc426ffd700 1 -- 172.21.15.32:0/1181361263 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc42c004290 con 0x7fc4380fbba0 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc438112dd0 con 0x7fc4380fbba0 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc426ffd700 1 -- 172.21.15.32:0/1181361263 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc42c01c3f0 con 0x7fc4380fbba0 2024-04-06T14:14:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.168+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc438113310 con 0x7fc4380fbba0 2024-04-06T14:14:34.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.169+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc438064970 con 0x7fc4380fbba0 2024-04-06T14:14:34.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.169+0000 7fc426ffd700 1 -- 172.21.15.32:0/1181361263 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc42c00bc50 con 0x7fc4380fbba0 2024-04-06T14:14:34.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.170+0000 7fc426ffd700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc4200692a0 0x7fc42006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:34.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.170+0000 7fc435d9b700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc4200692a0 0x7fc42006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:34.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.170+0000 7fc426ffd700 1 -- 172.21.15.32:0/1181361263 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc42c088730 con 0x7fc4380fbba0 2024-04-06T14:14:34.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.170+0000 7fc435d9b700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc4200692a0 0x7fc42006b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc428009fd0 tx=0x7fc428009ee0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:34.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.175+0000 7fc426ffd700 1 -- 172.21.15.32:0/1181361263 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc42c058440 con 0x7fc4380fbba0 2024-04-06T14:14:34.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.370+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc438113530 con 0x7fc4380fbba0 2024-04-06T14:14:34.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.370+0000 7fc426ffd700 1 -- 172.21.15.32:0/1181361263 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc42c057fd0 con 0x7fc4380fbba0 2024-04-06T14:14:34.370 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-06T14:14:34.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc4200692a0 msgr2=0x7fc42006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc4200692a0 0x7fc42006b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc428009fd0 tx=0x7fc428009ee0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 msgr2=0x7fc438065820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 0x7fc438065820 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009fd0 tx=0x7fc42c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:34.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 shutdown_connections 2024-04-06T14:14:34.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc4200692a0 0x7fc42006b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4380fbba0 0x7fc438065820 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 --2- 172.21.15.32:0/1181361263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4380fce50 0x7fc438112550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 >> 172.21.15.32:0/1181361263 conn(0x7fc4380f7300 msgr2=0x7fc4380f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:34.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 shutdown_connections 2024-04-06T14:14:34.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.372+0000 7fc43c9eb700 1 -- 172.21.15.32:0/1181361263 wait complete. 2024-04-06T14:14:34.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-06T14:14:34.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.511+0000 7f93922f3700 1 -- 172.21.15.32:0/4016036488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c065aa0 msgr2=0x7f938c0ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.511+0000 7f93922f3700 1 --2- 172.21.15.32:0/4016036488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c065aa0 0x7f938c0ff890 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f937c004940 tx=0x7f937c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.511+0000 7f93922f3700 1 -- 172.21.15.32:0/4016036488 shutdown_connections 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.511+0000 7f93922f3700 1 --2- 172.21.15.32:0/4016036488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f938c0ffdd0 0x7f938c1021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.511+0000 7f93922f3700 1 --2- 172.21.15.32:0/4016036488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c065aa0 0x7f938c0ff890 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.512+0000 7f93922f3700 1 -- 172.21.15.32:0/4016036488 >> 172.21.15.32:0/4016036488 conn(0x7f938c0f7300 msgr2=0x7f938c0f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.512+0000 7f93922f3700 1 -- 172.21.15.32:0/4016036488 shutdown_connections 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.512+0000 7f93922f3700 1 -- 172.21.15.32:0/4016036488 wait complete. 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.512+0000 7f93922f3700 1 Processor -- start 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.512+0000 7f93922f3700 1 -- start start 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.512+0000 7f93922f3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f938c065aa0 0x7f938c0fd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:34.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.512+0000 7f93922f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c0ffdd0 0x7f938c0fb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.512+0000 7f93922f3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c0fd6e0 con 0x7f938c0ffdd0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.512+0000 7f93922f3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c0fbcd0 con 0x7f938c065aa0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f938b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c0ffdd0 0x7f938c0fb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f938b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c0ffdd0 0x7f938c0fb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48460/0 (socket says 172.21.15.32:48460) 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f938b7fe700 1 -- 172.21.15.32:0/2947559910 learned_addr learned my addr 172.21.15.32:0/2947559910 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f938bfff700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f938c065aa0 0x7f938c0fd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f938b7fe700 1 -- 172.21.15.32:0/2947559910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f938c065aa0 msgr2=0x7f938c0fd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f938b7fe700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f938c065aa0 0x7f938c0fd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f938b7fe700 1 -- 172.21.15.32:0/2947559910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c005040 con 0x7f938c0ffdd0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f938b7fe700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c0ffdd0 0x7f938c0fb790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f937400c830 tx=0x7f9374004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f9388ff9700 1 -- 172.21.15.32:0/2947559910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93740106f0 con 0x7f938c0ffdd0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f9388ff9700 1 -- 172.21.15.32:0/2947559910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9374010d00 con 0x7f938c0ffdd0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938c0fbf80 con 0x7f938c0ffdd0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f9388ff9700 1 -- 172.21.15.32:0/2947559910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9374012e90 con 0x7f938c0ffdd0 2024-04-06T14:14:34.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.513+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938c0fc540 con 0x7f938c0ffdd0 2024-04-06T14:14:34.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.514+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938c064970 con 0x7f938c0ffdd0 2024-04-06T14:14:34.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.515+0000 7f9388ff9700 1 -- 172.21.15.32:0/2947559910 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9374010850 con 0x7f938c0ffdd0 2024-04-06T14:14:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.515+0000 7f9388ff9700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93780692a0 0x7f937806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.515+0000 7f9388ff9700 1 -- 172.21.15.32:0/2947559910 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9374087140 con 0x7f938c0ffdd0 2024-04-06T14:14:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.515+0000 7f938bfff700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93780692a0 0x7f937806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.515+0000 7f938bfff700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93780692a0 0x7f937806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f937c00b900 tx=0x7f937c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:34.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.520+0000 7f9388ff9700 1 -- 172.21.15.32:0/2947559910 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9374056f50 con 0x7f938c0ffdd0 2024-04-06T14:14:34.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.717+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f938c105650 con 0x7f938c0ffdd0 2024-04-06T14:14:34.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.718+0000 7f9388ff9700 1 -- 172.21.15.32:0/2947559910 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f9374056ae0 con 0x7f938c0ffdd0 2024-04-06T14:14:34.718 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-06T14:14:34.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93780692a0 msgr2=0x7f937806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93780692a0 0x7f937806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f937c00b900 tx=0x7f937c009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c0ffdd0 msgr2=0x7f938c0fb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c0ffdd0 0x7f938c0fb790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f937400c830 tx=0x7f9374004470 comp rx=0 tx=0).stop 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 shutdown_connections 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93780692a0 0x7f937806b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f938c065aa0 0x7f938c0fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 --2- 172.21.15.32:0/2947559910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f938c0ffdd0 0x7f938c0fb790 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 >> 172.21.15.32:0/2947559910 conn(0x7f938c0f7300 msgr2=0x7f938c0f7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 shutdown_connections 2024-04-06T14:14:34.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.720+0000 7f93922f3700 1 -- 172.21.15.32:0/2947559910 wait complete. 2024-04-06T14:14:34.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-06T14:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.857+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1865456793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fccc0 msgr2=0x7f37d40fd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.857+0000 7f37d8f9f700 1 --2- 172.21.15.32:0/1865456793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fccc0 0x7f37d40fd130 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f37c4004940 tx=0x7f37c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.857+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1865456793 shutdown_connections 2024-04-06T14:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.857+0000 7f37d8f9f700 1 --2- 172.21.15.32:0/1865456793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fccc0 0x7f37d40fd130 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.857+0000 7f37d8f9f700 1 --2- 172.21.15.32:0/1865456793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37d40fba10 0x7f37d40fbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.857+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1865456793 >> 172.21.15.32:0/1865456793 conn(0x7f37d40f7190 msgr2=0x7f37d40f95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.857+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1865456793 shutdown_connections 2024-04-06T14:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.857+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1865456793 wait complete. 2024-04-06T14:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.858+0000 7f37d8f9f700 1 Processor -- start 2024-04-06T14:14:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.858+0000 7f37d8f9f700 1 -- start start 2024-04-06T14:14:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.858+0000 7f37d8f9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fba10 0x7f37d410c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.858+0000 7f37d8f9f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37d40fccc0 0x7f37d410c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.858+0000 7f37d8f9f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d410cfd0 con 0x7f37d40fba10 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d8f9f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d410d140 con 0x7f37d40fccc0 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fba10 0x7f37d410c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fba10 0x7f37d410c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48484/0 (socket says 172.21.15.32:48484) 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d37fe700 1 -- 172.21.15.32:0/1416552138 learned_addr learned my addr 172.21.15.32:0/1416552138 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d2ffd700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37d40fccc0 0x7f37d410c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d37fe700 1 -- 172.21.15.32:0/1416552138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37d40fccc0 msgr2=0x7f37d410c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d37fe700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37d40fccc0 0x7f37d410c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d37fe700 1 -- 172.21.15.32:0/1416552138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c4005040 con 0x7f37d40fba10 2024-04-06T14:14:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d37fe700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fba10 0x7f37d410c490 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00c830 tx=0x7f37bc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37cbfff700 1 -- 172.21.15.32:0/1416552138 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37bc00a760 con 0x7f37d40fba10 2024-04-06T14:14:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37cbfff700 1 -- 172.21.15.32:0/1416552138 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37bc00ad70 con 0x7f37d40fba10 2024-04-06T14:14:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d4111ba0 con 0x7f37d40fba10 2024-04-06T14:14:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37cbfff700 1 -- 172.21.15.32:0/1416552138 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37bc0113a0 con 0x7f37d40fba10 2024-04-06T14:14:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.859+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d41120e0 con 0x7f37d40fba10 2024-04-06T14:14:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.860+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d404c710 con 0x7f37d40fba10 2024-04-06T14:14:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.861+0000 7f37cbfff700 1 -- 172.21.15.32:0/1416552138 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f37bc012400 con 0x7f37d40fba10 2024-04-06T14:14:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.861+0000 7f37cbfff700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f37c00692a0 0x7f37c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.861+0000 7f37cbfff700 1 -- 172.21.15.32:0/1416552138 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f37bc087270 con 0x7f37d40fba10 2024-04-06T14:14:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.861+0000 7f37d2ffd700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f37c00692a0 0x7f37c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.862+0000 7f37d2ffd700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f37c00692a0 0x7f37c006b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f37c400b3b0 tx=0x7f37c4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:34.865+0000 7f37cbfff700 1 -- 172.21.15.32:0/1416552138 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f37bc056fe0 con 0x7f37d40fba10 2024-04-06T14:14:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.061+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f37d41043d0 con 0x7f37d40fba10 2024-04-06T14:14:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.062+0000 7f37cbfff700 1 -- 172.21.15.32:0/1416552138 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f37bc056b70 con 0x7f37d40fba10 2024-04-06T14:14:35.062 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-06T14:14:35.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f37c00692a0 msgr2=0x7f37c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:35.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f37c00692a0 0x7f37c006b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f37c400b3b0 tx=0x7f37c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:35.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fba10 msgr2=0x7f37d410c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:35.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fba10 0x7f37d410c490 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00c830 tx=0x7f37bc004470 comp rx=0 tx=0).stop 2024-04-06T14:14:35.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 shutdown_connections 2024-04-06T14:14:35.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f37c00692a0 0x7f37c006b750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d40fba10 0x7f37d410c490 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 --2- 172.21.15.32:0/1416552138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37d40fccc0 0x7f37d410c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 >> 172.21.15.32:0/1416552138 conn(0x7f37d40f7190 msgr2=0x7f37d40f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:35.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 shutdown_connections 2024-04-06T14:14:35.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.064+0000 7f37d8f9f700 1 -- 172.21.15.32:0/1416552138 wait complete. 2024-04-06T14:14:35.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:34 smithi032 ceph-mon[39730]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-06T14:14:35.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:34 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/748864777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:14:35.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:34 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1181361263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:14:35.075 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:34 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2947559910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T14:14:35.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T14:14:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:34 smithi086 ceph-mon[47925]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-06T14:14:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:34 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/748864777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:14:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:34 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1181361263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:14:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:34 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2947559910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T14:14:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 -- 172.21.15.32:0/3701408700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fbbd0 msgr2=0x7f481c0fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 --2- 172.21.15.32:0/3701408700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fbbd0 0x7f481c0fbfe0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4818004940 tx=0x7f4818009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 -- 172.21.15.32:0/3701408700 shutdown_connections 2024-04-06T14:14:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 --2- 172.21.15.32:0/3701408700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f481c0fce80 0x7f481c0fd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 --2- 172.21.15.32:0/3701408700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fbbd0 0x7f481c0fbfe0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 -- 172.21.15.32:0/3701408700 >> 172.21.15.32:0/3701408700 conn(0x7f481c0f7310 msgr2=0x7f481c0f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 -- 172.21.15.32:0/3701408700 shutdown_connections 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 -- 172.21.15.32:0/3701408700 wait complete. 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 Processor -- start 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.197+0000 7f4824515700 1 -- start start 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4824515700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f481c0fbbd0 0x7f481c065820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4824515700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fce80 0x7f481c112550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4824515700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c112a90 con 0x7f481c0fce80 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4824515700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c112bd0 con 0x7f481c0fbbd0 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4821ab0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fce80 0x7f481c112550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4821ab0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fce80 0x7f481c112550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48508/0 (socket says 172.21.15.32:48508) 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4821ab0700 1 -- 172.21.15.32:0/2206197023 learned_addr learned my addr 172.21.15.32:0/2206197023 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f48222b1700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f481c0fbbd0 0x7f481c065820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4821ab0700 1 -- 172.21.15.32:0/2206197023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f481c0fbbd0 msgr2=0x7f481c065820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4821ab0700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f481c0fbbd0 0x7f481c065820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4821ab0700 1 -- 172.21.15.32:0/2206197023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4818005040 con 0x7f481c0fce80 2024-04-06T14:14:35.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f48222b1700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f481c0fbbd0 0x7f481c065820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:14:35.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4821ab0700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fce80 0x7f481c112550 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4810002a50 tx=0x7f48100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:35.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f480effd700 1 -- 172.21.15.32:0/2206197023 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4810007cc0 con 0x7f481c0fce80 2024-04-06T14:14:35.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f480effd700 1 -- 172.21.15.32:0/2206197023 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f481000f040 con 0x7f481c0fce80 2024-04-06T14:14:35.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c112e30 con 0x7f481c0fce80 2024-04-06T14:14:35.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f480effd700 1 -- 172.21.15.32:0/2206197023 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4810009540 con 0x7f481c0fce80 2024-04-06T14:14:35.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.198+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c1133c0 con 0x7f481c0fce80 2024-04-06T14:14:35.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.199+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c064970 con 0x7f481c0fce80 2024-04-06T14:14:35.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.200+0000 7f480effd700 1 -- 172.21.15.32:0/2206197023 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f48100096a0 con 0x7f481c0fce80 2024-04-06T14:14:35.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.200+0000 7f480effd700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f48080692a0 0x7f480806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:35.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.200+0000 7f480effd700 1 -- 172.21.15.32:0/2206197023 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4810086f40 con 0x7f481c0fce80 2024-04-06T14:14:35.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.200+0000 7f48222b1700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f48080692a0 0x7f480806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:35.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.200+0000 7f48222b1700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f48080692a0 0x7f480806b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f481800b3b0 tx=0x7f4818009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:35.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.204+0000 7f480effd700 1 -- 172.21.15.32:0/2206197023 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4810056cb0 con 0x7f481c0fce80 2024-04-06T14:14:35.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.447+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f481c104580 con 0x7f481c0fce80 2024-04-06T14:14:35.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.447+0000 7f480effd700 1 -- 172.21.15.32:0/2206197023 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4700 (secure 0 0 0) 0x7f4810056840 con 0x7f481c0fce80 2024-04-06T14:14:35.447 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:14:35.448 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14472,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/179354627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":179354627},{"type":"v1","addr":"172.21.15.32:6827","nonce":179354627}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:14:20.970955+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14460,"mds_1":24301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14460":{"gid":14460,"name":"cephfs.smithi032.mqoaof","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/1704327767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1704327767},{"type":"v1","addr":"172.21.15.32:6829","nonce":1704327767}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24301":{"gid":24301,"name":"cephfs.smithi086.nikfjj","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6825/3246813202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3246813202},{"type":"v1","addr":"172.21.15.86:6825","nonce":3246813202}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-04-06T14:14:35.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.449+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f48080692a0 msgr2=0x7f480806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.449+0000 7f4824515700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f48080692a0 0x7f480806b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f481800b3b0 tx=0x7f4818009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.449+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fce80 msgr2=0x7f481c112550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.449+0000 7f4824515700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fce80 0x7f481c112550 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4810002a50 tx=0x7f48100046b0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.450+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 shutdown_connections 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.450+0000 7f4824515700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f48080692a0 0x7f480806b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.450+0000 7f4824515700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f481c0fbbd0 0x7f481c065820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.450+0000 7f4824515700 1 --2- 172.21.15.32:0/2206197023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f481c0fce80 0x7f481c112550 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.450+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 >> 172.21.15.32:0/2206197023 conn(0x7f481c0f7310 msgr2=0x7f481c0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.450+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 shutdown_connections 2024-04-06T14:14:35.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:35.450+0000 7f4824515700 1 -- 172.21.15.32:0/2206197023 wait complete. 2024-04-06T14:14:35.451 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2024-04-06T14:14:35.462 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 18} 2024-04-06T14:14:35.462 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-06T14:14:35.488 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-06T14:14:35.489 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-06T14:14:35.489 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-06T14:14:35.489 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-06T14:14:35.489 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:35.489 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-06T14:14:35.489 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-06T14:14:35.489 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:35.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:35.490 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list 2024-04-06T14:14:35.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:35.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link delete ceph-brx 2024-04-06T14:14:35.592 INFO:teuthology.orchestra.run.smithi086.stderr:Cannot find device "ceph-brx" 2024-04-06T14:14:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T14:14:35.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:35.593 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-06T14:14:35.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:35.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-04-06T14:14:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-04-06T14:14:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T14:14:35.685 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-06T14:14:35.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph fs ls 2024-04-06T14:14:35.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:35 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1416552138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T14:14:35.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:35 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2206197023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T14:14:36.002 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:35 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1416552138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T14:14:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:35 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2206197023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T14:14:37.074 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:36 smithi032 ceph-mon[39730]: pgmap v203: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-06T14:14:37.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:36 smithi086 ceph-mon[47925]: pgmap v203: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-06T14:14:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.936+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3253995535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 msgr2=0x7f0cd80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.936+0000 7f0cdda2d700 1 --2- 172.21.15.32:0/3253995535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 0x7f0cd80fbd80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8004940 tx=0x7f0cc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.936+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3253995535 shutdown_connections 2024-04-06T14:14:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.936+0000 7f0cdda2d700 1 --2- 172.21.15.32:0/3253995535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 0x7f0cd80fbd80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.936+0000 7f0cdda2d700 1 --2- 172.21.15.32:0/3253995535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd8101d70 0x7f0cd8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.936+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3253995535 >> 172.21.15.32:0/3253995535 conn(0x7f0cd8072bc0 msgr2=0x7f0cd8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.936+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3253995535 shutdown_connections 2024-04-06T14:14:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.936+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3253995535 wait complete. 2024-04-06T14:14:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cdda2d700 1 Processor -- start 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cdda2d700 1 -- start start 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cdda2d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd8101d70 0x7f0cd81139d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cdda2d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 0x7f0cd8113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cdda2d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8114560 con 0x7f0cd8102710 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cdda2d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd810d990 con 0x7f0cd8101d70 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cd67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 0x7f0cd8113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cd67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 0x7f0cd8113f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48514/0 (socket says 172.21.15.32:48514) 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cd67fc700 1 -- 172.21.15.32:0/3436042448 learned_addr learned my addr 172.21.15.32:0/3436042448 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cd6ffd700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd8101d70 0x7f0cd81139d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cd67fc700 1 -- 172.21.15.32:0/3436042448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd8101d70 msgr2=0x7f0cd81139d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cd67fc700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd8101d70 0x7f0cd81139d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.937+0000 7f0cd67fc700 1 -- 172.21.15.32:0/3436042448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc8005040 con 0x7f0cd8102710 2024-04-06T14:14:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.938+0000 7f0cd67fc700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 0x7f0cd8113f10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80093b0 tx=0x7f0cc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.938+0000 7f0cbf7fe700 1 -- 172.21.15.32:0/3436042448 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cc800e040 con 0x7f0cd8102710 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.938+0000 7f0cbf7fe700 1 -- 172.21.15.32:0/3436042448 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cc8013420 con 0x7f0cd8102710 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.938+0000 7f0cbf7fe700 1 -- 172.21.15.32:0/3436042448 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cc801c630 con 0x7f0cd8102710 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.938+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd810dc10 con 0x7f0cd8102710 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.938+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd810e180 con 0x7f0cd8102710 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.939+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd80fd4c0 con 0x7f0cd8102710 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.939+0000 7f0cbf7fe700 1 -- 172.21.15.32:0/3436042448 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0cc800bb80 con 0x7f0cd8102710 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.940+0000 7f0cbf7fe700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0cc00692a0 0x7f0cc006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.940+0000 7f0cbf7fe700 1 -- 172.21.15.32:0/3436042448 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0cc80883d0 con 0x7f0cd8102710 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.940+0000 7f0cd6ffd700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0cc00692a0 0x7f0cc006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.940+0000 7f0cd6ffd700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0cc00692a0 0x7f0cc006b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009fd0 tx=0x7f0ccc009f60 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:37.944+0000 7f0cbf7fe700 1 -- 172.21.15.32:0/3436042448 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0cc8058140 con 0x7f0cd8102710 2024-04-06T14:14:38.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.184+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0cd8108d10 con 0x7f0cd8102710 2024-04-06T14:14:38.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.184+0000 7f0cbf7fe700 1 -- 172.21.15.32:0/3436042448 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f0cc8057cd0 con 0x7f0cd8102710 2024-04-06T14:14:38.184 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-06T14:14:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0cc00692a0 msgr2=0x7f0cc006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0cc00692a0 0x7f0cc006b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009fd0 tx=0x7f0ccc009f60 comp rx=0 tx=0).stop 2024-04-06T14:14:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 msgr2=0x7f0cd8113f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 0x7f0cd8113f10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80093b0 tx=0x7f0cc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 shutdown_connections 2024-04-06T14:14:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0cc00692a0 0x7f0cc006b750 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd8101d70 0x7f0cd81139d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 --2- 172.21.15.32:0/3436042448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0cd8102710 0x7f0cd8113f10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 >> 172.21.15.32:0/3436042448 conn(0x7f0cd8072bc0 msgr2=0x7f0cd8065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 shutdown_connections 2024-04-06T14:14:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:38.186+0000 7f0cdda2d700 1 -- 172.21.15.32:0/3436042448 wait complete. 2024-04-06T14:14:38.891 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-06T14:14:38.891 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T14:14:38.891 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-04-06T14:14:38.891 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T14:14:38.891 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:38.891 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-06T14:14:38.891 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-06T14:14:38.891 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T14:14:38.891 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T14:14:38.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:38.892 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-04-06T14:14:38.913 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T14:14:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T14:14:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T14:14:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-06T14:14:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42050sec preferred_lft 42050sec 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-06T14:14:38.914 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T14:14:38.915 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-04-06T14:14:38.915 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-04-06T14:14:38.915 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-06T14:14:38.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:38.915 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-06T14:14:38.915 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-06T14:14:38.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add name ceph-brx type bridge 2024-04-06T14:14:38.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr flush dev ceph-brx 2024-04-06T14:14:38.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set ceph-brx up 2024-04-06T14:14:38.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-06T14:14:38.916 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-06T14:14:39.068 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:38 smithi032 ceph-mon[39730]: pgmap v204: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T14:14:39.068 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:38 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3436042448' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-06T14:14:39.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:39.083 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T14:14:39.113 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-04-06T14:14:39.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:39.114 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-04-06T14:14:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:38 smithi086 ceph-mon[47925]: pgmap v204: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T14:14:39.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:38 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3436042448' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-06T14:14:39.187 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-04-06T14:14:39.187 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T14:14:39.187 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-06T14:14:39.187 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-06T14:14:39.187 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T14:14:39.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:39.188 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-06T14:14:39.188 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-06T14:14:39.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-06T14:14:39.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-06T14:14:39.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-06T14:14:39.188 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-06T14:14:39.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:39.459 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-06T14:14:39.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:39.479 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-04-06T14:14:39.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:39.537 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-06T14:14:39.537 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-06T14:14:39.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T14:14:39.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-06T14:14:39.538 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-06T14:14:39.634 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-06T14:14:39.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:39.635 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-06T14:14:39.635 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-06T14:14:39.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-06T14:14:39.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-06T14:14:39.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-06T14:14:39.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-06T14:14:39.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-06T14:14:39.635 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-06T14:14:39.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:39.828 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-06T14:14:39.828 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-06T14:14:39.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-04-06T14:14:39.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.0 master ceph-brx 2024-04-06T14:14:39.828 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-06T14:14:39.927 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-06T14:14:39.927 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T14:14:39.927 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:39.945 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T14:14:39.946 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T14:14:39.946 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:40.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-04-06T14:14:40.072 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-06T14:14:40.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-06T14:14:40.139 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-06T14:14:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-06T14:14:40.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:40.141 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-04-06T14:14:40.199 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-06T14:14:40.199 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-06T14:14:40.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-04-06T14:14:40.270 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-06T14:14:40.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-06T14:14:40.338 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-06T14:14:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-06T14:14:40.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:40.340 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-04-06T14:14:40.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2024-04-06T14:14:40.360+0000 7fb4b8051540 -1 init, newargv = 0x562bdc43fe60 newargc=15 2024-04-06T14:14:40.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[100693]: starting ceph client 2024-04-06T14:14:40.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[100693]: starting fuse 2024-04-06T14:14:40.383 INFO:teuthology.orchestra.run.smithi032.stdout:161 2024-04-06T14:14:40.383 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [161] 2024-04-06T14:14:40.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-04-06T14:14:40.384 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2024-04-06T14:14:40.384 DEBUG:teuthology.orchestra.run.smithi032:> import re 2024-04-06T14:14:40.384 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-04-06T14:14:40.384 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2024-04-06T14:14:40.384 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-06T14:14:40.384 DEBUG:teuthology.orchestra.run.smithi032:> def _find_admin_socket(client_name): 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2024-04-06T14:14:40.385 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-06T14:14:40.386 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-06T14:14:40.386 DEBUG:teuthology.orchestra.run.smithi032:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-06T14:14:40.386 DEBUG:teuthology.orchestra.run.smithi032:> contents = proc_f.read() 2024-04-06T14:14:40.386 DEBUG:teuthology.orchestra.run.smithi032:> if mountpoint in contents: 2024-04-06T14:14:40.386 DEBUG:teuthology.orchestra.run.smithi032:> return f 2024-04-06T14:14:40.386 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-06T14:14:40.386 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-06T14:14:40.386 DEBUG:teuthology.orchestra.run.smithi032:> print(_find_admin_socket("client.0")) 2024-04-06T14:14:40.387 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-04-06T14:14:40.508 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.100693.asok 2024-04-06T14:14:40.691 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.100693.asok 2024-04-06T14:14:40.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:40.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.100693.asok status 2024-04-06T14:14:40.849 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:14:40.849 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2024-04-06T14:14:40.849 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "6699495ce2af437c1ff648f1736d2457f417a1aa", 2024-04-06T14:14:40.849 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)", 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "100693", 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 14516, 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 14516 2024-04-06T14:14:40.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "192.168.0.1:0", 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 714996764 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "192.168.0.1:0", 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 714996764 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.14516 192.168.0.1:0/714996764", 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "192.168.0.1:0/714996764", 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2024-04-06T14:14:40.851 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 18, 2024-04-06T14:14:40.852 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 46, 2024-04-06T14:14:40.852 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2024-04-06T14:14:40.852 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklisted": false, 2024-04-06T14:14:40.852 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2024-04-06T14:14:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:14:40.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- ceph fs ls 2024-04-06T14:14:40.979 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:40 smithi032 ceph-mon[39730]: pgmap v205: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:40 smithi086 ceph-mon[47925]: pgmap v205: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:14:41.135 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.109+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/2339366577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81026c0 msgr2=0x7f9cd80fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.109+0000 7f9cdfc9a700 1 --2- 172.21.15.32:0/2339366577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81026c0 0x7f9cd80fbd30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc004940 tx=0x7f9ccc009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.109+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/2339366577 shutdown_connections 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.109+0000 7f9cdfc9a700 1 --2- 172.21.15.32:0/2339366577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81026c0 0x7f9cd80fbd30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.109+0000 7f9cdfc9a700 1 --2- 172.21.15.32:0/2339366577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd8101d20 0x7f9cd81020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.109+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/2339366577 >> 172.21.15.32:0/2339366577 conn(0x7f9cd80f7870 msgr2=0x7f9cd80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/2339366577 shutdown_connections 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/2339366577 wait complete. 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdfc9a700 1 Processor -- start 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdfc9a700 1 -- start start 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdfc9a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd8101d20 0x7f9cd81138e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdfc9a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd81026c0 0x7f9cd8113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdfc9a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd8114470 con 0x7f9cd8101d20 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdfc9a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd810da70 con 0x7f9cd81026c0 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdda36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd8101d20 0x7f9cd81138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdda36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd8101d20 0x7f9cd81138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48538/0 (socket says 172.21.15.32:48538) 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.110+0000 7f9cdda36700 1 -- 172.21.15.32:0/1840693602 learned_addr learned my addr 172.21.15.32:0/1840693602 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cdd235700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd81026c0 0x7f9cd8113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cdda36700 1 -- 172.21.15.32:0/1840693602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd81026c0 msgr2=0x7f9cd8113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cdda36700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd81026c0 0x7f9cd8113e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cdda36700 1 -- 172.21.15.32:0/1840693602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc005040 con 0x7f9cd8101d20 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cdda36700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd8101d20 0x7f9cd81138e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4002a50 tx=0x7f9cd40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cca7fc700 1 -- 172.21.15.32:0/1840693602 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd4007cc0 con 0x7f9cd8101d20 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cca7fc700 1 -- 172.21.15.32:0/1840693602 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cd400f040 con 0x7f9cd8101d20 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd810dd50 con 0x7f9cd8101d20 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cca7fc700 1 -- 172.21.15.32:0/1840693602 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd4009540 con 0x7f9cd8101d20 2024-04-06T14:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.111+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd810e310 con 0x7f9cd8101d20 2024-04-06T14:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.112+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd80fd470 con 0x7f9cd8101d20 2024-04-06T14:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.113+0000 7f9cca7fc700 1 -- 172.21.15.32:0/1840693602 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9cd40096a0 con 0x7f9cd8101d20 2024-04-06T14:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.114+0000 7f9cca7fc700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9cc40692a0 0x7f9cc406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.114+0000 7f9cca7fc700 1 -- 172.21.15.32:0/1840693602 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9cd40874f0 con 0x7f9cd8101d20 2024-04-06T14:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.114+0000 7f9cdd235700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9cc40692a0 0x7f9cc406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.114+0000 7f9cdd235700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9cc40692a0 0x7f9cc406b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9cd810ec30 tx=0x7f9ccc009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:43.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.118+0000 7f9cca7fc700 1 -- 172.21.15.32:0/1840693602 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9cd4057300 con 0x7f9cd8101d20 2024-04-06T14:14:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:42 smithi086 ceph-mon[47925]: pgmap v206: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T14:14:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:14:43.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:42 smithi032 ceph-mon[39730]: pgmap v206: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T14:14:43.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:14:43.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.360+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9cd8108cf0 con 0x7f9cd8101d20 2024-04-06T14:14:43.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.360+0000 7f9cca7fc700 1 -- 172.21.15.32:0/1840693602 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f9cd4056e90 con 0x7f9cd8101d20 2024-04-06T14:14:43.361 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-06T14:14:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9cc40692a0 msgr2=0x7f9cc406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9cc40692a0 0x7f9cc406b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9cd810ec30 tx=0x7f9ccc009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd8101d20 msgr2=0x7f9cd81138e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd8101d20 0x7f9cd81138e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4002a50 tx=0x7f9cd40046b0 comp rx=0 tx=0).stop 2024-04-06T14:14:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 shutdown_connections 2024-04-06T14:14:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f9cc40692a0 0x7f9cc406b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd8101d20 0x7f9cd81138e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 --2- 172.21.15.32:0/1840693602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd81026c0 0x7f9cd8113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:43.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 >> 172.21.15.32:0/1840693602 conn(0x7f9cd80f7870 msgr2=0x7f9cd80f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:43.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 shutdown_connections 2024-04-06T14:14:43.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:43.362+0000 7f9cdfc9a700 1 -- 172.21.15.32:0/1840693602 wait complete. 2024-04-06T14:14:44.044 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:43 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1840693602' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-06T14:14:44.065 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-06T14:14:44.066 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T14:14:44.066 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi086.front.sepia.ceph.com 2024-04-06T14:14:44.066 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-06T14:14:44.066 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:44.066 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-06T14:14:44.066 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-06T14:14:44.066 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T14:14:44.066 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-06T14:14:44.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:44.067 DEBUG:teuthology.orchestra.run.smithi086:> ip addr 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 ::1/128 scope host 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:d8:da:f6 brd ff:ff:ff:ff:ff:ff 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: altname enp7s0f0 2024-04-06T14:14:44.090 INFO:teuthology.orchestra.run.smithi086.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:d8:da:f7 brd ff:ff:ff:ff:ff:ff 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: altname enp7s0f1 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:8f:d8:08 brd ff:ff:ff:ff:ff:ff 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:8f:d8:09 brd ff:ff:ff:ff:ff:ff 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: inet 172.21.15.86/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft 42055sec preferred_lft 42055sec 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 fe80::ec4:7aff:fe8f:d809/64 scope link 2024-04-06T14:14:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-06T14:14:44.091 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-06T14:14:44.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:44.092 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-06T14:14:44.092 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-06T14:14:44.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link add name ceph-brx type bridge 2024-04-06T14:14:44.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip addr flush dev ceph-brx 2024-04-06T14:14:44.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set ceph-brx up 2024-04-06T14:14:44.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-06T14:14:44.092 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-06T14:14:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:43 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1840693602' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-06T14:14:44.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:44.278 DEBUG:teuthology.orchestra.run.smithi086:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T14:14:44.306 INFO:teuthology.orchestra.run.smithi086.stdout:1 2024-04-06T14:14:44.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:44.308 DEBUG:teuthology.orchestra.run.smithi086:> route 2024-04-06T14:14:44.394 INFO:teuthology.orchestra.run.smithi086.stdout:Kernel IP routing table 2024-04-06T14:14:44.395 INFO:teuthology.orchestra.run.smithi086.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T14:14:44.395 INFO:teuthology.orchestra.run.smithi086.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T14:14:44.395 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T14:14:44.395 INFO:teuthology.orchestra.run.smithi086.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T14:14:44.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:44.397 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-06T14:14:44.397 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-06T14:14:44.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T14:14:44.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T14:14:44.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T14:14:44.397 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-06T14:14:44.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:44.647 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list 2024-04-06T14:14:44.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:44.665 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list-id 2024-04-06T14:14:44.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:44.723 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-06T14:14:44.723 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-06T14:14:44.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-06T14:14:44.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-06T14:14:44.723 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-06T14:14:44.817 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-06T14:14:44.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:44.974 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-06T14:14:44.974 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-06T14:14:44.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-06T14:14:44.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-06T14:14:44.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-06T14:14:44.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-06T14:14:44.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-06T14:14:44.975 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-06T14:14:45.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:44 smithi086 ceph-mon[47925]: pgmap v207: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T14:14:45.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:45.210 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-06T14:14:45.210 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-06T14:14:45.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set brx.0 up 2024-04-06T14:14:45.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set dev brx.0 master ceph-brx 2024-04-06T14:14:45.210 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-06T14:14:45.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:44 smithi032 ceph-mon[39730]: pgmap v207: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T14:14:45.307 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-06T14:14:45.308 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T14:14:45.308 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:45.325 INFO:teuthology.orchestra.run.smithi086.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-06T14:14:45.326 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T14:14:45.326 DEBUG:teuthology.orchestra.run.smithi086:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:45.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo modprobe fuse 2024-04-06T14:14:45.452 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-06T14:14:45.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-06T14:14:45.519 INFO:teuthology.orchestra.run.smithi086.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-06T14:14:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-06T14:14:45.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:45.520 DEBUG:teuthology.orchestra.run.smithi086:> ls /sys/fs/fuse/connections 2024-04-06T14:14:45.578 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-06T14:14:45.579 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-06T14:14:45.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo modprobe fuse 2024-04-06T14:14:45.647 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-06T14:14:45.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-06T14:14:45.714 INFO:teuthology.orchestra.run.smithi086.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-06T14:14:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-06T14:14:45.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:45.716 DEBUG:teuthology.orchestra.run.smithi086:> ls /sys/fs/fuse/connections 2024-04-06T14:14:45.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi086.stderr:2024-04-06T14:14:45.719+0000 7f1003883540 -1 init, newargv = 0x55bc56c5dde0 newargc=15 2024-04-06T14:14:45.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi086.stderr:ceph-fuse[79576]: starting ceph client 2024-04-06T14:14:45.726 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi086.stderr:ceph-fuse[79576]: starting fuse 2024-04-06T14:14:45.737 INFO:teuthology.orchestra.run.smithi086.stdout:125 2024-04-06T14:14:45.737 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [125] 2024-04-06T14:14:45.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-04-06T14:14:45.737 DEBUG:teuthology.orchestra.run.smithi086:> import glob 2024-04-06T14:14:45.737 DEBUG:teuthology.orchestra.run.smithi086:> import re 2024-04-06T14:14:45.737 DEBUG:teuthology.orchestra.run.smithi086:> import os 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> import subprocess 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> def _find_admin_socket(client_name): 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> files = glob.glob(asok_path) 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> # Given a non-glob path, it better be there 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> if "*" not in asok_path: 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> assert(len(files) == 1) 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> return files[0] 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> 2024-04-06T14:14:45.738 DEBUG:teuthology.orchestra.run.smithi086:> for f in files: 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> contents = proc_f.read() 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> if mountpoint in contents: 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> return f 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> print(_find_admin_socket("client.1")) 2024-04-06T14:14:45.739 DEBUG:teuthology.orchestra.run.smithi086:> ' 2024-04-06T14:14:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:/var/run/ceph/ceph-client.1.79576.asok 2024-04-06T14:14:46.043 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.79576.asok 2024-04-06T14:14:46.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:46.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.79576.asok status 2024-04-06T14:14:46.195 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-06T14:14:46.195 INFO:teuthology.orchestra.run.smithi086.stdout: "metadata": { 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_sha1": "6699495ce2af437c1ff648f1736d2457f417a1aa", 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)", 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "entity_id": "1", 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "hostname": "smithi086", 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "pid": "79576", 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "root": "/" 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "dentry_count": 0, 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "dentry_pinned_count": 0, 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 24357, 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "inst": { 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "name": { 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "client", 2024-04-06T14:14:46.196 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 24357 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": { 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "192.168.0.1:0", 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 3886073944 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": { 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "192.168.0.1:0", 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 3886073944 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "inst_str": "client.24357 192.168.0.1:0/3886073944", 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "addr_str": "192.168.0.1:0/3886073944", 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "inode_count": 1, 2024-04-06T14:14:46.197 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_epoch": 18, 2024-04-06T14:14:46.198 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_epoch": 46, 2024-04-06T14:14:46.198 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_epoch_barrier": 0, 2024-04-06T14:14:46.198 INFO:teuthology.orchestra.run.smithi086.stdout: "blocklisted": false, 2024-04-06T14:14:46.198 INFO:teuthology.orchestra.run.smithi086.stdout: "fs_name": "cephfs" 2024-04-06T14:14:46.198 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-06T14:14:46.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:46.205 DEBUG:teuthology.orchestra.run.smithi032:> stat --file-system '--printf=%T 2024-04-06T14:14:46.205 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:46.231 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2024-04-06T14:14:46.232 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:46.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:46.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:46.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:46.311 DEBUG:teuthology.orchestra.run.smithi086:> stat --file-system '--printf=%T 2024-04-06T14:14:46.311 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:46.331 INFO:teuthology.orchestra.run.smithi086.stdout:fuseblk 2024-04-06T14:14:46.331 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:46.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:14:46.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:46 smithi086 ceph-mon[47925]: pgmap v208: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T14:14:47.175 INFO:teuthology.run_tasks:Running task print... 2024-04-06T14:14:47.186 INFO:teuthology.task.print:**** done client 2024-04-06T14:14:47.186 INFO:teuthology.run_tasks:Running task parallel... 2024-04-06T14:14:47.217 INFO:teuthology.task.parallel:starting parallel... 2024-04-06T14:14:47.217 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-06T14:14:47.218 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-06T14:14:47.218 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:14:47.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-06T14:14:47.219 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-06T14:14:47.219 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-06T14:14:47.222 INFO:tasks.workunit:Pulling workunits from ref 800ec6d64a7fa0648ad53edfc6e50b41fc4cd674 2024-04-06T14:14:47.222 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-06T14:14:47.223 INFO:tasks.workunit:timeout=3h 2024-04-06T14:14:47.223 INFO:tasks.workunit:cleanup=True 2024-04-06T14:14:47.223 DEBUG:teuthology.orchestra.run.smithi032:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:46 smithi032 ceph-mon[39730]: pgmap v208: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T14:14:47.247 INFO:teuthology.orchestra.run.smithi032.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:47.247 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-06T14:14:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:Device: a1h/161d Inode: 1 Links: 2 2024-04-06T14:14:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:14:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-06T14:14:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-06T14:14:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-06 14:14:06.064672602 +0000 2024-04-06T14:14:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-06 14:14:46.308520491 +0000 2024-04-06T14:14:47.248 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-06T14:14:47.248 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-06T14:14:47.248 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-06T14:14:47.325 DEBUG:teuthology.orchestra.run.smithi086:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:47.348 INFO:teuthology.orchestra.run.smithi086.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:47.348 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-06T14:14:47.348 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 7dh/125d Inode: 1 Links: 3 2024-04-06T14:14:47.348 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T14:14:47.348 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-06T14:14:47.348 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-06T14:14:47.348 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-06 14:14:47.320338990 +0000 2024-04-06T14:14:47.348 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-06 14:14:47.320338990 +0000 2024-04-06T14:14:47.349 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-04-06T14:14:47.349 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-06T14:14:47.349 DEBUG:teuthology.orchestra.run.smithi086:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-06T14:14:47.425 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 800ec6d64a7fa0648ad53edfc6e50b41fc4cd674 2024-04-06T14:14:47.426 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 800ec6d64a7fa0648ad53edfc6e50b41fc4cd674 2024-04-06T14:14:47.446 INFO:tasks.workunit.client.0.smithi032.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-06T14:14:47.485 INFO:tasks.workunit.client.1.smithi086.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-06T14:14:47.521 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:49.121 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:48 smithi032 ceph-mon[39730]: pgmap v209: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-06T14:14:49.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:48 smithi086 ceph-mon[47925]: pgmap v209: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-06T14:14:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.476+0000 7fd2020d3700 1 -- 172.21.15.32:0/1272797002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc066280 msgr2=0x7fd1fc0ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.476+0000 7fd2020d3700 1 --2- 172.21.15.32:0/1272797002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc066280 0x7fd1fc0ffa20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec004940 tx=0x7fd1ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.476+0000 7fd2020d3700 1 -- 172.21.15.32:0/1272797002 shutdown_connections 2024-04-06T14:14:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.476+0000 7fd2020d3700 1 --2- 172.21.15.32:0/1272797002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1fc0fff60 0x7fd1fc102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.476+0000 7fd2020d3700 1 --2- 172.21.15.32:0/1272797002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc066280 0x7fd1fc0ffa20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.476+0000 7fd2020d3700 1 -- 172.21.15.32:0/1272797002 >> 172.21.15.32:0/1272797002 conn(0x7fd1fc0f7430 msgr2=0x7fd1fc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:49.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.477+0000 7fd2020d3700 1 -- 172.21.15.32:0/1272797002 shutdown_connections 2024-04-06T14:14:49.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.477+0000 7fd2020d3700 1 -- 172.21.15.32:0/1272797002 wait complete. 2024-04-06T14:14:49.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.477+0000 7fd2020d3700 1 Processor -- start 2024-04-06T14:14:49.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.477+0000 7fd2020d3700 1 -- start start 2024-04-06T14:14:49.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.477+0000 7fd2020d3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1fc066280 0x7fd1fc10d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:49.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.477+0000 7fd2020d3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc0fff60 0x7fd1fc10df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:49.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.477+0000 7fd2020d3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc10e520 con 0x7fd1fc0fff60 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.477+0000 7fd2020d3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc10ed20 con 0x7fd1fc066280 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc0fff60 0x7fd1fc10df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc0fff60 0x7fd1fc10df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56948/0 (socket says 172.21.15.32:56948) 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1faffd700 1 -- 172.21.15.32:0/491836323 learned_addr learned my addr 172.21.15.32:0/491836323 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1fb7fe700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1fc066280 0x7fd1fc10d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1faffd700 1 -- 172.21.15.32:0/491836323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1fc066280 msgr2=0x7fd1fc10d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1faffd700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1fc066280 0x7fd1fc10d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1faffd700 1 -- 172.21.15.32:0/491836323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ec005040 con 0x7fd1fc0fff60 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1faffd700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc0fff60 0x7fd1fc10df20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000c830 tx=0x7fd1f0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1e3fff700 1 -- 172.21.15.32:0/491836323 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1f000a760 con 0x7fd1fc0fff60 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1e3fff700 1 -- 172.21.15.32:0/491836323 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1f000ad70 con 0x7fd1fc0fff60 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1fc10f000 con 0x7fd1fc0fff60 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd1e3fff700 1 -- 172.21.15.32:0/491836323 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1f00113a0 con 0x7fd1fc0fff60 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.478+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1fc10f540 con 0x7fd1fc0fff60 2024-04-06T14:14:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.479+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1fc05f5f0 con 0x7fd1fc0fff60 2024-04-06T14:14:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.480+0000 7fd1e3fff700 1 -- 172.21.15.32:0/491836323 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd1f0012400 con 0x7fd1fc0fff60 2024-04-06T14:14:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.480+0000 7fd1e3fff700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd1e40692a0 0x7fd1e406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.480+0000 7fd1e3fff700 1 -- 172.21.15.32:0/491836323 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd1f0087230 con 0x7fd1fc0fff60 2024-04-06T14:14:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.481+0000 7fd1fb7fe700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd1e40692a0 0x7fd1e406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.481+0000 7fd1fb7fe700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd1e40692a0 0x7fd1e406b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00b900 tx=0x7fd1ec009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.485+0000 7fd1e3fff700 1 -- 172.21.15.32:0/491836323 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd1f0056fa0 con 0x7fd1fc0fff60 2024-04-06T14:14:49.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.681+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fd1fc04bed0 con 0x7fd1fc0fff60 2024-04-06T14:14:49.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.681+0000 7fd1e3fff700 1 -- 172.21.15.32:0/491836323 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fd1f0056b30 con 0x7fd1fc0fff60 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.683+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd1e40692a0 msgr2=0x7fd1e406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.683+0000 7fd2020d3700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd1e40692a0 0x7fd1e406b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00b900 tx=0x7fd1ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.683+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc0fff60 msgr2=0x7fd1fc10df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.683+0000 7fd2020d3700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc0fff60 0x7fd1fc10df20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000c830 tx=0x7fd1f0004470 comp rx=0 tx=0).stop 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.683+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 shutdown_connections 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.683+0000 7fd2020d3700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fd1e40692a0 0x7fd1e406b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.683+0000 7fd2020d3700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1fc066280 0x7fd1fc10d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.683+0000 7fd2020d3700 1 --2- 172.21.15.32:0/491836323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1fc0fff60 0x7fd1fc10df20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.683+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 >> 172.21.15.32:0/491836323 conn(0x7fd1fc0f7430 msgr2=0x7fd1fc0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.684+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 shutdown_connections 2024-04-06T14:14:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:49.684+0000 7fd2020d3700 1 -- 172.21.15.32:0/491836323 wait complete. 2024-04-06T14:14:50.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:50 smithi086 ceph-mon[47925]: pgmap v210: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-06T14:14:50.386 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:50 smithi032 ceph-mon[39730]: pgmap v210: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-06T14:14:50.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-06T14:14:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:52.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.845+0000 7fc632d63700 1 -- 172.21.15.32:0/1046832768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 msgr2=0x7fc62c0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:52.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.845+0000 7fc632d63700 1 --2- 172.21.15.32:0/1046832768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 0x7fc62c0fc0e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc620004940 tx=0x7fc620009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:52.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.845+0000 7fc632d63700 1 -- 172.21.15.32:0/1046832768 shutdown_connections 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.845+0000 7fc632d63700 1 --2- 172.21.15.32:0/1046832768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc62c0fcf80 0x7fc62c0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.845+0000 7fc632d63700 1 --2- 172.21.15.32:0/1046832768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 0x7fc62c0fc0e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.845+0000 7fc632d63700 1 -- 172.21.15.32:0/1046832768 >> 172.21.15.32:0/1046832768 conn(0x7fc62c0f7430 msgr2=0x7fc62c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.845+0000 7fc632d63700 1 -- 172.21.15.32:0/1046832768 shutdown_connections 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.845+0000 7fc632d63700 1 -- 172.21.15.32:0/1046832768 wait complete. 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.846+0000 7fc632d63700 1 Processor -- start 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.846+0000 7fc632d63700 1 -- start start 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.846+0000 7fc632d63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 0x7fc62c10d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.846+0000 7fc632d63700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc62c0fcf80 0x7fc62c10dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.846+0000 7fc632d63700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62c10e4c0 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.846+0000 7fc632d63700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62c10ed10 con 0x7fc62c0fcf80 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.846+0000 7fc630aff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 0x7fc62c10d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.846+0000 7fc630aff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 0x7fc62c10d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56970/0 (socket says 172.21.15.32:56970) 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc630aff700 1 -- 172.21.15.32:0/415222724 learned_addr learned my addr 172.21.15.32:0/415222724 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc62bfff700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc62c0fcf80 0x7fc62c10dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc630aff700 1 -- 172.21.15.32:0/415222724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc62c0fcf80 msgr2=0x7fc62c10dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc630aff700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc62c0fcf80 0x7fc62c10dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc630aff700 1 -- 172.21.15.32:0/415222724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc620005040 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc62bfff700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc62c0fcf80 0x7fc62c10dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc630aff700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 0x7fc62c10d980 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc62000b3b0 tx=0x7fc620009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc6297fa700 1 -- 172.21.15.32:0/415222724 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc62000e040 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc6297fa700 1 -- 172.21.15.32:0/415222724 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc62000bd20 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62c10ef90 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc6297fa700 1 -- 172.21.15.32:0/415222724 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc62001c630 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.847+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc62c10f4d0 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.848+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc62c101b10 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.849+0000 7fc6297fa700 1 -- 172.21.15.32:0/415222724 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc620013420 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.849+0000 7fc6297fa700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc61c0692a0 0x7fc61c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:52.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.849+0000 7fc6297fa700 1 -- 172.21.15.32:0/415222724 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc620087fc0 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.849+0000 7fc62bfff700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc61c0692a0 0x7fc61c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:52.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.850+0000 7fc62bfff700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc61c0692a0 0x7fc61c06b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc618009fd0 tx=0x7fc618009ee0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:52.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:52.854+0000 7fc6297fa700 1 -- 172.21.15.32:0/415222724 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc620057cb0 con 0x7fc62c0fbcd0 2024-04-06T14:14:52.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:52 smithi032 ceph-mon[39730]: pgmap v211: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-04-06T14:14:53.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.051+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fc62c05f5f0 con 0x7fc62c0fbcd0 2024-04-06T14:14:53.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.051+0000 7fc6297fa700 1 -- 172.21.15.32:0/415222724 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7fc620057840 con 0x7fc62c0fbcd0 2024-04-06T14:14:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.053+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc61c0692a0 msgr2=0x7fc61c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.053+0000 7fc632d63700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc61c0692a0 0x7fc61c06b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc618009fd0 tx=0x7fc618009ee0 comp rx=0 tx=0).stop 2024-04-06T14:14:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.053+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 msgr2=0x7fc62c10d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.053+0000 7fc632d63700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 0x7fc62c10d980 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc62000b3b0 tx=0x7fc620009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.054+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 shutdown_connections 2024-04-06T14:14:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.054+0000 7fc632d63700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc61c0692a0 0x7fc61c06b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.054+0000 7fc632d63700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62c0fbcd0 0x7fc62c10d980 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.054+0000 7fc632d63700 1 --2- 172.21.15.32:0/415222724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc62c0fcf80 0x7fc62c10dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.054+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 >> 172.21.15.32:0/415222724 conn(0x7fc62c0f7430 msgr2=0x7fc62c065c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.054+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 shutdown_connections 2024-04-06T14:14:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:53.054+0000 7fc632d63700 1 -- 172.21.15.32:0/415222724 wait complete. 2024-04-06T14:14:53.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:52 smithi086 ceph-mon[47925]: pgmap v211: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-04-06T14:14:53.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-06T14:14:54.116 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:54 smithi032 ceph-mon[39730]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-06T14:14:55.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:54 smithi086 ceph-mon[47925]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-06T14:14:56.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.065+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/2945710935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 msgr2=0x7f1a180fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:56.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.065+0000 7f1a1e4aa700 1 --2- 172.21.15.32:0/2945710935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 0x7f1a180fc0e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1a08004940 tx=0x7f1a08009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:56.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.065+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/2945710935 shutdown_connections 2024-04-06T14:14:56.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.065+0000 7f1a1e4aa700 1 --2- 172.21.15.32:0/2945710935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a180fcf80 0x7f1a180fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:56.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.065+0000 7f1a1e4aa700 1 --2- 172.21.15.32:0/2945710935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 0x7f1a180fc0e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.065+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/2945710935 >> 172.21.15.32:0/2945710935 conn(0x7f1a180f7430 msgr2=0x7f1a180f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.065+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/2945710935 shutdown_connections 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/2945710935 wait complete. 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a1e4aa700 1 Processor -- start 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a1e4aa700 1 -- start start 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a1e4aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 0x7f1a1810da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a1e4aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a180fcf80 0x7f1a1810df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a1e4aa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1810e560 con 0x7f1a180fbcd0 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a1e4aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1810ed10 con 0x7f1a180fcf80 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 0x7f1a1810da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 0x7f1a1810da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58800/0 (socket says 172.21.15.32:58800) 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.066+0000 7f1a17fff700 1 -- 172.21.15.32:0/3485144636 learned_addr learned my addr 172.21.15.32:0/3485144636 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a177fe700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a180fcf80 0x7f1a1810df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a17fff700 1 -- 172.21.15.32:0/3485144636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a180fcf80 msgr2=0x7f1a1810df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a17fff700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a180fcf80 0x7f1a1810df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a17fff700 1 -- 172.21.15.32:0/3485144636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a08005040 con 0x7f1a180fbcd0 2024-04-06T14:14:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a17fff700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 0x7f1a1810da20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009380 tx=0x7f1a08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a14ff9700 1 -- 172.21.15.32:0/3485144636 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0800e040 con 0x7f1a180fbcd0 2024-04-06T14:14:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1810ef90 con 0x7f1a180fbcd0 2024-04-06T14:14:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a14ff9700 1 -- 172.21.15.32:0/3485144636 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a0800bd20 con 0x7f1a180fbcd0 2024-04-06T14:14:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a14ff9700 1 -- 172.21.15.32:0/3485144636 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0801c630 con 0x7f1a180fbcd0 2024-04-06T14:14:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.067+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1810f4d0 con 0x7f1a180fbcd0 2024-04-06T14:14:56.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.068+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a18101b10 con 0x7f1a180fbcd0 2024-04-06T14:14:56.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.069+0000 7f1a14ff9700 1 -- 172.21.15.32:0/3485144636 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1a08013420 con 0x7f1a180fbcd0 2024-04-06T14:14:56.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.069+0000 7f1a14ff9700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1a040692a0 0x7f1a0406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:56.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.069+0000 7f1a14ff9700 1 -- 172.21.15.32:0/3485144636 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1a08087f50 con 0x7f1a180fbcd0 2024-04-06T14:14:56.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.069+0000 7f1a177fe700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1a040692a0 0x7f1a0406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:56.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.070+0000 7f1a177fe700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1a040692a0 0x7f1a0406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a00007fd0 tx=0x7f1a00009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:56.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.074+0000 7f1a14ff9700 1 -- 172.21.15.32:0/3485144636 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1a080ba190 con 0x7f1a180fbcd0 2024-04-06T14:14:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.268+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f1a181047f0 con 0x7f1a180fbcd0 2024-04-06T14:14:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.268+0000 7f1a14ff9700 1 -- 172.21.15.32:0/3485144636 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f1a08057cc0 con 0x7f1a180fbcd0 2024-04-06T14:14:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1a040692a0 msgr2=0x7f1a0406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1a040692a0 0x7f1a0406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a00007fd0 tx=0x7f1a00009040 comp rx=0 tx=0).stop 2024-04-06T14:14:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 msgr2=0x7f1a1810da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 0x7f1a1810da20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009380 tx=0x7f1a08009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 shutdown_connections 2024-04-06T14:14:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1a040692a0 0x7f1a0406b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a180fbcd0 0x7f1a1810da20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 --2- 172.21.15.32:0/3485144636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a180fcf80 0x7f1a1810df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 >> 172.21.15.32:0/3485144636 conn(0x7f1a180f7430 msgr2=0x7f1a18065cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 shutdown_connections 2024-04-06T14:14:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:56.270+0000 7f1a1e4aa700 1 -- 172.21.15.32:0/3485144636 wait complete. 2024-04-06T14:14:56.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:56 smithi032 ceph-mon[39730]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-06T14:14:57.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:56 smithi086 ceph-mon[47925]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-06T14:14:57.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-06T14:14:57.419 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:14:57.951 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:14:58.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:57 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:14:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:14:58 smithi032 ceph-mon[39730]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-06T14:14:59.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:14:58 smithi086 ceph-mon[47925]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-06T14:14:59.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.291+0000 7f1c38b43700 1 -- 172.21.15.32:0/2105334949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 msgr2=0x7f1c340fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.291+0000 7f1c38b43700 1 --2- 172.21.15.32:0/2105334949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 0x7f1c340fd3f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1c24004940 tx=0x7f1c24009e30 comp rx=0 tx=0).stop 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.291+0000 7f1c38b43700 1 -- 172.21.15.32:0/2105334949 shutdown_connections 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.291+0000 7f1c38b43700 1 --2- 172.21.15.32:0/2105334949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 0x7f1c340fd3f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.291+0000 7f1c38b43700 1 --2- 172.21.15.32:0/2105334949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c340fbcd0 0x7f1c340fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.291+0000 7f1c38b43700 1 -- 172.21.15.32:0/2105334949 >> 172.21.15.32:0/2105334949 conn(0x7f1c340f7430 msgr2=0x7f1c340f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.291+0000 7f1c38b43700 1 -- 172.21.15.32:0/2105334949 shutdown_connections 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.291+0000 7f1c38b43700 1 -- 172.21.15.32:0/2105334949 wait complete. 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c38b43700 1 Processor -- start 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c38b43700 1 -- start start 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c38b43700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c340fbcd0 0x7f1c3410fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c38b43700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 0x7f1c34110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:59.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c38b43700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c34110680 con 0x7f1c340fcf80 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c38b43700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c34110ed0 con 0x7f1c340fbcd0 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c31d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 0x7f1c34110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c31d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 0x7f1c34110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58818/0 (socket says 172.21.15.32:58818) 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c31d9b700 1 -- 172.21.15.32:0/2745231007 learned_addr learned my addr 172.21.15.32:0/2745231007 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c3259c700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c340fbcd0 0x7f1c3410fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c31d9b700 1 -- 172.21.15.32:0/2745231007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c340fbcd0 msgr2=0x7f1c3410fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c31d9b700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c340fbcd0 0x7f1c3410fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.292+0000 7f1c31d9b700 1 -- 172.21.15.32:0/2745231007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c24005040 con 0x7f1c340fcf80 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.293+0000 7f1c31d9b700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 0x7f1c34110080 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400b3b0 tx=0x7f1c24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.293+0000 7f1c22ffd700 1 -- 172.21.15.32:0/2745231007 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c2400e040 con 0x7f1c340fcf80 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.293+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c34111150 con 0x7f1c340fcf80 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.293+0000 7f1c22ffd700 1 -- 172.21.15.32:0/2745231007 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c2400bd20 con 0x7f1c340fcf80 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.293+0000 7f1c22ffd700 1 -- 172.21.15.32:0/2745231007 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c2401c630 con 0x7f1c340fcf80 2024-04-06T14:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.293+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c34111690 con 0x7f1c340fcf80 2024-04-06T14:14:59.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.294+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3405f5f0 con 0x7f1c340fcf80 2024-04-06T14:14:59.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.294+0000 7f1c22ffd700 1 -- 172.21.15.32:0/2745231007 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1c24013420 con 0x7f1c340fcf80 2024-04-06T14:14:59.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.294+0000 7f1c22ffd700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1c1c069290 0x7f1c1c06b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:14:59.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.295+0000 7f1c22ffd700 1 -- 172.21.15.32:0/2745231007 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1c24087fe0 con 0x7f1c340fcf80 2024-04-06T14:14:59.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.295+0000 7f1c3259c700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1c1c069290 0x7f1c1c06b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:14:59.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.295+0000 7f1c3259c700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1c1c069290 0x7f1c1c06b740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1c28007fd0 tx=0x7f1c28009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:14:59.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.299+0000 7f1c22ffd700 1 -- 172.21.15.32:0/2745231007 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1c24057c90 con 0x7f1c340fcf80 2024-04-06T14:14:59.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:14:59.513+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c34111a20 con 0x7f1c1c069290 2024-04-06T14:15:01.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:00 smithi086 ceph-mon[47925]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:01.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:00 smithi086 ceph-mon[47925]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-04-06T14:15:01.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:00 smithi032 ceph-mon[39730]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:01.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:00 smithi032 ceph-mon[39730]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-04-06T14:15:03.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:02 smithi086 ceph-mon[47925]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-06T14:15:03.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:02 smithi032 ceph-mon[39730]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-06T14:15:05.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:04 smithi086 ceph-mon[47925]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T14:15:05.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:04 smithi032 ceph-mon[39730]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T14:15:07.073 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:06 smithi032 ceph-mon[39730]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T14:15:07.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:06 smithi086 ceph-mon[47925]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T14:15:09.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:08 smithi086 ceph-mon[47925]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T14:15:09.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:08 smithi032 ceph-mon[39730]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T14:15:11.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:10 smithi086 ceph-mon[47925]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:15:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:10 smithi032 ceph-mon[39730]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:15:13.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:12 smithi086 ceph-mon[47925]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T14:15:13.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:15:13.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:12 smithi032 ceph-mon[39730]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T14:15:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:15:15.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:14 smithi086 ceph-mon[47925]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:14 smithi032 ceph-mon[39730]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:17.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:16 smithi032 ceph-mon[39730]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:17.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:16 smithi086 ceph-mon[47925]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:19.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:18 smithi086 ceph-mon[47925]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:18 smithi032 ceph-mon[39730]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:21.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:20 smithi086 ceph-mon[47925]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:20 smithi032 ceph-mon[39730]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:23.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:22 smithi086 ceph-mon[47925]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:23.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:22 smithi032 ceph-mon[39730]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:25.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:24 smithi086 ceph-mon[47925]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:24 smithi032 ceph-mon[39730]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:27.073 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:26 smithi032 ceph-mon[39730]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:27.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:26 smithi086 ceph-mon[47925]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:28.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:27 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:15:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:27 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:15:29.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:28 smithi086 ceph-mon[47925]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:29.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:28 smithi032 ceph-mon[39730]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:31.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:30 smithi086 ceph-mon[47925]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:30 smithi032 ceph-mon[39730]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:33.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:32 smithi086 ceph-mon[47925]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:33.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:32 smithi032 ceph-mon[39730]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:34.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:33 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:15:34.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:33 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:15:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:34 smithi086 ceph-mon[47925]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:35.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:34 smithi032 ceph-mon[39730]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:36.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:35 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:15:36.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:35 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:15:36.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:35 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:36.176 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:35 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:15:36.176 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:35 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:15:36.176 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:35 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:36.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.659+0000 7f1c22ffd700 1 -- 172.21.15.32:0/2745231007 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f1c34111a20 con 0x7f1c1c069290 2024-04-06T14:15:36.660 INFO:teuthology.orchestra.run.smithi032.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.661+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1c1c069290 msgr2=0x7f1c1c06b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.661+0000 7f1c38b43700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1c1c069290 0x7f1c1c06b740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1c28007fd0 tx=0x7f1c28009040 comp rx=0 tx=0).stop 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.661+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 msgr2=0x7f1c34110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.661+0000 7f1c38b43700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 0x7f1c34110080 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400b3b0 tx=0x7f1c24009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.662+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 shutdown_connections 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.662+0000 7f1c38b43700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1c1c069290 0x7f1c1c06b740 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.662+0000 7f1c38b43700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c340fbcd0 0x7f1c3410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.662+0000 7f1c38b43700 1 --2- 172.21.15.32:0/2745231007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c340fcf80 0x7f1c34110080 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.662+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 >> 172.21.15.32:0/2745231007 conn(0x7f1c340f7430 msgr2=0x7f1c340f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.662+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 shutdown_connections 2024-04-06T14:15:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:36.662+0000 7f1c38b43700 1 -- 172.21.15.32:0/2745231007 wait complete. 2024-04-06T14:15:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:36 smithi032 ceph-mon[39730]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:15:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:15:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:15:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:36 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:37.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:36 smithi086 ceph-mon[47925]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:37.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:37.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:15:37.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:15:37.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:15:37.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:36 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:37.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-06T14:15:37.825 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:15:38.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:37 smithi086 ceph-mon[47925]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:15:38.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:37 smithi086 ceph-mon[47925]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:15:38.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:37 smithi032 ceph-mon[39730]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:15:38.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:37 smithi032 ceph-mon[39730]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:15:38.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:38 smithi032 ceph-mon[39730]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:39.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:38 smithi086 ceph-mon[47925]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:40 smithi032 ceph-mon[39730]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:15:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:40 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:40 smithi086 ceph-mon[47925]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:15:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:40 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:15:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.348+0000 7f0a57dd9700 1 -- 172.21.15.32:0/3308074050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 msgr2=0x7f0a500703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.348+0000 7f0a57dd9700 1 --2- 172.21.15.32:0/3308074050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 0x7f0a500703b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c004940 tx=0x7f0a4c009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.348+0000 7f0a57dd9700 1 -- 172.21.15.32:0/3308074050 shutdown_connections 2024-04-06T14:15:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.348+0000 7f0a57dd9700 1 --2- 172.21.15.32:0/3308074050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a500708f0 0x7f0a50070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.348+0000 7f0a57dd9700 1 --2- 172.21.15.32:0/3308074050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 0x7f0a500703b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.348+0000 7f0a57dd9700 1 -- 172.21.15.32:0/3308074050 >> 172.21.15.32:0/3308074050 conn(0x7f0a500f8580 msgr2=0x7f0a500fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.348+0000 7f0a57dd9700 1 -- 172.21.15.32:0/3308074050 shutdown_connections 2024-04-06T14:15:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.348+0000 7f0a57dd9700 1 -- 172.21.15.32:0/3308074050 wait complete. 2024-04-06T14:15:41.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a57dd9700 1 Processor -- start 2024-04-06T14:15:41.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a57dd9700 1 -- start start 2024-04-06T14:15:41.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a57dd9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a500708f0 0x7f0a5010fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a57dd9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 0x7f0a50110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a57dd9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a50110650 con 0x7f0a50071ea0 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a57dd9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a50110ea0 con 0x7f0a500708f0 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a55374700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 0x7f0a50110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a55374700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 0x7f0a50110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56212/0 (socket says 172.21.15.32:56212) 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a55b75700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a500708f0 0x7f0a5010fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.349+0000 7f0a55374700 1 -- 172.21.15.32:0/2327056051 learned_addr learned my addr 172.21.15.32:0/2327056051 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.350+0000 7f0a55374700 1 -- 172.21.15.32:0/2327056051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a500708f0 msgr2=0x7f0a5010fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.350+0000 7f0a55374700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a500708f0 0x7f0a5010fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.350+0000 7f0a55374700 1 -- 172.21.15.32:0/2327056051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c005040 con 0x7f0a50071ea0 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.350+0000 7f0a55374700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 0x7f0a50110050 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000eb40 tx=0x7f0a4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.350+0000 7f0a467fc700 1 -- 172.21.15.32:0/2327056051 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a40007cc0 con 0x7f0a50071ea0 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.350+0000 7f0a467fc700 1 -- 172.21.15.32:0/2327056051 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a4000f040 con 0x7f0a50071ea0 2024-04-06T14:15:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.350+0000 7f0a467fc700 1 -- 172.21.15.32:0/2327056051 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a4000c990 con 0x7f0a50071ea0 2024-04-06T14:15:41.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.350+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a50111120 con 0x7f0a50071ea0 2024-04-06T14:15:41.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.350+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a50111660 con 0x7f0a50071ea0 2024-04-06T14:15:41.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.352+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5005f5f0 con 0x7f0a50071ea0 2024-04-06T14:15:41.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.353+0000 7f0a467fc700 1 -- 172.21.15.32:0/2327056051 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0a4000caf0 con 0x7f0a50071ea0 2024-04-06T14:15:41.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.354+0000 7f0a467fc700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0a3c0692b0 0x7f0a3c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:41.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.354+0000 7f0a467fc700 1 -- 172.21.15.32:0/2327056051 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0a40088e10 con 0x7f0a50071ea0 2024-04-06T14:15:41.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.357+0000 7f0a55b75700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0a3c0692b0 0x7f0a3c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.358+0000 7f0a55b75700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0a3c0692b0 0x7f0a3c06b760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00b0d0 tx=0x7f0a4c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:41.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.362+0000 7f0a467fc700 1 -- 172.21.15.32:0/2327056051 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0a4008c060 con 0x7f0a50071ea0 2024-04-06T14:15:41.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.778+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a50111a50 con 0x7f0a3c0692b0 2024-04-06T14:15:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.780+0000 7f0a467fc700 1 -- 172.21.15.32:0/2327056051 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f0a50111a50 con 0x7f0a3c0692b0 2024-04-06T14:15:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0a3c0692b0 msgr2=0x7f0a3c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0a3c0692b0 0x7f0a3c06b760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00b0d0 tx=0x7f0a4c009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 msgr2=0x7f0a50110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 0x7f0a50110050 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000eb40 tx=0x7f0a4000b040 comp rx=0 tx=0).stop 2024-04-06T14:15:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 shutdown_connections 2024-04-06T14:15:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f0a3c0692b0 0x7f0a3c06b760 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a500708f0 0x7f0a5010fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 --2- 172.21.15.32:0/2327056051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a50071ea0 0x7f0a50110050 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 >> 172.21.15.32:0/2327056051 conn(0x7f0a500f8580 msgr2=0x7f0a500f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 shutdown_connections 2024-04-06T14:15:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.783+0000 7f0a57dd9700 1 -- 172.21.15.32:0/2327056051 wait complete. 2024-04-06T14:15:41.798 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:15:41.847 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:41 smithi086 ceph-mon[47925]: Upgrade: Target is version 18.2.2-1086-g052adcad (unknown) 2024-04-06T14:15:41.847 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:41 smithi086 ceph-mon[47925]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e'] 2024-04-06T14:15:41.847 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:41 smithi086 ceph-mon[47925]: Upgrade: Need to upgrade myself (mgr.smithi032.peohxu) 2024-04-06T14:15:41.847 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:41 smithi086 ceph-mon[47925]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e on smithi086 2024-04-06T14:15:41.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.949+0000 7f6321b07700 1 -- 172.21.15.32:0/3664184465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 msgr2=0x7f631c0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.949+0000 7f6321b07700 1 --2- 172.21.15.32:0/3664184465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 0x7f631c0fc0e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f630c004940 tx=0x7f630c009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.949+0000 7f6321b07700 1 -- 172.21.15.32:0/3664184465 shutdown_connections 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.949+0000 7f6321b07700 1 --2- 172.21.15.32:0/3664184465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f631c0fcf80 0x7f631c0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.949+0000 7f6321b07700 1 --2- 172.21.15.32:0/3664184465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 0x7f631c0fc0e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.949+0000 7f6321b07700 1 -- 172.21.15.32:0/3664184465 >> 172.21.15.32:0/3664184465 conn(0x7f631c0f7430 msgr2=0x7f631c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.949+0000 7f6321b07700 1 -- 172.21.15.32:0/3664184465 shutdown_connections 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.949+0000 7f6321b07700 1 -- 172.21.15.32:0/3664184465 wait complete. 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f6321b07700 1 Processor -- start 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f6321b07700 1 -- start start 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f6321b07700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 0x7f631c10b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f6321b07700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f631c0fcf80 0x7f631c10bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f6321b07700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631c10c1f0 con 0x7f631c0fbcd0 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f6321b07700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631c111ca0 con 0x7f631c0fcf80 2024-04-06T14:15:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f631b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 0x7f631c10b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:41.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f631b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 0x7f631c10b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56234/0 (socket says 172.21.15.32:56234) 2024-04-06T14:15:41.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f631b7fe700 1 -- 172.21.15.32:0/362227445 learned_addr learned my addr 172.21.15.32:0/362227445 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:15:41.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.950+0000 7f631affd700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f631c0fcf80 0x7f631c10bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:41.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f631affd700 1 -- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 msgr2=0x7f631c10b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:41.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f631affd700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 0x7f631c10b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:41.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f631affd700 1 -- 172.21.15.32:0/362227445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c005040 con 0x7f631c0fcf80 2024-04-06T14:15:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f631b7fe700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 0x7f631c10b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:15:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f631affd700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f631c0fcf80 0x7f631c10bba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6310009fd0 tx=0x7f631000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f6303fff700 1 -- 172.21.15.32:0/362227445 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6310004e90 con 0x7f631c0fcf80 2024-04-06T14:15:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631c111ea0 con 0x7f631c0fcf80 2024-04-06T14:15:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f6303fff700 1 -- 172.21.15.32:0/362227445 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f631000f040 con 0x7f631c0fcf80 2024-04-06T14:15:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f6303fff700 1 -- 172.21.15.32:0/362227445 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f631000c9b0 con 0x7f631c0fcf80 2024-04-06T14:15:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.951+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631c112430 con 0x7f631c0fcf80 2024-04-06T14:15:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.953+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631c05f5f0 con 0x7f631c0fcf80 2024-04-06T14:15:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.954+0000 7f6303fff700 1 -- 172.21.15.32:0/362227445 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6310007cc0 con 0x7f631c0fcf80 2024-04-06T14:15:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.954+0000 7f6303fff700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f63040692a0 0x7f630406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.954+0000 7f6303fff700 1 -- 172.21.15.32:0/362227445 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6310088c70 con 0x7f631c0fcf80 2024-04-06T14:15:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.954+0000 7f631b7fe700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f63040692a0 0x7f630406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.955+0000 7f631b7fe700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f63040692a0 0x7f630406b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f630c00b0d0 tx=0x7f630c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:41.963+0000 7f6303fff700 1 -- 172.21.15.32:0/362227445 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f631008c060 con 0x7f631c0fcf80 2024-04-06T14:15:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:41 smithi032 ceph-mon[39730]: Upgrade: Target is version 18.2.2-1086-g052adcad (unknown) 2024-04-06T14:15:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:41 smithi032 ceph-mon[39730]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e'] 2024-04-06T14:15:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:41 smithi032 ceph-mon[39730]: Upgrade: Need to upgrade myself (mgr.smithi032.peohxu) 2024-04-06T14:15:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:41 smithi032 ceph-mon[39730]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e on smithi086 2024-04-06T14:15:42.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.235+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f631c1014b0 con 0x7f63040692a0 2024-04-06T14:15:42.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.236+0000 7f6303fff700 1 -- 172.21.15.32:0/362227445 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f631c1014b0 con 0x7f63040692a0 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f63040692a0 msgr2=0x7f630406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f63040692a0 0x7f630406b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f630c00b0d0 tx=0x7f630c009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f631c0fcf80 msgr2=0x7f631c10bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f631c0fcf80 0x7f631c10bba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6310009fd0 tx=0x7f631000b040 comp rx=0 tx=0).stop 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 shutdown_connections 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f63040692a0 0x7f630406b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631c0fbcd0 0x7f631c10b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 --2- 172.21.15.32:0/362227445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f631c0fcf80 0x7f631c10bba0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 >> 172.21.15.32:0/362227445 conn(0x7f631c0f7430 msgr2=0x7f631c065b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 shutdown_connections 2024-04-06T14:15:42.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.238+0000 7f6321b07700 1 -- 172.21.15.32:0/362227445 wait complete. 2024-04-06T14:15:42.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.347+0000 7f28cd831700 1 -- 172.21.15.32:0/2678939977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 msgr2=0x7f28c80ff9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:42.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.347+0000 7f28cd831700 1 --2- 172.21.15.32:0/2678939977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 0x7f28c80ff9d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f28b8004940 tx=0x7f28b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:42.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.347+0000 7f28cd831700 1 -- 172.21.15.32:0/2678939977 shutdown_connections 2024-04-06T14:15:42.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.347+0000 7f28cd831700 1 --2- 172.21.15.32:0/2678939977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28c80fff10 0x7f28c8102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.347+0000 7f28cd831700 1 --2- 172.21.15.32:0/2678939977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 0x7f28c80ff9d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.347+0000 7f28cd831700 1 -- 172.21.15.32:0/2678939977 >> 172.21.15.32:0/2678939977 conn(0x7f28c80f7460 msgr2=0x7f28c80f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:42.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.347+0000 7f28cd831700 1 -- 172.21.15.32:0/2678939977 shutdown_connections 2024-04-06T14:15:42.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.347+0000 7f28cd831700 1 -- 172.21.15.32:0/2678939977 wait complete. 2024-04-06T14:15:42.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28cd831700 1 Processor -- start 2024-04-06T14:15:42.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28cd831700 1 -- start start 2024-04-06T14:15:42.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28cd831700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 0x7f28c80fd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:42.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28cd831700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28c80fff10 0x7f28c80fb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:42.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28cd831700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80fd820 con 0x7f28c8066f50 2024-04-06T14:15:42.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28cd831700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80fbe10 con 0x7f28c80fff10 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 0x7f28c80fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 0x7f28c80fd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56242/0 (socket says 172.21.15.32:56242) 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28c6ffd700 1 -- 172.21.15.32:0/2422705312 learned_addr learned my addr 172.21.15.32:0/2422705312 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.348+0000 7f28c67fc700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28c80fff10 0x7f28c80fb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28c6ffd700 1 -- 172.21.15.32:0/2422705312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28c80fff10 msgr2=0x7f28c80fb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28c6ffd700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28c80fff10 0x7f28c80fb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28c6ffd700 1 -- 172.21.15.32:0/2422705312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b8005040 con 0x7f28c8066f50 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28c67fc700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28c80fff10 0x7f28c80fb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28c6ffd700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 0x7f28c80fd220 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f28b80137b0 tx=0x7f28b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28af7fe700 1 -- 172.21.15.32:0/2422705312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28b8021070 con 0x7f28c8066f50 2024-04-06T14:15:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28af7fe700 1 -- 172.21.15.32:0/2422705312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28b801c400 con 0x7f28c8066f50 2024-04-06T14:15:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28af7fe700 1 -- 172.21.15.32:0/2422705312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28b800e070 con 0x7f28c8066f50 2024-04-06T14:15:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c80fc060 con 0x7f28c8066f50 2024-04-06T14:15:42.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.349+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c80fc620 con 0x7f28c8066f50 2024-04-06T14:15:42.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.351+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c804c710 con 0x7f28c8066f50 2024-04-06T14:15:42.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.352+0000 7f28af7fe700 1 -- 172.21.15.32:0/2422705312 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f28b801c9c0 con 0x7f28c8066f50 2024-04-06T14:15:42.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.352+0000 7f28af7fe700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f28b0069360 0x7f28b006b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:42.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.352+0000 7f28af7fe700 1 -- 172.21.15.32:0/2422705312 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f28b8088540 con 0x7f28c8066f50 2024-04-06T14:15:42.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.354+0000 7f28c67fc700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f28b0069360 0x7f28b006b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:42.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.354+0000 7f28c67fc700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f28b0069360 0x7f28b006b810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009b80 tx=0x7f28bc00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:42.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.360+0000 7f28af7fe700 1 -- 172.21.15.32:0/2422705312 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f28b8058400 con 0x7f28c8066f50 2024-04-06T14:15:42.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.609+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f28c80f8f50 con 0x7f28b0069360 2024-04-06T14:15:42.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.627+0000 7f28af7fe700 1 -- 172.21.15.32:0/2422705312 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f28c80f8f50 con 0x7f28b0069360 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (5m) 69s ago 7m 26.8M - 0.25.0 c8568f914cd2 d6f57d3d10f9 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 69s ago 7m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (6m) 76s ago 6m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (5m) 69s ago 7m 86.3M - 9.4.7 954c08fa6188 1cc1d256b791 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (98s) 69s ago 97s 10.4M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (90s) 69s ago 90s 15.8M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (95s) 76s ago 94s 16.7M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (88s) 76s ago 87s 13.4M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:9283 running (8m) 69s ago 8m 480M - 17.2.7-819-g6699495c ec300eaa768b 7737c4037eba 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283 running (6m) 76s ago 6m 432M - 17.2.7-819-g6699495c ec300eaa768b b0b7b2c2f6da 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 69s ago 8m 43.4M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (6m) 76s ago 6m 33.5M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:15:42.628 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (7m) 69s ago 7m 16.4M - 1.5.0 0da6a335fe13 41a44c41cd57 2024-04-06T14:15:42.629 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (6m) 76s ago 6m 18.9M - 1.5.0 0da6a335fe13 21ac74498650 2024-04-06T14:15:42.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (5m) 69s ago 5m 44.6M 4300M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:15:42.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 69s ago 5m 46.3M 4300M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:15:42.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (4m) 69s ago 4m 47.9M 4300M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:15:42.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (4m) 76s ago 4m 46.8M 5323M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:15:42.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (3m) 76s ago 3m 44.4M 5323M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:15:42.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (3m) 76s ago 3m 42.3M 5323M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:15:42.629 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (5m) 69s ago 6m 53.1M - 2.43.0 a07b618ecd1d 6742cb6c38a9 2024-04-06T14:15:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.629+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f28b0069360 msgr2=0x7f28b006b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.629+0000 7f28cd831700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f28b0069360 0x7f28b006b810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009b80 tx=0x7f28bc00b040 comp rx=0 tx=0).stop 2024-04-06T14:15:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.629+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 msgr2=0x7f28c80fd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.629+0000 7f28cd831700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 0x7f28c80fd220 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f28b80137b0 tx=0x7f28b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.630+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 shutdown_connections 2024-04-06T14:15:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.630+0000 7f28cd831700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f28b0069360 0x7f28b006b810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.630+0000 7f28cd831700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28c8066f50 0x7f28c80fd220 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.630+0000 7f28cd831700 1 --2- 172.21.15.32:0/2422705312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28c80fff10 0x7f28c80fb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.630+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 >> 172.21.15.32:0/2422705312 conn(0x7f28c80f7460 msgr2=0x7f28c80f7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.630+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 shutdown_connections 2024-04-06T14:15:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.630+0000 7f28cd831700 1 -- 172.21.15.32:0/2422705312 wait complete. 2024-04-06T14:15:42.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.749+0000 7fc620c55700 1 -- 172.21.15.32:0/4030319069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 msgr2=0x7fc61c104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:42.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.749+0000 7fc620c55700 1 --2- 172.21.15.32:0/4030319069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 0x7fc61c104740 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc610004940 tx=0x7fc610009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.749+0000 7fc620c55700 1 -- 172.21.15.32:0/4030319069 shutdown_connections 2024-04-06T14:15:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.749+0000 7fc620c55700 1 --2- 172.21.15.32:0/4030319069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 0x7fc61c104740 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.749+0000 7fc620c55700 1 --2- 172.21.15.32:0/4030319069 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc61c066e90 0x7fc61c101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.749+0000 7fc620c55700 1 -- 172.21.15.32:0/4030319069 >> 172.21.15.32:0/4030319069 conn(0x7fc61c0f7430 msgr2=0x7fc61c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.749+0000 7fc620c55700 1 -- 172.21.15.32:0/4030319069 shutdown_connections 2024-04-06T14:15:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.749+0000 7fc620c55700 1 -- 172.21.15.32:0/4030319069 wait complete. 2024-04-06T14:15:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.750+0000 7fc620c55700 1 Processor -- start 2024-04-06T14:15:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.750+0000 7fc620c55700 1 -- start start 2024-04-06T14:15:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.750+0000 7fc620c55700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc61c066e90 0x7fc61c10f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:42.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.750+0000 7fc620c55700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 0x7fc61c10ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:42.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.750+0000 7fc620c55700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c110520 con 0x7fc61c102360 2024-04-06T14:15:42.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.750+0000 7fc620c55700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c110690 con 0x7fc61c066e90 2024-04-06T14:15:42.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc619d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 0x7fc61c10ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:42.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc619d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 0x7fc61c10ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56258/0 (socket says 172.21.15.32:56258) 2024-04-06T14:15:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc619d9b700 1 -- 172.21.15.32:0/2939631982 learned_addr learned my addr 172.21.15.32:0/2939631982 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:15:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc619d9b700 1 -- 172.21.15.32:0/2939631982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc61c066e90 msgr2=0x7fc61c10f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc619d9b700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc61c066e90 0x7fc61c10f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc619d9b700 1 -- 172.21.15.32:0/2939631982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc610005040 con 0x7fc61c102360 2024-04-06T14:15:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc619d9b700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 0x7fc61c10ff20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc6100137b0 tx=0x7fc610009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:42.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc60affd700 1 -- 172.21.15.32:0/2939631982 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc61000e070 con 0x7fc61c102360 2024-04-06T14:15:42.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc60affd700 1 -- 172.21.15.32:0/2939631982 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6100093f0 con 0x7fc61c102360 2024-04-06T14:15:42.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc620c55700 1 -- 172.21.15.32:0/2939631982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61c115090 con 0x7fc61c102360 2024-04-06T14:15:42.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc60affd700 1 -- 172.21.15.32:0/2939631982 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc61000b410 con 0x7fc61c102360 2024-04-06T14:15:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.751+0000 7fc620c55700 1 -- 172.21.15.32:0/2939631982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c1155f0 con 0x7fc61c102360 2024-04-06T14:15:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.753+0000 7fc608ff9700 1 -- 172.21.15.32:0/2939631982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6000052b0 con 0x7fc61c102360 2024-04-06T14:15:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.753+0000 7fc60affd700 1 -- 172.21.15.32:0/2939631982 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc610013c40 con 0x7fc61c102360 2024-04-06T14:15:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.754+0000 7fc60affd700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc6040692a0 0x7fc60406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:42.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.754+0000 7fc60affd700 1 -- 172.21.15.32:0/2939631982 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc610088880 con 0x7fc61c102360 2024-04-06T14:15:42.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.757+0000 7fc61a59c700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc6040692a0 0x7fc60406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:42.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.757+0000 7fc61a59c700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc6040692a0 0x7fc60406b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc61c066bd0 tx=0x7fc60c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:42.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:42.759+0000 7fc60affd700 1 -- 172.21.15.32:0/2939631982 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc6100585d0 con 0x7fc61c102360 2024-04-06T14:15:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:42 smithi086 ceph-mon[47925]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:42 smithi086 ceph-mon[47925]: from='client.14540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:43.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:42 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:15:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.178+0000 7fc608ff9700 1 -- 172.21.15.32:0/2939631982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc600005d70 con 0x7fc61c102360 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.178+0000 7fc60affd700 1 -- 172.21.15.32:0/2939631982 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fc610004490 con 0x7fc61c102360 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:15:43.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:15:43.180 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:15:43.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:15:43.180 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:15:43.180 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 14 2024-04-06T14:15:43.180 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:15:43.180 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:15:43.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.181+0000 7fc608ff9700 1 -- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc6040692a0 msgr2=0x7fc60406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:43.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.181+0000 7fc608ff9700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc6040692a0 0x7fc60406b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc61c066bd0 tx=0x7fc60c00b040 comp rx=0 tx=0).stop 2024-04-06T14:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.181+0000 7fc608ff9700 1 -- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 msgr2=0x7fc61c10ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.181+0000 7fc608ff9700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 0x7fc61c10ff20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc6100137b0 tx=0x7fc610009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.181+0000 7fc608ff9700 1 -- 172.21.15.32:0/2939631982 shutdown_connections 2024-04-06T14:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.181+0000 7fc608ff9700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc6040692a0 0x7fc60406b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.181+0000 7fc608ff9700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc61c066e90 0x7fc61c10f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.181+0000 7fc608ff9700 1 --2- 172.21.15.32:0/2939631982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c102360 0x7fc61c10ff20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.181+0000 7fc608ff9700 1 -- 172.21.15.32:0/2939631982 >> 172.21.15.32:0/2939631982 conn(0x7fc61c0f7430 msgr2=0x7fc61c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.182+0000 7fc608ff9700 1 -- 172.21.15.32:0/2939631982 shutdown_connections 2024-04-06T14:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.182+0000 7fc608ff9700 1 -- 172.21.15.32:0/2939631982 wait complete. 2024-04-06T14:15:43.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:42 smithi032 ceph-mon[39730]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:43.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:42 smithi032 ceph-mon[39730]: from='client.14540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:43.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:42 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:15:43.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.349+0000 7fc133a52700 1 -- 172.21.15.32:0/1957311632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc12c0fdc10 msgr2=0x7fc12c0fc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:43.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.349+0000 7fc133a52700 1 --2- 172.21.15.32:0/1957311632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc12c0fdc10 0x7fc12c0fc020 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc11c004940 tx=0x7fc11c009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:43.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.349+0000 7fc133a52700 1 -- 172.21.15.32:0/1957311632 shutdown_connections 2024-04-06T14:15:43.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.349+0000 7fc133a52700 1 --2- 172.21.15.32:0/1957311632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc12c0fdc10 0x7fc12c0fc020 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:43.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.349+0000 7fc133a52700 1 --2- 172.21.15.32:0/1957311632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc12c0fd2c0 0x7fc12c0fd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.349+0000 7fc133a52700 1 -- 172.21.15.32:0/1957311632 >> 172.21.15.32:0/1957311632 conn(0x7fc12c0f6ba0 msgr2=0x7fc12c0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.349+0000 7fc133a52700 1 -- 172.21.15.32:0/1957311632 shutdown_connections 2024-04-06T14:15:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.349+0000 7fc133a52700 1 -- 172.21.15.32:0/1957311632 wait complete. 2024-04-06T14:15:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc133a52700 1 Processor -- start 2024-04-06T14:15:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc133a52700 1 -- start start 2024-04-06T14:15:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc133a52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc12c0fd2c0 0x7fc12c10d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc133a52700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc12c0fdc10 0x7fc12c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc133a52700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c10e340 con 0x7fc12c0fd2c0 2024-04-06T14:15:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc133a52700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c10e4b0 con 0x7fc12c0fdc10 2024-04-06T14:15:43.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc1317ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc12c0fd2c0 0x7fc12c10d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:43.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc1317ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc12c0fd2c0 0x7fc12c10d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56280/0 (socket says 172.21.15.32:56280) 2024-04-06T14:15:43.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc1317ee700 1 -- 172.21.15.32:0/2194272529 learned_addr learned my addr 172.21.15.32:0/2194272529 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:15:43.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc130fed700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc12c0fdc10 0x7fc12c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:43.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc130fed700 1 -- 172.21.15.32:0/2194272529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc12c0fd2c0 msgr2=0x7fc12c10d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:43.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc130fed700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc12c0fd2c0 0x7fc12c10d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:43.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.350+0000 7fc130fed700 1 -- 172.21.15.32:0/2194272529 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c005040 con 0x7fc12c0fdc10 2024-04-06T14:15:43.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.351+0000 7fc130fed700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc12c0fdc10 0x7fc12c10de00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc11c000c00 tx=0x7fc11c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:43.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.351+0000 7fc121ffb700 1 -- 172.21.15.32:0/2194272529 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc11c021070 con 0x7fc12c0fdc10 2024-04-06T14:15:43.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.351+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c10ed50 con 0x7fc12c0fdc10 2024-04-06T14:15:43.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.351+0000 7fc121ffb700 1 -- 172.21.15.32:0/2194272529 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc11c01c400 con 0x7fc12c0fdc10 2024-04-06T14:15:43.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.351+0000 7fc121ffb700 1 -- 172.21.15.32:0/2194272529 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc11c00e070 con 0x7fc12c0fdc10 2024-04-06T14:15:43.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.351+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c10f2e0 con 0x7fc12c0fdc10 2024-04-06T14:15:43.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.352+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c05f5f0 con 0x7fc12c0fdc10 2024-04-06T14:15:43.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.354+0000 7fc121ffb700 1 -- 172.21.15.32:0/2194272529 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc11c01c9c0 con 0x7fc12c0fdc10 2024-04-06T14:15:43.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.355+0000 7fc121ffb700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc1180692a0 0x7fc11806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:15:43.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.355+0000 7fc121ffb700 1 -- 172.21.15.32:0/2194272529 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc11c088c40 con 0x7fc12c0fdc10 2024-04-06T14:15:43.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.355+0000 7fc1317ee700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc1180692a0 0x7fc11806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:15:43.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.355+0000 7fc1317ee700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc1180692a0 0x7fc11806b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc12c10d690 tx=0x7fc12800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:15:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.362+0000 7fc121ffb700 1 -- 172.21.15.32:0/2194272529 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc11c054fc0 con 0x7fc12c0fdc10 2024-04-06T14:15:43.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.787+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc12c10f5c0 con 0x7fc1180692a0 2024-04-06T14:15:43.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.788+0000 7fc121ffb700 1 -- 172.21.15.32:0/2194272529 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fc12c10f5c0 con 0x7fc1180692a0 2024-04-06T14:15:43.789 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:15:43.789 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:15:43.789 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:15:43.789 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T14:15:43.789 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-06T14:15:43.789 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "0/2 daemons upgraded", 2024-04-06T14:15:43.789 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e image on host smithi086", 2024-04-06T14:15:43.789 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:15:43.789 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:15:43.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc1180692a0 msgr2=0x7fc11806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:43.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc1180692a0 0x7fc11806b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc12c10d690 tx=0x7fc12800b040 comp rx=0 tx=0).stop 2024-04-06T14:15:43.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc12c0fdc10 msgr2=0x7fc12c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:15:43.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc12c0fdc10 0x7fc12c10de00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc11c000c00 tx=0x7fc11c009e30 comp rx=0 tx=0).stop 2024-04-06T14:15:43.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 shutdown_connections 2024-04-06T14:15:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fc1180692a0 0x7fc11806b750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc12c0fd2c0 0x7fc12c10d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 --2- 172.21.15.32:0/2194272529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc12c0fdc10 0x7fc12c10de00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:15:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 >> 172.21.15.32:0/2194272529 conn(0x7fc12c0f6ba0 msgr2=0x7fc12c0f8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:15:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 shutdown_connections 2024-04-06T14:15:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:15:43.791+0000 7fc133a52700 1 -- 172.21.15.32:0/2194272529 wait complete. 2024-04-06T14:15:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:43 smithi086 ceph-mon[47925]: from='client.24375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:43 smithi086 ceph-mon[47925]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:44.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:43 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2939631982' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:15:44.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:43 smithi032 ceph-mon[39730]: from='client.24375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:44.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:43 smithi032 ceph-mon[39730]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:44.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:43 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2939631982' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:15:45.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:44 smithi086 ceph-mon[47925]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:45.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:44 smithi086 ceph-mon[47925]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:44 smithi032 ceph-mon[39730]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:44 smithi032 ceph-mon[39730]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:15:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:46 smithi086 ceph-mon[47925]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:46 smithi032 ceph-mon[39730]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:49.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:48 smithi086 ceph-mon[47925]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:48 smithi032 ceph-mon[39730]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:51.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:50 smithi086 ceph-mon[47925]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:50 smithi032 ceph-mon[39730]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:53.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:52 smithi086 ceph-mon[47925]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:53.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:52 smithi032 ceph-mon[39730]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:55.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:54 smithi086 ceph-mon[47925]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:55.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:54 smithi032 ceph-mon[39730]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:56.007 INFO:tasks.workunit.client.1.smithi086.stderr:Updating files: 96% (12030/12461) Updating files: 97% (12088/12461) Updating files: 98% (12212/12461) Updating files: 99% (12337/12461) Updating files: 100% (12461/12461) Updating files: 100% (12461/12461), done. 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr:Note: switching to '800ec6d64a7fa0648ad53edfc6e50b41fc4cd674'. 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr:state without impacting any branches by switching back to a branch. 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr: git switch -c 2024-04-06T14:15:56.411 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-04-06T14:15:56.412 INFO:tasks.workunit.client.1.smithi086.stderr:Or undo this operation with: 2024-04-06T14:15:56.412 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-04-06T14:15:56.412 INFO:tasks.workunit.client.1.smithi086.stderr: git switch - 2024-04-06T14:15:56.412 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-04-06T14:15:56.412 INFO:tasks.workunit.client.1.smithi086.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-06T14:15:56.412 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-04-06T14:15:56.412 INFO:tasks.workunit.client.1.smithi086.stderr:HEAD is now at 800ec6d64a7 Merge pull request #56347 from cbodley/wip-65003-reef 2024-04-06T14:15:56.417 DEBUG:teuthology.orchestra.run.smithi086:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-06T14:15:56.485 INFO:tasks.workunit.client.1.smithi086.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-06T14:15:56.488 INFO:tasks.workunit.client.1.smithi086.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-06T14:15:56.488 INFO:tasks.workunit.client.1.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-06T14:15:57.030 INFO:tasks.workunit.client.1.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-06T14:15:57.096 INFO:tasks.workunit.client.1.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-06T14:15:57.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:56 smithi086 ceph-mon[47925]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:57.134 INFO:tasks.workunit.client.1.smithi086.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-06T14:15:57.136 INFO:tasks.workunit.client.1.smithi086.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-06T14:15:57.136 INFO:tasks.workunit.client.1.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-06T14:15:57.173 INFO:tasks.workunit.client.1.smithi086.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-06T14:15:57.177 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-06T14:15:57.177 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-06T14:15:57.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:56 smithi032 ceph-mon[39730]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:57.234 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-06T14:15:57.236 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-06T14:15:57.236 DEBUG:teuthology.orchestra.run.smithi086:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=800ec6d64a7fa0648ad53edfc6e50b41fc4cd674 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-06T14:15:57.303 INFO:tasks.workunit.client.1.smithi086.stderr:+ mkdir -p fsstress 2024-04-06T14:15:57.305 INFO:tasks.workunit.client.1.smithi086.stderr:+ pushd fsstress 2024-04-06T14:15:57.306 INFO:tasks.workunit.client.1.smithi086.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-06T14:15:57.306 INFO:tasks.workunit.client.1.smithi086.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr:Note: switching to '800ec6d64a7fa0648ad53edfc6e50b41fc4cd674'. 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr:state without impacting any branches by switching back to a branch. 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr: git switch -c 2024-04-06T14:15:57.538 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-06T14:15:57.539 INFO:tasks.workunit.client.0.smithi032.stderr:Or undo this operation with: 2024-04-06T14:15:57.539 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-06T14:15:57.539 INFO:tasks.workunit.client.0.smithi032.stderr: git switch - 2024-04-06T14:15:57.539 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-06T14:15:57.539 INFO:tasks.workunit.client.0.smithi032.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-06T14:15:57.539 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-06T14:15:57.539 INFO:tasks.workunit.client.0.smithi032.stderr:HEAD is now at 800ec6d64a7 Merge pull request #56347 from cbodley/wip-65003-reef 2024-04-06T14:15:57.544 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-06T14:15:57.581 INFO:tasks.workunit.client.0.smithi032.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-06T14:15:57.583 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-06T14:15:57.583 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-06T14:15:57.839 INFO:tasks.workunit.client.1.smithi086.stderr:+ tar xzf ltp-full.tgz 2024-04-06T14:15:58.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:57 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:15:58.143 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-06T14:15:58.206 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-06T14:15:58.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:57 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:15:58.246 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-06T14:15:58.248 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-06T14:15:58.248 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-06T14:15:58.287 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-06T14:15:58.291 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:15:58.291 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-06T14:15:58.350 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-06T14:15:58.351 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-06T14:15:58.351 DEBUG:teuthology.orchestra.run.smithi032:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=800ec6d64a7fa0648ad53edfc6e50b41fc4cd674 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-06T14:15:58.420 INFO:tasks.workunit.client.0.smithi032.stderr:+ mkdir -p fsstress 2024-04-06T14:15:58.423 INFO:tasks.workunit.client.0.smithi032.stderr:+ pushd fsstress 2024-04-06T14:15:58.424 INFO:tasks.workunit.client.0.smithi032.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-06T14:15:58.425 INFO:tasks.workunit.client.0.smithi032.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-06T14:15:59.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:15:58 smithi086 ceph-mon[47925]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:59.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:15:58 smithi032 ceph-mon[39730]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:15:59.396 INFO:tasks.workunit.client.0.smithi032.stderr:+ tar xzf ltp-full.tgz 2024-04-06T14:16:01.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:00 smithi086 ceph-mon[47925]: pgmap v245: 65 pgs: 65 active+clean; 854 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 23 KiB/s wr, 3 op/s 2024-04-06T14:16:01.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:00 smithi032 ceph-mon[39730]: pgmap v245: 65 pgs: 65 active+clean; 854 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 23 KiB/s wr, 3 op/s 2024-04-06T14:16:03.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:02 smithi032 ceph-mon[39730]: pgmap v246: 65 pgs: 65 active+clean; 13 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.1 MiB/s wr, 31 op/s 2024-04-06T14:16:03.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:02 smithi086 ceph-mon[47925]: pgmap v246: 65 pgs: 65 active+clean; 13 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.1 MiB/s wr, 31 op/s 2024-04-06T14:16:05.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:04 smithi032 ceph-mon[39730]: pgmap v247: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.4 MiB/s wr, 67 op/s 2024-04-06T14:16:05.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:04 smithi086 ceph-mon[47925]: pgmap v247: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.4 MiB/s wr, 67 op/s 2024-04-06T14:16:07.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:06 smithi032 ceph-mon[39730]: pgmap v248: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.5 MiB/s wr, 77 op/s 2024-04-06T14:16:07.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:06 smithi086 ceph-mon[47925]: pgmap v248: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.5 MiB/s wr, 77 op/s 2024-04-06T14:16:09.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:08 smithi086 ceph-mon[47925]: pgmap v249: 65 pgs: 65 active+clean; 36 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3.0 MiB/s wr, 110 op/s 2024-04-06T14:16:09.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:08 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:16:09.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:09.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:16:09.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:16:09.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:08 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:16:09.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:08 smithi032 ceph-mon[39730]: pgmap v249: 65 pgs: 65 active+clean; 36 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3.0 MiB/s wr, 110 op/s 2024-04-06T14:16:09.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:08 smithi032 ceph-mon[39730]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:16:09.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:09.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:16:09.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:16:09.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:08 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:16:10.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:09 smithi086 ceph-mon[47925]: Upgrade: Updating mgr.smithi086.fyyhxm 2024-04-06T14:16:10.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:09 smithi086 ceph-mon[47925]: Deploying daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:16:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:09 smithi032 ceph-mon[39730]: Upgrade: Updating mgr.smithi086.fyyhxm 2024-04-06T14:16:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:09 smithi032 ceph-mon[39730]: Deploying daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:16:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:10 smithi032 ceph-mon[39730]: pgmap v250: 65 pgs: 65 active+clean; 45 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3.8 MiB/s wr, 141 op/s 2024-04-06T14:16:11.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:10 smithi086 ceph-mon[47925]: pgmap v250: 65 pgs: 65 active+clean; 45 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3.8 MiB/s wr, 141 op/s 2024-04-06T14:16:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:12 smithi032 ceph-mon[39730]: pgmap v251: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 5.0 MiB/s wr, 169 op/s 2024-04-06T14:16:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:12 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:16:13.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:12 smithi086 ceph-mon[47925]: pgmap v251: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 5.0 MiB/s wr, 169 op/s 2024-04-06T14:16:13.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:12 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.925+0000 7f1f21c8b700 1 -- 172.21.15.32:0/53276798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fcf80 msgr2=0x7f1f1c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.925+0000 7f1f21c8b700 1 --2- 172.21.15.32:0/53276798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fcf80 0x7f1f1c0fd3f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c004940 tx=0x7f1f0c009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.925+0000 7f1f21c8b700 1 -- 172.21.15.32:0/53276798 shutdown_connections 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.925+0000 7f1f21c8b700 1 --2- 172.21.15.32:0/53276798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fcf80 0x7f1f1c0fd3f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.925+0000 7f1f21c8b700 1 --2- 172.21.15.32:0/53276798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f1c0fbcd0 0x7f1f1c0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.925+0000 7f1f21c8b700 1 -- 172.21.15.32:0/53276798 >> 172.21.15.32:0/53276798 conn(0x7f1f1c0f7430 msgr2=0x7f1f1c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.925+0000 7f1f21c8b700 1 -- 172.21.15.32:0/53276798 shutdown_connections 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.926+0000 7f1f21c8b700 1 -- 172.21.15.32:0/53276798 wait complete. 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.926+0000 7f1f21c8b700 1 Processor -- start 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.926+0000 7f1f21c8b700 1 -- start start 2024-04-06T14:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.926+0000 7f1f21c8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fbcd0 0x7f1f1c065740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f21c8b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f1c0fcf80 0x7f1f1c065c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f21c8b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c06ff60 con 0x7f1f1c0fbcd0 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f21c8b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0700a0 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f1b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fbcd0 0x7f1f1c065740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f1b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fbcd0 0x7f1f1c065740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40458/0 (socket says 172.21.15.32:40458) 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f1b7fe700 1 -- 172.21.15.32:0/2926385378 learned_addr learned my addr 172.21.15.32:0/2926385378 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f1affd700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f1c0fcf80 0x7f1f1c065c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f1affd700 1 -- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fbcd0 msgr2=0x7f1f1c065740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f1affd700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fbcd0 0x7f1f1c065740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f1affd700 1 -- 172.21.15.32:0/2926385378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c005040 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f1b7fe700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fbcd0 0x7f1f1c065740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f1affd700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f1c0fcf80 0x7f1f1c065c80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0137b0 tx=0x7f1f0c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.927+0000 7f1f13fff700 1 -- 172.21.15.32:0/2926385378 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f0c021070 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.928+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c066300 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.928+0000 7f1f13fff700 1 -- 172.21.15.32:0/2926385378 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f0c01c400 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.928+0000 7f1f13fff700 1 -- 172.21.15.32:0/2926385378 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f0c00e070 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.928+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c06e560 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.929+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c05f5f0 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.929+0000 7f1f13fff700 1 -- 172.21.15.32:0/2926385378 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1f0c013a30 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.930+0000 7f1f13fff700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1f080692a0 0x7f1f0806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:13.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.930+0000 7f1f13fff700 1 -- 172.21.15.32:0/2926385378 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1f0c0883b0 con 0x7f1f1c0fcf80 2024-04-06T14:16:13.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.930+0000 7f1f1b7fe700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1f080692a0 0x7f1f0806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:13.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.930+0000 7f1f1b7fe700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1f080692a0 0x7f1f0806b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1f14004fd0 tx=0x7f1f1400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:13.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:13.935+0000 7f1f13fff700 1 -- 172.21.15.32:0/2926385378 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1f0c054630 con 0x7f1f1c0fcf80 2024-04-06T14:16:14.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.217+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f1c1012e0 con 0x7f1f080692a0 2024-04-06T14:16:14.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.219+0000 7f1f13fff700 1 -- 172.21.15.32:0/2926385378 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f1f1c1012e0 con 0x7f1f080692a0 2024-04-06T14:16:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1f080692a0 msgr2=0x7f1f0806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1f080692a0 0x7f1f0806b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1f14004fd0 tx=0x7f1f1400b040 comp rx=0 tx=0).stop 2024-04-06T14:16:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f1c0fcf80 msgr2=0x7f1f1c065c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f1c0fcf80 0x7f1f1c065c80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0137b0 tx=0x7f1f0c009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 shutdown_connections 2024-04-06T14:16:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f1f080692a0 0x7f1f0806b750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c0fbcd0 0x7f1f1c065740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 --2- 172.21.15.32:0/2926385378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f1c0fcf80 0x7f1f1c065c80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 >> 172.21.15.32:0/2926385378 conn(0x7f1f1c0f7430 msgr2=0x7f1f1c0f8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 shutdown_connections 2024-04-06T14:16:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.221+0000 7f1f21c8b700 1 -- 172.21.15.32:0/2926385378 wait complete. 2024-04-06T14:16:14.233 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:16:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.340+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/1932459184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 msgr2=0x7f3a380fae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.340+0000 7f3a3e4d7700 1 --2- 172.21.15.32:0/1932459184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 0x7f3a380fae90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c004940 tx=0x7f3a2c009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.340+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/1932459184 shutdown_connections 2024-04-06T14:16:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.340+0000 7f3a3e4d7700 1 --2- 172.21.15.32:0/1932459184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 0x7f3a380fae90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.340+0000 7f3a3e4d7700 1 --2- 172.21.15.32:0/1932459184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a380fe400 0x7f3a380fe810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.340+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/1932459184 >> 172.21.15.32:0/1932459184 conn(0x7f3a380f6ba0 msgr2=0x7f3a380f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.340+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/1932459184 shutdown_connections 2024-04-06T14:16:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.340+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/1932459184 wait complete. 2024-04-06T14:16:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a3e4d7700 1 Processor -- start 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a3e4d7700 1 -- start start 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a3e4d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a380fe400 0x7f3a3810d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a3e4d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 0x7f3a3810dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a3e4d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3810e3e0 con 0x7f3a380fed50 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a3e4d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3810ec30 con 0x7f3a380fe400 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 0x7f3a3810dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a37fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a380fe400 0x7f3a3810d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 0x7f3a3810dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40480/0 (socket says 172.21.15.32:40480) 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a37fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a380fe400 0x7f3a3810d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:53752/0 (socket says 172.21.15.32:53752) 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.341+0000 7f3a377fe700 1 -- 172.21.15.32:0/189526253 learned_addr learned my addr 172.21.15.32:0/189526253 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a377fe700 1 -- 172.21.15.32:0/189526253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a380fe400 msgr2=0x7f3a3810d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a377fe700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a380fe400 0x7f3a3810d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a377fe700 1 -- 172.21.15.32:0/189526253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c005040 con 0x7f3a380fed50 2024-04-06T14:16:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a37fff700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a380fe400 0x7f3a3810d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:16:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a377fe700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 0x7f3a3810dde0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0137b0 tx=0x7f3a2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a34ff9700 1 -- 172.21.15.32:0/189526253 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a2c00e070 con 0x7f3a380fed50 2024-04-06T14:16:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3810eeb0 con 0x7f3a380fed50 2024-04-06T14:16:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a34ff9700 1 -- 172.21.15.32:0/189526253 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a2c01c400 con 0x7f3a380fed50 2024-04-06T14:16:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a34ff9700 1 -- 172.21.15.32:0/189526253 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a2c00b650 con 0x7f3a380fed50 2024-04-06T14:16:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.342+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3810f390 con 0x7f3a380fed50 2024-04-06T14:16:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.343+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3805f5f0 con 0x7f3a380fed50 2024-04-06T14:16:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.344+0000 7f3a34ff9700 1 -- 172.21.15.32:0/189526253 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3a2c01c9e0 con 0x7f3a380fed50 2024-04-06T14:16:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.344+0000 7f3a34ff9700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3a200692a0 0x7f3a2006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.344+0000 7f3a34ff9700 1 -- 172.21.15.32:0/189526253 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3a2c0887e0 con 0x7f3a380fed50 2024-04-06T14:16:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.344+0000 7f3a37fff700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3a200692a0 0x7f3a2006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.345+0000 7f3a37fff700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3a200692a0 0x7f3a2006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009b80 tx=0x7f3a2800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:14.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.349+0000 7f3a34ff9700 1 -- 172.21.15.32:0/189526253 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3a2c058550 con 0x7f3a380fed50 2024-04-06T14:16:14.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.655+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a3810f6c0 con 0x7f3a200692a0 2024-04-06T14:16:14.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.656+0000 7f3a34ff9700 1 -- 172.21.15.32:0/189526253 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f3a3810f6c0 con 0x7f3a200692a0 2024-04-06T14:16:14.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.659+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3a200692a0 msgr2=0x7f3a2006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.659+0000 7f3a3e4d7700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3a200692a0 0x7f3a2006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009b80 tx=0x7f3a2800b040 comp rx=0 tx=0).stop 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.659+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 msgr2=0x7f3a3810dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.659+0000 7f3a3e4d7700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 0x7f3a3810dde0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0137b0 tx=0x7f3a2c009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.659+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 shutdown_connections 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.659+0000 7f3a3e4d7700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3a200692a0 0x7f3a2006b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.659+0000 7f3a3e4d7700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a380fe400 0x7f3a3810d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.659+0000 7f3a3e4d7700 1 --2- 172.21.15.32:0/189526253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a380fed50 0x7f3a3810dde0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.659+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 >> 172.21.15.32:0/189526253 conn(0x7f3a380f6ba0 msgr2=0x7f3a380f8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.660+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 shutdown_connections 2024-04-06T14:16:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.660+0000 7f3a3e4d7700 1 -- 172.21.15.32:0/189526253 wait complete. 2024-04-06T14:16:14.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.805+0000 7fdd13ea0700 1 -- 172.21.15.32:0/620649833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd0c0fbcd0 msgr2=0x7fdd0c0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:14.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.805+0000 7fdd13ea0700 1 --2- 172.21.15.32:0/620649833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd0c0fbcd0 0x7fdd0c0fc0e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdd08004940 tx=0x7fdd08009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.805+0000 7fdd13ea0700 1 -- 172.21.15.32:0/620649833 shutdown_connections 2024-04-06T14:16:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.805+0000 7fdd13ea0700 1 --2- 172.21.15.32:0/620649833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd0c0fcf80 0x7fdd0c0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.805+0000 7fdd13ea0700 1 --2- 172.21.15.32:0/620649833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd0c0fbcd0 0x7fdd0c0fc0e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.805+0000 7fdd13ea0700 1 -- 172.21.15.32:0/620649833 >> 172.21.15.32:0/620649833 conn(0x7fdd0c0f7430 msgr2=0x7fdd0c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.805+0000 7fdd13ea0700 1 -- 172.21.15.32:0/620649833 shutdown_connections 2024-04-06T14:16:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.805+0000 7fdd13ea0700 1 -- 172.21.15.32:0/620649833 wait complete. 2024-04-06T14:16:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.806+0000 7fdd13ea0700 1 Processor -- start 2024-04-06T14:16:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.806+0000 7fdd13ea0700 1 -- start start 2024-04-06T14:16:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.806+0000 7fdd13ea0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd0c0fbcd0 0x7fdd0c113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.806+0000 7fdd13ea0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd0c0fcf80 0x7fdd0c114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.806+0000 7fdd13ea0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c1151d0 con 0x7fdd0c0fbcd0 2024-04-06T14:16:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.806+0000 7fdd13ea0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c115340 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.806+0000 7fdd1143b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd0c0fcf80 0x7fdd0c114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.806+0000 7fdd1143b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd0c0fcf80 0x7fdd0c114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:43776/0 (socket says 172.21.15.32:43776) 2024-04-06T14:16:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.806+0000 7fdd1143b700 1 -- 172.21.15.32:0/1026938162 learned_addr learned my addr 172.21.15.32:0/1026938162 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.807+0000 7fdd11c3c700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd0c0fbcd0 0x7fdd0c113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.807+0000 7fdd1143b700 1 -- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd0c0fbcd0 msgr2=0x7fdd0c113f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.807+0000 7fdd1143b700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd0c0fbcd0 0x7fdd0c113f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.807+0000 7fdd1143b700 1 -- 172.21.15.32:0/1026938162 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd08005040 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.807+0000 7fdd1143b700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd0c0fcf80 0x7fdd0c114460 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00eb40 tx=0x7fdcfc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.808+0000 7fdd027fc700 1 -- 172.21.15.32:0/1026938162 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc007cc0 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.808+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0c115480 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.808+0000 7fdd027fc700 1 -- 172.21.15.32:0/1026938162 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcfc00f040 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.808+0000 7fdd027fc700 1 -- 172.21.15.32:0/1026938162 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc00c990 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.808+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0c1159e0 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.809+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0c101910 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.810+0000 7fdd027fc700 1 -- 172.21.15.32:0/1026938162 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fdcfc00cc50 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.810+0000 7fdd027fc700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcf80692b0 0x7fdcf806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.810+0000 7fdd027fc700 1 -- 172.21.15.32:0/1026938162 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdcfc086f20 con 0x7fdd0c0fcf80 2024-04-06T14:16:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.811+0000 7fdd11c3c700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcf80692b0 0x7fdcf806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:14.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.811+0000 7fdd11c3c700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcf80692b0 0x7fdcf806b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c113c90 tx=0x7fdd08009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:14.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:14.815+0000 7fdd027fc700 1 -- 172.21.15.32:0/1026938162 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdcfc04e190 con 0x7fdd0c0fcf80 2024-04-06T14:16:15.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.106+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd0c104830 con 0x7fdcf80692b0 2024-04-06T14:16:15.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.118+0000 7fdd027fc700 1 -- 172.21.15.32:0/1026938162 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3344 (secure 0 0 0) 0x7fdd080096f0 con 0x7fdcf80692b0 2024-04-06T14:16:15.118 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:16:15.118 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 102s ago 7m 26.8M - 0.25.0 c8568f914cd2 d6f57d3d10f9 2024-04-06T14:16:15.118 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 102s ago 8m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:16:15.118 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (7m) 109s ago 7m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:16:15.118 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 102s ago 7m 86.3M - 9.4.7 954c08fa6188 1cc1d256b791 2024-04-06T14:16:15.118 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (2m) 102s ago 2m 10.4M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (2m) 102s ago 2m 15.8M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (2m) 109s ago 2m 16.7M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (2m) 109s ago 2m 13.4M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:9283 running (9m) 102s ago 9m 480M - 17.2.7-819-g6699495c ec300eaa768b 7737c4037eba 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283 running (6m) 109s ago 6m 432M - 17.2.7-819-g6699495c ec300eaa768b b0b7b2c2f6da 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 102s ago 9m 43.4M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (6m) 109s ago 6m 33.5M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 102s ago 8m 16.4M - 1.5.0 0da6a335fe13 41a44c41cd57 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (7m) 109s ago 7m 18.9M - 1.5.0 0da6a335fe13 21ac74498650 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 102s ago 6m 44.6M 4300M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 102s ago 5m 46.3M 4300M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 102s ago 5m 47.9M 4300M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (4m) 109s ago 4m 46.8M 5323M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (4m) 109s ago 4m 44.4M 5323M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:16:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (3m) 109s ago 3m 42.3M 5323M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:16:15.120 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 102s ago 7m 53.1M - 2.43.0 a07b618ecd1d 6742cb6c38a9 2024-04-06T14:16:15.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcf80692b0 msgr2=0x7fdcf806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:15.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcf80692b0 0x7fdcf806b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c113c90 tx=0x7fdd08009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:15.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd0c0fcf80 msgr2=0x7fdd0c114460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:15.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd0c0fcf80 0x7fdd0c114460 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00eb40 tx=0x7fdcfc00b040 comp rx=0 tx=0).stop 2024-04-06T14:16:15.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 shutdown_connections 2024-04-06T14:16:15.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7fdcf80692b0 0x7fdcf806b760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd0c0fbcd0 0x7fdd0c113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 --2- 172.21.15.32:0/1026938162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd0c0fcf80 0x7fdd0c114460 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 >> 172.21.15.32:0/1026938162 conn(0x7fdd0c0f7430 msgr2=0x7fdd0c065c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:15.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 shutdown_connections 2024-04-06T14:16:15.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.120+0000 7fdd13ea0700 1 -- 172.21.15.32:0/1026938162 wait complete. 2024-04-06T14:16:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:14 smithi032 ceph-mon[39730]: pgmap v252: 65 pgs: 65 active+clean; 69 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.7 MiB/s wr, 179 op/s 2024-04-06T14:16:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.234+0000 7f93e8c7c700 1 -- 172.21.15.32:0/3077122121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40fff00 msgr2=0x7f93e4102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.234+0000 7f93e8c7c700 1 --2- 172.21.15.32:0/3077122121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40fff00 0x7f93e4102320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f93d8004880 tx=0x7f93d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 -- 172.21.15.32:0/3077122121 shutdown_connections 2024-04-06T14:16:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 --2- 172.21.15.32:0/3077122121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40fff00 0x7f93e4102320 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 --2- 172.21.15.32:0/3077122121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93e40662e0 0x7f93e40ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 -- 172.21.15.32:0/3077122121 >> 172.21.15.32:0/3077122121 conn(0x7f93e40f7430 msgr2=0x7f93e40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 -- 172.21.15.32:0/3077122121 shutdown_connections 2024-04-06T14:16:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 -- 172.21.15.32:0/3077122121 wait complete. 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 Processor -- start 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 -- start start 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93e40662e0 0x7f93e410d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40fff00 0x7f93e410de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e410e420 con 0x7f93e40662e0 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e8c7c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e410ec70 con 0x7f93e40fff00 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.235+0000 7f93e259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93e40662e0 0x7f93e410d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93e40662e0 0x7f93e410d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51494/0 (socket says 172.21.15.32:51494) 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e259c700 1 -- 172.21.15.32:0/773116611 learned_addr learned my addr 172.21.15.32:0/773116611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e1d9b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40fff00 0x7f93e410de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e1d9b700 1 -- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93e40662e0 msgr2=0x7f93e410d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e1d9b700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93e40662e0 0x7f93e410d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e1d9b700 1 -- 172.21.15.32:0/773116611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d8005040 con 0x7f93e40fff00 2024-04-06T14:16:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e259c700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93e40662e0 0x7f93e410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:16:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e1d9b700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40fff00 0x7f93e410de20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93d8000c00 tx=0x7f93d8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93d2ffd700 1 -- 172.21.15.32:0/773116611 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93d800e030 con 0x7f93e40fff00 2024-04-06T14:16:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93d2ffd700 1 -- 172.21.15.32:0/773116611 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93d801c400 con 0x7f93e40fff00 2024-04-06T14:16:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e8c7c700 1 -- 172.21.15.32:0/773116611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e410eef0 con 0x7f93e40fff00 2024-04-06T14:16:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93d2ffd700 1 -- 172.21.15.32:0/773116611 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93d8012600 con 0x7f93e40fff00 2024-04-06T14:16:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.236+0000 7f93e8c7c700 1 -- 172.21.15.32:0/773116611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e410f430 con 0x7f93e40fff00 2024-04-06T14:16:15.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.238+0000 7f93d0ff9700 1 -- 172.21.15.32:0/773116611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c80052b0 con 0x7f93e40fff00 2024-04-06T14:16:15.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.238+0000 7f93d2ffd700 1 -- 172.21.15.32:0/773116611 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f93d8013a20 con 0x7f93e40fff00 2024-04-06T14:16:15.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.238+0000 7f93d2ffd700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93cc0692a0 0x7f93cc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:15.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.239+0000 7f93d2ffd700 1 -- 172.21.15.32:0/773116611 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f93d8088c20 con 0x7f93e40fff00 2024-04-06T14:16:15.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.239+0000 7f93e259c700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93cc0692a0 0x7f93cc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:15.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.239+0000 7f93e259c700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93cc0692a0 0x7f93cc06b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009b80 tx=0x7f93d400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:15.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.244+0000 7f93d2ffd700 1 -- 172.21.15.32:0/773116611 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f93d80589b0 con 0x7f93e40fff00 2024-04-06T14:16:15.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:14 smithi086 ceph-mon[47925]: pgmap v252: 65 pgs: 65 active+clean; 69 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.7 MiB/s wr, 179 op/s 2024-04-06T14:16:15.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.600+0000 7f93d0ff9700 1 -- 172.21.15.32:0/773116611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f93c8005d70 con 0x7f93e40fff00 2024-04-06T14:16:15.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.600+0000 7f93d2ffd700 1 -- 172.21.15.32:0/773116611 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f93d8058540 con 0x7f93e40fff00 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:16:15.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:16:15.602 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:16:15.602 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:16:15.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:16:15.602 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:16:15.602 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 14 2024-04-06T14:16:15.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:16:15.602 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:16:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 -- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93cc0692a0 msgr2=0x7f93cc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93cc0692a0 0x7f93cc06b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009b80 tx=0x7f93d400b040 comp rx=0 tx=0).stop 2024-04-06T14:16:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 -- 172.21.15.32:0/773116611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40fff00 msgr2=0x7f93e410de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40fff00 0x7f93e410de20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93d8000c00 tx=0x7f93d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 -- 172.21.15.32:0/773116611 shutdown_connections 2024-04-06T14:16:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f93cc0692a0 0x7f93cc06b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93e40662e0 0x7f93e410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 --2- 172.21.15.32:0/773116611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40fff00 0x7f93e410de20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 -- 172.21.15.32:0/773116611 >> 172.21.15.32:0/773116611 conn(0x7f93e40f7430 msgr2=0x7f93e40f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:15.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 -- 172.21.15.32:0/773116611 shutdown_connections 2024-04-06T14:16:15.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.603+0000 7f93d0ff9700 1 -- 172.21.15.32:0/773116611 wait complete. 2024-04-06T14:16:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.711+0000 7f3bb7f83700 1 -- 172.21.15.32:0/487528367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00fff00 msgr2=0x7f3bb0102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.711+0000 7f3bb7f83700 1 --2- 172.21.15.32:0/487528367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00fff00 0x7f3bb0102320 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0004940 tx=0x7f3ba0009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.711+0000 7f3bb7f83700 1 -- 172.21.15.32:0/487528367 shutdown_connections 2024-04-06T14:16:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.711+0000 7f3bb7f83700 1 --2- 172.21.15.32:0/487528367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00fff00 0x7f3bb0102320 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.711+0000 7f3bb7f83700 1 --2- 172.21.15.32:0/487528367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb00662e0 0x7f3bb00ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.711+0000 7f3bb7f83700 1 -- 172.21.15.32:0/487528367 >> 172.21.15.32:0/487528367 conn(0x7f3bb00f7430 msgr2=0x7f3bb00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.711+0000 7f3bb7f83700 1 -- 172.21.15.32:0/487528367 shutdown_connections 2024-04-06T14:16:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.711+0000 7f3bb7f83700 1 -- 172.21.15.32:0/487528367 wait complete. 2024-04-06T14:16:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb7f83700 1 Processor -- start 2024-04-06T14:16:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb7f83700 1 -- start start 2024-04-06T14:16:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb7f83700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00662e0 0x7f3bb010d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb7f83700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb00fff00 0x7f3bb010de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb7f83700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb010e440 con 0x7f3bb00662e0 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb7f83700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb010ec90 con 0x7f3bb00fff00 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb5d1f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00662e0 0x7f3bb010d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb551e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb00fff00 0x7f3bb010de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb5d1f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00662e0 0x7f3bb010d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51504/0 (socket says 172.21.15.32:51504) 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb5d1f700 1 -- 172.21.15.32:0/2727734135 learned_addr learned my addr 172.21.15.32:0/2727734135 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb5d1f700 1 -- 172.21.15.32:0/2727734135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb00fff00 msgr2=0x7f3bb010de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb5d1f700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb00fff00 0x7f3bb010de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb5d1f700 1 -- 172.21.15.32:0/2727734135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba0005040 con 0x7f3bb00662e0 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.712+0000 7f3bb5d1f700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00662e0 0x7f3bb010d900 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00eb40 tx=0x7f3bac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.713+0000 7f3ba67fc700 1 -- 172.21.15.32:0/2727734135 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bac007cc0 con 0x7f3bb00662e0 2024-04-06T14:16:15.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.713+0000 7f3ba67fc700 1 -- 172.21.15.32:0/2727734135 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bac00f040 con 0x7f3bb00662e0 2024-04-06T14:16:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.713+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb010ef10 con 0x7f3bb00662e0 2024-04-06T14:16:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.713+0000 7f3ba67fc700 1 -- 172.21.15.32:0/2727734135 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bac00ca30 con 0x7f3bb00662e0 2024-04-06T14:16:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.713+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb010f450 con 0x7f3bb00662e0 2024-04-06T14:16:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.714+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb005f5f0 con 0x7f3bb00662e0 2024-04-06T14:16:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.714+0000 7f3ba67fc700 1 -- 172.21.15.32:0/2727734135 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3bac00cb90 con 0x7f3bb00662e0 2024-04-06T14:16:15.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.715+0000 7f3ba67fc700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3b9c0692a0 0x7f3b9c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:15.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.715+0000 7f3ba67fc700 1 -- 172.21.15.32:0/2727734135 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3bac086c30 con 0x7f3bb00662e0 2024-04-06T14:16:15.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.715+0000 7f3bb551e700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3b9c0692a0 0x7f3b9c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:15.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.715+0000 7f3bb551e700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3b9c0692a0 0x7f3b9c06b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0004890 tx=0x7f3ba0009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:15.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:15.721+0000 7f3ba67fc700 1 -- 172.21.15.32:0/2727734135 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3bac0569a0 con 0x7f3bb00662e0 2024-04-06T14:16:16.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.011+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 --> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bb010f7d0 con 0x7f3b9c0692a0 2024-04-06T14:16:16.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.012+0000 7f3ba67fc700 1 -- 172.21.15.32:0/2727734135 <== mgr.14206 v2:172.21.15.32:6800/3067649142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f3bb010f7d0 con 0x7f3b9c0692a0 2024-04-06T14:16:16.013 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:16:16.013 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:16:16.013 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:16:16.013 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T14:16:16.013 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-06T14:16:16.013 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "0/2 daemons upgraded", 2024-04-06T14:16:16.013 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mgr daemons", 2024-04-06T14:16:16.013 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:16:16.013 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:16:16.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.014+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3b9c0692a0 msgr2=0x7f3b9c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.014+0000 7f3bb7f83700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3b9c0692a0 0x7f3b9c06b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0004890 tx=0x7f3ba0009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.014+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00662e0 msgr2=0x7f3bb010d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.014+0000 7f3bb7f83700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00662e0 0x7f3bb010d900 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00eb40 tx=0x7f3bac00b040 comp rx=0 tx=0).stop 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.014+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 shutdown_connections 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.014+0000 7f3bb7f83700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:6800/3067649142,v1:172.21.15.32:6801/3067649142] conn(0x7f3b9c0692a0 0x7f3b9c06b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.015+0000 7f3bb7f83700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bb00662e0 0x7f3bb010d900 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.015+0000 7f3bb7f83700 1 --2- 172.21.15.32:0/2727734135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb00fff00 0x7f3bb010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.015+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 >> 172.21.15.32:0/2727734135 conn(0x7f3bb00f7430 msgr2=0x7f3bb00f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.015+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 shutdown_connections 2024-04-06T14:16:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:16.015+0000 7f3bb7f83700 1 -- 172.21.15.32:0/2727734135 wait complete. 2024-04-06T14:16:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:15 smithi032 ceph-mon[39730]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:15 smithi032 ceph-mon[39730]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:15 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/773116611' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:16:16.263 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:15 smithi086 ceph-mon[47925]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:16.263 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:15 smithi086 ceph-mon[47925]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:16.263 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:15 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/773116611' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:16:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:16 smithi032 ceph-mon[39730]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:16 smithi032 ceph-mon[39730]: pgmap v253: 65 pgs: 65 active+clean; 75 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 3.8 MiB/s wr, 159 op/s 2024-04-06T14:16:17.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:16 smithi086 ceph-mon[47925]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:17.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:16 smithi086 ceph-mon[47925]: pgmap v253: 65 pgs: 65 active+clean; 75 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 3.8 MiB/s wr, 159 op/s 2024-04-06T14:16:18.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:17 smithi086 ceph-mon[47925]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:18.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:17 smithi032 ceph-mon[39730]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:18 smithi032 ceph-mon[39730]: pgmap v254: 65 pgs: 65 active+clean; 84 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.6 MiB/s wr, 184 op/s 2024-04-06T14:16:19.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:18 smithi086 ceph-mon[47925]: pgmap v254: 65 pgs: 65 active+clean; 84 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.6 MiB/s wr, 184 op/s 2024-04-06T14:16:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:20 smithi032 ceph-mon[39730]: pgmap v255: 65 pgs: 65 active+clean; 103 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 5.6 MiB/s wr, 209 op/s 2024-04-06T14:16:21.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:20 smithi086 ceph-mon[47925]: pgmap v255: 65 pgs: 65 active+clean; 103 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 5.6 MiB/s wr, 209 op/s 2024-04-06T14:16:23.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:22 smithi032 ceph-mon[39730]: pgmap v256: 65 pgs: 65 active+clean; 103 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.9 MiB/s wr, 204 op/s 2024-04-06T14:16:23.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:22 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:23.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:22 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:23.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:22 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:16:23.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:22 smithi086 ceph-mon[47925]: pgmap v256: 65 pgs: 65 active+clean; 103 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.9 MiB/s wr, 204 op/s 2024-04-06T14:16:23.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:22 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:23.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:22 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:23.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:22 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:16:25.187 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:24 smithi086 ceph-mon[47925]: pgmap v257: 65 pgs: 65 active+clean; 115 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.7 MiB/s wr, 204 op/s 2024-04-06T14:16:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:24 smithi032 ceph-mon[39730]: pgmap v257: 65 pgs: 65 active+clean; 115 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.7 MiB/s wr, 204 op/s 2024-04-06T14:16:26.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:26.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:26.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:26 smithi086 ceph-mon[47925]: pgmap v258: 65 pgs: 65 active+clean; 116 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 3.9 MiB/s wr, 173 op/s 2024-04-06T14:16:26.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:26.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:26 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:26 smithi032 ceph-mon[39730]: pgmap v258: 65 pgs: 65 active+clean; 116 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 3.9 MiB/s wr, 173 op/s 2024-04-06T14:16:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:26 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:27.860 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:27 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:16:27.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:27 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:16:28.961 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:28 smithi086 ceph-mon[47925]: pgmap v259: 65 pgs: 65 active+clean; 122 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.1 MiB/s wr, 191 op/s 2024-04-06T14:16:29.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:28 smithi032 ceph-mon[39730]: pgmap v259: 65 pgs: 65 active+clean; 122 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.1 MiB/s wr, 191 op/s 2024-04-06T14:16:30.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: pgmap v260: 65 pgs: 65 active+clean; 135 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.4 MiB/s wr, 225 op/s 2024-04-06T14:16:30.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:30.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:16:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:16:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:16:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr fail", "who": "smithi032.peohxu"}]: dispatch 2024-04-06T14:16:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: osdmap e47: 6 total, 6 up, 6 in 2024-04-06T14:16:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "mgr fail", "who": "smithi032.peohxu"}]': finished 2024-04-06T14:16:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:30 smithi086 ceph-mon[47925]: mgrmap e20: smithi086.fyyhxm(active, starting, since 0.00663985s) 2024-04-06T14:16:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: pgmap v260: 65 pgs: 65 active+clean; 135 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 4.4 MiB/s wr, 225 op/s 2024-04-06T14:16:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:16:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:16:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' 2024-04-06T14:16:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:16:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr fail", "who": "smithi032.peohxu"}]: dispatch 2024-04-06T14:16:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: osdmap e47: 6 total, 6 up, 6 in 2024-04-06T14:16:30.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: from='mgr.14206 172.21.15.32:0/4088884969' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "mgr fail", "who": "smithi032.peohxu"}]': finished 2024-04-06T14:16:30.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:30 smithi032 ceph-mon[39730]: mgrmap e20: smithi086.fyyhxm(active, starting, since 0.00663985s) 2024-04-06T14:16:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: Active manager daemon smithi086.fyyhxm restarted 2024-04-06T14:16:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: Activating manager daemon smithi086.fyyhxm 2024-04-06T14:16:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: osdmap e48: 6 total, 6 up, 6 in 2024-04-06T14:16:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: mgrmap e21: smithi086.fyyhxm(active, starting, since 0.00570833s) 2024-04-06T14:16:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:16:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:16:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:16:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr metadata", "who": "smithi086.fyyhxm", "id": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:16:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: Manager daemon smithi086.fyyhxm is now available 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.fyyhxm/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.fyyhxm/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.fyyhxm/trash_purge_schedule"}]: dispatch 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.fyyhxm/trash_purge_schedule"}]: dispatch 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:32.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:31 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:32.133 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: Active manager daemon smithi086.fyyhxm restarted 2024-04-06T14:16:32.133 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: Activating manager daemon smithi086.fyyhxm 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: osdmap e48: 6 total, 6 up, 6 in 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: mgrmap e21: smithi086.fyyhxm(active, starting, since 0.00570833s) 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:16:32.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr metadata", "who": "smithi086.fyyhxm", "id": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: Manager daemon smithi086.fyyhxm is now available 2024-04-06T14:16:32.135 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:16:32.136 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:16:32.136 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.fyyhxm/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:16:32.136 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.fyyhxm/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:16:32.136 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.fyyhxm/trash_purge_schedule"}]: dispatch 2024-04-06T14:16:32.136 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.fyyhxm/trash_purge_schedule"}]: dispatch 2024-04-06T14:16:32.136 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:32.136 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:31 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:33.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:32 smithi086 ceph-mon[47925]: Deploying cephadm binary to smithi086 2024-04-06T14:16:33.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:32 smithi086 ceph-mon[47925]: Deploying cephadm binary to smithi032 2024-04-06T14:16:33.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:32 smithi086 ceph-mon[47925]: mgrmap e22: smithi086.fyyhxm(active, since 1.00974s) 2024-04-06T14:16:33.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:32 smithi086 ceph-mon[47925]: pgmap v3: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:16:33.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:32 smithi032 ceph-mon[39730]: Deploying cephadm binary to smithi086 2024-04-06T14:16:33.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:32 smithi032 ceph-mon[39730]: Deploying cephadm binary to smithi032 2024-04-06T14:16:33.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:32 smithi032 ceph-mon[39730]: mgrmap e22: smithi086.fyyhxm(active, since 1.00974s) 2024-04-06T14:16:33.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:32 smithi032 ceph-mon[39730]: pgmap v3: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:16:33.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:33 smithi032 ceph-mon[39730]: [06/Apr/2024:14:16:32] ENGINE Bus STARTING 2024-04-06T14:16:33.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:33 smithi032 ceph-mon[39730]: [06/Apr/2024:14:16:32] ENGINE Serving on https://172.21.15.86:7150 2024-04-06T14:16:33.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:33 smithi032 ceph-mon[39730]: [06/Apr/2024:14:16:32] ENGINE Serving on http://172.21.15.86:8765 2024-04-06T14:16:33.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:33 smithi032 ceph-mon[39730]: [06/Apr/2024:14:16:32] ENGINE Bus STARTED 2024-04-06T14:16:33.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:33 smithi032 ceph-mon[39730]: pgmap v4: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:16:34.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:33 smithi086 ceph-mon[47925]: [06/Apr/2024:14:16:32] ENGINE Bus STARTING 2024-04-06T14:16:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:33 smithi086 ceph-mon[47925]: [06/Apr/2024:14:16:32] ENGINE Serving on https://172.21.15.86:7150 2024-04-06T14:16:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:33 smithi086 ceph-mon[47925]: [06/Apr/2024:14:16:32] ENGINE Serving on http://172.21.15.86:8765 2024-04-06T14:16:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:33 smithi086 ceph-mon[47925]: [06/Apr/2024:14:16:32] ENGINE Bus STARTED 2024-04-06T14:16:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:33 smithi086 ceph-mon[47925]: pgmap v4: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:16:34.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:34 smithi032 ceph-mon[39730]: mgrmap e23: smithi086.fyyhxm(active, since 3s) 2024-04-06T14:16:35.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:34 smithi086 ceph-mon[47925]: mgrmap e23: smithi086.fyyhxm(active, since 3s) 2024-04-06T14:16:35.974 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:35 smithi032 ceph-mon[39730]: pgmap v5: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:16:36.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:35 smithi086 ceph-mon[47925]: pgmap v5: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:16:37.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:36 smithi086 ceph-mon[47925]: mgrmap e24: smithi086.fyyhxm(active, since 5s) 2024-04-06T14:16:37.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:36 smithi032 ceph-mon[39730]: mgrmap e24: smithi086.fyyhxm(active, since 5s) 2024-04-06T14:16:38.068 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:37 smithi032 ceph-mon[39730]: pgmap v6: 65 pgs: 65 active+clean; 161 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 676 KiB/s wr, 80 op/s 2024-04-06T14:16:38.068 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:37 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:38.069 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:37 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:38.069 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:37 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:38.069 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:37 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:38.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:37 smithi086 ceph-mon[47925]: pgmap v6: 65 pgs: 65 active+clean; 161 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 676 KiB/s wr, 80 op/s 2024-04-06T14:16:38.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:37 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:38.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:37 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:38.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:37 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:38.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:37 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:39.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:39 smithi032 ceph-mon[39730]: pgmap v7: 65 pgs: 65 active+clean; 178 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 2.9 MiB/s wr, 330 op/s 2024-04-06T14:16:39.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:39 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:39.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:39 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:39.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:39 smithi032 ceph-mon[39730]: Standby manager daemon smithi032.peohxu started 2024-04-06T14:16:39.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:39 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.32:0/4147948304' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.peohxu/crt"}]: dispatch 2024-04-06T14:16:39.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:39 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.32:0/4147948304' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:16:39.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:39 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.32:0/4147948304' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.peohxu/key"}]: dispatch 2024-04-06T14:16:39.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:39 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.32:0/4147948304' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:16:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:39 smithi086 ceph-mon[47925]: pgmap v7: 65 pgs: 65 active+clean; 178 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 2.9 MiB/s wr, 330 op/s 2024-04-06T14:16:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:39 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:39 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:39 smithi086 ceph-mon[47925]: Standby manager daemon smithi032.peohxu started 2024-04-06T14:16:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:39 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.32:0/4147948304' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.peohxu/crt"}]: dispatch 2024-04-06T14:16:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:39 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.32:0/4147948304' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:16:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:39 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.32:0/4147948304' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.peohxu/key"}]: dispatch 2024-04-06T14:16:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:39 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.32:0/4147948304' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:16:41.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:41.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:41.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: Adjusting osd_memory_target on smithi032 to 1570M 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: mgrmap e25: smithi086.fyyhxm(active, since 9s), standbys: smithi032.peohxu 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:16:41.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:41 smithi086 ceph-mon[47925]: pgmap v8: 65 pgs: 65 active+clean; 178 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 2.3 MiB/s wr, 256 op/s 2024-04-06T14:16:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: Adjusting osd_memory_target on smithi032 to 1570M 2024-04-06T14:16:41.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:41.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: mgrmap e25: smithi086.fyyhxm(active, since 9s), standbys: smithi032.peohxu 2024-04-06T14:16:41.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:16:41.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:41 smithi032 ceph-mon[39730]: pgmap v8: 65 pgs: 65 active+clean; 178 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 2.3 MiB/s wr, 256 op/s 2024-04-06T14:16:42.407 INFO:tasks.workunit.client.1.smithi086.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-06T14:16:42.415 INFO:tasks.workunit.client.1.smithi086.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-06T14:16:42.416 INFO:tasks.workunit.client.1.smithi086.stderr:+ make 2024-04-06T14:16:42.527 INFO:tasks.workunit.client.1.smithi086.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-06T14:16:43.059 INFO:tasks.workunit.client.1.smithi086.stderr:++ readlink -f fsstress 2024-04-06T14:16:43.061 INFO:tasks.workunit.client.1.smithi086.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-06T14:16:43.061 INFO:tasks.workunit.client.1.smithi086.stderr:+ popd 2024-04-06T14:16:43.062 INFO:tasks.workunit.client.1.smithi086.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-06T14:16:43.062 INFO:tasks.workunit.client.1.smithi086.stderr:+ popd 2024-04-06T14:16:43.063 INFO:tasks.workunit.client.1.smithi086.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-06T14:16:43.063 INFO:tasks.workunit.client.1.smithi086.stderr:++ mktemp -d -p . 2024-04-06T14:16:43.065 INFO:tasks.workunit.client.1.smithi086.stderr:+ T=./tmp.uBXlKvppeW 2024-04-06T14:16:43.065 INFO:tasks.workunit.client.1.smithi086.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.uBXlKvppeW -l 1 -n 1000 -p 10 -v 2024-04-06T14:16:43.069 INFO:tasks.workunit.client.1.smithi086.stdout:seed = 1712344390 2024-04-06T14:16:43.072 INFO:tasks.workunit.client.1.smithi086.stdout:0/0: dread - no filename 2024-04-06T14:16:43.074 INFO:tasks.workunit.client.1.smithi086.stdout:1/0: dread - no filename 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/1: mkdir d0 0 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/2: unlink - no file 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/3: fsync - no filename 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/4: stat d0 0 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/5: dwrite - no filename 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/6: fdatasync - no filename 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/7: dread - no filename 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/8: stat d0 0 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/9: truncate - no filename 2024-04-06T14:16:43.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/10: read - no filename 2024-04-06T14:16:43.076 INFO:tasks.workunit.client.1.smithi086.stdout:2/0: dread - no filename 2024-04-06T14:16:43.076 INFO:tasks.workunit.client.1.smithi086.stdout:2/1: write - no filename 2024-04-06T14:16:43.076 INFO:tasks.workunit.client.1.smithi086.stdout:2/2: rename - no filename 2024-04-06T14:16:43.076 INFO:tasks.workunit.client.1.smithi086.stdout:1/1: creat f0 x:0 0 0 2024-04-06T14:16:43.077 INFO:tasks.workunit.client.1.smithi086.stdout:0/11: mkdir d0/d1 0 2024-04-06T14:16:43.078 INFO:tasks.workunit.client.1.smithi086.stdout:0/12: chown d0 13719939 1 2024-04-06T14:16:43.078 INFO:tasks.workunit.client.1.smithi086.stdout:3/0: dread - no filename 2024-04-06T14:16:43.078 INFO:tasks.workunit.client.1.smithi086.stdout:3/1: stat - no entries 2024-04-06T14:16:43.079 INFO:tasks.workunit.client.1.smithi086.stdout:2/3: mkdir d0 0 2024-04-06T14:16:43.079 INFO:tasks.workunit.client.1.smithi086.stdout:2/4: dread - no filename 2024-04-06T14:16:43.079 INFO:tasks.workunit.client.1.smithi086.stdout:2/5: dread - no filename 2024-04-06T14:16:43.079 INFO:tasks.workunit.client.1.smithi086.stdout:2/6: dwrite - no filename 2024-04-06T14:16:43.079 INFO:tasks.workunit.client.1.smithi086.stdout:2/7: dread - no filename 2024-04-06T14:16:43.079 INFO:tasks.workunit.client.1.smithi086.stdout:2/8: truncate - no filename 2024-04-06T14:16:43.081 INFO:tasks.workunit.client.1.smithi086.stdout:4/0: dwrite - no filename 2024-04-06T14:16:43.081 INFO:tasks.workunit.client.1.smithi086.stdout:4/1: dread - no filename 2024-04-06T14:16:43.081 INFO:tasks.workunit.client.1.smithi086.stdout:4/2: write - no filename 2024-04-06T14:16:43.081 INFO:tasks.workunit.client.1.smithi086.stdout:4/3: truncate - no filename 2024-04-06T14:16:43.081 INFO:tasks.workunit.client.1.smithi086.stdout:4/4: dread - no filename 2024-04-06T14:16:43.081 INFO:tasks.workunit.client.1.smithi086.stdout:4/5: write - no filename 2024-04-06T14:16:43.082 INFO:tasks.workunit.client.1.smithi086.stdout:0/13: mkdir d0/d1/d2 0 2024-04-06T14:16:43.082 INFO:tasks.workunit.client.1.smithi086.stdout:4/6: chown . 4075 1 2024-04-06T14:16:43.084 INFO:tasks.workunit.client.1.smithi086.stdout:5/0: write - no filename 2024-04-06T14:16:43.085 INFO:tasks.workunit.client.1.smithi086.stdout:5/1: chown . 11 1 2024-04-06T14:16:43.085 INFO:tasks.workunit.client.1.smithi086.stdout:5/2: read - no filename 2024-04-06T14:16:43.085 INFO:tasks.workunit.client.1.smithi086.stdout:5/3: rmdir - no directory 2024-04-06T14:16:43.086 INFO:tasks.workunit.client.1.smithi086.stdout:5/4: rename - no filename 2024-04-06T14:16:43.086 INFO:tasks.workunit.client.1.smithi086.stdout:3/2: creat f0 x:0 0 0 2024-04-06T14:16:43.086 INFO:tasks.workunit.client.1.smithi086.stdout:6/0: dread - no filename 2024-04-06T14:16:43.086 INFO:tasks.workunit.client.1.smithi086.stdout:6/1: chown . 48728303 1 2024-04-06T14:16:43.086 INFO:tasks.workunit.client.1.smithi086.stdout:6/2: rename - no filename 2024-04-06T14:16:43.087 INFO:tasks.workunit.client.1.smithi086.stdout:6/3: dwrite - no filename 2024-04-06T14:16:43.087 INFO:tasks.workunit.client.1.smithi086.stdout:4/7: mkdir d0 0 2024-04-06T14:16:43.087 INFO:tasks.workunit.client.1.smithi086.stdout:4/8: write - no filename 2024-04-06T14:16:43.087 INFO:tasks.workunit.client.1.smithi086.stdout:3/3: read - f0 zero size 2024-04-06T14:16:43.088 INFO:tasks.workunit.client.1.smithi086.stdout:0/14: symlink d0/d1/d2/l3 0 2024-04-06T14:16:43.089 INFO:tasks.workunit.client.1.smithi086.stdout:0/15: chown d0/d1/d2 237 1 2024-04-06T14:16:43.089 INFO:tasks.workunit.client.1.smithi086.stdout:3/4: dread - f0 zero size 2024-04-06T14:16:43.091 INFO:tasks.workunit.client.1.smithi086.stdout:3/5: stat f0 0 2024-04-06T14:16:43.092 INFO:tasks.workunit.client.1.smithi086.stdout:1/2: dwrite f0 [0,4194304] 0 2024-04-06T14:16:43.093 INFO:tasks.workunit.client.1.smithi086.stdout:5/5: mkdir d0 0 2024-04-06T14:16:43.094 INFO:tasks.workunit.client.1.smithi086.stdout:4/9: creat d0/f1 x:0 0 0 2024-04-06T14:16:43.095 INFO:tasks.workunit.client.1.smithi086.stdout:0/16: mkdir d0/d1/d4 0 2024-04-06T14:16:43.095 INFO:tasks.workunit.client.1.smithi086.stdout:0/17: read - no filename 2024-04-06T14:16:43.098 INFO:tasks.workunit.client.1.smithi086.stdout:6/4: mknod c0 0 2024-04-06T14:16:43.098 INFO:tasks.workunit.client.1.smithi086.stdout:0/18: stat d0/d1/d4 0 2024-04-06T14:16:43.100 INFO:tasks.workunit.client.1.smithi086.stdout:8/0: dwrite - no filename 2024-04-06T14:16:43.101 INFO:tasks.workunit.client.1.smithi086.stdout:7/0: symlink l0 0 2024-04-06T14:16:43.101 INFO:tasks.workunit.client.1.smithi086.stdout:7/1: write - no filename 2024-04-06T14:16:43.102 INFO:tasks.workunit.client.1.smithi086.stdout:1/3: creat f1 x:0 0 0 2024-04-06T14:16:43.106 INFO:tasks.workunit.client.1.smithi086.stdout:5/6: creat d0/f1 x:0 0 0 2024-04-06T14:16:43.106 INFO:tasks.workunit.client.1.smithi086.stdout:9/0: unlink - no file 2024-04-06T14:16:43.106 INFO:tasks.workunit.client.1.smithi086.stdout:9/1: write - no filename 2024-04-06T14:16:43.106 INFO:tasks.workunit.client.1.smithi086.stdout:0/19: creat d0/d1/f5 x:0 0 0 2024-04-06T14:16:43.107 INFO:tasks.workunit.client.1.smithi086.stdout:9/2: chown . 76682 1 2024-04-06T14:16:43.107 INFO:tasks.workunit.client.1.smithi086.stdout:9/3: dwrite - no filename 2024-04-06T14:16:43.107 INFO:tasks.workunit.client.1.smithi086.stdout:9/4: rmdir - no directory 2024-04-06T14:16:43.107 INFO:tasks.workunit.client.1.smithi086.stdout:6/5: creat f1 x:0 0 0 2024-04-06T14:16:43.108 INFO:tasks.workunit.client.1.smithi086.stdout:5/7: chown d0 2 1 2024-04-06T14:16:43.109 INFO:tasks.workunit.client.1.smithi086.stdout:7/2: rename l0 to l1 0 2024-04-06T14:16:43.109 INFO:tasks.workunit.client.1.smithi086.stdout:6/6: chown c0 5614773 1 2024-04-06T14:16:43.110 INFO:tasks.workunit.client.1.smithi086.stdout:0/20: fdatasync d0/d1/f5 0 2024-04-06T14:16:43.111 INFO:tasks.workunit.client.1.smithi086.stdout:1/4: rename f0 to f2 0 2024-04-06T14:16:43.112 INFO:tasks.workunit.client.1.smithi086.stdout:8/1: mkdir d0 0 2024-04-06T14:16:43.113 INFO:tasks.workunit.client.1.smithi086.stdout:8/2: readlink - no filename 2024-04-06T14:16:43.114 INFO:tasks.workunit.client.1.smithi086.stdout:1/5: fdatasync f1 0 2024-04-06T14:16:43.114 INFO:tasks.workunit.client.1.smithi086.stdout:3/6: dwrite f0 [0,4194304] 0 2024-04-06T14:16:43.114 INFO:tasks.workunit.client.1.smithi086.stdout:7/3: creat f2 x:0 0 0 2024-04-06T14:16:43.116 INFO:tasks.workunit.client.1.smithi086.stdout:5/8: mkdir d0/d2 0 2024-04-06T14:16:43.116 INFO:tasks.workunit.client.1.smithi086.stdout:8/3: rename d0 to d0/d1 22 2024-04-06T14:16:43.120 INFO:tasks.workunit.client.1.smithi086.stdout:9/5: creat f0 x:0 0 0 2024-04-06T14:16:43.122 INFO:tasks.workunit.client.1.smithi086.stdout:1/6: truncate f1 646739 0 2024-04-06T14:16:43.122 INFO:tasks.workunit.client.1.smithi086.stdout:4/10: dwrite d0/f1 [0,4194304] 0 2024-04-06T14:16:43.123 INFO:tasks.workunit.client.1.smithi086.stdout:1/7: chown f1 1 1 2024-04-06T14:16:43.124 INFO:tasks.workunit.client.1.smithi086.stdout:5/9: rmdir d0/d2 0 2024-04-06T14:16:43.126 INFO:tasks.workunit.client.1.smithi086.stdout:8/4: creat d0/f2 x:0 0 0 2024-04-06T14:16:43.129 INFO:tasks.workunit.client.1.smithi086.stdout:0/21: truncate d0/d1/f5 678866 0 2024-04-06T14:16:43.129 INFO:tasks.workunit.client.1.smithi086.stdout:9/6: mknod c1 0 2024-04-06T14:16:43.131 INFO:tasks.workunit.client.1.smithi086.stdout:9/7: read - f0 zero size 2024-04-06T14:16:43.132 INFO:tasks.workunit.client.1.smithi086.stdout:4/11: creat d0/f2 x:0 0 0 2024-04-06T14:16:43.133 INFO:tasks.workunit.client.1.smithi086.stdout:0/22: rename d0/d1 to d0/d1/d6 22 2024-04-06T14:16:43.134 INFO:tasks.workunit.client.1.smithi086.stdout:5/10: creat d0/f3 x:0 0 0 2024-04-06T14:16:43.135 INFO:tasks.workunit.client.1.smithi086.stdout:6/7: dwrite f1 [0,4194304] 0 2024-04-06T14:16:43.136 INFO:tasks.workunit.client.1.smithi086.stdout:8/5: creat d0/f3 x:0 0 0 2024-04-06T14:16:43.143 INFO:tasks.workunit.client.1.smithi086.stdout:8/6: truncate d0/f2 394504 0 2024-04-06T14:16:43.144 INFO:tasks.workunit.client.1.smithi086.stdout:1/8: dwrite f1 [0,4194304] 0 2024-04-06T14:16:43.146 INFO:tasks.workunit.client.1.smithi086.stdout:5/11: creat d0/f4 x:0 0 0 2024-04-06T14:16:43.147 INFO:tasks.workunit.client.1.smithi086.stdout:9/8: link f0 f2 0 2024-04-06T14:16:43.147 INFO:tasks.workunit.client.1.smithi086.stdout:5/12: dread - d0/f3 zero size 2024-04-06T14:16:43.147 INFO:tasks.workunit.client.1.smithi086.stdout:0/23: mkdir d0/d1/d4/d7 0 2024-04-06T14:16:43.148 INFO:tasks.workunit.client.1.smithi086.stdout:8/7: dread - d0/f3 zero size 2024-04-06T14:16:43.150 INFO:tasks.workunit.client.1.smithi086.stdout:5/13: chown d0/f3 317777226 1 2024-04-06T14:16:43.153 INFO:tasks.workunit.client.1.smithi086.stdout:8/8: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:43.153 INFO:tasks.workunit.client.1.smithi086.stdout:8/9: readlink - no filename 2024-04-06T14:16:43.154 INFO:tasks.workunit.client.1.smithi086.stdout:6/8: dwrite f1 [0,4194304] 0 2024-04-06T14:16:43.154 INFO:tasks.workunit.client.1.smithi086.stdout:0/24: creat d0/d1/d4/f8 x:0 0 0 2024-04-06T14:16:43.154 INFO:tasks.workunit.client.1.smithi086.stdout:5/14: read - d0/f3 zero size 2024-04-06T14:16:43.203 INFO:tasks.workunit.client.1.smithi086.stdout:0/25: rename d0/d1 to d0/d1/d4/d9 22 2024-04-06T14:16:43.205 INFO:tasks.workunit.client.1.smithi086.stdout:8/10: symlink d0/l4 0 2024-04-06T14:16:43.205 INFO:tasks.workunit.client.1.smithi086.stdout:5/15: truncate d0/f4 96267 0 2024-04-06T14:16:43.205 INFO:tasks.workunit.client.1.smithi086.stdout:6/9: symlink l2 0 2024-04-06T14:16:43.205 INFO:tasks.workunit.client.1.smithi086.stdout:6/10: rmdir - no directory 2024-04-06T14:16:43.207 INFO:tasks.workunit.client.1.smithi086.stdout:7/4: fsync f2 0 2024-04-06T14:16:43.208 INFO:tasks.workunit.client.1.smithi086.stdout:4/12: fsync d0/f2 0 2024-04-06T14:16:43.210 INFO:tasks.workunit.client.1.smithi086.stdout:9/9: getdents . 0 2024-04-06T14:16:43.211 INFO:tasks.workunit.client.1.smithi086.stdout:0/26: creat d0/d1/d4/fa x:0 0 0 2024-04-06T14:16:43.212 INFO:tasks.workunit.client.1.smithi086.stdout:8/11: creat d0/f5 x:0 0 0 2024-04-06T14:16:43.213 INFO:tasks.workunit.client.1.smithi086.stdout:5/16: mknod d0/c5 0 2024-04-06T14:16:43.214 INFO:tasks.workunit.client.1.smithi086.stdout:3/7: dwrite f0 [4194304,4194304] 0 2024-04-06T14:16:43.214 INFO:tasks.workunit.client.1.smithi086.stdout:9/10: creat f3 x:0 0 0 2024-04-06T14:16:43.215 INFO:tasks.workunit.client.1.smithi086.stdout:8/12: write d0/f3 [183305,9289] 0 2024-04-06T14:16:43.216 INFO:tasks.workunit.client.1.smithi086.stdout:4/13: link d0/f2 d0/f3 0 2024-04-06T14:16:43.217 INFO:tasks.workunit.client.1.smithi086.stdout:8/13: stat d0 0 2024-04-06T14:16:43.217 INFO:tasks.workunit.client.1.smithi086.stdout:7/5: unlink l1 0 2024-04-06T14:16:43.217 INFO:tasks.workunit.client.1.smithi086.stdout:7/6: readlink - no filename 2024-04-06T14:16:43.218 INFO:tasks.workunit.client.1.smithi086.stdout:6/11: fdatasync f1 0 2024-04-06T14:16:43.222 INFO:tasks.workunit.client.1.smithi086.stdout:9/11: creat f4 x:0 0 0 2024-04-06T14:16:43.223 INFO:tasks.workunit.client.1.smithi086.stdout:4/14: truncate d0/f2 853170 0 2024-04-06T14:16:43.223 INFO:tasks.workunit.client.1.smithi086.stdout:6/12: write f1 [2001986,103621] 0 2024-04-06T14:16:43.224 INFO:tasks.workunit.client.1.smithi086.stdout:8/14: creat d0/f6 x:0 0 0 2024-04-06T14:16:43.226 INFO:tasks.workunit.client.1.smithi086.stdout:7/7: dread - f2 zero size 2024-04-06T14:16:43.227 INFO:tasks.workunit.client.1.smithi086.stdout:0/27: getdents d0/d1 0 2024-04-06T14:16:43.228 INFO:tasks.workunit.client.1.smithi086.stdout:9/12: mkdir d5 0 2024-04-06T14:16:43.229 INFO:tasks.workunit.client.1.smithi086.stdout:6/13: symlink l3 0 2024-04-06T14:16:43.231 INFO:tasks.workunit.client.1.smithi086.stdout:8/15: mknod d0/c7 0 2024-04-06T14:16:43.232 INFO:tasks.workunit.client.1.smithi086.stdout:4/15: creat d0/f4 x:0 0 0 2024-04-06T14:16:43.232 INFO:tasks.workunit.client.1.smithi086.stdout:7/8: creat f3 x:0 0 0 2024-04-06T14:16:43.233 INFO:tasks.workunit.client.1.smithi086.stdout:5/17: dwrite d0/f4 [0,4194304] 0 2024-04-06T14:16:43.233 INFO:tasks.workunit.client.1.smithi086.stdout:0/28: creat d0/d1/fb x:0 0 0 2024-04-06T14:16:43.233 INFO:tasks.workunit.client.1.smithi086.stdout:6/14: mknod c4 0 2024-04-06T14:16:43.236 INFO:tasks.workunit.client.1.smithi086.stdout:3/8: truncate f0 7883711 0 2024-04-06T14:16:43.237 INFO:tasks.workunit.client.1.smithi086.stdout:9/13: symlink d5/l6 0 2024-04-06T14:16:43.238 INFO:tasks.workunit.client.1.smithi086.stdout:7/9: rename f2 to f4 0 2024-04-06T14:16:43.241 INFO:tasks.workunit.client.1.smithi086.stdout:5/18: creat d0/f6 x:0 0 0 2024-04-06T14:16:43.241 INFO:tasks.workunit.client.1.smithi086.stdout:4/16: rmdir d0 39 2024-04-06T14:16:43.246 INFO:tasks.workunit.client.1.smithi086.stdout:5/19: write d0/f6 [498816,31775] 0 2024-04-06T14:16:43.252 INFO:tasks.workunit.client.1.smithi086.stdout:8/16: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:16:43.253 INFO:tasks.workunit.client.1.smithi086.stdout:3/9: rename f0 to f1 0 2024-04-06T14:16:43.253 INFO:tasks.workunit.client.1.smithi086.stdout:3/10: rmdir - no directory 2024-04-06T14:16:43.254 INFO:tasks.workunit.client.1.smithi086.stdout:0/29: dwrite d0/d1/f5 [0,4194304] 0 2024-04-06T14:16:43.263 INFO:tasks.workunit.client.1.smithi086.stdout:4/17: link d0/f4 d0/f5 0 2024-04-06T14:16:43.266 INFO:tasks.workunit.client.1.smithi086.stdout:0/30: symlink d0/d1/d4/d7/lc 0 2024-04-06T14:16:43.268 INFO:tasks.workunit.client.1.smithi086.stdout:7/10: dwrite f4 [0,4194304] 0 2024-04-06T14:16:43.269 INFO:tasks.workunit.client.1.smithi086.stdout:4/18: write d0/f5 [524881,77792] 0 2024-04-06T14:16:43.269 INFO:tasks.workunit.client.1.smithi086.stdout:8/17: getdents d0 0 2024-04-06T14:16:43.271 INFO:tasks.workunit.client.1.smithi086.stdout:0/31: dread - d0/d1/fb zero size 2024-04-06T14:16:43.274 INFO:tasks.workunit.client.1.smithi086.stdout:4/19: rename d0 to d0/d6 22 2024-04-06T14:16:43.275 INFO:tasks.workunit.client.1.smithi086.stdout:9/14: dwrite f0 [0,4194304] 0 2024-04-06T14:16:43.275 INFO:tasks.workunit.client.1.smithi086.stdout:8/18: mkdir d0/d8 0 2024-04-06T14:16:43.277 INFO:tasks.workunit.client.1.smithi086.stdout:9/15: chown f3 162496 1 2024-04-06T14:16:43.277 INFO:tasks.workunit.client.1.smithi086.stdout:4/20: chown d0/f5 134747492 1 2024-04-06T14:16:43.277 INFO:tasks.workunit.client.1.smithi086.stdout:0/32: chown d0/d1/d4/fa 415 1 2024-04-06T14:16:43.277 INFO:tasks.workunit.client.1.smithi086.stdout:9/16: stat f0 0 2024-04-06T14:16:43.279 INFO:tasks.workunit.client.1.smithi086.stdout:5/20: truncate d0/f6 396648 0 2024-04-06T14:16:43.283 INFO:tasks.workunit.client.1.smithi086.stdout:3/11: truncate f1 3806692 0 2024-04-06T14:16:43.283 INFO:tasks.workunit.client.1.smithi086.stdout:8/19: mknod d0/d8/c9 0 2024-04-06T14:16:43.285 INFO:tasks.workunit.client.1.smithi086.stdout:0/33: truncate d0/d1/d4/fa 754719 0 2024-04-06T14:16:43.285 INFO:tasks.workunit.client.1.smithi086.stdout:5/21: chown d0/f3 4641000 1 2024-04-06T14:16:43.287 INFO:tasks.workunit.client.1.smithi086.stdout:0/34: fsync d0/d1/d4/fa 0 2024-04-06T14:16:43.288 INFO:tasks.workunit.client.1.smithi086.stdout:7/11: dwrite f3 [0,4194304] 0 2024-04-06T14:16:43.290 INFO:tasks.workunit.client.1.smithi086.stdout:4/21: rename d0/f4 to d0/f7 0 2024-04-06T14:16:43.291 INFO:tasks.workunit.client.1.smithi086.stdout:0/35: dread d0/d1/d4/fa [0,4194304] 0 2024-04-06T14:16:43.291 INFO:tasks.workunit.client.1.smithi086.stdout:8/20: mkdir d0/d8/da 0 2024-04-06T14:16:43.291 INFO:tasks.workunit.client.1.smithi086.stdout:0/36: rename d0 to d0/dd 22 2024-04-06T14:16:43.292 INFO:tasks.workunit.client.1.smithi086.stdout:8/21: fsync d0/f5 0 2024-04-06T14:16:43.292 INFO:tasks.workunit.client.1.smithi086.stdout:5/22: mkdir d0/d7 0 2024-04-06T14:16:43.293 INFO:tasks.workunit.client.1.smithi086.stdout:8/22: dread - d0/f5 zero size 2024-04-06T14:16:43.298 INFO:tasks.workunit.client.1.smithi086.stdout:9/17: link f3 d5/f7 0 2024-04-06T14:16:43.300 INFO:tasks.workunit.client.1.smithi086.stdout:4/22: creat d0/f8 x:0 0 0 2024-04-06T14:16:43.301 INFO:tasks.workunit.client.1.smithi086.stdout:0/37: mkdir d0/d1/d4/de 0 2024-04-06T14:16:43.302 INFO:tasks.workunit.client.1.smithi086.stdout:0/38: write d0/d1/d4/fa [995850,47128] 0 2024-04-06T14:16:43.305 INFO:tasks.workunit.client.1.smithi086.stdout:9/18: write f3 [410573,104395] 0 2024-04-06T14:16:43.307 INFO:tasks.workunit.client.1.smithi086.stdout:4/23: unlink d0/f1 0 2024-04-06T14:16:43.308 INFO:tasks.workunit.client.1.smithi086.stdout:5/23: mkdir d0/d7/d8 0 2024-04-06T14:16:43.310 INFO:tasks.workunit.client.1.smithi086.stdout:3/12: write f1 [1076657,116213] 0 2024-04-06T14:16:43.311 INFO:tasks.workunit.client.1.smithi086.stdout:3/13: read f1 [1222293,57481] 0 2024-04-06T14:16:43.311 INFO:tasks.workunit.client.1.smithi086.stdout:0/39: unlink d0/d1/f5 0 2024-04-06T14:16:43.314 INFO:tasks.workunit.client.1.smithi086.stdout:4/24: symlink d0/l9 0 2024-04-06T14:16:43.317 INFO:tasks.workunit.client.1.smithi086.stdout:8/23: rmdir d0/d8/da 0 2024-04-06T14:16:43.319 INFO:tasks.workunit.client.1.smithi086.stdout:3/14: mknod c2 0 2024-04-06T14:16:43.320 INFO:tasks.workunit.client.1.smithi086.stdout:0/40: dwrite d0/d1/d4/f8 [0,4194304] 0 2024-04-06T14:16:43.326 INFO:tasks.workunit.client.1.smithi086.stdout:8/24: mkdir d0/db 0 2024-04-06T14:16:43.328 INFO:tasks.workunit.client.1.smithi086.stdout:0/41: creat d0/ff x:0 0 0 2024-04-06T14:16:43.329 INFO:tasks.workunit.client.1.smithi086.stdout:5/24: rmdir d0/d7/d8 0 2024-04-06T14:16:43.330 INFO:tasks.workunit.client.1.smithi086.stdout:8/25: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:43.331 INFO:tasks.workunit.client.1.smithi086.stdout:0/42: chown d0/d1/d4/fa 4 1 2024-04-06T14:16:43.335 INFO:tasks.workunit.client.1.smithi086.stdout:3/15: dwrite f1 [0,4194304] 0 2024-04-06T14:16:43.335 INFO:tasks.workunit.client.1.smithi086.stdout:8/26: mknod d0/cc 0 2024-04-06T14:16:43.337 INFO:tasks.workunit.client.1.smithi086.stdout:9/19: truncate f2 1014248 0 2024-04-06T14:16:43.342 INFO:tasks.workunit.client.1.smithi086.stdout:8/27: write d0/f3 [2043699,41203] 0 2024-04-06T14:16:43.343 INFO:tasks.workunit.client.1.smithi086.stdout:9/20: mknod d5/c8 0 2024-04-06T14:16:43.345 INFO:tasks.workunit.client.1.smithi086.stdout:8/28: chown d0 248606 1 2024-04-06T14:16:43.348 INFO:tasks.workunit.client.1.smithi086.stdout:9/21: chown d5/f7 261 1 2024-04-06T14:16:43.348 INFO:tasks.workunit.client.1.smithi086.stdout:5/25: rmdir d0/d7 0 2024-04-06T14:16:43.348 INFO:tasks.workunit.client.1.smithi086.stdout:5/26: readlink - no filename 2024-04-06T14:16:43.350 INFO:tasks.workunit.client.1.smithi086.stdout:0/43: truncate d0/d1/d4/f8 608916 0 2024-04-06T14:16:43.353 INFO:tasks.workunit.client.1.smithi086.stdout:3/16: dwrite f1 [4194304,4194304] 0 2024-04-06T14:16:43.353 INFO:tasks.workunit.client.1.smithi086.stdout:4/25: dwrite d0/f5 [0,4194304] 0 2024-04-06T14:16:43.356 INFO:tasks.workunit.client.1.smithi086.stdout:8/29: link d0/d8/c9 d0/db/cd 0 2024-04-06T14:16:43.357 INFO:tasks.workunit.client.1.smithi086.stdout:8/30: chown d0/cc 29 1 2024-04-06T14:16:43.359 INFO:tasks.workunit.client.1.smithi086.stdout:8/31: dread - d0/f6 zero size 2024-04-06T14:16:43.360 INFO:tasks.workunit.client.1.smithi086.stdout:3/17: creat f3 x:0 0 0 2024-04-06T14:16:43.360 INFO:tasks.workunit.client.1.smithi086.stdout:3/18: readlink - no filename 2024-04-06T14:16:43.361 INFO:tasks.workunit.client.1.smithi086.stdout:9/22: fdatasync f0 0 2024-04-06T14:16:43.362 INFO:tasks.workunit.client.1.smithi086.stdout:9/23: truncate f4 119485 0 2024-04-06T14:16:43.363 INFO:tasks.workunit.client.1.smithi086.stdout:9/24: readlink d5/l6 0 2024-04-06T14:16:43.371 INFO:tasks.workunit.client.1.smithi086.stdout:0/44: mknod d0/d1/d4/de/c10 0 2024-04-06T14:16:43.371 INFO:tasks.workunit.client.1.smithi086.stdout:9/25: write f4 [546292,61059] 0 2024-04-06T14:16:43.371 INFO:tasks.workunit.client.1.smithi086.stdout:8/32: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:43.372 INFO:tasks.workunit.client.1.smithi086.stdout:8/33: rename d0/c7 to d0/ce 0 2024-04-06T14:16:43.374 INFO:tasks.workunit.client.1.smithi086.stdout:9/26: dread f3 [0,4194304] 0 2024-04-06T14:16:43.374 INFO:tasks.workunit.client.1.smithi086.stdout:3/19: dwrite f1 [4194304,4194304] 0 2024-04-06T14:16:43.377 INFO:tasks.workunit.client.1.smithi086.stdout:3/20: dread - f3 zero size 2024-04-06T14:16:43.377 INFO:tasks.workunit.client.1.smithi086.stdout:8/34: symlink d0/lf 0 2024-04-06T14:16:43.385 INFO:tasks.workunit.client.1.smithi086.stdout:5/27: truncate d0/f4 2764538 0 2024-04-06T14:16:43.387 INFO:tasks.workunit.client.1.smithi086.stdout:0/45: dwrite d0/d1/d4/fa [0,4194304] 0 2024-04-06T14:16:43.388 INFO:tasks.workunit.client.1.smithi086.stdout:3/21: rename f3 to f4 0 2024-04-06T14:16:43.391 INFO:tasks.workunit.client.1.smithi086.stdout:9/27: link d5/c8 d5/c9 0 2024-04-06T14:16:43.401 INFO:tasks.workunit.client.1.smithi086.stdout:4/26: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:16:43.401 INFO:tasks.workunit.client.1.smithi086.stdout:8/35: dwrite d0/f5 [0,4194304] 0 2024-04-06T14:16:43.402 INFO:tasks.workunit.client.1.smithi086.stdout:9/28: chown d5/c9 105426 1 2024-04-06T14:16:43.404 INFO:tasks.workunit.client.1.smithi086.stdout:5/28: write d0/f6 [927207,104492] 0 2024-04-06T14:16:43.409 INFO:tasks.workunit.client.1.smithi086.stdout:9/29: dread f3 [0,4194304] 0 2024-04-06T14:16:43.409 INFO:tasks.workunit.client.1.smithi086.stdout:8/36: mkdir d0/db/d10 0 2024-04-06T14:16:43.411 INFO:tasks.workunit.client.1.smithi086.stdout:9/30: chown f4 33845 1 2024-04-06T14:16:43.412 INFO:tasks.workunit.client.1.smithi086.stdout:3/22: dread f1 [4194304,4194304] 0 2024-04-06T14:16:43.413 INFO:tasks.workunit.client.1.smithi086.stdout:4/27: creat d0/fa x:0 0 0 2024-04-06T14:16:43.413 INFO:tasks.workunit.client.1.smithi086.stdout:0/46: dwrite d0/ff [0,4194304] 0 2024-04-06T14:16:43.413 INFO:tasks.workunit.client.1.smithi086.stdout:9/31: chown f3 5805 1 2024-04-06T14:16:43.414 INFO:tasks.workunit.client.1.smithi086.stdout:8/37: rename d0 to d0/db/d10/d11 22 2024-04-06T14:16:43.414 INFO:tasks.workunit.client.1.smithi086.stdout:4/28: chown d0 8179 1 2024-04-06T14:16:43.415 INFO:tasks.workunit.client.1.smithi086.stdout:0/47: chown d0/d1/d4/d7 629474 1 2024-04-06T14:16:43.420 INFO:tasks.workunit.client.1.smithi086.stdout:3/23: fdatasync f4 0 2024-04-06T14:16:43.426 INFO:tasks.workunit.client.1.smithi086.stdout:9/32: link f2 d5/fa 0 2024-04-06T14:16:43.427 INFO:tasks.workunit.client.1.smithi086.stdout:0/48: truncate d0/d1/d4/f8 1321135 0 2024-04-06T14:16:43.433 INFO:tasks.workunit.client.1.smithi086.stdout:5/29: dread d0/f4 [0,4194304] 0 2024-04-06T14:16:43.435 INFO:tasks.workunit.client.1.smithi086.stdout:3/24: dwrite f4 [0,4194304] 0 2024-04-06T14:16:43.445 INFO:tasks.workunit.client.1.smithi086.stdout:0/49: dread d0/ff [0,4194304] 0 2024-04-06T14:16:43.447 INFO:tasks.workunit.client.1.smithi086.stdout:8/38: truncate d0/f3 3094740 0 2024-04-06T14:16:43.448 INFO:tasks.workunit.client.1.smithi086.stdout:5/30: dwrite d0/f1 [0,4194304] 0 2024-04-06T14:16:43.449 INFO:tasks.workunit.client.1.smithi086.stdout:3/25: truncate f1 4032209 0 2024-04-06T14:16:43.451 INFO:tasks.workunit.client.1.smithi086.stdout:9/33: dwrite d5/fa [0,4194304] 0 2024-04-06T14:16:43.451 INFO:tasks.workunit.client.1.smithi086.stdout:8/39: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:43.453 INFO:tasks.workunit.client.1.smithi086.stdout:3/26: mkdir d5 0 2024-04-06T14:16:43.453 INFO:tasks.workunit.client.1.smithi086.stdout:0/50: link d0/ff d0/d1/d4/d7/f11 0 2024-04-06T14:16:43.454 INFO:tasks.workunit.client.1.smithi086.stdout:9/34: rmdir d5 39 2024-04-06T14:16:43.456 INFO:tasks.workunit.client.1.smithi086.stdout:0/51: fdatasync d0/d1/d4/fa 0 2024-04-06T14:16:43.459 INFO:tasks.workunit.client.1.smithi086.stdout:5/31: getdents d0 0 2024-04-06T14:16:43.460 INFO:tasks.workunit.client.1.smithi086.stdout:9/35: mknod d5/cb 0 2024-04-06T14:16:43.461 INFO:tasks.workunit.client.1.smithi086.stdout:0/52: rmdir d0/d1/d4/d7 39 2024-04-06T14:16:43.462 INFO:tasks.workunit.client.1.smithi086.stdout:5/32: creat d0/f9 x:0 0 0 2024-04-06T14:16:43.464 INFO:tasks.workunit.client.1.smithi086.stdout:9/36: truncate d5/f7 1389758 0 2024-04-06T14:16:43.466 INFO:tasks.workunit.client.1.smithi086.stdout:3/27: dwrite f4 [4194304,4194304] 0 2024-04-06T14:16:43.471 INFO:tasks.workunit.client.1.smithi086.stdout:9/37: creat d5/fc x:0 0 0 2024-04-06T14:16:43.471 INFO:tasks.workunit.client.1.smithi086.stdout:0/53: creat d0/d1/d4/f12 x:0 0 0 2024-04-06T14:16:43.480 INFO:tasks.workunit.client.1.smithi086.stdout:9/38: rename d5/c9 to d5/cd 0 2024-04-06T14:16:43.480 INFO:tasks.workunit.client.1.smithi086.stdout:8/40: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:43.481 INFO:tasks.workunit.client.1.smithi086.stdout:0/54: rename d0/d1/d4/fa to d0/d1/f13 0 2024-04-06T14:16:43.481 INFO:tasks.workunit.client.1.smithi086.stdout:5/33: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:16:43.486 INFO:tasks.workunit.client.1.smithi086.stdout:8/41: stat d0/f3 0 2024-04-06T14:16:43.486 INFO:tasks.workunit.client.1.smithi086.stdout:9/39: unlink f4 0 2024-04-06T14:16:43.489 INFO:tasks.workunit.client.1.smithi086.stdout:9/40: truncate f0 5045092 0 2024-04-06T14:16:43.489 INFO:tasks.workunit.client.1.smithi086.stdout:5/34: rename d0/f9 to d0/fa 0 2024-04-06T14:16:43.490 INFO:tasks.workunit.client.1.smithi086.stdout:9/41: write f0 [686737,82586] 0 2024-04-06T14:16:43.491 INFO:tasks.workunit.client.1.smithi086.stdout:5/35: mkdir d0/db 0 2024-04-06T14:16:43.491 INFO:tasks.workunit.client.1.smithi086.stdout:3/28: dwrite f4 [0,4194304] 0 2024-04-06T14:16:43.494 INFO:tasks.workunit.client.1.smithi086.stdout:3/29: write f4 [8670979,10979] 0 2024-04-06T14:16:43.498 INFO:tasks.workunit.client.1.smithi086.stdout:5/36: write d0/f3 [4746689,11044] 0 2024-04-06T14:16:43.498 INFO:tasks.workunit.client.1.smithi086.stdout:9/42: unlink d5/fa 0 2024-04-06T14:16:43.501 INFO:tasks.workunit.client.1.smithi086.stdout:8/42: dread d0/f5 [0,4194304] 0 2024-04-06T14:16:43.502 INFO:tasks.workunit.client.1.smithi086.stdout:3/30: dread f4 [0,4194304] 0 2024-04-06T14:16:43.504 INFO:tasks.workunit.client.1.smithi086.stdout:5/37: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:43.506 INFO:tasks.workunit.client.1.smithi086.stdout:0/55: write d0/d1/d4/f8 [1230635,68475] 0 2024-04-06T14:16:43.508 INFO:tasks.workunit.client.1.smithi086.stdout:8/43: rename d0/f6 to d0/db/f12 0 2024-04-06T14:16:43.510 INFO:tasks.workunit.client.1.smithi086.stdout:5/38: symlink d0/db/lc 0 2024-04-06T14:16:43.510 INFO:tasks.workunit.client.1.smithi086.stdout:9/43: link c1 d5/ce 0 2024-04-06T14:16:43.511 INFO:tasks.workunit.client.1.smithi086.stdout:8/44: mkdir d0/d13 0 2024-04-06T14:16:43.512 INFO:tasks.workunit.client.1.smithi086.stdout:0/56: creat d0/d1/d4/d7/f14 x:0 0 0 2024-04-06T14:16:43.513 INFO:tasks.workunit.client.1.smithi086.stdout:3/31: dread f1 [0,4194304] 0 2024-04-06T14:16:43.513 INFO:tasks.workunit.client.1.smithi086.stdout:5/39: creat d0/fd x:0 0 0 2024-04-06T14:16:43.514 INFO:tasks.workunit.client.1.smithi086.stdout:9/44: symlink d5/lf 0 2024-04-06T14:16:43.515 INFO:tasks.workunit.client.1.smithi086.stdout:8/45: symlink d0/d8/l14 0 2024-04-06T14:16:43.515 INFO:tasks.workunit.client.1.smithi086.stdout:5/40: chown d0/db/lc 7 1 2024-04-06T14:16:43.515 INFO:tasks.workunit.client.1.smithi086.stdout:0/57: mkdir d0/d15 0 2024-04-06T14:16:43.517 INFO:tasks.workunit.client.1.smithi086.stdout:5/41: symlink d0/le 0 2024-04-06T14:16:43.518 INFO:tasks.workunit.client.1.smithi086.stdout:3/32: symlink d5/l6 0 2024-04-06T14:16:43.518 INFO:tasks.workunit.client.1.smithi086.stdout:9/45: dread f3 [0,4194304] 0 2024-04-06T14:16:43.518 INFO:tasks.workunit.client.1.smithi086.stdout:5/42: readlink d0/db/lc 0 2024-04-06T14:16:43.519 INFO:tasks.workunit.client.1.smithi086.stdout:8/46: mkdir d0/db/d10/d15 0 2024-04-06T14:16:43.522 INFO:tasks.workunit.client.1.smithi086.stdout:8/47: mkdir d0/d8/d16 0 2024-04-06T14:16:43.526 INFO:tasks.workunit.client.1.smithi086.stdout:5/43: dread d0/f6 [0,4194304] 0 2024-04-06T14:16:43.529 INFO:tasks.workunit.client.1.smithi086.stdout:0/58: dwrite d0/d1/d4/f8 [0,4194304] 0 2024-04-06T14:16:43.530 INFO:tasks.workunit.client.1.smithi086.stdout:5/44: readlink d0/db/lc 0 2024-04-06T14:16:43.533 INFO:tasks.workunit.client.1.smithi086.stdout:5/45: read - d0/fd zero size 2024-04-06T14:16:43.537 INFO:tasks.workunit.client.1.smithi086.stdout:0/59: fdatasync d0/d1/d4/d7/f11 0 2024-04-06T14:16:43.538 INFO:tasks.workunit.client.1.smithi086.stdout:5/46: write d0/fd [426868,130739] 0 2024-04-06T14:16:43.540 INFO:tasks.workunit.client.1.smithi086.stdout:9/46: dwrite f3 [0,4194304] 0 2024-04-06T14:16:43.542 INFO:tasks.workunit.client.1.smithi086.stdout:0/60: creat d0/f16 x:0 0 0 2024-04-06T14:16:43.543 INFO:tasks.workunit.client.1.smithi086.stdout:5/47: creat d0/db/ff x:0 0 0 2024-04-06T14:16:43.544 INFO:tasks.workunit.client.1.smithi086.stdout:8/48: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:16:43.579 INFO:tasks.workunit.client.1.smithi086.stdout:0/61: creat d0/f17 x:0 0 0 2024-04-06T14:16:43.580 INFO:tasks.workunit.client.1.smithi086.stdout:9/47: mkdir d5/d10 0 2024-04-06T14:16:43.580 INFO:tasks.workunit.client.1.smithi086.stdout:5/48: truncate d0/fa 31634 0 2024-04-06T14:16:43.581 INFO:tasks.workunit.client.1.smithi086.stdout:9/48: dread - d5/fc zero size 2024-04-06T14:16:43.581 INFO:tasks.workunit.client.1.smithi086.stdout:5/49: readlink d0/db/lc 0 2024-04-06T14:16:43.581 INFO:tasks.workunit.client.1.smithi086.stdout:3/33: dwrite f1 [0,4194304] 0 2024-04-06T14:16:43.583 INFO:tasks.workunit.client.1.smithi086.stdout:5/50: truncate d0/fd 641001 0 2024-04-06T14:16:43.590 INFO:tasks.workunit.client.1.smithi086.stdout:9/49: dwrite d5/f7 [0,4194304] 0 2024-04-06T14:16:43.730 INFO:tasks.workunit.client.1.smithi086.stdout:8/49: creat d0/f17 x:0 0 0 2024-04-06T14:16:43.730 INFO:tasks.workunit.client.1.smithi086.stdout:0/62: rmdir d0/d1/d4/de 39 2024-04-06T14:16:43.730 INFO:tasks.workunit.client.1.smithi086.stdout:0/63: stat d0/d1/fb 0 2024-04-06T14:16:43.731 INFO:tasks.workunit.client.1.smithi086.stdout:0/64: dread - d0/d1/d4/d7/f14 zero size 2024-04-06T14:16:43.732 INFO:tasks.workunit.client.1.smithi086.stdout:3/34: mkdir d5/d7 0 2024-04-06T14:16:43.734 INFO:tasks.workunit.client.1.smithi086.stdout:3/35: read f4 [1292648,11140] 0 2024-04-06T14:16:43.734 INFO:tasks.workunit.client.1.smithi086.stdout:5/51: symlink d0/db/l10 0 2024-04-06T14:16:43.738 INFO:tasks.workunit.client.1.smithi086.stdout:5/52: creat d0/f11 x:0 0 0 2024-04-06T14:16:43.738 INFO:tasks.workunit.client.1.smithi086.stdout:5/53: chown d0 26373431 1 2024-04-06T14:16:43.738 INFO:tasks.workunit.client.1.smithi086.stdout:3/36: symlink d5/l8 0 2024-04-06T14:16:43.739 INFO:tasks.workunit.client.1.smithi086.stdout:0/65: dwrite d0/f17 [0,4194304] 0 2024-04-06T14:16:43.744 INFO:tasks.workunit.client.1.smithi086.stdout:0/66: creat d0/d1/d4/d7/f18 x:0 0 0 2024-04-06T14:16:43.746 INFO:tasks.workunit.client.1.smithi086.stdout:8/50: link d0/d8/c9 d0/d13/c18 0 2024-04-06T14:16:43.747 INFO:tasks.workunit.client.1.smithi086.stdout:0/67: symlink d0/d1/l19 0 2024-04-06T14:16:43.748 INFO:tasks.workunit.client.1.smithi086.stdout:5/54: dwrite d0/fd [0,4194304] 0 2024-04-06T14:16:43.756 INFO:tasks.workunit.client.1.smithi086.stdout:0/68: truncate d0/d1/d4/f12 713605 0 2024-04-06T14:16:43.757 INFO:tasks.workunit.client.1.smithi086.stdout:5/55: dread d0/f6 [0,4194304] 0 2024-04-06T14:16:43.758 INFO:tasks.workunit.client.1.smithi086.stdout:5/56: write d0/f11 [231757,19298] 0 2024-04-06T14:16:43.758 INFO:tasks.workunit.client.1.smithi086.stdout:0/69: truncate d0/d1/d4/d7/f18 862857 0 2024-04-06T14:16:43.765 INFO:tasks.workunit.client.1.smithi086.stdout:0/70: dwrite d0/d1/d4/d7/f18 [0,4194304] 0 2024-04-06T14:16:43.766 INFO:tasks.workunit.client.1.smithi086.stdout:5/57: dwrite d0/fd [0,4194304] 0 2024-04-06T14:16:43.766 INFO:tasks.workunit.client.1.smithi086.stdout:5/58: chown d0/db/l10 369 1 2024-04-06T14:16:43.773 INFO:tasks.workunit.client.1.smithi086.stdout:0/71: dread d0/d1/d4/d7/f11 [0,4194304] 0 2024-04-06T14:16:43.775 INFO:tasks.workunit.client.1.smithi086.stdout:0/72: mkdir d0/d1/d2/d1a 0 2024-04-06T14:16:43.779 INFO:tasks.workunit.client.1.smithi086.stdout:0/73: write d0/d1/f13 [4673788,4470] 0 2024-04-06T14:16:43.819 INFO:tasks.workunit.client.1.smithi086.stdout:5/59: getdents d0 0 2024-04-06T14:16:43.820 INFO:tasks.workunit.client.1.smithi086.stdout:5/60: fdatasync d0/f1 0 2024-04-06T14:16:43.821 INFO:tasks.workunit.client.1.smithi086.stdout:0/74: rmdir d0/d1/d4 39 2024-04-06T14:16:43.822 INFO:tasks.workunit.client.1.smithi086.stdout:9/50: dwrite f0 [0,4194304] 0 2024-04-06T14:16:43.824 INFO:tasks.workunit.client.1.smithi086.stdout:5/61: mknod d0/db/c12 0 2024-04-06T14:16:43.825 INFO:tasks.workunit.client.1.smithi086.stdout:3/37: truncate f4 1741286 0 2024-04-06T14:16:43.827 INFO:tasks.workunit.client.1.smithi086.stdout:8/51: write d0/f2 [656009,64317] 0 2024-04-06T14:16:43.832 INFO:tasks.workunit.client.1.smithi086.stdout:5/62: creat d0/f13 x:0 0 0 2024-04-06T14:16:43.833 INFO:tasks.workunit.client.1.smithi086.stdout:8/52: creat d0/db/f19 x:0 0 0 2024-04-06T14:16:43.836 INFO:tasks.workunit.client.1.smithi086.stdout:9/51: dwrite d5/fc [0,4194304] 0 2024-04-06T14:16:43.837 INFO:tasks.workunit.client.1.smithi086.stdout:5/63: rename d0/f11 to d0/f14 0 2024-04-06T14:16:43.839 INFO:tasks.workunit.client.1.smithi086.stdout:9/52: write f0 [5678639,33624] 0 2024-04-06T14:16:43.839 INFO:tasks.workunit.client.1.smithi086.stdout:0/75: dwrite d0/d1/d4/f12 [0,4194304] 0 2024-04-06T14:16:43.842 INFO:tasks.workunit.client.1.smithi086.stdout:8/53: mknod d0/d8/d16/c1a 0 2024-04-06T14:16:43.844 INFO:tasks.workunit.client.1.smithi086.stdout:8/54: truncate d0/db/f12 38461 0 2024-04-06T14:16:43.845 INFO:tasks.workunit.client.1.smithi086.stdout:8/55: dread d0/db/f12 [0,4194304] 0 2024-04-06T14:16:43.846 INFO:tasks.workunit.client.1.smithi086.stdout:5/64: rename d0/c5 to d0/c15 0 2024-04-06T14:16:43.854 INFO:tasks.workunit.client.1.smithi086.stdout:5/65: mkdir d0/d16 0 2024-04-06T14:16:43.854 INFO:tasks.workunit.client.1.smithi086.stdout:9/53: link d5/l6 d5/d10/l11 0 2024-04-06T14:16:43.854 INFO:tasks.workunit.client.1.smithi086.stdout:9/54: fsync f0 0 2024-04-06T14:16:43.855 INFO:tasks.workunit.client.1.smithi086.stdout:8/56: symlink d0/db/d10/d15/l1b 0 2024-04-06T14:16:43.856 INFO:tasks.workunit.client.1.smithi086.stdout:5/66: mkdir d0/d17 0 2024-04-06T14:16:43.857 INFO:tasks.workunit.client.1.smithi086.stdout:8/57: chown d0/d13/c18 1224310 1 2024-04-06T14:16:43.858 INFO:tasks.workunit.client.1.smithi086.stdout:9/55: mknod d5/d10/c12 0 2024-04-06T14:16:43.861 INFO:tasks.workunit.client.1.smithi086.stdout:5/67: rename d0/db to d0/d16/d18 0 2024-04-06T14:16:43.861 INFO:tasks.workunit.client.1.smithi086.stdout:8/58: dread d0/f5 [0,4194304] 0 2024-04-06T14:16:43.862 INFO:tasks.workunit.client.1.smithi086.stdout:9/56: mkdir d5/d13 0 2024-04-06T14:16:43.862 INFO:tasks.workunit.client.1.smithi086.stdout:5/68: truncate d0/f14 884661 0 2024-04-06T14:16:43.863 INFO:tasks.workunit.client.1.smithi086.stdout:9/57: write d5/f7 [1633977,112637] 0 2024-04-06T14:16:43.865 INFO:tasks.workunit.client.1.smithi086.stdout:8/59: mknod d0/db/c1c 0 2024-04-06T14:16:43.868 INFO:tasks.workunit.client.1.smithi086.stdout:9/58: mkdir d5/d10/d14 0 2024-04-06T14:16:43.869 INFO:tasks.workunit.client.1.smithi086.stdout:3/38: truncate f4 1182033 0 2024-04-06T14:16:43.871 INFO:tasks.workunit.client.1.smithi086.stdout:0/76: truncate d0/f17 1865026 0 2024-04-06T14:16:43.872 INFO:tasks.workunit.client.1.smithi086.stdout:8/60: dwrite d0/db/f19 [0,4194304] 0 2024-04-06T14:16:43.874 INFO:tasks.workunit.client.1.smithi086.stdout:0/77: dread - d0/d1/d4/d7/f14 zero size 2024-04-06T14:16:43.875 INFO:tasks.workunit.client.1.smithi086.stdout:9/59: dread f3 [0,4194304] 0 2024-04-06T14:16:43.880 INFO:tasks.workunit.client.1.smithi086.stdout:8/61: unlink d0/ce 0 2024-04-06T14:16:43.882 INFO:tasks.workunit.client.1.smithi086.stdout:0/78: mkdir d0/d1/d1b 0 2024-04-06T14:16:43.884 INFO:tasks.workunit.client.1.smithi086.stdout:3/39: rmdir d5/d7 0 2024-04-06T14:16:43.886 INFO:tasks.workunit.client.1.smithi086.stdout:5/69: dwrite d0/f4 [0,4194304] 0 2024-04-06T14:16:43.887 INFO:tasks.workunit.client.1.smithi086.stdout:8/62: mknod d0/db/d10/c1d 0 2024-04-06T14:16:43.887 INFO:tasks.workunit.client.1.smithi086.stdout:5/70: chown d0/fa 122454749 1 2024-04-06T14:16:43.889 INFO:tasks.workunit.client.1.smithi086.stdout:5/71: dread - d0/f13 zero size 2024-04-06T14:16:43.889 INFO:tasks.workunit.client.1.smithi086.stdout:9/60: getdents d5/d10/d14 0 2024-04-06T14:16:43.890 INFO:tasks.workunit.client.1.smithi086.stdout:5/72: readlink d0/d16/d18/lc 0 2024-04-06T14:16:43.891 INFO:tasks.workunit.client.1.smithi086.stdout:0/79: symlink d0/l1c 0 2024-04-06T14:16:43.892 INFO:tasks.workunit.client.1.smithi086.stdout:3/40: creat d5/f9 x:0 0 0 2024-04-06T14:16:43.892 INFO:tasks.workunit.client.1.smithi086.stdout:0/80: fsync d0/d1/d4/d7/f14 0 2024-04-06T14:16:43.893 INFO:tasks.workunit.client.1.smithi086.stdout:3/41: rename d5 to d5/da 22 2024-04-06T14:16:43.893 INFO:tasks.workunit.client.1.smithi086.stdout:8/63: write d0/db/f12 [33246,87904] 0 2024-04-06T14:16:43.894 INFO:tasks.workunit.client.1.smithi086.stdout:0/81: truncate d0/d1/fb 2639 0 2024-04-06T14:16:43.894 INFO:tasks.workunit.client.1.smithi086.stdout:8/64: chown d0/d8/c9 42 1 2024-04-06T14:16:43.895 INFO:tasks.workunit.client.1.smithi086.stdout:3/42: write d5/f9 [198758,2735] 0 2024-04-06T14:16:43.895 INFO:tasks.workunit.client.1.smithi086.stdout:5/73: mkdir d0/d19 0 2024-04-06T14:16:43.895 INFO:tasks.workunit.client.1.smithi086.stdout:0/82: dread - d0/f16 zero size 2024-04-06T14:16:43.896 INFO:tasks.workunit.client.1.smithi086.stdout:8/65: stat d0/cc 0 2024-04-06T14:16:43.897 INFO:tasks.workunit.client.1.smithi086.stdout:9/61: getdents d5/d10/d14 0 2024-04-06T14:16:43.898 INFO:tasks.workunit.client.1.smithi086.stdout:0/83: write d0/d1/d4/d7/f14 [496082,100602] 0 2024-04-06T14:16:43.900 INFO:tasks.workunit.client.1.smithi086.stdout:8/66: mknod d0/c1e 0 2024-04-06T14:16:43.902 INFO:tasks.workunit.client.1.smithi086.stdout:0/84: chown d0/d1/d4/d7/f11 427 1 2024-04-06T14:16:43.902 INFO:tasks.workunit.client.1.smithi086.stdout:3/43: dread f1 [0,4194304] 0 2024-04-06T14:16:43.903 INFO:tasks.workunit.client.1.smithi086.stdout:5/74: dread d0/f1 [0,4194304] 0 2024-04-06T14:16:43.906 INFO:tasks.workunit.client.1.smithi086.stdout:3/44: symlink d5/lb 0 2024-04-06T14:16:43.906 INFO:tasks.workunit.client.1.smithi086.stdout:9/62: dread f0 [0,4194304] 0 2024-04-06T14:16:43.906 INFO:tasks.workunit.client.1.smithi086.stdout:5/75: write d0/f14 [1553437,129651] 0 2024-04-06T14:16:43.908 INFO:tasks.workunit.client.1.smithi086.stdout:8/67: dread d0/f5 [0,4194304] 0 2024-04-06T14:16:43.908 INFO:tasks.workunit.client.1.smithi086.stdout:5/76: write d0/f4 [832658,104960] 0 2024-04-06T14:16:43.910 INFO:tasks.workunit.client.1.smithi086.stdout:9/63: creat d5/d13/f15 x:0 0 0 2024-04-06T14:16:43.911 INFO:tasks.workunit.client.1.smithi086.stdout:3/45: dread f4 [0,4194304] 0 2024-04-06T14:16:43.911 INFO:tasks.workunit.client.1.smithi086.stdout:8/68: unlink d0/db/c1c 0 2024-04-06T14:16:43.913 INFO:tasks.workunit.client.1.smithi086.stdout:9/64: symlink d5/d13/l16 0 2024-04-06T14:16:43.915 INFO:tasks.workunit.client.1.smithi086.stdout:3/46: mknod d5/cc 0 2024-04-06T14:16:43.916 INFO:tasks.workunit.client.1.smithi086.stdout:5/77: link d0/f13 d0/d16/d18/f1a 0 2024-04-06T14:16:43.917 INFO:tasks.workunit.client.1.smithi086.stdout:8/69: rename d0/l4 to d0/d13/l1f 0 2024-04-06T14:16:43.918 INFO:tasks.workunit.client.1.smithi086.stdout:9/65: symlink d5/d13/l17 0 2024-04-06T14:16:43.921 INFO:tasks.workunit.client.1.smithi086.stdout:3/47: link d5/cc d5/cd 0 2024-04-06T14:16:43.924 INFO:tasks.workunit.client.1.smithi086.stdout:8/70: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:16:43.924 INFO:tasks.workunit.client.1.smithi086.stdout:9/66: dread f3 [0,4194304] 0 2024-04-06T14:16:43.924 INFO:tasks.workunit.client.1.smithi086.stdout:3/48: rename f1 to d5/fe 0 2024-04-06T14:16:43.927 INFO:tasks.workunit.client.1.smithi086.stdout:3/49: mkdir d5/df 0 2024-04-06T14:16:43.928 INFO:tasks.workunit.client.1.smithi086.stdout:8/71: dread d0/f5 [0,4194304] 0 2024-04-06T14:16:43.929 INFO:tasks.workunit.client.1.smithi086.stdout:9/67: fsync d5/f7 0 2024-04-06T14:16:43.931 INFO:tasks.workunit.client.1.smithi086.stdout:8/72: dread d0/db/f12 [0,4194304] 0 2024-04-06T14:16:43.932 INFO:tasks.workunit.client.1.smithi086.stdout:3/50: unlink d5/l8 0 2024-04-06T14:16:43.933 INFO:tasks.workunit.client.1.smithi086.stdout:8/73: dread d0/db/f12 [0,4194304] 0 2024-04-06T14:16:43.934 INFO:tasks.workunit.client.1.smithi086.stdout:9/68: mkdir d5/d18 0 2024-04-06T14:16:43.934 INFO:tasks.workunit.client.1.smithi086.stdout:3/51: fsync f4 0 2024-04-06T14:16:43.935 INFO:tasks.workunit.client.1.smithi086.stdout:3/52: fsync d5/f9 0 2024-04-06T14:16:43.936 INFO:tasks.workunit.client.1.smithi086.stdout:3/53: read f4 [159657,116155] 0 2024-04-06T14:16:43.942 INFO:tasks.workunit.client.1.smithi086.stdout:3/54: link c2 d5/c10 0 2024-04-06T14:16:43.944 INFO:tasks.workunit.client.1.smithi086.stdout:9/69: getdents d5 0 2024-04-06T14:16:43.947 INFO:tasks.workunit.client.1.smithi086.stdout:3/55: truncate d5/f9 187702 0 2024-04-06T14:16:43.948 INFO:tasks.workunit.client.1.smithi086.stdout:8/74: truncate d0/f3 357702 0 2024-04-06T14:16:43.948 INFO:tasks.workunit.client.1.smithi086.stdout:8/75: fdatasync d0/f17 0 2024-04-06T14:16:43.949 INFO:tasks.workunit.client.1.smithi086.stdout:3/56: truncate f4 1446902 0 2024-04-06T14:16:43.949 INFO:tasks.workunit.client.1.smithi086.stdout:9/70: mkdir d5/d10/d14/d19 0 2024-04-06T14:16:43.950 INFO:tasks.workunit.client.1.smithi086.stdout:9/71: chown f2 110 1 2024-04-06T14:16:43.951 INFO:tasks.workunit.client.1.smithi086.stdout:8/76: creat d0/db/d10/f20 x:0 0 0 2024-04-06T14:16:43.952 INFO:tasks.workunit.client.1.smithi086.stdout:9/72: mkdir d5/d13/d1a 0 2024-04-06T14:16:43.953 INFO:tasks.workunit.client.1.smithi086.stdout:8/77: creat d0/d13/f21 x:0 0 0 2024-04-06T14:16:43.954 INFO:tasks.workunit.client.1.smithi086.stdout:9/73: mkdir d5/d10/d14/d19/d1b 0 2024-04-06T14:16:43.955 INFO:tasks.workunit.client.1.smithi086.stdout:8/78: mkdir d0/d8/d22 0 2024-04-06T14:16:43.956 INFO:tasks.workunit.client.1.smithi086.stdout:8/79: readlink d0/db/d10/d15/l1b 0 2024-04-06T14:16:43.957 INFO:tasks.workunit.client.1.smithi086.stdout:9/74: creat d5/f1c x:0 0 0 2024-04-06T14:16:43.964 INFO:tasks.workunit.client.1.smithi086.stdout:9/75: creat d5/d18/f1d x:0 0 0 2024-04-06T14:16:43.964 INFO:tasks.workunit.client.1.smithi086.stdout:9/76: chown d5/fc 7 1 2024-04-06T14:16:43.965 INFO:tasks.workunit.client.1.smithi086.stdout:8/80: link d0/d8/l14 d0/d8/d16/l23 0 2024-04-06T14:16:43.968 INFO:tasks.workunit.client.1.smithi086.stdout:9/77: creat d5/d10/d14/d19/f1e x:0 0 0 2024-04-06T14:16:43.968 INFO:tasks.workunit.client.1.smithi086.stdout:8/81: chown d0/d8/d16/l23 93549347 1 2024-04-06T14:16:43.969 INFO:tasks.workunit.client.1.smithi086.stdout:3/57: write f4 [1076527,78991] 0 2024-04-06T14:16:43.972 INFO:tasks.workunit.client.1.smithi086.stdout:9/78: symlink d5/d10/d14/d19/d1b/l1f 0 2024-04-06T14:16:43.973 INFO:tasks.workunit.client.1.smithi086.stdout:8/82: unlink d0/d8/d16/l23 0 2024-04-06T14:16:43.975 INFO:tasks.workunit.client.1.smithi086.stdout:8/83: fsync d0/db/f12 0 2024-04-06T14:16:43.976 INFO:tasks.workunit.client.1.smithi086.stdout:8/84: dread - d0/db/d10/f20 zero size 2024-04-06T14:16:43.978 INFO:tasks.workunit.client.1.smithi086.stdout:3/58: write d5/f9 [456284,86622] 0 2024-04-06T14:16:43.978 INFO:tasks.workunit.client.1.smithi086.stdout:9/79: dwrite f2 [0,4194304] 0 2024-04-06T14:16:43.981 INFO:tasks.workunit.client.1.smithi086.stdout:3/59: creat d5/f11 x:0 0 0 2024-04-06T14:16:43.983 INFO:tasks.workunit.client.1.smithi086.stdout:3/60: dread - d5/f11 zero size 2024-04-06T14:16:43.985 INFO:tasks.workunit.client.1.smithi086.stdout:9/80: link f2 d5/f20 0 2024-04-06T14:16:43.985 INFO:tasks.workunit.client.1.smithi086.stdout:8/85: dwrite d0/db/d10/f20 [0,4194304] 0 2024-04-06T14:16:43.988 INFO:tasks.workunit.client.1.smithi086.stdout:9/81: write d5/f1c [245193,38833] 0 2024-04-06T14:16:43.988 INFO:tasks.workunit.client.1.smithi086.stdout:3/61: mknod d5/c12 0 2024-04-06T14:16:43.988 INFO:tasks.workunit.client.1.smithi086.stdout:8/86: fdatasync d0/f5 0 2024-04-06T14:16:43.989 INFO:tasks.workunit.client.1.smithi086.stdout:9/82: write d5/fc [4076639,51707] 0 2024-04-06T14:16:43.990 INFO:tasks.workunit.client.1.smithi086.stdout:8/87: dread d0/db/f12 [0,4194304] 0 2024-04-06T14:16:43.991 INFO:tasks.workunit.client.1.smithi086.stdout:3/62: creat d5/df/f13 x:0 0 0 2024-04-06T14:16:43.992 INFO:tasks.workunit.client.1.smithi086.stdout:3/63: stat d5 0 2024-04-06T14:16:43.993 INFO:tasks.workunit.client.1.smithi086.stdout:8/88: mknod d0/c24 0 2024-04-06T14:16:43.993 INFO:tasks.workunit.client.1.smithi086.stdout:3/64: chown d5/f11 5331862 1 2024-04-06T14:16:43.996 INFO:tasks.workunit.client.1.smithi086.stdout:3/65: write d5/df/f13 [134611,24847] 0 2024-04-06T14:16:43.998 INFO:tasks.workunit.client.1.smithi086.stdout:8/89: getdents d0/d8 0 2024-04-06T14:16:43.998 INFO:tasks.workunit.client.1.smithi086.stdout:9/83: dwrite f3 [0,4194304] 0 2024-04-06T14:16:44.000 INFO:tasks.workunit.client.1.smithi086.stdout:3/66: write d5/df/f13 [1106847,12343] 0 2024-04-06T14:16:44.004 INFO:tasks.workunit.client.1.smithi086.stdout:8/90: read d0/db/f12 [61995,36963] 0 2024-04-06T14:16:44.007 INFO:tasks.workunit.client.1.smithi086.stdout:8/91: write d0/f3 [420881,26430] 0 2024-04-06T14:16:44.009 INFO:tasks.workunit.client.1.smithi086.stdout:8/92: rmdir d0/d8 39 2024-04-06T14:16:44.010 INFO:tasks.workunit.client.1.smithi086.stdout:9/84: dwrite d5/f20 [0,4194304] 0 2024-04-06T14:16:44.012 INFO:tasks.workunit.client.1.smithi086.stdout:3/67: dwrite d5/df/f13 [0,4194304] 0 2024-04-06T14:16:44.012 INFO:tasks.workunit.client.1.smithi086.stdout:8/93: write d0/db/d10/f20 [639608,3997] 0 2024-04-06T14:16:44.017 INFO:tasks.workunit.client.1.smithi086.stdout:9/85: symlink d5/d10/d14/d19/d1b/l21 0 2024-04-06T14:16:44.021 INFO:tasks.workunit.client.1.smithi086.stdout:9/86: creat d5/d10/d14/f22 x:0 0 0 2024-04-06T14:16:44.021 INFO:tasks.workunit.client.1.smithi086.stdout:9/87: write f3 [2688166,89087] 0 2024-04-06T14:16:44.023 INFO:tasks.workunit.client.1.smithi086.stdout:3/68: getdents d5/df 0 2024-04-06T14:16:44.024 INFO:tasks.workunit.client.1.smithi086.stdout:9/88: dread - d5/d10/d14/d19/f1e zero size 2024-04-06T14:16:44.025 INFO:tasks.workunit.client.1.smithi086.stdout:8/94: link d0/lf d0/l25 0 2024-04-06T14:16:44.025 INFO:tasks.workunit.client.1.smithi086.stdout:8/95: truncate d0/db/f12 264919 0 2024-04-06T14:16:44.026 INFO:tasks.workunit.client.1.smithi086.stdout:9/89: stat c1 0 2024-04-06T14:16:44.028 INFO:tasks.workunit.client.1.smithi086.stdout:9/90: symlink d5/d10/l23 0 2024-04-06T14:16:44.029 INFO:tasks.workunit.client.1.smithi086.stdout:8/96: creat d0/f26 x:0 0 0 2024-04-06T14:16:44.031 INFO:tasks.workunit.client.1.smithi086.stdout:9/91: creat d5/d13/d1a/f24 x:0 0 0 2024-04-06T14:16:44.033 INFO:tasks.workunit.client.1.smithi086.stdout:8/97: mknod d0/c27 0 2024-04-06T14:16:44.036 INFO:tasks.workunit.client.1.smithi086.stdout:8/98: creat d0/db/d10/d15/f28 x:0 0 0 2024-04-06T14:16:44.036 INFO:tasks.workunit.client.1.smithi086.stdout:9/92: dread d5/fc [0,4194304] 0 2024-04-06T14:16:44.039 INFO:tasks.workunit.client.1.smithi086.stdout:9/93: mkdir d5/d10/d14/d19/d25 0 2024-04-06T14:16:44.039 INFO:tasks.workunit.client.1.smithi086.stdout:8/99: getdents d0/d8/d22 0 2024-04-06T14:16:44.040 INFO:tasks.workunit.client.1.smithi086.stdout:9/94: write d5/d13/d1a/f24 [329325,130541] 0 2024-04-06T14:16:44.041 INFO:tasks.workunit.client.1.smithi086.stdout:8/100: mknod d0/db/d10/c29 0 2024-04-06T14:16:44.042 INFO:tasks.workunit.client.1.smithi086.stdout:8/101: write d0/f2 [859511,14841] 0 2024-04-06T14:16:44.044 INFO:tasks.workunit.client.1.smithi086.stdout:8/102: symlink d0/db/l2a 0 2024-04-06T14:16:44.045 INFO:tasks.workunit.client.1.smithi086.stdout:8/103: write d0/f17 [702075,47371] 0 2024-04-06T14:16:44.045 INFO:tasks.workunit.client.1.smithi086.stdout:8/104: chown d0/d13/f21 2764 1 2024-04-06T14:16:44.046 INFO:tasks.workunit.client.1.smithi086.stdout:8/105: chown d0/c1e 479754 1 2024-04-06T14:16:44.047 INFO:tasks.workunit.client.1.smithi086.stdout:8/106: symlink d0/db/l2b 0 2024-04-06T14:16:44.051 INFO:tasks.workunit.client.1.smithi086.stdout:8/107: dread d0/db/f19 [0,4194304] 0 2024-04-06T14:16:44.052 INFO:tasks.workunit.client.1.smithi086.stdout:8/108: write d0/f3 [603905,107791] 0 2024-04-06T14:16:44.056 INFO:tasks.workunit.client.1.smithi086.stdout:8/109: rmdir d0/db/d10/d15 39 2024-04-06T14:16:44.060 INFO:tasks.workunit.client.1.smithi086.stdout:8/110: mknod d0/db/d10/d15/c2c 0 2024-04-06T14:16:44.066 INFO:tasks.workunit.client.1.smithi086.stdout:8/111: dwrite d0/db/f19 [0,4194304] 0 2024-04-06T14:16:44.070 INFO:tasks.workunit.client.1.smithi086.stdout:8/112: rename d0/db/d10/d15/c2c to d0/c2d 0 2024-04-06T14:16:44.071 INFO:tasks.workunit.client.1.smithi086.stdout:8/113: write d0/db/f19 [2500517,34706] 0 2024-04-06T14:16:44.073 INFO:tasks.workunit.client.1.smithi086.stdout:8/114: truncate d0/f26 683048 0 2024-04-06T14:16:44.075 INFO:tasks.workunit.client.1.smithi086.stdout:8/115: mknod d0/d8/d16/c2e 0 2024-04-06T14:16:44.076 INFO:tasks.workunit.client.1.smithi086.stdout:8/116: mknod d0/db/c2f 0 2024-04-06T14:16:44.082 INFO:tasks.workunit.client.1.smithi086.stdout:8/117: dwrite d0/f5 [0,4194304] 0 2024-04-06T14:16:44.083 INFO:tasks.workunit.client.1.smithi086.stdout:8/118: fsync d0/f3 0 2024-04-06T14:16:44.088 INFO:tasks.workunit.client.1.smithi086.stdout:8/119: rename d0/cc to d0/d8/d16/c30 0 2024-04-06T14:16:44.097 INFO:tasks.workunit.client.1.smithi086.stdout:8/120: chown d0/c2d 0 1 2024-04-06T14:16:44.098 INFO:tasks.workunit.client.1.smithi086.stdout:8/121: stat d0/db/f12 0 2024-04-06T14:16:44.101 INFO:tasks.workunit.client.1.smithi086.stdout:8/122: link d0/f2 d0/d13/f31 0 2024-04-06T14:16:44.102 INFO:tasks.workunit.client.1.smithi086.stdout:8/123: chown d0/d8/d16/c2e 8449 1 2024-04-06T14:16:44.104 INFO:tasks.workunit.client.1.smithi086.stdout:8/124: mkdir d0/db/d32 0 2024-04-06T14:16:44.104 INFO:tasks.workunit.client.1.smithi086.stdout:8/125: fsync d0/f17 0 2024-04-06T14:16:44.105 INFO:tasks.workunit.client.1.smithi086.stdout:8/126: write d0/f26 [570201,37187] 0 2024-04-06T14:16:44.106 INFO:tasks.workunit.client.1.smithi086.stdout:8/127: dread - d0/db/d10/d15/f28 zero size 2024-04-06T14:16:44.111 INFO:tasks.workunit.client.1.smithi086.stdout:8/128: link d0/d8/d16/c1a d0/db/d10/c33 0 2024-04-06T14:16:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:43 smithi086 ceph-mon[47925]: pgmap v9: 65 pgs: 65 active+clean; 200 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 416 op/s 2024-04-06T14:16:44.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:43 smithi032 ceph-mon[39730]: pgmap v9: 65 pgs: 65 active+clean; 200 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 4.0 MiB/s wr, 416 op/s 2024-04-06T14:16:46.124 INFO:tasks.workunit.client.0.smithi032.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-06T14:16:46.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:45 smithi086 ceph-mon[47925]: pgmap v10: 65 pgs: 65 active+clean; 200 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 3.6 MiB/s wr, 381 op/s 2024-04-06T14:16:46.134 INFO:tasks.workunit.client.0.smithi032.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-06T14:16:46.135 INFO:tasks.workunit.client.0.smithi032.stderr:+ make 2024-04-06T14:16:46.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.137+0000 7f53a9ca1700 1 -- 172.21.15.32:0/1155128313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4066260 msgr2=0x7f53a4101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.137+0000 7f53a9ca1700 1 --2- 172.21.15.32:0/1155128313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4066260 0x7f53a4101e40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5394004940 tx=0x7f5394009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:46.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.137+0000 7f53a9ca1700 1 -- 172.21.15.32:0/1155128313 shutdown_connections 2024-04-06T14:16:46.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.137+0000 7f53a9ca1700 1 --2- 172.21.15.32:0/1155128313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53a4102380 0x7f53a4104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.137+0000 7f53a9ca1700 1 --2- 172.21.15.32:0/1155128313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4066260 0x7f53a4101e40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.137+0000 7f53a9ca1700 1 -- 172.21.15.32:0/1155128313 >> 172.21.15.32:0/1155128313 conn(0x7f53a40f7430 msgr2=0x7f53a40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:46.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.137+0000 7f53a9ca1700 1 -- 172.21.15.32:0/1155128313 shutdown_connections 2024-04-06T14:16:46.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.137+0000 7f53a9ca1700 1 -- 172.21.15.32:0/1155128313 wait complete. 2024-04-06T14:16:46.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.137+0000 7f53a9ca1700 1 Processor -- start 2024-04-06T14:16:46.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a9ca1700 1 -- start start 2024-04-06T14:16:46.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a9ca1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53a4066260 0x7f53a410fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a9ca1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4102380 0x7f53a4110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a9ca1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a4110680 con 0x7f53a4102380 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a9ca1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a4110ed0 con 0x7f53a4066260 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4102380 0x7f53a4110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4102380 0x7f53a4110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48724/0 (socket says 172.21.15.32:48724) 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a2ffd700 1 -- 172.21.15.32:0/3456932309 learned_addr learned my addr 172.21.15.32:0/3456932309 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a37fe700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53a4066260 0x7f53a410fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a2ffd700 1 -- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53a4066260 msgr2=0x7f53a410fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a2ffd700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53a4066260 0x7f53a410fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a2ffd700 1 -- 172.21.15.32:0/3456932309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5394005040 con 0x7f53a4102380 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.138+0000 7f53a2ffd700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4102380 0x7f53a4110080 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f539800eb40 tx=0x7f539800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:46.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.139+0000 7f538bfff700 1 -- 172.21.15.32:0/3456932309 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5398007cc0 con 0x7f53a4102380 2024-04-06T14:16:46.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.139+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a4111150 con 0x7f53a4102380 2024-04-06T14:16:46.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.139+0000 7f538bfff700 1 -- 172.21.15.32:0/3456932309 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f539800f040 con 0x7f53a4102380 2024-04-06T14:16:46.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.139+0000 7f538bfff700 1 -- 172.21.15.32:0/3456932309 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f539800c990 con 0x7f53a4102380 2024-04-06T14:16:46.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.139+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a4111690 con 0x7f53a4102380 2024-04-06T14:16:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.140+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a405f5f0 con 0x7f53a4102380 2024-04-06T14:16:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.140+0000 7f538bfff700 1 -- 172.21.15.32:0/3456932309 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f539800cc00 con 0x7f53a4102380 2024-04-06T14:16:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.141+0000 7f538bfff700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f538c06aec0 0x7f538c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.141+0000 7f538bfff700 1 -- 172.21.15.32:0/3456932309 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f5398089490 con 0x7f53a4102380 2024-04-06T14:16:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.141+0000 7f53a37fe700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f538c06aec0 0x7f538c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:46.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.141+0000 7f53a37fe700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f538c06aec0 0x7f538c06d370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f53940137b0 tx=0x7f5394009e30 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:46.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.148+0000 7f538bfff700 1 -- 172.21.15.32:0/3456932309 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f539804f560 con 0x7f53a4102380 2024-04-06T14:16:46.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:45 smithi032 ceph-mon[39730]: pgmap v10: 65 pgs: 65 active+clean; 200 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 3.6 MiB/s wr, 381 op/s 2024-04-06T14:16:46.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.393+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f53a41119c0 con 0x7f538c06aec0 2024-04-06T14:16:46.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.395+0000 7f538bfff700 1 -- 172.21.15.32:0/3456932309 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f53a41119c0 con 0x7f538c06aec0 2024-04-06T14:16:46.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f538c06aec0 msgr2=0x7f538c06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f538c06aec0 0x7f538c06d370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f53940137b0 tx=0x7f5394009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:46.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4102380 msgr2=0x7f53a4110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4102380 0x7f53a4110080 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f539800eb40 tx=0x7f539800b040 comp rx=0 tx=0).stop 2024-04-06T14:16:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 shutdown_connections 2024-04-06T14:16:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f538c06aec0 0x7f538c06d370 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53a4066260 0x7f53a410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 --2- 172.21.15.32:0/3456932309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a4102380 0x7f53a4110080 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 >> 172.21.15.32:0/3456932309 conn(0x7f53a40f7430 msgr2=0x7f53a40f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 shutdown_connections 2024-04-06T14:16:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.398+0000 7f53a9ca1700 1 -- 172.21.15.32:0/3456932309 wait complete. 2024-04-06T14:16:46.410 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:16:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.521+0000 7f3a867db700 1 -- 172.21.15.32:0/3071552686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 msgr2=0x7f3a800fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.521+0000 7f3a867db700 1 --2- 172.21.15.32:0/3071552686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 0x7f3a800fd400 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3a74004880 tx=0x7f3a74009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.521+0000 7f3a867db700 1 -- 172.21.15.32:0/3071552686 shutdown_connections 2024-04-06T14:16:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.521+0000 7f3a867db700 1 --2- 172.21.15.32:0/3071552686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 0x7f3a800fd400 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.521+0000 7f3a867db700 1 --2- 172.21.15.32:0/3071552686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a800fbce0 0x7f3a800fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.521+0000 7f3a867db700 1 -- 172.21.15.32:0/3071552686 >> 172.21.15.32:0/3071552686 conn(0x7f3a800f7480 msgr2=0x7f3a800f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.521+0000 7f3a867db700 1 -- 172.21.15.32:0/3071552686 shutdown_connections 2024-04-06T14:16:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.521+0000 7f3a867db700 1 -- 172.21.15.32:0/3071552686 wait complete. 2024-04-06T14:16:46.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.521+0000 7f3a867db700 1 Processor -- start 2024-04-06T14:16:46.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a867db700 1 -- start start 2024-04-06T14:16:46.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a867db700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a800fbce0 0x7f3a8010d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:46.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a867db700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 0x7f3a8010de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:46.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a867db700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8010e480 con 0x7f3a800fcf90 2024-04-06T14:16:46.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a867db700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8010ec30 con 0x7f3a800fbce0 2024-04-06T14:16:46.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a7f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 0x7f3a8010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a7ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a800fbce0 0x7f3a8010d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a7f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 0x7f3a8010de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48742/0 (socket says 172.21.15.32:48742) 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a7f7fe700 1 -- 172.21.15.32:0/3076998509 learned_addr learned my addr 172.21.15.32:0/3076998509 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a7f7fe700 1 -- 172.21.15.32:0/3076998509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a800fbce0 msgr2=0x7f3a8010d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.522+0000 7f3a7f7fe700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a800fbce0 0x7f3a8010d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.523+0000 7f3a7f7fe700 1 -- 172.21.15.32:0/3076998509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a74005040 con 0x7f3a800fcf90 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.523+0000 7f3a7f7fe700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 0x7f3a8010de80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009c90 tx=0x7f3a74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.523+0000 7f3a7cff9700 1 -- 172.21.15.32:0/3076998509 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a7400e030 con 0x7f3a800fcf90 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.523+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8010eeb0 con 0x7f3a800fcf90 2024-04-06T14:16:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.523+0000 7f3a7cff9700 1 -- 172.21.15.32:0/3076998509 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a7401c400 con 0x7f3a800fcf90 2024-04-06T14:16:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.523+0000 7f3a7cff9700 1 -- 172.21.15.32:0/3076998509 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a74012620 con 0x7f3a800fcf90 2024-04-06T14:16:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.523+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8010f3f0 con 0x7f3a800fcf90 2024-04-06T14:16:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.524+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a80101b20 con 0x7f3a800fcf90 2024-04-06T14:16:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.525+0000 7f3a7cff9700 1 -- 172.21.15.32:0/3076998509 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f3a7401c570 con 0x7f3a800fcf90 2024-04-06T14:16:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.525+0000 7f3a7cff9700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f3a6806aec0 0x7f3a6806d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.525+0000 7f3a7cff9700 1 -- 172.21.15.32:0/3076998509 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f3a7408b1a0 con 0x7f3a800fcf90 2024-04-06T14:16:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.526+0000 7f3a7ffff700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f3a6806aec0 0x7f3a6806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.526+0000 7f3a7ffff700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f3a6806aec0 0x7f3a6806d370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009b80 tx=0x7f3a7000b040 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:46.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.530+0000 7f3a7cff9700 1 -- 172.21.15.32:0/3076998509 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3a74059260 con 0x7f3a800fcf90 2024-04-06T14:16:46.538 INFO:tasks.workunit.client.0.smithi032.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-06T14:16:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.830+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a8010f580 con 0x7f3a6806aec0 2024-04-06T14:16:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.831+0000 7f3a7cff9700 1 -- 172.21.15.32:0/3076998509 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f3a8010f580 con 0x7f3a6806aec0 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.833+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f3a6806aec0 msgr2=0x7f3a6806d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.833+0000 7f3a867db700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f3a6806aec0 0x7f3a6806d370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009b80 tx=0x7f3a7000b040 comp rx=0 tx=0).stop 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.833+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 msgr2=0x7f3a8010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.833+0000 7f3a867db700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 0x7f3a8010de80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009c90 tx=0x7f3a74009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.834+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 shutdown_connections 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.834+0000 7f3a867db700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f3a6806aec0 0x7f3a6806d370 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.834+0000 7f3a867db700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a800fbce0 0x7f3a8010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.834+0000 7f3a867db700 1 --2- 172.21.15.32:0/3076998509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a800fcf90 0x7f3a8010de80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.834+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 >> 172.21.15.32:0/3076998509 conn(0x7f3a800f7480 msgr2=0x7f3a80066480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.834+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 shutdown_connections 2024-04-06T14:16:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.834+0000 7f3a867db700 1 -- 172.21.15.32:0/3076998509 wait complete. 2024-04-06T14:16:46.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.948+0000 7fc25c7af700 1 -- 172.21.15.32:0/1982379191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fe430 msgr2=0x7fc2540fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.948+0000 7fc25c7af700 1 --2- 172.21.15.32:0/1982379191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fe430 0x7fc2540fe840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc250004940 tx=0x7fc250009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:46.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.948+0000 7fc25c7af700 1 -- 172.21.15.32:0/1982379191 shutdown_connections 2024-04-06T14:16:46.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.948+0000 7fc25c7af700 1 --2- 172.21.15.32:0/1982379191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2540fed80 0x7fc2540faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.948+0000 7fc25c7af700 1 --2- 172.21.15.32:0/1982379191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fe430 0x7fc2540fe840 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.948+0000 7fc25c7af700 1 -- 172.21.15.32:0/1982379191 >> 172.21.15.32:0/1982379191 conn(0x7fc2540f6bd0 msgr2=0x7fc2540f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:46.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.948+0000 7fc25c7af700 1 -- 172.21.15.32:0/1982379191 shutdown_connections 2024-04-06T14:16:46.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.948+0000 7fc25c7af700 1 -- 172.21.15.32:0/1982379191 wait complete. 2024-04-06T14:16:46.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.948+0000 7fc25c7af700 1 Processor -- start 2024-04-06T14:16:46.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc25c7af700 1 -- start start 2024-04-06T14:16:46.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc25c7af700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2540fe430 0x7fc25410d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:46.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc25c7af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fed80 0x7fc25410dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:46.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc25c7af700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25410e2d0 con 0x7fc2540fed80 2024-04-06T14:16:46.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc25c7af700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25410e440 con 0x7fc2540fe430 2024-04-06T14:16:46.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc259d4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fed80 0x7fc25410dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:46.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc259d4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fed80 0x7fc25410dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48756/0 (socket says 172.21.15.32:48756) 2024-04-06T14:16:46.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc259d4a700 1 -- 172.21.15.32:0/1360859602 learned_addr learned my addr 172.21.15.32:0/1360859602 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc259d4a700 1 -- 172.21.15.32:0/1360859602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2540fe430 msgr2=0x7fc25410d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc259d4a700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2540fe430 0x7fc25410d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.949+0000 7fc259d4a700 1 -- 172.21.15.32:0/1360859602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc250005040 con 0x7fc2540fed80 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.950+0000 7fc259d4a700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fed80 0x7fc25410dcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc24400eb40 tx=0x7fc24400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.950+0000 7fc24affd700 1 -- 172.21.15.32:0/1360859602 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc244007cc0 con 0x7fc2540fed80 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.950+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc254112e40 con 0x7fc2540fed80 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.950+0000 7fc24affd700 1 -- 172.21.15.32:0/1360859602 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc24400f040 con 0x7fc2540fed80 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.950+0000 7fc24affd700 1 -- 172.21.15.32:0/1360859602 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc244009570 con 0x7fc2540fed80 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.950+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2541133d0 con 0x7fc2540fed80 2024-04-06T14:16:46.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.951+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25404c710 con 0x7fc2540fed80 2024-04-06T14:16:46.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.953+0000 7fc24affd700 1 -- 172.21.15.32:0/1360859602 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7fc2440096d0 con 0x7fc2540fed80 2024-04-06T14:16:46.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.953+0000 7fc24affd700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fc24006aed0 0x7fc24006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:46.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.953+0000 7fc24affd700 1 -- 172.21.15.32:0/1360859602 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fc244089740 con 0x7fc2540fed80 2024-04-06T14:16:46.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.953+0000 7fc25a54b700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fc24006aed0 0x7fc24006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:46.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.954+0000 7fc25a54b700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fc24006aed0 0x7fc24006d380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc2500137b0 tx=0x7fc250009e30 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:46.958+0000 7fc24affd700 1 -- 172.21.15.32:0/1360859602 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc2440576d0 con 0x7fc2540fed80 2024-04-06T14:16:46.974 INFO:tasks.workunit.client.0.smithi032.stderr:++ readlink -f fsstress 2024-04-06T14:16:46.976 INFO:tasks.workunit.client.0.smithi032.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-06T14:16:46.976 INFO:tasks.workunit.client.0.smithi032.stderr:+ popd 2024-04-06T14:16:46.977 INFO:tasks.workunit.client.0.smithi032.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-06T14:16:46.977 INFO:tasks.workunit.client.0.smithi032.stderr:+ popd 2024-04-06T14:16:46.978 INFO:tasks.workunit.client.0.smithi032.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-06T14:16:46.978 INFO:tasks.workunit.client.0.smithi032.stderr:++ mktemp -d -p . 2024-04-06T14:16:46.981 INFO:tasks.workunit.client.0.smithi032.stderr:+ T=./tmp.mbqFNLcCvl 2024-04-06T14:16:46.981 INFO:tasks.workunit.client.0.smithi032.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.mbqFNLcCvl -l 1 -n 1000 -p 10 -v 2024-04-06T14:16:46.984 INFO:tasks.workunit.client.0.smithi032.stdout:seed = 1713263107 2024-04-06T14:16:47.049 INFO:tasks.workunit.client.0.smithi032.stdout:1/0: dread - no filename 2024-04-06T14:16:47.049 INFO:tasks.workunit.client.0.smithi032.stdout:1/1: dwrite - no filename 2024-04-06T14:16:47.049 INFO:tasks.workunit.client.0.smithi032.stdout:1/2: dread - no filename 2024-04-06T14:16:47.050 INFO:tasks.workunit.client.0.smithi032.stdout:0/0: creat f0 x:0 0 0 2024-04-06T14:16:47.050 INFO:tasks.workunit.client.0.smithi032.stdout:0/1: chown f0 5683276 1 2024-04-06T14:16:47.050 INFO:tasks.workunit.client.0.smithi032.stdout:0/2: fdatasync f0 0 2024-04-06T14:16:47.051 INFO:tasks.workunit.client.0.smithi032.stdout:1/3: mkdir d0 0 2024-04-06T14:16:47.052 INFO:tasks.workunit.client.0.smithi032.stdout:0/3: write f0 [948687,44084] 0 2024-04-06T14:16:47.054 INFO:tasks.workunit.client.0.smithi032.stdout:2/0: mkdir d0 0 2024-04-06T14:16:47.054 INFO:tasks.workunit.client.0.smithi032.stdout:4/0: dwrite - no filename 2024-04-06T14:16:47.054 INFO:tasks.workunit.client.0.smithi032.stdout:1/4: mkdir d0/d1 0 2024-04-06T14:16:47.055 INFO:tasks.workunit.client.0.smithi032.stdout:0/4: symlink l1 0 2024-04-06T14:16:47.055 INFO:tasks.workunit.client.0.smithi032.stdout:0/5: stat l1 0 2024-04-06T14:16:47.055 INFO:tasks.workunit.client.0.smithi032.stdout:0/6: fdatasync f0 0 2024-04-06T14:16:47.056 INFO:tasks.workunit.client.0.smithi032.stdout:3/0: symlink l0 0 2024-04-06T14:16:47.056 INFO:tasks.workunit.client.0.smithi032.stdout:3/1: write - no filename 2024-04-06T14:16:47.056 INFO:tasks.workunit.client.0.smithi032.stdout:5/0: write - no filename 2024-04-06T14:16:47.057 INFO:tasks.workunit.client.0.smithi032.stdout:5/1: write - no filename 2024-04-06T14:16:47.057 INFO:tasks.workunit.client.0.smithi032.stdout:5/2: link - no file 2024-04-06T14:16:47.057 INFO:tasks.workunit.client.0.smithi032.stdout:5/3: dread - no filename 2024-04-06T14:16:47.057 INFO:tasks.workunit.client.0.smithi032.stdout:5/4: read - no filename 2024-04-06T14:16:47.057 INFO:tasks.workunit.client.0.smithi032.stdout:5/5: write - no filename 2024-04-06T14:16:47.057 INFO:tasks.workunit.client.0.smithi032.stdout:0/7: write f0 [1569724,19150] 0 2024-04-06T14:16:47.057 INFO:tasks.workunit.client.0.smithi032.stdout:1/5: creat d0/f2 x:0 0 0 2024-04-06T14:16:47.058 INFO:tasks.workunit.client.0.smithi032.stdout:4/1: mkdir d0 0 2024-04-06T14:16:47.058 INFO:tasks.workunit.client.0.smithi032.stdout:4/2: truncate - no filename 2024-04-06T14:16:47.058 INFO:tasks.workunit.client.0.smithi032.stdout:4/3: stat d0 0 2024-04-06T14:16:47.058 INFO:tasks.workunit.client.0.smithi032.stdout:4/4: dread - no filename 2024-04-06T14:16:47.059 INFO:tasks.workunit.client.0.smithi032.stdout:2/1: creat d0/f1 x:0 0 0 2024-04-06T14:16:47.060 INFO:tasks.workunit.client.0.smithi032.stdout:6/0: unlink - no file 2024-04-06T14:16:47.060 INFO:tasks.workunit.client.0.smithi032.stdout:6/1: rename - no filename 2024-04-06T14:16:47.060 INFO:tasks.workunit.client.0.smithi032.stdout:6/2: dread - no filename 2024-04-06T14:16:47.060 INFO:tasks.workunit.client.0.smithi032.stdout:6/3: chown . 103 1 2024-04-06T14:16:47.060 INFO:tasks.workunit.client.0.smithi032.stdout:6/4: rmdir - no directory 2024-04-06T14:16:47.060 INFO:tasks.workunit.client.0.smithi032.stdout:6/5: unlink - no file 2024-04-06T14:16:47.060 INFO:tasks.workunit.client.0.smithi032.stdout:3/2: rename l0 to l1 0 2024-04-06T14:16:47.060 INFO:tasks.workunit.client.0.smithi032.stdout:3/3: dread - no filename 2024-04-06T14:16:47.060 INFO:tasks.workunit.client.0.smithi032.stdout:3/4: dread - no filename 2024-04-06T14:16:47.061 INFO:tasks.workunit.client.0.smithi032.stdout:3/5: dwrite - no filename 2024-04-06T14:16:47.061 INFO:tasks.workunit.client.0.smithi032.stdout:0/8: mknod c2 0 2024-04-06T14:16:47.062 INFO:tasks.workunit.client.0.smithi032.stdout:2/2: creat d0/f2 x:0 0 0 2024-04-06T14:16:47.062 INFO:tasks.workunit.client.0.smithi032.stdout:5/6: getdents . 0 2024-04-06T14:16:47.064 INFO:tasks.workunit.client.0.smithi032.stdout:0/9: mkdir d3 0 2024-04-06T14:16:47.065 INFO:tasks.workunit.client.0.smithi032.stdout:0/10: write f0 [2249545,32446] 0 2024-04-06T14:16:47.066 INFO:tasks.workunit.client.0.smithi032.stdout:2/3: symlink d0/l3 0 2024-04-06T14:16:47.066 INFO:tasks.workunit.client.0.smithi032.stdout:4/5: creat d0/f1 x:0 0 0 2024-04-06T14:16:47.066 INFO:tasks.workunit.client.0.smithi032.stdout:6/6: mknod c0 0 2024-04-06T14:16:47.066 INFO:tasks.workunit.client.0.smithi032.stdout:6/7: truncate - no filename 2024-04-06T14:16:47.066 INFO:tasks.workunit.client.0.smithi032.stdout:6/8: dwrite - no filename 2024-04-06T14:16:47.067 INFO:tasks.workunit.client.0.smithi032.stdout:6/9: dread - no filename 2024-04-06T14:16:47.068 INFO:tasks.workunit.client.0.smithi032.stdout:1/6: rmdir d0/d1 0 2024-04-06T14:16:47.068 INFO:tasks.workunit.client.0.smithi032.stdout:1/7: readlink - no filename 2024-04-06T14:16:47.068 INFO:tasks.workunit.client.0.smithi032.stdout:2/4: truncate d0/f1 959915 0 2024-04-06T14:16:47.069 INFO:tasks.workunit.client.0.smithi032.stdout:8/0: fdatasync - no filename 2024-04-06T14:16:47.069 INFO:tasks.workunit.client.0.smithi032.stdout:8/1: dwrite - no filename 2024-04-06T14:16:47.069 INFO:tasks.workunit.client.0.smithi032.stdout:8/2: rename - no filename 2024-04-06T14:16:47.069 INFO:tasks.workunit.client.0.smithi032.stdout:8/3: chown . 6 1 2024-04-06T14:16:47.070 INFO:tasks.workunit.client.0.smithi032.stdout:5/7: creat f0 x:0 0 0 2024-04-06T14:16:47.072 INFO:tasks.workunit.client.0.smithi032.stdout:6/10: mknod c1 0 2024-04-06T14:16:47.072 INFO:tasks.workunit.client.0.smithi032.stdout:7/0: symlink l0 0 2024-04-06T14:16:47.072 INFO:tasks.workunit.client.0.smithi032.stdout:7/1: write - no filename 2024-04-06T14:16:47.073 INFO:tasks.workunit.client.0.smithi032.stdout:4/6: symlink d0/l2 0 2024-04-06T14:16:47.075 INFO:tasks.workunit.client.0.smithi032.stdout:1/8: mknod d0/c3 0 2024-04-06T14:16:47.076 INFO:tasks.workunit.client.0.smithi032.stdout:9/0: stat - no entries 2024-04-06T14:16:47.076 INFO:tasks.workunit.client.0.smithi032.stdout:9/1: stat - no entries 2024-04-06T14:16:47.076 INFO:tasks.workunit.client.0.smithi032.stdout:4/7: write d0/f1 [142217,124658] 0 2024-04-06T14:16:47.076 INFO:tasks.workunit.client.0.smithi032.stdout:9/2: chown . 110907 1 2024-04-06T14:16:47.076 INFO:tasks.workunit.client.0.smithi032.stdout:9/3: dread - no filename 2024-04-06T14:16:47.076 INFO:tasks.workunit.client.0.smithi032.stdout:9/4: dwrite - no filename 2024-04-06T14:16:47.077 INFO:tasks.workunit.client.0.smithi032.stdout:2/5: rename d0/l3 to d0/l4 0 2024-04-06T14:16:47.077 INFO:tasks.workunit.client.0.smithi032.stdout:9/5: chown . 954 1 2024-04-06T14:16:47.077 INFO:tasks.workunit.client.0.smithi032.stdout:9/6: dread - no filename 2024-04-06T14:16:47.078 INFO:tasks.workunit.client.0.smithi032.stdout:8/4: mknod c0 0 2024-04-06T14:16:47.078 INFO:tasks.workunit.client.0.smithi032.stdout:5/8: creat f1 x:0 0 0 2024-04-06T14:16:47.078 INFO:tasks.workunit.client.0.smithi032.stdout:9/7: chown . 1 1 2024-04-06T14:16:47.078 INFO:tasks.workunit.client.0.smithi032.stdout:9/8: readlink - no filename 2024-04-06T14:16:47.079 INFO:tasks.workunit.client.0.smithi032.stdout:0/11: dwrite f0 [0,4194304] 0 2024-04-06T14:16:47.080 INFO:tasks.workunit.client.0.smithi032.stdout:2/6: read d0/f1 [20977,127663] 0 2024-04-06T14:16:47.081 INFO:tasks.workunit.client.0.smithi032.stdout:8/5: creat f1 x:0 0 0 2024-04-06T14:16:47.081 INFO:tasks.workunit.client.0.smithi032.stdout:4/8: creat d0/f3 x:0 0 0 2024-04-06T14:16:47.081 INFO:tasks.workunit.client.0.smithi032.stdout:5/9: truncate f0 898112 0 2024-04-06T14:16:47.082 INFO:tasks.workunit.client.0.smithi032.stdout:1/9: rename d0/c3 to d0/c4 0 2024-04-06T14:16:47.083 INFO:tasks.workunit.client.0.smithi032.stdout:2/7: truncate d0/f2 559096 0 2024-04-06T14:16:47.086 INFO:tasks.workunit.client.0.smithi032.stdout:9/9: symlink l0 0 2024-04-06T14:16:47.087 INFO:tasks.workunit.client.0.smithi032.stdout:1/10: mknod d0/c5 0 2024-04-06T14:16:47.088 INFO:tasks.workunit.client.0.smithi032.stdout:9/10: stat l0 0 2024-04-06T14:16:47.089 INFO:tasks.workunit.client.0.smithi032.stdout:4/9: symlink d0/l4 0 2024-04-06T14:16:47.091 INFO:tasks.workunit.client.0.smithi032.stdout:4/10: chown d0 1373203 1 2024-04-06T14:16:47.091 INFO:tasks.workunit.client.0.smithi032.stdout:9/11: mkdir d1 0 2024-04-06T14:16:47.091 INFO:tasks.workunit.client.0.smithi032.stdout:9/12: write - no filename 2024-04-06T14:16:47.091 INFO:tasks.workunit.client.0.smithi032.stdout:9/13: write - no filename 2024-04-06T14:16:47.091 INFO:tasks.workunit.client.0.smithi032.stdout:9/14: dwrite - no filename 2024-04-06T14:16:47.097 INFO:tasks.workunit.client.0.smithi032.stdout:1/11: write d0/f2 [263605,82230] 0 2024-04-06T14:16:47.097 INFO:tasks.workunit.client.0.smithi032.stdout:9/15: chown d1 2043 1 2024-04-06T14:16:47.097 INFO:tasks.workunit.client.0.smithi032.stdout:9/16: write - no filename 2024-04-06T14:16:47.097 INFO:tasks.workunit.client.0.smithi032.stdout:0/12: dwrite f0 [0,4194304] 0 2024-04-06T14:16:47.097 INFO:tasks.workunit.client.0.smithi032.stdout:4/11: write d0/f3 [531311,54877] 0 2024-04-06T14:16:47.101 INFO:tasks.workunit.client.0.smithi032.stdout:1/12: symlink d0/l6 0 2024-04-06T14:16:47.101 INFO:tasks.workunit.client.0.smithi032.stdout:8/6: dwrite f1 [0,4194304] 0 2024-04-06T14:16:47.103 INFO:tasks.workunit.client.0.smithi032.stdout:4/12: symlink d0/l5 0 2024-04-06T14:16:47.106 INFO:tasks.workunit.client.0.smithi032.stdout:8/7: symlink l2 0 2024-04-06T14:16:47.106 INFO:tasks.workunit.client.0.smithi032.stdout:8/8: chown c0 708542960 1 2024-04-06T14:16:47.107 INFO:tasks.workunit.client.0.smithi032.stdout:9/17: rmdir d1 0 2024-04-06T14:16:47.110 INFO:tasks.workunit.client.0.smithi032.stdout:8/9: mkdir d3 0 2024-04-06T14:16:47.111 INFO:tasks.workunit.client.0.smithi032.stdout:9/18: mkdir d2 0 2024-04-06T14:16:47.111 INFO:tasks.workunit.client.0.smithi032.stdout:9/19: dread - no filename 2024-04-06T14:16:47.111 INFO:tasks.workunit.client.0.smithi032.stdout:0/13: dwrite f0 [0,4194304] 0 2024-04-06T14:16:47.111 INFO:tasks.workunit.client.0.smithi032.stdout:8/10: read f1 [3913734,68568] 0 2024-04-06T14:16:47.113 INFO:tasks.workunit.client.0.smithi032.stdout:9/20: stat d2 0 2024-04-06T14:16:47.113 INFO:tasks.workunit.client.0.smithi032.stdout:9/21: write - no filename 2024-04-06T14:16:47.113 INFO:tasks.workunit.client.0.smithi032.stdout:9/22: truncate - no filename 2024-04-06T14:16:47.113 INFO:tasks.workunit.client.0.smithi032.stdout:9/23: write - no filename 2024-04-06T14:16:47.113 INFO:tasks.workunit.client.0.smithi032.stdout:9/24: dwrite - no filename 2024-04-06T14:16:47.115 INFO:tasks.workunit.client.0.smithi032.stdout:4/13: link d0/l4 d0/l6 0 2024-04-06T14:16:47.123 INFO:tasks.workunit.client.0.smithi032.stdout:4/14: rename d0/l6 to d0/l7 0 2024-04-06T14:16:47.123 INFO:tasks.workunit.client.0.smithi032.stdout:9/25: link l0 d2/l3 0 2024-04-06T14:16:47.124 INFO:tasks.workunit.client.0.smithi032.stdout:8/11: getdents d3 0 2024-04-06T14:16:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:16:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:47.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:47.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:16:47.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:46 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:16:47.129 INFO:tasks.workunit.client.0.smithi032.stdout:8/12: unlink f1 0 2024-04-06T14:16:47.129 INFO:tasks.workunit.client.0.smithi032.stdout:8/13: dread - no filename 2024-04-06T14:16:47.129 INFO:tasks.workunit.client.0.smithi032.stdout:8/14: dwrite - no filename 2024-04-06T14:16:47.129 INFO:tasks.workunit.client.0.smithi032.stdout:8/15: dread - no filename 2024-04-06T14:16:47.130 INFO:tasks.workunit.client.0.smithi032.stdout:9/26: creat d2/f4 x:0 0 0 2024-04-06T14:16:47.131 INFO:tasks.workunit.client.0.smithi032.stdout:8/16: mkdir d3/d4 0 2024-04-06T14:16:47.133 INFO:tasks.workunit.client.0.smithi032.stdout:8/17: symlink d3/l5 0 2024-04-06T14:16:47.133 INFO:tasks.workunit.client.0.smithi032.stdout:8/18: write - no filename 2024-04-06T14:16:47.135 INFO:tasks.workunit.client.0.smithi032.stdout:8/19: symlink d3/l6 0 2024-04-06T14:16:47.138 INFO:tasks.workunit.client.0.smithi032.stdout:9/27: dwrite d2/f4 [0,4194304] 0 2024-04-06T14:16:47.138 INFO:tasks.workunit.client.0.smithi032.stdout:8/20: mkdir d3/d4/d7 0 2024-04-06T14:16:47.138 INFO:tasks.workunit.client.0.smithi032.stdout:8/21: dwrite - no filename 2024-04-06T14:16:47.138 INFO:tasks.workunit.client.0.smithi032.stdout:8/22: fdatasync - no filename 2024-04-06T14:16:47.140 INFO:tasks.workunit.client.0.smithi032.stdout:9/28: mkdir d2/d5 0 2024-04-06T14:16:47.141 INFO:tasks.workunit.client.0.smithi032.stdout:8/23: rmdir d3/d4 39 2024-04-06T14:16:47.144 INFO:tasks.workunit.client.0.smithi032.stdout:9/29: creat d2/d5/f6 x:0 0 0 2024-04-06T14:16:47.148 INFO:tasks.workunit.client.0.smithi032.stdout:9/30: rename d2/l3 to d2/l7 0 2024-04-06T14:16:47.149 INFO:tasks.workunit.client.0.smithi032.stdout:9/31: truncate d2/d5/f6 996647 0 2024-04-06T14:16:47.153 INFO:tasks.workunit.client.0.smithi032.stdout:9/32: dread d2/f4 [0,4194304] 0 2024-04-06T14:16:47.155 INFO:tasks.workunit.client.0.smithi032.stdout:9/33: creat d2/f8 x:0 0 0 2024-04-06T14:16:47.157 INFO:tasks.workunit.client.0.smithi032.stdout:9/34: creat d2/d5/f9 x:0 0 0 2024-04-06T14:16:47.159 INFO:tasks.workunit.client.0.smithi032.stdout:9/35: creat d2/d5/fa x:0 0 0 2024-04-06T14:16:47.160 INFO:tasks.workunit.client.0.smithi032.stdout:9/36: symlink d2/lb 0 2024-04-06T14:16:47.168 INFO:tasks.workunit.client.0.smithi032.stdout:9/37: dwrite d2/f8 [0,4194304] 0 2024-04-06T14:16:47.170 INFO:tasks.workunit.client.0.smithi032.stdout:9/38: mknod d2/cc 0 2024-04-06T14:16:47.182 INFO:tasks.workunit.client.0.smithi032.stdout:9/39: dwrite d2/d5/fa [0,4194304] 0 2024-04-06T14:16:47.203 INFO:tasks.workunit.client.0.smithi032.stdout:4/15: fsync d0/f1 0 2024-04-06T14:16:47.205 INFO:tasks.workunit.client.0.smithi032.stdout:3/6: rename l1 to l2 0 2024-04-06T14:16:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:16:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T14:16:47.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:47.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:16:47.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:46 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:16:47.238 INFO:tasks.workunit.client.0.smithi032.stdout:5/10: getdents . 0 2024-04-06T14:16:47.238 INFO:tasks.workunit.client.0.smithi032.stdout:4/16: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:47.246 INFO:tasks.workunit.client.0.smithi032.stdout:0/14: dwrite f0 [4194304,4194304] 0 2024-04-06T14:16:47.257 INFO:tasks.workunit.client.0.smithi032.stdout:9/40: fsync d2/d5/f9 0 2024-04-06T14:16:47.257 INFO:tasks.workunit.client.0.smithi032.stdout:4/17: dwrite d0/f1 [0,4194304] 0 2024-04-06T14:16:47.258 INFO:tasks.workunit.client.0.smithi032.stdout:9/41: fsync d2/d5/f9 0 2024-04-06T14:16:47.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.266+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc254102500 con 0x7fc24006aed0 2024-04-06T14:16:47.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.281+0000 7fc24affd700 1 -- 172.21.15.32:0/1360859602 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fc254102500 con 0x7fc24006aed0 2024-04-06T14:16:47.281 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:16:47.281 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (7m) 8s ago 8m 26.8M - 0.25.0 c8568f914cd2 d6f57d3d10f9 2024-04-06T14:16:47.281 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 8s ago 8m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (7m) 10s ago 7m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 8s ago 8m 86.7M - 9.4.7 954c08fa6188 1cc1d256b791 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (2m) 8s ago 2m 11.4M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (2m) 8s ago 2m 108M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (2m) 10s ago 2m 108M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (2m) 10s ago 2m 14.2M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:9283 running (9m) 8s ago 9m 192M - 17.2.7-819-g6699495c ec300eaa768b 7737c4037eba 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283 running (29s) 10s ago 7m 486M - 18.2.2-1086-g052adcad 87261d5ece5a f68af56edf8d 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 8s ago 9m 47.3M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (7m) 10s ago 7m 38.6M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 8s ago 8m 16.4M - 1.5.0 0da6a335fe13 41a44c41cd57 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (7m) 10s ago 7m 18.9M - 1.5.0 0da6a335fe13 21ac74498650 2024-04-06T14:16:47.282 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 8s ago 6m 76.0M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:16:47.283 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 8s ago 6m 83.0M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:16:47.283 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 8s ago 5m 76.6M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:16:47.283 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (5m) 10s ago 5m 90.2M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:16:47.283 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (4m) 10s ago 4m 83.9M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:16:47.283 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (4m) 10s ago 4m 75.4M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:16:47.283 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 8s ago 7m 57.4M - 2.43.0 a07b618ecd1d 6742cb6c38a9 2024-04-06T14:16:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fc24006aed0 msgr2=0x7fc24006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fc24006aed0 0x7fc24006d380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc2500137b0 tx=0x7fc250009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fed80 msgr2=0x7fc25410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fed80 0x7fc25410dcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc24400eb40 tx=0x7fc24400b040 comp rx=0 tx=0).stop 2024-04-06T14:16:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 shutdown_connections 2024-04-06T14:16:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fc24006aed0 0x7fc24006d380 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2540fe430 0x7fc25410d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 --2- 172.21.15.32:0/1360859602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2540fed80 0x7fc25410dcd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 >> 172.21.15.32:0/1360859602 conn(0x7fc2540f6bd0 msgr2=0x7fc2540f82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:47.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 shutdown_connections 2024-04-06T14:16:47.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.285+0000 7fc25c7af700 1 -- 172.21.15.32:0/1360859602 wait complete. 2024-04-06T14:16:47.389 INFO:tasks.workunit.client.0.smithi032.stdout:3/7: mkdir d3 0 2024-04-06T14:16:47.390 INFO:tasks.workunit.client.0.smithi032.stdout:3/8: chown d3 5837189 1 2024-04-06T14:16:47.390 INFO:tasks.workunit.client.0.smithi032.stdout:2/8: rename d0/l4 to d0/l5 0 2024-04-06T14:16:47.391 INFO:tasks.workunit.client.0.smithi032.stdout:1/13: rename d0/c4 to d0/c7 0 2024-04-06T14:16:47.391 INFO:tasks.workunit.client.0.smithi032.stdout:2/9: chown d0/f1 1691 1 2024-04-06T14:16:47.392 INFO:tasks.workunit.client.0.smithi032.stdout:1/14: write d0/f2 [1387442,102939] 0 2024-04-06T14:16:47.400 INFO:tasks.workunit.client.0.smithi032.stdout:1/15: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:47.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.413+0000 7f07bccee700 1 -- 172.21.15.32:0/399269149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b80fbcd0 msgr2=0x7f07b80fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:47.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.413+0000 7f07bccee700 1 --2- 172.21.15.32:0/399269149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b80fbcd0 0x7f07b80fc0e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f07a8004940 tx=0x7f07a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:47.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.413+0000 7f07bccee700 1 -- 172.21.15.32:0/399269149 shutdown_connections 2024-04-06T14:16:47.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.413+0000 7f07bccee700 1 --2- 172.21.15.32:0/399269149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b80fcf80 0x7f07b80fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.413+0000 7f07bccee700 1 --2- 172.21.15.32:0/399269149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b80fbcd0 0x7f07b80fc0e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.413+0000 7f07bccee700 1 -- 172.21.15.32:0/399269149 >> 172.21.15.32:0/399269149 conn(0x7f07b80f7430 msgr2=0x7f07b80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:47.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.415+0000 7f07bccee700 1 -- 172.21.15.32:0/399269149 shutdown_connections 2024-04-06T14:16:47.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.415+0000 7f07bccee700 1 -- 172.21.15.32:0/399269149 wait complete. 2024-04-06T14:16:47.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.415+0000 7f07bccee700 1 Processor -- start 2024-04-06T14:16:47.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07bccee700 1 -- start start 2024-04-06T14:16:47.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07bccee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b80fbcd0 0x7f07b810db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:47.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07bccee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810e070 0x7f07b810eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:47.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07bccee700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b810e5c0 con 0x7f07b810e070 2024-04-06T14:16:47.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07bccee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b810e700 con 0x7f07b80fbcd0 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07b5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810e070 0x7f07b810eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07b659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b80fbcd0 0x7f07b810db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07b5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810e070 0x7f07b810eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48780/0 (socket says 172.21.15.32:48780) 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07b5d9b700 1 -- 172.21.15.32:0/2517352994 learned_addr learned my addr 172.21.15.32:0/2517352994 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07b659c700 1 -- 172.21.15.32:0/2517352994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810e070 msgr2=0x7f07b810eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07b659c700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810e070 0x7f07b810eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07b659c700 1 -- 172.21.15.32:0/2517352994 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a8005040 con 0x7f07b80fbcd0 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.416+0000 7f07b5d9b700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810e070 0x7f07b810eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.417+0000 7f07b659c700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b80fbcd0 0x7f07b810db30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f07a800b900 tx=0x7f07a8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.417+0000 7f07a6ffd700 1 -- 172.21.15.32:0/2517352994 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a800e030 con 0x7f07b80fbcd0 2024-04-06T14:16:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.417+0000 7f07a6ffd700 1 -- 172.21.15.32:0/2517352994 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07a80093b0 con 0x7f07b80fbcd0 2024-04-06T14:16:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.417+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b810f540 con 0x7f07b80fbcd0 2024-04-06T14:16:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.417+0000 7f07a6ffd700 1 -- 172.21.15.32:0/2517352994 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a8013e90 con 0x7f07b80fbcd0 2024-04-06T14:16:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.417+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b810fb00 con 0x7f07b80fbcd0 2024-04-06T14:16:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.418+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b8101b10 con 0x7f07b80fbcd0 2024-04-06T14:16:47.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.419+0000 7f07a6ffd700 1 -- 172.21.15.32:0/2517352994 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f07a800bb80 con 0x7f07b80fbcd0 2024-04-06T14:16:47.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.419+0000 7f07a6ffd700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f07a006aec0 0x7f07a006d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:47.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.419+0000 7f07a6ffd700 1 -- 172.21.15.32:0/2517352994 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f07a808b080 con 0x7f07b80fbcd0 2024-04-06T14:16:47.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.419+0000 7f07b5d9b700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f07a006aec0 0x7f07a006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:47.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.420+0000 7f07b5d9b700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f07a006aec0 0x7f07a006d370 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009ed0 tx=0x7f07ac0091a0 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:47.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.425+0000 7f07a6ffd700 1 -- 172.21.15.32:0/2517352994 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f07a8059020 con 0x7f07b80fbcd0 2024-04-06T14:16:47.463 INFO:tasks.workunit.client.0.smithi032.stdout:0/15: creat d3/f4 x:0 0 0 2024-04-06T14:16:47.463 INFO:tasks.workunit.client.0.smithi032.stdout:0/16: write f0 [7321797,47928] 0 2024-04-06T14:16:47.469 INFO:tasks.workunit.client.0.smithi032.stdout:1/16: unlink d0/c5 0 2024-04-06T14:16:47.471 INFO:tasks.workunit.client.0.smithi032.stdout:9/42: rename d2/d5/fa to d2/fd 0 2024-04-06T14:16:47.472 INFO:tasks.workunit.client.0.smithi032.stdout:2/10: rename d0 to d0/d6 22 2024-04-06T14:16:47.473 INFO:tasks.workunit.client.0.smithi032.stdout:3/9: creat d3/f4 x:0 0 0 2024-04-06T14:16:47.473 INFO:tasks.workunit.client.0.smithi032.stdout:3/10: write d3/f4 [118801,20080] 0 2024-04-06T14:16:47.476 INFO:tasks.workunit.client.0.smithi032.stdout:8/24: rename d3/d4 to d3/d8 0 2024-04-06T14:16:47.477 INFO:tasks.workunit.client.0.smithi032.stdout:0/17: creat d3/f5 x:0 0 0 2024-04-06T14:16:47.542 INFO:tasks.workunit.client.0.smithi032.stdout:9/43: readlink l0 0 2024-04-06T14:16:47.542 INFO:tasks.workunit.client.0.smithi032.stdout:3/11: dread d3/f4 [0,4194304] 0 2024-04-06T14:16:47.549 INFO:tasks.workunit.client.0.smithi032.stdout:1/17: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:16:47.562 INFO:tasks.workunit.client.0.smithi032.stdout:9/44: unlink d2/l7 0 2024-04-06T14:16:47.564 INFO:tasks.workunit.client.0.smithi032.stdout:9/45: fdatasync d2/f8 0 2024-04-06T14:16:47.564 INFO:tasks.workunit.client.0.smithi032.stdout:3/12: write d3/f4 [414391,42678] 0 2024-04-06T14:16:47.565 INFO:tasks.workunit.client.0.smithi032.stdout:1/18: rmdir d0 39 2024-04-06T14:16:47.566 INFO:tasks.workunit.client.0.smithi032.stdout:0/18: truncate f0 5394229 0 2024-04-06T14:16:47.567 INFO:tasks.workunit.client.0.smithi032.stdout:8/25: creat d3/f9 x:0 0 0 2024-04-06T14:16:47.568 INFO:tasks.workunit.client.0.smithi032.stdout:3/13: creat d3/f5 x:0 0 0 2024-04-06T14:16:47.569 INFO:tasks.workunit.client.0.smithi032.stdout:8/26: chown d3/d8 1028 1 2024-04-06T14:16:47.572 INFO:tasks.workunit.client.0.smithi032.stdout:4/18: dwrite d0/f1 [4194304,4194304] 0 2024-04-06T14:16:47.572 INFO:tasks.workunit.client.0.smithi032.stdout:9/46: write d2/f4 [1841086,124848] 0 2024-04-06T14:16:47.572 INFO:tasks.workunit.client.0.smithi032.stdout:1/19: read d0/f2 [4121232,125544] 0 2024-04-06T14:16:47.573 INFO:tasks.workunit.client.0.smithi032.stdout:3/14: mknod d3/c6 0 2024-04-06T14:16:47.573 INFO:tasks.workunit.client.0.smithi032.stdout:8/27: dread - d3/f9 zero size 2024-04-06T14:16:47.574 INFO:tasks.workunit.client.0.smithi032.stdout:3/15: chown d3 5341 1 2024-04-06T14:16:47.575 INFO:tasks.workunit.client.0.smithi032.stdout:0/19: link f0 d3/f6 0 2024-04-06T14:16:47.576 INFO:tasks.workunit.client.0.smithi032.stdout:8/28: dread - d3/f9 zero size 2024-04-06T14:16:47.577 INFO:tasks.workunit.client.0.smithi032.stdout:1/20: write d0/f2 [3516450,18665] 0 2024-04-06T14:16:47.578 INFO:tasks.workunit.client.0.smithi032.stdout:4/19: rename d0/l5 to d0/l8 0 2024-04-06T14:16:47.578 INFO:tasks.workunit.client.0.smithi032.stdout:8/29: rename d3/d8/d7 to d3/d8/d7/da 22 2024-04-06T14:16:47.578 INFO:tasks.workunit.client.0.smithi032.stdout:8/30: chown d3 54865843 1 2024-04-06T14:16:47.579 INFO:tasks.workunit.client.0.smithi032.stdout:9/47: symlink d2/le 0 2024-04-06T14:16:47.580 INFO:tasks.workunit.client.0.smithi032.stdout:0/20: truncate d3/f5 844907 0 2024-04-06T14:16:47.580 INFO:tasks.workunit.client.0.smithi032.stdout:3/16: symlink d3/l7 0 2024-04-06T14:16:47.581 INFO:tasks.workunit.client.0.smithi032.stdout:4/20: write d0/f1 [1141351,87010] 0 2024-04-06T14:16:47.582 INFO:tasks.workunit.client.0.smithi032.stdout:1/21: chown d0/c7 6778156 1 2024-04-06T14:16:47.582 INFO:tasks.workunit.client.0.smithi032.stdout:8/31: rmdir d3/d8 39 2024-04-06T14:16:47.585 INFO:tasks.workunit.client.0.smithi032.stdout:0/21: write d3/f5 [683574,51426] 0 2024-04-06T14:16:47.587 INFO:tasks.workunit.client.0.smithi032.stdout:1/22: write d0/f2 [1398182,76083] 0 2024-04-06T14:16:47.588 INFO:tasks.workunit.client.0.smithi032.stdout:9/48: rename d2/d5/f9 to d2/ff 0 2024-04-06T14:16:47.594 INFO:tasks.workunit.client.0.smithi032.stdout:8/32: mkdir d3/db 0 2024-04-06T14:16:47.598 INFO:tasks.workunit.client.0.smithi032.stdout:0/22: dread d3/f6 [0,4194304] 0 2024-04-06T14:16:47.599 INFO:tasks.workunit.client.0.smithi032.stdout:3/17: dwrite d3/f4 [0,4194304] 0 2024-04-06T14:16:47.600 INFO:tasks.workunit.client.0.smithi032.stdout:8/33: creat d3/d8/fc x:0 0 0 2024-04-06T14:16:47.601 INFO:tasks.workunit.client.0.smithi032.stdout:4/21: truncate d0/f1 3394516 0 2024-04-06T14:16:47.601 INFO:tasks.workunit.client.0.smithi032.stdout:3/18: write d3/f5 [788366,18192] 0 2024-04-06T14:16:47.602 INFO:tasks.workunit.client.0.smithi032.stdout:9/49: write d2/fd [3344000,84788] 0 2024-04-06T14:16:47.604 INFO:tasks.workunit.client.0.smithi032.stdout:1/23: truncate d0/f2 531808 0 2024-04-06T14:16:47.605 INFO:tasks.workunit.client.0.smithi032.stdout:9/50: fsync d2/f8 0 2024-04-06T14:16:47.605 INFO:tasks.workunit.client.0.smithi032.stdout:4/22: symlink d0/l9 0 2024-04-06T14:16:47.607 INFO:tasks.workunit.client.0.smithi032.stdout:3/19: rename d3/l7 to d3/l8 0 2024-04-06T14:16:47.611 INFO:tasks.workunit.client.0.smithi032.stdout:4/23: mknod d0/ca 0 2024-04-06T14:16:47.618 INFO:tasks.workunit.client.0.smithi032.stdout:0/23: dwrite d3/f4 [0,4194304] 0 2024-04-06T14:16:47.621 INFO:tasks.workunit.client.0.smithi032.stdout:4/24: chown d0/l4 804 1 2024-04-06T14:16:47.621 INFO:tasks.workunit.client.0.smithi032.stdout:1/24: write d0/f2 [1496166,71149] 0 2024-04-06T14:16:47.626 INFO:tasks.workunit.client.0.smithi032.stdout:9/51: getdents d2/d5 0 2024-04-06T14:16:47.629 INFO:tasks.workunit.client.0.smithi032.stdout:3/20: dwrite d3/f5 [0,4194304] 0 2024-04-06T14:16:47.629 INFO:tasks.workunit.client.0.smithi032.stdout:8/34: dwrite d3/d8/fc [0,4194304] 0 2024-04-06T14:16:47.631 INFO:tasks.workunit.client.0.smithi032.stdout:9/52: fdatasync d2/d5/f6 0 2024-04-06T14:16:47.632 INFO:tasks.workunit.client.0.smithi032.stdout:0/24: getdents d3 0 2024-04-06T14:16:47.632 INFO:tasks.workunit.client.0.smithi032.stdout:9/53: chown d2/lb 852507759 1 2024-04-06T14:16:47.632 INFO:tasks.workunit.client.0.smithi032.stdout:8/35: chown d3/d8/fc 128461 1 2024-04-06T14:16:47.633 INFO:tasks.workunit.client.0.smithi032.stdout:0/25: chown d3/f6 381783591 1 2024-04-06T14:16:47.638 INFO:tasks.workunit.client.0.smithi032.stdout:8/36: rmdir d3 39 2024-04-06T14:16:47.638 INFO:tasks.workunit.client.0.smithi032.stdout:0/26: mknod d3/c7 0 2024-04-06T14:16:47.639 INFO:tasks.workunit.client.0.smithi032.stdout:1/25: link d0/c7 d0/c8 0 2024-04-06T14:16:47.639 INFO:tasks.workunit.client.0.smithi032.stdout:1/26: chown d0 0 1 2024-04-06T14:16:47.640 INFO:tasks.workunit.client.0.smithi032.stdout:4/25: link d0/l4 d0/lb 0 2024-04-06T14:16:47.643 INFO:tasks.workunit.client.0.smithi032.stdout:0/27: fsync d3/f6 0 2024-04-06T14:16:47.644 INFO:tasks.workunit.client.0.smithi032.stdout:1/27: mkdir d0/d9 0 2024-04-06T14:16:47.646 INFO:tasks.workunit.client.0.smithi032.stdout:1/28: write d0/f2 [1220856,56141] 0 2024-04-06T14:16:47.648 INFO:tasks.workunit.client.0.smithi032.stdout:3/21: link l2 d3/l9 0 2024-04-06T14:16:47.649 INFO:tasks.workunit.client.0.smithi032.stdout:9/54: getdents d2 0 2024-04-06T14:16:47.651 INFO:tasks.workunit.client.0.smithi032.stdout:8/37: symlink d3/d8/d7/ld 0 2024-04-06T14:16:47.651 INFO:tasks.workunit.client.0.smithi032.stdout:1/29: mkdir d0/da 0 2024-04-06T14:16:47.653 INFO:tasks.workunit.client.0.smithi032.stdout:3/22: creat d3/fa x:0 0 0 2024-04-06T14:16:47.654 INFO:tasks.workunit.client.0.smithi032.stdout:9/55: creat d2/f10 x:0 0 0 2024-04-06T14:16:47.654 INFO:tasks.workunit.client.0.smithi032.stdout:1/30: mknod d0/cb 0 2024-04-06T14:16:47.654 INFO:tasks.workunit.client.0.smithi032.stdout:8/38: chown d3/d8/d7/ld 6870 1 2024-04-06T14:16:47.655 INFO:tasks.workunit.client.0.smithi032.stdout:4/26: dwrite d0/f1 [0,4194304] 0 2024-04-06T14:16:47.656 INFO:tasks.workunit.client.0.smithi032.stdout:9/56: mknod d2/c11 0 2024-04-06T14:16:47.657 INFO:tasks.workunit.client.0.smithi032.stdout:4/27: readlink d0/lb 0 2024-04-06T14:16:47.659 INFO:tasks.workunit.client.0.smithi032.stdout:9/57: chown d2/f10 4612624 1 2024-04-06T14:16:47.660 INFO:tasks.workunit.client.0.smithi032.stdout:8/39: rmdir d3/db 0 2024-04-06T14:16:47.661 INFO:tasks.workunit.client.0.smithi032.stdout:4/28: mknod d0/cc 0 2024-04-06T14:16:47.666 INFO:tasks.workunit.client.0.smithi032.stdout:3/23: dwrite d3/f5 [0,4194304] 0 2024-04-06T14:16:47.666 INFO:tasks.workunit.client.0.smithi032.stdout:8/40: creat d3/d8/d7/fe x:0 0 0 2024-04-06T14:16:47.673 INFO:tasks.workunit.client.0.smithi032.stdout:1/31: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:16:47.673 INFO:tasks.workunit.client.0.smithi032.stdout:3/24: unlink d3/l8 0 2024-04-06T14:16:47.676 INFO:tasks.workunit.client.0.smithi032.stdout:3/25: fdatasync d3/fa 0 2024-04-06T14:16:47.678 INFO:tasks.workunit.client.0.smithi032.stdout:4/29: dread d0/f1 [0,4194304] 0 2024-04-06T14:16:47.684 INFO:tasks.workunit.client.0.smithi032.stdout:9/58: dwrite d2/f4 [4194304,4194304] 0 2024-04-06T14:16:47.685 INFO:tasks.workunit.client.0.smithi032.stdout:4/30: symlink d0/ld 0 2024-04-06T14:16:47.688 INFO:tasks.workunit.client.0.smithi032.stdout:4/31: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:47.690 INFO:tasks.workunit.client.0.smithi032.stdout:4/32: readlink d0/l4 0 2024-04-06T14:16:47.690 INFO:tasks.workunit.client.0.smithi032.stdout:1/32: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:16:47.692 INFO:tasks.workunit.client.0.smithi032.stdout:9/59: link d2/f10 d2/d5/f12 0 2024-04-06T14:16:47.696 INFO:tasks.workunit.client.0.smithi032.stdout:3/26: rmdir d3 39 2024-04-06T14:16:47.697 INFO:tasks.workunit.client.0.smithi032.stdout:9/60: dread d2/d5/f6 [0,4194304] 0 2024-04-06T14:16:47.702 INFO:tasks.workunit.client.0.smithi032.stdout:8/41: truncate d3/d8/fc 2411511 0 2024-04-06T14:16:47.703 INFO:tasks.workunit.client.0.smithi032.stdout:1/33: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:47.703 INFO:tasks.workunit.client.0.smithi032.stdout:8/42: readlink l2 0 2024-04-06T14:16:47.704 INFO:tasks.workunit.client.0.smithi032.stdout:4/33: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:16:47.707 INFO:tasks.workunit.client.0.smithi032.stdout:3/27: rename d3 to d3/db 22 2024-04-06T14:16:47.708 INFO:tasks.workunit.client.0.smithi032.stdout:9/61: creat d2/f13 x:0 0 0 2024-04-06T14:16:47.709 INFO:tasks.workunit.client.0.smithi032.stdout:9/62: chown d2/le 342 1 2024-04-06T14:16:47.710 INFO:tasks.workunit.client.0.smithi032.stdout:9/63: truncate d2/f10 677968 0 2024-04-06T14:16:47.711 INFO:tasks.workunit.client.0.smithi032.stdout:4/34: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:47.714 INFO:tasks.workunit.client.0.smithi032.stdout:8/43: symlink d3/d8/d7/lf 0 2024-04-06T14:16:47.715 INFO:tasks.workunit.client.0.smithi032.stdout:3/28: creat d3/fc x:0 0 0 2024-04-06T14:16:47.716 INFO:tasks.workunit.client.0.smithi032.stdout:1/34: mknod d0/da/cc 0 2024-04-06T14:16:47.717 INFO:tasks.workunit.client.0.smithi032.stdout:9/64: creat d2/d5/f14 x:0 0 0 2024-04-06T14:16:47.717 INFO:tasks.workunit.client.0.smithi032.stdout:4/35: mknod d0/ce 0 2024-04-06T14:16:47.718 INFO:tasks.workunit.client.0.smithi032.stdout:9/65: readlink d2/le 0 2024-04-06T14:16:47.720 INFO:tasks.workunit.client.0.smithi032.stdout:1/35: chown d0/cb 501 1 2024-04-06T14:16:47.721 INFO:tasks.workunit.client.0.smithi032.stdout:9/66: mkdir d2/d15 0 2024-04-06T14:16:47.722 INFO:tasks.workunit.client.0.smithi032.stdout:4/36: mkdir d0/df 0 2024-04-06T14:16:47.724 INFO:tasks.workunit.client.0.smithi032.stdout:1/36: fsync d0/f2 0 2024-04-06T14:16:47.725 INFO:tasks.workunit.client.0.smithi032.stdout:8/44: creat d3/f10 x:0 0 0 2024-04-06T14:16:47.725 INFO:tasks.workunit.client.0.smithi032.stdout:1/37: fsync d0/f2 0 2024-04-06T14:16:47.725 INFO:tasks.workunit.client.0.smithi032.stdout:4/37: fdatasync d0/f1 0 2024-04-06T14:16:47.726 INFO:tasks.workunit.client.0.smithi032.stdout:3/29: dwrite d3/f4 [0,4194304] 0 2024-04-06T14:16:47.727 INFO:tasks.workunit.client.0.smithi032.stdout:1/38: fdatasync d0/f2 0 2024-04-06T14:16:47.730 INFO:tasks.workunit.client.0.smithi032.stdout:4/38: symlink d0/l10 0 2024-04-06T14:16:47.730 INFO:tasks.workunit.client.0.smithi032.stdout:4/39: chown d0 111154 1 2024-04-06T14:16:47.732 INFO:tasks.workunit.client.0.smithi032.stdout:3/30: symlink d3/ld 0 2024-04-06T14:16:47.737 INFO:tasks.workunit.client.0.smithi032.stdout:4/40: mknod d0/c11 0 2024-04-06T14:16:47.738 INFO:tasks.workunit.client.0.smithi032.stdout:3/31: mknod d3/ce 0 2024-04-06T14:16:47.739 INFO:tasks.workunit.client.0.smithi032.stdout:4/41: truncate d0/f3 5044843 0 2024-04-06T14:16:47.739 INFO:tasks.workunit.client.0.smithi032.stdout:1/39: creat d0/d9/fd x:0 0 0 2024-04-06T14:16:47.747 INFO:tasks.workunit.client.0.smithi032.stdout:1/40: write d0/f2 [1637285,4530] 0 2024-04-06T14:16:47.747 INFO:tasks.workunit.client.0.smithi032.stdout:3/32: readlink d3/l9 0 2024-04-06T14:16:47.748 INFO:tasks.workunit.client.0.smithi032.stdout:4/42: write d0/f3 [3520402,92117] 0 2024-04-06T14:16:47.749 INFO:tasks.workunit.client.0.smithi032.stdout:3/33: chown d3/ld 3216743 1 2024-04-06T14:16:47.752 INFO:tasks.workunit.client.0.smithi032.stdout:4/43: write d0/f3 [1559057,36851] 0 2024-04-06T14:16:47.753 INFO:tasks.workunit.client.0.smithi032.stdout:8/45: write d3/d8/fc [1614486,40968] 0 2024-04-06T14:16:47.754 INFO:tasks.workunit.client.0.smithi032.stdout:3/34: rename d3/ce to d3/cf 0 2024-04-06T14:16:47.757 INFO:tasks.workunit.client.0.smithi032.stdout:8/46: dread - d3/f10 zero size 2024-04-06T14:16:47.759 INFO:tasks.workunit.client.0.smithi032.stdout:8/47: fsync d3/d8/d7/fe 0 2024-04-06T14:16:47.760 INFO:tasks.workunit.client.0.smithi032.stdout:1/41: dwrite d0/d9/fd [0,4194304] 0 2024-04-06T14:16:47.762 INFO:tasks.workunit.client.0.smithi032.stdout:4/44: dread d0/f1 [0,4194304] 0 2024-04-06T14:16:47.763 INFO:tasks.workunit.client.0.smithi032.stdout:1/42: mknod d0/da/ce 0 2024-04-06T14:16:47.764 INFO:tasks.workunit.client.0.smithi032.stdout:8/48: creat d3/f11 x:0 0 0 2024-04-06T14:16:47.766 INFO:tasks.workunit.client.0.smithi032.stdout:8/49: chown d3/d8/fc 2 1 2024-04-06T14:16:47.768 INFO:tasks.workunit.client.0.smithi032.stdout:4/45: link d0/c11 d0/df/c12 0 2024-04-06T14:16:47.770 INFO:tasks.workunit.client.0.smithi032.stdout:4/46: write d0/f3 [657098,11412] 0 2024-04-06T14:16:47.770 INFO:tasks.workunit.client.0.smithi032.stdout:8/50: mkdir d3/d8/d12 0 2024-04-06T14:16:47.772 INFO:tasks.workunit.client.0.smithi032.stdout:1/43: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:16:47.773 INFO:tasks.workunit.client.0.smithi032.stdout:8/51: creat d3/d8/f13 x:0 0 0 2024-04-06T14:16:47.774 INFO:tasks.workunit.client.0.smithi032.stdout:4/47: creat d0/df/f13 x:0 0 0 2024-04-06T14:16:47.776 INFO:tasks.workunit.client.0.smithi032.stdout:8/52: mkdir d3/d8/d7/d14 0 2024-04-06T14:16:47.777 INFO:tasks.workunit.client.0.smithi032.stdout:4/48: symlink d0/l14 0 2024-04-06T14:16:47.779 INFO:tasks.workunit.client.0.smithi032.stdout:4/49: creat d0/df/f15 x:0 0 0 2024-04-06T14:16:47.780 INFO:tasks.workunit.client.0.smithi032.stdout:8/53: symlink d3/d8/d7/d14/l15 0 2024-04-06T14:16:47.782 INFO:tasks.workunit.client.0.smithi032.stdout:8/54: creat d3/d8/d7/f16 x:0 0 0 2024-04-06T14:16:47.783 INFO:tasks.workunit.client.0.smithi032.stdout:8/55: chown d3/d8/d7/ld 1341 1 2024-04-06T14:16:47.783 INFO:tasks.workunit.client.0.smithi032.stdout:4/50: unlink d0/c11 0 2024-04-06T14:16:47.784 INFO:tasks.workunit.client.0.smithi032.stdout:8/56: truncate d3/f10 234625 0 2024-04-06T14:16:47.791 INFO:tasks.workunit.client.0.smithi032.stdout:4/51: rmdir d0 39 2024-04-06T14:16:47.796 INFO:tasks.workunit.client.0.smithi032.stdout:1/44: fsync d0/f2 0 2024-04-06T14:16:47.798 INFO:tasks.workunit.client.0.smithi032.stdout:4/52: link d0/ca d0/c16 0 2024-04-06T14:16:47.800 INFO:tasks.workunit.client.0.smithi032.stdout:4/53: read d0/f1 [1337145,96433] 0 2024-04-06T14:16:47.803 INFO:tasks.workunit.client.0.smithi032.stdout:4/54: write d0/f3 [2129990,3661] 0 2024-04-06T14:16:47.805 INFO:tasks.workunit.client.0.smithi032.stdout:1/45: dwrite d0/d9/fd [4194304,4194304] 0 2024-04-06T14:16:47.807 INFO:tasks.workunit.client.0.smithi032.stdout:4/55: symlink d0/df/l17 0 2024-04-06T14:16:47.810 INFO:tasks.workunit.client.0.smithi032.stdout:1/46: link d0/cb d0/d9/cf 0 2024-04-06T14:16:47.854 INFO:tasks.workunit.client.0.smithi032.stdout:4/56: dread - d0/df/f15 zero size 2024-04-06T14:16:47.868 INFO:tasks.workunit.client.0.smithi032.stdout:1/47: truncate d0/f2 1124391 0 2024-04-06T14:16:47.871 INFO:tasks.workunit.client.0.smithi032.stdout:1/48: chown d0/c7 3211 1 2024-04-06T14:16:47.872 INFO:tasks.workunit.client.0.smithi032.stdout:1/49: write d0/d9/fd [509964,2955] 0 2024-04-06T14:16:47.873 INFO:tasks.workunit.client.0.smithi032.stdout:4/57: creat d0/df/f18 x:0 0 0 2024-04-06T14:16:47.883 INFO:tasks.workunit.client.0.smithi032.stdout:1/50: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:47.885 INFO:tasks.workunit.client.0.smithi032.stdout:1/51: mkdir d0/d10 0 2024-04-06T14:16:47.886 INFO:tasks.workunit.client.0.smithi032.stdout:4/58: truncate d0/f3 3148887 0 2024-04-06T14:16:47.896 INFO:tasks.workunit.client.0.smithi032.stdout:4/59: dwrite d0/df/f15 [0,4194304] 0 2024-04-06T14:16:47.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.900+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f07b804bed0 con 0x7f07b80fbcd0 2024-04-06T14:16:47.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.901+0000 7f07a6ffd700 1 -- 172.21.15.32:0/2517352994 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7f07a8053e60 con 0x7f07b80fbcd0 2024-04-06T14:16:47.901 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 1 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:16:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:16:47.903 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 13, 2024-04-06T14:16:47.903 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 1 2024-04-06T14:16:47.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:16:47.903 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:16:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f07a006aec0 msgr2=0x7f07a006d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f07a006aec0 0x7f07a006d370 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009ed0 tx=0x7f07ac0091a0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b80fbcd0 msgr2=0x7f07b810db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b80fbcd0 0x7f07b810db30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f07a800b900 tx=0x7f07a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 shutdown_connections 2024-04-06T14:16:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f07a006aec0 0x7f07a006d370 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b80fbcd0 0x7f07b810db30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 --2- 172.21.15.32:0/2517352994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810e070 0x7f07b810eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 >> 172.21.15.32:0/2517352994 conn(0x7f07b80f7430 msgr2=0x7f07b8065db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.904+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 shutdown_connections 2024-04-06T14:16:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:47.905+0000 7f07bccee700 1 -- 172.21.15.32:0/2517352994 wait complete. 2024-04-06T14:16:47.909 INFO:tasks.workunit.client.0.smithi032.stdout:4/60: chown d0/f3 1804464229 1 2024-04-06T14:16:47.911 INFO:tasks.workunit.client.0.smithi032.stdout:4/61: symlink d0/l19 0 2024-04-06T14:16:47.913 INFO:tasks.workunit.client.0.smithi032.stdout:4/62: mknod d0/df/c1a 0 2024-04-06T14:16:47.915 INFO:tasks.workunit.client.0.smithi032.stdout:4/63: creat d0/f1b x:0 0 0 2024-04-06T14:16:47.917 INFO:tasks.workunit.client.0.smithi032.stdout:4/64: write d0/f1b [128370,45457] 0 2024-04-06T14:16:47.917 INFO:tasks.workunit.client.0.smithi032.stdout:4/65: stat d0/lb 0 2024-04-06T14:16:47.917 INFO:tasks.workunit.client.0.smithi032.stdout:4/66: fsync d0/df/f15 0 2024-04-06T14:16:47.923 INFO:tasks.workunit.client.0.smithi032.stdout:4/67: write d0/f3 [2201210,2459] 0 2024-04-06T14:16:47.924 INFO:tasks.workunit.client.0.smithi032.stdout:4/68: creat d0/f1c x:0 0 0 2024-04-06T14:16:47.926 INFO:tasks.workunit.client.0.smithi032.stdout:4/69: dread - d0/df/f13 zero size 2024-04-06T14:16:47.927 INFO:tasks.workunit.client.0.smithi032.stdout:4/70: read - d0/f1c zero size 2024-04-06T14:16:47.929 INFO:tasks.workunit.client.0.smithi032.stdout:4/71: unlink d0/df/f13 0 2024-04-06T14:16:47.931 INFO:tasks.workunit.client.0.smithi032.stdout:4/72: mknod d0/c1d 0 2024-04-06T14:16:47.931 INFO:tasks.workunit.client.0.smithi032.stdout:4/73: stat d0/f3 0 2024-04-06T14:16:47.933 INFO:tasks.workunit.client.0.smithi032.stdout:4/74: write d0/f1 [2219166,46807] 0 2024-04-06T14:16:47.933 INFO:tasks.workunit.client.0.smithi032.stdout:4/75: fdatasync d0/f3 0 2024-04-06T14:16:47.943 INFO:tasks.workunit.client.0.smithi032.stdout:4/76: dwrite d0/f1b [0,4194304] 0 2024-04-06T14:16:47.944 INFO:tasks.workunit.client.0.smithi032.stdout:4/77: dread - d0/df/f18 zero size 2024-04-06T14:16:47.950 INFO:tasks.workunit.client.0.smithi032.stdout:4/78: dread d0/f1b [0,4194304] 0 2024-04-06T14:16:47.953 INFO:tasks.workunit.client.0.smithi032.stdout:4/79: creat d0/f1e x:0 0 0 2024-04-06T14:16:47.960 INFO:tasks.workunit.client.0.smithi032.stdout:4/80: truncate d0/f1 662008 0 2024-04-06T14:16:47.962 INFO:tasks.workunit.client.0.smithi032.stdout:4/81: unlink d0/l14 0 2024-04-06T14:16:47.962 INFO:tasks.workunit.client.0.smithi032.stdout:4/82: fdatasync d0/f3 0 2024-04-06T14:16:47.966 INFO:tasks.workunit.client.0.smithi032.stdout:4/83: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:47.966 INFO:tasks.workunit.client.0.smithi032.stdout:4/84: fsync d0/f1c 0 2024-04-06T14:16:47.978 INFO:tasks.workunit.client.0.smithi032.stdout:4/85: dwrite d0/f1 [0,4194304] 0 2024-04-06T14:16:47.982 INFO:tasks.workunit.client.0.smithi032.stdout:4/86: unlink d0/df/c12 0 2024-04-06T14:16:47.984 INFO:tasks.workunit.client.0.smithi032.stdout:4/87: mkdir d0/df/d1f 0 2024-04-06T14:16:47.985 INFO:tasks.workunit.client.0.smithi032.stdout:4/88: rename d0 to d0/df/d1f/d20 22 2024-04-06T14:16:47.985 INFO:tasks.workunit.client.0.smithi032.stdout:4/89: dread - d0/df/f18 zero size 2024-04-06T14:16:47.986 INFO:tasks.workunit.client.0.smithi032.stdout:4/90: readlink d0/l10 0 2024-04-06T14:16:47.988 INFO:tasks.workunit.client.0.smithi032.stdout:4/91: mknod d0/df/c21 0 2024-04-06T14:16:47.989 INFO:tasks.workunit.client.0.smithi032.stdout:4/92: write d0/f1e [53191,22333] 0 2024-04-06T14:16:47.990 INFO:tasks.workunit.client.0.smithi032.stdout:4/93: chown d0/l4 1 1 2024-04-06T14:16:47.993 INFO:tasks.workunit.client.0.smithi032.stdout:4/94: rename d0/l19 to d0/l22 0 2024-04-06T14:16:47.996 INFO:tasks.workunit.client.0.smithi032.stdout:4/95: mkdir d0/df/d23 0 2024-04-06T14:16:48.000 INFO:tasks.workunit.client.0.smithi032.stdout:4/96: fsync d0/f1 0 2024-04-06T14:16:48.001 INFO:tasks.workunit.client.0.smithi032.stdout:4/97: read - d0/df/f18 zero size 2024-04-06T14:16:48.003 INFO:tasks.workunit.client.0.smithi032.stdout:4/98: mknod d0/df/d23/c24 0 2024-04-06T14:16:48.005 INFO:tasks.workunit.client.0.smithi032.stdout:4/99: write d0/f1b [1524583,76534] 0 2024-04-06T14:16:48.008 INFO:tasks.workunit.client.0.smithi032.stdout:4/100: unlink d0/cc 0 2024-04-06T14:16:48.008 INFO:tasks.workunit.client.0.smithi032.stdout:4/101: readlink d0/l2 0 2024-04-06T14:16:48.009 INFO:tasks.workunit.client.0.smithi032.stdout:4/102: readlink d0/l22 0 2024-04-06T14:16:48.010 INFO:tasks.workunit.client.0.smithi032.stdout:4/103: write d0/f1e [717378,62241] 0 2024-04-06T14:16:48.011 INFO:tasks.workunit.client.0.smithi032.stdout:4/104: chown d0/df/l17 3327 1 2024-04-06T14:16:48.014 INFO:tasks.workunit.client.0.smithi032.stdout:4/105: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:48.018 INFO:tasks.workunit.client.0.smithi032.stdout:4/106: dread d0/f1b [0,4194304] 0 2024-04-06T14:16:48.018 INFO:tasks.workunit.client.0.smithi032.stdout:4/107: fdatasync d0/f1c 0 2024-04-06T14:16:48.019 INFO:tasks.workunit.client.0.smithi032.stdout:4/108: read d0/f3 [2967925,47993] 0 2024-04-06T14:16:48.021 INFO:tasks.workunit.client.0.smithi032.stdout:4/109: getdents d0/df/d1f 0 2024-04-06T14:16:48.024 INFO:tasks.workunit.client.0.smithi032.stdout:4/110: rename d0/f3 to d0/df/d1f/f25 0 2024-04-06T14:16:48.024 INFO:tasks.workunit.client.0.smithi032.stdout:4/111: fdatasync d0/f1c 0 2024-04-06T14:16:48.027 INFO:tasks.workunit.client.0.smithi032.stdout:4/112: unlink d0/df/f15 0 2024-04-06T14:16:48.028 INFO:tasks.workunit.client.0.smithi032.stdout:4/113: chown d0/l4 0 1 2024-04-06T14:16:48.030 INFO:tasks.workunit.client.0.smithi032.stdout:4/114: creat d0/df/d1f/f26 x:0 0 0 2024-04-06T14:16:48.034 INFO:tasks.workunit.client.0.smithi032.stdout:4/115: link d0/l9 d0/df/d1f/l27 0 2024-04-06T14:16:48.036 INFO:tasks.workunit.client.0.smithi032.stdout:4/116: mknod d0/df/c28 0 2024-04-06T14:16:48.039 INFO:tasks.workunit.client.0.smithi032.stdout:4/117: dread d0/f1 [0,4194304] 0 2024-04-06T14:16:48.041 INFO:tasks.workunit.client.0.smithi032.stdout:4/118: symlink d0/l29 0 2024-04-06T14:16:48.042 INFO:tasks.workunit.client.0.smithi032.stdout:4/119: dread - d0/df/f18 zero size 2024-04-06T14:16:48.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.045+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/7695889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 msgr2=0x7fb7c40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:48.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.045+0000 7fb7c8fd1700 1 --2- 172.21.15.32:0/7695889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 0x7fb7c40fc0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4004940 tx=0x7fb7b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:48.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/7695889 shutdown_connections 2024-04-06T14:16:48.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 --2- 172.21.15.32:0/7695889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7c40fcf80 0x7fb7c40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:48.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 --2- 172.21.15.32:0/7695889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 0x7fb7c40fc0e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:48.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/7695889 >> 172.21.15.32:0/7695889 conn(0x7fb7c40f7430 msgr2=0x7fb7c40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:48.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/7695889 shutdown_connections 2024-04-06T14:16:48.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/7695889 wait complete. 2024-04-06T14:16:48.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 Processor -- start 2024-04-06T14:16:48.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 -- start start 2024-04-06T14:16:48.049 INFO:tasks.workunit.client.0.smithi032.stdout:4/120: getdents d0/df/d23 0 2024-04-06T14:16:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 0x7fb7c40ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7c40fcf80 0x7fb7c40fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40fe870 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.046+0000 7fb7c8fd1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40fe9e0 con 0x7fb7c40fcf80 2024-04-06T14:16:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 0x7fb7c40ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c1d9b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7c40fcf80 0x7fb7c40fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 0x7fb7c40ffdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48802/0 (socket says 172.21.15.32:48802) 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c259c700 1 -- 172.21.15.32:0/2381447064 learned_addr learned my addr 172.21.15.32:0/2381447064 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c1d9b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7c40fcf80 0x7fb7c40fe2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:33120/0 (socket says 172.21.15.32:33120) 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c259c700 1 -- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7c40fcf80 msgr2=0x7fb7c40fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c259c700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7c40fcf80 0x7fb7c40fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c259c700 1 -- 172.21.15.32:0/2381447064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b4005040 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c1d9b700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7c40fcf80 0x7fb7c40fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c259c700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 0x7fb7c40ffdf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400b0d0 tx=0x7fb7b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7b2ffd700 1 -- 172.21.15.32:0/2381447064 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7b400e070 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c40fec60 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7b2ffd700 1 -- 172.21.15.32:0/2381447064 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7b400b410 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7b2ffd700 1 -- 172.21.15.32:0/2381447064 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7b4012600 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.047+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c41158d0 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.048+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c405f5f0 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.049+0000 7fb7b2ffd700 1 -- 172.21.15.32:0/2381447064 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7fb7b400b9f0 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.049+0000 7fb7b2ffd700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fb7ac06aec0 0x7fb7ac06d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:16:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.049+0000 7fb7b2ffd700 1 -- 172.21.15.32:0/2381447064 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fb7b408a480 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.049+0000 7fb7c1d9b700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fb7ac06aec0 0x7fb7ac06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:16:48.053 INFO:tasks.workunit.client.0.smithi032.stdout:4/121: mknod d0/df/c2a 0 2024-04-06T14:16:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.050+0000 7fb7c1d9b700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fb7ac06aec0 0x7fb7ac06d370 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009b80 tx=0x7fb7b800b040 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:16:48.054 INFO:tasks.workunit.client.0.smithi032.stdout:4/122: truncate d0/f1 1524282 0 2024-04-06T14:16:48.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.054+0000 7fb7b2ffd700 1 -- 172.21.15.32:0/2381447064 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fb7b4054aa0 con 0x7fb7c40fbcd0 2024-04-06T14:16:48.055 INFO:tasks.workunit.client.0.smithi032.stdout:4/123: mkdir d0/df/d2b 0 2024-04-06T14:16:48.057 INFO:tasks.workunit.client.0.smithi032.stdout:4/124: creat d0/df/d2b/f2c x:0 0 0 2024-04-06T14:16:48.058 INFO:tasks.workunit.client.0.smithi032.stdout:4/125: mknod d0/c2d 0 2024-04-06T14:16:48.063 INFO:tasks.workunit.client.0.smithi032.stdout:4/126: dread d0/f1e [0,4194304] 0 2024-04-06T14:16:48.064 INFO:tasks.workunit.client.0.smithi032.stdout:4/127: mkdir d0/df/d1f/d2e 0 2024-04-06T14:16:48.067 INFO:tasks.workunit.client.0.smithi032.stdout:4/128: rename d0/ce to d0/df/d1f/d2e/c2f 0 2024-04-06T14:16:48.068 INFO:tasks.workunit.client.0.smithi032.stdout:4/129: write d0/df/d1f/f26 [255264,117954] 0 2024-04-06T14:16:48.075 INFO:tasks.workunit.client.0.smithi032.stdout:4/130: dwrite d0/df/d1f/f26 [0,4194304] 0 2024-04-06T14:16:48.077 INFO:tasks.workunit.client.0.smithi032.stdout:4/131: symlink d0/df/d1f/d2e/l30 0 2024-04-06T14:16:48.078 INFO:tasks.workunit.client.0.smithi032.stdout:4/132: chown d0/c2d 0 1 2024-04-06T14:16:48.080 INFO:tasks.workunit.client.0.smithi032.stdout:4/133: mkdir d0/df/d23/d31 0 2024-04-06T14:16:48.085 INFO:tasks.workunit.client.0.smithi032.stdout:4/134: creat d0/df/d23/f32 x:0 0 0 2024-04-06T14:16:48.085 INFO:tasks.workunit.client.0.smithi032.stdout:4/135: dread - d0/df/f18 zero size 2024-04-06T14:16:48.087 INFO:tasks.workunit.client.0.smithi032.stdout:4/136: mknod d0/df/d2b/c33 0 2024-04-06T14:16:48.089 INFO:tasks.workunit.client.0.smithi032.stdout:4/137: creat d0/df/d1f/f34 x:0 0 0 2024-04-06T14:16:48.091 INFO:tasks.workunit.client.0.smithi032.stdout:4/138: mkdir d0/d35 0 2024-04-06T14:16:48.101 INFO:tasks.workunit.client.0.smithi032.stdout:4/139: dwrite d0/df/d23/f32 [0,4194304] 0 2024-04-06T14:16:48.102 INFO:tasks.workunit.client.0.smithi032.stdout:4/140: chown d0/df/d1f/f26 195469239 1 2024-04-06T14:16:48.109 INFO:tasks.workunit.client.0.smithi032.stdout:4/141: rename d0/l29 to d0/df/d23/d31/l36 0 2024-04-06T14:16:48.110 INFO:tasks.workunit.client.0.smithi032.stdout:4/142: dread - d0/df/f18 zero size 2024-04-06T14:16:48.111 INFO:tasks.workunit.client.0.smithi032.stdout:4/143: write d0/df/f18 [8277,69941] 0 2024-04-06T14:16:48.114 INFO:tasks.workunit.client.0.smithi032.stdout:4/144: creat d0/df/d23/d31/f37 x:0 0 0 2024-04-06T14:16:48.115 INFO:tasks.workunit.client.0.smithi032.stdout:4/145: symlink d0/df/d1f/d2e/l38 0 2024-04-06T14:16:48.118 INFO:tasks.workunit.client.0.smithi032.stdout:4/146: creat d0/df/f39 x:0 0 0 2024-04-06T14:16:48.118 INFO:tasks.workunit.client.0.smithi032.stdout:4/147: chown d0/c2d 7892595 1 2024-04-06T14:16:48.120 INFO:tasks.workunit.client.0.smithi032.stdout:4/148: rmdir d0/df/d1f/d2e 39 2024-04-06T14:16:48.121 INFO:tasks.workunit.client.0.smithi032.stdout:4/149: fsync d0/df/d23/f32 0 2024-04-06T14:16:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: Adjusting osd_memory_target on smithi086 to 2592M 2024-04-06T14:16:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:16:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:16:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:16:48.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:16:48.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: pgmap v11: 65 pgs: 65 active+clean; 209 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 5.2 MiB/s wr, 409 op/s 2024-04-06T14:16:48.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:48.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:48.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:47 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: Adjusting osd_memory_target on smithi086 to 2592M 2024-04-06T14:16:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:16:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:16:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:16:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:16:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: pgmap v11: 65 pgs: 65 active+clean; 209 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 5.2 MiB/s wr, 409 op/s 2024-04-06T14:16:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:48.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:47 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:48.368 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:16:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:16:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:16:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "1/2 daemons upgraded", 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.301+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7c405e400 con 0x7fb7ac06aec0 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.303+0000 7fb7b2ffd700 1 -- 172.21.15.32:0/2381447064 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fb7c405e400 con 0x7fb7ac06aec0 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fb7ac06aec0 msgr2=0x7fb7ac06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fb7ac06aec0 0x7fb7ac06d370 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009b80 tx=0x7fb7b800b040 comp rx=0 tx=0).stop 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 msgr2=0x7fb7c40ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 0x7fb7c40ffdf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400b0d0 tx=0x7fb7b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 shutdown_connections 2024-04-06T14:16:48.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c40fbcd0 0x7fb7c40ffdf0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:48.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fb7ac06aec0 0x7fb7ac06d370 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:48.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 --2- 172.21.15.32:0/2381447064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7c40fcf80 0x7fb7c40fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:16:48.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 >> 172.21.15.32:0/2381447064 conn(0x7fb7c40f7430 msgr2=0x7fb7c4065c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:16:48.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 shutdown_connections 2024-04-06T14:16:48.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:16:48.305+0000 7fb7c8fd1700 1 -- 172.21.15.32:0/2381447064 wait complete. 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2517352994' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:48 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:16:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2517352994' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:49.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:48 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:16:50.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:49 smithi086 ceph-mon[47925]: Reconfiguring node-exporter.smithi032 (dependencies changed)... 2024-04-06T14:16:50.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:49 smithi086 ceph-mon[47925]: Reconfiguring daemon node-exporter.smithi032 on smithi032 2024-04-06T14:16:50.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:49 smithi086 ceph-mon[47925]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:50.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:49 smithi086 ceph-mon[47925]: pgmap v12: 65 pgs: 65 active+clean; 332 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 295 KiB/s rd, 22 MiB/s wr, 453 op/s 2024-04-06T14:16:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:49 smithi032 ceph-mon[39730]: Reconfiguring node-exporter.smithi032 (dependencies changed)... 2024-04-06T14:16:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:49 smithi032 ceph-mon[39730]: Reconfiguring daemon node-exporter.smithi032 on smithi032 2024-04-06T14:16:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:49 smithi032 ceph-mon[39730]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:16:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:49 smithi032 ceph-mon[39730]: pgmap v12: 65 pgs: 65 active+clean; 332 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 295 KiB/s rd, 22 MiB/s wr, 453 op/s 2024-04-06T14:16:52.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:51 smithi086 ceph-mon[47925]: pgmap v13: 65 pgs: 65 active+clean; 332 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 278 KiB/s rd, 21 MiB/s wr, 293 op/s 2024-04-06T14:16:52.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:51 smithi032 ceph-mon[39730]: pgmap v13: 65 pgs: 65 active+clean; 332 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 278 KiB/s rd, 21 MiB/s wr, 293 op/s 2024-04-06T14:16:54.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:53 smithi086 ceph-mon[47925]: pgmap v14: 65 pgs: 65 active+clean; 399 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 494 KiB/s rd, 31 MiB/s wr, 318 op/s 2024-04-06T14:16:54.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:53 smithi032 ceph-mon[39730]: pgmap v14: 65 pgs: 65 active+clean; 399 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 494 KiB/s rd, 31 MiB/s wr, 318 op/s 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:1/52: sync 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:7/2: sync 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:8/57: sync 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:5/11: sync 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:6/11: sync 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:0/28: sync 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:2/11: sync 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:9/67: sync 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:3/35: sync 2024-04-06T14:16:55.654 INFO:tasks.workunit.client.0.smithi032.stdout:4/150: sync 2024-04-06T14:16:55.657 INFO:tasks.workunit.client.0.smithi032.stdout:7/3: unlink l0 0 2024-04-06T14:16:55.657 INFO:tasks.workunit.client.0.smithi032.stdout:7/4: dread - no filename 2024-04-06T14:16:55.657 INFO:tasks.workunit.client.0.smithi032.stdout:0/29: read d3/f4 [1919813,6919] 0 2024-04-06T14:16:55.658 INFO:tasks.workunit.client.0.smithi032.stdout:5/12: unlink f0 0 2024-04-06T14:16:55.659 INFO:tasks.workunit.client.0.smithi032.stdout:5/13: chown f1 14 1 2024-04-06T14:16:55.659 INFO:tasks.workunit.client.0.smithi032.stdout:6/12: mknod c2 0 2024-04-06T14:16:55.659 INFO:tasks.workunit.client.0.smithi032.stdout:6/13: fdatasync - no filename 2024-04-06T14:16:55.659 INFO:tasks.workunit.client.0.smithi032.stdout:6/14: dread - no filename 2024-04-06T14:16:55.661 INFO:tasks.workunit.client.0.smithi032.stdout:2/12: creat d0/f7 x:0 0 0 2024-04-06T14:16:55.661 INFO:tasks.workunit.client.0.smithi032.stdout:1/53: symlink d0/l11 0 2024-04-06T14:16:55.662 INFO:tasks.workunit.client.0.smithi032.stdout:4/151: creat d0/f3a x:0 0 0 2024-04-06T14:16:55.663 INFO:tasks.workunit.client.0.smithi032.stdout:3/36: mknod d3/c10 0 2024-04-06T14:16:55.663 INFO:tasks.workunit.client.0.smithi032.stdout:3/37: chown d3/fa 87 1 2024-04-06T14:16:55.664 INFO:tasks.workunit.client.0.smithi032.stdout:3/38: chown d3/l9 11401472 1 2024-04-06T14:16:55.664 INFO:tasks.workunit.client.0.smithi032.stdout:8/58: mknod d3/d8/d7/d14/c17 0 2024-04-06T14:16:55.665 INFO:tasks.workunit.client.0.smithi032.stdout:7/5: creat f1 x:0 0 0 2024-04-06T14:16:55.666 INFO:tasks.workunit.client.0.smithi032.stdout:0/30: write d3/f4 [3877615,61422] 0 2024-04-06T14:16:55.667 INFO:tasks.workunit.client.0.smithi032.stdout:5/14: dread - f1 zero size 2024-04-06T14:16:55.667 INFO:tasks.workunit.client.0.smithi032.stdout:6/15: symlink l3 0 2024-04-06T14:16:55.668 INFO:tasks.workunit.client.0.smithi032.stdout:6/16: chown l3 873058 1 2024-04-06T14:16:55.668 INFO:tasks.workunit.client.0.smithi032.stdout:6/17: dread - no filename 2024-04-06T14:16:55.669 INFO:tasks.workunit.client.0.smithi032.stdout:2/13: creat d0/f8 x:0 0 0 2024-04-06T14:16:55.669 INFO:tasks.workunit.client.0.smithi032.stdout:6/18: stat c2 0 2024-04-06T14:16:55.669 INFO:tasks.workunit.client.0.smithi032.stdout:2/14: chown d0 3304487 1 2024-04-06T14:16:55.670 INFO:tasks.workunit.client.0.smithi032.stdout:9/68: mknod d2/d5/c16 0 2024-04-06T14:16:55.671 INFO:tasks.workunit.client.0.smithi032.stdout:4/152: read d0/f1 [989738,66766] 0 2024-04-06T14:16:55.672 INFO:tasks.workunit.client.0.smithi032.stdout:3/39: symlink d3/l11 0 2024-04-06T14:16:55.675 INFO:tasks.workunit.client.0.smithi032.stdout:1/54: dread d0/d9/fd [4194304,4194304] 0 2024-04-06T14:16:55.676 INFO:tasks.workunit.client.0.smithi032.stdout:6/19: mkdir d4 0 2024-04-06T14:16:55.677 INFO:tasks.workunit.client.0.smithi032.stdout:1/55: read d0/d9/fd [2862707,108421] 0 2024-04-06T14:16:55.677 INFO:tasks.workunit.client.0.smithi032.stdout:6/20: chown c2 37 1 2024-04-06T14:16:55.677 INFO:tasks.workunit.client.0.smithi032.stdout:6/21: write - no filename 2024-04-06T14:16:55.678 INFO:tasks.workunit.client.0.smithi032.stdout:9/69: rename d2/cc to d2/d15/c17 0 2024-04-06T14:16:55.680 INFO:tasks.workunit.client.0.smithi032.stdout:3/40: write d3/f4 [4537839,108150] 0 2024-04-06T14:16:55.681 INFO:tasks.workunit.client.0.smithi032.stdout:4/153: creat d0/df/d2b/f3b x:0 0 0 2024-04-06T14:16:55.685 INFO:tasks.workunit.client.0.smithi032.stdout:3/41: dread d3/f4 [0,4194304] 0 2024-04-06T14:16:55.686 INFO:tasks.workunit.client.0.smithi032.stdout:9/70: truncate d2/f8 2405015 0 2024-04-06T14:16:55.691 INFO:tasks.workunit.client.0.smithi032.stdout:4/154: write d0/df/d23/d31/f37 [89190,11877] 0 2024-04-06T14:16:55.692 INFO:tasks.workunit.client.0.smithi032.stdout:6/22: mkdir d4/d5 0 2024-04-06T14:16:55.692 INFO:tasks.workunit.client.0.smithi032.stdout:6/23: truncate - no filename 2024-04-06T14:16:55.694 INFO:tasks.workunit.client.0.smithi032.stdout:3/42: creat d3/f12 x:0 0 0 2024-04-06T14:16:55.698 INFO:tasks.workunit.client.0.smithi032.stdout:0/31: getdents d3 0 2024-04-06T14:16:55.700 INFO:tasks.workunit.client.0.smithi032.stdout:4/155: rename d0/df/d1f/f26 to d0/df/d23/f3c 0 2024-04-06T14:16:55.700 INFO:tasks.workunit.client.0.smithi032.stdout:6/24: rename d4 to d4/d6 22 2024-04-06T14:16:55.700 INFO:tasks.workunit.client.0.smithi032.stdout:1/56: link d0/c7 d0/d10/c12 0 2024-04-06T14:16:55.702 INFO:tasks.workunit.client.0.smithi032.stdout:3/43: fdatasync d3/f4 0 2024-04-06T14:16:55.704 INFO:tasks.workunit.client.0.smithi032.stdout:6/25: mknod d4/c7 0 2024-04-06T14:16:55.704 INFO:tasks.workunit.client.0.smithi032.stdout:6/26: truncate - no filename 2024-04-06T14:16:55.704 INFO:tasks.workunit.client.0.smithi032.stdout:6/27: dread - no filename 2024-04-06T14:16:55.704 INFO:tasks.workunit.client.0.smithi032.stdout:6/28: fsync - no filename 2024-04-06T14:16:55.704 INFO:tasks.workunit.client.0.smithi032.stdout:6/29: dwrite - no filename 2024-04-06T14:16:55.706 INFO:tasks.workunit.client.0.smithi032.stdout:4/156: truncate d0/df/d1f/f34 560515 0 2024-04-06T14:16:55.708 INFO:tasks.workunit.client.0.smithi032.stdout:4/157: creat d0/df/d23/f3d x:0 0 0 2024-04-06T14:16:55.708 INFO:tasks.workunit.client.0.smithi032.stdout:7/6: getdents . 0 2024-04-06T14:16:55.710 INFO:tasks.workunit.client.0.smithi032.stdout:6/30: mkdir d4/d5/d8 0 2024-04-06T14:16:55.711 INFO:tasks.workunit.client.0.smithi032.stdout:6/31: truncate - no filename 2024-04-06T14:16:55.711 INFO:tasks.workunit.client.0.smithi032.stdout:6/32: fdatasync - no filename 2024-04-06T14:16:55.711 INFO:tasks.workunit.client.0.smithi032.stdout:6/33: dwrite - no filename 2024-04-06T14:16:55.711 INFO:tasks.workunit.client.0.smithi032.stdout:6/34: read - no filename 2024-04-06T14:16:55.712 INFO:tasks.workunit.client.0.smithi032.stdout:2/15: getdents d0 0 2024-04-06T14:16:55.713 INFO:tasks.workunit.client.0.smithi032.stdout:1/57: link d0/d9/fd d0/f13 0 2024-04-06T14:16:55.714 INFO:tasks.workunit.client.0.smithi032.stdout:6/35: symlink d4/d5/d8/l9 0 2024-04-06T14:16:55.715 INFO:tasks.workunit.client.0.smithi032.stdout:2/16: chown d0/f8 325 1 2024-04-06T14:16:55.717 INFO:tasks.workunit.client.0.smithi032.stdout:3/44: write d3/f4 [2398835,6843] 0 2024-04-06T14:16:55.717 INFO:tasks.workunit.client.0.smithi032.stdout:2/17: stat d0/l5 0 2024-04-06T14:16:55.718 INFO:tasks.workunit.client.0.smithi032.stdout:6/36: chown d4/d5/d8 139 1 2024-04-06T14:16:55.718 INFO:tasks.workunit.client.0.smithi032.stdout:6/37: dwrite - no filename 2024-04-06T14:16:55.721 INFO:tasks.workunit.client.0.smithi032.stdout:2/18: dread - d0/f8 zero size 2024-04-06T14:16:55.721 INFO:tasks.workunit.client.0.smithi032.stdout:6/38: chown d4 61218 1 2024-04-06T14:16:55.722 INFO:tasks.workunit.client.0.smithi032.stdout:9/71: write d2/d5/f12 [520213,23206] 0 2024-04-06T14:16:55.724 INFO:tasks.workunit.client.0.smithi032.stdout:7/7: dwrite f1 [0,4194304] 0 2024-04-06T14:16:55.724 INFO:tasks.workunit.client.0.smithi032.stdout:3/45: dread - d3/fc zero size 2024-04-06T14:16:55.726 INFO:tasks.workunit.client.0.smithi032.stdout:2/19: symlink d0/l9 0 2024-04-06T14:16:55.727 INFO:tasks.workunit.client.0.smithi032.stdout:6/39: rmdir d4/d5/d8 39 2024-04-06T14:16:55.727 INFO:tasks.workunit.client.0.smithi032.stdout:6/40: dwrite - no filename 2024-04-06T14:16:55.729 INFO:tasks.workunit.client.0.smithi032.stdout:9/72: unlink d2/f10 0 2024-04-06T14:16:55.730 INFO:tasks.workunit.client.0.smithi032.stdout:8/59: dwrite d3/f9 [0,4194304] 0 2024-04-06T14:16:55.731 INFO:tasks.workunit.client.0.smithi032.stdout:7/8: rename f1 to f2 0 2024-04-06T14:16:55.732 INFO:tasks.workunit.client.0.smithi032.stdout:8/60: stat c0 0 2024-04-06T14:16:55.732 INFO:tasks.workunit.client.0.smithi032.stdout:9/73: read - d2/ff zero size 2024-04-06T14:16:55.733 INFO:tasks.workunit.client.0.smithi032.stdout:2/20: dread d0/f1 [0,4194304] 0 2024-04-06T14:16:55.735 INFO:tasks.workunit.client.0.smithi032.stdout:3/46: rename d3/f12 to d3/f13 0 2024-04-06T14:16:55.741 INFO:tasks.workunit.client.0.smithi032.stdout:6/41: mknod d4/ca 0 2024-04-06T14:16:55.743 INFO:tasks.workunit.client.0.smithi032.stdout:4/158: dwrite d0/df/f18 [0,4194304] 0 2024-04-06T14:16:55.745 INFO:tasks.workunit.client.0.smithi032.stdout:7/9: fdatasync f2 0 2024-04-06T14:16:55.746 INFO:tasks.workunit.client.0.smithi032.stdout:8/61: symlink d3/d8/d12/l18 0 2024-04-06T14:16:55.748 INFO:tasks.workunit.client.0.smithi032.stdout:9/74: unlink d2/d15/c17 0 2024-04-06T14:16:55.751 INFO:tasks.workunit.client.0.smithi032.stdout:3/47: rename d3/cf to d3/c14 0 2024-04-06T14:16:55.754 INFO:tasks.workunit.client.0.smithi032.stdout:1/58: dwrite d0/f13 [0,4194304] 0 2024-04-06T14:16:55.757 INFO:tasks.workunit.client.0.smithi032.stdout:2/21: dwrite d0/f7 [0,4194304] 0 2024-04-06T14:16:55.758 INFO:tasks.workunit.client.0.smithi032.stdout:6/42: creat d4/d5/fb x:0 0 0 2024-04-06T14:16:55.761 INFO:tasks.workunit.client.0.smithi032.stdout:6/43: dread - d4/d5/fb zero size 2024-04-06T14:16:55.764 INFO:tasks.workunit.client.0.smithi032.stdout:2/22: mknod d0/ca 0 2024-04-06T14:16:55.765 INFO:tasks.workunit.client.0.smithi032.stdout:6/44: dread - d4/d5/fb zero size 2024-04-06T14:16:55.768 INFO:tasks.workunit.client.0.smithi032.stdout:1/59: creat d0/d10/f14 x:0 0 0 2024-04-06T14:16:55.769 INFO:tasks.workunit.client.0.smithi032.stdout:4/159: getdents d0/df/d1f 0 2024-04-06T14:16:55.770 INFO:tasks.workunit.client.0.smithi032.stdout:7/10: dwrite f2 [0,4194304] 0 2024-04-06T14:16:55.770 INFO:tasks.workunit.client.0.smithi032.stdout:7/11: rmdir - no directory 2024-04-06T14:16:55.771 INFO:tasks.workunit.client.0.smithi032.stdout:2/23: rename d0/f8 to d0/fb 0 2024-04-06T14:16:55.771 INFO:tasks.workunit.client.0.smithi032.stdout:6/45: truncate d4/d5/fb 658467 0 2024-04-06T14:16:55.772 INFO:tasks.workunit.client.0.smithi032.stdout:8/62: write d3/f10 [952593,100881] 0 2024-04-06T14:16:55.773 INFO:tasks.workunit.client.0.smithi032.stdout:1/60: write d0/d10/f14 [851592,40768] 0 2024-04-06T14:16:55.774 INFO:tasks.workunit.client.0.smithi032.stdout:3/48: dread d3/f5 [0,4194304] 0 2024-04-06T14:16:55.777 INFO:tasks.workunit.client.0.smithi032.stdout:1/61: write d0/d9/fd [9165978,18969] 0 2024-04-06T14:16:55.781 INFO:tasks.workunit.client.0.smithi032.stdout:3/49: symlink d3/l15 0 2024-04-06T14:16:55.781 INFO:tasks.workunit.client.0.smithi032.stdout:8/63: dread - d3/d8/f13 zero size 2024-04-06T14:16:55.781 INFO:tasks.workunit.client.0.smithi032.stdout:7/12: dread f2 [0,4194304] 0 2024-04-06T14:16:55.781 INFO:tasks.workunit.client.0.smithi032.stdout:6/46: unlink d4/d5/d8/l9 0 2024-04-06T14:16:55.783 INFO:tasks.workunit.client.0.smithi032.stdout:1/62: rename d0/l6 to d0/l15 0 2024-04-06T14:16:55.784 INFO:tasks.workunit.client.0.smithi032.stdout:3/50: chown d3/fc 3 1 2024-04-06T14:16:55.784 INFO:tasks.workunit.client.0.smithi032.stdout:6/47: fdatasync d4/d5/fb 0 2024-04-06T14:16:55.787 INFO:tasks.workunit.client.0.smithi032.stdout:8/64: creat d3/d8/d12/f19 x:0 0 0 2024-04-06T14:16:55.788 INFO:tasks.workunit.client.0.smithi032.stdout:3/51: mkdir d3/d16 0 2024-04-06T14:16:55.788 INFO:tasks.workunit.client.0.smithi032.stdout:7/13: fsync f2 0 2024-04-06T14:16:55.790 INFO:tasks.workunit.client.0.smithi032.stdout:8/65: readlink d3/l6 0 2024-04-06T14:16:55.792 INFO:tasks.workunit.client.0.smithi032.stdout:2/24: dwrite d0/fb [0,4194304] 0 2024-04-06T14:16:55.797 INFO:tasks.workunit.client.0.smithi032.stdout:7/14: rename f2 to f3 0 2024-04-06T14:16:55.797 INFO:tasks.workunit.client.0.smithi032.stdout:2/25: read d0/f1 [533066,115650] 0 2024-04-06T14:16:55.804 INFO:tasks.workunit.client.0.smithi032.stdout:8/66: mkdir d3/d8/d7/d1a 0 2024-04-06T14:16:55.805 INFO:tasks.workunit.client.0.smithi032.stdout:6/48: truncate d4/d5/fb 481764 0 2024-04-06T14:16:55.806 INFO:tasks.workunit.client.0.smithi032.stdout:2/26: creat d0/fc x:0 0 0 2024-04-06T14:16:55.808 INFO:tasks.workunit.client.0.smithi032.stdout:2/27: chown d0 6626678 1 2024-04-06T14:16:55.809 INFO:tasks.workunit.client.0.smithi032.stdout:6/49: chown l3 2 1 2024-04-06T14:16:55.810 INFO:tasks.workunit.client.0.smithi032.stdout:9/75: dwrite d2/d5/f12 [0,4194304] 0 2024-04-06T14:16:55.810 INFO:tasks.workunit.client.0.smithi032.stdout:3/52: symlink d3/d16/l17 0 2024-04-06T14:16:55.811 INFO:tasks.workunit.client.0.smithi032.stdout:4/160: dwrite d0/f1c [0,4194304] 0 2024-04-06T14:16:55.814 INFO:tasks.workunit.client.0.smithi032.stdout:1/63: dwrite d0/d9/fd [0,4194304] 0 2024-04-06T14:16:55.815 INFO:tasks.workunit.client.0.smithi032.stdout:7/15: link f3 f4 0 2024-04-06T14:16:55.815 INFO:tasks.workunit.client.0.smithi032.stdout:8/67: creat d3/d8/d12/f1b x:0 0 0 2024-04-06T14:16:55.817 INFO:tasks.workunit.client.0.smithi032.stdout:4/161: write d0/df/d2b/f3b [5904,42873] 0 2024-04-06T14:16:55.822 INFO:tasks.workunit.client.0.smithi032.stdout:7/16: write f3 [4510049,64385] 0 2024-04-06T14:16:55.822 INFO:tasks.workunit.client.0.smithi032.stdout:7/17: rmdir - no directory 2024-04-06T14:16:55.822 INFO:tasks.workunit.client.0.smithi032.stdout:6/50: dread d4/d5/fb [0,4194304] 0 2024-04-06T14:16:55.827 INFO:tasks.workunit.client.0.smithi032.stdout:6/51: dread d4/d5/fb [0,4194304] 0 2024-04-06T14:16:55.827 INFO:tasks.workunit.client.0.smithi032.stdout:1/64: dwrite d0/f13 [4194304,4194304] 0 2024-04-06T14:16:55.828 INFO:tasks.workunit.client.0.smithi032.stdout:9/76: creat d2/d5/f18 x:0 0 0 2024-04-06T14:16:55.832 INFO:tasks.workunit.client.0.smithi032.stdout:4/162: mknod d0/df/c3e 0 2024-04-06T14:16:55.833 INFO:tasks.workunit.client.0.smithi032.stdout:1/65: unlink d0/f13 0 2024-04-06T14:16:55.835 INFO:tasks.workunit.client.0.smithi032.stdout:6/52: creat d4/d5/d8/fc x:0 0 0 2024-04-06T14:16:55.837 INFO:tasks.workunit.client.0.smithi032.stdout:2/28: truncate d0/fb 2189199 0 2024-04-06T14:16:55.837 INFO:tasks.workunit.client.0.smithi032.stdout:7/18: dwrite f4 [0,4194304] 0 2024-04-06T14:16:55.838 INFO:tasks.workunit.client.0.smithi032.stdout:1/66: mkdir d0/d10/d16 0 2024-04-06T14:16:55.839 INFO:tasks.workunit.client.0.smithi032.stdout:3/53: dwrite d3/fa [0,4194304] 0 2024-04-06T14:16:55.840 INFO:tasks.workunit.client.0.smithi032.stdout:7/19: write f3 [1195372,82491] 0 2024-04-06T14:16:55.841 INFO:tasks.workunit.client.0.smithi032.stdout:6/53: dread d4/d5/fb [0,4194304] 0 2024-04-06T14:16:55.841 INFO:tasks.workunit.client.0.smithi032.stdout:2/29: symlink d0/ld 0 2024-04-06T14:16:55.843 INFO:tasks.workunit.client.0.smithi032.stdout:7/20: symlink l5 0 2024-04-06T14:16:55.843 INFO:tasks.workunit.client.0.smithi032.stdout:6/54: dread - d4/d5/d8/fc zero size 2024-04-06T14:16:55.843 INFO:tasks.workunit.client.0.smithi032.stdout:7/21: chown f3 957042987 1 2024-04-06T14:16:55.843 INFO:tasks.workunit.client.0.smithi032.stdout:6/55: chown c1 14 1 2024-04-06T14:16:55.844 INFO:tasks.workunit.client.0.smithi032.stdout:3/54: creat d3/d16/f18 x:0 0 0 2024-04-06T14:16:55.844 INFO:tasks.workunit.client.0.smithi032.stdout:7/22: write f3 [4489682,82917] 0 2024-04-06T14:16:55.849 INFO:tasks.workunit.client.0.smithi032.stdout:6/56: symlink d4/d5/d8/ld 0 2024-04-06T14:16:55.851 INFO:tasks.workunit.client.0.smithi032.stdout:6/57: chown c0 1 1 2024-04-06T14:16:55.856 INFO:tasks.workunit.client.0.smithi032.stdout:4/163: rmdir d0 39 2024-04-06T14:16:55.856 INFO:tasks.workunit.client.0.smithi032.stdout:8/68: dwrite d3/d8/fc [0,4194304] 0 2024-04-06T14:16:55.858 INFO:tasks.workunit.client.0.smithi032.stdout:3/55: dread d3/fa [0,4194304] 0 2024-04-06T14:16:55.858 INFO:tasks.workunit.client.0.smithi032.stdout:1/67: dwrite d0/d9/fd [0,4194304] 0 2024-04-06T14:16:55.859 INFO:tasks.workunit.client.0.smithi032.stdout:2/30: fsync d0/fb 0 2024-04-06T14:16:55.859 INFO:tasks.workunit.client.0.smithi032.stdout:8/69: dread - d3/d8/f13 zero size 2024-04-06T14:16:55.859 INFO:tasks.workunit.client.0.smithi032.stdout:6/58: truncate d4/d5/d8/fc 865784 0 2024-04-06T14:16:55.860 INFO:tasks.workunit.client.0.smithi032.stdout:1/68: stat d0/c7 0 2024-04-06T14:16:55.862 INFO:tasks.workunit.client.0.smithi032.stdout:2/31: write d0/fc [194881,111612] 0 2024-04-06T14:16:55.863 INFO:tasks.workunit.client.0.smithi032.stdout:3/56: rename d3/f5 to d3/f19 0 2024-04-06T14:16:55.864 INFO:tasks.workunit.client.0.smithi032.stdout:7/23: fdatasync f3 0 2024-04-06T14:16:55.866 INFO:tasks.workunit.client.0.smithi032.stdout:7/24: read f4 [4055616,108407] 0 2024-04-06T14:16:55.867 INFO:tasks.workunit.client.0.smithi032.stdout:1/69: creat d0/f17 x:0 0 0 2024-04-06T14:16:55.870 INFO:tasks.workunit.client.0.smithi032.stdout:2/32: symlink d0/le 0 2024-04-06T14:16:55.870 INFO:tasks.workunit.client.0.smithi032.stdout:4/164: dread d0/df/d1f/f25 [0,4194304] 0 2024-04-06T14:16:55.870 INFO:tasks.workunit.client.0.smithi032.stdout:1/70: write d0/f17 [539877,10824] 0 2024-04-06T14:16:55.874 INFO:tasks.workunit.client.0.smithi032.stdout:3/57: creat d3/d16/f1a x:0 0 0 2024-04-06T14:16:55.874 INFO:tasks.workunit.client.0.smithi032.stdout:8/70: dwrite d3/f9 [0,4194304] 0 2024-04-06T14:16:55.877 INFO:tasks.workunit.client.0.smithi032.stdout:3/58: fdatasync d3/d16/f1a 0 2024-04-06T14:16:55.877 INFO:tasks.workunit.client.0.smithi032.stdout:1/71: symlink d0/da/l18 0 2024-04-06T14:16:55.880 INFO:tasks.workunit.client.0.smithi032.stdout:8/71: creat d3/d8/d7/d14/f1c x:0 0 0 2024-04-06T14:16:55.881 INFO:tasks.workunit.client.0.smithi032.stdout:3/59: symlink d3/d16/l1b 0 2024-04-06T14:16:55.882 INFO:tasks.workunit.client.0.smithi032.stdout:1/72: symlink d0/d9/l19 0 2024-04-06T14:16:55.883 INFO:tasks.workunit.client.0.smithi032.stdout:7/25: dwrite f3 [0,4194304] 0 2024-04-06T14:16:55.885 INFO:tasks.workunit.client.0.smithi032.stdout:3/60: mknod d3/d16/c1c 0 2024-04-06T14:16:55.885 INFO:tasks.workunit.client.0.smithi032.stdout:7/26: write f3 [3271676,96197] 0 2024-04-06T14:16:55.887 INFO:tasks.workunit.client.0.smithi032.stdout:7/27: stat l5 0 2024-04-06T14:16:55.887 INFO:tasks.workunit.client.0.smithi032.stdout:2/33: dwrite d0/fc [0,4194304] 0 2024-04-06T14:16:55.887 INFO:tasks.workunit.client.0.smithi032.stdout:8/72: mkdir d3/d8/d7/d1a/d1d 0 2024-04-06T14:16:55.889 INFO:tasks.workunit.client.0.smithi032.stdout:3/61: write d3/f13 [872094,112319] 0 2024-04-06T14:16:55.893 INFO:tasks.workunit.client.0.smithi032.stdout:2/34: creat d0/ff x:0 0 0 2024-04-06T14:16:55.893 INFO:tasks.workunit.client.0.smithi032.stdout:6/59: write d4/d5/fb [86296,111728] 0 2024-04-06T14:16:55.893 INFO:tasks.workunit.client.0.smithi032.stdout:2/35: chown d0/l5 513104892 1 2024-04-06T14:16:55.896 INFO:tasks.workunit.client.0.smithi032.stdout:6/60: mknod d4/ce 0 2024-04-06T14:16:55.896 INFO:tasks.workunit.client.0.smithi032.stdout:4/165: write d0/df/d23/f32 [4136251,87380] 0 2024-04-06T14:16:55.898 INFO:tasks.workunit.client.0.smithi032.stdout:3/62: rename d3/l9 to d3/d16/l1d 0 2024-04-06T14:16:55.899 INFO:tasks.workunit.client.0.smithi032.stdout:2/36: mknod d0/c10 0 2024-04-06T14:16:55.900 INFO:tasks.workunit.client.0.smithi032.stdout:3/63: dread - d3/fc zero size 2024-04-06T14:16:55.900 INFO:tasks.workunit.client.0.smithi032.stdout:2/37: stat d0/fc 0 2024-04-06T14:16:55.902 INFO:tasks.workunit.client.0.smithi032.stdout:4/166: dread d0/df/d2b/f3b [0,4194304] 0 2024-04-06T14:16:55.903 INFO:tasks.workunit.client.0.smithi032.stdout:6/61: dread d4/d5/d8/fc [0,4194304] 0 2024-04-06T14:16:55.904 INFO:tasks.workunit.client.0.smithi032.stdout:2/38: unlink d0/ca 0 2024-04-06T14:16:55.904 INFO:tasks.workunit.client.0.smithi032.stdout:7/28: dwrite f4 [0,4194304] 0 2024-04-06T14:16:55.907 INFO:tasks.workunit.client.0.smithi032.stdout:4/167: unlink d0/df/d1f/l27 0 2024-04-06T14:16:55.911 INFO:tasks.workunit.client.0.smithi032.stdout:2/39: truncate d0/f2 1008432 0 2024-04-06T14:16:55.914 INFO:tasks.workunit.client.0.smithi032.stdout:8/73: write d3/d8/d7/fe [824673,49397] 0 2024-04-06T14:16:55.916 INFO:tasks.workunit.client.0.smithi032.stdout:2/40: symlink d0/l11 0 2024-04-06T14:16:55.917 INFO:tasks.workunit.client.0.smithi032.stdout:3/64: dwrite d3/f4 [0,4194304] 0 2024-04-06T14:16:55.921 INFO:tasks.workunit.client.0.smithi032.stdout:8/74: readlink d3/d8/d7/lf 0 2024-04-06T14:16:55.922 INFO:tasks.workunit.client.0.smithi032.stdout:7/29: dwrite f3 [0,4194304] 0 2024-04-06T14:16:55.924 INFO:tasks.workunit.client.0.smithi032.stdout:6/62: dwrite d4/d5/fb [0,4194304] 0 2024-04-06T14:16:55.924 INFO:tasks.workunit.client.0.smithi032.stdout:3/65: mkdir d3/d16/d1e 0 2024-04-06T14:16:55.927 INFO:tasks.workunit.client.0.smithi032.stdout:8/75: rename d3/d8 to d3/d8/d7/d14/d1e 22 2024-04-06T14:16:55.928 INFO:tasks.workunit.client.0.smithi032.stdout:5/15: sync 2024-04-06T14:16:55.928 INFO:tasks.workunit.client.0.smithi032.stdout:7/30: rename f4 to f6 0 2024-04-06T14:16:55.930 INFO:tasks.workunit.client.0.smithi032.stdout:4/168: dwrite d0/df/d23/f3d [0,4194304] 0 2024-04-06T14:16:55.936 INFO:tasks.workunit.client.0.smithi032.stdout:7/31: dread f3 [0,4194304] 0 2024-04-06T14:16:55.937 INFO:tasks.workunit.client.0.smithi032.stdout:5/16: symlink l2 0 2024-04-06T14:16:55.938 INFO:tasks.workunit.client.0.smithi032.stdout:8/76: rename d3/f11 to d3/d8/d7/d14/f1f 0 2024-04-06T14:16:55.939 INFO:tasks.workunit.client.0.smithi032.stdout:7/32: creat f7 x:0 0 0 2024-04-06T14:16:55.941 INFO:tasks.workunit.client.0.smithi032.stdout:5/17: creat f3 x:0 0 0 2024-04-06T14:16:55.942 INFO:tasks.workunit.client.0.smithi032.stdout:2/41: getdents d0 0 2024-04-06T14:16:55.943 INFO:tasks.workunit.client.0.smithi032.stdout:8/77: mknod d3/d8/d12/c20 0 2024-04-06T14:16:55.943 INFO:tasks.workunit.client.0.smithi032.stdout:2/42: chown d0/f7 321866 1 2024-04-06T14:16:55.944 INFO:tasks.workunit.client.0.smithi032.stdout:6/63: creat d4/d5/ff x:0 0 0 2024-04-06T14:16:55.945 INFO:tasks.workunit.client.0.smithi032.stdout:7/33: creat f8 x:0 0 0 2024-04-06T14:16:55.945 INFO:tasks.workunit.client.0.smithi032.stdout:7/34: chown f6 6158 1 2024-04-06T14:16:55.945 INFO:tasks.workunit.client.0.smithi032.stdout:5/18: symlink l4 0 2024-04-06T14:16:55.946 INFO:tasks.workunit.client.0.smithi032.stdout:7/35: stat f6 0 2024-04-06T14:16:55.946 INFO:tasks.workunit.client.0.smithi032.stdout:2/43: mkdir d0/d12 0 2024-04-06T14:16:55.946 INFO:tasks.workunit.client.0.smithi032.stdout:5/19: truncate f3 996091 0 2024-04-06T14:16:55.947 INFO:tasks.workunit.client.0.smithi032.stdout:8/78: symlink d3/d8/d7/d14/l21 0 2024-04-06T14:16:55.947 INFO:tasks.workunit.client.0.smithi032.stdout:8/79: chown d3/f10 929716 1 2024-04-06T14:16:55.948 INFO:tasks.workunit.client.0.smithi032.stdout:8/80: stat d3/d8/d12 0 2024-04-06T14:16:55.948 INFO:tasks.workunit.client.0.smithi032.stdout:7/36: creat f9 x:0 0 0 2024-04-06T14:16:55.949 INFO:tasks.workunit.client.0.smithi032.stdout:2/44: creat d0/f13 x:0 0 0 2024-04-06T14:16:55.949 INFO:tasks.workunit.client.0.smithi032.stdout:8/81: stat d3/d8/d7/f16 0 2024-04-06T14:16:55.949 INFO:tasks.workunit.client.0.smithi032.stdout:5/20: unlink l4 0 2024-04-06T14:16:55.950 INFO:tasks.workunit.client.0.smithi032.stdout:2/45: read d0/fb [1237041,99748] 0 2024-04-06T14:16:55.951 INFO:tasks.workunit.client.0.smithi032.stdout:7/37: creat fa x:0 0 0 2024-04-06T14:16:55.951 INFO:tasks.workunit.client.0.smithi032.stdout:2/46: write d0/ff [387080,31500] 0 2024-04-06T14:16:55.951 INFO:tasks.workunit.client.0.smithi032.stdout:3/66: write d3/fc [506987,31160] 0 2024-04-06T14:16:55.953 INFO:tasks.workunit.client.0.smithi032.stdout:2/47: write d0/f13 [904551,51519] 0 2024-04-06T14:16:55.954 INFO:tasks.workunit.client.0.smithi032.stdout:7/38: rename f8 to fb 0 2024-04-06T14:16:55.956 INFO:tasks.workunit.client.0.smithi032.stdout:6/64: truncate d4/d5/d8/fc 1021635 0 2024-04-06T14:16:55.957 INFO:tasks.workunit.client.0.smithi032.stdout:7/39: fdatasync f6 0 2024-04-06T14:16:55.958 INFO:tasks.workunit.client.0.smithi032.stdout:8/82: chown d3/d8/d7/d14/f1f 309 1 2024-04-06T14:16:55.959 INFO:tasks.workunit.client.0.smithi032.stdout:2/48: symlink d0/d12/l14 0 2024-04-06T14:16:56.090 INFO:tasks.workunit.client.0.smithi032.stdout:8/83: rename c0 to d3/d8/d7/d14/c22 0 2024-04-06T14:16:56.090 INFO:tasks.workunit.client.0.smithi032.stdout:7/40: dread f6 [0,4194304] 0 2024-04-06T14:16:56.091 INFO:tasks.workunit.client.0.smithi032.stdout:7/41: chown f9 23 1 2024-04-06T14:16:56.092 INFO:tasks.workunit.client.0.smithi032.stdout:2/49: dread d0/fb [0,4194304] 0 2024-04-06T14:16:56.093 INFO:tasks.workunit.client.0.smithi032.stdout:8/84: symlink d3/d8/d12/l23 0 2024-04-06T14:16:56.094 INFO:tasks.workunit.client.0.smithi032.stdout:7/42: rename f9 to fc 0 2024-04-06T14:16:56.098 INFO:tasks.workunit.client.0.smithi032.stdout:8/85: creat d3/d8/d7/d1a/f24 x:0 0 0 2024-04-06T14:16:56.102 INFO:tasks.workunit.client.0.smithi032.stdout:5/21: dwrite f1 [0,4194304] 0 2024-04-06T14:16:56.104 INFO:tasks.workunit.client.0.smithi032.stdout:6/65: dwrite d4/d5/fb [4194304,4194304] 0 2024-04-06T14:16:56.105 INFO:tasks.workunit.client.0.smithi032.stdout:5/22: creat f5 x:0 0 0 2024-04-06T14:16:56.105 INFO:tasks.workunit.client.0.smithi032.stdout:8/86: rmdir d3/d8/d7/d1a/d1d 0 2024-04-06T14:16:56.106 INFO:tasks.workunit.client.0.smithi032.stdout:2/50: dread d0/fc [0,4194304] 0 2024-04-06T14:16:56.108 INFO:tasks.workunit.client.0.smithi032.stdout:8/87: read - d3/d8/d7/d14/f1f zero size 2024-04-06T14:16:56.108 INFO:tasks.workunit.client.0.smithi032.stdout:2/51: chown d0/f7 4312324 1 2024-04-06T14:16:56.112 INFO:tasks.workunit.client.0.smithi032.stdout:2/52: creat d0/f15 x:0 0 0 2024-04-06T14:16:56.114 INFO:tasks.workunit.client.0.smithi032.stdout:5/23: dread f1 [0,4194304] 0 2024-04-06T14:16:56.115 INFO:tasks.workunit.client.0.smithi032.stdout:6/66: getdents d4 0 2024-04-06T14:16:56.116 INFO:tasks.workunit.client.0.smithi032.stdout:2/53: truncate d0/fb 3024676 0 2024-04-06T14:16:56.116 INFO:tasks.workunit.client.0.smithi032.stdout:6/67: write d4/d5/fb [6584270,14747] 0 2024-04-06T14:16:56.116 INFO:tasks.workunit.client.0.smithi032.stdout:8/88: link d3/d8/d7/d14/f1f d3/d8/f25 0 2024-04-06T14:16:56.117 INFO:tasks.workunit.client.0.smithi032.stdout:0/32: sync 2024-04-06T14:16:56.117 INFO:tasks.workunit.client.0.smithi032.stdout:2/54: stat d0/f15 0 2024-04-06T14:16:56.118 INFO:tasks.workunit.client.0.smithi032.stdout:8/89: chown l2 150267182 1 2024-04-06T14:16:56.119 INFO:tasks.workunit.client.0.smithi032.stdout:0/33: symlink d3/l8 0 2024-04-06T14:16:56.120 INFO:tasks.workunit.client.0.smithi032.stdout:0/34: rmdir d3 39 2024-04-06T14:16:56.121 INFO:tasks.workunit.client.0.smithi032.stdout:6/68: dread d4/d5/fb [0,4194304] 0 2024-04-06T14:16:56.123 INFO:tasks.workunit.client.0.smithi032.stdout:6/69: read - d4/d5/ff zero size 2024-04-06T14:16:56.123 INFO:tasks.workunit.client.0.smithi032.stdout:8/90: dread d3/d8/fc [0,4194304] 0 2024-04-06T14:16:56.124 INFO:tasks.workunit.client.0.smithi032.stdout:6/70: creat d4/f10 x:0 0 0 2024-04-06T14:16:56.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:55 smithi086 ceph-mon[47925]: pgmap v15: 65 pgs: 65 active+clean; 399 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 494 KiB/s rd, 29 MiB/s wr, 129 op/s 2024-04-06T14:16:56.128 INFO:tasks.workunit.client.0.smithi032.stdout:6/71: getdents d4/d5/d8 0 2024-04-06T14:16:56.130 INFO:tasks.workunit.client.0.smithi032.stdout:5/24: fdatasync f1 0 2024-04-06T14:16:56.131 INFO:tasks.workunit.client.0.smithi032.stdout:5/25: write f3 [1345975,127072] 0 2024-04-06T14:16:56.131 INFO:tasks.workunit.client.0.smithi032.stdout:6/72: rename d4/ca to d4/d5/c11 0 2024-04-06T14:16:56.132 INFO:tasks.workunit.client.0.smithi032.stdout:6/73: dread - d4/d5/ff zero size 2024-04-06T14:16:56.134 INFO:tasks.workunit.client.0.smithi032.stdout:6/74: fdatasync d4/d5/fb 0 2024-04-06T14:16:56.136 INFO:tasks.workunit.client.0.smithi032.stdout:6/75: fsync d4/d5/ff 0 2024-04-06T14:16:56.136 INFO:tasks.workunit.client.0.smithi032.stdout:0/35: write d3/f6 [5997789,27431] 0 2024-04-06T14:16:56.137 INFO:tasks.workunit.client.0.smithi032.stdout:8/91: write d3/d8/d7/d14/f1f [663578,66826] 0 2024-04-06T14:16:56.137 INFO:tasks.workunit.client.0.smithi032.stdout:0/36: stat l1 0 2024-04-06T14:16:56.139 INFO:tasks.workunit.client.0.smithi032.stdout:8/92: write d3/d8/f25 [496396,103506] 0 2024-04-06T14:16:56.140 INFO:tasks.workunit.client.0.smithi032.stdout:0/37: creat d3/f9 x:0 0 0 2024-04-06T14:16:56.140 INFO:tasks.workunit.client.0.smithi032.stdout:5/26: dwrite f1 [0,4194304] 0 2024-04-06T14:16:56.140 INFO:tasks.workunit.client.0.smithi032.stdout:6/76: mknod d4/d5/c12 0 2024-04-06T14:16:56.142 INFO:tasks.workunit.client.0.smithi032.stdout:0/38: mkdir d3/da 0 2024-04-06T14:16:56.143 INFO:tasks.workunit.client.0.smithi032.stdout:5/27: mknod c6 0 2024-04-06T14:16:56.143 INFO:tasks.workunit.client.0.smithi032.stdout:6/77: creat d4/f13 x:0 0 0 2024-04-06T14:16:56.144 INFO:tasks.workunit.client.0.smithi032.stdout:8/93: creat d3/d8/d7/f26 x:0 0 0 2024-04-06T14:16:56.145 INFO:tasks.workunit.client.0.smithi032.stdout:5/28: write f3 [915834,44381] 0 2024-04-06T14:16:56.145 INFO:tasks.workunit.client.0.smithi032.stdout:6/78: chown d4/f13 0 1 2024-04-06T14:16:56.149 INFO:tasks.workunit.client.0.smithi032.stdout:8/94: truncate d3/d8/d7/d14/f1f 1611849 0 2024-04-06T14:16:56.150 INFO:tasks.workunit.client.0.smithi032.stdout:6/79: mkdir d4/d5/d14 0 2024-04-06T14:16:56.152 INFO:tasks.workunit.client.0.smithi032.stdout:8/95: symlink d3/d8/d12/l27 0 2024-04-06T14:16:56.156 INFO:tasks.workunit.client.0.smithi032.stdout:8/96: dread - d3/d8/d7/d14/f1c zero size 2024-04-06T14:16:56.156 INFO:tasks.workunit.client.0.smithi032.stdout:5/29: dwrite f5 [0,4194304] 0 2024-04-06T14:16:56.157 INFO:tasks.workunit.client.0.smithi032.stdout:8/97: chown d3/d8 256 1 2024-04-06T14:16:56.158 INFO:tasks.workunit.client.0.smithi032.stdout:5/30: chown f3 4495492 1 2024-04-06T14:16:56.162 INFO:tasks.workunit.client.0.smithi032.stdout:5/31: read f5 [2675326,72398] 0 2024-04-06T14:16:56.163 INFO:tasks.workunit.client.0.smithi032.stdout:8/98: symlink d3/l28 0 2024-04-06T14:16:56.167 INFO:tasks.workunit.client.0.smithi032.stdout:6/80: dwrite d4/d5/ff [0,4194304] 0 2024-04-06T14:16:56.169 INFO:tasks.workunit.client.0.smithi032.stdout:0/39: dwrite d3/f5 [0,4194304] 0 2024-04-06T14:16:56.169 INFO:tasks.workunit.client.0.smithi032.stdout:6/81: readlink l3 0 2024-04-06T14:16:56.169 INFO:tasks.workunit.client.0.smithi032.stdout:8/99: creat d3/f29 x:0 0 0 2024-04-06T14:16:56.174 INFO:tasks.workunit.client.0.smithi032.stdout:6/82: write d4/d5/ff [3689767,74193] 0 2024-04-06T14:16:56.174 INFO:tasks.workunit.client.0.smithi032.stdout:8/100: chown d3/d8/d7/d14/c17 101396877 1 2024-04-06T14:16:56.179 INFO:tasks.workunit.client.0.smithi032.stdout:5/32: dwrite f5 [0,4194304] 0 2024-04-06T14:16:56.180 INFO:tasks.workunit.client.0.smithi032.stdout:8/101: symlink d3/d8/d7/d14/l2a 0 2024-04-06T14:16:56.183 INFO:tasks.workunit.client.0.smithi032.stdout:0/40: dread d3/f5 [0,4194304] 0 2024-04-06T14:16:56.187 INFO:tasks.workunit.client.0.smithi032.stdout:0/41: truncate d3/f9 92789 0 2024-04-06T14:16:56.189 INFO:tasks.workunit.client.0.smithi032.stdout:5/33: getdents . 0 2024-04-06T14:16:56.189 INFO:tasks.workunit.client.0.smithi032.stdout:8/102: link d3/l6 d3/d8/l2b 0 2024-04-06T14:16:56.190 INFO:tasks.workunit.client.0.smithi032.stdout:6/83: dwrite d4/f10 [0,4194304] 0 2024-04-06T14:16:56.192 INFO:tasks.workunit.client.0.smithi032.stdout:6/84: readlink d4/d5/d8/ld 0 2024-04-06T14:16:56.195 INFO:tasks.workunit.client.0.smithi032.stdout:0/42: dread f0 [0,4194304] 0 2024-04-06T14:16:56.199 INFO:tasks.workunit.client.0.smithi032.stdout:5/34: dread f3 [0,4194304] 0 2024-04-06T14:16:56.199 INFO:tasks.workunit.client.0.smithi032.stdout:5/35: readlink l2 0 2024-04-06T14:16:56.200 INFO:tasks.workunit.client.0.smithi032.stdout:0/43: symlink d3/da/lb 0 2024-04-06T14:16:56.204 INFO:tasks.workunit.client.0.smithi032.stdout:0/44: unlink c2 0 2024-04-06T14:16:56.204 INFO:tasks.workunit.client.0.smithi032.stdout:6/85: dwrite d4/d5/ff [0,4194304] 0 2024-04-06T14:16:56.205 INFO:tasks.workunit.client.0.smithi032.stdout:8/103: write d3/d8/d7/f16 [467221,31030] 0 2024-04-06T14:16:56.207 INFO:tasks.workunit.client.0.smithi032.stdout:5/36: dread f5 [0,4194304] 0 2024-04-06T14:16:56.208 INFO:tasks.workunit.client.0.smithi032.stdout:8/104: symlink d3/d8/l2c 0 2024-04-06T14:16:56.212 INFO:tasks.workunit.client.0.smithi032.stdout:5/37: dread f1 [0,4194304] 0 2024-04-06T14:16:56.212 INFO:tasks.workunit.client.0.smithi032.stdout:0/45: link d3/da/lb d3/da/lc 0 2024-04-06T14:16:56.214 INFO:tasks.workunit.client.0.smithi032.stdout:8/105: dread d3/d8/d7/fe [0,4194304] 0 2024-04-06T14:16:56.214 INFO:tasks.workunit.client.0.smithi032.stdout:5/38: write f5 [3764601,75498] 0 2024-04-06T14:16:56.214 INFO:tasks.workunit.client.0.smithi032.stdout:0/46: readlink d3/da/lb 0 2024-04-06T14:16:56.216 INFO:tasks.workunit.client.0.smithi032.stdout:0/47: chown d3/c7 562651 1 2024-04-06T14:16:56.216 INFO:tasks.workunit.client.0.smithi032.stdout:5/39: write f1 [3367884,791] 0 2024-04-06T14:16:56.217 INFO:tasks.workunit.client.0.smithi032.stdout:8/106: mknod d3/d8/d7/d1a/c2d 0 2024-04-06T14:16:56.217 INFO:tasks.workunit.client.0.smithi032.stdout:0/48: creat d3/fd x:0 0 0 2024-04-06T14:16:56.218 INFO:tasks.workunit.client.0.smithi032.stdout:5/40: chown f3 1360 1 2024-04-06T14:16:56.219 INFO:tasks.workunit.client.0.smithi032.stdout:5/41: creat f7 x:0 0 0 2024-04-06T14:16:56.221 INFO:tasks.workunit.client.0.smithi032.stdout:0/49: creat d3/da/fe x:0 0 0 2024-04-06T14:16:56.223 INFO:tasks.workunit.client.0.smithi032.stdout:5/42: symlink l8 0 2024-04-06T14:16:56.224 INFO:tasks.workunit.client.0.smithi032.stdout:0/50: creat d3/da/ff x:0 0 0 2024-04-06T14:16:56.225 INFO:tasks.workunit.client.0.smithi032.stdout:0/51: fsync d3/f9 0 2024-04-06T14:16:56.226 INFO:tasks.workunit.client.0.smithi032.stdout:6/86: truncate d4/d5/ff 2454501 0 2024-04-06T14:16:56.230 INFO:tasks.workunit.client.0.smithi032.stdout:5/43: creat f9 x:0 0 0 2024-04-06T14:16:56.230 INFO:tasks.workunit.client.0.smithi032.stdout:6/87: fdatasync d4/f13 0 2024-04-06T14:16:56.230 INFO:tasks.workunit.client.0.smithi032.stdout:0/52: creat d3/f10 x:0 0 0 2024-04-06T14:16:56.230 INFO:tasks.workunit.client.0.smithi032.stdout:5/44: creat fa x:0 0 0 2024-04-06T14:16:56.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:55 smithi032 ceph-mon[39730]: pgmap v15: 65 pgs: 65 active+clean; 399 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 494 KiB/s rd, 29 MiB/s wr, 129 op/s 2024-04-06T14:16:56.232 INFO:tasks.workunit.client.0.smithi032.stdout:8/107: truncate d3/f9 2442168 0 2024-04-06T14:16:56.236 INFO:tasks.workunit.client.0.smithi032.stdout:8/108: dread d3/f10 [0,4194304] 0 2024-04-06T14:16:56.236 INFO:tasks.workunit.client.0.smithi032.stdout:8/109: chown d3/d8/d7 9 1 2024-04-06T14:16:56.239 INFO:tasks.workunit.client.0.smithi032.stdout:6/88: dwrite d4/d5/d8/fc [0,4194304] 0 2024-04-06T14:16:56.239 INFO:tasks.workunit.client.0.smithi032.stdout:8/110: write d3/d8/d7/fe [595888,39840] 0 2024-04-06T14:16:56.243 INFO:tasks.workunit.client.0.smithi032.stdout:1/73: sync 2024-04-06T14:16:56.243 INFO:tasks.workunit.client.0.smithi032.stdout:9/77: sync 2024-04-06T14:16:56.243 INFO:tasks.workunit.client.0.smithi032.stdout:4/169: sync 2024-04-06T14:16:56.244 INFO:tasks.workunit.client.0.smithi032.stdout:8/111: creat d3/d8/f2e x:0 0 0 2024-04-06T14:16:56.247 INFO:tasks.workunit.client.0.smithi032.stdout:9/78: creat d2/f19 x:0 0 0 2024-04-06T14:16:56.248 INFO:tasks.workunit.client.0.smithi032.stdout:9/79: read d2/d5/f12 [1307593,57954] 0 2024-04-06T14:16:56.249 INFO:tasks.workunit.client.0.smithi032.stdout:4/170: unlink d0/df/d1f/d2e/l30 0 2024-04-06T14:16:56.249 INFO:tasks.workunit.client.0.smithi032.stdout:1/74: dread d0/f17 [0,4194304] 0 2024-04-06T14:16:56.250 INFO:tasks.workunit.client.0.smithi032.stdout:6/89: creat d4/d5/f15 x:0 0 0 2024-04-06T14:16:56.253 INFO:tasks.workunit.client.0.smithi032.stdout:4/171: unlink d0/l4 0 2024-04-06T14:16:56.255 INFO:tasks.workunit.client.0.smithi032.stdout:6/90: dread d4/f10 [0,4194304] 0 2024-04-06T14:16:56.255 INFO:tasks.workunit.client.0.smithi032.stdout:6/91: read d4/d5/ff [2374066,127079] 0 2024-04-06T14:16:56.256 INFO:tasks.workunit.client.0.smithi032.stdout:9/80: rename d2/f8 to d2/d15/f1a 0 2024-04-06T14:16:56.258 INFO:tasks.workunit.client.0.smithi032.stdout:6/92: symlink d4/l16 0 2024-04-06T14:16:56.259 INFO:tasks.workunit.client.0.smithi032.stdout:8/112: getdents d3 0 2024-04-06T14:16:56.260 INFO:tasks.workunit.client.0.smithi032.stdout:9/81: stat d2/d5/c16 0 2024-04-06T14:16:56.263 INFO:tasks.workunit.client.0.smithi032.stdout:6/93: symlink d4/l17 0 2024-04-06T14:16:56.263 INFO:tasks.workunit.client.0.smithi032.stdout:8/113: dread d3/d8/f25 [0,4194304] 0 2024-04-06T14:16:56.266 INFO:tasks.workunit.client.0.smithi032.stdout:1/75: write d0/f2 [519948,80354] 0 2024-04-06T14:16:56.266 INFO:tasks.workunit.client.0.smithi032.stdout:8/114: mkdir d3/d8/d12/d2f 0 2024-04-06T14:16:56.268 INFO:tasks.workunit.client.0.smithi032.stdout:9/82: truncate d2/d5/f6 960000 0 2024-04-06T14:16:56.271 INFO:tasks.workunit.client.0.smithi032.stdout:6/94: dwrite d4/f13 [0,4194304] 0 2024-04-06T14:16:56.272 INFO:tasks.workunit.client.0.smithi032.stdout:8/115: fsync d3/d8/f13 0 2024-04-06T14:16:56.272 INFO:tasks.workunit.client.0.smithi032.stdout:4/172: truncate d0/df/d23/f32 554957 0 2024-04-06T14:16:56.277 INFO:tasks.workunit.client.0.smithi032.stdout:8/116: chown d3/d8/d7/d14/f1f 4876 1 2024-04-06T14:16:56.284 INFO:tasks.workunit.client.0.smithi032.stdout:4/173: fdatasync d0/df/f18 0 2024-04-06T14:16:56.289 INFO:tasks.workunit.client.0.smithi032.stdout:9/83: getdents d2 0 2024-04-06T14:16:56.290 INFO:tasks.workunit.client.0.smithi032.stdout:9/84: truncate d2/ff 905229 0 2024-04-06T14:16:56.290 INFO:tasks.workunit.client.0.smithi032.stdout:8/117: dwrite d3/d8/d7/d14/f1f [0,4194304] 0 2024-04-06T14:16:56.292 INFO:tasks.workunit.client.0.smithi032.stdout:6/95: truncate d4/d5/d8/fc 974562 0 2024-04-06T14:16:56.293 INFO:tasks.workunit.client.0.smithi032.stdout:4/174: read d0/df/d23/f3c [2839674,39767] 0 2024-04-06T14:16:56.296 INFO:tasks.workunit.client.0.smithi032.stdout:8/118: creat d3/d8/d7/d1a/f30 x:0 0 0 2024-04-06T14:16:56.298 INFO:tasks.workunit.client.0.smithi032.stdout:6/96: unlink d4/d5/ff 0 2024-04-06T14:16:56.300 INFO:tasks.workunit.client.0.smithi032.stdout:4/175: creat d0/df/d1f/d2e/f3f x:0 0 0 2024-04-06T14:16:56.303 INFO:tasks.workunit.client.0.smithi032.stdout:4/176: creat d0/df/d23/f40 x:0 0 0 2024-04-06T14:16:56.307 INFO:tasks.workunit.client.0.smithi032.stdout:6/97: rmdir d4 39 2024-04-06T14:16:56.308 INFO:tasks.workunit.client.0.smithi032.stdout:9/85: write d2/d5/f14 [784356,104446] 0 2024-04-06T14:16:56.313 INFO:tasks.workunit.client.0.smithi032.stdout:9/86: creat d2/d5/f1b x:0 0 0 2024-04-06T14:16:56.318 INFO:tasks.workunit.client.0.smithi032.stdout:4/177: dwrite d0/df/f39 [0,4194304] 0 2024-04-06T14:16:56.318 INFO:tasks.workunit.client.0.smithi032.stdout:9/87: symlink d2/d5/l1c 0 2024-04-06T14:16:56.319 INFO:tasks.workunit.client.0.smithi032.stdout:6/98: dwrite d4/d5/f15 [0,4194304] 0 2024-04-06T14:16:56.322 INFO:tasks.workunit.client.0.smithi032.stdout:4/178: creat d0/df/d1f/d2e/f41 x:0 0 0 2024-04-06T14:16:56.324 INFO:tasks.workunit.client.0.smithi032.stdout:6/99: mkdir d4/d5/d18 0 2024-04-06T14:16:56.325 INFO:tasks.workunit.client.0.smithi032.stdout:9/88: creat d2/d5/f1d x:0 0 0 2024-04-06T14:16:56.325 INFO:tasks.workunit.client.0.smithi032.stdout:4/179: chown d0/l2 111 1 2024-04-06T14:16:56.330 INFO:tasks.workunit.client.0.smithi032.stdout:9/89: truncate d2/f13 358551 0 2024-04-06T14:16:56.332 INFO:tasks.workunit.client.0.smithi032.stdout:4/180: mkdir d0/df/d1f/d42 0 2024-04-06T14:16:56.332 INFO:tasks.workunit.client.0.smithi032.stdout:9/90: readlink l0 0 2024-04-06T14:16:56.334 INFO:tasks.workunit.client.0.smithi032.stdout:9/91: dread - d2/d5/f18 zero size 2024-04-06T14:16:56.334 INFO:tasks.workunit.client.0.smithi032.stdout:6/100: dwrite d4/d5/f15 [0,4194304] 0 2024-04-06T14:16:56.335 INFO:tasks.workunit.client.0.smithi032.stdout:4/181: truncate d0/df/d2b/f3b 175817 0 2024-04-06T14:16:56.336 INFO:tasks.workunit.client.0.smithi032.stdout:9/92: mkdir d2/d15/d1e 0 2024-04-06T14:16:56.337 INFO:tasks.workunit.client.0.smithi032.stdout:4/182: rmdir d0/df/d23/d31 39 2024-04-06T14:16:56.337 INFO:tasks.workunit.client.0.smithi032.stdout:9/93: readlink d2/le 0 2024-04-06T14:16:56.340 INFO:tasks.workunit.client.0.smithi032.stdout:4/183: mkdir d0/df/d2b/d43 0 2024-04-06T14:16:56.341 INFO:tasks.workunit.client.0.smithi032.stdout:6/101: dwrite d4/f13 [0,4194304] 0 2024-04-06T14:16:56.343 INFO:tasks.workunit.client.0.smithi032.stdout:9/94: creat d2/d15/d1e/f1f x:0 0 0 2024-04-06T14:16:56.345 INFO:tasks.workunit.client.0.smithi032.stdout:4/184: creat d0/f44 x:0 0 0 2024-04-06T14:16:56.345 INFO:tasks.workunit.client.0.smithi032.stdout:9/95: dread - d2/d5/f1b zero size 2024-04-06T14:16:56.350 INFO:tasks.workunit.client.0.smithi032.stdout:9/96: rename l0 to d2/d15/l20 0 2024-04-06T14:16:56.351 INFO:tasks.workunit.client.0.smithi032.stdout:4/185: mkdir d0/df/d1f/d42/d45 0 2024-04-06T14:16:56.351 INFO:tasks.workunit.client.0.smithi032.stdout:6/102: dwrite d4/f13 [4194304,4194304] 0 2024-04-06T14:16:56.352 INFO:tasks.workunit.client.0.smithi032.stdout:6/103: chown d4/f13 88615 1 2024-04-06T14:16:56.354 INFO:tasks.workunit.client.0.smithi032.stdout:6/104: rmdir d4 39 2024-04-06T14:16:56.356 INFO:tasks.workunit.client.0.smithi032.stdout:9/97: dread d2/f4 [0,4194304] 0 2024-04-06T14:16:56.357 INFO:tasks.workunit.client.0.smithi032.stdout:4/186: dread d0/f1 [0,4194304] 0 2024-04-06T14:16:56.359 INFO:tasks.workunit.client.0.smithi032.stdout:9/98: rename d2/f19 to d2/f21 0 2024-04-06T14:16:56.360 INFO:tasks.workunit.client.0.smithi032.stdout:9/99: chown d2/c11 1788 1 2024-04-06T14:16:56.363 INFO:tasks.workunit.client.0.smithi032.stdout:9/100: mknod d2/c22 0 2024-04-06T14:16:56.363 INFO:tasks.workunit.client.0.smithi032.stdout:9/101: chown d2/d15/d1e 82360 1 2024-04-06T14:16:56.365 INFO:tasks.workunit.client.0.smithi032.stdout:9/102: truncate d2/d15/d1e/f1f 688786 0 2024-04-06T14:16:56.365 INFO:tasks.workunit.client.0.smithi032.stdout:9/103: chown d2/d15/d1e/f1f 2325 1 2024-04-06T14:16:56.367 INFO:tasks.workunit.client.0.smithi032.stdout:6/105: truncate d4/f13 5902626 0 2024-04-06T14:16:56.371 INFO:tasks.workunit.client.0.smithi032.stdout:6/106: mkdir d4/d5/d19 0 2024-04-06T14:16:56.372 INFO:tasks.workunit.client.0.smithi032.stdout:3/67: sync 2024-04-06T14:16:56.372 INFO:tasks.workunit.client.0.smithi032.stdout:7/43: sync 2024-04-06T14:16:56.372 INFO:tasks.workunit.client.0.smithi032.stdout:2/55: sync 2024-04-06T14:16:56.372 INFO:tasks.workunit.client.0.smithi032.stdout:4/187: dwrite d0/df/d23/f3c [0,4194304] 0 2024-04-06T14:16:56.374 INFO:tasks.workunit.client.0.smithi032.stdout:6/107: creat d4/d5/d14/f1a x:0 0 0 2024-04-06T14:16:56.377 INFO:tasks.workunit.client.0.smithi032.stdout:7/44: rename fc to fd 0 2024-04-06T14:16:56.379 INFO:tasks.workunit.client.0.smithi032.stdout:3/68: rmdir d3/d16 39 2024-04-06T14:16:56.380 INFO:tasks.workunit.client.0.smithi032.stdout:2/56: fdatasync d0/fb 0 2024-04-06T14:16:56.381 INFO:tasks.workunit.client.0.smithi032.stdout:4/188: dread d0/f1 [0,4194304] 0 2024-04-06T14:16:56.382 INFO:tasks.workunit.client.0.smithi032.stdout:9/104: dwrite d2/d5/f6 [0,4194304] 0 2024-04-06T14:16:56.383 INFO:tasks.workunit.client.0.smithi032.stdout:2/57: readlink d0/d12/l14 0 2024-04-06T14:16:56.384 INFO:tasks.workunit.client.0.smithi032.stdout:4/189: fdatasync d0/df/d1f/d2e/f41 0 2024-04-06T14:16:56.385 INFO:tasks.workunit.client.0.smithi032.stdout:7/45: link fb fe 0 2024-04-06T14:16:56.386 INFO:tasks.workunit.client.0.smithi032.stdout:7/46: dread - fb zero size 2024-04-06T14:16:56.387 INFO:tasks.workunit.client.0.smithi032.stdout:4/190: dread - d0/df/d1f/d2e/f41 zero size 2024-04-06T14:16:56.387 INFO:tasks.workunit.client.0.smithi032.stdout:7/47: chown f6 5485570 1 2024-04-06T14:16:56.387 INFO:tasks.workunit.client.0.smithi032.stdout:9/105: rmdir d2/d15 39 2024-04-06T14:16:56.389 INFO:tasks.workunit.client.0.smithi032.stdout:6/108: truncate d4/d5/d8/fc 1070318 0 2024-04-06T14:16:56.392 INFO:tasks.workunit.client.0.smithi032.stdout:3/69: rename d3/f13 to d3/d16/d1e/f1f 0 2024-04-06T14:16:56.393 INFO:tasks.workunit.client.0.smithi032.stdout:2/58: write d0/f1 [1769263,48045] 0 2024-04-06T14:16:56.393 INFO:tasks.workunit.client.0.smithi032.stdout:3/70: dread - d3/d16/f1a zero size 2024-04-06T14:16:56.393 INFO:tasks.workunit.client.0.smithi032.stdout:4/191: chown d0/l8 8061655 1 2024-04-06T14:16:56.397 INFO:tasks.workunit.client.0.smithi032.stdout:7/48: link f6 ff 0 2024-04-06T14:16:56.398 INFO:tasks.workunit.client.0.smithi032.stdout:9/106: dread d2/d15/d1e/f1f [0,4194304] 0 2024-04-06T14:16:56.398 INFO:tasks.workunit.client.0.smithi032.stdout:4/192: stat d0/df/d1f/d2e/c2f 0 2024-04-06T14:16:56.399 INFO:tasks.workunit.client.0.smithi032.stdout:7/49: truncate fb 220997 0 2024-04-06T14:16:56.400 INFO:tasks.workunit.client.0.smithi032.stdout:6/109: rename c2 to d4/d5/c1b 0 2024-04-06T14:16:56.401 INFO:tasks.workunit.client.0.smithi032.stdout:3/71: creat d3/f20 x:0 0 0 2024-04-06T14:16:56.403 INFO:tasks.workunit.client.0.smithi032.stdout:7/50: creat f10 x:0 0 0 2024-04-06T14:16:56.403 INFO:tasks.workunit.client.0.smithi032.stdout:2/59: dwrite d0/f15 [0,4194304] 0 2024-04-06T14:16:56.405 INFO:tasks.workunit.client.0.smithi032.stdout:4/193: unlink d0/c2d 0 2024-04-06T14:16:56.411 INFO:tasks.workunit.client.0.smithi032.stdout:6/110: chown d4/d5/d8/fc 102377135 1 2024-04-06T14:16:56.414 INFO:tasks.workunit.client.0.smithi032.stdout:9/107: dread d2/f4 [4194304,4194304] 0 2024-04-06T14:16:56.416 INFO:tasks.workunit.client.0.smithi032.stdout:4/194: truncate d0/f44 981938 0 2024-04-06T14:16:56.417 INFO:tasks.workunit.client.0.smithi032.stdout:3/72: dwrite d3/d16/f1a [0,4194304] 0 2024-04-06T14:16:56.421 INFO:tasks.workunit.client.0.smithi032.stdout:4/195: readlink d0/l22 0 2024-04-06T14:16:56.423 INFO:tasks.workunit.client.0.smithi032.stdout:2/60: dwrite d0/ff [0,4194304] 0 2024-04-06T14:16:56.424 INFO:tasks.workunit.client.0.smithi032.stdout:3/73: unlink d3/d16/l17 0 2024-04-06T14:16:56.425 INFO:tasks.workunit.client.0.smithi032.stdout:6/111: dread d4/d5/d8/fc [0,4194304] 0 2024-04-06T14:16:56.425 INFO:tasks.workunit.client.0.smithi032.stdout:9/108: link d2/d5/c16 d2/d15/c23 0 2024-04-06T14:16:56.426 INFO:tasks.workunit.client.0.smithi032.stdout:4/196: stat d0/df/d23/f32 0 2024-04-06T14:16:56.427 INFO:tasks.workunit.client.0.smithi032.stdout:9/109: chown d2/d5/f1b 15 1 2024-04-06T14:16:56.428 INFO:tasks.workunit.client.0.smithi032.stdout:6/112: write d4/d5/f15 [829302,43282] 0 2024-04-06T14:16:56.428 INFO:tasks.workunit.client.0.smithi032.stdout:7/51: dwrite ff [0,4194304] 0 2024-04-06T14:16:56.429 INFO:tasks.workunit.client.0.smithi032.stdout:4/197: readlink d0/l10 0 2024-04-06T14:16:56.430 INFO:tasks.workunit.client.0.smithi032.stdout:6/113: write d4/d5/fb [7151891,52803] 0 2024-04-06T14:16:56.431 INFO:tasks.workunit.client.0.smithi032.stdout:4/198: chown d0/l10 3 1 2024-04-06T14:16:56.431 INFO:tasks.workunit.client.0.smithi032.stdout:9/110: chown d2/d15/f1a 22 1 2024-04-06T14:16:56.434 INFO:tasks.workunit.client.0.smithi032.stdout:7/52: creat f11 x:0 0 0 2024-04-06T14:16:56.434 INFO:tasks.workunit.client.0.smithi032.stdout:7/53: chown ff 1 1 2024-04-06T14:16:56.434 INFO:tasks.workunit.client.0.smithi032.stdout:7/54: stat fe 0 2024-04-06T14:16:56.434 INFO:tasks.workunit.client.0.smithi032.stdout:4/199: mkdir d0/df/d2b/d46 0 2024-04-06T14:16:56.435 INFO:tasks.workunit.client.0.smithi032.stdout:9/111: creat d2/d15/d1e/f24 x:0 0 0 2024-04-06T14:16:56.438 INFO:tasks.workunit.client.0.smithi032.stdout:7/55: creat f12 x:0 0 0 2024-04-06T14:16:56.440 INFO:tasks.workunit.client.0.smithi032.stdout:3/74: rmdir d3/d16 39 2024-04-06T14:16:56.440 INFO:tasks.workunit.client.0.smithi032.stdout:4/200: truncate d0/f1c 4821197 0 2024-04-06T14:16:56.441 INFO:tasks.workunit.client.0.smithi032.stdout:2/61: write d0/f2 [2050992,100206] 0 2024-04-06T14:16:56.443 INFO:tasks.workunit.client.0.smithi032.stdout:2/62: chown d0/f2 29753 1 2024-04-06T14:16:56.444 INFO:tasks.workunit.client.0.smithi032.stdout:9/112: mkdir d2/d25 0 2024-04-06T14:16:56.446 INFO:tasks.workunit.client.0.smithi032.stdout:2/63: mknod d0/c16 0 2024-04-06T14:16:56.450 INFO:tasks.workunit.client.0.smithi032.stdout:9/113: getdents d2/d5 0 2024-04-06T14:16:56.452 INFO:tasks.workunit.client.0.smithi032.stdout:2/64: creat d0/d12/f17 x:0 0 0 2024-04-06T14:16:56.454 INFO:tasks.workunit.client.0.smithi032.stdout:6/114: truncate d4/d5/fb 1694762 0 2024-04-06T14:16:56.454 INFO:tasks.workunit.client.0.smithi032.stdout:9/114: write d2/d5/f14 [967776,32415] 0 2024-04-06T14:16:56.455 INFO:tasks.workunit.client.0.smithi032.stdout:4/201: dwrite d0/df/d23/f3d [0,4194304] 0 2024-04-06T14:16:56.456 INFO:tasks.workunit.client.0.smithi032.stdout:7/56: dwrite fe [0,4194304] 0 2024-04-06T14:16:56.456 INFO:tasks.workunit.client.0.smithi032.stdout:3/75: truncate d3/fa 773704 0 2024-04-06T14:16:56.459 INFO:tasks.workunit.client.0.smithi032.stdout:6/115: mknod d4/d5/d14/c1c 0 2024-04-06T14:16:56.459 INFO:tasks.workunit.client.0.smithi032.stdout:3/76: dread - d3/f20 zero size 2024-04-06T14:16:56.460 INFO:tasks.workunit.client.0.smithi032.stdout:4/202: mkdir d0/d47 0 2024-04-06T14:16:56.461 INFO:tasks.workunit.client.0.smithi032.stdout:7/57: link fd f13 0 2024-04-06T14:16:56.462 INFO:tasks.workunit.client.0.smithi032.stdout:7/58: read f6 [73799,54966] 0 2024-04-06T14:16:56.463 INFO:tasks.workunit.client.0.smithi032.stdout:2/65: dread d0/ff [0,4194304] 0 2024-04-06T14:16:56.464 INFO:tasks.workunit.client.0.smithi032.stdout:5/45: sync 2024-04-06T14:16:56.464 INFO:tasks.workunit.client.0.smithi032.stdout:0/53: sync 2024-04-06T14:16:56.464 INFO:tasks.workunit.client.0.smithi032.stdout:7/59: chown ff 477146 1 2024-04-06T14:16:56.465 INFO:tasks.workunit.client.0.smithi032.stdout:6/116: mkdir d4/d5/d19/d1d 0 2024-04-06T14:16:56.467 INFO:tasks.workunit.client.0.smithi032.stdout:0/54: write f0 [1125797,105900] 0 2024-04-06T14:16:56.467 INFO:tasks.workunit.client.0.smithi032.stdout:6/117: truncate d4/f13 6019182 0 2024-04-06T14:16:56.468 INFO:tasks.workunit.client.0.smithi032.stdout:4/203: mkdir d0/df/d2b/d43/d48 0 2024-04-06T14:16:56.469 INFO:tasks.workunit.client.0.smithi032.stdout:3/77: creat d3/f21 x:0 0 0 2024-04-06T14:16:56.470 INFO:tasks.workunit.client.0.smithi032.stdout:0/55: truncate d3/da/ff 525781 0 2024-04-06T14:16:56.471 INFO:tasks.workunit.client.0.smithi032.stdout:6/118: write d4/d5/f15 [5216995,119896] 0 2024-04-06T14:16:56.472 INFO:tasks.workunit.client.0.smithi032.stdout:5/46: creat fb x:0 0 0 2024-04-06T14:16:56.475 INFO:tasks.workunit.client.0.smithi032.stdout:4/204: creat d0/df/d1f/d2e/f49 x:0 0 0 2024-04-06T14:16:56.478 INFO:tasks.workunit.client.0.smithi032.stdout:5/47: creat fc x:0 0 0 2024-04-06T14:16:56.480 INFO:tasks.workunit.client.0.smithi032.stdout:7/60: dwrite f13 [0,4194304] 0 2024-04-06T14:16:56.482 INFO:tasks.workunit.client.0.smithi032.stdout:6/119: symlink d4/d5/l1e 0 2024-04-06T14:16:56.490 INFO:tasks.workunit.client.0.smithi032.stdout:5/48: dwrite f1 [4194304,4194304] 0 2024-04-06T14:16:56.490 INFO:tasks.workunit.client.0.smithi032.stdout:2/66: truncate d0/fc 3542833 0 2024-04-06T14:16:56.490 INFO:tasks.workunit.client.0.smithi032.stdout:0/56: link d3/da/lb d3/l11 0 2024-04-06T14:16:56.491 INFO:tasks.workunit.client.0.smithi032.stdout:7/61: dread fb [0,4194304] 0 2024-04-06T14:16:56.491 INFO:tasks.workunit.client.0.smithi032.stdout:4/205: creat d0/df/d23/f4a x:0 0 0 2024-04-06T14:16:56.492 INFO:tasks.workunit.client.0.smithi032.stdout:7/62: read fb [607665,56997] 0 2024-04-06T14:16:56.492 INFO:tasks.workunit.client.0.smithi032.stdout:6/120: dread d4/f10 [0,4194304] 0 2024-04-06T14:16:56.493 INFO:tasks.workunit.client.0.smithi032.stdout:3/78: getdents d3/d16 0 2024-04-06T14:16:56.494 INFO:tasks.workunit.client.0.smithi032.stdout:2/67: creat d0/d12/f18 x:0 0 0 2024-04-06T14:16:56.494 INFO:tasks.workunit.client.0.smithi032.stdout:0/57: symlink d3/da/l12 0 2024-04-06T14:16:56.495 INFO:tasks.workunit.client.0.smithi032.stdout:7/63: mknod c14 0 2024-04-06T14:16:56.496 INFO:tasks.workunit.client.0.smithi032.stdout:6/121: fsync d4/d5/d8/fc 0 2024-04-06T14:16:56.497 INFO:tasks.workunit.client.0.smithi032.stdout:3/79: creat d3/d16/f22 x:0 0 0 2024-04-06T14:16:56.497 INFO:tasks.workunit.client.0.smithi032.stdout:6/122: stat d4/d5 0 2024-04-06T14:16:56.498 INFO:tasks.workunit.client.0.smithi032.stdout:4/206: dread d0/f1e [0,4194304] 0 2024-04-06T14:16:56.498 INFO:tasks.workunit.client.0.smithi032.stdout:7/64: creat f15 x:0 0 0 2024-04-06T14:16:56.498 INFO:tasks.workunit.client.0.smithi032.stdout:3/80: stat d3/ld 0 2024-04-06T14:16:56.500 INFO:tasks.workunit.client.0.smithi032.stdout:4/207: dread - d0/df/d1f/d2e/f3f zero size 2024-04-06T14:16:56.502 INFO:tasks.workunit.client.0.smithi032.stdout:2/68: rename d0/d12/f18 to d0/f19 0 2024-04-06T14:16:56.504 INFO:tasks.workunit.client.0.smithi032.stdout:6/123: write d4/f10 [1959530,26261] 0 2024-04-06T14:16:56.506 INFO:tasks.workunit.client.0.smithi032.stdout:6/124: write d4/d5/f15 [5722898,76752] 0 2024-04-06T14:16:56.509 INFO:tasks.workunit.client.0.smithi032.stdout:4/208: getdents d0/df/d2b/d43/d48 0 2024-04-06T14:16:56.510 INFO:tasks.workunit.client.0.smithi032.stdout:2/69: dread d0/f15 [0,4194304] 0 2024-04-06T14:16:56.512 INFO:tasks.workunit.client.0.smithi032.stdout:3/81: write d3/f19 [777490,58646] 0 2024-04-06T14:16:56.513 INFO:tasks.workunit.client.0.smithi032.stdout:7/65: dwrite fb [0,4194304] 0 2024-04-06T14:16:56.515 INFO:tasks.workunit.client.0.smithi032.stdout:4/209: mkdir d0/d47/d4b 0 2024-04-06T14:16:56.515 INFO:tasks.workunit.client.0.smithi032.stdout:6/125: truncate d4/d5/f15 1810221 0 2024-04-06T14:16:56.516 INFO:tasks.workunit.client.0.smithi032.stdout:2/70: rename d0/f7 to d0/d12/f1a 0 2024-04-06T14:16:56.520 INFO:tasks.workunit.client.0.smithi032.stdout:4/210: mknod d0/df/d1f/d42/c4c 0 2024-04-06T14:16:56.522 INFO:tasks.workunit.client.0.smithi032.stdout:2/71: rename d0/f19 to d0/f1b 0 2024-04-06T14:16:56.523 INFO:tasks.workunit.client.0.smithi032.stdout:4/211: stat d0/df/c1a 0 2024-04-06T14:16:56.524 INFO:tasks.workunit.client.0.smithi032.stdout:2/72: readlink d0/ld 0 2024-04-06T14:16:56.525 INFO:tasks.workunit.client.0.smithi032.stdout:7/66: dread fb [0,4194304] 0 2024-04-06T14:16:56.525 INFO:tasks.workunit.client.0.smithi032.stdout:3/82: dwrite d3/d16/f1a [0,4194304] 0 2024-04-06T14:16:56.526 INFO:tasks.workunit.client.0.smithi032.stdout:4/212: readlink d0/l9 0 2024-04-06T14:16:56.526 INFO:tasks.workunit.client.0.smithi032.stdout:7/67: chown fb 3 1 2024-04-06T14:16:56.529 INFO:tasks.workunit.client.0.smithi032.stdout:7/68: truncate f15 22699 0 2024-04-06T14:16:56.529 INFO:tasks.workunit.client.0.smithi032.stdout:4/213: chown d0/df/d23/c24 105 1 2024-04-06T14:16:56.529 INFO:tasks.workunit.client.0.smithi032.stdout:6/126: dwrite d4/f10 [0,4194304] 0 2024-04-06T14:16:56.530 INFO:tasks.workunit.client.0.smithi032.stdout:6/127: readlink d4/l17 0 2024-04-06T14:16:56.530 INFO:tasks.workunit.client.0.smithi032.stdout:2/73: rename d0/f1 to d0/d12/f1c 0 2024-04-06T14:16:56.532 INFO:tasks.workunit.client.0.smithi032.stdout:6/128: chown d4/ce 5262 1 2024-04-06T14:16:56.532 INFO:tasks.workunit.client.0.smithi032.stdout:4/214: chown d0/df/d1f/f34 89059 1 2024-04-06T14:16:56.536 INFO:tasks.workunit.client.0.smithi032.stdout:6/129: mkdir d4/d5/d8/d1f 0 2024-04-06T14:16:56.539 INFO:tasks.workunit.client.0.smithi032.stdout:3/83: dwrite d3/d16/f22 [0,4194304] 0 2024-04-06T14:16:56.540 INFO:tasks.workunit.client.0.smithi032.stdout:6/130: symlink d4/d5/l20 0 2024-04-06T14:16:56.541 INFO:tasks.workunit.client.0.smithi032.stdout:3/84: chown d3/d16 15767 1 2024-04-06T14:16:56.542 INFO:tasks.workunit.client.0.smithi032.stdout:4/215: link d0/df/d23/f3d d0/df/d23/d31/f4d 0 2024-04-06T14:16:56.543 INFO:tasks.workunit.client.0.smithi032.stdout:6/131: mkdir d4/d5/d19/d21 0 2024-04-06T14:16:56.546 INFO:tasks.workunit.client.0.smithi032.stdout:2/74: dwrite d0/d12/f1c [0,4194304] 0 2024-04-06T14:16:56.551 INFO:tasks.workunit.client.0.smithi032.stdout:2/75: mknod d0/c1d 0 2024-04-06T14:16:56.556 INFO:tasks.workunit.client.0.smithi032.stdout:4/216: dread d0/df/d23/f3c [0,4194304] 0 2024-04-06T14:16:56.557 INFO:tasks.workunit.client.0.smithi032.stdout:7/69: dwrite ff [4194304,4194304] 0 2024-04-06T14:16:56.558 INFO:tasks.workunit.client.0.smithi032.stdout:2/76: truncate d0/fc 2906932 0 2024-04-06T14:16:56.558 INFO:tasks.workunit.client.0.smithi032.stdout:4/217: readlink d0/l2 0 2024-04-06T14:16:56.559 INFO:tasks.workunit.client.0.smithi032.stdout:2/77: readlink d0/l5 0 2024-04-06T14:16:56.559 INFO:tasks.workunit.client.0.smithi032.stdout:7/70: write f11 [626104,78684] 0 2024-04-06T14:16:56.560 INFO:tasks.workunit.client.0.smithi032.stdout:6/132: dwrite d4/f13 [0,4194304] 0 2024-04-06T14:16:56.562 INFO:tasks.workunit.client.0.smithi032.stdout:4/218: unlink d0/df/f39 0 2024-04-06T14:16:56.564 INFO:tasks.workunit.client.0.smithi032.stdout:7/71: mkdir d16 0 2024-04-06T14:16:56.567 INFO:tasks.workunit.client.0.smithi032.stdout:6/133: symlink d4/d5/d14/l22 0 2024-04-06T14:16:56.567 INFO:tasks.workunit.client.0.smithi032.stdout:4/219: truncate d0/f1e 1634738 0 2024-04-06T14:16:56.568 INFO:tasks.workunit.client.0.smithi032.stdout:7/72: truncate f10 172551 0 2024-04-06T14:16:56.570 INFO:tasks.workunit.client.0.smithi032.stdout:2/78: dwrite d0/f13 [0,4194304] 0 2024-04-06T14:16:56.571 INFO:tasks.workunit.client.0.smithi032.stdout:2/79: rename d0 to d0/d12/d1e 22 2024-04-06T14:16:56.572 INFO:tasks.workunit.client.0.smithi032.stdout:4/220: creat d0/df/d1f/f4e x:0 0 0 2024-04-06T14:16:56.572 INFO:tasks.workunit.client.0.smithi032.stdout:7/73: mknod d16/c17 0 2024-04-06T14:16:56.574 INFO:tasks.workunit.client.0.smithi032.stdout:4/221: mkdir d0/df/d2b/d4f 0 2024-04-06T14:16:56.575 INFO:tasks.workunit.client.0.smithi032.stdout:2/80: write d0/ff [1259106,56986] 0 2024-04-06T14:16:56.575 INFO:tasks.workunit.client.0.smithi032.stdout:7/74: symlink d16/l18 0 2024-04-06T14:16:56.576 INFO:tasks.workunit.client.0.smithi032.stdout:6/134: truncate d4/d5/d8/fc 280266 0 2024-04-06T14:16:56.580 INFO:tasks.workunit.client.0.smithi032.stdout:8/119: sync 2024-04-06T14:16:56.581 INFO:tasks.workunit.client.0.smithi032.stdout:2/81: rename d0/f2 to d0/d12/f1f 0 2024-04-06T14:16:56.583 INFO:tasks.workunit.client.0.smithi032.stdout:1/76: sync 2024-04-06T14:16:56.583 INFO:tasks.workunit.client.0.smithi032.stdout:2/82: chown d0/f1b 16683 1 2024-04-06T14:16:56.583 INFO:tasks.workunit.client.0.smithi032.stdout:3/85: dwrite d3/fa [0,4194304] 0 2024-04-06T14:16:56.585 INFO:tasks.workunit.client.0.smithi032.stdout:4/222: rmdir d0/d35 0 2024-04-06T14:16:56.585 INFO:tasks.workunit.client.0.smithi032.stdout:2/83: dread - d0/d12/f17 zero size 2024-04-06T14:16:56.586 INFO:tasks.workunit.client.0.smithi032.stdout:2/84: stat d0/f1b 0 2024-04-06T14:16:56.588 INFO:tasks.workunit.client.0.smithi032.stdout:6/135: dread d4/f10 [0,4194304] 0 2024-04-06T14:16:56.589 INFO:tasks.workunit.client.0.smithi032.stdout:4/223: unlink d0/f1c 0 2024-04-06T14:16:56.590 INFO:tasks.workunit.client.0.smithi032.stdout:6/136: write d4/d5/d14/f1a [879108,121184] 0 2024-04-06T14:16:56.593 INFO:tasks.workunit.client.0.smithi032.stdout:2/85: creat d0/f20 x:0 0 0 2024-04-06T14:16:56.595 INFO:tasks.workunit.client.0.smithi032.stdout:3/86: dwrite d3/f19 [0,4194304] 0 2024-04-06T14:16:56.596 INFO:tasks.workunit.client.0.smithi032.stdout:4/224: truncate d0/df/d1f/d2e/f49 342863 0 2024-04-06T14:16:56.598 INFO:tasks.workunit.client.0.smithi032.stdout:2/86: creat d0/d12/f21 x:0 0 0 2024-04-06T14:16:56.598 INFO:tasks.workunit.client.0.smithi032.stdout:3/87: mknod d3/d16/d1e/c23 0 2024-04-06T14:16:56.599 INFO:tasks.workunit.client.0.smithi032.stdout:3/88: write d3/f19 [4103905,117969] 0 2024-04-06T14:16:56.602 INFO:tasks.workunit.client.0.smithi032.stdout:7/75: getdents d16 0 2024-04-06T14:16:56.603 INFO:tasks.workunit.client.0.smithi032.stdout:6/137: dread d4/d5/d8/fc [0,4194304] 0 2024-04-06T14:16:56.605 INFO:tasks.workunit.client.0.smithi032.stdout:2/87: creat d0/f22 x:0 0 0 2024-04-06T14:16:56.605 INFO:tasks.workunit.client.0.smithi032.stdout:2/88: dread - d0/d12/f17 zero size 2024-04-06T14:16:56.608 INFO:tasks.workunit.client.0.smithi032.stdout:7/76: creat d16/f19 x:0 0 0 2024-04-06T14:16:56.611 INFO:tasks.workunit.client.0.smithi032.stdout:8/120: write d3/f9 [2207181,85136] 0 2024-04-06T14:16:56.613 INFO:tasks.workunit.client.0.smithi032.stdout:6/138: mkdir d4/d5/d18/d23 0 2024-04-06T14:16:56.615 INFO:tasks.workunit.client.0.smithi032.stdout:2/89: link d0/ff d0/f23 0 2024-04-06T14:16:56.616 INFO:tasks.workunit.client.0.smithi032.stdout:3/89: creat d3/f24 x:0 0 0 2024-04-06T14:16:56.616 INFO:tasks.workunit.client.0.smithi032.stdout:1/77: truncate d0/f2 1051573 0 2024-04-06T14:16:56.617 INFO:tasks.workunit.client.0.smithi032.stdout:3/90: write d3/f21 [177552,63747] 0 2024-04-06T14:16:56.622 INFO:tasks.workunit.client.0.smithi032.stdout:7/77: link d16/l18 d16/l1a 0 2024-04-06T14:16:56.626 INFO:tasks.workunit.client.0.smithi032.stdout:2/90: link d0/f13 d0/f24 0 2024-04-06T14:16:56.627 INFO:tasks.workunit.client.0.smithi032.stdout:3/91: dwrite d3/f24 [0,4194304] 0 2024-04-06T14:16:56.629 INFO:tasks.workunit.client.0.smithi032.stdout:4/225: dwrite d0/df/d23/d31/f4d [0,4194304] 0 2024-04-06T14:16:56.630 INFO:tasks.workunit.client.0.smithi032.stdout:2/91: truncate d0/d12/f1f 2310874 0 2024-04-06T14:16:56.631 INFO:tasks.workunit.client.0.smithi032.stdout:2/92: chown d0/l9 252079 1 2024-04-06T14:16:56.632 INFO:tasks.workunit.client.0.smithi032.stdout:3/92: creat d3/d16/f25 x:0 0 0 2024-04-06T14:16:56.636 INFO:tasks.workunit.client.0.smithi032.stdout:4/226: unlink d0/df/d23/d31/f4d 0 2024-04-06T14:16:56.637 INFO:tasks.workunit.client.0.smithi032.stdout:8/121: truncate d3/d8/d7/d14/f1f 3345154 0 2024-04-06T14:16:56.642 INFO:tasks.workunit.client.0.smithi032.stdout:8/122: chown d3/d8/d12/c20 170 1 2024-04-06T14:16:56.642 INFO:tasks.workunit.client.0.smithi032.stdout:4/227: mkdir d0/df/d23/d31/d50 0 2024-04-06T14:16:56.644 INFO:tasks.workunit.client.0.smithi032.stdout:7/78: truncate fb 2737523 0 2024-04-06T14:16:56.646 INFO:tasks.workunit.client.0.smithi032.stdout:4/228: creat d0/df/d1f/f51 x:0 0 0 2024-04-06T14:16:56.646 INFO:tasks.workunit.client.0.smithi032.stdout:6/139: dwrite d4/d5/f15 [0,4194304] 0 2024-04-06T14:16:56.646 INFO:tasks.workunit.client.0.smithi032.stdout:1/78: write d0/f2 [320721,42882] 0 2024-04-06T14:16:56.647 INFO:tasks.workunit.client.0.smithi032.stdout:4/229: chown d0/l22 324 1 2024-04-06T14:16:56.648 INFO:tasks.workunit.client.0.smithi032.stdout:6/140: write d4/d5/f15 [1756925,97167] 0 2024-04-06T14:16:56.649 INFO:tasks.workunit.client.0.smithi032.stdout:7/79: symlink d16/l1b 0 2024-04-06T14:16:56.649 INFO:tasks.workunit.client.0.smithi032.stdout:1/79: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:56.653 INFO:tasks.workunit.client.0.smithi032.stdout:7/80: rename d16/c17 to d16/c1c 0 2024-04-06T14:16:56.657 INFO:tasks.workunit.client.0.smithi032.stdout:1/80: link d0/da/l18 d0/d9/l1a 0 2024-04-06T14:16:56.658 INFO:tasks.workunit.client.0.smithi032.stdout:8/123: read d3/d8/d7/d14/f1f [104744,37633] 0 2024-04-06T14:16:56.660 INFO:tasks.workunit.client.0.smithi032.stdout:2/93: write d0/fc [3817869,87670] 0 2024-04-06T14:16:56.661 INFO:tasks.workunit.client.0.smithi032.stdout:4/230: dwrite d0/df/d1f/f25 [0,4194304] 0 2024-04-06T14:16:56.663 INFO:tasks.workunit.client.0.smithi032.stdout:4/231: dread - d0/df/d23/f40 zero size 2024-04-06T14:16:56.663 INFO:tasks.workunit.client.0.smithi032.stdout:8/124: creat d3/f31 x:0 0 0 2024-04-06T14:16:56.663 INFO:tasks.workunit.client.0.smithi032.stdout:2/94: dread - d0/f22 zero size 2024-04-06T14:16:56.668 INFO:tasks.workunit.client.0.smithi032.stdout:8/125: chown d3/f31 134111818 1 2024-04-06T14:16:56.669 INFO:tasks.workunit.client.0.smithi032.stdout:4/232: truncate d0/df/f18 4488511 0 2024-04-06T14:16:56.669 INFO:tasks.workunit.client.0.smithi032.stdout:7/81: dwrite ff [0,4194304] 0 2024-04-06T14:16:56.672 INFO:tasks.workunit.client.0.smithi032.stdout:7/82: dread - d16/f19 zero size 2024-04-06T14:16:56.672 INFO:tasks.workunit.client.0.smithi032.stdout:4/233: fdatasync d0/f3a 0 2024-04-06T14:16:56.674 INFO:tasks.workunit.client.0.smithi032.stdout:2/95: getdents d0 0 2024-04-06T14:16:56.676 INFO:tasks.workunit.client.0.smithi032.stdout:1/81: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:16:56.677 INFO:tasks.workunit.client.0.smithi032.stdout:8/126: link d3/d8/fc d3/f32 0 2024-04-06T14:16:56.678 INFO:tasks.workunit.client.0.smithi032.stdout:2/96: creat d0/d12/f25 x:0 0 0 2024-04-06T14:16:56.679 INFO:tasks.workunit.client.0.smithi032.stdout:8/127: dread - d3/f29 zero size 2024-04-06T14:16:56.681 INFO:tasks.workunit.client.0.smithi032.stdout:4/234: mkdir d0/df/d2b/d4f/d52 0 2024-04-06T14:16:56.682 INFO:tasks.workunit.client.0.smithi032.stdout:1/82: truncate d0/f17 1185586 0 2024-04-06T14:16:56.682 INFO:tasks.workunit.client.0.smithi032.stdout:2/97: symlink d0/l26 0 2024-04-06T14:16:56.686 INFO:tasks.workunit.client.0.smithi032.stdout:7/83: dwrite ff [0,4194304] 0 2024-04-06T14:16:56.687 INFO:tasks.workunit.client.0.smithi032.stdout:1/83: dread d0/f17 [0,4194304] 0 2024-04-06T14:16:56.688 INFO:tasks.workunit.client.0.smithi032.stdout:7/84: chown l5 604504824 1 2024-04-06T14:16:56.689 INFO:tasks.workunit.client.0.smithi032.stdout:4/235: creat d0/df/d1f/d42/d45/f53 x:0 0 0 2024-04-06T14:16:56.689 INFO:tasks.workunit.client.0.smithi032.stdout:7/85: chown fd 2 1 2024-04-06T14:16:56.689 INFO:tasks.workunit.client.0.smithi032.stdout:8/128: link d3/d8/d12/l27 d3/d8/l33 0 2024-04-06T14:16:56.689 INFO:tasks.workunit.client.0.smithi032.stdout:8/129: fsync d3/f9 0 2024-04-06T14:16:56.690 INFO:tasks.workunit.client.0.smithi032.stdout:1/84: creat d0/d9/f1b x:0 0 0 2024-04-06T14:16:56.690 INFO:tasks.workunit.client.0.smithi032.stdout:2/98: dread d0/ff [0,4194304] 0 2024-04-06T14:16:56.691 INFO:tasks.workunit.client.0.smithi032.stdout:7/86: mknod d16/c1d 0 2024-04-06T14:16:56.692 INFO:tasks.workunit.client.0.smithi032.stdout:4/236: mknod d0/df/d2b/d43/c54 0 2024-04-06T14:16:56.692 INFO:tasks.workunit.client.0.smithi032.stdout:8/130: chown d3/d8/d7/d14/f1f 1042 1 2024-04-06T14:16:56.692 INFO:tasks.workunit.client.0.smithi032.stdout:1/85: rename d0/l11 to d0/l1c 0 2024-04-06T14:16:56.693 INFO:tasks.workunit.client.0.smithi032.stdout:8/131: truncate d3/f31 1016364 0 2024-04-06T14:16:56.693 INFO:tasks.workunit.client.0.smithi032.stdout:2/99: creat d0/f27 x:0 0 0 2024-04-06T14:16:56.694 INFO:tasks.workunit.client.0.smithi032.stdout:7/87: mknod d16/c1e 0 2024-04-06T14:16:56.695 INFO:tasks.workunit.client.0.smithi032.stdout:4/237: readlink d0/lb 0 2024-04-06T14:16:56.700 INFO:tasks.workunit.client.0.smithi032.stdout:4/238: mknod d0/df/d23/d31/d50/c55 0 2024-04-06T14:16:56.702 INFO:tasks.workunit.client.0.smithi032.stdout:8/132: dread d3/d8/fc [0,4194304] 0 2024-04-06T14:16:56.704 INFO:tasks.workunit.client.0.smithi032.stdout:2/100: dwrite d0/d12/f25 [0,4194304] 0 2024-04-06T14:16:56.704 INFO:tasks.workunit.client.0.smithi032.stdout:8/133: write d3/f9 [2009078,46710] 0 2024-04-06T14:16:56.706 INFO:tasks.workunit.client.0.smithi032.stdout:4/239: write d0/df/d23/f3d [796130,56906] 0 2024-04-06T14:16:56.706 INFO:tasks.workunit.client.0.smithi032.stdout:2/101: chown d0/c10 3832 1 2024-04-06T14:16:56.707 INFO:tasks.workunit.client.0.smithi032.stdout:8/134: truncate d3/f29 53800 0 2024-04-06T14:16:56.708 INFO:tasks.workunit.client.0.smithi032.stdout:4/240: write d0/df/d2b/f3b [159411,81426] 0 2024-04-06T14:16:56.710 INFO:tasks.workunit.client.0.smithi032.stdout:4/241: chown d0/df/d23 61 1 2024-04-06T14:16:56.710 INFO:tasks.workunit.client.0.smithi032.stdout:7/88: fdatasync ff 0 2024-04-06T14:16:56.710 INFO:tasks.workunit.client.0.smithi032.stdout:2/102: rename d0/f15 to d0/f28 0 2024-04-06T14:16:56.711 INFO:tasks.workunit.client.0.smithi032.stdout:4/242: chown d0/df/c2a 520139 1 2024-04-06T14:16:56.713 INFO:tasks.workunit.client.0.smithi032.stdout:1/86: truncate d0/d9/fd 8624425 0 2024-04-06T14:16:56.715 INFO:tasks.workunit.client.0.smithi032.stdout:2/103: unlink d0/c1d 0 2024-04-06T14:16:56.719 INFO:tasks.workunit.client.0.smithi032.stdout:4/243: rename d0/l10 to d0/df/d1f/l56 0 2024-04-06T14:16:56.721 INFO:tasks.workunit.client.0.smithi032.stdout:8/135: dwrite d3/d8/d7/d1a/f30 [0,4194304] 0 2024-04-06T14:16:56.721 INFO:tasks.workunit.client.0.smithi032.stdout:7/89: dread fd [0,4194304] 0 2024-04-06T14:16:56.722 INFO:tasks.workunit.client.0.smithi032.stdout:2/104: mknod d0/c29 0 2024-04-06T14:16:56.723 INFO:tasks.workunit.client.0.smithi032.stdout:4/244: dread d0/df/d1f/d2e/f49 [0,4194304] 0 2024-04-06T14:16:56.723 INFO:tasks.workunit.client.0.smithi032.stdout:1/87: unlink d0/d9/cf 0 2024-04-06T14:16:56.726 INFO:tasks.workunit.client.0.smithi032.stdout:7/90: rmdir d16 39 2024-04-06T14:16:56.727 INFO:tasks.workunit.client.0.smithi032.stdout:8/136: symlink d3/d8/l34 0 2024-04-06T14:16:56.728 INFO:tasks.workunit.client.0.smithi032.stdout:8/137: write d3/d8/f2e [393162,24008] 0 2024-04-06T14:16:56.728 INFO:tasks.workunit.client.0.smithi032.stdout:2/105: rename d0/f27 to d0/d12/f2a 0 2024-04-06T14:16:56.732 INFO:tasks.workunit.client.0.smithi032.stdout:4/245: mkdir d0/df/d23/d31/d50/d57 0 2024-04-06T14:16:56.734 INFO:tasks.workunit.client.0.smithi032.stdout:8/138: creat d3/d8/f35 x:0 0 0 2024-04-06T14:16:56.735 INFO:tasks.workunit.client.0.smithi032.stdout:2/106: mknod d0/d12/c2b 0 2024-04-06T14:16:56.736 INFO:tasks.workunit.client.0.smithi032.stdout:2/107: dread - d0/d12/f21 zero size 2024-04-06T14:16:56.736 INFO:tasks.workunit.client.0.smithi032.stdout:1/88: mknod d0/d10/d16/c1d 0 2024-04-06T14:16:56.737 INFO:tasks.workunit.client.0.smithi032.stdout:1/89: chown d0/f17 1008520 1 2024-04-06T14:16:56.737 INFO:tasks.workunit.client.0.smithi032.stdout:1/90: chown d0/da 5588 1 2024-04-06T14:16:56.738 INFO:tasks.workunit.client.0.smithi032.stdout:5/49: sync 2024-04-06T14:16:56.738 INFO:tasks.workunit.client.0.smithi032.stdout:0/58: sync 2024-04-06T14:16:56.738 INFO:tasks.workunit.client.0.smithi032.stdout:9/115: sync 2024-04-06T14:16:56.738 INFO:tasks.workunit.client.0.smithi032.stdout:7/91: creat d16/f1f x:0 0 0 2024-04-06T14:16:56.739 INFO:tasks.workunit.client.0.smithi032.stdout:4/246: truncate d0/df/d2b/f2c 989476 0 2024-04-06T14:16:56.739 INFO:tasks.workunit.client.0.smithi032.stdout:5/50: fsync f1 0 2024-04-06T14:16:56.740 INFO:tasks.workunit.client.0.smithi032.stdout:5/51: chown fa 0 1 2024-04-06T14:16:56.742 INFO:tasks.workunit.client.0.smithi032.stdout:0/59: unlink l1 0 2024-04-06T14:16:56.743 INFO:tasks.workunit.client.0.smithi032.stdout:8/139: symlink d3/d8/d12/d2f/l36 0 2024-04-06T14:16:56.744 INFO:tasks.workunit.client.0.smithi032.stdout:7/92: mkdir d16/d20 0 2024-04-06T14:16:56.745 INFO:tasks.workunit.client.0.smithi032.stdout:7/93: readlink l5 0 2024-04-06T14:16:56.746 INFO:tasks.workunit.client.0.smithi032.stdout:8/140: dread - d3/d8/f35 zero size 2024-04-06T14:16:56.746 INFO:tasks.workunit.client.0.smithi032.stdout:4/247: fsync d0/df/d1f/f34 0 2024-04-06T14:16:56.748 INFO:tasks.workunit.client.0.smithi032.stdout:7/94: chown f6 139 1 2024-04-06T14:16:56.748 INFO:tasks.workunit.client.0.smithi032.stdout:5/52: creat fd x:0 0 0 2024-04-06T14:16:56.748 INFO:tasks.workunit.client.0.smithi032.stdout:4/248: stat d0/f1 0 2024-04-06T14:16:56.749 INFO:tasks.workunit.client.0.smithi032.stdout:0/60: rename d3/f6 to d3/da/f13 0 2024-04-06T14:16:56.749 INFO:tasks.workunit.client.0.smithi032.stdout:2/108: rmdir d0 39 2024-04-06T14:16:56.751 INFO:tasks.workunit.client.0.smithi032.stdout:5/53: truncate f7 549573 0 2024-04-06T14:16:56.752 INFO:tasks.workunit.client.0.smithi032.stdout:8/141: symlink d3/d8/d12/l37 0 2024-04-06T14:16:56.753 INFO:tasks.workunit.client.0.smithi032.stdout:9/116: dwrite d2/f21 [0,4194304] 0 2024-04-06T14:16:56.754 INFO:tasks.workunit.client.0.smithi032.stdout:8/142: write d3/f29 [34504,112784] 0 2024-04-06T14:16:56.754 INFO:tasks.workunit.client.0.smithi032.stdout:2/109: chown d0 12833 1 2024-04-06T14:16:56.757 INFO:tasks.workunit.client.0.smithi032.stdout:4/249: unlink d0/df/d1f/d2e/f3f 0 2024-04-06T14:16:56.759 INFO:tasks.workunit.client.0.smithi032.stdout:8/143: write d3/d8/d12/f19 [527358,127373] 0 2024-04-06T14:16:56.760 INFO:tasks.workunit.client.0.smithi032.stdout:3/93: sync 2024-04-06T14:16:56.760 INFO:tasks.workunit.client.0.smithi032.stdout:6/141: sync 2024-04-06T14:16:56.760 INFO:tasks.workunit.client.0.smithi032.stdout:3/94: readlink d3/d16/l1b 0 2024-04-06T14:16:56.760 INFO:tasks.workunit.client.0.smithi032.stdout:5/54: rename fd to fe 0 2024-04-06T14:16:56.760 INFO:tasks.workunit.client.0.smithi032.stdout:3/95: stat d3/d16/f22 0 2024-04-06T14:16:56.761 INFO:tasks.workunit.client.0.smithi032.stdout:1/91: getdents d0 0 2024-04-06T14:16:56.762 INFO:tasks.workunit.client.0.smithi032.stdout:4/250: dread d0/f1 [0,4194304] 0 2024-04-06T14:16:56.763 INFO:tasks.workunit.client.0.smithi032.stdout:0/61: rename d3/da/f13 to d3/f14 0 2024-04-06T14:16:56.763 INFO:tasks.workunit.client.0.smithi032.stdout:4/251: write d0/df/d2b/f3b [357742,85107] 0 2024-04-06T14:16:56.764 INFO:tasks.workunit.client.0.smithi032.stdout:9/117: fsync d2/fd 0 2024-04-06T14:16:56.770 INFO:tasks.workunit.client.0.smithi032.stdout:5/55: creat ff x:0 0 0 2024-04-06T14:16:56.773 INFO:tasks.workunit.client.0.smithi032.stdout:1/92: creat d0/f1e x:0 0 0 2024-04-06T14:16:56.777 INFO:tasks.workunit.client.0.smithi032.stdout:4/252: mkdir d0/df/d2b/d46/d58 0 2024-04-06T14:16:56.778 INFO:tasks.workunit.client.0.smithi032.stdout:9/118: dwrite d2/d15/d1e/f24 [0,4194304] 0 2024-04-06T14:16:56.781 INFO:tasks.workunit.client.0.smithi032.stdout:2/110: dwrite d0/f28 [0,4194304] 0 2024-04-06T14:16:56.783 INFO:tasks.workunit.client.0.smithi032.stdout:6/142: truncate d4/d5/d14/f1a 909579 0 2024-04-06T14:16:56.783 INFO:tasks.workunit.client.0.smithi032.stdout:3/96: creat d3/f26 x:0 0 0 2024-04-06T14:16:56.786 INFO:tasks.workunit.client.0.smithi032.stdout:8/144: dwrite d3/f32 [0,4194304] 0 2024-04-06T14:16:56.790 INFO:tasks.workunit.client.0.smithi032.stdout:4/253: dread - d0/df/d1f/d42/d45/f53 zero size 2024-04-06T14:16:56.791 INFO:tasks.workunit.client.0.smithi032.stdout:2/111: fdatasync d0/f22 0 2024-04-06T14:16:56.794 INFO:tasks.workunit.client.0.smithi032.stdout:0/62: truncate d3/f14 1178789 0 2024-04-06T14:16:56.794 INFO:tasks.workunit.client.0.smithi032.stdout:6/143: write d4/f13 [3471606,49912] 0 2024-04-06T14:16:56.797 INFO:tasks.workunit.client.0.smithi032.stdout:1/93: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:56.798 INFO:tasks.workunit.client.0.smithi032.stdout:3/97: symlink d3/d16/d1e/l27 0 2024-04-06T14:16:56.800 INFO:tasks.workunit.client.0.smithi032.stdout:2/112: creat d0/d12/f2c x:0 0 0 2024-04-06T14:16:56.800 INFO:tasks.workunit.client.0.smithi032.stdout:5/56: getdents . 0 2024-04-06T14:16:56.801 INFO:tasks.workunit.client.0.smithi032.stdout:8/145: rename d3/f9 to d3/d8/d12/f38 0 2024-04-06T14:16:56.802 INFO:tasks.workunit.client.0.smithi032.stdout:3/98: dread - d3/f20 zero size 2024-04-06T14:16:56.802 INFO:tasks.workunit.client.0.smithi032.stdout:1/94: dread - d0/d9/f1b zero size 2024-04-06T14:16:56.802 INFO:tasks.workunit.client.0.smithi032.stdout:4/254: symlink d0/df/d2b/d4f/l59 0 2024-04-06T14:16:56.805 INFO:tasks.workunit.client.0.smithi032.stdout:9/119: dread d2/d15/d1e/f24 [0,4194304] 0 2024-04-06T14:16:56.806 INFO:tasks.workunit.client.0.smithi032.stdout:5/57: creat f10 x:0 0 0 2024-04-06T14:16:56.808 INFO:tasks.workunit.client.0.smithi032.stdout:8/146: dread d3/d8/d7/fe [0,4194304] 0 2024-04-06T14:16:56.809 INFO:tasks.workunit.client.0.smithi032.stdout:5/58: chown fc 92200816 1 2024-04-06T14:16:56.809 INFO:tasks.workunit.client.0.smithi032.stdout:4/255: mkdir d0/df/d1f/d5a 0 2024-04-06T14:16:56.810 INFO:tasks.workunit.client.0.smithi032.stdout:6/144: dwrite d4/f13 [0,4194304] 0 2024-04-06T14:16:56.810 INFO:tasks.workunit.client.0.smithi032.stdout:9/120: mkdir d2/d26 0 2024-04-06T14:16:56.811 INFO:tasks.workunit.client.0.smithi032.stdout:5/59: chown fc 51 1 2024-04-06T14:16:56.814 INFO:tasks.workunit.client.0.smithi032.stdout:0/63: getdents d3/da 0 2024-04-06T14:16:56.816 INFO:tasks.workunit.client.0.smithi032.stdout:5/60: creat f11 x:0 0 0 2024-04-06T14:16:56.817 INFO:tasks.workunit.client.0.smithi032.stdout:8/147: creat d3/d8/d12/d2f/f39 x:0 0 0 2024-04-06T14:16:56.818 INFO:tasks.workunit.client.0.smithi032.stdout:9/121: write d2/d5/f6 [3713317,78716] 0 2024-04-06T14:16:56.818 INFO:tasks.workunit.client.0.smithi032.stdout:1/95: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:56.819 INFO:tasks.workunit.client.0.smithi032.stdout:2/113: dread d0/f23 [0,4194304] 0 2024-04-06T14:16:56.819 INFO:tasks.workunit.client.0.smithi032.stdout:0/64: unlink d3/f5 0 2024-04-06T14:16:56.820 INFO:tasks.workunit.client.0.smithi032.stdout:0/65: stat d3/da/fe 0 2024-04-06T14:16:56.821 INFO:tasks.workunit.client.0.smithi032.stdout:0/66: fsync d3/f9 0 2024-04-06T14:16:56.822 INFO:tasks.workunit.client.0.smithi032.stdout:3/99: dread d3/f4 [0,4194304] 0 2024-04-06T14:16:56.823 INFO:tasks.workunit.client.0.smithi032.stdout:5/61: creat f12 x:0 0 0 2024-04-06T14:16:56.823 INFO:tasks.workunit.client.0.smithi032.stdout:8/148: mkdir d3/d8/d3a 0 2024-04-06T14:16:56.824 INFO:tasks.workunit.client.0.smithi032.stdout:3/100: rename d3 to d3/d16/d28 22 2024-04-06T14:16:56.824 INFO:tasks.workunit.client.0.smithi032.stdout:6/145: symlink d4/d5/d8/d1f/l24 0 2024-04-06T14:16:56.824 INFO:tasks.workunit.client.0.smithi032.stdout:9/122: dread d2/ff [0,4194304] 0 2024-04-06T14:16:56.825 INFO:tasks.workunit.client.0.smithi032.stdout:1/96: mkdir d0/d9/d1f 0 2024-04-06T14:16:56.825 INFO:tasks.workunit.client.0.smithi032.stdout:2/114: mknod d0/c2d 0 2024-04-06T14:16:56.826 INFO:tasks.workunit.client.0.smithi032.stdout:4/256: mkdir d0/df/d23/d31/d50/d57/d5b 0 2024-04-06T14:16:56.826 INFO:tasks.workunit.client.0.smithi032.stdout:0/67: truncate d3/da/fe 633937 0 2024-04-06T14:16:56.826 INFO:tasks.workunit.client.0.smithi032.stdout:5/62: mknod c13 0 2024-04-06T14:16:56.827 INFO:tasks.workunit.client.0.smithi032.stdout:6/146: chown c1 8633712 1 2024-04-06T14:16:56.827 INFO:tasks.workunit.client.0.smithi032.stdout:1/97: chown d0/l15 7 1 2024-04-06T14:16:56.829 INFO:tasks.workunit.client.0.smithi032.stdout:6/147: fdatasync d4/d5/f15 0 2024-04-06T14:16:56.830 INFO:tasks.workunit.client.0.smithi032.stdout:8/149: creat d3/d8/d3a/f3b x:0 0 0 2024-04-06T14:16:56.830 INFO:tasks.workunit.client.0.smithi032.stdout:2/115: symlink d0/l2e 0 2024-04-06T14:16:56.831 INFO:tasks.workunit.client.0.smithi032.stdout:5/63: mknod c14 0 2024-04-06T14:16:56.831 INFO:tasks.workunit.client.0.smithi032.stdout:5/64: rmdir - no directory 2024-04-06T14:16:56.832 INFO:tasks.workunit.client.0.smithi032.stdout:0/68: symlink d3/l15 0 2024-04-06T14:16:56.833 INFO:tasks.workunit.client.0.smithi032.stdout:6/148: chown d4/d5/d8/fc 40593428 1 2024-04-06T14:16:56.834 INFO:tasks.workunit.client.0.smithi032.stdout:0/69: chown d3/f4 26455 1 2024-04-06T14:16:56.834 INFO:tasks.workunit.client.0.smithi032.stdout:8/150: mkdir d3/d8/d3a/d3c 0 2024-04-06T14:16:56.835 INFO:tasks.workunit.client.0.smithi032.stdout:6/149: read d4/f13 [4515729,2640] 0 2024-04-06T14:16:56.836 INFO:tasks.workunit.client.0.smithi032.stdout:5/65: write f12 [512967,44011] 0 2024-04-06T14:16:56.836 INFO:tasks.workunit.client.0.smithi032.stdout:8/151: readlink d3/l5 0 2024-04-06T14:16:56.837 INFO:tasks.workunit.client.0.smithi032.stdout:9/123: write d2/d15/f1a [1086104,9776] 0 2024-04-06T14:16:56.838 INFO:tasks.workunit.client.0.smithi032.stdout:3/101: dread d3/d16/d1e/f1f [0,4194304] 0 2024-04-06T14:16:56.840 INFO:tasks.workunit.client.0.smithi032.stdout:9/124: fsync d2/d5/f18 0 2024-04-06T14:16:56.841 INFO:tasks.workunit.client.0.smithi032.stdout:5/66: symlink l15 0 2024-04-06T14:16:56.841 INFO:tasks.workunit.client.0.smithi032.stdout:5/67: rmdir - no directory 2024-04-06T14:16:56.842 INFO:tasks.workunit.client.0.smithi032.stdout:0/70: rmdir d3 39 2024-04-06T14:16:56.842 INFO:tasks.workunit.client.0.smithi032.stdout:6/150: fdatasync d4/f13 0 2024-04-06T14:16:56.846 INFO:tasks.workunit.client.0.smithi032.stdout:4/257: dwrite d0/df/d23/f3c [0,4194304] 0 2024-04-06T14:16:56.846 INFO:tasks.workunit.client.0.smithi032.stdout:2/116: link d0/l9 d0/l2f 0 2024-04-06T14:16:56.846 INFO:tasks.workunit.client.0.smithi032.stdout:6/151: chown c1 280 1 2024-04-06T14:16:56.846 INFO:tasks.workunit.client.0.smithi032.stdout:1/98: truncate d0/f17 596900 0 2024-04-06T14:16:56.846 INFO:tasks.workunit.client.0.smithi032.stdout:2/117: read d0/d12/f1c [3500290,116439] 0 2024-04-06T14:16:56.847 INFO:tasks.workunit.client.0.smithi032.stdout:6/152: stat l3 0 2024-04-06T14:16:56.847 INFO:tasks.workunit.client.0.smithi032.stdout:9/125: creat d2/d15/f27 x:0 0 0 2024-04-06T14:16:56.848 INFO:tasks.workunit.client.0.smithi032.stdout:5/68: mkdir d16 0 2024-04-06T14:16:56.851 INFO:tasks.workunit.client.0.smithi032.stdout:7/95: sync 2024-04-06T14:16:56.853 INFO:tasks.workunit.client.0.smithi032.stdout:7/96: chown f7 2177 1 2024-04-06T14:16:56.854 INFO:tasks.workunit.client.0.smithi032.stdout:2/118: mkdir d0/d30 0 2024-04-06T14:16:56.855 INFO:tasks.workunit.client.0.smithi032.stdout:7/97: chown d16/l1b 316 1 2024-04-06T14:16:56.856 INFO:tasks.workunit.client.0.smithi032.stdout:7/98: fdatasync f7 0 2024-04-06T14:16:56.856 INFO:tasks.workunit.client.0.smithi032.stdout:0/71: write d3/da/fe [149966,20136] 0 2024-04-06T14:16:56.857 INFO:tasks.workunit.client.0.smithi032.stdout:3/102: mknod d3/c29 0 2024-04-06T14:16:56.859 INFO:tasks.workunit.client.0.smithi032.stdout:0/72: read - d3/fd zero size 2024-04-06T14:16:56.861 INFO:tasks.workunit.client.0.smithi032.stdout:8/152: dwrite d3/d8/f35 [0,4194304] 0 2024-04-06T14:16:56.865 INFO:tasks.workunit.client.0.smithi032.stdout:7/99: rename f13 to d16/f21 0 2024-04-06T14:16:56.866 INFO:tasks.workunit.client.0.smithi032.stdout:5/69: creat d16/f17 x:0 0 0 2024-04-06T14:16:56.867 INFO:tasks.workunit.client.0.smithi032.stdout:6/153: dread d4/d5/f15 [0,4194304] 0 2024-04-06T14:16:56.867 INFO:tasks.workunit.client.0.smithi032.stdout:4/258: dread d0/df/d1f/f25 [0,4194304] 0 2024-04-06T14:16:56.870 INFO:tasks.workunit.client.0.smithi032.stdout:7/100: chown fa 33821030 1 2024-04-06T14:16:56.870 INFO:tasks.workunit.client.0.smithi032.stdout:5/70: chown l8 1807 1 2024-04-06T14:16:56.871 INFO:tasks.workunit.client.0.smithi032.stdout:3/103: rename d3/f26 to d3/d16/f2a 0 2024-04-06T14:16:56.874 INFO:tasks.workunit.client.0.smithi032.stdout:5/71: chown f9 436401233 1 2024-04-06T14:16:56.875 INFO:tasks.workunit.client.0.smithi032.stdout:9/126: dwrite d2/d5/f14 [0,4194304] 0 2024-04-06T14:16:56.875 INFO:tasks.workunit.client.0.smithi032.stdout:0/73: link d3/fd d3/da/f16 0 2024-04-06T14:16:56.879 INFO:tasks.workunit.client.0.smithi032.stdout:7/101: mkdir d16/d22 0 2024-04-06T14:16:56.882 INFO:tasks.workunit.client.0.smithi032.stdout:5/72: write f1 [26411,30587] 0 2024-04-06T14:16:56.883 INFO:tasks.workunit.client.0.smithi032.stdout:0/74: readlink d3/l15 0 2024-04-06T14:16:56.884 INFO:tasks.workunit.client.0.smithi032.stdout:1/99: write d0/f17 [894758,47665] 0 2024-04-06T14:16:56.886 INFO:tasks.workunit.client.0.smithi032.stdout:5/73: chown f3 108129 1 2024-04-06T14:16:56.887 INFO:tasks.workunit.client.0.smithi032.stdout:2/119: dwrite d0/d12/f1c [0,4194304] 0 2024-04-06T14:16:56.888 INFO:tasks.workunit.client.0.smithi032.stdout:4/259: creat d0/f5c x:0 0 0 2024-04-06T14:16:56.890 INFO:tasks.workunit.client.0.smithi032.stdout:0/75: chown f0 60 1 2024-04-06T14:16:56.891 INFO:tasks.workunit.client.0.smithi032.stdout:6/154: dread d4/d5/d14/f1a [0,4194304] 0 2024-04-06T14:16:56.893 INFO:tasks.workunit.client.0.smithi032.stdout:7/102: rename f3 to d16/d20/f23 0 2024-04-06T14:16:56.894 INFO:tasks.workunit.client.0.smithi032.stdout:9/127: mknod d2/d15/d1e/c28 0 2024-04-06T14:16:56.896 INFO:tasks.workunit.client.0.smithi032.stdout:5/74: mkdir d16/d18 0 2024-04-06T14:16:56.898 INFO:tasks.workunit.client.0.smithi032.stdout:8/153: dwrite d3/d8/d12/f1b [0,4194304] 0 2024-04-06T14:16:56.899 INFO:tasks.workunit.client.0.smithi032.stdout:5/75: dread - d16/f17 zero size 2024-04-06T14:16:56.899 INFO:tasks.workunit.client.0.smithi032.stdout:3/104: link d3/d16/l1b d3/d16/d1e/l2b 0 2024-04-06T14:16:56.899 INFO:tasks.workunit.client.0.smithi032.stdout:2/120: creat d0/d12/f31 x:0 0 0 2024-04-06T14:16:56.900 INFO:tasks.workunit.client.0.smithi032.stdout:6/155: mkdir d4/d5/d19/d25 0 2024-04-06T14:16:56.900 INFO:tasks.workunit.client.0.smithi032.stdout:0/76: unlink d3/da/f16 0 2024-04-06T14:16:56.902 INFO:tasks.workunit.client.0.smithi032.stdout:3/105: chown d3/f20 0 1 2024-04-06T14:16:56.903 INFO:tasks.workunit.client.0.smithi032.stdout:7/103: rename d16/f21 to d16/d20/f24 0 2024-04-06T14:16:56.906 INFO:tasks.workunit.client.0.smithi032.stdout:4/260: unlink d0/ca 0 2024-04-06T14:16:56.912 INFO:tasks.workunit.client.0.smithi032.stdout:2/121: mknod d0/c32 0 2024-04-06T14:16:56.912 INFO:tasks.workunit.client.0.smithi032.stdout:5/76: dwrite f5 [0,4194304] 0 2024-04-06T14:16:56.913 INFO:tasks.workunit.client.0.smithi032.stdout:6/156: fsync d4/d5/d8/fc 0 2024-04-06T14:16:56.914 INFO:tasks.workunit.client.0.smithi032.stdout:0/77: symlink d3/da/l17 0 2024-04-06T14:16:56.921 INFO:tasks.workunit.client.0.smithi032.stdout:3/106: dwrite d3/d16/f18 [0,4194304] 0 2024-04-06T14:16:56.923 INFO:tasks.workunit.client.0.smithi032.stdout:5/77: truncate f11 327597 0 2024-04-06T14:16:56.923 INFO:tasks.workunit.client.0.smithi032.stdout:5/78: stat c6 0 2024-04-06T14:16:56.924 INFO:tasks.workunit.client.0.smithi032.stdout:5/79: readlink l2 0 2024-04-06T14:16:56.925 INFO:tasks.workunit.client.0.smithi032.stdout:9/128: write d2/d15/d1e/f1f [1117882,36034] 0 2024-04-06T14:16:56.925 INFO:tasks.workunit.client.0.smithi032.stdout:1/100: link d0/d10/d16/c1d d0/c20 0 2024-04-06T14:16:56.929 INFO:tasks.workunit.client.0.smithi032.stdout:0/78: creat d3/da/f18 x:0 0 0 2024-04-06T14:16:56.929 INFO:tasks.workunit.client.0.smithi032.stdout:5/80: rename f12 to d16/f19 0 2024-04-06T14:16:56.930 INFO:tasks.workunit.client.0.smithi032.stdout:4/261: symlink d0/df/d23/l5d 0 2024-04-06T14:16:56.930 INFO:tasks.workunit.client.0.smithi032.stdout:8/154: rmdir d3/d8/d3a/d3c 0 2024-04-06T14:16:56.930 INFO:tasks.workunit.client.0.smithi032.stdout:3/107: creat d3/d16/f2c x:0 0 0 2024-04-06T14:16:56.931 INFO:tasks.workunit.client.0.smithi032.stdout:8/155: chown d3/d8/d7/ld 1043514871 1 2024-04-06T14:16:56.933 INFO:tasks.workunit.client.0.smithi032.stdout:9/129: creat d2/d15/f29 x:0 0 0 2024-04-06T14:16:56.933 INFO:tasks.workunit.client.0.smithi032.stdout:9/130: chown d2/f4 249 1 2024-04-06T14:16:56.934 INFO:tasks.workunit.client.0.smithi032.stdout:6/157: fsync d4/d5/d14/f1a 0 2024-04-06T14:16:56.935 INFO:tasks.workunit.client.0.smithi032.stdout:1/101: symlink d0/d10/d16/l21 0 2024-04-06T14:16:56.940 INFO:tasks.workunit.client.0.smithi032.stdout:5/81: unlink c6 0 2024-04-06T14:16:56.942 INFO:tasks.workunit.client.0.smithi032.stdout:5/82: write fa [792772,118457] 0 2024-04-06T14:16:56.944 INFO:tasks.workunit.client.0.smithi032.stdout:0/79: unlink d3/da/l12 0 2024-04-06T14:16:56.945 INFO:tasks.workunit.client.0.smithi032.stdout:0/80: write d3/f10 [44910,105153] 0 2024-04-06T14:16:56.945 INFO:tasks.workunit.client.0.smithi032.stdout:3/108: unlink d3/d16/f22 0 2024-04-06T14:16:56.946 INFO:tasks.workunit.client.0.smithi032.stdout:3/109: fdatasync d3/d16/f18 0 2024-04-06T14:16:56.946 INFO:tasks.workunit.client.0.smithi032.stdout:8/156: mknod d3/d8/d7/c3d 0 2024-04-06T14:16:56.948 INFO:tasks.workunit.client.0.smithi032.stdout:9/131: creat d2/d15/d1e/f2a x:0 0 0 2024-04-06T14:16:56.949 INFO:tasks.workunit.client.0.smithi032.stdout:7/104: getdents d16/d20 0 2024-04-06T14:16:56.951 INFO:tasks.workunit.client.0.smithi032.stdout:4/262: mkdir d0/df/d23/d31/d50/d5e 0 2024-04-06T14:16:56.952 INFO:tasks.workunit.client.0.smithi032.stdout:0/81: creat d3/f19 x:0 0 0 2024-04-06T14:16:56.954 INFO:tasks.workunit.client.0.smithi032.stdout:8/157: mknod d3/d8/c3e 0 2024-04-06T14:16:56.955 INFO:tasks.workunit.client.0.smithi032.stdout:6/158: symlink d4/d5/d19/d1d/l26 0 2024-04-06T14:16:56.958 INFO:tasks.workunit.client.0.smithi032.stdout:1/102: unlink d0/d9/l1a 0 2024-04-06T14:16:56.958 INFO:tasks.workunit.client.0.smithi032.stdout:6/159: chown d4/d5/fb 122468 1 2024-04-06T14:16:56.958 INFO:tasks.workunit.client.0.smithi032.stdout:9/132: dwrite d2/f21 [0,4194304] 0 2024-04-06T14:16:56.959 INFO:tasks.workunit.client.0.smithi032.stdout:5/83: getdents d16/d18 0 2024-04-06T14:16:56.962 INFO:tasks.workunit.client.0.smithi032.stdout:7/105: dwrite f15 [0,4194304] 0 2024-04-06T14:16:56.963 INFO:tasks.workunit.client.0.smithi032.stdout:8/158: creat d3/d8/d12/f3f x:0 0 0 2024-04-06T14:16:56.965 INFO:tasks.workunit.client.0.smithi032.stdout:9/133: creat d2/d15/f2b x:0 0 0 2024-04-06T14:16:56.965 INFO:tasks.workunit.client.0.smithi032.stdout:8/159: readlink d3/d8/d7/lf 0 2024-04-06T14:16:56.966 INFO:tasks.workunit.client.0.smithi032.stdout:4/263: dread - d0/f5c zero size 2024-04-06T14:16:56.966 INFO:tasks.workunit.client.0.smithi032.stdout:3/110: creat d3/f2d x:0 0 0 2024-04-06T14:16:56.970 INFO:tasks.workunit.client.0.smithi032.stdout:3/111: stat d3/d16/d1e/f1f 0 2024-04-06T14:16:56.970 INFO:tasks.workunit.client.0.smithi032.stdout:6/160: write d4/d5/d14/f1a [493430,29652] 0 2024-04-06T14:16:56.971 INFO:tasks.workunit.client.0.smithi032.stdout:7/106: write d16/f19 [289737,59227] 0 2024-04-06T14:16:56.979 INFO:tasks.workunit.client.0.smithi032.stdout:1/103: creat d0/d9/d1f/f22 x:0 0 0 2024-04-06T14:16:56.979 INFO:tasks.workunit.client.0.smithi032.stdout:0/82: getdents d3 0 2024-04-06T14:16:56.983 INFO:tasks.workunit.client.0.smithi032.stdout:3/112: creat d3/d16/f2e x:0 0 0 2024-04-06T14:16:56.983 INFO:tasks.workunit.client.0.smithi032.stdout:1/104: fsync d0/f1e 0 2024-04-06T14:16:56.986 INFO:tasks.workunit.client.0.smithi032.stdout:6/161: write d4/f13 [5615048,34837] 0 2024-04-06T14:16:56.988 INFO:tasks.workunit.client.0.smithi032.stdout:7/107: rmdir d16/d22 0 2024-04-06T14:16:56.991 INFO:tasks.workunit.client.0.smithi032.stdout:8/160: link d3/l6 d3/d8/d12/l40 0 2024-04-06T14:16:56.991 INFO:tasks.workunit.client.0.smithi032.stdout:9/134: dwrite d2/d5/f18 [0,4194304] 0 2024-04-06T14:16:56.992 INFO:tasks.workunit.client.0.smithi032.stdout:1/105: creat d0/d9/f23 x:0 0 0 2024-04-06T14:16:56.993 INFO:tasks.workunit.client.0.smithi032.stdout:3/113: rename d3/c29 to d3/d16/d1e/c2f 0 2024-04-06T14:16:56.994 INFO:tasks.workunit.client.0.smithi032.stdout:0/83: dread d3/da/ff [0,4194304] 0 2024-04-06T14:16:56.994 INFO:tasks.workunit.client.0.smithi032.stdout:5/84: dwrite d16/f19 [0,4194304] 0 2024-04-06T14:16:56.995 INFO:tasks.workunit.client.0.smithi032.stdout:4/264: dwrite d0/df/d1f/f25 [0,4194304] 0 2024-04-06T14:16:57.000 INFO:tasks.workunit.client.0.smithi032.stdout:1/106: truncate d0/d9/f23 1010957 0 2024-04-06T14:16:57.001 INFO:tasks.workunit.client.0.smithi032.stdout:3/114: write d3/f24 [2370390,8693] 0 2024-04-06T14:16:57.002 INFO:tasks.workunit.client.0.smithi032.stdout:7/108: symlink d16/l25 0 2024-04-06T14:16:57.003 INFO:tasks.workunit.client.0.smithi032.stdout:1/107: truncate d0/d9/d1f/f22 446369 0 2024-04-06T14:16:57.003 INFO:tasks.workunit.client.0.smithi032.stdout:0/84: symlink d3/da/l1a 0 2024-04-06T14:16:57.004 INFO:tasks.workunit.client.0.smithi032.stdout:8/161: rmdir d3/d8/d12 39 2024-04-06T14:16:57.007 INFO:tasks.workunit.client.0.smithi032.stdout:6/162: creat d4/d5/d18/d23/f27 x:0 0 0 2024-04-06T14:16:57.007 INFO:tasks.workunit.client.0.smithi032.stdout:7/109: write d16/f19 [1248774,1961] 0 2024-04-06T14:16:57.009 INFO:tasks.workunit.client.0.smithi032.stdout:9/135: rmdir d2/d26 0 2024-04-06T14:16:57.009 INFO:tasks.workunit.client.0.smithi032.stdout:7/110: write f11 [1559152,63151] 0 2024-04-06T14:16:57.010 INFO:tasks.workunit.client.0.smithi032.stdout:1/108: creat d0/d9/f24 x:0 0 0 2024-04-06T14:16:57.012 INFO:tasks.workunit.client.0.smithi032.stdout:9/136: write d2/f21 [407508,120107] 0 2024-04-06T14:16:57.012 INFO:tasks.workunit.client.0.smithi032.stdout:6/163: write d4/d5/d18/d23/f27 [141261,76229] 0 2024-04-06T14:16:57.012 INFO:tasks.workunit.client.0.smithi032.stdout:0/85: mknod d3/da/c1b 0 2024-04-06T14:16:57.013 INFO:tasks.workunit.client.0.smithi032.stdout:8/162: symlink d3/l41 0 2024-04-06T14:16:57.013 INFO:tasks.workunit.client.0.smithi032.stdout:2/122: sync 2024-04-06T14:16:57.015 INFO:tasks.workunit.client.0.smithi032.stdout:7/111: mkdir d16/d20/d26 0 2024-04-06T14:16:57.018 INFO:tasks.workunit.client.0.smithi032.stdout:2/123: mknod d0/c33 0 2024-04-06T14:16:57.020 INFO:tasks.workunit.client.0.smithi032.stdout:4/265: link d0/df/c21 d0/d47/d4b/c5f 0 2024-04-06T14:16:57.020 INFO:tasks.workunit.client.0.smithi032.stdout:8/163: write d3/d8/d12/f1b [474664,75268] 0 2024-04-06T14:16:57.021 INFO:tasks.workunit.client.0.smithi032.stdout:3/115: dread d3/d16/f1a [0,4194304] 0 2024-04-06T14:16:57.021 INFO:tasks.workunit.client.0.smithi032.stdout:2/124: read - d0/f22 zero size 2024-04-06T14:16:57.021 INFO:tasks.workunit.client.0.smithi032.stdout:6/164: dread d4/d5/d14/f1a [0,4194304] 0 2024-04-06T14:16:57.026 INFO:tasks.workunit.client.0.smithi032.stdout:2/125: chown d0/f1b 51 1 2024-04-06T14:16:57.026 INFO:tasks.workunit.client.0.smithi032.stdout:1/109: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:57.026 INFO:tasks.workunit.client.0.smithi032.stdout:4/266: write d0/f44 [124542,85471] 0 2024-04-06T14:16:57.028 INFO:tasks.workunit.client.0.smithi032.stdout:3/116: symlink d3/d16/d1e/l30 0 2024-04-06T14:16:57.032 INFO:tasks.workunit.client.0.smithi032.stdout:0/86: dwrite d3/f10 [0,4194304] 0 2024-04-06T14:16:57.032 INFO:tasks.workunit.client.0.smithi032.stdout:7/112: dread f6 [4194304,4194304] 0 2024-04-06T14:16:57.033 INFO:tasks.workunit.client.0.smithi032.stdout:9/137: dwrite d2/d15/d1e/f1f [0,4194304] 0 2024-04-06T14:16:57.038 INFO:tasks.workunit.client.0.smithi032.stdout:4/267: creat d0/df/d23/f60 x:0 0 0 2024-04-06T14:16:57.039 INFO:tasks.workunit.client.0.smithi032.stdout:7/113: write d16/f19 [1513006,17267] 0 2024-04-06T14:16:57.039 INFO:tasks.workunit.client.0.smithi032.stdout:8/164: dread d3/d8/d7/d14/f1f [0,4194304] 0 2024-04-06T14:16:57.040 INFO:tasks.workunit.client.0.smithi032.stdout:9/138: chown d2/d15/f27 12902391 1 2024-04-06T14:16:57.040 INFO:tasks.workunit.client.0.smithi032.stdout:3/117: rename d3/c10 to d3/d16/d1e/c31 0 2024-04-06T14:16:57.041 INFO:tasks.workunit.client.0.smithi032.stdout:7/114: dread - f12 zero size 2024-04-06T14:16:57.042 INFO:tasks.workunit.client.0.smithi032.stdout:9/139: readlink d2/lb 0 2024-04-06T14:16:57.043 INFO:tasks.workunit.client.0.smithi032.stdout:8/165: readlink d3/d8/d7/ld 0 2024-04-06T14:16:57.045 INFO:tasks.workunit.client.0.smithi032.stdout:0/87: rename d3/l8 to d3/da/l1c 0 2024-04-06T14:16:57.045 INFO:tasks.workunit.client.0.smithi032.stdout:1/110: dwrite d0/d9/f24 [0,4194304] 0 2024-04-06T14:16:57.050 INFO:tasks.workunit.client.0.smithi032.stdout:7/115: creat d16/f27 x:0 0 0 2024-04-06T14:16:57.050 INFO:tasks.workunit.client.0.smithi032.stdout:9/140: mknod d2/d15/c2c 0 2024-04-06T14:16:57.050 INFO:tasks.workunit.client.0.smithi032.stdout:8/166: chown d3/d8/d7/d14/l15 1 1 2024-04-06T14:16:57.050 INFO:tasks.workunit.client.0.smithi032.stdout:2/126: dread d0/d12/f1a [0,4194304] 0 2024-04-06T14:16:57.050 INFO:tasks.workunit.client.0.smithi032.stdout:4/268: creat d0/df/d2b/d4f/d52/f61 x:0 0 0 2024-04-06T14:16:57.051 INFO:tasks.workunit.client.0.smithi032.stdout:1/111: unlink d0/c8 0 2024-04-06T14:16:57.053 INFO:tasks.workunit.client.0.smithi032.stdout:3/118: getdents d3 0 2024-04-06T14:16:57.054 INFO:tasks.workunit.client.0.smithi032.stdout:7/116: rename ff to d16/d20/f28 0 2024-04-06T14:16:57.054 INFO:tasks.workunit.client.0.smithi032.stdout:8/167: write d3/d8/d7/d1a/f30 [3673548,78555] 0 2024-04-06T14:16:57.055 INFO:tasks.workunit.client.0.smithi032.stdout:0/88: link d3/f4 d3/da/f1d 0 2024-04-06T14:16:57.056 INFO:tasks.workunit.client.0.smithi032.stdout:8/168: write d3/d8/f35 [1638141,86420] 0 2024-04-06T14:16:57.056 INFO:tasks.workunit.client.0.smithi032.stdout:1/112: creat d0/d10/f25 x:0 0 0 2024-04-06T14:16:57.058 INFO:tasks.workunit.client.0.smithi032.stdout:1/113: chown d0/d9/f1b 177675 1 2024-04-06T14:16:57.059 INFO:tasks.workunit.client.0.smithi032.stdout:8/169: fdatasync d3/d8/d7/d1a/f24 0 2024-04-06T14:16:57.063 INFO:tasks.workunit.client.0.smithi032.stdout:3/119: creat d3/d16/d1e/f32 x:0 0 0 2024-04-06T14:16:57.067 INFO:tasks.workunit.client.0.smithi032.stdout:9/141: getdents d2/d25 0 2024-04-06T14:16:57.067 INFO:tasks.workunit.client.0.smithi032.stdout:3/120: write d3/f20 [551113,9046] 0 2024-04-06T14:16:57.067 INFO:tasks.workunit.client.0.smithi032.stdout:4/269: mknod d0/df/d2b/d46/d58/c62 0 2024-04-06T14:16:57.068 INFO:tasks.workunit.client.0.smithi032.stdout:2/127: dwrite d0/d12/f1f [0,4194304] 0 2024-04-06T14:16:57.070 INFO:tasks.workunit.client.0.smithi032.stdout:8/170: creat d3/d8/d3a/f42 x:0 0 0 2024-04-06T14:16:57.073 INFO:tasks.workunit.client.0.smithi032.stdout:1/114: dwrite d0/d9/f1b [0,4194304] 0 2024-04-06T14:16:57.075 INFO:tasks.workunit.client.0.smithi032.stdout:3/121: creat d3/d16/f33 x:0 0 0 2024-04-06T14:16:57.075 INFO:tasks.workunit.client.0.smithi032.stdout:1/115: readlink d0/l1c 0 2024-04-06T14:16:57.077 INFO:tasks.workunit.client.0.smithi032.stdout:7/117: dwrite fd [0,4194304] 0 2024-04-06T14:16:57.077 INFO:tasks.workunit.client.0.smithi032.stdout:4/270: unlink d0/l8 0 2024-04-06T14:16:57.081 INFO:tasks.workunit.client.0.smithi032.stdout:2/128: rename d0/f20 to d0/d30/f34 0 2024-04-06T14:16:57.082 INFO:tasks.workunit.client.0.smithi032.stdout:9/142: creat d2/f2d x:0 0 0 2024-04-06T14:16:57.082 INFO:tasks.workunit.client.0.smithi032.stdout:3/122: creat d3/d16/d1e/f34 x:0 0 0 2024-04-06T14:16:57.085 INFO:tasks.workunit.client.0.smithi032.stdout:1/116: creat d0/d10/f26 x:0 0 0 2024-04-06T14:16:57.088 INFO:tasks.workunit.client.0.smithi032.stdout:7/118: dread f11 [0,4194304] 0 2024-04-06T14:16:57.091 INFO:tasks.workunit.client.0.smithi032.stdout:3/123: creat d3/d16/f35 x:0 0 0 2024-04-06T14:16:57.091 INFO:tasks.workunit.client.0.smithi032.stdout:1/117: fsync d0/f2 0 2024-04-06T14:16:57.094 INFO:tasks.workunit.client.0.smithi032.stdout:3/124: fsync d3/f19 0 2024-04-06T14:16:57.095 INFO:tasks.workunit.client.0.smithi032.stdout:0/89: dwrite d3/fd [0,4194304] 0 2024-04-06T14:16:57.097 INFO:tasks.workunit.client.0.smithi032.stdout:1/118: stat d0/d9/f24 0 2024-04-06T14:16:57.101 INFO:tasks.workunit.client.0.smithi032.stdout:4/271: dwrite d0/df/d23/f3d [0,4194304] 0 2024-04-06T14:16:57.104 INFO:tasks.workunit.client.0.smithi032.stdout:2/129: dwrite d0/d12/f21 [0,4194304] 0 2024-04-06T14:16:57.104 INFO:tasks.workunit.client.0.smithi032.stdout:9/143: dread d2/d5/f6 [0,4194304] 0 2024-04-06T14:16:57.105 INFO:tasks.workunit.client.0.smithi032.stdout:3/125: symlink d3/d16/d1e/l36 0 2024-04-06T14:16:57.106 INFO:tasks.workunit.client.0.smithi032.stdout:1/119: mknod d0/da/c27 0 2024-04-06T14:16:57.109 INFO:tasks.workunit.client.0.smithi032.stdout:4/272: fdatasync d0/df/d2b/f3b 0 2024-04-06T14:16:57.109 INFO:tasks.workunit.client.0.smithi032.stdout:0/90: rename d3/f9 to d3/da/f1e 0 2024-04-06T14:16:57.110 INFO:tasks.workunit.client.0.smithi032.stdout:3/126: write d3/f20 [769009,56140] 0 2024-04-06T14:16:57.111 INFO:tasks.workunit.client.0.smithi032.stdout:2/130: write d0/d12/f1f [114632,17681] 0 2024-04-06T14:16:57.112 INFO:tasks.workunit.client.0.smithi032.stdout:7/119: dwrite d16/d20/f23 [4194304,4194304] 0 2024-04-06T14:16:57.113 INFO:tasks.workunit.client.0.smithi032.stdout:1/120: creat d0/da/f28 x:0 0 0 2024-04-06T14:16:57.113 INFO:tasks.workunit.client.0.smithi032.stdout:9/144: creat d2/f2e x:0 0 0 2024-04-06T14:16:57.114 INFO:tasks.workunit.client.0.smithi032.stdout:2/131: read d0/ff [2576448,105263] 0 2024-04-06T14:16:57.115 INFO:tasks.workunit.client.0.smithi032.stdout:3/127: write d3/d16/f18 [3993010,129099] 0 2024-04-06T14:16:57.116 INFO:tasks.workunit.client.0.smithi032.stdout:1/121: chown d0/l15 1484828855 1 2024-04-06T14:16:57.117 INFO:tasks.workunit.client.0.smithi032.stdout:1/122: fsync d0/d9/d1f/f22 0 2024-04-06T14:16:57.123 INFO:tasks.workunit.client.0.smithi032.stdout:4/273: creat d0/d47/d4b/f63 x:0 0 0 2024-04-06T14:16:57.123 INFO:tasks.workunit.client.0.smithi032.stdout:7/120: mknod d16/d20/d26/c29 0 2024-04-06T14:16:57.123 INFO:tasks.workunit.client.0.smithi032.stdout:2/132: mknod d0/c35 0 2024-04-06T14:16:57.124 INFO:tasks.workunit.client.0.smithi032.stdout:9/145: dwrite d2/d15/f1a [0,4194304] 0 2024-04-06T14:16:57.124 INFO:tasks.workunit.client.0.smithi032.stdout:1/123: symlink d0/d9/d1f/l29 0 2024-04-06T14:16:57.125 INFO:tasks.workunit.client.0.smithi032.stdout:2/133: readlink d0/d12/l14 0 2024-04-06T14:16:57.126 INFO:tasks.workunit.client.0.smithi032.stdout:9/146: fsync d2/d5/f1d 0 2024-04-06T14:16:57.126 INFO:tasks.workunit.client.0.smithi032.stdout:7/121: mknod d16/d20/c2a 0 2024-04-06T14:16:57.127 INFO:tasks.workunit.client.0.smithi032.stdout:1/124: write d0/d10/f26 [524672,83906] 0 2024-04-06T14:16:57.130 INFO:tasks.workunit.client.0.smithi032.stdout:7/122: fdatasync fb 0 2024-04-06T14:16:57.131 INFO:tasks.workunit.client.0.smithi032.stdout:2/134: truncate d0/d12/f21 5199484 0 2024-04-06T14:16:57.131 INFO:tasks.workunit.client.0.smithi032.stdout:9/147: dread - d2/f2e zero size 2024-04-06T14:16:57.132 INFO:tasks.workunit.client.0.smithi032.stdout:1/125: mknod d0/d10/c2a 0 2024-04-06T14:16:57.132 INFO:tasks.workunit.client.0.smithi032.stdout:7/123: dread - d16/f27 zero size 2024-04-06T14:16:57.136 INFO:tasks.workunit.client.0.smithi032.stdout:2/135: rename d0/d30/f34 to d0/d30/f36 0 2024-04-06T14:16:57.136 INFO:tasks.workunit.client.0.smithi032.stdout:4/274: dwrite d0/df/d23/d31/f37 [0,4194304] 0 2024-04-06T14:16:57.137 INFO:tasks.workunit.client.0.smithi032.stdout:4/275: readlink d0/df/d23/l5d 0 2024-04-06T14:16:57.138 INFO:tasks.workunit.client.0.smithi032.stdout:9/148: rename d2/ff to d2/d15/f2f 0 2024-04-06T14:16:57.140 INFO:tasks.workunit.client.0.smithi032.stdout:9/149: fsync d2/f2e 0 2024-04-06T14:16:57.142 INFO:tasks.workunit.client.0.smithi032.stdout:2/136: dread d0/d12/f1c [0,4194304] 0 2024-04-06T14:16:57.143 INFO:tasks.workunit.client.0.smithi032.stdout:2/137: fsync d0/d12/f17 0 2024-04-06T14:16:57.143 INFO:tasks.workunit.client.0.smithi032.stdout:2/138: stat d0/c32 0 2024-04-06T14:16:57.145 INFO:tasks.workunit.client.0.smithi032.stdout:4/276: creat d0/df/d23/d31/d50/d57/f64 x:0 0 0 2024-04-06T14:16:57.146 INFO:tasks.workunit.client.0.smithi032.stdout:9/150: write d2/d15/f2f [1271541,69691] 0 2024-04-06T14:16:57.146 INFO:tasks.workunit.client.0.smithi032.stdout:4/277: write d0/df/d1f/f25 [2567832,39456] 0 2024-04-06T14:16:57.147 INFO:tasks.workunit.client.0.smithi032.stdout:1/126: link d0/da/l18 d0/d10/d16/l2b 0 2024-04-06T14:16:57.149 INFO:tasks.workunit.client.0.smithi032.stdout:4/278: write d0/df/d23/d31/f37 [3953692,24208] 0 2024-04-06T14:16:57.149 INFO:tasks.workunit.client.0.smithi032.stdout:2/139: rmdir d0 39 2024-04-06T14:16:57.150 INFO:tasks.workunit.client.0.smithi032.stdout:7/124: getdents d16/d20/d26 0 2024-04-06T14:16:57.152 INFO:tasks.workunit.client.0.smithi032.stdout:1/127: creat d0/f2c x:0 0 0 2024-04-06T14:16:57.153 INFO:tasks.workunit.client.0.smithi032.stdout:4/279: chown d0/f3a 518518843 1 2024-04-06T14:16:57.153 INFO:tasks.workunit.client.0.smithi032.stdout:9/151: symlink d2/d25/l30 0 2024-04-06T14:16:57.154 INFO:tasks.workunit.client.0.smithi032.stdout:7/125: mkdir d16/d2b 0 2024-04-06T14:16:57.155 INFO:tasks.workunit.client.0.smithi032.stdout:1/128: dread d0/f17 [0,4194304] 0 2024-04-06T14:16:57.155 INFO:tasks.workunit.client.0.smithi032.stdout:2/140: rmdir d0/d30 39 2024-04-06T14:16:57.156 INFO:tasks.workunit.client.0.smithi032.stdout:9/152: truncate d2/d15/f27 973724 0 2024-04-06T14:16:57.156 INFO:tasks.workunit.client.0.smithi032.stdout:1/129: chown d0/da 443 1 2024-04-06T14:16:57.156 INFO:tasks.workunit.client.0.smithi032.stdout:9/153: fsync d2/f2d 0 2024-04-06T14:16:57.158 INFO:tasks.workunit.client.0.smithi032.stdout:4/280: rename d0/df/d1f/f51 to d0/df/d2b/d4f/f65 0 2024-04-06T14:16:57.159 INFO:tasks.workunit.client.0.smithi032.stdout:7/126: mknod d16/c2c 0 2024-04-06T14:16:57.164 INFO:tasks.workunit.client.0.smithi032.stdout:9/154: mkdir d2/d25/d31 0 2024-04-06T14:16:57.164 INFO:tasks.workunit.client.0.smithi032.stdout:4/281: rmdir d0/df/d2b/d4f/d52 39 2024-04-06T14:16:57.164 INFO:tasks.workunit.client.0.smithi032.stdout:4/282: write d0/df/d23/d31/d50/d57/f64 [378858,47426] 0 2024-04-06T14:16:57.164 INFO:tasks.workunit.client.0.smithi032.stdout:7/127: mkdir d16/d20/d26/d2d 0 2024-04-06T14:16:57.165 INFO:tasks.workunit.client.0.smithi032.stdout:4/283: fsync d0/f44 0 2024-04-06T14:16:57.165 INFO:tasks.workunit.client.0.smithi032.stdout:2/141: mknod d0/d30/c37 0 2024-04-06T14:16:57.168 INFO:tasks.workunit.client.0.smithi032.stdout:7/128: dread fe [0,4194304] 0 2024-04-06T14:16:57.170 INFO:tasks.workunit.client.0.smithi032.stdout:4/284: mkdir d0/df/d2b/d66 0 2024-04-06T14:16:57.172 INFO:tasks.workunit.client.0.smithi032.stdout:1/130: getdents d0/d9 0 2024-04-06T14:16:57.172 INFO:tasks.workunit.client.0.smithi032.stdout:2/142: dwrite d0/f22 [0,4194304] 0 2024-04-06T14:16:57.173 INFO:tasks.workunit.client.0.smithi032.stdout:7/129: chown d16/c1c 26 1 2024-04-06T14:16:57.177 INFO:tasks.workunit.client.0.smithi032.stdout:7/130: truncate fb 1425065 0 2024-04-06T14:16:57.177 INFO:tasks.workunit.client.0.smithi032.stdout:9/155: getdents d2/d15 0 2024-04-06T14:16:57.179 INFO:tasks.workunit.client.0.smithi032.stdout:2/143: rename d0/c16 to d0/c38 0 2024-04-06T14:16:57.182 INFO:tasks.workunit.client.0.smithi032.stdout:2/144: dread - d0/d30/f36 zero size 2024-04-06T14:16:57.184 INFO:tasks.workunit.client.0.smithi032.stdout:9/156: creat d2/d25/f32 x:0 0 0 2024-04-06T14:16:57.185 INFO:tasks.workunit.client.0.smithi032.stdout:4/285: link d0/df/d1f/d42/c4c d0/df/d23/d31/d50/d57/d5b/c67 0 2024-04-06T14:16:57.186 INFO:tasks.workunit.client.0.smithi032.stdout:1/131: dwrite d0/d10/f25 [0,4194304] 0 2024-04-06T14:16:57.190 INFO:tasks.workunit.client.0.smithi032.stdout:4/286: creat d0/df/d23/d31/f68 x:0 0 0 2024-04-06T14:16:57.193 INFO:tasks.workunit.client.0.smithi032.stdout:7/131: truncate fd 1972010 0 2024-04-06T14:16:57.194 INFO:tasks.workunit.client.0.smithi032.stdout:4/287: unlink d0/df/f18 0 2024-04-06T14:16:57.195 INFO:tasks.workunit.client.0.smithi032.stdout:2/145: dwrite d0/f28 [0,4194304] 0 2024-04-06T14:16:57.198 INFO:tasks.workunit.client.0.smithi032.stdout:9/157: link d2/f21 d2/d15/d1e/f33 0 2024-04-06T14:16:57.200 INFO:tasks.workunit.client.0.smithi032.stdout:7/132: mknod d16/c2e 0 2024-04-06T14:16:57.200 INFO:tasks.workunit.client.0.smithi032.stdout:9/158: chown d2/d15/f2f 2 1 2024-04-06T14:16:57.201 INFO:tasks.workunit.client.0.smithi032.stdout:7/133: fdatasync fa 0 2024-04-06T14:16:57.201 INFO:tasks.workunit.client.0.smithi032.stdout:9/159: dread - d2/d15/f2b zero size 2024-04-06T14:16:57.202 INFO:tasks.workunit.client.0.smithi032.stdout:7/134: dread - d16/f27 zero size 2024-04-06T14:16:57.203 INFO:tasks.workunit.client.0.smithi032.stdout:7/135: stat d16/c2c 0 2024-04-06T14:16:57.204 INFO:tasks.workunit.client.0.smithi032.stdout:7/136: fsync f15 0 2024-04-06T14:16:57.208 INFO:tasks.workunit.client.0.smithi032.stdout:7/137: stat d16/l18 0 2024-04-06T14:16:57.209 INFO:tasks.workunit.client.0.smithi032.stdout:9/160: mknod d2/d25/d31/c34 0 2024-04-06T14:16:57.210 INFO:tasks.workunit.client.0.smithi032.stdout:7/138: mknod d16/c2f 0 2024-04-06T14:16:57.211 INFO:tasks.workunit.client.0.smithi032.stdout:1/132: truncate d0/d9/f23 675291 0 2024-04-06T14:16:57.212 INFO:tasks.workunit.client.0.smithi032.stdout:4/288: link d0/df/c21 d0/df/d1f/d5a/c69 0 2024-04-06T14:16:57.212 INFO:tasks.workunit.client.0.smithi032.stdout:9/161: symlink d2/d5/l35 0 2024-04-06T14:16:57.214 INFO:tasks.workunit.client.0.smithi032.stdout:1/133: readlink d0/da/l18 0 2024-04-06T14:16:57.215 INFO:tasks.workunit.client.0.smithi032.stdout:7/139: mkdir d16/d20/d26/d2d/d30 0 2024-04-06T14:16:57.215 INFO:tasks.workunit.client.0.smithi032.stdout:4/289: read d0/df/d23/d31/d50/d57/f64 [162278,31379] 0 2024-04-06T14:16:57.218 INFO:tasks.workunit.client.0.smithi032.stdout:7/140: write f10 [1102316,23437] 0 2024-04-06T14:16:57.218 INFO:tasks.workunit.client.0.smithi032.stdout:1/134: symlink d0/d10/d16/l2d 0 2024-04-06T14:16:57.219 INFO:tasks.workunit.client.0.smithi032.stdout:2/146: dwrite d0/f13 [4194304,4194304] 0 2024-04-06T14:16:57.220 INFO:tasks.workunit.client.0.smithi032.stdout:1/135: write d0/d9/f1b [2331512,100638] 0 2024-04-06T14:16:57.221 INFO:tasks.workunit.client.0.smithi032.stdout:4/290: creat d0/df/d23/d31/d50/d57/f6a x:0 0 0 2024-04-06T14:16:57.227 INFO:tasks.workunit.client.0.smithi032.stdout:2/147: dwrite d0/f22 [4194304,4194304] 0 2024-04-06T14:16:57.229 INFO:tasks.workunit.client.0.smithi032.stdout:4/291: symlink d0/df/d2b/d43/l6b 0 2024-04-06T14:16:57.232 INFO:tasks.workunit.client.0.smithi032.stdout:2/148: mkdir d0/d30/d39 0 2024-04-06T14:16:57.233 INFO:tasks.workunit.client.0.smithi032.stdout:7/141: dread fd [0,4194304] 0 2024-04-06T14:16:57.233 INFO:tasks.workunit.client.0.smithi032.stdout:9/162: read d2/d5/f12 [582851,10949] 0 2024-04-06T14:16:57.233 INFO:tasks.workunit.client.0.smithi032.stdout:2/149: fdatasync d0/fc 0 2024-04-06T14:16:57.298 INFO:tasks.workunit.client.0.smithi032.stdout:1/136: truncate d0/f2 2766950 0 2024-04-06T14:16:57.299 INFO:tasks.workunit.client.0.smithi032.stdout:7/142: creat d16/d20/d26/f31 x:0 0 0 2024-04-06T14:16:57.299 INFO:tasks.workunit.client.0.smithi032.stdout:2/150: mkdir d0/d12/d3a 0 2024-04-06T14:16:57.300 INFO:tasks.workunit.client.0.smithi032.stdout:7/143: truncate f7 107151 0 2024-04-06T14:16:57.303 INFO:tasks.workunit.client.0.smithi032.stdout:7/144: read fb [433726,113470] 0 2024-04-06T14:16:57.303 INFO:tasks.workunit.client.0.smithi032.stdout:4/292: getdents d0/df/d23/d31/d50/d57/d5b 0 2024-04-06T14:16:57.305 INFO:tasks.workunit.client.0.smithi032.stdout:7/145: fdatasync d16/f19 0 2024-04-06T14:16:57.306 INFO:tasks.workunit.client.0.smithi032.stdout:7/146: stat d16/d2b 0 2024-04-06T14:16:57.309 INFO:tasks.workunit.client.0.smithi032.stdout:2/151: dwrite d0/d12/f2c [0,4194304] 0 2024-04-06T14:16:57.309 INFO:tasks.workunit.client.0.smithi032.stdout:1/137: dread d0/d9/f23 [0,4194304] 0 2024-04-06T14:16:57.310 INFO:tasks.workunit.client.0.smithi032.stdout:4/293: unlink d0/f1b 0 2024-04-06T14:16:57.311 INFO:tasks.workunit.client.0.smithi032.stdout:2/152: write d0/f22 [1788304,63048] 0 2024-04-06T14:16:57.312 INFO:tasks.workunit.client.0.smithi032.stdout:7/147: creat d16/d20/d26/d2d/f32 x:0 0 0 2024-04-06T14:16:57.315 INFO:tasks.workunit.client.0.smithi032.stdout:1/138: creat d0/da/f2e x:0 0 0 2024-04-06T14:16:57.315 INFO:tasks.workunit.client.0.smithi032.stdout:4/294: write d0/df/d2b/f3b [1403745,117221] 0 2024-04-06T14:16:57.315 INFO:tasks.workunit.client.0.smithi032.stdout:9/163: dwrite d2/f13 [0,4194304] 0 2024-04-06T14:16:57.317 INFO:tasks.workunit.client.0.smithi032.stdout:2/153: write d0/d30/f36 [521596,29071] 0 2024-04-06T14:16:57.317 INFO:tasks.workunit.client.0.smithi032.stdout:1/139: fdatasync d0/da/f28 0 2024-04-06T14:16:57.318 INFO:tasks.workunit.client.0.smithi032.stdout:2/154: chown d0/c32 103 1 2024-04-06T14:16:57.321 INFO:tasks.workunit.client.0.smithi032.stdout:9/164: creat d2/d25/f36 x:0 0 0 2024-04-06T14:16:57.322 INFO:tasks.workunit.client.0.smithi032.stdout:7/148: fsync d16/d20/d26/d2d/f32 0 2024-04-06T14:16:57.324 INFO:tasks.workunit.client.0.smithi032.stdout:4/295: write d0/df/d23/d31/d50/d57/f64 [348202,11625] 0 2024-04-06T14:16:57.327 INFO:tasks.workunit.client.0.smithi032.stdout:9/165: rename d2/d25/f36 to d2/d5/f37 0 2024-04-06T14:16:57.329 INFO:tasks.workunit.client.0.smithi032.stdout:7/149: dread d16/d20/f23 [4194304,4194304] 0 2024-04-06T14:16:57.333 INFO:tasks.workunit.client.0.smithi032.stdout:4/296: rename d0/df/d2b/d43/l6b to d0/d47/l6c 0 2024-04-06T14:16:57.334 INFO:tasks.workunit.client.0.smithi032.stdout:9/166: symlink d2/d15/l38 0 2024-04-06T14:16:57.336 INFO:tasks.workunit.client.0.smithi032.stdout:7/150: rename d16/f19 to d16/d20/d26/d2d/d30/f33 0 2024-04-06T14:16:57.337 INFO:tasks.workunit.client.0.smithi032.stdout:4/297: symlink d0/df/d2b/d43/d48/l6d 0 2024-04-06T14:16:57.340 INFO:tasks.workunit.client.0.smithi032.stdout:4/298: creat d0/df/d2b/d46/f6e x:0 0 0 2024-04-06T14:16:57.342 INFO:tasks.workunit.client.0.smithi032.stdout:7/151: dwrite f7 [0,4194304] 0 2024-04-06T14:16:57.343 INFO:tasks.workunit.client.0.smithi032.stdout:4/299: write d0/df/d23/d31/d50/d57/f6a [66474,127564] 0 2024-04-06T14:16:57.346 INFO:tasks.workunit.client.0.smithi032.stdout:7/152: dread fb [0,4194304] 0 2024-04-06T14:16:57.349 INFO:tasks.workunit.client.0.smithi032.stdout:4/300: dread d0/df/d23/f3d [0,4194304] 0 2024-04-06T14:16:57.350 INFO:tasks.workunit.client.0.smithi032.stdout:4/301: chown d0/df/d1f/f34 555 1 2024-04-06T14:16:57.351 INFO:tasks.workunit.client.0.smithi032.stdout:9/167: dread d2/f4 [4194304,4194304] 0 2024-04-06T14:16:57.352 INFO:tasks.workunit.client.0.smithi032.stdout:9/168: fsync d2/d5/f14 0 2024-04-06T14:16:57.355 INFO:tasks.workunit.client.0.smithi032.stdout:4/302: rename d0/df/d23/d31/d50/c55 to d0/df/d2b/d4f/d52/c6f 0 2024-04-06T14:16:57.356 INFO:tasks.workunit.client.0.smithi032.stdout:7/153: write fe [2018206,103266] 0 2024-04-06T14:16:57.358 INFO:tasks.workunit.client.0.smithi032.stdout:9/169: getdents d2/d25/d31 0 2024-04-06T14:16:57.358 INFO:tasks.workunit.client.0.smithi032.stdout:7/154: read f10 [783633,77804] 0 2024-04-06T14:16:57.359 INFO:tasks.workunit.client.0.smithi032.stdout:4/303: rename d0/f3a to d0/df/d2b/f70 0 2024-04-06T14:16:57.361 INFO:tasks.workunit.client.0.smithi032.stdout:9/170: mkdir d2/d25/d31/d39 0 2024-04-06T14:16:57.361 INFO:tasks.workunit.client.0.smithi032.stdout:9/171: fsync d2/d25/f32 0 2024-04-06T14:16:57.362 INFO:tasks.workunit.client.0.smithi032.stdout:9/172: dread - d2/d5/f37 zero size 2024-04-06T14:16:57.362 INFO:tasks.workunit.client.0.smithi032.stdout:7/155: rmdir d16/d2b 0 2024-04-06T14:16:57.363 INFO:tasks.workunit.client.0.smithi032.stdout:9/173: chown d2/d25/d31 168349 1 2024-04-06T14:16:57.363 INFO:tasks.workunit.client.0.smithi032.stdout:9/174: truncate d2/f2d 884674 0 2024-04-06T14:16:57.364 INFO:tasks.workunit.client.0.smithi032.stdout:7/156: symlink d16/d20/d26/d2d/l34 0 2024-04-06T14:16:57.366 INFO:tasks.workunit.client.0.smithi032.stdout:7/157: fdatasync f10 0 2024-04-06T14:16:57.369 INFO:tasks.workunit.client.0.smithi032.stdout:4/304: truncate d0/df/d23/f3d 2978942 0 2024-04-06T14:16:57.371 INFO:tasks.workunit.client.0.smithi032.stdout:7/158: dread f10 [0,4194304] 0 2024-04-06T14:16:57.372 INFO:tasks.workunit.client.0.smithi032.stdout:4/305: mknod d0/df/c71 0 2024-04-06T14:16:57.372 INFO:tasks.workunit.client.0.smithi032.stdout:9/175: dwrite d2/d5/f18 [0,4194304] 0 2024-04-06T14:16:57.372 INFO:tasks.workunit.client.0.smithi032.stdout:7/159: dread - d16/d20/d26/f31 zero size 2024-04-06T14:16:57.374 INFO:tasks.workunit.client.0.smithi032.stdout:9/176: chown d2/d5/f6 0 1 2024-04-06T14:16:57.374 INFO:tasks.workunit.client.0.smithi032.stdout:7/160: write d16/d20/d26/f31 [336812,81528] 0 2024-04-06T14:16:57.375 INFO:tasks.workunit.client.0.smithi032.stdout:4/306: write d0/df/d2b/f70 [382339,117117] 0 2024-04-06T14:16:57.375 INFO:tasks.workunit.client.0.smithi032.stdout:9/177: fdatasync d2/f13 0 2024-04-06T14:16:57.376 INFO:tasks.workunit.client.0.smithi032.stdout:7/161: write d16/f1f [544136,88370] 0 2024-04-06T14:16:57.376 INFO:tasks.workunit.client.0.smithi032.stdout:9/178: truncate d2/f2d 945255 0 2024-04-06T14:16:57.378 INFO:tasks.workunit.client.0.smithi032.stdout:7/162: read d16/d20/d26/d2d/d30/f33 [776128,49278] 0 2024-04-06T14:16:57.384 INFO:tasks.workunit.client.0.smithi032.stdout:9/179: creat d2/d25/d31/f3a x:0 0 0 2024-04-06T14:16:57.385 INFO:tasks.workunit.client.0.smithi032.stdout:9/180: dread - d2/f2e zero size 2024-04-06T14:16:57.386 INFO:tasks.workunit.client.0.smithi032.stdout:9/181: creat d2/d25/f3b x:0 0 0 2024-04-06T14:16:57.389 INFO:tasks.workunit.client.0.smithi032.stdout:9/182: symlink d2/d25/l3c 0 2024-04-06T14:16:57.390 INFO:tasks.workunit.client.0.smithi032.stdout:7/163: dwrite f6 [4194304,4194304] 0 2024-04-06T14:16:57.394 INFO:tasks.workunit.client.0.smithi032.stdout:7/164: creat d16/d20/d26/d2d/d30/f35 x:0 0 0 2024-04-06T14:16:57.398 INFO:tasks.workunit.client.0.smithi032.stdout:4/307: dwrite d0/df/d23/f3d [0,4194304] 0 2024-04-06T14:16:57.400 INFO:tasks.workunit.client.0.smithi032.stdout:7/165: truncate d16/d20/d26/d2d/d30/f33 1535132 0 2024-04-06T14:16:57.400 INFO:tasks.workunit.client.0.smithi032.stdout:9/183: dwrite d2/d5/f6 [0,4194304] 0 2024-04-06T14:16:57.402 INFO:tasks.workunit.client.0.smithi032.stdout:9/184: write d2/f2d [1360703,62427] 0 2024-04-06T14:16:57.404 INFO:tasks.workunit.client.0.smithi032.stdout:7/166: creat d16/d20/d26/d2d/d30/f36 x:0 0 0 2024-04-06T14:16:57.407 INFO:tasks.workunit.client.0.smithi032.stdout:9/185: mknod d2/c3d 0 2024-04-06T14:16:57.410 INFO:tasks.workunit.client.0.smithi032.stdout:7/167: rename f12 to d16/d20/f37 0 2024-04-06T14:16:57.414 INFO:tasks.workunit.client.0.smithi032.stdout:9/186: creat d2/d5/f3e x:0 0 0 2024-04-06T14:16:57.414 INFO:tasks.workunit.client.0.smithi032.stdout:4/308: truncate d0/df/d23/f3c 4119343 0 2024-04-06T14:16:57.415 INFO:tasks.workunit.client.0.smithi032.stdout:9/187: read - d2/f2e zero size 2024-04-06T14:16:57.417 INFO:tasks.workunit.client.0.smithi032.stdout:9/188: mknod d2/d15/c3f 0 2024-04-06T14:16:57.419 INFO:tasks.workunit.client.0.smithi032.stdout:4/309: write d0/df/d23/d31/d50/d57/f64 [988646,23384] 0 2024-04-06T14:16:57.419 INFO:tasks.workunit.client.0.smithi032.stdout:7/168: dwrite d16/d20/d26/d2d/d30/f35 [0,4194304] 0 2024-04-06T14:16:57.421 INFO:tasks.workunit.client.0.smithi032.stdout:9/189: write d2/d5/f1d [831062,9878] 0 2024-04-06T14:16:57.422 INFO:tasks.workunit.client.0.smithi032.stdout:4/310: write d0/df/d23/f4a [389178,85374] 0 2024-04-06T14:16:57.427 INFO:tasks.workunit.client.0.smithi032.stdout:7/169: dwrite f15 [0,4194304] 0 2024-04-06T14:16:57.428 INFO:tasks.workunit.client.0.smithi032.stdout:9/190: creat d2/d25/d31/f40 x:0 0 0 2024-04-06T14:16:57.429 INFO:tasks.workunit.client.0.smithi032.stdout:4/311: creat d0/df/d2b/f72 x:0 0 0 2024-04-06T14:16:57.430 INFO:tasks.workunit.client.0.smithi032.stdout:9/191: write d2/d15/f2f [1744847,15173] 0 2024-04-06T14:16:57.435 INFO:tasks.workunit.client.0.smithi032.stdout:7/170: link f6 d16/d20/d26/d2d/d30/f38 0 2024-04-06T14:16:57.436 INFO:tasks.workunit.client.0.smithi032.stdout:4/312: write d0/df/d2b/d4f/f65 [750052,8369] 0 2024-04-06T14:16:57.438 INFO:tasks.workunit.client.0.smithi032.stdout:7/171: write d16/d20/f23 [2071389,12565] 0 2024-04-06T14:16:57.438 INFO:tasks.workunit.client.0.smithi032.stdout:4/313: rmdir d0/df/d1f/d5a 39 2024-04-06T14:16:57.440 INFO:tasks.workunit.client.0.smithi032.stdout:4/314: mknod d0/df/d23/d31/d50/d57/c73 0 2024-04-06T14:16:57.441 INFO:tasks.workunit.client.0.smithi032.stdout:4/315: chown d0/df/d1f/d2e/c2f 9919605 1 2024-04-06T14:16:57.441 INFO:tasks.workunit.client.0.smithi032.stdout:4/316: chown d0/df/c1a 1066558366 1 2024-04-06T14:16:57.443 INFO:tasks.workunit.client.0.smithi032.stdout:4/317: symlink d0/df/d23/d31/d50/d57/l74 0 2024-04-06T14:16:57.445 INFO:tasks.workunit.client.0.smithi032.stdout:9/192: rmdir d2/d25 39 2024-04-06T14:16:57.445 INFO:tasks.workunit.client.0.smithi032.stdout:4/318: creat d0/df/d1f/d42/d45/f75 x:0 0 0 2024-04-06T14:16:57.446 INFO:tasks.workunit.client.0.smithi032.stdout:9/193: mkdir d2/d15/d1e/d41 0 2024-04-06T14:16:57.450 INFO:tasks.workunit.client.0.smithi032.stdout:9/194: mknod d2/d25/d31/d39/c42 0 2024-04-06T14:16:57.453 INFO:tasks.workunit.client.0.smithi032.stdout:4/319: dwrite d0/df/d1f/d2e/f41 [0,4194304] 0 2024-04-06T14:16:57.453 INFO:tasks.workunit.client.0.smithi032.stdout:9/195: creat d2/d15/f43 x:0 0 0 2024-04-06T14:16:57.457 INFO:tasks.workunit.client.0.smithi032.stdout:7/172: dwrite fd [0,4194304] 0 2024-04-06T14:16:57.473 INFO:tasks.workunit.client.0.smithi032.stdout:9/196: truncate d2/d15/f1a 2841446 0 2024-04-06T14:16:57.473 INFO:tasks.workunit.client.0.smithi032.stdout:7/173: truncate f6 3792661 0 2024-04-06T14:16:57.475 INFO:tasks.workunit.client.0.smithi032.stdout:7/174: creat d16/d20/d26/f39 x:0 0 0 2024-04-06T14:16:57.478 INFO:tasks.workunit.client.0.smithi032.stdout:7/175: rename f15 to d16/d20/d26/f3a 0 2024-04-06T14:16:57.478 INFO:tasks.workunit.client.0.smithi032.stdout:9/197: link d2/f4 d2/d5/f44 0 2024-04-06T14:16:57.478 INFO:tasks.workunit.client.0.smithi032.stdout:7/176: read - d16/d20/d26/d2d/f32 zero size 2024-04-06T14:16:57.479 INFO:tasks.workunit.client.0.smithi032.stdout:9/198: creat d2/d25/f45 x:0 0 0 2024-04-06T14:16:57.480 INFO:tasks.workunit.client.0.smithi032.stdout:7/177: mknod d16/d20/d26/d2d/c3b 0 2024-04-06T14:16:57.481 INFO:tasks.workunit.client.0.smithi032.stdout:7/178: symlink d16/d20/l3c 0 2024-04-06T14:16:57.482 INFO:tasks.workunit.client.0.smithi032.stdout:7/179: fdatasync d16/f27 0 2024-04-06T14:16:57.482 INFO:tasks.workunit.client.0.smithi032.stdout:7/180: chown d16/d20/d26/f3a 6714 1 2024-04-06T14:16:57.484 INFO:tasks.workunit.client.0.smithi032.stdout:7/181: symlink d16/d20/d26/d2d/d30/l3d 0 2024-04-06T14:16:57.485 INFO:tasks.workunit.client.0.smithi032.stdout:7/182: write d16/f1f [1069471,125379] 0 2024-04-06T14:16:57.490 INFO:tasks.workunit.client.0.smithi032.stdout:7/183: dwrite fa [0,4194304] 0 2024-04-06T14:16:57.491 INFO:tasks.workunit.client.0.smithi032.stdout:7/184: stat f7 0 2024-04-06T14:16:57.494 INFO:tasks.workunit.client.0.smithi032.stdout:7/185: mkdir d16/d20/d26/d2d/d30/d3e 0 2024-04-06T14:16:57.497 INFO:tasks.workunit.client.0.smithi032.stdout:7/186: creat d16/d20/d26/d2d/f3f x:0 0 0 2024-04-06T14:16:57.506 INFO:tasks.workunit.client.0.smithi032.stdout:5/85: sync 2024-04-06T14:16:57.506 INFO:tasks.workunit.client.0.smithi032.stdout:6/165: sync 2024-04-06T14:16:57.507 INFO:tasks.workunit.client.0.smithi032.stdout:5/86: chown d16/d18 1671 1 2024-04-06T14:16:57.507 INFO:tasks.workunit.client.0.smithi032.stdout:7/187: stat f6 0 2024-04-06T14:16:57.508 INFO:tasks.workunit.client.0.smithi032.stdout:7/188: truncate d16/f27 536361 0 2024-04-06T14:16:57.508 INFO:tasks.workunit.client.0.smithi032.stdout:7/189: read fe [1329948,45649] 0 2024-04-06T14:16:57.509 INFO:tasks.workunit.client.0.smithi032.stdout:5/87: mknod d16/c1a 0 2024-04-06T14:16:57.509 INFO:tasks.workunit.client.0.smithi032.stdout:5/88: read - fe zero size 2024-04-06T14:16:57.509 INFO:tasks.workunit.client.0.smithi032.stdout:7/190: write d16/f27 [1571991,382] 0 2024-04-06T14:16:57.510 INFO:tasks.workunit.client.0.smithi032.stdout:6/166: mkdir d4/d5/d28 0 2024-04-06T14:16:57.511 INFO:tasks.workunit.client.0.smithi032.stdout:6/167: read d4/d5/d14/f1a [6492,28954] 0 2024-04-06T14:16:57.512 INFO:tasks.workunit.client.0.smithi032.stdout:7/191: creat d16/f40 x:0 0 0 2024-04-06T14:16:57.514 INFO:tasks.workunit.client.0.smithi032.stdout:5/89: symlink d16/d18/l1b 0 2024-04-06T14:16:57.515 INFO:tasks.workunit.client.0.smithi032.stdout:5/90: write f7 [735298,10530] 0 2024-04-06T14:16:57.515 INFO:tasks.workunit.client.0.smithi032.stdout:7/192: creat d16/d20/d26/d2d/f41 x:0 0 0 2024-04-06T14:16:57.517 INFO:tasks.workunit.client.0.smithi032.stdout:5/91: creat d16/d18/f1c x:0 0 0 2024-04-06T14:16:57.517 INFO:tasks.workunit.client.0.smithi032.stdout:7/193: symlink d16/d20/d26/d2d/d30/l42 0 2024-04-06T14:16:57.518 INFO:tasks.workunit.client.0.smithi032.stdout:5/92: dread - d16/f17 zero size 2024-04-06T14:16:57.518 INFO:tasks.workunit.client.0.smithi032.stdout:7/194: write fd [1095682,104506] 0 2024-04-06T14:16:57.520 INFO:tasks.workunit.client.0.smithi032.stdout:6/168: dread d4/d5/fb [0,4194304] 0 2024-04-06T14:16:57.520 INFO:tasks.workunit.client.0.smithi032.stdout:5/93: creat d16/f1d x:0 0 0 2024-04-06T14:16:57.521 INFO:tasks.workunit.client.0.smithi032.stdout:6/169: chown d4/d5/d19/d25 148128 1 2024-04-06T14:16:57.521 INFO:tasks.workunit.client.0.smithi032.stdout:7/195: unlink d16/f1f 0 2024-04-06T14:16:57.522 INFO:tasks.workunit.client.0.smithi032.stdout:5/94: write f10 [590690,65967] 0 2024-04-06T14:16:57.522 INFO:tasks.workunit.client.0.smithi032.stdout:5/95: dread - fe zero size 2024-04-06T14:16:57.523 INFO:tasks.workunit.client.0.smithi032.stdout:6/170: chown d4/d5/d14/c1c 4 1 2024-04-06T14:16:57.523 INFO:tasks.workunit.client.0.smithi032.stdout:5/96: fdatasync f5 0 2024-04-06T14:16:57.523 INFO:tasks.workunit.client.0.smithi032.stdout:6/171: stat d4/d5/l20 0 2024-04-06T14:16:57.528 INFO:tasks.workunit.client.0.smithi032.stdout:5/97: rmdir d16 39 2024-04-06T14:16:57.529 INFO:tasks.workunit.client.0.smithi032.stdout:5/98: dread f11 [0,4194304] 0 2024-04-06T14:16:57.530 INFO:tasks.workunit.client.0.smithi032.stdout:5/99: read - fe zero size 2024-04-06T14:16:57.531 INFO:tasks.workunit.client.0.smithi032.stdout:6/172: dwrite d4/d5/f15 [0,4194304] 0 2024-04-06T14:16:57.536 INFO:tasks.workunit.client.0.smithi032.stdout:5/100: rename d16/f19 to d16/d18/f1e 0 2024-04-06T14:16:57.538 INFO:tasks.workunit.client.0.smithi032.stdout:5/101: mknod d16/d18/c1f 0 2024-04-06T14:16:57.540 INFO:tasks.workunit.client.0.smithi032.stdout:5/102: fdatasync d16/f1d 0 2024-04-06T14:16:57.542 INFO:tasks.workunit.client.0.smithi032.stdout:5/103: creat d16/d18/f20 x:0 0 0 2024-04-06T14:16:57.542 INFO:tasks.workunit.client.0.smithi032.stdout:7/196: dread d16/d20/d26/d2d/d30/f38 [0,4194304] 0 2024-04-06T14:16:57.544 INFO:tasks.workunit.client.0.smithi032.stdout:7/197: mknod d16/d20/d26/d2d/d30/c43 0 2024-04-06T14:16:57.546 INFO:tasks.workunit.client.0.smithi032.stdout:7/198: creat d16/d20/d26/f44 x:0 0 0 2024-04-06T14:16:57.549 INFO:tasks.workunit.client.0.smithi032.stdout:7/199: creat d16/d20/d26/d2d/d30/d3e/f45 x:0 0 0 2024-04-06T14:16:57.555 INFO:tasks.workunit.client.0.smithi032.stdout:5/104: dwrite f11 [0,4194304] 0 2024-04-06T14:16:57.555 INFO:tasks.workunit.client.0.smithi032.stdout:7/200: dread d16/d20/f28 [0,4194304] 0 2024-04-06T14:16:57.559 INFO:tasks.workunit.client.0.smithi032.stdout:5/105: link f11 d16/d18/f21 0 2024-04-06T14:16:57.559 INFO:tasks.workunit.client.0.smithi032.stdout:7/201: write d16/d20/f23 [3233443,81724] 0 2024-04-06T14:16:57.561 INFO:tasks.workunit.client.0.smithi032.stdout:5/106: symlink d16/d18/l22 0 2024-04-06T14:16:57.563 INFO:tasks.workunit.client.0.smithi032.stdout:7/202: creat d16/d20/d26/d2d/d30/f46 x:0 0 0 2024-04-06T14:16:57.564 INFO:tasks.workunit.client.0.smithi032.stdout:5/107: creat d16/d18/f23 x:0 0 0 2024-04-06T14:16:57.566 INFO:tasks.workunit.client.0.smithi032.stdout:5/108: creat d16/f24 x:0 0 0 2024-04-06T14:16:57.570 INFO:tasks.workunit.client.0.smithi032.stdout:7/203: dwrite d16/d20/d26/d2d/d30/f46 [0,4194304] 0 2024-04-06T14:16:57.571 INFO:tasks.workunit.client.0.smithi032.stdout:7/204: readlink d16/l1b 0 2024-04-06T14:16:57.571 INFO:tasks.workunit.client.0.smithi032.stdout:5/109: link d16/c1a d16/d18/c25 0 2024-04-06T14:16:57.572 INFO:tasks.workunit.client.0.smithi032.stdout:7/205: write d16/d20/d26/d2d/f3f [355526,4314] 0 2024-04-06T14:16:57.574 INFO:tasks.workunit.client.0.smithi032.stdout:5/110: symlink d16/l26 0 2024-04-06T14:16:57.576 INFO:tasks.workunit.client.0.smithi032.stdout:5/111: mknod d16/d18/c27 0 2024-04-06T14:16:57.577 INFO:tasks.workunit.client.0.smithi032.stdout:7/206: rename d16/d20/d26/d2d/d30/f33 to d16/d20/f47 0 2024-04-06T14:16:57.578 INFO:tasks.workunit.client.0.smithi032.stdout:5/112: mkdir d16/d18/d28 0 2024-04-06T14:16:57.578 INFO:tasks.workunit.client.0.smithi032.stdout:5/113: chown ff 2023281 1 2024-04-06T14:16:57.580 INFO:tasks.workunit.client.0.smithi032.stdout:5/114: creat d16/d18/f29 x:0 0 0 2024-04-06T14:16:57.581 INFO:tasks.workunit.client.0.smithi032.stdout:5/115: unlink f7 0 2024-04-06T14:16:57.582 INFO:tasks.workunit.client.0.smithi032.stdout:5/116: write fc [621855,122279] 0 2024-04-06T14:16:57.583 INFO:tasks.workunit.client.0.smithi032.stdout:5/117: read - d16/f1d zero size 2024-04-06T14:16:57.587 INFO:tasks.workunit.client.0.smithi032.stdout:5/118: getdents d16 0 2024-04-06T14:16:57.588 INFO:tasks.workunit.client.0.smithi032.stdout:5/119: chown d16/d18/f29 175737 1 2024-04-06T14:16:57.588 INFO:tasks.workunit.client.0.smithi032.stdout:5/120: fdatasync d16/f1d 0 2024-04-06T14:16:57.592 INFO:tasks.workunit.client.0.smithi032.stdout:5/121: truncate d16/d18/f21 1126764 0 2024-04-06T14:16:57.596 INFO:tasks.workunit.client.0.smithi032.stdout:5/122: link d16/d18/c25 d16/d18/d28/c2a 0 2024-04-06T14:16:57.596 INFO:tasks.workunit.client.0.smithi032.stdout:5/123: dread - fb zero size 2024-04-06T14:16:57.598 INFO:tasks.workunit.client.0.smithi032.stdout:5/124: link fe d16/d18/f2b 0 2024-04-06T14:16:57.603 INFO:tasks.workunit.client.0.smithi032.stdout:5/125: dwrite f3 [0,4194304] 0 2024-04-06T14:16:57.604 INFO:tasks.workunit.client.0.smithi032.stdout:5/126: chown fe 9 1 2024-04-06T14:16:57.605 INFO:tasks.workunit.client.0.smithi032.stdout:5/127: chown d16/d18/f29 30 1 2024-04-06T14:16:57.605 INFO:tasks.workunit.client.0.smithi032.stdout:5/128: chown d16/f1d 32 1 2024-04-06T14:16:57.607 INFO:tasks.workunit.client.0.smithi032.stdout:5/129: creat d16/f2c x:0 0 0 2024-04-06T14:16:57.608 INFO:tasks.workunit.client.0.smithi032.stdout:8/171: sync 2024-04-06T14:16:57.609 INFO:tasks.workunit.client.0.smithi032.stdout:0/91: sync 2024-04-06T14:16:57.611 INFO:tasks.workunit.client.0.smithi032.stdout:0/92: creat d3/da/f1f x:0 0 0 2024-04-06T14:16:57.612 INFO:tasks.workunit.client.0.smithi032.stdout:8/172: mknod d3/d8/d3a/c43 0 2024-04-06T14:16:57.614 INFO:tasks.workunit.client.0.smithi032.stdout:0/93: fsync d3/da/ff 0 2024-04-06T14:16:57.615 INFO:tasks.workunit.client.0.smithi032.stdout:5/130: dwrite f1 [4194304,4194304] 0 2024-04-06T14:16:57.616 INFO:tasks.workunit.client.0.smithi032.stdout:8/173: creat d3/d8/d7/d14/f44 x:0 0 0 2024-04-06T14:16:57.617 INFO:tasks.workunit.client.0.smithi032.stdout:0/94: chown d3/da/l1c 4570858 1 2024-04-06T14:16:57.617 INFO:tasks.workunit.client.0.smithi032.stdout:5/131: write f10 [1471000,53767] 0 2024-04-06T14:16:57.618 INFO:tasks.workunit.client.0.smithi032.stdout:8/174: write d3/d8/d12/f3f [867328,67369] 0 2024-04-06T14:16:57.622 INFO:tasks.workunit.client.0.smithi032.stdout:0/95: getdents d3 0 2024-04-06T14:16:57.623 INFO:tasks.workunit.client.0.smithi032.stdout:8/175: creat d3/d8/d7/d14/f45 x:0 0 0 2024-04-06T14:16:57.625 INFO:tasks.workunit.client.0.smithi032.stdout:0/96: creat d3/f20 x:0 0 0 2024-04-06T14:16:57.627 INFO:tasks.workunit.client.0.smithi032.stdout:8/176: mkdir d3/d8/d12/d2f/d46 0 2024-04-06T14:16:57.628 INFO:tasks.workunit.client.0.smithi032.stdout:5/132: getdents d16/d18/d28 0 2024-04-06T14:16:57.630 INFO:tasks.workunit.client.0.smithi032.stdout:5/133: write d16/d18/f20 [904460,53439] 0 2024-04-06T14:16:57.631 INFO:tasks.workunit.client.0.smithi032.stdout:8/177: link d3/d8/d12/f3f d3/d8/d7/f47 0 2024-04-06T14:16:57.632 INFO:tasks.workunit.client.0.smithi032.stdout:0/97: link d3/da/lb d3/l21 0 2024-04-06T14:16:57.634 INFO:tasks.workunit.client.0.smithi032.stdout:0/98: truncate f0 1349309 0 2024-04-06T14:16:57.634 INFO:tasks.workunit.client.0.smithi032.stdout:8/178: mknod d3/d8/d7/c48 0 2024-04-06T14:16:57.636 INFO:tasks.workunit.client.0.smithi032.stdout:0/99: mknod d3/da/c22 0 2024-04-06T14:16:57.637 INFO:tasks.workunit.client.0.smithi032.stdout:5/134: dwrite f5 [0,4194304] 0 2024-04-06T14:16:57.637 INFO:tasks.workunit.client.0.smithi032.stdout:8/179: symlink d3/d8/d12/d2f/l49 0 2024-04-06T14:16:57.639 INFO:tasks.workunit.client.0.smithi032.stdout:8/180: rmdir d3/d8/d12/d2f 39 2024-04-06T14:16:57.646 INFO:tasks.workunit.client.0.smithi032.stdout:0/100: rename f0 to d3/da/f23 0 2024-04-06T14:16:57.648 INFO:tasks.workunit.client.0.smithi032.stdout:8/181: dwrite d3/d8/f2e [0,4194304] 0 2024-04-06T14:16:57.648 INFO:tasks.workunit.client.0.smithi032.stdout:5/135: write fe [440598,119658] 0 2024-04-06T14:16:57.649 INFO:tasks.workunit.client.0.smithi032.stdout:5/136: write f5 [3098360,34618] 0 2024-04-06T14:16:57.649 INFO:tasks.workunit.client.0.smithi032.stdout:8/182: stat d3/d8/d7/d1a/c2d 0 2024-04-06T14:16:57.652 INFO:tasks.workunit.client.0.smithi032.stdout:5/137: write f5 [100979,68627] 0 2024-04-06T14:16:57.652 INFO:tasks.workunit.client.0.smithi032.stdout:5/138: stat d16/d18/l1b 0 2024-04-06T14:16:57.655 INFO:tasks.workunit.client.0.smithi032.stdout:5/139: rename d16/d18/l22 to d16/d18/l2d 0 2024-04-06T14:16:57.661 INFO:tasks.workunit.client.0.smithi032.stdout:8/183: dwrite d3/d8/d7/d1a/f24 [0,4194304] 0 2024-04-06T14:16:57.661 INFO:tasks.workunit.client.0.smithi032.stdout:0/101: dread d3/f14 [0,4194304] 0 2024-04-06T14:16:57.663 INFO:tasks.workunit.client.0.smithi032.stdout:8/184: read d3/d8/d12/f3f [141574,94029] 0 2024-04-06T14:16:57.663 INFO:tasks.workunit.client.0.smithi032.stdout:0/102: creat d3/f24 x:0 0 0 2024-04-06T14:16:57.664 INFO:tasks.workunit.client.0.smithi032.stdout:5/140: getdents d16/d18 0 2024-04-06T14:16:57.665 INFO:tasks.workunit.client.0.smithi032.stdout:5/141: read f5 [692809,47923] 0 2024-04-06T14:16:57.666 INFO:tasks.workunit.client.0.smithi032.stdout:5/142: write d16/d18/f2b [1260271,73330] 0 2024-04-06T14:16:57.667 INFO:tasks.workunit.client.0.smithi032.stdout:0/103: mkdir d3/da/d25 0 2024-04-06T14:16:57.668 INFO:tasks.workunit.client.0.smithi032.stdout:0/104: truncate d3/da/f1d 4696319 0 2024-04-06T14:16:57.670 INFO:tasks.workunit.client.0.smithi032.stdout:8/185: mkdir d3/d8/d12/d2f/d4a 0 2024-04-06T14:16:57.673 INFO:tasks.workunit.client.0.smithi032.stdout:0/105: creat d3/da/f26 x:0 0 0 2024-04-06T14:16:57.676 INFO:tasks.workunit.client.0.smithi032.stdout:5/143: dwrite fa [0,4194304] 0 2024-04-06T14:16:57.677 INFO:tasks.workunit.client.0.smithi032.stdout:5/144: fdatasync fe 0 2024-04-06T14:16:57.679 INFO:tasks.workunit.client.0.smithi032.stdout:0/106: dread d3/da/fe [0,4194304] 0 2024-04-06T14:16:57.679 INFO:tasks.workunit.client.0.smithi032.stdout:8/186: dread d3/d8/d7/d1a/f24 [0,4194304] 0 2024-04-06T14:16:57.680 INFO:tasks.workunit.client.0.smithi032.stdout:5/145: mkdir d16/d18/d28/d2e 0 2024-04-06T14:16:57.680 INFO:tasks.workunit.client.0.smithi032.stdout:5/146: fdatasync d16/d18/f1c 0 2024-04-06T14:16:57.681 INFO:tasks.workunit.client.0.smithi032.stdout:0/107: symlink d3/l27 0 2024-04-06T14:16:57.682 INFO:tasks.workunit.client.0.smithi032.stdout:5/147: write d16/f2c [582603,86540] 0 2024-04-06T14:16:57.684 INFO:tasks.workunit.client.0.smithi032.stdout:5/148: write d16/d18/f2b [1331722,103689] 0 2024-04-06T14:16:57.685 INFO:tasks.workunit.client.0.smithi032.stdout:5/149: fdatasync d16/f1d 0 2024-04-06T14:16:57.687 INFO:tasks.workunit.client.0.smithi032.stdout:0/108: link d3/da/c22 d3/da/d25/c28 0 2024-04-06T14:16:57.688 INFO:tasks.workunit.client.0.smithi032.stdout:5/150: getdents d16/d18/d28/d2e 0 2024-04-06T14:16:57.689 INFO:tasks.workunit.client.0.smithi032.stdout:5/151: truncate d16/d18/f20 1515016 0 2024-04-06T14:16:57.689 INFO:tasks.workunit.client.0.smithi032.stdout:0/109: chown d3/l21 1541697 1 2024-04-06T14:16:57.691 INFO:tasks.workunit.client.0.smithi032.stdout:5/152: mknod d16/d18/c2f 0 2024-04-06T14:16:57.693 INFO:tasks.workunit.client.0.smithi032.stdout:5/153: truncate d16/f1d 393617 0 2024-04-06T14:16:57.694 INFO:tasks.workunit.client.0.smithi032.stdout:0/110: write d3/da/f23 [1584815,123498] 0 2024-04-06T14:16:57.696 INFO:tasks.workunit.client.0.smithi032.stdout:0/111: dread - d3/f24 zero size 2024-04-06T14:16:57.696 INFO:tasks.workunit.client.0.smithi032.stdout:8/187: dwrite d3/d8/d7/d14/f1f [0,4194304] 0 2024-04-06T14:16:57.697 INFO:tasks.workunit.client.0.smithi032.stdout:5/154: rename d16/d18/f29 to d16/d18/f30 0 2024-04-06T14:16:57.697 INFO:tasks.workunit.client.0.smithi032.stdout:8/188: fdatasync d3/d8/d3a/f42 0 2024-04-06T14:16:57.698 INFO:tasks.workunit.client.0.smithi032.stdout:5/155: truncate d16/f17 12224 0 2024-04-06T14:16:57.700 INFO:tasks.workunit.client.0.smithi032.stdout:8/189: fdatasync d3/d8/d12/f38 0 2024-04-06T14:16:57.702 INFO:tasks.workunit.client.0.smithi032.stdout:5/156: creat d16/d18/f31 x:0 0 0 2024-04-06T14:16:57.704 INFO:tasks.workunit.client.0.smithi032.stdout:8/190: mkdir d3/d8/d3a/d4b 0 2024-04-06T14:16:57.704 INFO:tasks.workunit.client.0.smithi032.stdout:5/157: write d16/f17 [818301,73232] 0 2024-04-06T14:16:57.706 INFO:tasks.workunit.client.0.smithi032.stdout:0/112: dwrite d3/f14 [0,4194304] 0 2024-04-06T14:16:57.706 INFO:tasks.workunit.client.0.smithi032.stdout:8/191: chown d3/d8/d7/d1a/c2d 11326 1 2024-04-06T14:16:57.708 INFO:tasks.workunit.client.0.smithi032.stdout:5/158: mknod d16/d18/d28/c32 0 2024-04-06T14:16:57.709 INFO:tasks.workunit.client.0.smithi032.stdout:0/113: unlink d3/da/l1a 0 2024-04-06T14:16:57.709 INFO:tasks.workunit.client.0.smithi032.stdout:0/114: stat d3/fd 0 2024-04-06T14:16:57.710 INFO:tasks.workunit.client.0.smithi032.stdout:5/159: mkdir d16/d18/d33 0 2024-04-06T14:16:57.712 INFO:tasks.workunit.client.0.smithi032.stdout:0/115: symlink d3/da/l29 0 2024-04-06T14:16:57.712 INFO:tasks.workunit.client.0.smithi032.stdout:8/192: dread d3/d8/d12/f3f [0,4194304] 0 2024-04-06T14:16:57.714 INFO:tasks.workunit.client.0.smithi032.stdout:5/160: creat d16/d18/d33/f34 x:0 0 0 2024-04-06T14:16:57.717 INFO:tasks.workunit.client.0.smithi032.stdout:0/116: creat d3/da/d25/f2a x:0 0 0 2024-04-06T14:16:57.718 INFO:tasks.workunit.client.0.smithi032.stdout:5/161: mkdir d16/d18/d33/d35 0 2024-04-06T14:16:57.719 INFO:tasks.workunit.client.0.smithi032.stdout:0/117: mkdir d3/da/d25/d2b 0 2024-04-06T14:16:57.720 INFO:tasks.workunit.client.0.smithi032.stdout:5/162: symlink d16/l36 0 2024-04-06T14:16:57.721 INFO:tasks.workunit.client.0.smithi032.stdout:0/118: symlink d3/da/l2c 0 2024-04-06T14:16:57.722 INFO:tasks.workunit.client.0.smithi032.stdout:0/119: creat d3/da/f2d x:0 0 0 2024-04-06T14:16:57.726 INFO:tasks.workunit.client.0.smithi032.stdout:8/193: rmdir d3/d8/d3a 39 2024-04-06T14:16:57.729 INFO:tasks.workunit.client.0.smithi032.stdout:0/120: dwrite d3/da/f18 [0,4194304] 0 2024-04-06T14:16:57.731 INFO:tasks.workunit.client.0.smithi032.stdout:8/194: creat d3/d8/d3a/f4c x:0 0 0 2024-04-06T14:16:57.732 INFO:tasks.workunit.client.0.smithi032.stdout:0/121: creat d3/f2e x:0 0 0 2024-04-06T14:16:57.733 INFO:tasks.workunit.client.0.smithi032.stdout:0/122: stat d3/f4 0 2024-04-06T14:16:57.736 INFO:tasks.workunit.client.0.smithi032.stdout:1/140: sync 2024-04-06T14:16:57.736 INFO:tasks.workunit.client.0.smithi032.stdout:3/128: sync 2024-04-06T14:16:57.736 INFO:tasks.workunit.client.0.smithi032.stdout:8/195: truncate d3/d8/d12/f38 960694 0 2024-04-06T14:16:57.739 INFO:tasks.workunit.client.0.smithi032.stdout:0/123: rename d3/da/lc to d3/da/l2f 0 2024-04-06T14:16:57.741 INFO:tasks.workunit.client.0.smithi032.stdout:0/124: read d3/da/f18 [1636382,62195] 0 2024-04-06T14:16:57.743 INFO:tasks.workunit.client.0.smithi032.stdout:3/129: link d3/f21 d3/f37 0 2024-04-06T14:16:57.748 INFO:tasks.workunit.client.0.smithi032.stdout:8/196: dwrite d3/d8/d7/d14/f45 [0,4194304] 0 2024-04-06T14:16:57.750 INFO:tasks.workunit.client.0.smithi032.stdout:5/163: dwrite f5 [0,4194304] 0 2024-04-06T14:16:57.750 INFO:tasks.workunit.client.0.smithi032.stdout:1/141: write d0/d9/fd [2828357,13235] 0 2024-04-06T14:16:57.751 INFO:tasks.workunit.client.0.smithi032.stdout:8/197: stat d3/d8/l33 0 2024-04-06T14:16:57.752 INFO:tasks.workunit.client.0.smithi032.stdout:8/198: readlink d3/l41 0 2024-04-06T14:16:57.754 INFO:tasks.workunit.client.0.smithi032.stdout:5/164: rmdir d16/d18/d33 39 2024-04-06T14:16:57.756 INFO:tasks.workunit.client.0.smithi032.stdout:5/165: write d16/f1d [1412632,110767] 0 2024-04-06T14:16:57.756 INFO:tasks.workunit.client.0.smithi032.stdout:1/142: unlink d0/l1c 0 2024-04-06T14:16:57.757 INFO:tasks.workunit.client.0.smithi032.stdout:0/125: link d3/l27 d3/da/l30 0 2024-04-06T14:16:57.758 INFO:tasks.workunit.client.0.smithi032.stdout:1/143: chown d0/da/cc 937688822 1 2024-04-06T14:16:57.759 INFO:tasks.workunit.client.0.smithi032.stdout:1/144: dread - d0/f2c zero size 2024-04-06T14:16:57.761 INFO:tasks.workunit.client.0.smithi032.stdout:3/130: creat d3/d16/f38 x:0 0 0 2024-04-06T14:16:57.761 INFO:tasks.workunit.client.0.smithi032.stdout:1/145: write d0/d9/f24 [2691730,46620] 0 2024-04-06T14:16:57.762 INFO:tasks.workunit.client.0.smithi032.stdout:0/126: mknod d3/da/d25/d2b/c31 0 2024-04-06T14:16:57.762 INFO:tasks.workunit.client.0.smithi032.stdout:8/199: creat d3/d8/d12/f4d x:0 0 0 2024-04-06T14:16:57.764 INFO:tasks.workunit.client.0.smithi032.stdout:1/146: write d0/da/f28 [617855,97014] 0 2024-04-06T14:16:57.764 INFO:tasks.workunit.client.0.smithi032.stdout:5/166: rename d16/d18/d28/d2e to d16/d18/d33/d35/d37 0 2024-04-06T14:16:57.765 INFO:tasks.workunit.client.0.smithi032.stdout:1/147: stat d0/d9/fd 0 2024-04-06T14:16:57.766 INFO:tasks.workunit.client.0.smithi032.stdout:3/131: creat d3/d16/d1e/f39 x:0 0 0 2024-04-06T14:16:57.767 INFO:tasks.workunit.client.0.smithi032.stdout:8/200: dread d3/d8/d12/f3f [0,4194304] 0 2024-04-06T14:16:57.767 INFO:tasks.workunit.client.0.smithi032.stdout:1/148: write d0/d9/d1f/f22 [990732,56545] 0 2024-04-06T14:16:57.768 INFO:tasks.workunit.client.0.smithi032.stdout:0/127: creat d3/da/d25/d2b/f32 x:0 0 0 2024-04-06T14:16:57.771 INFO:tasks.workunit.client.0.smithi032.stdout:1/149: mknod d0/da/c2f 0 2024-04-06T14:16:57.772 INFO:tasks.workunit.client.0.smithi032.stdout:8/201: symlink d3/d8/d7/l4e 0 2024-04-06T14:16:57.772 INFO:tasks.workunit.client.0.smithi032.stdout:1/150: readlink d0/d10/d16/l2d 0 2024-04-06T14:16:57.773 INFO:tasks.workunit.client.0.smithi032.stdout:3/132: symlink d3/l3a 0 2024-04-06T14:16:57.773 INFO:tasks.workunit.client.0.smithi032.stdout:0/128: mknod d3/da/d25/d2b/c33 0 2024-04-06T14:16:57.774 INFO:tasks.workunit.client.0.smithi032.stdout:8/202: readlink d3/d8/d12/l23 0 2024-04-06T14:16:57.774 INFO:tasks.workunit.client.0.smithi032.stdout:5/167: dwrite d16/d18/f20 [0,4194304] 0 2024-04-06T14:16:57.775 INFO:tasks.workunit.client.0.smithi032.stdout:0/129: stat d3/f20 0 2024-04-06T14:16:57.776 INFO:tasks.workunit.client.0.smithi032.stdout:1/151: creat d0/d10/f30 x:0 0 0 2024-04-06T14:16:57.776 INFO:tasks.workunit.client.0.smithi032.stdout:5/168: chown f1 7368815 1 2024-04-06T14:16:57.777 INFO:tasks.workunit.client.0.smithi032.stdout:3/133: unlink d3/d16/f2e 0 2024-04-06T14:16:57.778 INFO:tasks.workunit.client.0.smithi032.stdout:3/134: stat d3/d16/d1e/f32 0 2024-04-06T14:16:57.779 INFO:tasks.workunit.client.0.smithi032.stdout:5/169: creat d16/f38 x:0 0 0 2024-04-06T14:16:57.780 INFO:tasks.workunit.client.0.smithi032.stdout:1/152: symlink d0/da/l31 0 2024-04-06T14:16:57.781 INFO:tasks.workunit.client.0.smithi032.stdout:5/170: unlink d16/l26 0 2024-04-06T14:16:57.783 INFO:tasks.workunit.client.0.smithi032.stdout:3/135: mkdir d3/d3b 0 2024-04-06T14:16:57.784 INFO:tasks.workunit.client.0.smithi032.stdout:1/153: creat d0/d10/f32 x:0 0 0 2024-04-06T14:16:57.785 INFO:tasks.workunit.client.0.smithi032.stdout:8/203: getdents d3/d8/d7/d1a 0 2024-04-06T14:16:57.785 INFO:tasks.workunit.client.0.smithi032.stdout:5/171: mkdir d16/d18/d33/d35/d37/d39 0 2024-04-06T14:16:57.788 INFO:tasks.workunit.client.0.smithi032.stdout:1/154: symlink d0/d10/d16/l33 0 2024-04-06T14:16:57.790 INFO:tasks.workunit.client.0.smithi032.stdout:8/204: write d3/f10 [1210400,127949] 0 2024-04-06T14:16:57.790 INFO:tasks.workunit.client.0.smithi032.stdout:0/130: getdents d3/da/d25/d2b 0 2024-04-06T14:16:57.791 INFO:tasks.workunit.client.0.smithi032.stdout:3/136: mkdir d3/d3b/d3c 0 2024-04-06T14:16:57.792 INFO:tasks.workunit.client.0.smithi032.stdout:8/205: write d3/d8/d12/f4d [408211,97606] 0 2024-04-06T14:16:57.792 INFO:tasks.workunit.client.0.smithi032.stdout:1/155: symlink d0/da/l34 0 2024-04-06T14:16:57.794 INFO:tasks.workunit.client.0.smithi032.stdout:3/137: mknod d3/d3b/c3d 0 2024-04-06T14:16:57.798 INFO:tasks.workunit.client.0.smithi032.stdout:0/131: dwrite d3/da/d25/f2a [0,4194304] 0 2024-04-06T14:16:57.798 INFO:tasks.workunit.client.0.smithi032.stdout:0/132: readlink d3/l15 0 2024-04-06T14:16:57.799 INFO:tasks.workunit.client.0.smithi032.stdout:0/133: dread - d3/f24 zero size 2024-04-06T14:16:57.800 INFO:tasks.workunit.client.0.smithi032.stdout:3/138: unlink d3/l3a 0 2024-04-06T14:16:57.802 INFO:tasks.workunit.client.0.smithi032.stdout:8/206: getdents d3/d8/d12/d2f/d46 0 2024-04-06T14:16:57.803 INFO:tasks.workunit.client.0.smithi032.stdout:0/134: symlink d3/da/l34 0 2024-04-06T14:16:57.804 INFO:tasks.workunit.client.0.smithi032.stdout:5/172: truncate f5 589156 0 2024-04-06T14:16:57.807 INFO:tasks.workunit.client.0.smithi032.stdout:8/207: creat d3/d8/d3a/f4f x:0 0 0 2024-04-06T14:16:57.807 INFO:tasks.workunit.client.0.smithi032.stdout:1/156: write d0/d9/f23 [1393575,54149] 0 2024-04-06T14:16:57.808 INFO:tasks.workunit.client.0.smithi032.stdout:5/173: mknod d16/c3a 0 2024-04-06T14:16:57.810 INFO:tasks.workunit.client.0.smithi032.stdout:3/139: dread d3/fc [0,4194304] 0 2024-04-06T14:16:57.810 INFO:tasks.workunit.client.0.smithi032.stdout:1/157: mkdir d0/d35 0 2024-04-06T14:16:57.814 INFO:tasks.workunit.client.0.smithi032.stdout:3/140: mknod d3/d16/d1e/c3e 0 2024-04-06T14:16:57.815 INFO:tasks.workunit.client.0.smithi032.stdout:5/174: dread d16/f2c [0,4194304] 0 2024-04-06T14:16:57.815 INFO:tasks.workunit.client.0.smithi032.stdout:0/135: dwrite d3/da/d25/d2b/f32 [0,4194304] 0 2024-04-06T14:16:57.817 INFO:tasks.workunit.client.0.smithi032.stdout:2/155: sync 2024-04-06T14:16:57.818 INFO:tasks.workunit.client.0.smithi032.stdout:4/320: sync 2024-04-06T14:16:57.821 INFO:tasks.workunit.client.0.smithi032.stdout:0/136: mkdir d3/da/d35 0 2024-04-06T14:16:57.821 INFO:tasks.workunit.client.0.smithi032.stdout:8/208: dwrite d3/d8/d3a/f4c [0,4194304] 0 2024-04-06T14:16:57.822 INFO:tasks.workunit.client.0.smithi032.stdout:2/156: truncate d0/f1b 490482 0 2024-04-06T14:16:57.826 INFO:tasks.workunit.client.0.smithi032.stdout:1/158: dwrite d0/d9/f24 [0,4194304] 0 2024-04-06T14:16:57.827 INFO:tasks.workunit.client.0.smithi032.stdout:0/137: write d3/da/f1e [237089,32282] 0 2024-04-06T14:16:57.828 INFO:tasks.workunit.client.0.smithi032.stdout:3/141: rename d3/d16/f1a to d3/f3f 0 2024-04-06T14:16:57.829 INFO:tasks.workunit.client.0.smithi032.stdout:5/175: symlink d16/d18/d33/d35/d37/d39/l3b 0 2024-04-06T14:16:57.829 INFO:tasks.workunit.client.0.smithi032.stdout:3/142: dread - d3/d16/f35 zero size 2024-04-06T14:16:57.832 INFO:tasks.workunit.client.0.smithi032.stdout:2/157: symlink d0/d12/l3b 0 2024-04-06T14:16:57.833 INFO:tasks.workunit.client.0.smithi032.stdout:1/159: creat d0/f36 x:0 0 0 2024-04-06T14:16:57.834 INFO:tasks.workunit.client.0.smithi032.stdout:1/160: chown d0/d9/l19 735837186 1 2024-04-06T14:16:57.834 INFO:tasks.workunit.client.0.smithi032.stdout:0/138: mkdir d3/da/d25/d36 0 2024-04-06T14:16:57.836 INFO:tasks.workunit.client.0.smithi032.stdout:3/143: creat d3/d3b/f40 x:0 0 0 2024-04-06T14:16:57.837 INFO:tasks.workunit.client.0.smithi032.stdout:4/321: getdents d0 0 2024-04-06T14:16:57.840 INFO:tasks.workunit.client.0.smithi032.stdout:2/158: symlink d0/l3c 0 2024-04-06T14:16:57.845 INFO:tasks.workunit.client.0.smithi032.stdout:3/144: dwrite d3/f19 [4194304,4194304] 0 2024-04-06T14:16:57.848 INFO:tasks.workunit.client.0.smithi032.stdout:0/139: symlink d3/da/d25/d36/l37 0 2024-04-06T14:16:57.853 INFO:tasks.workunit.client.0.smithi032.stdout:5/176: rename ff to d16/d18/f3c 0 2024-04-06T14:16:57.855 INFO:tasks.workunit.client.0.smithi032.stdout:2/159: chown d0/l9 1692564 1 2024-04-06T14:16:57.858 INFO:tasks.workunit.client.0.smithi032.stdout:6/173: sync 2024-04-06T14:16:57.858 INFO:tasks.workunit.client.0.smithi032.stdout:9/199: sync 2024-04-06T14:16:57.858 INFO:tasks.workunit.client.0.smithi032.stdout:7/207: sync 2024-04-06T14:16:57.859 INFO:tasks.workunit.client.0.smithi032.stdout:8/209: dread d3/d8/d12/f1b [0,4194304] 0 2024-04-06T14:16:57.859 INFO:tasks.workunit.client.0.smithi032.stdout:9/200: stat d2/d15/c3f 0 2024-04-06T14:16:57.865 INFO:tasks.workunit.client.0.smithi032.stdout:1/161: rename d0/d10/f32 to d0/f37 0 2024-04-06T14:16:57.868 INFO:tasks.workunit.client.0.smithi032.stdout:7/208: mkdir d16/d20/d48 0 2024-04-06T14:16:57.896 INFO:tasks.workunit.client.0.smithi032.stdout:2/160: symlink d0/d30/d39/l3d 0 2024-04-06T14:16:57.897 INFO:tasks.workunit.client.0.smithi032.stdout:0/140: write d3/da/ff [1528840,38983] 0 2024-04-06T14:16:57.900 INFO:tasks.workunit.client.0.smithi032.stdout:0/141: truncate d3/f24 229659 0 2024-04-06T14:16:57.900 INFO:tasks.workunit.client.0.smithi032.stdout:6/174: unlink d4/d5/c1b 0 2024-04-06T14:16:57.900 INFO:tasks.workunit.client.0.smithi032.stdout:1/162: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:57.901 INFO:tasks.workunit.client.0.smithi032.stdout:6/175: readlink d4/d5/l20 0 2024-04-06T14:16:57.901 INFO:tasks.workunit.client.0.smithi032.stdout:0/142: truncate d3/f2e 879086 0 2024-04-06T14:16:57.904 INFO:tasks.workunit.client.0.smithi032.stdout:2/161: dwrite d0/f24 [4194304,4194304] 0 2024-04-06T14:16:57.905 INFO:tasks.workunit.client.0.smithi032.stdout:6/176: dread d4/d5/d18/d23/f27 [0,4194304] 0 2024-04-06T14:16:57.928 INFO:tasks.workunit.client.0.smithi032.stdout:7/209: creat d16/f49 x:0 0 0 2024-04-06T14:16:57.930 INFO:tasks.workunit.client.0.smithi032.stdout:9/201: fsync d2/d5/f44 0 2024-04-06T14:16:57.931 INFO:tasks.workunit.client.0.smithi032.stdout:8/210: symlink d3/d8/d12/d2f/d46/l50 0 2024-04-06T14:16:57.932 INFO:tasks.workunit.client.0.smithi032.stdout:5/177: creat d16/d18/f3d x:0 0 0 2024-04-06T14:16:57.934 INFO:tasks.workunit.client.0.smithi032.stdout:3/145: getdents d3/d16 0 2024-04-06T14:16:57.936 INFO:tasks.workunit.client.0.smithi032.stdout:5/178: dread fc [0,4194304] 0 2024-04-06T14:16:57.936 INFO:tasks.workunit.client.0.smithi032.stdout:0/143: readlink d3/l27 0 2024-04-06T14:16:57.937 INFO:tasks.workunit.client.0.smithi032.stdout:1/163: creat d0/da/f38 x:0 0 0 2024-04-06T14:16:57.940 INFO:tasks.workunit.client.0.smithi032.stdout:9/202: write d2/d5/f1b [46061,97422] 0 2024-04-06T14:16:57.941 INFO:tasks.workunit.client.0.smithi032.stdout:4/322: rename d0/df/d2b/d4f/d52 to d0/df/d76 0 2024-04-06T14:16:57.943 INFO:tasks.workunit.client.0.smithi032.stdout:4/323: write d0/df/d23/d31/d50/d57/f64 [403913,98837] 0 2024-04-06T14:16:57.944 INFO:tasks.workunit.client.0.smithi032.stdout:3/146: mknod d3/d3b/c41 0 2024-04-06T14:16:57.945 INFO:tasks.workunit.client.0.smithi032.stdout:5/179: creat d16/d18/d33/d35/d37/d39/f3e x:0 0 0 2024-04-06T14:16:57.946 INFO:tasks.workunit.client.0.smithi032.stdout:0/144: mknod d3/da/d25/d2b/c38 0 2024-04-06T14:16:57.946 INFO:tasks.workunit.client.0.smithi032.stdout:6/177: write d4/d5/d8/fc [1158283,125662] 0 2024-04-06T14:16:57.947 INFO:tasks.workunit.client.0.smithi032.stdout:6/178: write d4/d5/d8/fc [1780098,106571] 0 2024-04-06T14:16:57.948 INFO:tasks.workunit.client.0.smithi032.stdout:7/210: creat d16/d20/d48/f4a x:0 0 0 2024-04-06T14:16:57.949 INFO:tasks.workunit.client.0.smithi032.stdout:2/162: rename d0 to d0/d3e 22 2024-04-06T14:16:57.949 INFO:tasks.workunit.client.0.smithi032.stdout:9/203: creat d2/d25/f46 x:0 0 0 2024-04-06T14:16:57.950 INFO:tasks.workunit.client.0.smithi032.stdout:4/324: rename d0 to d0/df/d2b/d66/d77 22 2024-04-06T14:16:57.951 INFO:tasks.workunit.client.0.smithi032.stdout:9/204: truncate d2/d5/f14 4857033 0 2024-04-06T14:16:57.951 INFO:tasks.workunit.client.0.smithi032.stdout:3/147: mknod d3/d3b/c42 0 2024-04-06T14:16:57.951 INFO:tasks.workunit.client.0.smithi032.stdout:5/180: mknod d16/d18/d28/c3f 0 2024-04-06T14:16:57.953 INFO:tasks.workunit.client.0.smithi032.stdout:0/145: creat d3/da/d25/d36/f39 x:0 0 0 2024-04-06T14:16:57.954 INFO:tasks.workunit.client.0.smithi032.stdout:7/211: creat d16/f4b x:0 0 0 2024-04-06T14:16:57.955 INFO:tasks.workunit.client.0.smithi032.stdout:8/211: creat d3/d8/d12/f51 x:0 0 0 2024-04-06T14:16:57.955 INFO:tasks.workunit.client.0.smithi032.stdout:4/325: mknod d0/df/d1f/d42/c78 0 2024-04-06T14:16:57.958 INFO:tasks.workunit.client.0.smithi032.stdout:6/179: creat d4/d5/d28/f29 x:0 0 0 2024-04-06T14:16:57.959 INFO:tasks.workunit.client.0.smithi032.stdout:0/146: creat d3/da/d25/d2b/f3a x:0 0 0 2024-04-06T14:16:57.959 INFO:tasks.workunit.client.0.smithi032.stdout:9/205: dwrite d2/d25/f45 [0,4194304] 0 2024-04-06T14:16:57.959 INFO:tasks.workunit.client.0.smithi032.stdout:4/326: fsync d0/df/d1f/d42/d45/f75 0 2024-04-06T14:16:57.961 INFO:tasks.workunit.client.0.smithi032.stdout:2/163: rename d0/c38 to d0/d30/d39/c3f 0 2024-04-06T14:16:57.964 INFO:tasks.workunit.client.0.smithi032.stdout:1/164: getdents d0 0 2024-04-06T14:16:57.965 INFO:tasks.workunit.client.0.smithi032.stdout:6/180: rmdir d4/d5/d14 39 2024-04-06T14:16:57.965 INFO:tasks.workunit.client.0.smithi032.stdout:2/164: stat d0/d12/f2a 0 2024-04-06T14:16:57.967 INFO:tasks.workunit.client.0.smithi032.stdout:3/148: creat d3/f43 x:0 0 0 2024-04-06T14:16:57.968 INFO:tasks.workunit.client.0.smithi032.stdout:0/147: mknod d3/da/d25/d36/c3b 0 2024-04-06T14:16:57.969 INFO:tasks.workunit.client.0.smithi032.stdout:1/165: write d0/d10/f26 [1505227,113692] 0 2024-04-06T14:16:57.970 INFO:tasks.workunit.client.0.smithi032.stdout:9/206: mkdir d2/d15/d1e/d47 0 2024-04-06T14:16:57.970 INFO:tasks.workunit.client.0.smithi032.stdout:2/165: write d0/d12/f17 [138740,51393] 0 2024-04-06T14:16:57.971 INFO:tasks.workunit.client.0.smithi032.stdout:7/212: rename d16/d20/d26/d2d/c3b to d16/d20/d26/c4c 0 2024-04-06T14:16:57.972 INFO:tasks.workunit.client.0.smithi032.stdout:3/149: chown d3/d16/f2a 24712 1 2024-04-06T14:16:57.972 INFO:tasks.workunit.client.0.smithi032.stdout:8/212: dwrite d3/d8/d7/f16 [0,4194304] 0 2024-04-06T14:16:57.974 INFO:tasks.workunit.client.0.smithi032.stdout:1/166: symlink d0/d10/l39 0 2024-04-06T14:16:57.976 INFO:tasks.workunit.client.0.smithi032.stdout:2/166: truncate d0/f28 5039970 0 2024-04-06T14:16:57.978 INFO:tasks.workunit.client.0.smithi032.stdout:0/148: write d3/da/fe [356936,55862] 0 2024-04-06T14:16:57.983 INFO:tasks.workunit.client.0.smithi032.stdout:9/207: symlink d2/d5/l48 0 2024-04-06T14:16:57.984 INFO:tasks.workunit.client.0.smithi032.stdout:8/213: creat d3/d8/d12/d2f/d46/f52 x:0 0 0 2024-04-06T14:16:57.986 INFO:tasks.workunit.client.0.smithi032.stdout:4/327: dwrite d0/df/d23/f3d [0,4194304] 0 2024-04-06T14:16:57.986 INFO:tasks.workunit.client.0.smithi032.stdout:1/167: creat d0/d9/d1f/f3a x:0 0 0 2024-04-06T14:16:57.987 INFO:tasks.workunit.client.0.smithi032.stdout:6/181: dwrite d4/d5/f15 [4194304,4194304] 0 2024-04-06T14:16:57.989 INFO:tasks.workunit.client.0.smithi032.stdout:2/167: creat d0/d12/d3a/f40 x:0 0 0 2024-04-06T14:16:57.991 INFO:tasks.workunit.client.0.smithi032.stdout:8/214: write d3/f31 [331692,112701] 0 2024-04-06T14:16:57.992 INFO:tasks.workunit.client.0.smithi032.stdout:9/208: rmdir d2/d15/d1e 39 2024-04-06T14:16:57.993 INFO:tasks.workunit.client.0.smithi032.stdout:2/168: fdatasync d0/d30/f36 0 2024-04-06T14:16:57.993 INFO:tasks.workunit.client.0.smithi032.stdout:6/182: chown d4/d5/d8/d1f 11559335 1 2024-04-06T14:16:57.997 INFO:tasks.workunit.client.0.smithi032.stdout:1/168: creat d0/d9/d1f/f3b x:0 0 0 2024-04-06T14:16:57.998 INFO:tasks.workunit.client.0.smithi032.stdout:6/183: readlink d4/d5/l20 0 2024-04-06T14:16:58.001 INFO:tasks.workunit.client.0.smithi032.stdout:3/150: dwrite d3/f3f [0,4194304] 0 2024-04-06T14:16:58.001 INFO:tasks.workunit.client.0.smithi032.stdout:0/149: getdents d3/da/d25/d2b 0 2024-04-06T14:16:58.002 INFO:tasks.workunit.client.0.smithi032.stdout:7/213: dwrite d16/d20/d26/f3a [0,4194304] 0 2024-04-06T14:16:58.003 INFO:tasks.workunit.client.0.smithi032.stdout:7/214: readlink d16/l18 0 2024-04-06T14:16:58.004 INFO:tasks.workunit.client.0.smithi032.stdout:0/150: chown d3/da/d25/d2b/f32 4 1 2024-04-06T14:16:58.004 INFO:tasks.workunit.client.0.smithi032.stdout:7/215: chown d16/l1a 680851 1 2024-04-06T14:16:58.005 INFO:tasks.workunit.client.0.smithi032.stdout:8/215: unlink d3/d8/d7/d14/f1c 0 2024-04-06T14:16:58.006 INFO:tasks.workunit.client.0.smithi032.stdout:4/328: mknod d0/df/d23/d31/d50/d57/d5b/c79 0 2024-04-06T14:16:58.006 INFO:tasks.workunit.client.0.smithi032.stdout:3/151: rename d3/d3b/c3d to d3/d16/d1e/c44 0 2024-04-06T14:16:58.008 INFO:tasks.workunit.client.0.smithi032.stdout:7/216: symlink d16/d20/l4d 0 2024-04-06T14:16:58.009 INFO:tasks.workunit.client.0.smithi032.stdout:4/329: fsync d0/df/d2b/d46/f6e 0 2024-04-06T14:16:58.010 INFO:tasks.workunit.client.0.smithi032.stdout:1/169: link d0/f2c d0/d10/d16/f3c 0 2024-04-06T14:16:58.010 INFO:tasks.workunit.client.0.smithi032.stdout:4/330: dread - d0/df/d1f/d42/d45/f53 zero size 2024-04-06T14:16:58.011 INFO:tasks.workunit.client.0.smithi032.stdout:9/209: creat d2/d5/f49 x:0 0 0 2024-04-06T14:16:58.012 INFO:tasks.workunit.client.0.smithi032.stdout:1/170: dread d0/f17 [0,4194304] 0 2024-04-06T14:16:58.013 INFO:tasks.workunit.client.0.smithi032.stdout:1/171: write d0/d9/f1b [4364331,28256] 0 2024-04-06T14:16:58.017 INFO:tasks.workunit.client.0.smithi032.stdout:8/216: creat d3/d8/d12/d2f/d46/f53 x:0 0 0 2024-04-06T14:16:58.018 INFO:tasks.workunit.client.0.smithi032.stdout:6/184: creat d4/d5/f2a x:0 0 0 2024-04-06T14:16:58.018 INFO:tasks.workunit.client.0.smithi032.stdout:7/217: write d16/d20/f47 [1617856,121041] 0 2024-04-06T14:16:58.018 INFO:tasks.workunit.client.0.smithi032.stdout:2/169: getdents d0/d30 0 2024-04-06T14:16:58.113 INFO:tasks.workunit.client.0.smithi032.stdout:3/152: symlink d3/l45 0 2024-04-06T14:16:58.114 INFO:tasks.workunit.client.0.smithi032.stdout:9/210: read d2/d5/f12 [253830,25300] 0 2024-04-06T14:16:58.114 INFO:tasks.workunit.client.0.smithi032.stdout:6/185: mkdir d4/d5/d8/d2b 0 2024-04-06T14:16:58.115 INFO:tasks.workunit.client.0.smithi032.stdout:3/153: write d3/d16/f38 [792957,117675] 0 2024-04-06T14:16:58.115 INFO:tasks.workunit.client.0.smithi032.stdout:0/151: truncate d3/da/f1d 4168600 0 2024-04-06T14:16:58.116 INFO:tasks.workunit.client.0.smithi032.stdout:0/152: rename d3/da to d3/da/d35/d3c 22 2024-04-06T14:16:58.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:57 smithi086 ceph-mon[47925]: pgmap v16: 65 pgs: 65 active+clean; 399 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 494 KiB/s rd, 29 MiB/s wr, 129 op/s 2024-04-06T14:16:58.165 INFO:tasks.workunit.client.0.smithi032.stdout:8/217: getdents d3/d8/d3a/d4b 0 2024-04-06T14:16:58.165 INFO:tasks.workunit.client.0.smithi032.stdout:7/218: getdents d16/d20/d26 0 2024-04-06T14:16:58.166 INFO:tasks.workunit.client.0.smithi032.stdout:5/181: sync 2024-04-06T14:16:58.168 INFO:tasks.workunit.client.0.smithi032.stdout:0/153: dread - d3/da/d25/d2b/f3a zero size 2024-04-06T14:16:58.168 INFO:tasks.workunit.client.0.smithi032.stdout:6/186: truncate d4/f10 3539591 0 2024-04-06T14:16:58.168 INFO:tasks.workunit.client.0.smithi032.stdout:9/211: mkdir d2/d15/d1e/d47/d4a 0 2024-04-06T14:16:58.169 INFO:tasks.workunit.client.0.smithi032.stdout:2/170: truncate d0/d12/f1a 2463881 0 2024-04-06T14:16:58.169 INFO:tasks.workunit.client.0.smithi032.stdout:7/219: stat d16/d20/d26/c29 0 2024-04-06T14:16:58.169 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:57 smithi032 ceph-mon[39730]: pgmap v16: 65 pgs: 65 active+clean; 399 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 494 KiB/s rd, 29 MiB/s wr, 129 op/s 2024-04-06T14:16:58.170 INFO:tasks.workunit.client.0.smithi032.stdout:8/218: chown d3/d8/l2b 5572 1 2024-04-06T14:16:58.171 INFO:tasks.workunit.client.0.smithi032.stdout:0/154: fsync d3/da/d25/d36/f39 0 2024-04-06T14:16:58.172 INFO:tasks.workunit.client.0.smithi032.stdout:2/171: chown d0/d12/d3a/f40 1887194 1 2024-04-06T14:16:58.172 INFO:tasks.workunit.client.0.smithi032.stdout:5/182: mkdir d16/d18/d33/d35/d37/d39/d40 0 2024-04-06T14:16:58.172 INFO:tasks.workunit.client.0.smithi032.stdout:5/183: readlink l2 0 2024-04-06T14:16:58.172 INFO:tasks.workunit.client.0.smithi032.stdout:2/172: stat d0/c2d 0 2024-04-06T14:16:58.173 INFO:tasks.workunit.client.0.smithi032.stdout:3/154: dwrite d3/f3f [0,4194304] 0 2024-04-06T14:16:58.173 INFO:tasks.workunit.client.0.smithi032.stdout:7/220: rename d16/c2f to d16/c4e 0 2024-04-06T14:16:58.176 INFO:tasks.workunit.client.0.smithi032.stdout:9/212: creat d2/d15/d1e/d47/f4b x:0 0 0 2024-04-06T14:16:58.177 INFO:tasks.workunit.client.0.smithi032.stdout:6/187: getdents d4 0 2024-04-06T14:16:58.177 INFO:tasks.workunit.client.0.smithi032.stdout:8/219: creat d3/d8/d7/d1a/f54 x:0 0 0 2024-04-06T14:16:58.228 INFO:tasks.workunit.client.0.smithi032.stdout:9/213: chown d2/d15/l38 1204 1 2024-04-06T14:16:58.230 INFO:tasks.workunit.client.0.smithi032.stdout:5/184: dread d16/d18/f20 [0,4194304] 0 2024-04-06T14:16:58.231 INFO:tasks.workunit.client.0.smithi032.stdout:9/214: write d2/d15/f29 [36186,92979] 0 2024-04-06T14:16:58.232 INFO:tasks.workunit.client.0.smithi032.stdout:2/173: dwrite d0/fc [0,4194304] 0 2024-04-06T14:16:58.234 INFO:tasks.workunit.client.0.smithi032.stdout:5/185: write d16/f2c [302491,36484] 0 2024-04-06T14:16:58.236 INFO:tasks.workunit.client.0.smithi032.stdout:2/174: creat d0/d12/f41 x:0 0 0 2024-04-06T14:16:58.236 INFO:tasks.workunit.client.0.smithi032.stdout:6/188: link d4/ce d4/d5/d18/d23/c2c 0 2024-04-06T14:16:58.238 INFO:tasks.workunit.client.0.smithi032.stdout:5/186: write d16/d18/d33/d35/d37/d39/f3e [962737,89998] 0 2024-04-06T14:16:58.238 INFO:tasks.workunit.client.0.smithi032.stdout:3/155: truncate d3/f24 1124177 0 2024-04-06T14:16:58.239 INFO:tasks.workunit.client.0.smithi032.stdout:9/215: write d2/f4 [8950735,37984] 0 2024-04-06T14:16:58.240 INFO:tasks.workunit.client.0.smithi032.stdout:5/187: write f10 [1656284,79168] 0 2024-04-06T14:16:58.240 INFO:tasks.workunit.client.0.smithi032.stdout:2/175: creat d0/d12/d3a/f42 x:0 0 0 2024-04-06T14:16:58.241 INFO:tasks.workunit.client.0.smithi032.stdout:5/188: fdatasync fb 0 2024-04-06T14:16:58.242 INFO:tasks.workunit.client.0.smithi032.stdout:3/156: mknod d3/d3b/c46 0 2024-04-06T14:16:58.243 INFO:tasks.workunit.client.0.smithi032.stdout:9/216: truncate d2/d15/f2b 341051 0 2024-04-06T14:16:58.247 INFO:tasks.workunit.client.0.smithi032.stdout:3/157: creat d3/d16/d1e/f47 x:0 0 0 2024-04-06T14:16:58.249 INFO:tasks.workunit.client.0.smithi032.stdout:6/189: write d4/d5/d18/d23/f27 [689044,38699] 0 2024-04-06T14:16:58.249 INFO:tasks.workunit.client.0.smithi032.stdout:6/190: chown d4/d5 3978 1 2024-04-06T14:16:58.250 INFO:tasks.workunit.client.0.smithi032.stdout:8/220: dread d3/d8/fc [0,4194304] 0 2024-04-06T14:16:58.250 INFO:tasks.workunit.client.0.smithi032.stdout:3/158: creat d3/d3b/f48 x:0 0 0 2024-04-06T14:16:58.251 INFO:tasks.workunit.client.0.smithi032.stdout:5/189: rmdir d16/d18/d33/d35/d37/d39/d40 0 2024-04-06T14:16:58.252 INFO:tasks.workunit.client.0.smithi032.stdout:3/159: write d3/d16/f35 [374373,45152] 0 2024-04-06T14:16:58.254 INFO:tasks.workunit.client.0.smithi032.stdout:8/221: creat d3/d8/d12/d2f/d46/f55 x:0 0 0 2024-04-06T14:16:58.256 INFO:tasks.workunit.client.0.smithi032.stdout:9/217: getdents d2/d25 0 2024-04-06T14:16:58.256 INFO:tasks.workunit.client.0.smithi032.stdout:8/222: dread d3/d8/d7/f47 [0,4194304] 0 2024-04-06T14:16:58.257 INFO:tasks.workunit.client.0.smithi032.stdout:5/190: creat d16/d18/d33/f41 x:0 0 0 2024-04-06T14:16:58.262 INFO:tasks.workunit.client.0.smithi032.stdout:9/218: creat d2/d15/d1e/d47/f4c x:0 0 0 2024-04-06T14:16:58.262 INFO:tasks.workunit.client.0.smithi032.stdout:6/191: write d4/f10 [1101154,55751] 0 2024-04-06T14:16:58.263 INFO:tasks.workunit.client.0.smithi032.stdout:5/191: rmdir d16/d18/d33/d35 39 2024-04-06T14:16:58.264 INFO:tasks.workunit.client.0.smithi032.stdout:2/176: dread d0/f13 [0,4194304] 0 2024-04-06T14:16:58.265 INFO:tasks.workunit.client.0.smithi032.stdout:8/223: rename d3/d8/d12/d2f/d46/f52 to d3/d8/d3a/f56 0 2024-04-06T14:16:58.269 INFO:tasks.workunit.client.0.smithi032.stdout:2/177: fdatasync d0/f24 0 2024-04-06T14:16:58.269 INFO:tasks.workunit.client.0.smithi032.stdout:3/160: link d3/d16/l1b d3/d16/d1e/l49 0 2024-04-06T14:16:58.270 INFO:tasks.workunit.client.0.smithi032.stdout:3/161: dread - d3/d16/d1e/f32 zero size 2024-04-06T14:16:58.272 INFO:tasks.workunit.client.0.smithi032.stdout:9/219: creat d2/d15/d1e/d47/d4a/f4d x:0 0 0 2024-04-06T14:16:58.273 INFO:tasks.workunit.client.0.smithi032.stdout:9/220: chown d2/d15/d1e/d41 3341350 1 2024-04-06T14:16:58.273 INFO:tasks.workunit.client.0.smithi032.stdout:6/192: mknod d4/d5/d19/d25/c2d 0 2024-04-06T14:16:58.274 INFO:tasks.workunit.client.0.smithi032.stdout:2/178: creat d0/d30/f43 x:0 0 0 2024-04-06T14:16:58.274 INFO:tasks.workunit.client.0.smithi032.stdout:3/162: rename d3/d16/d1e/c3e to d3/d3b/c4a 0 2024-04-06T14:16:58.277 INFO:tasks.workunit.client.0.smithi032.stdout:6/193: creat d4/d5/d19/f2e x:0 0 0 2024-04-06T14:16:58.277 INFO:tasks.workunit.client.0.smithi032.stdout:9/221: truncate d2/fd 4629912 0 2024-04-06T14:16:58.279 INFO:tasks.workunit.client.0.smithi032.stdout:2/179: creat d0/d30/d39/f44 x:0 0 0 2024-04-06T14:16:58.280 INFO:tasks.workunit.client.0.smithi032.stdout:2/180: write d0/f13 [5187627,11920] 0 2024-04-06T14:16:58.280 INFO:tasks.workunit.client.0.smithi032.stdout:5/192: truncate f3 2041884 0 2024-04-06T14:16:58.281 INFO:tasks.workunit.client.0.smithi032.stdout:2/181: fsync d0/d12/f1f 0 2024-04-06T14:16:58.282 INFO:tasks.workunit.client.0.smithi032.stdout:3/163: creat d3/d16/f4b x:0 0 0 2024-04-06T14:16:58.282 INFO:tasks.workunit.client.0.smithi032.stdout:6/194: fsync d4/d5/fb 0 2024-04-06T14:16:58.282 INFO:tasks.workunit.client.0.smithi032.stdout:9/222: rmdir d2/d25/d31/d39 39 2024-04-06T14:16:58.283 INFO:tasks.workunit.client.0.smithi032.stdout:2/182: dread - d0/d12/f31 zero size 2024-04-06T14:16:58.283 INFO:tasks.workunit.client.0.smithi032.stdout:9/223: readlink d2/le 0 2024-04-06T14:16:58.285 INFO:tasks.workunit.client.0.smithi032.stdout:5/193: rename fc to d16/d18/d33/d35/f42 0 2024-04-06T14:16:58.285 INFO:tasks.workunit.client.0.smithi032.stdout:2/183: write d0/d12/f1f [2249223,8303] 0 2024-04-06T14:16:58.286 INFO:tasks.workunit.client.0.smithi032.stdout:9/224: symlink d2/d15/d1e/d47/d4a/l4e 0 2024-04-06T14:16:58.288 INFO:tasks.workunit.client.0.smithi032.stdout:2/184: chown d0/l11 5 1 2024-04-06T14:16:58.289 INFO:tasks.workunit.client.0.smithi032.stdout:3/164: dread d3/f20 [0,4194304] 0 2024-04-06T14:16:58.290 INFO:tasks.workunit.client.0.smithi032.stdout:9/225: truncate d2/d5/f1d 1681128 0 2024-04-06T14:16:58.292 INFO:tasks.workunit.client.0.smithi032.stdout:3/165: chown d3/l11 21308 1 2024-04-06T14:16:58.292 INFO:tasks.workunit.client.0.smithi032.stdout:6/195: dwrite d4/d5/f2a [0,4194304] 0 2024-04-06T14:16:58.293 INFO:tasks.workunit.client.0.smithi032.stdout:3/166: chown d3/d16/f2c 0 1 2024-04-06T14:16:58.293 INFO:tasks.workunit.client.0.smithi032.stdout:2/185: rename d0/d30/d39/f44 to d0/d12/d3a/f45 0 2024-04-06T14:16:58.294 INFO:tasks.workunit.client.0.smithi032.stdout:9/226: symlink d2/d15/l4f 0 2024-04-06T14:16:58.296 INFO:tasks.workunit.client.0.smithi032.stdout:9/227: dread - d2/d15/d1e/d47/d4a/f4d zero size 2024-04-06T14:16:58.297 INFO:tasks.workunit.client.0.smithi032.stdout:6/196: mkdir d4/d5/d8/d2f 0 2024-04-06T14:16:58.297 INFO:tasks.workunit.client.0.smithi032.stdout:6/197: chown d4 7744125 1 2024-04-06T14:16:58.298 INFO:tasks.workunit.client.0.smithi032.stdout:3/167: readlink d3/l15 0 2024-04-06T14:16:58.300 INFO:tasks.workunit.client.0.smithi032.stdout:3/168: truncate d3/d3b/f40 219948 0 2024-04-06T14:16:58.305 INFO:tasks.workunit.client.0.smithi032.stdout:9/228: mknod d2/d5/c50 0 2024-04-06T14:16:58.305 INFO:tasks.workunit.client.0.smithi032.stdout:6/198: dwrite d4/d5/fb [0,4194304] 0 2024-04-06T14:16:58.307 INFO:tasks.workunit.client.0.smithi032.stdout:5/194: truncate fa 1685741 0 2024-04-06T14:16:58.308 INFO:tasks.workunit.client.0.smithi032.stdout:6/199: chown d4/d5/l1e 24 1 2024-04-06T14:16:58.309 INFO:tasks.workunit.client.0.smithi032.stdout:5/195: read d16/d18/f1e [1583894,119534] 0 2024-04-06T14:16:58.310 INFO:tasks.workunit.client.0.smithi032.stdout:3/169: dwrite d3/d3b/f40 [0,4194304] 0 2024-04-06T14:16:58.310 INFO:tasks.workunit.client.0.smithi032.stdout:9/229: mknod d2/c51 0 2024-04-06T14:16:58.311 INFO:tasks.workunit.client.0.smithi032.stdout:3/170: stat d3/l11 0 2024-04-06T14:16:58.312 INFO:tasks.workunit.client.0.smithi032.stdout:3/171: read - d3/d16/f4b zero size 2024-04-06T14:16:58.315 INFO:tasks.workunit.client.0.smithi032.stdout:6/200: creat d4/d5/d19/d21/f30 x:0 0 0 2024-04-06T14:16:58.316 INFO:tasks.workunit.client.0.smithi032.stdout:3/172: mknod d3/d3b/c4c 0 2024-04-06T14:16:58.319 INFO:tasks.workunit.client.0.smithi032.stdout:3/173: dread - d3/d16/f33 zero size 2024-04-06T14:16:58.319 INFO:tasks.workunit.client.0.smithi032.stdout:6/201: chown d4/ce 3514 1 2024-04-06T14:16:58.319 INFO:tasks.workunit.client.0.smithi032.stdout:5/196: getdents d16/d18/d33/d35 0 2024-04-06T14:16:58.321 INFO:tasks.workunit.client.0.smithi032.stdout:5/197: read d16/f2c [594845,14683] 0 2024-04-06T14:16:58.321 INFO:tasks.workunit.client.0.smithi032.stdout:2/186: dwrite d0/d12/f1a [0,4194304] 0 2024-04-06T14:16:58.322 INFO:tasks.workunit.client.0.smithi032.stdout:6/202: fdatasync d4/d5/d28/f29 0 2024-04-06T14:16:58.326 INFO:tasks.workunit.client.0.smithi032.stdout:3/174: rename d3/d16/d1e/l27 to d3/d3b/l4d 0 2024-04-06T14:16:58.330 INFO:tasks.workunit.client.0.smithi032.stdout:6/203: link d4/d5/d14/f1a d4/d5/d18/f31 0 2024-04-06T14:16:58.331 INFO:tasks.workunit.client.0.smithi032.stdout:5/198: dwrite f1 [4194304,4194304] 0 2024-04-06T14:16:58.339 INFO:tasks.workunit.client.0.smithi032.stdout:2/187: dread d0/d12/f2c [0,4194304] 0 2024-04-06T14:16:58.343 INFO:tasks.workunit.client.0.smithi032.stdout:2/188: read - d0/d12/d3a/f42 zero size 2024-04-06T14:16:58.343 INFO:tasks.workunit.client.0.smithi032.stdout:6/204: truncate d4/d5/d19/d21/f30 1012162 0 2024-04-06T14:16:58.345 INFO:tasks.workunit.client.0.smithi032.stdout:2/189: mknod d0/d30/c46 0 2024-04-06T14:16:58.346 INFO:tasks.workunit.client.0.smithi032.stdout:3/175: dwrite d3/d16/f18 [0,4194304] 0 2024-04-06T14:16:58.347 INFO:tasks.workunit.client.0.smithi032.stdout:9/230: dwrite d2/fd [0,4194304] 0 2024-04-06T14:16:58.350 INFO:tasks.workunit.client.0.smithi032.stdout:5/199: truncate d16/f2c 383702 0 2024-04-06T14:16:58.350 INFO:tasks.workunit.client.0.smithi032.stdout:3/176: symlink d3/d16/d1e/l4e 0 2024-04-06T14:16:58.353 INFO:tasks.workunit.client.0.smithi032.stdout:3/177: readlink d3/d16/d1e/l4e 0 2024-04-06T14:16:58.357 INFO:tasks.workunit.client.0.smithi032.stdout:9/231: getdents d2 0 2024-04-06T14:16:58.357 INFO:tasks.workunit.client.0.smithi032.stdout:5/200: creat d16/d18/d33/d35/d37/f43 x:0 0 0 2024-04-06T14:16:58.358 INFO:tasks.workunit.client.0.smithi032.stdout:6/205: dwrite d4/d5/f15 [0,4194304] 0 2024-04-06T14:16:58.360 INFO:tasks.workunit.client.0.smithi032.stdout:3/178: mknod d3/d16/c4f 0 2024-04-06T14:16:58.362 INFO:tasks.workunit.client.0.smithi032.stdout:3/179: write d3/f3f [4183615,46276] 0 2024-04-06T14:16:58.363 INFO:tasks.workunit.client.0.smithi032.stdout:6/206: unlink d4/l16 0 2024-04-06T14:16:58.363 INFO:tasks.workunit.client.0.smithi032.stdout:3/180: write d3/d16/f25 [868323,121448] 0 2024-04-06T14:16:58.364 INFO:tasks.workunit.client.0.smithi032.stdout:5/201: creat d16/d18/d33/d35/f44 x:0 0 0 2024-04-06T14:16:58.367 INFO:tasks.workunit.client.0.smithi032.stdout:9/232: creat d2/d25/d31/d39/f52 x:0 0 0 2024-04-06T14:16:58.367 INFO:tasks.workunit.client.0.smithi032.stdout:9/233: chown d2/d25 1 1 2024-04-06T14:16:58.369 INFO:tasks.workunit.client.0.smithi032.stdout:6/207: symlink d4/d5/d19/d25/l32 0 2024-04-06T14:16:58.372 INFO:tasks.workunit.client.0.smithi032.stdout:1/172: sync 2024-04-06T14:16:58.372 INFO:tasks.workunit.client.0.smithi032.stdout:4/331: sync 2024-04-06T14:16:58.373 INFO:tasks.workunit.client.0.smithi032.stdout:6/208: truncate d4/d5/d18/d23/f27 1277996 0 2024-04-06T14:16:58.376 INFO:tasks.workunit.client.0.smithi032.stdout:5/202: rename d16/d18/d28 to d16/d18/d33/d45 0 2024-04-06T14:16:58.376 INFO:tasks.workunit.client.0.smithi032.stdout:4/332: truncate d0/df/d23/f40 940190 0 2024-04-06T14:16:58.377 INFO:tasks.workunit.client.0.smithi032.stdout:6/209: creat d4/d5/d8/d1f/f33 x:0 0 0 2024-04-06T14:16:58.378 INFO:tasks.workunit.client.0.smithi032.stdout:2/190: dread d0/f23 [0,4194304] 0 2024-04-06T14:16:58.378 INFO:tasks.workunit.client.0.smithi032.stdout:1/173: rename d0/d9/fd to d0/d10/f3d 0 2024-04-06T14:16:58.380 INFO:tasks.workunit.client.0.smithi032.stdout:3/181: dwrite d3/f4 [0,4194304] 0 2024-04-06T14:16:58.380 INFO:tasks.workunit.client.0.smithi032.stdout:5/203: mkdir d16/d18/d33/d35/d37/d46 0 2024-04-06T14:16:58.382 INFO:tasks.workunit.client.0.smithi032.stdout:5/204: chown d16/f1d 0 1 2024-04-06T14:16:58.382 INFO:tasks.workunit.client.0.smithi032.stdout:5/205: dread - d16/d18/f1c zero size 2024-04-06T14:16:58.382 INFO:tasks.workunit.client.0.smithi032.stdout:5/206: stat l8 0 2024-04-06T14:16:58.385 INFO:tasks.workunit.client.0.smithi032.stdout:6/210: mkdir d4/d5/d34 0 2024-04-06T14:16:58.387 INFO:tasks.workunit.client.0.smithi032.stdout:2/191: dread d0/d12/f17 [0,4194304] 0 2024-04-06T14:16:58.390 INFO:tasks.workunit.client.0.smithi032.stdout:3/182: symlink d3/l50 0 2024-04-06T14:16:58.391 INFO:tasks.workunit.client.0.smithi032.stdout:9/234: write d2/d15/f1a [2642293,128412] 0 2024-04-06T14:16:58.395 INFO:tasks.workunit.client.0.smithi032.stdout:5/207: rename fa to d16/d18/d33/d35/d37/d39/f47 0 2024-04-06T14:16:58.399 INFO:tasks.workunit.client.0.smithi032.stdout:2/192: dwrite d0/d12/f17 [0,4194304] 0 2024-04-06T14:16:58.400 INFO:tasks.workunit.client.0.smithi032.stdout:3/183: link d3/f19 d3/d3b/f51 0 2024-04-06T14:16:58.400 INFO:tasks.workunit.client.0.smithi032.stdout:1/174: write d0/f2 [3084300,87109] 0 2024-04-06T14:16:58.402 INFO:tasks.workunit.client.0.smithi032.stdout:2/193: fsync d0/f24 0 2024-04-06T14:16:58.402 INFO:tasks.workunit.client.0.smithi032.stdout:5/208: rename l15 to d16/d18/d33/d35/d37/l48 0 2024-04-06T14:16:58.404 INFO:tasks.workunit.client.0.smithi032.stdout:6/211: write d4/d5/d18/f31 [411250,11919] 0 2024-04-06T14:16:58.405 INFO:tasks.workunit.client.0.smithi032.stdout:2/194: fsync d0/d12/f31 0 2024-04-06T14:16:58.405 INFO:tasks.workunit.client.0.smithi032.stdout:3/184: creat d3/d16/d1e/f52 x:0 0 0 2024-04-06T14:16:58.406 INFO:tasks.workunit.client.0.smithi032.stdout:4/333: dwrite d0/df/d1f/f34 [0,4194304] 0 2024-04-06T14:16:58.407 INFO:tasks.workunit.client.0.smithi032.stdout:5/209: mknod d16/d18/c49 0 2024-04-06T14:16:58.408 INFO:tasks.workunit.client.0.smithi032.stdout:2/195: mknod d0/d30/c47 0 2024-04-06T14:16:58.409 INFO:tasks.workunit.client.0.smithi032.stdout:4/334: chown d0/df/d23/d31/d50 225603 1 2024-04-06T14:16:58.410 INFO:tasks.workunit.client.0.smithi032.stdout:3/185: mknod d3/d3b/c53 0 2024-04-06T14:16:58.410 INFO:tasks.workunit.client.0.smithi032.stdout:2/196: write d0/d12/f41 [281282,93836] 0 2024-04-06T14:16:58.411 INFO:tasks.workunit.client.0.smithi032.stdout:6/212: mkdir d4/d5/d35 0 2024-04-06T14:16:58.413 INFO:tasks.workunit.client.0.smithi032.stdout:6/213: write d4/d5/d28/f29 [910380,32245] 0 2024-04-06T14:16:58.415 INFO:tasks.workunit.client.0.smithi032.stdout:6/214: write d4/d5/d14/f1a [751677,42020] 0 2024-04-06T14:16:58.417 INFO:tasks.workunit.client.0.smithi032.stdout:2/197: dread d0/f24 [0,4194304] 0 2024-04-06T14:16:58.417 INFO:tasks.workunit.client.0.smithi032.stdout:6/215: write d4/d5/f15 [4316760,49459] 0 2024-04-06T14:16:58.417 INFO:tasks.workunit.client.0.smithi032.stdout:4/335: truncate d0/df/d2b/f2c 1663965 0 2024-04-06T14:16:58.420 INFO:tasks.workunit.client.0.smithi032.stdout:6/216: mknod d4/d5/d19/c36 0 2024-04-06T14:16:58.421 INFO:tasks.workunit.client.0.smithi032.stdout:4/336: mknod d0/df/d1f/d2e/c7a 0 2024-04-06T14:16:58.424 INFO:tasks.workunit.client.0.smithi032.stdout:4/337: fdatasync d0/df/d23/f60 0 2024-04-06T14:16:58.426 INFO:tasks.workunit.client.0.smithi032.stdout:3/186: dwrite d3/d3b/f51 [4194304,4194304] 0 2024-04-06T14:16:58.429 INFO:tasks.workunit.client.0.smithi032.stdout:9/235: dread d2/d15/d1e/f1f [0,4194304] 0 2024-04-06T14:16:58.430 INFO:tasks.workunit.client.0.smithi032.stdout:2/198: dwrite d0/d30/f43 [0,4194304] 0 2024-04-06T14:16:58.430 INFO:tasks.workunit.client.0.smithi032.stdout:4/338: write d0/df/d2b/d46/f6e [198514,5948] 0 2024-04-06T14:16:58.435 INFO:tasks.workunit.client.0.smithi032.stdout:9/236: write d2/d5/f6 [5060197,84041] 0 2024-04-06T14:16:58.437 INFO:tasks.workunit.client.0.smithi032.stdout:3/187: creat d3/d3b/f54 x:0 0 0 2024-04-06T14:16:58.438 INFO:tasks.workunit.client.0.smithi032.stdout:5/210: rename d16/d18/d33/d35/d37 to d16/d18/d33/d45/d4a 0 2024-04-06T14:16:58.438 INFO:tasks.workunit.client.0.smithi032.stdout:4/339: symlink d0/df/d1f/d42/d45/l7b 0 2024-04-06T14:16:58.439 INFO:tasks.workunit.client.0.smithi032.stdout:6/217: dwrite d4/d5/d19/f2e [0,4194304] 0 2024-04-06T14:16:58.440 INFO:tasks.workunit.client.0.smithi032.stdout:9/237: creat d2/f53 x:0 0 0 2024-04-06T14:16:58.441 INFO:tasks.workunit.client.0.smithi032.stdout:2/199: unlink d0/le 0 2024-04-06T14:16:58.443 INFO:tasks.workunit.client.0.smithi032.stdout:2/200: chown d0/l11 32912658 1 2024-04-06T14:16:58.443 INFO:tasks.workunit.client.0.smithi032.stdout:3/188: rename d3/d3b/f48 to d3/d16/f55 0 2024-04-06T14:16:58.446 INFO:tasks.workunit.client.0.smithi032.stdout:5/211: symlink d16/d18/d33/d35/l4b 0 2024-04-06T14:16:58.452 INFO:tasks.workunit.client.0.smithi032.stdout:5/212: write d16/d18/d33/d45/d4a/d39/f3e [519355,30986] 0 2024-04-06T14:16:58.452 INFO:tasks.workunit.client.0.smithi032.stdout:3/189: symlink d3/l56 0 2024-04-06T14:16:58.452 INFO:tasks.workunit.client.0.smithi032.stdout:1/175: dwrite d0/d10/d16/f3c [0,4194304] 0 2024-04-06T14:16:58.453 INFO:tasks.workunit.client.0.smithi032.stdout:4/340: dwrite d0/df/d23/f3d [4194304,4194304] 0 2024-04-06T14:16:58.453 INFO:tasks.workunit.client.0.smithi032.stdout:6/218: link d4/d5/d19/d21/f30 d4/d5/d8/d1f/f37 0 2024-04-06T14:16:58.457 INFO:tasks.workunit.client.0.smithi032.stdout:4/341: truncate d0/d47/d4b/f63 1024092 0 2024-04-06T14:16:58.459 INFO:tasks.workunit.client.0.smithi032.stdout:6/219: symlink d4/d5/d19/d21/l38 0 2024-04-06T14:16:58.459 INFO:tasks.workunit.client.0.smithi032.stdout:4/342: stat d0/df/d1f/d42 0 2024-04-06T14:16:58.460 INFO:tasks.workunit.client.0.smithi032.stdout:1/176: read - d0/f37 zero size 2024-04-06T14:16:58.463 INFO:tasks.workunit.client.0.smithi032.stdout:5/213: dread d16/d18/d33/d45/d4a/d39/f47 [0,4194304] 0 2024-04-06T14:16:58.463 INFO:tasks.workunit.client.0.smithi032.stdout:2/201: write d0/f23 [4785601,74126] 0 2024-04-06T14:16:58.464 INFO:tasks.workunit.client.0.smithi032.stdout:1/177: mkdir d0/da/d3e 0 2024-04-06T14:16:58.465 INFO:tasks.workunit.client.0.smithi032.stdout:1/178: dread - d0/f1e zero size 2024-04-06T14:16:58.466 INFO:tasks.workunit.client.0.smithi032.stdout:6/220: creat d4/d5/d35/f39 x:0 0 0 2024-04-06T14:16:58.467 INFO:tasks.workunit.client.0.smithi032.stdout:5/214: symlink d16/d18/d33/d45/l4c 0 2024-04-06T14:16:58.468 INFO:tasks.workunit.client.0.smithi032.stdout:1/179: chown d0/d10/d16/l2b 1 1 2024-04-06T14:16:58.472 INFO:tasks.workunit.client.0.smithi032.stdout:5/215: chown d16/d18/d33/f34 0 1 2024-04-06T14:16:58.472 INFO:tasks.workunit.client.0.smithi032.stdout:7/221: sync 2024-04-06T14:16:58.472 INFO:tasks.workunit.client.0.smithi032.stdout:0/155: sync 2024-04-06T14:16:58.475 INFO:tasks.workunit.client.0.smithi032.stdout:5/216: read - f9 zero size 2024-04-06T14:16:58.476 INFO:tasks.workunit.client.0.smithi032.stdout:4/343: dwrite d0/df/d23/d31/d50/d57/f6a [0,4194304] 0 2024-04-06T14:16:58.478 INFO:tasks.workunit.client.0.smithi032.stdout:1/180: truncate d0/d9/d1f/f3a 509639 0 2024-04-06T14:16:58.481 INFO:tasks.workunit.client.0.smithi032.stdout:2/202: dread d0/d30/f43 [0,4194304] 0 2024-04-06T14:16:58.481 INFO:tasks.workunit.client.0.smithi032.stdout:9/238: dwrite d2/d15/d1e/d47/d4a/f4d [0,4194304] 0 2024-04-06T14:16:58.482 INFO:tasks.workunit.client.0.smithi032.stdout:0/156: creat d3/da/d25/d2b/f3d x:0 0 0 2024-04-06T14:16:58.487 INFO:tasks.workunit.client.0.smithi032.stdout:2/203: stat d0/d12/f2a 0 2024-04-06T14:16:58.488 INFO:tasks.workunit.client.0.smithi032.stdout:6/221: dwrite d4/f10 [0,4194304] 0 2024-04-06T14:16:58.488 INFO:tasks.workunit.client.0.smithi032.stdout:5/217: fsync d16/d18/f21 0 2024-04-06T14:16:58.490 INFO:tasks.workunit.client.0.smithi032.stdout:0/157: read - d3/da/d25/d2b/f3a zero size 2024-04-06T14:16:58.491 INFO:tasks.workunit.client.0.smithi032.stdout:6/222: fdatasync d4/f13 0 2024-04-06T14:16:58.493 INFO:tasks.workunit.client.0.smithi032.stdout:7/222: dread fb [0,4194304] 0 2024-04-06T14:16:58.493 INFO:tasks.workunit.client.0.smithi032.stdout:9/239: truncate d2/d25/f32 107206 0 2024-04-06T14:16:58.493 INFO:tasks.workunit.client.0.smithi032.stdout:0/158: stat d3/c7 0 2024-04-06T14:16:58.494 INFO:tasks.workunit.client.0.smithi032.stdout:2/204: creat d0/d12/d3a/f48 x:0 0 0 2024-04-06T14:16:58.495 INFO:tasks.workunit.client.0.smithi032.stdout:5/218: mkdir d16/d18/d33/d35/d4d 0 2024-04-06T14:16:58.496 INFO:tasks.workunit.client.0.smithi032.stdout:7/223: chown d16/d20/d48/f4a 3377 1 2024-04-06T14:16:58.497 INFO:tasks.workunit.client.0.smithi032.stdout:0/159: read d3/da/f1e [111020,46934] 0 2024-04-06T14:16:58.498 INFO:tasks.workunit.client.0.smithi032.stdout:6/223: getdents d4/d5/d8/d2b 0 2024-04-06T14:16:58.498 INFO:tasks.workunit.client.0.smithi032.stdout:4/344: dread d0/f1e [0,4194304] 0 2024-04-06T14:16:58.499 INFO:tasks.workunit.client.0.smithi032.stdout:5/219: dread - d16/d18/d33/f41 zero size 2024-04-06T14:16:58.499 INFO:tasks.workunit.client.0.smithi032.stdout:2/205: fsync d0/d12/d3a/f45 0 2024-04-06T14:16:58.501 INFO:tasks.workunit.client.0.smithi032.stdout:9/240: write d2/d5/f37 [754169,89081] 0 2024-04-06T14:16:58.501 INFO:tasks.workunit.client.0.smithi032.stdout:1/181: dread d0/d10/d16/f3c [0,4194304] 0 2024-04-06T14:16:58.502 INFO:tasks.workunit.client.0.smithi032.stdout:2/206: stat d0/d12/f1c 0 2024-04-06T14:16:58.503 INFO:tasks.workunit.client.0.smithi032.stdout:6/224: symlink d4/d5/d8/d2f/l3a 0 2024-04-06T14:16:58.503 INFO:tasks.workunit.client.0.smithi032.stdout:9/241: dread - d2/d15/d1e/d47/f4b zero size 2024-04-06T14:16:58.504 INFO:tasks.workunit.client.0.smithi032.stdout:5/220: creat d16/d18/d33/d45/d4a/f4e x:0 0 0 2024-04-06T14:16:58.505 INFO:tasks.workunit.client.0.smithi032.stdout:4/345: mknod d0/df/d2b/d4f/c7c 0 2024-04-06T14:16:58.505 INFO:tasks.workunit.client.0.smithi032.stdout:1/182: fdatasync d0/d10/d16/f3c 0 2024-04-06T14:16:58.507 INFO:tasks.workunit.client.0.smithi032.stdout:1/183: chown d0/d10/f14 67506351 1 2024-04-06T14:16:58.507 INFO:tasks.workunit.client.0.smithi032.stdout:2/207: rename d0/d12/c2b to d0/d12/d3a/c49 0 2024-04-06T14:16:58.507 INFO:tasks.workunit.client.0.smithi032.stdout:5/221: rename d16 to d16/d4f 22 2024-04-06T14:16:58.507 INFO:tasks.workunit.client.0.smithi032.stdout:9/242: mknod d2/d15/d1e/c54 0 2024-04-06T14:16:58.510 INFO:tasks.workunit.client.0.smithi032.stdout:1/184: mkdir d0/d10/d16/d3f 0 2024-04-06T14:16:58.510 INFO:tasks.workunit.client.0.smithi032.stdout:2/208: write d0/d12/d3a/f42 [769687,110083] 0 2024-04-06T14:16:58.511 INFO:tasks.workunit.client.0.smithi032.stdout:9/243: mkdir d2/d15/d55 0 2024-04-06T14:16:58.512 INFO:tasks.workunit.client.0.smithi032.stdout:5/222: readlink d16/d18/d33/d45/d4a/l48 0 2024-04-06T14:16:58.515 INFO:tasks.workunit.client.0.smithi032.stdout:0/160: dread d3/fd [0,4194304] 0 2024-04-06T14:16:58.515 INFO:tasks.workunit.client.0.smithi032.stdout:4/346: dwrite d0/df/d1f/d42/d45/f53 [0,4194304] 0 2024-04-06T14:16:58.518 INFO:tasks.workunit.client.0.smithi032.stdout:1/185: truncate d0/f2c 3364110 0 2024-04-06T14:16:58.520 INFO:tasks.workunit.client.0.smithi032.stdout:2/209: rmdir d0 39 2024-04-06T14:16:58.522 INFO:tasks.workunit.client.0.smithi032.stdout:5/223: dwrite d16/d18/d33/d35/f44 [0,4194304] 0 2024-04-06T14:16:58.522 INFO:tasks.workunit.client.0.smithi032.stdout:4/347: unlink d0/df/d2b/f70 0 2024-04-06T14:16:58.524 INFO:tasks.workunit.client.0.smithi032.stdout:0/161: rename d3/da/d25/d2b/c31 to d3/da/d25/c3e 0 2024-04-06T14:16:58.524 INFO:tasks.workunit.client.0.smithi032.stdout:0/162: readlink d3/da/l2c 0 2024-04-06T14:16:58.525 INFO:tasks.workunit.client.0.smithi032.stdout:0/163: truncate d3/f20 97347 0 2024-04-06T14:16:58.526 INFO:tasks.workunit.client.0.smithi032.stdout:0/164: write d3/da/d25/d2b/f3d [41607,41862] 0 2024-04-06T14:16:58.529 INFO:tasks.workunit.client.0.smithi032.stdout:5/224: creat d16/f50 x:0 0 0 2024-04-06T14:16:58.531 INFO:tasks.workunit.client.0.smithi032.stdout:0/165: rename d3/da/l30 to d3/da/d25/d2b/l3f 0 2024-04-06T14:16:58.534 INFO:tasks.workunit.client.0.smithi032.stdout:5/225: symlink d16/d18/d33/d45/l51 0 2024-04-06T14:16:58.535 INFO:tasks.workunit.client.0.smithi032.stdout:1/186: link d0/d10/d16/l2d d0/da/d3e/l40 0 2024-04-06T14:16:58.536 INFO:tasks.workunit.client.0.smithi032.stdout:0/166: dread d3/f14 [0,4194304] 0 2024-04-06T14:16:58.537 INFO:tasks.workunit.client.0.smithi032.stdout:2/210: link d0/c2d d0/d12/d3a/c4a 0 2024-04-06T14:16:58.539 INFO:tasks.workunit.client.0.smithi032.stdout:1/187: dread d0/da/f28 [0,4194304] 0 2024-04-06T14:16:58.539 INFO:tasks.workunit.client.0.smithi032.stdout:2/211: fdatasync d0/d12/f1c 0 2024-04-06T14:16:58.543 INFO:tasks.workunit.client.0.smithi032.stdout:1/188: write d0/f2 [2217421,129144] 0 2024-04-06T14:16:58.546 INFO:tasks.workunit.client.0.smithi032.stdout:5/226: dwrite f10 [0,4194304] 0 2024-04-06T14:16:58.546 INFO:tasks.workunit.client.0.smithi032.stdout:4/348: truncate d0/df/d1f/f25 2013282 0 2024-04-06T14:16:58.548 INFO:tasks.workunit.client.0.smithi032.stdout:5/227: dread - d16/f50 zero size 2024-04-06T14:16:58.548 INFO:tasks.workunit.client.0.smithi032.stdout:1/189: unlink d0/d10/f14 0 2024-04-06T14:16:58.550 INFO:tasks.workunit.client.0.smithi032.stdout:1/190: mkdir d0/d41 0 2024-04-06T14:16:58.550 INFO:tasks.workunit.client.0.smithi032.stdout:4/349: write d0/df/d1f/d42/d45/f75 [387754,81884] 0 2024-04-06T14:16:58.551 INFO:tasks.workunit.client.0.smithi032.stdout:0/167: dwrite d3/da/ff [0,4194304] 0 2024-04-06T14:16:58.552 INFO:tasks.workunit.client.0.smithi032.stdout:4/350: chown d0/d47/d4b 339061 1 2024-04-06T14:16:58.553 INFO:tasks.workunit.client.0.smithi032.stdout:1/191: fdatasync d0/d10/f3d 0 2024-04-06T14:16:58.555 INFO:tasks.workunit.client.0.smithi032.stdout:5/228: rename d16/d18/d33/d45/d4a to d16/d18/d33/d45/d52 0 2024-04-06T14:16:58.556 INFO:tasks.workunit.client.0.smithi032.stdout:5/229: write f10 [1769094,114462] 0 2024-04-06T14:16:58.557 INFO:tasks.workunit.client.0.smithi032.stdout:0/168: creat d3/da/d25/d2b/f40 x:0 0 0 2024-04-06T14:16:58.562 INFO:tasks.workunit.client.0.smithi032.stdout:0/169: mkdir d3/da/d41 0 2024-04-06T14:16:58.563 INFO:tasks.workunit.client.0.smithi032.stdout:5/230: truncate d16/d18/d33/d35/f42 385787 0 2024-04-06T14:16:58.563 INFO:tasks.workunit.client.0.smithi032.stdout:5/231: stat d16/d18/c27 0 2024-04-06T14:16:58.564 INFO:tasks.workunit.client.0.smithi032.stdout:1/192: rmdir d0/d35 0 2024-04-06T14:16:58.566 INFO:tasks.workunit.client.0.smithi032.stdout:0/170: link d3/da/fe d3/da/d41/f42 0 2024-04-06T14:16:58.568 INFO:tasks.workunit.client.0.smithi032.stdout:5/232: dread d16/d18/d33/d35/f44 [0,4194304] 0 2024-04-06T14:16:58.569 INFO:tasks.workunit.client.0.smithi032.stdout:1/193: rename d0/d9/d1f/f22 to d0/d10/d16/f42 0 2024-04-06T14:16:58.569 INFO:tasks.workunit.client.0.smithi032.stdout:5/233: chown d16/f38 184 1 2024-04-06T14:16:58.569 INFO:tasks.workunit.client.0.smithi032.stdout:1/194: dread - d0/f36 zero size 2024-04-06T14:16:58.572 INFO:tasks.workunit.client.0.smithi032.stdout:1/195: read d0/d10/d16/f3c [3014720,13568] 0 2024-04-06T14:16:58.573 INFO:tasks.workunit.client.0.smithi032.stdout:0/171: creat d3/da/d41/f43 x:0 0 0 2024-04-06T14:16:58.573 INFO:tasks.workunit.client.0.smithi032.stdout:1/196: readlink d0/da/l18 0 2024-04-06T14:16:58.576 INFO:tasks.workunit.client.0.smithi032.stdout:0/172: mkdir d3/da/d25/d2b/d44 0 2024-04-06T14:16:58.576 INFO:tasks.workunit.client.0.smithi032.stdout:5/234: dread d16/d18/f21 [0,4194304] 0 2024-04-06T14:16:58.580 INFO:tasks.workunit.client.0.smithi032.stdout:4/351: truncate d0/df/d1f/d42/d45/f75 437150 0 2024-04-06T14:16:58.581 INFO:tasks.workunit.client.0.smithi032.stdout:4/352: stat d0/df/c71 0 2024-04-06T14:16:58.583 INFO:tasks.workunit.client.0.smithi032.stdout:1/197: link d0/l15 d0/d10/d16/l43 0 2024-04-06T14:16:58.587 INFO:tasks.workunit.client.0.smithi032.stdout:0/173: rmdir d3/da/d25/d2b/d44 0 2024-04-06T14:16:58.587 INFO:tasks.workunit.client.0.smithi032.stdout:0/174: chown d3/c7 15118192 1 2024-04-06T14:16:58.587 INFO:tasks.workunit.client.0.smithi032.stdout:1/198: creat d0/da/f44 x:0 0 0 2024-04-06T14:16:58.590 INFO:tasks.workunit.client.0.smithi032.stdout:0/175: creat d3/f45 x:0 0 0 2024-04-06T14:16:58.591 INFO:tasks.workunit.client.0.smithi032.stdout:0/176: chown d3/da/l34 1093621 1 2024-04-06T14:16:58.591 INFO:tasks.workunit.client.0.smithi032.stdout:5/235: getdents d16/d18/d33/d45/d52 0 2024-04-06T14:16:58.594 INFO:tasks.workunit.client.0.smithi032.stdout:4/353: getdents d0/df/d2b/d4f 0 2024-04-06T14:16:58.594 INFO:tasks.workunit.client.0.smithi032.stdout:1/199: dwrite d0/d9/f23 [0,4194304] 0 2024-04-06T14:16:58.596 INFO:tasks.workunit.client.0.smithi032.stdout:0/177: mkdir d3/d46 0 2024-04-06T14:16:58.597 INFO:tasks.workunit.client.0.smithi032.stdout:4/354: chown d0/df/d1f/d42/d45/f53 181 1 2024-04-06T14:16:58.599 INFO:tasks.workunit.client.0.smithi032.stdout:1/200: write d0/da/f44 [1025832,118236] 0 2024-04-06T14:16:58.599 INFO:tasks.workunit.client.0.smithi032.stdout:4/355: stat d0/df/l17 0 2024-04-06T14:16:58.599 INFO:tasks.workunit.client.0.smithi032.stdout:0/178: symlink d3/da/d41/l47 0 2024-04-06T14:16:58.603 INFO:tasks.workunit.client.0.smithi032.stdout:5/236: dwrite d16/d18/d33/f34 [0,4194304] 0 2024-04-06T14:16:58.603 INFO:tasks.workunit.client.0.smithi032.stdout:1/201: truncate d0/f17 1607991 0 2024-04-06T14:16:58.604 INFO:tasks.workunit.client.0.smithi032.stdout:0/179: getdents d3/d46 0 2024-04-06T14:16:58.605 INFO:tasks.workunit.client.0.smithi032.stdout:4/356: symlink d0/df/d23/d31/d50/d57/l7d 0 2024-04-06T14:16:58.606 INFO:tasks.workunit.client.0.smithi032.stdout:1/202: creat d0/da/f45 x:0 0 0 2024-04-06T14:16:58.606 INFO:tasks.workunit.client.0.smithi032.stdout:4/357: chown d0/df/d2b/c33 0 1 2024-04-06T14:16:58.606 INFO:tasks.workunit.client.0.smithi032.stdout:5/237: creat d16/d18/d33/d45/d52/f53 x:0 0 0 2024-04-06T14:16:58.607 INFO:tasks.workunit.client.0.smithi032.stdout:1/203: readlink d0/da/l18 0 2024-04-06T14:16:58.609 INFO:tasks.workunit.client.0.smithi032.stdout:5/238: dread - d16/d18/d33/d45/d52/f4e zero size 2024-04-06T14:16:58.609 INFO:tasks.workunit.client.0.smithi032.stdout:4/358: unlink d0/l22 0 2024-04-06T14:16:58.613 INFO:tasks.workunit.client.0.smithi032.stdout:0/180: dwrite d3/f19 [0,4194304] 0 2024-04-06T14:16:58.615 INFO:tasks.workunit.client.0.smithi032.stdout:5/239: dread d16/d18/d33/d45/d52/d39/f47 [0,4194304] 0 2024-04-06T14:16:58.616 INFO:tasks.workunit.client.0.smithi032.stdout:1/204: getdents d0/d10/d16 0 2024-04-06T14:16:58.616 INFO:tasks.workunit.client.0.smithi032.stdout:0/181: fsync d3/da/d41/f42 0 2024-04-06T14:16:58.619 INFO:tasks.workunit.client.0.smithi032.stdout:1/205: creat d0/d9/d1f/f46 x:0 0 0 2024-04-06T14:16:58.621 INFO:tasks.workunit.client.0.smithi032.stdout:5/240: dread d16/d18/f20 [0,4194304] 0 2024-04-06T14:16:58.623 INFO:tasks.workunit.client.0.smithi032.stdout:8/224: sync 2024-04-06T14:16:58.624 INFO:tasks.workunit.client.0.smithi032.stdout:8/225: chown d3/d8/f35 3496577 1 2024-04-06T14:16:58.624 INFO:tasks.workunit.client.0.smithi032.stdout:0/182: link d3/l27 d3/da/d25/l48 0 2024-04-06T14:16:58.626 INFO:tasks.workunit.client.0.smithi032.stdout:5/241: mkdir d16/d18/d33/d45/d52/d46/d54 0 2024-04-06T14:16:58.627 INFO:tasks.workunit.client.0.smithi032.stdout:5/242: chown d16/f17 517 1 2024-04-06T14:16:58.627 INFO:tasks.workunit.client.0.smithi032.stdout:8/226: truncate d3/d8/d7/d14/f44 754189 0 2024-04-06T14:16:58.627 INFO:tasks.workunit.client.0.smithi032.stdout:1/206: rmdir d0/d9/d1f 39 2024-04-06T14:16:58.628 INFO:tasks.workunit.client.0.smithi032.stdout:4/359: dread d0/df/d1f/d2e/f49 [0,4194304] 0 2024-04-06T14:16:58.629 INFO:tasks.workunit.client.0.smithi032.stdout:1/207: write d0/f2 [2479850,68920] 0 2024-04-06T14:16:58.629 INFO:tasks.workunit.client.0.smithi032.stdout:4/360: readlink d0/df/d1f/d42/d45/l7b 0 2024-04-06T14:16:58.630 INFO:tasks.workunit.client.0.smithi032.stdout:0/183: rmdir d3/d46 0 2024-04-06T14:16:58.634 INFO:tasks.workunit.client.0.smithi032.stdout:0/184: dread d3/da/fe [0,4194304] 0 2024-04-06T14:16:58.634 INFO:tasks.workunit.client.0.smithi032.stdout:5/243: rename d16/c3a to d16/d18/c55 0 2024-04-06T14:16:58.635 INFO:tasks.workunit.client.0.smithi032.stdout:1/208: creat d0/d9/d1f/f47 x:0 0 0 2024-04-06T14:16:58.637 INFO:tasks.workunit.client.0.smithi032.stdout:4/361: truncate d0/df/d2b/f2c 1946581 0 2024-04-06T14:16:58.638 INFO:tasks.workunit.client.0.smithi032.stdout:5/244: fdatasync d16/d18/d33/d45/d52/d39/f3e 0 2024-04-06T14:16:58.640 INFO:tasks.workunit.client.0.smithi032.stdout:5/245: write f10 [1786670,124362] 0 2024-04-06T14:16:58.640 INFO:tasks.workunit.client.0.smithi032.stdout:8/227: dwrite d3/d8/d7/f47 [0,4194304] 0 2024-04-06T14:16:58.644 INFO:tasks.workunit.client.0.smithi032.stdout:1/209: rename d0/da/ce to d0/d10/d16/d3f/c48 0 2024-04-06T14:16:58.647 INFO:tasks.workunit.client.0.smithi032.stdout:8/228: symlink d3/d8/d12/d2f/l57 0 2024-04-06T14:16:58.649 INFO:tasks.workunit.client.0.smithi032.stdout:0/185: dwrite d3/da/fe [0,4194304] 0 2024-04-06T14:16:58.651 INFO:tasks.workunit.client.0.smithi032.stdout:8/229: symlink d3/l58 0 2024-04-06T14:16:58.653 INFO:tasks.workunit.client.0.smithi032.stdout:1/210: rename d0/d10/d16/c1d to d0/d10/d16/c49 0 2024-04-06T14:16:58.654 INFO:tasks.workunit.client.0.smithi032.stdout:4/362: getdents d0/df/d1f 0 2024-04-06T14:16:58.654 INFO:tasks.workunit.client.0.smithi032.stdout:0/186: rmdir d3/da/d25/d2b 39 2024-04-06T14:16:58.655 INFO:tasks.workunit.client.0.smithi032.stdout:1/211: chown d0/d9/f1b 165389 1 2024-04-06T14:16:58.655 INFO:tasks.workunit.client.0.smithi032.stdout:4/363: stat d0/df/d2b/d46 0 2024-04-06T14:16:58.655 INFO:tasks.workunit.client.0.smithi032.stdout:8/230: mknod d3/d8/d12/d2f/d4a/c59 0 2024-04-06T14:16:58.656 INFO:tasks.workunit.client.0.smithi032.stdout:1/212: chown d0/da/l18 0 1 2024-04-06T14:16:58.657 INFO:tasks.workunit.client.0.smithi032.stdout:0/187: rmdir d3 39 2024-04-06T14:16:58.660 INFO:tasks.workunit.client.0.smithi032.stdout:8/231: unlink d3/f10 0 2024-04-06T14:16:58.660 INFO:tasks.workunit.client.0.smithi032.stdout:4/364: dread d0/f1e [0,4194304] 0 2024-04-06T14:16:58.661 INFO:tasks.workunit.client.0.smithi032.stdout:5/246: write f11 [178066,12575] 0 2024-04-06T14:16:58.662 INFO:tasks.workunit.client.0.smithi032.stdout:5/247: dread - d16/d18/f31 zero size 2024-04-06T14:16:58.664 INFO:tasks.workunit.client.0.smithi032.stdout:1/213: getdents d0/d41 0 2024-04-06T14:16:58.666 INFO:tasks.workunit.client.0.smithi032.stdout:1/214: creat d0/d9/d1f/f4a x:0 0 0 2024-04-06T14:16:58.666 INFO:tasks.workunit.client.0.smithi032.stdout:8/232: dread d3/d8/d12/f4d [0,4194304] 0 2024-04-06T14:16:58.668 INFO:tasks.workunit.client.0.smithi032.stdout:1/215: chown d0/d10/d16/f3c 1387342 1 2024-04-06T14:16:58.670 INFO:tasks.workunit.client.0.smithi032.stdout:0/188: getdents d3/da/d41 0 2024-04-06T14:16:58.671 INFO:tasks.workunit.client.0.smithi032.stdout:0/189: truncate d3/da/d41/f43 95010 0 2024-04-06T14:16:58.671 INFO:tasks.workunit.client.0.smithi032.stdout:8/233: creat d3/d8/d7/d1a/f5a x:0 0 0 2024-04-06T14:16:58.674 INFO:tasks.workunit.client.0.smithi032.stdout:5/248: rename f3 to d16/d18/d33/f56 0 2024-04-06T14:16:58.675 INFO:tasks.workunit.client.0.smithi032.stdout:1/216: mknod d0/d10/c4b 0 2024-04-06T14:16:58.675 INFO:tasks.workunit.client.0.smithi032.stdout:5/249: readlink d16/d18/d33/d45/l51 0 2024-04-06T14:16:58.676 INFO:tasks.workunit.client.0.smithi032.stdout:0/190: mknod d3/da/c49 0 2024-04-06T14:16:58.680 INFO:tasks.workunit.client.0.smithi032.stdout:4/365: dwrite d0/f1 [0,4194304] 0 2024-04-06T14:16:58.681 INFO:tasks.workunit.client.0.smithi032.stdout:5/250: creat d16/d18/d33/d45/f57 x:0 0 0 2024-04-06T14:16:58.683 INFO:tasks.workunit.client.0.smithi032.stdout:8/234: creat d3/d8/d12/f5b x:0 0 0 2024-04-06T14:16:58.683 INFO:tasks.workunit.client.0.smithi032.stdout:1/217: dread d0/d10/f26 [0,4194304] 0 2024-04-06T14:16:58.684 INFO:tasks.workunit.client.0.smithi032.stdout:8/235: truncate d3/d8/d3a/f42 920649 0 2024-04-06T14:16:58.690 INFO:tasks.workunit.client.0.smithi032.stdout:4/366: creat d0/df/d2b/d46/d58/f7e x:0 0 0 2024-04-06T14:16:58.691 INFO:tasks.workunit.client.0.smithi032.stdout:1/218: creat d0/f4c x:0 0 0 2024-04-06T14:16:58.692 INFO:tasks.workunit.client.0.smithi032.stdout:1/219: truncate d0/f37 965923 0 2024-04-06T14:16:58.692 INFO:tasks.workunit.client.0.smithi032.stdout:8/236: creat d3/d8/d3a/f5c x:0 0 0 2024-04-06T14:16:58.693 INFO:tasks.workunit.client.0.smithi032.stdout:1/220: read - d0/da/f45 zero size 2024-04-06T14:16:58.694 INFO:tasks.workunit.client.0.smithi032.stdout:5/251: rmdir d16/d18 39 2024-04-06T14:16:58.695 INFO:tasks.workunit.client.0.smithi032.stdout:4/367: symlink d0/d47/l7f 0 2024-04-06T14:16:58.696 INFO:tasks.workunit.client.0.smithi032.stdout:1/221: dread - d0/da/f45 zero size 2024-04-06T14:16:58.696 INFO:tasks.workunit.client.0.smithi032.stdout:8/237: mknod d3/d8/d7/d1a/c5d 0 2024-04-06T14:16:58.697 INFO:tasks.workunit.client.0.smithi032.stdout:4/368: read - d0/df/d2b/d46/d58/f7e zero size 2024-04-06T14:16:58.702 INFO:tasks.workunit.client.0.smithi032.stdout:8/238: unlink d3/d8/d7/d1a/c5d 0 2024-04-06T14:16:58.703 INFO:tasks.workunit.client.0.smithi032.stdout:1/222: rename d0/da/f44 to d0/f4d 0 2024-04-06T14:16:58.703 INFO:tasks.workunit.client.0.smithi032.stdout:0/191: dwrite d3/da/f1d [0,4194304] 0 2024-04-06T14:16:58.704 INFO:tasks.workunit.client.0.smithi032.stdout:4/369: chown d0/df/d23/d31/l36 247 1 2024-04-06T14:16:58.705 INFO:tasks.workunit.client.0.smithi032.stdout:1/223: write d0/d9/d1f/f4a [257698,43626] 0 2024-04-06T14:16:58.707 INFO:tasks.workunit.client.0.smithi032.stdout:4/370: rename d0/df/d23/c24 to d0/df/d76/c80 0 2024-04-06T14:16:58.708 INFO:tasks.workunit.client.0.smithi032.stdout:1/224: symlink d0/d9/d1f/l4e 0 2024-04-06T14:16:58.712 INFO:tasks.workunit.client.0.smithi032.stdout:8/239: getdents d3/d8/d12 0 2024-04-06T14:16:58.712 INFO:tasks.workunit.client.0.smithi032.stdout:0/192: getdents d3/da/d25/d36 0 2024-04-06T14:16:58.713 INFO:tasks.workunit.client.0.smithi032.stdout:1/225: mknod d0/d9/d1f/c4f 0 2024-04-06T14:16:58.715 INFO:tasks.workunit.client.0.smithi032.stdout:8/240: chown d3/d8/d7/d1a/c2d 90187 1 2024-04-06T14:16:58.716 INFO:tasks.workunit.client.0.smithi032.stdout:5/252: dwrite d16/f2c [0,4194304] 0 2024-04-06T14:16:58.716 INFO:tasks.workunit.client.0.smithi032.stdout:0/193: chown d3/da/d25/c28 66292432 1 2024-04-06T14:16:58.717 INFO:tasks.workunit.client.0.smithi032.stdout:8/241: write d3/d8/d12/f3f [2693586,80542] 0 2024-04-06T14:16:58.718 INFO:tasks.workunit.client.0.smithi032.stdout:1/226: creat d0/da/d3e/f50 x:0 0 0 2024-04-06T14:16:58.720 INFO:tasks.workunit.client.0.smithi032.stdout:0/194: chown d3/l11 9699643 1 2024-04-06T14:16:58.721 INFO:tasks.workunit.client.0.smithi032.stdout:1/227: creat d0/d10/d16/f51 x:0 0 0 2024-04-06T14:16:58.723 INFO:tasks.workunit.client.0.smithi032.stdout:8/242: creat d3/d8/d3a/d4b/f5e x:0 0 0 2024-04-06T14:16:58.725 INFO:tasks.workunit.client.0.smithi032.stdout:1/228: rmdir d0/da/d3e 39 2024-04-06T14:16:58.727 INFO:tasks.workunit.client.0.smithi032.stdout:8/243: truncate d3/d8/d7/d14/f45 4488610 0 2024-04-06T14:16:58.728 INFO:tasks.workunit.client.0.smithi032.stdout:5/253: rename d16/d18/d33/d45/d52/d46 to d16/d58 0 2024-04-06T14:16:58.728 INFO:tasks.workunit.client.0.smithi032.stdout:0/195: dwrite d3/da/f2d [0,4194304] 0 2024-04-06T14:16:58.729 INFO:tasks.workunit.client.0.smithi032.stdout:8/244: chown d3/d8/d12/f38 5427823 1 2024-04-06T14:16:58.729 INFO:tasks.workunit.client.0.smithi032.stdout:5/254: chown d16/d18/d33/d45/c32 13 1 2024-04-06T14:16:58.732 INFO:tasks.workunit.client.0.smithi032.stdout:3/190: sync 2024-04-06T14:16:58.732 INFO:tasks.workunit.client.0.smithi032.stdout:5/255: creat d16/d58/f59 x:0 0 0 2024-04-06T14:16:58.734 INFO:tasks.workunit.client.0.smithi032.stdout:1/229: truncate d0/f37 562358 0 2024-04-06T14:16:58.735 INFO:tasks.workunit.client.0.smithi032.stdout:8/245: creat d3/d8/f5f x:0 0 0 2024-04-06T14:16:58.738 INFO:tasks.workunit.client.0.smithi032.stdout:3/191: rename d3/d16/f38 to d3/d3b/d3c/f57 0 2024-04-06T14:16:58.738 INFO:tasks.workunit.client.0.smithi032.stdout:1/230: rmdir d0/da/d3e 39 2024-04-06T14:16:58.739 INFO:tasks.workunit.client.0.smithi032.stdout:1/231: chown d0/da/l18 243 1 2024-04-06T14:16:58.740 INFO:tasks.workunit.client.0.smithi032.stdout:8/246: mkdir d3/d8/d12/d2f/d4a/d60 0 2024-04-06T14:16:58.741 INFO:tasks.workunit.client.0.smithi032.stdout:8/247: truncate d3/d8/d3a/f4c 4796141 0 2024-04-06T14:16:58.742 INFO:tasks.workunit.client.0.smithi032.stdout:3/192: rename d3/l45 to d3/d3b/d3c/l58 0 2024-04-06T14:16:58.743 INFO:tasks.workunit.client.0.smithi032.stdout:1/232: unlink d0/d9/f1b 0 2024-04-06T14:16:58.744 INFO:tasks.workunit.client.0.smithi032.stdout:8/248: creat d3/d8/d7/d1a/f61 x:0 0 0 2024-04-06T14:16:58.751 INFO:tasks.workunit.client.0.smithi032.stdout:5/256: truncate d16/d18/f1e 3055337 0 2024-04-06T14:16:58.751 INFO:tasks.workunit.client.0.smithi032.stdout:3/193: link d3/f37 d3/d3b/d3c/f59 0 2024-04-06T14:16:58.755 INFO:tasks.workunit.client.0.smithi032.stdout:8/249: fsync d3/d8/d7/d1a/f61 0 2024-04-06T14:16:58.860 INFO:tasks.workunit.client.0.smithi032.stdout:3/194: write d3/f20 [154062,91737] 0 2024-04-06T14:16:58.861 INFO:tasks.workunit.client.0.smithi032.stdout:5/257: truncate f10 1323211 0 2024-04-06T14:16:58.863 INFO:tasks.workunit.client.0.smithi032.stdout:8/250: creat d3/d8/d12/d2f/f62 x:0 0 0 2024-04-06T14:16:58.865 INFO:tasks.workunit.client.0.smithi032.stdout:5/258: symlink d16/d18/l5a 0 2024-04-06T14:16:58.866 INFO:tasks.workunit.client.0.smithi032.stdout:1/233: dwrite d0/d10/d16/f3c [0,4194304] 0 2024-04-06T14:16:58.867 INFO:tasks.workunit.client.0.smithi032.stdout:3/195: dread d3/f19 [4194304,4194304] 0 2024-04-06T14:16:58.869 INFO:tasks.workunit.client.0.smithi032.stdout:5/259: creat d16/d18/d33/d45/d52/d39/f5b x:0 0 0 2024-04-06T14:16:58.870 INFO:tasks.workunit.client.0.smithi032.stdout:3/196: mknod d3/d16/c5a 0 2024-04-06T14:16:58.873 INFO:tasks.workunit.client.0.smithi032.stdout:1/234: link d0/d10/d16/f42 d0/da/f52 0 2024-04-06T14:16:58.873 INFO:tasks.workunit.client.0.smithi032.stdout:1/235: dread - d0/f36 zero size 2024-04-06T14:16:58.874 INFO:tasks.workunit.client.0.smithi032.stdout:1/236: readlink d0/d10/l39 0 2024-04-06T14:16:58.875 INFO:tasks.workunit.client.0.smithi032.stdout:3/197: dread d3/f4 [0,4194304] 0 2024-04-06T14:16:58.876 INFO:tasks.workunit.client.0.smithi032.stdout:3/198: truncate d3/d3b/f54 964310 0 2024-04-06T14:16:58.876 INFO:tasks.workunit.client.0.smithi032.stdout:1/237: mknod d0/d10/d16/d3f/c53 0 2024-04-06T14:16:58.877 INFO:tasks.workunit.client.0.smithi032.stdout:1/238: readlink d0/da/l18 0 2024-04-06T14:16:58.878 INFO:tasks.workunit.client.0.smithi032.stdout:3/199: symlink d3/d16/d1e/l5b 0 2024-04-06T14:16:58.879 INFO:tasks.workunit.client.0.smithi032.stdout:1/239: readlink d0/d10/d16/l2d 0 2024-04-06T14:16:58.879 INFO:tasks.workunit.client.0.smithi032.stdout:3/200: write d3/d16/d1e/f32 [37374,71213] 0 2024-04-06T14:16:58.881 INFO:tasks.workunit.client.0.smithi032.stdout:3/201: creat d3/d16/f5c x:0 0 0 2024-04-06T14:16:58.882 INFO:tasks.workunit.client.0.smithi032.stdout:1/240: mknod d0/d9/c54 0 2024-04-06T14:16:58.882 INFO:tasks.workunit.client.0.smithi032.stdout:3/202: truncate d3/f4 5138701 0 2024-04-06T14:16:58.883 INFO:tasks.workunit.client.0.smithi032.stdout:3/203: readlink l2 0 2024-04-06T14:16:58.891 INFO:tasks.workunit.client.0.smithi032.stdout:1/241: write d0/f17 [2625147,70906] 0 2024-04-06T14:16:58.898 INFO:tasks.workunit.client.0.smithi032.stdout:1/242: dwrite d0/d10/f30 [0,4194304] 0 2024-04-06T14:16:58.899 INFO:tasks.workunit.client.0.smithi032.stdout:1/243: mknod d0/d9/c55 0 2024-04-06T14:16:58.899 INFO:tasks.workunit.client.0.smithi032.stdout:1/244: stat d0/da/f52 0 2024-04-06T14:16:58.904 INFO:tasks.workunit.client.0.smithi032.stdout:1/245: write d0/f37 [1333727,25295] 0 2024-04-06T14:16:58.907 INFO:tasks.workunit.client.0.smithi032.stdout:1/246: unlink d0/c7 0 2024-04-06T14:16:58.913 INFO:tasks.workunit.client.0.smithi032.stdout:1/247: dwrite d0/da/f28 [0,4194304] 0 2024-04-06T14:16:58.914 INFO:tasks.workunit.client.0.smithi032.stdout:1/248: stat d0/d41 0 2024-04-06T14:16:58.914 INFO:tasks.workunit.client.0.smithi032.stdout:1/249: chown d0/da/f38 1396395 1 2024-04-06T14:16:58.915 INFO:tasks.workunit.client.0.smithi032.stdout:7/224: sync 2024-04-06T14:16:58.916 INFO:tasks.workunit.client.0.smithi032.stdout:1/250: dread - d0/d9/d1f/f46 zero size 2024-04-06T14:16:58.916 INFO:tasks.workunit.client.0.smithi032.stdout:7/225: readlink d16/l18 0 2024-04-06T14:16:58.920 INFO:tasks.workunit.client.0.smithi032.stdout:1/251: rmdir d0 39 2024-04-06T14:16:58.922 INFO:tasks.workunit.client.0.smithi032.stdout:7/226: creat d16/d20/d26/d2d/f4f x:0 0 0 2024-04-06T14:16:58.923 INFO:tasks.workunit.client.0.smithi032.stdout:7/227: dread - d16/d20/d26/d2d/f4f zero size 2024-04-06T14:16:58.927 INFO:tasks.workunit.client.0.smithi032.stdout:1/252: rename d0/l15 to d0/d10/l56 0 2024-04-06T14:16:58.929 INFO:tasks.workunit.client.0.smithi032.stdout:7/228: dwrite fd [0,4194304] 0 2024-04-06T14:16:58.931 INFO:tasks.workunit.client.0.smithi032.stdout:1/253: rename d0/d9/d1f/f4a to d0/d9/f57 0 2024-04-06T14:16:58.932 INFO:tasks.workunit.client.0.smithi032.stdout:1/254: fdatasync d0/d9/f24 0 2024-04-06T14:16:58.933 INFO:tasks.workunit.client.0.smithi032.stdout:7/229: mknod d16/d20/d26/d2d/d30/d3e/c50 0 2024-04-06T14:16:58.935 INFO:tasks.workunit.client.0.smithi032.stdout:7/230: symlink d16/d20/d26/d2d/l51 0 2024-04-06T14:16:58.937 INFO:tasks.workunit.client.0.smithi032.stdout:7/231: chown d16/d20/d26/d2d/d30/f38 6 1 2024-04-06T14:16:58.938 INFO:tasks.workunit.client.0.smithi032.stdout:1/255: write d0/d10/d16/f3c [644123,15877] 0 2024-04-06T14:16:58.939 INFO:tasks.workunit.client.0.smithi032.stdout:7/232: creat d16/d20/d26/d2d/d30/f52 x:0 0 0 2024-04-06T14:16:58.940 INFO:tasks.workunit.client.0.smithi032.stdout:7/233: truncate f10 1831754 0 2024-04-06T14:16:58.941 INFO:tasks.workunit.client.0.smithi032.stdout:1/256: dread - d0/da/d3e/f50 zero size 2024-04-06T14:16:58.941 INFO:tasks.workunit.client.0.smithi032.stdout:7/234: write d16/d20/d26/d2d/f4f [856319,37255] 0 2024-04-06T14:16:58.942 INFO:tasks.workunit.client.0.smithi032.stdout:7/235: write f10 [942739,55624] 0 2024-04-06T14:16:58.944 INFO:tasks.workunit.client.0.smithi032.stdout:7/236: write d16/d20/d26/d2d/f3f [157397,86164] 0 2024-04-06T14:16:58.944 INFO:tasks.workunit.client.0.smithi032.stdout:1/257: mkdir d0/da/d58 0 2024-04-06T14:16:58.946 INFO:tasks.workunit.client.0.smithi032.stdout:7/237: creat d16/d20/d26/d2d/d30/f53 x:0 0 0 2024-04-06T14:16:58.947 INFO:tasks.workunit.client.0.smithi032.stdout:1/258: mkdir d0/d41/d59 0 2024-04-06T14:16:58.949 INFO:tasks.workunit.client.0.smithi032.stdout:1/259: symlink d0/da/d3e/l5a 0 2024-04-06T14:16:58.950 INFO:tasks.workunit.client.0.smithi032.stdout:1/260: write d0/f17 [371295,71795] 0 2024-04-06T14:16:58.953 INFO:tasks.workunit.client.0.smithi032.stdout:7/238: dwrite d16/d20/d26/d2d/f3f [0,4194304] 0 2024-04-06T14:16:58.954 INFO:tasks.workunit.client.0.smithi032.stdout:7/239: fsync f6 0 2024-04-06T14:16:58.956 INFO:tasks.workunit.client.0.smithi032.stdout:1/261: getdents d0/d10/d16/d3f 0 2024-04-06T14:16:58.957 INFO:tasks.workunit.client.0.smithi032.stdout:1/262: fdatasync d0/d9/d1f/f3b 0 2024-04-06T14:16:58.957 INFO:tasks.workunit.client.0.smithi032.stdout:1/263: stat d0/d10/d16/f3c 0 2024-04-06T14:16:58.957 INFO:tasks.workunit.client.0.smithi032.stdout:7/240: mkdir d16/d20/d54 0 2024-04-06T14:16:58.959 INFO:tasks.workunit.client.0.smithi032.stdout:7/241: write d16/d20/d48/f4a [67237,114886] 0 2024-04-06T14:16:58.959 INFO:tasks.workunit.client.0.smithi032.stdout:1/264: mkdir d0/d41/d5b 0 2024-04-06T14:16:58.962 INFO:tasks.workunit.client.0.smithi032.stdout:1/265: rename d0/d9/f57 to d0/d41/f5c 0 2024-04-06T14:16:58.963 INFO:tasks.workunit.client.0.smithi032.stdout:7/242: rename d16/d20/d26/d2d/d30 to d16/d20/d26/d2d/d30/d55 22 2024-04-06T14:16:58.965 INFO:tasks.workunit.client.0.smithi032.stdout:7/243: symlink d16/d20/l56 0 2024-04-06T14:16:58.965 INFO:tasks.workunit.client.0.smithi032.stdout:7/244: readlink d16/l25 0 2024-04-06T14:16:58.966 INFO:tasks.workunit.client.0.smithi032.stdout:1/266: dread d0/d9/f23 [0,4194304] 0 2024-04-06T14:16:58.967 INFO:tasks.workunit.client.0.smithi032.stdout:1/267: fdatasync d0/f36 0 2024-04-06T14:16:58.968 INFO:tasks.workunit.client.0.smithi032.stdout:1/268: dread - d0/d9/d1f/f46 zero size 2024-04-06T14:16:58.968 INFO:tasks.workunit.client.0.smithi032.stdout:7/245: rename d16/d20/d26/d2d/d30/f35 to d16/d20/d54/f57 0 2024-04-06T14:16:58.974 INFO:tasks.workunit.client.0.smithi032.stdout:1/269: write d0/d10/f26 [49566,78478] 0 2024-04-06T14:16:58.976 INFO:tasks.workunit.client.0.smithi032.stdout:7/246: rmdir d16/d20/d26/d2d 39 2024-04-06T14:16:58.976 INFO:tasks.workunit.client.0.smithi032.stdout:7/247: write d16/f4b [876550,120756] 0 2024-04-06T14:16:58.977 INFO:tasks.workunit.client.0.smithi032.stdout:1/270: write d0/d9/f23 [2253160,322] 0 2024-04-06T14:16:58.978 INFO:tasks.workunit.client.0.smithi032.stdout:1/271: stat d0/d10/f30 0 2024-04-06T14:16:58.979 INFO:tasks.workunit.client.0.smithi032.stdout:7/248: dread - d16/d20/d26/d2d/d30/f52 zero size 2024-04-06T14:16:58.980 INFO:tasks.workunit.client.0.smithi032.stdout:7/249: chown d16/d20/d48/f4a 15 1 2024-04-06T14:16:58.981 INFO:tasks.workunit.client.0.smithi032.stdout:1/272: creat d0/d10/d16/d3f/f5d x:0 0 0 2024-04-06T14:16:58.981 INFO:tasks.workunit.client.0.smithi032.stdout:1/273: fdatasync d0/f37 0 2024-04-06T14:16:58.984 INFO:tasks.workunit.client.0.smithi032.stdout:7/250: truncate f6 2243630 0 2024-04-06T14:16:58.984 INFO:tasks.workunit.client.0.smithi032.stdout:1/274: mknod d0/da/d3e/c5e 0 2024-04-06T14:16:58.985 INFO:tasks.workunit.client.0.smithi032.stdout:1/275: write d0/da/f38 [250869,35361] 0 2024-04-06T14:16:58.987 INFO:tasks.workunit.client.0.smithi032.stdout:7/251: creat d16/d20/d26/f58 x:0 0 0 2024-04-06T14:16:58.988 INFO:tasks.workunit.client.0.smithi032.stdout:7/252: fdatasync d16/d20/f24 0 2024-04-06T14:16:58.991 INFO:tasks.workunit.client.0.smithi032.stdout:7/253: truncate d16/d20/d26/d2d/f4f 1477532 0 2024-04-06T14:16:58.992 INFO:tasks.workunit.client.0.smithi032.stdout:1/276: dwrite d0/f2c [4194304,4194304] 0 2024-04-06T14:16:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:7/254: mkdir d16/d20/d26/d2d/d30/d3e/d59 0 2024-04-06T14:16:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:1/277: creat d0/d10/d16/f5f x:0 0 0 2024-04-06T14:16:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:7/255: creat d16/d20/d48/f5a x:0 0 0 2024-04-06T14:16:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:1/278: write d0/f2c [327319,59594] 0 2024-04-06T14:16:58.997 INFO:tasks.workunit.client.0.smithi032.stdout:7/256: write d16/d20/d26/f31 [1390895,16159] 0 2024-04-06T14:16:58.999 INFO:tasks.workunit.client.0.smithi032.stdout:1/279: creat d0/d41/d59/f60 x:0 0 0 2024-04-06T14:16:59.001 INFO:tasks.workunit.client.0.smithi032.stdout:7/257: symlink d16/d20/d54/l5b 0 2024-04-06T14:16:59.002 INFO:tasks.workunit.client.0.smithi032.stdout:7/258: chown d16/d20/d26/c29 15525090 1 2024-04-06T14:16:59.003 INFO:tasks.workunit.client.0.smithi032.stdout:1/280: mknod d0/d9/c61 0 2024-04-06T14:16:59.003 INFO:tasks.workunit.client.0.smithi032.stdout:1/281: dread - d0/f1e zero size 2024-04-06T14:16:59.004 INFO:tasks.workunit.client.0.smithi032.stdout:7/259: mknod d16/d20/d26/d2d/d30/c5c 0 2024-04-06T14:16:59.004 INFO:tasks.workunit.client.0.smithi032.stdout:1/282: readlink d0/d9/d1f/l4e 0 2024-04-06T14:16:59.009 INFO:tasks.workunit.client.0.smithi032.stdout:7/260: write fe [207551,62042] 0 2024-04-06T14:16:59.011 INFO:tasks.workunit.client.0.smithi032.stdout:7/261: rename d16/d20/d54/l5b to d16/d20/l5d 0 2024-04-06T14:16:59.013 INFO:tasks.workunit.client.0.smithi032.stdout:7/262: truncate d16/d20/d26/d2d/f32 74974 0 2024-04-06T14:16:59.015 INFO:tasks.workunit.client.0.smithi032.stdout:7/263: rename f6 to d16/d20/f5e 0 2024-04-06T14:16:59.080 INFO:tasks.workunit.client.0.smithi032.stdout:6/225: sync 2024-04-06T14:16:59.080 INFO:tasks.workunit.client.0.smithi032.stdout:9/244: sync 2024-04-06T14:16:59.083 INFO:tasks.workunit.client.0.smithi032.stdout:9/245: dread d2/d5/f37 [0,4194304] 0 2024-04-06T14:16:59.084 INFO:tasks.workunit.client.0.smithi032.stdout:6/226: dread d4/d5/d18/d23/f27 [0,4194304] 0 2024-04-06T14:16:59.084 INFO:tasks.workunit.client.0.smithi032.stdout:9/246: chown d2/d15/d55 31175309 1 2024-04-06T14:16:59.085 INFO:tasks.workunit.client.0.smithi032.stdout:6/227: write d4/d5/d14/f1a [67610,104914] 0 2024-04-06T14:16:59.085 INFO:tasks.workunit.client.0.smithi032.stdout:9/247: write d2/d15/d1e/d47/f4c [710959,43655] 0 2024-04-06T14:16:59.087 INFO:tasks.workunit.client.0.smithi032.stdout:6/228: truncate d4/d5/d8/d1f/f33 76908 0 2024-04-06T14:16:59.088 INFO:tasks.workunit.client.0.smithi032.stdout:6/229: dread - d4/d5/d35/f39 zero size 2024-04-06T14:16:59.089 INFO:tasks.workunit.client.0.smithi032.stdout:9/248: creat d2/d5/f56 x:0 0 0 2024-04-06T14:16:59.092 INFO:tasks.workunit.client.0.smithi032.stdout:9/249: link d2/d15/f43 d2/d25/d31/f57 0 2024-04-06T14:16:59.094 INFO:tasks.workunit.client.0.smithi032.stdout:6/230: dwrite d4/f10 [4194304,4194304] 0 2024-04-06T14:16:59.095 INFO:tasks.workunit.client.0.smithi032.stdout:9/250: write d2/fd [1405150,102819] 0 2024-04-06T14:16:59.098 INFO:tasks.workunit.client.0.smithi032.stdout:9/251: creat d2/d25/d31/d39/f58 x:0 0 0 2024-04-06T14:16:59.101 INFO:tasks.workunit.client.0.smithi032.stdout:9/252: mkdir d2/d15/d55/d59 0 2024-04-06T14:16:59.101 INFO:tasks.workunit.client.0.smithi032.stdout:9/253: readlink d2/d25/l30 0 2024-04-06T14:16:59.103 INFO:tasks.workunit.client.0.smithi032.stdout:6/231: truncate d4/d5/f15 879799 0 2024-04-06T14:16:59.103 INFO:tasks.workunit.client.0.smithi032.stdout:9/254: creat d2/d25/d31/d39/f5a x:0 0 0 2024-04-06T14:16:59.105 INFO:tasks.workunit.client.0.smithi032.stdout:9/255: chown d2/d15/l20 0 1 2024-04-06T14:16:59.108 INFO:tasks.workunit.client.0.smithi032.stdout:9/256: unlink d2/d15/f2f 0 2024-04-06T14:16:59.108 INFO:tasks.workunit.client.0.smithi032.stdout:6/232: dwrite d4/f10 [0,4194304] 0 2024-04-06T14:16:59.113 INFO:tasks.workunit.client.0.smithi032.stdout:9/257: mknod d2/d25/d31/c5b 0 2024-04-06T14:16:59.117 INFO:tasks.workunit.client.0.smithi032.stdout:9/258: getdents d2/d15/d1e/d47/d4a 0 2024-04-06T14:16:59.119 INFO:tasks.workunit.client.0.smithi032.stdout:6/233: chown d4/d5/f15 10 1 2024-04-06T14:16:59.121 INFO:tasks.workunit.client.0.smithi032.stdout:9/259: getdents d2 0 2024-04-06T14:16:59.122 INFO:tasks.workunit.client.0.smithi032.stdout:4/371: sync 2024-04-06T14:16:59.122 INFO:tasks.workunit.client.0.smithi032.stdout:0/196: sync 2024-04-06T14:16:59.122 INFO:tasks.workunit.client.0.smithi032.stdout:2/212: sync 2024-04-06T14:16:59.123 INFO:tasks.workunit.client.0.smithi032.stdout:8/251: sync 2024-04-06T14:16:59.125 INFO:tasks.workunit.client.0.smithi032.stdout:9/260: mknod d2/d15/d1e/d41/c5c 0 2024-04-06T14:16:59.131 INFO:tasks.workunit.client.0.smithi032.stdout:9/261: fdatasync d2/d15/f2b 0 2024-04-06T14:16:59.132 INFO:tasks.workunit.client.0.smithi032.stdout:8/252: mknod d3/d8/c63 0 2024-04-06T14:16:59.135 INFO:tasks.workunit.client.0.smithi032.stdout:6/234: dwrite d4/d5/d18/f31 [0,4194304] 0 2024-04-06T14:16:59.135 INFO:tasks.workunit.client.0.smithi032.stdout:0/197: mkdir d3/da/d25/d2b/d4a 0 2024-04-06T14:16:59.136 INFO:tasks.workunit.client.0.smithi032.stdout:8/253: stat d3/d8/f25 0 2024-04-06T14:16:59.138 INFO:tasks.workunit.client.0.smithi032.stdout:6/235: chown d4 0 1 2024-04-06T14:16:59.140 INFO:tasks.workunit.client.0.smithi032.stdout:4/372: getdents d0/df/d2b/d43 0 2024-04-06T14:16:59.141 INFO:tasks.workunit.client.0.smithi032.stdout:8/254: mknod d3/d8/d7/c64 0 2024-04-06T14:16:59.142 INFO:tasks.workunit.client.0.smithi032.stdout:2/213: dwrite d0/d12/d3a/f42 [0,4194304] 0 2024-04-06T14:16:59.142 INFO:tasks.workunit.client.0.smithi032.stdout:6/236: write d4/d5/d8/d1f/f33 [535414,49058] 0 2024-04-06T14:16:59.144 INFO:tasks.workunit.client.0.smithi032.stdout:8/255: write d3/f31 [207058,110952] 0 2024-04-06T14:16:59.148 INFO:tasks.workunit.client.0.smithi032.stdout:2/214: fsync d0/d12/f2a 0 2024-04-06T14:16:59.150 INFO:tasks.workunit.client.0.smithi032.stdout:4/373: mknod d0/df/d23/c81 0 2024-04-06T14:16:59.155 INFO:tasks.workunit.client.0.smithi032.stdout:8/256: rmdir d3/d8/d7 39 2024-04-06T14:16:59.156 INFO:tasks.workunit.client.0.smithi032.stdout:0/198: dread d3/da/d25/f2a [0,4194304] 0 2024-04-06T14:16:59.156 INFO:tasks.workunit.client.0.smithi032.stdout:4/374: unlink d0/df/d2b/d4f/l59 0 2024-04-06T14:16:59.157 INFO:tasks.workunit.client.0.smithi032.stdout:9/262: dwrite d2/d5/f12 [4194304,4194304] 0 2024-04-06T14:16:59.157 INFO:tasks.workunit.client.0.smithi032.stdout:2/215: mknod d0/d12/c4b 0 2024-04-06T14:16:59.159 INFO:tasks.workunit.client.0.smithi032.stdout:8/257: creat d3/d8/d12/d2f/f65 x:0 0 0 2024-04-06T14:16:59.161 INFO:tasks.workunit.client.0.smithi032.stdout:0/199: unlink d3/da/l2f 0 2024-04-06T14:16:59.162 INFO:tasks.workunit.client.0.smithi032.stdout:2/216: write d0/d12/f21 [3976274,64774] 0 2024-04-06T14:16:59.165 INFO:tasks.workunit.client.0.smithi032.stdout:8/258: dread - d3/d8/d12/d2f/f39 zero size 2024-04-06T14:16:59.167 INFO:tasks.workunit.client.0.smithi032.stdout:2/217: creat d0/d12/f4c x:0 0 0 2024-04-06T14:16:59.167 INFO:tasks.workunit.client.0.smithi032.stdout:0/200: write d3/da/d25/f2a [468651,128395] 0 2024-04-06T14:16:59.170 INFO:tasks.workunit.client.0.smithi032.stdout:6/237: dwrite d4/d5/d8/d1f/f37 [0,4194304] 0 2024-04-06T14:16:59.173 INFO:tasks.workunit.client.0.smithi032.stdout:4/375: dwrite d0/df/d1f/d2e/f41 [4194304,4194304] 0 2024-04-06T14:16:59.174 INFO:tasks.workunit.client.0.smithi032.stdout:0/201: mkdir d3/da/d35/d4b 0 2024-04-06T14:16:59.175 INFO:tasks.workunit.client.0.smithi032.stdout:2/218: rename d0/d12/d3a/f45 to d0/d30/f4d 0 2024-04-06T14:16:59.176 INFO:tasks.workunit.client.0.smithi032.stdout:8/259: link d3/d8/d3a/f42 d3/d8/d12/f66 0 2024-04-06T14:16:59.181 INFO:tasks.workunit.client.0.smithi032.stdout:6/238: rename d4/ce to d4/d5/d8/d2b/c3b 0 2024-04-06T14:16:59.288 INFO:tasks.workunit.client.0.smithi032.stdout:0/202: symlink d3/da/d25/d2b/d4a/l4c 0 2024-04-06T14:16:59.288 INFO:tasks.workunit.client.0.smithi032.stdout:6/239: chown d4/d5/d18/f31 368 1 2024-04-06T14:16:59.288 INFO:tasks.workunit.client.0.smithi032.stdout:2/219: dwrite d0/fc [0,4194304] 0 2024-04-06T14:16:59.288 INFO:tasks.workunit.client.0.smithi032.stdout:0/203: dread d3/f2e [0,4194304] 0 2024-04-06T14:16:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:2/220: mkdir d0/d4e 0 2024-04-06T14:16:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:2/221: fsync d0/d12/d3a/f48 0 2024-04-06T14:16:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:0/204: rename d3/da/d25/d2b/d4a/l4c to d3/da/d35/d4b/l4d 0 2024-04-06T14:16:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:6/240: dwrite d4/d5/d8/fc [0,4194304] 0 2024-04-06T14:16:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:9/263: dread d2/d5/f12 [0,4194304] 0 2024-04-06T14:16:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:0/205: write d3/f10 [1197898,64302] 0 2024-04-06T14:16:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:9/264: truncate d2/d15/f2b 1360735 0 2024-04-06T14:16:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:8/260: rename d3/d8/d3a/c43 to d3/d8/d12/d2f/d46/c67 0 2024-04-06T14:16:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:5/260: sync 2024-04-06T14:16:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:7/264: sync 2024-04-06T14:16:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:3/204: sync 2024-04-06T14:16:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:1/283: sync 2024-04-06T14:16:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:0/206: link d3/da/f1d d3/da/d35/f4e 0 2024-04-06T14:16:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:6/241: truncate d4/d5/d18/f31 1639963 0 2024-04-06T14:16:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:3/205: chown d3/d16/f2a 186 1 2024-04-06T14:16:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:8/261: symlink d3/d8/d12/d2f/d4a/d60/l68 0 2024-04-06T14:16:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:7/265: unlink d16/c1d 0 2024-04-06T14:16:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:6/242: chown d4/d5/d28 1744401795 1 2024-04-06T14:16:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:5/261: chown f10 25931 1 2024-04-06T14:16:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:1/284: rename d0/f2 to d0/d10/d16/f62 0 2024-04-06T14:16:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:0/207: symlink d3/da/d25/d36/l4f 0 2024-04-06T14:16:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:5/262: dread - d16/d18/f3c zero size 2024-04-06T14:16:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:8/262: mkdir d3/d69 0 2024-04-06T14:16:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:1/285: dread - d0/d41/d59/f60 zero size 2024-04-06T14:16:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:0/208: fdatasync d3/f4 0 2024-04-06T14:16:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:3/206: dread d3/d3b/d3c/f59 [0,4194304] 0 2024-04-06T14:16:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:7/266: mkdir d16/d20/d26/d5f 0 2024-04-06T14:16:59.292 INFO:tasks.workunit.client.0.smithi032.stdout:9/265: dwrite d2/d15/f43 [0,4194304] 0 2024-04-06T14:16:59.292 INFO:tasks.workunit.client.0.smithi032.stdout:1/286: write d0/f1e [160365,2747] 0 2024-04-06T14:16:59.292 INFO:tasks.workunit.client.0.smithi032.stdout:6/243: dread d4/d5/d8/d1f/f33 [0,4194304] 0 2024-04-06T14:16:59.292 INFO:tasks.workunit.client.0.smithi032.stdout:0/209: mknod d3/da/c50 0 2024-04-06T14:16:59.292 INFO:tasks.workunit.client.0.smithi032.stdout:7/267: dwrite d16/d20/d26/d2d/f3f [0,4194304] 0 2024-04-06T14:16:59.292 INFO:tasks.workunit.client.0.smithi032.stdout:5/263: truncate d16/d18/d33/d45/d52/d39/f47 843561 0 2024-04-06T14:16:59.293 INFO:tasks.workunit.client.0.smithi032.stdout:9/266: dwrite d2/d15/d1e/f1f [0,4194304] 0 2024-04-06T14:16:59.293 INFO:tasks.workunit.client.0.smithi032.stdout:1/287: creat d0/da/f63 x:0 0 0 2024-04-06T14:16:59.295 INFO:tasks.workunit.client.0.smithi032.stdout:8/263: truncate d3/d8/f2e 237686 0 2024-04-06T14:16:59.296 INFO:tasks.workunit.client.0.smithi032.stdout:1/288: dread d0/d9/f23 [0,4194304] 0 2024-04-06T14:16:59.297 INFO:tasks.workunit.client.0.smithi032.stdout:7/268: rename d16/d20/l5d to d16/d20/l60 0 2024-04-06T14:16:59.298 INFO:tasks.workunit.client.0.smithi032.stdout:1/289: write d0/f1e [3846,90163] 0 2024-04-06T14:16:59.298 INFO:tasks.workunit.client.0.smithi032.stdout:4/376: sync 2024-04-06T14:16:59.302 INFO:tasks.workunit.client.0.smithi032.stdout:7/269: dread - d16/d20/d26/d2d/d30/f52 zero size 2024-04-06T14:16:59.303 INFO:tasks.workunit.client.0.smithi032.stdout:5/264: mknod d16/d18/d33/d35/d4d/c5c 0 2024-04-06T14:16:59.304 INFO:tasks.workunit.client.0.smithi032.stdout:1/290: write d0/d9/d1f/f47 [737958,98314] 0 2024-04-06T14:16:59.304 INFO:tasks.workunit.client.0.smithi032.stdout:8/264: mknod d3/d8/d12/d2f/d46/c6a 0 2024-04-06T14:16:59.304 INFO:tasks.workunit.client.0.smithi032.stdout:5/265: chown d16/d18/c2f 3191 1 2024-04-06T14:16:59.304 INFO:tasks.workunit.client.0.smithi032.stdout:7/270: chown d16/d20/d48/f5a 553 1 2024-04-06T14:16:59.305 INFO:tasks.workunit.client.0.smithi032.stdout:7/271: chown d16/f40 1045774 1 2024-04-06T14:16:59.305 INFO:tasks.workunit.client.0.smithi032.stdout:9/267: link d2/d5/f37 d2/d15/d1e/d47/f5d 0 2024-04-06T14:16:59.306 INFO:tasks.workunit.client.0.smithi032.stdout:7/272: stat f10 0 2024-04-06T14:16:59.308 INFO:tasks.workunit.client.0.smithi032.stdout:1/291: creat d0/d41/d59/f64 x:0 0 0 2024-04-06T14:16:59.309 INFO:tasks.workunit.client.0.smithi032.stdout:3/207: dwrite d3/f24 [0,4194304] 0 2024-04-06T14:16:59.310 INFO:tasks.workunit.client.0.smithi032.stdout:8/265: chown d3/d8/d7/d14/f45 30199620 1 2024-04-06T14:16:59.310 INFO:tasks.workunit.client.0.smithi032.stdout:4/377: rename d0/d47 to d0/d82 0 2024-04-06T14:16:59.314 INFO:tasks.workunit.client.0.smithi032.stdout:5/266: rename d16/d18/d33/d45/d52/d39/l3b to d16/d18/d33/d45/l5d 0 2024-04-06T14:16:59.314 INFO:tasks.workunit.client.0.smithi032.stdout:6/244: dread d4/d5/d14/f1a [0,4194304] 0 2024-04-06T14:16:59.314 INFO:tasks.workunit.client.0.smithi032.stdout:0/210: dwrite d3/f2e [0,4194304] 0 2024-04-06T14:16:59.316 INFO:tasks.workunit.client.0.smithi032.stdout:7/273: chown d16/d20/f5e 4619340 1 2024-04-06T14:16:59.317 INFO:tasks.workunit.client.0.smithi032.stdout:0/211: read - d3/da/d25/d2b/f40 zero size 2024-04-06T14:16:59.320 INFO:tasks.workunit.client.0.smithi032.stdout:9/268: symlink d2/d15/d1e/d47/d4a/l5e 0 2024-04-06T14:16:59.321 INFO:tasks.workunit.client.0.smithi032.stdout:3/208: mknod d3/d3b/c5d 0 2024-04-06T14:16:59.325 INFO:tasks.workunit.client.0.smithi032.stdout:3/209: write d3/d16/f2c [101041,32523] 0 2024-04-06T14:16:59.325 INFO:tasks.workunit.client.0.smithi032.stdout:5/267: mkdir d16/d18/d33/d45/d5e 0 2024-04-06T14:16:59.325 INFO:tasks.workunit.client.0.smithi032.stdout:0/212: dwrite d3/da/d25/d2b/f32 [0,4194304] 0 2024-04-06T14:16:59.326 INFO:tasks.workunit.client.0.smithi032.stdout:3/210: truncate d3/d16/f33 923697 0 2024-04-06T14:16:59.327 INFO:tasks.workunit.client.0.smithi032.stdout:0/213: read d3/da/f23 [3541389,75187] 0 2024-04-06T14:16:59.327 INFO:tasks.workunit.client.0.smithi032.stdout:6/245: mknod d4/d5/d8/d1f/c3c 0 2024-04-06T14:16:59.328 INFO:tasks.workunit.client.0.smithi032.stdout:7/274: mkdir d16/d20/d48/d61 0 2024-04-06T14:16:59.330 INFO:tasks.workunit.client.0.smithi032.stdout:8/266: mknod d3/d8/d12/c6b 0 2024-04-06T14:16:59.334 INFO:tasks.workunit.client.0.smithi032.stdout:9/269: mkdir d2/d15/d55/d5f 0 2024-04-06T14:16:59.335 INFO:tasks.workunit.client.0.smithi032.stdout:1/292: creat d0/da/f65 x:0 0 0 2024-04-06T14:16:59.337 INFO:tasks.workunit.client.0.smithi032.stdout:5/268: unlink d16/d18/d33/d45/f57 0 2024-04-06T14:16:59.337 INFO:tasks.workunit.client.0.smithi032.stdout:0/214: fsync d3/da/f23 0 2024-04-06T14:16:59.339 INFO:tasks.workunit.client.0.smithi032.stdout:6/246: creat d4/d5/d19/d1d/f3d x:0 0 0 2024-04-06T14:16:59.339 INFO:tasks.workunit.client.0.smithi032.stdout:7/275: symlink d16/d20/d26/d2d/l62 0 2024-04-06T14:16:59.346 INFO:tasks.workunit.client.0.smithi032.stdout:5/269: rmdir d16/d18/d33/d45/d52 39 2024-04-06T14:16:59.346 INFO:tasks.workunit.client.0.smithi032.stdout:7/276: dread - d16/d20/d26/f58 zero size 2024-04-06T14:16:59.346 INFO:tasks.workunit.client.0.smithi032.stdout:3/211: rename d3/d16/d1e/f1f to d3/f5e 0 2024-04-06T14:16:59.346 INFO:tasks.workunit.client.0.smithi032.stdout:1/293: mkdir d0/d10/d66 0 2024-04-06T14:16:59.347 INFO:tasks.workunit.client.0.smithi032.stdout:9/270: dwrite d2/d15/f1a [0,4194304] 0 2024-04-06T14:16:59.348 INFO:tasks.workunit.client.0.smithi032.stdout:0/215: creat d3/da/d25/d36/f51 x:0 0 0 2024-04-06T14:16:59.349 INFO:tasks.workunit.client.0.smithi032.stdout:4/378: dread d0/d82/d4b/f63 [0,4194304] 0 2024-04-06T14:16:59.349 INFO:tasks.workunit.client.0.smithi032.stdout:6/247: mknod d4/d5/d14/c3e 0 2024-04-06T14:16:59.351 INFO:tasks.workunit.client.0.smithi032.stdout:5/270: truncate d16/d18/f30 481009 0 2024-04-06T14:16:59.352 INFO:tasks.workunit.client.0.smithi032.stdout:7/277: mkdir d16/d20/d63 0 2024-04-06T14:16:59.356 INFO:tasks.workunit.client.0.smithi032.stdout:9/271: symlink d2/d15/d1e/d47/d4a/l60 0 2024-04-06T14:16:59.356 INFO:tasks.workunit.client.0.smithi032.stdout:0/216: mknod d3/da/d25/d2b/d4a/c52 0 2024-04-06T14:16:59.356 INFO:tasks.workunit.client.0.smithi032.stdout:4/379: chown d0/f1e 197189863 1 2024-04-06T14:16:59.357 INFO:tasks.workunit.client.0.smithi032.stdout:5/271: creat d16/d58/f5f x:0 0 0 2024-04-06T14:16:59.361 INFO:tasks.workunit.client.0.smithi032.stdout:0/217: stat d3/da/d25/d2b 0 2024-04-06T14:16:59.362 INFO:tasks.workunit.client.0.smithi032.stdout:9/272: dread - d2/d25/d31/d39/f5a zero size 2024-04-06T14:16:59.364 INFO:tasks.workunit.client.0.smithi032.stdout:3/212: dwrite d3/f24 [0,4194304] 0 2024-04-06T14:16:59.364 INFO:tasks.workunit.client.0.smithi032.stdout:7/278: link fe d16/d20/d26/d2d/f64 0 2024-04-06T14:16:59.365 INFO:tasks.workunit.client.0.smithi032.stdout:4/380: truncate d0/df/d23/f60 309000 0 2024-04-06T14:16:59.365 INFO:tasks.workunit.client.0.smithi032.stdout:0/218: creat d3/da/f53 x:0 0 0 2024-04-06T14:16:59.366 INFO:tasks.workunit.client.0.smithi032.stdout:1/294: dwrite d0/d9/f24 [0,4194304] 0 2024-04-06T14:16:59.368 INFO:tasks.workunit.client.0.smithi032.stdout:9/273: read d2/d5/f44 [7673599,9070] 0 2024-04-06T14:16:59.368 INFO:tasks.workunit.client.0.smithi032.stdout:5/272: creat d16/d58/d54/f60 x:0 0 0 2024-04-06T14:16:59.371 INFO:tasks.workunit.client.0.smithi032.stdout:8/267: dwrite d3/d8/d12/f66 [0,4194304] 0 2024-04-06T14:16:59.373 INFO:tasks.workunit.client.0.smithi032.stdout:2/222: sync 2024-04-06T14:16:59.373 INFO:tasks.workunit.client.0.smithi032.stdout:4/381: dread d0/df/d23/f60 [0,4194304] 0 2024-04-06T14:16:59.374 INFO:tasks.workunit.client.0.smithi032.stdout:6/248: truncate d4/d5/d8/fc 859026 0 2024-04-06T14:16:59.375 INFO:tasks.workunit.client.0.smithi032.stdout:9/274: truncate d2/d15/f2b 1595279 0 2024-04-06T14:16:59.379 INFO:tasks.workunit.client.0.smithi032.stdout:7/279: creat d16/d20/d26/d5f/f65 x:0 0 0 2024-04-06T14:16:59.379 INFO:tasks.workunit.client.0.smithi032.stdout:6/249: write d4/d5/d35/f39 [944121,1455] 0 2024-04-06T14:16:59.379 INFO:tasks.workunit.client.0.smithi032.stdout:5/273: mknod d16/d18/c61 0 2024-04-06T14:16:59.380 INFO:tasks.workunit.client.0.smithi032.stdout:1/295: write d0/d10/d16/f5f [349424,94917] 0 2024-04-06T14:16:59.380 INFO:tasks.workunit.client.0.smithi032.stdout:4/382: write d0/df/d23/f4a [1159499,46516] 0 2024-04-06T14:16:59.380 INFO:tasks.workunit.client.0.smithi032.stdout:9/275: write d2/d15/d1e/d47/f4b [887861,62559] 0 2024-04-06T14:16:59.382 INFO:tasks.workunit.client.0.smithi032.stdout:3/213: rename d3/d3b/d3c/f59 to d3/d3b/d3c/f5f 0 2024-04-06T14:16:59.383 INFO:tasks.workunit.client.0.smithi032.stdout:5/274: write fe [2300325,24806] 0 2024-04-06T14:16:59.385 INFO:tasks.workunit.client.0.smithi032.stdout:4/383: read - d0/df/d2b/f72 zero size 2024-04-06T14:16:59.386 INFO:tasks.workunit.client.0.smithi032.stdout:2/223: mkdir d0/d4f 0 2024-04-06T14:16:59.387 INFO:tasks.workunit.client.0.smithi032.stdout:2/224: stat d0/fb 0 2024-04-06T14:16:59.387 INFO:tasks.workunit.client.0.smithi032.stdout:8/268: fsync d3/d8/f2e 0 2024-04-06T14:16:59.388 INFO:tasks.workunit.client.0.smithi032.stdout:1/296: write d0/f4d [1813791,119866] 0 2024-04-06T14:16:59.389 INFO:tasks.workunit.client.0.smithi032.stdout:3/214: stat d3/c14 0 2024-04-06T14:16:59.391 INFO:tasks.workunit.client.0.smithi032.stdout:7/280: symlink d16/d20/d48/d61/l66 0 2024-04-06T14:16:59.391 INFO:tasks.workunit.client.0.smithi032.stdout:3/215: readlink d3/l15 0 2024-04-06T14:16:59.392 INFO:tasks.workunit.client.0.smithi032.stdout:0/219: getdents d3/da/d25/d2b 0 2024-04-06T14:16:59.392 INFO:tasks.workunit.client.0.smithi032.stdout:4/384: symlink d0/df/d1f/d2e/l83 0 2024-04-06T14:16:59.394 INFO:tasks.workunit.client.0.smithi032.stdout:8/269: mknod d3/d8/d3a/d4b/c6c 0 2024-04-06T14:16:59.395 INFO:tasks.workunit.client.0.smithi032.stdout:1/297: rename d0 to d0/d10/d67 22 2024-04-06T14:16:59.395 INFO:tasks.workunit.client.0.smithi032.stdout:9/276: link d2/lb d2/d25/l61 0 2024-04-06T14:16:59.395 INFO:tasks.workunit.client.0.smithi032.stdout:3/216: write d3/f20 [279734,72218] 0 2024-04-06T14:16:59.396 INFO:tasks.workunit.client.1.smithi086.stdout:7/12: sync 2024-04-06T14:16:59.396 INFO:tasks.workunit.client.1.smithi086.stdout:0/85: sync 2024-04-06T14:16:59.396 INFO:tasks.workunit.client.1.smithi086.stdout:6/15: sync 2024-04-06T14:16:59.396 INFO:tasks.workunit.client.1.smithi086.stdout:1/9: sync 2024-04-06T14:16:59.396 INFO:tasks.workunit.client.1.smithi086.stdout:8/129: sync 2024-04-06T14:16:59.396 INFO:tasks.workunit.client.1.smithi086.stdout:9/95: sync 2024-04-06T14:16:59.396 INFO:tasks.workunit.client.1.smithi086.stdout:3/69: sync 2024-04-06T14:16:59.396 INFO:tasks.workunit.client.1.smithi086.stdout:5/78: sync 2024-04-06T14:16:59.397 INFO:tasks.workunit.client.1.smithi086.stdout:2/9: sync 2024-04-06T14:16:59.397 INFO:tasks.workunit.client.1.smithi086.stdout:4/29: sync 2024-04-06T14:16:59.397 INFO:tasks.workunit.client.0.smithi032.stdout:6/250: dwrite d4/f13 [0,4194304] 0 2024-04-06T14:16:59.397 INFO:tasks.workunit.client.0.smithi032.stdout:0/220: readlink d3/l11 0 2024-04-06T14:16:59.399 INFO:tasks.workunit.client.0.smithi032.stdout:4/385: creat d0/df/d2b/d46/d58/f84 x:0 0 0 2024-04-06T14:16:59.399 INFO:tasks.workunit.client.0.smithi032.stdout:3/217: fdatasync d3/d16/f35 0 2024-04-06T14:16:59.400 INFO:tasks.workunit.client.0.smithi032.stdout:2/225: rename d0/fc to d0/d30/f50 0 2024-04-06T14:16:59.401 INFO:tasks.workunit.client.1.smithi086.stdout:6/16: creat f5 x:0 0 0 2024-04-06T14:16:59.401 INFO:tasks.workunit.client.0.smithi032.stdout:0/221: fdatasync d3/da/d25/d36/f51 0 2024-04-06T14:16:59.401 INFO:tasks.workunit.client.0.smithi032.stdout:5/275: link d16/d18/d33/d45/d52/f4e d16/f62 0 2024-04-06T14:16:59.402 INFO:tasks.workunit.client.0.smithi032.stdout:4/386: readlink d0/df/d23/d31/d50/d57/l74 0 2024-04-06T14:16:59.403 INFO:tasks.workunit.client.0.smithi032.stdout:2/226: dread - d0/d12/d3a/f40 zero size 2024-04-06T14:16:59.404 INFO:tasks.workunit.client.0.smithi032.stdout:6/251: fdatasync d4/d5/d19/d1d/f3d 0 2024-04-06T14:16:59.405 INFO:tasks.workunit.client.0.smithi032.stdout:5/276: dread - f9 zero size 2024-04-06T14:16:59.406 INFO:tasks.workunit.client.1.smithi086.stdout:2/10: symlink d0/l1 0 2024-04-06T14:16:59.406 INFO:tasks.workunit.client.0.smithi032.stdout:2/227: chown d0/c32 43006012 1 2024-04-06T14:16:59.407 INFO:tasks.workunit.client.1.smithi086.stdout:1/10: chown f1 0 1 2024-04-06T14:16:59.408 INFO:tasks.workunit.client.1.smithi086.stdout:8/130: chown d0/d13/f21 111029 1 2024-04-06T14:16:59.410 INFO:tasks.workunit.client.1.smithi086.stdout:4/30: symlink d0/lb 0 2024-04-06T14:16:59.411 INFO:tasks.workunit.client.0.smithi032.stdout:1/298: symlink d0/d10/d66/l68 0 2024-04-06T14:16:59.411 INFO:tasks.workunit.client.0.smithi032.stdout:4/387: creat d0/d82/f85 x:0 0 0 2024-04-06T14:16:59.412 INFO:tasks.workunit.client.0.smithi032.stdout:1/299: fdatasync d0/f17 0 2024-04-06T14:16:59.414 INFO:tasks.workunit.client.0.smithi032.stdout:8/270: creat d3/d8/f6d x:0 0 0 2024-04-06T14:16:59.416 INFO:tasks.workunit.client.1.smithi086.stdout:9/96: rename d5/f7 to d5/d18/f26 0 2024-04-06T14:16:59.418 INFO:tasks.workunit.client.0.smithi032.stdout:9/277: creat d2/d15/d55/d5f/f62 x:0 0 0 2024-04-06T14:16:59.418 INFO:tasks.workunit.client.0.smithi032.stdout:8/271: chown d3/d8/d3a/f4f 13 1 2024-04-06T14:16:59.418 INFO:tasks.workunit.client.0.smithi032.stdout:6/252: mknod d4/d5/d19/d21/c3f 0 2024-04-06T14:16:59.419 INFO:tasks.workunit.client.1.smithi086.stdout:4/31: creat d0/fc x:0 0 0 2024-04-06T14:16:59.420 INFO:tasks.workunit.client.0.smithi032.stdout:5/277: symlink d16/d18/d33/d35/l63 0 2024-04-06T14:16:59.422 INFO:tasks.workunit.client.1.smithi086.stdout:3/70: dwrite d5/fe [0,4194304] 0 2024-04-06T14:16:59.433 INFO:tasks.workunit.client.1.smithi086.stdout:4/32: fdatasync d0/f7 0 2024-04-06T14:16:59.433 INFO:tasks.workunit.client.1.smithi086.stdout:2/11: getdents d0 0 2024-04-06T14:16:59.434 INFO:tasks.workunit.client.1.smithi086.stdout:0/86: write d0/d1/d4/d7/f14 [615791,43921] 0 2024-04-06T14:16:59.436 INFO:tasks.workunit.client.0.smithi032.stdout:2/228: dwrite d0/f1b [0,4194304] 0 2024-04-06T14:16:59.437 INFO:tasks.workunit.client.0.smithi032.stdout:9/278: dwrite d2/d25/f32 [0,4194304] 0 2024-04-06T14:16:59.437 INFO:tasks.workunit.client.1.smithi086.stdout:8/131: dread d0/db/d10/f20 [0,4194304] 0 2024-04-06T14:16:59.438 INFO:tasks.workunit.client.0.smithi032.stdout:6/253: dwrite d4/d5/d8/d1f/f37 [0,4194304] 0 2024-04-06T14:16:59.438 INFO:tasks.workunit.client.0.smithi032.stdout:2/229: chown d0/d12/f25 3 1 2024-04-06T14:16:59.438 INFO:tasks.workunit.client.1.smithi086.stdout:6/17: dwrite f5 [0,4194304] 0 2024-04-06T14:16:59.438 INFO:tasks.workunit.client.0.smithi032.stdout:0/222: creat d3/da/d25/f54 x:0 0 0 2024-04-06T14:16:59.439 INFO:tasks.workunit.client.0.smithi032.stdout:6/254: stat d4/d5/d34 0 2024-04-06T14:16:59.440 INFO:tasks.workunit.client.1.smithi086.stdout:5/79: dwrite d0/fa [0,4194304] 0 2024-04-06T14:16:59.443 INFO:tasks.workunit.client.0.smithi032.stdout:8/272: dread d3/f32 [0,4194304] 0 2024-04-06T14:16:59.443 INFO:tasks.workunit.client.0.smithi032.stdout:4/388: fdatasync d0/df/d2b/d4f/f65 0 2024-04-06T14:16:59.445 INFO:tasks.workunit.client.1.smithi086.stdout:6/18: chown l3 63465 1 2024-04-06T14:16:59.446 INFO:tasks.workunit.client.1.smithi086.stdout:0/87: creat d0/d1/d2/d1a/f1d x:0 0 0 2024-04-06T14:16:59.446 INFO:tasks.workunit.client.0.smithi032.stdout:3/218: truncate d3/f24 3644798 0 2024-04-06T14:16:59.447 INFO:tasks.workunit.client.1.smithi086.stdout:5/80: mkdir d0/d1b 0 2024-04-06T14:16:59.449 INFO:tasks.workunit.client.1.smithi086.stdout:8/132: chown d0/db/d10/d15/f28 44436908 1 2024-04-06T14:16:59.449 INFO:tasks.workunit.client.1.smithi086.stdout:2/12: symlink d0/l2 0 2024-04-06T14:16:59.450 INFO:tasks.workunit.client.0.smithi032.stdout:9/279: creat d2/d15/d1e/d41/f63 x:0 0 0 2024-04-06T14:16:59.452 INFO:tasks.workunit.client.1.smithi086.stdout:7/13: dwrite f4 [0,4194304] 0 2024-04-06T14:16:59.452 INFO:tasks.workunit.client.1.smithi086.stdout:6/19: mkdir d6 0 2024-04-06T14:16:59.453 INFO:tasks.workunit.client.0.smithi032.stdout:0/223: creat d3/f55 x:0 0 0 2024-04-06T14:16:59.454 INFO:tasks.workunit.client.0.smithi032.stdout:8/273: dwrite d3/f29 [0,4194304] 0 2024-04-06T14:16:59.455 INFO:tasks.workunit.client.0.smithi032.stdout:6/255: creat d4/d5/d8/d2b/f40 x:0 0 0 2024-04-06T14:16:59.456 INFO:tasks.workunit.client.0.smithi032.stdout:3/219: dread d3/f5e [0,4194304] 0 2024-04-06T14:16:59.456 INFO:tasks.workunit.client.0.smithi032.stdout:1/300: creat d0/da/f69 x:0 0 0 2024-04-06T14:16:59.458 INFO:tasks.workunit.client.0.smithi032.stdout:4/389: creat d0/df/d2b/d43/d48/f86 x:0 0 0 2024-04-06T14:16:59.460 INFO:tasks.workunit.client.1.smithi086.stdout:5/81: mknod d0/c1c 0 2024-04-06T14:16:59.461 INFO:tasks.workunit.client.0.smithi032.stdout:0/224: creat d3/da/f56 x:0 0 0 2024-04-06T14:16:59.461 INFO:tasks.workunit.client.0.smithi032.stdout:6/256: write d4/d5/d8/d2b/f40 [379685,80362] 0 2024-04-06T14:16:59.461 INFO:tasks.workunit.client.1.smithi086.stdout:5/82: stat d0 0 2024-04-06T14:16:59.462 INFO:tasks.workunit.client.0.smithi032.stdout:8/274: symlink d3/d8/d7/l6e 0 2024-04-06T14:16:59.463 INFO:tasks.workunit.client.1.smithi086.stdout:6/20: unlink c4 0 2024-04-06T14:16:59.464 INFO:tasks.workunit.client.0.smithi032.stdout:5/278: dread fe [0,4194304] 0 2024-04-06T14:16:59.464 INFO:tasks.workunit.client.1.smithi086.stdout:7/14: creat f5 x:0 0 0 2024-04-06T14:16:59.465 INFO:tasks.workunit.client.0.smithi032.stdout:3/220: dread d3/f37 [0,4194304] 0 2024-04-06T14:16:59.466 INFO:tasks.workunit.client.1.smithi086.stdout:7/15: write f4 [5187690,14845] 0 2024-04-06T14:16:59.466 INFO:tasks.workunit.client.1.smithi086.stdout:0/88: read d0/d1/d4/f8 [494210,1543] 0 2024-04-06T14:16:59.466 INFO:tasks.workunit.client.1.smithi086.stdout:8/133: unlink d0/db/d10/c29 0 2024-04-06T14:16:59.467 INFO:tasks.workunit.client.0.smithi032.stdout:2/230: write d0/fb [3069959,100669] 0 2024-04-06T14:16:59.468 INFO:tasks.workunit.client.0.smithi032.stdout:8/275: fsync d3/d8/d7/d1a/f5a 0 2024-04-06T14:16:59.470 INFO:tasks.workunit.client.0.smithi032.stdout:5/279: stat d16/d58/d54/f60 0 2024-04-06T14:16:59.474 INFO:tasks.workunit.client.0.smithi032.stdout:4/390: dwrite d0/df/d23/f40 [0,4194304] 0 2024-04-06T14:16:59.475 INFO:tasks.workunit.client.0.smithi032.stdout:0/225: dread d3/f14 [0,4194304] 0 2024-04-06T14:16:59.481 INFO:tasks.workunit.client.0.smithi032.stdout:0/226: dwrite d3/da/d25/d36/f39 [0,4194304] 0 2024-04-06T14:16:59.483 INFO:tasks.workunit.client.0.smithi032.stdout:6/257: rename d4/d5/d8/fc to d4/d5/d28/f41 0 2024-04-06T14:16:59.485 INFO:tasks.workunit.client.0.smithi032.stdout:5/280: symlink d16/d58/l64 0 2024-04-06T14:16:59.488 INFO:tasks.workunit.client.0.smithi032.stdout:5/281: dread f10 [0,4194304] 0 2024-04-06T14:16:59.489 INFO:tasks.workunit.client.0.smithi032.stdout:0/227: readlink d3/l27 0 2024-04-06T14:16:59.489 INFO:tasks.workunit.client.0.smithi032.stdout:4/391: readlink d0/l7 0 2024-04-06T14:16:59.490 INFO:tasks.workunit.client.0.smithi032.stdout:3/221: fdatasync d3/f21 0 2024-04-06T14:16:59.491 INFO:tasks.workunit.client.0.smithi032.stdout:0/228: dread d3/f20 [0,4194304] 0 2024-04-06T14:16:59.491 INFO:tasks.workunit.client.1.smithi086.stdout:3/71: dwrite d5/df/f13 [4194304,4194304] 0 2024-04-06T14:16:59.491 INFO:tasks.workunit.client.1.smithi086.stdout:9/97: dwrite d5/d13/d1a/f24 [0,4194304] 0 2024-04-06T14:16:59.492 INFO:tasks.workunit.client.0.smithi032.stdout:1/301: rename d0/d9/d1f/f46 to d0/da/d3e/f6a 0 2024-04-06T14:16:59.493 INFO:tasks.workunit.client.0.smithi032.stdout:0/229: dread d3/f20 [0,4194304] 0 2024-04-06T14:16:59.494 INFO:tasks.workunit.client.1.smithi086.stdout:2/13: link d0/l2 d0/l3 0 2024-04-06T14:16:59.494 INFO:tasks.workunit.client.1.smithi086.stdout:2/14: fdatasync - no filename 2024-04-06T14:16:59.494 INFO:tasks.workunit.client.1.smithi086.stdout:2/15: write - no filename 2024-04-06T14:16:59.494 INFO:tasks.workunit.client.1.smithi086.stdout:5/83: mknod d0/d16/d18/c1d 0 2024-04-06T14:16:59.496 INFO:tasks.workunit.client.0.smithi032.stdout:6/258: mknod d4/d5/d35/c42 0 2024-04-06T14:16:59.496 INFO:tasks.workunit.client.0.smithi032.stdout:0/230: chown d3/da/d25/d2b/l3f 495 1 2024-04-06T14:16:59.497 INFO:tasks.workunit.client.0.smithi032.stdout:1/302: write d0/d10/d16/f51 [845261,37453] 0 2024-04-06T14:16:59.497 INFO:tasks.workunit.client.0.smithi032.stdout:0/231: chown d3/da/d41/f43 5599 1 2024-04-06T14:16:59.498 INFO:tasks.workunit.client.0.smithi032.stdout:3/222: symlink d3/d16/l60 0 2024-04-06T14:16:59.499 INFO:tasks.workunit.client.0.smithi032.stdout:6/259: write d4/d5/d8/d2b/f40 [296520,33000] 0 2024-04-06T14:16:59.502 INFO:tasks.workunit.client.1.smithi086.stdout:5/84: write d0/fa [4031166,5853] 0 2024-04-06T14:16:59.506 INFO:tasks.workunit.client.1.smithi086.stdout:4/33: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:16:59.507 INFO:tasks.workunit.client.1.smithi086.stdout:3/72: rename d5/cc to d5/c14 0 2024-04-06T14:16:59.507 INFO:tasks.workunit.client.1.smithi086.stdout:2/16: creat d0/f4 x:0 0 0 2024-04-06T14:16:59.507 INFO:tasks.workunit.client.1.smithi086.stdout:9/98: getdents d5/d10/d14/d19/d25 0 2024-04-06T14:16:59.507 INFO:tasks.workunit.client.1.smithi086.stdout:0/89: truncate d0/f17 2179989 0 2024-04-06T14:16:59.507 INFO:tasks.workunit.client.0.smithi032.stdout:3/223: unlink d3/d16/l1d 0 2024-04-06T14:16:59.507 INFO:tasks.workunit.client.0.smithi032.stdout:2/231: getdents d0/d30/d39 0 2024-04-06T14:16:59.508 INFO:tasks.workunit.client.1.smithi086.stdout:3/73: rmdir d5 39 2024-04-06T14:16:59.509 INFO:tasks.workunit.client.0.smithi032.stdout:2/232: write d0/f24 [3523790,113659] 0 2024-04-06T14:16:59.509 INFO:tasks.workunit.client.1.smithi086.stdout:8/134: dread d0/f2 [0,4194304] 0 2024-04-06T14:16:59.509 INFO:tasks.workunit.client.1.smithi086.stdout:2/17: truncate d0/f4 57463 0 2024-04-06T14:16:59.509 INFO:tasks.workunit.client.1.smithi086.stdout:9/99: read d5/f20 [1641780,81621] 0 2024-04-06T14:16:59.511 INFO:tasks.workunit.client.0.smithi032.stdout:8/276: rename d3/d8/fc to d3/d8/d12/f6f 0 2024-04-06T14:16:59.512 INFO:tasks.workunit.client.0.smithi032.stdout:5/282: rename d16/d18/d33/d35 to d16/d18/d33/d35/d4d/d65 22 2024-04-06T14:16:59.512 INFO:tasks.workunit.client.1.smithi086.stdout:0/90: symlink d0/d15/l1e 0 2024-04-06T14:16:59.512 INFO:tasks.workunit.client.0.smithi032.stdout:0/232: mkdir d3/da/d25/d57 0 2024-04-06T14:16:59.515 INFO:tasks.workunit.client.0.smithi032.stdout:1/303: creat d0/da/f6b x:0 0 0 2024-04-06T14:16:59.517 INFO:tasks.workunit.client.1.smithi086.stdout:4/34: dwrite d0/fc [0,4194304] 0 2024-04-06T14:16:59.517 INFO:tasks.workunit.client.0.smithi032.stdout:6/260: rename c1 to d4/d5/d34/c43 0 2024-04-06T14:16:59.518 INFO:tasks.workunit.client.0.smithi032.stdout:8/277: truncate d3/d8/f6d 203712 0 2024-04-06T14:16:59.520 INFO:tasks.workunit.client.0.smithi032.stdout:1/304: creat d0/da/d3e/f6c x:0 0 0 2024-04-06T14:16:59.521 INFO:tasks.workunit.client.0.smithi032.stdout:0/233: mknod d3/da/d25/c58 0 2024-04-06T14:16:59.522 INFO:tasks.workunit.client.1.smithi086.stdout:4/35: creat d0/fd x:0 0 0 2024-04-06T14:16:59.523 INFO:tasks.workunit.client.0.smithi032.stdout:6/261: symlink d4/d5/d19/d21/l44 0 2024-04-06T14:16:59.525 INFO:tasks.workunit.client.1.smithi086.stdout:4/36: mkdir d0/de 0 2024-04-06T14:16:59.526 INFO:tasks.workunit.client.0.smithi032.stdout:1/305: creat d0/d10/d16/f6d x:0 0 0 2024-04-06T14:16:59.526 INFO:tasks.workunit.client.1.smithi086.stdout:2/18: link d0/l2 d0/l5 0 2024-04-06T14:16:59.526 INFO:tasks.workunit.client.0.smithi032.stdout:5/283: dwrite d16/d18/d33/f34 [0,4194304] 0 2024-04-06T14:16:59.526 INFO:tasks.workunit.client.1.smithi086.stdout:4/37: chown d0/l9 886564677 1 2024-04-06T14:16:59.527 INFO:tasks.workunit.client.0.smithi032.stdout:5/284: dread - d16/d18/f3d zero size 2024-04-06T14:16:59.528 INFO:tasks.workunit.client.0.smithi032.stdout:8/278: mknod d3/d69/c70 0 2024-04-06T14:16:59.528 INFO:tasks.workunit.client.0.smithi032.stdout:1/306: write d0/d9/d1f/f47 [235920,111239] 0 2024-04-06T14:16:59.529 INFO:tasks.workunit.client.0.smithi032.stdout:6/262: mkdir d4/d5/d19/d1d/d45 0 2024-04-06T14:16:59.530 INFO:tasks.workunit.client.0.smithi032.stdout:3/224: link d3/d16/d1e/c44 d3/d16/d1e/c61 0 2024-04-06T14:16:59.533 INFO:tasks.workunit.client.1.smithi086.stdout:2/19: creat d0/f6 x:0 0 0 2024-04-06T14:16:59.533 INFO:tasks.workunit.client.1.smithi086.stdout:4/38: link d0/l9 d0/de/lf 0 2024-04-06T14:16:59.533 INFO:tasks.workunit.client.1.smithi086.stdout:2/20: dread - d0/f6 zero size 2024-04-06T14:16:59.535 INFO:tasks.workunit.client.0.smithi032.stdout:3/225: creat d3/d3b/f62 x:0 0 0 2024-04-06T14:16:59.539 INFO:tasks.workunit.client.1.smithi086.stdout:2/21: rename d0/l3 to d0/l7 0 2024-04-06T14:16:59.539 INFO:tasks.workunit.client.0.smithi032.stdout:7/281: sync 2024-04-06T14:16:59.539 INFO:tasks.workunit.client.0.smithi032.stdout:5/285: dread f5 [0,4194304] 0 2024-04-06T14:16:59.540 INFO:tasks.workunit.client.0.smithi032.stdout:6/263: dread d4/d5/d18/d23/f27 [0,4194304] 0 2024-04-06T14:16:59.540 INFO:tasks.workunit.client.1.smithi086.stdout:7/16: fsync f5 0 2024-04-06T14:16:59.542 INFO:tasks.workunit.client.0.smithi032.stdout:3/226: write d3/d16/f25 [1917366,116028] 0 2024-04-06T14:16:59.542 INFO:tasks.workunit.client.1.smithi086.stdout:2/22: dread d0/f4 [0,4194304] 0 2024-04-06T14:16:59.543 INFO:tasks.workunit.client.0.smithi032.stdout:1/307: dwrite d0/f37 [0,4194304] 0 2024-04-06T14:16:59.544 INFO:tasks.workunit.client.1.smithi086.stdout:7/17: dread f4 [0,4194304] 0 2024-04-06T14:16:59.546 INFO:tasks.workunit.client.0.smithi032.stdout:8/279: dwrite d3/d8/f6d [0,4194304] 0 2024-04-06T14:16:59.548 INFO:tasks.workunit.client.0.smithi032.stdout:7/282: write d16/d20/d26/f3a [3212928,116000] 0 2024-04-06T14:16:59.550 INFO:tasks.workunit.client.1.smithi086.stdout:7/18: dread f3 [0,4194304] 0 2024-04-06T14:16:59.553 INFO:tasks.workunit.client.0.smithi032.stdout:6/264: creat d4/d5/d19/f46 x:0 0 0 2024-04-06T14:16:59.554 INFO:tasks.workunit.client.0.smithi032.stdout:5/286: mknod d16/d18/d33/d45/d52/d39/c66 0 2024-04-06T14:16:59.564 INFO:tasks.workunit.client.0.smithi032.stdout:6/265: symlink d4/d5/d34/l47 0 2024-04-06T14:16:59.566 INFO:tasks.workunit.client.0.smithi032.stdout:3/227: dwrite d3/f43 [0,4194304] 0 2024-04-06T14:16:59.567 INFO:tasks.workunit.client.0.smithi032.stdout:5/287: symlink d16/d58/d54/l67 0 2024-04-06T14:16:59.570 INFO:tasks.workunit.client.0.smithi032.stdout:8/280: dwrite d3/d8/f25 [4194304,4194304] 0 2024-04-06T14:16:59.570 INFO:tasks.workunit.client.0.smithi032.stdout:1/308: dwrite d0/da/f65 [0,4194304] 0 2024-04-06T14:16:59.570 INFO:tasks.workunit.client.0.smithi032.stdout:4/392: truncate d0/df/d23/f3d 7693788 0 2024-04-06T14:16:59.571 INFO:tasks.workunit.client.0.smithi032.stdout:3/228: chown d3/d16/f2a 29353638 1 2024-04-06T14:16:59.571 INFO:tasks.workunit.client.0.smithi032.stdout:5/288: unlink d16/f62 0 2024-04-06T14:16:59.573 INFO:tasks.workunit.client.0.smithi032.stdout:3/229: chown d3/d16/c4f 611444 1 2024-04-06T14:16:59.573 INFO:tasks.workunit.client.0.smithi032.stdout:4/393: read - d0/df/d2b/d46/d58/f84 zero size 2024-04-06T14:16:59.573 INFO:tasks.workunit.client.0.smithi032.stdout:1/309: fsync d0/d41/d59/f60 0 2024-04-06T14:16:59.575 INFO:tasks.workunit.client.0.smithi032.stdout:3/230: readlink d3/d16/d1e/l5b 0 2024-04-06T14:16:59.579 INFO:tasks.workunit.client.0.smithi032.stdout:3/231: mkdir d3/d16/d63 0 2024-04-06T14:16:59.583 INFO:tasks.workunit.client.0.smithi032.stdout:5/289: dwrite d16/d18/d33/d45/d52/f53 [0,4194304] 0 2024-04-06T14:16:59.583 INFO:tasks.workunit.client.0.smithi032.stdout:1/310: mknod d0/d9/c6e 0 2024-04-06T14:16:59.583 INFO:tasks.workunit.client.0.smithi032.stdout:4/394: dread d0/f44 [0,4194304] 0 2024-04-06T14:16:59.584 INFO:tasks.workunit.client.0.smithi032.stdout:5/290: dread - d16/d18/f3d zero size 2024-04-06T14:16:59.586 INFO:tasks.workunit.client.0.smithi032.stdout:3/232: write d3/d16/f55 [521068,27775] 0 2024-04-06T14:16:59.587 INFO:tasks.workunit.client.0.smithi032.stdout:6/266: dread d4/d5/d35/f39 [0,4194304] 0 2024-04-06T14:16:59.588 INFO:tasks.workunit.client.0.smithi032.stdout:1/311: symlink d0/d10/d16/d3f/l6f 0 2024-04-06T14:16:59.589 INFO:tasks.workunit.client.0.smithi032.stdout:4/395: fsync d0/df/d23/f60 0 2024-04-06T14:16:59.589 INFO:tasks.workunit.client.0.smithi032.stdout:3/233: write d3/f3f [3768103,107467] 0 2024-04-06T14:16:59.590 INFO:tasks.workunit.client.0.smithi032.stdout:1/312: fsync d0/d9/d1f/f47 0 2024-04-06T14:16:59.593 INFO:tasks.workunit.client.0.smithi032.stdout:6/267: dread d4/d5/d28/f41 [0,4194304] 0 2024-04-06T14:16:59.596 INFO:tasks.workunit.client.0.smithi032.stdout:0/234: truncate d3/da/ff 2536277 0 2024-04-06T14:16:59.597 INFO:tasks.workunit.client.0.smithi032.stdout:4/396: creat d0/df/d23/d31/f87 x:0 0 0 2024-04-06T14:16:59.599 INFO:tasks.workunit.client.0.smithi032.stdout:1/313: rename d0/da/f2e to d0/d10/d66/f70 0 2024-04-06T14:16:59.602 INFO:tasks.workunit.client.0.smithi032.stdout:0/235: dwrite d3/f24 [0,4194304] 0 2024-04-06T14:16:59.603 INFO:tasks.workunit.client.0.smithi032.stdout:0/236: read d3/f10 [434948,18855] 0 2024-04-06T14:16:59.604 INFO:tasks.workunit.client.0.smithi032.stdout:7/283: write d16/d20/f23 [2534900,102119] 0 2024-04-06T14:16:59.606 INFO:tasks.workunit.client.0.smithi032.stdout:6/268: creat d4/d5/d19/d1d/d45/f48 x:0 0 0 2024-04-06T14:16:59.607 INFO:tasks.workunit.client.0.smithi032.stdout:1/314: mkdir d0/d41/d71 0 2024-04-06T14:16:59.608 INFO:tasks.workunit.client.0.smithi032.stdout:1/315: stat d0/d9/c55 0 2024-04-06T14:16:59.609 INFO:tasks.workunit.client.0.smithi032.stdout:5/291: getdents d16/d58 0 2024-04-06T14:16:59.613 INFO:tasks.workunit.client.0.smithi032.stdout:7/284: mknod d16/d20/d48/d61/c67 0 2024-04-06T14:16:59.613 INFO:tasks.workunit.client.0.smithi032.stdout:7/285: chown d16/d20/f47 4866080 1 2024-04-06T14:16:59.615 INFO:tasks.workunit.client.0.smithi032.stdout:6/269: rename d4/d5/d8/d2b/f40 to d4/d5/d18/f49 0 2024-04-06T14:16:59.618 INFO:tasks.workunit.client.0.smithi032.stdout:6/270: dread d4/f10 [0,4194304] 0 2024-04-06T14:16:59.620 INFO:tasks.workunit.client.0.smithi032.stdout:6/271: write d4/d5/d19/d1d/d45/f48 [648398,97431] 0 2024-04-06T14:16:59.621 INFO:tasks.workunit.client.0.smithi032.stdout:8/281: truncate d3/d8/d7/f47 2901915 0 2024-04-06T14:16:59.622 INFO:tasks.workunit.client.0.smithi032.stdout:7/286: stat d16/d20/l60 0 2024-04-06T14:16:59.623 INFO:tasks.workunit.client.0.smithi032.stdout:1/316: truncate d0/da/f52 344812 0 2024-04-06T14:16:59.625 INFO:tasks.workunit.client.0.smithi032.stdout:6/272: creat d4/d5/d19/d1d/d45/f4a x:0 0 0 2024-04-06T14:16:59.628 INFO:tasks.workunit.client.0.smithi032.stdout:7/287: unlink d16/d20/d26/d2d/l62 0 2024-04-06T14:16:59.629 INFO:tasks.workunit.client.0.smithi032.stdout:0/237: getdents d3/da/d35/d4b 0 2024-04-06T14:16:59.632 INFO:tasks.workunit.client.0.smithi032.stdout:7/288: write d16/f4b [415657,80849] 0 2024-04-06T14:16:59.632 INFO:tasks.workunit.client.0.smithi032.stdout:1/317: dwrite d0/d10/d16/f6d [0,4194304] 0 2024-04-06T14:16:59.634 INFO:tasks.workunit.client.0.smithi032.stdout:0/238: symlink d3/da/d35/l59 0 2024-04-06T14:16:59.635 INFO:tasks.workunit.client.0.smithi032.stdout:3/234: truncate d3/f19 3277914 0 2024-04-06T14:16:59.638 INFO:tasks.workunit.client.0.smithi032.stdout:1/318: write d0/da/f38 [819629,104190] 0 2024-04-06T14:16:59.642 INFO:tasks.workunit.client.0.smithi032.stdout:7/289: dread - d16/d20/d26/d2d/d30/f53 zero size 2024-04-06T14:16:59.642 INFO:tasks.workunit.client.0.smithi032.stdout:3/235: write d3/f20 [766258,127609] 0 2024-04-06T14:16:59.644 INFO:tasks.workunit.client.0.smithi032.stdout:6/273: dwrite d4/d5/d19/d1d/d45/f48 [0,4194304] 0 2024-04-06T14:16:59.648 INFO:tasks.workunit.client.0.smithi032.stdout:7/290: chown fb 18670 1 2024-04-06T14:16:59.650 INFO:tasks.workunit.client.0.smithi032.stdout:8/282: dwrite d3/d8/d12/f4d [0,4194304] 0 2024-04-06T14:16:59.651 INFO:tasks.workunit.client.0.smithi032.stdout:3/236: symlink d3/l64 0 2024-04-06T14:16:59.654 INFO:tasks.workunit.client.0.smithi032.stdout:7/291: fsync d16/f27 0 2024-04-06T14:16:59.656 INFO:tasks.workunit.client.0.smithi032.stdout:4/397: dwrite d0/df/d23/f3c [0,4194304] 0 2024-04-06T14:16:59.657 INFO:tasks.workunit.client.0.smithi032.stdout:6/274: write d4/d5/fb [4773940,94748] 0 2024-04-06T14:16:59.659 INFO:tasks.workunit.client.0.smithi032.stdout:5/292: dwrite f10 [0,4194304] 0 2024-04-06T14:16:59.660 INFO:tasks.workunit.client.0.smithi032.stdout:1/319: dread d0/d10/d16/f3c [0,4194304] 0 2024-04-06T14:16:59.664 INFO:tasks.workunit.client.0.smithi032.stdout:7/292: chown d16/d20/d26/c29 0 1 2024-04-06T14:16:59.666 INFO:tasks.workunit.client.0.smithi032.stdout:7/293: write d16/d20/d26/f3a [2013133,75605] 0 2024-04-06T14:16:59.671 INFO:tasks.workunit.client.0.smithi032.stdout:3/237: rename d3/d16/d1e to d3/d65 0 2024-04-06T14:16:59.672 INFO:tasks.workunit.client.0.smithi032.stdout:5/293: dwrite d16/d18/f2b [0,4194304] 0 2024-04-06T14:16:59.673 INFO:tasks.workunit.client.0.smithi032.stdout:7/294: symlink d16/d20/d26/d2d/l68 0 2024-04-06T14:16:59.674 INFO:tasks.workunit.client.0.smithi032.stdout:8/283: getdents d3/d8/d12/d2f/d4a 0 2024-04-06T14:16:59.677 INFO:tasks.workunit.client.0.smithi032.stdout:6/275: rename d4/d5/d19/d1d/d45 to d4/d5/d8/d4b 0 2024-04-06T14:16:59.678 INFO:tasks.workunit.client.0.smithi032.stdout:3/238: creat d3/d65/f66 x:0 0 0 2024-04-06T14:16:59.679 INFO:tasks.workunit.client.0.smithi032.stdout:3/239: write d3/d3b/f40 [2837081,52052] 0 2024-04-06T14:16:59.680 INFO:tasks.workunit.client.0.smithi032.stdout:3/240: chown d3/d3b/d3c/f5f 16063 1 2024-04-06T14:16:59.681 INFO:tasks.workunit.client.0.smithi032.stdout:5/294: unlink d16/f17 0 2024-04-06T14:16:59.684 INFO:tasks.workunit.client.0.smithi032.stdout:7/295: creat d16/d20/d26/d2d/f69 x:0 0 0 2024-04-06T14:16:59.686 INFO:tasks.workunit.client.0.smithi032.stdout:7/296: write d16/f4b [1728912,6400] 0 2024-04-06T14:16:59.686 INFO:tasks.workunit.client.0.smithi032.stdout:1/320: truncate d0/d10/f25 1866048 0 2024-04-06T14:16:59.687 INFO:tasks.workunit.client.0.smithi032.stdout:4/398: link d0/df/d2b/c33 d0/df/d23/c88 0 2024-04-06T14:16:59.690 INFO:tasks.workunit.client.0.smithi032.stdout:3/241: rename d3/l11 to d3/d3b/l67 0 2024-04-06T14:16:59.692 INFO:tasks.workunit.client.0.smithi032.stdout:6/276: dwrite d4/d5/d8/d4b/f48 [0,4194304] 0 2024-04-06T14:16:59.695 INFO:tasks.workunit.client.0.smithi032.stdout:7/297: dread d16/d20/d26/f31 [0,4194304] 0 2024-04-06T14:16:59.698 INFO:tasks.workunit.client.0.smithi032.stdout:6/277: dread d4/d5/d35/f39 [0,4194304] 0 2024-04-06T14:16:59.700 INFO:tasks.workunit.client.0.smithi032.stdout:4/399: rename d0/df/d2b/d43/d48 to d0/df/d23/d31/d50/d5e/d89 0 2024-04-06T14:16:59.700 INFO:tasks.workunit.client.0.smithi032.stdout:6/278: readlink d4/d5/d8/ld 0 2024-04-06T14:16:59.701 INFO:tasks.workunit.client.0.smithi032.stdout:8/284: dwrite d3/d8/d12/f1b [0,4194304] 0 2024-04-06T14:16:59.701 INFO:tasks.workunit.client.0.smithi032.stdout:3/242: creat d3/d16/f68 x:0 0 0 2024-04-06T14:16:59.705 INFO:tasks.workunit.client.0.smithi032.stdout:5/295: creat d16/d18/f68 x:0 0 0 2024-04-06T14:16:59.711 INFO:tasks.workunit.client.0.smithi032.stdout:3/243: creat d3/d65/f69 x:0 0 0 2024-04-06T14:16:59.711 INFO:tasks.workunit.client.0.smithi032.stdout:5/296: fsync d16/d18/d33/d45/d52/d39/f5b 0 2024-04-06T14:16:59.711 INFO:tasks.workunit.client.0.smithi032.stdout:1/321: rmdir d0/da/d58 0 2024-04-06T14:16:59.711 INFO:tasks.workunit.client.0.smithi032.stdout:6/279: fdatasync d4/d5/d8/d4b/f48 0 2024-04-06T14:16:59.711 INFO:tasks.workunit.client.0.smithi032.stdout:3/244: read - d3/d65/f52 zero size 2024-04-06T14:16:59.711 INFO:tasks.workunit.client.0.smithi032.stdout:5/297: dread - f9 zero size 2024-04-06T14:16:59.711 INFO:tasks.workunit.client.0.smithi032.stdout:4/400: rename d0/df/d2b/d4f to d0/df/d23/d31/d50/d57/d8a 0 2024-04-06T14:16:59.711 INFO:tasks.workunit.client.0.smithi032.stdout:8/285: dwrite d3/d8/d7/f16 [0,4194304] 0 2024-04-06T14:16:59.712 INFO:tasks.workunit.client.0.smithi032.stdout:1/322: chown d0/d10/d16/d3f/c53 497 1 2024-04-06T14:16:59.712 INFO:tasks.workunit.client.0.smithi032.stdout:8/286: readlink d3/d8/l33 0 2024-04-06T14:16:59.716 INFO:tasks.workunit.client.0.smithi032.stdout:2/233: sync 2024-04-06T14:16:59.716 INFO:tasks.workunit.client.0.smithi032.stdout:9/280: sync 2024-04-06T14:16:59.719 INFO:tasks.workunit.client.0.smithi032.stdout:1/323: mkdir d0/d9/d1f/d72 0 2024-04-06T14:16:59.720 INFO:tasks.workunit.client.0.smithi032.stdout:1/324: chown d0/d41 0 1 2024-04-06T14:16:59.725 INFO:tasks.workunit.client.0.smithi032.stdout:1/325: chown d0/da/d3e/c5e 5 1 2024-04-06T14:16:59.726 INFO:tasks.workunit.client.0.smithi032.stdout:3/245: dwrite d3/d3b/f62 [0,4194304] 0 2024-04-06T14:16:59.727 INFO:tasks.workunit.client.0.smithi032.stdout:7/298: truncate d16/d20/f47 851857 0 2024-04-06T14:16:59.728 INFO:tasks.workunit.client.0.smithi032.stdout:6/280: creat d4/d5/f4c x:0 0 0 2024-04-06T14:16:59.729 INFO:tasks.workunit.client.0.smithi032.stdout:4/401: dwrite d0/df/d23/d31/f37 [0,4194304] 0 2024-04-06T14:16:59.731 INFO:tasks.workunit.client.0.smithi032.stdout:2/234: creat d0/f51 x:0 0 0 2024-04-06T14:16:59.732 INFO:tasks.workunit.client.0.smithi032.stdout:7/299: stat d16/d20/d63 0 2024-04-06T14:16:59.734 INFO:tasks.workunit.client.0.smithi032.stdout:1/326: write d0/d10/d16/f6d [4938943,35533] 0 2024-04-06T14:16:59.734 INFO:tasks.workunit.client.0.smithi032.stdout:9/281: creat d2/d5/f64 x:0 0 0 2024-04-06T14:16:59.735 INFO:tasks.workunit.client.0.smithi032.stdout:2/235: fsync d0/d12/f1c 0 2024-04-06T14:16:59.737 INFO:tasks.workunit.client.0.smithi032.stdout:4/402: write d0/df/d76/f61 [114098,124395] 0 2024-04-06T14:16:59.737 INFO:tasks.workunit.client.0.smithi032.stdout:5/298: getdents d16/d18/d33/d45/d52/d39 0 2024-04-06T14:16:59.738 INFO:tasks.workunit.client.0.smithi032.stdout:8/287: dwrite d3/d8/d3a/d4b/f5e [0,4194304] 0 2024-04-06T14:16:59.741 INFO:tasks.workunit.client.0.smithi032.stdout:1/327: write d0/f17 [3507287,55968] 0 2024-04-06T14:16:59.742 INFO:tasks.workunit.client.0.smithi032.stdout:6/281: creat d4/d5/d14/f4d x:0 0 0 2024-04-06T14:16:59.742 INFO:tasks.workunit.client.0.smithi032.stdout:3/246: creat d3/f6a x:0 0 0 2024-04-06T14:16:59.744 INFO:tasks.workunit.client.0.smithi032.stdout:6/282: stat d4/d5/d18 0 2024-04-06T14:16:59.745 INFO:tasks.workunit.client.1.smithi086.stdout:6/21: dwrite f1 [0,4194304] 0 2024-04-06T14:16:59.746 INFO:tasks.workunit.client.0.smithi032.stdout:8/288: write d3/d8/d7/d1a/f54 [209982,65207] 0 2024-04-06T14:16:59.746 INFO:tasks.workunit.client.0.smithi032.stdout:6/283: chown d4/d5/c12 51414 1 2024-04-06T14:16:59.749 INFO:tasks.workunit.client.0.smithi032.stdout:9/282: mkdir d2/d15/d1e/d65 0 2024-04-06T14:16:59.750 INFO:tasks.workunit.client.0.smithi032.stdout:7/300: rename d16/d20/d26/d2d/d30/l42 to d16/d20/d48/l6a 0 2024-04-06T14:16:59.751 INFO:tasks.workunit.client.0.smithi032.stdout:8/289: truncate d3/d8/d12/d2f/f39 539611 0 2024-04-06T14:16:59.751 INFO:tasks.workunit.client.0.smithi032.stdout:5/299: symlink d16/d58/d54/l69 0 2024-04-06T14:16:59.752 INFO:tasks.workunit.client.0.smithi032.stdout:1/328: rename d0/d10/d66 to d0/d10/d66/d73 22 2024-04-06T14:16:59.752 INFO:tasks.workunit.client.1.smithi086.stdout:3/74: dwrite f4 [0,4194304] 0 2024-04-06T14:16:59.755 INFO:tasks.workunit.client.0.smithi032.stdout:2/236: mkdir d0/d4e/d52 0 2024-04-06T14:16:59.756 INFO:tasks.workunit.client.0.smithi032.stdout:3/247: symlink d3/l6b 0 2024-04-06T14:16:59.756 INFO:tasks.workunit.client.0.smithi032.stdout:8/290: fsync d3/d8/d3a/f4c 0 2024-04-06T14:16:59.757 INFO:tasks.workunit.client.0.smithi032.stdout:1/329: write d0/f4c [549035,39660] 0 2024-04-06T14:16:59.757 INFO:tasks.workunit.client.0.smithi032.stdout:7/301: write d16/d20/d26/d2d/d30/f46 [4721723,53292] 0 2024-04-06T14:16:59.759 INFO:tasks.workunit.client.0.smithi032.stdout:8/291: readlink d3/d8/d12/d2f/l57 0 2024-04-06T14:16:59.760 INFO:tasks.workunit.client.0.smithi032.stdout:7/302: read fd [2095266,103106] 0 2024-04-06T14:16:59.761 INFO:tasks.workunit.client.0.smithi032.stdout:4/403: creat d0/df/f8b x:0 0 0 2024-04-06T14:16:59.761 INFO:tasks.workunit.client.1.smithi086.stdout:7/19: getdents . 0 2024-04-06T14:16:59.762 INFO:tasks.workunit.client.0.smithi032.stdout:7/303: chown d16/d20/l60 77147327 1 2024-04-06T14:16:59.762 INFO:tasks.workunit.client.0.smithi032.stdout:4/404: read d0/df/d1f/d2e/f49 [164544,96980] 0 2024-04-06T14:16:59.764 INFO:tasks.workunit.client.0.smithi032.stdout:5/300: creat d16/d18/d33/d35/f6a x:0 0 0 2024-04-06T14:16:59.765 INFO:tasks.workunit.client.1.smithi086.stdout:5/85: dwrite d0/f4 [0,4194304] 0 2024-04-06T14:16:59.765 INFO:tasks.workunit.client.0.smithi032.stdout:2/237: creat d0/d4e/f53 x:0 0 0 2024-04-06T14:16:59.766 INFO:tasks.workunit.client.1.smithi086.stdout:0/91: dwrite d0/d1/fb [0,4194304] 0 2024-04-06T14:16:59.766 INFO:tasks.workunit.client.0.smithi032.stdout:3/248: creat d3/d3b/f6c x:0 0 0 2024-04-06T14:16:59.766 INFO:tasks.workunit.client.1.smithi086.stdout:9/100: dwrite d5/d10/d14/f22 [0,4194304] 0 2024-04-06T14:16:59.767 INFO:tasks.workunit.client.0.smithi032.stdout:2/238: chown d0/f28 492 1 2024-04-06T14:16:59.769 INFO:tasks.workunit.client.1.smithi086.stdout:8/135: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:16:59.769 INFO:tasks.workunit.client.0.smithi032.stdout:2/239: read d0/d12/d3a/f42 [1161809,36282] 0 2024-04-06T14:16:59.770 INFO:tasks.workunit.client.0.smithi032.stdout:8/292: rmdir d3 39 2024-04-06T14:16:59.771 INFO:tasks.workunit.client.0.smithi032.stdout:2/240: chown d0/d4f 27209675 1 2024-04-06T14:16:59.772 INFO:tasks.workunit.client.0.smithi032.stdout:4/405: dwrite d0/df/d23/f4a [0,4194304] 0 2024-04-06T14:16:59.774 INFO:tasks.workunit.client.1.smithi086.stdout:3/75: chown d5/f9 8 1 2024-04-06T14:16:59.775 INFO:tasks.workunit.client.1.smithi086.stdout:2/23: link d0/l7 d0/l8 0 2024-04-06T14:16:59.775 INFO:tasks.workunit.client.0.smithi032.stdout:7/304: symlink d16/d20/l6b 0 2024-04-06T14:16:59.776 INFO:tasks.workunit.client.1.smithi086.stdout:8/136: fdatasync d0/f2 0 2024-04-06T14:16:59.777 INFO:tasks.workunit.client.1.smithi086.stdout:9/101: rmdir d5/d10/d14/d19/d1b 39 2024-04-06T14:16:59.778 INFO:tasks.workunit.client.1.smithi086.stdout:6/22: creat d6/f7 x:0 0 0 2024-04-06T14:16:59.780 INFO:tasks.workunit.client.1.smithi086.stdout:8/137: write d0/f3 [5033986,127794] 0 2024-04-06T14:16:59.780 INFO:tasks.workunit.client.1.smithi086.stdout:7/20: dread f3 [0,4194304] 0 2024-04-06T14:16:59.781 INFO:tasks.workunit.client.0.smithi032.stdout:7/305: chown d16/d20/d26/d2d/d30/l3d 13862767 1 2024-04-06T14:16:59.781 INFO:tasks.workunit.client.1.smithi086.stdout:1/11: sync 2024-04-06T14:16:59.782 INFO:tasks.workunit.client.0.smithi032.stdout:4/406: fsync d0/df/d23/d31/d50/d57/f6a 0 2024-04-06T14:16:59.783 INFO:tasks.workunit.client.1.smithi086.stdout:5/86: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:59.783 INFO:tasks.workunit.client.1.smithi086.stdout:0/92: truncate d0/d1/d4/f12 3336770 0 2024-04-06T14:16:59.784 INFO:tasks.workunit.client.0.smithi032.stdout:0/239: sync 2024-04-06T14:16:59.784 INFO:tasks.workunit.client.1.smithi086.stdout:6/23: symlink d6/l8 0 2024-04-06T14:16:59.786 INFO:tasks.workunit.client.0.smithi032.stdout:1/330: dwrite d0/d10/f3d [8388608,4194304] 0 2024-04-06T14:16:59.786 INFO:tasks.workunit.client.1.smithi086.stdout:9/102: dread - d5/d13/f15 zero size 2024-04-06T14:16:59.786 INFO:tasks.workunit.client.0.smithi032.stdout:9/283: link d2/d15/d1e/d47/d4a/l4e d2/d15/d1e/d47/d4a/l66 0 2024-04-06T14:16:59.787 INFO:tasks.workunit.client.1.smithi086.stdout:2/24: rename d0/l2 to d0/l9 0 2024-04-06T14:16:59.787 INFO:tasks.workunit.client.0.smithi032.stdout:2/241: mknod d0/d30/c54 0 2024-04-06T14:16:59.788 INFO:tasks.workunit.client.0.smithi032.stdout:0/240: readlink d3/da/l34 0 2024-04-06T14:16:59.788 INFO:tasks.workunit.client.1.smithi086.stdout:3/76: dread d5/fe [0,4194304] 0 2024-04-06T14:16:59.789 INFO:tasks.workunit.client.0.smithi032.stdout:8/293: fsync d3/f29 0 2024-04-06T14:16:59.789 INFO:tasks.workunit.client.1.smithi086.stdout:5/87: unlink d0/f13 0 2024-04-06T14:16:59.790 INFO:tasks.workunit.client.1.smithi086.stdout:2/25: chown d0/f6 59485886 1 2024-04-06T14:16:59.790 INFO:tasks.workunit.client.1.smithi086.stdout:0/93: symlink d0/d1/d2/d1a/l1f 0 2024-04-06T14:16:59.790 INFO:tasks.workunit.client.0.smithi032.stdout:3/249: mknod d3/d16/d63/c6d 0 2024-04-06T14:16:59.791 INFO:tasks.workunit.client.0.smithi032.stdout:9/284: fdatasync d2/d25/d31/f3a 0 2024-04-06T14:16:59.791 INFO:tasks.workunit.client.0.smithi032.stdout:7/306: unlink d16/d20/d26/d2d/l51 0 2024-04-06T14:16:59.791 INFO:tasks.workunit.client.1.smithi086.stdout:0/94: stat d0/f16 0 2024-04-06T14:16:59.792 INFO:tasks.workunit.client.0.smithi032.stdout:2/242: dread - d0/d30/f4d zero size 2024-04-06T14:16:59.792 INFO:tasks.workunit.client.0.smithi032.stdout:0/241: readlink d3/da/d35/l59 0 2024-04-06T14:16:59.793 INFO:tasks.workunit.client.0.smithi032.stdout:5/301: creat d16/f6b x:0 0 0 2024-04-06T14:16:59.794 INFO:tasks.workunit.client.0.smithi032.stdout:4/407: creat d0/df/d1f/d42/d45/f8c x:0 0 0 2024-04-06T14:16:59.795 INFO:tasks.workunit.client.0.smithi032.stdout:1/331: dread d0/f1e [0,4194304] 0 2024-04-06T14:16:59.796 INFO:tasks.workunit.client.1.smithi086.stdout:3/77: rmdir d5 39 2024-04-06T14:16:59.797 INFO:tasks.workunit.client.1.smithi086.stdout:7/21: dread f4 [0,4194304] 0 2024-04-06T14:16:59.797 INFO:tasks.workunit.client.0.smithi032.stdout:5/302: readlink d16/d18/d33/d35/l4b 0 2024-04-06T14:16:59.798 INFO:tasks.workunit.client.1.smithi086.stdout:8/138: creat d0/f34 x:0 0 0 2024-04-06T14:16:59.799 INFO:tasks.workunit.client.0.smithi032.stdout:3/250: mknod d3/d3b/d3c/c6e 0 2024-04-06T14:16:59.799 INFO:tasks.workunit.client.1.smithi086.stdout:9/103: truncate d5/fc 4761149 0 2024-04-06T14:16:59.800 INFO:tasks.workunit.client.0.smithi032.stdout:1/332: dread - d0/d41/d59/f60 zero size 2024-04-06T14:16:59.801 INFO:tasks.workunit.client.0.smithi032.stdout:7/307: symlink d16/d20/d26/d2d/d30/d3e/l6c 0 2024-04-06T14:16:59.801 INFO:tasks.workunit.client.1.smithi086.stdout:0/95: mknod d0/d1/d4/d7/c20 0 2024-04-06T14:16:59.801 INFO:tasks.workunit.client.1.smithi086.stdout:3/78: stat d5/f9 0 2024-04-06T14:16:59.802 INFO:tasks.workunit.client.1.smithi086.stdout:1/12: dread f1 [0,4194304] 0 2024-04-06T14:16:59.803 INFO:tasks.workunit.client.0.smithi032.stdout:2/243: mknod d0/d4e/c55 0 2024-04-06T14:16:59.803 INFO:tasks.workunit.client.0.smithi032.stdout:7/308: dread - d16/d20/d26/f39 zero size 2024-04-06T14:16:59.805 INFO:tasks.workunit.client.1.smithi086.stdout:7/22: unlink f3 0 2024-04-06T14:16:59.805 INFO:tasks.workunit.client.0.smithi032.stdout:4/408: unlink d0/df/d23/d31/f37 0 2024-04-06T14:16:59.805 INFO:tasks.workunit.client.1.smithi086.stdout:9/104: creat d5/f27 x:0 0 0 2024-04-06T14:16:59.806 INFO:tasks.workunit.client.1.smithi086.stdout:6/24: rmdir d6 39 2024-04-06T14:16:59.808 INFO:tasks.workunit.client.0.smithi032.stdout:3/251: creat d3/d16/f6f x:0 0 0 2024-04-06T14:16:59.809 INFO:tasks.workunit.client.1.smithi086.stdout:2/26: chown d0/l9 390 1 2024-04-06T14:16:59.809 INFO:tasks.workunit.client.0.smithi032.stdout:2/244: symlink d0/d12/l56 0 2024-04-06T14:16:59.811 INFO:tasks.workunit.client.0.smithi032.stdout:8/294: getdents d3/d8/d7/d1a 0 2024-04-06T14:16:59.811 INFO:tasks.workunit.client.1.smithi086.stdout:3/79: fsync d5/f9 0 2024-04-06T14:16:59.813 INFO:tasks.workunit.client.1.smithi086.stdout:1/13: mknod c3 0 2024-04-06T14:16:59.813 INFO:tasks.workunit.client.0.smithi032.stdout:5/303: symlink d16/l6c 0 2024-04-06T14:16:59.814 INFO:tasks.workunit.client.0.smithi032.stdout:0/242: dread d3/f4 [0,4194304] 0 2024-04-06T14:16:59.814 INFO:tasks.workunit.client.0.smithi032.stdout:1/333: symlink d0/d10/l74 0 2024-04-06T14:16:59.815 INFO:tasks.workunit.client.0.smithi032.stdout:6/284: dwrite d4/d5/d18/d23/f27 [0,4194304] 0 2024-04-06T14:16:59.815 INFO:tasks.workunit.client.0.smithi032.stdout:2/245: write d0/d12/f17 [147855,20813] 0 2024-04-06T14:16:59.816 INFO:tasks.workunit.client.0.smithi032.stdout:8/295: stat d3/d8/d3a/f4c 0 2024-04-06T14:16:59.816 INFO:tasks.workunit.client.1.smithi086.stdout:7/23: creat f6 x:0 0 0 2024-04-06T14:16:59.817 INFO:tasks.workunit.client.0.smithi032.stdout:3/252: creat d3/d16/f70 x:0 0 0 2024-04-06T14:16:59.817 INFO:tasks.workunit.client.0.smithi032.stdout:5/304: fsync d16/d18/d33/d35/f6a 0 2024-04-06T14:16:59.820 INFO:tasks.workunit.client.0.smithi032.stdout:1/334: dread - d0/da/d3e/f6c zero size 2024-04-06T14:16:59.822 INFO:tasks.workunit.client.1.smithi086.stdout:0/96: mknod d0/d1/d4/c21 0 2024-04-06T14:16:59.823 INFO:tasks.workunit.client.1.smithi086.stdout:2/27: symlink d0/la 0 2024-04-06T14:16:59.824 INFO:tasks.workunit.client.1.smithi086.stdout:5/88: dwrite d0/f1 [0,4194304] 0 2024-04-06T14:16:59.825 INFO:tasks.workunit.client.0.smithi032.stdout:4/409: link d0/df/d23/d31/d50/d57/f6a d0/df/d23/d31/f8d 0 2024-04-06T14:16:59.825 INFO:tasks.workunit.client.1.smithi086.stdout:2/28: chown d0/f4 1 1 2024-04-06T14:16:59.825 INFO:tasks.workunit.client.1.smithi086.stdout:8/139: write d0/d13/f21 [1010513,18281] 0 2024-04-06T14:16:59.826 INFO:tasks.workunit.client.1.smithi086.stdout:3/80: symlink d5/l15 0 2024-04-06T14:16:59.826 INFO:tasks.workunit.client.0.smithi032.stdout:6/285: mkdir d4/d5/d19/d1d/d4e 0 2024-04-06T14:16:59.827 INFO:tasks.workunit.client.1.smithi086.stdout:1/14: write f2 [1981467,5061] 0 2024-04-06T14:16:59.828 INFO:tasks.workunit.client.0.smithi032.stdout:8/296: unlink d3/d8/d7/f26 0 2024-04-06T14:16:59.829 INFO:tasks.workunit.client.0.smithi032.stdout:9/285: write d2/f21 [4219485,22737] 0 2024-04-06T14:16:59.829 INFO:tasks.workunit.client.0.smithi032.stdout:5/305: dwrite d16/f6b [0,4194304] 0 2024-04-06T14:16:59.831 INFO:tasks.workunit.client.1.smithi086.stdout:7/24: mkdir d7 0 2024-04-06T14:16:59.832 INFO:tasks.workunit.client.1.smithi086.stdout:7/25: rename d7 to d7/d8 22 2024-04-06T14:16:59.832 INFO:tasks.workunit.client.1.smithi086.stdout:4/39: sync 2024-04-06T14:16:59.832 INFO:tasks.workunit.client.0.smithi032.stdout:5/306: dread - d16/d18/f23 zero size 2024-04-06T14:16:59.832 INFO:tasks.workunit.client.0.smithi032.stdout:2/246: rmdir d0/d12/d3a 39 2024-04-06T14:16:59.833 INFO:tasks.workunit.client.0.smithi032.stdout:3/253: symlink d3/d16/d63/l71 0 2024-04-06T14:16:59.833 INFO:tasks.workunit.client.1.smithi086.stdout:1/15: dread f1 [0,4194304] 0 2024-04-06T14:16:59.834 INFO:tasks.workunit.client.1.smithi086.stdout:9/105: write d5/d13/f15 [861217,129880] 0 2024-04-06T14:16:59.834 INFO:tasks.workunit.client.0.smithi032.stdout:2/247: chown d0/c32 0 1 2024-04-06T14:16:59.835 INFO:tasks.workunit.client.1.smithi086.stdout:6/25: mkdir d6/d9 0 2024-04-06T14:16:59.835 INFO:tasks.workunit.client.0.smithi032.stdout:0/243: link d3/da/f1f d3/f5a 0 2024-04-06T14:16:59.836 INFO:tasks.workunit.client.1.smithi086.stdout:6/26: chown f1 27594364 1 2024-04-06T14:16:59.838 INFO:tasks.workunit.client.0.smithi032.stdout:0/244: chown d3/da/l2c 37 1 2024-04-06T14:16:59.839 INFO:tasks.workunit.client.1.smithi086.stdout:3/81: symlink d5/df/l16 0 2024-04-06T14:16:59.840 INFO:tasks.workunit.client.0.smithi032.stdout:4/410: mkdir d0/d82/d8e 0 2024-04-06T14:16:59.840 INFO:tasks.workunit.client.1.smithi086.stdout:3/82: fsync d5/df/f13 0 2024-04-06T14:16:59.840 INFO:tasks.workunit.client.1.smithi086.stdout:8/140: truncate d0/db/d10/d15/f28 1023527 0 2024-04-06T14:16:59.841 INFO:tasks.workunit.client.1.smithi086.stdout:3/83: write f4 [684093,70507] 0 2024-04-06T14:16:59.842 INFO:tasks.workunit.client.1.smithi086.stdout:8/141: chown d0/db/d32 63545026 1 2024-04-06T14:16:59.843 INFO:tasks.workunit.client.1.smithi086.stdout:7/26: unlink f4 0 2024-04-06T14:16:59.844 INFO:tasks.workunit.client.0.smithi032.stdout:6/286: creat d4/d5/d28/f4f x:0 0 0 2024-04-06T14:16:59.844 INFO:tasks.workunit.client.0.smithi032.stdout:8/297: rmdir d3/d8/d12/d2f 39 2024-04-06T14:16:59.850 INFO:tasks.workunit.client.1.smithi086.stdout:1/16: creat f4 x:0 0 0 2024-04-06T14:16:59.850 INFO:tasks.workunit.client.1.smithi086.stdout:5/89: rmdir d0 39 2024-04-06T14:16:59.851 INFO:tasks.workunit.client.0.smithi032.stdout:0/245: mknod d3/da/d41/c5b 0 2024-04-06T14:16:59.851 INFO:tasks.workunit.client.0.smithi032.stdout:5/307: write d16/d18/f20 [470718,99816] 0 2024-04-06T14:16:59.852 INFO:tasks.workunit.client.1.smithi086.stdout:7/27: dwrite f5 [0,4194304] 0 2024-04-06T14:16:59.853 INFO:tasks.workunit.client.0.smithi032.stdout:9/286: creat d2/d15/d55/d59/f67 x:0 0 0 2024-04-06T14:16:59.853 INFO:tasks.workunit.client.0.smithi032.stdout:3/254: dwrite d3/d65/f47 [0,4194304] 0 2024-04-06T14:16:59.855 INFO:tasks.workunit.client.1.smithi086.stdout:9/106: dread d5/d13/f15 [0,4194304] 0 2024-04-06T14:16:59.856 INFO:tasks.workunit.client.0.smithi032.stdout:4/411: creat d0/df/d1f/d5a/f8f x:0 0 0 2024-04-06T14:16:59.857 INFO:tasks.workunit.client.0.smithi032.stdout:8/298: readlink d3/d8/d12/d2f/l57 0 2024-04-06T14:16:59.859 INFO:tasks.workunit.client.0.smithi032.stdout:6/287: creat d4/d5/d18/d23/f50 x:0 0 0 2024-04-06T14:16:59.859 INFO:tasks.workunit.client.0.smithi032.stdout:0/246: chown d3/da/d25/d36/f51 0 1 2024-04-06T14:16:59.860 INFO:tasks.workunit.client.1.smithi086.stdout:0/97: creat d0/d1/d4/f22 x:0 0 0 2024-04-06T14:16:59.861 INFO:tasks.workunit.client.1.smithi086.stdout:7/28: dread f5 [0,4194304] 0 2024-04-06T14:16:59.862 INFO:tasks.workunit.client.1.smithi086.stdout:6/27: creat d6/d9/fa x:0 0 0 2024-04-06T14:16:59.862 INFO:tasks.workunit.client.1.smithi086.stdout:1/17: dwrite f2 [4194304,4194304] 0 2024-04-06T14:16:59.862 INFO:tasks.workunit.client.0.smithi032.stdout:9/287: read d2/d15/d1e/f1f [991252,32529] 0 2024-04-06T14:16:59.863 INFO:tasks.workunit.client.0.smithi032.stdout:5/308: chown d16/d18/d33/d45/d52/d39 37155062 1 2024-04-06T14:16:59.863 INFO:tasks.workunit.client.1.smithi086.stdout:0/98: stat d0/d1/f13 0 2024-04-06T14:16:59.863 INFO:tasks.workunit.client.1.smithi086.stdout:7/29: dread - f6 zero size 2024-04-06T14:16:59.864 INFO:tasks.workunit.client.1.smithi086.stdout:0/99: chown d0/l1c 3 1 2024-04-06T14:16:59.864 INFO:tasks.workunit.client.1.smithi086.stdout:7/30: truncate f6 779424 0 2024-04-06T14:16:59.865 INFO:tasks.workunit.client.0.smithi032.stdout:6/288: read d4/d5/d19/d21/f30 [3700626,126129] 0 2024-04-06T14:16:59.865 INFO:tasks.workunit.client.0.smithi032.stdout:1/335: dwrite d0/f1e [0,4194304] 0 2024-04-06T14:16:59.866 INFO:tasks.workunit.client.0.smithi032.stdout:5/309: dread - d16/f24 zero size 2024-04-06T14:16:59.867 INFO:tasks.workunit.client.1.smithi086.stdout:2/29: link d0/l9 d0/lb 0 2024-04-06T14:16:59.874 INFO:tasks.workunit.client.0.smithi032.stdout:3/255: link d3/f24 d3/d16/f72 0 2024-04-06T14:16:59.877 INFO:tasks.workunit.client.1.smithi086.stdout:9/107: symlink d5/d13/l28 0 2024-04-06T14:16:59.878 INFO:tasks.workunit.client.0.smithi032.stdout:2/248: dread d0/f28 [0,4194304] 0 2024-04-06T14:16:59.878 INFO:tasks.workunit.client.0.smithi032.stdout:8/299: creat d3/d8/d7/d1a/f71 x:0 0 0 2024-04-06T14:16:59.879 INFO:tasks.workunit.client.1.smithi086.stdout:4/40: write d0/f8 [654372,7507] 0 2024-04-06T14:16:59.880 INFO:tasks.workunit.client.0.smithi032.stdout:0/247: truncate d3/da/d41/f43 859565 0 2024-04-06T14:16:59.881 INFO:tasks.workunit.client.1.smithi086.stdout:6/28: mkdir d6/d9/db 0 2024-04-06T14:16:59.882 INFO:tasks.workunit.client.1.smithi086.stdout:6/29: readlink d6/l8 0 2024-04-06T14:16:59.882 INFO:tasks.workunit.client.0.smithi032.stdout:9/288: write d2/f4 [4478578,75396] 0 2024-04-06T14:16:59.882 INFO:tasks.workunit.client.1.smithi086.stdout:1/18: symlink l5 0 2024-04-06T14:16:59.883 INFO:tasks.workunit.client.0.smithi032.stdout:9/289: readlink d2/d25/l30 0 2024-04-06T14:16:59.883 INFO:tasks.workunit.client.1.smithi086.stdout:6/30: stat d6/d9 0 2024-04-06T14:16:59.884 INFO:tasks.workunit.client.1.smithi086.stdout:0/100: truncate d0/d1/d4/f8 4154223 0 2024-04-06T14:16:59.886 INFO:tasks.workunit.client.1.smithi086.stdout:2/30: mknod d0/cc 0 2024-04-06T14:16:59.887 INFO:tasks.workunit.client.1.smithi086.stdout:3/84: link d5/l6 d5/df/l17 0 2024-04-06T14:16:59.888 INFO:tasks.workunit.client.0.smithi032.stdout:0/248: chown d3/da/d25/d2b/f3d 109715 1 2024-04-06T14:16:59.890 INFO:tasks.workunit.client.1.smithi086.stdout:4/41: dwrite d0/f2 [4194304,4194304] 0 2024-04-06T14:16:59.891 INFO:tasks.workunit.client.0.smithi032.stdout:1/336: dread d0/f2c [4194304,4194304] 0 2024-04-06T14:16:59.892 INFO:tasks.workunit.client.0.smithi032.stdout:4/412: dwrite d0/df/d23/d31/d50/d5e/d89/f86 [0,4194304] 0 2024-04-06T14:16:59.892 INFO:tasks.workunit.client.1.smithi086.stdout:8/142: dread d0/db/d10/d15/f28 [0,4194304] 0 2024-04-06T14:16:59.893 INFO:tasks.workunit.client.0.smithi032.stdout:9/290: dread d2/d15/d1e/d47/f5d [0,4194304] 0 2024-04-06T14:16:59.894 INFO:tasks.workunit.client.0.smithi032.stdout:4/413: chown d0/df/d1f/d2e/f41 14633305 1 2024-04-06T14:16:59.895 INFO:tasks.workunit.client.0.smithi032.stdout:9/291: stat d2/d25/d31/d39/f5a 0 2024-04-06T14:16:59.895 INFO:tasks.workunit.client.1.smithi086.stdout:2/31: creat d0/fd x:0 0 0 2024-04-06T14:16:59.896 INFO:tasks.workunit.client.1.smithi086.stdout:2/32: dread - d0/f6 zero size 2024-04-06T14:16:59.896 INFO:tasks.workunit.client.0.smithi032.stdout:9/292: readlink d2/d5/l35 0 2024-04-06T14:16:59.897 INFO:tasks.workunit.client.1.smithi086.stdout:1/19: dwrite f2 [4194304,4194304] 0 2024-04-06T14:16:59.898 INFO:tasks.workunit.client.1.smithi086.stdout:2/33: write d0/f6 [858002,127897] 0 2024-04-06T14:16:59.900 INFO:tasks.workunit.client.1.smithi086.stdout:2/34: truncate d0/f6 1384067 0 2024-04-06T14:16:59.901 INFO:tasks.workunit.client.1.smithi086.stdout:8/143: dread d0/f3 [0,4194304] 0 2024-04-06T14:16:59.905 INFO:tasks.workunit.client.1.smithi086.stdout:5/90: truncate d0/f1 464246 0 2024-04-06T14:16:59.907 INFO:tasks.workunit.client.1.smithi086.stdout:2/35: creat d0/fe x:0 0 0 2024-04-06T14:16:59.908 INFO:tasks.workunit.client.1.smithi086.stdout:4/42: fsync d0/f2 0 2024-04-06T14:16:59.908 INFO:tasks.workunit.client.1.smithi086.stdout:3/85: link d5/df/l16 d5/df/l18 0 2024-04-06T14:16:59.910 INFO:tasks.workunit.client.1.smithi086.stdout:1/20: dwrite f2 [0,4194304] 0 2024-04-06T14:16:59.910 INFO:tasks.workunit.client.1.smithi086.stdout:8/144: fdatasync d0/db/d10/f20 0 2024-04-06T14:16:59.913 INFO:tasks.workunit.client.1.smithi086.stdout:9/108: write f3 [4728432,33302] 0 2024-04-06T14:16:59.914 INFO:tasks.workunit.client.1.smithi086.stdout:2/36: mkdir d0/df 0 2024-04-06T14:16:59.914 INFO:tasks.workunit.client.1.smithi086.stdout:5/91: mknod d0/d19/c1e 0 2024-04-06T14:16:59.915 INFO:tasks.workunit.client.1.smithi086.stdout:3/86: fsync d5/fe 0 2024-04-06T14:16:59.916 INFO:tasks.workunit.client.1.smithi086.stdout:3/87: fdatasync f4 0 2024-04-06T14:16:59.916 INFO:tasks.workunit.client.1.smithi086.stdout:4/43: dread d0/f8 [0,4194304] 0 2024-04-06T14:16:59.917 INFO:tasks.workunit.client.1.smithi086.stdout:5/92: rmdir d0 39 2024-04-06T14:16:59.919 INFO:tasks.workunit.client.1.smithi086.stdout:8/145: mknod d0/d8/d22/c35 0 2024-04-06T14:16:59.923 INFO:tasks.workunit.client.1.smithi086.stdout:5/93: write d0/fa [2614773,29222] 0 2024-04-06T14:16:59.924 INFO:tasks.workunit.client.1.smithi086.stdout:3/88: symlink d5/l19 0 2024-04-06T14:16:59.925 INFO:tasks.workunit.client.1.smithi086.stdout:1/21: link c3 c6 0 2024-04-06T14:16:59.929 INFO:tasks.workunit.client.0.smithi032.stdout:5/310: creat d16/d58/f6d x:0 0 0 2024-04-06T14:16:59.929 INFO:tasks.workunit.client.1.smithi086.stdout:8/146: dread d0/db/f12 [0,4194304] 0 2024-04-06T14:16:59.930 INFO:tasks.workunit.client.0.smithi032.stdout:3/256: symlink d3/d3b/d3c/l73 0 2024-04-06T14:16:59.931 INFO:tasks.workunit.client.0.smithi032.stdout:5/311: read fe [796393,19241] 0 2024-04-06T14:16:59.931 INFO:tasks.workunit.client.0.smithi032.stdout:5/312: chown d16/d58 247 1 2024-04-06T14:16:59.932 INFO:tasks.workunit.client.0.smithi032.stdout:3/257: write d3/d16/f5c [907833,118555] 0 2024-04-06T14:16:59.932 INFO:tasks.workunit.client.0.smithi032.stdout:8/300: creat d3/d8/d7/d1a/f72 x:0 0 0 2024-04-06T14:16:59.933 INFO:tasks.workunit.client.1.smithi086.stdout:1/22: creat f7 x:0 0 0 2024-04-06T14:16:59.934 INFO:tasks.workunit.client.1.smithi086.stdout:5/94: creat d0/d1b/f1f x:0 0 0 2024-04-06T14:16:59.935 INFO:tasks.workunit.client.0.smithi032.stdout:0/249: symlink d3/da/d41/l5c 0 2024-04-06T14:16:59.936 INFO:tasks.workunit.client.1.smithi086.stdout:8/147: mknod d0/d8/d22/c36 0 2024-04-06T14:16:59.936 INFO:tasks.workunit.client.1.smithi086.stdout:8/148: rename d0 to d0/d8/d37 22 2024-04-06T14:16:59.936 INFO:tasks.workunit.client.1.smithi086.stdout:4/44: link d0/f5 d0/f10 0 2024-04-06T14:16:59.937 INFO:tasks.workunit.client.0.smithi032.stdout:1/337: mkdir d0/da/d3e/d75 0 2024-04-06T14:16:59.941 INFO:tasks.workunit.client.0.smithi032.stdout:9/293: unlink d2/d15/c3f 0 2024-04-06T14:16:59.942 INFO:tasks.workunit.client.0.smithi032.stdout:6/289: dwrite d4/f10 [0,4194304] 0 2024-04-06T14:16:59.944 INFO:tasks.workunit.client.0.smithi032.stdout:2/249: mkdir d0/d4e/d52/d57 0 2024-04-06T14:16:59.945 INFO:tasks.workunit.client.0.smithi032.stdout:9/294: fsync d2/d25/f32 0 2024-04-06T14:16:59.945 INFO:tasks.workunit.client.0.smithi032.stdout:5/313: creat d16/d58/f6e x:0 0 0 2024-04-06T14:16:59.945 INFO:tasks.workunit.client.1.smithi086.stdout:4/45: rename d0/fc to d0/de/f11 0 2024-04-06T14:16:59.945 INFO:tasks.workunit.client.1.smithi086.stdout:9/109: dwrite d5/f20 [0,4194304] 0 2024-04-06T14:16:59.949 INFO:tasks.workunit.client.0.smithi032.stdout:4/414: dread d0/df/d1f/d42/d45/f75 [0,4194304] 0 2024-04-06T14:16:59.949 INFO:tasks.workunit.client.0.smithi032.stdout:8/301: symlink d3/d8/d7/d14/l73 0 2024-04-06T14:16:59.950 INFO:tasks.workunit.client.1.smithi086.stdout:9/110: truncate d5/f27 403364 0 2024-04-06T14:16:59.950 INFO:tasks.workunit.client.1.smithi086.stdout:3/89: dwrite d5/f11 [0,4194304] 0 2024-04-06T14:16:59.951 INFO:tasks.workunit.client.1.smithi086.stdout:4/46: creat d0/f12 x:0 0 0 2024-04-06T14:16:59.952 INFO:tasks.workunit.client.0.smithi032.stdout:5/314: write d16/d58/f5f [223448,99648] 0 2024-04-06T14:16:59.952 INFO:tasks.workunit.client.1.smithi086.stdout:9/111: write d5/d13/d1a/f24 [5152677,87307] 0 2024-04-06T14:16:59.953 INFO:tasks.workunit.client.0.smithi032.stdout:3/258: rename d3/f19 to d3/d65/f74 0 2024-04-06T14:16:59.954 INFO:tasks.workunit.client.1.smithi086.stdout:3/90: chown d5/f9 2 1 2024-04-06T14:16:59.954 INFO:tasks.workunit.client.1.smithi086.stdout:4/47: creat d0/f13 x:0 0 0 2024-04-06T14:16:59.957 INFO:tasks.workunit.client.1.smithi086.stdout:9/112: write d5/d10/d14/f22 [2449811,6158] 0 2024-04-06T14:16:59.957 INFO:tasks.workunit.client.1.smithi086.stdout:9/113: stat c1 0 2024-04-06T14:16:59.959 INFO:tasks.workunit.client.0.smithi032.stdout:0/250: dwrite d3/da/f26 [0,4194304] 0 2024-04-06T14:16:59.961 INFO:tasks.workunit.client.1.smithi086.stdout:3/91: dread f4 [0,4194304] 0 2024-04-06T14:16:59.961 INFO:tasks.workunit.client.1.smithi086.stdout:4/48: symlink d0/de/l14 0 2024-04-06T14:16:59.962 INFO:tasks.workunit.client.1.smithi086.stdout:1/23: write f1 [5089440,108649] 0 2024-04-06T14:16:59.962 INFO:tasks.workunit.client.1.smithi086.stdout:9/114: mknod d5/d13/c29 0 2024-04-06T14:16:59.963 INFO:tasks.workunit.client.1.smithi086.stdout:5/95: write d0/f14 [1697674,42154] 0 2024-04-06T14:16:59.964 INFO:tasks.workunit.client.1.smithi086.stdout:8/149: write d0/db/f12 [716620,69769] 0 2024-04-06T14:16:59.964 INFO:tasks.workunit.client.1.smithi086.stdout:1/24: creat f8 x:0 0 0 2024-04-06T14:16:59.967 INFO:tasks.workunit.client.1.smithi086.stdout:4/49: mkdir d0/d15 0 2024-04-06T14:16:59.967 INFO:tasks.workunit.client.0.smithi032.stdout:9/295: creat d2/d15/d1e/d47/d4a/f68 x:0 0 0 2024-04-06T14:16:59.967 INFO:tasks.workunit.client.0.smithi032.stdout:4/415: mknod d0/df/d23/d31/d50/d5e/d89/c90 0 2024-04-06T14:16:59.971 INFO:tasks.workunit.client.0.smithi032.stdout:6/290: rename d4/d5/d19/d1d to d4/d5/d8/d1f/d51 0 2024-04-06T14:16:59.971 INFO:tasks.workunit.client.1.smithi086.stdout:4/50: fdatasync d0/f2 0 2024-04-06T14:16:59.974 INFO:tasks.workunit.client.1.smithi086.stdout:3/92: fsync d5/f9 0 2024-04-06T14:16:59.974 INFO:tasks.workunit.client.0.smithi032.stdout:1/338: dread d0/d10/d16/f6d [0,4194304] 0 2024-04-06T14:16:59.974 INFO:tasks.workunit.client.0.smithi032.stdout:2/250: dwrite d0/f51 [0,4194304] 0 2024-04-06T14:16:59.976 INFO:tasks.workunit.client.0.smithi032.stdout:5/315: creat d16/d58/f6f x:0 0 0 2024-04-06T14:16:59.977 INFO:tasks.workunit.client.1.smithi086.stdout:5/96: dwrite d0/f14 [0,4194304] 0 2024-04-06T14:16:59.979 INFO:tasks.workunit.client.1.smithi086.stdout:9/115: rename d5/c8 to d5/d13/d1a/c2a 0 2024-04-06T14:16:59.980 INFO:tasks.workunit.client.1.smithi086.stdout:3/93: rmdir d5 39 2024-04-06T14:16:59.980 INFO:tasks.workunit.client.1.smithi086.stdout:1/25: dwrite f4 [0,4194304] 0 2024-04-06T14:16:59.981 INFO:tasks.workunit.client.1.smithi086.stdout:9/116: read f0 [3080041,109264] 0 2024-04-06T14:16:59.981 INFO:tasks.workunit.client.1.smithi086.stdout:4/51: rename d0/de to d0/de/d16 22 2024-04-06T14:16:59.981 INFO:tasks.workunit.client.0.smithi032.stdout:0/251: creat d3/f5d x:0 0 0 2024-04-06T14:16:59.981 INFO:tasks.workunit.client.0.smithi032.stdout:3/259: write d3/d16/f72 [606800,66152] 0 2024-04-06T14:16:59.984 INFO:tasks.workunit.client.0.smithi032.stdout:2/251: unlink d0/c10 0 2024-04-06T14:16:59.984 INFO:tasks.workunit.client.0.smithi032.stdout:6/291: truncate d4/d5/f4c 771198 0 2024-04-06T14:16:59.984 INFO:tasks.workunit.client.0.smithi032.stdout:3/260: chown d3/f4 160268530 1 2024-04-06T14:16:59.986 INFO:tasks.workunit.client.0.smithi032.stdout:0/252: fdatasync d3/da/f1e 0 2024-04-06T14:16:59.987 INFO:tasks.workunit.client.0.smithi032.stdout:5/316: unlink d16/d18/d33/d45/c3f 0 2024-04-06T14:16:59.989 INFO:tasks.workunit.client.0.smithi032.stdout:1/339: link d0/d10/d16/d3f/l6f d0/d10/d66/l76 0 2024-04-06T14:16:59.989 INFO:tasks.workunit.client.1.smithi086.stdout:5/97: creat d0/d1b/f20 x:0 0 0 2024-04-06T14:16:59.989 INFO:tasks.workunit.client.0.smithi032.stdout:5/317: write d16/d58/f59 [837360,2999] 0 2024-04-06T14:16:59.990 INFO:tasks.workunit.client.0.smithi032.stdout:9/296: getdents d2/d15/d55 0 2024-04-06T14:16:59.991 INFO:tasks.workunit.client.0.smithi032.stdout:2/252: creat d0/d30/d39/f58 x:0 0 0 2024-04-06T14:16:59.992 INFO:tasks.workunit.client.1.smithi086.stdout:1/26: creat f9 x:0 0 0 2024-04-06T14:16:59.993 INFO:tasks.workunit.client.0.smithi032.stdout:0/253: creat d3/da/f5e x:0 0 0 2024-04-06T14:16:59.994 INFO:tasks.workunit.client.1.smithi086.stdout:4/52: creat d0/f17 x:0 0 0 2024-04-06T14:16:59.994 INFO:tasks.workunit.client.1.smithi086.stdout:9/117: creat d5/d13/f2b x:0 0 0 2024-04-06T14:16:59.995 INFO:tasks.workunit.client.1.smithi086.stdout:4/53: dread - d0/fd zero size 2024-04-06T14:16:59.995 INFO:tasks.workunit.client.0.smithi032.stdout:6/292: rename d4/d5/d14/f4d to d4/d5/d18/f52 0 2024-04-06T14:16:59.995 INFO:tasks.workunit.client.0.smithi032.stdout:3/261: mknod d3/c75 0 2024-04-06T14:16:59.995 INFO:tasks.workunit.client.0.smithi032.stdout:5/318: truncate d16/f38 976712 0 2024-04-06T14:16:59.997 INFO:tasks.workunit.client.1.smithi086.stdout:1/27: dread f2 [4194304,4194304] 0 2024-04-06T14:16:59.998 INFO:tasks.workunit.client.0.smithi032.stdout:6/293: write d4/d5/d8/d4b/f4a [805991,125306] 0 2024-04-06T14:16:59.998 INFO:tasks.workunit.client.1.smithi086.stdout:8/150: write d0/f17 [142303,39648] 0 2024-04-06T14:16:59.999 INFO:tasks.workunit.client.0.smithi032.stdout:2/253: mkdir d0/d30/d39/d59 0 2024-04-06T14:17:00.000 INFO:tasks.workunit.client.0.smithi032.stdout:1/340: getdents d0/d41/d5b 0 2024-04-06T14:17:00.002 INFO:tasks.workunit.client.1.smithi086.stdout:3/94: write d5/f9 [495219,25953] 0 2024-04-06T14:17:00.003 INFO:tasks.workunit.client.0.smithi032.stdout:6/294: mkdir d4/d5/d8/d1f/d53 0 2024-04-06T14:17:00.009 INFO:tasks.workunit.client.0.smithi032.stdout:3/262: mknod d3/c76 0 2024-04-06T14:17:00.009 INFO:tasks.workunit.client.0.smithi032.stdout:5/319: symlink d16/d18/d33/d45/l70 0 2024-04-06T14:17:00.009 INFO:tasks.workunit.client.0.smithi032.stdout:8/302: dread d3/d8/d12/f3f [0,4194304] 0 2024-04-06T14:17:00.009 INFO:tasks.workunit.client.1.smithi086.stdout:3/95: chown d5/fe 909 1 2024-04-06T14:17:00.009 INFO:tasks.workunit.client.1.smithi086.stdout:9/118: chown d5/l6 136799988 1 2024-04-06T14:17:00.009 INFO:tasks.workunit.client.1.smithi086.stdout:3/96: write d5/fe [3955105,42807] 0 2024-04-06T14:17:00.009 INFO:tasks.workunit.client.1.smithi086.stdout:1/28: creat fa x:0 0 0 2024-04-06T14:17:00.009 INFO:tasks.workunit.client.1.smithi086.stdout:1/29: fdatasync f8 0 2024-04-06T14:17:00.010 INFO:tasks.workunit.client.0.smithi032.stdout:5/320: write d16/f50 [891450,107995] 0 2024-04-06T14:17:00.010 INFO:tasks.workunit.client.1.smithi086.stdout:4/54: truncate d0/f2 4145900 0 2024-04-06T14:17:00.010 INFO:tasks.workunit.client.1.smithi086.stdout:1/30: write f8 [669882,94386] 0 2024-04-06T14:17:00.011 INFO:tasks.workunit.client.1.smithi086.stdout:9/119: mknod d5/d18/c2c 0 2024-04-06T14:17:00.012 INFO:tasks.workunit.client.0.smithi032.stdout:2/254: truncate d0/d4e/f53 705420 0 2024-04-06T14:17:00.012 INFO:tasks.workunit.client.1.smithi086.stdout:4/55: read d0/f8 [190839,30077] 0 2024-04-06T14:17:00.012 INFO:tasks.workunit.client.1.smithi086.stdout:1/31: chown f4 923860181 1 2024-04-06T14:17:00.013 INFO:tasks.workunit.client.1.smithi086.stdout:5/98: truncate d0/d16/d18/ff 529385 0 2024-04-06T14:17:00.014 INFO:tasks.workunit.client.1.smithi086.stdout:4/56: stat d0/f13 0 2024-04-06T14:17:00.015 INFO:tasks.workunit.client.1.smithi086.stdout:3/97: dread d5/fe [0,4194304] 0 2024-04-06T14:17:00.016 INFO:tasks.workunit.client.0.smithi032.stdout:3/263: creat d3/d3b/d3c/f77 x:0 0 0 2024-04-06T14:17:00.016 INFO:tasks.workunit.client.0.smithi032.stdout:4/416: truncate d0/df/d23/f3c 280849 0 2024-04-06T14:17:00.017 INFO:tasks.workunit.client.1.smithi086.stdout:8/151: link d0/d8/l14 d0/db/d10/d15/l38 0 2024-04-06T14:17:00.018 INFO:tasks.workunit.client.1.smithi086.stdout:8/152: dread - d0/f34 zero size 2024-04-06T14:17:00.023 INFO:tasks.workunit.client.1.smithi086.stdout:4/57: symlink d0/l18 0 2024-04-06T14:17:00.023 INFO:tasks.workunit.client.1.smithi086.stdout:3/98: creat d5/df/f1a x:0 0 0 2024-04-06T14:17:00.023 INFO:tasks.workunit.client.1.smithi086.stdout:1/32: dwrite f2 [0,4194304] 0 2024-04-06T14:17:00.026 INFO:tasks.workunit.client.1.smithi086.stdout:5/99: mkdir d0/d16/d18/d21 0 2024-04-06T14:17:00.026 INFO:tasks.workunit.client.0.smithi032.stdout:6/295: write d4/d5/d28/f41 [1373291,101387] 0 2024-04-06T14:17:00.028 INFO:tasks.workunit.client.1.smithi086.stdout:1/33: write f4 [4376001,128184] 0 2024-04-06T14:17:00.030 INFO:tasks.workunit.client.0.smithi032.stdout:3/264: rmdir d3 39 2024-04-06T14:17:00.030 INFO:tasks.workunit.client.0.smithi032.stdout:5/321: dwrite d16/d18/f31 [0,4194304] 0 2024-04-06T14:17:00.034 INFO:tasks.workunit.client.0.smithi032.stdout:4/417: symlink d0/df/d23/d31/l91 0 2024-04-06T14:17:00.034 INFO:tasks.workunit.client.0.smithi032.stdout:9/297: dread d2/f4 [0,4194304] 0 2024-04-06T14:17:00.034 INFO:tasks.workunit.client.1.smithi086.stdout:1/34: dread - fa zero size 2024-04-06T14:17:00.034 INFO:tasks.workunit.client.1.smithi086.stdout:1/35: rmdir - no directory 2024-04-06T14:17:00.035 INFO:tasks.workunit.client.0.smithi032.stdout:8/303: getdents d3/d8/d12/d2f/d4a 0 2024-04-06T14:17:00.036 INFO:tasks.workunit.client.0.smithi032.stdout:2/255: rename d0/d12 to d0/d5a 0 2024-04-06T14:17:00.036 INFO:tasks.workunit.client.0.smithi032.stdout:1/341: dwrite d0/da/f38 [0,4194304] 0 2024-04-06T14:17:00.037 INFO:tasks.workunit.client.0.smithi032.stdout:6/296: fdatasync d4/d5/d8/d1f/f33 0 2024-04-06T14:17:00.042 INFO:tasks.workunit.client.0.smithi032.stdout:0/254: dwrite d3/f10 [0,4194304] 0 2024-04-06T14:17:00.044 INFO:tasks.workunit.client.1.smithi086.stdout:1/36: mkdir db 0 2024-04-06T14:17:00.045 INFO:tasks.workunit.client.1.smithi086.stdout:1/37: read f8 [724287,64711] 0 2024-04-06T14:17:00.046 INFO:tasks.workunit.client.0.smithi032.stdout:6/297: mkdir d4/d5/d28/d54 0 2024-04-06T14:17:00.046 INFO:tasks.workunit.client.0.smithi032.stdout:1/342: symlink d0/d10/d16/d3f/l77 0 2024-04-06T14:17:00.047 INFO:tasks.workunit.client.0.smithi032.stdout:9/298: rename d2/d15/l38 to d2/d15/d1e/d47/l69 0 2024-04-06T14:17:00.048 INFO:tasks.workunit.client.0.smithi032.stdout:6/298: chown d4/d5/c12 62 1 2024-04-06T14:17:00.050 INFO:tasks.workunit.client.0.smithi032.stdout:4/418: dread d0/f1e [0,4194304] 0 2024-04-06T14:17:00.051 INFO:tasks.workunit.client.0.smithi032.stdout:2/256: creat d0/d5a/d3a/f5b x:0 0 0 2024-04-06T14:17:00.052 INFO:tasks.workunit.client.0.smithi032.stdout:1/343: read d0/da/f65 [4115651,3422] 0 2024-04-06T14:17:00.054 INFO:tasks.workunit.client.0.smithi032.stdout:1/344: chown d0/d10/d16/f6d 2477 1 2024-04-06T14:17:00.055 INFO:tasks.workunit.client.0.smithi032.stdout:6/299: creat d4/d5/d19/f55 x:0 0 0 2024-04-06T14:17:00.059 INFO:tasks.workunit.client.0.smithi032.stdout:8/304: rename d3/d8/d3a/d4b/f5e to d3/d8/d12/d2f/d46/f74 0 2024-04-06T14:17:00.059 INFO:tasks.workunit.client.0.smithi032.stdout:5/322: link d16/d18/l1b d16/l71 0 2024-04-06T14:17:00.065 INFO:tasks.workunit.client.0.smithi032.stdout:2/257: rename d0/d30/f50 to d0/d4e/d52/d57/f5c 0 2024-04-06T14:17:00.065 INFO:tasks.workunit.client.0.smithi032.stdout:0/255: write d3/da/f18 [4642658,11092] 0 2024-04-06T14:17:00.066 INFO:tasks.workunit.client.0.smithi032.stdout:2/258: dread - d0/d5a/f31 zero size 2024-04-06T14:17:00.066 INFO:tasks.workunit.client.0.smithi032.stdout:0/256: readlink d3/da/l2c 0 2024-04-06T14:17:00.068 INFO:tasks.workunit.client.0.smithi032.stdout:4/419: creat d0/df/f92 x:0 0 0 2024-04-06T14:17:00.068 INFO:tasks.workunit.client.0.smithi032.stdout:1/345: mkdir d0/d10/d78 0 2024-04-06T14:17:00.069 INFO:tasks.workunit.client.0.smithi032.stdout:6/300: creat d4/d5/d8/d1f/d51/d4e/f56 x:0 0 0 2024-04-06T14:17:00.071 INFO:tasks.workunit.client.0.smithi032.stdout:5/323: mknod d16/d18/c72 0 2024-04-06T14:17:00.072 INFO:tasks.workunit.client.0.smithi032.stdout:4/420: symlink d0/df/d1f/d42/d45/l93 0 2024-04-06T14:17:00.073 INFO:tasks.workunit.client.0.smithi032.stdout:2/259: mknod d0/d30/d39/d59/c5d 0 2024-04-06T14:17:00.074 INFO:tasks.workunit.client.0.smithi032.stdout:6/301: write d4/d5/d28/f41 [2222302,119550] 0 2024-04-06T14:17:00.075 INFO:tasks.workunit.client.0.smithi032.stdout:1/346: symlink d0/da/l79 0 2024-04-06T14:17:00.075 INFO:tasks.workunit.client.0.smithi032.stdout:8/305: dwrite d3/d8/d12/f6f [0,4194304] 0 2024-04-06T14:17:00.078 INFO:tasks.workunit.client.0.smithi032.stdout:0/257: getdents d3/da/d25/d2b/d4a 0 2024-04-06T14:17:00.079 INFO:tasks.workunit.client.0.smithi032.stdout:4/421: write d0/df/d1f/f34 [2043442,110843] 0 2024-04-06T14:17:00.080 INFO:tasks.workunit.client.0.smithi032.stdout:0/258: stat d3/f14 0 2024-04-06T14:17:00.080 INFO:tasks.workunit.client.0.smithi032.stdout:9/299: dwrite d2/d15/d1e/d47/f5d [0,4194304] 0 2024-04-06T14:17:00.081 INFO:tasks.workunit.client.0.smithi032.stdout:5/324: mknod d16/d18/d33/d45/d5e/c73 0 2024-04-06T14:17:00.083 INFO:tasks.workunit.client.0.smithi032.stdout:1/347: write d0/d10/d16/f51 [1199339,117307] 0 2024-04-06T14:17:00.083 INFO:tasks.workunit.client.0.smithi032.stdout:6/302: creat d4/d5/d19/d25/f57 x:0 0 0 2024-04-06T14:17:00.085 INFO:tasks.workunit.client.0.smithi032.stdout:9/300: chown d2/d5/f18 192 1 2024-04-06T14:17:00.086 INFO:tasks.workunit.client.0.smithi032.stdout:1/348: chown d0/d41/d71 5 1 2024-04-06T14:17:00.087 INFO:tasks.workunit.client.0.smithi032.stdout:8/306: mkdir d3/d8/d12/d2f/d4a/d60/d75 0 2024-04-06T14:17:00.087 INFO:tasks.workunit.client.0.smithi032.stdout:0/259: chown d3/da/d25/d36/l4f 60 1 2024-04-06T14:17:00.090 INFO:tasks.workunit.client.0.smithi032.stdout:8/307: chown d3/d8/l2c 11001 1 2024-04-06T14:17:00.091 INFO:tasks.workunit.client.0.smithi032.stdout:6/303: fdatasync d4/d5/d19/d21/f30 0 2024-04-06T14:17:00.093 INFO:tasks.workunit.client.0.smithi032.stdout:0/260: mknod d3/da/c5f 0 2024-04-06T14:17:00.095 INFO:tasks.workunit.client.0.smithi032.stdout:4/422: link d0/df/d23/f32 d0/df/d1f/d2e/f94 0 2024-04-06T14:17:00.096 INFO:tasks.workunit.client.0.smithi032.stdout:5/325: truncate d16/d18/d33/d45/d52/f43 139988 0 2024-04-06T14:17:00.096 INFO:tasks.workunit.client.0.smithi032.stdout:8/308: read - d3/d8/d7/d1a/f61 zero size 2024-04-06T14:17:00.099 INFO:tasks.workunit.client.0.smithi032.stdout:6/304: mknod d4/d5/d19/d25/c58 0 2024-04-06T14:17:00.099 INFO:tasks.workunit.client.0.smithi032.stdout:0/261: dread - d3/da/d25/d36/f51 zero size 2024-04-06T14:17:00.101 INFO:tasks.workunit.client.0.smithi032.stdout:2/260: dwrite d0/d30/d39/f58 [0,4194304] 0 2024-04-06T14:17:00.101 INFO:tasks.workunit.client.0.smithi032.stdout:5/326: read - d16/d58/f6f zero size 2024-04-06T14:17:00.107 INFO:tasks.workunit.client.0.smithi032.stdout:0/262: symlink d3/da/d35/l60 0 2024-04-06T14:17:00.108 INFO:tasks.workunit.client.0.smithi032.stdout:6/305: creat d4/d5/d35/f59 x:0 0 0 2024-04-06T14:17:00.109 INFO:tasks.workunit.client.0.smithi032.stdout:9/301: dwrite d2/d15/d1e/f2a [0,4194304] 0 2024-04-06T14:17:00.110 INFO:tasks.workunit.client.0.smithi032.stdout:5/327: mkdir d16/d58/d54/d74 0 2024-04-06T14:17:00.110 INFO:tasks.workunit.client.1.smithi086.stdout:1/38: creat db/fc x:0 0 0 2024-04-06T14:17:00.111 INFO:tasks.workunit.client.1.smithi086.stdout:5/100: truncate d0/f3 725994 0 2024-04-06T14:17:00.113 INFO:tasks.workunit.client.0.smithi032.stdout:1/349: dread d0/d10/f3d [4194304,4194304] 0 2024-04-06T14:17:00.113 INFO:tasks.workunit.client.1.smithi086.stdout:3/99: truncate d5/f9 23235 0 2024-04-06T14:17:00.114 INFO:tasks.workunit.client.0.smithi032.stdout:2/261: unlink d0/c2d 0 2024-04-06T14:17:00.114 INFO:tasks.workunit.client.1.smithi086.stdout:8/153: dwrite d0/f5 [0,4194304] 0 2024-04-06T14:17:00.115 INFO:tasks.workunit.client.0.smithi032.stdout:0/263: creat d3/da/d25/d2b/f61 x:0 0 0 2024-04-06T14:17:00.115 INFO:tasks.workunit.client.0.smithi032.stdout:2/262: dread d0/d4e/f53 [0,4194304] 0 2024-04-06T14:17:00.116 INFO:tasks.workunit.client.0.smithi032.stdout:5/328: mknod d16/d58/d54/c75 0 2024-04-06T14:17:00.116 INFO:tasks.workunit.client.0.smithi032.stdout:2/263: fdatasync d0/ff 0 2024-04-06T14:17:00.117 INFO:tasks.workunit.client.0.smithi032.stdout:6/306: symlink d4/d5/d14/l5a 0 2024-04-06T14:17:00.119 INFO:tasks.workunit.client.0.smithi032.stdout:9/302: symlink d2/d25/d31/d39/l6a 0 2024-04-06T14:17:00.119 INFO:tasks.workunit.client.1.smithi086.stdout:1/39: creat db/fd x:0 0 0 2024-04-06T14:17:00.120 INFO:tasks.workunit.client.1.smithi086.stdout:5/101: mkdir d0/d19/d22 0 2024-04-06T14:17:00.122 INFO:tasks.workunit.client.1.smithi086.stdout:8/154: rename d0/db/d10 to d0/d8/d16/d39 0 2024-04-06T14:17:00.125 INFO:tasks.workunit.client.1.smithi086.stdout:4/58: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:17:00.126 INFO:tasks.workunit.client.0.smithi032.stdout:6/307: creat d4/d5/d28/f5b x:0 0 0 2024-04-06T14:17:00.126 INFO:tasks.workunit.client.1.smithi086.stdout:4/59: write d0/f17 [1036865,48670] 0 2024-04-06T14:17:00.128 INFO:tasks.workunit.client.0.smithi032.stdout:9/303: unlink d2/f2e 0 2024-04-06T14:17:00.128 INFO:tasks.workunit.client.0.smithi032.stdout:6/308: fsync d4/d5/d19/f46 0 2024-04-06T14:17:00.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:16:59 smithi086 ceph-mon[47925]: pgmap v17: 65 pgs: 65 active+clean; 545 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 936 KiB/s rd, 52 MiB/s wr, 141 op/s 2024-04-06T14:17:00.129 INFO:tasks.workunit.client.1.smithi086.stdout:1/40: rename f9 to db/fe 0 2024-04-06T14:17:00.129 INFO:tasks.workunit.client.1.smithi086.stdout:1/41: readlink l5 0 2024-04-06T14:17:00.129 INFO:tasks.workunit.client.0.smithi032.stdout:6/309: dread - d4/d5/d8/d1f/d51/f3d zero size 2024-04-06T14:17:00.130 INFO:tasks.workunit.client.0.smithi032.stdout:8/309: link d3/d8/d7/d14/f44 d3/d8/d12/d2f/d46/f76 0 2024-04-06T14:17:00.131 INFO:tasks.workunit.client.1.smithi086.stdout:5/102: rename d0/d1b to d0/d16/d18/d21/d23 0 2024-04-06T14:17:00.133 INFO:tasks.workunit.client.1.smithi086.stdout:1/42: unlink db/fd 0 2024-04-06T14:17:00.136 INFO:tasks.workunit.client.1.smithi086.stdout:8/155: creat d0/d8/d16/d39/f3a x:0 0 0 2024-04-06T14:17:00.139 INFO:tasks.workunit.client.0.smithi032.stdout:2/264: mknod d0/c5e 0 2024-04-06T14:17:00.146 INFO:tasks.workunit.client.1.smithi086.stdout:5/103: dread d0/f1 [0,4194304] 0 2024-04-06T14:17:00.146 INFO:tasks.workunit.client.1.smithi086.stdout:3/100: read d5/f9 [13350,125331] 0 2024-04-06T14:17:00.148 INFO:tasks.workunit.client.0.smithi032.stdout:5/329: link d16/d18/d33/d45/c2a d16/d18/d33/d45/d52/c76 0 2024-04-06T14:17:00.148 INFO:tasks.workunit.client.1.smithi086.stdout:8/156: mkdir d0/d3b 0 2024-04-06T14:17:00.149 INFO:tasks.workunit.client.1.smithi086.stdout:8/157: chown d0/db/cd 16513581 1 2024-04-06T14:17:00.149 INFO:tasks.workunit.client.1.smithi086.stdout:3/101: write d5/df/f13 [341998,30018] 0 2024-04-06T14:17:00.150 INFO:tasks.workunit.client.0.smithi032.stdout:8/310: dwrite d3/d8/d3a/f3b [0,4194304] 0 2024-04-06T14:17:00.150 INFO:tasks.workunit.client.0.smithi032.stdout:5/330: truncate fb 100675 0 2024-04-06T14:17:00.151 INFO:tasks.workunit.client.1.smithi086.stdout:5/104: write d0/d16/d18/d21/d23/f1f [1042623,18830] 0 2024-04-06T14:17:00.153 INFO:tasks.workunit.client.0.smithi032.stdout:2/265: dwrite d0/d5a/f1f [0,4194304] 0 2024-04-06T14:17:00.158 INFO:tasks.workunit.client.0.smithi032.stdout:8/311: creat d3/d8/d12/d2f/f77 x:0 0 0 2024-04-06T14:17:00.159 INFO:tasks.workunit.client.1.smithi086.stdout:8/158: unlink d0/d13/c18 0 2024-04-06T14:17:00.159 INFO:tasks.workunit.client.1.smithi086.stdout:1/43: getdents db 0 2024-04-06T14:17:00.162 INFO:tasks.workunit.client.1.smithi086.stdout:3/102: mknod d5/df/c1b 0 2024-04-06T14:17:00.163 INFO:tasks.workunit.client.0.smithi032.stdout:2/266: rename d0/d4e/d52/d57/f5c to d0/d30/d39/d59/f5f 0 2024-04-06T14:17:00.164 INFO:tasks.workunit.client.0.smithi032.stdout:6/310: truncate d4/d5/d18/f49 375680 0 2024-04-06T14:17:00.164 INFO:tasks.workunit.client.1.smithi086.stdout:5/105: readlink d0/d16/d18/l10 0 2024-04-06T14:17:00.165 INFO:tasks.workunit.client.0.smithi032.stdout:7/309: sync 2024-04-06T14:17:00.165 INFO:tasks.workunit.client.0.smithi032.stdout:8/312: creat d3/d8/d7/d1a/f78 x:0 0 0 2024-04-06T14:17:00.168 INFO:tasks.workunit.client.0.smithi032.stdout:6/311: dread - d4/d5/d19/f46 zero size 2024-04-06T14:17:00.168 INFO:tasks.workunit.client.0.smithi032.stdout:9/304: dwrite d2/f2d [0,4194304] 0 2024-04-06T14:17:00.170 INFO:tasks.workunit.client.0.smithi032.stdout:6/312: chown d4/d5/d8 11586 1 2024-04-06T14:17:00.170 INFO:tasks.workunit.client.1.smithi086.stdout:3/103: dwrite d5/f11 [0,4194304] 0 2024-04-06T14:17:00.171 INFO:tasks.workunit.client.0.smithi032.stdout:8/313: write d3/d8/d3a/f42 [1553486,43258] 0 2024-04-06T14:17:00.174 INFO:tasks.workunit.client.1.smithi086.stdout:5/106: mknod d0/c24 0 2024-04-06T14:17:00.174 INFO:tasks.workunit.client.1.smithi086.stdout:8/159: link d0/f3 d0/d8/f3c 0 2024-04-06T14:17:00.176 INFO:tasks.workunit.client.0.smithi032.stdout:8/314: fsync d3/d8/d3a/f4c 0 2024-04-06T14:17:00.177 INFO:tasks.workunit.client.0.smithi032.stdout:6/313: creat d4/d5/d28/f5c x:0 0 0 2024-04-06T14:17:00.179 INFO:tasks.workunit.client.1.smithi086.stdout:5/107: dread - d0/d16/d18/f1a zero size 2024-04-06T14:17:00.181 INFO:tasks.workunit.client.1.smithi086.stdout:1/44: write f8 [236232,58529] 0 2024-04-06T14:17:00.181 INFO:tasks.workunit.client.1.smithi086.stdout:5/108: write d0/fa [2298929,24890] 0 2024-04-06T14:17:00.182 INFO:tasks.workunit.client.0.smithi032.stdout:9/305: mkdir d2/d15/d1e/d65/d6b 0 2024-04-06T14:17:00.183 INFO:tasks.workunit.client.0.smithi032.stdout:7/310: write fe [814247,40199] 0 2024-04-06T14:17:00.183 INFO:tasks.workunit.client.0.smithi032.stdout:2/267: dwrite d0/d5a/f2a [0,4194304] 0 2024-04-06T14:17:00.184 INFO:tasks.workunit.client.1.smithi086.stdout:8/160: creat d0/d3b/f3d x:0 0 0 2024-04-06T14:17:00.190 INFO:tasks.workunit.client.0.smithi032.stdout:7/311: fsync d16/d20/d26/d2d/d30/f46 0 2024-04-06T14:17:00.190 INFO:tasks.workunit.client.0.smithi032.stdout:6/314: creat d4/d5/d19/d25/f5d x:0 0 0 2024-04-06T14:17:00.192 INFO:tasks.workunit.client.1.smithi086.stdout:1/45: dwrite f7 [0,4194304] 0 2024-04-06T14:17:00.194 INFO:tasks.workunit.client.1.smithi086.stdout:8/161: rename d0/f3 to d0/db/d32/f3e 0 2024-04-06T14:17:00.196 INFO:tasks.workunit.client.0.smithi032.stdout:8/315: link d3/d8/d12/f5b d3/f79 0 2024-04-06T14:17:00.196 INFO:tasks.workunit.client.0.smithi032.stdout:7/312: mkdir d16/d20/d48/d6d 0 2024-04-06T14:17:00.197 INFO:tasks.workunit.client.0.smithi032.stdout:6/315: rmdir d4/d5/d8/d2b 39 2024-04-06T14:17:00.199 INFO:tasks.workunit.client.1.smithi086.stdout:1/46: chown c3 0 1 2024-04-06T14:17:00.200 INFO:tasks.workunit.client.1.smithi086.stdout:8/162: unlink d0/db/l2a 0 2024-04-06T14:17:00.202 INFO:tasks.workunit.client.1.smithi086.stdout:1/47: creat db/ff x:0 0 0 2024-04-06T14:17:00.204 INFO:tasks.workunit.client.0.smithi032.stdout:7/313: mkdir d16/d20/d6e 0 2024-04-06T14:17:00.204 INFO:tasks.workunit.client.0.smithi032.stdout:6/316: rmdir d4/d5/d28 39 2024-04-06T14:17:00.205 INFO:tasks.workunit.client.1.smithi086.stdout:1/48: write f1 [4785348,128189] 0 2024-04-06T14:17:00.207 INFO:tasks.workunit.client.1.smithi086.stdout:1/49: fdatasync f4 0 2024-04-06T14:17:00.207 INFO:tasks.workunit.client.0.smithi032.stdout:6/317: write d4/d5/d8/d4b/f4a [1561403,111579] 0 2024-04-06T14:17:00.208 INFO:tasks.workunit.client.0.smithi032.stdout:8/316: creat d3/d8/f7a x:0 0 0 2024-04-06T14:17:00.209 INFO:tasks.workunit.client.0.smithi032.stdout:8/317: write d3/d8/d12/d2f/f62 [934728,93142] 0 2024-04-06T14:17:00.210 INFO:tasks.workunit.client.1.smithi086.stdout:1/50: fdatasync db/fe 0 2024-04-06T14:17:00.211 INFO:tasks.workunit.client.0.smithi032.stdout:9/306: truncate d2/d15/d1e/f2a 4128990 0 2024-04-06T14:17:00.213 INFO:tasks.workunit.client.0.smithi032.stdout:6/318: unlink d4/c7 0 2024-04-06T14:17:00.213 INFO:tasks.workunit.client.1.smithi086.stdout:1/51: creat db/f10 x:0 0 0 2024-04-06T14:17:00.214 INFO:tasks.workunit.client.1.smithi086.stdout:3/104: dwrite d5/f9 [0,4194304] 0 2024-04-06T14:17:00.215 INFO:tasks.workunit.client.1.smithi086.stdout:8/163: dwrite d0/f17 [0,4194304] 0 2024-04-06T14:17:00.216 INFO:tasks.workunit.client.0.smithi032.stdout:9/307: chown d2/d25/f3b 184 1 2024-04-06T14:17:00.216 INFO:tasks.workunit.client.1.smithi086.stdout:3/105: fsync f4 0 2024-04-06T14:17:00.217 INFO:tasks.workunit.client.1.smithi086.stdout:8/164: write d0/f34 [562791,3455] 0 2024-04-06T14:17:00.217 INFO:tasks.workunit.client.1.smithi086.stdout:1/52: mkdir db/d11 0 2024-04-06T14:17:00.218 INFO:tasks.workunit.client.0.smithi032.stdout:2/268: truncate d0/d30/d39/f58 2982472 0 2024-04-06T14:17:00.219 INFO:tasks.workunit.client.1.smithi086.stdout:3/106: stat d5/c14 0 2024-04-06T14:17:00.223 INFO:tasks.workunit.client.1.smithi086.stdout:3/107: write d5/df/f13 [3036592,63902] 0 2024-04-06T14:17:00.225 INFO:tasks.workunit.client.0.smithi032.stdout:6/319: mkdir d4/d5/d8/d1f/d51/d5e 0 2024-04-06T14:17:00.225 INFO:tasks.workunit.client.1.smithi086.stdout:3/108: write d5/f9 [2745775,54516] 0 2024-04-06T14:17:00.226 INFO:tasks.workunit.client.1.smithi086.stdout:8/165: rename d0/d8/d16/c1a to d0/d8/d16/d39/d15/c3f 0 2024-04-06T14:17:00.230 INFO:tasks.workunit.client.1.smithi086.stdout:3/109: creat d5/df/f1c x:0 0 0 2024-04-06T14:17:00.230 INFO:tasks.workunit.client.1.smithi086.stdout:1/53: dwrite db/f10 [0,4194304] 0 2024-04-06T14:17:00.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:16:59 smithi032 ceph-mon[39730]: pgmap v17: 65 pgs: 65 active+clean; 545 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 936 KiB/s rd, 52 MiB/s wr, 141 op/s 2024-04-06T14:17:00.232 INFO:tasks.workunit.client.1.smithi086.stdout:8/166: creat d0/d13/f40 x:0 0 0 2024-04-06T14:17:00.233 INFO:tasks.workunit.client.1.smithi086.stdout:3/110: chown f4 139 1 2024-04-06T14:17:00.235 INFO:tasks.workunit.client.1.smithi086.stdout:3/111: write d5/f11 [2595569,12642] 0 2024-04-06T14:17:00.236 INFO:tasks.workunit.client.0.smithi032.stdout:6/320: dread d4/d5/d8/d4b/f4a [0,4194304] 0 2024-04-06T14:17:00.237 INFO:tasks.workunit.client.1.smithi086.stdout:1/54: creat db/d11/f12 x:0 0 0 2024-04-06T14:17:00.240 INFO:tasks.workunit.client.0.smithi032.stdout:9/308: getdents d2/d5 0 2024-04-06T14:17:00.240 INFO:tasks.workunit.client.0.smithi032.stdout:7/314: dwrite d16/d20/f47 [0,4194304] 0 2024-04-06T14:17:00.241 INFO:tasks.workunit.client.1.smithi086.stdout:3/112: dread f4 [0,4194304] 0 2024-04-06T14:17:00.248 INFO:tasks.workunit.client.0.smithi032.stdout:2/269: dread d0/f13 [0,4194304] 0 2024-04-06T14:17:00.248 INFO:tasks.workunit.client.1.smithi086.stdout:3/113: dwrite d5/f11 [0,4194304] 0 2024-04-06T14:17:00.251 INFO:tasks.workunit.client.0.smithi032.stdout:8/318: dwrite d3/d8/d12/f38 [0,4194304] 0 2024-04-06T14:17:00.252 INFO:tasks.workunit.client.1.smithi086.stdout:3/114: chown d5/df/f1c 138 1 2024-04-06T14:17:00.253 INFO:tasks.workunit.client.1.smithi086.stdout:3/115: rmdir d5 39 2024-04-06T14:17:00.253 INFO:tasks.workunit.client.0.smithi032.stdout:6/321: link d4/d5/d28/f5c d4/d5/d34/f5f 0 2024-04-06T14:17:00.255 INFO:tasks.workunit.client.0.smithi032.stdout:7/315: getdents d16/d20/d26 0 2024-04-06T14:17:00.255 INFO:tasks.workunit.client.1.smithi086.stdout:3/116: fdatasync d5/df/f1a 0 2024-04-06T14:17:00.256 INFO:tasks.workunit.client.1.smithi086.stdout:3/117: write d5/f11 [1951342,15486] 0 2024-04-06T14:17:00.257 INFO:tasks.workunit.client.0.smithi032.stdout:9/309: dwrite d2/d5/f44 [0,4194304] 0 2024-04-06T14:17:00.258 INFO:tasks.workunit.client.0.smithi032.stdout:8/319: chown d3/d8/d12/d2f/f77 18639 1 2024-04-06T14:17:00.261 INFO:tasks.workunit.client.1.smithi086.stdout:3/118: link d5/df/f1a d5/df/f1d 0 2024-04-06T14:17:00.262 INFO:tasks.workunit.client.0.smithi032.stdout:9/310: read d2/f2d [384547,110177] 0 2024-04-06T14:17:00.262 INFO:tasks.workunit.client.1.smithi086.stdout:3/119: fdatasync d5/fe 0 2024-04-06T14:17:00.264 INFO:tasks.workunit.client.1.smithi086.stdout:3/120: symlink d5/df/l1e 0 2024-04-06T14:17:00.265 INFO:tasks.workunit.client.0.smithi032.stdout:9/311: truncate d2/d25/f45 4669535 0 2024-04-06T14:17:00.266 INFO:tasks.workunit.client.0.smithi032.stdout:9/312: fsync d2/d25/d31/f40 0 2024-04-06T14:17:00.281 INFO:tasks.workunit.client.1.smithi086.stdout:6/31: sync 2024-04-06T14:17:00.281 INFO:tasks.workunit.client.1.smithi086.stdout:7/31: sync 2024-04-06T14:17:00.285 INFO:tasks.workunit.client.0.smithi032.stdout:2/270: fsync d0/d5a/d3a/f42 0 2024-04-06T14:17:00.286 INFO:tasks.workunit.client.1.smithi086.stdout:7/32: write f5 [542597,118930] 0 2024-04-06T14:17:00.286 INFO:tasks.workunit.client.0.smithi032.stdout:8/320: mkdir d3/d8/d3a/d7b 0 2024-04-06T14:17:00.287 INFO:tasks.workunit.client.1.smithi086.stdout:6/32: creat d6/d9/db/fc x:0 0 0 2024-04-06T14:17:00.289 INFO:tasks.workunit.client.0.smithi032.stdout:7/316: mknod d16/d20/d6e/c6f 0 2024-04-06T14:17:00.290 INFO:tasks.workunit.client.0.smithi032.stdout:2/271: symlink d0/d4e/d52/l60 0 2024-04-06T14:17:00.292 INFO:tasks.workunit.client.0.smithi032.stdout:8/321: mknod d3/d8/d7/c7c 0 2024-04-06T14:17:00.293 INFO:tasks.workunit.client.0.smithi032.stdout:8/322: chown d3/d8/d12/d2f/f62 670 1 2024-04-06T14:17:00.293 INFO:tasks.workunit.client.1.smithi086.stdout:6/33: unlink c0 0 2024-04-06T14:17:00.294 INFO:tasks.workunit.client.0.smithi032.stdout:8/323: readlink d3/d8/d12/d2f/l57 0 2024-04-06T14:17:00.294 INFO:tasks.workunit.client.0.smithi032.stdout:7/317: mkdir d16/d20/d26/d70 0 2024-04-06T14:17:00.295 INFO:tasks.workunit.client.0.smithi032.stdout:8/324: readlink d3/d8/l2b 0 2024-04-06T14:17:00.297 INFO:tasks.workunit.client.0.smithi032.stdout:6/322: getdents d4/d5/d18 0 2024-04-06T14:17:00.297 INFO:tasks.workunit.client.0.smithi032.stdout:8/325: write d3/d8/d12/f4d [4826691,106332] 0 2024-04-06T14:17:00.298 INFO:tasks.workunit.client.1.smithi086.stdout:7/33: dwrite f6 [0,4194304] 0 2024-04-06T14:17:00.300 INFO:tasks.workunit.client.1.smithi086.stdout:6/34: link l2 d6/d9/ld 0 2024-04-06T14:17:00.301 INFO:tasks.workunit.client.1.smithi086.stdout:6/35: write f1 [4928882,22847] 0 2024-04-06T14:17:00.301 INFO:tasks.workunit.client.0.smithi032.stdout:7/318: mkdir d16/d20/d48/d71 0 2024-04-06T14:17:00.302 INFO:tasks.workunit.client.0.smithi032.stdout:7/319: write d16/d20/d48/f4a [687368,98541] 0 2024-04-06T14:17:00.303 INFO:tasks.workunit.client.0.smithi032.stdout:6/323: mkdir d4/d5/d19/d25/d60 0 2024-04-06T14:17:00.304 INFO:tasks.workunit.client.0.smithi032.stdout:6/324: chown d4/d5/d8/d1f/f37 35 1 2024-04-06T14:17:00.305 INFO:tasks.workunit.client.1.smithi086.stdout:6/36: write d6/d9/fa [329305,118683] 0 2024-04-06T14:17:00.305 INFO:tasks.workunit.client.1.smithi086.stdout:7/34: link f6 d7/f9 0 2024-04-06T14:17:00.305 INFO:tasks.workunit.client.0.smithi032.stdout:2/272: creat d0/d5a/d3a/f61 x:0 0 0 2024-04-06T14:17:00.307 INFO:tasks.workunit.client.1.smithi086.stdout:6/37: mknod d6/d9/ce 0 2024-04-06T14:17:00.307 INFO:tasks.workunit.client.0.smithi032.stdout:6/325: write d4/d5/d19/f2e [2352364,102903] 0 2024-04-06T14:17:00.308 INFO:tasks.workunit.client.1.smithi086.stdout:6/38: dread - d6/d9/db/fc zero size 2024-04-06T14:17:00.308 INFO:tasks.workunit.client.0.smithi032.stdout:8/326: rename d3/d8/d12/d2f/d46/c67 to d3/d8/d12/d2f/d4a/d60/d75/c7d 0 2024-04-06T14:17:00.310 INFO:tasks.workunit.client.1.smithi086.stdout:6/39: write d6/d9/db/fc [858191,56714] 0 2024-04-06T14:17:00.312 INFO:tasks.workunit.client.0.smithi032.stdout:8/327: mknod d3/c7e 0 2024-04-06T14:17:00.314 INFO:tasks.workunit.client.0.smithi032.stdout:2/273: truncate d0/d5a/f1c 2995400 0 2024-04-06T14:17:00.317 INFO:tasks.workunit.client.1.smithi086.stdout:3/121: rmdir d5/df 39 2024-04-06T14:17:00.320 INFO:tasks.workunit.client.1.smithi086.stdout:6/40: dread d6/d9/db/fc [0,4194304] 0 2024-04-06T14:17:00.321 INFO:tasks.workunit.client.1.smithi086.stdout:7/35: truncate f6 2787235 0 2024-04-06T14:17:00.321 INFO:tasks.workunit.client.0.smithi032.stdout:9/313: truncate d2/f21 2008978 0 2024-04-06T14:17:00.323 INFO:tasks.workunit.client.1.smithi086.stdout:6/41: symlink d6/d9/lf 0 2024-04-06T14:17:00.327 INFO:tasks.workunit.client.0.smithi032.stdout:7/320: rmdir d16/d20/d26/d2d/d30/d3e 39 2024-04-06T14:17:00.327 INFO:tasks.workunit.client.1.smithi086.stdout:6/42: write f5 [4935733,117372] 0 2024-04-06T14:17:00.327 INFO:tasks.workunit.client.1.smithi086.stdout:6/43: dread d6/d9/db/fc [0,4194304] 0 2024-04-06T14:17:00.327 INFO:tasks.workunit.client.0.smithi032.stdout:7/321: stat d16/d20/d26/d2d/d30/d3e/l6c 0 2024-04-06T14:17:00.328 INFO:tasks.workunit.client.1.smithi086.stdout:6/44: write d6/f7 [434300,41037] 0 2024-04-06T14:17:00.328 INFO:tasks.workunit.client.0.smithi032.stdout:7/322: read d16/d20/f24 [640894,15537] 0 2024-04-06T14:17:00.329 INFO:tasks.workunit.client.0.smithi032.stdout:7/323: readlink d16/d20/l4d 0 2024-04-06T14:17:00.330 INFO:tasks.workunit.client.0.smithi032.stdout:2/274: truncate d0/fb 1243661 0 2024-04-06T14:17:00.330 INFO:tasks.workunit.client.0.smithi032.stdout:7/324: dread - d16/f40 zero size 2024-04-06T14:17:00.331 INFO:tasks.workunit.client.1.smithi086.stdout:6/45: rename d6/d9/fa to d6/d9/f10 0 2024-04-06T14:17:00.332 INFO:tasks.workunit.client.0.smithi032.stdout:2/275: mknod d0/d4e/c62 0 2024-04-06T14:17:00.334 INFO:tasks.workunit.client.1.smithi086.stdout:6/46: rename l3 to d6/l11 0 2024-04-06T14:17:00.334 INFO:tasks.workunit.client.0.smithi032.stdout:2/276: mkdir d0/d4e/d63 0 2024-04-06T14:17:00.338 INFO:tasks.workunit.client.1.smithi086.stdout:6/47: getdents d6/d9/db 0 2024-04-06T14:17:00.338 INFO:tasks.workunit.client.0.smithi032.stdout:7/325: dwrite fb [0,4194304] 0 2024-04-06T14:17:00.340 INFO:tasks.workunit.client.1.smithi086.stdout:6/48: mknod d6/c12 0 2024-04-06T14:17:00.341 INFO:tasks.workunit.client.0.smithi032.stdout:7/326: write d16/d20/f23 [2180108,11500] 0 2024-04-06T14:17:00.341 INFO:tasks.workunit.client.0.smithi032.stdout:9/314: dread d2/d5/f18 [0,4194304] 0 2024-04-06T14:17:00.345 INFO:tasks.workunit.client.0.smithi032.stdout:9/315: write d2/d15/d1e/d47/f4b [372544,77293] 0 2024-04-06T14:17:00.346 INFO:tasks.workunit.client.1.smithi086.stdout:6/49: mkdir d6/d13 0 2024-04-06T14:17:00.351 INFO:tasks.workunit.client.0.smithi032.stdout:7/327: creat d16/d20/d63/f72 x:0 0 0 2024-04-06T14:17:00.354 INFO:tasks.workunit.client.0.smithi032.stdout:9/316: mkdir d2/d15/d1e/d41/d6c 0 2024-04-06T14:17:00.354 INFO:tasks.workunit.client.1.smithi086.stdout:6/50: dwrite d6/d9/db/fc [0,4194304] 0 2024-04-06T14:17:00.354 INFO:tasks.workunit.client.0.smithi032.stdout:2/277: dwrite d0/f23 [0,4194304] 0 2024-04-06T14:17:00.355 INFO:tasks.workunit.client.1.smithi086.stdout:6/51: chown d6/d9/ce 28256756 1 2024-04-06T14:17:00.357 INFO:tasks.workunit.client.1.smithi086.stdout:6/52: mknod d6/d9/db/c14 0 2024-04-06T14:17:00.359 INFO:tasks.workunit.client.0.smithi032.stdout:9/317: write d2/f4 [5964679,8245] 0 2024-04-06T14:17:00.359 INFO:tasks.workunit.client.1.smithi086.stdout:6/53: read d6/d9/f10 [111498,96834] 0 2024-04-06T14:17:00.360 INFO:tasks.workunit.client.1.smithi086.stdout:6/54: chown d6/l8 653 1 2024-04-06T14:17:00.363 INFO:tasks.workunit.client.0.smithi032.stdout:9/318: dread - d2/d5/f3e zero size 2024-04-06T14:17:00.363 INFO:tasks.workunit.client.0.smithi032.stdout:7/328: dwrite d16/d20/d48/f5a [0,4194304] 0 2024-04-06T14:17:00.370 INFO:tasks.workunit.client.1.smithi086.stdout:6/55: dwrite f1 [4194304,4194304] 0 2024-04-06T14:17:00.371 INFO:tasks.workunit.client.0.smithi032.stdout:7/329: write d16/d20/d26/d2d/f64 [134519,109527] 0 2024-04-06T14:17:00.374 INFO:tasks.workunit.client.1.smithi086.stdout:6/56: mknod d6/d9/c15 0 2024-04-06T14:17:00.375 INFO:tasks.workunit.client.1.smithi086.stdout:6/57: stat d6/d9/db/c14 0 2024-04-06T14:17:00.377 INFO:tasks.workunit.client.1.smithi086.stdout:6/58: chown d6/d9/c15 2 1 2024-04-06T14:17:00.380 INFO:tasks.workunit.client.1.smithi086.stdout:6/59: unlink l2 0 2024-04-06T14:17:00.381 INFO:tasks.workunit.client.1.smithi086.stdout:6/60: write f5 [6088619,109430] 0 2024-04-06T14:17:00.384 INFO:tasks.workunit.client.1.smithi086.stdout:6/61: unlink d6/d9/ce 0 2024-04-06T14:17:00.391 INFO:tasks.workunit.client.1.smithi086.stdout:6/62: dwrite d6/f7 [0,4194304] 0 2024-04-06T14:17:00.393 INFO:tasks.workunit.client.1.smithi086.stdout:6/63: creat d6/d9/db/f16 x:0 0 0 2024-04-06T14:17:00.395 INFO:tasks.workunit.client.1.smithi086.stdout:6/64: creat d6/d9/f17 x:0 0 0 2024-04-06T14:17:00.397 INFO:tasks.workunit.client.1.smithi086.stdout:0/101: sync 2024-04-06T14:17:00.397 INFO:tasks.workunit.client.1.smithi086.stdout:2/37: sync 2024-04-06T14:17:00.397 INFO:tasks.workunit.client.1.smithi086.stdout:4/60: sync 2024-04-06T14:17:00.397 INFO:tasks.workunit.client.1.smithi086.stdout:9/120: sync 2024-04-06T14:17:00.398 INFO:tasks.workunit.client.1.smithi086.stdout:0/102: stat d0/d1/d4/d7/f14 0 2024-04-06T14:17:00.400 INFO:tasks.workunit.client.1.smithi086.stdout:9/121: creat d5/d13/f2d x:0 0 0 2024-04-06T14:17:00.402 INFO:tasks.workunit.client.1.smithi086.stdout:4/61: dread d0/f17 [0,4194304] 0 2024-04-06T14:17:00.422 INFO:tasks.workunit.client.1.smithi086.stdout:0/103: mkdir d0/d23 0 2024-04-06T14:17:00.422 INFO:tasks.workunit.client.0.smithi032.stdout:9/319: dwrite d2/d15/f29 [0,4194304] 0 2024-04-06T14:17:00.423 INFO:tasks.workunit.client.0.smithi032.stdout:3/265: sync 2024-04-06T14:17:00.423 INFO:tasks.workunit.client.1.smithi086.stdout:6/65: fdatasync d6/f7 0 2024-04-06T14:17:00.424 INFO:tasks.workunit.client.1.smithi086.stdout:9/122: rename d5/lf to d5/d10/d14/d19/d25/l2e 0 2024-04-06T14:17:00.425 INFO:tasks.workunit.client.0.smithi032.stdout:9/320: stat d2/d15/d1e/d47/d4a 0 2024-04-06T14:17:00.428 INFO:tasks.workunit.client.1.smithi086.stdout:4/62: unlink d0/f8 0 2024-04-06T14:17:00.428 INFO:tasks.workunit.client.0.smithi032.stdout:9/321: creat d2/d15/d1e/f6d x:0 0 0 2024-04-06T14:17:00.432 INFO:tasks.workunit.client.0.smithi032.stdout:2/278: dread d0/d5a/f2c [0,4194304] 0 2024-04-06T14:17:00.433 INFO:tasks.workunit.client.0.smithi032.stdout:7/330: getdents d16/d20/d26/d2d/d30/d3e 0 2024-04-06T14:17:00.435 INFO:tasks.workunit.client.1.smithi086.stdout:9/123: dread d5/f20 [4194304,4194304] 0 2024-04-06T14:17:00.436 INFO:tasks.workunit.client.0.smithi032.stdout:9/322: creat d2/d15/d55/f6e x:0 0 0 2024-04-06T14:17:00.437 INFO:tasks.workunit.client.1.smithi086.stdout:0/104: creat d0/d1/d4/de/f24 x:0 0 0 2024-04-06T14:17:00.437 INFO:tasks.workunit.client.0.smithi032.stdout:3/266: dwrite d3/d3b/d3c/f77 [0,4194304] 0 2024-04-06T14:17:00.439 INFO:tasks.workunit.client.1.smithi086.stdout:0/105: write d0/d1/d4/de/f24 [314746,15189] 0 2024-04-06T14:17:00.440 INFO:tasks.workunit.client.0.smithi032.stdout:3/267: fdatasync d3/d16/f68 0 2024-04-06T14:17:00.440 INFO:tasks.workunit.client.1.smithi086.stdout:6/66: rmdir d6/d9 39 2024-04-06T14:17:00.441 INFO:tasks.workunit.client.1.smithi086.stdout:0/106: write d0/d1/d2/d1a/f1d [802076,32012] 0 2024-04-06T14:17:00.445 INFO:tasks.workunit.client.0.smithi032.stdout:3/268: dread d3/f5e [0,4194304] 0 2024-04-06T14:17:00.446 INFO:tasks.workunit.client.0.smithi032.stdout:9/323: fdatasync d2/d15/d1e/f2a 0 2024-04-06T14:17:00.446 INFO:tasks.workunit.client.1.smithi086.stdout:6/67: chown f1 39310 1 2024-04-06T14:17:00.446 INFO:tasks.workunit.client.1.smithi086.stdout:0/107: write d0/d1/d2/d1a/f1d [1373676,33767] 0 2024-04-06T14:17:00.446 INFO:tasks.workunit.client.1.smithi086.stdout:9/124: dread d5/fc [0,4194304] 0 2024-04-06T14:17:00.447 INFO:tasks.workunit.client.0.smithi032.stdout:2/279: getdents d0/d4e/d52 0 2024-04-06T14:17:00.448 INFO:tasks.workunit.client.0.smithi032.stdout:7/331: link d16/d20/d26/d2d/d30/f53 d16/d20/d26/d2d/d30/d3e/f73 0 2024-04-06T14:17:00.452 INFO:tasks.workunit.client.0.smithi032.stdout:7/332: creat d16/d20/d48/d61/f74 x:0 0 0 2024-04-06T14:17:00.453 INFO:tasks.workunit.client.0.smithi032.stdout:2/280: symlink d0/d5a/d3a/l64 0 2024-04-06T14:17:00.453 INFO:tasks.workunit.client.1.smithi086.stdout:9/125: creat d5/d10/d14/d19/d25/f2f x:0 0 0 2024-04-06T14:17:00.454 INFO:tasks.workunit.client.0.smithi032.stdout:3/269: getdents d3/d3b 0 2024-04-06T14:17:00.455 INFO:tasks.workunit.client.1.smithi086.stdout:2/38: dwrite d0/f4 [0,4194304] 0 2024-04-06T14:17:00.456 INFO:tasks.workunit.client.1.smithi086.stdout:0/108: mkdir d0/d1/d25 0 2024-04-06T14:17:00.456 INFO:tasks.workunit.client.1.smithi086.stdout:2/39: write d0/fd [36031,40498] 0 2024-04-06T14:17:00.459 INFO:tasks.workunit.client.0.smithi032.stdout:2/281: mkdir d0/d4e/d52/d65 0 2024-04-06T14:17:00.460 INFO:tasks.workunit.client.0.smithi032.stdout:7/333: rename d16/d20/d48/d61/f74 to d16/d20/d26/d2d/f75 0 2024-04-06T14:17:00.461 INFO:tasks.workunit.client.0.smithi032.stdout:9/324: dwrite d2/fd [0,4194304] 0 2024-04-06T14:17:00.462 INFO:tasks.workunit.client.0.smithi032.stdout:7/334: fsync d16/f40 0 2024-04-06T14:17:00.462 INFO:tasks.workunit.client.0.smithi032.stdout:3/270: creat d3/d3b/d3c/f78 x:0 0 0 2024-04-06T14:17:00.462 INFO:tasks.workunit.client.1.smithi086.stdout:9/126: dwrite d5/d18/f26 [0,4194304] 0 2024-04-06T14:17:00.464 INFO:tasks.workunit.client.0.smithi032.stdout:9/325: chown d2/d5/f6 0 1 2024-04-06T14:17:00.464 INFO:tasks.workunit.client.0.smithi032.stdout:2/282: dread - d0/d5a/d3a/f40 zero size 2024-04-06T14:17:00.465 INFO:tasks.workunit.client.1.smithi086.stdout:4/63: write d0/f7 [1367151,69894] 0 2024-04-06T14:17:00.467 INFO:tasks.workunit.client.1.smithi086.stdout:2/40: readlink d0/lb 0 2024-04-06T14:17:00.474 INFO:tasks.workunit.client.0.smithi032.stdout:9/326: chown d2/d5/c50 27685853 1 2024-04-06T14:17:00.474 INFO:tasks.workunit.client.0.smithi032.stdout:7/335: truncate d16/d20/f24 4270925 0 2024-04-06T14:17:00.474 INFO:tasks.workunit.client.0.smithi032.stdout:3/271: rename d3/f6a to d3/d3b/d3c/f79 0 2024-04-06T14:17:00.475 INFO:tasks.workunit.client.0.smithi032.stdout:2/283: mknod d0/d30/d39/c66 0 2024-04-06T14:17:00.476 INFO:tasks.workunit.client.1.smithi086.stdout:6/68: truncate f1 8340855 0 2024-04-06T14:17:00.477 INFO:tasks.workunit.client.0.smithi032.stdout:2/284: fsync d0/d5a/d3a/f40 0 2024-04-06T14:17:00.478 INFO:tasks.workunit.client.1.smithi086.stdout:2/41: dwrite d0/f6 [0,4194304] 0 2024-04-06T14:17:00.478 INFO:tasks.workunit.client.1.smithi086.stdout:4/64: readlink d0/l9 0 2024-04-06T14:17:00.480 INFO:tasks.workunit.client.0.smithi032.stdout:3/272: creat d3/f7a x:0 0 0 2024-04-06T14:17:00.485 INFO:tasks.workunit.client.0.smithi032.stdout:7/336: dwrite d16/d20/d26/d2d/f69 [0,4194304] 0 2024-04-06T14:17:00.485 INFO:tasks.workunit.client.0.smithi032.stdout:2/285: unlink d0/d5a/l14 0 2024-04-06T14:17:00.487 INFO:tasks.workunit.client.0.smithi032.stdout:3/273: symlink d3/d3b/d3c/l7b 0 2024-04-06T14:17:00.487 INFO:tasks.workunit.client.1.smithi086.stdout:2/42: rename d0/l8 to d0/df/l10 0 2024-04-06T14:17:00.487 INFO:tasks.workunit.client.1.smithi086.stdout:4/65: mkdir d0/d15/d19 0 2024-04-06T14:17:00.488 INFO:tasks.workunit.client.0.smithi032.stdout:7/337: creat d16/d20/d54/f76 x:0 0 0 2024-04-06T14:17:00.488 INFO:tasks.workunit.client.1.smithi086.stdout:4/66: fsync d0/f10 0 2024-04-06T14:17:00.490 INFO:tasks.workunit.client.0.smithi032.stdout:7/338: stat d16/d20/d26/d2d/f75 0 2024-04-06T14:17:00.492 INFO:tasks.workunit.client.1.smithi086.stdout:2/43: symlink d0/df/l11 0 2024-04-06T14:17:00.492 INFO:tasks.workunit.client.0.smithi032.stdout:2/286: mkdir d0/d30/d39/d59/d67 0 2024-04-06T14:17:00.493 INFO:tasks.workunit.client.0.smithi032.stdout:9/327: dwrite d2/f13 [0,4194304] 0 2024-04-06T14:17:00.494 INFO:tasks.workunit.client.0.smithi032.stdout:7/339: truncate d16/f49 858895 0 2024-04-06T14:17:00.499 INFO:tasks.workunit.client.0.smithi032.stdout:3/274: dwrite d3/d65/f69 [0,4194304] 0 2024-04-06T14:17:00.499 INFO:tasks.workunit.client.0.smithi032.stdout:9/328: mknod d2/d25/c6f 0 2024-04-06T14:17:00.499 INFO:tasks.workunit.client.0.smithi032.stdout:7/340: symlink d16/d20/d63/l77 0 2024-04-06T14:17:00.500 INFO:tasks.workunit.client.1.smithi086.stdout:2/44: read d0/f6 [2807316,73349] 0 2024-04-06T14:17:00.502 INFO:tasks.workunit.client.0.smithi032.stdout:2/287: symlink d0/d5a/l68 0 2024-04-06T14:17:00.503 INFO:tasks.workunit.client.0.smithi032.stdout:9/329: truncate d2/d15/f27 1420538 0 2024-04-06T14:17:00.507 INFO:tasks.workunit.client.0.smithi032.stdout:7/341: symlink d16/d20/l78 0 2024-04-06T14:17:00.508 INFO:tasks.workunit.client.0.smithi032.stdout:3/275: rename d3/l64 to d3/d65/l7c 0 2024-04-06T14:17:00.508 INFO:tasks.workunit.client.0.smithi032.stdout:2/288: chown d0/fb 17265215 1 2024-04-06T14:17:00.509 INFO:tasks.workunit.client.1.smithi086.stdout:6/69: dwrite d6/d9/f10 [0,4194304] 0 2024-04-06T14:17:00.511 INFO:tasks.workunit.client.0.smithi032.stdout:3/276: write d3/d16/f70 [315810,79515] 0 2024-04-06T14:17:00.511 INFO:tasks.workunit.client.1.smithi086.stdout:9/127: dwrite d5/f20 [0,4194304] 0 2024-04-06T14:17:00.511 INFO:tasks.workunit.client.1.smithi086.stdout:0/109: dwrite d0/d1/d4/d7/f11 [0,4194304] 0 2024-04-06T14:17:00.513 INFO:tasks.workunit.client.0.smithi032.stdout:2/289: rmdir d0/d30 39 2024-04-06T14:17:00.515 INFO:tasks.workunit.client.0.smithi032.stdout:3/277: write d3/d16/f6f [168686,11715] 0 2024-04-06T14:17:00.518 INFO:tasks.workunit.client.0.smithi032.stdout:9/330: dwrite d2/d15/d55/d59/f67 [0,4194304] 0 2024-04-06T14:17:00.518 INFO:tasks.workunit.client.0.smithi032.stdout:7/342: creat d16/d20/d48/d6d/f79 x:0 0 0 2024-04-06T14:17:00.518 INFO:tasks.workunit.client.1.smithi086.stdout:0/110: mkdir d0/d15/d26 0 2024-04-06T14:17:00.519 INFO:tasks.workunit.client.1.smithi086.stdout:4/67: dwrite d0/de/f11 [4194304,4194304] 0 2024-04-06T14:17:00.520 INFO:tasks.workunit.client.0.smithi032.stdout:3/278: stat d3/d65/c2f 0 2024-04-06T14:17:00.521 INFO:tasks.workunit.client.1.smithi086.stdout:6/70: mknod d6/c18 0 2024-04-06T14:17:00.522 INFO:tasks.workunit.client.1.smithi086.stdout:8/167: sync 2024-04-06T14:17:00.522 INFO:tasks.workunit.client.1.smithi086.stdout:5/109: sync 2024-04-06T14:17:00.523 INFO:tasks.workunit.client.1.smithi086.stdout:9/128: mknod d5/d10/d14/d19/c30 0 2024-04-06T14:17:00.530 INFO:tasks.workunit.client.0.smithi032.stdout:9/331: stat d2/c3d 0 2024-04-06T14:17:00.530 INFO:tasks.workunit.client.0.smithi032.stdout:7/343: write d16/d20/d48/d6d/f79 [1032738,125840] 0 2024-04-06T14:17:00.531 INFO:tasks.workunit.client.1.smithi086.stdout:5/110: write d0/d16/d18/d21/d23/f20 [309812,23485] 0 2024-04-06T14:17:00.531 INFO:tasks.workunit.client.0.smithi032.stdout:3/279: mkdir d3/d65/d7d 0 2024-04-06T14:17:00.532 INFO:tasks.workunit.client.1.smithi086.stdout:2/45: getdents d0 0 2024-04-06T14:17:00.533 INFO:tasks.workunit.client.1.smithi086.stdout:4/68: dwrite d0/f7 [0,4194304] 0 2024-04-06T14:17:00.535 INFO:tasks.workunit.client.1.smithi086.stdout:0/111: dwrite d0/d1/d4/f22 [0,4194304] 0 2024-04-06T14:17:00.538 INFO:tasks.workunit.client.0.smithi032.stdout:7/344: unlink d16/d20/d26/d2d/l68 0 2024-04-06T14:17:00.538 INFO:tasks.workunit.client.1.smithi086.stdout:4/69: write d0/f17 [1466403,19204] 0 2024-04-06T14:17:00.539 INFO:tasks.workunit.client.1.smithi086.stdout:5/111: mknod d0/d16/d18/c25 0 2024-04-06T14:17:00.544 INFO:tasks.workunit.client.0.smithi032.stdout:3/280: dread d3/d16/f55 [0,4194304] 0 2024-04-06T14:17:00.544 INFO:tasks.workunit.client.1.smithi086.stdout:2/46: truncate d0/fe 1047976 0 2024-04-06T14:17:00.545 INFO:tasks.workunit.client.1.smithi086.stdout:1/55: sync 2024-04-06T14:17:00.545 INFO:tasks.workunit.client.1.smithi086.stdout:0/112: creat d0/d1/d4/d7/f27 x:0 0 0 2024-04-06T14:17:00.546 INFO:tasks.workunit.client.1.smithi086.stdout:6/71: getdents d6/d9/db 0 2024-04-06T14:17:00.548 INFO:tasks.workunit.client.1.smithi086.stdout:4/70: creat d0/de/f1a x:0 0 0 2024-04-06T14:17:00.548 INFO:tasks.workunit.client.1.smithi086.stdout:4/71: chown d0 208587078 1 2024-04-06T14:17:00.549 INFO:tasks.workunit.client.1.smithi086.stdout:7/36: sync 2024-04-06T14:17:00.549 INFO:tasks.workunit.client.1.smithi086.stdout:3/122: sync 2024-04-06T14:17:00.549 INFO:tasks.workunit.client.0.smithi032.stdout:7/345: rmdir d16/d20/d26 39 2024-04-06T14:17:00.549 INFO:tasks.workunit.client.1.smithi086.stdout:4/72: readlink d0/de/l14 0 2024-04-06T14:17:00.550 INFO:tasks.workunit.client.0.smithi032.stdout:9/332: getdents d2/d15 0 2024-04-06T14:17:00.553 INFO:tasks.workunit.client.0.smithi032.stdout:9/333: write d2/d15/d55/d5f/f62 [432813,59397] 0 2024-04-06T14:17:00.553 INFO:tasks.workunit.client.1.smithi086.stdout:2/47: unlink d0/f4 0 2024-04-06T14:17:00.554 INFO:tasks.workunit.client.1.smithi086.stdout:1/56: dwrite db/d11/f12 [0,4194304] 0 2024-04-06T14:17:00.555 INFO:tasks.workunit.client.1.smithi086.stdout:1/57: write f7 [3136630,40151] 0 2024-04-06T14:17:00.556 INFO:tasks.workunit.client.1.smithi086.stdout:1/58: dread - db/fe zero size 2024-04-06T14:17:00.557 INFO:tasks.workunit.client.1.smithi086.stdout:1/59: truncate fa 444042 0 2024-04-06T14:17:00.558 INFO:tasks.workunit.client.1.smithi086.stdout:0/113: symlink d0/l28 0 2024-04-06T14:17:00.560 INFO:tasks.workunit.client.1.smithi086.stdout:0/114: write d0/d1/d2/d1a/f1d [391557,72322] 0 2024-04-06T14:17:00.560 INFO:tasks.workunit.client.1.smithi086.stdout:3/123: dread - d5/df/f1d zero size 2024-04-06T14:17:00.560 INFO:tasks.workunit.client.1.smithi086.stdout:8/168: write d0/db/d32/f3e [4210287,49196] 0 2024-04-06T14:17:00.561 INFO:tasks.workunit.client.1.smithi086.stdout:4/73: symlink d0/l1b 0 2024-04-06T14:17:00.563 INFO:tasks.workunit.client.1.smithi086.stdout:8/169: write d0/d8/d16/d39/f3a [519296,20303] 0 2024-04-06T14:17:00.564 INFO:tasks.workunit.client.0.smithi032.stdout:3/281: link d3/d65/f47 d3/f7e 0 2024-04-06T14:17:00.565 INFO:tasks.workunit.client.1.smithi086.stdout:2/48: creat d0/f12 x:0 0 0 2024-04-06T14:17:00.566 INFO:tasks.workunit.client.0.smithi032.stdout:7/346: dwrite d16/d20/d48/f4a [0,4194304] 0 2024-04-06T14:17:00.568 INFO:tasks.workunit.client.1.smithi086.stdout:9/129: getdents d5/d10/d14/d19/d1b 0 2024-04-06T14:17:00.569 INFO:tasks.workunit.client.1.smithi086.stdout:1/60: unlink l5 0 2024-04-06T14:17:00.570 INFO:tasks.workunit.client.0.smithi032.stdout:2/290: dread d0/d30/d39/d59/f5f [0,4194304] 0 2024-04-06T14:17:00.571 INFO:tasks.workunit.client.0.smithi032.stdout:0/264: sync 2024-04-06T14:17:00.571 INFO:tasks.workunit.client.0.smithi032.stdout:1/350: sync 2024-04-06T14:17:00.571 INFO:tasks.workunit.client.0.smithi032.stdout:4/423: sync 2024-04-06T14:17:00.571 INFO:tasks.workunit.client.1.smithi086.stdout:7/37: mkdir d7/da 0 2024-04-06T14:17:00.572 INFO:tasks.workunit.client.1.smithi086.stdout:3/124: dwrite d5/df/f13 [0,4194304] 0 2024-04-06T14:17:00.573 INFO:tasks.workunit.client.1.smithi086.stdout:3/125: dread - d5/df/f1d zero size 2024-04-06T14:17:00.576 INFO:tasks.workunit.client.1.smithi086.stdout:4/74: rmdir d0/d15 39 2024-04-06T14:17:00.578 INFO:tasks.workunit.client.0.smithi032.stdout:3/282: write d3/f5e [1416748,110970] 0 2024-04-06T14:17:00.581 INFO:tasks.workunit.client.0.smithi032.stdout:9/334: dwrite d2/d15/f43 [0,4194304] 0 2024-04-06T14:17:00.582 INFO:tasks.workunit.client.1.smithi086.stdout:2/49: creat d0/df/f13 x:0 0 0 2024-04-06T14:17:00.583 INFO:tasks.workunit.client.1.smithi086.stdout:6/72: rename f1 to d6/d9/db/f19 0 2024-04-06T14:17:00.583 INFO:tasks.workunit.client.0.smithi032.stdout:7/347: write d16/d20/d26/d2d/d30/f38 [1556320,63527] 0 2024-04-06T14:17:00.583 INFO:tasks.workunit.client.0.smithi032.stdout:9/335: chown d2/d25/f45 1468453 1 2024-04-06T14:17:00.584 INFO:tasks.workunit.client.1.smithi086.stdout:1/61: mkdir db/d13 0 2024-04-06T14:17:00.585 INFO:tasks.workunit.client.0.smithi032.stdout:2/291: creat d0/d4e/f69 x:0 0 0 2024-04-06T14:17:00.585 INFO:tasks.workunit.client.0.smithi032.stdout:7/348: fsync d16/d20/d26/d2d/f3f 0 2024-04-06T14:17:00.586 INFO:tasks.workunit.client.1.smithi086.stdout:9/130: dread f2 [0,4194304] 0 2024-04-06T14:17:00.586 INFO:tasks.workunit.client.1.smithi086.stdout:5/112: dwrite d0/f1 [0,4194304] 0 2024-04-06T14:17:00.587 INFO:tasks.workunit.client.0.smithi032.stdout:9/336: write d2/d15/d1e/d47/d4a/f4d [1477056,37539] 0 2024-04-06T14:17:00.590 INFO:tasks.workunit.client.1.smithi086.stdout:5/113: chown d0 22 1 2024-04-06T14:17:00.592 INFO:tasks.workunit.client.0.smithi032.stdout:9/337: mknod d2/d15/c70 0 2024-04-06T14:17:00.593 INFO:tasks.workunit.client.1.smithi086.stdout:3/126: mknod d5/c1f 0 2024-04-06T14:17:00.595 INFO:tasks.workunit.client.1.smithi086.stdout:0/115: getdents d0/d1/d2/d1a 0 2024-04-06T14:17:00.595 INFO:tasks.workunit.client.1.smithi086.stdout:3/127: write d5/df/f1c [389018,67541] 0 2024-04-06T14:17:00.595 INFO:tasks.workunit.client.1.smithi086.stdout:6/73: mkdir d6/d13/d1a 0 2024-04-06T14:17:00.595 INFO:tasks.workunit.client.1.smithi086.stdout:7/38: dwrite f5 [0,4194304] 0 2024-04-06T14:17:00.596 INFO:tasks.workunit.client.1.smithi086.stdout:7/39: readlink - no filename 2024-04-06T14:17:00.597 INFO:tasks.workunit.client.1.smithi086.stdout:5/114: creat d0/d19/f26 x:0 0 0 2024-04-06T14:17:00.597 INFO:tasks.workunit.client.1.smithi086.stdout:7/40: chown f5 29 1 2024-04-06T14:17:00.598 INFO:tasks.workunit.client.1.smithi086.stdout:6/74: dread - d6/d9/db/f16 zero size 2024-04-06T14:17:00.598 INFO:tasks.workunit.client.1.smithi086.stdout:3/128: rmdir d5/df 39 2024-04-06T14:17:00.600 INFO:tasks.workunit.client.0.smithi032.stdout:7/349: dwrite d16/d20/d54/f76 [0,4194304] 0 2024-04-06T14:17:00.600 INFO:tasks.workunit.client.0.smithi032.stdout:7/350: chown d16/d20/d26 19 1 2024-04-06T14:17:00.601 INFO:tasks.workunit.client.1.smithi086.stdout:4/75: creat d0/d15/d19/f1c x:0 0 0 2024-04-06T14:17:00.601 INFO:tasks.workunit.client.0.smithi032.stdout:3/283: link d3/d3b/d3c/c6e d3/d3b/c7f 0 2024-04-06T14:17:00.603 INFO:tasks.workunit.client.0.smithi032.stdout:3/284: rmdir d3/d65 39 2024-04-06T14:17:00.604 INFO:tasks.workunit.client.0.smithi032.stdout:7/351: chown d16/d20/d26/d2d/d30/d3e 44 1 2024-04-06T14:17:00.605 INFO:tasks.workunit.client.1.smithi086.stdout:0/116: mkdir d0/d15/d26/d29 0 2024-04-06T14:17:00.606 INFO:tasks.workunit.client.0.smithi032.stdout:7/352: readlink d16/d20/d48/l6a 0 2024-04-06T14:17:00.609 INFO:tasks.workunit.client.1.smithi086.stdout:5/115: symlink d0/d16/d18/l27 0 2024-04-06T14:17:00.609 INFO:tasks.workunit.client.0.smithi032.stdout:1/351: truncate d0/d10/d16/f6d 4173177 0 2024-04-06T14:17:00.610 INFO:tasks.workunit.client.1.smithi086.stdout:7/41: fdatasync f6 0 2024-04-06T14:17:00.612 INFO:tasks.workunit.client.1.smithi086.stdout:6/75: link d6/f7 d6/d13/d1a/f1b 0 2024-04-06T14:17:00.613 INFO:tasks.workunit.client.1.smithi086.stdout:9/131: link d5/d10/d14/d19/c30 d5/d10/d14/d19/c31 0 2024-04-06T14:17:00.614 INFO:tasks.workunit.client.1.smithi086.stdout:0/117: mknod d0/d15/d26/c2a 0 2024-04-06T14:17:00.614 INFO:tasks.workunit.client.0.smithi032.stdout:2/292: dread d0/d30/f43 [0,4194304] 0 2024-04-06T14:17:00.615 INFO:tasks.workunit.client.1.smithi086.stdout:3/129: creat d5/f20 x:0 0 0 2024-04-06T14:17:00.616 INFO:tasks.workunit.client.1.smithi086.stdout:7/42: dread f5 [0,4194304] 0 2024-04-06T14:17:00.616 INFO:tasks.workunit.client.0.smithi032.stdout:7/353: mkdir d16/d20/d7a 0 2024-04-06T14:17:00.617 INFO:tasks.workunit.client.1.smithi086.stdout:5/116: unlink d0/le 0 2024-04-06T14:17:00.618 INFO:tasks.workunit.client.0.smithi032.stdout:9/338: fdatasync d2/d15/d1e/d47/f5d 0 2024-04-06T14:17:00.620 INFO:tasks.workunit.client.0.smithi032.stdout:2/293: write d0/d5a/d3a/f48 [833061,117801] 0 2024-04-06T14:17:00.622 INFO:tasks.workunit.client.1.smithi086.stdout:4/76: creat d0/f1d x:0 0 0 2024-04-06T14:17:00.623 INFO:tasks.workunit.client.1.smithi086.stdout:2/50: getdents d0/df 0 2024-04-06T14:17:00.624 INFO:tasks.workunit.client.0.smithi032.stdout:3/285: dwrite d3/d16/f2a [0,4194304] 0 2024-04-06T14:17:00.624 INFO:tasks.workunit.client.1.smithi086.stdout:9/132: write d5/f20 [4780266,42760] 0 2024-04-06T14:17:00.625 INFO:tasks.workunit.client.1.smithi086.stdout:2/51: stat d0/f12 0 2024-04-06T14:17:00.625 INFO:tasks.workunit.client.1.smithi086.stdout:9/133: chown f2 249 1 2024-04-06T14:17:00.625 INFO:tasks.workunit.client.0.smithi032.stdout:7/354: rename d16/d20/d26/c4c to d16/d20/d48/d61/c7b 0 2024-04-06T14:17:00.625 INFO:tasks.workunit.client.0.smithi032.stdout:3/286: write d3/d16/f35 [357037,107766] 0 2024-04-06T14:17:00.626 INFO:tasks.workunit.client.0.smithi032.stdout:9/339: write d2/d15/d55/d5f/f62 [1381995,83288] 0 2024-04-06T14:17:00.627 INFO:tasks.workunit.client.0.smithi032.stdout:3/287: chown d3/fc 3 1 2024-04-06T14:17:00.628 INFO:tasks.workunit.client.0.smithi032.stdout:2/294: mkdir d0/d5a/d6a 0 2024-04-06T14:17:00.630 INFO:tasks.workunit.client.1.smithi086.stdout:0/118: rename d0/l28 to d0/d15/d26/l2b 0 2024-04-06T14:17:00.632 INFO:tasks.workunit.client.1.smithi086.stdout:7/43: fdatasync f6 0 2024-04-06T14:17:00.633 INFO:tasks.workunit.client.0.smithi032.stdout:9/340: mkdir d2/d15/d55/d5f/d71 0 2024-04-06T14:17:00.633 INFO:tasks.workunit.client.0.smithi032.stdout:1/352: truncate d0/f17 2851980 0 2024-04-06T14:17:00.633 INFO:tasks.workunit.client.1.smithi086.stdout:0/119: chown d0/d1/d4/de/f24 5 1 2024-04-06T14:17:00.634 INFO:tasks.workunit.client.1.smithi086.stdout:1/62: truncate f7 135315 0 2024-04-06T14:17:00.635 INFO:tasks.workunit.client.1.smithi086.stdout:5/117: read d0/f6 [748241,108138] 0 2024-04-06T14:17:00.636 INFO:tasks.workunit.client.0.smithi032.stdout:2/295: mknod d0/d5a/d3a/c6b 0 2024-04-06T14:17:00.636 INFO:tasks.workunit.client.1.smithi086.stdout:2/52: dwrite d0/df/f13 [0,4194304] 0 2024-04-06T14:17:00.636 INFO:tasks.workunit.client.0.smithi032.stdout:2/296: fdatasync d0/d5a/f41 0 2024-04-06T14:17:00.638 INFO:tasks.workunit.client.0.smithi032.stdout:9/341: fsync d2/d15/d1e/f2a 0 2024-04-06T14:17:00.640 INFO:tasks.workunit.client.1.smithi086.stdout:9/134: creat d5/d18/f32 x:0 0 0 2024-04-06T14:17:00.641 INFO:tasks.workunit.client.1.smithi086.stdout:3/130: mknod d5/c21 0 2024-04-06T14:17:00.645 INFO:tasks.workunit.client.1.smithi086.stdout:1/63: dwrite f8 [0,4194304] 0 2024-04-06T14:17:00.647 INFO:tasks.workunit.client.1.smithi086.stdout:4/77: mknod d0/c1e 0 2024-04-06T14:17:00.648 INFO:tasks.workunit.client.1.smithi086.stdout:5/118: dread d0/d16/d18/d21/d23/f20 [0,4194304] 0 2024-04-06T14:17:00.649 INFO:tasks.workunit.client.1.smithi086.stdout:4/78: truncate d0/fd 484338 0 2024-04-06T14:17:00.649 INFO:tasks.workunit.client.0.smithi032.stdout:9/342: rmdir d2/d5 39 2024-04-06T14:17:00.650 INFO:tasks.workunit.client.1.smithi086.stdout:4/79: chown d0/f17 26 1 2024-04-06T14:17:00.651 INFO:tasks.workunit.client.1.smithi086.stdout:5/119: fsync d0/d16/d18/d21/d23/f20 0 2024-04-06T14:17:00.651 INFO:tasks.workunit.client.1.smithi086.stdout:2/53: creat d0/df/f14 x:0 0 0 2024-04-06T14:17:00.651 INFO:tasks.workunit.client.0.smithi032.stdout:3/288: truncate d3/f43 2695066 0 2024-04-06T14:17:00.652 INFO:tasks.workunit.client.0.smithi032.stdout:2/297: dwrite d0/f28 [0,4194304] 0 2024-04-06T14:17:00.655 INFO:tasks.workunit.client.1.smithi086.stdout:7/44: link f6 d7/fb 0 2024-04-06T14:17:00.656 INFO:tasks.workunit.client.1.smithi086.stdout:7/45: chown f6 60104712 1 2024-04-06T14:17:00.659 INFO:tasks.workunit.client.0.smithi032.stdout:1/353: rmdir d0/d41/d71 0 2024-04-06T14:17:00.660 INFO:tasks.workunit.client.0.smithi032.stdout:1/354: chown d0/d9/d1f/l4e 105481388 1 2024-04-06T14:17:00.660 INFO:tasks.workunit.client.1.smithi086.stdout:9/135: symlink d5/d13/d1a/l33 0 2024-04-06T14:17:00.660 INFO:tasks.workunit.client.1.smithi086.stdout:6/76: truncate f5 5204344 0 2024-04-06T14:17:00.661 INFO:tasks.workunit.client.1.smithi086.stdout:4/80: symlink d0/d15/d19/l1f 0 2024-04-06T14:17:00.662 INFO:tasks.workunit.client.1.smithi086.stdout:2/54: creat d0/f15 x:0 0 0 2024-04-06T14:17:00.663 INFO:tasks.workunit.client.0.smithi032.stdout:3/289: truncate d3/f2d 738451 0 2024-04-06T14:17:00.665 INFO:tasks.workunit.client.0.smithi032.stdout:7/355: getdents d16/d20/d26/d5f 0 2024-04-06T14:17:00.665 INFO:tasks.workunit.client.0.smithi032.stdout:1/355: write d0/d9/d1f/f3a [1346855,51564] 0 2024-04-06T14:17:00.665 INFO:tasks.workunit.client.1.smithi086.stdout:3/131: mknod d5/c22 0 2024-04-06T14:17:00.665 INFO:tasks.workunit.client.1.smithi086.stdout:4/81: dread d0/fd [0,4194304] 0 2024-04-06T14:17:00.665 INFO:tasks.workunit.client.1.smithi086.stdout:4/82: stat d0/d15/d19/l1f 0 2024-04-06T14:17:00.667 INFO:tasks.workunit.client.0.smithi032.stdout:9/343: chown d2/d15/d1e/d47/d4a/l5e 3 1 2024-04-06T14:17:00.667 INFO:tasks.workunit.client.1.smithi086.stdout:4/83: chown d0/de/lf 38570797 1 2024-04-06T14:17:00.667 INFO:tasks.workunit.client.1.smithi086.stdout:9/136: symlink d5/d13/l34 0 2024-04-06T14:17:00.668 INFO:tasks.workunit.client.0.smithi032.stdout:2/298: creat d0/f6c x:0 0 0 2024-04-06T14:17:00.669 INFO:tasks.workunit.client.1.smithi086.stdout:2/55: mknod d0/c16 0 2024-04-06T14:17:00.670 INFO:tasks.workunit.client.0.smithi032.stdout:7/356: write d16/d20/d54/f57 [1598078,113568] 0 2024-04-06T14:17:00.672 INFO:tasks.workunit.client.1.smithi086.stdout:7/46: rename f6 to d7/da/fc 0 2024-04-06T14:17:00.672 INFO:tasks.workunit.client.1.smithi086.stdout:6/77: rename d6/d9 to d6/d9/db/d1c 22 2024-04-06T14:17:00.673 INFO:tasks.workunit.client.1.smithi086.stdout:4/84: dread - d0/fa zero size 2024-04-06T14:17:00.673 INFO:tasks.workunit.client.0.smithi032.stdout:3/290: creat d3/f80 x:0 0 0 2024-04-06T14:17:00.674 INFO:tasks.workunit.client.0.smithi032.stdout:2/299: mknod d0/d5a/c6d 0 2024-04-06T14:17:00.674 INFO:tasks.workunit.client.1.smithi086.stdout:9/137: creat d5/d13/f35 x:0 0 0 2024-04-06T14:17:00.675 INFO:tasks.workunit.client.1.smithi086.stdout:9/138: stat d5/d13/f2b 0 2024-04-06T14:17:00.676 INFO:tasks.workunit.client.0.smithi032.stdout:3/291: chown d3/f80 4 1 2024-04-06T14:17:00.676 INFO:tasks.workunit.client.0.smithi032.stdout:9/344: truncate d2/d25/d31/f3a 842375 0 2024-04-06T14:17:00.677 INFO:tasks.workunit.client.1.smithi086.stdout:7/47: dread d7/fb [0,4194304] 0 2024-04-06T14:17:00.678 INFO:tasks.workunit.client.1.smithi086.stdout:2/56: rmdir d0/df 39 2024-04-06T14:17:00.679 INFO:tasks.workunit.client.1.smithi086.stdout:9/139: dread f2 [4194304,4194304] 0 2024-04-06T14:17:00.679 INFO:tasks.workunit.client.0.smithi032.stdout:3/292: fdatasync d3/d3b/f40 0 2024-04-06T14:17:00.680 INFO:tasks.workunit.client.0.smithi032.stdout:1/356: dwrite d0/d41/d59/f60 [0,4194304] 0 2024-04-06T14:17:00.681 INFO:tasks.workunit.client.1.smithi086.stdout:9/140: dread - d5/d13/f35 zero size 2024-04-06T14:17:00.681 INFO:tasks.workunit.client.0.smithi032.stdout:7/357: dread f11 [0,4194304] 0 2024-04-06T14:17:00.682 INFO:tasks.workunit.client.0.smithi032.stdout:9/345: readlink d2/d5/l35 0 2024-04-06T14:17:00.683 INFO:tasks.workunit.client.0.smithi032.stdout:2/300: dread d0/f23 [4194304,4194304] 0 2024-04-06T14:17:00.684 INFO:tasks.workunit.client.0.smithi032.stdout:1/357: write d0/d9/f24 [2300225,105992] 0 2024-04-06T14:17:00.684 INFO:tasks.workunit.client.1.smithi086.stdout:0/120: truncate d0/d1/d4/d7/f14 385599 0 2024-04-06T14:17:00.686 INFO:tasks.workunit.client.1.smithi086.stdout:5/120: dread d0/fd [0,4194304] 0 2024-04-06T14:17:00.686 INFO:tasks.workunit.client.0.smithi032.stdout:7/358: read d16/d20/d26/d2d/d30/f38 [271067,3697] 0 2024-04-06T14:17:00.687 INFO:tasks.workunit.client.1.smithi086.stdout:0/121: write d0/d1/d2/d1a/f1d [2056818,88728] 0 2024-04-06T14:17:00.687 INFO:tasks.workunit.client.1.smithi086.stdout:4/85: rmdir d0 39 2024-04-06T14:17:00.688 INFO:tasks.workunit.client.1.smithi086.stdout:0/122: readlink d0/d1/d2/l3 0 2024-04-06T14:17:00.689 INFO:tasks.workunit.client.0.smithi032.stdout:9/346: write d2/d5/f1b [899745,39880] 0 2024-04-06T14:17:00.689 INFO:tasks.workunit.client.1.smithi086.stdout:6/78: truncate d6/d9/f17 866964 0 2024-04-06T14:17:00.689 INFO:tasks.workunit.client.0.smithi032.stdout:3/293: chown d3/d3b/d3c/c6e 126 1 2024-04-06T14:17:00.691 INFO:tasks.workunit.client.0.smithi032.stdout:9/347: fdatasync d2/d15/d55/f6e 0 2024-04-06T14:17:00.692 INFO:tasks.workunit.client.0.smithi032.stdout:2/301: mkdir d0/d6e 0 2024-04-06T14:17:00.696 INFO:tasks.workunit.client.1.smithi086.stdout:9/141: rename f3 to d5/d18/f36 0 2024-04-06T14:17:00.696 INFO:tasks.workunit.client.1.smithi086.stdout:2/57: dread d0/df/f13 [0,4194304] 0 2024-04-06T14:17:00.696 INFO:tasks.workunit.client.0.smithi032.stdout:2/302: chown d0/d5a/d3a/l64 131343319 1 2024-04-06T14:17:00.696 INFO:tasks.workunit.client.0.smithi032.stdout:9/348: mknod d2/d15/d55/d5f/d71/c72 0 2024-04-06T14:17:00.697 INFO:tasks.workunit.client.0.smithi032.stdout:1/358: getdents d0/d10/d66 0 2024-04-06T14:17:00.697 INFO:tasks.workunit.client.1.smithi086.stdout:9/142: read - d5/d13/f2d zero size 2024-04-06T14:17:00.697 INFO:tasks.workunit.client.1.smithi086.stdout:4/86: readlink d0/l9 0 2024-04-06T14:17:00.699 INFO:tasks.workunit.client.0.smithi032.stdout:1/359: fdatasync d0/f4c 0 2024-04-06T14:17:00.699 INFO:tasks.workunit.client.0.smithi032.stdout:9/349: unlink d2/d25/f46 0 2024-04-06T14:17:00.699 INFO:tasks.workunit.client.0.smithi032.stdout:7/359: dread d16/d20/d26/d2d/f69 [0,4194304] 0 2024-04-06T14:17:00.700 INFO:tasks.workunit.client.0.smithi032.stdout:2/303: rename d0/d30/d39 to d0/d30/d39/d59/d67/d6f 22 2024-04-06T14:17:00.702 INFO:tasks.workunit.client.1.smithi086.stdout:3/132: truncate d5/f9 1962371 0 2024-04-06T14:17:00.703 INFO:tasks.workunit.client.1.smithi086.stdout:3/133: read - d5/f20 zero size 2024-04-06T14:17:00.703 INFO:tasks.workunit.client.1.smithi086.stdout:4/87: dwrite d0/de/f1a [0,4194304] 0 2024-04-06T14:17:00.749 INFO:tasks.workunit.client.0.smithi032.stdout:5/331: sync 2024-04-06T14:17:00.754 INFO:tasks.workunit.client.1.smithi086.stdout:5/121: rename d0/d19/d22 to d0/d16/d18/d21/d23/d28 0 2024-04-06T14:17:00.754 INFO:tasks.workunit.client.1.smithi086.stdout:6/79: mknod d6/d9/c1d 0 2024-04-06T14:17:00.756 INFO:tasks.workunit.client.1.smithi086.stdout:3/134: fdatasync d5/df/f13 0 2024-04-06T14:17:00.757 INFO:tasks.workunit.client.1.smithi086.stdout:7/48: getdents d7 0 2024-04-06T14:17:00.758 INFO:tasks.workunit.client.1.smithi086.stdout:0/123: rename d0/d1/fb to d0/d15/d26/f2c 0 2024-04-06T14:17:00.759 INFO:tasks.workunit.client.1.smithi086.stdout:6/80: creat d6/d9/db/f1e x:0 0 0 2024-04-06T14:17:00.761 INFO:tasks.workunit.client.1.smithi086.stdout:4/88: dwrite d0/f13 [0,4194304] 0 2024-04-06T14:17:00.762 INFO:tasks.workunit.client.1.smithi086.stdout:3/135: mknod d5/df/c23 0 2024-04-06T14:17:00.762 INFO:tasks.workunit.client.1.smithi086.stdout:6/81: dread d6/d9/f17 [0,4194304] 0 2024-04-06T14:17:00.763 INFO:tasks.workunit.client.1.smithi086.stdout:0/124: truncate d0/d1/d4/de/f24 1189679 0 2024-04-06T14:17:00.763 INFO:tasks.workunit.client.1.smithi086.stdout:7/49: fsync d7/f9 0 2024-04-06T14:17:00.765 INFO:tasks.workunit.client.1.smithi086.stdout:6/82: write d6/d9/f10 [3381740,62606] 0 2024-04-06T14:17:00.768 INFO:tasks.workunit.client.1.smithi086.stdout:2/58: rename d0/l9 to d0/df/l17 0 2024-04-06T14:17:00.768 INFO:tasks.workunit.client.1.smithi086.stdout:5/122: rename d0/d16 to d0/d16/d18/d21/d23/d29 22 2024-04-06T14:17:00.768 INFO:tasks.workunit.client.1.smithi086.stdout:4/89: dwrite d0/de/f1a [0,4194304] 0 2024-04-06T14:17:00.771 INFO:tasks.workunit.client.1.smithi086.stdout:2/59: dread d0/f6 [0,4194304] 0 2024-04-06T14:17:00.775 INFO:tasks.workunit.client.1.smithi086.stdout:0/125: dread d0/d1/d4/d7/f14 [0,4194304] 0 2024-04-06T14:17:00.776 INFO:tasks.workunit.client.1.smithi086.stdout:4/90: mkdir d0/d15/d20 0 2024-04-06T14:17:00.777 INFO:tasks.workunit.client.1.smithi086.stdout:3/136: truncate d5/f9 1712315 0 2024-04-06T14:17:00.777 INFO:tasks.workunit.client.1.smithi086.stdout:3/137: fdatasync d5/f11 0 2024-04-06T14:17:00.779 INFO:tasks.workunit.client.1.smithi086.stdout:3/138: truncate f4 4991012 0 2024-04-06T14:17:00.780 INFO:tasks.workunit.client.1.smithi086.stdout:3/139: write f4 [3553340,69345] 0 2024-04-06T14:17:00.780 INFO:tasks.workunit.client.1.smithi086.stdout:2/60: rename d0/f15 to d0/df/f18 0 2024-04-06T14:17:00.782 INFO:tasks.workunit.client.1.smithi086.stdout:6/83: mknod d6/d13/d1a/c1f 0 2024-04-06T14:17:00.784 INFO:tasks.workunit.client.1.smithi086.stdout:3/140: write d5/df/f1c [982990,97919] 0 2024-04-06T14:17:00.786 INFO:tasks.workunit.client.1.smithi086.stdout:7/50: dwrite d7/fb [0,4194304] 0 2024-04-06T14:17:00.787 INFO:tasks.workunit.client.1.smithi086.stdout:0/126: creat d0/d15/d26/d29/f2d x:0 0 0 2024-04-06T14:17:00.791 INFO:tasks.workunit.client.0.smithi032.stdout:3/294: dwrite d3/d16/f68 [0,4194304] 0 2024-04-06T14:17:00.791 INFO:tasks.workunit.client.1.smithi086.stdout:7/51: chown d7/da 0 1 2024-04-06T14:17:00.791 INFO:tasks.workunit.client.1.smithi086.stdout:3/141: write d5/df/f1c [1688707,90057] 0 2024-04-06T14:17:00.795 INFO:tasks.workunit.client.1.smithi086.stdout:7/52: stat d7/da/fc 0 2024-04-06T14:17:00.797 INFO:tasks.workunit.client.0.smithi032.stdout:7/360: dread d16/d20/d26/d2d/f64 [0,4194304] 0 2024-04-06T14:17:00.797 INFO:tasks.workunit.client.1.smithi086.stdout:0/127: creat d0/d1/d4/de/f2e x:0 0 0 2024-04-06T14:17:00.799 INFO:tasks.workunit.client.0.smithi032.stdout:7/361: fsync d16/d20/d48/f4a 0 2024-04-06T14:17:00.799 INFO:tasks.workunit.client.1.smithi086.stdout:2/61: dwrite d0/fd [0,4194304] 0 2024-04-06T14:17:00.800 INFO:tasks.workunit.client.0.smithi032.stdout:9/350: mkdir d2/d15/d1e/d47/d4a/d73 0 2024-04-06T14:17:00.802 INFO:tasks.workunit.client.0.smithi032.stdout:3/295: symlink d3/d16/l81 0 2024-04-06T14:17:00.802 INFO:tasks.workunit.client.0.smithi032.stdout:1/360: rename d0/da/f65 to d0/d10/d16/f7a 0 2024-04-06T14:17:00.803 INFO:tasks.workunit.client.1.smithi086.stdout:7/53: mknod d7/da/cd 0 2024-04-06T14:17:00.803 INFO:tasks.workunit.client.0.smithi032.stdout:3/296: fsync d3/d16/f18 0 2024-04-06T14:17:00.805 INFO:tasks.workunit.client.0.smithi032.stdout:3/297: chown d3/d16/f72 42 1 2024-04-06T14:17:00.805 INFO:tasks.workunit.client.1.smithi086.stdout:6/84: dwrite d6/d9/db/f16 [0,4194304] 0 2024-04-06T14:17:00.807 INFO:tasks.workunit.client.1.smithi086.stdout:2/62: mkdir d0/d19 0 2024-04-06T14:17:00.809 INFO:tasks.workunit.client.1.smithi086.stdout:7/54: rmdir d7 39 2024-04-06T14:17:00.813 INFO:tasks.workunit.client.1.smithi086.stdout:4/91: dwrite d0/fa [0,4194304] 0 2024-04-06T14:17:00.813 INFO:tasks.workunit.client.1.smithi086.stdout:6/85: unlink d6/c12 0 2024-04-06T14:17:00.813 INFO:tasks.workunit.client.1.smithi086.stdout:3/142: dwrite d5/df/f1c [0,4194304] 0 2024-04-06T14:17:00.814 INFO:tasks.workunit.client.0.smithi032.stdout:7/362: write d16/d20/d26/f31 [2383988,93507] 0 2024-04-06T14:17:00.814 INFO:tasks.workunit.client.0.smithi032.stdout:9/351: mkdir d2/d15/d1e/d41/d6c/d74 0 2024-04-06T14:17:00.814 INFO:tasks.workunit.client.0.smithi032.stdout:3/298: rename d3/d65 to d3/d65/d7d/d82 22 2024-04-06T14:17:00.814 INFO:tasks.workunit.client.0.smithi032.stdout:5/332: getdents d16/d18/d33/d45 0 2024-04-06T14:17:00.815 INFO:tasks.workunit.client.1.smithi086.stdout:4/92: stat d0/l18 0 2024-04-06T14:17:00.816 INFO:tasks.workunit.client.1.smithi086.stdout:4/93: stat d0/f3 0 2024-04-06T14:17:00.818 INFO:tasks.workunit.client.0.smithi032.stdout:9/352: creat d2/d5/f75 x:0 0 0 2024-04-06T14:17:00.819 INFO:tasks.workunit.client.0.smithi032.stdout:7/363: mknod d16/d20/d48/d71/c7c 0 2024-04-06T14:17:00.821 INFO:tasks.workunit.client.1.smithi086.stdout:7/55: dread d7/da/fc [0,4194304] 0 2024-04-06T14:17:00.822 INFO:tasks.workunit.client.0.smithi032.stdout:1/361: dwrite d0/d10/f30 [0,4194304] 0 2024-04-06T14:17:00.824 INFO:tasks.workunit.client.1.smithi086.stdout:6/86: dread f5 [0,4194304] 0 2024-04-06T14:17:00.825 INFO:tasks.workunit.client.1.smithi086.stdout:4/94: mkdir d0/de/d21 0 2024-04-06T14:17:00.827 INFO:tasks.workunit.client.0.smithi032.stdout:9/353: chown d2/f13 703986 1 2024-04-06T14:17:00.827 INFO:tasks.workunit.client.1.smithi086.stdout:3/143: symlink d5/l24 0 2024-04-06T14:17:00.829 INFO:tasks.workunit.client.0.smithi032.stdout:9/354: chown d2/d15/d55/d5f 23 1 2024-04-06T14:17:00.829 INFO:tasks.workunit.client.0.smithi032.stdout:7/364: truncate fb 4270692 0 2024-04-06T14:17:00.830 INFO:tasks.workunit.client.1.smithi086.stdout:8/170: sync 2024-04-06T14:17:00.832 INFO:tasks.workunit.client.1.smithi086.stdout:2/63: rmdir d0/d19 0 2024-04-06T14:17:00.833 INFO:tasks.workunit.client.0.smithi032.stdout:1/362: write d0/d41/d59/f64 [676547,112198] 0 2024-04-06T14:17:00.833 INFO:tasks.workunit.client.0.smithi032.stdout:3/299: dwrite d3/d16/f4b [0,4194304] 0 2024-04-06T14:17:00.834 INFO:tasks.workunit.client.1.smithi086.stdout:7/56: symlink d7/le 0 2024-04-06T14:17:00.834 INFO:tasks.workunit.client.1.smithi086.stdout:7/57: readlink d7/le 0 2024-04-06T14:17:00.835 INFO:tasks.workunit.client.1.smithi086.stdout:7/58: chown d7/fb 15503 1 2024-04-06T14:17:00.836 INFO:tasks.workunit.client.0.smithi032.stdout:5/333: read d16/d18/d33/d35/f42 [8632,23178] 0 2024-04-06T14:17:00.838 INFO:tasks.workunit.client.0.smithi032.stdout:2/304: dwrite d0/d5a/d3a/f42 [0,4194304] 0 2024-04-06T14:17:00.840 INFO:tasks.workunit.client.1.smithi086.stdout:2/64: dwrite d0/fd [0,4194304] 0 2024-04-06T14:17:00.841 INFO:tasks.workunit.client.1.smithi086.stdout:2/65: dread - d0/df/f18 zero size 2024-04-06T14:17:00.845 INFO:tasks.workunit.client.0.smithi032.stdout:2/305: unlink d0/l11 0 2024-04-06T14:17:00.849 INFO:tasks.workunit.client.0.smithi032.stdout:2/306: fsync d0/d5a/f25 0 2024-04-06T14:17:00.851 INFO:tasks.workunit.client.1.smithi086.stdout:7/59: mknod d7/da/cf 0 2024-04-06T14:17:00.853 INFO:tasks.workunit.client.1.smithi086.stdout:0/128: dwrite d0/f17 [0,4194304] 0 2024-04-06T14:17:00.853 INFO:tasks.workunit.client.0.smithi032.stdout:9/355: getdents d2 0 2024-04-06T14:17:00.853 INFO:tasks.workunit.client.0.smithi032.stdout:5/334: truncate d16/d18/d33/f34 4460494 0 2024-04-06T14:17:00.853 INFO:tasks.workunit.client.0.smithi032.stdout:7/365: dwrite d16/d20/d26/d2d/f32 [0,4194304] 0 2024-04-06T14:17:00.854 INFO:tasks.workunit.client.0.smithi032.stdout:9/356: chown d2/d15/d1e/d47/f5d 4316 1 2024-04-06T14:17:00.854 INFO:tasks.workunit.client.1.smithi086.stdout:0/129: stat d0/d1/d4/de/c10 0 2024-04-06T14:17:00.855 INFO:tasks.workunit.client.1.smithi086.stdout:8/171: link d0/f34 d0/d8/d16/f41 0 2024-04-06T14:17:00.857 INFO:tasks.workunit.client.0.smithi032.stdout:5/335: write d16/d18/d33/f34 [3021680,38354] 0 2024-04-06T14:17:00.857 INFO:tasks.workunit.client.0.smithi032.stdout:2/307: symlink d0/d4e/l70 0 2024-04-06T14:17:00.862 INFO:tasks.workunit.client.1.smithi086.stdout:6/87: mkdir d6/d20 0 2024-04-06T14:17:00.862 INFO:tasks.workunit.client.1.smithi086.stdout:3/144: link d5/df/f1d d5/df/f25 0 2024-04-06T14:17:00.865 INFO:tasks.workunit.client.1.smithi086.stdout:8/172: mknod d0/d8/c42 0 2024-04-06T14:17:00.866 INFO:tasks.workunit.client.0.smithi032.stdout:9/357: write d2/d5/f1b [359397,32427] 0 2024-04-06T14:17:00.866 INFO:tasks.workunit.client.1.smithi086.stdout:0/130: creat d0/f2f x:0 0 0 2024-04-06T14:17:00.867 INFO:tasks.workunit.client.1.smithi086.stdout:3/145: unlink d5/lb 0 2024-04-06T14:17:00.867 INFO:tasks.workunit.client.0.smithi032.stdout:7/366: dwrite d16/f4b [0,4194304] 0 2024-04-06T14:17:00.868 INFO:tasks.workunit.client.0.smithi032.stdout:5/336: link d16/d18/d33/d45/d52/f53 d16/f77 0 2024-04-06T14:17:00.869 INFO:tasks.workunit.client.1.smithi086.stdout:6/88: creat d6/d13/d1a/f21 x:0 0 0 2024-04-06T14:17:00.869 INFO:tasks.workunit.client.1.smithi086.stdout:8/173: unlink d0/d8/d16/d39/c1d 0 2024-04-06T14:17:00.870 INFO:tasks.workunit.client.0.smithi032.stdout:5/337: readlink d16/d18/d33/d45/d52/l48 0 2024-04-06T14:17:00.871 INFO:tasks.workunit.client.1.smithi086.stdout:3/146: readlink d5/df/l16 0 2024-04-06T14:17:00.871 INFO:tasks.workunit.client.0.smithi032.stdout:7/367: write d16/d20/d26/f31 [1257155,61746] 0 2024-04-06T14:17:00.872 INFO:tasks.workunit.client.1.smithi086.stdout:0/131: mknod d0/d15/c30 0 2024-04-06T14:17:00.873 INFO:tasks.workunit.client.1.smithi086.stdout:6/89: mkdir d6/d13/d1a/d22 0 2024-04-06T14:17:00.873 INFO:tasks.workunit.client.0.smithi032.stdout:7/368: write d16/f4b [9444,109036] 0 2024-04-06T14:17:00.874 INFO:tasks.workunit.client.0.smithi032.stdout:9/358: rename d2/c3d to d2/d15/d1e/d41/c76 0 2024-04-06T14:17:00.877 INFO:tasks.workunit.client.1.smithi086.stdout:7/60: dwrite d7/fb [4194304,4194304] 0 2024-04-06T14:17:00.877 INFO:tasks.workunit.client.1.smithi086.stdout:8/174: dread d0/d8/d16/d39/d15/f28 [0,4194304] 0 2024-04-06T14:17:00.877 INFO:tasks.workunit.client.1.smithi086.stdout:3/147: dread - d5/df/f1a zero size 2024-04-06T14:17:00.878 INFO:tasks.workunit.client.0.smithi032.stdout:7/369: symlink d16/d20/d26/l7d 0 2024-04-06T14:17:00.878 INFO:tasks.workunit.client.0.smithi032.stdout:1/363: dread d0/d10/d16/f3c [4194304,4194304] 0 2024-04-06T14:17:00.879 INFO:tasks.workunit.client.1.smithi086.stdout:7/61: read f5 [3184902,83878] 0 2024-04-06T14:17:00.879 INFO:tasks.workunit.client.0.smithi032.stdout:7/370: readlink d16/l25 0 2024-04-06T14:17:00.880 INFO:tasks.workunit.client.1.smithi086.stdout:6/90: mknod d6/c23 0 2024-04-06T14:17:00.881 INFO:tasks.workunit.client.1.smithi086.stdout:2/66: rmdir d0 39 2024-04-06T14:17:00.881 INFO:tasks.workunit.client.0.smithi032.stdout:5/338: rename fb to d16/d18/d33/f78 0 2024-04-06T14:17:00.882 INFO:tasks.workunit.client.0.smithi032.stdout:5/339: chown d16/d58/d54/d74 4862006 1 2024-04-06T14:17:00.882 INFO:tasks.workunit.client.1.smithi086.stdout:0/132: dread d0/d15/d26/f2c [0,4194304] 0 2024-04-06T14:17:00.885 INFO:tasks.workunit.client.0.smithi032.stdout:1/364: mknod d0/d9/d1f/c7b 0 2024-04-06T14:17:00.885 INFO:tasks.workunit.client.0.smithi032.stdout:7/371: mknod d16/d20/d6e/c7e 0 2024-04-06T14:17:00.886 INFO:tasks.workunit.client.1.smithi086.stdout:7/62: mkdir d7/d10 0 2024-04-06T14:17:00.886 INFO:tasks.workunit.client.0.smithi032.stdout:1/365: chown d0/d10/d66/l68 924516116 1 2024-04-06T14:17:00.888 INFO:tasks.workunit.client.0.smithi032.stdout:9/359: getdents d2/d25 0 2024-04-06T14:17:00.888 INFO:tasks.workunit.client.0.smithi032.stdout:5/340: symlink d16/d18/d33/d45/l79 0 2024-04-06T14:17:00.889 INFO:tasks.workunit.client.1.smithi086.stdout:3/148: mkdir d5/d26 0 2024-04-06T14:17:00.890 INFO:tasks.workunit.client.0.smithi032.stdout:5/341: write d16/f38 [1119729,69133] 0 2024-04-06T14:17:00.890 INFO:tasks.workunit.client.0.smithi032.stdout:7/372: creat d16/d20/d26/d70/f7f x:0 0 0 2024-04-06T14:17:00.891 INFO:tasks.workunit.client.0.smithi032.stdout:1/366: mkdir d0/d9/d1f/d7c 0 2024-04-06T14:17:00.892 INFO:tasks.workunit.client.1.smithi086.stdout:0/133: unlink d0/d1/d4/f22 0 2024-04-06T14:17:00.892 INFO:tasks.workunit.client.0.smithi032.stdout:5/342: write f1 [7857266,93098] 0 2024-04-06T14:17:00.892 INFO:tasks.workunit.client.1.smithi086.stdout:0/134: dread - d0/d15/d26/d29/f2d zero size 2024-04-06T14:17:00.893 INFO:tasks.workunit.client.0.smithi032.stdout:1/367: dread - d0/da/f69 zero size 2024-04-06T14:17:00.893 INFO:tasks.workunit.client.0.smithi032.stdout:9/360: write d2/d5/f3e [891378,56205] 0 2024-04-06T14:17:00.894 INFO:tasks.workunit.client.1.smithi086.stdout:6/91: symlink d6/d20/l24 0 2024-04-06T14:17:00.895 INFO:tasks.workunit.client.0.smithi032.stdout:5/343: dread - d16/d58/f6f zero size 2024-04-06T14:17:00.896 INFO:tasks.workunit.client.1.smithi086.stdout:3/149: rename d5/l24 to d5/df/l27 0 2024-04-06T14:17:00.897 INFO:tasks.workunit.client.0.smithi032.stdout:7/373: rename d16/d20/d26/d2d/d30 to d16/d20/d26/d2d/d30/d80 22 2024-04-06T14:17:00.897 INFO:tasks.workunit.client.0.smithi032.stdout:9/361: fsync d2/d5/f64 0 2024-04-06T14:17:00.897 INFO:tasks.workunit.client.1.smithi086.stdout:0/135: creat d0/d15/d26/d29/f31 x:0 0 0 2024-04-06T14:17:00.898 INFO:tasks.workunit.client.0.smithi032.stdout:1/368: write d0/d10/d16/d3f/f5d [301560,29724] 0 2024-04-06T14:17:00.899 INFO:tasks.workunit.client.1.smithi086.stdout:0/136: dread - d0/d1/d4/de/f2e zero size 2024-04-06T14:17:00.899 INFO:tasks.workunit.client.0.smithi032.stdout:7/374: rmdir d16/d20 39 2024-04-06T14:17:00.899 INFO:tasks.workunit.client.0.smithi032.stdout:1/369: chown d0/da/f38 47524622 1 2024-04-06T14:17:00.902 INFO:tasks.workunit.client.1.smithi086.stdout:6/92: rename d6/d9/ld to d6/d13/d1a/l25 0 2024-04-06T14:17:00.903 INFO:tasks.workunit.client.1.smithi086.stdout:3/150: dread d5/df/f13 [4194304,4194304] 0 2024-04-06T14:17:00.904 INFO:tasks.workunit.client.1.smithi086.stdout:0/137: truncate d0/d1/d4/d7/f14 1038218 0 2024-04-06T14:17:00.905 INFO:tasks.workunit.client.1.smithi086.stdout:6/93: creat d6/d13/f26 x:0 0 0 2024-04-06T14:17:00.906 INFO:tasks.workunit.client.1.smithi086.stdout:3/151: getdents d5/d26 0 2024-04-06T14:17:00.907 INFO:tasks.workunit.client.0.smithi032.stdout:7/375: mknod d16/d20/d26/d70/c81 0 2024-04-06T14:17:00.907 INFO:tasks.workunit.client.0.smithi032.stdout:2/308: dread d0/d5a/f1c [0,4194304] 0 2024-04-06T14:17:00.908 INFO:tasks.workunit.client.0.smithi032.stdout:1/370: link d0/d9/c6e d0/d9/d1f/c7d 0 2024-04-06T14:17:00.908 INFO:tasks.workunit.client.0.smithi032.stdout:5/344: dread d16/d18/f31 [0,4194304] 0 2024-04-06T14:17:00.909 INFO:tasks.workunit.client.1.smithi086.stdout:3/152: stat d5/df/l17 0 2024-04-06T14:17:00.911 INFO:tasks.workunit.client.1.smithi086.stdout:4/95: truncate d0/de/f1a 2913770 0 2024-04-06T14:17:00.911 INFO:tasks.workunit.client.1.smithi086.stdout:1/64: sync 2024-04-06T14:17:00.911 INFO:tasks.workunit.client.0.smithi032.stdout:1/371: rmdir d0/d9 39 2024-04-06T14:17:00.914 INFO:tasks.workunit.client.1.smithi086.stdout:3/153: truncate d5/f11 3032453 0 2024-04-06T14:17:00.915 INFO:tasks.workunit.client.0.smithi032.stdout:9/362: dwrite d2/d15/d1e/d41/f63 [0,4194304] 0 2024-04-06T14:17:00.915 INFO:tasks.workunit.client.0.smithi032.stdout:3/300: dwrite d3/f37 [0,4194304] 0 2024-04-06T14:17:00.918 INFO:tasks.workunit.client.0.smithi032.stdout:2/309: symlink d0/d4e/d52/l71 0 2024-04-06T14:17:00.919 INFO:tasks.workunit.client.1.smithi086.stdout:3/154: symlink d5/d26/l28 0 2024-04-06T14:17:00.920 INFO:tasks.workunit.client.0.smithi032.stdout:3/301: fsync d3/f7a 0 2024-04-06T14:17:00.923 INFO:tasks.workunit.client.0.smithi032.stdout:3/302: write d3/d16/f33 [577703,47059] 0 2024-04-06T14:17:00.923 INFO:tasks.workunit.client.0.smithi032.stdout:9/363: mkdir d2/d15/d1e/d47/d77 0 2024-04-06T14:17:00.924 INFO:tasks.workunit.client.1.smithi086.stdout:3/155: dread d5/df/f13 [0,4194304] 0 2024-04-06T14:17:00.926 INFO:tasks.workunit.client.0.smithi032.stdout:1/372: dwrite d0/da/f69 [0,4194304] 0 2024-04-06T14:17:00.927 INFO:tasks.workunit.client.0.smithi032.stdout:9/364: truncate d2/d15/d1e/f6d 441342 0 2024-04-06T14:17:00.929 INFO:tasks.workunit.client.0.smithi032.stdout:9/365: stat d2/d25/d31/f3a 0 2024-04-06T14:17:00.929 INFO:tasks.workunit.client.0.smithi032.stdout:2/310: creat d0/d6e/f72 x:0 0 0 2024-04-06T14:17:00.929 INFO:tasks.workunit.client.0.smithi032.stdout:2/311: read d0/d30/f43 [534400,109304] 0 2024-04-06T14:17:00.930 INFO:tasks.workunit.client.1.smithi086.stdout:6/94: getdents d6 0 2024-04-06T14:17:00.930 INFO:tasks.workunit.client.0.smithi032.stdout:3/303: fsync d3/fc 0 2024-04-06T14:17:00.932 INFO:tasks.workunit.client.1.smithi086.stdout:8/175: write d0/d13/f31 [1144676,61832] 0 2024-04-06T14:17:00.938 INFO:tasks.workunit.client.0.smithi032.stdout:2/312: mknod d0/d6e/c73 0 2024-04-06T14:17:00.939 INFO:tasks.workunit.client.0.smithi032.stdout:5/345: unlink d16/d18/d33/d45/l51 0 2024-04-06T14:17:00.939 INFO:tasks.workunit.client.0.smithi032.stdout:7/376: write d16/d20/f23 [1856392,116669] 0 2024-04-06T14:17:00.939 INFO:tasks.workunit.client.0.smithi032.stdout:5/346: write d16/d18/f20 [2321353,40088] 0 2024-04-06T14:17:00.939 INFO:tasks.workunit.client.1.smithi086.stdout:7/63: truncate f5 1710634 0 2024-04-06T14:17:00.939 INFO:tasks.workunit.client.1.smithi086.stdout:6/95: creat d6/d13/f27 x:0 0 0 2024-04-06T14:17:00.940 INFO:tasks.workunit.client.0.smithi032.stdout:3/304: creat d3/d16/d63/f83 x:0 0 0 2024-04-06T14:17:00.942 INFO:tasks.workunit.client.1.smithi086.stdout:6/96: creat d6/d9/db/f28 x:0 0 0 2024-04-06T14:17:00.942 INFO:tasks.workunit.client.0.smithi032.stdout:1/373: unlink d0/d9/d1f/f3b 0 2024-04-06T14:17:00.943 INFO:tasks.workunit.client.0.smithi032.stdout:9/366: creat d2/d15/d1e/d47/d77/f78 x:0 0 0 2024-04-06T14:17:00.944 INFO:tasks.workunit.client.1.smithi086.stdout:0/138: dread d0/d1/d4/d7/f14 [0,4194304] 0 2024-04-06T14:17:00.944 INFO:tasks.workunit.client.1.smithi086.stdout:2/67: dwrite d0/df/f13 [4194304,4194304] 0 2024-04-06T14:17:00.944 INFO:tasks.workunit.client.0.smithi032.stdout:7/377: creat d16/d20/d54/f82 x:0 0 0 2024-04-06T14:17:00.946 INFO:tasks.workunit.client.1.smithi086.stdout:8/176: rename d0/d8/d16/d39/d15 to d0/d8/d22/d43 0 2024-04-06T14:17:00.947 INFO:tasks.workunit.client.0.smithi032.stdout:5/347: creat d16/d58/f7a x:0 0 0 2024-04-06T14:17:00.948 INFO:tasks.workunit.client.1.smithi086.stdout:2/68: creat d0/df/f1a x:0 0 0 2024-04-06T14:17:00.949 INFO:tasks.workunit.client.1.smithi086.stdout:8/177: mknod d0/d8/d22/d43/c44 0 2024-04-06T14:17:00.950 INFO:tasks.workunit.client.0.smithi032.stdout:2/313: mkdir d0/d30/d39/d59/d67/d74 0 2024-04-06T14:17:00.950 INFO:tasks.workunit.client.1.smithi086.stdout:0/139: mknod d0/d23/c32 0 2024-04-06T14:17:00.950 INFO:tasks.workunit.client.1.smithi086.stdout:7/64: mkdir d7/d10/d11 0 2024-04-06T14:17:00.951 INFO:tasks.workunit.client.0.smithi032.stdout:2/314: stat d0/d4e/d52/l71 0 2024-04-06T14:17:00.951 INFO:tasks.workunit.client.0.smithi032.stdout:9/367: dread - d2/d5/f49 zero size 2024-04-06T14:17:00.952 INFO:tasks.workunit.client.0.smithi032.stdout:2/315: chown d0/d5a/f1a 274 1 2024-04-06T14:17:00.952 INFO:tasks.workunit.client.1.smithi086.stdout:0/140: write d0/f17 [1469162,341] 0 2024-04-06T14:17:00.952 INFO:tasks.workunit.client.0.smithi032.stdout:3/305: mknod d3/c84 0 2024-04-06T14:17:00.953 INFO:tasks.workunit.client.0.smithi032.stdout:2/316: fsync d0/d4e/f69 0 2024-04-06T14:17:00.954 INFO:tasks.workunit.client.0.smithi032.stdout:5/348: mknod d16/d18/d33/d35/c7b 0 2024-04-06T14:17:00.954 INFO:tasks.workunit.client.0.smithi032.stdout:9/368: truncate d2/d15/d1e/d47/f4b 1104111 0 2024-04-06T14:17:00.954 INFO:tasks.workunit.client.1.smithi086.stdout:2/69: unlink d0/lb 0 2024-04-06T14:17:00.955 INFO:tasks.workunit.client.0.smithi032.stdout:1/374: creat d0/d9/d1f/d72/f7e x:0 0 0 2024-04-06T14:17:00.956 INFO:tasks.workunit.client.0.smithi032.stdout:7/378: creat d16/d20/d7a/f83 x:0 0 0 2024-04-06T14:17:00.956 INFO:tasks.workunit.client.1.smithi086.stdout:0/141: mkdir d0/d15/d26/d33 0 2024-04-06T14:17:00.958 INFO:tasks.workunit.client.1.smithi086.stdout:2/70: dread d0/fe [0,4194304] 0 2024-04-06T14:17:00.958 INFO:tasks.workunit.client.1.smithi086.stdout:8/178: getdents d0/db/d32 0 2024-04-06T14:17:00.959 INFO:tasks.workunit.client.1.smithi086.stdout:0/142: mknod d0/d23/c34 0 2024-04-06T14:17:00.960 INFO:tasks.workunit.client.0.smithi032.stdout:2/317: mknod d0/c75 0 2024-04-06T14:17:00.960 INFO:tasks.workunit.client.0.smithi032.stdout:7/379: chown d16/d20/d26/d2d/d30/d3e 8 1 2024-04-06T14:17:00.960 INFO:tasks.workunit.client.1.smithi086.stdout:7/65: dwrite d7/f9 [0,4194304] 0 2024-04-06T14:17:00.961 INFO:tasks.workunit.client.0.smithi032.stdout:5/349: symlink d16/d58/d54/d74/l7c 0 2024-04-06T14:17:00.962 INFO:tasks.workunit.client.0.smithi032.stdout:1/375: unlink d0/d9/d1f/c7d 0 2024-04-06T14:17:00.962 INFO:tasks.workunit.client.1.smithi086.stdout:2/71: creat d0/df/f1b x:0 0 0 2024-04-06T14:17:00.963 INFO:tasks.workunit.client.1.smithi086.stdout:7/66: chown d7/da 206191375 1 2024-04-06T14:17:00.964 INFO:tasks.workunit.client.0.smithi032.stdout:3/306: dwrite d3/d65/f69 [4194304,4194304] 0 2024-04-06T14:17:00.964 INFO:tasks.workunit.client.1.smithi086.stdout:7/67: readlink d7/le 0 2024-04-06T14:17:00.965 INFO:tasks.workunit.client.1.smithi086.stdout:2/72: fsync d0/df/f18 0 2024-04-06T14:17:00.965 INFO:tasks.workunit.client.0.smithi032.stdout:9/369: rename d2/d15/d1e/d47/d4a/f4d to d2/d15/d1e/d41/d6c/d74/f79 0 2024-04-06T14:17:00.968 INFO:tasks.workunit.client.1.smithi086.stdout:0/143: symlink d0/l35 0 2024-04-06T14:17:00.969 INFO:tasks.workunit.client.0.smithi032.stdout:2/318: mknod d0/d5a/d3a/c76 0 2024-04-06T14:17:00.969 INFO:tasks.workunit.client.0.smithi032.stdout:9/370: readlink d2/d25/d31/d39/l6a 0 2024-04-06T14:17:00.972 INFO:tasks.workunit.client.0.smithi032.stdout:5/350: rmdir d16/d18/d33/d45/d52 39 2024-04-06T14:17:00.972 INFO:tasks.workunit.client.0.smithi032.stdout:1/376: read d0/f17 [1948335,84150] 0 2024-04-06T14:17:00.976 INFO:tasks.workunit.client.1.smithi086.stdout:8/179: dwrite d0/d3b/f3d [0,4194304] 0 2024-04-06T14:17:00.976 INFO:tasks.workunit.client.0.smithi032.stdout:2/319: creat d0/d4e/d52/f77 x:0 0 0 2024-04-06T14:17:00.978 INFO:tasks.workunit.client.0.smithi032.stdout:3/307: dwrite d3/d65/f66 [0,4194304] 0 2024-04-06T14:17:00.979 INFO:tasks.workunit.client.1.smithi086.stdout:7/68: symlink d7/d10/d11/l12 0 2024-04-06T14:17:00.982 INFO:tasks.workunit.client.0.smithi032.stdout:3/308: write d3/f80 [168368,107785] 0 2024-04-06T14:17:00.983 INFO:tasks.workunit.client.1.smithi086.stdout:2/73: creat d0/f1c x:0 0 0 2024-04-06T14:17:00.992 INFO:tasks.workunit.client.0.smithi032.stdout:8/328: sync 2024-04-06T14:17:00.992 INFO:tasks.workunit.client.0.smithi032.stdout:6/326: sync 2024-04-06T14:17:00.992 INFO:tasks.workunit.client.0.smithi032.stdout:3/309: rename d3/d16/f25 to d3/d3b/d3c/f85 0 2024-04-06T14:17:00.994 INFO:tasks.workunit.client.0.smithi032.stdout:5/351: truncate d16/d18/d33/d45/d52/d39/f47 1264640 0 2024-04-06T14:17:00.995 INFO:tasks.workunit.client.1.smithi086.stdout:2/74: creat d0/df/f1d x:0 0 0 2024-04-06T14:17:00.995 INFO:tasks.workunit.client.1.smithi086.stdout:0/144: dwrite d0/d1/d4/d7/f11 [0,4194304] 0 2024-04-06T14:17:00.996 INFO:tasks.workunit.client.0.smithi032.stdout:3/310: chown d3/d16/d63/c6d 1811461 1 2024-04-06T14:17:00.997 INFO:tasks.workunit.client.1.smithi086.stdout:3/156: dwrite d5/df/f25 [0,4194304] 0 2024-04-06T14:17:01.003 INFO:tasks.workunit.client.0.smithi032.stdout:9/371: dwrite d2/d5/f18 [4194304,4194304] 0 2024-04-06T14:17:01.003 INFO:tasks.workunit.client.0.smithi032.stdout:6/327: mkdir d4/d5/d8/d1f/d61 0 2024-04-06T14:17:01.003 INFO:tasks.workunit.client.0.smithi032.stdout:2/320: dwrite d0/f23 [4194304,4194304] 0 2024-04-06T14:17:01.003 INFO:tasks.workunit.client.1.smithi086.stdout:3/157: chown d5/df/l18 8997809 1 2024-04-06T14:17:01.003 INFO:tasks.workunit.client.1.smithi086.stdout:7/69: dread d7/da/fc [0,4194304] 0 2024-04-06T14:17:01.003 INFO:tasks.workunit.client.1.smithi086.stdout:0/145: truncate d0/f17 4253605 0 2024-04-06T14:17:01.003 INFO:tasks.workunit.client.1.smithi086.stdout:6/97: dwrite d6/d13/d1a/f1b [0,4194304] 0 2024-04-06T14:17:01.006 INFO:tasks.workunit.client.1.smithi086.stdout:2/75: creat d0/f1e x:0 0 0 2024-04-06T14:17:01.006 INFO:tasks.workunit.client.0.smithi032.stdout:7/380: truncate d16/d20/d26/d2d/f69 2678413 0 2024-04-06T14:17:01.011 INFO:tasks.workunit.client.0.smithi032.stdout:3/311: mknod d3/d3b/c86 0 2024-04-06T14:17:01.011 INFO:tasks.workunit.client.0.smithi032.stdout:8/329: dwrite d3/f32 [4194304,4194304] 0 2024-04-06T14:17:01.011 INFO:tasks.workunit.client.0.smithi032.stdout:9/372: symlink d2/d15/d55/d59/l7a 0 2024-04-06T14:17:01.013 INFO:tasks.workunit.client.0.smithi032.stdout:2/321: write d0/d5a/d3a/f5b [678814,101179] 0 2024-04-06T14:17:01.014 INFO:tasks.workunit.client.1.smithi086.stdout:8/180: dwrite d0/d8/d16/f41 [0,4194304] 0 2024-04-06T14:17:01.015 INFO:tasks.workunit.client.0.smithi032.stdout:7/381: write d16/d20/d26/f58 [818698,85305] 0 2024-04-06T14:17:01.015 INFO:tasks.workunit.client.0.smithi032.stdout:2/322: stat d0/d4f 0 2024-04-06T14:17:01.017 INFO:tasks.workunit.client.0.smithi032.stdout:7/382: read f11 [887799,74480] 0 2024-04-06T14:17:01.018 INFO:tasks.workunit.client.1.smithi086.stdout:8/181: stat d0/d8/d16/f41 0 2024-04-06T14:17:01.018 INFO:tasks.workunit.client.1.smithi086.stdout:6/98: dread d6/d9/db/f16 [0,4194304] 0 2024-04-06T14:17:01.018 INFO:tasks.workunit.client.1.smithi086.stdout:3/158: rename d5/df/f1d to d5/f29 0 2024-04-06T14:17:01.019 INFO:tasks.workunit.client.0.smithi032.stdout:1/377: truncate d0/da/f38 710176 0 2024-04-06T14:17:01.019 INFO:tasks.workunit.client.1.smithi086.stdout:0/146: symlink d0/d1/d4/d7/l36 0 2024-04-06T14:17:01.020 INFO:tasks.workunit.client.1.smithi086.stdout:6/99: write d6/d9/db/f1e [442818,72008] 0 2024-04-06T14:17:01.023 INFO:tasks.workunit.client.0.smithi032.stdout:2/323: dread - d0/d5a/d3a/f61 zero size 2024-04-06T14:17:01.024 INFO:tasks.workunit.client.0.smithi032.stdout:3/312: dwrite d3/fc [0,4194304] 0 2024-04-06T14:17:01.025 INFO:tasks.workunit.client.0.smithi032.stdout:3/313: fdatasync d3/d16/f72 0 2024-04-06T14:17:01.027 INFO:tasks.workunit.client.1.smithi086.stdout:3/159: mkdir d5/d26/d2a 0 2024-04-06T14:17:01.027 INFO:tasks.workunit.client.0.smithi032.stdout:7/383: rename d16/d20/l4d to d16/d20/d54/l84 0 2024-04-06T14:17:01.029 INFO:tasks.workunit.client.0.smithi032.stdout:9/373: dread d2/d15/f2b [0,4194304] 0 2024-04-06T14:17:01.029 INFO:tasks.workunit.client.0.smithi032.stdout:7/384: fsync d16/d20/d7a/f83 0 2024-04-06T14:17:01.030 INFO:tasks.workunit.client.0.smithi032.stdout:9/374: readlink d2/le 0 2024-04-06T14:17:01.030 INFO:tasks.workunit.client.0.smithi032.stdout:7/385: fsync d16/f40 0 2024-04-06T14:17:01.033 INFO:tasks.workunit.client.1.smithi086.stdout:6/100: rename d6/c23 to d6/d9/db/c29 0 2024-04-06T14:17:01.033 INFO:tasks.workunit.client.1.smithi086.stdout:3/160: mkdir d5/d2b 0 2024-04-06T14:17:01.035 INFO:tasks.workunit.client.0.smithi032.stdout:2/324: rename d0/d4e/d52 to d0/d30/d39/d59/d67/d74/d78 0 2024-04-06T14:17:01.035 INFO:tasks.workunit.client.0.smithi032.stdout:3/314: dread d3/d16/f2c [0,4194304] 0 2024-04-06T14:17:01.037 INFO:tasks.workunit.client.0.smithi032.stdout:9/375: chown d2/d5/l1c 35856 1 2024-04-06T14:17:01.042 INFO:tasks.workunit.client.1.smithi086.stdout:3/161: creat d5/df/f2c x:0 0 0 2024-04-06T14:17:01.042 INFO:tasks.workunit.client.1.smithi086.stdout:2/76: rmdir d0/df 39 2024-04-06T14:17:01.042 INFO:tasks.workunit.client.0.smithi032.stdout:1/378: rename d0/d9/d1f/c7b to d0/d10/d16/c7f 0 2024-04-06T14:17:01.042 INFO:tasks.workunit.client.1.smithi086.stdout:3/162: chown d5/c12 162747 1 2024-04-06T14:17:01.043 INFO:tasks.workunit.client.1.smithi086.stdout:2/77: stat d0/cc 0 2024-04-06T14:17:01.043 INFO:tasks.workunit.client.0.smithi032.stdout:2/325: fsync d0/d4e/f53 0 2024-04-06T14:17:01.043 INFO:tasks.workunit.client.0.smithi032.stdout:3/315: mkdir d3/d3b/d3c/d87 0 2024-04-06T14:17:01.044 INFO:tasks.workunit.client.0.smithi032.stdout:5/352: dwrite d16/f77 [0,4194304] 0 2024-04-06T14:17:01.046 INFO:tasks.workunit.client.0.smithi032.stdout:7/386: symlink d16/l85 0 2024-04-06T14:17:01.046 INFO:tasks.workunit.client.0.smithi032.stdout:2/326: write d0/d5a/d3a/f42 [4353547,101659] 0 2024-04-06T14:17:01.047 INFO:tasks.workunit.client.0.smithi032.stdout:9/376: symlink d2/d15/l7b 0 2024-04-06T14:17:01.047 INFO:tasks.workunit.client.1.smithi086.stdout:2/78: mknod d0/c1f 0 2024-04-06T14:17:01.048 INFO:tasks.workunit.client.1.smithi086.stdout:3/163: creat d5/f2d x:0 0 0 2024-04-06T14:17:01.049 INFO:tasks.workunit.client.0.smithi032.stdout:3/316: rename d3/d16/f4b to d3/d65/f88 0 2024-04-06T14:17:01.051 INFO:tasks.workunit.client.0.smithi032.stdout:3/317: read d3/d16/f18 [1476025,117336] 0 2024-04-06T14:17:01.053 INFO:tasks.workunit.client.1.smithi086.stdout:9/143: sync 2024-04-06T14:17:01.053 INFO:tasks.workunit.client.1.smithi086.stdout:5/123: sync 2024-04-06T14:17:01.054 INFO:tasks.workunit.client.0.smithi032.stdout:2/327: truncate d0/d5a/f1c 2765347 0 2024-04-06T14:17:01.056 INFO:tasks.workunit.client.0.smithi032.stdout:3/318: mkdir d3/d65/d89 0 2024-04-06T14:17:01.056 INFO:tasks.workunit.client.0.smithi032.stdout:9/377: dwrite d2/f13 [0,4194304] 0 2024-04-06T14:17:01.058 INFO:tasks.workunit.client.0.smithi032.stdout:3/319: truncate d3/d16/f70 779113 0 2024-04-06T14:17:01.058 INFO:tasks.workunit.client.0.smithi032.stdout:7/387: mkdir d16/d20/d86 0 2024-04-06T14:17:01.058 INFO:tasks.workunit.client.0.smithi032.stdout:2/328: mknod d0/d30/c79 0 2024-04-06T14:17:01.058 INFO:tasks.workunit.client.1.smithi086.stdout:2/79: mkdir d0/d20 0 2024-04-06T14:17:01.059 INFO:tasks.workunit.client.1.smithi086.stdout:5/124: fdatasync d0/d16/d18/d21/d23/f20 0 2024-04-06T14:17:01.059 INFO:tasks.workunit.client.1.smithi086.stdout:9/144: dread - d5/d10/d14/d19/d25/f2f zero size 2024-04-06T14:17:01.059 INFO:tasks.workunit.client.0.smithi032.stdout:9/378: stat d2/d15/l20 0 2024-04-06T14:17:01.060 INFO:tasks.workunit.client.1.smithi086.stdout:9/145: chown d5/d13/l34 1844985 1 2024-04-06T14:17:01.060 INFO:tasks.workunit.client.1.smithi086.stdout:5/125: dread - d0/d16/d18/f1a zero size 2024-04-06T14:17:01.061 INFO:tasks.workunit.client.1.smithi086.stdout:3/164: dread d5/fe [0,4194304] 0 2024-04-06T14:17:01.061 INFO:tasks.workunit.client.0.smithi032.stdout:2/329: write d0/f1b [2670245,85022] 0 2024-04-06T14:17:01.062 INFO:tasks.workunit.client.1.smithi086.stdout:5/126: dread - d0/d16/d18/f1a zero size 2024-04-06T14:17:01.064 INFO:tasks.workunit.client.0.smithi032.stdout:6/328: write d4/d5/d18/f31 [465503,15188] 0 2024-04-06T14:17:01.066 INFO:tasks.workunit.client.1.smithi086.stdout:9/146: symlink d5/d10/l37 0 2024-04-06T14:17:01.067 INFO:tasks.workunit.client.0.smithi032.stdout:2/330: write d0/d5a/d3a/f61 [741602,45558] 0 2024-04-06T14:17:01.067 INFO:tasks.workunit.client.1.smithi086.stdout:3/165: symlink d5/d2b/l2e 0 2024-04-06T14:17:01.068 INFO:tasks.workunit.client.1.smithi086.stdout:5/127: dread d0/fd [0,4194304] 0 2024-04-06T14:17:01.069 INFO:tasks.workunit.client.0.smithi032.stdout:3/320: getdents d3/d65/d89 0 2024-04-06T14:17:01.070 INFO:tasks.workunit.client.1.smithi086.stdout:5/128: chown d0/d16/d18/c25 115316657 1 2024-04-06T14:17:01.070 INFO:tasks.workunit.client.1.smithi086.stdout:9/147: unlink d5/d10/c12 0 2024-04-06T14:17:01.072 INFO:tasks.workunit.client.0.smithi032.stdout:9/379: mkdir d2/d15/d1e/d47/d4a/d73/d7c 0 2024-04-06T14:17:01.072 INFO:tasks.workunit.client.0.smithi032.stdout:6/329: fdatasync d4/d5/d19/d21/f30 0 2024-04-06T14:17:01.072 INFO:tasks.workunit.client.1.smithi086.stdout:3/166: write d5/df/f25 [2288946,127547] 0 2024-04-06T14:17:01.073 INFO:tasks.workunit.client.0.smithi032.stdout:7/388: dwrite d16/d20/d26/d2d/f41 [0,4194304] 0 2024-04-06T14:17:01.074 INFO:tasks.workunit.client.0.smithi032.stdout:8/330: dread d3/f29 [0,4194304] 0 2024-04-06T14:17:01.074 INFO:tasks.workunit.client.0.smithi032.stdout:2/331: mknod d0/d30/d39/d59/d67/c7a 0 2024-04-06T14:17:01.075 INFO:tasks.workunit.client.1.smithi086.stdout:0/147: rmdir d0/d1/d4 39 2024-04-06T14:17:01.076 INFO:tasks.workunit.client.1.smithi086.stdout:8/182: write d0/d8/d22/d43/f28 [2025375,104375] 0 2024-04-06T14:17:01.077 INFO:tasks.workunit.client.0.smithi032.stdout:3/321: truncate d3/d65/f39 54150 0 2024-04-06T14:17:01.078 INFO:tasks.workunit.client.0.smithi032.stdout:7/389: stat d16/d20/d54 0 2024-04-06T14:17:01.078 INFO:tasks.workunit.client.0.smithi032.stdout:6/330: dread - d4/d5/d8/d1f/d51/f3d zero size 2024-04-06T14:17:01.078 INFO:tasks.workunit.client.0.smithi032.stdout:9/380: unlink d2/fd 0 2024-04-06T14:17:01.080 INFO:tasks.workunit.client.1.smithi086.stdout:5/129: mkdir d0/d16/d18/d2a 0 2024-04-06T14:17:01.081 INFO:tasks.workunit.client.1.smithi086.stdout:3/167: dread d5/f9 [0,4194304] 0 2024-04-06T14:17:01.082 INFO:tasks.workunit.client.0.smithi032.stdout:2/332: mkdir d0/d30/d7b 0 2024-04-06T14:17:01.085 INFO:tasks.workunit.client.0.smithi032.stdout:9/381: write d2/d5/f14 [1715670,86117] 0 2024-04-06T14:17:01.087 INFO:tasks.workunit.client.0.smithi032.stdout:8/331: dread d3/d8/d12/d2f/f62 [0,4194304] 0 2024-04-06T14:17:01.087 INFO:tasks.workunit.client.1.smithi086.stdout:2/80: rmdir d0 39 2024-04-06T14:17:01.090 INFO:tasks.workunit.client.1.smithi086.stdout:3/168: creat d5/df/f2f x:0 0 0 2024-04-06T14:17:01.090 INFO:tasks.workunit.client.0.smithi032.stdout:7/390: dwrite d16/d20/d26/d2d/d30/f46 [0,4194304] 0 2024-04-06T14:17:01.092 INFO:tasks.workunit.client.1.smithi086.stdout:9/148: getdents d5/d10/d14 0 2024-04-06T14:17:01.092 INFO:tasks.workunit.client.1.smithi086.stdout:6/101: write d6/d13/f27 [57235,41183] 0 2024-04-06T14:17:01.094 INFO:tasks.workunit.client.0.smithi032.stdout:2/333: mkdir d0/d30/d39/d59/d67/d74/d78/d57/d7c 0 2024-04-06T14:17:01.094 INFO:tasks.workunit.client.1.smithi086.stdout:8/183: dwrite d0/d8/f3c [4194304,4194304] 0 2024-04-06T14:17:01.096 INFO:tasks.workunit.client.0.smithi032.stdout:0/265: sync 2024-04-06T14:17:01.096 INFO:tasks.workunit.client.0.smithi032.stdout:4/424: sync 2024-04-06T14:17:01.097 INFO:tasks.workunit.client.1.smithi086.stdout:8/184: chown d0/d3b 360332 1 2024-04-06T14:17:01.098 INFO:tasks.workunit.client.1.smithi086.stdout:0/148: getdents d0/d1/d2 0 2024-04-06T14:17:01.098 INFO:tasks.workunit.client.0.smithi032.stdout:2/334: write d0/d4e/f69 [1010509,90409] 0 2024-04-06T14:17:01.101 INFO:tasks.workunit.client.0.smithi032.stdout:7/391: symlink d16/d20/d26/d2d/d30/d3e/l87 0 2024-04-06T14:17:01.101 INFO:tasks.workunit.client.0.smithi032.stdout:6/331: link d4/d5/d8/d1f/l24 d4/d5/d19/d21/l62 0 2024-04-06T14:17:01.101 INFO:tasks.workunit.client.1.smithi086.stdout:3/169: write d5/df/f2f [401524,114703] 0 2024-04-06T14:17:01.101 INFO:tasks.workunit.client.1.smithi086.stdout:6/102: fsync d6/d9/db/f1e 0 2024-04-06T14:17:01.103 INFO:tasks.workunit.client.0.smithi032.stdout:5/353: dwrite d16/d18/d33/d35/f42 [0,4194304] 0 2024-04-06T14:17:01.103 INFO:tasks.workunit.client.1.smithi086.stdout:7/70: dwrite f5 [0,4194304] 0 2024-04-06T14:17:01.104 INFO:tasks.workunit.client.1.smithi086.stdout:9/149: unlink d5/d13/l28 0 2024-04-06T14:17:01.104 INFO:tasks.workunit.client.0.smithi032.stdout:5/354: write d16/f38 [1691734,76857] 0 2024-04-06T14:17:01.109 INFO:tasks.workunit.client.1.smithi086.stdout:2/81: rename d0/f1c to d0/df/f21 0 2024-04-06T14:17:01.109 INFO:tasks.workunit.client.1.smithi086.stdout:8/185: symlink d0/db/l45 0 2024-04-06T14:17:01.110 INFO:tasks.workunit.client.1.smithi086.stdout:5/130: write d0/fd [252955,1075] 0 2024-04-06T14:17:01.113 INFO:tasks.workunit.client.0.smithi032.stdout:0/266: unlink d3/da/c22 0 2024-04-06T14:17:01.113 INFO:tasks.workunit.client.1.smithi086.stdout:7/71: creat d7/da/f13 x:0 0 0 2024-04-06T14:17:01.115 INFO:tasks.workunit.client.0.smithi032.stdout:4/425: creat d0/df/d1f/f95 x:0 0 0 2024-04-06T14:17:01.116 INFO:tasks.workunit.client.0.smithi032.stdout:2/335: creat d0/d30/d39/d59/d67/d74/d78/d57/f7d x:0 0 0 2024-04-06T14:17:01.117 INFO:tasks.workunit.client.0.smithi032.stdout:6/332: creat d4/d5/d8/d1f/f63 x:0 0 0 2024-04-06T14:17:01.120 INFO:tasks.workunit.client.1.smithi086.stdout:5/131: dread - d0/d19/f26 zero size 2024-04-06T14:17:01.120 INFO:tasks.workunit.client.1.smithi086.stdout:0/149: unlink d0/d1/d4/d7/f18 0 2024-04-06T14:17:01.120 INFO:tasks.workunit.client.1.smithi086.stdout:8/186: write d0/d3b/f3d [513059,110510] 0 2024-04-06T14:17:01.121 INFO:tasks.workunit.client.1.smithi086.stdout:3/170: symlink d5/d26/d2a/l30 0 2024-04-06T14:17:01.121 INFO:tasks.workunit.client.1.smithi086.stdout:9/150: symlink d5/d13/d1a/l38 0 2024-04-06T14:17:01.122 INFO:tasks.workunit.client.0.smithi032.stdout:3/322: truncate d3/d65/f32 89819 0 2024-04-06T14:17:01.123 INFO:tasks.workunit.client.0.smithi032.stdout:4/426: write d0/df/d23/d31/d50/d5e/d89/f86 [2792734,100970] 0 2024-04-06T14:17:01.125 INFO:tasks.workunit.client.1.smithi086.stdout:7/72: symlink d7/da/l14 0 2024-04-06T14:17:01.125 INFO:tasks.workunit.client.1.smithi086.stdout:9/151: chown d5/d13/f2d 4 1 2024-04-06T14:17:01.126 INFO:tasks.workunit.client.1.smithi086.stdout:2/82: dwrite d0/df/f13 [4194304,4194304] 0 2024-04-06T14:17:01.126 INFO:tasks.workunit.client.1.smithi086.stdout:6/103: dread d6/d13/d1a/f1b [0,4194304] 0 2024-04-06T14:17:01.128 INFO:tasks.workunit.client.1.smithi086.stdout:3/171: unlink d5/l15 0 2024-04-06T14:17:01.129 INFO:tasks.workunit.client.0.smithi032.stdout:5/355: mknod d16/d18/d33/d45/c7d 0 2024-04-06T14:17:01.129 INFO:tasks.workunit.client.0.smithi032.stdout:1/379: dwrite d0/da/f38 [0,4194304] 0 2024-04-06T14:17:01.130 INFO:tasks.workunit.client.1.smithi086.stdout:5/132: symlink d0/d16/l2b 0 2024-04-06T14:17:01.130 INFO:tasks.workunit.client.1.smithi086.stdout:3/172: read d5/df/f13 [8126461,54633] 0 2024-04-06T14:17:01.130 INFO:tasks.workunit.client.1.smithi086.stdout:8/187: read d0/d13/f21 [722855,76953] 0 2024-04-06T14:17:01.131 INFO:tasks.workunit.client.0.smithi032.stdout:7/392: dread d16/d20/d26/d2d/f69 [0,4194304] 0 2024-04-06T14:17:01.131 INFO:tasks.workunit.client.1.smithi086.stdout:3/173: stat d5/c1f 0 2024-04-06T14:17:01.133 INFO:tasks.workunit.client.0.smithi032.stdout:8/332: getdents d3/d8 0 2024-04-06T14:17:01.134 INFO:tasks.workunit.client.0.smithi032.stdout:0/267: dread d3/f2e [0,4194304] 0 2024-04-06T14:17:01.134 INFO:tasks.workunit.client.1.smithi086.stdout:6/104: creat d6/d13/f2a x:0 0 0 2024-04-06T14:17:01.135 INFO:tasks.workunit.client.1.smithi086.stdout:6/105: read - d6/d13/f26 zero size 2024-04-06T14:17:01.135 INFO:tasks.workunit.client.0.smithi032.stdout:6/333: creat d4/d5/d18/f64 x:0 0 0 2024-04-06T14:17:01.137 INFO:tasks.workunit.client.1.smithi086.stdout:7/73: write d7/f9 [4926885,97603] 0 2024-04-06T14:17:01.137 INFO:tasks.workunit.client.0.smithi032.stdout:0/268: truncate d3/da/d25/d2b/f61 629294 0 2024-04-06T14:17:01.137 INFO:tasks.workunit.client.1.smithi086.stdout:7/74: readlink d7/d10/d11/l12 0 2024-04-06T14:17:01.139 INFO:tasks.workunit.client.1.smithi086.stdout:0/150: mknod d0/d1/d1b/c37 0 2024-04-06T14:17:01.140 INFO:tasks.workunit.client.0.smithi032.stdout:2/336: symlink d0/l7e 0 2024-04-06T14:17:01.140 INFO:tasks.workunit.client.1.smithi086.stdout:2/83: mknod d0/d20/c22 0 2024-04-06T14:17:01.142 INFO:tasks.workunit.client.1.smithi086.stdout:8/188: creat d0/d3b/f46 x:0 0 0 2024-04-06T14:17:01.146 INFO:tasks.workunit.client.1.smithi086.stdout:3/174: unlink d5/c10 0 2024-04-06T14:17:01.146 INFO:tasks.workunit.client.1.smithi086.stdout:8/189: creat d0/d8/d22/d43/f47 x:0 0 0 2024-04-06T14:17:01.146 INFO:tasks.workunit.client.1.smithi086.stdout:9/152: getdents d5/d10/d14/d19 0 2024-04-06T14:17:01.147 INFO:tasks.workunit.client.0.smithi032.stdout:4/427: chown d0/df/d2b/f2c 68542 1 2024-04-06T14:17:01.147 INFO:tasks.workunit.client.0.smithi032.stdout:0/269: dwrite d3/da/d25/f54 [0,4194304] 0 2024-04-06T14:17:01.147 INFO:tasks.workunit.client.0.smithi032.stdout:4/428: fdatasync d0/df/d2b/d46/f6e 0 2024-04-06T14:17:01.150 INFO:tasks.workunit.client.0.smithi032.stdout:2/337: dread d0/f23 [0,4194304] 0 2024-04-06T14:17:01.150 INFO:tasks.workunit.client.0.smithi032.stdout:8/333: stat d3/d8/d12/l37 0 2024-04-06T14:17:01.150 INFO:tasks.workunit.client.0.smithi032.stdout:0/270: write d3/f4 [4349598,116796] 0 2024-04-06T14:17:01.151 INFO:tasks.workunit.client.1.smithi086.stdout:6/106: creat d6/f2b x:0 0 0 2024-04-06T14:17:01.151 INFO:tasks.workunit.client.0.smithi032.stdout:4/429: stat d0/df/d2b/d46/d58/c62 0 2024-04-06T14:17:01.152 INFO:tasks.workunit.client.0.smithi032.stdout:2/338: dread - d0/d30/f4d zero size 2024-04-06T14:17:01.153 INFO:tasks.workunit.client.1.smithi086.stdout:0/151: rename d0/d1/d2/d1a/f1d to d0/d1/f38 0 2024-04-06T14:17:01.153 INFO:tasks.workunit.client.1.smithi086.stdout:8/190: dread d0/f17 [0,4194304] 0 2024-04-06T14:17:01.153 INFO:tasks.workunit.client.0.smithi032.stdout:6/334: mkdir d4/d5/d28/d65 0 2024-04-06T14:17:01.154 INFO:tasks.workunit.client.1.smithi086.stdout:9/153: creat d5/d13/f39 x:0 0 0 2024-04-06T14:17:01.156 INFO:tasks.workunit.client.1.smithi086.stdout:8/191: dread - d0/d8/d22/d43/f47 zero size 2024-04-06T14:17:01.157 INFO:tasks.workunit.client.1.smithi086.stdout:8/192: truncate d0/f2 1391292 0 2024-04-06T14:17:01.160 INFO:tasks.workunit.client.0.smithi032.stdout:3/323: truncate d3/d16/f72 2597763 0 2024-04-06T14:17:01.162 INFO:tasks.workunit.client.1.smithi086.stdout:3/175: rename d5/f29 to d5/d26/d2a/f31 0 2024-04-06T14:17:01.163 INFO:tasks.workunit.client.1.smithi086.stdout:8/193: write d0/d13/f40 [490965,29725] 0 2024-04-06T14:17:01.164 INFO:tasks.workunit.client.0.smithi032.stdout:3/324: stat d3/d16/l81 0 2024-04-06T14:17:01.166 INFO:tasks.workunit.client.1.smithi086.stdout:9/154: truncate d5/fc 4141684 0 2024-04-06T14:17:01.168 INFO:tasks.workunit.client.1.smithi086.stdout:5/133: truncate d0/f4 3652033 0 2024-04-06T14:17:01.168 INFO:tasks.workunit.client.0.smithi032.stdout:2/339: dwrite d0/d5a/f2a [0,4194304] 0 2024-04-06T14:17:01.168 INFO:tasks.workunit.client.0.smithi032.stdout:5/356: getdents d16/d18/d33/d45/d52/d39 0 2024-04-06T14:17:01.169 INFO:tasks.workunit.client.1.smithi086.stdout:8/194: chown d0/f5 16742076 1 2024-04-06T14:17:01.170 INFO:tasks.workunit.client.0.smithi032.stdout:8/334: mknod d3/d8/d7/c7f 0 2024-04-06T14:17:01.172 INFO:tasks.workunit.client.1.smithi086.stdout:8/195: write d0/d8/d16/f41 [2312590,17595] 0 2024-04-06T14:17:01.172 INFO:tasks.workunit.client.1.smithi086.stdout:6/107: dwrite d6/d13/f27 [0,4194304] 0 2024-04-06T14:17:01.173 INFO:tasks.workunit.client.0.smithi032.stdout:0/271: write d3/fd [4787531,34837] 0 2024-04-06T14:17:01.173 INFO:tasks.workunit.client.1.smithi086.stdout:3/176: unlink d5/l19 0 2024-04-06T14:17:01.174 INFO:tasks.workunit.client.1.smithi086.stdout:3/177: readlink d5/d26/d2a/l30 0 2024-04-06T14:17:01.174 INFO:tasks.workunit.client.0.smithi032.stdout:6/335: dread d4/f13 [0,4194304] 0 2024-04-06T14:17:01.175 INFO:tasks.workunit.client.0.smithi032.stdout:8/335: write d3/d8/d7/d1a/f5a [99598,67338] 0 2024-04-06T14:17:01.175 INFO:tasks.workunit.client.0.smithi032.stdout:5/357: dread d16/d18/d33/d45/d52/f43 [0,4194304] 0 2024-04-06T14:17:01.177 INFO:tasks.workunit.client.0.smithi032.stdout:0/272: truncate d3/da/f1f 673204 0 2024-04-06T14:17:01.178 INFO:tasks.workunit.client.0.smithi032.stdout:2/340: mkdir d0/d30/d7f 0 2024-04-06T14:17:01.185 INFO:tasks.workunit.client.0.smithi032.stdout:7/393: link d16/c1c d16/d20/d26/c88 0 2024-04-06T14:17:01.185 INFO:tasks.workunit.client.0.smithi032.stdout:2/341: dread d0/d5a/d3a/f5b [0,4194304] 0 2024-04-06T14:17:01.188 INFO:tasks.workunit.client.0.smithi032.stdout:7/394: write d16/d20/d54/f57 [3672392,48592] 0 2024-04-06T14:17:01.194 INFO:tasks.workunit.client.1.smithi086.stdout:8/196: rename d0/d8/d22/d43/l38 to d0/d8/d22/d43/l48 0 2024-04-06T14:17:01.195 INFO:tasks.workunit.client.1.smithi086.stdout:9/155: link d5/d13/d1a/l33 d5/d13/d1a/l3a 0 2024-04-06T14:17:01.196 INFO:tasks.workunit.client.0.smithi032.stdout:1/380: truncate d0/d41/d59/f64 698304 0 2024-04-06T14:17:01.198 INFO:tasks.workunit.client.1.smithi086.stdout:0/152: dwrite d0/d1/d4/d7/f14 [0,4194304] 0 2024-04-06T14:17:01.199 INFO:tasks.workunit.client.1.smithi086.stdout:9/156: dread - d5/d13/f2b zero size 2024-04-06T14:17:01.200 INFO:tasks.workunit.client.0.smithi032.stdout:2/342: dwrite d0/d5a/d3a/f61 [0,4194304] 0 2024-04-06T14:17:01.200 INFO:tasks.workunit.client.1.smithi086.stdout:5/134: read d0/f4 [470791,74337] 0 2024-04-06T14:17:01.201 INFO:tasks.workunit.client.0.smithi032.stdout:2/343: chown d0/d4f 739204 1 2024-04-06T14:17:01.202 INFO:tasks.workunit.client.0.smithi032.stdout:6/336: creat d4/d5/d8/d1f/d61/f66 x:0 0 0 2024-04-06T14:17:01.203 INFO:tasks.workunit.client.0.smithi032.stdout:4/430: write d0/df/d23/f32 [745680,29248] 0 2024-04-06T14:17:01.205 INFO:tasks.workunit.client.0.smithi032.stdout:8/336: link d3/d8/d12/l37 d3/d8/l80 0 2024-04-06T14:17:01.205 INFO:tasks.workunit.client.0.smithi032.stdout:3/325: fsync d3/f4 0 2024-04-06T14:17:01.206 INFO:tasks.workunit.client.1.smithi086.stdout:8/197: dread d0/db/f12 [0,4194304] 0 2024-04-06T14:17:01.206 INFO:tasks.workunit.client.1.smithi086.stdout:9/157: rmdir d5/d10/d14/d19/d1b 39 2024-04-06T14:17:01.207 INFO:tasks.workunit.client.0.smithi032.stdout:1/381: mknod d0/d41/c80 0 2024-04-06T14:17:01.207 INFO:tasks.workunit.client.0.smithi032.stdout:6/337: chown d4/d5/d8/d1f 979705539 1 2024-04-06T14:17:01.209 INFO:tasks.workunit.client.0.smithi032.stdout:5/358: dread f1 [0,4194304] 0 2024-04-06T14:17:01.209 INFO:tasks.workunit.client.0.smithi032.stdout:0/273: dread d3/da/f26 [0,4194304] 0 2024-04-06T14:17:01.210 INFO:tasks.workunit.client.0.smithi032.stdout:2/344: mkdir d0/d30/d39/d59/d67/d80 0 2024-04-06T14:17:01.210 INFO:tasks.workunit.client.0.smithi032.stdout:3/326: chown d3/d3b/f54 507 1 2024-04-06T14:17:01.211 INFO:tasks.workunit.client.1.smithi086.stdout:5/135: readlink d0/d16/l2b 0 2024-04-06T14:17:01.212 INFO:tasks.workunit.client.1.smithi086.stdout:6/108: dwrite f5 [0,4194304] 0 2024-04-06T14:17:01.214 INFO:tasks.workunit.client.0.smithi032.stdout:1/382: write d0/d9/f24 [793599,82569] 0 2024-04-06T14:17:01.214 INFO:tasks.workunit.client.1.smithi086.stdout:8/198: symlink d0/db/d32/l49 0 2024-04-06T14:17:01.215 INFO:tasks.workunit.client.0.smithi032.stdout:5/359: chown d16/f2c 206952480 1 2024-04-06T14:17:01.217 INFO:tasks.workunit.client.1.smithi086.stdout:0/153: getdents d0/d1/d2/d1a 0 2024-04-06T14:17:01.217 INFO:tasks.workunit.client.1.smithi086.stdout:5/136: fsync d0/d16/d18/f1a 0 2024-04-06T14:17:01.218 INFO:tasks.workunit.client.1.smithi086.stdout:5/137: truncate d0/f1 4573279 0 2024-04-06T14:17:01.218 INFO:tasks.workunit.client.1.smithi086.stdout:8/199: chown d0/d8/d16/d39 418239 1 2024-04-06T14:17:01.220 INFO:tasks.workunit.client.1.smithi086.stdout:5/138: symlink d0/l2c 0 2024-04-06T14:17:01.221 INFO:tasks.workunit.client.0.smithi032.stdout:4/431: rename d0/c16 to d0/df/c96 0 2024-04-06T14:17:01.222 INFO:tasks.workunit.client.1.smithi086.stdout:8/200: read - d0/d3b/f46 zero size 2024-04-06T14:17:01.222 INFO:tasks.workunit.client.0.smithi032.stdout:0/274: dwrite d3/da/f1e [0,4194304] 0 2024-04-06T14:17:01.223 INFO:tasks.workunit.client.1.smithi086.stdout:9/158: dread d5/f20 [0,4194304] 0 2024-04-06T14:17:01.225 INFO:tasks.workunit.client.1.smithi086.stdout:5/139: mkdir d0/d17/d2d 0 2024-04-06T14:17:01.226 INFO:tasks.workunit.client.0.smithi032.stdout:0/275: dread d3/f2e [0,4194304] 0 2024-04-06T14:17:01.227 INFO:tasks.workunit.client.0.smithi032.stdout:0/276: chown d3/fd 23806 1 2024-04-06T14:17:01.227 INFO:tasks.workunit.client.1.smithi086.stdout:6/109: dwrite d6/d13/d1a/f21 [0,4194304] 0 2024-04-06T14:17:01.228 INFO:tasks.workunit.client.0.smithi032.stdout:5/360: rmdir d16 39 2024-04-06T14:17:01.228 INFO:tasks.workunit.client.1.smithi086.stdout:9/159: mknod d5/d10/c3b 0 2024-04-06T14:17:01.232 INFO:tasks.workunit.client.1.smithi086.stdout:5/140: dread d0/d16/d18/ff [0,4194304] 0 2024-04-06T14:17:01.232 INFO:tasks.workunit.client.1.smithi086.stdout:6/110: write d6/d9/f10 [5073826,80958] 0 2024-04-06T14:17:01.234 INFO:tasks.workunit.client.0.smithi032.stdout:7/395: write d16/d20/d26/d2d/d30/d3e/f73 [787539,73365] 0 2024-04-06T14:17:01.235 INFO:tasks.workunit.client.1.smithi086.stdout:5/141: chown d0/d16/d18/c25 0 1 2024-04-06T14:17:01.235 INFO:tasks.workunit.client.0.smithi032.stdout:1/383: creat d0/da/d3e/d75/f81 x:0 0 0 2024-04-06T14:17:01.235 INFO:tasks.workunit.client.1.smithi086.stdout:9/160: chown d5/d10/l23 28 1 2024-04-06T14:17:01.237 INFO:tasks.workunit.client.1.smithi086.stdout:6/111: write d6/f7 [1712477,76339] 0 2024-04-06T14:17:01.237 INFO:tasks.workunit.client.0.smithi032.stdout:8/337: write d3/d8/d7/f16 [1190156,126604] 0 2024-04-06T14:17:01.238 INFO:tasks.workunit.client.1.smithi086.stdout:5/142: creat d0/d16/d18/d21/f2e x:0 0 0 2024-04-06T14:17:01.241 INFO:tasks.workunit.client.0.smithi032.stdout:7/396: write d16/d20/f5e [3429764,83982] 0 2024-04-06T14:17:01.243 INFO:tasks.workunit.client.0.smithi032.stdout:3/327: truncate d3/f21 593902 0 2024-04-06T14:17:01.244 INFO:tasks.workunit.client.1.smithi086.stdout:4/96: sync 2024-04-06T14:17:01.248 INFO:tasks.workunit.client.1.smithi086.stdout:5/143: symlink d0/d16/l2f 0 2024-04-06T14:17:01.249 INFO:tasks.workunit.client.1.smithi086.stdout:8/201: write d0/d13/f21 [647556,110108] 0 2024-04-06T14:17:01.249 INFO:tasks.workunit.client.0.smithi032.stdout:2/345: rename d0/d5a/c6d to d0/d5a/d3a/c81 0 2024-04-06T14:17:01.249 INFO:tasks.workunit.client.0.smithi032.stdout:1/384: dwrite d0/d10/d16/f5f [0,4194304] 0 2024-04-06T14:17:01.249 INFO:tasks.workunit.client.1.smithi086.stdout:0/154: truncate d0/f17 1637672 0 2024-04-06T14:17:01.250 INFO:tasks.workunit.client.1.smithi086.stdout:6/112: write d6/d13/f27 [5048036,30378] 0 2024-04-06T14:17:01.251 INFO:tasks.workunit.client.1.smithi086.stdout:5/144: mknod d0/c30 0 2024-04-06T14:17:01.251 INFO:tasks.workunit.client.0.smithi032.stdout:8/338: creat d3/d8/d7/f81 x:0 0 0 2024-04-06T14:17:01.253 INFO:tasks.workunit.client.1.smithi086.stdout:0/155: write d0/d15/d26/d29/f31 [976436,85747] 0 2024-04-06T14:17:01.253 INFO:tasks.workunit.client.1.smithi086.stdout:8/202: symlink d0/d8/d16/l4a 0 2024-04-06T14:17:01.253 INFO:tasks.workunit.client.0.smithi032.stdout:1/385: write d0/f1e [4813005,106750] 0 2024-04-06T14:17:01.253 INFO:tasks.workunit.client.0.smithi032.stdout:3/328: chown d3/fa 25 1 2024-04-06T14:17:01.254 INFO:tasks.workunit.client.1.smithi086.stdout:8/203: readlink d0/db/l2b 0 2024-04-06T14:17:01.255 INFO:tasks.workunit.client.1.smithi086.stdout:9/161: link d5/f27 d5/d10/d14/d19/d1b/f3c 0 2024-04-06T14:17:01.256 INFO:tasks.workunit.client.1.smithi086.stdout:5/145: fsync d0/d16/d18/ff 0 2024-04-06T14:17:01.256 INFO:tasks.workunit.client.0.smithi032.stdout:3/329: truncate d3/d65/f34 231093 0 2024-04-06T14:17:01.256 INFO:tasks.workunit.client.1.smithi086.stdout:8/204: chown d0/f2 62748340 1 2024-04-06T14:17:01.257 INFO:tasks.workunit.client.0.smithi032.stdout:0/277: creat d3/da/d25/d2b/d4a/f62 x:0 0 0 2024-04-06T14:17:01.258 INFO:tasks.workunit.client.0.smithi032.stdout:6/338: rename d4/d5/d8/d1f/d51/d4e to d4/d5/d19/d25/d67 0 2024-04-06T14:17:01.258 INFO:tasks.workunit.client.1.smithi086.stdout:5/146: chown d0/c1c 41429998 1 2024-04-06T14:17:01.260 INFO:tasks.workunit.client.0.smithi032.stdout:7/397: dwrite d16/d20/d54/f76 [0,4194304] 0 2024-04-06T14:17:01.261 INFO:tasks.workunit.client.0.smithi032.stdout:4/432: stat d0/df/d1f/d42/d45 0 2024-04-06T14:17:01.262 INFO:tasks.workunit.client.1.smithi086.stdout:0/156: creat d0/d1/d1b/f39 x:0 0 0 2024-04-06T14:17:01.262 INFO:tasks.workunit.client.1.smithi086.stdout:9/162: dread - d5/d10/d14/d19/f1e zero size 2024-04-06T14:17:01.263 INFO:tasks.workunit.client.0.smithi032.stdout:3/330: symlink d3/d16/d63/l8a 0 2024-04-06T14:17:01.263 INFO:tasks.workunit.client.1.smithi086.stdout:6/113: rename d6/d9/db/c29 to d6/d13/d1a/d22/c2c 0 2024-04-06T14:17:01.264 INFO:tasks.workunit.client.0.smithi032.stdout:7/398: write d16/d20/f5e [2234413,97153] 0 2024-04-06T14:17:01.268 INFO:tasks.workunit.client.0.smithi032.stdout:6/339: mknod d4/d5/d8/d4b/c68 0 2024-04-06T14:17:01.268 INFO:tasks.workunit.client.1.smithi086.stdout:9/163: symlink d5/d10/d14/d19/d1b/l3d 0 2024-04-06T14:17:01.269 INFO:tasks.workunit.client.0.smithi032.stdout:0/278: dwrite d3/da/d25/d36/f39 [0,4194304] 0 2024-04-06T14:17:01.270 INFO:tasks.workunit.client.1.smithi086.stdout:6/114: symlink d6/d13/d1a/l2d 0 2024-04-06T14:17:01.271 INFO:tasks.workunit.client.0.smithi032.stdout:0/279: write d3/fd [4537358,9160] 0 2024-04-06T14:17:01.272 INFO:tasks.workunit.client.0.smithi032.stdout:0/280: dread - d3/f55 zero size 2024-04-06T14:17:01.273 INFO:tasks.workunit.client.1.smithi086.stdout:0/157: mknod d0/d1/c3a 0 2024-04-06T14:17:01.274 INFO:tasks.workunit.client.1.smithi086.stdout:9/164: truncate f0 6679646 0 2024-04-06T14:17:01.277 INFO:tasks.workunit.client.0.smithi032.stdout:4/433: rmdir d0/df/d1f/d42/d45 39 2024-04-06T14:17:01.277 INFO:tasks.workunit.client.1.smithi086.stdout:8/205: getdents d0/db 0 2024-04-06T14:17:01.278 INFO:tasks.workunit.client.1.smithi086.stdout:6/115: symlink d6/d9/db/l2e 0 2024-04-06T14:17:01.278 INFO:tasks.workunit.client.1.smithi086.stdout:0/158: symlink d0/d1/d25/l3b 0 2024-04-06T14:17:01.278 INFO:tasks.workunit.client.1.smithi086.stdout:6/116: dread - d6/f2b zero size 2024-04-06T14:17:01.278 INFO:tasks.workunit.client.1.smithi086.stdout:0/159: stat d0/d1/f38 0 2024-04-06T14:17:01.279 INFO:tasks.workunit.client.1.smithi086.stdout:8/206: readlink d0/d8/d22/d43/l1b 0 2024-04-06T14:17:01.279 INFO:tasks.workunit.client.1.smithi086.stdout:6/117: write d6/d13/d1a/f21 [2705897,35295] 0 2024-04-06T14:17:01.280 INFO:tasks.workunit.client.0.smithi032.stdout:7/399: chown d16/c4e 891587 1 2024-04-06T14:17:01.282 INFO:tasks.workunit.client.0.smithi032.stdout:1/386: creat d0/da/f82 x:0 0 0 2024-04-06T14:17:01.282 INFO:tasks.workunit.client.0.smithi032.stdout:1/387: fsync d0/f4d 0 2024-04-06T14:17:01.282 INFO:tasks.workunit.client.1.smithi086.stdout:0/160: fsync d0/d1/d4/f12 0 2024-04-06T14:17:01.283 INFO:tasks.workunit.client.0.smithi032.stdout:6/340: mkdir d4/d69 0 2024-04-06T14:17:01.283 INFO:tasks.workunit.client.0.smithi032.stdout:1/388: read d0/d10/d16/f5f [1114227,1248] 0 2024-04-06T14:17:01.285 INFO:tasks.workunit.client.1.smithi086.stdout:6/118: link d6/d9/f10 d6/d9/db/f2f 0 2024-04-06T14:17:01.285 INFO:tasks.workunit.client.1.smithi086.stdout:8/207: dread d0/d8/d16/f41 [0,4194304] 0 2024-04-06T14:17:01.286 INFO:tasks.workunit.client.1.smithi086.stdout:0/161: creat d0/d1/d4/de/f3c x:0 0 0 2024-04-06T14:17:01.286 INFO:tasks.workunit.client.0.smithi032.stdout:2/346: write d0/d4e/f53 [1616634,953] 0 2024-04-06T14:17:01.288 INFO:tasks.workunit.client.1.smithi086.stdout:6/119: mknod d6/c30 0 2024-04-06T14:17:01.289 INFO:tasks.workunit.client.1.smithi086.stdout:0/162: mknod d0/d15/d26/d29/c3d 0 2024-04-06T14:17:01.290 INFO:tasks.workunit.client.0.smithi032.stdout:7/400: creat d16/d20/d48/d6d/f89 x:0 0 0 2024-04-06T14:17:01.293 INFO:tasks.workunit.client.1.smithi086.stdout:0/163: creat d0/d15/f3e x:0 0 0 2024-04-06T14:17:01.295 INFO:tasks.workunit.client.0.smithi032.stdout:7/401: creat d16/d20/d26/d5f/f8a x:0 0 0 2024-04-06T14:17:01.296 INFO:tasks.workunit.client.0.smithi032.stdout:2/347: dwrite d0/d30/f36 [0,4194304] 0 2024-04-06T14:17:01.296 INFO:tasks.workunit.client.1.smithi086.stdout:0/164: creat d0/f3f x:0 0 0 2024-04-06T14:17:01.299 INFO:tasks.workunit.client.1.smithi086.stdout:5/147: truncate d0/fa 1753756 0 2024-04-06T14:17:01.300 INFO:tasks.workunit.client.0.smithi032.stdout:4/434: rename d0/lb to d0/df/d1f/d2e/l97 0 2024-04-06T14:17:01.300 INFO:tasks.workunit.client.1.smithi086.stdout:6/120: dwrite d6/d9/db/f2f [0,4194304] 0 2024-04-06T14:17:01.301 INFO:tasks.workunit.client.0.smithi032.stdout:1/389: link d0/d10/d66/f70 d0/d9/f83 0 2024-04-06T14:17:01.301 INFO:tasks.workunit.client.1.smithi086.stdout:0/165: fdatasync d0/ff 0 2024-04-06T14:17:01.304 INFO:tasks.workunit.client.0.smithi032.stdout:2/348: creat d0/d30/d39/d59/f82 x:0 0 0 2024-04-06T14:17:01.304 INFO:tasks.workunit.client.1.smithi086.stdout:5/148: unlink d0/fd 0 2024-04-06T14:17:01.305 INFO:tasks.workunit.client.0.smithi032.stdout:2/349: chown d0/d4e/c62 870047 1 2024-04-06T14:17:01.305 INFO:tasks.workunit.client.0.smithi032.stdout:7/402: rename d16/d20/d26/d2d/d30/c5c to d16/d20/d26/d70/c8b 0 2024-04-06T14:17:01.306 INFO:tasks.workunit.client.1.smithi086.stdout:0/166: dread - d0/d1/d4/de/f2e zero size 2024-04-06T14:17:01.308 INFO:tasks.workunit.client.1.smithi086.stdout:0/167: truncate d0/f2f 957006 0 2024-04-06T14:17:01.308 INFO:tasks.workunit.client.0.smithi032.stdout:1/390: symlink d0/da/d3e/l84 0 2024-04-06T14:17:01.308 INFO:tasks.workunit.client.1.smithi086.stdout:5/149: symlink d0/d17/d2d/l31 0 2024-04-06T14:17:01.310 INFO:tasks.workunit.client.1.smithi086.stdout:0/168: creat d0/f40 x:0 0 0 2024-04-06T14:17:01.310 INFO:tasks.workunit.client.1.smithi086.stdout:6/121: dread d6/d9/db/f19 [4194304,4194304] 0 2024-04-06T14:17:01.311 INFO:tasks.workunit.client.0.smithi032.stdout:7/403: mknod d16/d20/d86/c8c 0 2024-04-06T14:17:01.312 INFO:tasks.workunit.client.1.smithi086.stdout:5/150: symlink d0/d17/d2d/l32 0 2024-04-06T14:17:01.312 INFO:tasks.workunit.client.0.smithi032.stdout:0/281: truncate d3/da/f1d 1586261 0 2024-04-06T14:17:01.313 INFO:tasks.workunit.client.1.smithi086.stdout:6/122: truncate d6/f7 4632306 0 2024-04-06T14:17:01.313 INFO:tasks.workunit.client.1.smithi086.stdout:6/123: stat d6/d9/db/c14 0 2024-04-06T14:17:01.313 INFO:tasks.workunit.client.1.smithi086.stdout:6/124: chown d6/d13/d1a/l2d 2118640669 1 2024-04-06T14:17:01.315 INFO:tasks.workunit.client.0.smithi032.stdout:1/391: read - d0/da/d3e/d75/f81 zero size 2024-04-06T14:17:01.317 INFO:tasks.workunit.client.1.smithi086.stdout:6/125: dread d6/d9/db/f2f [4194304,4194304] 0 2024-04-06T14:17:01.318 INFO:tasks.workunit.client.0.smithi032.stdout:7/404: mkdir d16/d20/d48/d8d 0 2024-04-06T14:17:01.319 INFO:tasks.workunit.client.1.smithi086.stdout:8/208: truncate d0/f17 112467 0 2024-04-06T14:17:01.320 INFO:tasks.workunit.client.1.smithi086.stdout:8/209: fdatasync d0/d3b/f3d 0 2024-04-06T14:17:01.321 INFO:tasks.workunit.client.1.smithi086.stdout:8/210: fdatasync d0/d13/f21 0 2024-04-06T14:17:01.323 INFO:tasks.workunit.client.0.smithi032.stdout:2/350: dwrite d0/d30/f43 [4194304,4194304] 0 2024-04-06T14:17:01.324 INFO:tasks.workunit.client.1.smithi086.stdout:0/169: dwrite d0/d15/d26/f2c [0,4194304] 0 2024-04-06T14:17:01.332 INFO:tasks.workunit.client.1.smithi086.stdout:8/211: dwrite d0/d8/d22/d43/f28 [0,4194304] 0 2024-04-06T14:17:01.332 INFO:tasks.workunit.client.1.smithi086.stdout:6/126: creat d6/d9/f31 x:0 0 0 2024-04-06T14:17:01.334 INFO:tasks.workunit.client.1.smithi086.stdout:8/212: fdatasync d0/d3b/f3d 0 2024-04-06T14:17:01.335 INFO:tasks.workunit.client.1.smithi086.stdout:6/127: fsync d6/d9/f17 0 2024-04-06T14:17:01.339 INFO:tasks.workunit.client.1.smithi086.stdout:8/213: write d0/d8/d16/f41 [2274922,83988] 0 2024-04-06T14:17:01.340 INFO:tasks.workunit.client.1.smithi086.stdout:5/151: dwrite d0/d16/d18/f1a [0,4194304] 0 2024-04-06T14:17:01.342 INFO:tasks.workunit.client.1.smithi086.stdout:6/128: getdents d6 0 2024-04-06T14:17:01.342 INFO:tasks.workunit.client.1.smithi086.stdout:5/152: stat d0/d17 0 2024-04-06T14:17:01.342 INFO:tasks.workunit.client.1.smithi086.stdout:8/214: creat d0/d13/f4b x:0 0 0 2024-04-06T14:17:01.343 INFO:tasks.workunit.client.1.smithi086.stdout:6/129: symlink d6/d20/l32 0 2024-04-06T14:17:01.346 INFO:tasks.workunit.client.1.smithi086.stdout:5/153: mknod d0/d17/d2d/c33 0 2024-04-06T14:17:01.348 INFO:tasks.workunit.client.1.smithi086.stdout:6/130: dread d6/f7 [0,4194304] 0 2024-04-06T14:17:01.350 INFO:tasks.workunit.client.1.smithi086.stdout:5/154: fdatasync d0/d16/d18/d21/d23/f20 0 2024-04-06T14:17:01.353 INFO:tasks.workunit.client.1.smithi086.stdout:6/131: rmdir d6/d13/d1a 39 2024-04-06T14:17:01.354 INFO:tasks.workunit.client.1.smithi086.stdout:6/132: readlink d6/d9/lf 0 2024-04-06T14:17:01.355 INFO:tasks.workunit.client.1.smithi086.stdout:5/155: symlink d0/d16/d18/l34 0 2024-04-06T14:17:01.357 INFO:tasks.workunit.client.1.smithi086.stdout:5/156: write d0/f14 [613452,19163] 0 2024-04-06T14:17:01.359 INFO:tasks.workunit.client.1.smithi086.stdout:6/133: write d6/d13/d1a/f21 [2123288,104709] 0 2024-04-06T14:17:01.359 INFO:tasks.workunit.client.1.smithi086.stdout:5/157: chown d0/d16/d18/d21/f2e 21 1 2024-04-06T14:17:01.362 INFO:tasks.workunit.client.1.smithi086.stdout:6/134: mknod d6/d13/d1a/c33 0 2024-04-06T14:17:01.362 INFO:tasks.workunit.client.1.smithi086.stdout:6/135: write d6/d9/f31 [98509,88116] 0 2024-04-06T14:17:01.363 INFO:tasks.workunit.client.1.smithi086.stdout:6/136: read - d6/d13/f26 zero size 2024-04-06T14:17:01.364 INFO:tasks.workunit.client.1.smithi086.stdout:8/215: rmdir d0 39 2024-04-06T14:17:01.365 INFO:tasks.workunit.client.1.smithi086.stdout:6/137: chown d6/d9/lf 46 1 2024-04-06T14:17:01.367 INFO:tasks.workunit.client.1.smithi086.stdout:8/216: dread - d0/d13/f4b zero size 2024-04-06T14:17:01.368 INFO:tasks.workunit.client.1.smithi086.stdout:5/158: dwrite d0/d16/d18/d21/d23/f1f [0,4194304] 0 2024-04-06T14:17:01.369 INFO:tasks.workunit.client.1.smithi086.stdout:6/138: fsync d6/d9/f10 0 2024-04-06T14:17:01.371 INFO:tasks.workunit.client.1.smithi086.stdout:8/217: unlink d0/d13/f31 0 2024-04-06T14:17:01.373 INFO:tasks.workunit.client.1.smithi086.stdout:6/139: mkdir d6/d13/d34 0 2024-04-06T14:17:01.373 INFO:tasks.workunit.client.1.smithi086.stdout:5/159: dread d0/f1 [0,4194304] 0 2024-04-06T14:17:01.375 INFO:tasks.workunit.client.1.smithi086.stdout:6/140: write d6/d13/f26 [818798,91508] 0 2024-04-06T14:17:01.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:01 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:01.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:01 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:01.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:01 smithi086 ceph-mon[47925]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-04-06T14:17:01.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:01 smithi086 ceph-mon[47925]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-04-06T14:17:01.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:01 smithi086 ceph-mon[47925]: pgmap v18: 65 pgs: 65 active+clean; 545 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 658 KiB/s rd, 35 MiB/s wr, 64 op/s 2024-04-06T14:17:01.378 INFO:tasks.workunit.client.1.smithi086.stdout:8/218: dread d0/f26 [0,4194304] 0 2024-04-06T14:17:01.408 INFO:tasks.workunit.client.1.smithi086.stdout:5/160: unlink d0/d16/d18/d21/d23/f20 0 2024-04-06T14:17:01.410 INFO:tasks.workunit.client.1.smithi086.stdout:8/219: rmdir d0/db 39 2024-04-06T14:17:01.412 INFO:tasks.workunit.client.0.smithi032.stdout:1/392: rename d0/da/f38 to d0/d10/d16/d3f/f85 0 2024-04-06T14:17:01.412 INFO:tasks.workunit.client.0.smithi032.stdout:6/341: dwrite d4/d5/d34/f5f [0,4194304] 0 2024-04-06T14:17:01.413 INFO:tasks.workunit.client.0.smithi032.stdout:3/331: dwrite d3/d65/f32 [0,4194304] 0 2024-04-06T14:17:01.413 INFO:tasks.workunit.client.1.smithi086.stdout:6/141: truncate d6/d9/f10 3491016 0 2024-04-06T14:17:01.414 INFO:tasks.workunit.client.1.smithi086.stdout:8/220: readlink d0/db/l2b 0 2024-04-06T14:17:01.415 INFO:tasks.workunit.client.1.smithi086.stdout:8/221: readlink d0/db/l2b 0 2024-04-06T14:17:01.417 INFO:tasks.workunit.client.1.smithi086.stdout:5/161: mkdir d0/d16/d18/d35 0 2024-04-06T14:17:01.420 INFO:tasks.workunit.client.1.smithi086.stdout:6/142: creat d6/d13/d1a/d22/f35 x:0 0 0 2024-04-06T14:17:01.420 INFO:tasks.workunit.client.1.smithi086.stdout:6/143: chown d6/d9 12988907 1 2024-04-06T14:17:01.421 INFO:tasks.workunit.client.0.smithi032.stdout:1/393: fsync d0/d10/f26 0 2024-04-06T14:17:01.421 INFO:tasks.workunit.client.1.smithi086.stdout:8/222: rmdir d0 39 2024-04-06T14:17:01.424 INFO:tasks.workunit.client.0.smithi032.stdout:1/394: dread - d0/d9/f83 zero size 2024-04-06T14:17:01.426 INFO:tasks.workunit.client.0.smithi032.stdout:2/351: dwrite d0/d5a/f1f [4194304,4194304] 0 2024-04-06T14:17:01.426 INFO:tasks.workunit.client.0.smithi032.stdout:3/332: mknod d3/d65/d89/c8b 0 2024-04-06T14:17:01.428 INFO:tasks.workunit.client.1.smithi086.stdout:5/162: rename d0/d16/l2f to d0/d17/d2d/l36 0 2024-04-06T14:17:01.429 INFO:tasks.workunit.client.1.smithi086.stdout:5/163: symlink d0/d16/d18/d21/d23/l37 0 2024-04-06T14:17:01.431 INFO:tasks.workunit.client.1.smithi086.stdout:8/223: creat d0/d3b/f4c x:0 0 0 2024-04-06T14:17:01.431 INFO:tasks.workunit.client.0.smithi032.stdout:3/333: dread - d3/d3b/d3c/f79 zero size 2024-04-06T14:17:01.433 INFO:tasks.workunit.client.1.smithi086.stdout:8/224: fsync d0/d8/d22/d43/f28 0 2024-04-06T14:17:01.434 INFO:tasks.workunit.client.0.smithi032.stdout:6/342: dwrite d4/d5/d8/d1f/d51/f3d [0,4194304] 0 2024-04-06T14:17:01.435 INFO:tasks.workunit.client.1.smithi086.stdout:8/225: mkdir d0/d8/d22/d43/d4d 0 2024-04-06T14:17:01.436 INFO:tasks.workunit.client.0.smithi032.stdout:2/352: dread d0/d30/f43 [0,4194304] 0 2024-04-06T14:17:01.436 INFO:tasks.workunit.client.0.smithi032.stdout:1/395: rename d0/d10/d16/l33 to d0/l86 0 2024-04-06T14:17:01.437 INFO:tasks.workunit.client.0.smithi032.stdout:6/343: write d4/d5/d19/d21/f30 [864734,118649] 0 2024-04-06T14:17:01.441 INFO:tasks.workunit.client.0.smithi032.stdout:6/344: write d4/d5/d18/d23/f50 [779387,102393] 0 2024-04-06T14:17:01.443 INFO:tasks.workunit.client.0.smithi032.stdout:7/405: dread d16/d20/d26/f3a [0,4194304] 0 2024-04-06T14:17:01.443 INFO:tasks.workunit.client.0.smithi032.stdout:3/334: rmdir d3/d3b/d3c/d87 0 2024-04-06T14:17:01.444 INFO:tasks.workunit.client.0.smithi032.stdout:1/396: symlink d0/d41/d5b/l87 0 2024-04-06T14:17:01.446 INFO:tasks.workunit.client.0.smithi032.stdout:6/345: getdents d4/d5/d8/d1f/d51/d5e 0 2024-04-06T14:17:01.447 INFO:tasks.workunit.client.1.smithi086.stdout:8/226: dwrite d0/db/f12 [0,4194304] 0 2024-04-06T14:17:01.448 INFO:tasks.workunit.client.0.smithi032.stdout:3/335: creat d3/d65/f8c x:0 0 0 2024-04-06T14:17:01.448 INFO:tasks.workunit.client.0.smithi032.stdout:1/397: fdatasync d0/da/f6b 0 2024-04-06T14:17:01.449 INFO:tasks.workunit.client.0.smithi032.stdout:7/406: truncate d16/d20/d54/f76 4583489 0 2024-04-06T14:17:01.450 INFO:tasks.workunit.client.1.smithi086.stdout:1/65: sync 2024-04-06T14:17:01.453 INFO:tasks.workunit.client.1.smithi086.stdout:6/144: dread d6/d9/db/f2f [0,4194304] 0 2024-04-06T14:17:01.454 INFO:tasks.workunit.client.0.smithi032.stdout:1/398: symlink d0/da/d3e/d75/l88 0 2024-04-06T14:17:01.454 INFO:tasks.workunit.client.1.smithi086.stdout:5/164: truncate d0/d16/d18/d21/d23/f1f 2528788 0 2024-04-06T14:17:01.456 INFO:tasks.workunit.client.1.smithi086.stdout:5/165: chown d0/f6 8431 1 2024-04-06T14:17:01.456 INFO:tasks.workunit.client.0.smithi032.stdout:7/407: write d16/d20/d26/d70/f7f [242745,111216] 0 2024-04-06T14:17:01.457 INFO:tasks.workunit.client.0.smithi032.stdout:2/353: dwrite d0/d6e/f72 [0,4194304] 0 2024-04-06T14:17:01.459 INFO:tasks.workunit.client.1.smithi086.stdout:8/227: getdents d0/d8 0 2024-04-06T14:17:01.460 INFO:tasks.workunit.client.0.smithi032.stdout:6/346: rename d4/d5/d18/d23/f27 to d4/d5/f6a 0 2024-04-06T14:17:01.460 INFO:tasks.workunit.client.0.smithi032.stdout:1/399: chown d0/d9/f24 1397808 1 2024-04-06T14:17:01.461 INFO:tasks.workunit.client.1.smithi086.stdout:8/228: chown d0/d3b/f4c 836059 1 2024-04-06T14:17:01.464 INFO:tasks.workunit.client.0.smithi032.stdout:7/408: mknod d16/d20/d63/c8e 0 2024-04-06T14:17:01.464 INFO:tasks.workunit.client.0.smithi032.stdout:1/400: chown d0/da/d3e/l84 403689 1 2024-04-06T14:17:01.466 INFO:tasks.workunit.client.0.smithi032.stdout:6/347: read d4/d5/d18/f31 [1172528,104950] 0 2024-04-06T14:17:01.467 INFO:tasks.workunit.client.1.smithi086.stdout:6/145: dwrite d6/d13/d1a/d22/f35 [0,4194304] 0 2024-04-06T14:17:01.467 INFO:tasks.workunit.client.1.smithi086.stdout:1/66: creat db/d13/f14 x:0 0 0 2024-04-06T14:17:01.467 INFO:tasks.workunit.client.1.smithi086.stdout:1/67: write db/fe [24734,76976] 0 2024-04-06T14:17:01.467 INFO:tasks.workunit.client.1.smithi086.stdout:8/229: mknod d0/d8/d16/c4e 0 2024-04-06T14:17:01.467 INFO:tasks.workunit.client.1.smithi086.stdout:1/68: stat c6 0 2024-04-06T14:17:01.468 INFO:tasks.workunit.client.0.smithi032.stdout:3/336: dwrite d3/d16/f18 [0,4194304] 0 2024-04-06T14:17:01.471 INFO:tasks.workunit.client.1.smithi086.stdout:6/146: mkdir d6/d9/d36 0 2024-04-06T14:17:01.471 INFO:tasks.workunit.client.1.smithi086.stdout:8/230: truncate d0/d13/f4b 531791 0 2024-04-06T14:17:01.471 INFO:tasks.workunit.client.1.smithi086.stdout:1/69: mknod db/d11/c15 0 2024-04-06T14:17:01.472 INFO:tasks.workunit.client.1.smithi086.stdout:6/147: symlink d6/d13/d1a/l37 0 2024-04-06T14:17:01.473 INFO:tasks.workunit.client.0.smithi032.stdout:1/401: unlink d0/d9/c55 0 2024-04-06T14:17:01.473 INFO:tasks.workunit.client.0.smithi032.stdout:6/348: mknod d4/d5/d8/d2f/c6b 0 2024-04-06T14:17:01.473 INFO:tasks.workunit.client.0.smithi032.stdout:3/337: rmdir d3/d3b/d3c 39 2024-04-06T14:17:01.476 INFO:tasks.workunit.client.0.smithi032.stdout:2/354: dwrite d0/d5a/f1a [4194304,4194304] 0 2024-04-06T14:17:01.477 INFO:tasks.workunit.client.0.smithi032.stdout:3/338: mkdir d3/d16/d63/d8d 0 2024-04-06T14:17:01.477 INFO:tasks.workunit.client.0.smithi032.stdout:3/339: chown d3/d65/d7d 966908015 1 2024-04-06T14:17:01.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:01 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:01.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:01 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:01.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:01 smithi032 ceph-mon[39730]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-04-06T14:17:01.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:01 smithi032 ceph-mon[39730]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-04-06T14:17:01.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:01 smithi032 ceph-mon[39730]: pgmap v18: 65 pgs: 65 active+clean; 545 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 658 KiB/s rd, 35 MiB/s wr, 64 op/s 2024-04-06T14:17:01.481 INFO:tasks.workunit.client.0.smithi032.stdout:1/402: rename d0/d9/c61 to d0/d41/d59/c89 0 2024-04-06T14:17:01.481 INFO:tasks.workunit.client.0.smithi032.stdout:3/340: chown d3/d65/f52 1923839 1 2024-04-06T14:17:01.481 INFO:tasks.workunit.client.0.smithi032.stdout:6/349: symlink d4/d5/d8/d2b/l6c 0 2024-04-06T14:17:01.482 INFO:tasks.workunit.client.0.smithi032.stdout:2/355: write d0/f24 [4468574,42036] 0 2024-04-06T14:17:01.483 INFO:tasks.workunit.client.0.smithi032.stdout:2/356: chown d0/d5a/d3a 20668 1 2024-04-06T14:17:01.484 INFO:tasks.workunit.client.0.smithi032.stdout:3/341: creat d3/d16/d63/f8e x:0 0 0 2024-04-06T14:17:01.486 INFO:tasks.workunit.client.0.smithi032.stdout:6/350: creat d4/f6d x:0 0 0 2024-04-06T14:17:01.490 INFO:tasks.workunit.client.1.smithi086.stdout:1/70: rmdir db/d11 39 2024-04-06T14:17:01.490 INFO:tasks.workunit.client.1.smithi086.stdout:1/71: readlink - no filename 2024-04-06T14:17:01.491 INFO:tasks.workunit.client.1.smithi086.stdout:6/148: write d6/d9/db/f19 [4898501,19407] 0 2024-04-06T14:17:01.491 INFO:tasks.workunit.client.1.smithi086.stdout:5/166: write d0/d16/d18/d21/d23/f1f [682737,95763] 0 2024-04-06T14:17:01.492 INFO:tasks.workunit.client.0.smithi032.stdout:1/403: dwrite d0/da/d3e/f6a [0,4194304] 0 2024-04-06T14:17:01.494 INFO:tasks.workunit.client.0.smithi032.stdout:6/351: unlink d4/d5/d8/d1f/d51/f3d 0 2024-04-06T14:17:01.495 INFO:tasks.workunit.client.1.smithi086.stdout:5/167: write d0/d16/d18/f1a [2072517,6103] 0 2024-04-06T14:17:01.496 INFO:tasks.workunit.client.0.smithi032.stdout:6/352: readlink d4/d5/d8/d2f/l3a 0 2024-04-06T14:17:01.496 INFO:tasks.workunit.client.1.smithi086.stdout:6/149: unlink d6/d9/db/fc 0 2024-04-06T14:17:01.497 INFO:tasks.workunit.client.0.smithi032.stdout:3/342: write d3/d3b/d3c/f79 [710605,21283] 0 2024-04-06T14:17:01.498 INFO:tasks.workunit.client.0.smithi032.stdout:1/404: rename d0/d10/d16/f62 to d0/da/d3e/d75/f8a 0 2024-04-06T14:17:01.500 INFO:tasks.workunit.client.0.smithi032.stdout:6/353: mkdir d4/d5/d28/d65/d6e 0 2024-04-06T14:17:01.500 INFO:tasks.workunit.client.1.smithi086.stdout:8/231: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:17:01.501 INFO:tasks.workunit.client.1.smithi086.stdout:1/72: creat db/f16 x:0 0 0 2024-04-06T14:17:01.501 INFO:tasks.workunit.client.0.smithi032.stdout:3/343: dread d3/d16/f2c [0,4194304] 0 2024-04-06T14:17:01.502 INFO:tasks.workunit.client.0.smithi032.stdout:1/405: write d0/d10/f3d [7855207,85570] 0 2024-04-06T14:17:01.502 INFO:tasks.workunit.client.0.smithi032.stdout:6/354: dread - d4/d5/d35/f59 zero size 2024-04-06T14:17:01.503 INFO:tasks.workunit.client.0.smithi032.stdout:1/406: chown d0/d10/d66/l68 288 1 2024-04-06T14:17:01.504 INFO:tasks.workunit.client.0.smithi032.stdout:1/407: write d0/d10/d16/f51 [1274816,112735] 0 2024-04-06T14:17:01.505 INFO:tasks.workunit.client.1.smithi086.stdout:6/150: creat d6/d9/db/f38 x:0 0 0 2024-04-06T14:17:01.506 INFO:tasks.workunit.client.0.smithi032.stdout:3/344: creat d3/d3b/d3c/f8f x:0 0 0 2024-04-06T14:17:01.507 INFO:tasks.workunit.client.0.smithi032.stdout:3/345: chown d3/d3b/c7f 61616861 1 2024-04-06T14:17:01.508 INFO:tasks.workunit.client.0.smithi032.stdout:6/355: mkdir d4/d5/d28/d65/d6f 0 2024-04-06T14:17:01.509 INFO:tasks.workunit.client.1.smithi086.stdout:1/73: creat db/d11/f17 x:0 0 0 2024-04-06T14:17:01.510 INFO:tasks.workunit.client.0.smithi032.stdout:3/346: creat d3/d3b/d3c/f90 x:0 0 0 2024-04-06T14:17:01.511 INFO:tasks.workunit.client.1.smithi086.stdout:8/232: dread d0/d13/f21 [0,4194304] 0 2024-04-06T14:17:01.511 INFO:tasks.workunit.client.0.smithi032.stdout:3/347: fdatasync d3/d3b/f62 0 2024-04-06T14:17:01.512 INFO:tasks.workunit.client.0.smithi032.stdout:6/356: unlink d4/d5/d8/d4b/f48 0 2024-04-06T14:17:01.513 INFO:tasks.workunit.client.1.smithi086.stdout:1/74: creat db/f18 x:0 0 0 2024-04-06T14:17:01.514 INFO:tasks.workunit.client.1.smithi086.stdout:8/233: mknod d0/d13/c4f 0 2024-04-06T14:17:01.514 INFO:tasks.workunit.client.0.smithi032.stdout:3/348: symlink d3/d65/d89/l91 0 2024-04-06T14:17:01.515 INFO:tasks.workunit.client.1.smithi086.stdout:1/75: creat db/d11/f19 x:0 0 0 2024-04-06T14:17:01.516 INFO:tasks.workunit.client.0.smithi032.stdout:1/408: dread d0/d10/d16/f3c [0,4194304] 0 2024-04-06T14:17:01.516 INFO:tasks.workunit.client.1.smithi086.stdout:1/76: read - db/f16 zero size 2024-04-06T14:17:01.518 INFO:tasks.workunit.client.1.smithi086.stdout:8/234: dread d0/f34 [0,4194304] 0 2024-04-06T14:17:01.518 INFO:tasks.workunit.client.1.smithi086.stdout:8/235: stat d0/c27 0 2024-04-06T14:17:01.519 INFO:tasks.workunit.client.1.smithi086.stdout:8/236: read d0/f34 [3074641,1259] 0 2024-04-06T14:17:01.523 INFO:tasks.workunit.client.0.smithi032.stdout:6/357: read d4/d5/d18/f31 [440267,95417] 0 2024-04-06T14:17:01.525 INFO:tasks.workunit.client.1.smithi086.stdout:8/237: unlink d0/f17 0 2024-04-06T14:17:01.526 INFO:tasks.workunit.client.0.smithi032.stdout:3/349: dwrite d3/d3b/d3c/f78 [0,4194304] 0 2024-04-06T14:17:01.526 INFO:tasks.workunit.client.1.smithi086.stdout:5/168: write d0/f6 [594670,107538] 0 2024-04-06T14:17:01.528 INFO:tasks.workunit.client.1.smithi086.stdout:8/238: symlink d0/d8/d16/l50 0 2024-04-06T14:17:01.529 INFO:tasks.workunit.client.0.smithi032.stdout:1/409: dwrite d0/da/d3e/d75/f81 [0,4194304] 0 2024-04-06T14:17:01.532 INFO:tasks.workunit.client.1.smithi086.stdout:8/239: fdatasync d0/f26 0 2024-04-06T14:17:01.535 INFO:tasks.workunit.client.1.smithi086.stdout:5/169: dwrite d0/d16/d18/d21/d23/f1f [0,4194304] 0 2024-04-06T14:17:01.535 INFO:tasks.workunit.client.1.smithi086.stdout:8/240: write d0/d8/d22/d43/f28 [3512713,42900] 0 2024-04-06T14:17:01.536 INFO:tasks.workunit.client.0.smithi032.stdout:2/357: dwrite d0/d30/d39/f58 [0,4194304] 0 2024-04-06T14:17:01.538 INFO:tasks.workunit.client.0.smithi032.stdout:2/358: readlink d0/ld 0 2024-04-06T14:17:01.539 INFO:tasks.workunit.client.0.smithi032.stdout:6/358: dwrite d4/d5/d19/f2e [0,4194304] 0 2024-04-06T14:17:01.540 INFO:tasks.workunit.client.0.smithi032.stdout:3/350: link d3/d16/c4f d3/d65/c92 0 2024-04-06T14:17:01.540 INFO:tasks.workunit.client.0.smithi032.stdout:1/410: write d0/d10/d16/d3f/f85 [3484092,71265] 0 2024-04-06T14:17:01.542 INFO:tasks.workunit.client.1.smithi086.stdout:5/170: fdatasync d0/d16/d18/d21/f2e 0 2024-04-06T14:17:01.542 INFO:tasks.workunit.client.0.smithi032.stdout:6/359: truncate d4/d5/d18/f64 220485 0 2024-04-06T14:17:01.544 INFO:tasks.workunit.client.1.smithi086.stdout:8/241: creat d0/d3b/f51 x:0 0 0 2024-04-06T14:17:01.544 INFO:tasks.workunit.client.0.smithi032.stdout:6/360: readlink d4/d5/d8/d1f/d51/l26 0 2024-04-06T14:17:01.545 INFO:tasks.workunit.client.0.smithi032.stdout:3/351: creat d3/d65/f93 x:0 0 0 2024-04-06T14:17:01.545 INFO:tasks.workunit.client.0.smithi032.stdout:9/382: sync 2024-04-06T14:17:01.545 INFO:tasks.workunit.client.1.smithi086.stdout:1/77: dwrite f7 [0,4194304] 0 2024-04-06T14:17:01.547 INFO:tasks.workunit.client.1.smithi086.stdout:5/171: truncate d0/d19/f26 870034 0 2024-04-06T14:17:01.547 INFO:tasks.workunit.client.0.smithi032.stdout:2/359: dwrite d0/d4e/f69 [0,4194304] 0 2024-04-06T14:17:01.556 INFO:tasks.workunit.client.0.smithi032.stdout:9/383: dwrite d2/d25/d31/f3a [0,4194304] 0 2024-04-06T14:17:01.559 INFO:tasks.workunit.client.0.smithi032.stdout:2/360: dwrite d0/d30/f36 [0,4194304] 0 2024-04-06T14:17:01.563 INFO:tasks.workunit.client.1.smithi086.stdout:8/242: getdents d0/d8/d16 0 2024-04-06T14:17:01.566 INFO:tasks.workunit.client.1.smithi086.stdout:1/78: dwrite db/fe [0,4194304] 0 2024-04-06T14:17:01.567 INFO:tasks.workunit.client.0.smithi032.stdout:9/384: rmdir d2/d15/d55 39 2024-04-06T14:17:01.568 INFO:tasks.workunit.client.0.smithi032.stdout:3/352: link d3/d65/l30 d3/d65/d89/l94 0 2024-04-06T14:17:01.569 INFO:tasks.workunit.client.1.smithi086.stdout:5/172: dread d0/d16/d18/d21/d23/f1f [0,4194304] 0 2024-04-06T14:17:01.569 INFO:tasks.workunit.client.0.smithi032.stdout:3/353: stat d3/d3b/d3c/l73 0 2024-04-06T14:17:01.571 INFO:tasks.workunit.client.0.smithi032.stdout:2/361: mknod d0/d30/d39/d59/d67/d74/d78/d57/d7c/c83 0 2024-04-06T14:17:01.577 INFO:tasks.workunit.client.1.smithi086.stdout:8/243: mkdir d0/d8/d16/d52 0 2024-04-06T14:17:01.578 INFO:tasks.workunit.client.1.smithi086.stdout:1/79: mkdir db/d11/d1a 0 2024-04-06T14:17:01.578 INFO:tasks.workunit.client.1.smithi086.stdout:5/173: symlink d0/d16/d18/d21/d23/d28/l38 0 2024-04-06T14:17:01.578 INFO:tasks.workunit.client.1.smithi086.stdout:5/174: chown d0/f4 14293 1 2024-04-06T14:17:01.578 INFO:tasks.workunit.client.0.smithi032.stdout:9/385: link d2/d25/c6f d2/d15/d1e/d47/d77/c7d 0 2024-04-06T14:17:01.580 INFO:tasks.workunit.client.0.smithi032.stdout:9/386: write d2/d15/d1e/d47/d4a/f68 [927176,60050] 0 2024-04-06T14:17:01.581 INFO:tasks.workunit.client.1.smithi086.stdout:5/175: stat d0/d17/d2d/l36 0 2024-04-06T14:17:01.582 INFO:tasks.workunit.client.1.smithi086.stdout:1/80: creat db/d11/d1a/f1b x:0 0 0 2024-04-06T14:17:01.584 INFO:tasks.workunit.client.0.smithi032.stdout:3/354: rmdir d3/d16/d63/d8d 0 2024-04-06T14:17:01.585 INFO:tasks.workunit.client.1.smithi086.stdout:5/176: rename d0/f14 to d0/d16/f39 0 2024-04-06T14:17:01.586 INFO:tasks.workunit.client.0.smithi032.stdout:9/387: rename d2/d5 to d2/d25/d7e 0 2024-04-06T14:17:01.586 INFO:tasks.workunit.client.1.smithi086.stdout:8/244: creat d0/d8/d22/d43/d4d/f53 x:0 0 0 2024-04-06T14:17:01.587 INFO:tasks.workunit.client.0.smithi032.stdout:9/388: fdatasync d2/d15/f43 0 2024-04-06T14:17:01.588 INFO:tasks.workunit.client.1.smithi086.stdout:1/81: mknod db/d11/d1a/c1c 0 2024-04-06T14:17:01.588 INFO:tasks.workunit.client.0.smithi032.stdout:3/355: link d3/d3b/f40 d3/d65/f95 0 2024-04-06T14:17:01.589 INFO:tasks.workunit.client.0.smithi032.stdout:9/389: write d2/d15/d1e/d41/f63 [537270,119616] 0 2024-04-06T14:17:01.591 INFO:tasks.workunit.client.1.smithi086.stdout:8/245: fdatasync d0/db/f19 0 2024-04-06T14:17:01.592 INFO:tasks.workunit.client.0.smithi032.stdout:3/356: dread d3/f37 [0,4194304] 0 2024-04-06T14:17:01.592 INFO:tasks.workunit.client.0.smithi032.stdout:9/390: truncate d2/d15/d55/f6e 238361 0 2024-04-06T14:17:01.592 INFO:tasks.workunit.client.1.smithi086.stdout:1/82: mkdir db/d11/d1d 0 2024-04-06T14:17:01.593 INFO:tasks.workunit.client.0.smithi032.stdout:3/357: truncate d3/d3b/f6c 301713 0 2024-04-06T14:17:01.596 INFO:tasks.workunit.client.1.smithi086.stdout:1/83: getdents db/d11/d1d 0 2024-04-06T14:17:01.597 INFO:tasks.workunit.client.0.smithi032.stdout:3/358: getdents d3/d3b 0 2024-04-06T14:17:01.598 INFO:tasks.workunit.client.0.smithi032.stdout:3/359: write d3/d65/f34 [1212633,56887] 0 2024-04-06T14:17:01.600 INFO:tasks.workunit.client.0.smithi032.stdout:9/391: dwrite d2/d25/d7e/f1b [0,4194304] 0 2024-04-06T14:17:01.601 INFO:tasks.workunit.client.0.smithi032.stdout:3/360: creat d3/d3b/d3c/f96 x:0 0 0 2024-04-06T14:17:01.602 INFO:tasks.workunit.client.0.smithi032.stdout:3/361: truncate d3/f7a 396404 0 2024-04-06T14:17:01.603 INFO:tasks.workunit.client.1.smithi086.stdout:1/84: dwrite db/fe [0,4194304] 0 2024-04-06T14:17:01.604 INFO:tasks.workunit.client.0.smithi032.stdout:9/392: mkdir d2/d15/d1e/d65/d6b/d7f 0 2024-04-06T14:17:01.608 INFO:tasks.workunit.client.1.smithi086.stdout:1/85: unlink db/f18 0 2024-04-06T14:17:01.609 INFO:tasks.workunit.client.0.smithi032.stdout:9/393: fsync d2/d15/f29 0 2024-04-06T14:17:01.610 INFO:tasks.workunit.client.0.smithi032.stdout:3/362: dwrite d3/d3b/d3c/f90 [0,4194304] 0 2024-04-06T14:17:01.610 INFO:tasks.workunit.client.1.smithi086.stdout:1/86: creat db/d13/f1e x:0 0 0 2024-04-06T14:17:01.611 INFO:tasks.workunit.client.1.smithi086.stdout:1/87: stat db/fc 0 2024-04-06T14:17:01.612 INFO:tasks.workunit.client.1.smithi086.stdout:1/88: chown db/d13/f14 2 1 2024-04-06T14:17:01.612 INFO:tasks.workunit.client.0.smithi032.stdout:6/361: write d4/d5/d18/f49 [1026534,104704] 0 2024-04-06T14:17:01.613 INFO:tasks.workunit.client.1.smithi086.stdout:1/89: write db/f16 [294608,60603] 0 2024-04-06T14:17:01.615 INFO:tasks.workunit.client.1.smithi086.stdout:1/90: write db/ff [302962,47501] 0 2024-04-06T14:17:01.615 INFO:tasks.workunit.client.0.smithi032.stdout:3/363: creat d3/d65/d89/f97 x:0 0 0 2024-04-06T14:17:01.619 INFO:tasks.workunit.client.0.smithi032.stdout:6/362: mkdir d4/d5/d8/d1f/d51/d70 0 2024-04-06T14:17:01.620 INFO:tasks.workunit.client.1.smithi086.stdout:1/91: mkdir db/d11/d1f 0 2024-04-06T14:17:01.620 INFO:tasks.workunit.client.0.smithi032.stdout:3/364: mknod d3/c98 0 2024-04-06T14:17:01.620 INFO:tasks.workunit.client.1.smithi086.stdout:1/92: fsync f2 0 2024-04-06T14:17:01.622 INFO:tasks.workunit.client.0.smithi032.stdout:1/411: dwrite d0/d10/d16/f7a [0,4194304] 0 2024-04-06T14:17:01.626 INFO:tasks.workunit.client.0.smithi032.stdout:1/412: chown d0/d10/l56 116 1 2024-04-06T14:17:01.626 INFO:tasks.workunit.client.0.smithi032.stdout:9/394: dwrite d2/d25/d7e/f37 [0,4194304] 0 2024-04-06T14:17:01.628 INFO:tasks.workunit.client.0.smithi032.stdout:1/413: chown d0/da/f63 22750261 1 2024-04-06T14:17:01.629 INFO:tasks.workunit.client.0.smithi032.stdout:1/414: dread - d0/da/f82 zero size 2024-04-06T14:17:01.632 INFO:tasks.workunit.client.1.smithi086.stdout:5/177: getdents d0/d16/d18/d21/d23/d28 0 2024-04-06T14:17:01.632 INFO:tasks.workunit.client.0.smithi032.stdout:3/365: dread d3/d16/f5c [0,4194304] 0 2024-04-06T14:17:01.633 INFO:tasks.workunit.client.0.smithi032.stdout:2/362: truncate d0/d5a/d3a/f42 3618620 0 2024-04-06T14:17:01.634 INFO:tasks.workunit.client.1.smithi086.stdout:8/246: write d0/d8/d16/d39/f20 [2759247,122851] 0 2024-04-06T14:17:01.634 INFO:tasks.workunit.client.0.smithi032.stdout:1/415: symlink d0/d9/d1f/d72/l8b 0 2024-04-06T14:17:01.635 INFO:tasks.workunit.client.1.smithi086.stdout:8/247: fsync d0/d8/d22/d43/d4d/f53 0 2024-04-06T14:17:01.635 INFO:tasks.workunit.client.0.smithi032.stdout:2/363: fdatasync d0/d5a/f2a 0 2024-04-06T14:17:01.636 INFO:tasks.workunit.client.0.smithi032.stdout:9/395: creat d2/d15/d1e/d47/d4a/d73/d7c/f80 x:0 0 0 2024-04-06T14:17:01.637 INFO:tasks.workunit.client.0.smithi032.stdout:9/396: stat d2/c11 0 2024-04-06T14:17:01.637 INFO:tasks.workunit.client.1.smithi086.stdout:7/75: sync 2024-04-06T14:17:01.637 INFO:tasks.workunit.client.1.smithi086.stdout:8/248: write d0/d8/d22/d43/d4d/f53 [782626,77879] 0 2024-04-06T14:17:01.637 INFO:tasks.workunit.client.1.smithi086.stdout:5/178: creat d0/d16/d18/d35/f3a x:0 0 0 2024-04-06T14:17:01.638 INFO:tasks.workunit.client.1.smithi086.stdout:5/179: chown d0/c30 66 1 2024-04-06T14:17:01.640 INFO:tasks.workunit.client.0.smithi032.stdout:6/363: dread d4/d5/f15 [0,4194304] 0 2024-04-06T14:17:01.640 INFO:tasks.workunit.client.0.smithi032.stdout:2/364: read d0/f22 [2502840,23249] 0 2024-04-06T14:17:01.641 INFO:tasks.workunit.client.1.smithi086.stdout:5/180: mknod d0/d16/c3b 0 2024-04-06T14:17:01.642 INFO:tasks.workunit.client.1.smithi086.stdout:8/249: mknod d0/d8/d22/d43/d4d/c54 0 2024-04-06T14:17:01.643 INFO:tasks.workunit.client.0.smithi032.stdout:3/366: link d3/d3b/c42 d3/d65/d89/c99 0 2024-04-06T14:17:01.644 INFO:tasks.workunit.client.0.smithi032.stdout:2/365: chown d0/c29 7 1 2024-04-06T14:17:01.644 INFO:tasks.workunit.client.1.smithi086.stdout:8/250: write d0/d13/f4b [850025,9988] 0 2024-04-06T14:17:01.645 INFO:tasks.workunit.client.1.smithi086.stdout:1/93: fdatasync db/ff 0 2024-04-06T14:17:01.645 INFO:tasks.workunit.client.1.smithi086.stdout:7/76: dwrite d7/fb [0,4194304] 0 2024-04-06T14:17:01.646 INFO:tasks.workunit.client.0.smithi032.stdout:9/397: dwrite d2/d25/d31/f57 [0,4194304] 0 2024-04-06T14:17:01.647 INFO:tasks.workunit.client.1.smithi086.stdout:8/251: rmdir d0/d8/d16 39 2024-04-06T14:17:01.648 INFO:tasks.workunit.client.0.smithi032.stdout:2/366: unlink d0/d5a/l56 0 2024-04-06T14:17:01.649 INFO:tasks.workunit.client.1.smithi086.stdout:1/94: mknod db/d11/c20 0 2024-04-06T14:17:01.650 INFO:tasks.workunit.client.0.smithi032.stdout:9/398: truncate d2/d25/d31/d39/f5a 697563 0 2024-04-06T14:17:01.651 INFO:tasks.workunit.client.1.smithi086.stdout:7/77: creat d7/d10/d11/f15 x:0 0 0 2024-04-06T14:17:01.652 INFO:tasks.workunit.client.0.smithi032.stdout:1/416: dread d0/d9/f24 [0,4194304] 0 2024-04-06T14:17:01.653 INFO:tasks.workunit.client.0.smithi032.stdout:9/399: dread d2/d15/d55/f6e [0,4194304] 0 2024-04-06T14:17:01.653 INFO:tasks.workunit.client.0.smithi032.stdout:1/417: fsync d0/d10/f30 0 2024-04-06T14:17:01.654 INFO:tasks.workunit.client.0.smithi032.stdout:2/367: mknod d0/d4e/d63/c84 0 2024-04-06T14:17:01.655 INFO:tasks.workunit.client.0.smithi032.stdout:6/364: getdents d4/d5/d8/d2f 0 2024-04-06T14:17:01.656 INFO:tasks.workunit.client.0.smithi032.stdout:9/400: read - d2/d15/d1e/d47/d4a/d73/d7c/f80 zero size 2024-04-06T14:17:01.657 INFO:tasks.workunit.client.1.smithi086.stdout:7/78: rmdir d7/da 39 2024-04-06T14:17:01.657 INFO:tasks.workunit.client.1.smithi086.stdout:8/252: dread d0/d13/f40 [0,4194304] 0 2024-04-06T14:17:01.658 INFO:tasks.workunit.client.0.smithi032.stdout:9/401: chown d2/d15/d1e/d47/d77 5722298 1 2024-04-06T14:17:01.660 INFO:tasks.workunit.client.1.smithi086.stdout:7/79: creat d7/d10/d11/f16 x:0 0 0 2024-04-06T14:17:01.661 INFO:tasks.workunit.client.1.smithi086.stdout:2/84: sync 2024-04-06T14:17:01.661 INFO:tasks.workunit.client.1.smithi086.stdout:0/170: sync 2024-04-06T14:17:01.661 INFO:tasks.workunit.client.1.smithi086.stdout:9/165: sync 2024-04-06T14:17:01.661 INFO:tasks.workunit.client.1.smithi086.stdout:6/151: sync 2024-04-06T14:17:01.661 INFO:tasks.workunit.client.1.smithi086.stdout:4/97: sync 2024-04-06T14:17:01.662 INFO:tasks.workunit.client.1.smithi086.stdout:3/178: sync 2024-04-06T14:17:01.662 INFO:tasks.workunit.client.0.smithi032.stdout:9/402: rmdir d2/d15/d55/d59 39 2024-04-06T14:17:01.663 INFO:tasks.workunit.client.1.smithi086.stdout:9/166: readlink d5/d10/l23 0 2024-04-06T14:17:01.665 INFO:tasks.workunit.client.0.smithi032.stdout:2/368: dwrite d0/d5a/f25 [0,4194304] 0 2024-04-06T14:17:01.665 INFO:tasks.workunit.client.0.smithi032.stdout:9/403: chown d2/d25/d31/d39/f52 1871832 1 2024-04-06T14:17:01.665 INFO:tasks.workunit.client.1.smithi086.stdout:7/80: symlink d7/d10/d11/l17 0 2024-04-06T14:17:01.665 INFO:tasks.workunit.client.1.smithi086.stdout:1/95: rmdir db/d13 39 2024-04-06T14:17:01.665 INFO:tasks.workunit.client.1.smithi086.stdout:9/167: readlink d5/d10/d14/d19/d1b/l1f 0 2024-04-06T14:17:01.666 INFO:tasks.workunit.client.0.smithi032.stdout:2/369: chown d0/c75 75755855 1 2024-04-06T14:17:01.666 INFO:tasks.workunit.client.1.smithi086.stdout:8/253: creat d0/d8/d16/d52/f55 x:0 0 0 2024-04-06T14:17:01.667 INFO:tasks.workunit.client.1.smithi086.stdout:8/254: fsync d0/d3b/f46 0 2024-04-06T14:17:01.669 INFO:tasks.workunit.client.1.smithi086.stdout:0/171: unlink d0/d1/f38 0 2024-04-06T14:17:01.671 INFO:tasks.workunit.client.1.smithi086.stdout:6/152: mknod d6/d13/d1a/d22/c39 0 2024-04-06T14:17:01.672 INFO:tasks.workunit.client.1.smithi086.stdout:3/179: fdatasync d5/d26/d2a/f31 0 2024-04-06T14:17:01.673 INFO:tasks.workunit.client.0.smithi032.stdout:2/370: symlink d0/d30/d39/d59/d67/d74/d78/l85 0 2024-04-06T14:17:01.675 INFO:tasks.workunit.client.1.smithi086.stdout:3/180: read f4 [3141956,5675] 0 2024-04-06T14:17:01.675 INFO:tasks.workunit.client.1.smithi086.stdout:6/153: write d6/d13/d1a/d22/f35 [5118276,92013] 0 2024-04-06T14:17:01.676 INFO:tasks.workunit.client.0.smithi032.stdout:9/404: rename d2/d15/d1e/d47 to d2/d15/d55/d5f/d71/d81 0 2024-04-06T14:17:01.676 INFO:tasks.workunit.client.1.smithi086.stdout:4/98: rename d0/f13 to d0/d15/d20/f22 0 2024-04-06T14:17:01.677 INFO:tasks.workunit.client.1.smithi086.stdout:7/81: symlink d7/d10/l18 0 2024-04-06T14:17:01.679 INFO:tasks.workunit.client.1.smithi086.stdout:5/181: write d0/fa [349613,98756] 0 2024-04-06T14:17:01.679 INFO:tasks.workunit.client.1.smithi086.stdout:3/181: write d5/df/f1c [2296249,5695] 0 2024-04-06T14:17:01.680 INFO:tasks.workunit.client.0.smithi032.stdout:9/405: fsync d2/d25/d7e/f56 0 2024-04-06T14:17:01.681 INFO:tasks.workunit.client.0.smithi032.stdout:6/365: truncate d4/f10 6684477 0 2024-04-06T14:17:01.682 INFO:tasks.workunit.client.0.smithi032.stdout:2/371: dwrite d0/d5a/f2a [0,4194304] 0 2024-04-06T14:17:01.682 INFO:tasks.workunit.client.1.smithi086.stdout:8/255: dwrite d0/d8/d22/d43/d4d/f53 [0,4194304] 0 2024-04-06T14:17:01.684 INFO:tasks.workunit.client.1.smithi086.stdout:1/96: mknod db/d11/d1f/c21 0 2024-04-06T14:17:01.685 INFO:tasks.workunit.client.1.smithi086.stdout:8/256: chown d0/db/d32/f3e 1779317 1 2024-04-06T14:17:01.687 INFO:tasks.workunit.client.1.smithi086.stdout:0/172: rename d0/d1/d4/d7/c20 to d0/d15/c41 0 2024-04-06T14:17:01.689 INFO:tasks.workunit.client.0.smithi032.stdout:2/372: link d0/fb d0/d6e/f86 0 2024-04-06T14:17:01.690 INFO:tasks.workunit.client.0.smithi032.stdout:2/373: chown d0/d5a/d3a 4002 1 2024-04-06T14:17:01.692 INFO:tasks.workunit.client.0.smithi032.stdout:6/366: dread d4/d5/d19/f2e [0,4194304] 0 2024-04-06T14:17:01.692 INFO:tasks.workunit.client.0.smithi032.stdout:9/406: link d2/d25/d31/d39/c42 d2/d25/d7e/c82 0 2024-04-06T14:17:01.692 INFO:tasks.workunit.client.1.smithi086.stdout:9/168: creat d5/d10/d14/d19/d25/f3e x:0 0 0 2024-04-06T14:17:01.695 INFO:tasks.workunit.client.0.smithi032.stdout:6/367: symlink d4/d5/d18/l71 0 2024-04-06T14:17:01.695 INFO:tasks.workunit.client.0.smithi032.stdout:9/407: fdatasync d2/d15/d55/d5f/d71/d81/d4a/d73/d7c/f80 0 2024-04-06T14:17:01.696 INFO:tasks.workunit.client.0.smithi032.stdout:6/368: readlink l3 0 2024-04-06T14:17:01.697 INFO:tasks.workunit.client.0.smithi032.stdout:2/374: creat d0/d30/d39/d59/d67/d74/d78/d57/d7c/f87 x:0 0 0 2024-04-06T14:17:01.698 INFO:tasks.workunit.client.0.smithi032.stdout:6/369: chown d4/d5/d34/l47 1346 1 2024-04-06T14:17:01.700 INFO:tasks.workunit.client.1.smithi086.stdout:3/182: rmdir d5/d26/d2a 39 2024-04-06T14:17:01.700 INFO:tasks.workunit.client.0.smithi032.stdout:1/418: dread d0/d9/f23 [0,4194304] 0 2024-04-06T14:17:01.702 INFO:tasks.workunit.client.0.smithi032.stdout:6/370: creat d4/d5/d8/f72 x:0 0 0 2024-04-06T14:17:01.703 INFO:tasks.workunit.client.1.smithi086.stdout:1/97: mknod db/d11/d1a/c22 0 2024-04-06T14:17:01.704 INFO:tasks.workunit.client.1.smithi086.stdout:8/257: chown d0/d8/d22/d43/c3f 1847 1 2024-04-06T14:17:01.704 INFO:tasks.workunit.client.1.smithi086.stdout:1/98: chown db/d11/d1d 46059 1 2024-04-06T14:17:01.705 INFO:tasks.workunit.client.0.smithi032.stdout:2/375: rename d0/f51 to d0/d30/d39/d59/d67/d80/f88 0 2024-04-06T14:17:01.705 INFO:tasks.workunit.client.1.smithi086.stdout:1/99: stat f7 0 2024-04-06T14:17:01.705 INFO:tasks.workunit.client.0.smithi032.stdout:2/376: dread - d0/d5a/f4c zero size 2024-04-06T14:17:01.705 INFO:tasks.workunit.client.1.smithi086.stdout:4/99: fsync d0/de/f1a 0 2024-04-06T14:17:01.706 INFO:tasks.workunit.client.0.smithi032.stdout:1/419: symlink d0/d41/d5b/l8c 0 2024-04-06T14:17:01.706 INFO:tasks.workunit.client.0.smithi032.stdout:2/377: dread - d0/d30/d39/d59/f82 zero size 2024-04-06T14:17:01.706 INFO:tasks.workunit.client.1.smithi086.stdout:2/85: getdents d0/d20 0 2024-04-06T14:17:01.707 INFO:tasks.workunit.client.1.smithi086.stdout:1/100: chown db/d11/d1f/c21 176909367 1 2024-04-06T14:17:01.707 INFO:tasks.workunit.client.0.smithi032.stdout:2/378: dread - d0/d5a/f31 zero size 2024-04-06T14:17:01.707 INFO:tasks.workunit.client.1.smithi086.stdout:9/169: creat d5/d18/f3f x:0 0 0 2024-04-06T14:17:01.709 INFO:tasks.workunit.client.1.smithi086.stdout:1/101: truncate db/d11/d1a/f1b 209297 0 2024-04-06T14:17:01.711 INFO:tasks.workunit.client.0.smithi032.stdout:6/371: mknod d4/d5/d18/c73 0 2024-04-06T14:17:01.713 INFO:tasks.workunit.client.0.smithi032.stdout:6/372: write d4/d5/d28/f29 [1319900,70755] 0 2024-04-06T14:17:01.713 INFO:tasks.workunit.client.1.smithi086.stdout:4/100: symlink d0/de/l23 0 2024-04-06T14:17:01.715 INFO:tasks.workunit.client.0.smithi032.stdout:6/373: fsync d4/d5/d28/f4f 0 2024-04-06T14:17:01.715 INFO:tasks.workunit.client.1.smithi086.stdout:2/86: creat d0/d20/f23 x:0 0 0 2024-04-06T14:17:01.715 INFO:tasks.workunit.client.0.smithi032.stdout:2/379: getdents d0/d4f 0 2024-04-06T14:17:01.717 INFO:tasks.workunit.client.1.smithi086.stdout:9/170: mknod d5/d10/d14/d19/d25/c40 0 2024-04-06T14:17:01.717 INFO:tasks.workunit.client.0.smithi032.stdout:9/408: truncate d2/d25/d7e/f1d 1378814 0 2024-04-06T14:17:01.717 INFO:tasks.workunit.client.0.smithi032.stdout:2/380: stat d0/d5a/f2a 0 2024-04-06T14:17:01.718 INFO:tasks.workunit.client.1.smithi086.stdout:5/182: dwrite d0/d16/d18/ff [0,4194304] 0 2024-04-06T14:17:01.720 INFO:tasks.workunit.client.1.smithi086.stdout:6/154: rename d6/d13/d1a/l37 to d6/d13/l3a 0 2024-04-06T14:17:01.720 INFO:tasks.workunit.client.0.smithi032.stdout:6/374: write d4/d5/d8/d1f/f63 [601642,103057] 0 2024-04-06T14:17:01.721 INFO:tasks.workunit.client.0.smithi032.stdout:9/409: write d2/d25/d31/f3a [252067,6881] 0 2024-04-06T14:17:01.721 INFO:tasks.workunit.client.0.smithi032.stdout:1/420: dwrite d0/d41/f5c [0,4194304] 0 2024-04-06T14:17:01.722 INFO:tasks.workunit.client.1.smithi086.stdout:7/82: truncate d7/fb 6236769 0 2024-04-06T14:17:01.723 INFO:tasks.workunit.client.1.smithi086.stdout:6/155: write d6/f2b [1020059,52563] 0 2024-04-06T14:17:01.725 INFO:tasks.workunit.client.1.smithi086.stdout:5/183: dread - d0/d16/d18/d35/f3a zero size 2024-04-06T14:17:01.725 INFO:tasks.workunit.client.1.smithi086.stdout:3/183: creat d5/d26/d2a/f32 x:0 0 0 2024-04-06T14:17:01.725 INFO:tasks.workunit.client.1.smithi086.stdout:7/83: write d7/d10/d11/f16 [631692,75524] 0 2024-04-06T14:17:01.728 INFO:tasks.workunit.client.1.smithi086.stdout:7/84: dread - d7/d10/d11/f15 zero size 2024-04-06T14:17:01.729 INFO:tasks.workunit.client.1.smithi086.stdout:8/258: dread d0/d8/f3c [0,4194304] 0 2024-04-06T14:17:01.734 INFO:tasks.workunit.client.0.smithi032.stdout:1/421: dwrite d0/f4d [0,4194304] 0 2024-04-06T14:17:01.734 INFO:tasks.workunit.client.1.smithi086.stdout:0/173: rename d0/d1/d4/de/f24 to d0/d23/f42 0 2024-04-06T14:17:01.735 INFO:tasks.workunit.client.1.smithi086.stdout:1/102: creat db/d13/f23 x:0 0 0 2024-04-06T14:17:01.735 INFO:tasks.workunit.client.1.smithi086.stdout:1/103: fsync f1 0 2024-04-06T14:17:01.735 INFO:tasks.workunit.client.1.smithi086.stdout:6/156: mknod d6/d13/d1a/c3b 0 2024-04-06T14:17:01.737 INFO:tasks.workunit.client.1.smithi086.stdout:3/184: rmdir d5/d2b 39 2024-04-06T14:17:01.738 INFO:tasks.workunit.client.1.smithi086.stdout:6/157: write d6/d9/f31 [972610,126645] 0 2024-04-06T14:17:01.741 INFO:tasks.workunit.client.1.smithi086.stdout:7/85: write d7/da/f13 [189198,86814] 0 2024-04-06T14:17:01.742 INFO:tasks.workunit.client.1.smithi086.stdout:8/259: rmdir d0/d8 39 2024-04-06T14:17:01.742 INFO:tasks.workunit.client.1.smithi086.stdout:8/260: chown d0/c2d 1994 1 2024-04-06T14:17:01.744 INFO:tasks.workunit.client.1.smithi086.stdout:4/101: mkdir d0/de/d21/d24 0 2024-04-06T14:17:01.747 INFO:tasks.workunit.client.0.smithi032.stdout:2/381: dread d0/f28 [0,4194304] 0 2024-04-06T14:17:01.749 INFO:tasks.workunit.client.1.smithi086.stdout:9/171: rename d5/d18/f36 to d5/d10/d14/d19/d1b/f41 0 2024-04-06T14:17:01.751 INFO:tasks.workunit.client.0.smithi032.stdout:1/422: dread d0/d10/f30 [0,4194304] 0 2024-04-06T14:17:01.753 INFO:tasks.workunit.client.1.smithi086.stdout:5/184: link d0/d16/d18/d21/f2e d0/d19/f3c 0 2024-04-06T14:17:01.758 INFO:tasks.workunit.client.0.smithi032.stdout:9/410: dread d2/d15/d1e/f1f [0,4194304] 0 2024-04-06T14:17:01.758 INFO:tasks.workunit.client.0.smithi032.stdout:2/382: dwrite d0/d5a/f2a [0,4194304] 0 2024-04-06T14:17:01.759 INFO:tasks.workunit.client.1.smithi086.stdout:3/185: unlink d5/df/f1a 0 2024-04-06T14:17:01.759 INFO:tasks.workunit.client.1.smithi086.stdout:3/186: fdatasync d5/df/f2c 0 2024-04-06T14:17:01.759 INFO:tasks.workunit.client.1.smithi086.stdout:7/86: creat d7/da/f19 x:0 0 0 2024-04-06T14:17:01.760 INFO:tasks.workunit.client.0.smithi032.stdout:9/411: rename d2/d25 to d2/d15/d1e/d65/d83 0 2024-04-06T14:17:01.761 INFO:tasks.workunit.client.0.smithi032.stdout:1/423: write d0/d10/d16/f6d [3222947,28643] 0 2024-04-06T14:17:01.764 INFO:tasks.workunit.client.1.smithi086.stdout:9/172: rmdir d5/d10/d14/d19/d25 39 2024-04-06T14:17:01.765 INFO:tasks.workunit.client.0.smithi032.stdout:2/383: rename d0/f28 to d0/d30/d39/d59/d67/d74/d78/f89 0 2024-04-06T14:17:01.767 INFO:tasks.workunit.client.1.smithi086.stdout:1/104: truncate f1 2509870 0 2024-04-06T14:17:01.767 INFO:tasks.workunit.client.0.smithi032.stdout:1/424: dread d0/f2c [0,4194304] 0 2024-04-06T14:17:01.768 INFO:tasks.workunit.client.1.smithi086.stdout:4/102: dread d0/de/f11 [0,4194304] 0 2024-04-06T14:17:01.768 INFO:tasks.workunit.client.0.smithi032.stdout:9/412: rename d2/d15/d1e/c54 to d2/d15/d55/d5f/d71/d81/d4a/c84 0 2024-04-06T14:17:01.770 INFO:tasks.workunit.client.1.smithi086.stdout:0/174: write d0/f16 [375919,64723] 0 2024-04-06T14:17:01.771 INFO:tasks.workunit.client.1.smithi086.stdout:5/185: mknod d0/d16/d18/d35/c3d 0 2024-04-06T14:17:01.772 INFO:tasks.workunit.client.0.smithi032.stdout:2/384: dread d0/d4e/f69 [0,4194304] 0 2024-04-06T14:17:01.773 INFO:tasks.workunit.client.0.smithi032.stdout:2/385: chown d0/d5a/f25 4476787 1 2024-04-06T14:17:01.775 INFO:tasks.workunit.client.1.smithi086.stdout:8/261: mknod d0/c56 0 2024-04-06T14:17:01.775 INFO:tasks.workunit.client.0.smithi032.stdout:2/386: fdatasync d0/d30/d39/f58 0 2024-04-06T14:17:01.776 INFO:tasks.workunit.client.1.smithi086.stdout:8/262: write d0/f2 [2894446,39714] 0 2024-04-06T14:17:01.777 INFO:tasks.workunit.client.0.smithi032.stdout:1/425: dwrite d0/f4d [0,4194304] 0 2024-04-06T14:17:01.777 INFO:tasks.workunit.client.0.smithi032.stdout:2/387: write d0/d5a/d3a/f48 [394514,98714] 0 2024-04-06T14:17:01.779 INFO:tasks.workunit.client.1.smithi086.stdout:1/105: mkdir db/d11/d24 0 2024-04-06T14:17:01.779 INFO:tasks.workunit.client.1.smithi086.stdout:1/106: readlink - no filename 2024-04-06T14:17:01.779 INFO:tasks.workunit.client.0.smithi032.stdout:9/413: getdents d2/d15/d55/d59 0 2024-04-06T14:17:01.779 INFO:tasks.workunit.client.1.smithi086.stdout:5/186: dwrite d0/d19/f26 [0,4194304] 0 2024-04-06T14:17:01.781 INFO:tasks.workunit.client.0.smithi032.stdout:1/426: read d0/f1e [4591545,78690] 0 2024-04-06T14:17:01.781 INFO:tasks.workunit.client.0.smithi032.stdout:1/427: chown d0/d9/d1f/d72/l8b 0 1 2024-04-06T14:17:01.782 INFO:tasks.workunit.client.1.smithi086.stdout:5/187: dread - d0/d19/f3c zero size 2024-04-06T14:17:01.783 INFO:tasks.workunit.client.0.smithi032.stdout:1/428: write d0/da/d3e/f6a [617589,16286] 0 2024-04-06T14:17:01.785 INFO:tasks.workunit.client.1.smithi086.stdout:4/103: rmdir d0/d15/d20 39 2024-04-06T14:17:01.790 INFO:tasks.workunit.client.0.smithi032.stdout:2/388: getdents d0/d30/d39/d59/d67 0 2024-04-06T14:17:01.790 INFO:tasks.workunit.client.1.smithi086.stdout:5/188: creat d0/d19/f3e x:0 0 0 2024-04-06T14:17:01.790 INFO:tasks.workunit.client.1.smithi086.stdout:1/107: dwrite fa [0,4194304] 0 2024-04-06T14:17:01.792 INFO:tasks.workunit.client.0.smithi032.stdout:2/389: truncate d0/d5a/d3a/f48 1457218 0 2024-04-06T14:17:01.792 INFO:tasks.workunit.client.0.smithi032.stdout:1/429: creat d0/d10/f8d x:0 0 0 2024-04-06T14:17:01.793 INFO:tasks.workunit.client.1.smithi086.stdout:1/108: read - db/d13/f1e zero size 2024-04-06T14:17:01.793 INFO:tasks.workunit.client.0.smithi032.stdout:1/430: fdatasync d0/d10/f26 0 2024-04-06T14:17:01.795 INFO:tasks.workunit.client.1.smithi086.stdout:8/263: symlink d0/d8/d16/l57 0 2024-04-06T14:17:01.795 INFO:tasks.workunit.client.0.smithi032.stdout:1/431: chown d0/da/d3e/l84 2 1 2024-04-06T14:17:01.795 INFO:tasks.workunit.client.0.smithi032.stdout:2/390: truncate d0/ff 7678908 0 2024-04-06T14:17:01.796 INFO:tasks.workunit.client.1.smithi086.stdout:0/175: rename d0/d1/d4/f12 to d0/d1/d4/f43 0 2024-04-06T14:17:01.797 INFO:tasks.workunit.client.0.smithi032.stdout:1/432: read d0/da/d3e/d75/f81 [1891958,99921] 0 2024-04-06T14:17:01.797 INFO:tasks.workunit.client.1.smithi086.stdout:5/189: creat d0/f3f x:0 0 0 2024-04-06T14:17:01.801 INFO:tasks.workunit.client.1.smithi086.stdout:1/109: rename db/d11/f19 to db/d11/d1a/f25 0 2024-04-06T14:17:01.802 INFO:tasks.workunit.client.1.smithi086.stdout:8/264: read d0/d8/d22/d43/d4d/f53 [1083800,76476] 0 2024-04-06T14:17:01.802 INFO:tasks.workunit.client.1.smithi086.stdout:0/176: write d0/d1/d4/d7/f14 [3408065,98049] 0 2024-04-06T14:17:01.804 INFO:tasks.workunit.client.0.smithi032.stdout:2/391: write d0/d30/f43 [5618785,12643] 0 2024-04-06T14:17:01.805 INFO:tasks.workunit.client.0.smithi032.stdout:9/414: dread d2/d15/d1e/d65/d83/f45 [0,4194304] 0 2024-04-06T14:17:01.806 INFO:tasks.workunit.client.1.smithi086.stdout:3/187: write d5/f9 [2573718,89133] 0 2024-04-06T14:17:01.806 INFO:tasks.workunit.client.1.smithi086.stdout:1/110: fdatasync db/ff 0 2024-04-06T14:17:01.807 INFO:tasks.workunit.client.0.smithi032.stdout:2/392: chown d0/d5a/c4b 1980689666 1 2024-04-06T14:17:01.808 INFO:tasks.workunit.client.0.smithi032.stdout:1/433: dwrite d0/d10/d16/d3f/f5d [0,4194304] 0 2024-04-06T14:17:01.809 INFO:tasks.workunit.client.1.smithi086.stdout:1/111: write fa [3484837,116456] 0 2024-04-06T14:17:01.809 INFO:tasks.workunit.client.0.smithi032.stdout:2/393: readlink d0/l9 0 2024-04-06T14:17:01.810 INFO:tasks.workunit.client.1.smithi086.stdout:8/265: mkdir d0/d13/d58 0 2024-04-06T14:17:01.811 INFO:tasks.workunit.client.0.smithi032.stdout:2/394: write d0/d5a/f25 [2846056,125321] 0 2024-04-06T14:17:01.811 INFO:tasks.workunit.client.1.smithi086.stdout:4/104: dwrite d0/de/f11 [4194304,4194304] 0 2024-04-06T14:17:01.813 INFO:tasks.workunit.client.1.smithi086.stdout:4/105: write d0/f5 [2334250,20688] 0 2024-04-06T14:17:01.813 INFO:tasks.workunit.client.0.smithi032.stdout:1/434: creat d0/d41/d59/f8e x:0 0 0 2024-04-06T14:17:01.815 INFO:tasks.workunit.client.1.smithi086.stdout:9/173: truncate d5/d10/d14/f22 2709500 0 2024-04-06T14:17:01.817 INFO:tasks.workunit.client.0.smithi032.stdout:9/415: creat d2/d15/d55/f85 x:0 0 0 2024-04-06T14:17:01.818 INFO:tasks.workunit.client.1.smithi086.stdout:7/87: dread d7/fb [0,4194304] 0 2024-04-06T14:17:01.820 INFO:tasks.workunit.client.1.smithi086.stdout:0/177: unlink d0/d15/d26/d29/f2d 0 2024-04-06T14:17:01.821 INFO:tasks.workunit.client.1.smithi086.stdout:8/266: rmdir d0/d8 39 2024-04-06T14:17:01.823 INFO:tasks.workunit.client.0.smithi032.stdout:2/395: creat d0/f8a x:0 0 0 2024-04-06T14:17:01.826 INFO:tasks.workunit.client.1.smithi086.stdout:1/112: dwrite db/d11/d1a/f1b [0,4194304] 0 2024-04-06T14:17:01.827 INFO:tasks.workunit.client.0.smithi032.stdout:1/435: dwrite d0/da/f69 [0,4194304] 0 2024-04-06T14:17:01.827 INFO:tasks.workunit.client.0.smithi032.stdout:2/396: mknod d0/d30/d7b/c8b 0 2024-04-06T14:17:01.828 INFO:tasks.workunit.client.1.smithi086.stdout:1/113: chown db/d11/c20 27 1 2024-04-06T14:17:01.830 INFO:tasks.workunit.client.0.smithi032.stdout:9/416: creat d2/d15/d55/f86 x:0 0 0 2024-04-06T14:17:01.831 INFO:tasks.workunit.client.1.smithi086.stdout:7/88: mknod d7/d10/d11/c1a 0 2024-04-06T14:17:01.832 INFO:tasks.workunit.client.0.smithi032.stdout:2/397: mkdir d0/d30/d39/d59/d67/d74/d78/d8c 0 2024-04-06T14:17:01.833 INFO:tasks.workunit.client.0.smithi032.stdout:2/398: chown d0/d30/d39/d59/c5d 9172988 1 2024-04-06T14:17:01.834 INFO:tasks.workunit.client.0.smithi032.stdout:2/399: fdatasync d0/d5a/f1f 0 2024-04-06T14:17:01.835 INFO:tasks.workunit.client.1.smithi086.stdout:9/174: unlink d5/d10/d14/d19/d1b/f3c 0 2024-04-06T14:17:01.835 INFO:tasks.workunit.client.0.smithi032.stdout:9/417: creat d2/d15/d55/d5f/d71/d81/d4a/d73/f87 x:0 0 0 2024-04-06T14:17:01.836 INFO:tasks.workunit.client.1.smithi086.stdout:0/178: dwrite d0/d1/d1b/f39 [0,4194304] 0 2024-04-06T14:17:01.838 INFO:tasks.workunit.client.0.smithi032.stdout:1/436: dread d0/d9/d1f/f47 [0,4194304] 0 2024-04-06T14:17:01.839 INFO:tasks.workunit.client.1.smithi086.stdout:0/179: stat d0/f40 0 2024-04-06T14:17:01.842 INFO:tasks.workunit.client.1.smithi086.stdout:3/188: rename d5/d26/d2a to d5/d33 0 2024-04-06T14:17:01.843 INFO:tasks.workunit.client.1.smithi086.stdout:5/190: truncate d0/d19/f26 4012753 0 2024-04-06T14:17:01.843 INFO:tasks.workunit.client.0.smithi032.stdout:1/437: write d0/f2c [7995958,74108] 0 2024-04-06T14:17:01.845 INFO:tasks.workunit.client.0.smithi032.stdout:1/438: stat d0/d10 0 2024-04-06T14:17:01.845 INFO:tasks.workunit.client.0.smithi032.stdout:9/418: dwrite d2/d15/d1e/d65/d83/d7e/f1b [0,4194304] 0 2024-04-06T14:17:01.847 INFO:tasks.workunit.client.0.smithi032.stdout:9/419: write d2/d15/d1e/d65/d83/d7e/f14 [601111,53278] 0 2024-04-06T14:17:01.849 INFO:tasks.workunit.client.1.smithi086.stdout:7/89: dread d7/fb [0,4194304] 0 2024-04-06T14:17:01.849 INFO:tasks.workunit.client.1.smithi086.stdout:1/114: link db/d11/d1a/c1c db/d11/c26 0 2024-04-06T14:17:01.851 INFO:tasks.workunit.client.0.smithi032.stdout:2/400: rename d0/d30/d7b to d0/d6e/d8d 0 2024-04-06T14:17:01.852 INFO:tasks.workunit.client.1.smithi086.stdout:8/267: rename d0/d8/d16/l57 to d0/d8/d16/d52/l59 0 2024-04-06T14:17:01.853 INFO:tasks.workunit.client.1.smithi086.stdout:7/90: mkdir d7/d10/d1b 0 2024-04-06T14:17:01.855 INFO:tasks.workunit.client.1.smithi086.stdout:3/189: unlink d5/df/l17 0 2024-04-06T14:17:01.856 INFO:tasks.workunit.client.1.smithi086.stdout:1/115: creat db/d11/f27 x:0 0 0 2024-04-06T14:17:01.856 INFO:tasks.workunit.client.1.smithi086.stdout:0/180: dwrite d0/d1/d4/d7/f11 [0,4194304] 0 2024-04-06T14:17:01.858 INFO:tasks.workunit.client.0.smithi032.stdout:2/401: dwrite d0/d5a/f25 [0,4194304] 0 2024-04-06T14:17:01.861 INFO:tasks.workunit.client.1.smithi086.stdout:1/116: dread - db/d13/f23 zero size 2024-04-06T14:17:01.864 INFO:tasks.workunit.client.1.smithi086.stdout:4/106: dwrite d0/fd [0,4194304] 0 2024-04-06T14:17:01.864 INFO:tasks.workunit.client.1.smithi086.stdout:7/91: mkdir d7/d10/d11/d1c 0 2024-04-06T14:17:01.864 INFO:tasks.workunit.client.1.smithi086.stdout:8/268: write d0/db/d32/f3e [4950777,54448] 0 2024-04-06T14:17:01.867 INFO:tasks.workunit.client.1.smithi086.stdout:0/181: mkdir d0/d1/d1b/d44 0 2024-04-06T14:17:01.871 INFO:tasks.workunit.client.1.smithi086.stdout:8/269: fsync d0/d8/d22/d43/f28 0 2024-04-06T14:17:01.871 INFO:tasks.workunit.client.1.smithi086.stdout:1/117: stat db/d11/d1f/c21 0 2024-04-06T14:17:01.871 INFO:tasks.workunit.client.1.smithi086.stdout:4/107: mknod d0/d15/d19/c25 0 2024-04-06T14:17:01.872 INFO:tasks.workunit.client.1.smithi086.stdout:1/118: chown db/fe 5150 1 2024-04-06T14:17:01.874 INFO:tasks.workunit.client.0.smithi032.stdout:9/420: read d2/d15/d1e/d65/d83/d7e/f14 [2469179,119459] 0 2024-04-06T14:17:01.875 INFO:tasks.workunit.client.1.smithi086.stdout:0/182: rename d0/f2f to d0/d23/f45 0 2024-04-06T14:17:01.875 INFO:tasks.workunit.client.1.smithi086.stdout:3/190: dread d5/df/f1c [0,4194304] 0 2024-04-06T14:17:01.876 INFO:tasks.workunit.client.0.smithi032.stdout:9/421: truncate d2/d15/d55/f85 412996 0 2024-04-06T14:17:01.876 INFO:tasks.workunit.client.1.smithi086.stdout:9/175: fsync d5/d10/d14/f22 0 2024-04-06T14:17:01.878 INFO:tasks.workunit.client.1.smithi086.stdout:4/108: creat d0/de/f26 x:0 0 0 2024-04-06T14:17:01.879 INFO:tasks.workunit.client.1.smithi086.stdout:7/92: dread f5 [0,4194304] 0 2024-04-06T14:17:01.879 INFO:tasks.workunit.client.0.smithi032.stdout:9/422: unlink d2/d15/d55/d5f/d71/d81/f5d 0 2024-04-06T14:17:01.881 INFO:tasks.workunit.client.1.smithi086.stdout:8/270: mkdir d0/d5a 0 2024-04-06T14:17:01.883 INFO:tasks.workunit.client.1.smithi086.stdout:3/191: read d5/df/f25 [306152,68734] 0 2024-04-06T14:17:01.884 INFO:tasks.workunit.client.1.smithi086.stdout:9/176: mknod d5/d10/d14/d19/c42 0 2024-04-06T14:17:01.884 INFO:tasks.workunit.client.1.smithi086.stdout:0/183: getdents d0/d15/d26/d33 0 2024-04-06T14:17:01.885 INFO:tasks.workunit.client.0.smithi032.stdout:1/439: write d0/d41/d59/f64 [632254,109961] 0 2024-04-06T14:17:01.886 INFO:tasks.workunit.client.1.smithi086.stdout:3/192: chown d5/df/l1e 0 1 2024-04-06T14:17:01.886 INFO:tasks.workunit.client.1.smithi086.stdout:1/119: link db/d11/c20 db/d11/d24/c28 0 2024-04-06T14:17:01.886 INFO:tasks.workunit.client.1.smithi086.stdout:9/177: dread - d5/d18/f3f zero size 2024-04-06T14:17:01.887 INFO:tasks.workunit.client.1.smithi086.stdout:4/109: dwrite d0/f7 [0,4194304] 0 2024-04-06T14:17:01.889 INFO:tasks.workunit.client.0.smithi032.stdout:1/440: getdents d0/da/d3e/d75 0 2024-04-06T14:17:01.890 INFO:tasks.workunit.client.0.smithi032.stdout:1/441: stat d0/da/d3e/d75/l88 0 2024-04-06T14:17:01.891 INFO:tasks.workunit.client.1.smithi086.stdout:4/110: write d0/f2 [2148714,26415] 0 2024-04-06T14:17:01.891 INFO:tasks.workunit.client.1.smithi086.stdout:1/120: rename db/d11/d1a/f1b to db/d13/f29 0 2024-04-06T14:17:01.893 INFO:tasks.workunit.client.1.smithi086.stdout:2/87: sync 2024-04-06T14:17:01.893 INFO:tasks.workunit.client.1.smithi086.stdout:3/193: creat d5/f34 x:0 0 0 2024-04-06T14:17:01.894 INFO:tasks.workunit.client.1.smithi086.stdout:2/88: fsync d0/d20/f23 0 2024-04-06T14:17:01.894 INFO:tasks.workunit.client.0.smithi032.stdout:1/442: mkdir d0/d9/d1f/d7c/d8f 0 2024-04-06T14:17:01.895 INFO:tasks.workunit.client.1.smithi086.stdout:2/89: write d0/df/f13 [7719495,15023] 0 2024-04-06T14:17:01.897 INFO:tasks.workunit.client.1.smithi086.stdout:1/121: unlink db/d11/c15 0 2024-04-06T14:17:01.898 INFO:tasks.workunit.client.1.smithi086.stdout:5/191: truncate d0/d19/f26 4286043 0 2024-04-06T14:17:01.901 INFO:tasks.workunit.client.1.smithi086.stdout:9/178: rename d5/fc to d5/d10/d14/d19/d1b/f43 0 2024-04-06T14:17:01.904 INFO:tasks.workunit.client.1.smithi086.stdout:1/122: dwrite db/f10 [4194304,4194304] 0 2024-04-06T14:17:01.904 INFO:tasks.workunit.client.1.smithi086.stdout:1/123: readlink - no filename 2024-04-06T14:17:01.904 INFO:tasks.workunit.client.1.smithi086.stdout:9/179: chown f2 555 1 2024-04-06T14:17:01.904 INFO:tasks.workunit.client.1.smithi086.stdout:2/90: symlink d0/l24 0 2024-04-06T14:17:01.905 INFO:tasks.workunit.client.1.smithi086.stdout:3/194: creat d5/d33/f35 x:0 0 0 2024-04-06T14:17:01.906 INFO:tasks.workunit.client.1.smithi086.stdout:9/180: creat d5/d13/f44 x:0 0 0 2024-04-06T14:17:01.913 INFO:tasks.workunit.client.0.smithi032.stdout:1/443: unlink d0/d9/d1f/f47 0 2024-04-06T14:17:01.915 INFO:tasks.workunit.client.1.smithi086.stdout:1/124: dwrite f4 [0,4194304] 0 2024-04-06T14:17:01.915 INFO:tasks.workunit.client.1.smithi086.stdout:1/125: chown f7 9681 1 2024-04-06T14:17:01.915 INFO:tasks.workunit.client.1.smithi086.stdout:9/181: creat d5/d10/d14/d19/d1b/f45 x:0 0 0 2024-04-06T14:17:01.918 INFO:tasks.workunit.client.0.smithi032.stdout:1/444: rename d0/da/f82 to d0/f90 0 2024-04-06T14:17:01.919 INFO:tasks.workunit.client.1.smithi086.stdout:2/91: dwrite d0/df/f21 [0,4194304] 0 2024-04-06T14:17:01.927 INFO:tasks.workunit.client.1.smithi086.stdout:1/126: dwrite fa [4194304,4194304] 0 2024-04-06T14:17:01.927 INFO:tasks.workunit.client.1.smithi086.stdout:1/127: readlink - no filename 2024-04-06T14:17:01.930 INFO:tasks.workunit.client.0.smithi032.stdout:1/445: link d0/da/d3e/d75/f81 d0/d9/d1f/f91 0 2024-04-06T14:17:01.933 INFO:tasks.workunit.client.0.smithi032.stdout:1/446: rename d0/d10/d16/l2b to d0/da/d3e/d75/l92 0 2024-04-06T14:17:01.936 INFO:tasks.workunit.client.1.smithi086.stdout:1/128: mkdir db/d11/d1a/d2a 0 2024-04-06T14:17:01.938 INFO:tasks.workunit.client.1.smithi086.stdout:2/92: dwrite d0/f6 [0,4194304] 0 2024-04-06T14:17:01.940 INFO:tasks.workunit.client.1.smithi086.stdout:1/129: rmdir db/d11/d1f 39 2024-04-06T14:17:01.943 INFO:tasks.workunit.client.1.smithi086.stdout:8/271: dwrite d0/d8/d22/d43/d4d/f53 [0,4194304] 0 2024-04-06T14:17:01.943 INFO:tasks.workunit.client.1.smithi086.stdout:9/182: write d5/d10/d14/d19/f1e [71826,62449] 0 2024-04-06T14:17:01.943 INFO:tasks.workunit.client.1.smithi086.stdout:2/93: creat d0/d20/f25 x:0 0 0 2024-04-06T14:17:01.950 INFO:tasks.workunit.client.1.smithi086.stdout:5/192: dwrite d0/d16/d18/d21/f2e [0,4194304] 0 2024-04-06T14:17:01.951 INFO:tasks.workunit.client.0.smithi032.stdout:1/447: dread d0/f17 [0,4194304] 0 2024-04-06T14:17:01.954 INFO:tasks.workunit.client.1.smithi086.stdout:1/130: dwrite db/d13/f29 [0,4194304] 0 2024-04-06T14:17:01.955 INFO:tasks.workunit.client.1.smithi086.stdout:8/272: rename d0/db/f19 to d0/db/f5b 0 2024-04-06T14:17:01.956 INFO:tasks.workunit.client.1.smithi086.stdout:2/94: creat d0/f26 x:0 0 0 2024-04-06T14:17:01.957 INFO:tasks.workunit.client.1.smithi086.stdout:5/193: write d0/f4 [101059,109775] 0 2024-04-06T14:17:01.963 INFO:tasks.workunit.client.1.smithi086.stdout:2/95: truncate d0/df/f18 1009583 0 2024-04-06T14:17:01.963 INFO:tasks.workunit.client.1.smithi086.stdout:2/96: read d0/f6 [2030464,129543] 0 2024-04-06T14:17:01.963 INFO:tasks.workunit.client.1.smithi086.stdout:2/97: read d0/df/f18 [513338,99821] 0 2024-04-06T14:17:01.964 INFO:tasks.workunit.client.1.smithi086.stdout:4/111: dread d0/f3 [0,4194304] 0 2024-04-06T14:17:01.966 INFO:tasks.workunit.client.1.smithi086.stdout:2/98: mkdir d0/d20/d27 0 2024-04-06T14:17:01.966 INFO:tasks.workunit.client.1.smithi086.stdout:2/99: chown d0/df/l11 0 1 2024-04-06T14:17:01.967 INFO:tasks.workunit.client.1.smithi086.stdout:2/100: dread - d0/df/f1b zero size 2024-04-06T14:17:01.968 INFO:tasks.workunit.client.0.smithi032.stdout:1/448: dread d0/d10/d16/f6d [0,4194304] 0 2024-04-06T14:17:01.971 INFO:tasks.workunit.client.0.smithi032.stdout:1/449: creat d0/d9/d1f/f93 x:0 0 0 2024-04-06T14:17:01.971 INFO:tasks.workunit.client.1.smithi086.stdout:4/112: fdatasync d0/f2 0 2024-04-06T14:17:01.972 INFO:tasks.workunit.client.0.smithi032.stdout:1/450: fsync d0/d41/d59/f60 0 2024-04-06T14:17:01.973 INFO:tasks.workunit.client.0.smithi032.stdout:1/451: chown d0/d9/d1f/d7c/d8f 18396810 1 2024-04-06T14:17:01.973 INFO:tasks.workunit.client.1.smithi086.stdout:2/101: rmdir d0/d20 39 2024-04-06T14:17:01.975 INFO:tasks.workunit.client.0.smithi032.stdout:1/452: mkdir d0/d9/d94 0 2024-04-06T14:17:01.977 INFO:tasks.workunit.client.0.smithi032.stdout:1/453: readlink d0/da/d3e/l84 0 2024-04-06T14:17:01.977 INFO:tasks.workunit.client.0.smithi032.stdout:1/454: readlink d0/da/l31 0 2024-04-06T14:17:01.977 INFO:tasks.workunit.client.1.smithi086.stdout:8/273: truncate d0/d8/d16/d39/f20 2525192 0 2024-04-06T14:17:01.978 INFO:tasks.workunit.client.1.smithi086.stdout:5/194: dread d0/fa [0,4194304] 0 2024-04-06T14:17:01.978 INFO:tasks.workunit.client.1.smithi086.stdout:4/113: creat d0/d15/d20/f27 x:0 0 0 2024-04-06T14:17:01.979 INFO:tasks.workunit.client.0.smithi032.stdout:1/455: creat d0/d9/d1f/d7c/f95 x:0 0 0 2024-04-06T14:17:01.980 INFO:tasks.workunit.client.1.smithi086.stdout:8/274: readlink d0/lf 0 2024-04-06T14:17:01.980 INFO:tasks.workunit.client.1.smithi086.stdout:1/131: dwrite f1 [0,4194304] 0 2024-04-06T14:17:01.980 INFO:tasks.workunit.client.1.smithi086.stdout:5/195: chown d0/d19/f3e 819 1 2024-04-06T14:17:01.981 INFO:tasks.workunit.client.1.smithi086.stdout:2/102: rename d0/df/l17 to d0/df/l28 0 2024-04-06T14:17:01.981 INFO:tasks.workunit.client.0.smithi032.stdout:1/456: mkdir d0/d41/d5b/d96 0 2024-04-06T14:17:01.984 INFO:tasks.workunit.client.1.smithi086.stdout:2/103: mknod d0/d20/c29 0 2024-04-06T14:17:01.984 INFO:tasks.workunit.client.1.smithi086.stdout:5/196: stat d0/d16/d18/d21/d23/f1f 0 2024-04-06T14:17:01.986 INFO:tasks.workunit.client.0.smithi032.stdout:1/457: link d0/d10/l39 d0/d41/d5b/l97 0 2024-04-06T14:17:01.987 INFO:tasks.workunit.client.1.smithi086.stdout:4/114: dwrite d0/f10 [0,4194304] 0 2024-04-06T14:17:01.987 INFO:tasks.workunit.client.0.smithi032.stdout:5/361: sync 2024-04-06T14:17:01.987 INFO:tasks.workunit.client.0.smithi032.stdout:8/339: sync 2024-04-06T14:17:01.990 INFO:tasks.workunit.client.1.smithi086.stdout:1/132: link db/d11/f17 db/d11/d1f/f2b 0 2024-04-06T14:17:01.990 INFO:tasks.workunit.client.0.smithi032.stdout:5/362: chown d16/d18/c49 121 1 2024-04-06T14:17:01.990 INFO:tasks.workunit.client.0.smithi032.stdout:5/363: chown f11 1628 1 2024-04-06T14:17:01.991 INFO:tasks.workunit.client.1.smithi086.stdout:5/197: unlink d0/d16/d18/d21/f2e 0 2024-04-06T14:17:01.992 INFO:tasks.workunit.client.0.smithi032.stdout:5/364: truncate d16/d58/f6d 377013 0 2024-04-06T14:17:01.992 INFO:tasks.workunit.client.0.smithi032.stdout:8/340: stat d3/d8/d12/d2f/d4a/d60/l68 0 2024-04-06T14:17:01.993 INFO:tasks.workunit.client.1.smithi086.stdout:5/198: dread - d0/d19/f3e zero size 2024-04-06T14:17:01.994 INFO:tasks.workunit.client.1.smithi086.stdout:4/115: mknod d0/de/d21/d24/c28 0 2024-04-06T14:17:01.995 INFO:tasks.workunit.client.1.smithi086.stdout:2/104: creat d0/f2a x:0 0 0 2024-04-06T14:17:01.995 INFO:tasks.workunit.client.1.smithi086.stdout:5/199: truncate d0/f6 1664423 0 2024-04-06T14:17:01.995 INFO:tasks.workunit.client.0.smithi032.stdout:5/365: rename d16/d18/c1f to d16/d58/c7e 0 2024-04-06T14:17:01.997 INFO:tasks.workunit.client.1.smithi086.stdout:1/133: dread f1 [0,4194304] 0 2024-04-06T14:17:01.997 INFO:tasks.workunit.client.1.smithi086.stdout:4/116: mknod d0/de/c29 0 2024-04-06T14:17:02.000 INFO:tasks.workunit.client.0.smithi032.stdout:5/366: rename d16/d18/l1b to d16/d58/d54/l7f 0 2024-04-06T14:17:02.001 INFO:tasks.workunit.client.1.smithi086.stdout:1/134: truncate db/d11/d1a/f25 117228 0 2024-04-06T14:17:02.001 INFO:tasks.workunit.client.1.smithi086.stdout:4/117: mknod d0/de/d21/d24/c2a 0 2024-04-06T14:17:02.004 INFO:tasks.workunit.client.0.smithi032.stdout:5/367: rename d16/d18/d33/d45/l4c to d16/d58/l80 0 2024-04-06T14:17:02.005 INFO:tasks.workunit.client.1.smithi086.stdout:1/135: write db/d13/f1e [874008,33264] 0 2024-04-06T14:17:02.006 INFO:tasks.workunit.client.1.smithi086.stdout:2/105: dwrite d0/df/f14 [0,4194304] 0 2024-04-06T14:17:02.007 INFO:tasks.workunit.client.1.smithi086.stdout:4/118: truncate d0/d15/d19/f1c 125594 0 2024-04-06T14:17:02.009 INFO:tasks.workunit.client.0.smithi032.stdout:1/458: rmdir d0/d10/d16 39 2024-04-06T14:17:02.015 INFO:tasks.workunit.client.1.smithi086.stdout:2/106: dwrite d0/f1e [0,4194304] 0 2024-04-06T14:17:02.017 INFO:tasks.workunit.client.1.smithi086.stdout:1/136: symlink db/d11/d24/l2c 0 2024-04-06T14:17:02.020 INFO:tasks.workunit.client.0.smithi032.stdout:5/368: getdents d16/d58 0 2024-04-06T14:17:02.023 INFO:tasks.workunit.client.1.smithi086.stdout:2/107: chown d0/df/l10 1376204 1 2024-04-06T14:17:02.024 INFO:tasks.workunit.client.0.smithi032.stdout:5/369: write f11 [1445716,104263] 0 2024-04-06T14:17:02.025 INFO:tasks.workunit.client.0.smithi032.stdout:1/459: dwrite d0/f2c [0,4194304] 0 2024-04-06T14:17:02.026 INFO:tasks.workunit.client.0.smithi032.stdout:8/341: dwrite d3/d8/f13 [0,4194304] 0 2024-04-06T14:17:02.026 INFO:tasks.workunit.client.0.smithi032.stdout:5/370: chown c14 78218 1 2024-04-06T14:17:02.028 INFO:tasks.workunit.client.1.smithi086.stdout:1/137: rename db/d11/d24/l2c to db/d11/d1f/l2d 0 2024-04-06T14:17:02.031 INFO:tasks.workunit.client.1.smithi086.stdout:4/119: getdents d0/d15 0 2024-04-06T14:17:02.033 INFO:tasks.workunit.client.1.smithi086.stdout:2/108: dwrite d0/df/f1a [0,4194304] 0 2024-04-06T14:17:02.035 INFO:tasks.workunit.client.0.smithi032.stdout:8/342: getdents d3/d8/d12/d2f 0 2024-04-06T14:17:02.037 INFO:tasks.workunit.client.0.smithi032.stdout:1/460: dwrite d0/d9/f24 [0,4194304] 0 2024-04-06T14:17:02.039 INFO:tasks.workunit.client.1.smithi086.stdout:1/138: creat db/d13/f2e x:0 0 0 2024-04-06T14:17:02.039 INFO:tasks.workunit.client.1.smithi086.stdout:2/109: mkdir d0/d20/d27/d2b 0 2024-04-06T14:17:02.041 INFO:tasks.workunit.client.1.smithi086.stdout:2/110: readlink d0/df/l10 0 2024-04-06T14:17:02.042 INFO:tasks.workunit.client.0.smithi032.stdout:1/461: mkdir d0/d10/d16/d3f/d98 0 2024-04-06T14:17:02.043 INFO:tasks.workunit.client.1.smithi086.stdout:1/139: chown db/d11/c26 93 1 2024-04-06T14:17:02.043 INFO:tasks.workunit.client.1.smithi086.stdout:4/120: dwrite d0/f10 [0,4194304] 0 2024-04-06T14:17:02.045 INFO:tasks.workunit.client.0.smithi032.stdout:1/462: mknod d0/d9/d1f/d72/c99 0 2024-04-06T14:17:02.045 INFO:tasks.workunit.client.1.smithi086.stdout:1/140: fdatasync f1 0 2024-04-06T14:17:02.047 INFO:tasks.workunit.client.0.smithi032.stdout:1/463: rmdir d0/da 39 2024-04-06T14:17:02.048 INFO:tasks.workunit.client.1.smithi086.stdout:2/111: creat d0/f2c x:0 0 0 2024-04-06T14:17:02.048 INFO:tasks.workunit.client.1.smithi086.stdout:2/112: dread - d0/df/f1b zero size 2024-04-06T14:17:02.049 INFO:tasks.workunit.client.1.smithi086.stdout:2/113: read - d0/d20/f23 zero size 2024-04-06T14:17:02.052 INFO:tasks.workunit.client.0.smithi032.stdout:1/464: rename d0/f4c to d0/d41/d59/f9a 0 2024-04-06T14:17:02.052 INFO:tasks.workunit.client.1.smithi086.stdout:2/114: creat d0/f2d x:0 0 0 2024-04-06T14:17:02.052 INFO:tasks.workunit.client.1.smithi086.stdout:2/115: fdatasync d0/df/f1d 0 2024-04-06T14:17:02.053 INFO:tasks.workunit.client.0.smithi032.stdout:1/465: dread - d0/d41/d59/f8e zero size 2024-04-06T14:17:02.055 INFO:tasks.workunit.client.0.smithi032.stdout:1/466: write d0/d10/d16/f51 [374224,88123] 0 2024-04-06T14:17:02.055 INFO:tasks.workunit.client.1.smithi086.stdout:2/116: getdents d0/d20 0 2024-04-06T14:17:02.056 INFO:tasks.workunit.client.1.smithi086.stdout:2/117: chown d0/df/l28 126 1 2024-04-06T14:17:02.083 INFO:tasks.workunit.client.1.smithi086.stdout:6/158: sync 2024-04-06T14:17:02.085 INFO:tasks.workunit.client.0.smithi032.stdout:1/467: dread d0/d41/d59/f64 [0,4194304] 0 2024-04-06T14:17:02.086 INFO:tasks.workunit.client.1.smithi086.stdout:6/159: read d6/f7 [1419667,119257] 0 2024-04-06T14:17:02.087 INFO:tasks.workunit.client.1.smithi086.stdout:2/118: dwrite d0/f2a [0,4194304] 0 2024-04-06T14:17:02.087 INFO:tasks.workunit.client.0.smithi032.stdout:1/468: read d0/d9/d1f/f91 [3270890,14580] 0 2024-04-06T14:17:02.088 INFO:tasks.workunit.client.0.smithi032.stdout:5/371: dread d16/d18/d33/f34 [0,4194304] 0 2024-04-06T14:17:02.088 INFO:tasks.workunit.client.0.smithi032.stdout:1/469: stat d0/d41/f5c 0 2024-04-06T14:17:02.089 INFO:tasks.workunit.client.1.smithi086.stdout:1/141: write db/d11/d1f/f2b [599761,80702] 0 2024-04-06T14:17:02.090 INFO:tasks.workunit.client.0.smithi032.stdout:5/372: rename d16 to d16/d18/d33/d81 22 2024-04-06T14:17:02.091 INFO:tasks.workunit.client.1.smithi086.stdout:2/119: creat d0/d20/f2e x:0 0 0 2024-04-06T14:17:02.093 INFO:tasks.workunit.client.1.smithi086.stdout:2/120: truncate d0/f12 521937 0 2024-04-06T14:17:02.094 INFO:tasks.workunit.client.1.smithi086.stdout:6/160: link d6/d13/f26 d6/f3c 0 2024-04-06T14:17:02.095 INFO:tasks.workunit.client.1.smithi086.stdout:2/121: fdatasync d0/f2d 0 2024-04-06T14:17:02.096 INFO:tasks.workunit.client.1.smithi086.stdout:1/142: dwrite db/d11/f17 [0,4194304] 0 2024-04-06T14:17:02.097 INFO:tasks.workunit.client.0.smithi032.stdout:1/470: dwrite d0/f36 [0,4194304] 0 2024-04-06T14:17:02.100 INFO:tasks.workunit.client.1.smithi086.stdout:1/143: symlink db/d11/d1f/l2f 0 2024-04-06T14:17:02.101 INFO:tasks.workunit.client.1.smithi086.stdout:1/144: fsync db/fc 0 2024-04-06T14:17:02.104 INFO:tasks.workunit.client.1.smithi086.stdout:2/122: dwrite d0/f2d [0,4194304] 0 2024-04-06T14:17:02.105 INFO:tasks.workunit.client.0.smithi032.stdout:5/373: dwrite d16/f24 [0,4194304] 0 2024-04-06T14:17:02.106 INFO:tasks.workunit.client.1.smithi086.stdout:2/123: write d0/f2c [903256,112975] 0 2024-04-06T14:17:02.108 INFO:tasks.workunit.client.1.smithi086.stdout:1/145: mknod db/c30 0 2024-04-06T14:17:02.109 INFO:tasks.workunit.client.1.smithi086.stdout:6/161: stat d6/f2b 0 2024-04-06T14:17:02.109 INFO:tasks.workunit.client.0.smithi032.stdout:5/374: fsync f5 0 2024-04-06T14:17:02.111 INFO:tasks.workunit.client.1.smithi086.stdout:2/124: creat d0/d20/d27/f2f x:0 0 0 2024-04-06T14:17:02.114 INFO:tasks.workunit.client.1.smithi086.stdout:1/146: getdents db/d11/d1a/d2a 0 2024-04-06T14:17:02.116 INFO:tasks.workunit.client.0.smithi032.stdout:1/471: dread d0/d10/f26 [0,4194304] 0 2024-04-06T14:17:02.119 INFO:tasks.workunit.client.1.smithi086.stdout:6/162: rmdir d6/d13/d34 0 2024-04-06T14:17:02.119 INFO:tasks.workunit.client.0.smithi032.stdout:5/375: dwrite d16/d18/f20 [0,4194304] 0 2024-04-06T14:17:02.121 INFO:tasks.workunit.client.1.smithi086.stdout:2/125: getdents d0/d20 0 2024-04-06T14:17:02.123 INFO:tasks.workunit.client.1.smithi086.stdout:6/163: symlink d6/d13/l3d 0 2024-04-06T14:17:02.124 INFO:tasks.workunit.client.1.smithi086.stdout:2/126: creat d0/d20/f30 x:0 0 0 2024-04-06T14:17:02.126 INFO:tasks.workunit.client.1.smithi086.stdout:6/164: write d6/d9/db/f1e [323091,58580] 0 2024-04-06T14:17:02.126 INFO:tasks.workunit.client.0.smithi032.stdout:1/472: rename d0/d9/d1f/l4e to d0/l9b 0 2024-04-06T14:17:02.126 INFO:tasks.workunit.client.1.smithi086.stdout:2/127: truncate d0/df/f21 4825868 0 2024-04-06T14:17:02.131 INFO:tasks.workunit.client.1.smithi086.stdout:1/147: truncate db/f10 1597118 0 2024-04-06T14:17:02.134 INFO:tasks.workunit.client.1.smithi086.stdout:6/165: dwrite d6/d9/f31 [0,4194304] 0 2024-04-06T14:17:02.134 INFO:tasks.workunit.client.1.smithi086.stdout:2/128: rename d0/df/l11 to d0/l31 0 2024-04-06T14:17:02.136 INFO:tasks.workunit.client.0.smithi032.stdout:1/473: dwrite d0/da/d3e/f6c [0,4194304] 0 2024-04-06T14:17:02.136 INFO:tasks.workunit.client.1.smithi086.stdout:2/129: symlink d0/df/l32 0 2024-04-06T14:17:02.138 INFO:tasks.workunit.client.0.smithi032.stdout:1/474: symlink d0/d41/l9c 0 2024-04-06T14:17:02.139 INFO:tasks.workunit.client.0.smithi032.stdout:1/475: chown d0/f36 65672458 1 2024-04-06T14:17:02.144 INFO:tasks.workunit.client.0.smithi032.stdout:1/476: unlink d0/d41/d59/c89 0 2024-04-06T14:17:02.149 INFO:tasks.workunit.client.1.smithi086.stdout:2/130: rename d0/d20/f30 to d0/f33 0 2024-04-06T14:17:02.150 INFO:tasks.workunit.client.0.smithi032.stdout:1/477: dread d0/d10/d16/f3c [0,4194304] 0 2024-04-06T14:17:02.151 INFO:tasks.workunit.client.1.smithi086.stdout:2/131: symlink d0/df/l34 0 2024-04-06T14:17:02.154 INFO:tasks.workunit.client.0.smithi032.stdout:1/478: truncate d0/d10/f25 1744703 0 2024-04-06T14:17:02.155 INFO:tasks.workunit.client.0.smithi032.stdout:1/479: chown d0/da/f6b 0 1 2024-04-06T14:17:02.155 INFO:tasks.workunit.client.1.smithi086.stdout:2/132: mkdir d0/d20/d27/d2b/d35 0 2024-04-06T14:17:02.155 INFO:tasks.workunit.client.1.smithi086.stdout:2/133: readlink d0/l7 0 2024-04-06T14:17:02.156 INFO:tasks.workunit.client.1.smithi086.stdout:2/134: readlink d0/l7 0 2024-04-06T14:17:02.156 INFO:tasks.workunit.client.0.smithi032.stdout:1/480: write d0/d41/d59/f60 [3875410,113025] 0 2024-04-06T14:17:02.158 INFO:tasks.workunit.client.1.smithi086.stdout:2/135: mkdir d0/d20/d27/d2b/d35/d36 0 2024-04-06T14:17:02.161 INFO:tasks.workunit.client.0.smithi032.stdout:1/481: unlink d0/d10/d16/f5f 0 2024-04-06T14:17:02.164 INFO:tasks.workunit.client.0.smithi032.stdout:5/376: dread d16/d18/f2b [0,4194304] 0 2024-04-06T14:17:02.164 INFO:tasks.workunit.client.0.smithi032.stdout:1/482: rmdir d0/d9/d1f/d72 39 2024-04-06T14:17:02.167 INFO:tasks.workunit.client.0.smithi032.stdout:7/409: sync 2024-04-06T14:17:02.167 INFO:tasks.workunit.client.0.smithi032.stdout:0/282: sync 2024-04-06T14:17:02.167 INFO:tasks.workunit.client.0.smithi032.stdout:3/367: sync 2024-04-06T14:17:02.167 INFO:tasks.workunit.client.0.smithi032.stdout:4/435: sync 2024-04-06T14:17:02.168 INFO:tasks.workunit.client.0.smithi032.stdout:7/410: readlink d16/d20/l56 0 2024-04-06T14:17:02.169 INFO:tasks.workunit.client.1.smithi086.stdout:2/136: dwrite d0/f2d [0,4194304] 0 2024-04-06T14:17:02.169 INFO:tasks.workunit.client.0.smithi032.stdout:1/483: mknod d0/d10/d16/d3f/d98/c9d 0 2024-04-06T14:17:02.169 INFO:tasks.workunit.client.0.smithi032.stdout:0/283: chown d3/da/d25/d36 0 1 2024-04-06T14:17:02.171 INFO:tasks.workunit.client.1.smithi086.stdout:1/148: dread db/f10 [0,4194304] 0 2024-04-06T14:17:02.173 INFO:tasks.workunit.client.0.smithi032.stdout:7/411: write d16/d20/d48/d6d/f89 [860291,92535] 0 2024-04-06T14:17:02.173 INFO:tasks.workunit.client.0.smithi032.stdout:5/377: chown d16/d18/d33/d45/d52/f4e 325078 1 2024-04-06T14:17:02.174 INFO:tasks.workunit.client.0.smithi032.stdout:3/368: rename d3/fc to d3/d16/f9a 0 2024-04-06T14:17:02.177 INFO:tasks.workunit.client.0.smithi032.stdout:4/436: symlink d0/df/d2b/d46/d58/l98 0 2024-04-06T14:17:02.178 INFO:tasks.workunit.client.0.smithi032.stdout:0/284: creat d3/f63 x:0 0 0 2024-04-06T14:17:02.179 INFO:tasks.workunit.client.0.smithi032.stdout:7/412: fsync d16/d20/d26/d2d/d30/f46 0 2024-04-06T14:17:02.180 INFO:tasks.workunit.client.1.smithi086.stdout:1/149: dwrite db/d13/f29 [0,4194304] 0 2024-04-06T14:17:02.180 INFO:tasks.workunit.client.0.smithi032.stdout:0/285: stat d3/da/d25/d36/f51 0 2024-04-06T14:17:02.181 INFO:tasks.workunit.client.0.smithi032.stdout:1/484: rename d0/d9/d1f/l29 to d0/d41/d59/l9e 0 2024-04-06T14:17:02.182 INFO:tasks.workunit.client.0.smithi032.stdout:0/286: fdatasync d3/da/f1e 0 2024-04-06T14:17:02.187 INFO:tasks.workunit.client.0.smithi032.stdout:3/369: dwrite d3/d65/f34 [0,4194304] 0 2024-04-06T14:17:02.188 INFO:tasks.workunit.client.1.smithi086.stdout:1/150: dread db/d11/d1f/f2b [0,4194304] 0 2024-04-06T14:17:02.189 INFO:tasks.workunit.client.0.smithi032.stdout:7/413: symlink d16/d20/d7a/l8f 0 2024-04-06T14:17:02.191 INFO:tasks.workunit.client.1.smithi086.stdout:1/151: mknod db/d11/d1a/c31 0 2024-04-06T14:17:02.191 INFO:tasks.workunit.client.1.smithi086.stdout:1/152: chown db/d11/d1a/d2a 96 1 2024-04-06T14:17:02.194 INFO:tasks.workunit.client.0.smithi032.stdout:0/287: creat d3/da/f64 x:0 0 0 2024-04-06T14:17:02.194 INFO:tasks.workunit.client.0.smithi032.stdout:5/378: link d16/d18/f23 d16/d18/d33/d35/d4d/f82 0 2024-04-06T14:17:02.197 INFO:tasks.workunit.client.0.smithi032.stdout:4/437: creat d0/df/d1f/d42/f99 x:0 0 0 2024-04-06T14:17:02.197 INFO:tasks.workunit.client.1.smithi086.stdout:1/153: symlink db/d11/d1d/l32 0 2024-04-06T14:17:02.201 INFO:tasks.workunit.client.0.smithi032.stdout:5/379: symlink d16/d58/l83 0 2024-04-06T14:17:02.202 INFO:tasks.workunit.client.0.smithi032.stdout:3/370: link d3/f7e d3/d3b/f9b 0 2024-04-06T14:17:02.203 INFO:tasks.workunit.client.0.smithi032.stdout:7/414: write d16/d20/d7a/f83 [2945,17079] 0 2024-04-06T14:17:02.204 INFO:tasks.workunit.client.1.smithi086.stdout:1/154: rmdir db 39 2024-04-06T14:17:02.205 INFO:tasks.workunit.client.0.smithi032.stdout:1/485: getdents d0 0 2024-04-06T14:17:02.205 INFO:tasks.workunit.client.1.smithi086.stdout:1/155: readlink db/d11/d1f/l2f 0 2024-04-06T14:17:02.206 INFO:tasks.workunit.client.0.smithi032.stdout:4/438: mknod d0/df/d1f/d42/d45/c9a 0 2024-04-06T14:17:02.206 INFO:tasks.workunit.client.1.smithi086.stdout:1/156: chown c6 435 1 2024-04-06T14:17:02.206 INFO:tasks.workunit.client.1.smithi086.stdout:1/157: stat f2 0 2024-04-06T14:17:02.207 INFO:tasks.workunit.client.0.smithi032.stdout:5/380: rename d16/d18/d33/d45/d5e to d16/d18/d33/d45/d5e/d84 22 2024-04-06T14:17:02.210 INFO:tasks.workunit.client.0.smithi032.stdout:7/415: creat d16/d20/d26/d5f/f90 x:0 0 0 2024-04-06T14:17:02.211 INFO:tasks.workunit.client.0.smithi032.stdout:3/371: mkdir d3/d65/d7d/d9c 0 2024-04-06T14:17:02.212 INFO:tasks.workunit.client.0.smithi032.stdout:1/486: dread d0/d10/d16/f51 [0,4194304] 0 2024-04-06T14:17:02.213 INFO:tasks.workunit.client.1.smithi086.stdout:1/158: dwrite db/d11/d1f/f2b [0,4194304] 0 2024-04-06T14:17:02.213 INFO:tasks.workunit.client.0.smithi032.stdout:0/288: dread d3/f19 [0,4194304] 0 2024-04-06T14:17:02.214 INFO:tasks.workunit.client.1.smithi086.stdout:7/93: sync 2024-04-06T14:17:02.215 INFO:tasks.workunit.client.0.smithi032.stdout:7/416: creat d16/d20/d26/d5f/f91 x:0 0 0 2024-04-06T14:17:02.215 INFO:tasks.workunit.client.0.smithi032.stdout:3/372: mknod d3/d65/d7d/c9d 0 2024-04-06T14:17:02.215 INFO:tasks.workunit.client.0.smithi032.stdout:1/487: chown d0/d9/d1f/f3a 1448153 1 2024-04-06T14:17:02.216 INFO:tasks.workunit.client.0.smithi032.stdout:5/381: dwrite d16/f77 [0,4194304] 0 2024-04-06T14:17:02.216 INFO:tasks.workunit.client.1.smithi086.stdout:7/94: truncate d7/da/f19 314493 0 2024-04-06T14:17:02.218 INFO:tasks.workunit.client.0.smithi032.stdout:0/289: unlink d3/da/fe 0 2024-04-06T14:17:02.218 INFO:tasks.workunit.client.0.smithi032.stdout:1/488: read d0/f37 [1986822,40313] 0 2024-04-06T14:17:02.220 INFO:tasks.workunit.client.1.smithi086.stdout:7/95: truncate d7/d10/d11/f15 546311 0 2024-04-06T14:17:02.220 INFO:tasks.workunit.client.0.smithi032.stdout:7/417: dread - d16/d20/d26/f44 zero size 2024-04-06T14:17:02.221 INFO:tasks.workunit.client.1.smithi086.stdout:1/159: dwrite db/fc [0,4194304] 0 2024-04-06T14:17:02.221 INFO:tasks.workunit.client.0.smithi032.stdout:3/373: creat d3/d16/f9e x:0 0 0 2024-04-06T14:17:02.223 INFO:tasks.workunit.client.0.smithi032.stdout:1/489: readlink d0/d41/d59/l9e 0 2024-04-06T14:17:02.227 INFO:tasks.workunit.client.0.smithi032.stdout:0/290: creat d3/da/d25/d2b/d4a/f65 x:0 0 0 2024-04-06T14:17:02.227 INFO:tasks.workunit.client.0.smithi032.stdout:3/374: chown d3/d65/f47 2004742 1 2024-04-06T14:17:02.228 INFO:tasks.workunit.client.0.smithi032.stdout:0/291: write d3/da/d25/d36/f39 [4564014,19552] 0 2024-04-06T14:17:02.229 INFO:tasks.workunit.client.1.smithi086.stdout:7/96: dread d7/fb [0,4194304] 0 2024-04-06T14:17:02.232 INFO:tasks.workunit.client.1.smithi086.stdout:1/160: dread db/f16 [0,4194304] 0 2024-04-06T14:17:02.233 INFO:tasks.workunit.client.1.smithi086.stdout:1/161: write db/d13/f23 [962067,70865] 0 2024-04-06T14:17:02.234 INFO:tasks.workunit.client.0.smithi032.stdout:0/292: unlink d3/da/l1c 0 2024-04-06T14:17:02.234 INFO:tasks.workunit.client.1.smithi086.stdout:7/97: dread f5 [0,4194304] 0 2024-04-06T14:17:02.235 INFO:tasks.workunit.client.1.smithi086.stdout:7/98: chown d7/da 15 1 2024-04-06T14:17:02.235 INFO:tasks.workunit.client.0.smithi032.stdout:1/490: creat d0/d9/d1f/d72/f9f x:0 0 0 2024-04-06T14:17:02.236 INFO:tasks.workunit.client.0.smithi032.stdout:3/375: dwrite d3/d65/f66 [4194304,4194304] 0 2024-04-06T14:17:02.238 INFO:tasks.workunit.client.1.smithi086.stdout:7/99: truncate d7/da/fc 1410130 0 2024-04-06T14:17:02.238 INFO:tasks.workunit.client.0.smithi032.stdout:1/491: fdatasync d0/d10/f3d 0 2024-04-06T14:17:02.238 INFO:tasks.workunit.client.1.smithi086.stdout:7/100: chown d7/d10/d11 3 1 2024-04-06T14:17:02.240 INFO:tasks.workunit.client.0.smithi032.stdout:3/376: getdents d3/d65/d7d/d9c 0 2024-04-06T14:17:02.241 INFO:tasks.workunit.client.0.smithi032.stdout:1/492: mknod d0/d41/d5b/d96/ca0 0 2024-04-06T14:17:02.245 INFO:tasks.workunit.client.0.smithi032.stdout:3/377: rename d3/d65/c61 to d3/d65/d89/c9f 0 2024-04-06T14:17:02.248 INFO:tasks.workunit.client.0.smithi032.stdout:1/493: rename d0/f17 to d0/d10/d16/fa1 0 2024-04-06T14:17:02.251 INFO:tasks.workunit.client.0.smithi032.stdout:3/378: dwrite d3/f5e [0,4194304] 0 2024-04-06T14:17:02.273 INFO:tasks.workunit.client.0.smithi032.stdout:5/382: truncate d16/d18/d33/d45/d52/f53 2296671 0 2024-04-06T14:17:02.274 INFO:tasks.workunit.client.1.smithi086.stdout:7/101: write d7/da/fc [2398556,91841] 0 2024-04-06T14:17:02.275 INFO:tasks.workunit.client.0.smithi032.stdout:0/293: write d3/da/d35/f4e [1834589,35192] 0 2024-04-06T14:17:02.278 INFO:tasks.workunit.client.0.smithi032.stdout:3/379: write d3/d3b/d3c/f5f [1344938,113382] 0 2024-04-06T14:17:02.279 INFO:tasks.workunit.client.1.smithi086.stdout:7/102: rename d7/da to d7/d1d 0 2024-04-06T14:17:02.281 INFO:tasks.workunit.client.1.smithi086.stdout:7/103: write d7/d1d/fc [619557,128994] 0 2024-04-06T14:17:02.283 INFO:tasks.workunit.client.0.smithi032.stdout:3/380: truncate d3/d65/f8c 338285 0 2024-04-06T14:17:02.283 INFO:tasks.workunit.client.0.smithi032.stdout:5/383: dwrite d16/d18/d33/d45/d52/d39/f5b [0,4194304] 0 2024-04-06T14:17:02.283 INFO:tasks.workunit.client.1.smithi086.stdout:7/104: creat d7/d10/d11/d1c/f1e x:0 0 0 2024-04-06T14:17:02.284 INFO:tasks.workunit.client.1.smithi086.stdout:7/105: dread - d7/d10/d11/d1c/f1e zero size 2024-04-06T14:17:02.287 INFO:tasks.workunit.client.0.smithi032.stdout:1/494: dread d0/f4d [0,4194304] 0 2024-04-06T14:17:02.287 INFO:tasks.workunit.client.1.smithi086.stdout:7/106: mknod d7/d10/c1f 0 2024-04-06T14:17:02.287 INFO:tasks.workunit.client.0.smithi032.stdout:3/381: mknod d3/ca0 0 2024-04-06T14:17:02.288 INFO:tasks.workunit.client.1.smithi086.stdout:7/107: chown d7/le 234529023 1 2024-04-06T14:17:02.290 INFO:tasks.workunit.client.1.smithi086.stdout:7/108: symlink d7/d10/l20 0 2024-04-06T14:17:02.290 INFO:tasks.workunit.client.0.smithi032.stdout:0/294: link d3/da/d25/c58 d3/da/d35/d4b/c66 0 2024-04-06T14:17:02.292 INFO:tasks.workunit.client.1.smithi086.stdout:7/109: dread d7/d10/d11/f15 [0,4194304] 0 2024-04-06T14:17:02.292 INFO:tasks.workunit.client.0.smithi032.stdout:5/384: creat d16/d18/d33/d45/f85 x:0 0 0 2024-04-06T14:17:02.293 INFO:tasks.workunit.client.0.smithi032.stdout:1/495: write d0/da/d3e/d75/f8a [3276231,116742] 0 2024-04-06T14:17:02.295 INFO:tasks.workunit.client.0.smithi032.stdout:3/382: dwrite d3/d65/f66 [4194304,4194304] 0 2024-04-06T14:17:02.296 INFO:tasks.workunit.client.1.smithi086.stdout:7/110: dread d7/f9 [0,4194304] 0 2024-04-06T14:17:02.300 INFO:tasks.workunit.client.0.smithi032.stdout:0/295: link d3/l21 d3/da/d25/d2b/d4a/l67 0 2024-04-06T14:17:02.301 INFO:tasks.workunit.client.0.smithi032.stdout:0/296: truncate d3/da/f23 4701663 0 2024-04-06T14:17:02.302 INFO:tasks.workunit.client.0.smithi032.stdout:0/297: read - d3/f45 zero size 2024-04-06T14:17:02.304 INFO:tasks.workunit.client.0.smithi032.stdout:3/383: getdents d3/d16 0 2024-04-06T14:17:02.306 INFO:tasks.workunit.client.0.smithi032.stdout:3/384: readlink d3/d65/l30 0 2024-04-06T14:17:02.307 INFO:tasks.workunit.client.0.smithi032.stdout:5/385: dread d16/f38 [0,4194304] 0 2024-04-06T14:17:02.311 INFO:tasks.workunit.client.0.smithi032.stdout:3/385: fsync d3/d3b/f51 0 2024-04-06T14:17:02.311 INFO:tasks.workunit.client.1.smithi086.stdout:7/111: dwrite d7/d1d/f19 [0,4194304] 0 2024-04-06T14:17:02.313 INFO:tasks.workunit.client.0.smithi032.stdout:3/386: stat d3/d16/f33 0 2024-04-06T14:17:02.314 INFO:tasks.workunit.client.0.smithi032.stdout:1/496: dwrite d0/d9/f83 [0,4194304] 0 2024-04-06T14:17:02.314 INFO:tasks.workunit.client.0.smithi032.stdout:3/387: stat d3/d16/f2c 0 2024-04-06T14:17:02.315 INFO:tasks.workunit.client.1.smithi086.stdout:7/112: mkdir d7/d10/d11/d21 0 2024-04-06T14:17:02.317 INFO:tasks.workunit.client.0.smithi032.stdout:2/402: sync 2024-04-06T14:17:02.317 INFO:tasks.workunit.client.0.smithi032.stdout:6/375: sync 2024-04-06T14:17:02.317 INFO:tasks.workunit.client.0.smithi032.stdout:8/343: sync 2024-04-06T14:17:02.317 INFO:tasks.workunit.client.0.smithi032.stdout:9/423: sync 2024-04-06T14:17:02.318 INFO:tasks.workunit.client.0.smithi032.stdout:5/386: dwrite d16/d18/f20 [4194304,4194304] 0 2024-04-06T14:17:02.318 INFO:tasks.workunit.client.0.smithi032.stdout:1/497: dread - d0/d10/f8d zero size 2024-04-06T14:17:02.319 INFO:tasks.workunit.client.0.smithi032.stdout:3/388: creat d3/d16/d63/fa1 x:0 0 0 2024-04-06T14:17:02.319 INFO:tasks.workunit.client.0.smithi032.stdout:9/424: readlink d2/d15/l20 0 2024-04-06T14:17:02.321 INFO:tasks.workunit.client.1.smithi086.stdout:7/113: rename d7/d10/d11/f15 to d7/d10/d11/d1c/f22 0 2024-04-06T14:17:02.323 INFO:tasks.workunit.client.1.smithi086.stdout:7/114: read d7/d1d/fc [418579,7087] 0 2024-04-06T14:17:02.324 INFO:tasks.workunit.client.0.smithi032.stdout:8/344: mkdir d3/d8/d12/d2f/d4a/d60/d82 0 2024-04-06T14:17:02.327 INFO:tasks.workunit.client.0.smithi032.stdout:1/498: creat d0/d9/d1f/d72/fa2 x:0 0 0 2024-04-06T14:17:02.327 INFO:tasks.workunit.client.0.smithi032.stdout:8/345: chown d3/d8/d7/d14/l21 329 1 2024-04-06T14:17:02.327 INFO:tasks.workunit.client.0.smithi032.stdout:0/298: truncate d3/da/d25/f2a 2744651 0 2024-04-06T14:17:02.328 INFO:tasks.workunit.client.1.smithi086.stdout:7/115: rename d7/d10/l18 to d7/d10/d11/d1c/l23 0 2024-04-06T14:17:02.328 INFO:tasks.workunit.client.0.smithi032.stdout:9/425: mkdir d2/d15/d1e/d41/d6c/d88 0 2024-04-06T14:17:02.328 INFO:tasks.workunit.client.0.smithi032.stdout:3/389: creat d3/d65/d89/fa2 x:0 0 0 2024-04-06T14:17:02.328 INFO:tasks.workunit.client.1.smithi086.stdout:7/116: fsync d7/d1d/f13 0 2024-04-06T14:17:02.331 INFO:tasks.workunit.client.1.smithi086.stdout:7/117: mkdir d7/d10/d11/d1c/d24 0 2024-04-06T14:17:02.331 INFO:tasks.workunit.client.1.smithi086.stdout:7/118: stat d7/le 0 2024-04-06T14:17:02.338 INFO:tasks.workunit.client.1.smithi086.stdout:7/119: dwrite d7/d10/d11/d1c/f1e [0,4194304] 0 2024-04-06T14:17:02.339 INFO:tasks.workunit.client.1.smithi086.stdout:7/120: stat d7/d10/d11/d1c/f22 0 2024-04-06T14:17:02.340 INFO:tasks.workunit.client.0.smithi032.stdout:6/376: link d4/d5/f6a d4/d5/d18/f74 0 2024-04-06T14:17:02.340 INFO:tasks.workunit.client.0.smithi032.stdout:0/299: creat d3/da/d25/d36/f68 x:0 0 0 2024-04-06T14:17:02.341 INFO:tasks.workunit.client.1.smithi086.stdout:7/121: mknod d7/d10/c25 0 2024-04-06T14:17:02.342 INFO:tasks.workunit.client.0.smithi032.stdout:6/377: write d4/d5/d28/f5c [2423078,52470] 0 2024-04-06T14:17:02.343 INFO:tasks.workunit.client.0.smithi032.stdout:0/300: chown d3/da/l2c 194979021 1 2024-04-06T14:17:02.344 INFO:tasks.workunit.client.0.smithi032.stdout:8/346: mkdir d3/d8/d12/d83 0 2024-04-06T14:17:02.345 INFO:tasks.workunit.client.1.smithi086.stdout:7/122: symlink d7/d10/d11/d1c/l26 0 2024-04-06T14:17:02.346 INFO:tasks.workunit.client.1.smithi086.stdout:7/123: write d7/fb [2524266,38965] 0 2024-04-06T14:17:02.349 INFO:tasks.workunit.client.1.smithi086.stdout:7/124: rename d7/d1d/cd to d7/d10/d1b/c27 0 2024-04-06T14:17:02.352 INFO:tasks.workunit.client.1.smithi086.stdout:7/125: link d7/d10/l20 d7/d10/d11/l28 0 2024-04-06T14:17:02.358 INFO:tasks.workunit.client.1.smithi086.stdout:7/126: write d7/d10/d11/d1c/f22 [330024,20563] 0 2024-04-06T14:17:02.362 INFO:tasks.workunit.client.1.smithi086.stdout:7/127: dread f5 [0,4194304] 0 2024-04-06T14:17:02.365 INFO:tasks.workunit.client.1.smithi086.stdout:7/128: dread d7/d1d/f19 [0,4194304] 0 2024-04-06T14:17:02.371 INFO:tasks.workunit.client.1.smithi086.stdout:7/129: write d7/d1d/f19 [4475159,24841] 0 2024-04-06T14:17:02.371 INFO:tasks.workunit.client.1.smithi086.stdout:7/130: rmdir d7/d10/d1b 39 2024-04-06T14:17:02.371 INFO:tasks.workunit.client.1.smithi086.stdout:7/131: unlink d7/d10/d11/d1c/l23 0 2024-04-06T14:17:02.371 INFO:tasks.workunit.client.1.smithi086.stdout:7/132: fdatasync d7/d1d/f19 0 2024-04-06T14:17:02.373 INFO:tasks.workunit.client.1.smithi086.stdout:7/133: mknod d7/d1d/c29 0 2024-04-06T14:17:02.374 INFO:tasks.workunit.client.1.smithi086.stdout:0/184: sync 2024-04-06T14:17:02.374 INFO:tasks.workunit.client.1.smithi086.stdout:3/195: sync 2024-04-06T14:17:02.375 INFO:tasks.workunit.client.1.smithi086.stdout:7/134: symlink d7/d10/d11/d21/l2a 0 2024-04-06T14:17:02.375 INFO:tasks.workunit.client.1.smithi086.stdout:3/196: rmdir d5/d33 39 2024-04-06T14:17:02.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:02 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:17:02.378 INFO:tasks.workunit.client.1.smithi086.stdout:3/197: mkdir d5/d36 0 2024-04-06T14:17:02.381 INFO:tasks.workunit.client.1.smithi086.stdout:0/185: dwrite d0/ff [4194304,4194304] 0 2024-04-06T14:17:02.381 INFO:tasks.workunit.client.1.smithi086.stdout:7/135: getdents d7/d10/d1b 0 2024-04-06T14:17:02.382 INFO:tasks.workunit.client.1.smithi086.stdout:0/186: stat d0/d1/d1b/c37 0 2024-04-06T14:17:02.382 INFO:tasks.workunit.client.1.smithi086.stdout:7/136: truncate d7/d1d/f13 555867 0 2024-04-06T14:17:02.383 INFO:tasks.workunit.client.1.smithi086.stdout:0/187: dread d0/d23/f45 [0,4194304] 0 2024-04-06T14:17:02.385 INFO:tasks.workunit.client.1.smithi086.stdout:3/198: rename d5/d33/f31 to d5/f37 0 2024-04-06T14:17:02.386 INFO:tasks.workunit.client.1.smithi086.stdout:7/137: symlink d7/d10/l2b 0 2024-04-06T14:17:02.386 INFO:tasks.workunit.client.1.smithi086.stdout:0/188: rmdir d0/d1/d2/d1a 39 2024-04-06T14:17:02.389 INFO:tasks.workunit.client.1.smithi086.stdout:7/138: dread d7/f9 [0,4194304] 0 2024-04-06T14:17:02.391 INFO:tasks.workunit.client.1.smithi086.stdout:0/189: getdents d0/d1/d25 0 2024-04-06T14:17:02.394 INFO:tasks.workunit.client.1.smithi086.stdout:3/199: dwrite d5/df/f1c [0,4194304] 0 2024-04-06T14:17:02.396 INFO:tasks.workunit.client.1.smithi086.stdout:3/200: write d5/df/f13 [7485144,88241] 0 2024-04-06T14:17:02.397 INFO:tasks.workunit.client.1.smithi086.stdout:7/139: dread d7/d1d/f19 [0,4194304] 0 2024-04-06T14:17:02.398 INFO:tasks.workunit.client.1.smithi086.stdout:3/201: write d5/d33/f35 [931213,152] 0 2024-04-06T14:17:02.399 INFO:tasks.workunit.client.1.smithi086.stdout:3/202: fsync d5/df/f2c 0 2024-04-06T14:17:02.404 INFO:tasks.workunit.client.1.smithi086.stdout:0/190: dwrite d0/d1/f13 [4194304,4194304] 0 2024-04-06T14:17:02.404 INFO:tasks.workunit.client.1.smithi086.stdout:7/140: rename d7/d10/d1b to d7/d10/d11/d1c/d24/d2c 0 2024-04-06T14:17:02.405 INFO:tasks.workunit.client.1.smithi086.stdout:3/203: write f4 [2300422,117493] 0 2024-04-06T14:17:02.406 INFO:tasks.workunit.client.1.smithi086.stdout:3/204: truncate d5/f9 3194736 0 2024-04-06T14:17:02.407 INFO:tasks.workunit.client.1.smithi086.stdout:0/191: mkdir d0/d1/d1b/d46 0 2024-04-06T14:17:02.409 INFO:tasks.workunit.client.1.smithi086.stdout:7/141: dread d7/fb [0,4194304] 0 2024-04-06T14:17:02.409 INFO:tasks.workunit.client.1.smithi086.stdout:7/142: stat d7/d10/d11 0 2024-04-06T14:17:02.410 INFO:tasks.workunit.client.1.smithi086.stdout:0/192: rename d0/d1 to d0/d1/d2/d47 22 2024-04-06T14:17:02.412 INFO:tasks.workunit.client.1.smithi086.stdout:7/143: mknod d7/d1d/c2d 0 2024-04-06T14:17:02.415 INFO:tasks.workunit.client.1.smithi086.stdout:0/193: rename d0/d15/d26/d29 to d0/d23/d48 0 2024-04-06T14:17:02.416 INFO:tasks.workunit.client.1.smithi086.stdout:0/194: dread - d0/d1/d4/de/f2e zero size 2024-04-06T14:17:02.417 INFO:tasks.workunit.client.1.smithi086.stdout:3/205: getdents d5/d2b 0 2024-04-06T14:17:02.419 INFO:tasks.workunit.client.1.smithi086.stdout:7/144: creat d7/d10/d11/d1c/f2e x:0 0 0 2024-04-06T14:17:02.420 INFO:tasks.workunit.client.0.smithi032.stdout:9/426: rename d2/d15/f27 to d2/d15/d1e/d41/d6c/d88/f89 0 2024-04-06T14:17:02.420 INFO:tasks.workunit.client.1.smithi086.stdout:3/206: symlink d5/l38 0 2024-04-06T14:17:02.421 INFO:tasks.workunit.client.0.smithi032.stdout:6/378: mknod d4/d5/d18/d23/c75 0 2024-04-06T14:17:02.421 INFO:tasks.workunit.client.1.smithi086.stdout:3/207: chown d5 191861440 1 2024-04-06T14:17:02.422 INFO:tasks.workunit.client.0.smithi032.stdout:1/499: dwrite d0/da/d3e/f50 [0,4194304] 0 2024-04-06T14:17:02.424 INFO:tasks.workunit.client.0.smithi032.stdout:3/390: dwrite d3/d65/d89/fa2 [0,4194304] 0 2024-04-06T14:17:02.425 INFO:tasks.workunit.client.0.smithi032.stdout:1/500: truncate d0/d10/f26 2522767 0 2024-04-06T14:17:02.425 INFO:tasks.workunit.client.0.smithi032.stdout:3/391: chown d3/d3b/d3c/f96 827 1 2024-04-06T14:17:02.426 INFO:tasks.workunit.client.0.smithi032.stdout:1/501: readlink d0/d10/l56 0 2024-04-06T14:17:02.426 INFO:tasks.workunit.client.1.smithi086.stdout:7/145: rmdir d7/d10/d11 39 2024-04-06T14:17:02.428 INFO:tasks.workunit.client.1.smithi086.stdout:0/195: truncate d0/d23/d48/f31 1058187 0 2024-04-06T14:17:02.429 INFO:tasks.workunit.client.1.smithi086.stdout:3/208: dwrite d5/df/f1c [4194304,4194304] 0 2024-04-06T14:17:02.431 INFO:tasks.workunit.client.0.smithi032.stdout:5/387: dread f10 [0,4194304] 0 2024-04-06T14:17:02.431 INFO:tasks.workunit.client.0.smithi032.stdout:5/388: chown d16/d18/d33/f34 1 1 2024-04-06T14:17:02.434 INFO:tasks.workunit.client.1.smithi086.stdout:7/146: mknod d7/d10/d11/d1c/c2f 0 2024-04-06T14:17:02.434 INFO:tasks.workunit.client.0.smithi032.stdout:6/379: mknod d4/d5/d19/c76 0 2024-04-06T14:17:02.434 INFO:tasks.workunit.client.0.smithi032.stdout:9/427: mknod d2/d15/d55/d5f/d71/c8a 0 2024-04-06T14:17:02.435 INFO:tasks.workunit.client.0.smithi032.stdout:1/502: creat d0/d10/d16/d3f/fa3 x:0 0 0 2024-04-06T14:17:02.437 INFO:tasks.workunit.client.0.smithi032.stdout:9/428: chown d2/d15/d55/f85 77827216 1 2024-04-06T14:17:02.438 INFO:tasks.workunit.client.1.smithi086.stdout:3/209: symlink d5/d33/l39 0 2024-04-06T14:17:02.438 INFO:tasks.workunit.client.0.smithi032.stdout:8/347: mkdir d3/d8/d12/d2f/d4a/d60/d82/d84 0 2024-04-06T14:17:02.439 INFO:tasks.workunit.client.0.smithi032.stdout:0/301: mknod d3/da/d41/c69 0 2024-04-06T14:17:02.442 INFO:tasks.workunit.client.0.smithi032.stdout:5/389: write d16/d18/f30 [973588,125797] 0 2024-04-06T14:17:02.442 INFO:tasks.workunit.client.0.smithi032.stdout:1/503: symlink d0/da/d3e/la4 0 2024-04-06T14:17:02.442 INFO:tasks.workunit.client.1.smithi086.stdout:7/147: mkdir d7/d10/d30 0 2024-04-06T14:17:02.442 INFO:tasks.workunit.client.0.smithi032.stdout:9/429: symlink d2/d15/d1e/d41/d6c/l8b 0 2024-04-06T14:17:02.444 INFO:tasks.workunit.client.0.smithi032.stdout:0/302: fsync d3/da/d25/d2b/d4a/f62 0 2024-04-06T14:17:02.444 INFO:tasks.workunit.client.0.smithi032.stdout:5/390: truncate f9 421581 0 2024-04-06T14:17:02.445 INFO:tasks.workunit.client.0.smithi032.stdout:6/380: fdatasync d4/f10 0 2024-04-06T14:17:02.445 INFO:tasks.workunit.client.0.smithi032.stdout:3/392: dwrite d3/d3b/d3c/f85 [0,4194304] 0 2024-04-06T14:17:02.447 INFO:tasks.workunit.client.0.smithi032.stdout:6/381: stat l3 0 2024-04-06T14:17:02.447 INFO:tasks.workunit.client.1.smithi086.stdout:3/210: creat d5/d33/f3a x:0 0 0 2024-04-06T14:17:02.448 INFO:tasks.workunit.client.0.smithi032.stdout:8/348: write d3/d8/d12/f51 [254428,87979] 0 2024-04-06T14:17:02.450 INFO:tasks.workunit.client.0.smithi032.stdout:1/504: creat d0/d10/d16/fa5 x:0 0 0 2024-04-06T14:17:02.451 INFO:tasks.workunit.client.1.smithi086.stdout:7/148: dwrite d7/d10/d11/d1c/f1e [0,4194304] 0 2024-04-06T14:17:02.451 INFO:tasks.workunit.client.0.smithi032.stdout:8/349: write d3/d8/d12/d2f/d46/f53 [946503,72873] 0 2024-04-06T14:17:02.452 INFO:tasks.workunit.client.0.smithi032.stdout:0/303: creat d3/da/f6a x:0 0 0 2024-04-06T14:17:02.453 INFO:tasks.workunit.client.0.smithi032.stdout:3/393: mknod d3/d16/ca3 0 2024-04-06T14:17:02.453 INFO:tasks.workunit.client.0.smithi032.stdout:0/304: write d3/da/d25/d2b/f32 [2823484,61233] 0 2024-04-06T14:17:02.456 INFO:tasks.workunit.client.0.smithi032.stdout:6/382: mkdir d4/d5/d8/d2b/d77 0 2024-04-06T14:17:02.458 INFO:tasks.workunit.client.0.smithi032.stdout:8/350: write d3/d8/f7a [403348,64608] 0 2024-04-06T14:17:02.459 INFO:tasks.workunit.client.1.smithi086.stdout:3/211: dwrite d5/d33/f3a [0,4194304] 0 2024-04-06T14:17:02.461 INFO:tasks.workunit.client.0.smithi032.stdout:3/394: dwrite d3/d3b/d3c/f78 [0,4194304] 0 2024-04-06T14:17:02.461 INFO:tasks.workunit.client.1.smithi086.stdout:7/149: symlink d7/d10/d11/d1c/l31 0 2024-04-06T14:17:02.463 INFO:tasks.workunit.client.1.smithi086.stdout:7/150: readlink d7/d1d/l14 0 2024-04-06T14:17:02.465 INFO:tasks.workunit.client.0.smithi032.stdout:8/351: mkdir d3/d8/d12/d2f/d4a/d60/d75/d85 0 2024-04-06T14:17:02.466 INFO:tasks.workunit.client.0.smithi032.stdout:8/352: readlink d3/d8/d12/d2f/d46/l50 0 2024-04-06T14:17:02.468 INFO:tasks.workunit.client.1.smithi086.stdout:3/212: rename d5/f11 to d5/d36/f3b 0 2024-04-06T14:17:02.469 INFO:tasks.workunit.client.0.smithi032.stdout:6/383: read d4/d5/d19/d21/f30 [598995,51279] 0 2024-04-06T14:17:02.469 INFO:tasks.workunit.client.1.smithi086.stdout:3/213: dread - d5/df/f2c zero size 2024-04-06T14:17:02.473 INFO:tasks.workunit.client.0.smithi032.stdout:3/395: creat d3/d3b/fa4 x:0 0 0 2024-04-06T14:17:02.473 INFO:tasks.workunit.client.0.smithi032.stdout:6/384: chown d4/d5/d19/f2e 485219671 1 2024-04-06T14:17:02.473 INFO:tasks.workunit.client.0.smithi032.stdout:3/396: chown d3/ca0 1 1 2024-04-06T14:17:02.474 INFO:tasks.workunit.client.1.smithi086.stdout:7/151: rename d7/d10/d11/l12 to d7/d10/l32 0 2024-04-06T14:17:02.476 INFO:tasks.workunit.client.1.smithi086.stdout:5/200: sync 2024-04-06T14:17:02.476 INFO:tasks.workunit.client.1.smithi086.stdout:3/214: fdatasync d5/df/f25 0 2024-04-06T14:17:02.476 INFO:tasks.workunit.client.1.smithi086.stdout:4/121: sync 2024-04-06T14:17:02.477 INFO:tasks.workunit.client.0.smithi032.stdout:6/385: creat d4/d5/d14/f78 x:0 0 0 2024-04-06T14:17:02.478 INFO:tasks.workunit.client.0.smithi032.stdout:3/397: symlink d3/d3b/d3c/la5 0 2024-04-06T14:17:02.479 INFO:tasks.workunit.client.1.smithi086.stdout:3/215: readlink d5/df/l16 0 2024-04-06T14:17:02.479 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:02 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:17:02.480 INFO:tasks.workunit.client.0.smithi032.stdout:8/353: dwrite d3/d8/d7/d1a/f61 [0,4194304] 0 2024-04-06T14:17:02.480 INFO:tasks.workunit.client.1.smithi086.stdout:3/216: chown d5/d33 233 1 2024-04-06T14:17:02.481 INFO:tasks.workunit.client.0.smithi032.stdout:8/354: stat d3/d8/d12/d2f/d46/f74 0 2024-04-06T14:17:02.481 INFO:tasks.workunit.client.0.smithi032.stdout:0/305: rename d3/l21 to d3/da/d41/l6b 0 2024-04-06T14:17:02.482 INFO:tasks.workunit.client.0.smithi032.stdout:3/398: chown d3/d3b/c41 38 1 2024-04-06T14:17:02.483 INFO:tasks.workunit.client.0.smithi032.stdout:6/386: symlink d4/d5/d8/d1f/d61/l79 0 2024-04-06T14:17:02.483 INFO:tasks.workunit.client.1.smithi086.stdout:3/217: rmdir d5/d36 39 2024-04-06T14:17:02.485 INFO:tasks.workunit.client.0.smithi032.stdout:0/306: dread d3/da/d25/d2b/f3d [0,4194304] 0 2024-04-06T14:17:02.486 INFO:tasks.workunit.client.0.smithi032.stdout:6/387: mknod d4/d5/d28/d65/c7a 0 2024-04-06T14:17:02.487 INFO:tasks.workunit.client.1.smithi086.stdout:4/122: dwrite d0/f1d [0,4194304] 0 2024-04-06T14:17:02.487 INFO:tasks.workunit.client.0.smithi032.stdout:3/399: mknod d3/d65/d7d/d9c/ca6 0 2024-04-06T14:17:02.489 INFO:tasks.workunit.client.1.smithi086.stdout:5/201: dread d0/d19/f3c [0,4194304] 0 2024-04-06T14:17:02.490 INFO:tasks.workunit.client.1.smithi086.stdout:4/123: read d0/d15/d19/f1c [45888,34203] 0 2024-04-06T14:17:02.490 INFO:tasks.workunit.client.1.smithi086.stdout:9/183: sync 2024-04-06T14:17:02.490 INFO:tasks.workunit.client.1.smithi086.stdout:8/275: sync 2024-04-06T14:17:02.491 INFO:tasks.workunit.client.0.smithi032.stdout:8/355: dwrite d3/d8/d7/d14/f1f [0,4194304] 0 2024-04-06T14:17:02.494 INFO:tasks.workunit.client.1.smithi086.stdout:6/166: sync 2024-04-06T14:17:02.494 INFO:tasks.workunit.client.1.smithi086.stdout:9/184: chown d5/d18/c2c 7605957 1 2024-04-06T14:17:02.495 INFO:tasks.workunit.client.1.smithi086.stdout:1/162: sync 2024-04-06T14:17:02.495 INFO:tasks.workunit.client.1.smithi086.stdout:2/137: sync 2024-04-06T14:17:02.495 INFO:tasks.workunit.client.0.smithi032.stdout:3/400: symlink d3/la7 0 2024-04-06T14:17:02.497 INFO:tasks.workunit.client.0.smithi032.stdout:8/356: rename d3/d8/d7/d14/c17 to d3/d8/d12/d83/c86 0 2024-04-06T14:17:02.497 INFO:tasks.workunit.client.1.smithi086.stdout:6/167: readlink d6/l11 0 2024-04-06T14:17:02.499 INFO:tasks.workunit.client.0.smithi032.stdout:0/307: mknod d3/da/d25/d57/c6c 0 2024-04-06T14:17:02.500 INFO:tasks.workunit.client.0.smithi032.stdout:5/391: write f5 [1586992,18344] 0 2024-04-06T14:17:02.500 INFO:tasks.workunit.client.1.smithi086.stdout:9/185: mknod d5/d13/c46 0 2024-04-06T14:17:02.501 INFO:tasks.workunit.client.1.smithi086.stdout:0/196: dread d0/d23/d48/f31 [0,4194304] 0 2024-04-06T14:17:02.502 INFO:tasks.workunit.client.0.smithi032.stdout:5/392: dread - d16/d18/f3d zero size 2024-04-06T14:17:02.502 INFO:tasks.workunit.client.0.smithi032.stdout:3/401: mknod d3/d3b/ca8 0 2024-04-06T14:17:02.503 INFO:tasks.workunit.client.1.smithi086.stdout:2/138: mkdir d0/d20/d27/d37 0 2024-04-06T14:17:02.505 INFO:tasks.workunit.client.1.smithi086.stdout:6/168: readlink d6/l8 0 2024-04-06T14:17:02.506 INFO:tasks.workunit.client.0.smithi032.stdout:3/402: mknod d3/d16/ca9 0 2024-04-06T14:17:02.507 INFO:tasks.workunit.client.0.smithi032.stdout:9/430: dread d2/d15/d1e/d41/f63 [0,4194304] 0 2024-04-06T14:17:02.507 INFO:tasks.workunit.client.1.smithi086.stdout:4/124: dwrite d0/f7 [0,4194304] 0 2024-04-06T14:17:02.508 INFO:tasks.workunit.client.0.smithi032.stdout:0/308: mknod d3/da/d25/c6d 0 2024-04-06T14:17:02.509 INFO:tasks.workunit.client.1.smithi086.stdout:4/125: chown d0/de 5915 1 2024-04-06T14:17:02.510 INFO:tasks.workunit.client.0.smithi032.stdout:6/388: dread d4/d5/f2a [0,4194304] 0 2024-04-06T14:17:02.510 INFO:tasks.workunit.client.1.smithi086.stdout:4/126: write d0/f5 [3059408,116705] 0 2024-04-06T14:17:02.511 INFO:tasks.workunit.client.0.smithi032.stdout:9/431: dread - d2/d15/d55/d5f/d71/d81/d4a/d73/d7c/f80 zero size 2024-04-06T14:17:02.511 INFO:tasks.workunit.client.0.smithi032.stdout:0/309: chown d3/da/d25/c6d 115812939 1 2024-04-06T14:17:02.513 INFO:tasks.workunit.client.1.smithi086.stdout:3/218: write d5/f37 [3643858,89353] 0 2024-04-06T14:17:02.513 INFO:tasks.workunit.client.0.smithi032.stdout:9/432: readlink d2/d15/d55/d59/l7a 0 2024-04-06T14:17:02.514 INFO:tasks.workunit.client.0.smithi032.stdout:8/357: getdents d3/d8/d12/d2f/d46 0 2024-04-06T14:17:02.518 INFO:tasks.workunit.client.1.smithi086.stdout:6/169: dwrite d6/f2b [0,4194304] 0 2024-04-06T14:17:02.518 INFO:tasks.workunit.client.1.smithi086.stdout:0/197: mknod d0/d15/d26/d33/c49 0 2024-04-06T14:17:02.519 INFO:tasks.workunit.client.0.smithi032.stdout:0/310: symlink d3/da/d25/d2b/d4a/l6e 0 2024-04-06T14:17:02.522 INFO:tasks.workunit.client.1.smithi086.stdout:8/276: link d0/d8/d22/c36 d0/d8/d22/d43/c5c 0 2024-04-06T14:17:02.522 INFO:tasks.workunit.client.0.smithi032.stdout:5/393: getdents d16/d58/d54/d74 0 2024-04-06T14:17:02.525 INFO:tasks.workunit.client.1.smithi086.stdout:2/139: link d0/d20/c29 d0/d20/d27/d37/c38 0 2024-04-06T14:17:02.525 INFO:tasks.workunit.client.0.smithi032.stdout:6/389: dwrite d4/d5/d19/d25/d67/f56 [0,4194304] 0 2024-04-06T14:17:02.526 INFO:tasks.workunit.client.1.smithi086.stdout:2/140: write d0/f2d [4962383,105076] 0 2024-04-06T14:17:02.529 INFO:tasks.workunit.client.0.smithi032.stdout:3/403: dread d3/d65/f69 [0,4194304] 0 2024-04-06T14:17:02.529 INFO:tasks.workunit.client.1.smithi086.stdout:1/163: getdents db 0 2024-04-06T14:17:02.530 INFO:tasks.workunit.client.0.smithi032.stdout:8/358: creat d3/d8/f87 x:0 0 0 2024-04-06T14:17:02.531 INFO:tasks.workunit.client.0.smithi032.stdout:9/433: dread d2/d15/f43 [0,4194304] 0 2024-04-06T14:17:02.532 INFO:tasks.workunit.client.0.smithi032.stdout:5/394: mkdir d16/d18/d33/d45/d52/d39/d86 0 2024-04-06T14:17:02.533 INFO:tasks.workunit.client.0.smithi032.stdout:3/404: chown d3/d16/f55 23 1 2024-04-06T14:17:02.533 INFO:tasks.workunit.client.1.smithi086.stdout:0/198: mknod d0/d1/d1b/d46/c4a 0 2024-04-06T14:17:02.533 INFO:tasks.workunit.client.1.smithi086.stdout:8/277: creat d0/db/d32/f5d x:0 0 0 2024-04-06T14:17:02.534 INFO:tasks.workunit.client.0.smithi032.stdout:6/390: creat d4/d5/d8/d1f/d51/f7b x:0 0 0 2024-04-06T14:17:02.536 INFO:tasks.workunit.client.1.smithi086.stdout:9/186: write d5/f27 [1266593,54461] 0 2024-04-06T14:17:02.538 INFO:tasks.workunit.client.1.smithi086.stdout:7/152: rename d7/d10/l32 to d7/d10/l33 0 2024-04-06T14:17:02.538 INFO:tasks.workunit.client.0.smithi032.stdout:5/395: write d16/d18/d33/f41 [281556,2794] 0 2024-04-06T14:17:02.539 INFO:tasks.workunit.client.1.smithi086.stdout:5/202: dwrite d0/d19/f26 [4194304,4194304] 0 2024-04-06T14:17:02.540 INFO:tasks.workunit.client.0.smithi032.stdout:8/359: rmdir d3/d8/d12/d2f/d4a 39 2024-04-06T14:17:02.540 INFO:tasks.workunit.client.0.smithi032.stdout:0/311: dwrite d3/da/d25/d2b/f32 [0,4194304] 0 2024-04-06T14:17:02.541 INFO:tasks.workunit.client.1.smithi086.stdout:2/141: rename d0/df/l34 to d0/d20/d27/l39 0 2024-04-06T14:17:02.541 INFO:tasks.workunit.client.0.smithi032.stdout:6/391: mkdir d4/d5/d8/d4b/d7c 0 2024-04-06T14:17:02.543 INFO:tasks.workunit.client.0.smithi032.stdout:4/439: sync 2024-04-06T14:17:02.543 INFO:tasks.workunit.client.0.smithi032.stdout:7/418: sync 2024-04-06T14:17:02.545 INFO:tasks.workunit.client.0.smithi032.stdout:0/312: chown d3/f14 0 1 2024-04-06T14:17:02.548 INFO:tasks.workunit.client.1.smithi086.stdout:6/170: creat d6/d13/f3e x:0 0 0 2024-04-06T14:17:02.548 INFO:tasks.workunit.client.1.smithi086.stdout:0/199: creat d0/d15/d26/f4b x:0 0 0 2024-04-06T14:17:02.549 INFO:tasks.workunit.client.1.smithi086.stdout:4/127: getdents d0/d15 0 2024-04-06T14:17:02.549 INFO:tasks.workunit.client.0.smithi032.stdout:8/360: dread - d3/d8/f5f zero size 2024-04-06T14:17:02.549 INFO:tasks.workunit.client.0.smithi032.stdout:3/405: dwrite d3/d16/f18 [4194304,4194304] 0 2024-04-06T14:17:02.549 INFO:tasks.workunit.client.0.smithi032.stdout:4/440: chown d0/d82 3 1 2024-04-06T14:17:02.550 INFO:tasks.workunit.client.0.smithi032.stdout:8/361: chown d3/f79 932 1 2024-04-06T14:17:02.550 INFO:tasks.workunit.client.0.smithi032.stdout:3/406: dread - d3/d65/d89/f97 zero size 2024-04-06T14:17:02.553 INFO:tasks.workunit.client.0.smithi032.stdout:3/407: chown d3/f80 0 1 2024-04-06T14:17:02.553 INFO:tasks.workunit.client.1.smithi086.stdout:5/203: rmdir d0/d19 39 2024-04-06T14:17:02.553 INFO:tasks.workunit.client.1.smithi086.stdout:3/219: getdents d5/d33 0 2024-04-06T14:17:02.554 INFO:tasks.workunit.client.1.smithi086.stdout:1/164: dwrite db/fe [0,4194304] 0 2024-04-06T14:17:02.555 INFO:tasks.workunit.client.0.smithi032.stdout:6/392: creat d4/d5/d19/f7d x:0 0 0 2024-04-06T14:17:02.556 INFO:tasks.workunit.client.0.smithi032.stdout:4/441: rename d0/d82/f85 to d0/df/d1f/d2e/f9b 0 2024-04-06T14:17:02.556 INFO:tasks.workunit.client.0.smithi032.stdout:3/408: write d3/d65/f34 [1373718,121408] 0 2024-04-06T14:17:02.556 INFO:tasks.workunit.client.0.smithi032.stdout:5/396: mkdir d16/d18/d33/d45/d87 0 2024-04-06T14:17:02.559 INFO:tasks.workunit.client.1.smithi086.stdout:5/204: chown d0/fa 9786 1 2024-04-06T14:17:02.559 INFO:tasks.workunit.client.1.smithi086.stdout:4/128: truncate d0/de/f26 777647 0 2024-04-06T14:17:02.560 INFO:tasks.workunit.client.1.smithi086.stdout:0/200: creat d0/d15/f4c x:0 0 0 2024-04-06T14:17:02.560 INFO:tasks.workunit.client.0.smithi032.stdout:8/362: creat d3/d8/d12/d2f/d46/f88 x:0 0 0 2024-04-06T14:17:02.560 INFO:tasks.workunit.client.0.smithi032.stdout:7/419: mkdir d16/d92 0 2024-04-06T14:17:02.561 INFO:tasks.workunit.client.1.smithi086.stdout:2/142: creat d0/d20/d27/d2b/d35/d36/f3a x:0 0 0 2024-04-06T14:17:02.561 INFO:tasks.workunit.client.0.smithi032.stdout:3/409: symlink d3/d16/laa 0 2024-04-06T14:17:02.562 INFO:tasks.workunit.client.1.smithi086.stdout:6/171: unlink d6/d9/db/f38 0 2024-04-06T14:17:02.562 INFO:tasks.workunit.client.0.smithi032.stdout:6/393: mknod d4/d5/d8/d1f/c7e 0 2024-04-06T14:17:02.563 INFO:tasks.workunit.client.1.smithi086.stdout:9/187: dwrite d5/d13/f2b [0,4194304] 0 2024-04-06T14:17:02.564 INFO:tasks.workunit.client.0.smithi032.stdout:5/397: fdatasync d16/d18/d33/d45/d52/f4e 0 2024-04-06T14:17:02.564 INFO:tasks.workunit.client.0.smithi032.stdout:6/394: read - d4/d5/d8/d1f/d51/f7b zero size 2024-04-06T14:17:02.566 INFO:tasks.workunit.client.0.smithi032.stdout:9/434: getdents d2/d15/d1e/d41/d6c/d88 0 2024-04-06T14:17:02.567 INFO:tasks.workunit.client.0.smithi032.stdout:7/420: dread d16/d20/d48/d6d/f79 [0,4194304] 0 2024-04-06T14:17:02.567 INFO:tasks.workunit.client.0.smithi032.stdout:6/395: write d4/d5/d28/f5b [122261,110675] 0 2024-04-06T14:17:02.570 INFO:tasks.workunit.client.1.smithi086.stdout:8/278: dread d0/db/d32/f3e [0,4194304] 0 2024-04-06T14:17:02.571 INFO:tasks.workunit.client.1.smithi086.stdout:4/129: dread - d0/d15/d20/f27 zero size 2024-04-06T14:17:02.571 INFO:tasks.workunit.client.0.smithi032.stdout:4/442: rename d0/df/d23/d31/d50/d5e/d89 to d0/df/d23/d31/d50/d57/d5b/d9c 0 2024-04-06T14:17:02.572 INFO:tasks.workunit.client.0.smithi032.stdout:3/410: mknod d3/d16/d63/cab 0 2024-04-06T14:17:02.578 INFO:tasks.workunit.client.1.smithi086.stdout:9/188: read d5/d13/d1a/f24 [1895724,102417] 0 2024-04-06T14:17:02.579 INFO:tasks.workunit.client.1.smithi086.stdout:6/172: chown d6/d20/l32 47042456 1 2024-04-06T14:17:02.579 INFO:tasks.workunit.client.1.smithi086.stdout:3/220: dwrite f4 [0,4194304] 0 2024-04-06T14:17:02.579 INFO:tasks.workunit.client.1.smithi086.stdout:8/279: creat d0/db/f5e x:0 0 0 2024-04-06T14:17:02.579 INFO:tasks.workunit.client.0.smithi032.stdout:5/398: mkdir d16/d18/d33/d35/d88 0 2024-04-06T14:17:02.580 INFO:tasks.workunit.client.0.smithi032.stdout:5/399: readlink d16/d18/l5a 0 2024-04-06T14:17:02.580 INFO:tasks.workunit.client.0.smithi032.stdout:7/421: creat d16/d20/d6e/f93 x:0 0 0 2024-04-06T14:17:02.581 INFO:tasks.workunit.client.0.smithi032.stdout:4/443: dwrite d0/df/d23/d31/d50/d57/f64 [0,4194304] 0 2024-04-06T14:17:02.581 INFO:tasks.workunit.client.1.smithi086.stdout:3/221: dread - d5/f34 zero size 2024-04-06T14:17:02.581 INFO:tasks.workunit.client.1.smithi086.stdout:4/130: mkdir d0/d15/d2b 0 2024-04-06T14:17:02.581 INFO:tasks.workunit.client.1.smithi086.stdout:6/173: fdatasync d6/d13/d1a/d22/f35 0 2024-04-06T14:17:02.582 INFO:tasks.workunit.client.0.smithi032.stdout:5/400: fdatasync d16/f24 0 2024-04-06T14:17:02.583 INFO:tasks.workunit.client.0.smithi032.stdout:6/396: symlink d4/d5/d18/d23/l7f 0 2024-04-06T14:17:02.583 INFO:tasks.workunit.client.0.smithi032.stdout:7/422: fsync d16/d20/d26/d2d/f4f 0 2024-04-06T14:17:02.584 INFO:tasks.workunit.client.1.smithi086.stdout:9/189: rmdir d5/d13 39 2024-04-06T14:17:02.584 INFO:tasks.workunit.client.0.smithi032.stdout:5/401: write d16/d58/f5f [707128,21923] 0 2024-04-06T14:17:02.587 INFO:tasks.workunit.client.1.smithi086.stdout:5/205: getdents d0/d16/d18 0 2024-04-06T14:17:02.587 INFO:tasks.workunit.client.0.smithi032.stdout:5/402: chown d16/d58/d54/d74 129 1 2024-04-06T14:17:02.589 INFO:tasks.workunit.client.0.smithi032.stdout:9/435: rename d2/d15/d1e/c28 to d2/d15/d55/d5f/d71/d81/d4a/d73/d7c/c8c 0 2024-04-06T14:17:02.590 INFO:tasks.workunit.client.1.smithi086.stdout:6/174: creat d6/d13/d1a/d22/f3f x:0 0 0 2024-04-06T14:17:02.591 INFO:tasks.workunit.client.1.smithi086.stdout:5/206: dread - d0/d16/d18/d35/f3a zero size 2024-04-06T14:17:02.594 INFO:tasks.workunit.client.1.smithi086.stdout:4/131: mkdir d0/d2c 0 2024-04-06T14:17:02.594 INFO:tasks.workunit.client.1.smithi086.stdout:5/207: chown d0/d16/d18/d21/d23/d28/l38 1324798023 1 2024-04-06T14:17:02.595 INFO:tasks.workunit.client.0.smithi032.stdout:0/313: dread d3/da/d25/d36/f39 [0,4194304] 0 2024-04-06T14:17:02.595 INFO:tasks.workunit.client.0.smithi032.stdout:6/397: dwrite d4/d5/d8/f72 [0,4194304] 0 2024-04-06T14:17:02.595 INFO:tasks.workunit.client.1.smithi086.stdout:9/190: mkdir d5/d13/d47 0 2024-04-06T14:17:02.596 INFO:tasks.workunit.client.0.smithi032.stdout:0/314: read - d3/f55 zero size 2024-04-06T14:17:02.597 INFO:tasks.workunit.client.1.smithi086.stdout:5/208: fdatasync d0/f4 0 2024-04-06T14:17:02.598 INFO:tasks.workunit.client.1.smithi086.stdout:4/132: creat d0/de/d21/d24/f2d x:0 0 0 2024-04-06T14:17:02.598 INFO:tasks.workunit.client.1.smithi086.stdout:9/191: write d5/d13/f39 [942594,83344] 0 2024-04-06T14:17:02.600 INFO:tasks.workunit.client.1.smithi086.stdout:8/280: dwrite d0/d13/f4b [0,4194304] 0 2024-04-06T14:17:02.600 INFO:tasks.workunit.client.0.smithi032.stdout:4/444: creat d0/df/d23/d31/d50/d57/d5b/d9c/f9d x:0 0 0 2024-04-06T14:17:02.601 INFO:tasks.workunit.client.1.smithi086.stdout:5/209: chown d0/d16/d18/l10 145 1 2024-04-06T14:17:02.602 INFO:tasks.workunit.client.1.smithi086.stdout:8/281: chown d0/d3b/f3d 5237 1 2024-04-06T14:17:02.602 INFO:tasks.workunit.client.0.smithi032.stdout:6/398: mkdir d4/d5/d14/d80 0 2024-04-06T14:17:02.605 INFO:tasks.workunit.client.0.smithi032.stdout:5/403: getdents d16/d18/d33/d45/d52/d39/d86 0 2024-04-06T14:17:02.605 INFO:tasks.workunit.client.0.smithi032.stdout:7/423: symlink d16/d20/d26/l94 0 2024-04-06T14:17:02.605 INFO:tasks.workunit.client.0.smithi032.stdout:5/404: readlink d16/l6c 0 2024-04-06T14:17:02.606 INFO:tasks.workunit.client.0.smithi032.stdout:7/424: fdatasync d16/d20/d26/d2d/f3f 0 2024-04-06T14:17:02.608 INFO:tasks.workunit.client.1.smithi086.stdout:8/282: mkdir d0/d8/d16/d52/d5f 0 2024-04-06T14:17:02.610 INFO:tasks.workunit.client.1.smithi086.stdout:3/222: fsync f4 0 2024-04-06T14:17:02.611 INFO:tasks.workunit.client.0.smithi032.stdout:6/399: unlink d4/d5/d34/l47 0 2024-04-06T14:17:02.612 INFO:tasks.workunit.client.0.smithi032.stdout:5/405: creat d16/d18/d33/d45/d5e/f89 x:0 0 0 2024-04-06T14:17:02.613 INFO:tasks.workunit.client.1.smithi086.stdout:5/210: symlink d0/d16/d18/d21/d23/l40 0 2024-04-06T14:17:02.613 INFO:tasks.workunit.client.1.smithi086.stdout:3/223: fdatasync d5/d33/f35 0 2024-04-06T14:17:02.614 INFO:tasks.workunit.client.1.smithi086.stdout:8/283: write d0/d3b/f3d [1614967,86401] 0 2024-04-06T14:17:02.615 INFO:tasks.workunit.client.0.smithi032.stdout:6/400: mknod d4/d5/d8/d2f/c81 0 2024-04-06T14:17:02.616 INFO:tasks.workunit.client.0.smithi032.stdout:6/401: write d4/d5/d28/f41 [2185990,116064] 0 2024-04-06T14:17:02.617 INFO:tasks.workunit.client.0.smithi032.stdout:7/425: mkdir d16/d20/d48/d8d/d95 0 2024-04-06T14:17:02.618 INFO:tasks.workunit.client.0.smithi032.stdout:7/426: chown d16/d20/d54/f76 39569920 1 2024-04-06T14:17:02.620 INFO:tasks.workunit.client.1.smithi086.stdout:8/284: dread - d0/d8/d22/d43/f47 zero size 2024-04-06T14:17:02.620 INFO:tasks.workunit.client.1.smithi086.stdout:4/133: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:17:02.621 INFO:tasks.workunit.client.1.smithi086.stdout:2/143: dwrite d0/f6 [0,4194304] 0 2024-04-06T14:17:02.621 INFO:tasks.workunit.client.0.smithi032.stdout:3/411: write d3/d65/f74 [1450077,10377] 0 2024-04-06T14:17:02.621 INFO:tasks.workunit.client.1.smithi086.stdout:3/224: rmdir d5/d2b 39 2024-04-06T14:17:02.622 INFO:tasks.workunit.client.0.smithi032.stdout:3/412: read - d3/d65/f93 zero size 2024-04-06T14:17:02.622 INFO:tasks.workunit.client.1.smithi086.stdout:9/192: dwrite d5/d18/f26 [0,4194304] 0 2024-04-06T14:17:02.623 INFO:tasks.workunit.client.0.smithi032.stdout:3/413: write d3/d16/f68 [5094505,7082] 0 2024-04-06T14:17:02.624 INFO:tasks.workunit.client.0.smithi032.stdout:6/402: mkdir d4/d5/d18/d23/d82 0 2024-04-06T14:17:02.624 INFO:tasks.workunit.client.0.smithi032.stdout:4/445: link d0/df/d1f/l56 d0/df/d23/d31/d50/d57/d5b/l9e 0 2024-04-06T14:17:02.626 INFO:tasks.workunit.client.0.smithi032.stdout:4/446: write d0/df/d2b/d46/f6e [969669,82696] 0 2024-04-06T14:17:02.626 INFO:tasks.workunit.client.0.smithi032.stdout:7/427: rmdir d16/d20/d26/d2d/d30/d3e 39 2024-04-06T14:17:02.628 INFO:tasks.workunit.client.1.smithi086.stdout:0/201: dread d0/d1/d4/f8 [0,4194304] 0 2024-04-06T14:17:02.629 INFO:tasks.workunit.client.0.smithi032.stdout:4/447: write d0/df/d2b/d46/d58/f7e [842260,38817] 0 2024-04-06T14:17:02.629 INFO:tasks.workunit.client.1.smithi086.stdout:3/225: dread - d5/df/f2c zero size 2024-04-06T14:17:02.632 INFO:tasks.workunit.client.1.smithi086.stdout:4/134: unlink d0/f1d 0 2024-04-06T14:17:02.633 INFO:tasks.workunit.client.1.smithi086.stdout:9/193: read - d5/d18/f1d zero size 2024-04-06T14:17:02.634 INFO:tasks.workunit.client.0.smithi032.stdout:5/406: link d16/d18/d33/d45/d52/f53 d16/d18/d33/f8a 0 2024-04-06T14:17:02.636 INFO:tasks.workunit.client.1.smithi086.stdout:3/226: write d5/fe [5191167,31102] 0 2024-04-06T14:17:02.637 INFO:tasks.workunit.client.1.smithi086.stdout:6/175: dwrite d6/f2b [4194304,4194304] 0 2024-04-06T14:17:02.637 INFO:tasks.workunit.client.0.smithi032.stdout:6/403: mkdir d4/d5/d83 0 2024-04-06T14:17:02.638 INFO:tasks.workunit.client.1.smithi086.stdout:2/144: creat d0/d20/d27/d2b/d35/d36/f3b x:0 0 0 2024-04-06T14:17:02.642 INFO:tasks.workunit.client.1.smithi086.stdout:2/145: readlink d0/l5 0 2024-04-06T14:17:02.642 INFO:tasks.workunit.client.1.smithi086.stdout:0/202: truncate d0/d1/d4/d7/f27 1030086 0 2024-04-06T14:17:02.642 INFO:tasks.workunit.client.1.smithi086.stdout:4/135: unlink d0/fd 0 2024-04-06T14:17:02.642 INFO:tasks.workunit.client.0.smithi032.stdout:4/448: creat d0/df/d23/d31/d50/d57/d5b/d9c/f9f x:0 0 0 2024-04-06T14:17:02.643 INFO:tasks.workunit.client.1.smithi086.stdout:5/211: dread d0/d16/d18/d21/d23/f1f [0,4194304] 0 2024-04-06T14:17:02.643 INFO:tasks.workunit.client.1.smithi086.stdout:9/194: creat d5/d18/f48 x:0 0 0 2024-04-06T14:17:02.645 INFO:tasks.workunit.client.1.smithi086.stdout:6/176: chown d6/d9/f10 32856373 1 2024-04-06T14:17:02.646 INFO:tasks.workunit.client.1.smithi086.stdout:2/146: chown d0 120 1 2024-04-06T14:17:02.647 INFO:tasks.workunit.client.0.smithi032.stdout:4/449: mkdir d0/df/d23/d31/d50/da0 0 2024-04-06T14:17:02.648 INFO:tasks.workunit.client.1.smithi086.stdout:8/285: link d0/d8/l14 d0/db/d32/l60 0 2024-04-06T14:17:02.649 INFO:tasks.workunit.client.1.smithi086.stdout:2/147: chown d0/df/l10 49600150 1 2024-04-06T14:17:02.650 INFO:tasks.workunit.client.1.smithi086.stdout:0/203: symlink d0/d1/d1b/d46/l4d 0 2024-04-06T14:17:02.650 INFO:tasks.workunit.client.1.smithi086.stdout:8/286: rmdir d0/d8/d16/d39 39 2024-04-06T14:17:02.651 INFO:tasks.workunit.client.1.smithi086.stdout:2/148: write d0/d20/f2e [360368,36603] 0 2024-04-06T14:17:02.652 INFO:tasks.workunit.client.0.smithi032.stdout:9/436: dwrite d2/d15/d1e/d65/d83/d7e/f1d [0,4194304] 0 2024-04-06T14:17:02.652 INFO:tasks.workunit.client.1.smithi086.stdout:8/287: write d0/f2 [2730809,67396] 0 2024-04-06T14:17:02.653 INFO:tasks.workunit.client.1.smithi086.stdout:3/227: rename d5/f9 to d5/d26/f3c 0 2024-04-06T14:17:02.656 INFO:tasks.workunit.client.0.smithi032.stdout:9/437: fsync d2/d15/d1e/d65/d83/f32 0 2024-04-06T14:17:02.658 INFO:tasks.workunit.client.1.smithi086.stdout:6/177: creat d6/f40 x:0 0 0 2024-04-06T14:17:02.659 INFO:tasks.workunit.client.0.smithi032.stdout:4/450: dwrite d0/df/d23/d31/d50/d57/d8a/f65 [0,4194304] 0 2024-04-06T14:17:02.664 INFO:tasks.workunit.client.0.smithi032.stdout:4/451: write d0/df/d23/d31/d50/d57/d5b/d9c/f86 [3137463,102218] 0 2024-04-06T14:17:02.665 INFO:tasks.workunit.client.1.smithi086.stdout:8/288: unlink d0/d8/d16/d39/f20 0 2024-04-06T14:17:02.668 INFO:tasks.workunit.client.1.smithi086.stdout:6/178: mknod d6/d13/d1a/c41 0 2024-04-06T14:17:02.669 INFO:tasks.workunit.client.1.smithi086.stdout:3/228: creat d5/d26/f3d x:0 0 0 2024-04-06T14:17:02.670 INFO:tasks.workunit.client.1.smithi086.stdout:2/149: getdents d0/d20/d27 0 2024-04-06T14:17:02.670 INFO:tasks.workunit.client.0.smithi032.stdout:9/438: dread d2/d15/d55/d5f/d71/d81/f4c [0,4194304] 0 2024-04-06T14:17:02.671 INFO:tasks.workunit.client.1.smithi086.stdout:3/229: write d5/df/f25 [1602885,70290] 0 2024-04-06T14:17:02.673 INFO:tasks.workunit.client.1.smithi086.stdout:3/230: write d5/f20 [463157,8628] 0 2024-04-06T14:17:02.673 INFO:tasks.workunit.client.1.smithi086.stdout:8/289: dread d0/d8/d22/d43/d4d/f53 [0,4194304] 0 2024-04-06T14:17:02.674 INFO:tasks.workunit.client.1.smithi086.stdout:3/231: readlink d5/df/l1e 0 2024-04-06T14:17:02.675 INFO:tasks.workunit.client.1.smithi086.stdout:3/232: write d5/df/f25 [3598405,2931] 0 2024-04-06T14:17:02.676 INFO:tasks.workunit.client.1.smithi086.stdout:3/233: fdatasync d5/d33/f35 0 2024-04-06T14:17:02.676 INFO:tasks.workunit.client.1.smithi086.stdout:3/234: chown d5/d33 97377931 1 2024-04-06T14:17:02.677 INFO:tasks.workunit.client.1.smithi086.stdout:6/179: rmdir d6/d9 39 2024-04-06T14:17:02.682 INFO:tasks.workunit.client.1.smithi086.stdout:0/204: write d0/f17 [1243078,32544] 0 2024-04-06T14:17:02.683 INFO:tasks.workunit.client.0.smithi032.stdout:2/403: sync 2024-04-06T14:17:02.685 INFO:tasks.workunit.client.1.smithi086.stdout:6/180: mkdir d6/d13/d42 0 2024-04-06T14:17:02.685 INFO:tasks.workunit.client.1.smithi086.stdout:8/290: rmdir d0/d13/d58 0 2024-04-06T14:17:02.686 INFO:tasks.workunit.client.1.smithi086.stdout:2/150: link d0/df/l32 d0/l3c 0 2024-04-06T14:17:02.686 INFO:tasks.workunit.client.1.smithi086.stdout:8/291: chown d0/db/f12 1149768 1 2024-04-06T14:17:02.690 INFO:tasks.workunit.client.1.smithi086.stdout:0/205: rename d0/d23 to d0/d15/d4e 0 2024-04-06T14:17:02.691 INFO:tasks.workunit.client.1.smithi086.stdout:3/235: rename d5/f34 to d5/f3e 0 2024-04-06T14:17:02.691 INFO:tasks.workunit.client.1.smithi086.stdout:3/236: chown d5/df/f25 55 1 2024-04-06T14:17:02.692 INFO:tasks.workunit.client.1.smithi086.stdout:2/151: symlink d0/d20/l3d 0 2024-04-06T14:17:02.693 INFO:tasks.workunit.client.1.smithi086.stdout:8/292: creat d0/d8/d22/f61 x:0 0 0 2024-04-06T14:17:02.693 INFO:tasks.workunit.client.1.smithi086.stdout:0/206: creat d0/d15/f4f x:0 0 0 2024-04-06T14:17:02.694 INFO:tasks.workunit.client.1.smithi086.stdout:0/207: chown d0/d1/d4 259008228 1 2024-04-06T14:17:02.695 INFO:tasks.workunit.client.1.smithi086.stdout:3/237: mkdir d5/d33/d3f 0 2024-04-06T14:17:02.696 INFO:tasks.workunit.client.1.smithi086.stdout:2/152: rename d0/f2c to d0/d20/d27/d2b/d35/d36/f3e 0 2024-04-06T14:17:02.697 INFO:tasks.workunit.client.1.smithi086.stdout:8/293: mkdir d0/d8/d22/d43/d4d/d62 0 2024-04-06T14:17:02.697 INFO:tasks.workunit.client.1.smithi086.stdout:2/153: dread - d0/d20/f25 zero size 2024-04-06T14:17:02.700 INFO:tasks.workunit.client.1.smithi086.stdout:2/154: symlink d0/l3f 0 2024-04-06T14:17:02.700 INFO:tasks.workunit.client.1.smithi086.stdout:6/181: dwrite d6/d9/db/f2f [0,4194304] 0 2024-04-06T14:17:02.701 INFO:tasks.workunit.client.1.smithi086.stdout:8/294: truncate d0/db/d32/f5d 435048 0 2024-04-06T14:17:02.702 INFO:tasks.workunit.client.1.smithi086.stdout:6/182: chown d6/d9/db/f16 3 1 2024-04-06T14:17:02.702 INFO:tasks.workunit.client.1.smithi086.stdout:8/295: read - d0/d8/d16/d52/f55 zero size 2024-04-06T14:17:02.702 INFO:tasks.workunit.client.1.smithi086.stdout:0/208: rename d0/l1c to d0/d1/d2/d1a/l50 0 2024-04-06T14:17:02.703 INFO:tasks.workunit.client.1.smithi086.stdout:2/155: fsync d0/fe 0 2024-04-06T14:17:02.705 INFO:tasks.workunit.client.1.smithi086.stdout:8/296: unlink d0/d8/d16/d39/f3a 0 2024-04-06T14:17:02.708 INFO:tasks.workunit.client.1.smithi086.stdout:0/209: mkdir d0/d1/d2/d1a/d51 0 2024-04-06T14:17:02.710 INFO:tasks.workunit.client.0.smithi032.stdout:3/414: write d3/d16/f72 [890990,112815] 0 2024-04-06T14:17:02.711 INFO:tasks.workunit.client.0.smithi032.stdout:7/428: truncate fa 11243 0 2024-04-06T14:17:02.712 INFO:tasks.workunit.client.1.smithi086.stdout:6/183: mkdir d6/d9/d43 0 2024-04-06T14:17:02.712 INFO:tasks.workunit.client.1.smithi086.stdout:2/156: dwrite d0/d20/d27/d2b/d35/d36/f3b [0,4194304] 0 2024-04-06T14:17:02.713 INFO:tasks.workunit.client.1.smithi086.stdout:2/157: stat d0/d20/d27/l39 0 2024-04-06T14:17:02.714 INFO:tasks.workunit.client.1.smithi086.stdout:8/297: symlink d0/d3b/l63 0 2024-04-06T14:17:02.716 INFO:tasks.workunit.client.0.smithi032.stdout:9/439: creat d2/d15/d55/d5f/d71/d81/d4a/f8d x:0 0 0 2024-04-06T14:17:02.717 INFO:tasks.workunit.client.0.smithi032.stdout:6/404: link d4/d5/d34/c43 d4/d5/d14/d80/c84 0 2024-04-06T14:17:02.718 INFO:tasks.workunit.client.1.smithi086.stdout:8/298: chown d0/db/l2b 1066992 1 2024-04-06T14:17:02.718 INFO:tasks.workunit.client.0.smithi032.stdout:6/405: fdatasync d4/d5/f4c 0 2024-04-06T14:17:02.719 INFO:tasks.workunit.client.1.smithi086.stdout:0/210: mkdir d0/d1/d1b/d46/d52 0 2024-04-06T14:17:02.720 INFO:tasks.workunit.client.1.smithi086.stdout:6/184: mknod d6/d13/d1a/c44 0 2024-04-06T14:17:02.721 INFO:tasks.workunit.client.0.smithi032.stdout:9/440: mkdir d2/d15/d55/d5f/d71/d81/d4a/d8e 0 2024-04-06T14:17:02.722 INFO:tasks.workunit.client.1.smithi086.stdout:2/158: rmdir d0/d20/d27/d2b/d35/d36 39 2024-04-06T14:17:02.727 INFO:tasks.workunit.client.0.smithi032.stdout:7/429: rename d16/d20/d48/d61 to d16/d20/d26/d2d/d30/d3e/d96 0 2024-04-06T14:17:02.728 INFO:tasks.workunit.client.1.smithi086.stdout:8/299: creat d0/d8/d22/d43/f64 x:0 0 0 2024-04-06T14:17:02.729 INFO:tasks.workunit.client.0.smithi032.stdout:2/404: dwrite d0/d30/f4d [0,4194304] 0 2024-04-06T14:17:02.729 INFO:tasks.workunit.client.1.smithi086.stdout:0/211: unlink d0/d15/d26/c2a 0 2024-04-06T14:17:02.730 INFO:tasks.workunit.client.0.smithi032.stdout:9/441: chown d2/d15/d1e/d65/d83/d31/d39/f52 2 1 2024-04-06T14:17:02.731 INFO:tasks.workunit.client.0.smithi032.stdout:4/452: getdents d0/df/d23/d31 0 2024-04-06T14:17:02.733 INFO:tasks.workunit.client.0.smithi032.stdout:5/407: dwrite d16/d18/f1e [0,4194304] 0 2024-04-06T14:17:02.734 INFO:tasks.workunit.client.0.smithi032.stdout:2/405: unlink d0/l2f 0 2024-04-06T14:17:02.735 INFO:tasks.workunit.client.0.smithi032.stdout:9/442: write d2/d15/d1e/d65/d83/d7e/f1b [5156878,9421] 0 2024-04-06T14:17:02.736 INFO:tasks.workunit.client.0.smithi032.stdout:7/430: write d16/f4b [2629711,11741] 0 2024-04-06T14:17:02.739 INFO:tasks.workunit.client.0.smithi032.stdout:6/406: link d4/d5/d8/d1f/f63 d4/d5/d28/d54/f85 0 2024-04-06T14:17:02.742 INFO:tasks.workunit.client.0.smithi032.stdout:5/408: rename f10 to d16/d18/d33/f8b 0 2024-04-06T14:17:02.743 INFO:tasks.workunit.client.0.smithi032.stdout:6/407: write d4/d5/d18/d23/f50 [1634383,74327] 0 2024-04-06T14:17:02.743 INFO:tasks.workunit.client.0.smithi032.stdout:2/406: creat d0/d5a/f8e x:0 0 0 2024-04-06T14:17:02.744 INFO:tasks.workunit.client.1.smithi086.stdout:8/300: mkdir d0/d8/d22/d65 0 2024-04-06T14:17:02.747 INFO:tasks.workunit.client.1.smithi086.stdout:2/159: creat d0/d20/d27/d2b/d35/d36/f40 x:0 0 0 2024-04-06T14:17:02.747 INFO:tasks.workunit.client.0.smithi032.stdout:7/431: mkdir d16/d20/d26/d2d/d30/d97 0 2024-04-06T14:17:02.748 INFO:tasks.workunit.client.0.smithi032.stdout:5/409: read d16/d18/f21 [303680,90401] 0 2024-04-06T14:17:02.750 INFO:tasks.workunit.client.0.smithi032.stdout:7/432: chown d16/d20/d26/d70 52215511 1 2024-04-06T14:17:02.751 INFO:tasks.workunit.client.0.smithi032.stdout:9/443: dwrite d2/d15/d55/d5f/d71/d81/d77/f78 [0,4194304] 0 2024-04-06T14:17:02.752 INFO:tasks.workunit.client.1.smithi086.stdout:2/160: dread d0/f1e [0,4194304] 0 2024-04-06T14:17:02.753 INFO:tasks.workunit.client.0.smithi032.stdout:2/407: link d0/d5a/f8e d0/d30/d39/f8f 0 2024-04-06T14:17:02.754 INFO:tasks.workunit.client.1.smithi086.stdout:3/238: rename d5/df/l27 to d5/d33/l40 0 2024-04-06T14:17:02.755 INFO:tasks.workunit.client.0.smithi032.stdout:7/433: unlink d16/d20/f23 0 2024-04-06T14:17:02.756 INFO:tasks.workunit.client.0.smithi032.stdout:5/410: creat d16/d18/d33/d45/d87/f8c x:0 0 0 2024-04-06T14:17:02.756 INFO:tasks.workunit.client.1.smithi086.stdout:7/153: sync 2024-04-06T14:17:02.758 INFO:tasks.workunit.client.0.smithi032.stdout:5/411: chown d16/d18/d33/d45/d52 48935 1 2024-04-06T14:17:02.758 INFO:tasks.workunit.client.0.smithi032.stdout:9/444: mkdir d2/d15/d55/d5f/d71/d81/d4a/d8f 0 2024-04-06T14:17:02.760 INFO:tasks.workunit.client.0.smithi032.stdout:6/408: dwrite d4/d5/d14/f1a [0,4194304] 0 2024-04-06T14:17:02.761 INFO:tasks.workunit.client.1.smithi086.stdout:6/185: getdents d6/d13 0 2024-04-06T14:17:02.761 INFO:tasks.workunit.client.1.smithi086.stdout:7/154: stat d7/d1d 0 2024-04-06T14:17:02.762 INFO:tasks.workunit.client.0.smithi032.stdout:2/408: creat d0/d30/d39/d59/d67/d74/d78/d57/f90 x:0 0 0 2024-04-06T14:17:02.762 INFO:tasks.workunit.client.1.smithi086.stdout:8/301: truncate d0/db/f5b 4788618 0 2024-04-06T14:17:02.763 INFO:tasks.workunit.client.1.smithi086.stdout:6/186: chown d6/d9/d43 527 1 2024-04-06T14:17:02.764 INFO:tasks.workunit.client.1.smithi086.stdout:0/212: link d0/d15/d4e/d48/c3d d0/d1/c53 0 2024-04-06T14:17:02.765 INFO:tasks.workunit.client.0.smithi032.stdout:5/412: mkdir d16/d18/d33/d45/d52/d8d 0 2024-04-06T14:17:02.766 INFO:tasks.workunit.client.1.smithi086.stdout:7/155: write d7/d1d/f13 [59724,88715] 0 2024-04-06T14:17:02.766 INFO:tasks.workunit.client.0.smithi032.stdout:7/434: symlink d16/l98 0 2024-04-06T14:17:02.768 INFO:tasks.workunit.client.0.smithi032.stdout:9/445: fsync d2/d15/d1e/d41/f63 0 2024-04-06T14:17:02.768 INFO:tasks.workunit.client.1.smithi086.stdout:2/161: getdents d0/d20 0 2024-04-06T14:17:02.768 INFO:tasks.workunit.client.0.smithi032.stdout:4/453: truncate d0/df/d1f/d2e/f41 806863 0 2024-04-06T14:17:02.769 INFO:tasks.workunit.client.1.smithi086.stdout:2/162: dread - d0/df/f1d zero size 2024-04-06T14:17:02.769 INFO:tasks.workunit.client.1.smithi086.stdout:0/213: stat d0/d1/d2/d1a/l1f 0 2024-04-06T14:17:02.771 INFO:tasks.workunit.client.1.smithi086.stdout:6/187: mkdir d6/d9/d36/d45 0 2024-04-06T14:17:02.771 INFO:tasks.workunit.client.0.smithi032.stdout:6/409: mkdir d4/d5/d8/d4b/d86 0 2024-04-06T14:17:02.772 INFO:tasks.workunit.client.1.smithi086.stdout:3/239: dwrite d5/df/f1c [4194304,4194304] 0 2024-04-06T14:17:02.772 INFO:tasks.workunit.client.1.smithi086.stdout:2/163: chown d0/d20/d27/d37 99 1 2024-04-06T14:17:02.773 INFO:tasks.workunit.client.0.smithi032.stdout:2/409: read d0/d5a/f1c [151777,21813] 0 2024-04-06T14:17:02.773 INFO:tasks.workunit.client.1.smithi086.stdout:7/156: mknod d7/d10/c34 0 2024-04-06T14:17:02.774 INFO:tasks.workunit.client.0.smithi032.stdout:5/413: rename d16/d58/d54/l67 to d16/d18/d33/d45/d52/d39/l8e 0 2024-04-06T14:17:02.777 INFO:tasks.workunit.client.0.smithi032.stdout:7/435: write d16/d20/d26/f39 [589050,122965] 0 2024-04-06T14:17:02.779 INFO:tasks.workunit.client.1.smithi086.stdout:3/240: dread d5/df/f2f [0,4194304] 0 2024-04-06T14:17:02.781 INFO:tasks.workunit.client.1.smithi086.stdout:3/241: readlink d5/l38 0 2024-04-06T14:17:02.781 INFO:tasks.workunit.client.1.smithi086.stdout:8/302: dread d0/d13/f21 [0,4194304] 0 2024-04-06T14:17:02.781 INFO:tasks.workunit.client.1.smithi086.stdout:6/188: creat d6/d20/f46 x:0 0 0 2024-04-06T14:17:02.782 INFO:tasks.workunit.client.1.smithi086.stdout:7/157: dread d7/d10/d11/f16 [0,4194304] 0 2024-04-06T14:17:02.782 INFO:tasks.workunit.client.1.smithi086.stdout:2/164: symlink d0/d20/d27/d2b/d35/d36/l41 0 2024-04-06T14:17:02.783 INFO:tasks.workunit.client.0.smithi032.stdout:9/446: rename d2/d15/d1e/d41/c5c to d2/d15/d55/d5f/d71/d81/c90 0 2024-04-06T14:17:02.783 INFO:tasks.workunit.client.1.smithi086.stdout:3/242: fdatasync d5/df/f25 0 2024-04-06T14:17:02.784 INFO:tasks.workunit.client.0.smithi032.stdout:5/414: dwrite d16/d18/f30 [0,4194304] 0 2024-04-06T14:17:02.784 INFO:tasks.workunit.client.1.smithi086.stdout:6/189: truncate d6/d13/f26 1591438 0 2024-04-06T14:17:02.785 INFO:tasks.workunit.client.1.smithi086.stdout:2/165: truncate d0/df/f1a 4424785 0 2024-04-06T14:17:02.785 INFO:tasks.workunit.client.0.smithi032.stdout:4/454: chown d0/df/d1f/d5a/c69 13462840 1 2024-04-06T14:17:02.786 INFO:tasks.workunit.client.1.smithi086.stdout:2/166: dread - d0/f26 zero size 2024-04-06T14:17:02.787 INFO:tasks.workunit.client.1.smithi086.stdout:7/158: creat d7/d10/d30/f35 x:0 0 0 2024-04-06T14:17:02.788 INFO:tasks.workunit.client.0.smithi032.stdout:7/436: mknod d16/d20/d48/d71/c99 0 2024-04-06T14:17:02.788 INFO:tasks.workunit.client.1.smithi086.stdout:6/190: mknod d6/d20/c47 0 2024-04-06T14:17:02.788 INFO:tasks.workunit.client.0.smithi032.stdout:4/455: chown d0/d82/l7f 421767706 1 2024-04-06T14:17:02.789 INFO:tasks.workunit.client.1.smithi086.stdout:3/243: mknod d5/d2b/c41 0 2024-04-06T14:17:02.789 INFO:tasks.workunit.client.1.smithi086.stdout:3/244: dread - d5/df/f2c zero size 2024-04-06T14:17:02.790 INFO:tasks.workunit.client.1.smithi086.stdout:8/303: creat d0/d8/d16/d39/f66 x:0 0 0 2024-04-06T14:17:02.790 INFO:tasks.workunit.client.0.smithi032.stdout:6/410: rename d4/d5/fb to d4/d5/d19/d25/f87 0 2024-04-06T14:17:02.793 INFO:tasks.workunit.client.1.smithi086.stdout:2/167: creat d0/d20/d27/d37/f42 x:0 0 0 2024-04-06T14:17:02.793 INFO:tasks.workunit.client.1.smithi086.stdout:6/191: mknod d6/d9/c48 0 2024-04-06T14:17:02.795 INFO:tasks.workunit.client.1.smithi086.stdout:3/245: mkdir d5/d42 0 2024-04-06T14:17:02.795 INFO:tasks.workunit.client.0.smithi032.stdout:9/447: truncate d2/d15/d1e/d65/d83/d7e/f12 8440540 0 2024-04-06T14:17:02.797 INFO:tasks.workunit.client.1.smithi086.stdout:2/168: rename d0/df/f1d to d0/d20/d27/f43 0 2024-04-06T14:17:02.800 INFO:tasks.workunit.client.0.smithi032.stdout:9/448: write d2/d15/d1e/d65/d83/d31/d39/f58 [808111,29959] 0 2024-04-06T14:17:02.800 INFO:tasks.workunit.client.0.smithi032.stdout:4/456: dwrite d0/df/d1f/d42/d45/f53 [0,4194304] 0 2024-04-06T14:17:02.800 INFO:tasks.workunit.client.1.smithi086.stdout:2/169: fdatasync d0/d20/d27/d37/f42 0 2024-04-06T14:17:02.802 INFO:tasks.workunit.client.1.smithi086.stdout:7/159: dwrite d7/d10/d11/f16 [0,4194304] 0 2024-04-06T14:17:02.802 INFO:tasks.workunit.client.0.smithi032.stdout:9/449: read d2/d15/d1e/d65/d83/d31/d39/f5a [113784,45420] 0 2024-04-06T14:17:02.803 INFO:tasks.workunit.client.1.smithi086.stdout:6/192: link d6/d13/d1a/c1f d6/d13/d1a/c49 0 2024-04-06T14:17:02.804 INFO:tasks.workunit.client.1.smithi086.stdout:6/193: fsync d6/d9/db/f19 0 2024-04-06T14:17:02.806 INFO:tasks.workunit.client.1.smithi086.stdout:7/160: mkdir d7/d10/d11/d1c/d36 0 2024-04-06T14:17:02.811 INFO:tasks.workunit.client.1.smithi086.stdout:7/161: symlink d7/d10/d11/d1c/l37 0 2024-04-06T14:17:02.813 INFO:tasks.workunit.client.1.smithi086.stdout:2/170: dwrite d0/d20/d27/d2b/d35/d36/f3b [0,4194304] 0 2024-04-06T14:17:02.819 INFO:tasks.workunit.client.1.smithi086.stdout:7/162: creat d7/d1d/f38 x:0 0 0 2024-04-06T14:17:02.821 INFO:tasks.workunit.client.1.smithi086.stdout:2/171: dread d0/df/f18 [0,4194304] 0 2024-04-06T14:17:02.821 INFO:tasks.workunit.client.1.smithi086.stdout:7/163: fsync d7/d1d/fc 0 2024-04-06T14:17:02.824 INFO:tasks.workunit.client.1.smithi086.stdout:2/172: fsync d0/df/f18 0 2024-04-06T14:17:02.825 INFO:tasks.workunit.client.1.smithi086.stdout:2/173: write d0/fd [251828,58451] 0 2024-04-06T14:17:02.825 INFO:tasks.workunit.client.1.smithi086.stdout:7/164: creat d7/d10/d11/d1c/d24/d2c/f39 x:0 0 0 2024-04-06T14:17:02.826 INFO:tasks.workunit.client.1.smithi086.stdout:3/246: dread d5/df/f13 [4194304,4194304] 0 2024-04-06T14:17:02.826 INFO:tasks.workunit.client.1.smithi086.stdout:8/304: dread d0/d8/d16/f41 [0,4194304] 0 2024-04-06T14:17:02.829 INFO:tasks.workunit.client.1.smithi086.stdout:3/247: mknod d5/d26/c43 0 2024-04-06T14:17:02.832 INFO:tasks.workunit.client.1.smithi086.stdout:7/165: dread d7/fb [0,4194304] 0 2024-04-06T14:17:02.833 INFO:tasks.workunit.client.1.smithi086.stdout:3/248: read d5/df/f13 [1354165,54052] 0 2024-04-06T14:17:02.838 INFO:tasks.workunit.client.1.smithi086.stdout:2/174: dwrite d0/df/f1b [0,4194304] 0 2024-04-06T14:17:02.838 INFO:tasks.workunit.client.1.smithi086.stdout:7/166: unlink d7/d10/d11/l28 0 2024-04-06T14:17:02.839 INFO:tasks.workunit.client.1.smithi086.stdout:8/305: dwrite d0/d8/d16/d52/f55 [0,4194304] 0 2024-04-06T14:17:02.841 INFO:tasks.workunit.client.1.smithi086.stdout:2/175: stat d0/df/f1a 0 2024-04-06T14:17:02.844 INFO:tasks.workunit.client.1.smithi086.stdout:2/176: mkdir d0/d20/d44 0 2024-04-06T14:17:02.850 INFO:tasks.workunit.client.1.smithi086.stdout:3/249: dwrite d5/d33/f35 [0,4194304] 0 2024-04-06T14:17:02.853 INFO:tasks.workunit.client.1.smithi086.stdout:8/306: link d0/d8/c9 d0/d8/d16/d52/d5f/c67 0 2024-04-06T14:17:02.855 INFO:tasks.workunit.client.1.smithi086.stdout:7/167: dwrite d7/d1d/f19 [0,4194304] 0 2024-04-06T14:17:02.857 INFO:tasks.workunit.client.1.smithi086.stdout:8/307: readlink d0/d8/d22/d43/l48 0 2024-04-06T14:17:02.857 INFO:tasks.workunit.client.1.smithi086.stdout:3/250: dread d5/f37 [0,4194304] 0 2024-04-06T14:17:02.858 INFO:tasks.workunit.client.1.smithi086.stdout:7/168: symlink d7/d1d/l3a 0 2024-04-06T14:17:02.859 INFO:tasks.workunit.client.1.smithi086.stdout:7/169: chown f5 3495 1 2024-04-06T14:17:02.860 INFO:tasks.workunit.client.1.smithi086.stdout:3/251: write d5/fe [2628041,47964] 0 2024-04-06T14:17:02.861 INFO:tasks.workunit.client.1.smithi086.stdout:7/170: read - d7/d10/d11/d1c/f2e zero size 2024-04-06T14:17:02.862 INFO:tasks.workunit.client.1.smithi086.stdout:3/252: fdatasync d5/d26/f3c 0 2024-04-06T14:17:02.863 INFO:tasks.workunit.client.1.smithi086.stdout:7/171: symlink d7/d1d/l3b 0 2024-04-06T14:17:02.865 INFO:tasks.workunit.client.1.smithi086.stdout:7/172: mkdir d7/d10/d11/d1c/d24/d2c/d3c 0 2024-04-06T14:17:02.868 INFO:tasks.workunit.client.1.smithi086.stdout:7/173: mknod d7/d10/d11/d1c/d24/d2c/c3d 0 2024-04-06T14:17:02.869 INFO:tasks.workunit.client.1.smithi086.stdout:7/174: creat d7/d1d/f3e x:0 0 0 2024-04-06T14:17:02.872 INFO:tasks.workunit.client.0.smithi032.stdout:5/415: mkdir d16/d58/d54/d74/d8f 0 2024-04-06T14:17:02.872 INFO:tasks.workunit.client.1.smithi086.stdout:7/175: mkdir d7/d10/d11/d1c/d24/d2c/d3f 0 2024-04-06T14:17:02.872 INFO:tasks.workunit.client.0.smithi032.stdout:5/416: chown d16/d18/d33/d45/d52/f43 1468 1 2024-04-06T14:17:02.878 INFO:tasks.workunit.client.1.smithi086.stdout:7/176: creat d7/d10/d11/d1c/d24/d2c/d3f/f40 x:0 0 0 2024-04-06T14:17:02.878 INFO:tasks.workunit.client.1.smithi086.stdout:8/308: write d0/f26 [1010632,105185] 0 2024-04-06T14:17:02.879 INFO:tasks.workunit.client.1.smithi086.stdout:7/177: chown d7/d1d 0 1 2024-04-06T14:17:02.905 INFO:tasks.workunit.client.1.smithi086.stdout:1/165: sync 2024-04-06T14:17:02.913 INFO:tasks.workunit.client.0.smithi032.stdout:2/410: dwrite d0/fb [0,4194304] 0 2024-04-06T14:17:02.919 INFO:tasks.workunit.client.0.smithi032.stdout:9/450: mknod d2/d15/d1e/c91 0 2024-04-06T14:17:02.920 INFO:tasks.workunit.client.0.smithi032.stdout:9/451: write d2/d15/d55/d5f/d71/d81/d4a/d73/d7c/f80 [122930,81583] 0 2024-04-06T14:17:02.928 INFO:tasks.workunit.client.1.smithi086.stdout:2/177: dwrite d0/df/f18 [0,4194304] 0 2024-04-06T14:17:02.928 INFO:tasks.workunit.client.1.smithi086.stdout:7/178: unlink d7/d10/d11/d1c/f22 0 2024-04-06T14:17:02.930 INFO:tasks.workunit.client.1.smithi086.stdout:1/166: write db/f16 [917047,105660] 0 2024-04-06T14:17:02.933 INFO:tasks.workunit.client.1.smithi086.stdout:3/253: dwrite d5/d36/f3b [0,4194304] 0 2024-04-06T14:17:02.935 INFO:tasks.workunit.client.1.smithi086.stdout:1/167: dread - db/d13/f2e zero size 2024-04-06T14:17:02.935 INFO:tasks.workunit.client.0.smithi032.stdout:1/505: sync 2024-04-06T14:17:02.936 INFO:tasks.workunit.client.0.smithi032.stdout:9/452: write d2/f13 [3173817,26440] 0 2024-04-06T14:17:02.940 INFO:tasks.workunit.client.1.smithi086.stdout:8/309: dwrite d0/d8/d22/d43/f64 [0,4194304] 0 2024-04-06T14:17:02.941 INFO:tasks.workunit.client.0.smithi032.stdout:2/411: mknod d0/d4f/c91 0 2024-04-06T14:17:02.941 INFO:tasks.workunit.client.0.smithi032.stdout:7/437: write d16/d20/d26/d2d/f4f [1655224,128402] 0 2024-04-06T14:17:02.942 INFO:tasks.workunit.client.0.smithi032.stdout:2/412: stat d0/d5a/d3a/c6b 0 2024-04-06T14:17:02.947 INFO:tasks.workunit.client.0.smithi032.stdout:9/453: rmdir d2/d15/d55/d5f/d71/d81/d4a 39 2024-04-06T14:17:02.948 INFO:tasks.workunit.client.1.smithi086.stdout:7/179: dread d7/d1d/f19 [0,4194304] 0 2024-04-06T14:17:02.948 INFO:tasks.workunit.client.1.smithi086.stdout:8/310: creat d0/d13/f68 x:0 0 0 2024-04-06T14:17:02.951 INFO:tasks.workunit.client.1.smithi086.stdout:1/168: creat db/d11/f33 x:0 0 0 2024-04-06T14:17:02.951 INFO:tasks.workunit.client.1.smithi086.stdout:2/178: dwrite d0/d20/d27/d37/f42 [0,4194304] 0 2024-04-06T14:17:02.952 INFO:tasks.workunit.client.1.smithi086.stdout:8/311: write d0/d8/d16/f41 [4227604,34918] 0 2024-04-06T14:17:02.954 INFO:tasks.workunit.client.1.smithi086.stdout:3/254: rename d5/d33/f3a to d5/f44 0 2024-04-06T14:17:02.954 INFO:tasks.workunit.client.1.smithi086.stdout:1/169: write f4 [4396043,76985] 0 2024-04-06T14:17:02.956 INFO:tasks.workunit.client.1.smithi086.stdout:7/180: creat d7/d1d/f41 x:0 0 0 2024-04-06T14:17:02.957 INFO:tasks.workunit.client.1.smithi086.stdout:7/181: write d7/f9 [706614,2657] 0 2024-04-06T14:17:02.957 INFO:tasks.workunit.client.1.smithi086.stdout:1/170: write f8 [3110095,24770] 0 2024-04-06T14:17:02.960 INFO:tasks.workunit.client.1.smithi086.stdout:7/182: fdatasync d7/d10/d11/d1c/f2e 0 2024-04-06T14:17:02.962 INFO:tasks.workunit.client.1.smithi086.stdout:2/179: truncate d0/df/f18 4447236 0 2024-04-06T14:17:02.962 INFO:tasks.workunit.client.1.smithi086.stdout:2/180: readlink d0/l24 0 2024-04-06T14:17:02.963 INFO:tasks.workunit.client.1.smithi086.stdout:2/181: fsync d0/d20/d27/d2b/d35/d36/f3b 0 2024-04-06T14:17:02.966 INFO:tasks.workunit.client.1.smithi086.stdout:2/182: truncate d0/d20/f25 259461 0 2024-04-06T14:17:02.966 INFO:tasks.workunit.client.1.smithi086.stdout:8/312: creat d0/d8/d16/d52/f69 x:0 0 0 2024-04-06T14:17:02.966 INFO:tasks.workunit.client.1.smithi086.stdout:1/171: mknod db/d11/d24/c34 0 2024-04-06T14:17:02.968 INFO:tasks.workunit.client.1.smithi086.stdout:7/183: mknod d7/d10/d11/d1c/d24/d2c/d3f/c42 0 2024-04-06T14:17:02.969 INFO:tasks.workunit.client.1.smithi086.stdout:3/255: creat d5/df/f45 x:0 0 0 2024-04-06T14:17:02.970 INFO:tasks.workunit.client.1.smithi086.stdout:8/313: rmdir d0/d3b 39 2024-04-06T14:17:02.971 INFO:tasks.workunit.client.1.smithi086.stdout:7/184: symlink d7/d10/d11/d21/l43 0 2024-04-06T14:17:02.976 INFO:tasks.workunit.client.1.smithi086.stdout:8/314: rename d0/d8/d22/c35 to d0/d8/d22/c6a 0 2024-04-06T14:17:02.976 INFO:tasks.workunit.client.1.smithi086.stdout:8/315: stat d0/c27 0 2024-04-06T14:17:02.978 INFO:tasks.workunit.client.1.smithi086.stdout:7/185: unlink d7/d10/d11/d1c/d24/d2c/d3f/f40 0 2024-04-06T14:17:02.978 INFO:tasks.workunit.client.1.smithi086.stdout:7/186: stat d7/d10/l2b 0 2024-04-06T14:17:02.980 INFO:tasks.workunit.client.0.smithi032.stdout:1/506: rename d0/f37 to d0/d10/d16/fa6 0 2024-04-06T14:17:02.980 INFO:tasks.workunit.client.1.smithi086.stdout:3/256: unlink d5/d33/l40 0 2024-04-06T14:17:02.982 INFO:tasks.workunit.client.0.smithi032.stdout:9/454: dread d2/f4 [0,4194304] 0 2024-04-06T14:17:02.987 INFO:tasks.workunit.client.0.smithi032.stdout:9/455: mkdir d2/d15/d1e/d65/d6b/d92 0 2024-04-06T14:17:02.987 INFO:tasks.workunit.client.0.smithi032.stdout:2/413: truncate d0/f13 8148335 0 2024-04-06T14:17:02.987 INFO:tasks.workunit.client.0.smithi032.stdout:2/414: chown d0/l3c 11112778 1 2024-04-06T14:17:02.987 INFO:tasks.workunit.client.1.smithi086.stdout:7/187: chown d7/d10/l33 70400 1 2024-04-06T14:17:02.988 INFO:tasks.workunit.client.0.smithi032.stdout:9/456: rmdir d2/d15/d55/d5f/d71/d81/d77 39 2024-04-06T14:17:02.988 INFO:tasks.workunit.client.0.smithi032.stdout:2/415: truncate d0/f8a 556469 0 2024-04-06T14:17:02.989 INFO:tasks.workunit.client.1.smithi086.stdout:7/188: symlink d7/l44 0 2024-04-06T14:17:02.991 INFO:tasks.workunit.client.1.smithi086.stdout:8/316: getdents d0/d8/d16/d52 0 2024-04-06T14:17:02.992 INFO:tasks.workunit.client.0.smithi032.stdout:9/457: chown d2/d15/d55/d5f/d71/d81/l69 39 1 2024-04-06T14:17:02.993 INFO:tasks.workunit.client.1.smithi086.stdout:1/172: truncate db/d11/f17 2786390 0 2024-04-06T14:17:02.994 INFO:tasks.workunit.client.0.smithi032.stdout:2/416: dwrite d0/d5a/f2a [0,4194304] 0 2024-04-06T14:17:02.995 INFO:tasks.workunit.client.1.smithi086.stdout:8/317: chown d0/d8/d16/d52/f69 545 1 2024-04-06T14:17:02.995 INFO:tasks.workunit.client.1.smithi086.stdout:3/257: dwrite d5/d33/f32 [0,4194304] 0 2024-04-06T14:17:02.996 INFO:tasks.workunit.client.1.smithi086.stdout:7/189: mknod d7/d10/d11/d1c/d24/c45 0 2024-04-06T14:17:02.999 INFO:tasks.workunit.client.1.smithi086.stdout:1/173: write db/d11/f27 [743307,93687] 0 2024-04-06T14:17:03.001 INFO:tasks.workunit.client.0.smithi032.stdout:7/438: symlink d16/d20/d48/l9a 0 2024-04-06T14:17:03.003 INFO:tasks.workunit.client.1.smithi086.stdout:7/190: creat d7/d10/d30/f46 x:0 0 0 2024-04-06T14:17:03.004 INFO:tasks.workunit.client.1.smithi086.stdout:3/258: write d5/d26/f3d [586462,68935] 0 2024-04-06T14:17:03.005 INFO:tasks.workunit.client.1.smithi086.stdout:8/318: mknod d0/d8/d16/d39/c6b 0 2024-04-06T14:17:03.007 INFO:tasks.workunit.client.0.smithi032.stdout:9/458: write d2/d15/d1e/f2a [789054,5453] 0 2024-04-06T14:17:03.008 INFO:tasks.workunit.client.1.smithi086.stdout:2/183: dwrite d0/f33 [0,4194304] 0 2024-04-06T14:17:03.010 INFO:tasks.workunit.client.0.smithi032.stdout:7/439: rename d16/d20/d48/f5a to d16/d20/d26/d2d/f9b 0 2024-04-06T14:17:03.011 INFO:tasks.workunit.client.1.smithi086.stdout:1/174: rename db/d13/f2e to db/d11/d1a/d2a/f35 0 2024-04-06T14:17:03.011 INFO:tasks.workunit.client.1.smithi086.stdout:7/191: mknod d7/d10/d30/c47 0 2024-04-06T14:17:03.012 INFO:tasks.workunit.client.0.smithi032.stdout:9/459: mkdir d2/d15/d55/d59/d93 0 2024-04-06T14:17:03.014 INFO:tasks.workunit.client.1.smithi086.stdout:3/259: creat d5/df/f46 x:0 0 0 2024-04-06T14:17:03.015 INFO:tasks.workunit.client.0.smithi032.stdout:9/460: fsync d2/f2d 0 2024-04-06T14:17:03.015 INFO:tasks.workunit.client.0.smithi032.stdout:1/507: dread d0/d9/f83 [0,4194304] 0 2024-04-06T14:17:03.015 INFO:tasks.workunit.client.1.smithi086.stdout:1/175: read db/d11/d1a/f25 [103806,66260] 0 2024-04-06T14:17:03.017 INFO:tasks.workunit.client.1.smithi086.stdout:3/260: write f4 [5658954,100944] 0 2024-04-06T14:17:03.017 INFO:tasks.workunit.client.0.smithi032.stdout:1/508: chown d0/d9/d1f/d72 5100862 1 2024-04-06T14:17:03.018 INFO:tasks.workunit.client.0.smithi032.stdout:7/440: truncate d16/d20/d26/d2d/d30/f38 888060 0 2024-04-06T14:17:03.019 INFO:tasks.workunit.client.1.smithi086.stdout:7/192: rmdir d7/d10/d11/d1c/d24 39 2024-04-06T14:17:03.020 INFO:tasks.workunit.client.1.smithi086.stdout:2/184: dread d0/d20/d27/d2b/d35/d36/f3b [0,4194304] 0 2024-04-06T14:17:03.020 INFO:tasks.workunit.client.1.smithi086.stdout:7/193: dread - d7/d1d/f38 zero size 2024-04-06T14:17:03.021 INFO:tasks.workunit.client.1.smithi086.stdout:2/185: stat d0/d20/d27/d37 0 2024-04-06T14:17:03.022 INFO:tasks.workunit.client.0.smithi032.stdout:7/441: readlink d16/d20/l3c 0 2024-04-06T14:17:03.022 INFO:tasks.workunit.client.1.smithi086.stdout:1/176: creat db/d11/d1a/f36 x:0 0 0 2024-04-06T14:17:03.023 INFO:tasks.workunit.client.0.smithi032.stdout:9/461: readlink d2/d15/d55/d5f/d71/d81/d4a/l4e 0 2024-04-06T14:17:03.023 INFO:tasks.workunit.client.0.smithi032.stdout:1/509: mknod d0/d10/d16/d3f/d98/ca7 0 2024-04-06T14:17:03.024 INFO:tasks.workunit.client.1.smithi086.stdout:2/186: creat d0/d20/d27/d2b/f45 x:0 0 0 2024-04-06T14:17:03.026 INFO:tasks.workunit.client.1.smithi086.stdout:7/194: rename d7/d1d/f41 to d7/d10/d11/d1c/d24/d2c/f48 0 2024-04-06T14:17:03.028 INFO:tasks.workunit.client.1.smithi086.stdout:2/187: dread d0/f33 [0,4194304] 0 2024-04-06T14:17:03.029 INFO:tasks.workunit.client.1.smithi086.stdout:1/177: symlink db/d11/l37 0 2024-04-06T14:17:03.030 INFO:tasks.workunit.client.0.smithi032.stdout:2/417: dread d0/d30/f43 [4194304,4194304] 0 2024-04-06T14:17:03.030 INFO:tasks.workunit.client.0.smithi032.stdout:9/462: dread d2/d15/d1e/d41/f63 [0,4194304] 0 2024-04-06T14:17:03.031 INFO:tasks.workunit.client.1.smithi086.stdout:1/178: dread db/f10 [0,4194304] 0 2024-04-06T14:17:03.031 INFO:tasks.workunit.client.1.smithi086.stdout:7/195: symlink d7/d10/d11/d1c/l49 0 2024-04-06T14:17:03.032 INFO:tasks.workunit.client.0.smithi032.stdout:1/510: dwrite d0/da/d3e/d75/f8a [0,4194304] 0 2024-04-06T14:17:03.033 INFO:tasks.workunit.client.1.smithi086.stdout:7/196: fdatasync d7/d10/d11/d1c/d24/d2c/f39 0 2024-04-06T14:17:03.033 INFO:tasks.workunit.client.1.smithi086.stdout:2/188: link d0/d20/d27/d2b/d35/d36/f40 d0/d20/d44/f46 0 2024-04-06T14:17:03.033 INFO:tasks.workunit.client.1.smithi086.stdout:8/319: dread d0/db/f12 [0,4194304] 0 2024-04-06T14:17:03.034 INFO:tasks.workunit.client.0.smithi032.stdout:8/363: sync 2024-04-06T14:17:03.094 INFO:tasks.workunit.client.1.smithi086.stdout:1/179: mknod db/d11/d24/c38 0 2024-04-06T14:17:03.095 INFO:tasks.workunit.client.1.smithi086.stdout:9/195: sync 2024-04-06T14:17:03.095 INFO:tasks.workunit.client.1.smithi086.stdout:4/136: sync 2024-04-06T14:17:03.095 INFO:tasks.workunit.client.1.smithi086.stdout:5/212: sync 2024-04-06T14:17:03.096 INFO:tasks.workunit.client.0.smithi032.stdout:7/442: creat d16/d20/d54/f9c x:0 0 0 2024-04-06T14:17:03.096 INFO:tasks.workunit.client.1.smithi086.stdout:1/180: write f7 [850144,21850] 0 2024-04-06T14:17:03.097 INFO:tasks.workunit.client.0.smithi032.stdout:2/418: symlink d0/d6e/l92 0 2024-04-06T14:17:03.097 INFO:tasks.workunit.client.1.smithi086.stdout:4/137: stat d0/c1e 0 2024-04-06T14:17:03.099 INFO:tasks.workunit.client.0.smithi032.stdout:1/511: creat d0/d10/d16/d3f/d98/fa8 x:0 0 0 2024-04-06T14:17:03.100 INFO:tasks.workunit.client.0.smithi032.stdout:1/512: fsync d0/d9/f24 0 2024-04-06T14:17:03.103 INFO:tasks.workunit.client.1.smithi086.stdout:1/181: creat db/f39 x:0 0 0 2024-04-06T14:17:03.104 INFO:tasks.workunit.client.0.smithi032.stdout:7/443: mkdir d16/d20/d63/d9d 0 2024-04-06T14:17:03.105 INFO:tasks.workunit.client.0.smithi032.stdout:1/513: mkdir d0/d9/d1f/d7c/da9 0 2024-04-06T14:17:03.106 INFO:tasks.workunit.client.0.smithi032.stdout:8/364: dread d3/d8/d7/d14/f1f [0,4194304] 0 2024-04-06T14:17:03.106 INFO:tasks.workunit.client.1.smithi086.stdout:8/320: creat d0/d8/d16/f6c x:0 0 0 2024-04-06T14:17:03.106 INFO:tasks.workunit.client.1.smithi086.stdout:4/138: dread d0/f2 [0,4194304] 0 2024-04-06T14:17:03.107 INFO:tasks.workunit.client.0.smithi032.stdout:7/444: symlink d16/d20/d26/d2d/l9e 0 2024-04-06T14:17:03.108 INFO:tasks.workunit.client.0.smithi032.stdout:8/365: write d3/d8/d12/d2f/f39 [480291,47833] 0 2024-04-06T14:17:03.112 INFO:tasks.workunit.client.1.smithi086.stdout:4/139: mknod d0/de/d21/c2e 0 2024-04-06T14:17:03.113 INFO:tasks.workunit.client.0.smithi032.stdout:7/445: symlink d16/d20/d26/d2d/l9f 0 2024-04-06T14:17:03.113 INFO:tasks.workunit.client.1.smithi086.stdout:1/182: dwrite f8 [0,4194304] 0 2024-04-06T14:17:03.113 INFO:tasks.workunit.client.1.smithi086.stdout:9/196: link d5/d10/l11 d5/d18/l49 0 2024-04-06T14:17:03.114 INFO:tasks.workunit.client.0.smithi032.stdout:8/366: mkdir d3/d8/d12/d83/d89 0 2024-04-06T14:17:03.114 INFO:tasks.workunit.client.1.smithi086.stdout:9/197: stat d5/d18/f1d 0 2024-04-06T14:17:03.117 INFO:tasks.workunit.client.1.smithi086.stdout:9/198: write d5/d10/d14/d19/d1b/f45 [501013,105675] 0 2024-04-06T14:17:03.118 INFO:tasks.workunit.client.1.smithi086.stdout:7/197: truncate d7/d10/d11/f16 1515580 0 2024-04-06T14:17:03.119 INFO:tasks.workunit.client.1.smithi086.stdout:4/140: mknod d0/de/c2f 0 2024-04-06T14:17:03.120 INFO:tasks.workunit.client.0.smithi032.stdout:8/367: truncate d3/d8/f2e 1141522 0 2024-04-06T14:17:03.121 INFO:tasks.workunit.client.1.smithi086.stdout:2/189: dread d0/df/f1a [0,4194304] 0 2024-04-06T14:17:03.122 INFO:tasks.workunit.client.1.smithi086.stdout:4/141: readlink d0/l9 0 2024-04-06T14:17:03.122 INFO:tasks.workunit.client.1.smithi086.stdout:8/321: truncate d0/d8/d22/d43/f28 2352420 0 2024-04-06T14:17:03.125 INFO:tasks.workunit.client.1.smithi086.stdout:5/213: dwrite d0/f1 [0,4194304] 0 2024-04-06T14:17:03.126 INFO:tasks.workunit.client.0.smithi032.stdout:8/368: mkdir d3/d8/d7/d8a 0 2024-04-06T14:17:03.127 INFO:tasks.workunit.client.1.smithi086.stdout:7/198: chown d7/d10/d11/d1c/c2f 31 1 2024-04-06T14:17:03.128 INFO:tasks.workunit.client.0.smithi032.stdout:2/419: dwrite d0/d30/d39/f8f [0,4194304] 0 2024-04-06T14:17:03.129 INFO:tasks.workunit.client.1.smithi086.stdout:2/190: write d0/d20/d27/d2b/d35/d36/f3b [87968,120433] 0 2024-04-06T14:17:03.129 INFO:tasks.workunit.client.1.smithi086.stdout:7/199: write d7/d1d/f13 [134938,72662] 0 2024-04-06T14:17:03.131 INFO:tasks.workunit.client.0.smithi032.stdout:1/514: dwrite d0/da/d3e/d75/f81 [0,4194304] 0 2024-04-06T14:17:03.133 INFO:tasks.workunit.client.1.smithi086.stdout:4/142: rename d0/de/d21 to d0/d2c/d30 0 2024-04-06T14:17:03.133 INFO:tasks.workunit.client.0.smithi032.stdout:2/420: chown d0/d5a/c4b 12228 1 2024-04-06T14:17:03.134 INFO:tasks.workunit.client.0.smithi032.stdout:2/421: readlink d0/d30/d39/d59/d67/d74/d78/l60 0 2024-04-06T14:17:03.136 INFO:tasks.workunit.client.0.smithi032.stdout:2/422: chown d0/d30/c79 132 1 2024-04-06T14:17:03.136 INFO:tasks.workunit.client.1.smithi086.stdout:8/322: unlink d0/d8/d22/d43/f64 0 2024-04-06T14:17:03.139 INFO:tasks.workunit.client.0.smithi032.stdout:8/369: creat d3/d8/d12/d2f/d4a/f8b x:0 0 0 2024-04-06T14:17:03.141 INFO:tasks.workunit.client.0.smithi032.stdout:1/515: mkdir d0/d9/d1f/d7c/d8f/daa 0 2024-04-06T14:17:03.143 INFO:tasks.workunit.client.1.smithi086.stdout:1/183: mknod db/d11/d1f/c3a 0 2024-04-06T14:17:03.144 INFO:tasks.workunit.client.0.smithi032.stdout:2/423: creat d0/d30/d39/d59/d67/d74/d78/d65/f93 x:0 0 0 2024-04-06T14:17:03.147 INFO:tasks.workunit.client.0.smithi032.stdout:8/370: unlink d3/d8/l33 0 2024-04-06T14:17:03.148 INFO:tasks.workunit.client.1.smithi086.stdout:4/143: creat d0/d2c/d30/d24/f31 x:0 0 0 2024-04-06T14:17:03.150 INFO:tasks.workunit.client.1.smithi086.stdout:8/323: write d0/d3b/f4c [280017,82397] 0 2024-04-06T14:17:03.150 INFO:tasks.workunit.client.0.smithi032.stdout:2/424: rename d0/d30/d39/d59/d67/c7a to d0/d4e/c94 0 2024-04-06T14:17:03.151 INFO:tasks.workunit.client.1.smithi086.stdout:1/184: write db/d13/f14 [842835,40632] 0 2024-04-06T14:17:03.152 INFO:tasks.workunit.client.0.smithi032.stdout:8/371: write d3/d8/f7a [227237,5004] 0 2024-04-06T14:17:03.153 INFO:tasks.workunit.client.0.smithi032.stdout:2/425: dread - d0/d30/d39/d59/d67/d74/d78/f77 zero size 2024-04-06T14:17:03.153 INFO:tasks.workunit.client.1.smithi086.stdout:1/185: chown db/f39 54 1 2024-04-06T14:17:03.153 INFO:tasks.workunit.client.1.smithi086.stdout:5/214: creat d0/f41 x:0 0 0 2024-04-06T14:17:03.157 INFO:tasks.workunit.client.1.smithi086.stdout:1/186: dread - db/d11/d1a/d2a/f35 zero size 2024-04-06T14:17:03.157 INFO:tasks.workunit.client.1.smithi086.stdout:2/191: getdents d0/d20/d27/d2b/d35 0 2024-04-06T14:17:03.158 INFO:tasks.workunit.client.0.smithi032.stdout:7/446: dread d16/d20/d26/d2d/d30/f38 [0,4194304] 0 2024-04-06T14:17:03.159 INFO:tasks.workunit.client.1.smithi086.stdout:1/187: readlink db/d11/d1f/l2f 0 2024-04-06T14:17:03.160 INFO:tasks.workunit.client.1.smithi086.stdout:4/144: dwrite d0/fa [0,4194304] 0 2024-04-06T14:17:03.160 INFO:tasks.workunit.client.1.smithi086.stdout:8/324: mkdir d0/d8/d22/d43/d6d 0 2024-04-06T14:17:03.161 INFO:tasks.workunit.client.1.smithi086.stdout:1/188: fdatasync db/d11/f27 0 2024-04-06T14:17:03.162 INFO:tasks.workunit.client.1.smithi086.stdout:4/145: write d0/f12 [51713,35593] 0 2024-04-06T14:17:03.163 INFO:tasks.workunit.client.0.smithi032.stdout:2/426: write d0/ff [5792829,76762] 0 2024-04-06T14:17:03.163 INFO:tasks.workunit.client.1.smithi086.stdout:8/325: read d0/f34 [834289,8580] 0 2024-04-06T14:17:03.163 INFO:tasks.workunit.client.0.smithi032.stdout:8/372: dread d3/d8/d7/d14/f1f [4194304,4194304] 0 2024-04-06T14:17:03.164 INFO:tasks.workunit.client.1.smithi086.stdout:5/215: mknod d0/c42 0 2024-04-06T14:17:03.164 INFO:tasks.workunit.client.0.smithi032.stdout:7/447: write d16/d20/d26/d2d/f75 [273452,26149] 0 2024-04-06T14:17:03.164 INFO:tasks.workunit.client.0.smithi032.stdout:6/411: sync 2024-04-06T14:17:03.164 INFO:tasks.workunit.client.0.smithi032.stdout:3/415: sync 2024-04-06T14:17:03.164 INFO:tasks.workunit.client.0.smithi032.stdout:0/315: sync 2024-04-06T14:17:03.166 INFO:tasks.workunit.client.1.smithi086.stdout:8/326: readlink d0/db/l45 0 2024-04-06T14:17:03.168 INFO:tasks.workunit.client.0.smithi032.stdout:2/427: write d0/d6e/f86 [4504644,69523] 0 2024-04-06T14:17:03.169 INFO:tasks.workunit.client.1.smithi086.stdout:2/192: dwrite d0/f1e [0,4194304] 0 2024-04-06T14:17:03.170 INFO:tasks.workunit.client.1.smithi086.stdout:4/146: unlink d0/f7 0 2024-04-06T14:17:03.172 INFO:tasks.workunit.client.0.smithi032.stdout:3/416: write d3/f2d [1528248,95230] 0 2024-04-06T14:17:03.173 INFO:tasks.workunit.client.0.smithi032.stdout:1/516: dwrite d0/d10/d16/fa1 [0,4194304] 0 2024-04-06T14:17:03.175 INFO:tasks.workunit.client.1.smithi086.stdout:1/189: creat db/d11/d1a/d2a/f3b x:0 0 0 2024-04-06T14:17:03.176 INFO:tasks.workunit.client.0.smithi032.stdout:6/412: rename d4/f6d to d4/d5/d19/d25/f88 0 2024-04-06T14:17:03.176 INFO:tasks.workunit.client.0.smithi032.stdout:2/428: unlink d0/d5a/c4b 0 2024-04-06T14:17:03.179 INFO:tasks.workunit.client.1.smithi086.stdout:7/200: truncate d7/f9 106041 0 2024-04-06T14:17:03.179 INFO:tasks.workunit.client.0.smithi032.stdout:1/517: chown d0/da/f6b 0 1 2024-04-06T14:17:03.180 INFO:tasks.workunit.client.0.smithi032.stdout:8/373: write d3/d8/d12/d2f/d46/f74 [4599536,40306] 0 2024-04-06T14:17:03.182 INFO:tasks.workunit.client.1.smithi086.stdout:5/216: dread d0/f4 [0,4194304] 0 2024-04-06T14:17:03.185 INFO:tasks.workunit.client.0.smithi032.stdout:3/417: link d3/f4 d3/fac 0 2024-04-06T14:17:03.187 INFO:tasks.workunit.client.0.smithi032.stdout:8/374: read - d3/d8/d12/d2f/f77 zero size 2024-04-06T14:17:03.187 INFO:tasks.workunit.client.0.smithi032.stdout:0/316: dwrite d3/f63 [0,4194304] 0 2024-04-06T14:17:03.188 INFO:tasks.workunit.client.0.smithi032.stdout:1/518: write d0/d10/d16/f51 [389043,63267] 0 2024-04-06T14:17:03.189 INFO:tasks.workunit.client.1.smithi086.stdout:2/193: creat d0/d20/d27/d2b/d35/f47 x:0 0 0 2024-04-06T14:17:03.189 INFO:tasks.workunit.client.1.smithi086.stdout:8/327: mkdir d0/d8/d22/d65/d6e 0 2024-04-06T14:17:03.193 INFO:tasks.workunit.client.1.smithi086.stdout:8/328: fdatasync d0/f2 0 2024-04-06T14:17:03.193 INFO:tasks.workunit.client.1.smithi086.stdout:7/201: creat d7/d10/d30/f4a x:0 0 0 2024-04-06T14:17:03.194 INFO:tasks.workunit.client.1.smithi086.stdout:2/194: mkdir d0/df/d48 0 2024-04-06T14:17:03.194 INFO:tasks.workunit.client.1.smithi086.stdout:5/217: dread d0/d16/d18/d21/d23/f1f [0,4194304] 0 2024-04-06T14:17:03.194 INFO:tasks.workunit.client.1.smithi086.stdout:1/190: symlink db/l3c 0 2024-04-06T14:17:03.194 INFO:tasks.workunit.client.1.smithi086.stdout:7/202: creat d7/d10/d30/f4b x:0 0 0 2024-04-06T14:17:03.195 INFO:tasks.workunit.client.0.smithi032.stdout:6/413: dread d4/d5/d8/d4b/f4a [0,4194304] 0 2024-04-06T14:17:03.195 INFO:tasks.workunit.client.0.smithi032.stdout:0/317: dread - d3/da/d25/d2b/f40 zero size 2024-04-06T14:17:03.195 INFO:tasks.workunit.client.0.smithi032.stdout:2/429: fsync d0/d30/d39/d59/d67/d74/d78/f89 0 2024-04-06T14:17:03.200 INFO:tasks.workunit.client.1.smithi086.stdout:2/195: dwrite d0/df/f1b [0,4194304] 0 2024-04-06T14:17:03.200 INFO:tasks.workunit.client.0.smithi032.stdout:7/448: dwrite d16/d20/d26/d2d/d30/f53 [0,4194304] 0 2024-04-06T14:17:03.206 INFO:tasks.workunit.client.1.smithi086.stdout:5/218: fsync d0/d19/f3c 0 2024-04-06T14:17:03.207 INFO:tasks.workunit.client.1.smithi086.stdout:8/329: mknod d0/d8/d16/d52/d5f/c6f 0 2024-04-06T14:17:03.207 INFO:tasks.workunit.client.1.smithi086.stdout:7/203: symlink d7/d10/d11/d1c/d36/l4c 0 2024-04-06T14:17:03.208 INFO:tasks.workunit.client.1.smithi086.stdout:1/191: creat db/d13/f3d x:0 0 0 2024-04-06T14:17:03.210 INFO:tasks.workunit.client.0.smithi032.stdout:7/449: mknod d16/d20/d63/ca0 0 2024-04-06T14:17:03.211 INFO:tasks.workunit.client.0.smithi032.stdout:6/414: creat d4/d5/d28/d54/f89 x:0 0 0 2024-04-06T14:17:03.212 INFO:tasks.workunit.client.0.smithi032.stdout:0/318: chown d3/da/d35/d4b/c66 95972 1 2024-04-06T14:17:03.213 INFO:tasks.workunit.client.1.smithi086.stdout:1/192: mknod db/d13/c3e 0 2024-04-06T14:17:03.214 INFO:tasks.workunit.client.1.smithi086.stdout:7/204: mkdir d7/d10/d11/d1c/d24/d4d 0 2024-04-06T14:17:03.215 INFO:tasks.workunit.client.1.smithi086.stdout:1/193: write f7 [3826199,40780] 0 2024-04-06T14:17:03.215 INFO:tasks.workunit.client.0.smithi032.stdout:1/519: dwrite d0/d10/f30 [0,4194304] 0 2024-04-06T14:17:03.216 INFO:tasks.workunit.client.0.smithi032.stdout:6/415: chown d4/d5/d19/d25 684195701 1 2024-04-06T14:17:03.216 INFO:tasks.workunit.client.1.smithi086.stdout:7/205: fdatasync d7/d10/d11/d1c/d24/d2c/f39 0 2024-04-06T14:17:03.216 INFO:tasks.workunit.client.1.smithi086.stdout:8/330: chown d0/d8/d22/c36 3882 1 2024-04-06T14:17:03.218 INFO:tasks.workunit.client.0.smithi032.stdout:0/319: creat d3/da/f6f x:0 0 0 2024-04-06T14:17:03.219 INFO:tasks.workunit.client.1.smithi086.stdout:5/219: truncate d0/f3 383317 0 2024-04-06T14:17:03.219 INFO:tasks.workunit.client.1.smithi086.stdout:5/220: fsync d0/d16/d18/d35/f3a 0 2024-04-06T14:17:03.220 INFO:tasks.workunit.client.1.smithi086.stdout:1/194: read db/f16 [49823,52733] 0 2024-04-06T14:17:03.221 INFO:tasks.workunit.client.0.smithi032.stdout:7/450: chown d16/d20/d63/c8e 13549999 1 2024-04-06T14:17:03.222 INFO:tasks.workunit.client.1.smithi086.stdout:8/331: symlink d0/d8/d16/d52/l70 0 2024-04-06T14:17:03.223 INFO:tasks.workunit.client.0.smithi032.stdout:8/375: dwrite d3/d8/d12/f38 [0,4194304] 0 2024-04-06T14:17:03.225 INFO:tasks.workunit.client.1.smithi086.stdout:7/206: creat d7/d10/d11/d1c/d24/f4e x:0 0 0 2024-04-06T14:17:03.228 INFO:tasks.workunit.client.0.smithi032.stdout:1/520: link d0/da/c27 d0/d10/cab 0 2024-04-06T14:17:03.230 INFO:tasks.workunit.client.0.smithi032.stdout:8/376: chown d3/d8/d12/d2f/l57 136 1 2024-04-06T14:17:03.230 INFO:tasks.workunit.client.0.smithi032.stdout:2/430: dread d0/d5a/f1f [0,4194304] 0 2024-04-06T14:17:03.232 INFO:tasks.workunit.client.0.smithi032.stdout:1/521: write d0/d10/f3d [344470,50912] 0 2024-04-06T14:17:03.233 INFO:tasks.workunit.client.0.smithi032.stdout:6/416: dwrite d4/d5/d28/f41 [0,4194304] 0 2024-04-06T14:17:03.234 INFO:tasks.workunit.client.1.smithi086.stdout:7/207: rename d7/d10/d11/d1c/d24/d2c/d3f/c42 to d7/d10/d11/d1c/d24/c4f 0 2024-04-06T14:17:03.236 INFO:tasks.workunit.client.1.smithi086.stdout:5/221: dwrite d0/d16/d18/d21/d23/f1f [4194304,4194304] 0 2024-04-06T14:17:03.238 INFO:tasks.workunit.client.0.smithi032.stdout:6/417: read - d4/d5/d19/f46 zero size 2024-04-06T14:17:03.238 INFO:tasks.workunit.client.0.smithi032.stdout:1/522: chown d0/d9/f24 1413 1 2024-04-06T14:17:03.242 INFO:tasks.workunit.client.0.smithi032.stdout:6/418: readlink d4/d5/d8/d1f/d51/l26 0 2024-04-06T14:17:03.245 INFO:tasks.workunit.client.0.smithi032.stdout:7/451: dwrite d16/d20/d48/d6d/f79 [0,4194304] 0 2024-04-06T14:17:03.247 INFO:tasks.workunit.client.1.smithi086.stdout:0/214: sync 2024-04-06T14:17:03.248 INFO:tasks.workunit.client.1.smithi086.stdout:6/194: sync 2024-04-06T14:17:03.251 INFO:tasks.workunit.client.0.smithi032.stdout:1/523: unlink d0/l86 0 2024-04-06T14:17:03.251 INFO:tasks.workunit.client.1.smithi086.stdout:4/147: dwrite d0/de/f1a [0,4194304] 0 2024-04-06T14:17:03.253 INFO:tasks.workunit.client.1.smithi086.stdout:1/195: creat db/d11/f3f x:0 0 0 2024-04-06T14:17:03.257 INFO:tasks.workunit.client.0.smithi032.stdout:8/377: dwrite d3/d8/d12/d2f/d46/f53 [0,4194304] 0 2024-04-06T14:17:03.258 INFO:tasks.workunit.client.1.smithi086.stdout:5/222: mkdir d0/d43 0 2024-04-06T14:17:03.260 INFO:tasks.workunit.client.1.smithi086.stdout:0/215: dwrite d0/f3f [0,4194304] 0 2024-04-06T14:17:03.261 INFO:tasks.workunit.client.0.smithi032.stdout:2/431: dwrite d0/d30/d39/d59/f5f [0,4194304] 0 2024-04-06T14:17:03.263 INFO:tasks.workunit.client.1.smithi086.stdout:7/208: dwrite d7/d1d/f19 [0,4194304] 0 2024-04-06T14:17:03.268 INFO:tasks.workunit.client.0.smithi032.stdout:1/524: truncate d0/d10/f25 1040295 0 2024-04-06T14:17:03.270 INFO:tasks.workunit.client.0.smithi032.stdout:6/419: dwrite d4/d5/d18/f64 [0,4194304] 0 2024-04-06T14:17:03.274 INFO:tasks.workunit.client.1.smithi086.stdout:1/196: dread db/f10 [0,4194304] 0 2024-04-06T14:17:03.275 INFO:tasks.workunit.client.0.smithi032.stdout:1/525: fsync d0/d10/d66/f70 0 2024-04-06T14:17:03.275 INFO:tasks.workunit.client.1.smithi086.stdout:5/223: symlink d0/d16/d18/d21/l44 0 2024-04-06T14:17:03.278 INFO:tasks.workunit.client.1.smithi086.stdout:0/216: symlink d0/l54 0 2024-04-06T14:17:03.279 INFO:tasks.workunit.client.1.smithi086.stdout:1/197: read db/fe [972558,80118] 0 2024-04-06T14:17:03.281 INFO:tasks.workunit.client.0.smithi032.stdout:6/420: creat d4/d5/d19/d25/d67/f8a x:0 0 0 2024-04-06T14:17:03.283 INFO:tasks.workunit.client.0.smithi032.stdout:1/526: readlink d0/da/d3e/la4 0 2024-04-06T14:17:03.284 INFO:tasks.workunit.client.1.smithi086.stdout:0/217: dread d0/ff [4194304,4194304] 0 2024-04-06T14:17:03.284 INFO:tasks.workunit.client.1.smithi086.stdout:5/224: rmdir d0/d16/d18/d21/d23/d28 39 2024-04-06T14:17:03.286 INFO:tasks.workunit.client.1.smithi086.stdout:5/225: chown d0/d16/d18/d35/f3a 198276 1 2024-04-06T14:17:03.287 INFO:tasks.workunit.client.0.smithi032.stdout:6/421: creat d4/d5/d28/d54/f8b x:0 0 0 2024-04-06T14:17:03.288 INFO:tasks.workunit.client.0.smithi032.stdout:4/457: sync 2024-04-06T14:17:03.289 INFO:tasks.workunit.client.0.smithi032.stdout:9/463: sync 2024-04-06T14:17:03.289 INFO:tasks.workunit.client.0.smithi032.stdout:5/417: sync 2024-04-06T14:17:03.293 INFO:tasks.workunit.client.0.smithi032.stdout:6/422: chown d4/d5/d28/f5c 15561810 1 2024-04-06T14:17:03.293 INFO:tasks.workunit.client.1.smithi086.stdout:8/332: rename d0/d8 to d0/d71 0 2024-04-06T14:17:03.294 INFO:tasks.workunit.client.1.smithi086.stdout:0/218: rmdir d0/d1/d4/de 39 2024-04-06T14:17:03.295 INFO:tasks.workunit.client.1.smithi086.stdout:0/219: dread - d0/d15/f4f zero size 2024-04-06T14:17:03.295 INFO:tasks.workunit.client.0.smithi032.stdout:6/423: stat d4/d5/d8/d1f/d53 0 2024-04-06T14:17:03.295 INFO:tasks.workunit.client.0.smithi032.stdout:9/464: creat d2/d15/d1e/d65/d83/f94 x:0 0 0 2024-04-06T14:17:03.297 INFO:tasks.workunit.client.0.smithi032.stdout:1/527: link d0/d10/c12 d0/d9/d1f/d72/cac 0 2024-04-06T14:17:03.298 INFO:tasks.workunit.client.1.smithi086.stdout:5/226: readlink d0/d16/d18/d21/d23/d28/l38 0 2024-04-06T14:17:03.300 INFO:tasks.workunit.client.0.smithi032.stdout:6/424: creat d4/d5/d8/d1f/f8c x:0 0 0 2024-04-06T14:17:03.300 INFO:tasks.workunit.client.0.smithi032.stdout:7/452: dwrite f7 [4194304,4194304] 0 2024-04-06T14:17:03.301 INFO:tasks.workunit.client.1.smithi086.stdout:7/209: getdents d7 0 2024-04-06T14:17:03.302 INFO:tasks.workunit.client.0.smithi032.stdout:9/465: mkdir d2/d15/d55/d59/d93/d95 0 2024-04-06T14:17:03.303 INFO:tasks.workunit.client.0.smithi032.stdout:5/418: getdents d16/d18/d33/d45/d87 0 2024-04-06T14:17:03.304 INFO:tasks.workunit.client.1.smithi086.stdout:5/227: dread d0/d19/f3c [0,4194304] 0 2024-04-06T14:17:03.305 INFO:tasks.workunit.client.0.smithi032.stdout:6/425: mkdir d4/d5/d14/d8d 0 2024-04-06T14:17:03.305 INFO:tasks.workunit.client.0.smithi032.stdout:5/419: stat d16/d18/d33/d45/d52/d39/c66 0 2024-04-06T14:17:03.307 INFO:tasks.workunit.client.1.smithi086.stdout:1/198: link db/d11/d1a/c1c db/d11/d24/c40 0 2024-04-06T14:17:03.308 INFO:tasks.workunit.client.0.smithi032.stdout:8/378: truncate d3/d8/d12/d2f/f39 177750 0 2024-04-06T14:17:03.310 INFO:tasks.workunit.client.0.smithi032.stdout:2/432: dread d0/d30/f43 [0,4194304] 0 2024-04-06T14:17:03.311 INFO:tasks.workunit.client.1.smithi086.stdout:7/210: rmdir d7/d10/d11 39 2024-04-06T14:17:03.313 INFO:tasks.workunit.client.1.smithi086.stdout:6/195: truncate d6/d9/f31 1061425 0 2024-04-06T14:17:03.314 INFO:tasks.workunit.client.0.smithi032.stdout:8/379: readlink d3/d8/d12/l23 0 2024-04-06T14:17:03.314 INFO:tasks.workunit.client.0.smithi032.stdout:4/458: dread d0/df/d1f/f25 [0,4194304] 0 2024-04-06T14:17:03.315 INFO:tasks.workunit.client.0.smithi032.stdout:7/453: rename d16/d20/l60 to d16/d20/la1 0 2024-04-06T14:17:03.316 INFO:tasks.workunit.client.1.smithi086.stdout:5/228: write d0/d16/f39 [1974584,86751] 0 2024-04-06T14:17:03.317 INFO:tasks.workunit.client.0.smithi032.stdout:6/426: rename d4/d5/d14/f78 to d4/d5/d8/f8e 0 2024-04-06T14:17:03.320 INFO:tasks.workunit.client.1.smithi086.stdout:2/196: dwrite d0/d20/d27/d2b/d35/d36/f40 [0,4194304] 0 2024-04-06T14:17:03.322 INFO:tasks.workunit.client.0.smithi032.stdout:6/427: rmdir d4/d5/d28/d65 39 2024-04-06T14:17:03.322 INFO:tasks.workunit.client.1.smithi086.stdout:8/333: creat d0/f72 x:0 0 0 2024-04-06T14:17:03.323 INFO:tasks.workunit.client.0.smithi032.stdout:6/428: dread - d4/d5/d8/d1f/d51/f7b zero size 2024-04-06T14:17:03.324 INFO:tasks.workunit.client.0.smithi032.stdout:8/380: mkdir d3/d8/d3a/d7b/d8c 0 2024-04-06T14:17:03.325 INFO:tasks.workunit.client.0.smithi032.stdout:6/429: read d4/f13 [4116951,57117] 0 2024-04-06T14:17:03.326 INFO:tasks.workunit.client.0.smithi032.stdout:6/430: readlink d4/d5/d8/d2f/l3a 0 2024-04-06T14:17:03.327 INFO:tasks.workunit.client.1.smithi086.stdout:6/196: write d6/d9/db/f16 [189015,32991] 0 2024-04-06T14:17:03.327 INFO:tasks.workunit.client.0.smithi032.stdout:9/466: rename d2/d15/d55/d5f/d71/d81/d4a/d73/d7c/c8c to d2/d15/d55/d5f/d71/d81/c96 0 2024-04-06T14:17:03.328 INFO:tasks.workunit.client.1.smithi086.stdout:0/220: write d0/d1/d4/f8 [4278495,101530] 0 2024-04-06T14:17:03.329 INFO:tasks.workunit.client.1.smithi086.stdout:5/229: link d0/d16/d18/d35/f3a d0/d16/d18/d21/f45 0 2024-04-06T14:17:03.330 INFO:tasks.workunit.client.1.smithi086.stdout:4/148: truncate d0/d15/d20/f27 154773 0 2024-04-06T14:17:03.331 INFO:tasks.workunit.client.1.smithi086.stdout:2/197: link d0/f1e d0/d20/d44/f49 0 2024-04-06T14:17:03.333 INFO:tasks.workunit.client.0.smithi032.stdout:1/528: rename d0/d10/d16/f42 to d0/da/d3e/fad 0 2024-04-06T14:17:03.334 INFO:tasks.workunit.client.0.smithi032.stdout:5/420: write d16/d18/d33/d45/d52/f4e [291862,33536] 0 2024-04-06T14:17:03.335 INFO:tasks.workunit.client.0.smithi032.stdout:8/381: dwrite d3/d8/f87 [0,4194304] 0 2024-04-06T14:17:03.337 INFO:tasks.workunit.client.1.smithi086.stdout:0/221: symlink d0/d1/l55 0 2024-04-06T14:17:03.338 INFO:tasks.workunit.client.0.smithi032.stdout:5/421: stat d16/d18/d33/f34 0 2024-04-06T14:17:03.340 INFO:tasks.workunit.client.0.smithi032.stdout:6/431: mknod d4/d5/d8/d1f/d53/c8f 0 2024-04-06T14:17:03.340 INFO:tasks.workunit.client.1.smithi086.stdout:7/211: dwrite d7/d10/d11/d1c/f2e [0,4194304] 0 2024-04-06T14:17:03.342 INFO:tasks.workunit.client.0.smithi032.stdout:2/433: truncate d0/d5a/d3a/f48 566052 0 2024-04-06T14:17:03.343 INFO:tasks.workunit.client.1.smithi086.stdout:2/198: mkdir d0/d20/d27/d2b/d35/d36/d4a 0 2024-04-06T14:17:03.344 INFO:tasks.workunit.client.0.smithi032.stdout:4/459: write d0/df/d23/f3d [6530357,49854] 0 2024-04-06T14:17:03.345 INFO:tasks.workunit.client.0.smithi032.stdout:2/434: readlink d0/d4e/l70 0 2024-04-06T14:17:03.346 INFO:tasks.workunit.client.0.smithi032.stdout:9/467: dwrite d2/f53 [0,4194304] 0 2024-04-06T14:17:03.349 INFO:tasks.workunit.client.0.smithi032.stdout:6/432: creat d4/d5/d8/d2b/f90 x:0 0 0 2024-04-06T14:17:03.350 INFO:tasks.workunit.client.1.smithi086.stdout:6/197: getdents d6/d20 0 2024-04-06T14:17:03.351 INFO:tasks.workunit.client.1.smithi086.stdout:8/334: write d0/d71/d22/d43/f28 [2702476,3831] 0 2024-04-06T14:17:03.351 INFO:tasks.workunit.client.0.smithi032.stdout:2/435: mkdir d0/d4f/d95 0 2024-04-06T14:17:03.352 INFO:tasks.workunit.client.1.smithi086.stdout:7/212: fsync d7/d10/d11/d1c/d24/f4e 0 2024-04-06T14:17:03.353 INFO:tasks.workunit.client.1.smithi086.stdout:2/199: unlink d0/d20/d27/f2f 0 2024-04-06T14:17:03.354 INFO:tasks.workunit.client.1.smithi086.stdout:0/222: dread d0/d1/d4/d7/f11 [4194304,4194304] 0 2024-04-06T14:17:03.355 INFO:tasks.workunit.client.1.smithi086.stdout:8/335: readlink d0/d71/d16/l4a 0 2024-04-06T14:17:03.355 INFO:tasks.workunit.client.0.smithi032.stdout:8/382: rename d3/d8/c63 to d3/d8/d12/d2f/d4a/d60/d82/d84/c8d 0 2024-04-06T14:17:03.355 INFO:tasks.workunit.client.1.smithi086.stdout:1/199: dwrite db/f16 [0,4194304] 0 2024-04-06T14:17:03.358 INFO:tasks.workunit.client.0.smithi032.stdout:9/468: truncate d2/d15/d55/f6e 226334 0 2024-04-06T14:17:03.358 INFO:tasks.workunit.client.0.smithi032.stdout:5/422: dwrite d16/f38 [0,4194304] 0 2024-04-06T14:17:03.360 INFO:tasks.workunit.client.1.smithi086.stdout:0/223: stat d0/d1/d1b/d46/l4d 0 2024-04-06T14:17:03.360 INFO:tasks.workunit.client.1.smithi086.stdout:2/200: write d0/f2a [4653331,89346] 0 2024-04-06T14:17:03.362 INFO:tasks.workunit.client.0.smithi032.stdout:6/433: write d4/d5/d19/d25/f87 [5644088,60278] 0 2024-04-06T14:17:03.363 INFO:tasks.workunit.client.1.smithi086.stdout:7/213: creat d7/d10/d30/f50 x:0 0 0 2024-04-06T14:17:03.363 INFO:tasks.workunit.client.0.smithi032.stdout:4/460: getdents d0/df/d23/d31/d50/da0 0 2024-04-06T14:17:03.363 INFO:tasks.workunit.client.0.smithi032.stdout:5/423: read - d16/d18/f23 zero size 2024-04-06T14:17:03.364 INFO:tasks.workunit.client.1.smithi086.stdout:6/198: mkdir d6/d9/d36/d45/d4a 0 2024-04-06T14:17:03.365 INFO:tasks.workunit.client.1.smithi086.stdout:1/200: write db/d11/f12 [3004328,67172] 0 2024-04-06T14:17:03.366 INFO:tasks.workunit.client.0.smithi032.stdout:8/383: creat d3/f8e x:0 0 0 2024-04-06T14:17:03.367 INFO:tasks.workunit.client.0.smithi032.stdout:1/529: dread d0/d10/f25 [0,4194304] 0 2024-04-06T14:17:03.370 INFO:tasks.workunit.client.1.smithi086.stdout:1/201: write f2 [5935746,67209] 0 2024-04-06T14:17:03.373 INFO:tasks.workunit.client.1.smithi086.stdout:2/201: symlink d0/l4b 0 2024-04-06T14:17:03.375 INFO:tasks.workunit.client.1.smithi086.stdout:6/199: rename d6/d13/f3e to d6/d13/d1a/f4b 0 2024-04-06T14:17:03.375 INFO:tasks.workunit.client.1.smithi086.stdout:7/214: symlink d7/d10/d11/d1c/d24/d2c/d3c/l51 0 2024-04-06T14:17:03.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:03 smithi086 ceph-mon[47925]: pgmap v19: 65 pgs: 65 active+clean; 1.3 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 19 MiB/s rd, 153 MiB/s wr, 251 op/s 2024-04-06T14:17:03.380 INFO:tasks.workunit.client.1.smithi086.stdout:1/202: fsync db/fc 0 2024-04-06T14:17:03.383 INFO:tasks.workunit.client.1.smithi086.stdout:4/149: dwrite d0/f5 [0,4194304] 0 2024-04-06T14:17:03.385 INFO:tasks.workunit.client.1.smithi086.stdout:1/203: chown db/d11/d1a/d2a/f3b 10335 1 2024-04-06T14:17:03.386 INFO:tasks.workunit.client.1.smithi086.stdout:0/224: dwrite d0/f40 [0,4194304] 0 2024-04-06T14:17:03.389 INFO:tasks.workunit.client.1.smithi086.stdout:4/150: chown d0/c1e 45341 1 2024-04-06T14:17:03.390 INFO:tasks.workunit.client.1.smithi086.stdout:8/336: dwrite d0/db/d32/f3e [0,4194304] 0 2024-04-06T14:17:03.395 INFO:tasks.workunit.client.1.smithi086.stdout:1/204: readlink db/d11/d1f/l2d 0 2024-04-06T14:17:03.398 INFO:tasks.workunit.client.1.smithi086.stdout:4/151: chown d0/d2c/d30/c2e 6 1 2024-04-06T14:17:03.398 INFO:tasks.workunit.client.1.smithi086.stdout:0/225: rmdir d0/d1/d25 39 2024-04-06T14:17:03.401 INFO:tasks.workunit.client.1.smithi086.stdout:7/215: dwrite d7/d10/d30/f46 [0,4194304] 0 2024-04-06T14:17:03.402 INFO:tasks.workunit.client.1.smithi086.stdout:8/337: symlink d0/d71/d22/d43/l73 0 2024-04-06T14:17:03.404 INFO:tasks.workunit.client.1.smithi086.stdout:6/200: dread - d6/d13/d1a/d22/f3f zero size 2024-04-06T14:17:03.404 INFO:tasks.workunit.client.1.smithi086.stdout:2/202: dread d0/d20/d44/f49 [0,4194304] 0 2024-04-06T14:17:03.405 INFO:tasks.workunit.client.1.smithi086.stdout:4/152: write d0/d2c/d30/d24/f2d [42102,25465] 0 2024-04-06T14:17:03.405 INFO:tasks.workunit.client.1.smithi086.stdout:7/216: dread - d7/d1d/f3e zero size 2024-04-06T14:17:03.407 INFO:tasks.workunit.client.1.smithi086.stdout:1/205: dread db/ff [0,4194304] 0 2024-04-06T14:17:03.408 INFO:tasks.workunit.client.1.smithi086.stdout:4/153: stat d0/d15 0 2024-04-06T14:17:03.408 INFO:tasks.workunit.client.1.smithi086.stdout:0/226: getdents d0 0 2024-04-06T14:17:03.408 INFO:tasks.workunit.client.1.smithi086.stdout:7/217: fsync d7/d1d/f38 0 2024-04-06T14:17:03.408 INFO:tasks.workunit.client.1.smithi086.stdout:8/338: creat d0/d71/d22/f74 x:0 0 0 2024-04-06T14:17:03.410 INFO:tasks.workunit.client.1.smithi086.stdout:0/227: write d0/f16 [511349,68384] 0 2024-04-06T14:17:03.412 INFO:tasks.workunit.client.1.smithi086.stdout:4/154: truncate d0/f5 4642601 0 2024-04-06T14:17:03.412 INFO:tasks.workunit.client.1.smithi086.stdout:2/203: creat d0/d20/f4c x:0 0 0 2024-04-06T14:17:03.413 INFO:tasks.workunit.client.1.smithi086.stdout:7/218: fdatasync d7/d10/d11/d1c/d24/d2c/f39 0 2024-04-06T14:17:03.414 INFO:tasks.workunit.client.0.smithi032.stdout:8/384: symlink d3/d8/d3a/l8f 0 2024-04-06T14:17:03.415 INFO:tasks.workunit.client.1.smithi086.stdout:1/206: unlink db/d11/c26 0 2024-04-06T14:17:03.415 INFO:tasks.workunit.client.1.smithi086.stdout:4/155: readlink d0/l9 0 2024-04-06T14:17:03.420 INFO:tasks.workunit.client.1.smithi086.stdout:6/201: rename d6/d9/db/f1e to d6/d9/d36/d45/d4a/f4c 0 2024-04-06T14:17:03.421 INFO:tasks.workunit.client.1.smithi086.stdout:2/204: mkdir d0/d20/d27/d2b/d35/d36/d4d 0 2024-04-06T14:17:03.422 INFO:tasks.workunit.client.1.smithi086.stdout:8/339: mkdir d0/d71/d22/d43/d6d/d75 0 2024-04-06T14:17:03.423 INFO:tasks.workunit.client.0.smithi032.stdout:4/461: dwrite d0/df/d2b/d46/d58/f7e [0,4194304] 0 2024-04-06T14:17:03.423 INFO:tasks.workunit.client.0.smithi032.stdout:2/436: dread d0/d5a/f1a [0,4194304] 0 2024-04-06T14:17:03.427 INFO:tasks.workunit.client.1.smithi086.stdout:1/207: creat db/d11/f41 x:0 0 0 2024-04-06T14:17:03.427 INFO:tasks.workunit.client.1.smithi086.stdout:6/202: mknod d6/d13/d1a/d22/c4d 0 2024-04-06T14:17:03.428 INFO:tasks.workunit.client.0.smithi032.stdout:8/385: mkdir d3/d8/d12/d2f/d4a/d60/d90 0 2024-04-06T14:17:03.428 INFO:tasks.workunit.client.0.smithi032.stdout:5/424: creat d16/f90 x:0 0 0 2024-04-06T14:17:03.429 INFO:tasks.workunit.client.0.smithi032.stdout:6/434: rmdir d4/d5/d28/d65/d6f 0 2024-04-06T14:17:03.429 INFO:tasks.workunit.client.0.smithi032.stdout:7/454: dread d16/d20/d26/d2d/f69 [0,4194304] 0 2024-04-06T14:17:03.430 INFO:tasks.workunit.client.1.smithi086.stdout:4/156: dwrite d0/f5 [0,4194304] 0 2024-04-06T14:17:03.432 INFO:tasks.workunit.client.0.smithi032.stdout:6/435: stat d4/d5/d28/d54/f8b 0 2024-04-06T14:17:03.434 INFO:tasks.workunit.client.0.smithi032.stdout:4/462: rmdir d0/df/d23/d31/d50/d57/d5b/d9c 39 2024-04-06T14:17:03.435 INFO:tasks.workunit.client.0.smithi032.stdout:1/530: dwrite d0/d9/d1f/d72/fa2 [0,4194304] 0 2024-04-06T14:17:03.435 INFO:tasks.workunit.client.1.smithi086.stdout:0/228: dwrite d0/d15/d4e/f42 [0,4194304] 0 2024-04-06T14:17:03.437 INFO:tasks.workunit.client.0.smithi032.stdout:5/425: dread d16/d18/d33/f34 [4194304,4194304] 0 2024-04-06T14:17:03.441 INFO:tasks.workunit.client.0.smithi032.stdout:5/426: write d16/d18/f20 [3266326,64427] 0 2024-04-06T14:17:03.441 INFO:tasks.workunit.client.1.smithi086.stdout:7/219: dwrite d7/d10/d30/f35 [0,4194304] 0 2024-04-06T14:17:03.442 INFO:tasks.workunit.client.0.smithi032.stdout:2/437: unlink d0/d30/d39/d59/d67/d74/d78/f89 0 2024-04-06T14:17:03.444 INFO:tasks.workunit.client.0.smithi032.stdout:2/438: chown d0/d5a/f25 0 1 2024-04-06T14:17:03.444 INFO:tasks.workunit.client.0.smithi032.stdout:7/455: dwrite f7 [4194304,4194304] 0 2024-04-06T14:17:03.444 INFO:tasks.workunit.client.1.smithi086.stdout:7/220: write d7/d10/d30/f4a [81011,24309] 0 2024-04-06T14:17:03.448 INFO:tasks.workunit.client.1.smithi086.stdout:1/208: dread - db/f39 zero size 2024-04-06T14:17:03.449 INFO:tasks.workunit.client.0.smithi032.stdout:6/436: mknod d4/d5/d28/d65/d6e/c91 0 2024-04-06T14:17:03.449 INFO:tasks.workunit.client.0.smithi032.stdout:8/386: creat d3/d8/d12/d2f/d4a/d60/d90/f91 x:0 0 0 2024-04-06T14:17:03.450 INFO:tasks.workunit.client.0.smithi032.stdout:1/531: getdents d0/d9/d94 0 2024-04-06T14:17:03.456 INFO:tasks.workunit.client.0.smithi032.stdout:2/439: link d0/d6e/f86 d0/d30/d39/d59/d67/d74/d78/d57/f96 0 2024-04-06T14:17:03.458 INFO:tasks.workunit.client.0.smithi032.stdout:8/387: symlink d3/d8/d3a/l92 0 2024-04-06T14:17:03.458 INFO:tasks.workunit.client.0.smithi032.stdout:6/437: write d4/d5/d28/f4f [366748,121416] 0 2024-04-06T14:17:03.458 INFO:tasks.workunit.client.1.smithi086.stdout:4/157: dwrite d0/de/f11 [0,4194304] 0 2024-04-06T14:17:03.458 INFO:tasks.workunit.client.1.smithi086.stdout:2/205: creat d0/d20/d27/d2b/d35/f4e x:0 0 0 2024-04-06T14:17:03.460 INFO:tasks.workunit.client.0.smithi032.stdout:5/427: dwrite d16/d18/d33/d35/f6a [0,4194304] 0 2024-04-06T14:17:03.461 INFO:tasks.workunit.client.1.smithi086.stdout:0/229: creat d0/d1/d2/d1a/f56 x:0 0 0 2024-04-06T14:17:03.464 INFO:tasks.workunit.client.0.smithi032.stdout:7/456: dwrite d16/d20/d26/f39 [0,4194304] 0 2024-04-06T14:17:03.465 INFO:tasks.workunit.client.0.smithi032.stdout:2/440: unlink d0/d30/d39/d59/d67/d80/f88 0 2024-04-06T14:17:03.466 INFO:tasks.workunit.client.1.smithi086.stdout:8/340: symlink d0/d71/d16/d39/l76 0 2024-04-06T14:17:03.466 INFO:tasks.workunit.client.1.smithi086.stdout:0/230: stat d0/d15 0 2024-04-06T14:17:03.466 INFO:tasks.workunit.client.0.smithi032.stdout:2/441: write d0/d5a/f21 [665887,62115] 0 2024-04-06T14:17:03.468 INFO:tasks.workunit.client.1.smithi086.stdout:8/341: read d0/d13/f40 [111814,126313] 0 2024-04-06T14:17:03.469 INFO:tasks.workunit.client.0.smithi032.stdout:7/457: creat d16/d20/d26/d5f/fa2 x:0 0 0 2024-04-06T14:17:03.469 INFO:tasks.workunit.client.0.smithi032.stdout:2/442: write d0/d30/f36 [715045,85241] 0 2024-04-06T14:17:03.470 INFO:tasks.workunit.client.0.smithi032.stdout:6/438: creat d4/d5/d28/d65/f92 x:0 0 0 2024-04-06T14:17:03.470 INFO:tasks.workunit.client.0.smithi032.stdout:1/532: getdents d0/d9/d1f/d7c/d8f 0 2024-04-06T14:17:03.472 INFO:tasks.workunit.client.0.smithi032.stdout:2/443: write d0/d5a/f8e [1568966,83920] 0 2024-04-06T14:17:03.474 INFO:tasks.workunit.client.1.smithi086.stdout:4/158: creat d0/d15/d20/f32 x:0 0 0 2024-04-06T14:17:03.475 INFO:tasks.workunit.client.0.smithi032.stdout:2/444: write d0/f8a [600813,12573] 0 2024-04-06T14:17:03.475 INFO:tasks.workunit.client.0.smithi032.stdout:8/388: rename d3/d8/d7/d1a/f54 to d3/d8/f93 0 2024-04-06T14:17:03.479 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:03 smithi032 ceph-mon[39730]: pgmap v19: 65 pgs: 65 active+clean; 1.3 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 19 MiB/s rd, 153 MiB/s wr, 251 op/s 2024-04-06T14:17:03.480 INFO:tasks.workunit.client.1.smithi086.stdout:3/261: sync 2024-04-06T14:17:03.480 INFO:tasks.workunit.client.0.smithi032.stdout:1/533: unlink d0/d10/f26 0 2024-04-06T14:17:03.481 INFO:tasks.workunit.client.1.smithi086.stdout:8/342: rmdir d0/d71/d16/d39 39 2024-04-06T14:17:03.481 INFO:tasks.workunit.client.1.smithi086.stdout:6/203: getdents d6/d9 0 2024-04-06T14:17:03.481 INFO:tasks.workunit.client.0.smithi032.stdout:8/389: dread - d3/d8/d7/d1a/f72 zero size 2024-04-06T14:17:03.481 INFO:tasks.workunit.client.0.smithi032.stdout:6/439: dwrite d4/d5/d28/f41 [0,4194304] 0 2024-04-06T14:17:03.482 INFO:tasks.workunit.client.1.smithi086.stdout:4/159: dread d0/d15/d19/f1c [0,4194304] 0 2024-04-06T14:17:03.483 INFO:tasks.workunit.client.1.smithi086.stdout:2/206: creat d0/d20/d27/d2b/d35/d36/d4d/f4f x:0 0 0 2024-04-06T14:17:03.483 INFO:tasks.workunit.client.0.smithi032.stdout:7/458: creat d16/d20/d48/fa3 x:0 0 0 2024-04-06T14:17:03.483 INFO:tasks.workunit.client.1.smithi086.stdout:6/204: read d6/d9/db/f2f [3931352,14489] 0 2024-04-06T14:17:03.485 INFO:tasks.workunit.client.0.smithi032.stdout:7/459: truncate d16/d20/d26/d5f/f91 142306 0 2024-04-06T14:17:03.486 INFO:tasks.workunit.client.0.smithi032.stdout:5/428: rename d16/l36 to d16/d18/d33/d45/d52/d39/l91 0 2024-04-06T14:17:03.489 INFO:tasks.workunit.client.1.smithi086.stdout:4/160: write d0/de/f11 [7356965,119424] 0 2024-04-06T14:17:03.490 INFO:tasks.workunit.client.0.smithi032.stdout:8/390: creat d3/d8/d12/d2f/d4a/f94 x:0 0 0 2024-04-06T14:17:03.491 INFO:tasks.workunit.client.0.smithi032.stdout:6/440: symlink d4/d5/d35/l93 0 2024-04-06T14:17:03.495 INFO:tasks.workunit.client.1.smithi086.stdout:6/205: chown d6/d13/d1a/d22/f3f 1 1 2024-04-06T14:17:03.497 INFO:tasks.workunit.client.1.smithi086.stdout:3/262: dwrite d5/d26/f3c [0,4194304] 0 2024-04-06T14:17:03.498 INFO:tasks.workunit.client.0.smithi032.stdout:1/534: symlink d0/lae 0 2024-04-06T14:17:03.498 INFO:tasks.workunit.client.1.smithi086.stdout:8/343: dwrite d0/db/f5e [0,4194304] 0 2024-04-06T14:17:03.498 INFO:tasks.workunit.client.1.smithi086.stdout:4/161: symlink d0/d15/d2b/l33 0 2024-04-06T14:17:03.499 INFO:tasks.workunit.client.0.smithi032.stdout:8/391: truncate d3/d8/d7/d1a/f30 2491365 0 2024-04-06T14:17:03.499 INFO:tasks.workunit.client.0.smithi032.stdout:1/535: readlink d0/da/l79 0 2024-04-06T14:17:03.499 INFO:tasks.workunit.client.1.smithi086.stdout:3/263: fdatasync d5/f37 0 2024-04-06T14:17:03.501 INFO:tasks.workunit.client.0.smithi032.stdout:2/445: creat d0/d5a/f97 x:0 0 0 2024-04-06T14:17:03.502 INFO:tasks.workunit.client.1.smithi086.stdout:8/344: readlink d0/d71/d16/d52/l59 0 2024-04-06T14:17:03.503 INFO:tasks.workunit.client.0.smithi032.stdout:5/429: link d16/d18/d33/d35/f42 d16/d18/d33/d45/d52/f92 0 2024-04-06T14:17:03.504 INFO:tasks.workunit.client.1.smithi086.stdout:3/264: write d5/d33/f32 [4642707,27922] 0 2024-04-06T14:17:03.504 INFO:tasks.workunit.client.1.smithi086.stdout:3/265: write d5/f2d [450482,28372] 0 2024-04-06T14:17:03.505 INFO:tasks.workunit.client.0.smithi032.stdout:6/441: mknod d4/d5/d8/c94 0 2024-04-06T14:17:03.508 INFO:tasks.workunit.client.0.smithi032.stdout:7/460: dwrite d16/d20/d54/f82 [0,4194304] 0 2024-04-06T14:17:03.508 INFO:tasks.workunit.client.1.smithi086.stdout:7/221: write d7/d1d/fc [499578,3010] 0 2024-04-06T14:17:03.509 INFO:tasks.workunit.client.1.smithi086.stdout:8/345: truncate d0/d3b/f4c 652968 0 2024-04-06T14:17:03.511 INFO:tasks.workunit.client.0.smithi032.stdout:8/392: dread - d3/d8/d12/d2f/d46/f55 zero size 2024-04-06T14:17:03.513 INFO:tasks.workunit.client.1.smithi086.stdout:6/206: dread d6/d9/db/f2f [0,4194304] 0 2024-04-06T14:17:03.514 INFO:tasks.workunit.client.1.smithi086.stdout:1/209: truncate f7 2295471 0 2024-04-06T14:17:03.515 INFO:tasks.workunit.client.0.smithi032.stdout:1/536: write d0/da/f45 [407782,81696] 0 2024-04-06T14:17:03.516 INFO:tasks.workunit.client.0.smithi032.stdout:2/446: dread d0/f8a [0,4194304] 0 2024-04-06T14:17:03.520 INFO:tasks.workunit.client.1.smithi086.stdout:4/162: dwrite d0/d15/d20/f22 [0,4194304] 0 2024-04-06T14:17:03.520 INFO:tasks.workunit.client.1.smithi086.stdout:2/207: truncate d0/d20/d44/f49 942933 0 2024-04-06T14:17:03.521 INFO:tasks.workunit.client.0.smithi032.stdout:4/463: dwrite d0/df/d23/f4a [4194304,4194304] 0 2024-04-06T14:17:03.521 INFO:tasks.workunit.client.0.smithi032.stdout:8/393: unlink d3/d8/d7/d1a/f61 0 2024-04-06T14:17:03.523 INFO:tasks.workunit.client.0.smithi032.stdout:8/394: fdatasync d3/d8/d7/d1a/f72 0 2024-04-06T14:17:03.523 INFO:tasks.workunit.client.1.smithi086.stdout:0/231: dread d0/d1/d4/d7/f14 [0,4194304] 0 2024-04-06T14:17:03.523 INFO:tasks.workunit.client.1.smithi086.stdout:6/207: write d6/d13/d1a/f21 [4521481,70859] 0 2024-04-06T14:17:03.524 INFO:tasks.workunit.client.0.smithi032.stdout:8/395: fdatasync d3/d8/f87 0 2024-04-06T14:17:03.526 INFO:tasks.workunit.client.0.smithi032.stdout:8/396: readlink d3/d8/d7/d14/l73 0 2024-04-06T14:17:03.526 INFO:tasks.workunit.client.1.smithi086.stdout:6/208: dread - d6/d13/d1a/f4b zero size 2024-04-06T14:17:03.527 INFO:tasks.workunit.client.0.smithi032.stdout:5/430: dread f11 [0,4194304] 0 2024-04-06T14:17:03.528 INFO:tasks.workunit.client.0.smithi032.stdout:5/431: chown d16/d58/f6e 145808 1 2024-04-06T14:17:03.528 INFO:tasks.workunit.client.1.smithi086.stdout:3/266: dwrite d5/df/f2c [0,4194304] 0 2024-04-06T14:17:03.529 INFO:tasks.workunit.client.0.smithi032.stdout:5/432: read d16/d18/d33/d35/f42 [908076,119756] 0 2024-04-06T14:17:03.538 INFO:tasks.workunit.client.1.smithi086.stdout:7/222: mkdir d7/d10/d11/d1c/d24/d2c/d3c/d52 0 2024-04-06T14:17:03.538 INFO:tasks.workunit.client.1.smithi086.stdout:1/210: write db/d11/d1a/f25 [786352,7993] 0 2024-04-06T14:17:03.540 INFO:tasks.workunit.client.1.smithi086.stdout:0/232: creat d0/d15/d26/d33/f57 x:0 0 0 2024-04-06T14:17:03.541 INFO:tasks.workunit.client.0.smithi032.stdout:1/537: unlink d0/f36 0 2024-04-06T14:17:03.542 INFO:tasks.workunit.client.0.smithi032.stdout:6/442: mknod d4/d5/d8/d1f/d51/d5e/c95 0 2024-04-06T14:17:03.544 INFO:tasks.workunit.client.1.smithi086.stdout:2/208: symlink d0/l50 0 2024-04-06T14:17:03.545 INFO:tasks.workunit.client.1.smithi086.stdout:2/209: chown d0/d20/d27/d2b/f45 7856450 1 2024-04-06T14:17:03.548 INFO:tasks.workunit.client.0.smithi032.stdout:2/447: symlink d0/d5a/d3a/l98 0 2024-04-06T14:17:03.551 INFO:tasks.workunit.client.1.smithi086.stdout:3/267: dwrite d5/f37 [0,4194304] 0 2024-04-06T14:17:03.551 INFO:tasks.workunit.client.0.smithi032.stdout:4/464: mkdir d0/df/d2b/d66/da1 0 2024-04-06T14:17:03.552 INFO:tasks.workunit.client.1.smithi086.stdout:1/211: unlink db/d11/d1a/f36 0 2024-04-06T14:17:03.553 INFO:tasks.workunit.client.0.smithi032.stdout:5/433: creat d16/d18/d33/d45/d52/d39/d86/f93 x:0 0 0 2024-04-06T14:17:03.554 INFO:tasks.workunit.client.0.smithi032.stdout:2/448: mknod d0/d30/d39/d59/d67/d80/c99 0 2024-04-06T14:17:03.555 INFO:tasks.workunit.client.1.smithi086.stdout:2/210: creat d0/df/d48/f51 x:0 0 0 2024-04-06T14:17:03.555 INFO:tasks.workunit.client.0.smithi032.stdout:5/434: dread - d16/d18/f1c zero size 2024-04-06T14:17:03.555 INFO:tasks.workunit.client.1.smithi086.stdout:2/211: dread - d0/d20/d27/d2b/f45 zero size 2024-04-06T14:17:03.559 INFO:tasks.workunit.client.1.smithi086.stdout:4/163: getdents d0/d2c/d30 0 2024-04-06T14:17:03.561 INFO:tasks.workunit.client.0.smithi032.stdout:4/465: dread d0/df/d76/f61 [0,4194304] 0 2024-04-06T14:17:03.562 INFO:tasks.workunit.client.1.smithi086.stdout:2/212: dread d0/df/f1a [0,4194304] 0 2024-04-06T14:17:03.563 INFO:tasks.workunit.client.1.smithi086.stdout:2/213: chown d0/l50 97499166 1 2024-04-06T14:17:03.563 INFO:tasks.workunit.client.0.smithi032.stdout:4/466: chown d0/df/d76 1139349 1 2024-04-06T14:17:03.565 INFO:tasks.workunit.client.1.smithi086.stdout:4/164: unlink d0/d2c/d30/d24/c28 0 2024-04-06T14:17:03.566 INFO:tasks.workunit.client.1.smithi086.stdout:7/223: link d7/d10/d11/c1a d7/d10/d11/d1c/c53 0 2024-04-06T14:17:03.566 INFO:tasks.workunit.client.1.smithi086.stdout:4/165: dread - d0/d15/d20/f32 zero size 2024-04-06T14:17:03.567 INFO:tasks.workunit.client.0.smithi032.stdout:4/467: mknod d0/df/d1f/d2e/ca2 0 2024-04-06T14:17:03.567 INFO:tasks.workunit.client.0.smithi032.stdout:2/449: dwrite d0/d5a/f4c [0,4194304] 0 2024-04-06T14:17:03.567 INFO:tasks.workunit.client.0.smithi032.stdout:3/418: sync 2024-04-06T14:17:03.567 INFO:tasks.workunit.client.1.smithi086.stdout:4/166: fdatasync d0/d15/d20/f27 0 2024-04-06T14:17:03.568 INFO:tasks.workunit.client.1.smithi086.stdout:7/224: mknod d7/d10/d11/c54 0 2024-04-06T14:17:03.569 INFO:tasks.workunit.client.0.smithi032.stdout:5/435: dread d16/d18/f30 [0,4194304] 0 2024-04-06T14:17:03.572 INFO:tasks.workunit.client.0.smithi032.stdout:3/419: write d3/d65/d89/f97 [675045,120306] 0 2024-04-06T14:17:03.574 INFO:tasks.workunit.client.0.smithi032.stdout:5/436: dread - d16/d18/d33/d45/f85 zero size 2024-04-06T14:17:03.576 INFO:tasks.workunit.client.0.smithi032.stdout:4/468: mkdir d0/df/d23/d31/d50/d5e/da3 0 2024-04-06T14:17:03.576 INFO:tasks.workunit.client.0.smithi032.stdout:7/461: dwrite fe [4194304,4194304] 0 2024-04-06T14:17:03.577 INFO:tasks.workunit.client.0.smithi032.stdout:4/469: chown d0/d82/l7f 1466282180 1 2024-04-06T14:17:03.580 INFO:tasks.workunit.client.1.smithi086.stdout:5/230: sync 2024-04-06T14:17:03.580 INFO:tasks.workunit.client.1.smithi086.stdout:9/199: sync 2024-04-06T14:17:03.582 INFO:tasks.workunit.client.0.smithi032.stdout:3/420: mkdir d3/d65/d7d/d9c/dad 0 2024-04-06T14:17:03.584 INFO:tasks.workunit.client.1.smithi086.stdout:9/200: read - d5/d18/f3f zero size 2024-04-06T14:17:03.584 INFO:tasks.workunit.client.0.smithi032.stdout:5/437: write d16/d18/d33/f78 [770190,58269] 0 2024-04-06T14:17:03.585 INFO:tasks.workunit.client.1.smithi086.stdout:7/225: dwrite d7/d10/d11/d1c/d24/d2c/f39 [0,4194304] 0 2024-04-06T14:17:03.586 INFO:tasks.workunit.client.0.smithi032.stdout:7/462: dread d16/d20/f5e [0,4194304] 0 2024-04-06T14:17:03.586 INFO:tasks.workunit.client.1.smithi086.stdout:4/167: dread d0/f2 [0,4194304] 0 2024-04-06T14:17:03.587 INFO:tasks.workunit.client.1.smithi086.stdout:5/231: write d0/d19/f26 [548478,105295] 0 2024-04-06T14:17:03.587 INFO:tasks.workunit.client.0.smithi032.stdout:4/470: truncate d0/df/d23/d31/f68 655335 0 2024-04-06T14:17:03.587 INFO:tasks.workunit.client.0.smithi032.stdout:2/450: link d0/f22 d0/d30/d39/d59/d67/d74/f9a 0 2024-04-06T14:17:03.588 INFO:tasks.workunit.client.1.smithi086.stdout:0/233: truncate d0/d1/d1b/f39 1263960 0 2024-04-06T14:17:03.590 INFO:tasks.workunit.client.1.smithi086.stdout:6/209: dwrite d6/f7 [0,4194304] 0 2024-04-06T14:17:03.590 INFO:tasks.workunit.client.1.smithi086.stdout:3/268: truncate d5/f2d 236140 0 2024-04-06T14:17:03.591 INFO:tasks.workunit.client.0.smithi032.stdout:3/421: creat d3/fae x:0 0 0 2024-04-06T14:17:03.591 INFO:tasks.workunit.client.1.smithi086.stdout:4/168: fdatasync d0/d2c/d30/d24/f2d 0 2024-04-06T14:17:03.593 INFO:tasks.workunit.client.0.smithi032.stdout:4/471: mknod d0/ca4 0 2024-04-06T14:17:03.593 INFO:tasks.workunit.client.1.smithi086.stdout:0/234: write d0/f3f [3208664,30037] 0 2024-04-06T14:17:03.594 INFO:tasks.workunit.client.0.smithi032.stdout:7/463: symlink d16/d20/d63/d9d/la4 0 2024-04-06T14:17:03.596 INFO:tasks.workunit.client.0.smithi032.stdout:6/443: dread d4/d5/f6a [0,4194304] 0 2024-04-06T14:17:03.596 INFO:tasks.workunit.client.0.smithi032.stdout:4/472: chown d0/df/d2b/d46/d58 7 1 2024-04-06T14:17:03.596 INFO:tasks.workunit.client.0.smithi032.stdout:8/397: dread d3/d8/d7/d1a/f30 [0,4194304] 0 2024-04-06T14:17:03.597 INFO:tasks.workunit.client.0.smithi032.stdout:5/438: getdents d16/d18/d33/d35 0 2024-04-06T14:17:03.602 INFO:tasks.workunit.client.0.smithi032.stdout:7/464: chown d16/d20/d26/d2d/d30/d3e/d96 243471 1 2024-04-06T14:17:03.602 INFO:tasks.workunit.client.0.smithi032.stdout:6/444: write d4/d5/d28/d54/f89 [925970,49867] 0 2024-04-06T14:17:03.604 INFO:tasks.workunit.client.1.smithi086.stdout:4/169: mkdir d0/d15/d20/d34 0 2024-04-06T14:17:03.605 INFO:tasks.workunit.client.1.smithi086.stdout:3/269: link d5/d26/f3d d5/d33/f47 0 2024-04-06T14:17:03.607 INFO:tasks.workunit.client.1.smithi086.stdout:4/170: write d0/de/f11 [4049102,106447] 0 2024-04-06T14:17:03.608 INFO:tasks.workunit.client.1.smithi086.stdout:6/210: dwrite d6/f40 [0,4194304] 0 2024-04-06T14:17:03.610 INFO:tasks.workunit.client.1.smithi086.stdout:9/201: dread d5/d10/d14/d19/f1e [0,4194304] 0 2024-04-06T14:17:03.610 INFO:tasks.workunit.client.0.smithi032.stdout:6/445: mkdir d4/d5/d19/d96 0 2024-04-06T14:17:03.610 INFO:tasks.workunit.client.0.smithi032.stdout:4/473: dread d0/df/d1f/d42/d45/f53 [0,4194304] 0 2024-04-06T14:17:03.610 INFO:tasks.workunit.client.0.smithi032.stdout:5/439: getdents d16/d58/d54/d74/d8f 0 2024-04-06T14:17:03.612 INFO:tasks.workunit.client.1.smithi086.stdout:3/270: write d5/f3e [719879,77104] 0 2024-04-06T14:17:03.613 INFO:tasks.workunit.client.0.smithi032.stdout:2/451: dread d0/d5a/f25 [0,4194304] 0 2024-04-06T14:17:03.613 INFO:tasks.workunit.client.1.smithi086.stdout:6/211: rmdir d6/d20 39 2024-04-06T14:17:03.614 INFO:tasks.workunit.client.0.smithi032.stdout:5/440: dread - d16/d58/f7a zero size 2024-04-06T14:17:03.615 INFO:tasks.workunit.client.1.smithi086.stdout:3/271: write d5/d33/f32 [2353064,21379] 0 2024-04-06T14:17:03.615 INFO:tasks.workunit.client.1.smithi086.stdout:1/212: dread f2 [0,4194304] 0 2024-04-06T14:17:03.615 INFO:tasks.workunit.client.1.smithi086.stdout:9/202: fsync d5/d10/d14/d19/f1e 0 2024-04-06T14:17:03.616 INFO:tasks.workunit.client.0.smithi032.stdout:7/465: creat d16/d20/d7a/fa5 x:0 0 0 2024-04-06T14:17:03.617 INFO:tasks.workunit.client.0.smithi032.stdout:4/474: fdatasync d0/df/d23/f4a 0 2024-04-06T14:17:03.618 INFO:tasks.workunit.client.1.smithi086.stdout:0/235: getdents d0/d15/d26/d33 0 2024-04-06T14:17:03.619 INFO:tasks.workunit.client.0.smithi032.stdout:7/466: write f10 [2195825,26860] 0 2024-04-06T14:17:03.620 INFO:tasks.workunit.client.0.smithi032.stdout:6/446: mknod d4/d5/d14/d80/c97 0 2024-04-06T14:17:03.620 INFO:tasks.workunit.client.0.smithi032.stdout:8/398: dwrite d3/d8/d12/d2f/d4a/f94 [0,4194304] 0 2024-04-06T14:17:03.621 INFO:tasks.workunit.client.1.smithi086.stdout:3/272: read d5/fe [3918032,99413] 0 2024-04-06T14:17:03.622 INFO:tasks.workunit.client.0.smithi032.stdout:3/422: dread d3/d3b/d3c/f5f [0,4194304] 0 2024-04-06T14:17:03.624 INFO:tasks.workunit.client.1.smithi086.stdout:3/273: write d5/f20 [1098857,15498] 0 2024-04-06T14:17:03.625 INFO:tasks.workunit.client.1.smithi086.stdout:9/203: rmdir d5/d13 39 2024-04-06T14:17:03.625 INFO:tasks.workunit.client.1.smithi086.stdout:5/232: truncate d0/d19/f26 3374927 0 2024-04-06T14:17:03.627 INFO:tasks.workunit.client.0.smithi032.stdout:3/423: write d3/d3b/d3c/f90 [3674085,103766] 0 2024-04-06T14:17:03.627 INFO:tasks.workunit.client.1.smithi086.stdout:1/213: dwrite db/d13/f1e [0,4194304] 0 2024-04-06T14:17:03.628 INFO:tasks.workunit.client.1.smithi086.stdout:3/274: chown d5/d36/f3b 904698978 1 2024-04-06T14:17:03.629 INFO:tasks.workunit.client.1.smithi086.stdout:5/233: fdatasync d0/d16/d18/d21/d23/f1f 0 2024-04-06T14:17:03.629 INFO:tasks.workunit.client.0.smithi032.stdout:4/475: creat d0/df/d1f/d2e/fa5 x:0 0 0 2024-04-06T14:17:03.630 INFO:tasks.workunit.client.0.smithi032.stdout:3/424: fsync d3/d16/f72 0 2024-04-06T14:17:03.631 INFO:tasks.workunit.client.0.smithi032.stdout:2/452: symlink d0/d30/d39/d59/d67/d80/l9b 0 2024-04-06T14:17:03.633 INFO:tasks.workunit.client.0.smithi032.stdout:8/399: read d3/d8/d7/d1a/f24 [2068294,70562] 0 2024-04-06T14:17:03.635 INFO:tasks.workunit.client.1.smithi086.stdout:9/204: symlink d5/d10/l4a 0 2024-04-06T14:17:03.636 INFO:tasks.workunit.client.0.smithi032.stdout:7/467: dwrite d16/d20/d26/d5f/f8a [0,4194304] 0 2024-04-06T14:17:03.638 INFO:tasks.workunit.client.0.smithi032.stdout:5/441: rename d16/d18/d33/d45/d52/d8d to d16/d58/d54/d94 0 2024-04-06T14:17:03.641 INFO:tasks.workunit.client.0.smithi032.stdout:4/476: fdatasync d0/df/d2b/f3b 0 2024-04-06T14:17:03.641 INFO:tasks.workunit.client.1.smithi086.stdout:9/205: fsync f0 0 2024-04-06T14:17:03.642 INFO:tasks.workunit.client.0.smithi032.stdout:8/400: write d3/d8/d7/d1a/f78 [901316,120380] 0 2024-04-06T14:17:03.642 INFO:tasks.workunit.client.1.smithi086.stdout:5/234: creat d0/d16/d18/d2a/f46 x:0 0 0 2024-04-06T14:17:03.644 INFO:tasks.workunit.client.0.smithi032.stdout:7/468: chown d16/d20/f37 11275 1 2024-04-06T14:17:03.647 INFO:tasks.workunit.client.1.smithi086.stdout:0/236: dwrite d0/d15/d4e/d48/f31 [0,4194304] 0 2024-04-06T14:17:03.648 INFO:tasks.workunit.client.0.smithi032.stdout:3/425: dwrite d3/d16/d63/f83 [0,4194304] 0 2024-04-06T14:17:03.648 INFO:tasks.workunit.client.0.smithi032.stdout:7/469: dread - d16/d20/d26/d5f/f90 zero size 2024-04-06T14:17:03.649 INFO:tasks.workunit.client.1.smithi086.stdout:9/206: chown d5/d10/d14/d19/d25 1254 1 2024-04-06T14:17:03.649 INFO:tasks.workunit.client.0.smithi032.stdout:2/453: dwrite d0/d5a/f41 [0,4194304] 0 2024-04-06T14:17:03.649 INFO:tasks.workunit.client.1.smithi086.stdout:9/207: chown d5/d10/d14/f22 221406545 1 2024-04-06T14:17:03.650 INFO:tasks.workunit.client.1.smithi086.stdout:5/235: getdents d0/d16/d18 0 2024-04-06T14:17:03.652 INFO:tasks.workunit.client.1.smithi086.stdout:9/208: dread - d5/d18/f32 zero size 2024-04-06T14:17:03.652 INFO:tasks.workunit.client.1.smithi086.stdout:5/236: chown d0/d16/d18/l10 907591 1 2024-04-06T14:17:03.653 INFO:tasks.workunit.client.1.smithi086.stdout:5/237: truncate d0/f3f 239775 0 2024-04-06T14:17:03.660 INFO:tasks.workunit.client.0.smithi032.stdout:5/442: mkdir d16/d58/d54/d74/d8f/d95 0 2024-04-06T14:17:03.662 INFO:tasks.workunit.client.0.smithi032.stdout:4/477: dwrite d0/df/d23/f40 [4194304,4194304] 0 2024-04-06T14:17:03.663 INFO:tasks.workunit.client.1.smithi086.stdout:9/209: unlink d5/d13/f35 0 2024-04-06T14:17:03.664 INFO:tasks.workunit.client.0.smithi032.stdout:7/470: rmdir d16/d20/d26/d2d/d30/d3e/d59 0 2024-04-06T14:17:03.667 INFO:tasks.workunit.client.0.smithi032.stdout:3/426: rename d3/d16/f68 to d3/faf 0 2024-04-06T14:17:03.669 INFO:tasks.workunit.client.0.smithi032.stdout:8/401: dwrite d3/d8/f93 [0,4194304] 0 2024-04-06T14:17:03.669 INFO:tasks.workunit.client.1.smithi086.stdout:9/210: link d5/d18/f48 d5/d10/d14/d19/f4b 0 2024-04-06T14:17:03.670 INFO:tasks.workunit.client.0.smithi032.stdout:4/478: creat d0/df/d2b/d43/fa6 x:0 0 0 2024-04-06T14:17:03.672 INFO:tasks.workunit.client.0.smithi032.stdout:7/471: mkdir d16/d20/d7a/da6 0 2024-04-06T14:17:03.672 INFO:tasks.workunit.client.0.smithi032.stdout:8/402: chown d3/d8/d12/f4d 118714 1 2024-04-06T14:17:03.673 INFO:tasks.workunit.client.1.smithi086.stdout:9/211: symlink d5/d13/d1a/l4c 0 2024-04-06T14:17:03.683 INFO:tasks.workunit.client.1.smithi086.stdout:1/214: dread db/fc [0,4194304] 0 2024-04-06T14:17:03.683 INFO:tasks.workunit.client.0.smithi032.stdout:8/403: creat d3/d8/d12/d2f/d4a/d60/d82/f95 x:0 0 0 2024-04-06T14:17:03.684 INFO:tasks.workunit.client.0.smithi032.stdout:3/427: dread d3/d16/f18 [4194304,4194304] 0 2024-04-06T14:17:03.685 INFO:tasks.workunit.client.0.smithi032.stdout:7/472: rename d16/d20/d48/d8d/d95 to d16/d20/d26/d2d/d30/d3e/da7 0 2024-04-06T14:17:03.686 INFO:tasks.workunit.client.1.smithi086.stdout:9/212: dwrite d5/d18/f3f [0,4194304] 0 2024-04-06T14:17:03.686 INFO:tasks.workunit.client.1.smithi086.stdout:1/215: mkdir db/d11/d42 0 2024-04-06T14:17:03.686 INFO:tasks.workunit.client.0.smithi032.stdout:0/320: sync 2024-04-06T14:17:03.688 INFO:tasks.workunit.client.0.smithi032.stdout:4/479: creat d0/d82/d8e/fa7 x:0 0 0 2024-04-06T14:17:03.688 INFO:tasks.workunit.client.0.smithi032.stdout:6/447: truncate d4/f13 6000625 0 2024-04-06T14:17:03.689 INFO:tasks.workunit.client.1.smithi086.stdout:9/213: write d5/d10/d14/d19/d1b/f41 [3400491,44379] 0 2024-04-06T14:17:03.689 INFO:tasks.workunit.client.0.smithi032.stdout:0/321: chown d3/f20 27 1 2024-04-06T14:17:03.690 INFO:tasks.workunit.client.1.smithi086.stdout:9/214: dread - d5/d10/d14/d19/d25/f3e zero size 2024-04-06T14:17:03.691 INFO:tasks.workunit.client.1.smithi086.stdout:3/275: truncate d5/d33/f32 1989000 0 2024-04-06T14:17:03.691 INFO:tasks.workunit.client.1.smithi086.stdout:3/276: stat d5/d36 0 2024-04-06T14:17:03.692 INFO:tasks.workunit.client.0.smithi032.stdout:4/480: read - d0/df/d1f/d5a/f8f zero size 2024-04-06T14:17:03.692 INFO:tasks.workunit.client.1.smithi086.stdout:3/277: chown d5/c12 1785 1 2024-04-06T14:17:03.693 INFO:tasks.workunit.client.1.smithi086.stdout:1/216: dread db/d11/d1a/f25 [0,4194304] 0 2024-04-06T14:17:03.694 INFO:tasks.workunit.client.0.smithi032.stdout:3/428: creat d3/d16/fb0 x:0 0 0 2024-04-06T14:17:03.694 INFO:tasks.workunit.client.0.smithi032.stdout:8/404: dwrite d3/d8/d12/f38 [0,4194304] 0 2024-04-06T14:17:03.694 INFO:tasks.workunit.client.1.smithi086.stdout:9/215: unlink d5/ce 0 2024-04-06T14:17:03.695 INFO:tasks.workunit.client.0.smithi032.stdout:7/473: symlink d16/d20/d26/d70/la8 0 2024-04-06T14:17:03.697 INFO:tasks.workunit.client.1.smithi086.stdout:5/238: truncate d0/d16/f39 838816 0 2024-04-06T14:17:03.698 INFO:tasks.workunit.client.0.smithi032.stdout:0/322: symlink d3/da/d25/d57/l70 0 2024-04-06T14:17:03.699 INFO:tasks.workunit.client.0.smithi032.stdout:4/481: write d0/df/d23/f3d [2374811,104248] 0 2024-04-06T14:17:03.700 INFO:tasks.workunit.client.1.smithi086.stdout:1/217: rename db/d13/f29 to db/d11/d1a/d2a/f43 0 2024-04-06T14:17:03.701 INFO:tasks.workunit.client.0.smithi032.stdout:0/323: write d3/f63 [3795138,82757] 0 2024-04-06T14:17:03.701 INFO:tasks.workunit.client.1.smithi086.stdout:9/216: rmdir d5/d10/d14/d19 39 2024-04-06T14:17:03.702 INFO:tasks.workunit.client.0.smithi032.stdout:6/448: dread d4/d5/d35/f39 [0,4194304] 0 2024-04-06T14:17:03.703 INFO:tasks.workunit.client.0.smithi032.stdout:4/482: chown d0/df/d23/c81 2740286 1 2024-04-06T14:17:03.706 INFO:tasks.workunit.client.0.smithi032.stdout:5/443: write d16/d18/d33/d45/d52/f92 [95580,16675] 0 2024-04-06T14:17:03.707 INFO:tasks.workunit.client.0.smithi032.stdout:3/429: dread d3/d65/d89/f97 [0,4194304] 0 2024-04-06T14:17:03.708 INFO:tasks.workunit.client.0.smithi032.stdout:6/449: stat d4/d5/d19 0 2024-04-06T14:17:03.708 INFO:tasks.workunit.client.1.smithi086.stdout:1/218: rename db/d11/f3f to db/d11/d1f/f44 0 2024-04-06T14:17:03.709 INFO:tasks.workunit.client.0.smithi032.stdout:2/454: truncate d0/d30/d39/d59/f5f 1166528 0 2024-04-06T14:17:03.710 INFO:tasks.workunit.client.0.smithi032.stdout:4/483: rmdir d0/df/d23/d31/d50 39 2024-04-06T14:17:03.710 INFO:tasks.workunit.client.1.smithi086.stdout:3/278: creat d5/f48 x:0 0 0 2024-04-06T14:17:03.710 INFO:tasks.workunit.client.1.smithi086.stdout:1/219: dread db/d11/d1a/f25 [0,4194304] 0 2024-04-06T14:17:03.711 INFO:tasks.workunit.client.0.smithi032.stdout:0/324: write d3/da/d25/d36/f39 [2848537,95708] 0 2024-04-06T14:17:03.712 INFO:tasks.workunit.client.1.smithi086.stdout:9/217: dread - d5/d10/d14/d19/d25/f3e zero size 2024-04-06T14:17:03.713 INFO:tasks.workunit.client.0.smithi032.stdout:6/450: chown d4/d5/d14/l22 46330 1 2024-04-06T14:17:03.713 INFO:tasks.workunit.client.0.smithi032.stdout:2/455: chown d0/d30/c54 6120 1 2024-04-06T14:17:03.715 INFO:tasks.workunit.client.1.smithi086.stdout:5/239: getdents d0/d19 0 2024-04-06T14:17:03.715 INFO:tasks.workunit.client.0.smithi032.stdout:7/474: dread d16/d20/d26/d5f/f8a [0,4194304] 0 2024-04-06T14:17:03.715 INFO:tasks.workunit.client.0.smithi032.stdout:3/430: truncate d3/d16/f5c 1608899 0 2024-04-06T14:17:03.716 INFO:tasks.workunit.client.1.smithi086.stdout:1/220: read db/d11/f17 [1080473,112346] 0 2024-04-06T14:17:03.716 INFO:tasks.workunit.client.0.smithi032.stdout:5/444: dread f11 [0,4194304] 0 2024-04-06T14:17:03.716 INFO:tasks.workunit.client.0.smithi032.stdout:8/405: dread d3/d8/d7/fe [0,4194304] 0 2024-04-06T14:17:03.717 INFO:tasks.workunit.client.0.smithi032.stdout:2/456: chown d0/d30/d39/d59 1934563 1 2024-04-06T14:17:03.720 INFO:tasks.workunit.client.0.smithi032.stdout:2/457: stat d0/d30/f36 0 2024-04-06T14:17:03.722 INFO:tasks.workunit.client.1.smithi086.stdout:1/221: creat db/d11/d24/f45 x:0 0 0 2024-04-06T14:17:03.723 INFO:tasks.workunit.client.1.smithi086.stdout:9/218: mkdir d5/d10/d14/d19/d4d 0 2024-04-06T14:17:03.723 INFO:tasks.workunit.client.1.smithi086.stdout:0/237: dwrite d0/d1/d1b/f39 [0,4194304] 0 2024-04-06T14:17:03.725 INFO:tasks.workunit.client.0.smithi032.stdout:0/325: dread d3/da/f1d [0,4194304] 0 2024-04-06T14:17:03.725 INFO:tasks.workunit.client.0.smithi032.stdout:3/431: rename d3/d3b to d3/d65/d7d/d9c/dad/db1 0 2024-04-06T14:17:03.726 INFO:tasks.workunit.client.0.smithi032.stdout:2/458: write d0/d30/d39/d59/d67/d74/d78/d57/f90 [979322,60416] 0 2024-04-06T14:17:03.726 INFO:tasks.workunit.client.1.smithi086.stdout:3/279: dwrite d5/df/f2f [0,4194304] 0 2024-04-06T14:17:03.728 INFO:tasks.workunit.client.1.smithi086.stdout:1/222: fdatasync db/d11/d24/f45 0 2024-04-06T14:17:03.731 INFO:tasks.workunit.client.0.smithi032.stdout:6/451: getdents d4/d5/d28 0 2024-04-06T14:17:03.735 INFO:tasks.workunit.client.1.smithi086.stdout:3/280: mkdir d5/d36/d49 0 2024-04-06T14:17:03.737 INFO:tasks.workunit.client.0.smithi032.stdout:3/432: dwrite d3/d65/f39 [0,4194304] 0 2024-04-06T14:17:03.738 INFO:tasks.workunit.client.1.smithi086.stdout:1/223: rename db/d11/d1f/c3a to db/d11/d1d/c46 0 2024-04-06T14:17:03.738 INFO:tasks.workunit.client.1.smithi086.stdout:1/224: stat c6 0 2024-04-06T14:17:03.739 INFO:tasks.workunit.client.0.smithi032.stdout:3/433: chown d3/c76 7846 1 2024-04-06T14:17:03.740 INFO:tasks.workunit.client.0.smithi032.stdout:2/459: rmdir d0/d30/d39/d59/d67/d74/d78 39 2024-04-06T14:17:03.741 INFO:tasks.workunit.client.1.smithi086.stdout:9/219: creat d5/d10/d14/f4e x:0 0 0 2024-04-06T14:17:03.741 INFO:tasks.workunit.client.0.smithi032.stdout:6/452: rmdir d4/d5/d14 39 2024-04-06T14:17:03.741 INFO:tasks.workunit.client.1.smithi086.stdout:0/238: dwrite d0/d1/d4/d7/f14 [0,4194304] 0 2024-04-06T14:17:03.743 INFO:tasks.workunit.client.0.smithi032.stdout:8/406: link d3/d8/d12/d2f/d46/f74 d3/f96 0 2024-04-06T14:17:03.746 INFO:tasks.workunit.client.0.smithi032.stdout:5/445: dwrite d16/d18/d33/d35/f44 [0,4194304] 0 2024-04-06T14:17:03.746 INFO:tasks.workunit.client.0.smithi032.stdout:2/460: symlink d0/d4f/l9c 0 2024-04-06T14:17:03.748 INFO:tasks.workunit.client.0.smithi032.stdout:7/475: rename d16/d20/d26/d70/c8b to d16/d20/d63/ca9 0 2024-04-06T14:17:03.749 INFO:tasks.workunit.client.0.smithi032.stdout:6/453: creat d4/f98 x:0 0 0 2024-04-06T14:17:03.749 INFO:tasks.workunit.client.0.smithi032.stdout:4/484: dwrite d0/df/d23/d31/f8d [0,4194304] 0 2024-04-06T14:17:03.749 INFO:tasks.workunit.client.1.smithi086.stdout:9/220: read d5/d18/f26 [3387086,105904] 0 2024-04-06T14:17:03.750 INFO:tasks.workunit.client.0.smithi032.stdout:2/461: fdatasync d0/d5a/f8e 0 2024-04-06T14:17:03.751 INFO:tasks.workunit.client.1.smithi086.stdout:1/225: creat db/d11/d1a/d2a/f47 x:0 0 0 2024-04-06T14:17:03.751 INFO:tasks.workunit.client.1.smithi086.stdout:5/240: truncate d0/f1 1235983 0 2024-04-06T14:17:03.752 INFO:tasks.workunit.client.1.smithi086.stdout:5/241: dread - d0/f41 zero size 2024-04-06T14:17:03.752 INFO:tasks.workunit.client.0.smithi032.stdout:0/326: getdents d3/da/d35/d4b 0 2024-04-06T14:17:03.753 INFO:tasks.workunit.client.1.smithi086.stdout:9/221: creat d5/d13/f4f x:0 0 0 2024-04-06T14:17:03.754 INFO:tasks.workunit.client.0.smithi032.stdout:3/434: write d3/d65/d7d/d9c/dad/db1/d3c/f57 [541753,50558] 0 2024-04-06T14:17:03.754 INFO:tasks.workunit.client.1.smithi086.stdout:0/239: symlink d0/d1/d1b/d44/l58 0 2024-04-06T14:17:03.755 INFO:tasks.workunit.client.1.smithi086.stdout:3/281: getdents d5/d33 0 2024-04-06T14:17:03.755 INFO:tasks.workunit.client.1.smithi086.stdout:1/226: stat db/d11/d1a/d2a/f35 0 2024-04-06T14:17:03.756 INFO:tasks.workunit.client.0.smithi032.stdout:3/435: stat d3/d65/d89/l94 0 2024-04-06T14:17:03.759 INFO:tasks.workunit.client.0.smithi032.stdout:7/476: write d16/d20/d26/d5f/f90 [431573,19277] 0 2024-04-06T14:17:03.759 INFO:tasks.workunit.client.0.smithi032.stdout:6/454: mknod d4/c99 0 2024-04-06T14:17:03.759 INFO:tasks.workunit.client.0.smithi032.stdout:8/407: symlink d3/d8/d3a/d4b/l97 0 2024-04-06T14:17:03.761 INFO:tasks.workunit.client.0.smithi032.stdout:4/485: chown d0/l9 244 1 2024-04-06T14:17:03.764 INFO:tasks.workunit.client.0.smithi032.stdout:8/408: write d3/d8/d3a/f42 [4359426,118235] 0 2024-04-06T14:17:03.764 INFO:tasks.workunit.client.0.smithi032.stdout:0/327: rmdir d3/da/d25/d2b/d4a 39 2024-04-06T14:17:03.766 INFO:tasks.workunit.client.0.smithi032.stdout:6/455: symlink d4/d5/d8/d1f/d51/d5e/l9a 0 2024-04-06T14:17:03.769 INFO:tasks.workunit.client.0.smithi032.stdout:3/436: rename d3/l50 to d3/d65/d7d/d9c/dad/db1/lb2 0 2024-04-06T14:17:03.772 INFO:tasks.workunit.client.0.smithi032.stdout:8/409: symlink d3/d8/d12/d2f/l98 0 2024-04-06T14:17:03.775 INFO:tasks.workunit.client.0.smithi032.stdout:3/437: write d3/d65/d7d/d9c/dad/db1/f62 [3765071,21547] 0 2024-04-06T14:17:03.776 INFO:tasks.workunit.client.0.smithi032.stdout:3/438: write d3/d16/fb0 [673048,1450] 0 2024-04-06T14:17:03.776 INFO:tasks.workunit.client.0.smithi032.stdout:1/538: sync 2024-04-06T14:17:03.776 INFO:tasks.workunit.client.0.smithi032.stdout:9/469: sync 2024-04-06T14:17:03.778 INFO:tasks.workunit.client.1.smithi086.stdout:0/240: unlink d0/d1/d1b/c37 0 2024-04-06T14:17:03.779 INFO:tasks.workunit.client.0.smithi032.stdout:7/477: rename d16/l18 to d16/d20/d26/d2d/d30/laa 0 2024-04-06T14:17:03.780 INFO:tasks.workunit.client.0.smithi032.stdout:9/470: read d2/d15/d1e/d41/f63 [4001111,94518] 0 2024-04-06T14:17:03.782 INFO:tasks.workunit.client.0.smithi032.stdout:5/446: truncate d16/d18/d33/d35/f6a 361377 0 2024-04-06T14:17:03.782 INFO:tasks.workunit.client.1.smithi086.stdout:1/227: getdents db/d11/d42 0 2024-04-06T14:17:03.785 INFO:tasks.workunit.client.0.smithi032.stdout:6/456: rename d4/d5/d8/d4b/d86 to d4/d5/d8/d2f/d9b 0 2024-04-06T14:17:03.786 INFO:tasks.workunit.client.1.smithi086.stdout:1/228: write db/d11/d1a/d2a/f35 [958574,86844] 0 2024-04-06T14:17:03.787 INFO:tasks.workunit.client.0.smithi032.stdout:3/439: creat d3/d65/d7d/d9c/dad/fb3 x:0 0 0 2024-04-06T14:17:03.787 INFO:tasks.workunit.client.0.smithi032.stdout:6/457: write d4/f98 [379590,62607] 0 2024-04-06T14:17:03.788 INFO:tasks.workunit.client.1.smithi086.stdout:5/242: write d0/d16/d18/d35/f3a [92469,97714] 0 2024-04-06T14:17:03.788 INFO:tasks.workunit.client.0.smithi032.stdout:7/478: creat d16/d20/d63/d9d/fab x:0 0 0 2024-04-06T14:17:03.790 INFO:tasks.workunit.client.0.smithi032.stdout:0/328: truncate d3/f10 2913601 0 2024-04-06T14:17:03.791 INFO:tasks.workunit.client.1.smithi086.stdout:0/241: dwrite d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:03.791 INFO:tasks.workunit.client.0.smithi032.stdout:8/410: write d3/d8/d12/d2f/d46/f74 [5082995,125332] 0 2024-04-06T14:17:03.793 INFO:tasks.workunit.client.1.smithi086.stdout:1/229: symlink db/d11/d1f/l48 0 2024-04-06T14:17:03.796 INFO:tasks.workunit.client.0.smithi032.stdout:4/486: rename d0/df/d23/d31/d50/d57/d5b/d9c/f9d to d0/df/d2b/fa8 0 2024-04-06T14:17:03.798 INFO:tasks.workunit.client.0.smithi032.stdout:3/440: chown d3/d65/c31 4982 1 2024-04-06T14:17:03.798 INFO:tasks.workunit.client.0.smithi032.stdout:3/441: readlink d3/d16/d63/l8a 0 2024-04-06T14:17:03.799 INFO:tasks.workunit.client.1.smithi086.stdout:5/243: dwrite d0/f4 [0,4194304] 0 2024-04-06T14:17:03.800 INFO:tasks.workunit.client.0.smithi032.stdout:6/458: mkdir d4/d5/d19/d25/d9c 0 2024-04-06T14:17:03.801 INFO:tasks.workunit.client.1.smithi086.stdout:1/230: mknod db/c49 0 2024-04-06T14:17:03.801 INFO:tasks.workunit.client.0.smithi032.stdout:0/329: symlink d3/da/l71 0 2024-04-06T14:17:03.802 INFO:tasks.workunit.client.1.smithi086.stdout:0/242: dread d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:03.802 INFO:tasks.workunit.client.0.smithi032.stdout:1/539: dread d0/d10/d16/d3f/f5d [0,4194304] 0 2024-04-06T14:17:03.806 INFO:tasks.workunit.client.1.smithi086.stdout:0/243: read d0/f16 [306911,46884] 0 2024-04-06T14:17:03.806 INFO:tasks.workunit.client.1.smithi086.stdout:5/244: write d0/d16/d18/d2a/f46 [741438,128468] 0 2024-04-06T14:17:03.807 INFO:tasks.workunit.client.0.smithi032.stdout:0/330: write d3/da/d25/d2b/f40 [90155,128302] 0 2024-04-06T14:17:03.807 INFO:tasks.workunit.client.0.smithi032.stdout:4/487: mknod d0/df/d23/d31/d50/d57/ca9 0 2024-04-06T14:17:03.807 INFO:tasks.workunit.client.0.smithi032.stdout:3/442: rename d3/f2d to d3/d16/d63/fb4 0 2024-04-06T14:17:03.807 INFO:tasks.workunit.client.0.smithi032.stdout:7/479: symlink d16/d20/d26/d2d/d30/d97/lac 0 2024-04-06T14:17:03.807 INFO:tasks.workunit.client.1.smithi086.stdout:0/244: fdatasync d0/d1/d1b/f39 0 2024-04-06T14:17:03.808 INFO:tasks.workunit.client.1.smithi086.stdout:1/231: mknod db/d11/d1a/c4a 0 2024-04-06T14:17:03.809 INFO:tasks.workunit.client.1.smithi086.stdout:5/245: creat d0/d16/f47 x:0 0 0 2024-04-06T14:17:03.809 INFO:tasks.workunit.client.0.smithi032.stdout:5/447: creat d16/f96 x:0 0 0 2024-04-06T14:17:03.811 INFO:tasks.workunit.client.0.smithi032.stdout:5/448: chown f9 23906417 1 2024-04-06T14:17:03.813 INFO:tasks.workunit.client.1.smithi086.stdout:0/245: symlink d0/d1/d4/d7/l59 0 2024-04-06T14:17:03.813 INFO:tasks.workunit.client.1.smithi086.stdout:5/246: dread d0/d16/d18/d21/f45 [0,4194304] 0 2024-04-06T14:17:03.813 INFO:tasks.workunit.client.1.smithi086.stdout:8/346: sync 2024-04-06T14:17:03.814 INFO:tasks.workunit.client.1.smithi086.stdout:0/246: chown d0/f17 0 1 2024-04-06T14:17:03.814 INFO:tasks.workunit.client.0.smithi032.stdout:9/471: getdents d2/d15/d55/d59 0 2024-04-06T14:17:03.814 INFO:tasks.workunit.client.0.smithi032.stdout:5/449: chown d16/d18/d33/d35/l4b 78 1 2024-04-06T14:17:03.814 INFO:tasks.workunit.client.0.smithi032.stdout:3/443: read d3/d65/f66 [2626857,15594] 0 2024-04-06T14:17:03.814 INFO:tasks.workunit.client.0.smithi032.stdout:4/488: creat d0/df/d23/d31/d50/d5e/faa x:0 0 0 2024-04-06T14:17:03.815 INFO:tasks.workunit.client.1.smithi086.stdout:8/347: readlink d0/d71/d16/d52/l70 0 2024-04-06T14:17:03.815 INFO:tasks.workunit.client.1.smithi086.stdout:5/247: write d0/d16/d18/d2a/f46 [384598,48191] 0 2024-04-06T14:17:03.816 INFO:tasks.workunit.client.0.smithi032.stdout:4/489: write d0/df/d1f/d2e/fa5 [177782,125624] 0 2024-04-06T14:17:03.819 INFO:tasks.workunit.client.1.smithi086.stdout:0/247: write d0/d1/d4/de/f3c [291154,83028] 0 2024-04-06T14:17:03.821 INFO:tasks.workunit.client.1.smithi086.stdout:8/348: symlink d0/d3b/l77 0 2024-04-06T14:17:03.822 INFO:tasks.workunit.client.1.smithi086.stdout:0/248: read - d0/d15/f3e zero size 2024-04-06T14:17:03.823 INFO:tasks.workunit.client.1.smithi086.stdout:5/248: mknod d0/d43/c48 0 2024-04-06T14:17:03.824 INFO:tasks.workunit.client.1.smithi086.stdout:0/249: chown d0/d15/d26/f2c 1322389 1 2024-04-06T14:17:03.824 INFO:tasks.workunit.client.0.smithi032.stdout:7/480: dread d16/d20/d26/f31 [0,4194304] 0 2024-04-06T14:17:03.824 INFO:tasks.workunit.client.0.smithi032.stdout:9/472: creat d2/f97 x:0 0 0 2024-04-06T14:17:03.825 INFO:tasks.workunit.client.1.smithi086.stdout:8/349: mkdir d0/d71/d22/d43/d6d/d75/d78 0 2024-04-06T14:17:03.825 INFO:tasks.workunit.client.0.smithi032.stdout:5/450: symlink d16/d58/d54/d74/l97 0 2024-04-06T14:17:03.825 INFO:tasks.workunit.client.1.smithi086.stdout:8/350: chown d0/db/f5e 0 1 2024-04-06T14:17:03.826 INFO:tasks.workunit.client.1.smithi086.stdout:5/249: symlink d0/d16/d18/l49 0 2024-04-06T14:17:03.827 INFO:tasks.workunit.client.0.smithi032.stdout:3/444: symlink d3/d65/lb5 0 2024-04-06T14:17:03.827 INFO:tasks.workunit.client.1.smithi086.stdout:0/250: creat d0/d1/d1b/d44/f5a x:0 0 0 2024-04-06T14:17:03.828 INFO:tasks.workunit.client.1.smithi086.stdout:0/251: fsync d0/d15/f4c 0 2024-04-06T14:17:03.829 INFO:tasks.workunit.client.0.smithi032.stdout:1/540: creat d0/faf x:0 0 0 2024-04-06T14:17:03.829 INFO:tasks.workunit.client.1.smithi086.stdout:8/351: unlink d0/d71/d16/d52/f55 0 2024-04-06T14:17:03.829 INFO:tasks.workunit.client.0.smithi032.stdout:8/411: dread d3/f32 [4194304,4194304] 0 2024-04-06T14:17:03.830 INFO:tasks.workunit.client.0.smithi032.stdout:9/473: dread d2/f53 [0,4194304] 0 2024-04-06T14:17:03.830 INFO:tasks.workunit.client.0.smithi032.stdout:8/412: dread - d3/f8e zero size 2024-04-06T14:17:03.831 INFO:tasks.workunit.client.1.smithi086.stdout:1/232: dread f7 [0,4194304] 0 2024-04-06T14:17:03.832 INFO:tasks.workunit.client.1.smithi086.stdout:8/352: mkdir d0/d3b/d79 0 2024-04-06T14:17:03.832 INFO:tasks.workunit.client.0.smithi032.stdout:6/459: write d4/f10 [2473499,101189] 0 2024-04-06T14:17:03.833 INFO:tasks.workunit.client.1.smithi086.stdout:8/353: chown d0/lf 2 1 2024-04-06T14:17:03.834 INFO:tasks.workunit.client.0.smithi032.stdout:7/481: write d16/d20/d26/f31 [78094,97254] 0 2024-04-06T14:17:03.836 INFO:tasks.workunit.client.1.smithi086.stdout:1/233: rmdir db/d13 39 2024-04-06T14:17:03.836 INFO:tasks.workunit.client.0.smithi032.stdout:5/451: creat d16/d18/d33/d45/d52/d39/d86/f98 x:0 0 0 2024-04-06T14:17:03.837 INFO:tasks.workunit.client.1.smithi086.stdout:8/354: chown d0/db/c2f 1618 1 2024-04-06T14:17:03.839 INFO:tasks.workunit.client.1.smithi086.stdout:1/234: unlink db/d11/d1f/l2f 0 2024-04-06T14:17:03.839 INFO:tasks.workunit.client.1.smithi086.stdout:0/252: dwrite d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:03.841 INFO:tasks.workunit.client.1.smithi086.stdout:8/355: mknod d0/d71/d22/d43/d4d/c7a 0 2024-04-06T14:17:03.842 INFO:tasks.workunit.client.1.smithi086.stdout:8/356: chown d0/db/l2b 14 1 2024-04-06T14:17:03.843 INFO:tasks.workunit.client.0.smithi032.stdout:7/482: unlink d16/d20/d54/f76 0 2024-04-06T14:17:03.846 INFO:tasks.workunit.client.0.smithi032.stdout:3/445: dwrite d3/d16/f55 [0,4194304] 0 2024-04-06T14:17:03.847 INFO:tasks.workunit.client.0.smithi032.stdout:1/541: getdents d0/d9/d94 0 2024-04-06T14:17:03.850 INFO:tasks.workunit.client.0.smithi032.stdout:6/460: fsync d4/f13 0 2024-04-06T14:17:03.852 INFO:tasks.workunit.client.0.smithi032.stdout:0/331: dwrite d3/da/d25/f2a [0,4194304] 0 2024-04-06T14:17:03.854 INFO:tasks.workunit.client.1.smithi086.stdout:1/235: write db/fe [4605402,65862] 0 2024-04-06T14:17:03.855 INFO:tasks.workunit.client.0.smithi032.stdout:4/490: write d0/df/d1f/d42/d45/f75 [189283,35487] 0 2024-04-06T14:17:03.856 INFO:tasks.workunit.client.0.smithi032.stdout:7/483: rmdir d16/d20/d26/d2d/d30/d97 39 2024-04-06T14:17:03.856 INFO:tasks.workunit.client.0.smithi032.stdout:9/474: dread d2/d15/d1e/f1f [0,4194304] 0 2024-04-06T14:17:03.860 INFO:tasks.workunit.client.1.smithi086.stdout:1/236: dread f8 [0,4194304] 0 2024-04-06T14:17:03.860 INFO:tasks.workunit.client.0.smithi032.stdout:7/484: write d16/d20/d26/d5f/f65 [562118,35207] 0 2024-04-06T14:17:03.862 INFO:tasks.workunit.client.1.smithi086.stdout:1/237: write db/d11/f27 [334368,114119] 0 2024-04-06T14:17:03.863 INFO:tasks.workunit.client.0.smithi032.stdout:0/332: symlink d3/l72 0 2024-04-06T14:17:03.864 INFO:tasks.workunit.client.0.smithi032.stdout:4/491: write d0/df/f92 [179119,60204] 0 2024-04-06T14:17:03.867 INFO:tasks.workunit.client.0.smithi032.stdout:7/485: creat d16/d20/d54/fad x:0 0 0 2024-04-06T14:17:03.867 INFO:tasks.workunit.client.1.smithi086.stdout:1/238: creat db/d11/d1a/f4b x:0 0 0 2024-04-06T14:17:03.869 INFO:tasks.workunit.client.0.smithi032.stdout:1/542: link d0/d9/c54 d0/d10/d16/cb0 0 2024-04-06T14:17:03.870 INFO:tasks.workunit.client.0.smithi032.stdout:0/333: write d3/f4 [490072,79872] 0 2024-04-06T14:17:03.870 INFO:tasks.workunit.client.0.smithi032.stdout:4/492: rmdir d0/df/d1f/d5a 39 2024-04-06T14:17:03.872 INFO:tasks.workunit.client.1.smithi086.stdout:1/239: creat db/d11/d1a/d2a/f4c x:0 0 0 2024-04-06T14:17:03.873 INFO:tasks.workunit.client.1.smithi086.stdout:8/357: dread d0/f2 [0,4194304] 0 2024-04-06T14:17:03.875 INFO:tasks.workunit.client.1.smithi086.stdout:1/240: mknod db/d13/c4d 0 2024-04-06T14:17:03.878 INFO:tasks.workunit.client.1.smithi086.stdout:8/358: rename d0/d71/d16/d39/l76 to d0/l7b 0 2024-04-06T14:17:03.882 INFO:tasks.workunit.client.1.smithi086.stdout:1/241: dwrite db/d13/f3d [0,4194304] 0 2024-04-06T14:17:03.883 INFO:tasks.workunit.client.1.smithi086.stdout:8/359: getdents d0/db/d32 0 2024-04-06T14:17:03.884 INFO:tasks.workunit.client.1.smithi086.stdout:8/360: readlink d0/d71/d22/d43/l48 0 2024-04-06T14:17:03.887 INFO:tasks.workunit.client.1.smithi086.stdout:8/361: creat d0/d13/f7c x:0 0 0 2024-04-06T14:17:03.889 INFO:tasks.workunit.client.1.smithi086.stdout:1/242: rename db/d11/d24/c40 to db/d13/c4e 0 2024-04-06T14:17:03.891 INFO:tasks.workunit.client.1.smithi086.stdout:8/362: mkdir d0/d71/d22/d43/d6d/d75/d7d 0 2024-04-06T14:17:03.892 INFO:tasks.workunit.client.1.smithi086.stdout:1/243: mknod db/d11/d1d/c4f 0 2024-04-06T14:17:03.892 INFO:tasks.workunit.client.1.smithi086.stdout:8/363: write d0/d71/d22/d43/f28 [13446,123794] 0 2024-04-06T14:17:03.895 INFO:tasks.workunit.client.1.smithi086.stdout:8/364: mknod d0/d71/d22/d43/d4d/c7e 0 2024-04-06T14:17:03.896 INFO:tasks.workunit.client.1.smithi086.stdout:8/365: stat d0/d71/d22/d43/f47 0 2024-04-06T14:17:03.896 INFO:tasks.workunit.client.1.smithi086.stdout:1/244: dread db/d13/f3d [0,4194304] 0 2024-04-06T14:17:03.896 INFO:tasks.workunit.client.1.smithi086.stdout:8/366: stat d0/d3b/l77 0 2024-04-06T14:17:03.898 INFO:tasks.workunit.client.1.smithi086.stdout:1/245: write db/d11/d1a/d2a/f47 [768938,114733] 0 2024-04-06T14:17:03.898 INFO:tasks.workunit.client.1.smithi086.stdout:8/367: mkdir d0/d71/d22/d43/d7f 0 2024-04-06T14:17:03.901 INFO:tasks.workunit.client.1.smithi086.stdout:8/368: write d0/d71/d22/f74 [947520,44652] 0 2024-04-06T14:17:03.901 INFO:tasks.workunit.client.1.smithi086.stdout:8/369: fsync d0/db/f5e 0 2024-04-06T14:17:03.904 INFO:tasks.workunit.client.1.smithi086.stdout:1/246: dread db/d13/f1e [0,4194304] 0 2024-04-06T14:17:03.904 INFO:tasks.workunit.client.1.smithi086.stdout:8/370: creat d0/d71/d22/d43/d6d/f80 x:0 0 0 2024-04-06T14:17:03.905 INFO:tasks.workunit.client.0.smithi032.stdout:8/413: dwrite d3/d8/d12/f5b [0,4194304] 0 2024-04-06T14:17:03.906 INFO:tasks.workunit.client.0.smithi032.stdout:6/461: getdents d4/d5/d19 0 2024-04-06T14:17:03.906 INFO:tasks.workunit.client.1.smithi086.stdout:8/371: fsync d0/db/f12 0 2024-04-06T14:17:03.906 INFO:tasks.workunit.client.0.smithi032.stdout:7/486: symlink d16/d20/d48/lae 0 2024-04-06T14:17:03.907 INFO:tasks.workunit.client.1.smithi086.stdout:8/372: dread - d0/d3b/f51 zero size 2024-04-06T14:17:03.907 INFO:tasks.workunit.client.1.smithi086.stdout:1/247: creat db/d11/d42/f50 x:0 0 0 2024-04-06T14:17:03.910 INFO:tasks.workunit.client.0.smithi032.stdout:0/334: readlink d3/da/d25/d36/l37 0 2024-04-06T14:17:03.910 INFO:tasks.workunit.client.0.smithi032.stdout:9/475: dwrite d2/f97 [0,4194304] 0 2024-04-06T14:17:03.911 INFO:tasks.workunit.client.1.smithi086.stdout:1/248: creat db/d11/d1f/f51 x:0 0 0 2024-04-06T14:17:03.913 INFO:tasks.workunit.client.0.smithi032.stdout:5/452: dwrite d16/d18/d33/f56 [0,4194304] 0 2024-04-06T14:17:03.916 INFO:tasks.workunit.client.1.smithi086.stdout:1/249: mkdir db/d11/d1a/d2a/d52 0 2024-04-06T14:17:03.916 INFO:tasks.workunit.client.0.smithi032.stdout:8/414: read d3/d8/d12/d2f/d4a/f94 [1622995,127806] 0 2024-04-06T14:17:03.917 INFO:tasks.workunit.client.0.smithi032.stdout:3/446: dread d3/d65/f74 [0,4194304] 0 2024-04-06T14:17:03.917 INFO:tasks.workunit.client.1.smithi086.stdout:8/373: dwrite d0/db/f12 [4194304,4194304] 0 2024-04-06T14:17:03.917 INFO:tasks.workunit.client.0.smithi032.stdout:5/453: chown d16/d18/f23 2065965 1 2024-04-06T14:17:03.922 INFO:tasks.workunit.client.0.smithi032.stdout:3/447: write d3/d16/f33 [568358,29530] 0 2024-04-06T14:17:03.925 INFO:tasks.workunit.client.1.smithi086.stdout:1/250: mkdir db/d11/d42/d53 0 2024-04-06T14:17:03.925 INFO:tasks.workunit.client.1.smithi086.stdout:1/251: fdatasync db/d11/d1a/d2a/f47 0 2024-04-06T14:17:03.925 INFO:tasks.workunit.client.0.smithi032.stdout:7/487: creat d16/d20/d26/d2d/faf x:0 0 0 2024-04-06T14:17:03.926 INFO:tasks.workunit.client.0.smithi032.stdout:1/543: dwrite d0/d41/d59/f8e [0,4194304] 0 2024-04-06T14:17:03.926 INFO:tasks.workunit.client.0.smithi032.stdout:0/335: unlink d3/da/f1e 0 2024-04-06T14:17:03.927 INFO:tasks.workunit.client.0.smithi032.stdout:3/448: read - d3/d65/f93 zero size 2024-04-06T14:17:03.929 INFO:tasks.workunit.client.0.smithi032.stdout:5/454: symlink d16/d18/d33/d45/d52/d39/l99 0 2024-04-06T14:17:03.929 INFO:tasks.workunit.client.0.smithi032.stdout:6/462: dwrite d4/d5/d28/d54/f89 [0,4194304] 0 2024-04-06T14:17:03.931 INFO:tasks.workunit.client.0.smithi032.stdout:5/455: readlink d16/d18/l5a 0 2024-04-06T14:17:03.931 INFO:tasks.workunit.client.0.smithi032.stdout:9/476: link d2/f13 d2/f98 0 2024-04-06T14:17:03.932 INFO:tasks.workunit.client.0.smithi032.stdout:0/336: chown d3/da/d41/c69 45 1 2024-04-06T14:17:03.932 INFO:tasks.workunit.client.0.smithi032.stdout:7/488: write d16/d20/d26/d2d/f64 [760103,98554] 0 2024-04-06T14:17:03.936 INFO:tasks.workunit.client.0.smithi032.stdout:4/493: getdents d0/df/d2b 0 2024-04-06T14:17:03.938 INFO:tasks.workunit.client.0.smithi032.stdout:3/449: unlink d3/d16/d63/fb4 0 2024-04-06T14:17:03.938 INFO:tasks.workunit.client.1.smithi086.stdout:1/252: dwrite db/d11/d1f/f51 [0,4194304] 0 2024-04-06T14:17:03.940 INFO:tasks.workunit.client.0.smithi032.stdout:5/456: unlink d16/f1d 0 2024-04-06T14:17:03.940 INFO:tasks.workunit.client.1.smithi086.stdout:1/253: chown db/d11/l37 10 1 2024-04-06T14:17:03.942 INFO:tasks.workunit.client.0.smithi032.stdout:1/544: rename d0/d10/d16/d3f/l77 to d0/d41/d59/lb1 0 2024-04-06T14:17:03.947 INFO:tasks.workunit.client.0.smithi032.stdout:6/463: unlink d4/d5/d28/f4f 0 2024-04-06T14:17:03.949 INFO:tasks.workunit.client.0.smithi032.stdout:0/337: mknod d3/da/d41/c73 0 2024-04-06T14:17:03.950 INFO:tasks.workunit.client.0.smithi032.stdout:7/489: dwrite d16/d20/d26/d2d/d30/f53 [0,4194304] 0 2024-04-06T14:17:03.951 INFO:tasks.workunit.client.0.smithi032.stdout:8/415: getdents d3/d8/d12 0 2024-04-06T14:17:03.952 INFO:tasks.workunit.client.0.smithi032.stdout:9/477: symlink d2/d15/d55/d5f/d71/d81/d77/l99 0 2024-04-06T14:17:03.953 INFO:tasks.workunit.client.0.smithi032.stdout:4/494: truncate d0/df/d23/f3c 588621 0 2024-04-06T14:17:03.955 INFO:tasks.workunit.client.0.smithi032.stdout:1/545: chown d0/f2c 3000 1 2024-04-06T14:17:03.955 INFO:tasks.workunit.client.0.smithi032.stdout:6/464: mkdir d4/d5/d18/d23/d82/d9d 0 2024-04-06T14:17:03.956 INFO:tasks.workunit.client.0.smithi032.stdout:0/338: chown d3/da/d25/d2b/f32 3646286 1 2024-04-06T14:17:03.959 INFO:tasks.workunit.client.0.smithi032.stdout:7/490: creat d16/d20/d26/d5f/fb0 x:0 0 0 2024-04-06T14:17:03.959 INFO:tasks.workunit.client.0.smithi032.stdout:0/339: symlink d3/l74 0 2024-04-06T14:17:03.961 INFO:tasks.workunit.client.0.smithi032.stdout:6/465: stat d4/d5/d8/d1f/f8c 0 2024-04-06T14:17:03.961 INFO:tasks.workunit.client.0.smithi032.stdout:5/457: dwrite d16/d18/d33/f78 [0,4194304] 0 2024-04-06T14:17:03.963 INFO:tasks.workunit.client.0.smithi032.stdout:1/546: write d0/d10/d16/d3f/f85 [2048805,57831] 0 2024-04-06T14:17:03.964 INFO:tasks.workunit.client.1.smithi086.stdout:8/374: write d0/f34 [1853737,60441] 0 2024-04-06T14:17:03.966 INFO:tasks.workunit.client.0.smithi032.stdout:4/495: write d0/df/d2b/d43/fa6 [655,32697] 0 2024-04-06T14:17:03.967 INFO:tasks.workunit.client.0.smithi032.stdout:8/416: truncate d3/d8/d7/d1a/f71 584926 0 2024-04-06T14:17:03.969 INFO:tasks.workunit.client.0.smithi032.stdout:3/450: dwrite d3/d65/d89/f97 [0,4194304] 0 2024-04-06T14:17:03.970 INFO:tasks.workunit.client.1.smithi086.stdout:1/254: dwrite db/d11/f17 [0,4194304] 0 2024-04-06T14:17:03.971 INFO:tasks.workunit.client.1.smithi086.stdout:8/375: symlink d0/d71/d22/d43/d6d/d75/l81 0 2024-04-06T14:17:03.971 INFO:tasks.workunit.client.0.smithi032.stdout:0/340: mkdir d3/da/d35/d75 0 2024-04-06T14:17:03.973 INFO:tasks.workunit.client.1.smithi086.stdout:8/376: dread d0/db/d32/f5d [0,4194304] 0 2024-04-06T14:17:03.973 INFO:tasks.workunit.client.1.smithi086.stdout:1/255: creat db/d13/f54 x:0 0 0 2024-04-06T14:17:03.974 INFO:tasks.workunit.client.0.smithi032.stdout:7/491: mknod d16/d92/cb1 0 2024-04-06T14:17:03.975 INFO:tasks.workunit.client.1.smithi086.stdout:2/214: sync 2024-04-06T14:17:03.977 INFO:tasks.workunit.client.1.smithi086.stdout:1/256: write f8 [3097042,95006] 0 2024-04-06T14:17:03.978 INFO:tasks.workunit.client.0.smithi032.stdout:0/341: truncate d3/da/d25/d36/f68 720348 0 2024-04-06T14:17:03.980 INFO:tasks.workunit.client.0.smithi032.stdout:3/451: mknod d3/d65/cb6 0 2024-04-06T14:17:03.981 INFO:tasks.workunit.client.0.smithi032.stdout:8/417: truncate d3/d8/d3a/f4f 579351 0 2024-04-06T14:17:03.982 INFO:tasks.workunit.client.1.smithi086.stdout:8/377: mkdir d0/d71/d82 0 2024-04-06T14:17:03.982 INFO:tasks.workunit.client.1.smithi086.stdout:2/215: chown d0/d20/d27/d2b/d35/d36/f3b 47 1 2024-04-06T14:17:03.982 INFO:tasks.workunit.client.1.smithi086.stdout:8/378: read d0/d71/d16/f41 [1203167,47668] 0 2024-04-06T14:17:03.982 INFO:tasks.workunit.client.1.smithi086.stdout:2/216: fsync d0/df/f18 0 2024-04-06T14:17:03.984 INFO:tasks.workunit.client.1.smithi086.stdout:2/217: mkdir d0/d20/d27/d37/d52 0 2024-04-06T14:17:03.985 INFO:tasks.workunit.client.0.smithi032.stdout:4/496: write d0/df/d23/d31/d50/d57/d5b/d9c/f9f [787624,19480] 0 2024-04-06T14:17:03.986 INFO:tasks.workunit.client.0.smithi032.stdout:0/342: creat d3/f76 x:0 0 0 2024-04-06T14:17:03.986 INFO:tasks.workunit.client.1.smithi086.stdout:8/379: link d0/d71/f3c d0/d71/d22/f83 0 2024-04-06T14:17:03.988 INFO:tasks.workunit.client.0.smithi032.stdout:1/547: dwrite d0/d41/f5c [0,4194304] 0 2024-04-06T14:17:03.988 INFO:tasks.workunit.client.0.smithi032.stdout:0/343: stat d3/da/l71 0 2024-04-06T14:17:03.989 INFO:tasks.workunit.client.0.smithi032.stdout:9/478: write d2/f21 [1688950,94252] 0 2024-04-06T14:17:03.990 INFO:tasks.workunit.client.0.smithi032.stdout:3/452: mknod d3/d65/cb7 0 2024-04-06T14:17:03.990 INFO:tasks.workunit.client.0.smithi032.stdout:7/492: dread d16/d20/d26/f39 [0,4194304] 0 2024-04-06T14:17:03.992 INFO:tasks.workunit.client.1.smithi086.stdout:2/218: dwrite d0/d20/f23 [0,4194304] 0 2024-04-06T14:17:03.994 INFO:tasks.workunit.client.0.smithi032.stdout:5/458: creat d16/d18/f9a x:0 0 0 2024-04-06T14:17:03.994 INFO:tasks.workunit.client.0.smithi032.stdout:7/493: fsync d16/d20/d26/d2d/d30/d3e/f73 0 2024-04-06T14:17:03.995 INFO:tasks.workunit.client.0.smithi032.stdout:9/479: write d2/d15/d1e/d41/f63 [1400484,130695] 0 2024-04-06T14:17:03.997 INFO:tasks.workunit.client.0.smithi032.stdout:6/466: link d4/d5/d19/d25/c58 d4/d5/d19/c9e 0 2024-04-06T14:17:03.997 INFO:tasks.workunit.client.0.smithi032.stdout:4/497: rmdir d0/df/d76 39 2024-04-06T14:17:03.998 INFO:tasks.workunit.client.0.smithi032.stdout:0/344: mknod d3/c77 0 2024-04-06T14:17:04.001 INFO:tasks.workunit.client.0.smithi032.stdout:1/548: symlink d0/da/d3e/d75/lb2 0 2024-04-06T14:17:04.001 INFO:tasks.workunit.client.0.smithi032.stdout:3/453: mkdir d3/d65/d7d/d9c/db8 0 2024-04-06T14:17:04.001 INFO:tasks.workunit.client.1.smithi086.stdout:2/219: mkdir d0/df/d48/d53 0 2024-04-06T14:17:04.002 INFO:tasks.workunit.client.1.smithi086.stdout:8/380: mkdir d0/d71/d22/d65/d6e/d84 0 2024-04-06T14:17:04.002 INFO:tasks.workunit.client.0.smithi032.stdout:4/498: read d0/df/d23/d31/f8d [2317481,10371] 0 2024-04-06T14:17:04.005 INFO:tasks.workunit.client.1.smithi086.stdout:2/220: creat d0/df/d48/d53/f54 x:0 0 0 2024-04-06T14:17:04.005 INFO:tasks.workunit.client.0.smithi032.stdout:0/345: dread - d3/da/f6f zero size 2024-04-06T14:17:04.006 INFO:tasks.workunit.client.1.smithi086.stdout:8/381: symlink d0/d71/d22/d43/d4d/l85 0 2024-04-06T14:17:04.007 INFO:tasks.workunit.client.0.smithi032.stdout:9/480: write d2/d15/d1e/d65/d83/d31/d39/f58 [1836426,86464] 0 2024-04-06T14:17:04.007 INFO:tasks.workunit.client.0.smithi032.stdout:8/418: write d3/d8/d12/f3f [499798,115899] 0 2024-04-06T14:17:04.009 INFO:tasks.workunit.client.1.smithi086.stdout:8/382: symlink d0/d71/d16/d52/d5f/l86 0 2024-04-06T14:17:04.010 INFO:tasks.workunit.client.1.smithi086.stdout:1/257: dread db/d11/f12 [0,4194304] 0 2024-04-06T14:17:04.013 INFO:tasks.workunit.client.0.smithi032.stdout:0/346: fsync d3/da/d25/d2b/f32 0 2024-04-06T14:17:04.013 INFO:tasks.workunit.client.0.smithi032.stdout:3/454: write d3/f7a [1178947,105322] 0 2024-04-06T14:17:04.014 INFO:tasks.workunit.client.0.smithi032.stdout:6/467: rmdir d4/d5/d8/d1f/d51/d70 0 2024-04-06T14:17:04.015 INFO:tasks.workunit.client.1.smithi086.stdout:8/383: stat d0/db/d32/l49 0 2024-04-06T14:17:04.016 INFO:tasks.workunit.client.0.smithi032.stdout:7/494: dwrite d16/f40 [0,4194304] 0 2024-04-06T14:17:04.019 INFO:tasks.workunit.client.0.smithi032.stdout:9/481: write d2/d15/d1e/d65/d83/d7e/f56 [575350,18589] 0 2024-04-06T14:17:04.020 INFO:tasks.workunit.client.1.smithi086.stdout:8/384: read d0/f5 [3100040,16491] 0 2024-04-06T14:17:04.020 INFO:tasks.workunit.client.1.smithi086.stdout:2/221: dwrite d0/d20/d27/d2b/d35/f4e [0,4194304] 0 2024-04-06T14:17:04.020 INFO:tasks.workunit.client.0.smithi032.stdout:6/468: stat d4/d5/d19/d25/d67/f8a 0 2024-04-06T14:17:04.021 INFO:tasks.workunit.client.0.smithi032.stdout:0/347: creat d3/f78 x:0 0 0 2024-04-06T14:17:04.025 INFO:tasks.workunit.client.0.smithi032.stdout:5/459: dread d16/d18/d33/f8a [0,4194304] 0 2024-04-06T14:17:04.029 INFO:tasks.workunit.client.0.smithi032.stdout:8/419: dwrite d3/d8/d7/f47 [0,4194304] 0 2024-04-06T14:17:04.029 INFO:tasks.workunit.client.1.smithi086.stdout:8/385: unlink d0/d71/c42 0 2024-04-06T14:17:04.030 INFO:tasks.workunit.client.1.smithi086.stdout:1/258: dwrite db/d13/f3d [0,4194304] 0 2024-04-06T14:17:04.032 INFO:tasks.workunit.client.0.smithi032.stdout:3/455: creat d3/fb9 x:0 0 0 2024-04-06T14:17:04.033 INFO:tasks.workunit.client.1.smithi086.stdout:8/386: mknod d0/db/c87 0 2024-04-06T14:17:04.033 INFO:tasks.workunit.client.1.smithi086.stdout:2/222: getdents d0/d20/d27/d2b/d35/d36/d4a 0 2024-04-06T14:17:04.033 INFO:tasks.workunit.client.0.smithi032.stdout:2/462: sync 2024-04-06T14:17:04.035 INFO:tasks.workunit.client.1.smithi086.stdout:1/259: mkdir db/d13/d55 0 2024-04-06T14:17:04.035 INFO:tasks.workunit.client.0.smithi032.stdout:5/460: symlink d16/d18/d33/d45/d87/l9b 0 2024-04-06T14:17:04.035 INFO:tasks.workunit.client.0.smithi032.stdout:1/549: dwrite d0/f1e [0,4194304] 0 2024-04-06T14:17:04.036 INFO:tasks.workunit.client.0.smithi032.stdout:6/469: unlink d4/d5/d8/d1f/l24 0 2024-04-06T14:17:04.038 INFO:tasks.workunit.client.0.smithi032.stdout:4/499: rename d0/df/d23/d31/d50/d57/d5b/c79 to d0/df/d23/d31/d50/d57/d8a/cab 0 2024-04-06T14:17:04.039 INFO:tasks.workunit.client.1.smithi086.stdout:8/387: creat d0/d71/d22/d43/d7f/f88 x:0 0 0 2024-04-06T14:17:04.040 INFO:tasks.workunit.client.1.smithi086.stdout:1/260: rename c3 to db/d11/d1d/c56 0 2024-04-06T14:17:04.041 INFO:tasks.workunit.client.1.smithi086.stdout:1/261: fsync db/d11/d24/f45 0 2024-04-06T14:17:04.045 INFO:tasks.workunit.client.0.smithi032.stdout:5/461: creat d16/d18/d33/d35/d4d/f9c x:0 0 0 2024-04-06T14:17:04.046 INFO:tasks.workunit.client.1.smithi086.stdout:1/262: write db/d13/f14 [1340824,78685] 0 2024-04-06T14:17:04.046 INFO:tasks.workunit.client.0.smithi032.stdout:1/550: chown d0/d41/d5b/l97 14 1 2024-04-06T14:17:04.046 INFO:tasks.workunit.client.0.smithi032.stdout:6/470: stat d4/d5/d19/c9e 0 2024-04-06T14:17:04.047 INFO:tasks.workunit.client.0.smithi032.stdout:3/456: unlink d3/d65/f74 0 2024-04-06T14:17:04.047 INFO:tasks.workunit.client.0.smithi032.stdout:8/420: unlink d3/d8/d12/l27 0 2024-04-06T14:17:04.051 INFO:tasks.workunit.client.1.smithi086.stdout:1/263: creat db/d11/d42/d53/f57 x:0 0 0 2024-04-06T14:17:04.052 INFO:tasks.workunit.client.0.smithi032.stdout:6/471: creat d4/d5/d34/f9f x:0 0 0 2024-04-06T14:17:04.054 INFO:tasks.workunit.client.0.smithi032.stdout:1/551: unlink d0/d9/d1f/d72/cac 0 2024-04-06T14:17:04.055 INFO:tasks.workunit.client.0.smithi032.stdout:3/457: truncate d3/f4 5094703 0 2024-04-06T14:17:04.056 INFO:tasks.workunit.client.1.smithi086.stdout:1/264: rename db/d11/d1a/f25 to db/d11/d1d/f58 0 2024-04-06T14:17:04.058 INFO:tasks.workunit.client.0.smithi032.stdout:6/472: creat d4/d5/fa0 x:0 0 0 2024-04-06T14:17:04.061 INFO:tasks.workunit.client.0.smithi032.stdout:2/463: dread d0/d6e/f86 [0,4194304] 0 2024-04-06T14:17:04.066 INFO:tasks.workunit.client.0.smithi032.stdout:3/458: link d3/d65/l5b d3/d65/d7d/d9c/dad/db1/d3c/lba 0 2024-04-06T14:17:04.066 INFO:tasks.workunit.client.0.smithi032.stdout:8/421: dread d3/f31 [0,4194304] 0 2024-04-06T14:17:04.069 INFO:tasks.workunit.client.0.smithi032.stdout:3/459: symlink d3/lbb 0 2024-04-06T14:17:04.070 INFO:tasks.workunit.client.0.smithi032.stdout:1/552: dwrite d0/d9/d1f/d72/f7e [0,4194304] 0 2024-04-06T14:17:04.070 INFO:tasks.workunit.client.0.smithi032.stdout:2/464: creat d0/d5a/f9d x:0 0 0 2024-04-06T14:17:04.071 INFO:tasks.workunit.client.0.smithi032.stdout:8/422: fsync d3/d8/d7/d1a/f71 0 2024-04-06T14:17:04.073 INFO:tasks.workunit.client.1.smithi086.stdout:2/223: dread d0/df/f14 [0,4194304] 0 2024-04-06T14:17:04.073 INFO:tasks.workunit.client.0.smithi032.stdout:5/462: dread d16/d18/d33/d35/f42 [0,4194304] 0 2024-04-06T14:17:04.075 INFO:tasks.workunit.client.0.smithi032.stdout:3/460: stat d3/d65/c92 0 2024-04-06T14:17:04.075 INFO:tasks.workunit.client.0.smithi032.stdout:7/495: dread f7 [0,4194304] 0 2024-04-06T14:17:04.075 INFO:tasks.workunit.client.1.smithi086.stdout:1/265: dread f4 [0,4194304] 0 2024-04-06T14:17:04.075 INFO:tasks.workunit.client.1.smithi086.stdout:2/224: chown d0/d20/d27/d2b/f45 72378 1 2024-04-06T14:17:04.076 INFO:tasks.workunit.client.0.smithi032.stdout:0/348: write d3/da/f1f [1325147,6754] 0 2024-04-06T14:17:04.077 INFO:tasks.workunit.client.1.smithi086.stdout:8/388: write d0/db/d32/f5d [972382,29986] 0 2024-04-06T14:17:04.078 INFO:tasks.workunit.client.0.smithi032.stdout:6/473: dwrite d4/d5/d18/f74 [4194304,4194304] 0 2024-04-06T14:17:04.079 INFO:tasks.workunit.client.1.smithi086.stdout:2/225: fdatasync d0/df/f14 0 2024-04-06T14:17:04.079 INFO:tasks.workunit.client.0.smithi032.stdout:8/423: write d3/d8/d7/d1a/f71 [791970,54798] 0 2024-04-06T14:17:04.081 INFO:tasks.workunit.client.1.smithi086.stdout:8/389: mknod d0/d71/d22/d43/d7f/c89 0 2024-04-06T14:17:04.081 INFO:tasks.workunit.client.0.smithi032.stdout:2/465: stat d0/d30/d39/d59/d67/d74/d78/d57/f90 0 2024-04-06T14:17:04.082 INFO:tasks.workunit.client.0.smithi032.stdout:6/474: stat d4/d5/d8/d1f/f33 0 2024-04-06T14:17:04.082 INFO:tasks.workunit.client.0.smithi032.stdout:8/424: chown d3/d8/d7/d14/l73 747 1 2024-04-06T14:17:04.083 INFO:tasks.workunit.client.1.smithi086.stdout:2/226: creat d0/d20/d27/d2b/d35/d36/f55 x:0 0 0 2024-04-06T14:17:04.084 INFO:tasks.workunit.client.0.smithi032.stdout:8/425: chown d3/d8/d12/d2f/d4a/d60/d82/f95 307234 1 2024-04-06T14:17:04.088 INFO:tasks.workunit.client.0.smithi032.stdout:4/500: write d0/df/d23/d31/d50/d57/f6a [2840052,130991] 0 2024-04-06T14:17:04.089 INFO:tasks.workunit.client.1.smithi086.stdout:8/390: creat d0/d5a/f8a x:0 0 0 2024-04-06T14:17:04.090 INFO:tasks.workunit.client.0.smithi032.stdout:0/349: unlink d3/f24 0 2024-04-06T14:17:04.093 INFO:tasks.workunit.client.0.smithi032.stdout:5/463: creat d16/d18/d33/d35/f9d x:0 0 0 2024-04-06T14:17:04.093 INFO:tasks.workunit.client.1.smithi086.stdout:1/266: truncate db/f16 257698 0 2024-04-06T14:17:04.093 INFO:tasks.workunit.client.1.smithi086.stdout:1/267: readlink db/d11/d1f/l48 0 2024-04-06T14:17:04.094 INFO:tasks.workunit.client.1.smithi086.stdout:1/268: readlink db/l3c 0 2024-04-06T14:17:04.094 INFO:tasks.workunit.client.0.smithi032.stdout:3/461: mkdir d3/d65/d7d/d9c/dbc 0 2024-04-06T14:17:04.097 INFO:tasks.workunit.client.0.smithi032.stdout:1/553: creat d0/d9/d1f/d7c/da9/fb3 x:0 0 0 2024-04-06T14:17:04.098 INFO:tasks.workunit.client.1.smithi086.stdout:1/269: truncate db/ff 333041 0 2024-04-06T14:17:04.098 INFO:tasks.workunit.client.1.smithi086.stdout:1/270: readlink db/d11/d1f/l48 0 2024-04-06T14:17:04.099 INFO:tasks.workunit.client.1.smithi086.stdout:1/271: readlink db/l3c 0 2024-04-06T14:17:04.099 INFO:tasks.workunit.client.0.smithi032.stdout:2/466: chown d0/l9 740044905 1 2024-04-06T14:17:04.100 INFO:tasks.workunit.client.0.smithi032.stdout:6/475: creat d4/d5/d18/d23/fa1 x:0 0 0 2024-04-06T14:17:04.100 INFO:tasks.workunit.client.1.smithi086.stdout:1/272: truncate db/d11/d1d/f58 1192651 0 2024-04-06T14:17:04.102 INFO:tasks.workunit.client.0.smithi032.stdout:0/350: dwrite d3/f63 [4194304,4194304] 0 2024-04-06T14:17:04.104 INFO:tasks.workunit.client.1.smithi086.stdout:1/273: rename db/d11/d1a/d2a/f35 to db/d13/d55/f59 0 2024-04-06T14:17:04.110 INFO:tasks.workunit.client.0.smithi032.stdout:3/462: dwrite d3/d65/f32 [0,4194304] 0 2024-04-06T14:17:04.112 INFO:tasks.workunit.client.0.smithi032.stdout:5/464: creat d16/d18/d33/d45/d52/f9e x:0 0 0 2024-04-06T14:17:04.112 INFO:tasks.workunit.client.1.smithi086.stdout:1/274: dwrite db/d11/d24/f45 [0,4194304] 0 2024-04-06T14:17:04.114 INFO:tasks.workunit.client.0.smithi032.stdout:2/467: dwrite d0/d5a/f41 [4194304,4194304] 0 2024-04-06T14:17:04.123 INFO:tasks.workunit.client.0.smithi032.stdout:5/465: dwrite d16/d18/d33/d45/d52/d39/d86/f93 [0,4194304] 0 2024-04-06T14:17:04.124 INFO:tasks.workunit.client.1.smithi086.stdout:1/275: dwrite db/d11/d1f/f2b [0,4194304] 0 2024-04-06T14:17:04.125 INFO:tasks.workunit.client.0.smithi032.stdout:5/466: fdatasync d16/d18/d33/d35/d4d/f9c 0 2024-04-06T14:17:04.130 INFO:tasks.workunit.client.1.smithi086.stdout:1/276: unlink f8 0 2024-04-06T14:17:04.131 INFO:tasks.workunit.client.0.smithi032.stdout:4/501: truncate d0/df/d1f/d2e/f9b 885048 0 2024-04-06T14:17:04.133 INFO:tasks.workunit.client.0.smithi032.stdout:7/496: creat d16/d20/d26/fb2 x:0 0 0 2024-04-06T14:17:04.137 INFO:tasks.workunit.client.0.smithi032.stdout:1/554: mknod d0/d9/d1f/d7c/d8f/daa/cb4 0 2024-04-06T14:17:04.138 INFO:tasks.workunit.client.0.smithi032.stdout:8/426: creat d3/d8/f99 x:0 0 0 2024-04-06T14:17:04.139 INFO:tasks.workunit.client.0.smithi032.stdout:0/351: mkdir d3/da/d25/d2b/d4a/d79 0 2024-04-06T14:17:04.139 INFO:tasks.workunit.client.0.smithi032.stdout:6/476: creat d4/d5/d8/d2b/d77/fa2 x:0 0 0 2024-04-06T14:17:04.140 INFO:tasks.workunit.client.0.smithi032.stdout:1/555: write d0/da/d3e/f6a [1161355,47272] 0 2024-04-06T14:17:04.141 INFO:tasks.workunit.client.0.smithi032.stdout:7/497: mkdir d16/d92/db3 0 2024-04-06T14:17:04.141 INFO:tasks.workunit.client.0.smithi032.stdout:4/502: chown d0/df/d23/f3c 244692566 1 2024-04-06T14:17:04.142 INFO:tasks.workunit.client.0.smithi032.stdout:2/468: creat d0/d30/d39/d59/d67/d74/d78/d8c/f9e x:0 0 0 2024-04-06T14:17:04.145 INFO:tasks.workunit.client.0.smithi032.stdout:8/427: read d3/d8/f2e [169044,41036] 0 2024-04-06T14:17:04.146 INFO:tasks.workunit.client.0.smithi032.stdout:3/463: symlink d3/d65/d7d/d9c/dad/db1/lbd 0 2024-04-06T14:17:04.147 INFO:tasks.workunit.client.0.smithi032.stdout:5/467: rename d16/f77 to d16/d18/d33/d45/f9f 0 2024-04-06T14:17:04.148 INFO:tasks.workunit.client.0.smithi032.stdout:4/503: readlink d0/df/d1f/l56 0 2024-04-06T14:17:04.151 INFO:tasks.workunit.client.0.smithi032.stdout:2/469: creat d0/d30/d39/d59/d67/d74/d78/d8c/f9f x:0 0 0 2024-04-06T14:17:04.152 INFO:tasks.workunit.client.0.smithi032.stdout:8/428: stat d3/d8/l80 0 2024-04-06T14:17:04.153 INFO:tasks.workunit.client.0.smithi032.stdout:7/498: mknod d16/d92/db3/cb4 0 2024-04-06T14:17:04.155 INFO:tasks.workunit.client.0.smithi032.stdout:0/352: rmdir d3/da/d25/d2b 39 2024-04-06T14:17:04.155 INFO:tasks.workunit.client.0.smithi032.stdout:7/499: dread - d16/d20/f37 zero size 2024-04-06T14:17:04.155 INFO:tasks.workunit.client.0.smithi032.stdout:3/464: dwrite d3/d65/d89/f97 [4194304,4194304] 0 2024-04-06T14:17:04.156 INFO:tasks.workunit.client.0.smithi032.stdout:3/465: stat d3/fae 0 2024-04-06T14:17:04.157 INFO:tasks.workunit.client.0.smithi032.stdout:0/353: fdatasync d3/da/d25/d36/f68 0 2024-04-06T14:17:04.157 INFO:tasks.workunit.client.0.smithi032.stdout:6/477: creat d4/d5/d28/d65/fa3 x:0 0 0 2024-04-06T14:17:04.160 INFO:tasks.workunit.client.0.smithi032.stdout:8/429: chown d3/d8/f35 244588 1 2024-04-06T14:17:04.162 INFO:tasks.workunit.client.1.smithi086.stdout:7/226: sync 2024-04-06T14:17:04.162 INFO:tasks.workunit.client.1.smithi086.stdout:4/171: sync 2024-04-06T14:17:04.162 INFO:tasks.workunit.client.0.smithi032.stdout:5/468: symlink d16/d18/d33/d35/d88/la0 0 2024-04-06T14:17:04.165 INFO:tasks.workunit.client.0.smithi032.stdout:5/469: dread - d16/d18/f9a zero size 2024-04-06T14:17:04.166 INFO:tasks.workunit.client.1.smithi086.stdout:4/172: read d0/f3 [1852898,58369] 0 2024-04-06T14:17:04.171 INFO:tasks.workunit.client.0.smithi032.stdout:5/470: write d16/d58/f7a [211753,62091] 0 2024-04-06T14:17:04.171 INFO:tasks.workunit.client.0.smithi032.stdout:1/556: truncate d0/d10/d16/fa1 811226 0 2024-04-06T14:17:04.171 INFO:tasks.workunit.client.1.smithi086.stdout:4/173: creat d0/d15/d20/d34/f35 x:0 0 0 2024-04-06T14:17:04.173 INFO:tasks.workunit.client.1.smithi086.stdout:7/227: rmdir d7/d10/d11/d1c/d24/d2c/d3f 0 2024-04-06T14:17:04.176 INFO:tasks.workunit.client.0.smithi032.stdout:0/354: read d3/da/d41/f42 [1186745,129055] 0 2024-04-06T14:17:04.177 INFO:tasks.workunit.client.0.smithi032.stdout:4/504: rename d0/df/d23/d31/d50/d57/d5b/d9c/f86 to d0/df/d23/fac 0 2024-04-06T14:17:04.177 INFO:tasks.workunit.client.0.smithi032.stdout:7/500: dwrite d16/d20/d26/d5f/fa2 [0,4194304] 0 2024-04-06T14:17:04.179 INFO:tasks.workunit.client.0.smithi032.stdout:5/471: read d16/d18/d33/d45/d52/f43 [93672,106571] 0 2024-04-06T14:17:04.180 INFO:tasks.workunit.client.0.smithi032.stdout:2/470: write d0/f22 [6328595,37020] 0 2024-04-06T14:17:04.181 INFO:tasks.workunit.client.0.smithi032.stdout:0/355: chown d3/da/f1f 1204088765 1 2024-04-06T14:17:04.181 INFO:tasks.workunit.client.0.smithi032.stdout:1/557: mknod d0/d9/d94/cb5 0 2024-04-06T14:17:04.183 INFO:tasks.workunit.client.0.smithi032.stdout:3/466: write d3/d65/d7d/d9c/dad/db1/d3c/f57 [821521,20621] 0 2024-04-06T14:17:04.184 INFO:tasks.workunit.client.0.smithi032.stdout:5/472: chown d16/d18/c72 16196206 1 2024-04-06T14:17:04.186 INFO:tasks.workunit.client.1.smithi086.stdout:7/228: getdents d7/d10/d11/d1c 0 2024-04-06T14:17:04.187 INFO:tasks.workunit.client.0.smithi032.stdout:8/430: rename d3/d8/l2b to d3/d8/d12/d83/l9a 0 2024-04-06T14:17:04.187 INFO:tasks.workunit.client.0.smithi032.stdout:1/558: fdatasync d0/da/d3e/f50 0 2024-04-06T14:17:04.187 INFO:tasks.workunit.client.0.smithi032.stdout:3/467: chown d3/d65/c31 1 1 2024-04-06T14:17:04.193 INFO:tasks.workunit.client.0.smithi032.stdout:2/471: mknod d0/d30/d39/d59/d67/d74/d78/d8c/ca0 0 2024-04-06T14:17:04.194 INFO:tasks.workunit.client.1.smithi086.stdout:7/229: creat d7/d10/d11/d1c/d24/d2c/d3c/d52/f55 x:0 0 0 2024-04-06T14:17:04.197 INFO:tasks.workunit.client.0.smithi032.stdout:6/478: rename d4/d5/d28/d54/f8b to d4/d5/d19/d25/d67/fa4 0 2024-04-06T14:17:04.197 INFO:tasks.workunit.client.0.smithi032.stdout:8/431: symlink d3/d8/d12/d2f/d4a/d60/l9b 0 2024-04-06T14:17:04.198 INFO:tasks.workunit.client.1.smithi086.stdout:4/174: dread d0/de/f1a [0,4194304] 0 2024-04-06T14:17:04.199 INFO:tasks.workunit.client.0.smithi032.stdout:8/432: fdatasync d3/d8/d12/f51 0 2024-04-06T14:17:04.201 INFO:tasks.workunit.client.0.smithi032.stdout:0/356: write d3/da/d41/f42 [2063730,60691] 0 2024-04-06T14:17:04.202 INFO:tasks.workunit.client.1.smithi086.stdout:7/230: truncate f5 3115593 0 2024-04-06T14:17:04.202 INFO:tasks.workunit.client.0.smithi032.stdout:7/501: rename d16/d20/d48/d71 to d16/d20/d26/d2d/d30/d3e/da7/db5 0 2024-04-06T14:17:04.202 INFO:tasks.workunit.client.1.smithi086.stdout:7/231: chown d7/d10/d11/d1c/f2e 608 1 2024-04-06T14:17:04.204 INFO:tasks.workunit.client.1.smithi086.stdout:4/175: creat d0/f36 x:0 0 0 2024-04-06T14:17:04.206 INFO:tasks.workunit.client.1.smithi086.stdout:7/232: unlink d7/le 0 2024-04-06T14:17:04.206 INFO:tasks.workunit.client.0.smithi032.stdout:6/479: creat d4/d5/d8/d2f/d9b/fa5 x:0 0 0 2024-04-06T14:17:04.206 INFO:tasks.workunit.client.0.smithi032.stdout:3/468: dwrite d3/d65/d7d/d9c/dad/db1/d3c/f79 [0,4194304] 0 2024-04-06T14:17:04.207 INFO:tasks.workunit.client.1.smithi086.stdout:4/176: creat d0/d15/f37 x:0 0 0 2024-04-06T14:17:04.214 INFO:tasks.workunit.client.0.smithi032.stdout:5/473: dread d16/d18/f1e [0,4194304] 0 2024-04-06T14:17:04.214 INFO:tasks.workunit.client.0.smithi032.stdout:8/433: write d3/d8/d12/d2f/d46/f74 [1497781,64071] 0 2024-04-06T14:17:04.214 INFO:tasks.workunit.client.0.smithi032.stdout:5/474: stat l2 0 2024-04-06T14:17:04.215 INFO:tasks.workunit.client.0.smithi032.stdout:7/502: creat d16/d20/d48/d8d/fb6 x:0 0 0 2024-04-06T14:17:04.215 INFO:tasks.workunit.client.1.smithi086.stdout:4/177: unlink d0/fa 0 2024-04-06T14:17:04.215 INFO:tasks.workunit.client.1.smithi086.stdout:4/178: readlink d0/l9 0 2024-04-06T14:17:04.216 INFO:tasks.workunit.client.0.smithi032.stdout:4/505: dwrite d0/d82/d4b/f63 [0,4194304] 0 2024-04-06T14:17:04.217 INFO:tasks.workunit.client.0.smithi032.stdout:2/472: dwrite d0/d30/d39/f58 [0,4194304] 0 2024-04-06T14:17:04.217 INFO:tasks.workunit.client.1.smithi086.stdout:4/179: mkdir d0/d2c/d38 0 2024-04-06T14:17:04.218 INFO:tasks.workunit.client.1.smithi086.stdout:4/180: chown d0/d15/d19/f1c 33426 1 2024-04-06T14:17:04.222 INFO:tasks.workunit.client.0.smithi032.stdout:7/503: chown d16/d20/d26/d70 12724208 1 2024-04-06T14:17:04.224 INFO:tasks.workunit.client.0.smithi032.stdout:5/475: chown d16/d18/d33/d45/f85 12 1 2024-04-06T14:17:04.224 INFO:tasks.workunit.client.0.smithi032.stdout:2/473: read - d0/d30/d39/d59/f82 zero size 2024-04-06T14:17:04.226 INFO:tasks.workunit.client.0.smithi032.stdout:1/559: dread d0/d10/d66/f70 [0,4194304] 0 2024-04-06T14:17:04.230 INFO:tasks.workunit.client.0.smithi032.stdout:0/357: dwrite d3/da/d25/f2a [0,4194304] 0 2024-04-06T14:17:04.232 INFO:tasks.workunit.client.0.smithi032.stdout:2/474: truncate d0/d5a/d3a/f40 101873 0 2024-04-06T14:17:04.232 INFO:tasks.workunit.client.0.smithi032.stdout:3/469: dwrite d3/d16/f5c [0,4194304] 0 2024-04-06T14:17:04.235 INFO:tasks.workunit.client.1.smithi086.stdout:7/233: dwrite d7/d1d/fc [0,4194304] 0 2024-04-06T14:17:04.238 INFO:tasks.workunit.client.1.smithi086.stdout:4/181: truncate d0/de/f11 6800146 0 2024-04-06T14:17:04.240 INFO:tasks.workunit.client.1.smithi086.stdout:4/182: truncate d0/d2c/d30/d24/f2d 138030 0 2024-04-06T14:17:04.241 INFO:tasks.workunit.client.1.smithi086.stdout:7/234: stat d7/d10/d30/c47 0 2024-04-06T14:17:04.243 INFO:tasks.workunit.client.1.smithi086.stdout:4/183: symlink d0/d15/d2b/l39 0 2024-04-06T14:17:04.243 INFO:tasks.workunit.client.1.smithi086.stdout:7/235: dread - d7/d10/d30/f4b zero size 2024-04-06T14:17:04.246 INFO:tasks.workunit.client.1.smithi086.stdout:4/184: creat d0/d15/d20/f3a x:0 0 0 2024-04-06T14:17:04.248 INFO:tasks.workunit.client.1.smithi086.stdout:4/185: creat d0/d2c/d30/f3b x:0 0 0 2024-04-06T14:17:04.249 INFO:tasks.workunit.client.1.smithi086.stdout:4/186: creat d0/d15/f3c x:0 0 0 2024-04-06T14:17:04.250 INFO:tasks.workunit.client.1.smithi086.stdout:4/187: dread - d0/d15/f3c zero size 2024-04-06T14:17:04.256 INFO:tasks.workunit.client.1.smithi086.stdout:4/188: dwrite d0/f17 [0,4194304] 0 2024-04-06T14:17:04.260 INFO:tasks.workunit.client.1.smithi086.stdout:7/236: fsync d7/f9 0 2024-04-06T14:17:04.263 INFO:tasks.workunit.client.0.smithi032.stdout:9/482: sync 2024-04-06T14:17:04.267 INFO:tasks.workunit.client.1.smithi086.stdout:7/237: dwrite d7/d10/d11/d1c/d24/d2c/d3c/d52/f55 [0,4194304] 0 2024-04-06T14:17:04.271 INFO:tasks.workunit.client.1.smithi086.stdout:7/238: rename d7/d10/c25 to d7/d10/d11/d1c/d24/d2c/c56 0 2024-04-06T14:17:04.273 INFO:tasks.workunit.client.1.smithi086.stdout:4/189: write d0/de/f11 [7135767,103172] 0 2024-04-06T14:17:04.275 INFO:tasks.workunit.client.1.smithi086.stdout:4/190: mknod d0/d2c/d38/c3d 0 2024-04-06T14:17:04.276 INFO:tasks.workunit.client.1.smithi086.stdout:4/191: write d0/de/f11 [6764600,70905] 0 2024-04-06T14:17:04.279 INFO:tasks.workunit.client.1.smithi086.stdout:4/192: fdatasync d0/d15/d19/f1c 0 2024-04-06T14:17:04.281 INFO:tasks.workunit.client.1.smithi086.stdout:4/193: mknod d0/d2c/d38/c3e 0 2024-04-06T14:17:04.283 INFO:tasks.workunit.client.1.smithi086.stdout:4/194: unlink d0/d15/d20/f32 0 2024-04-06T14:17:04.288 INFO:tasks.workunit.client.1.smithi086.stdout:4/195: dread d0/f10 [0,4194304] 0 2024-04-06T14:17:04.289 INFO:tasks.workunit.client.1.smithi086.stdout:4/196: rmdir d0/d2c 39 2024-04-06T14:17:04.290 INFO:tasks.workunit.client.1.smithi086.stdout:4/197: symlink d0/de/l3f 0 2024-04-06T14:17:04.291 INFO:tasks.workunit.client.1.smithi086.stdout:4/198: write d0/d15/f37 [533391,28338] 0 2024-04-06T14:17:04.295 INFO:tasks.workunit.client.0.smithi032.stdout:6/480: creat d4/d5/d8/d4b/fa6 x:0 0 0 2024-04-06T14:17:04.295 INFO:tasks.workunit.client.0.smithi032.stdout:4/506: symlink d0/df/d2b/d43/lad 0 2024-04-06T14:17:04.296 INFO:tasks.workunit.client.0.smithi032.stdout:6/481: write d4/d5/d35/f59 [208088,72080] 0 2024-04-06T14:17:04.297 INFO:tasks.workunit.client.0.smithi032.stdout:8/434: mknod d3/d8/d7/d14/c9c 0 2024-04-06T14:17:04.297 INFO:tasks.workunit.client.1.smithi086.stdout:4/199: dwrite d0/d15/d20/d34/f35 [0,4194304] 0 2024-04-06T14:17:04.299 INFO:tasks.workunit.client.0.smithi032.stdout:7/504: rmdir d16/d20/d26/d2d 39 2024-04-06T14:17:04.299 INFO:tasks.workunit.client.0.smithi032.stdout:7/505: readlink d16/d20/d26/l94 0 2024-04-06T14:17:04.302 INFO:tasks.workunit.client.0.smithi032.stdout:3/470: symlink d3/d65/d7d/lbe 0 2024-04-06T14:17:04.306 INFO:tasks.workunit.client.0.smithi032.stdout:4/507: unlink d0/df/d2b/f2c 0 2024-04-06T14:17:04.306 INFO:tasks.workunit.client.0.smithi032.stdout:6/482: mkdir d4/d5/d8/d1f/d61/da7 0 2024-04-06T14:17:04.308 INFO:tasks.workunit.client.0.smithi032.stdout:8/435: creat d3/d8/d7/d14/f9d x:0 0 0 2024-04-06T14:17:04.309 INFO:tasks.workunit.client.0.smithi032.stdout:7/506: mknod d16/d92/cb7 0 2024-04-06T14:17:04.310 INFO:tasks.workunit.client.0.smithi032.stdout:2/475: write d0/d5a/f1c [716627,62036] 0 2024-04-06T14:17:04.310 INFO:tasks.workunit.client.0.smithi032.stdout:2/476: stat d0/ff 0 2024-04-06T14:17:04.311 INFO:tasks.workunit.client.0.smithi032.stdout:1/560: rename d0/d10/c2a to d0/d9/d1f/d72/cb6 0 2024-04-06T14:17:04.311 INFO:tasks.workunit.client.0.smithi032.stdout:2/477: chown d0/d30/d39/d59/d67/d80/c99 18008 1 2024-04-06T14:17:04.312 INFO:tasks.workunit.client.0.smithi032.stdout:0/358: mknod d3/da/d25/d2b/d4a/c7a 0 2024-04-06T14:17:04.312 INFO:tasks.workunit.client.0.smithi032.stdout:0/359: chown d3/da/f1f 458151048 1 2024-04-06T14:17:04.336 INFO:tasks.workunit.client.1.smithi086.stdout:6/212: sync 2024-04-06T14:17:04.336 INFO:tasks.workunit.client.1.smithi086.stdout:6/213: chown d6/d13/d1a/d22/f3f 1149642 1 2024-04-06T14:17:04.338 INFO:tasks.workunit.client.1.smithi086.stdout:6/214: mknod d6/d9/d36/c4e 0 2024-04-06T14:17:04.340 INFO:tasks.workunit.client.1.smithi086.stdout:6/215: rename d6/d13/d1a/d22/c4d to d6/d9/c4f 0 2024-04-06T14:17:04.348 INFO:tasks.workunit.client.1.smithi086.stdout:6/216: dread d6/d9/db/f2f [0,4194304] 0 2024-04-06T14:17:04.351 INFO:tasks.workunit.client.1.smithi086.stdout:6/217: symlink d6/d13/d42/l50 0 2024-04-06T14:17:04.352 INFO:tasks.workunit.client.1.smithi086.stdout:6/218: chown d6/l11 1232426605 1 2024-04-06T14:17:04.357 INFO:tasks.workunit.client.1.smithi086.stdout:6/219: truncate d6/d13/d1a/f21 156990 0 2024-04-06T14:17:04.358 INFO:tasks.workunit.client.1.smithi086.stdout:6/220: chown d6/d20/f46 466513 1 2024-04-06T14:17:04.359 INFO:tasks.workunit.client.1.smithi086.stdout:6/221: fdatasync d6/d9/db/f2f 0 2024-04-06T14:17:04.360 INFO:tasks.workunit.client.1.smithi086.stdout:6/222: rmdir d6/d13 39 2024-04-06T14:17:04.362 INFO:tasks.workunit.client.1.smithi086.stdout:6/223: creat d6/d20/f51 x:0 0 0 2024-04-06T14:17:04.364 INFO:tasks.workunit.client.1.smithi086.stdout:6/224: mkdir d6/d13/d42/d52 0 2024-04-06T14:17:04.365 INFO:tasks.workunit.client.1.smithi086.stdout:6/225: rmdir d6/d13/d42 39 2024-04-06T14:17:04.369 INFO:tasks.workunit.client.1.smithi086.stdout:6/226: rename d6/d13/f2a to d6/d9/d43/f53 0 2024-04-06T14:17:04.375 INFO:tasks.workunit.client.1.smithi086.stdout:6/227: dread d6/d9/f31 [0,4194304] 0 2024-04-06T14:17:04.376 INFO:tasks.workunit.client.1.smithi086.stdout:6/228: symlink d6/d9/db/l54 0 2024-04-06T14:17:04.380 INFO:tasks.workunit.client.1.smithi086.stdout:6/229: rename d6/d9/d36 to d6/d13/d42/d55 0 2024-04-06T14:17:04.383 INFO:tasks.workunit.client.1.smithi086.stdout:6/230: unlink d6/d13/d1a/f4b 0 2024-04-06T14:17:04.383 INFO:tasks.workunit.client.1.smithi086.stdout:6/231: dread - d6/d20/f51 zero size 2024-04-06T14:17:04.392 INFO:tasks.workunit.client.1.smithi086.stdout:6/232: creat d6/d9/d43/f56 x:0 0 0 2024-04-06T14:17:04.393 INFO:tasks.workunit.client.1.smithi086.stdout:6/233: read d6/d9/db/f2f [2988508,87881] 0 2024-04-06T14:17:04.393 INFO:tasks.workunit.client.1.smithi086.stdout:6/234: chown d6/d13/d1a/d22/f35 0 1 2024-04-06T14:17:04.395 INFO:tasks.workunit.client.1.smithi086.stdout:6/235: mkdir d6/d57 0 2024-04-06T14:17:04.400 INFO:tasks.workunit.client.1.smithi086.stdout:6/236: truncate d6/f2b 4654395 0 2024-04-06T14:17:04.403 INFO:tasks.workunit.client.1.smithi086.stdout:6/237: getdents d6/d13 0 2024-04-06T14:17:04.404 INFO:tasks.workunit.client.1.smithi086.stdout:6/238: write d6/d13/d1a/f1b [821987,93085] 0 2024-04-06T14:17:04.406 INFO:tasks.workunit.client.1.smithi086.stdout:6/239: truncate d6/d13/d42/d55/d45/d4a/f4c 927113 0 2024-04-06T14:17:04.408 INFO:tasks.workunit.client.1.smithi086.stdout:6/240: unlink d6/d13/d1a/c44 0 2024-04-06T14:17:04.408 INFO:tasks.workunit.client.1.smithi086.stdout:6/241: readlink d6/d13/d42/l50 0 2024-04-06T14:17:04.410 INFO:tasks.workunit.client.1.smithi086.stdout:6/242: creat d6/d57/f58 x:0 0 0 2024-04-06T14:17:04.410 INFO:tasks.workunit.client.1.smithi086.stdout:6/243: chown d6/d20/l24 1551 1 2024-04-06T14:17:04.418 INFO:tasks.workunit.client.0.smithi032.stdout:5/476: creat d16/d18/d33/d45/fa1 x:0 0 0 2024-04-06T14:17:04.418 INFO:tasks.workunit.client.0.smithi032.stdout:8/436: mkdir d3/d8/d7/d9e 0 2024-04-06T14:17:04.419 INFO:tasks.workunit.client.0.smithi032.stdout:7/507: symlink d16/d20/d48/d6d/lb8 0 2024-04-06T14:17:04.420 INFO:tasks.workunit.client.0.smithi032.stdout:3/471: creat d3/d65/d7d/d9c/db8/fbf x:0 0 0 2024-04-06T14:17:04.420 INFO:tasks.workunit.client.0.smithi032.stdout:2/478: rename d0/d5a/f17 to d0/d30/d39/d59/d67/d74/d78/d57/d7c/fa1 0 2024-04-06T14:17:04.421 INFO:tasks.workunit.client.0.smithi032.stdout:9/483: truncate d2/d15/d1e/d65/d83/d31/f3a 917106 0 2024-04-06T14:17:04.421 INFO:tasks.workunit.client.0.smithi032.stdout:4/508: mknod d0/df/d23/d31/d50/d57/d5b/cae 0 2024-04-06T14:17:04.421 INFO:tasks.workunit.client.0.smithi032.stdout:8/437: chown d3/d8/d7/f81 2092786 1 2024-04-06T14:17:04.463 INFO:tasks.workunit.client.0.smithi032.stdout:1/561: mknod d0/d9/cb7 0 2024-04-06T14:17:04.464 INFO:tasks.workunit.client.0.smithi032.stdout:3/472: chown d3/d65/d89/fa2 198387599 1 2024-04-06T14:17:04.464 INFO:tasks.workunit.client.0.smithi032.stdout:5/477: creat d16/d18/d33/d45/d52/fa2 x:0 0 0 2024-04-06T14:17:04.465 INFO:tasks.workunit.client.0.smithi032.stdout:4/509: dread - d0/d82/d8e/fa7 zero size 2024-04-06T14:17:04.466 INFO:tasks.workunit.client.0.smithi032.stdout:1/562: rmdir d0/d41 39 2024-04-06T14:17:04.467 INFO:tasks.workunit.client.0.smithi032.stdout:7/508: write d16/d20/d26/d2d/faf [62345,53069] 0 2024-04-06T14:17:04.467 INFO:tasks.workunit.client.0.smithi032.stdout:0/360: dwrite d3/da/d25/d2b/f40 [0,4194304] 0 2024-04-06T14:17:04.468 INFO:tasks.workunit.client.1.smithi086.stdout:9/222: sync 2024-04-06T14:17:04.468 INFO:tasks.workunit.client.1.smithi086.stdout:5/250: sync 2024-04-06T14:17:04.468 INFO:tasks.workunit.client.1.smithi086.stdout:3/282: sync 2024-04-06T14:17:04.469 INFO:tasks.workunit.client.0.smithi032.stdout:3/473: chown d3/d16/d63/f8e 599584944 1 2024-04-06T14:17:04.469 INFO:tasks.workunit.client.1.smithi086.stdout:3/283: readlink d5/d2b/l2e 0 2024-04-06T14:17:04.470 INFO:tasks.workunit.client.0.smithi032.stdout:5/478: mknod d16/d58/d54/d74/ca3 0 2024-04-06T14:17:04.471 INFO:tasks.workunit.client.0.smithi032.stdout:7/509: creat d16/d20/d7a/fb9 x:0 0 0 2024-04-06T14:17:04.472 INFO:tasks.workunit.client.1.smithi086.stdout:9/223: unlink d5/d10/c3b 0 2024-04-06T14:17:04.474 INFO:tasks.workunit.client.1.smithi086.stdout:5/251: fsync d0/f3 0 2024-04-06T14:17:04.474 INFO:tasks.workunit.client.0.smithi032.stdout:9/484: creat d2/d15/d55/f9a x:0 0 0 2024-04-06T14:17:04.475 INFO:tasks.workunit.client.1.smithi086.stdout:3/284: dwrite d5/f37 [0,4194304] 0 2024-04-06T14:17:04.476 INFO:tasks.workunit.client.1.smithi086.stdout:9/224: symlink d5/d10/d14/d19/d1b/l50 0 2024-04-06T14:17:04.477 INFO:tasks.workunit.client.0.smithi032.stdout:7/510: write d16/d20/d26/d5f/f90 [456520,59315] 0 2024-04-06T14:17:04.478 INFO:tasks.workunit.client.0.smithi032.stdout:2/479: getdents d0/d30/d39/d59/d67/d74/d78/d57 0 2024-04-06T14:17:04.484 INFO:tasks.workunit.client.0.smithi032.stdout:9/485: creat d2/d15/d1e/d65/d6b/f9b x:0 0 0 2024-04-06T14:17:04.484 INFO:tasks.workunit.client.0.smithi032.stdout:5/479: dwrite d16/d18/d33/d45/fa1 [0,4194304] 0 2024-04-06T14:17:04.485 INFO:tasks.workunit.client.0.smithi032.stdout:3/474: rename d3/d65/d7d/d9c/dad/db1/c7f to d3/d65/d89/cc0 0 2024-04-06T14:17:04.487 INFO:tasks.workunit.client.0.smithi032.stdout:3/475: chown d3/d65/f39 351002109 1 2024-04-06T14:17:04.489 INFO:tasks.workunit.client.0.smithi032.stdout:2/480: mkdir d0/d5a/d3a/da2 0 2024-04-06T14:17:04.489 INFO:tasks.workunit.client.0.smithi032.stdout:0/361: link d3/da/d35/d4b/c66 d3/da/d25/d2b/d4a/c7b 0 2024-04-06T14:17:04.492 INFO:tasks.workunit.client.0.smithi032.stdout:5/480: write d16/d18/d33/d45/d52/f92 [2853491,109008] 0 2024-04-06T14:17:04.493 INFO:tasks.workunit.client.0.smithi032.stdout:0/362: write d3/da/f1d [1299196,22742] 0 2024-04-06T14:17:04.497 INFO:tasks.workunit.client.0.smithi032.stdout:3/476: creat d3/fc1 x:0 0 0 2024-04-06T14:17:04.499 INFO:tasks.workunit.client.1.smithi086.stdout:5/252: write d0/f1 [960354,91997] 0 2024-04-06T14:17:04.499 INFO:tasks.workunit.client.0.smithi032.stdout:7/511: dwrite d16/d20/d26/d5f/f8a [0,4194304] 0 2024-04-06T14:17:04.500 INFO:tasks.workunit.client.1.smithi086.stdout:3/285: truncate d5/df/f13 1871697 0 2024-04-06T14:17:04.501 INFO:tasks.workunit.client.0.smithi032.stdout:5/481: mkdir d16/d18/d33/da4 0 2024-04-06T14:17:04.501 INFO:tasks.workunit.client.0.smithi032.stdout:7/512: stat f11 0 2024-04-06T14:17:04.501 INFO:tasks.workunit.client.1.smithi086.stdout:5/253: write d0/d19/f3e [658701,91397] 0 2024-04-06T14:17:04.503 INFO:tasks.workunit.client.0.smithi032.stdout:3/477: read d3/d16/f9a [2874138,38297] 0 2024-04-06T14:17:04.503 INFO:tasks.workunit.client.0.smithi032.stdout:2/481: rmdir d0/d30/d39 39 2024-04-06T14:17:04.503 INFO:tasks.workunit.client.0.smithi032.stdout:5/482: stat d16/d18/d33/d35/l63 0 2024-04-06T14:17:04.504 INFO:tasks.workunit.client.0.smithi032.stdout:2/482: chown d0/d5a/d3a 178059958 1 2024-04-06T14:17:04.505 INFO:tasks.workunit.client.1.smithi086.stdout:9/225: dwrite d5/f20 [0,4194304] 0 2024-04-06T14:17:04.508 INFO:tasks.workunit.client.1.smithi086.stdout:5/254: rename d0/c24 to d0/d17/c4a 0 2024-04-06T14:17:04.511 INFO:tasks.workunit.client.1.smithi086.stdout:5/255: creat d0/d19/f4b x:0 0 0 2024-04-06T14:17:04.511 INFO:tasks.workunit.client.0.smithi032.stdout:7/513: mkdir d16/d20/d26/d2d/d30/d3e/d96/dba 0 2024-04-06T14:17:04.512 INFO:tasks.workunit.client.0.smithi032.stdout:5/483: chown d16/d58/d54/l7f 1 1 2024-04-06T14:17:04.513 INFO:tasks.workunit.client.0.smithi032.stdout:5/484: dread - d16/d18/d33/d35/d4d/f9c zero size 2024-04-06T14:17:04.514 INFO:tasks.workunit.client.1.smithi086.stdout:3/286: dwrite d5/df/f46 [0,4194304] 0 2024-04-06T14:17:04.515 INFO:tasks.workunit.client.0.smithi032.stdout:2/483: symlink d0/d30/la3 0 2024-04-06T14:17:04.517 INFO:tasks.workunit.client.1.smithi086.stdout:5/256: creat d0/d16/d18/f4c x:0 0 0 2024-04-06T14:17:04.518 INFO:tasks.workunit.client.0.smithi032.stdout:5/485: write f1 [7743564,117813] 0 2024-04-06T14:17:04.523 INFO:tasks.workunit.client.0.smithi032.stdout:7/514: truncate d16/d20/d26/d2d/d30/f46 14193 0 2024-04-06T14:17:04.524 INFO:tasks.workunit.client.1.smithi086.stdout:4/200: sync 2024-04-06T14:17:04.524 INFO:tasks.workunit.client.1.smithi086.stdout:1/277: sync 2024-04-06T14:17:04.524 INFO:tasks.workunit.client.1.smithi086.stdout:2/227: sync 2024-04-06T14:17:04.524 INFO:tasks.workunit.client.1.smithi086.stdout:6/244: sync 2024-04-06T14:17:04.524 INFO:tasks.workunit.client.1.smithi086.stdout:7/239: sync 2024-04-06T14:17:04.524 INFO:tasks.workunit.client.1.smithi086.stdout:0/253: sync 2024-04-06T14:17:04.524 INFO:tasks.workunit.client.1.smithi086.stdout:8/391: sync 2024-04-06T14:17:04.524 INFO:tasks.workunit.client.1.smithi086.stdout:5/257: truncate d0/d16/d18/d21/f45 85361 0 2024-04-06T14:17:04.526 INFO:tasks.workunit.client.1.smithi086.stdout:0/254: dread - d0/d15/f3e zero size 2024-04-06T14:17:04.526 INFO:tasks.workunit.client.0.smithi032.stdout:2/484: dwrite d0/d5a/f25 [0,4194304] 0 2024-04-06T14:17:04.528 INFO:tasks.workunit.client.1.smithi086.stdout:7/240: fsync d7/d10/d11/d1c/f2e 0 2024-04-06T14:17:04.529 INFO:tasks.workunit.client.0.smithi032.stdout:2/485: write d0/f6c [812920,115609] 0 2024-04-06T14:17:04.531 INFO:tasks.workunit.client.0.smithi032.stdout:4/510: truncate d0/df/d23/f32 130866 0 2024-04-06T14:17:04.531 INFO:tasks.workunit.client.1.smithi086.stdout:2/228: chown d0/d20/d27/d2b/d35/f4e 164 1 2024-04-06T14:17:04.533 INFO:tasks.workunit.client.1.smithi086.stdout:0/255: chown d0/d15/d4e/c32 219491765 1 2024-04-06T14:17:04.533 INFO:tasks.workunit.client.1.smithi086.stdout:6/245: chown d6/d13/d42/l50 213 1 2024-04-06T14:17:04.534 INFO:tasks.workunit.client.0.smithi032.stdout:4/511: dread - d0/df/d23/d31/d50/d5e/faa zero size 2024-04-06T14:17:04.534 INFO:tasks.workunit.client.0.smithi032.stdout:7/515: mkdir d16/d20/d26/d2d/d30/dbb 0 2024-04-06T14:17:04.534 INFO:tasks.workunit.client.1.smithi086.stdout:4/201: write d0/f12 [233461,92738] 0 2024-04-06T14:17:04.535 INFO:tasks.workunit.client.0.smithi032.stdout:7/516: fdatasync d16/f49 0 2024-04-06T14:17:04.536 INFO:tasks.workunit.client.1.smithi086.stdout:8/392: creat d0/d3b/f8b x:0 0 0 2024-04-06T14:17:04.536 INFO:tasks.workunit.client.0.smithi032.stdout:1/563: truncate d0/d41/f5c 1444314 0 2024-04-06T14:17:04.537 INFO:tasks.workunit.client.1.smithi086.stdout:3/287: dwrite d5/d36/f3b [4194304,4194304] 0 2024-04-06T14:17:04.538 INFO:tasks.workunit.client.1.smithi086.stdout:9/226: write d5/d10/d14/d19/d1b/f43 [5099680,7355] 0 2024-04-06T14:17:04.539 INFO:tasks.workunit.client.0.smithi032.stdout:1/564: write d0/da/d3e/f6c [529107,108324] 0 2024-04-06T14:17:04.540 INFO:tasks.workunit.client.0.smithi032.stdout:2/486: mknod d0/d30/d39/d59/d67/ca4 0 2024-04-06T14:17:04.541 INFO:tasks.workunit.client.1.smithi086.stdout:0/256: chown d0/d15/d26/d33/f57 1022 1 2024-04-06T14:17:04.544 INFO:tasks.workunit.client.0.smithi032.stdout:4/512: rename d0/df/d2b/d66/da1 to d0/d82/daf 0 2024-04-06T14:17:04.545 INFO:tasks.workunit.client.0.smithi032.stdout:5/486: link d16/d18/d33/d45/d52/c76 d16/d18/d33/d35/ca5 0 2024-04-06T14:17:04.546 INFO:tasks.workunit.client.0.smithi032.stdout:7/517: dread d16/d20/d26/d5f/f65 [0,4194304] 0 2024-04-06T14:17:04.547 INFO:tasks.workunit.client.1.smithi086.stdout:9/227: readlink d5/d10/l11 0 2024-04-06T14:17:04.547 INFO:tasks.workunit.client.1.smithi086.stdout:8/393: dread - d0/d71/d16/d39/f66 zero size 2024-04-06T14:17:04.548 INFO:tasks.workunit.client.0.smithi032.stdout:9/486: truncate d2/f21 1347810 0 2024-04-06T14:17:04.548 INFO:tasks.workunit.client.1.smithi086.stdout:6/246: mknod d6/d13/d42/d55/c59 0 2024-04-06T14:17:04.549 INFO:tasks.workunit.client.0.smithi032.stdout:1/565: dwrite d0/d10/d16/f51 [0,4194304] 0 2024-04-06T14:17:04.551 INFO:tasks.workunit.client.0.smithi032.stdout:7/518: truncate d16/f27 2398524 0 2024-04-06T14:17:04.551 INFO:tasks.workunit.client.1.smithi086.stdout:1/278: dread db/d11/d24/f45 [0,4194304] 0 2024-04-06T14:17:04.551 INFO:tasks.workunit.client.1.smithi086.stdout:0/257: chown d0/d15/d4e/c34 214525 1 2024-04-06T14:17:04.552 INFO:tasks.workunit.client.1.smithi086.stdout:5/258: getdents d0/d16/d18/d21/d23 0 2024-04-06T14:17:04.553 INFO:tasks.workunit.client.0.smithi032.stdout:1/566: fsync d0/da/d3e/f6a 0 2024-04-06T14:17:04.555 INFO:tasks.workunit.client.1.smithi086.stdout:4/202: dwrite d0/d15/d20/f22 [0,4194304] 0 2024-04-06T14:17:04.555 INFO:tasks.workunit.client.1.smithi086.stdout:9/228: stat d5/d10/d14/d19/d1b/l3d 0 2024-04-06T14:17:04.556 INFO:tasks.workunit.client.0.smithi032.stdout:0/363: write d3/f10 [5590,104721] 0 2024-04-06T14:17:04.557 INFO:tasks.workunit.client.0.smithi032.stdout:5/487: mknod d16/d58/d54/d74/ca6 0 2024-04-06T14:17:04.557 INFO:tasks.workunit.client.0.smithi032.stdout:3/478: truncate d3/f37 1354052 0 2024-04-06T14:17:04.559 INFO:tasks.workunit.client.0.smithi032.stdout:3/479: write d3/d16/fb0 [1096494,24821] 0 2024-04-06T14:17:04.560 INFO:tasks.workunit.client.1.smithi086.stdout:9/229: chown d5/d13/d1a/l38 4 1 2024-04-06T14:17:04.561 INFO:tasks.workunit.client.1.smithi086.stdout:0/258: fsync d0/d1/d4/d7/f11 0 2024-04-06T14:17:04.562 INFO:tasks.workunit.client.0.smithi032.stdout:4/513: unlink d0/df/d23/c88 0 2024-04-06T14:17:04.563 INFO:tasks.workunit.client.1.smithi086.stdout:3/288: dwrite f4 [0,4194304] 0 2024-04-06T14:17:04.563 INFO:tasks.workunit.client.1.smithi086.stdout:5/259: creat d0/d16/d18/d35/f4d x:0 0 0 2024-04-06T14:17:04.563 INFO:tasks.workunit.client.0.smithi032.stdout:3/480: write d3/d65/d7d/d9c/dad/fb3 [915255,55982] 0 2024-04-06T14:17:04.565 INFO:tasks.workunit.client.0.smithi032.stdout:3/481: dread - d3/fb9 zero size 2024-04-06T14:17:04.567 INFO:tasks.workunit.client.0.smithi032.stdout:7/519: write d16/d20/d26/d2d/f9b [4889677,124188] 0 2024-04-06T14:17:04.568 INFO:tasks.workunit.client.1.smithi086.stdout:4/203: creat d0/f40 x:0 0 0 2024-04-06T14:17:04.568 INFO:tasks.workunit.client.1.smithi086.stdout:9/230: dread - d5/d10/d14/d19/d25/f2f zero size 2024-04-06T14:17:04.569 INFO:tasks.workunit.client.1.smithi086.stdout:3/289: write d5/f2d [368154,108748] 0 2024-04-06T14:17:04.570 INFO:tasks.workunit.client.0.smithi032.stdout:8/438: sync 2024-04-06T14:17:04.570 INFO:tasks.workunit.client.0.smithi032.stdout:6/483: sync 2024-04-06T14:17:04.571 INFO:tasks.workunit.client.1.smithi086.stdout:4/204: read d0/de/f1a [441332,13359] 0 2024-04-06T14:17:04.571 INFO:tasks.workunit.client.1.smithi086.stdout:5/260: creat d0/d16/f4e x:0 0 0 2024-04-06T14:17:04.573 INFO:tasks.workunit.client.0.smithi032.stdout:8/439: chown d3/d8/d12/f38 379221923 1 2024-04-06T14:17:04.575 INFO:tasks.workunit.client.1.smithi086.stdout:8/394: dwrite d0/d71/d16/d39/f66 [0,4194304] 0 2024-04-06T14:17:04.578 INFO:tasks.workunit.client.0.smithi032.stdout:6/484: read - d4/d5/d28/d65/fa3 zero size 2024-04-06T14:17:04.578 INFO:tasks.workunit.client.0.smithi032.stdout:8/440: write d3/d8/d3a/f42 [1934906,86320] 0 2024-04-06T14:17:04.578 INFO:tasks.workunit.client.1.smithi086.stdout:0/259: mkdir d0/d1/d2/d5b 0 2024-04-06T14:17:04.580 INFO:tasks.workunit.client.0.smithi032.stdout:5/488: rename d16/l6c to d16/d18/d33/d35/la7 0 2024-04-06T14:17:04.581 INFO:tasks.workunit.client.1.smithi086.stdout:0/260: chown d0/d15/d4e/c34 18004 1 2024-04-06T14:17:04.585 INFO:tasks.workunit.client.0.smithi032.stdout:5/489: chown d16/d18/d33/d45/d52/d39 35786966 1 2024-04-06T14:17:04.585 INFO:tasks.workunit.client.1.smithi086.stdout:9/231: rmdir d5/d10 39 2024-04-06T14:17:04.585 INFO:tasks.workunit.client.1.smithi086.stdout:0/261: dread - d0/d15/d26/f4b zero size 2024-04-06T14:17:04.587 INFO:tasks.workunit.client.0.smithi032.stdout:8/441: fsync d3/d8/d12/d2f/f77 0 2024-04-06T14:17:04.588 INFO:tasks.workunit.client.0.smithi032.stdout:1/567: fsync d0/d10/d16/fa6 0 2024-04-06T14:17:04.588 INFO:tasks.workunit.client.0.smithi032.stdout:3/482: creat d3/d65/d7d/d9c/db8/fc2 x:0 0 0 2024-04-06T14:17:04.588 INFO:tasks.workunit.client.0.smithi032.stdout:7/520: dwrite d16/d20/d26/f44 [0,4194304] 0 2024-04-06T14:17:04.589 INFO:tasks.workunit.client.1.smithi086.stdout:8/395: dread d0/d71/d16/d39/f66 [0,4194304] 0 2024-04-06T14:17:04.590 INFO:tasks.workunit.client.0.smithi032.stdout:2/487: truncate d0/d30/f43 7726416 0 2024-04-06T14:17:04.591 INFO:tasks.workunit.client.1.smithi086.stdout:9/232: unlink d5/f27 0 2024-04-06T14:17:04.591 INFO:tasks.workunit.client.1.smithi086.stdout:4/205: dread d0/d2c/d30/d24/f2d [0,4194304] 0 2024-04-06T14:17:04.592 INFO:tasks.workunit.client.1.smithi086.stdout:0/262: mknod d0/d1/d2/d1a/c5c 0 2024-04-06T14:17:04.592 INFO:tasks.workunit.client.1.smithi086.stdout:6/247: dwrite d6/f3c [0,4194304] 0 2024-04-06T14:17:04.594 INFO:tasks.workunit.client.0.smithi032.stdout:8/442: write d3/d8/d7/d1a/f71 [1847602,27329] 0 2024-04-06T14:17:04.597 INFO:tasks.workunit.client.1.smithi086.stdout:5/261: rmdir d0/d16 39 2024-04-06T14:17:04.597 INFO:tasks.workunit.client.0.smithi032.stdout:6/485: creat d4/d5/d8/d1f/d51/fa8 x:0 0 0 2024-04-06T14:17:04.598 INFO:tasks.workunit.client.0.smithi032.stdout:9/487: truncate d2/d15/d1e/f1f 216778 0 2024-04-06T14:17:04.598 INFO:tasks.workunit.client.1.smithi086.stdout:8/396: symlink d0/d13/l8c 0 2024-04-06T14:17:04.599 INFO:tasks.workunit.client.1.smithi086.stdout:0/263: fdatasync d0/d1/d4/d7/f27 0 2024-04-06T14:17:04.600 INFO:tasks.workunit.client.1.smithi086.stdout:0/264: stat d0/d1/c3a 0 2024-04-06T14:17:04.601 INFO:tasks.workunit.client.1.smithi086.stdout:4/206: unlink d0/d2c/d30/d24/c2a 0 2024-04-06T14:17:04.602 INFO:tasks.workunit.client.0.smithi032.stdout:9/488: dread - d2/d15/d1e/d65/d83/d31/f40 zero size 2024-04-06T14:17:04.603 INFO:tasks.workunit.client.1.smithi086.stdout:4/207: read d0/f12 [323004,22689] 0 2024-04-06T14:17:04.603 INFO:tasks.workunit.client.0.smithi032.stdout:9/489: readlink d2/d15/l7b 0 2024-04-06T14:17:04.604 INFO:tasks.workunit.client.1.smithi086.stdout:0/265: dread - d0/d15/f3e zero size 2024-04-06T14:17:04.604 INFO:tasks.workunit.client.1.smithi086.stdout:6/248: read - d6/d9/d43/f56 zero size 2024-04-06T14:17:04.606 INFO:tasks.workunit.client.0.smithi032.stdout:1/568: mkdir d0/d10/d16/db8 0 2024-04-06T14:17:04.608 INFO:tasks.workunit.client.1.smithi086.stdout:3/290: dwrite d5/d26/f3d [0,4194304] 0 2024-04-06T14:17:04.609 INFO:tasks.workunit.client.0.smithi032.stdout:6/486: symlink d4/d5/d34/la9 0 2024-04-06T14:17:04.609 INFO:tasks.workunit.client.0.smithi032.stdout:8/443: write d3/d8/d12/d2f/f62 [603997,72982] 0 2024-04-06T14:17:04.609 INFO:tasks.workunit.client.1.smithi086.stdout:3/291: chown d5/df/l1e 487 1 2024-04-06T14:17:04.612 INFO:tasks.workunit.client.1.smithi086.stdout:4/208: symlink d0/d2c/d30/d24/l41 0 2024-04-06T14:17:04.612 INFO:tasks.workunit.client.1.smithi086.stdout:8/397: dread d0/db/d32/f5d [0,4194304] 0 2024-04-06T14:17:04.613 INFO:tasks.workunit.client.1.smithi086.stdout:4/209: readlink d0/l9 0 2024-04-06T14:17:04.614 INFO:tasks.workunit.client.0.smithi032.stdout:7/521: symlink d16/d20/lbc 0 2024-04-06T14:17:04.614 INFO:tasks.workunit.client.1.smithi086.stdout:6/249: symlink d6/d13/d42/l5a 0 2024-04-06T14:17:04.614 INFO:tasks.workunit.client.1.smithi086.stdout:3/292: write d5/d33/f47 [2629622,14911] 0 2024-04-06T14:17:04.616 INFO:tasks.workunit.client.1.smithi086.stdout:0/266: mkdir d0/d1/d1b/d44/d5d 0 2024-04-06T14:17:04.617 INFO:tasks.workunit.client.0.smithi032.stdout:5/490: creat d16/d18/d33/fa8 x:0 0 0 2024-04-06T14:17:04.618 INFO:tasks.workunit.client.0.smithi032.stdout:4/514: write d0/df/d23/d31/f68 [1537264,33646] 0 2024-04-06T14:17:04.622 INFO:tasks.workunit.client.0.smithi032.stdout:4/515: chown d0/df/d23/d31/l36 129823 1 2024-04-06T14:17:04.622 INFO:tasks.workunit.client.0.smithi032.stdout:0/364: dread d3/da/d25/f54 [0,4194304] 0 2024-04-06T14:17:04.624 INFO:tasks.workunit.client.0.smithi032.stdout:5/491: symlink d16/d58/d54/d94/la9 0 2024-04-06T14:17:04.625 INFO:tasks.workunit.client.1.smithi086.stdout:5/262: dwrite d0/d19/f3e [0,4194304] 0 2024-04-06T14:17:04.625 INFO:tasks.workunit.client.1.smithi086.stdout:9/233: write d5/f1c [772483,59909] 0 2024-04-06T14:17:04.626 INFO:tasks.workunit.client.0.smithi032.stdout:9/490: unlink d2/d15/d1e/d65/d83/d7e/l48 0 2024-04-06T14:17:04.628 INFO:tasks.workunit.client.1.smithi086.stdout:3/293: write d5/d33/f47 [1361331,32017] 0 2024-04-06T14:17:04.629 INFO:tasks.workunit.client.1.smithi086.stdout:3/294: write d5/f48 [933241,33481] 0 2024-04-06T14:17:04.629 INFO:tasks.workunit.client.0.smithi032.stdout:3/483: rename d3/f80 to d3/d65/d7d/d9c/dad/db1/fc3 0 2024-04-06T14:17:04.637 INFO:tasks.workunit.client.0.smithi032.stdout:6/487: dread d4/d5/d28/d54/f85 [0,4194304] 0 2024-04-06T14:17:04.637 INFO:tasks.workunit.client.0.smithi032.stdout:1/569: link d0/d9/d1f/f91 d0/d10/d16/d3f/fb9 0 2024-04-06T14:17:04.638 INFO:tasks.workunit.client.1.smithi086.stdout:6/250: creat d6/d13/d42/d55/f5b x:0 0 0 2024-04-06T14:17:04.638 INFO:tasks.workunit.client.1.smithi086.stdout:0/267: dread d0/d1/d4/de/f3c [0,4194304] 0 2024-04-06T14:17:04.638 INFO:tasks.workunit.client.1.smithi086.stdout:4/210: write d0/f3 [4786682,85025] 0 2024-04-06T14:17:04.639 INFO:tasks.workunit.client.1.smithi086.stdout:4/211: truncate d0/d2c/d30/f3b 413993 0 2024-04-06T14:17:04.640 INFO:tasks.workunit.client.0.smithi032.stdout:5/492: mknod d16/d58/d54/d94/caa 0 2024-04-06T14:17:04.641 INFO:tasks.workunit.client.1.smithi086.stdout:3/295: link d5/d26/c43 d5/d33/c4a 0 2024-04-06T14:17:04.642 INFO:tasks.workunit.client.0.smithi032.stdout:0/365: mkdir d3/da/d25/d2b/d4a/d79/d7c 0 2024-04-06T14:17:04.642 INFO:tasks.workunit.client.0.smithi032.stdout:1/570: dwrite d0/d9/d1f/d7c/f95 [0,4194304] 0 2024-04-06T14:17:04.644 INFO:tasks.workunit.client.1.smithi086.stdout:4/212: dread d0/de/f26 [0,4194304] 0 2024-04-06T14:17:04.645 INFO:tasks.workunit.client.0.smithi032.stdout:3/484: rmdir d3/d65/d89 39 2024-04-06T14:17:04.648 INFO:tasks.workunit.client.1.smithi086.stdout:5/263: write d0/f3 [229559,40774] 0 2024-04-06T14:17:04.649 INFO:tasks.workunit.client.1.smithi086.stdout:0/268: dwrite d0/d15/d26/d33/f57 [0,4194304] 0 2024-04-06T14:17:04.650 INFO:tasks.workunit.client.1.smithi086.stdout:0/269: dread - d0/d1/d1b/d44/f5a zero size 2024-04-06T14:17:04.651 INFO:tasks.workunit.client.0.smithi032.stdout:9/491: dread d2/d15/d1e/d65/d83/d31/d39/f58 [0,4194304] 0 2024-04-06T14:17:04.652 INFO:tasks.workunit.client.0.smithi032.stdout:2/488: dwrite d0/d30/f43 [0,4194304] 0 2024-04-06T14:17:04.653 INFO:tasks.workunit.client.1.smithi086.stdout:4/213: write d0/de/f26 [730799,70225] 0 2024-04-06T14:17:04.653 INFO:tasks.workunit.client.1.smithi086.stdout:8/398: dread d0/d71/f3c [4194304,4194304] 0 2024-04-06T14:17:04.654 INFO:tasks.workunit.client.1.smithi086.stdout:0/270: write d0/d15/d26/f2c [2651298,19531] 0 2024-04-06T14:17:04.654 INFO:tasks.workunit.client.0.smithi032.stdout:4/516: rename d0/df/d1f/d2e/f41 to d0/df/d2b/d46/d58/fb0 0 2024-04-06T14:17:04.655 INFO:tasks.workunit.client.1.smithi086.stdout:4/214: write d0/f3 [3656285,15213] 0 2024-04-06T14:17:04.656 INFO:tasks.workunit.client.0.smithi032.stdout:5/493: chown d16/d18/d33/d45/c32 250208889 1 2024-04-06T14:17:04.657 INFO:tasks.workunit.client.0.smithi032.stdout:4/517: stat d0/df/d23/d31/d50/d5e 0 2024-04-06T14:17:04.658 INFO:tasks.workunit.client.1.smithi086.stdout:5/264: rename d0/d16/d18/d21/d23/f1f to d0/d17/d2d/f4f 0 2024-04-06T14:17:04.659 INFO:tasks.workunit.client.1.smithi086.stdout:5/265: write d0/f3 [1353905,59629] 0 2024-04-06T14:17:04.660 INFO:tasks.workunit.client.1.smithi086.stdout:0/271: creat d0/d15/d4e/f5e x:0 0 0 2024-04-06T14:17:04.660 INFO:tasks.workunit.client.0.smithi032.stdout:0/366: write d3/f2e [4897742,74003] 0 2024-04-06T14:17:04.662 INFO:tasks.workunit.client.1.smithi086.stdout:5/266: chown d0/d16/d18/f1a 68 1 2024-04-06T14:17:04.664 INFO:tasks.workunit.client.1.smithi086.stdout:7/241: sync 2024-04-06T14:17:04.664 INFO:tasks.workunit.client.0.smithi032.stdout:6/488: creat d4/d5/d19/d96/faa x:0 0 0 2024-04-06T14:17:04.665 INFO:tasks.workunit.client.0.smithi032.stdout:8/444: write d3/d8/d7/fe [1526658,119195] 0 2024-04-06T14:17:04.665 INFO:tasks.workunit.client.0.smithi032.stdout:7/522: getdents d16/d20/d26/d2d/d30/d3e/da7 0 2024-04-06T14:17:04.666 INFO:tasks.workunit.client.0.smithi032.stdout:9/492: mkdir d2/d15/d1e/d65/d9c 0 2024-04-06T14:17:04.666 INFO:tasks.workunit.client.1.smithi086.stdout:8/399: dwrite d0/d71/f3c [0,4194304] 0 2024-04-06T14:17:04.666 INFO:tasks.workunit.client.1.smithi086.stdout:5/267: getdents d0/d16 0 2024-04-06T14:17:04.667 INFO:tasks.workunit.client.0.smithi032.stdout:0/367: write d3/da/d25/d2b/f32 [2630990,129073] 0 2024-04-06T14:17:04.667 INFO:tasks.workunit.client.1.smithi086.stdout:7/242: mknod d7/d10/c57 0 2024-04-06T14:17:04.668 INFO:tasks.workunit.client.1.smithi086.stdout:0/272: dread d0/d15/d4e/d48/f31 [0,4194304] 0 2024-04-06T14:17:04.668 INFO:tasks.workunit.client.1.smithi086.stdout:7/243: chown d7/d10/d30/f50 14543565 1 2024-04-06T14:17:04.668 INFO:tasks.workunit.client.1.smithi086.stdout:5/268: chown d0/d43/c48 164 1 2024-04-06T14:17:04.670 INFO:tasks.workunit.client.1.smithi086.stdout:0/273: write d0/d15/d26/f4b [417117,93106] 0 2024-04-06T14:17:04.673 INFO:tasks.workunit.client.0.smithi032.stdout:1/571: rename d0/d9/d1f/d72/fa2 to d0/da/fba 0 2024-04-06T14:17:04.674 INFO:tasks.workunit.client.1.smithi086.stdout:5/269: mkdir d0/d16/d18/d2a/d50 0 2024-04-06T14:17:04.675 INFO:tasks.workunit.client.0.smithi032.stdout:4/518: unlink d0/f1e 0 2024-04-06T14:17:04.675 INFO:tasks.workunit.client.0.smithi032.stdout:2/489: read d0/d30/d39/d59/d67/d74/d78/d57/d7c/fa1 [2201613,62472] 0 2024-04-06T14:17:04.676 INFO:tasks.workunit.client.0.smithi032.stdout:5/494: dwrite d16/d18/d33/d45/d87/f8c [0,4194304] 0 2024-04-06T14:17:04.678 INFO:tasks.workunit.client.0.smithi032.stdout:6/489: write d4/d5/d8/d1f/f37 [2902067,28857] 0 2024-04-06T14:17:04.678 INFO:tasks.workunit.client.1.smithi086.stdout:0/274: fsync d0/f16 0 2024-04-06T14:17:04.679 INFO:tasks.workunit.client.1.smithi086.stdout:0/275: fsync d0/f17 0 2024-04-06T14:17:04.682 INFO:tasks.workunit.client.1.smithi086.stdout:5/270: symlink d0/d17/d2d/l51 0 2024-04-06T14:17:04.682 INFO:tasks.workunit.client.0.smithi032.stdout:8/445: mknod d3/d8/d12/d2f/d4a/d60/d82/d84/c9f 0 2024-04-06T14:17:04.682 INFO:tasks.workunit.client.0.smithi032.stdout:5/495: fsync d16/d18/d33/d35/f44 0 2024-04-06T14:17:04.685 INFO:tasks.workunit.client.1.smithi086.stdout:5/271: dread - d0/d16/d18/f4c zero size 2024-04-06T14:17:04.688 INFO:tasks.workunit.client.0.smithi032.stdout:0/368: chown d3/da/d41/c5b 0 1 2024-04-06T14:17:04.690 INFO:tasks.workunit.client.1.smithi086.stdout:5/272: mkdir d0/d17/d2d/d52 0 2024-04-06T14:17:04.691 INFO:tasks.workunit.client.0.smithi032.stdout:6/490: chown d4/d5/d19/c9e 2428312 1 2024-04-06T14:17:04.693 INFO:tasks.workunit.client.0.smithi032.stdout:8/446: mkdir d3/d8/d12/d83/d89/da0 0 2024-04-06T14:17:04.693 INFO:tasks.workunit.client.1.smithi086.stdout:0/276: dwrite d0/ff [8388608,4194304] 0 2024-04-06T14:17:04.694 INFO:tasks.workunit.client.0.smithi032.stdout:3/485: dwrite d3/d65/d7d/d9c/dad/db1/f6c [0,4194304] 0 2024-04-06T14:17:04.696 INFO:tasks.workunit.client.0.smithi032.stdout:9/493: dread d2/d15/d1e/f33 [0,4194304] 0 2024-04-06T14:17:04.696 INFO:tasks.workunit.client.1.smithi086.stdout:3/296: rmdir d5/d33/d3f 0 2024-04-06T14:17:04.697 INFO:tasks.workunit.client.0.smithi032.stdout:7/523: dwrite d16/d20/d26/d5f/fb0 [0,4194304] 0 2024-04-06T14:17:04.697 INFO:tasks.workunit.client.0.smithi032.stdout:2/490: write d0/d30/d39/d59/d67/d74/d78/d57/f90 [855827,40011] 0 2024-04-06T14:17:04.698 INFO:tasks.workunit.client.1.smithi086.stdout:9/234: dwrite d5/d10/d14/d19/f4b [0,4194304] 0 2024-04-06T14:17:04.699 INFO:tasks.workunit.client.0.smithi032.stdout:5/496: chown d16/d18/d33/fa8 3358610 1 2024-04-06T14:17:04.699 INFO:tasks.workunit.client.0.smithi032.stdout:6/491: chown d4/d5/d19/d25 10350 1 2024-04-06T14:17:04.702 INFO:tasks.workunit.client.0.smithi032.stdout:0/369: write d3/da/d25/d2b/f32 [505929,78687] 0 2024-04-06T14:17:04.703 INFO:tasks.workunit.client.1.smithi086.stdout:5/273: symlink d0/d19/l53 0 2024-04-06T14:17:04.705 INFO:tasks.workunit.client.0.smithi032.stdout:3/486: chown d3/d65/d7d/d9c/dad/db1/f9b 15 1 2024-04-06T14:17:04.707 INFO:tasks.workunit.client.1.smithi086.stdout:0/277: creat d0/d1/d1b/d44/d5d/f5f x:0 0 0 2024-04-06T14:17:04.708 INFO:tasks.workunit.client.0.smithi032.stdout:7/524: truncate d16/d20/d26/f58 1932366 0 2024-04-06T14:17:04.708 INFO:tasks.workunit.client.1.smithi086.stdout:3/297: rename d5/d26/c43 to d5/d36/c4b 0 2024-04-06T14:17:04.710 INFO:tasks.workunit.client.0.smithi032.stdout:0/370: write d3/da/d25/d2b/d4a/f62 [643246,96234] 0 2024-04-06T14:17:04.710 INFO:tasks.workunit.client.0.smithi032.stdout:9/494: symlink d2/d15/d55/d59/l9d 0 2024-04-06T14:17:04.712 INFO:tasks.workunit.client.1.smithi086.stdout:5/274: fdatasync d0/d19/f3c 0 2024-04-06T14:17:04.712 INFO:tasks.workunit.client.1.smithi086.stdout:9/235: creat d5/d10/d14/f51 x:0 0 0 2024-04-06T14:17:04.712 INFO:tasks.workunit.client.1.smithi086.stdout:9/236: stat d5/d13/f4f 0 2024-04-06T14:17:04.713 INFO:tasks.workunit.client.0.smithi032.stdout:1/572: dread d0/f2c [4194304,4194304] 0 2024-04-06T14:17:04.713 INFO:tasks.workunit.client.0.smithi032.stdout:8/447: creat d3/d8/d12/d2f/d4a/d60/d82/fa1 x:0 0 0 2024-04-06T14:17:04.721 INFO:tasks.workunit.client.1.smithi086.stdout:5/275: readlink d0/d16/d18/l34 0 2024-04-06T14:17:04.722 INFO:tasks.workunit.client.1.smithi086.stdout:3/298: fdatasync d5/d33/f35 0 2024-04-06T14:17:04.722 INFO:tasks.workunit.client.1.smithi086.stdout:0/278: dwrite d0/d1/d2/d1a/f56 [0,4194304] 0 2024-04-06T14:17:04.722 INFO:tasks.workunit.client.1.smithi086.stdout:5/276: chown d0/d16/d18/f1a 1 1 2024-04-06T14:17:04.723 INFO:tasks.workunit.client.0.smithi032.stdout:3/487: rename d3/d65/f88 to d3/d65/d7d/d9c/dad/db1/d3c/fc4 0 2024-04-06T14:17:04.724 INFO:tasks.workunit.client.1.smithi086.stdout:5/277: readlink d0/d16/d18/lc 0 2024-04-06T14:17:04.728 INFO:tasks.workunit.client.0.smithi032.stdout:1/573: symlink d0/d9/d1f/d7c/lbb 0 2024-04-06T14:17:04.729 INFO:tasks.workunit.client.0.smithi032.stdout:6/492: symlink d4/d5/d8/d1f/d61/da7/lab 0 2024-04-06T14:17:04.729 INFO:tasks.workunit.client.1.smithi086.stdout:0/279: unlink d0/d1/d1b/d46/c4a 0 2024-04-06T14:17:04.730 INFO:tasks.workunit.client.1.smithi086.stdout:9/237: link d5/l6 d5/d18/l52 0 2024-04-06T14:17:04.730 INFO:tasks.workunit.client.1.smithi086.stdout:5/278: chown d0/d19/f3e 145151255 1 2024-04-06T14:17:04.730 INFO:tasks.workunit.client.0.smithi032.stdout:8/448: fsync d3/d8/d12/f6f 0 2024-04-06T14:17:04.731 INFO:tasks.workunit.client.1.smithi086.stdout:3/299: dwrite d5/f3e [0,4194304] 0 2024-04-06T14:17:04.733 INFO:tasks.workunit.client.0.smithi032.stdout:1/574: chown d0/d9/d1f/d72/f7e 1 1 2024-04-06T14:17:04.733 INFO:tasks.workunit.client.1.smithi086.stdout:0/280: rmdir d0/d1/d2/d1a 39 2024-04-06T14:17:04.735 INFO:tasks.workunit.client.0.smithi032.stdout:7/525: mkdir d16/d20/dbd 0 2024-04-06T14:17:04.735 INFO:tasks.workunit.client.1.smithi086.stdout:9/238: creat d5/d10/f53 x:0 0 0 2024-04-06T14:17:04.736 INFO:tasks.workunit.client.0.smithi032.stdout:0/371: dwrite d3/da/d41/f43 [0,4194304] 0 2024-04-06T14:17:04.736 INFO:tasks.workunit.client.0.smithi032.stdout:3/488: dwrite d3/fc1 [0,4194304] 0 2024-04-06T14:17:04.736 INFO:tasks.workunit.client.1.smithi086.stdout:9/239: chown d5/d18 1 1 2024-04-06T14:17:04.737 INFO:tasks.workunit.client.1.smithi086.stdout:5/279: unlink d0/d16/d18/d21/l44 0 2024-04-06T14:17:04.739 INFO:tasks.workunit.client.0.smithi032.stdout:9/495: mknod d2/d15/d1e/d65/d83/d31/c9e 0 2024-04-06T14:17:04.739 INFO:tasks.workunit.client.0.smithi032.stdout:6/493: truncate d4/d5/d8/d1f/d61/f66 369907 0 2024-04-06T14:17:04.741 INFO:tasks.workunit.client.1.smithi086.stdout:3/300: rename d5/d33/f35 to d5/d42/f4c 0 2024-04-06T14:17:04.742 INFO:tasks.workunit.client.1.smithi086.stdout:3/301: read f4 [1582890,107795] 0 2024-04-06T14:17:04.743 INFO:tasks.workunit.client.0.smithi032.stdout:5/497: creat d16/fab x:0 0 0 2024-04-06T14:17:04.744 INFO:tasks.workunit.client.0.smithi032.stdout:2/491: link d0/d4f/c91 d0/d6e/d8d/ca5 0 2024-04-06T14:17:04.745 INFO:tasks.workunit.client.0.smithi032.stdout:7/526: symlink d16/d92/lbe 0 2024-04-06T14:17:04.747 INFO:tasks.workunit.client.0.smithi032.stdout:6/494: write d4/d5/fa0 [950014,127674] 0 2024-04-06T14:17:04.747 INFO:tasks.workunit.client.0.smithi032.stdout:8/449: mknod d3/d8/d12/d2f/d4a/ca2 0 2024-04-06T14:17:04.748 INFO:tasks.workunit.client.1.smithi086.stdout:3/302: mknod d5/d42/c4d 0 2024-04-06T14:17:04.749 INFO:tasks.workunit.client.0.smithi032.stdout:3/489: mkdir d3/d65/d7d/d9c/dad/dc5 0 2024-04-06T14:17:04.750 INFO:tasks.workunit.client.1.smithi086.stdout:0/281: creat d0/d1/d2/d1a/d51/f60 x:0 0 0 2024-04-06T14:17:04.752 INFO:tasks.workunit.client.1.smithi086.stdout:5/280: link d0/f1 d0/f54 0 2024-04-06T14:17:04.754 INFO:tasks.workunit.client.0.smithi032.stdout:2/492: symlink d0/la6 0 2024-04-06T14:17:04.754 INFO:tasks.workunit.client.1.smithi086.stdout:3/303: write d5/f44 [1878441,36700] 0 2024-04-06T14:17:04.755 INFO:tasks.workunit.client.1.smithi086.stdout:5/281: rmdir d0/d16/d18/d2a 39 2024-04-06T14:17:04.756 INFO:tasks.workunit.client.0.smithi032.stdout:7/527: creat d16/d20/d48/fbf x:0 0 0 2024-04-06T14:17:04.757 INFO:tasks.workunit.client.0.smithi032.stdout:6/495: write d4/d5/f4c [997320,120050] 0 2024-04-06T14:17:04.757 INFO:tasks.workunit.client.0.smithi032.stdout:8/450: read d3/d8/f6d [2926777,92456] 0 2024-04-06T14:17:04.758 INFO:tasks.workunit.client.0.smithi032.stdout:1/575: dread d0/d10/d16/f51 [0,4194304] 0 2024-04-06T14:17:04.762 INFO:tasks.workunit.client.1.smithi086.stdout:0/282: dwrite d0/d1/d1b/d44/f5a [0,4194304] 0 2024-04-06T14:17:04.762 INFO:tasks.workunit.client.0.smithi032.stdout:0/372: dwrite d3/f5a [0,4194304] 0 2024-04-06T14:17:04.765 INFO:tasks.workunit.client.0.smithi032.stdout:1/576: symlink d0/da/d3e/d75/lbc 0 2024-04-06T14:17:04.770 INFO:tasks.workunit.client.0.smithi032.stdout:6/496: rmdir d4/d5/d35 39 2024-04-06T14:17:04.771 INFO:tasks.workunit.client.1.smithi086.stdout:5/282: dwrite d0/d16/d18/ff [0,4194304] 0 2024-04-06T14:17:04.773 INFO:tasks.workunit.client.0.smithi032.stdout:1/577: dread - d0/f90 zero size 2024-04-06T14:17:04.774 INFO:tasks.workunit.client.1.smithi086.stdout:0/283: dread d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:04.774 INFO:tasks.workunit.client.0.smithi032.stdout:8/451: creat d3/d8/d12/d2f/d4a/d60/d82/fa3 x:0 0 0 2024-04-06T14:17:04.777 INFO:tasks.workunit.client.1.smithi086.stdout:5/283: dread d0/f4 [0,4194304] 0 2024-04-06T14:17:04.778 INFO:tasks.workunit.client.1.smithi086.stdout:5/284: chown d0/f4 244377 1 2024-04-06T14:17:04.779 INFO:tasks.workunit.client.0.smithi032.stdout:7/528: dwrite fb [0,4194304] 0 2024-04-06T14:17:04.779 INFO:tasks.workunit.client.0.smithi032.stdout:3/490: getdents d3/d65/d7d/d9c 0 2024-04-06T14:17:04.783 INFO:tasks.workunit.client.0.smithi032.stdout:2/493: dwrite d0/d6e/f72 [0,4194304] 0 2024-04-06T14:17:04.783 INFO:tasks.workunit.client.0.smithi032.stdout:5/498: write d16/d18/f23 [949716,29519] 0 2024-04-06T14:17:04.785 INFO:tasks.workunit.client.1.smithi086.stdout:3/304: truncate d5/df/f2c 728126 0 2024-04-06T14:17:04.785 INFO:tasks.workunit.client.1.smithi086.stdout:0/284: write d0/d1/d4/f43 [51474,60888] 0 2024-04-06T14:17:04.786 INFO:tasks.workunit.client.0.smithi032.stdout:8/452: mknod d3/d8/d7/ca4 0 2024-04-06T14:17:04.788 INFO:tasks.workunit.client.0.smithi032.stdout:1/578: creat d0/da/d3e/fbd x:0 0 0 2024-04-06T14:17:04.788 INFO:tasks.workunit.client.0.smithi032.stdout:6/497: mkdir d4/d5/d18/d23/d82/d9d/dac 0 2024-04-06T14:17:04.791 INFO:tasks.workunit.client.0.smithi032.stdout:9/496: dread d2/d15/d1e/d65/d83/f32 [0,4194304] 0 2024-04-06T14:17:04.791 INFO:tasks.workunit.client.0.smithi032.stdout:3/491: fdatasync d3/f4 0 2024-04-06T14:17:04.791 INFO:tasks.workunit.client.1.smithi086.stdout:5/285: creat d0/d16/d18/d21/d23/f55 x:0 0 0 2024-04-06T14:17:04.792 INFO:tasks.workunit.client.0.smithi032.stdout:5/499: chown d16/d58/d54/d74/l97 244 1 2024-04-06T14:17:04.794 INFO:tasks.workunit.client.1.smithi086.stdout:0/285: creat d0/d1/d25/f61 x:0 0 0 2024-04-06T14:17:04.794 INFO:tasks.workunit.client.1.smithi086.stdout:0/286: chown d0/d1 1 1 2024-04-06T14:17:04.797 INFO:tasks.workunit.client.0.smithi032.stdout:1/579: fsync d0/d9/d1f/f91 0 2024-04-06T14:17:04.799 INFO:tasks.workunit.client.0.smithi032.stdout:2/494: creat d0/d30/d39/d59/d67/d74/d78/d65/fa7 x:0 0 0 2024-04-06T14:17:04.800 INFO:tasks.workunit.client.0.smithi032.stdout:3/492: mkdir d3/d65/d7d/dc6 0 2024-04-06T14:17:04.802 INFO:tasks.workunit.client.0.smithi032.stdout:3/493: write d3/fae [297905,29216] 0 2024-04-06T14:17:04.803 INFO:tasks.workunit.client.0.smithi032.stdout:6/498: dwrite d4/d5/d18/f64 [0,4194304] 0 2024-04-06T14:17:04.804 INFO:tasks.workunit.client.1.smithi086.stdout:0/287: dwrite d0/d15/f3e [0,4194304] 0 2024-04-06T14:17:04.805 INFO:tasks.workunit.client.1.smithi086.stdout:3/305: dread f4 [4194304,4194304] 0 2024-04-06T14:17:04.806 INFO:tasks.workunit.client.0.smithi032.stdout:5/500: fdatasync d16/d18/d33/d35/f6a 0 2024-04-06T14:17:04.806 INFO:tasks.workunit.client.0.smithi032.stdout:3/494: write d3/f24 [47639,102923] 0 2024-04-06T14:17:04.806 INFO:tasks.workunit.client.1.smithi086.stdout:0/288: dread - d0/d15/d4e/f5e zero size 2024-04-06T14:17:04.807 INFO:tasks.workunit.client.0.smithi032.stdout:2/495: rmdir d0/d30/d39/d59/d67/d74/d78/d57 39 2024-04-06T14:17:04.809 INFO:tasks.workunit.client.1.smithi086.stdout:0/289: chown d0/d15/c30 7989 1 2024-04-06T14:17:04.809 INFO:tasks.workunit.client.1.smithi086.stdout:3/306: mkdir d5/d2b/d4e 0 2024-04-06T14:17:04.809 INFO:tasks.workunit.client.0.smithi032.stdout:8/453: write d3/d8/d12/d2f/d4a/f94 [4983705,27732] 0 2024-04-06T14:17:04.810 INFO:tasks.workunit.client.1.smithi086.stdout:0/290: readlink d0/d15/l1e 0 2024-04-06T14:17:04.813 INFO:tasks.workunit.client.0.smithi032.stdout:6/499: mkdir d4/d5/d83/dad 0 2024-04-06T14:17:04.814 INFO:tasks.workunit.client.0.smithi032.stdout:1/580: creat d0/d9/fbe x:0 0 0 2024-04-06T14:17:04.816 INFO:tasks.workunit.client.0.smithi032.stdout:2/496: creat d0/d30/d39/d59/d67/d80/fa8 x:0 0 0 2024-04-06T14:17:04.816 INFO:tasks.workunit.client.0.smithi032.stdout:5/501: symlink d16/d18/d33/d45/d5e/lac 0 2024-04-06T14:17:04.817 INFO:tasks.workunit.client.1.smithi086.stdout:0/291: dwrite d0/f3f [0,4194304] 0 2024-04-06T14:17:04.818 INFO:tasks.workunit.client.0.smithi032.stdout:7/529: dwrite d16/d20/d26/d2d/d30/f36 [0,4194304] 0 2024-04-06T14:17:04.819 INFO:tasks.workunit.client.0.smithi032.stdout:2/497: write d0/d5a/f1f [8588472,104928] 0 2024-04-06T14:17:04.819 INFO:tasks.workunit.client.0.smithi032.stdout:8/454: creat d3/d8/fa5 x:0 0 0 2024-04-06T14:17:04.821 INFO:tasks.workunit.client.0.smithi032.stdout:1/581: creat d0/d9/d94/fbf x:0 0 0 2024-04-06T14:17:04.822 INFO:tasks.workunit.client.0.smithi032.stdout:7/530: write d16/d20/d26/d2d/f75 [103839,82348] 0 2024-04-06T14:17:04.822 INFO:tasks.workunit.client.0.smithi032.stdout:8/455: chown d3/d8/d12/c6b 375 1 2024-04-06T14:17:04.823 INFO:tasks.workunit.client.0.smithi032.stdout:6/500: write d4/d5/d19/d25/d67/fa4 [597174,52417] 0 2024-04-06T14:17:04.825 INFO:tasks.workunit.client.0.smithi032.stdout:3/495: link d3/d16/d63/f8e d3/d65/d7d/d9c/dad/db1/fc7 0 2024-04-06T14:17:04.825 INFO:tasks.workunit.client.1.smithi086.stdout:2/229: sync 2024-04-06T14:17:04.826 INFO:tasks.workunit.client.0.smithi032.stdout:1/582: chown d0/da/d3e/c5e 66 1 2024-04-06T14:17:04.828 INFO:tasks.workunit.client.0.smithi032.stdout:2/498: rename d0/d5a/d3a/c81 to d0/d6e/d8d/ca9 0 2024-04-06T14:17:04.828 INFO:tasks.workunit.client.1.smithi086.stdout:2/230: mknod d0/df/d48/c56 0 2024-04-06T14:17:04.829 INFO:tasks.workunit.client.0.smithi032.stdout:1/583: readlink d0/d41/d5b/l97 0 2024-04-06T14:17:04.830 INFO:tasks.workunit.client.0.smithi032.stdout:8/456: creat d3/d8/d12/d2f/d4a/d60/d82/fa6 x:0 0 0 2024-04-06T14:17:04.831 INFO:tasks.workunit.client.0.smithi032.stdout:1/584: fdatasync d0/d10/d66/f70 0 2024-04-06T14:17:04.831 INFO:tasks.workunit.client.0.smithi032.stdout:6/501: stat d4/d5/d19/d25/l32 0 2024-04-06T14:17:04.831 INFO:tasks.workunit.client.1.smithi086.stdout:2/231: symlink d0/d20/d27/d2b/d35/d36/d4d/l57 0 2024-04-06T14:17:04.834 INFO:tasks.workunit.client.0.smithi032.stdout:3/496: getdents d3/d65/d7d/d9c/dad/dc5 0 2024-04-06T14:17:04.837 INFO:tasks.workunit.client.0.smithi032.stdout:1/585: stat d0/d10/d16/d3f/f5d 0 2024-04-06T14:17:04.838 INFO:tasks.workunit.client.1.smithi086.stdout:5/286: rmdir d0 39 2024-04-06T14:17:04.838 INFO:tasks.workunit.client.0.smithi032.stdout:9/497: truncate d2/d15/d55/d59/f67 1402504 0 2024-04-06T14:17:04.840 INFO:tasks.workunit.client.0.smithi032.stdout:7/531: mknod d16/d20/d26/d2d/d30/d3e/d96/dba/cc0 0 2024-04-06T14:17:04.843 INFO:tasks.workunit.client.0.smithi032.stdout:3/497: stat l2 0 2024-04-06T14:17:04.845 INFO:tasks.workunit.client.1.smithi086.stdout:3/307: rename d5/d2b to d5/d26/d4f 0 2024-04-06T14:17:04.845 INFO:tasks.workunit.client.0.smithi032.stdout:7/532: chown d16/d20/d48/d6d/f89 268 1 2024-04-06T14:17:04.847 INFO:tasks.workunit.client.0.smithi032.stdout:3/498: write d3/f7a [708837,67042] 0 2024-04-06T14:17:04.847 INFO:tasks.workunit.client.0.smithi032.stdout:6/502: dwrite d4/f98 [0,4194304] 0 2024-04-06T14:17:04.849 INFO:tasks.workunit.client.1.smithi086.stdout:5/287: dwrite d0/d16/d18/d21/d23/f55 [0,4194304] 0 2024-04-06T14:17:04.852 INFO:tasks.workunit.client.1.smithi086.stdout:3/308: creat d5/f50 x:0 0 0 2024-04-06T14:17:04.855 INFO:tasks.workunit.client.1.smithi086.stdout:3/309: write d5/d26/f3c [4483326,116246] 0 2024-04-06T14:17:04.856 INFO:tasks.workunit.client.0.smithi032.stdout:1/586: getdents d0/da 0 2024-04-06T14:17:04.856 INFO:tasks.workunit.client.0.smithi032.stdout:3/499: creat d3/d65/d7d/d9c/dad/fc8 x:0 0 0 2024-04-06T14:17:04.857 INFO:tasks.workunit.client.1.smithi086.stdout:2/232: dread d0/f6 [0,4194304] 0 2024-04-06T14:17:04.858 INFO:tasks.workunit.client.0.smithi032.stdout:6/503: creat d4/d5/d19/d25/d9c/fae x:0 0 0 2024-04-06T14:17:04.859 INFO:tasks.workunit.client.1.smithi086.stdout:2/233: chown d0/df/f1a 75 1 2024-04-06T14:17:04.860 INFO:tasks.workunit.client.0.smithi032.stdout:9/498: write d2/d15/d1e/d65/d83/f3b [244318,114214] 0 2024-04-06T14:17:04.861 INFO:tasks.workunit.client.1.smithi086.stdout:5/288: dwrite d0/d16/f47 [0,4194304] 0 2024-04-06T14:17:04.863 INFO:tasks.workunit.client.1.smithi086.stdout:2/234: symlink d0/d20/d27/d2b/d35/d36/l58 0 2024-04-06T14:17:04.864 INFO:tasks.workunit.client.1.smithi086.stdout:2/235: readlink d0/l1 0 2024-04-06T14:17:04.864 INFO:tasks.workunit.client.0.smithi032.stdout:1/587: mknod d0/d9/d1f/d7c/d8f/daa/cc0 0 2024-04-06T14:17:04.864 INFO:tasks.workunit.client.0.smithi032.stdout:8/457: dread d3/d8/d7/d14/f1f [0,4194304] 0 2024-04-06T14:17:04.864 INFO:tasks.workunit.client.1.smithi086.stdout:5/289: creat d0/d17/d2d/f56 x:0 0 0 2024-04-06T14:17:04.865 INFO:tasks.workunit.client.1.smithi086.stdout:2/236: write d0/f2a [3667534,14303] 0 2024-04-06T14:17:04.866 INFO:tasks.workunit.client.1.smithi086.stdout:5/290: write d0/f41 [388564,121387] 0 2024-04-06T14:17:04.868 INFO:tasks.workunit.client.0.smithi032.stdout:2/499: dwrite d0/f13 [4194304,4194304] 0 2024-04-06T14:17:04.868 INFO:tasks.workunit.client.0.smithi032.stdout:8/458: dread - d3/d8/d3a/f56 zero size 2024-04-06T14:17:04.868 INFO:tasks.workunit.client.1.smithi086.stdout:2/237: unlink d0/l1 0 2024-04-06T14:17:04.870 INFO:tasks.workunit.client.0.smithi032.stdout:1/588: rmdir d0/d41/d59 39 2024-04-06T14:17:04.871 INFO:tasks.workunit.client.1.smithi086.stdout:2/238: write d0/f33 [2294894,75793] 0 2024-04-06T14:17:04.872 INFO:tasks.workunit.client.0.smithi032.stdout:7/533: dwrite d16/d20/d26/d2d/f9b [0,4194304] 0 2024-04-06T14:17:04.874 INFO:tasks.workunit.client.1.smithi086.stdout:2/239: chown d0/df/d48 43086780 1 2024-04-06T14:17:04.874 INFO:tasks.workunit.client.0.smithi032.stdout:9/499: creat d2/d15/d55/d5f/d71/d81/d4a/d73/f9f x:0 0 0 2024-04-06T14:17:04.874 INFO:tasks.workunit.client.0.smithi032.stdout:6/504: getdents d4/d5/d8 0 2024-04-06T14:17:04.876 INFO:tasks.workunit.client.0.smithi032.stdout:8/459: creat d3/d8/d7/fa7 x:0 0 0 2024-04-06T14:17:04.876 INFO:tasks.workunit.client.1.smithi086.stdout:5/291: dwrite d0/f3f [0,4194304] 0 2024-04-06T14:17:04.876 INFO:tasks.workunit.client.0.smithi032.stdout:2/500: rmdir d0/d30/d39/d59 39 2024-04-06T14:17:04.879 INFO:tasks.workunit.client.0.smithi032.stdout:7/534: read - d16/d20/d63/d9d/fab zero size 2024-04-06T14:17:04.880 INFO:tasks.workunit.client.1.smithi086.stdout:2/240: mkdir d0/d20/d27/d2b/d35/d36/d4d/d59 0 2024-04-06T14:17:04.882 INFO:tasks.workunit.client.0.smithi032.stdout:6/505: rename d4/d5/d19/c9e to d4/d5/d18/caf 0 2024-04-06T14:17:04.883 INFO:tasks.workunit.client.0.smithi032.stdout:6/506: read - d4/d5/d34/f9f zero size 2024-04-06T14:17:04.886 INFO:tasks.workunit.client.0.smithi032.stdout:9/500: symlink d2/d15/d1e/d65/d6b/d7f/la0 0 2024-04-06T14:17:04.888 INFO:tasks.workunit.client.0.smithi032.stdout:9/501: dread - d2/d15/d1e/d65/d83/d31/f40 zero size 2024-04-06T14:17:04.889 INFO:tasks.workunit.client.1.smithi086.stdout:2/241: dwrite d0/df/f14 [0,4194304] 0 2024-04-06T14:17:04.890 INFO:tasks.workunit.client.0.smithi032.stdout:8/460: rename d3/d8/d12/f6f to d3/d8/d12/d2f/d46/fa8 0 2024-04-06T14:17:04.892 INFO:tasks.workunit.client.0.smithi032.stdout:6/507: creat d4/d5/d18/d23/d82/fb0 x:0 0 0 2024-04-06T14:17:04.894 INFO:tasks.workunit.client.0.smithi032.stdout:1/589: write d0/d10/d16/fa6 [2728680,5870] 0 2024-04-06T14:17:04.895 INFO:tasks.workunit.client.0.smithi032.stdout:7/535: rmdir d16/d20/d7a 39 2024-04-06T14:17:04.897 INFO:tasks.workunit.client.0.smithi032.stdout:7/536: chown d16/d20/d26/f3a 4595565 1 2024-04-06T14:17:04.897 INFO:tasks.workunit.client.0.smithi032.stdout:9/502: chown d2/d15/d55/d5f/d71/d81/c90 12793982 1 2024-04-06T14:17:04.897 INFO:tasks.workunit.client.0.smithi032.stdout:2/501: truncate d0/d30/d39/d59/f5f 1422799 0 2024-04-06T14:17:04.899 INFO:tasks.workunit.client.0.smithi032.stdout:8/461: unlink d3/d8/d7/fe 0 2024-04-06T14:17:04.901 INFO:tasks.workunit.client.0.smithi032.stdout:1/590: mknod d0/d10/d16/d3f/d98/cc1 0 2024-04-06T14:17:04.903 INFO:tasks.workunit.client.0.smithi032.stdout:9/503: creat d2/d15/d55/d5f/d71/d81/d4a/d73/fa1 x:0 0 0 2024-04-06T14:17:04.905 INFO:tasks.workunit.client.0.smithi032.stdout:6/508: dwrite d4/f10 [0,4194304] 0 2024-04-06T14:17:04.906 INFO:tasks.workunit.client.1.smithi086.stdout:5/292: dread d0/d19/f3c [0,4194304] 0 2024-04-06T14:17:04.908 INFO:tasks.workunit.client.0.smithi032.stdout:2/502: fdatasync d0/f13 0 2024-04-06T14:17:04.908 INFO:tasks.workunit.client.0.smithi032.stdout:8/462: rmdir d3/d8/d12/d2f/d4a/d60/d75 39 2024-04-06T14:17:04.909 INFO:tasks.workunit.client.1.smithi086.stdout:5/293: mknod d0/d19/c57 0 2024-04-06T14:17:04.910 INFO:tasks.workunit.client.0.smithi032.stdout:8/463: readlink d3/d8/d7/l4e 0 2024-04-06T14:17:04.910 INFO:tasks.workunit.client.0.smithi032.stdout:2/503: dread - d0/d5a/f9d zero size 2024-04-06T14:17:04.911 INFO:tasks.workunit.client.0.smithi032.stdout:7/537: creat d16/d20/d7a/da6/fc1 x:0 0 0 2024-04-06T14:17:04.911 INFO:tasks.workunit.client.0.smithi032.stdout:9/504: creat d2/d15/d55/d5f/d71/d81/d4a/d8f/fa2 x:0 0 0 2024-04-06T14:17:04.914 INFO:tasks.workunit.client.0.smithi032.stdout:8/464: rmdir d3/d8/d7/d1a 39 2024-04-06T14:17:04.914 INFO:tasks.workunit.client.1.smithi086.stdout:5/294: mknod d0/d17/d2d/c58 0 2024-04-06T14:17:04.914 INFO:tasks.workunit.client.0.smithi032.stdout:9/505: chown d2/d15/d55/d5f/d71/d81/d4a/f8d 222112688 1 2024-04-06T14:17:04.917 INFO:tasks.workunit.client.0.smithi032.stdout:9/506: fdatasync d2/d15/d1e/d65/d83/d7e/f64 0 2024-04-06T14:17:04.918 INFO:tasks.workunit.client.1.smithi086.stdout:5/295: symlink d0/d16/d18/d21/d23/d28/l59 0 2024-04-06T14:17:04.918 INFO:tasks.workunit.client.0.smithi032.stdout:2/504: getdents d0/d6e/d8d 0 2024-04-06T14:17:04.920 INFO:tasks.workunit.client.0.smithi032.stdout:7/538: rename d16/d20/c2a to d16/d20/d26/d2d/d30/d3e/cc2 0 2024-04-06T14:17:04.926 INFO:tasks.workunit.client.1.smithi086.stdout:5/296: dread d0/d16/d18/d2a/f46 [0,4194304] 0 2024-04-06T14:17:04.926 INFO:tasks.workunit.client.0.smithi032.stdout:7/539: write d16/d20/d63/d9d/fab [497140,8995] 0 2024-04-06T14:17:04.927 INFO:tasks.workunit.client.1.smithi086.stdout:2/242: dread d0/f33 [0,4194304] 0 2024-04-06T14:17:04.929 INFO:tasks.workunit.client.0.smithi032.stdout:8/465: dwrite d3/d8/d12/d2f/d46/f88 [0,4194304] 0 2024-04-06T14:17:04.934 INFO:tasks.workunit.client.0.smithi032.stdout:2/505: dwrite d0/d5a/f41 [0,4194304] 0 2024-04-06T14:17:04.934 INFO:tasks.workunit.client.1.smithi086.stdout:5/297: dwrite d0/d17/d2d/f4f [0,4194304] 0 2024-04-06T14:17:04.935 INFO:tasks.workunit.client.0.smithi032.stdout:7/540: symlink d16/d20/d63/lc3 0 2024-04-06T14:17:04.937 INFO:tasks.workunit.client.0.smithi032.stdout:8/466: write d3/d8/d7/f47 [4935896,71908] 0 2024-04-06T14:17:04.940 INFO:tasks.workunit.client.1.smithi086.stdout:2/243: truncate d0/df/f1b 2034627 0 2024-04-06T14:17:04.942 INFO:tasks.workunit.client.1.smithi086.stdout:5/298: rename d0/d17/c4a to d0/d17/d2d/c5a 0 2024-04-06T14:17:04.943 INFO:tasks.workunit.client.1.smithi086.stdout:2/244: write d0/d20/d27/d2b/d35/d36/f55 [530612,25930] 0 2024-04-06T14:17:04.943 INFO:tasks.workunit.client.0.smithi032.stdout:1/591: dread d0/d10/d16/f7a [0,4194304] 0 2024-04-06T14:17:04.944 INFO:tasks.workunit.client.1.smithi086.stdout:2/245: chown d0/d20/d27/d2b/d35/d36/d4a 1437 1 2024-04-06T14:17:04.944 INFO:tasks.workunit.client.1.smithi086.stdout:2/246: stat d0/l7 0 2024-04-06T14:17:04.945 INFO:tasks.workunit.client.1.smithi086.stdout:5/299: creat d0/d16/d18/d35/f5b x:0 0 0 2024-04-06T14:17:04.945 INFO:tasks.workunit.client.0.smithi032.stdout:2/506: creat d0/d30/d39/faa x:0 0 0 2024-04-06T14:17:04.946 INFO:tasks.workunit.client.0.smithi032.stdout:1/592: stat d0/d10/d16 0 2024-04-06T14:17:04.947 INFO:tasks.workunit.client.0.smithi032.stdout:9/507: dread d2/d15/d1e/d65/d83/d7e/f14 [0,4194304] 0 2024-04-06T14:17:04.948 INFO:tasks.workunit.client.1.smithi086.stdout:2/247: mknod d0/d20/d27/d2b/d35/d36/d4a/c5a 0 2024-04-06T14:17:04.951 INFO:tasks.workunit.client.0.smithi032.stdout:8/467: dwrite d3/d8/f7a [0,4194304] 0 2024-04-06T14:17:04.956 INFO:tasks.workunit.client.0.smithi032.stdout:4/519: sync 2024-04-06T14:17:04.958 INFO:tasks.workunit.client.0.smithi032.stdout:9/508: link d2/d15/d1e/d65/d83/d7e/l1c d2/d15/d1e/d41/la3 0 2024-04-06T14:17:04.961 INFO:tasks.workunit.client.0.smithi032.stdout:2/507: dwrite d0/d30/d39/d59/f82 [0,4194304] 0 2024-04-06T14:17:04.963 INFO:tasks.workunit.client.0.smithi032.stdout:1/593: write d0/da/f63 [327202,21945] 0 2024-04-06T14:17:04.965 INFO:tasks.workunit.client.0.smithi032.stdout:9/509: symlink d2/d15/d1e/d41/la4 0 2024-04-06T14:17:04.968 INFO:tasks.workunit.client.0.smithi032.stdout:2/508: mknod d0/d30/d39/d59/d67/d74/d78/d65/cab 0 2024-04-06T14:17:04.968 INFO:tasks.workunit.client.0.smithi032.stdout:8/468: dwrite d3/d8/d7/d1a/f71 [0,4194304] 0 2024-04-06T14:17:04.969 INFO:tasks.workunit.client.0.smithi032.stdout:1/594: mkdir d0/d9/d94/dc2 0 2024-04-06T14:17:04.974 INFO:tasks.workunit.client.1.smithi086.stdout:5/300: dread d0/d16/d18/f1a [0,4194304] 0 2024-04-06T14:17:04.975 INFO:tasks.workunit.client.1.smithi086.stdout:5/301: dread - d0/d16/d18/d35/f5b zero size 2024-04-06T14:17:04.975 INFO:tasks.workunit.client.0.smithi032.stdout:1/595: mkdir d0/dc3 0 2024-04-06T14:17:04.978 INFO:tasks.workunit.client.1.smithi086.stdout:5/302: creat d0/d43/f5c x:0 0 0 2024-04-06T14:17:04.978 INFO:tasks.workunit.client.0.smithi032.stdout:8/469: symlink d3/d8/d12/d83/d89/da0/la9 0 2024-04-06T14:17:04.979 INFO:tasks.workunit.client.0.smithi032.stdout:1/596: mkdir d0/da/d3e/d75/dc4 0 2024-04-06T14:17:04.980 INFO:tasks.workunit.client.0.smithi032.stdout:4/520: link d0/df/d23/d31/d50/d57/d8a/cab d0/df/d23/d31/d50/d57/cb1 0 2024-04-06T14:17:04.981 INFO:tasks.workunit.client.0.smithi032.stdout:2/509: read d0/fb [4420258,54052] 0 2024-04-06T14:17:04.981 INFO:tasks.workunit.client.0.smithi032.stdout:9/510: dread d2/f98 [0,4194304] 0 2024-04-06T14:17:04.981 INFO:tasks.workunit.client.1.smithi086.stdout:5/303: mknod d0/d16/d18/d2a/c5d 0 2024-04-06T14:17:04.981 INFO:tasks.workunit.client.1.smithi086.stdout:2/248: dread d0/f2d [0,4194304] 0 2024-04-06T14:17:04.983 INFO:tasks.workunit.client.0.smithi032.stdout:8/470: rename d3/d8/d7/d1a/f71 to d3/d8/d12/faa 0 2024-04-06T14:17:04.983 INFO:tasks.workunit.client.0.smithi032.stdout:4/521: truncate d0/df/f92 1265779 0 2024-04-06T14:17:04.984 INFO:tasks.workunit.client.1.smithi086.stdout:2/249: dread d0/fe [0,4194304] 0 2024-04-06T14:17:04.984 INFO:tasks.workunit.client.1.smithi086.stdout:5/304: readlink d0/d16/d18/d21/d23/l37 0 2024-04-06T14:17:04.984 INFO:tasks.workunit.client.1.smithi086.stdout:6/251: sync 2024-04-06T14:17:04.984 INFO:tasks.workunit.client.1.smithi086.stdout:1/279: sync 2024-04-06T14:17:04.985 INFO:tasks.workunit.client.1.smithi086.stdout:2/250: write d0/d20/d44/f46 [2458100,37463] 0 2024-04-06T14:17:04.986 INFO:tasks.workunit.client.0.smithi032.stdout:1/597: truncate d0/d41/d59/f9a 1540168 0 2024-04-06T14:17:04.987 INFO:tasks.workunit.client.1.smithi086.stdout:1/280: chown db/d13/f3d 42993453 1 2024-04-06T14:17:04.987 INFO:tasks.workunit.client.1.smithi086.stdout:6/252: read - d6/d9/d43/f56 zero size 2024-04-06T14:17:04.987 INFO:tasks.workunit.client.0.smithi032.stdout:9/511: mknod d2/d15/d55/d5f/d71/d81/d4a/d8e/ca5 0 2024-04-06T14:17:04.988 INFO:tasks.workunit.client.1.smithi086.stdout:6/253: stat d6/d13/f27 0 2024-04-06T14:17:04.988 INFO:tasks.workunit.client.0.smithi032.stdout:8/471: creat d3/d8/d7/d8a/fab x:0 0 0 2024-04-06T14:17:04.989 INFO:tasks.workunit.client.0.smithi032.stdout:4/522: mknod d0/cb2 0 2024-04-06T14:17:04.990 INFO:tasks.workunit.client.0.smithi032.stdout:2/510: dwrite d0/d5a/f1f [4194304,4194304] 0 2024-04-06T14:17:04.990 INFO:tasks.workunit.client.1.smithi086.stdout:6/254: chown d6/d13/d42 1 1 2024-04-06T14:17:04.991 INFO:tasks.workunit.client.0.smithi032.stdout:1/598: unlink d0/d9/d1f/f3a 0 2024-04-06T14:17:04.994 INFO:tasks.workunit.client.0.smithi032.stdout:4/523: mkdir d0/d82/d4b/db3 0 2024-04-06T14:17:04.997 INFO:tasks.workunit.client.0.smithi032.stdout:9/512: mkdir d2/d15/d1e/d65/d83/d31/d39/da6 0 2024-04-06T14:17:05.000 INFO:tasks.workunit.client.1.smithi086.stdout:0/292: sync 2024-04-06T14:17:05.000 INFO:tasks.workunit.client.1.smithi086.stdout:4/215: sync 2024-04-06T14:17:05.001 INFO:tasks.workunit.client.1.smithi086.stdout:7/244: sync 2024-04-06T14:17:05.001 INFO:tasks.workunit.client.1.smithi086.stdout:5/305: dwrite d0/d16/d18/d2a/f46 [0,4194304] 0 2024-04-06T14:17:05.001 INFO:tasks.workunit.client.1.smithi086.stdout:4/216: readlink d0/l9 0 2024-04-06T14:17:05.001 INFO:tasks.workunit.client.1.smithi086.stdout:8/400: sync 2024-04-06T14:17:05.001 INFO:tasks.workunit.client.1.smithi086.stdout:9/240: sync 2024-04-06T14:17:05.001 INFO:tasks.workunit.client.1.smithi086.stdout:3/310: sync 2024-04-06T14:17:05.002 INFO:tasks.workunit.client.0.smithi032.stdout:1/599: mkdir d0/d9/d1f/d7c/d8f/dc5 0 2024-04-06T14:17:05.004 INFO:tasks.workunit.client.1.smithi086.stdout:2/251: dwrite d0/d20/f25 [0,4194304] 0 2024-04-06T14:17:05.005 INFO:tasks.workunit.client.0.smithi032.stdout:1/600: chown d0/d9/d1f/d7c/d8f/dc5 101280132 1 2024-04-06T14:17:05.006 INFO:tasks.workunit.client.0.smithi032.stdout:9/513: mkdir d2/d15/d1e/d65/d83/d7e/da7 0 2024-04-06T14:17:05.007 INFO:tasks.workunit.client.1.smithi086.stdout:1/281: write db/ff [107898,16634] 0 2024-04-06T14:17:05.007 INFO:tasks.workunit.client.1.smithi086.stdout:8/401: chown d0/d71/d16/d52 578129 1 2024-04-06T14:17:05.007 INFO:tasks.workunit.client.0.smithi032.stdout:9/514: chown d2/d15/d1e/d65/d83/d7e/f12 131417 1 2024-04-06T14:17:05.008 INFO:tasks.workunit.client.0.smithi032.stdout:8/472: rename d3/d8/d12/d2f/d4a/d60/d82/fa3 to d3/d8/fac 0 2024-04-06T14:17:05.009 INFO:tasks.workunit.client.0.smithi032.stdout:9/515: fdatasync d2/d15/d55/f86 0 2024-04-06T14:17:05.009 INFO:tasks.workunit.client.0.smithi032.stdout:1/601: fsync d0/f2c 0 2024-04-06T14:17:05.011 INFO:tasks.workunit.client.0.smithi032.stdout:2/511: getdents d0/d30/d39 0 2024-04-06T14:17:05.013 INFO:tasks.workunit.client.0.smithi032.stdout:8/473: chown d3/d8/d12/d2f/d4a/d60/l68 0 1 2024-04-06T14:17:05.013 INFO:tasks.workunit.client.0.smithi032.stdout:4/524: dread d0/df/d23/d31/d50/d57/f6a [0,4194304] 0 2024-04-06T14:17:05.014 INFO:tasks.workunit.client.1.smithi086.stdout:0/293: creat d0/d1/d4/de/f62 x:0 0 0 2024-04-06T14:17:05.014 INFO:tasks.workunit.client.1.smithi086.stdout:9/241: mknod d5/d10/d14/d19/d25/c54 0 2024-04-06T14:17:05.014 INFO:tasks.workunit.client.1.smithi086.stdout:3/311: unlink d5/f48 0 2024-04-06T14:17:05.015 INFO:tasks.workunit.client.0.smithi032.stdout:9/516: chown d2/d15/d1e/d65/d83/f32 488 1 2024-04-06T14:17:05.017 INFO:tasks.workunit.client.1.smithi086.stdout:4/217: symlink d0/l42 0 2024-04-06T14:17:05.017 INFO:tasks.workunit.client.1.smithi086.stdout:2/252: write d0/df/f14 [1964725,7958] 0 2024-04-06T14:17:05.018 INFO:tasks.workunit.client.1.smithi086.stdout:6/255: dwrite d6/d13/d1a/d22/f35 [0,4194304] 0 2024-04-06T14:17:05.018 INFO:tasks.workunit.client.0.smithi032.stdout:8/474: unlink d3/d8/d12/d83/d89/da0/la9 0 2024-04-06T14:17:05.020 INFO:tasks.workunit.client.0.smithi032.stdout:4/525: stat d0/df/d2b/d46/d58/fb0 0 2024-04-06T14:17:05.021 INFO:tasks.workunit.client.0.smithi032.stdout:9/517: stat d2/d15/l7b 0 2024-04-06T14:17:05.023 INFO:tasks.workunit.client.0.smithi032.stdout:1/602: getdents d0/d41/d5b/d96 0 2024-04-06T14:17:05.024 INFO:tasks.workunit.client.0.smithi032.stdout:4/526: rmdir d0/df/d23 39 2024-04-06T14:17:05.024 INFO:tasks.workunit.client.0.smithi032.stdout:2/512: dwrite d0/d5a/f1f [0,4194304] 0 2024-04-06T14:17:05.025 INFO:tasks.workunit.client.1.smithi086.stdout:0/294: rmdir d0/d1/d1b 39 2024-04-06T14:17:05.026 INFO:tasks.workunit.client.0.smithi032.stdout:1/603: readlink d0/da/d3e/d75/l88 0 2024-04-06T14:17:05.031 INFO:tasks.workunit.client.1.smithi086.stdout:7/245: dwrite d7/d10/d11/d1c/f1e [0,4194304] 0 2024-04-06T14:17:05.031 INFO:tasks.workunit.client.0.smithi032.stdout:8/475: getdents d3/d8/d7 0 2024-04-06T14:17:05.031 INFO:tasks.workunit.client.0.smithi032.stdout:2/513: truncate d0/fb 1643330 0 2024-04-06T14:17:05.033 INFO:tasks.workunit.client.0.smithi032.stdout:8/476: readlink d3/d8/d7/l6e 0 2024-04-06T14:17:05.035 INFO:tasks.workunit.client.0.smithi032.stdout:4/527: rename d0/df/d1f/d2e/l38 to d0/df/d23/d31/d50/d5e/da3/lb4 0 2024-04-06T14:17:05.035 INFO:tasks.workunit.client.1.smithi086.stdout:4/218: rmdir d0/d15/d19 39 2024-04-06T14:17:05.035 INFO:tasks.workunit.client.0.smithi032.stdout:1/604: getdents d0/d41/d5b/d96 0 2024-04-06T14:17:05.037 INFO:tasks.workunit.client.1.smithi086.stdout:6/256: write d6/d9/db/f28 [320059,24747] 0 2024-04-06T14:17:05.037 INFO:tasks.workunit.client.1.smithi086.stdout:1/282: dwrite db/d13/f54 [0,4194304] 0 2024-04-06T14:17:05.037 INFO:tasks.workunit.client.0.smithi032.stdout:9/518: dwrite d2/d15/d55/d5f/d71/d81/d4a/d73/f9f [0,4194304] 0 2024-04-06T14:17:05.038 INFO:tasks.workunit.client.0.smithi032.stdout:4/528: write d0/df/d1f/d2e/fa5 [1043658,21470] 0 2024-04-06T14:17:05.038 INFO:tasks.workunit.client.0.smithi032.stdout:2/514: creat d0/d30/d7f/fac x:0 0 0 2024-04-06T14:17:05.040 INFO:tasks.workunit.client.1.smithi086.stdout:0/295: write d0/d1/d1b/d44/f5a [5239911,83208] 0 2024-04-06T14:17:05.041 INFO:tasks.workunit.client.0.smithi032.stdout:1/605: truncate d0/d9/d1f/d72/f7e 4672441 0 2024-04-06T14:17:05.042 INFO:tasks.workunit.client.0.smithi032.stdout:1/606: chown d0/d10/d16/l43 334108 1 2024-04-06T14:17:05.043 INFO:tasks.workunit.client.1.smithi086.stdout:8/402: rename d0/d71/d22/d43/c3f to d0/d71/d16/d39/c8d 0 2024-04-06T14:17:05.045 INFO:tasks.workunit.client.0.smithi032.stdout:9/519: readlink d2/d15/l4f 0 2024-04-06T14:17:05.046 INFO:tasks.workunit.client.0.smithi032.stdout:9/520: stat d2/f21 0 2024-04-06T14:17:05.046 INFO:tasks.workunit.client.1.smithi086.stdout:9/242: dwrite d5/d18/f48 [4194304,4194304] 0 2024-04-06T14:17:05.046 INFO:tasks.workunit.client.1.smithi086.stdout:9/243: chown d5/cb 126229 1 2024-04-06T14:17:05.047 INFO:tasks.workunit.client.0.smithi032.stdout:4/529: readlink d0/df/d1f/d2e/l97 0 2024-04-06T14:17:05.049 INFO:tasks.workunit.client.0.smithi032.stdout:1/607: write d0/d10/d16/f51 [2945540,60525] 0 2024-04-06T14:17:05.050 INFO:tasks.workunit.client.0.smithi032.stdout:9/521: symlink d2/d15/d1e/d41/d6c/d74/la8 0 2024-04-06T14:17:05.051 INFO:tasks.workunit.client.1.smithi086.stdout:2/253: link d0/df/f13 d0/d20/d44/f5b 0 2024-04-06T14:17:05.051 INFO:tasks.workunit.client.1.smithi086.stdout:1/283: creat db/d11/d1f/f5a x:0 0 0 2024-04-06T14:17:05.051 INFO:tasks.workunit.client.1.smithi086.stdout:7/246: dread d7/d10/d30/f4a [0,4194304] 0 2024-04-06T14:17:05.051 INFO:tasks.workunit.client.1.smithi086.stdout:4/219: truncate d0/d2c/d30/d24/f2d 364226 0 2024-04-06T14:17:05.051 INFO:tasks.workunit.client.1.smithi086.stdout:4/220: dread - d0/d15/d20/f3a zero size 2024-04-06T14:17:05.052 INFO:tasks.workunit.client.0.smithi032.stdout:1/608: chown d0/da/d3e/l84 2787360 1 2024-04-06T14:17:05.054 INFO:tasks.workunit.client.0.smithi032.stdout:9/522: dread - d2/d15/d55/d5f/d71/d81/d4a/d73/f87 zero size 2024-04-06T14:17:05.055 INFO:tasks.workunit.client.1.smithi086.stdout:3/312: getdents d5/df 0 2024-04-06T14:17:05.056 INFO:tasks.workunit.client.1.smithi086.stdout:8/403: creat d0/d71/d22/f8e x:0 0 0 2024-04-06T14:17:05.058 INFO:tasks.workunit.client.0.smithi032.stdout:4/530: mkdir d0/df/d23/d31/d50/d57/d8a/db5 0 2024-04-06T14:17:05.058 INFO:tasks.workunit.client.0.smithi032.stdout:1/609: dread d0/da/d3e/fad [0,4194304] 0 2024-04-06T14:17:05.059 INFO:tasks.workunit.client.1.smithi086.stdout:2/254: fdatasync d0/f2d 0 2024-04-06T14:17:05.060 INFO:tasks.workunit.client.0.smithi032.stdout:4/531: stat d0/df/d23/d31/d50/d57/d5b/d9c/l6d 0 2024-04-06T14:17:05.061 INFO:tasks.workunit.client.1.smithi086.stdout:5/306: dread d0/f4 [0,4194304] 0 2024-04-06T14:17:05.064 INFO:tasks.workunit.client.1.smithi086.stdout:1/284: mkdir db/d11/d1a/d2a/d52/d5b 0 2024-04-06T14:17:05.067 INFO:tasks.workunit.client.1.smithi086.stdout:6/257: rename d6/d57 to d6/d9/d5c 0 2024-04-06T14:17:05.068 INFO:tasks.workunit.client.1.smithi086.stdout:5/307: dread - d0/d16/f4e zero size 2024-04-06T14:17:05.068 INFO:tasks.workunit.client.0.smithi032.stdout:4/532: rename d0/df/d23/d31/d50/d57/d5b/d9c/l6d to d0/d82/d8e/lb6 0 2024-04-06T14:17:05.069 INFO:tasks.workunit.client.1.smithi086.stdout:3/313: mknod d5/d26/d4f/d4e/c51 0 2024-04-06T14:17:05.070 INFO:tasks.workunit.client.0.smithi032.stdout:9/523: dread d2/d15/d55/d5f/d71/d81/f4c [0,4194304] 0 2024-04-06T14:17:05.075 INFO:tasks.workunit.client.1.smithi086.stdout:7/247: dwrite d7/d10/d30/f50 [0,4194304] 0 2024-04-06T14:17:05.078 INFO:tasks.workunit.client.0.smithi032.stdout:9/524: write d2/d15/d1e/d65/d83/d7e/f6 [5255530,116542] 0 2024-04-06T14:17:05.080 INFO:tasks.workunit.client.1.smithi086.stdout:0/296: truncate d0/d1/d1b/d44/f5a 3189748 0 2024-04-06T14:17:05.082 INFO:tasks.workunit.client.0.smithi032.stdout:4/533: dread d0/df/d2b/d46/d58/fb0 [0,4194304] 0 2024-04-06T14:17:05.083 INFO:tasks.workunit.client.0.smithi032.stdout:2/515: dwrite d0/d30/d39/d59/d67/d74/d78/d57/f96 [0,4194304] 0 2024-04-06T14:17:05.083 INFO:tasks.workunit.client.0.smithi032.stdout:9/525: rmdir d2/d15/d1e/d65/d83/d31 39 2024-04-06T14:17:05.086 INFO:tasks.workunit.client.0.smithi032.stdout:9/526: write d2/d15/d1e/d65/d6b/f9b [121574,103769] 0 2024-04-06T14:17:05.088 INFO:tasks.workunit.client.0.smithi032.stdout:4/534: getdents d0/df/d23/d31/d50/d57 0 2024-04-06T14:17:05.090 INFO:tasks.workunit.client.0.smithi032.stdout:9/527: creat d2/d15/d55/d5f/d71/d81/d4a/d73/fa9 x:0 0 0 2024-04-06T14:17:05.095 INFO:tasks.workunit.client.0.smithi032.stdout:4/535: rename d0/df/d23/d31/d50/d57/d5b/c67 to d0/df/d23/d31/d50/d57/d5b/cb7 0 2024-04-06T14:17:05.097 INFO:tasks.workunit.client.0.smithi032.stdout:9/528: dread d2/d15/d55/d5f/d71/d81/d4a/d73/d7c/f80 [0,4194304] 0 2024-04-06T14:17:05.100 INFO:tasks.workunit.client.0.smithi032.stdout:9/529: symlink d2/d15/d55/d5f/d71/d81/d4a/d8f/laa 0 2024-04-06T14:17:05.101 INFO:tasks.workunit.client.0.smithi032.stdout:4/536: dwrite d0/df/d23/d31/d50/d57/f6a [0,4194304] 0 2024-04-06T14:17:05.102 INFO:tasks.workunit.client.0.smithi032.stdout:9/530: write d2/d15/d55/f9a [981750,40391] 0 2024-04-06T14:17:05.106 INFO:tasks.workunit.client.0.smithi032.stdout:9/531: dread - d2/d15/d1e/d65/d83/d31/d39/f52 zero size 2024-04-06T14:17:05.110 INFO:tasks.workunit.client.0.smithi032.stdout:9/532: creat d2/d15/d55/d5f/d71/d81/fab x:0 0 0 2024-04-06T14:17:05.110 INFO:tasks.workunit.client.0.smithi032.stdout:4/537: dwrite d0/df/d1f/d42/d45/f8c [0,4194304] 0 2024-04-06T14:17:05.117 INFO:tasks.workunit.client.0.smithi032.stdout:4/538: rename d0/df/d23/d31/f8d to d0/df/d23/fb8 0 2024-04-06T14:17:05.117 INFO:tasks.workunit.client.0.smithi032.stdout:9/533: dwrite d2/d15/d1e/d41/f63 [4194304,4194304] 0 2024-04-06T14:17:05.119 INFO:tasks.workunit.client.0.smithi032.stdout:9/534: chown d2/d15/d55/d5f/d71/d81/d77/l99 304066703 1 2024-04-06T14:17:05.122 INFO:tasks.workunit.client.0.smithi032.stdout:9/535: dread - d2/d15/d1e/d65/d83/d7e/f64 zero size 2024-04-06T14:17:05.125 INFO:tasks.workunit.client.1.smithi086.stdout:6/258: write d6/d9/d43/f53 [428187,71142] 0 2024-04-06T14:17:05.126 INFO:tasks.workunit.client.0.smithi032.stdout:4/539: dwrite d0/df/d1f/d42/d45/f53 [0,4194304] 0 2024-04-06T14:17:05.128 INFO:tasks.workunit.client.1.smithi086.stdout:9/244: truncate d5/d10/d14/d19/f4b 542798 0 2024-04-06T14:17:05.134 INFO:tasks.workunit.client.0.smithi032.stdout:9/536: rename d2/d15/d55/d5f/d71/d81/d4a/c84 to d2/d15/d55/cac 0 2024-04-06T14:17:05.136 INFO:tasks.workunit.client.1.smithi086.stdout:8/404: dwrite d0/d3b/f51 [0,4194304] 0 2024-04-06T14:17:05.137 INFO:tasks.workunit.client.0.smithi032.stdout:4/540: dwrite d0/df/d76/f61 [0,4194304] 0 2024-04-06T14:17:05.137 INFO:tasks.workunit.client.1.smithi086.stdout:7/248: creat d7/d10/d11/d1c/f58 x:0 0 0 2024-04-06T14:17:05.138 INFO:tasks.workunit.client.1.smithi086.stdout:4/221: dwrite d0/f2 [4194304,4194304] 0 2024-04-06T14:17:05.140 INFO:tasks.workunit.client.1.smithi086.stdout:5/308: read d0/d16/d18/d21/f45 [54143,107023] 0 2024-04-06T14:17:05.142 INFO:tasks.workunit.client.1.smithi086.stdout:7/249: chown d7/d1d/l14 242 1 2024-04-06T14:17:05.144 INFO:tasks.workunit.client.0.smithi032.stdout:9/537: mknod d2/d15/d1e/d65/d9c/cad 0 2024-04-06T14:17:05.144 INFO:tasks.workunit.client.1.smithi086.stdout:9/245: stat d5/d13/d1a/l38 0 2024-04-06T14:17:05.144 INFO:tasks.workunit.client.1.smithi086.stdout:6/259: creat d6/d13/f5d x:0 0 0 2024-04-06T14:17:05.145 INFO:tasks.workunit.client.1.smithi086.stdout:0/297: write d0/d15/f4c [110609,105676] 0 2024-04-06T14:17:05.145 INFO:tasks.workunit.client.0.smithi032.stdout:4/541: creat d0/df/fb9 x:0 0 0 2024-04-06T14:17:05.148 INFO:tasks.workunit.client.1.smithi086.stdout:1/285: dwrite db/d11/d1f/f5a [0,4194304] 0 2024-04-06T14:17:05.150 INFO:tasks.workunit.client.0.smithi032.stdout:9/538: link d2/d15/d55/d5f/d71/d81/d4a/d8f/fa2 d2/d15/d55/d5f/d71/d81/d4a/d8e/fae 0 2024-04-06T14:17:05.151 INFO:tasks.workunit.client.1.smithi086.stdout:5/309: mkdir d0/d5e 0 2024-04-06T14:17:05.153 INFO:tasks.workunit.client.1.smithi086.stdout:7/250: write d7/d10/d30/f50 [3748712,56653] 0 2024-04-06T14:17:05.153 INFO:tasks.workunit.client.1.smithi086.stdout:1/286: mkdir db/d11/d1f/d5c 0 2024-04-06T14:17:05.153 INFO:tasks.workunit.client.0.smithi032.stdout:9/539: dread - d2/d15/d55/d5f/d71/d81/d4a/d73/f87 zero size 2024-04-06T14:17:05.154 INFO:tasks.workunit.client.1.smithi086.stdout:9/246: mkdir d5/d13/d47/d55 0 2024-04-06T14:17:05.155 INFO:tasks.workunit.client.0.smithi032.stdout:4/542: dwrite d0/df/d1f/f95 [0,4194304] 0 2024-04-06T14:17:05.156 INFO:tasks.workunit.client.1.smithi086.stdout:4/222: dread d0/d15/d20/f22 [0,4194304] 0 2024-04-06T14:17:05.156 INFO:tasks.workunit.client.1.smithi086.stdout:6/260: mknod d6/c5e 0 2024-04-06T14:17:05.158 INFO:tasks.workunit.client.1.smithi086.stdout:0/298: creat d0/d1/d2/d1a/f63 x:0 0 0 2024-04-06T14:17:05.158 INFO:tasks.workunit.client.0.smithi032.stdout:4/543: write d0/df/d23/fac [4718521,122344] 0 2024-04-06T14:17:05.158 INFO:tasks.workunit.client.0.smithi032.stdout:9/540: getdents d2/d15/d1e/d65/d6b/d92 0 2024-04-06T14:17:05.160 INFO:tasks.workunit.client.1.smithi086.stdout:4/223: write d0/f17 [3926678,38594] 0 2024-04-06T14:17:05.160 INFO:tasks.workunit.client.0.smithi032.stdout:4/544: read d0/df/d2b/d46/d58/fb0 [333335,52856] 0 2024-04-06T14:17:05.160 INFO:tasks.workunit.client.1.smithi086.stdout:6/261: write d6/f3c [600001,96263] 0 2024-04-06T14:17:05.161 INFO:tasks.workunit.client.1.smithi086.stdout:8/405: rename d0/d71/d22/d43/d4d/f53 to d0/d71/d22/d65/d6e/f8f 0 2024-04-06T14:17:05.165 INFO:tasks.workunit.client.0.smithi032.stdout:9/541: creat d2/d15/d1e/d65/d83/d7e/faf x:0 0 0 2024-04-06T14:17:05.166 INFO:tasks.workunit.client.1.smithi086.stdout:4/224: symlink d0/d15/d20/l43 0 2024-04-06T14:17:05.167 INFO:tasks.workunit.client.1.smithi086.stdout:9/247: readlink d5/d13/d1a/l3a 0 2024-04-06T14:17:05.167 INFO:tasks.workunit.client.0.smithi032.stdout:4/545: creat d0/fba x:0 0 0 2024-04-06T14:17:05.168 INFO:tasks.workunit.client.1.smithi086.stdout:1/287: symlink db/d11/d1a/d2a/l5d 0 2024-04-06T14:17:05.170 INFO:tasks.workunit.client.1.smithi086.stdout:8/406: truncate d0/d3b/f4c 1035173 0 2024-04-06T14:17:05.171 INFO:tasks.workunit.client.0.smithi032.stdout:9/542: creat d2/d15/d55/d5f/d71/d81/fb0 x:0 0 0 2024-04-06T14:17:05.171 INFO:tasks.workunit.client.0.smithi032.stdout:4/546: mknod d0/d82/d4b/db3/cbb 0 2024-04-06T14:17:05.172 INFO:tasks.workunit.client.1.smithi086.stdout:4/225: creat d0/d15/d20/f44 x:0 0 0 2024-04-06T14:17:05.172 INFO:tasks.workunit.client.1.smithi086.stdout:0/299: chown d0/d15/d4e/d48/c3d 87857 1 2024-04-06T14:17:05.173 INFO:tasks.workunit.client.1.smithi086.stdout:7/251: dwrite d7/d10/d30/f35 [0,4194304] 0 2024-04-06T14:17:05.173 INFO:tasks.workunit.client.1.smithi086.stdout:9/248: mknod d5/d10/d14/d19/d1b/c56 0 2024-04-06T14:17:05.174 INFO:tasks.workunit.client.1.smithi086.stdout:5/310: dwrite d0/d16/d18/d21/d23/f55 [0,4194304] 0 2024-04-06T14:17:05.176 INFO:tasks.workunit.client.1.smithi086.stdout:8/407: write d0/d71/d22/d43/f28 [3328676,49164] 0 2024-04-06T14:17:05.177 INFO:tasks.workunit.client.1.smithi086.stdout:5/311: chown d0/d16/d18/d2a 56 1 2024-04-06T14:17:05.178 INFO:tasks.workunit.client.0.smithi032.stdout:4/547: creat d0/df/fbc x:0 0 0 2024-04-06T14:17:05.178 INFO:tasks.workunit.client.1.smithi086.stdout:5/312: write d0/f3f [10654,17780] 0 2024-04-06T14:17:05.181 INFO:tasks.workunit.client.1.smithi086.stdout:9/249: symlink d5/d13/d47/l57 0 2024-04-06T14:17:05.188 INFO:tasks.workunit.client.0.smithi032.stdout:4/548: write d0/f44 [1218331,60350] 0 2024-04-06T14:17:05.188 INFO:tasks.workunit.client.1.smithi086.stdout:2/255: fdatasync d0/df/f14 0 2024-04-06T14:17:05.189 INFO:tasks.workunit.client.1.smithi086.stdout:5/313: dwrite d0/f6 [0,4194304] 0 2024-04-06T14:17:05.192 INFO:tasks.workunit.client.1.smithi086.stdout:2/256: write d0/d20/d27/d2b/d35/d36/f40 [2601256,32259] 0 2024-04-06T14:17:05.195 INFO:tasks.workunit.client.0.smithi032.stdout:4/549: creat d0/d82/d4b/fbd x:0 0 0 2024-04-06T14:17:05.206 INFO:tasks.workunit.client.1.smithi086.stdout:9/250: rmdir d5/d10/d14/d19/d4d 0 2024-04-06T14:17:05.207 INFO:tasks.workunit.client.1.smithi086.stdout:9/251: chown d5/d10/l4a 2 1 2024-04-06T14:17:05.209 INFO:tasks.workunit.client.1.smithi086.stdout:9/252: chown d5/d18/f32 1229054 1 2024-04-06T14:17:05.210 INFO:tasks.workunit.client.0.smithi032.stdout:4/550: dread d0/df/d1f/d42/d45/f8c [0,4194304] 0 2024-04-06T14:17:05.212 INFO:tasks.workunit.client.1.smithi086.stdout:0/300: dread d0/f16 [0,4194304] 0 2024-04-06T14:17:05.213 INFO:tasks.workunit.client.0.smithi032.stdout:4/551: stat d0/df/d23/d31 0 2024-04-06T14:17:05.213 INFO:tasks.workunit.client.1.smithi086.stdout:7/252: symlink d7/d10/d11/d1c/d36/l59 0 2024-04-06T14:17:05.214 INFO:tasks.workunit.client.1.smithi086.stdout:2/257: dwrite d0/f6 [0,4194304] 0 2024-04-06T14:17:05.214 INFO:tasks.workunit.client.0.smithi032.stdout:2/516: dread d0/d4e/f53 [0,4194304] 0 2024-04-06T14:17:05.215 INFO:tasks.workunit.client.1.smithi086.stdout:2/258: readlink d0/l7 0 2024-04-06T14:17:05.216 INFO:tasks.workunit.client.1.smithi086.stdout:4/226: fsync d0/f2 0 2024-04-06T14:17:05.216 INFO:tasks.workunit.client.0.smithi032.stdout:4/552: mkdir d0/df/d2b/dbe 0 2024-04-06T14:17:05.217 INFO:tasks.workunit.client.1.smithi086.stdout:0/301: truncate d0/d1/d4/de/f62 152024 0 2024-04-06T14:17:05.218 INFO:tasks.workunit.client.1.smithi086.stdout:4/227: stat d0/f17 0 2024-04-06T14:17:05.220 INFO:tasks.workunit.client.1.smithi086.stdout:2/259: dread - d0/d20/d27/d2b/d35/d36/d4d/f4f zero size 2024-04-06T14:17:05.220 INFO:tasks.workunit.client.1.smithi086.stdout:6/262: mkdir d6/d13/d5f 0 2024-04-06T14:17:05.222 INFO:tasks.workunit.client.0.smithi032.stdout:4/553: unlink d0/df/d2b/d46/f6e 0 2024-04-06T14:17:05.224 INFO:tasks.workunit.client.1.smithi086.stdout:2/260: read - d0/d20/d27/d2b/d35/d36/d4d/f4f zero size 2024-04-06T14:17:05.225 INFO:tasks.workunit.client.0.smithi032.stdout:2/517: dwrite d0/d30/d39/d59/d67/d74/f9a [0,4194304] 0 2024-04-06T14:17:05.225 INFO:tasks.workunit.client.0.smithi032.stdout:9/543: dread d2/f2d [0,4194304] 0 2024-04-06T14:17:05.228 INFO:tasks.workunit.client.1.smithi086.stdout:4/228: dwrite d0/f40 [0,4194304] 0 2024-04-06T14:17:05.228 INFO:tasks.workunit.client.1.smithi086.stdout:0/302: dread d0/d15/d26/d33/f57 [0,4194304] 0 2024-04-06T14:17:05.228 INFO:tasks.workunit.client.1.smithi086.stdout:4/229: readlink d0/l42 0 2024-04-06T14:17:05.229 INFO:tasks.workunit.client.1.smithi086.stdout:7/253: mknod d7/d10/d11/c5a 0 2024-04-06T14:17:05.230 INFO:tasks.workunit.client.1.smithi086.stdout:9/253: mkdir d5/d58 0 2024-04-06T14:17:05.233 INFO:tasks.workunit.client.1.smithi086.stdout:0/303: write d0/d1/d4/d7/f14 [1593304,60076] 0 2024-04-06T14:17:05.234 INFO:tasks.workunit.client.1.smithi086.stdout:1/288: rmdir db/d11/d1a/d2a 39 2024-04-06T14:17:05.237 INFO:tasks.workunit.client.1.smithi086.stdout:0/304: write d0/d1/d2/d1a/f63 [761851,99146] 0 2024-04-06T14:17:05.238 INFO:tasks.workunit.client.1.smithi086.stdout:9/254: dread d5/d10/d14/d19/f1e [0,4194304] 0 2024-04-06T14:17:05.240 INFO:tasks.workunit.client.1.smithi086.stdout:2/261: readlink d0/df/l32 0 2024-04-06T14:17:05.244 INFO:tasks.workunit.client.1.smithi086.stdout:9/255: chown d5/d10/d14/d19/c42 0 1 2024-04-06T14:17:05.244 INFO:tasks.workunit.client.1.smithi086.stdout:2/262: chown d0/la 1113730 1 2024-04-06T14:17:05.244 INFO:tasks.workunit.client.1.smithi086.stdout:1/289: dread db/d13/f54 [0,4194304] 0 2024-04-06T14:17:05.245 INFO:tasks.workunit.client.1.smithi086.stdout:9/256: fdatasync d5/d13/f2d 0 2024-04-06T14:17:05.247 INFO:tasks.workunit.client.0.smithi032.stdout:4/554: truncate d0/df/d23/d31/d50/d57/f64 1583416 0 2024-04-06T14:17:05.247 INFO:tasks.workunit.client.1.smithi086.stdout:2/263: write d0/d20/f4c [524133,31936] 0 2024-04-06T14:17:05.250 INFO:tasks.workunit.client.1.smithi086.stdout:7/254: rmdir d7/d10/d11/d1c/d24/d2c 39 2024-04-06T14:17:05.250 INFO:tasks.workunit.client.0.smithi032.stdout:4/555: fsync d0/df/d1f/f34 0 2024-04-06T14:17:05.251 INFO:tasks.workunit.client.1.smithi086.stdout:0/305: dread d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:05.252 INFO:tasks.workunit.client.1.smithi086.stdout:2/264: fsync d0/d20/d27/d2b/d35/d36/f55 0 2024-04-06T14:17:05.253 INFO:tasks.workunit.client.0.smithi032.stdout:2/518: dwrite d0/d5a/d3a/f5b [0,4194304] 0 2024-04-06T14:17:05.256 INFO:tasks.workunit.client.1.smithi086.stdout:5/314: truncate d0/d16/d18/d2a/f46 336819 0 2024-04-06T14:17:05.256 INFO:tasks.workunit.client.1.smithi086.stdout:9/257: dwrite d5/d13/f39 [0,4194304] 0 2024-04-06T14:17:05.257 INFO:tasks.workunit.client.0.smithi032.stdout:4/556: creat d0/df/d23/d31/d50/d5e/fbf x:0 0 0 2024-04-06T14:17:05.260 INFO:tasks.workunit.client.0.smithi032.stdout:4/557: fsync d0/df/d2b/f72 0 2024-04-06T14:17:05.262 INFO:tasks.workunit.client.0.smithi032.stdout:9/544: dread d2/d15/d55/d5f/d71/d81/d77/f78 [0,4194304] 0 2024-04-06T14:17:05.266 INFO:tasks.workunit.client.0.smithi032.stdout:4/558: mknod d0/df/d2b/d46/cc0 0 2024-04-06T14:17:05.267 INFO:tasks.workunit.client.0.smithi032.stdout:2/519: link d0/d5a/d3a/f42 d0/d30/d39/d59/d67/d80/fad 0 2024-04-06T14:17:05.267 INFO:tasks.workunit.client.0.smithi032.stdout:2/520: chown d0/l26 14008956 1 2024-04-06T14:17:05.269 INFO:tasks.workunit.client.1.smithi086.stdout:7/255: dwrite d7/d10/d30/f4a [0,4194304] 0 2024-04-06T14:17:05.269 INFO:tasks.workunit.client.1.smithi086.stdout:0/306: mkdir d0/d1/d4/d64 0 2024-04-06T14:17:05.270 INFO:tasks.workunit.client.1.smithi086.stdout:2/265: getdents d0/d20/d27/d37/d52 0 2024-04-06T14:17:05.271 INFO:tasks.workunit.client.0.smithi032.stdout:9/545: rename d2/d15/d1e/d65/d83/d7e/c82 to d2/d15/d1e/d65/d6b/d7f/cb1 0 2024-04-06T14:17:05.272 INFO:tasks.workunit.client.1.smithi086.stdout:2/266: chown d0/d20/d27/d2b/d35/d36/d4d/f4f 7 1 2024-04-06T14:17:05.272 INFO:tasks.workunit.client.0.smithi032.stdout:4/559: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc1 0 2024-04-06T14:17:05.273 INFO:tasks.workunit.client.1.smithi086.stdout:9/258: read d5/d18/f26 [85098,72661] 0 2024-04-06T14:17:05.273 INFO:tasks.workunit.client.1.smithi086.stdout:5/315: creat d0/d16/d18/d2a/d50/f5f x:0 0 0 2024-04-06T14:17:05.275 INFO:tasks.workunit.client.1.smithi086.stdout:2/267: chown d0/df/d48 1261722 1 2024-04-06T14:17:05.275 INFO:tasks.workunit.client.1.smithi086.stdout:1/290: getdents db/d11/d1a 0 2024-04-06T14:17:05.276 INFO:tasks.workunit.client.1.smithi086.stdout:7/256: mknod d7/d10/d11/d21/c5b 0 2024-04-06T14:17:05.277 INFO:tasks.workunit.client.1.smithi086.stdout:0/307: symlink d0/d1/d4/d64/l65 0 2024-04-06T14:17:05.277 INFO:tasks.workunit.client.0.smithi032.stdout:4/560: mkdir d0/d82/daf/dc2 0 2024-04-06T14:17:05.277 INFO:tasks.workunit.client.0.smithi032.stdout:9/546: read d2/d15/d1e/d41/f63 [269579,85789] 0 2024-04-06T14:17:05.278 INFO:tasks.workunit.client.1.smithi086.stdout:9/259: fdatasync d5/d18/f1d 0 2024-04-06T14:17:05.278 INFO:tasks.workunit.client.1.smithi086.stdout:2/268: write d0/df/f18 [1130790,43056] 0 2024-04-06T14:17:05.279 INFO:tasks.workunit.client.1.smithi086.stdout:1/291: mknod db/d11/d1d/c5e 0 2024-04-06T14:17:05.279 INFO:tasks.workunit.client.0.smithi032.stdout:4/561: read d0/df/d2b/d43/fa6 [20007,19412] 0 2024-04-06T14:17:05.282 INFO:tasks.workunit.client.1.smithi086.stdout:7/257: truncate f5 1815166 0 2024-04-06T14:17:05.283 INFO:tasks.workunit.client.1.smithi086.stdout:1/292: rmdir db/d11/d1f 39 2024-04-06T14:17:05.285 INFO:tasks.workunit.client.0.smithi032.stdout:4/562: rename d0/f44 to d0/fc3 0 2024-04-06T14:17:05.285 INFO:tasks.workunit.client.1.smithi086.stdout:6/263: truncate d6/d9/db/f16 3277747 0 2024-04-06T14:17:05.289 INFO:tasks.workunit.client.0.smithi032.stdout:4/563: rename d0/df/c28 to d0/df/d23/d31/d50/d57/d5b/d9c/dc1/cc4 0 2024-04-06T14:17:05.289 INFO:tasks.workunit.client.1.smithi086.stdout:4/230: dwrite d0/d15/d19/f1c [0,4194304] 0 2024-04-06T14:17:05.290 INFO:tasks.workunit.client.1.smithi086.stdout:4/231: readlink d0/l42 0 2024-04-06T14:17:05.292 INFO:tasks.workunit.client.1.smithi086.stdout:1/293: creat db/d11/d1d/f5f x:0 0 0 2024-04-06T14:17:05.293 INFO:tasks.workunit.client.1.smithi086.stdout:2/269: link d0/df/f18 d0/d20/d27/d37/d52/f5c 0 2024-04-06T14:17:05.293 INFO:tasks.workunit.client.0.smithi032.stdout:4/564: creat d0/d82/fc5 x:0 0 0 2024-04-06T14:17:05.294 INFO:tasks.workunit.client.0.smithi032.stdout:2/521: truncate d0/d30/f43 3977292 0 2024-04-06T14:17:05.294 INFO:tasks.workunit.client.1.smithi086.stdout:9/260: dwrite d5/d13/f44 [0,4194304] 0 2024-04-06T14:17:05.295 INFO:tasks.workunit.client.1.smithi086.stdout:5/316: rmdir d0/d16/d18/d2a/d50 39 2024-04-06T14:17:05.295 INFO:tasks.workunit.client.0.smithi032.stdout:2/522: fsync d0/d5a/f31 0 2024-04-06T14:17:05.296 INFO:tasks.workunit.client.1.smithi086.stdout:6/264: write d6/d13/f27 [5234422,112208] 0 2024-04-06T14:17:05.296 INFO:tasks.workunit.client.0.smithi032.stdout:9/547: write d2/f2d [4119442,17045] 0 2024-04-06T14:17:05.297 INFO:tasks.workunit.client.1.smithi086.stdout:4/232: rmdir d0/d15 39 2024-04-06T14:17:05.298 INFO:tasks.workunit.client.0.smithi032.stdout:9/548: dread - d2/d15/d1e/d65/d83/d7e/f75 zero size 2024-04-06T14:17:05.298 INFO:tasks.workunit.client.0.smithi032.stdout:4/565: unlink d0/df/d1f/f4e 0 2024-04-06T14:17:05.300 INFO:tasks.workunit.client.1.smithi086.stdout:2/270: truncate d0/d20/d27/d2b/d35/f47 767964 0 2024-04-06T14:17:05.301 INFO:tasks.workunit.client.0.smithi032.stdout:4/566: write d0/df/d1f/d42/d45/f53 [1008399,1725] 0 2024-04-06T14:17:05.302 INFO:tasks.workunit.client.1.smithi086.stdout:2/271: readlink d0/d20/d27/d2b/d35/d36/d4d/l57 0 2024-04-06T14:17:05.303 INFO:tasks.workunit.client.0.smithi032.stdout:2/523: getdents d0/d30/d7f 0 2024-04-06T14:17:05.304 INFO:tasks.workunit.client.1.smithi086.stdout:2/272: chown d0/d20/d27/d2b/d35/d36/d4a 2 1 2024-04-06T14:17:05.304 INFO:tasks.workunit.client.0.smithi032.stdout:4/567: write d0/df/d23/d31/d50/d5e/faa [379342,59179] 0 2024-04-06T14:17:05.305 INFO:tasks.workunit.client.1.smithi086.stdout:1/294: rename db/d11/f33 to db/d11/d24/f60 0 2024-04-06T14:17:05.305 INFO:tasks.workunit.client.0.smithi032.stdout:2/524: chown d0/l9 0 1 2024-04-06T14:17:05.307 INFO:tasks.workunit.client.0.smithi032.stdout:9/549: getdents d2/d15/d55/d59 0 2024-04-06T14:17:05.308 INFO:tasks.workunit.client.0.smithi032.stdout:2/525: mkdir d0/d6e/dae 0 2024-04-06T14:17:05.310 INFO:tasks.workunit.client.1.smithi086.stdout:9/261: mkdir d5/d10/d14/d19/d25/d59 0 2024-04-06T14:17:05.310 INFO:tasks.workunit.client.0.smithi032.stdout:9/550: fdatasync d2/d15/d55/d5f/d71/d81/fab 0 2024-04-06T14:17:05.311 INFO:tasks.workunit.client.0.smithi032.stdout:4/568: rename d0/df/d2b/d46/cc0 to d0/df/d23/d31/d50/d57/cc6 0 2024-04-06T14:17:05.313 INFO:tasks.workunit.client.1.smithi086.stdout:7/258: mkdir d7/d10/d11/d1c/d24/d4d/d5c 0 2024-04-06T14:17:05.313 INFO:tasks.workunit.client.1.smithi086.stdout:0/308: truncate d0/d1/d4/f43 614519 0 2024-04-06T14:17:05.314 INFO:tasks.workunit.client.0.smithi032.stdout:9/551: write d2/d15/d55/d5f/d71/d81/d4a/d73/fa1 [505530,25039] 0 2024-04-06T14:17:05.314 INFO:tasks.workunit.client.0.smithi032.stdout:2/526: truncate d0/d30/d39/d59/d67/d74/d78/d65/f93 452876 0 2024-04-06T14:17:05.315 INFO:tasks.workunit.client.0.smithi032.stdout:4/569: dread - d0/df/d23/d31/d50/d5e/fbf zero size 2024-04-06T14:17:05.315 INFO:tasks.workunit.client.1.smithi086.stdout:2/273: write d0/df/f1a [1676558,100890] 0 2024-04-06T14:17:05.316 INFO:tasks.workunit.client.1.smithi086.stdout:2/274: chown d0/d20/d27/d2b/d35/f47 0 1 2024-04-06T14:17:05.316 INFO:tasks.workunit.client.1.smithi086.stdout:5/317: rename d0/d16/d18/f4c to d0/d5e/f60 0 2024-04-06T14:17:05.318 INFO:tasks.workunit.client.1.smithi086.stdout:1/295: symlink db/d11/d42/l61 0 2024-04-06T14:17:05.319 INFO:tasks.workunit.client.0.smithi032.stdout:9/552: rename d2/d15/d55/d5f/d71/d81/d4a/f68 to d2/d15/d1e/d65/d6b/fb2 0 2024-04-06T14:17:05.320 INFO:tasks.workunit.client.1.smithi086.stdout:9/262: creat d5/d13/d1a/f5a x:0 0 0 2024-04-06T14:17:05.322 INFO:tasks.workunit.client.1.smithi086.stdout:7/259: write d7/fb [991050,32525] 0 2024-04-06T14:17:05.323 INFO:tasks.workunit.client.1.smithi086.stdout:0/309: creat d0/d15/d26/d33/f66 x:0 0 0 2024-04-06T14:17:05.324 INFO:tasks.workunit.client.0.smithi032.stdout:4/570: getdents d0/d82/d4b 0 2024-04-06T14:17:05.325 INFO:tasks.workunit.client.1.smithi086.stdout:7/260: write d7/d10/d30/f4a [4560767,7974] 0 2024-04-06T14:17:05.328 INFO:tasks.workunit.client.1.smithi086.stdout:6/265: rename d6/d13/d42/d55 to d6/d13/d5f/d60 0 2024-04-06T14:17:05.330 INFO:tasks.workunit.client.0.smithi032.stdout:4/571: creat d0/df/d23/fc7 x:0 0 0 2024-04-06T14:17:05.334 INFO:tasks.workunit.client.0.smithi032.stdout:9/553: dwrite d2/d15/f1a [0,4194304] 0 2024-04-06T14:17:05.337 INFO:tasks.workunit.client.1.smithi086.stdout:6/266: dwrite d6/d20/f51 [0,4194304] 0 2024-04-06T14:17:05.339 INFO:tasks.workunit.client.1.smithi086.stdout:9/263: dread d5/d10/d14/d19/d1b/f43 [0,4194304] 0 2024-04-06T14:17:05.339 INFO:tasks.workunit.client.0.smithi032.stdout:4/572: getdents d0/df/d23/d31/d50/d57/d5b/d9c 0 2024-04-06T14:17:05.341 INFO:tasks.workunit.client.1.smithi086.stdout:7/261: chown d7/d10/d11/d1c/l31 251 1 2024-04-06T14:17:05.343 INFO:tasks.workunit.client.0.smithi032.stdout:4/573: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc8 0 2024-04-06T14:17:05.343 INFO:tasks.workunit.client.1.smithi086.stdout:7/262: chown d7/d10/d11/d1c/f58 174506252 1 2024-04-06T14:17:05.345 INFO:tasks.workunit.client.1.smithi086.stdout:4/233: link d0/d15/d2b/l33 d0/d2c/d38/l45 0 2024-04-06T14:17:05.348 INFO:tasks.workunit.client.0.smithi032.stdout:9/554: truncate d2/f98 1486288 0 2024-04-06T14:17:05.348 INFO:tasks.workunit.client.1.smithi086.stdout:9/264: readlink d5/l6 0 2024-04-06T14:17:05.349 INFO:tasks.workunit.client.0.smithi032.stdout:9/555: write d2/d15/f29 [2118834,104291] 0 2024-04-06T14:17:05.349 INFO:tasks.workunit.client.1.smithi086.stdout:2/275: link d0/df/l32 d0/d20/d27/d37/l5d 0 2024-04-06T14:17:05.349 INFO:tasks.workunit.client.1.smithi086.stdout:7/263: creat d7/d10/d11/d1c/d24/d4d/f5d x:0 0 0 2024-04-06T14:17:05.350 INFO:tasks.workunit.client.1.smithi086.stdout:0/310: creat d0/d1/d2/f67 x:0 0 0 2024-04-06T14:17:05.350 INFO:tasks.workunit.client.0.smithi032.stdout:4/574: creat d0/df/d23/d31/d50/d57/d5b/fc9 x:0 0 0 2024-04-06T14:17:05.351 INFO:tasks.workunit.client.0.smithi032.stdout:9/556: dread - d2/d15/d55/d5f/d71/d81/fb0 zero size 2024-04-06T14:17:05.351 INFO:tasks.workunit.client.1.smithi086.stdout:0/311: chown d0/d1/d2/d1a/d51 1 1 2024-04-06T14:17:05.352 INFO:tasks.workunit.client.1.smithi086.stdout:6/267: creat d6/d13/d1a/f61 x:0 0 0 2024-04-06T14:17:05.352 INFO:tasks.workunit.client.1.smithi086.stdout:4/234: creat d0/de/f46 x:0 0 0 2024-04-06T14:17:05.353 INFO:tasks.workunit.client.0.smithi032.stdout:4/575: write d0/df/d23/d31/d50/d5e/fbf [414338,19786] 0 2024-04-06T14:17:05.354 INFO:tasks.workunit.client.0.smithi032.stdout:9/557: creat d2/d15/d1e/d65/d6b/d92/fb3 x:0 0 0 2024-04-06T14:17:05.354 INFO:tasks.workunit.client.1.smithi086.stdout:6/268: dread - d6/d9/d5c/f58 zero size 2024-04-06T14:17:05.355 INFO:tasks.workunit.client.0.smithi032.stdout:4/576: chown d0/df/d2b/c33 110275832 1 2024-04-06T14:17:05.355 INFO:tasks.workunit.client.0.smithi032.stdout:4/577: chown d0/d82/l7f 46 1 2024-04-06T14:17:05.355 INFO:tasks.workunit.client.0.smithi032.stdout:9/558: mknod d2/cb4 0 2024-04-06T14:17:05.358 INFO:tasks.workunit.client.0.smithi032.stdout:9/559: write d2/f97 [4391429,14780] 0 2024-04-06T14:17:05.364 INFO:tasks.workunit.client.0.smithi032.stdout:4/578: dwrite d0/d82/fc5 [0,4194304] 0 2024-04-06T14:17:05.364 INFO:tasks.workunit.client.0.smithi032.stdout:4/579: stat d0/df/d2b/d43/fa6 0 2024-04-06T14:17:05.365 INFO:tasks.workunit.client.0.smithi032.stdout:4/580: read - d0/df/fb9 zero size 2024-04-06T14:17:05.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:05 smithi086 ceph-mon[47925]: pgmap v20: 65 pgs: 65 active+clean; 1.3 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 19 MiB/s rd, 142 MiB/s wr, 227 op/s 2024-04-06T14:17:05.380 INFO:tasks.workunit.client.1.smithi086.stdout:3/314: sync 2024-04-06T14:17:05.382 INFO:tasks.workunit.client.1.smithi086.stdout:4/235: write d0/f10 [1395274,56159] 0 2024-04-06T14:17:05.383 INFO:tasks.workunit.client.1.smithi086.stdout:5/318: write d0/d16/d18/d2a/f46 [80274,20089] 0 2024-04-06T14:17:05.385 INFO:tasks.workunit.client.0.smithi032.stdout:9/560: creat d2/d15/d1e/d65/d83/d7e/fb5 x:0 0 0 2024-04-06T14:17:05.387 INFO:tasks.workunit.client.1.smithi086.stdout:5/319: stat d0/d43/c48 0 2024-04-06T14:17:05.388 INFO:tasks.workunit.client.1.smithi086.stdout:9/265: getdents d5/d13 0 2024-04-06T14:17:05.388 INFO:tasks.workunit.client.0.smithi032.stdout:9/561: stat d2/d15/d55/d5f/d71/d81/d4a/d73/fa9 0 2024-04-06T14:17:05.389 INFO:tasks.workunit.client.1.smithi086.stdout:2/276: getdents d0/d20/d27/d2b/d35/d36 0 2024-04-06T14:17:05.389 INFO:tasks.workunit.client.1.smithi086.stdout:0/312: dwrite d0/f3f [0,4194304] 0 2024-04-06T14:17:05.390 INFO:tasks.workunit.client.0.smithi032.stdout:4/581: creat d0/df/d23/d31/d50/d57/d8a/fca x:0 0 0 2024-04-06T14:17:05.390 INFO:tasks.workunit.client.1.smithi086.stdout:7/264: dwrite d7/d10/d30/f50 [4194304,4194304] 0 2024-04-06T14:17:05.390 INFO:tasks.workunit.client.1.smithi086.stdout:6/269: mkdir d6/d13/d5f/d60/d45/d4a/d62 0 2024-04-06T14:17:05.391 INFO:tasks.workunit.client.1.smithi086.stdout:5/320: fsync d0/d19/f3e 0 2024-04-06T14:17:05.392 INFO:tasks.workunit.client.0.smithi032.stdout:9/562: fsync d2/d15/f43 0 2024-04-06T14:17:05.394 INFO:tasks.workunit.client.1.smithi086.stdout:2/277: chown d0/df/f14 442492 1 2024-04-06T14:17:05.396 INFO:tasks.workunit.client.1.smithi086.stdout:5/321: truncate d0/d16/d18/d2a/f46 1375554 0 2024-04-06T14:17:05.396 INFO:tasks.workunit.client.1.smithi086.stdout:0/313: rmdir d0/d1 39 2024-04-06T14:17:05.397 INFO:tasks.workunit.client.1.smithi086.stdout:0/314: stat d0/f40 0 2024-04-06T14:17:05.402 INFO:tasks.workunit.client.1.smithi086.stdout:2/278: dread d0/d20/f2e [0,4194304] 0 2024-04-06T14:17:05.403 INFO:tasks.workunit.client.1.smithi086.stdout:5/322: truncate d0/d19/f26 3421960 0 2024-04-06T14:17:05.403 INFO:tasks.workunit.client.1.smithi086.stdout:7/265: read d7/d10/d11/d1c/d24/d2c/f39 [2441293,39965] 0 2024-04-06T14:17:05.404 INFO:tasks.workunit.client.1.smithi086.stdout:6/270: mkdir d6/d9/d63 0 2024-04-06T14:17:05.405 INFO:tasks.workunit.client.1.smithi086.stdout:5/323: fdatasync d0/d16/d18/d2a/f46 0 2024-04-06T14:17:05.406 INFO:tasks.workunit.client.1.smithi086.stdout:4/236: getdents d0 0 2024-04-06T14:17:05.406 INFO:tasks.workunit.client.1.smithi086.stdout:9/266: mknod d5/d10/d14/d19/c5b 0 2024-04-06T14:17:05.407 INFO:tasks.workunit.client.1.smithi086.stdout:6/271: fdatasync d6/d13/d1a/d22/f35 0 2024-04-06T14:17:05.408 INFO:tasks.workunit.client.1.smithi086.stdout:4/237: stat d0/de/lf 0 2024-04-06T14:17:05.408 INFO:tasks.workunit.client.1.smithi086.stdout:9/267: stat d5/d18/f32 0 2024-04-06T14:17:05.409 INFO:tasks.workunit.client.1.smithi086.stdout:2/279: mkdir d0/df/d5e 0 2024-04-06T14:17:05.411 INFO:tasks.workunit.client.1.smithi086.stdout:4/238: truncate d0/d2c/d30/d24/f31 436160 0 2024-04-06T14:17:05.412 INFO:tasks.workunit.client.1.smithi086.stdout:3/315: truncate d5/df/f2f 1277657 0 2024-04-06T14:17:05.413 INFO:tasks.workunit.client.1.smithi086.stdout:4/239: chown d0/d2c/d30/f3b 1858 1 2024-04-06T14:17:05.413 INFO:tasks.workunit.client.1.smithi086.stdout:5/324: dread d0/d19/f3c [0,4194304] 0 2024-04-06T14:17:05.414 INFO:tasks.workunit.client.1.smithi086.stdout:3/316: write d5/df/f45 [509370,18197] 0 2024-04-06T14:17:05.418 INFO:tasks.workunit.client.1.smithi086.stdout:2/280: dread d0/d20/f2e [0,4194304] 0 2024-04-06T14:17:05.418 INFO:tasks.workunit.client.1.smithi086.stdout:3/317: dread d5/f3e [0,4194304] 0 2024-04-06T14:17:05.419 INFO:tasks.workunit.client.1.smithi086.stdout:3/318: write d5/d36/f3b [2394617,95190] 0 2024-04-06T14:17:05.420 INFO:tasks.workunit.client.1.smithi086.stdout:3/319: chown d5/d33 81 1 2024-04-06T14:17:05.422 INFO:tasks.workunit.client.1.smithi086.stdout:5/325: mknod d0/d16/d18/d21/d23/d28/c61 0 2024-04-06T14:17:05.422 INFO:tasks.workunit.client.1.smithi086.stdout:7/266: truncate d7/d10/d11/d1c/d24/d2c/f39 1765384 0 2024-04-06T14:17:05.422 INFO:tasks.workunit.client.1.smithi086.stdout:5/326: stat d0/d5e/f60 0 2024-04-06T14:17:05.423 INFO:tasks.workunit.client.1.smithi086.stdout:6/272: truncate d6/d9/db/f16 3674647 0 2024-04-06T14:17:05.425 INFO:tasks.workunit.client.1.smithi086.stdout:0/315: link d0/d1/d25/l3b d0/d1/d4/d7/l68 0 2024-04-06T14:17:05.426 INFO:tasks.workunit.client.1.smithi086.stdout:2/281: symlink d0/d20/d27/d37/l5f 0 2024-04-06T14:17:05.426 INFO:tasks.workunit.client.1.smithi086.stdout:2/282: stat d0/d20/d27/d2b/d35/f47 0 2024-04-06T14:17:05.428 INFO:tasks.workunit.client.1.smithi086.stdout:2/283: write d0/d20/d27/d2b/d35/d36/f40 [1933430,2565] 0 2024-04-06T14:17:05.429 INFO:tasks.workunit.client.1.smithi086.stdout:3/320: truncate d5/df/f13 373657 0 2024-04-06T14:17:05.432 INFO:tasks.workunit.client.1.smithi086.stdout:5/327: creat d0/d43/f62 x:0 0 0 2024-04-06T14:17:05.434 INFO:tasks.workunit.client.1.smithi086.stdout:6/273: mkdir d6/d13/d1a/d64 0 2024-04-06T14:17:05.436 INFO:tasks.workunit.client.1.smithi086.stdout:0/316: creat d0/d1/d25/f69 x:0 0 0 2024-04-06T14:17:05.436 INFO:tasks.workunit.client.1.smithi086.stdout:7/267: dwrite d7/d10/d11/d1c/f1e [0,4194304] 0 2024-04-06T14:17:05.440 INFO:tasks.workunit.client.1.smithi086.stdout:2/284: rmdir d0/d20/d27/d2b/d35/d36/d4a 39 2024-04-06T14:17:05.441 INFO:tasks.workunit.client.1.smithi086.stdout:3/321: stat d5/d33/f47 0 2024-04-06T14:17:05.443 INFO:tasks.workunit.client.1.smithi086.stdout:5/328: creat d0/d16/f63 x:0 0 0 2024-04-06T14:17:05.445 INFO:tasks.workunit.client.1.smithi086.stdout:9/268: getdents d5/d10/d14 0 2024-04-06T14:17:05.446 INFO:tasks.workunit.client.1.smithi086.stdout:4/240: getdents d0/d15/d19 0 2024-04-06T14:17:05.447 INFO:tasks.workunit.client.1.smithi086.stdout:2/285: mknod d0/d20/d27/c60 0 2024-04-06T14:17:05.451 INFO:tasks.workunit.client.1.smithi086.stdout:0/317: dwrite d0/d1/d1b/d44/d5d/f5f [0,4194304] 0 2024-04-06T14:17:05.451 INFO:tasks.workunit.client.1.smithi086.stdout:9/269: write d5/d10/d14/d19/d1b/f45 [1635858,24296] 0 2024-04-06T14:17:05.454 INFO:tasks.workunit.client.1.smithi086.stdout:3/322: creat d5/d33/f52 x:0 0 0 2024-04-06T14:17:05.455 INFO:tasks.workunit.client.1.smithi086.stdout:0/318: write d0/d15/f4f [40113,121446] 0 2024-04-06T14:17:05.456 INFO:tasks.workunit.client.1.smithi086.stdout:3/323: chown d5/f3e 348291 1 2024-04-06T14:17:05.456 INFO:tasks.workunit.client.1.smithi086.stdout:4/241: getdents d0/d15 0 2024-04-06T14:17:05.459 INFO:tasks.workunit.client.0.smithi032.stdout:3/500: sync 2024-04-06T14:17:05.459 INFO:tasks.workunit.client.0.smithi032.stdout:5/502: sync 2024-04-06T14:17:05.459 INFO:tasks.workunit.client.0.smithi032.stdout:0/373: sync 2024-04-06T14:17:05.460 INFO:tasks.workunit.client.1.smithi086.stdout:3/324: symlink d5/d42/l53 0 2024-04-06T14:17:05.460 INFO:tasks.workunit.client.1.smithi086.stdout:5/329: link d0/d16/d18/l10 d0/l64 0 2024-04-06T14:17:05.460 INFO:tasks.workunit.client.0.smithi032.stdout:5/503: chown d16/d18/d33/d35/d88/la0 0 1 2024-04-06T14:17:05.463 INFO:tasks.workunit.client.1.smithi086.stdout:7/268: dread d7/d1d/fc [0,4194304] 0 2024-04-06T14:17:05.463 INFO:tasks.workunit.client.1.smithi086.stdout:0/319: unlink d0/d1/d4/d7/l68 0 2024-04-06T14:17:05.465 INFO:tasks.workunit.client.1.smithi086.stdout:5/330: mkdir d0/d16/d65 0 2024-04-06T14:17:05.468 INFO:tasks.workunit.client.1.smithi086.stdout:7/269: fsync d7/d10/d11/d1c/d24/d2c/f39 0 2024-04-06T14:17:05.469 INFO:tasks.workunit.client.1.smithi086.stdout:0/320: write d0/d15/d4e/d48/f31 [968462,97105] 0 2024-04-06T14:17:05.470 INFO:tasks.workunit.client.1.smithi086.stdout:5/331: write d0/d19/f4b [670723,14129] 0 2024-04-06T14:17:05.473 INFO:tasks.workunit.client.0.smithi032.stdout:0/374: creat d3/da/d25/d57/f7d x:0 0 0 2024-04-06T14:17:05.473 INFO:tasks.workunit.client.1.smithi086.stdout:2/286: dread d0/df/f1a [0,4194304] 0 2024-04-06T14:17:05.475 INFO:tasks.workunit.client.0.smithi032.stdout:0/375: write d3/da/d25/d2b/d4a/f62 [1408169,49284] 0 2024-04-06T14:17:05.476 INFO:tasks.workunit.client.1.smithi086.stdout:0/321: chown d0/d1/d25/f61 9 1 2024-04-06T14:17:05.476 INFO:tasks.workunit.client.1.smithi086.stdout:5/332: truncate d0/d19/f3e 4396454 0 2024-04-06T14:17:05.477 INFO:tasks.workunit.client.1.smithi086.stdout:6/274: dwrite d6/d13/d1a/f21 [0,4194304] 0 2024-04-06T14:17:05.478 INFO:tasks.workunit.client.1.smithi086.stdout:3/325: write d5/df/f2f [2100175,33600] 0 2024-04-06T14:17:05.479 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:05 smithi032 ceph-mon[39730]: pgmap v20: 65 pgs: 65 active+clean; 1.3 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 19 MiB/s rd, 142 MiB/s wr, 227 op/s 2024-04-06T14:17:05.481 INFO:tasks.workunit.client.1.smithi086.stdout:7/270: rename d7/d10/d11/d1c/d24/c45 to d7/d1d/c5e 0 2024-04-06T14:17:05.481 INFO:tasks.workunit.client.1.smithi086.stdout:2/287: creat d0/d20/f61 x:0 0 0 2024-04-06T14:17:05.482 INFO:tasks.workunit.client.1.smithi086.stdout:3/326: readlink d5/d33/l39 0 2024-04-06T14:17:05.483 INFO:tasks.workunit.client.1.smithi086.stdout:7/271: stat d7/d10/d30/c47 0 2024-04-06T14:17:05.483 INFO:tasks.workunit.client.1.smithi086.stdout:6/275: rmdir d6/d9/d43 39 2024-04-06T14:17:05.489 INFO:tasks.workunit.client.1.smithi086.stdout:7/272: symlink d7/d10/d30/l5f 0 2024-04-06T14:17:05.489 INFO:tasks.workunit.client.1.smithi086.stdout:9/270: dread d5/f20 [0,4194304] 0 2024-04-06T14:17:05.490 INFO:tasks.workunit.client.1.smithi086.stdout:6/276: creat d6/d13/d42/d52/f65 x:0 0 0 2024-04-06T14:17:05.494 INFO:tasks.workunit.client.0.smithi032.stdout:9/563: link d2/d15/d55/f85 d2/d15/d1e/d65/d83/d31/d39/da6/fb6 0 2024-04-06T14:17:05.496 INFO:tasks.workunit.client.1.smithi086.stdout:2/288: getdents d0/d20/d44 0 2024-04-06T14:17:05.497 INFO:tasks.workunit.client.0.smithi032.stdout:5/504: link d16/f90 d16/d18/d33/d35/d88/fad 0 2024-04-06T14:17:05.498 INFO:tasks.workunit.client.1.smithi086.stdout:0/322: dwrite d0/d1/d25/f61 [0,4194304] 0 2024-04-06T14:17:05.499 INFO:tasks.workunit.client.0.smithi032.stdout:9/564: creat d2/d15/d1e/d41/d6c/fb7 x:0 0 0 2024-04-06T14:17:05.500 INFO:tasks.workunit.client.0.smithi032.stdout:9/565: chown d2/d15/d1e/d41/d6c/d74/f79 0 1 2024-04-06T14:17:05.501 INFO:tasks.workunit.client.1.smithi086.stdout:6/277: write d6/d13/d1a/f21 [329424,81363] 0 2024-04-06T14:17:05.502 INFO:tasks.workunit.client.0.smithi032.stdout:5/505: unlink d16/l71 0 2024-04-06T14:17:05.503 INFO:tasks.workunit.client.1.smithi086.stdout:7/273: creat d7/d10/d11/d1c/f60 x:0 0 0 2024-04-06T14:17:05.504 INFO:tasks.workunit.client.1.smithi086.stdout:9/271: truncate d5/d18/f26 1928183 0 2024-04-06T14:17:05.504 INFO:tasks.workunit.client.1.smithi086.stdout:6/278: stat d6/d13/d42/d52 0 2024-04-06T14:17:05.507 INFO:tasks.workunit.client.1.smithi086.stdout:3/327: dread d5/d42/f4c [0,4194304] 0 2024-04-06T14:17:05.507 INFO:tasks.workunit.client.1.smithi086.stdout:6/279: fdatasync d6/d13/d1a/d22/f3f 0 2024-04-06T14:17:05.508 INFO:tasks.workunit.client.0.smithi032.stdout:9/566: rename d2/d15/d55/d5f/d71 to d2/d15/d55/d59/d93/d95/db8 0 2024-04-06T14:17:05.509 INFO:tasks.workunit.client.1.smithi086.stdout:3/328: dread - d5/f50 zero size 2024-04-06T14:17:05.509 INFO:tasks.workunit.client.1.smithi086.stdout:9/272: unlink d5/d10/d14/d19/c42 0 2024-04-06T14:17:05.509 INFO:tasks.workunit.client.1.smithi086.stdout:2/289: dread d0/f33 [0,4194304] 0 2024-04-06T14:17:05.511 INFO:tasks.workunit.client.1.smithi086.stdout:9/273: chown d5/d10/d14/d19/d25/f2f 1887402 1 2024-04-06T14:17:05.511 INFO:tasks.workunit.client.1.smithi086.stdout:2/290: stat d0/d20/d27/d2b/d35/d36/l58 0 2024-04-06T14:17:05.512 INFO:tasks.workunit.client.0.smithi032.stdout:9/567: dread d2/f2d [0,4194304] 0 2024-04-06T14:17:05.512 INFO:tasks.workunit.client.0.smithi032.stdout:5/506: mkdir d16/d18/dae 0 2024-04-06T14:17:05.513 INFO:tasks.workunit.client.0.smithi032.stdout:5/507: chown d16/d58/d54/f60 27 1 2024-04-06T14:17:05.514 INFO:tasks.workunit.client.1.smithi086.stdout:0/323: fsync d0/d1/d25/f61 0 2024-04-06T14:17:05.514 INFO:tasks.workunit.client.1.smithi086.stdout:9/274: symlink d5/d10/d14/d19/d25/l5c 0 2024-04-06T14:17:05.514 INFO:tasks.workunit.client.0.smithi032.stdout:9/568: write d2/d15/d55/d59/d93/d95/db8/d81/d4a/f8d [698387,120057] 0 2024-04-06T14:17:05.515 INFO:tasks.workunit.client.1.smithi086.stdout:0/324: stat d0/d15/d26 0 2024-04-06T14:17:05.516 INFO:tasks.workunit.client.0.smithi032.stdout:9/569: write d2/f97 [3474694,25305] 0 2024-04-06T14:17:05.517 INFO:tasks.workunit.client.1.smithi086.stdout:9/275: write d5/d10/d14/d19/d1b/f45 [896540,42204] 0 2024-04-06T14:17:05.518 INFO:tasks.workunit.client.1.smithi086.stdout:2/291: creat d0/d20/d27/d2b/d35/f62 x:0 0 0 2024-04-06T14:17:05.519 INFO:tasks.workunit.client.0.smithi032.stdout:5/508: rename c14 to d16/d18/d33/caf 0 2024-04-06T14:17:05.521 INFO:tasks.workunit.client.1.smithi086.stdout:3/329: truncate d5/f3e 306815 0 2024-04-06T14:17:05.522 INFO:tasks.workunit.client.0.smithi032.stdout:9/570: creat d2/fb9 x:0 0 0 2024-04-06T14:17:05.522 INFO:tasks.workunit.client.1.smithi086.stdout:3/330: stat d5/f50 0 2024-04-06T14:17:05.524 INFO:tasks.workunit.client.1.smithi086.stdout:0/325: truncate d0/d1/d1b/f39 4117896 0 2024-04-06T14:17:05.526 INFO:tasks.workunit.client.0.smithi032.stdout:9/571: dread - d2/d15/d1e/d65/d83/d31/d39/f52 zero size 2024-04-06T14:17:05.527 INFO:tasks.workunit.client.0.smithi032.stdout:3/501: write d3/f43 [3283067,39708] 0 2024-04-06T14:17:05.528 INFO:tasks.workunit.client.1.smithi086.stdout:7/274: dread d7/d10/d11/d1c/f2e [0,4194304] 0 2024-04-06T14:17:05.530 INFO:tasks.workunit.client.1.smithi086.stdout:8/408: sync 2024-04-06T14:17:05.530 INFO:tasks.workunit.client.1.smithi086.stdout:8/409: chown d0/f34 523580313 1 2024-04-06T14:17:05.532 INFO:tasks.workunit.client.1.smithi086.stdout:2/292: rename d0/d20/f4c to d0/d20/d27/d2b/d35/d36/d4d/d59/f63 0 2024-04-06T14:17:05.533 INFO:tasks.workunit.client.1.smithi086.stdout:6/280: dread d6/d9/db/f19 [0,4194304] 0 2024-04-06T14:17:05.533 INFO:tasks.workunit.client.1.smithi086.stdout:0/326: creat d0/d15/f6a x:0 0 0 2024-04-06T14:17:05.534 INFO:tasks.workunit.client.0.smithi032.stdout:5/509: dwrite d16/d18/d33/fa8 [0,4194304] 0 2024-04-06T14:17:05.535 INFO:tasks.workunit.client.1.smithi086.stdout:6/281: readlink d6/d9/db/l54 0 2024-04-06T14:17:05.535 INFO:tasks.workunit.client.1.smithi086.stdout:0/327: write d0/d15/f6a [769350,30311] 0 2024-04-06T14:17:05.536 INFO:tasks.workunit.client.1.smithi086.stdout:9/276: creat d5/d10/d14/d19/f5d x:0 0 0 2024-04-06T14:17:05.537 INFO:tasks.workunit.client.0.smithi032.stdout:3/502: mkdir d3/d65/d7d/d9c/dad/db1/d3c/dc9 0 2024-04-06T14:17:05.537 INFO:tasks.workunit.client.0.smithi032.stdout:4/582: dwrite d0/df/d23/f32 [0,4194304] 0 2024-04-06T14:17:05.537 INFO:tasks.workunit.client.1.smithi086.stdout:6/282: readlink d6/d13/d42/l5a 0 2024-04-06T14:17:05.539 INFO:tasks.workunit.client.1.smithi086.stdout:6/283: fdatasync d6/d9/d5c/f58 0 2024-04-06T14:17:05.541 INFO:tasks.workunit.client.0.smithi032.stdout:4/583: truncate d0/df/d23/f32 4522467 0 2024-04-06T14:17:05.544 INFO:tasks.workunit.client.1.smithi086.stdout:6/284: mknod d6/d13/d42/d52/c66 0 2024-04-06T14:17:05.545 INFO:tasks.workunit.client.1.smithi086.stdout:6/285: stat d6/d9/db/f19 0 2024-04-06T14:17:05.545 INFO:tasks.workunit.client.0.smithi032.stdout:9/572: rename d2/d15/d55/d59/d93/d95/db8/d81/c90 to d2/d15/d1e/d65/d6b/d7f/cba 0 2024-04-06T14:17:05.546 INFO:tasks.workunit.client.1.smithi086.stdout:8/410: mkdir d0/d71/d16/d39/d90 0 2024-04-06T14:17:05.547 INFO:tasks.workunit.client.1.smithi086.stdout:3/331: dread d5/d26/f3d [0,4194304] 0 2024-04-06T14:17:05.549 INFO:tasks.workunit.client.0.smithi032.stdout:9/573: symlink d2/d15/d55/lbb 0 2024-04-06T14:17:05.550 INFO:tasks.workunit.client.1.smithi086.stdout:6/286: unlink d6/d13/d1a/f61 0 2024-04-06T14:17:05.553 INFO:tasks.workunit.client.0.smithi032.stdout:3/503: chown d3/d65/d7d/d9c/dad/db1/d3c/f79 3 1 2024-04-06T14:17:05.553 INFO:tasks.workunit.client.0.smithi032.stdout:4/584: dwrite d0/df/d23/d31/f68 [0,4194304] 0 2024-04-06T14:17:05.554 INFO:tasks.workunit.client.1.smithi086.stdout:8/411: rename d0/db/d32/l60 to d0/d71/d22/d43/l91 0 2024-04-06T14:17:05.555 INFO:tasks.workunit.client.0.smithi032.stdout:9/574: creat d2/d15/d1e/d65/d6b/d92/fbc x:0 0 0 2024-04-06T14:17:05.556 INFO:tasks.workunit.client.1.smithi086.stdout:6/287: chown d6/c18 11 1 2024-04-06T14:17:05.557 INFO:tasks.workunit.client.1.smithi086.stdout:2/293: getdents d0/d20/d27/d37 0 2024-04-06T14:17:05.559 INFO:tasks.workunit.client.0.smithi032.stdout:5/510: dwrite d16/d18/d33/d35/d88/fad [0,4194304] 0 2024-04-06T14:17:05.560 INFO:tasks.workunit.client.1.smithi086.stdout:3/332: link d5/fe d5/d36/f54 0 2024-04-06T14:17:05.561 INFO:tasks.workunit.client.0.smithi032.stdout:3/504: symlink d3/d65/d7d/d9c/dad/db1/d3c/dc9/lca 0 2024-04-06T14:17:05.561 INFO:tasks.workunit.client.0.smithi032.stdout:5/511: dread d16/d58/f6d [0,4194304] 0 2024-04-06T14:17:05.561 INFO:tasks.workunit.client.1.smithi086.stdout:6/288: mknod d6/d20/c67 0 2024-04-06T14:17:05.562 INFO:tasks.workunit.client.1.smithi086.stdout:6/289: stat d6/d20/c47 0 2024-04-06T14:17:05.563 INFO:tasks.workunit.client.1.smithi086.stdout:9/277: truncate d5/d13/f2b 1375981 0 2024-04-06T14:17:05.563 INFO:tasks.workunit.client.1.smithi086.stdout:3/333: write d5/d42/f4c [3966487,57429] 0 2024-04-06T14:17:05.564 INFO:tasks.workunit.client.1.smithi086.stdout:9/278: chown d5/d13/d1a 97822 1 2024-04-06T14:17:05.570 INFO:tasks.workunit.client.0.smithi032.stdout:4/585: link d0/df/d23/d31/d50/d57/d5b/cae d0/df/d23/d31/d50/d57/d8a/ccb 0 2024-04-06T14:17:05.571 INFO:tasks.workunit.client.1.smithi086.stdout:6/290: mknod d6/c68 0 2024-04-06T14:17:05.571 INFO:tasks.workunit.client.1.smithi086.stdout:3/334: rmdir d5/d26/d4f 39 2024-04-06T14:17:05.573 INFO:tasks.workunit.client.1.smithi086.stdout:2/294: dwrite d0/f2d [0,4194304] 0 2024-04-06T14:17:05.573 INFO:tasks.workunit.client.0.smithi032.stdout:4/586: symlink d0/df/d2b/d66/lcc 0 2024-04-06T14:17:05.574 INFO:tasks.workunit.client.1.smithi086.stdout:3/335: readlink d5/df/l16 0 2024-04-06T14:17:05.575 INFO:tasks.workunit.client.1.smithi086.stdout:3/336: fdatasync d5/d33/f52 0 2024-04-06T14:17:05.577 INFO:tasks.workunit.client.0.smithi032.stdout:5/512: truncate d16/d18/f1e 357440 0 2024-04-06T14:17:05.577 INFO:tasks.workunit.client.1.smithi086.stdout:0/328: dwrite d0/d1/d4/f43 [0,4194304] 0 2024-04-06T14:17:05.578 INFO:tasks.workunit.client.0.smithi032.stdout:4/587: truncate d0/df/d23/d31/f87 864857 0 2024-04-06T14:17:05.580 INFO:tasks.workunit.client.1.smithi086.stdout:8/412: readlink d0/d71/d22/d43/l91 0 2024-04-06T14:17:05.581 INFO:tasks.workunit.client.1.smithi086.stdout:2/295: symlink d0/d20/d27/d2b/l64 0 2024-04-06T14:17:05.581 INFO:tasks.workunit.client.1.smithi086.stdout:8/413: dread - d0/d13/f7c zero size 2024-04-06T14:17:05.582 INFO:tasks.workunit.client.0.smithi032.stdout:4/588: fsync d0/d82/d4b/fbd 0 2024-04-06T14:17:05.582 INFO:tasks.workunit.client.1.smithi086.stdout:6/291: dread d6/d9/f31 [0,4194304] 0 2024-04-06T14:17:05.583 INFO:tasks.workunit.client.0.smithi032.stdout:5/513: mknod d16/d58/d54/d74/d8f/cb0 0 2024-04-06T14:17:05.583 INFO:tasks.workunit.client.1.smithi086.stdout:3/337: creat d5/d42/f55 x:0 0 0 2024-04-06T14:17:05.584 INFO:tasks.workunit.client.0.smithi032.stdout:4/589: dread - d0/df/d2b/f72 zero size 2024-04-06T14:17:05.586 INFO:tasks.workunit.client.0.smithi032.stdout:3/505: dwrite d3/f37 [0,4194304] 0 2024-04-06T14:17:05.587 INFO:tasks.workunit.client.0.smithi032.stdout:4/590: readlink d0/df/d1f/d42/d45/l7b 0 2024-04-06T14:17:05.588 INFO:tasks.workunit.client.0.smithi032.stdout:4/591: stat d0/df/d2b/d46/d58/l98 0 2024-04-06T14:17:05.588 INFO:tasks.workunit.client.0.smithi032.stdout:5/514: symlink d16/d18/lb1 0 2024-04-06T14:17:05.589 INFO:tasks.workunit.client.1.smithi086.stdout:8/414: creat d0/db/f92 x:0 0 0 2024-04-06T14:17:05.590 INFO:tasks.workunit.client.0.smithi032.stdout:4/592: readlink d0/df/d1f/d42/d45/l93 0 2024-04-06T14:17:05.591 INFO:tasks.workunit.client.1.smithi086.stdout:9/279: truncate d5/d10/d14/d19/d1b/f43 559549 0 2024-04-06T14:17:05.593 INFO:tasks.workunit.client.1.smithi086.stdout:2/296: rename d0/d20/d27/d2b/d35/d36/d4d/d59/f63 to d0/d20/d27/d2b/f65 0 2024-04-06T14:17:05.593 INFO:tasks.workunit.client.1.smithi086.stdout:8/415: chown d0/d71/d22/d65/d6e 225976 1 2024-04-06T14:17:05.593 INFO:tasks.workunit.client.0.smithi032.stdout:5/515: mkdir d16/d58/d54/d94/db2 0 2024-04-06T14:17:05.594 INFO:tasks.workunit.client.1.smithi086.stdout:9/280: readlink d5/d13/d1a/l33 0 2024-04-06T14:17:05.595 INFO:tasks.workunit.client.0.smithi032.stdout:3/506: mkdir d3/d65/d89/dcb 0 2024-04-06T14:17:05.598 INFO:tasks.workunit.client.1.smithi086.stdout:6/292: dwrite d6/d13/f5d [0,4194304] 0 2024-04-06T14:17:05.600 INFO:tasks.workunit.client.1.smithi086.stdout:6/293: chown d6/f40 316 1 2024-04-06T14:17:05.602 INFO:tasks.workunit.client.1.smithi086.stdout:8/416: fdatasync d0/d71/d16/d39/f66 0 2024-04-06T14:17:05.602 INFO:tasks.workunit.client.1.smithi086.stdout:9/281: mkdir d5/d13/d47/d5e 0 2024-04-06T14:17:05.603 INFO:tasks.workunit.client.1.smithi086.stdout:6/294: creat d6/d9/db/f69 x:0 0 0 2024-04-06T14:17:05.604 INFO:tasks.workunit.client.0.smithi032.stdout:5/516: unlink d16/d18/d33/d45/d52/f53 0 2024-04-06T14:17:05.604 INFO:tasks.workunit.client.1.smithi086.stdout:2/297: dread d0/f2d [0,4194304] 0 2024-04-06T14:17:05.604 INFO:tasks.workunit.client.1.smithi086.stdout:9/282: mkdir d5/d13/d1a/d5f 0 2024-04-06T14:17:05.607 INFO:tasks.workunit.client.0.smithi032.stdout:3/507: read d3/fa [3503168,111248] 0 2024-04-06T14:17:05.610 INFO:tasks.workunit.client.1.smithi086.stdout:6/295: dread d6/d13/d1a/d22/f35 [0,4194304] 0 2024-04-06T14:17:05.613 INFO:tasks.workunit.client.0.smithi032.stdout:4/593: truncate d0/df/d1f/d2e/fa5 569239 0 2024-04-06T14:17:05.616 INFO:tasks.workunit.client.0.smithi032.stdout:5/517: dread d16/d18/f30 [0,4194304] 0 2024-04-06T14:17:05.619 INFO:tasks.workunit.client.1.smithi086.stdout:3/338: dread d5/df/f25 [0,4194304] 0 2024-04-06T14:17:05.620 INFO:tasks.workunit.client.0.smithi032.stdout:3/508: link d3/d65/d7d/d9c/dad/db1/f62 d3/d65/d7d/d9c/dad/dc5/fcc 0 2024-04-06T14:17:05.621 INFO:tasks.workunit.client.0.smithi032.stdout:3/509: write d3/d65/d89/fa2 [4597622,14698] 0 2024-04-06T14:17:05.625 INFO:tasks.workunit.client.0.smithi032.stdout:5/518: getdents d16/d58/d54/d94/db2 0 2024-04-06T14:17:05.627 INFO:tasks.workunit.client.1.smithi086.stdout:0/329: dread d0/d1/d2/d1a/f56 [0,4194304] 0 2024-04-06T14:17:05.628 INFO:tasks.workunit.client.0.smithi032.stdout:4/594: fsync d0/df/d1f/d2e/f9b 0 2024-04-06T14:17:05.628 INFO:tasks.workunit.client.0.smithi032.stdout:5/519: write d16/d18/f31 [3929015,76350] 0 2024-04-06T14:17:05.630 INFO:tasks.workunit.client.0.smithi032.stdout:4/595: mknod d0/d82/d4b/ccd 0 2024-04-06T14:17:05.631 INFO:tasks.workunit.client.0.smithi032.stdout:3/510: getdents d3/d16 0 2024-04-06T14:17:05.632 INFO:tasks.workunit.client.0.smithi032.stdout:5/520: symlink d16/d18/d33/d35/d88/lb3 0 2024-04-06T14:17:05.633 INFO:tasks.workunit.client.0.smithi032.stdout:4/596: symlink d0/d82/d8e/lce 0 2024-04-06T14:17:05.634 INFO:tasks.workunit.client.0.smithi032.stdout:3/511: mkdir d3/d65/d7d/d9c/dbc/dcd 0 2024-04-06T14:17:05.635 INFO:tasks.workunit.client.0.smithi032.stdout:4/597: dread - d0/df/d2b/d46/d58/f84 zero size 2024-04-06T14:17:05.636 INFO:tasks.workunit.client.0.smithi032.stdout:5/521: mknod d16/d18/d33/d45/d52/d39/cb4 0 2024-04-06T14:17:05.637 INFO:tasks.workunit.client.0.smithi032.stdout:3/512: symlink d3/d65/d7d/d9c/db8/lce 0 2024-04-06T14:17:05.638 INFO:tasks.workunit.client.0.smithi032.stdout:5/522: symlink d16/d18/dae/lb5 0 2024-04-06T14:17:05.639 INFO:tasks.workunit.client.0.smithi032.stdout:3/513: mkdir d3/dcf 0 2024-04-06T14:17:05.639 INFO:tasks.workunit.client.0.smithi032.stdout:5/523: write d16/d18/d33/d35/d4d/f82 [30146,103185] 0 2024-04-06T14:17:05.640 INFO:tasks.workunit.client.0.smithi032.stdout:5/524: write d16/d18/f3d [441978,111301] 0 2024-04-06T14:17:05.641 INFO:tasks.workunit.client.0.smithi032.stdout:4/598: rename d0/ld to d0/df/lcf 0 2024-04-06T14:17:05.641 INFO:tasks.workunit.client.0.smithi032.stdout:5/525: stat d16/d18 0 2024-04-06T14:17:05.643 INFO:tasks.workunit.client.0.smithi032.stdout:4/599: mknod d0/df/d23/d31/d50/d5e/cd0 0 2024-04-06T14:17:05.645 INFO:tasks.workunit.client.0.smithi032.stdout:5/526: link d16/d58/f6d d16/d18/d33/d45/d52/d39/fb6 0 2024-04-06T14:17:05.648 INFO:tasks.workunit.client.0.smithi032.stdout:3/514: getdents d3/d65/d7d/d9c/db8 0 2024-04-06T14:17:05.648 INFO:tasks.workunit.client.0.smithi032.stdout:5/527: dread d16/d58/f7a [0,4194304] 0 2024-04-06T14:17:05.655 INFO:tasks.workunit.client.0.smithi032.stdout:5/528: dread d16/d18/d33/f8a [0,4194304] 0 2024-04-06T14:17:05.656 INFO:tasks.workunit.client.0.smithi032.stdout:5/529: dread - d16/d18/d33/d35/d4d/f9c zero size 2024-04-06T14:17:05.662 INFO:tasks.workunit.client.0.smithi032.stdout:5/530: dwrite d16/d18/d33/d35/d88/fad [0,4194304] 0 2024-04-06T14:17:05.664 INFO:tasks.workunit.client.0.smithi032.stdout:6/509: sync 2024-04-06T14:17:05.664 INFO:tasks.workunit.client.0.smithi032.stdout:8/477: sync 2024-04-06T14:17:05.664 INFO:tasks.workunit.client.0.smithi032.stdout:7/541: sync 2024-04-06T14:17:05.664 INFO:tasks.workunit.client.1.smithi086.stdout:2/298: mknod d0/d20/d44/c66 0 2024-04-06T14:17:05.665 INFO:tasks.workunit.client.0.smithi032.stdout:8/478: chown d3/d8/d12/d2f/d4a/d60/d82/f95 825526 1 2024-04-06T14:17:05.667 INFO:tasks.workunit.client.1.smithi086.stdout:3/339: fdatasync d5/df/f2c 0 2024-04-06T14:17:05.668 INFO:tasks.workunit.client.0.smithi032.stdout:5/531: creat d16/d18/d33/d45/d52/d39/fb7 x:0 0 0 2024-04-06T14:17:05.668 INFO:tasks.workunit.client.0.smithi032.stdout:6/510: write d4/d5/d8/d1f/f33 [1302904,38268] 0 2024-04-06T14:17:05.669 INFO:tasks.workunit.client.0.smithi032.stdout:5/532: chown d16/d18/f3c 1316 1 2024-04-06T14:17:05.671 INFO:tasks.workunit.client.1.smithi086.stdout:6/296: link d6/d9/db/f2f d6/d13/d5f/d60/d45/d4a/d62/f6a 0 2024-04-06T14:17:05.671 INFO:tasks.workunit.client.0.smithi032.stdout:7/542: rename d16/d20/d26/c29 to d16/d20/d63/cc4 0 2024-04-06T14:17:05.673 INFO:tasks.workunit.client.1.smithi086.stdout:6/297: chown d6/d9/db/l2e 821522 1 2024-04-06T14:17:05.673 INFO:tasks.workunit.client.1.smithi086.stdout:2/299: symlink d0/d20/d44/l67 0 2024-04-06T14:17:05.673 INFO:tasks.workunit.client.0.smithi032.stdout:6/511: creat d4/d5/d28/d65/fb1 x:0 0 0 2024-04-06T14:17:05.675 INFO:tasks.workunit.client.1.smithi086.stdout:3/340: dread d5/f2d [0,4194304] 0 2024-04-06T14:17:05.676 INFO:tasks.workunit.client.0.smithi032.stdout:7/543: chown d16/d20/d48/fbf 12 1 2024-04-06T14:17:05.676 INFO:tasks.workunit.client.0.smithi032.stdout:5/533: creat d16/d58/d54/d94/db2/fb8 x:0 0 0 2024-04-06T14:17:05.676 INFO:tasks.workunit.client.1.smithi086.stdout:9/283: chown d5/d13/f2b 3294 1 2024-04-06T14:17:05.676 INFO:tasks.workunit.client.0.smithi032.stdout:6/512: chown d4/d5/d18/d23/d82 697336718 1 2024-04-06T14:17:05.678 INFO:tasks.workunit.client.1.smithi086.stdout:9/284: write d5/d10/d14/d19/f1e [122850,60295] 0 2024-04-06T14:17:05.678 INFO:tasks.workunit.client.1.smithi086.stdout:6/298: unlink d6/d9/f31 0 2024-04-06T14:17:05.679 INFO:tasks.workunit.client.0.smithi032.stdout:8/479: dwrite d3/d8/d12/d2f/d46/fa8 [8388608,4194304] 0 2024-04-06T14:17:05.679 INFO:tasks.workunit.client.0.smithi032.stdout:7/544: rmdir d16/d20/d26/d5f 39 2024-04-06T14:17:05.681 INFO:tasks.workunit.client.0.smithi032.stdout:6/513: creat d4/d5/d8/d2f/fb2 x:0 0 0 2024-04-06T14:17:05.681 INFO:tasks.workunit.client.1.smithi086.stdout:2/300: dwrite d0/d20/f25 [4194304,4194304] 0 2024-04-06T14:17:05.682 INFO:tasks.workunit.client.0.smithi032.stdout:7/545: mknod d16/d20/d48/d6d/cc5 0 2024-04-06T14:17:05.684 INFO:tasks.workunit.client.1.smithi086.stdout:0/330: link d0/d15/d4e/d48/f31 d0/d1/d4/f6b 0 2024-04-06T14:17:05.684 INFO:tasks.workunit.client.1.smithi086.stdout:3/341: fdatasync d5/f2d 0 2024-04-06T14:17:05.686 INFO:tasks.workunit.client.1.smithi086.stdout:6/299: creat d6/d9/d5c/f6b x:0 0 0 2024-04-06T14:17:05.688 INFO:tasks.workunit.client.1.smithi086.stdout:0/331: fsync d0/d1/d2/d1a/d51/f60 0 2024-04-06T14:17:05.689 INFO:tasks.workunit.client.1.smithi086.stdout:0/332: truncate d0/d15/d4e/f5e 259326 0 2024-04-06T14:17:05.690 INFO:tasks.workunit.client.1.smithi086.stdout:3/342: mkdir d5/d36/d56 0 2024-04-06T14:17:05.691 INFO:tasks.workunit.client.1.smithi086.stdout:9/285: creat d5/d10/d14/d19/d25/d59/f60 x:0 0 0 2024-04-06T14:17:05.693 INFO:tasks.workunit.client.0.smithi032.stdout:8/480: link d3/d8/f5f d3/d69/fad 0 2024-04-06T14:17:05.694 INFO:tasks.workunit.client.0.smithi032.stdout:6/514: creat d4/d5/d14/fb3 x:0 0 0 2024-04-06T14:17:05.695 INFO:tasks.workunit.client.1.smithi086.stdout:2/301: creat d0/d20/d27/d2b/d35/d36/f68 x:0 0 0 2024-04-06T14:17:05.695 INFO:tasks.workunit.client.1.smithi086.stdout:0/333: truncate d0/d1/d4/de/f3c 1099005 0 2024-04-06T14:17:05.696 INFO:tasks.workunit.client.0.smithi032.stdout:8/481: symlink d3/lae 0 2024-04-06T14:17:05.697 INFO:tasks.workunit.client.1.smithi086.stdout:2/302: write d0/d20/d27/d37/f42 [4402264,55014] 0 2024-04-06T14:17:05.697 INFO:tasks.workunit.client.1.smithi086.stdout:3/343: rename d5/d26/d4f/d4e to d5/d36/d49/d57 0 2024-04-06T14:17:05.698 INFO:tasks.workunit.client.1.smithi086.stdout:9/286: creat d5/d13/d47/d5e/f61 x:0 0 0 2024-04-06T14:17:05.698 INFO:tasks.workunit.client.0.smithi032.stdout:6/515: mkdir d4/d5/d8/d2f/d9b/db4 0 2024-04-06T14:17:05.699 INFO:tasks.workunit.client.0.smithi032.stdout:7/546: link d16/d20/d26/d2d/d30/d3e/da7/db5/c7c d16/d20/cc6 0 2024-04-06T14:17:05.700 INFO:tasks.workunit.client.1.smithi086.stdout:0/334: write d0/d15/d4e/d48/f31 [486870,25979] 0 2024-04-06T14:17:05.701 INFO:tasks.workunit.client.1.smithi086.stdout:3/344: mkdir d5/d36/d49/d58 0 2024-04-06T14:17:05.701 INFO:tasks.workunit.client.1.smithi086.stdout:9/287: mknod d5/d13/d1a/c62 0 2024-04-06T14:17:05.702 INFO:tasks.workunit.client.0.smithi032.stdout:6/516: symlink d4/d5/d8/d2b/lb5 0 2024-04-06T14:17:05.702 INFO:tasks.workunit.client.1.smithi086.stdout:2/303: mknod d0/d20/d27/d2b/d35/d36/d4d/d59/c69 0 2024-04-06T14:17:05.703 INFO:tasks.workunit.client.0.smithi032.stdout:8/482: dwrite d3/f96 [0,4194304] 0 2024-04-06T14:17:05.704 INFO:tasks.workunit.client.0.smithi032.stdout:6/517: chown d4/d5/d28/d65/d6e/c91 17334852 1 2024-04-06T14:17:05.706 INFO:tasks.workunit.client.0.smithi032.stdout:7/547: creat d16/d20/d26/d5f/fc7 x:0 0 0 2024-04-06T14:17:05.707 INFO:tasks.workunit.client.0.smithi032.stdout:7/548: write d16/d20/d26/f31 [2033999,1884] 0 2024-04-06T14:17:05.709 INFO:tasks.workunit.client.1.smithi086.stdout:2/304: dwrite d0/fd [4194304,4194304] 0 2024-04-06T14:17:05.710 INFO:tasks.workunit.client.0.smithi032.stdout:8/483: unlink d3/d8/d12/faa 0 2024-04-06T14:17:05.710 INFO:tasks.workunit.client.1.smithi086.stdout:9/288: rename d5/d13/f2b to d5/d13/d1a/f63 0 2024-04-06T14:17:05.711 INFO:tasks.workunit.client.1.smithi086.stdout:9/289: read - d5/d10/d14/f51 zero size 2024-04-06T14:17:05.712 INFO:tasks.workunit.client.0.smithi032.stdout:6/518: mknod d4/d5/d8/d4b/d7c/cb6 0 2024-04-06T14:17:05.713 INFO:tasks.workunit.client.1.smithi086.stdout:9/290: dread - d5/d10/d14/d19/d25/d59/f60 zero size 2024-04-06T14:17:05.714 INFO:tasks.workunit.client.0.smithi032.stdout:8/484: readlink d3/d8/d12/d83/l9a 0 2024-04-06T14:17:05.715 INFO:tasks.workunit.client.1.smithi086.stdout:3/345: truncate d5/df/f13 1341425 0 2024-04-06T14:17:05.719 INFO:tasks.workunit.client.0.smithi032.stdout:7/549: dwrite f10 [0,4194304] 0 2024-04-06T14:17:05.720 INFO:tasks.workunit.client.1.smithi086.stdout:0/335: fsync d0/d1/d4/de/f3c 0 2024-04-06T14:17:05.720 INFO:tasks.workunit.client.0.smithi032.stdout:7/550: chown d16/d20/d26/fb2 952697 1 2024-04-06T14:17:05.721 INFO:tasks.workunit.client.0.smithi032.stdout:8/485: mkdir d3/d8/d12/d83/d89/da0/daf 0 2024-04-06T14:17:05.722 INFO:tasks.workunit.client.1.smithi086.stdout:9/291: dwrite d5/d18/f3f [4194304,4194304] 0 2024-04-06T14:17:05.724 INFO:tasks.workunit.client.0.smithi032.stdout:7/551: mknod d16/d20/d26/d2d/d30/d3e/d96/cc8 0 2024-04-06T14:17:05.724 INFO:tasks.workunit.client.1.smithi086.stdout:2/305: symlink d0/d20/d27/d2b/d35/d36/l6a 0 2024-04-06T14:17:05.725 INFO:tasks.workunit.client.0.smithi032.stdout:7/552: readlink d16/d20/d26/d2d/d30/l3d 0 2024-04-06T14:17:05.728 INFO:tasks.workunit.client.1.smithi086.stdout:3/346: mkdir d5/d42/d59 0 2024-04-06T14:17:05.729 INFO:tasks.workunit.client.1.smithi086.stdout:6/300: dwrite d6/d13/d5f/d60/d45/d4a/d62/f6a [0,4194304] 0 2024-04-06T14:17:05.730 INFO:tasks.workunit.client.1.smithi086.stdout:9/292: chown d5/d13/f2d 134485635 1 2024-04-06T14:17:05.733 INFO:tasks.workunit.client.1.smithi086.stdout:3/347: mkdir d5/d36/d5a 0 2024-04-06T14:17:05.734 INFO:tasks.workunit.client.1.smithi086.stdout:9/293: write d5/d13/d1a/f24 [3918075,14292] 0 2024-04-06T14:17:05.734 INFO:tasks.workunit.client.1.smithi086.stdout:3/348: readlink d5/l38 0 2024-04-06T14:17:05.737 INFO:tasks.workunit.client.0.smithi032.stdout:7/553: link d16/d20/d26/d2d/d30/d3e/d96/c67 d16/cc9 0 2024-04-06T14:17:05.737 INFO:tasks.workunit.client.0.smithi032.stdout:8/486: dread d3/d8/d7/f16 [0,4194304] 0 2024-04-06T14:17:05.738 INFO:tasks.workunit.client.0.smithi032.stdout:7/554: stat d16/d20/d48/d8d 0 2024-04-06T14:17:05.741 INFO:tasks.workunit.client.1.smithi086.stdout:6/301: dwrite d6/d13/f27 [4194304,4194304] 0 2024-04-06T14:17:05.742 INFO:tasks.workunit.client.1.smithi086.stdout:2/306: link d0/d20/d27/d2b/d35/d36/d4a/c5a d0/df/c6b 0 2024-04-06T14:17:05.744 INFO:tasks.workunit.client.1.smithi086.stdout:3/349: dread d5/df/f25 [0,4194304] 0 2024-04-06T14:17:05.745 INFO:tasks.workunit.client.1.smithi086.stdout:3/350: chown f4 3 1 2024-04-06T14:17:05.745 INFO:tasks.workunit.client.0.smithi032.stdout:8/487: creat d3/d8/d12/fb0 x:0 0 0 2024-04-06T14:17:05.745 INFO:tasks.workunit.client.0.smithi032.stdout:8/488: dread - d3/d8/d12/d2f/f77 zero size 2024-04-06T14:17:05.745 INFO:tasks.workunit.client.1.smithi086.stdout:9/294: dwrite d5/d13/d47/d5e/f61 [0,4194304] 0 2024-04-06T14:17:05.747 INFO:tasks.workunit.client.1.smithi086.stdout:2/307: mknod d0/df/d48/c6c 0 2024-04-06T14:17:05.751 INFO:tasks.workunit.client.0.smithi032.stdout:6/519: dwrite d4/f13 [0,4194304] 0 2024-04-06T14:17:05.752 INFO:tasks.workunit.client.1.smithi086.stdout:6/302: creat d6/d9/f6c x:0 0 0 2024-04-06T14:17:05.752 INFO:tasks.workunit.client.1.smithi086.stdout:2/308: stat d0/df/f1b 0 2024-04-06T14:17:05.752 INFO:tasks.workunit.client.1.smithi086.stdout:1/296: sync 2024-04-06T14:17:05.753 INFO:tasks.workunit.client.1.smithi086.stdout:3/351: mknod d5/d36/d56/c5b 0 2024-04-06T14:17:05.755 INFO:tasks.workunit.client.0.smithi032.stdout:8/489: rmdir d3/d8/d12/d2f/d46 39 2024-04-06T14:17:05.759 INFO:tasks.workunit.client.0.smithi032.stdout:8/490: dread - d3/d8/d12/fb0 zero size 2024-04-06T14:17:05.759 INFO:tasks.workunit.client.1.smithi086.stdout:1/297: write db/d13/d55/f59 [1679902,21962] 0 2024-04-06T14:17:05.760 INFO:tasks.workunit.client.0.smithi032.stdout:6/520: fsync d4/d5/d18/f52 0 2024-04-06T14:17:05.762 INFO:tasks.workunit.client.1.smithi086.stdout:1/298: chown db/f39 2032 1 2024-04-06T14:17:05.763 INFO:tasks.workunit.client.1.smithi086.stdout:6/303: dwrite d6/d9/db/f69 [0,4194304] 0 2024-04-06T14:17:05.764 INFO:tasks.workunit.client.1.smithi086.stdout:2/309: rename d0/l4b to d0/d20/d27/d2b/d35/l6d 0 2024-04-06T14:17:05.765 INFO:tasks.workunit.client.0.smithi032.stdout:8/491: mknod d3/d8/d12/d83/d89/da0/daf/cb1 0 2024-04-06T14:17:05.765 INFO:tasks.workunit.client.0.smithi032.stdout:6/521: chown d4/d5/d8/d1f/f63 2 1 2024-04-06T14:17:05.771 INFO:tasks.workunit.client.0.smithi032.stdout:6/522: symlink d4/d5/d8/lb7 0 2024-04-06T14:17:05.772 INFO:tasks.workunit.client.0.smithi032.stdout:7/555: truncate d16/d20/d26/d5f/fa2 2746087 0 2024-04-06T14:17:05.772 INFO:tasks.workunit.client.0.smithi032.stdout:8/492: mknod d3/d8/d12/d2f/d4a/cb2 0 2024-04-06T14:17:05.773 INFO:tasks.workunit.client.1.smithi086.stdout:9/295: truncate d5/d13/f15 279199 0 2024-04-06T14:17:05.773 INFO:tasks.workunit.client.0.smithi032.stdout:6/523: fsync d4/d5/d19/f46 0 2024-04-06T14:17:05.774 INFO:tasks.workunit.client.1.smithi086.stdout:3/352: dwrite d5/f2d [0,4194304] 0 2024-04-06T14:17:05.774 INFO:tasks.workunit.client.0.smithi032.stdout:7/556: readlink d16/d20/d26/d2d/d30/laa 0 2024-04-06T14:17:05.775 INFO:tasks.workunit.client.0.smithi032.stdout:8/493: fdatasync d3/d8/d12/d2f/d46/f74 0 2024-04-06T14:17:05.779 INFO:tasks.workunit.client.0.smithi032.stdout:7/557: creat d16/d20/d26/d2d/d30/d97/fca x:0 0 0 2024-04-06T14:17:05.779 INFO:tasks.workunit.client.1.smithi086.stdout:3/353: mkdir d5/d36/d5c 0 2024-04-06T14:17:05.782 INFO:tasks.workunit.client.0.smithi032.stdout:8/494: rename d3/d8/d7/d1a/f24 to d3/d8/d7/d1a/fb3 0 2024-04-06T14:17:05.782 INFO:tasks.workunit.client.0.smithi032.stdout:8/495: chown d3/d69 119 1 2024-04-06T14:17:05.782 INFO:tasks.workunit.client.1.smithi086.stdout:9/296: creat d5/d13/d47/d55/f64 x:0 0 0 2024-04-06T14:17:05.785 INFO:tasks.workunit.client.0.smithi032.stdout:7/558: getdents d16/d20/d26/d2d/d30/dbb 0 2024-04-06T14:17:05.785 INFO:tasks.workunit.client.0.smithi032.stdout:8/496: mknod d3/d69/cb4 0 2024-04-06T14:17:05.785 INFO:tasks.workunit.client.1.smithi086.stdout:1/299: dwrite db/d11/d42/d53/f57 [0,4194304] 0 2024-04-06T14:17:05.785 INFO:tasks.workunit.client.0.smithi032.stdout:6/524: link d4/d5/d35/c42 d4/d5/d83/dad/cb8 0 2024-04-06T14:17:05.786 INFO:tasks.workunit.client.0.smithi032.stdout:8/497: rename d3 to d3/d8/d12/d83/db5 22 2024-04-06T14:17:05.787 INFO:tasks.workunit.client.0.smithi032.stdout:8/498: read - d3/d8/d7/f81 zero size 2024-04-06T14:17:05.789 INFO:tasks.workunit.client.0.smithi032.stdout:7/559: creat d16/d20/d48/d6d/fcb x:0 0 0 2024-04-06T14:17:05.791 INFO:tasks.workunit.client.1.smithi086.stdout:6/304: dwrite d6/f3c [4194304,4194304] 0 2024-04-06T14:17:05.791 INFO:tasks.workunit.client.1.smithi086.stdout:2/310: link d0/d20/d27/d2b/d35/d36/d4d/l57 d0/df/d48/l6e 0 2024-04-06T14:17:05.791 INFO:tasks.workunit.client.1.smithi086.stdout:6/305: fsync d6/d13/f27 0 2024-04-06T14:17:05.791 INFO:tasks.workunit.client.1.smithi086.stdout:3/354: mkdir d5/d33/d5d 0 2024-04-06T14:17:05.792 INFO:tasks.workunit.client.0.smithi032.stdout:7/560: symlink d16/d20/d26/d5f/lcc 0 2024-04-06T14:17:05.792 INFO:tasks.workunit.client.0.smithi032.stdout:6/525: dread d4/d5/fa0 [0,4194304] 0 2024-04-06T14:17:05.793 INFO:tasks.workunit.client.0.smithi032.stdout:7/561: chown d16/d20/d26/d2d/d30/l3d 6320 1 2024-04-06T14:17:05.797 INFO:tasks.workunit.client.1.smithi086.stdout:2/311: mkdir d0/d20/d27/d2b/d35/d36/d4d/d6f 0 2024-04-06T14:17:05.799 INFO:tasks.workunit.client.1.smithi086.stdout:1/300: dread db/d11/d1f/f5a [0,4194304] 0 2024-04-06T14:17:05.799 INFO:tasks.workunit.client.1.smithi086.stdout:3/355: creat d5/d36/d5c/f5e x:0 0 0 2024-04-06T14:17:05.800 INFO:tasks.workunit.client.0.smithi032.stdout:7/562: rename d16/d20/d63/d9d/fab to d16/d92/db3/fcd 0 2024-04-06T14:17:05.801 INFO:tasks.workunit.client.1.smithi086.stdout:6/306: dread f5 [4194304,4194304] 0 2024-04-06T14:17:05.802 INFO:tasks.workunit.client.1.smithi086.stdout:2/312: truncate d0/df/f18 3585832 0 2024-04-06T14:17:05.802 INFO:tasks.workunit.client.1.smithi086.stdout:1/301: mknod db/d11/d24/c62 0 2024-04-06T14:17:05.804 INFO:tasks.workunit.client.1.smithi086.stdout:3/356: rename d5/df/f25 to d5/d36/d49/f5f 0 2024-04-06T14:17:05.805 INFO:tasks.workunit.client.1.smithi086.stdout:6/307: creat d6/d13/d5f/d60/f6d x:0 0 0 2024-04-06T14:17:05.807 INFO:tasks.workunit.client.1.smithi086.stdout:3/357: unlink d5/df/f2f 0 2024-04-06T14:17:05.808 INFO:tasks.workunit.client.1.smithi086.stdout:6/308: creat d6/d13/d1a/d22/f6e x:0 0 0 2024-04-06T14:17:05.809 INFO:tasks.workunit.client.1.smithi086.stdout:3/358: rename d5/fe to d5/d42/f60 0 2024-04-06T14:17:05.810 INFO:tasks.workunit.client.1.smithi086.stdout:2/313: dread d0/d20/d27/d2b/d35/d36/f3e [0,4194304] 0 2024-04-06T14:17:05.810 INFO:tasks.workunit.client.1.smithi086.stdout:6/309: symlink d6/d9/d63/l6f 0 2024-04-06T14:17:05.811 INFO:tasks.workunit.client.1.smithi086.stdout:3/359: creat d5/d26/d4f/f61 x:0 0 0 2024-04-06T14:17:05.818 INFO:tasks.workunit.client.1.smithi086.stdout:9/297: dread d5/d13/d1a/f24 [0,4194304] 0 2024-04-06T14:17:05.822 INFO:tasks.workunit.client.0.smithi032.stdout:6/526: mkdir d4/d5/d8/d1f/d61/db9 0 2024-04-06T14:17:05.825 INFO:tasks.workunit.client.0.smithi032.stdout:7/563: write fa [942512,120922] 0 2024-04-06T14:17:05.825 INFO:tasks.workunit.client.0.smithi032.stdout:6/527: dread d4/f98 [0,4194304] 0 2024-04-06T14:17:05.827 INFO:tasks.workunit.client.0.smithi032.stdout:7/564: write d16/d20/d7a/da6/fc1 [225782,26620] 0 2024-04-06T14:17:05.830 INFO:tasks.workunit.client.0.smithi032.stdout:7/565: rmdir d16/d20/d48 39 2024-04-06T14:17:05.830 INFO:tasks.workunit.client.0.smithi032.stdout:7/566: chown d16/c1c 8583002 1 2024-04-06T14:17:05.831 INFO:tasks.workunit.client.0.smithi032.stdout:6/528: write d4/d5/d19/f55 [426680,54257] 0 2024-04-06T14:17:05.833 INFO:tasks.workunit.client.0.smithi032.stdout:7/567: mknod d16/d20/d6e/cce 0 2024-04-06T14:17:05.834 INFO:tasks.workunit.client.0.smithi032.stdout:7/568: chown d16/d20/d26/d5f/f90 661677 1 2024-04-06T14:17:05.835 INFO:tasks.workunit.client.0.smithi032.stdout:7/569: write d16/d20/d26/d5f/fc7 [1044593,89479] 0 2024-04-06T14:17:05.836 INFO:tasks.workunit.client.0.smithi032.stdout:6/529: truncate d4/f10 1624320 0 2024-04-06T14:17:05.838 INFO:tasks.workunit.client.0.smithi032.stdout:7/570: fsync d16/d20/d48/d8d/fb6 0 2024-04-06T14:17:05.838 INFO:tasks.workunit.client.0.smithi032.stdout:6/530: mknod d4/d5/d14/cba 0 2024-04-06T14:17:05.839 INFO:tasks.workunit.client.0.smithi032.stdout:7/571: fsync d16/d20/d54/f57 0 2024-04-06T14:17:05.849 INFO:tasks.workunit.client.0.smithi032.stdout:7/572: link d16/d20/d63/l77 d16/d20/d26/d2d/d30/d3e/da7/lcf 0 2024-04-06T14:17:05.849 INFO:tasks.workunit.client.0.smithi032.stdout:7/573: dread - d16/d20/d48/d8d/fb6 zero size 2024-04-06T14:17:05.849 INFO:tasks.workunit.client.0.smithi032.stdout:7/574: mknod d16/d20/d26/d70/cd0 0 2024-04-06T14:17:05.849 INFO:tasks.workunit.client.0.smithi032.stdout:7/575: dread - d16/d20/d54/f9c zero size 2024-04-06T14:17:05.849 INFO:tasks.workunit.client.0.smithi032.stdout:7/576: write d16/d20/d26/f44 [3766546,1064] 0 2024-04-06T14:17:05.852 INFO:tasks.workunit.client.0.smithi032.stdout:7/577: rmdir d16/d20/d63 39 2024-04-06T14:17:05.854 INFO:tasks.workunit.client.0.smithi032.stdout:6/531: dread d4/d5/d18/f31 [0,4194304] 0 2024-04-06T14:17:05.857 INFO:tasks.workunit.client.0.smithi032.stdout:6/532: creat d4/d5/d83/dad/fbb x:0 0 0 2024-04-06T14:17:05.858 INFO:tasks.workunit.client.0.smithi032.stdout:7/578: dread d16/d20/d26/f58 [0,4194304] 0 2024-04-06T14:17:05.859 INFO:tasks.workunit.client.0.smithi032.stdout:6/533: unlink d4/d5/d8/d1f/c3c 0 2024-04-06T14:17:05.860 INFO:tasks.workunit.client.0.smithi032.stdout:7/579: creat d16/d20/d48/d8d/fd1 x:0 0 0 2024-04-06T14:17:05.861 INFO:tasks.workunit.client.0.smithi032.stdout:7/580: creat d16/d20/d26/d70/fd2 x:0 0 0 2024-04-06T14:17:05.862 INFO:tasks.workunit.client.0.smithi032.stdout:6/534: mkdir d4/d5/d19/d21/dbc 0 2024-04-06T14:17:05.863 INFO:tasks.workunit.client.0.smithi032.stdout:7/581: fsync d16/d20/d26/d2d/f69 0 2024-04-06T14:17:05.864 INFO:tasks.workunit.client.0.smithi032.stdout:6/535: creat d4/d5/d18/d23/d82/d9d/fbd x:0 0 0 2024-04-06T14:17:05.867 INFO:tasks.workunit.client.0.smithi032.stdout:6/536: read d4/d5/d19/f55 [384328,127963] 0 2024-04-06T14:17:05.871 INFO:tasks.workunit.client.0.smithi032.stdout:6/537: creat d4/d5/d28/fbe x:0 0 0 2024-04-06T14:17:05.875 INFO:tasks.workunit.client.0.smithi032.stdout:6/538: write d4/d5/f15 [1749926,98015] 0 2024-04-06T14:17:05.875 INFO:tasks.workunit.client.0.smithi032.stdout:7/582: dread d16/d20/d48/d6d/f79 [0,4194304] 0 2024-04-06T14:17:05.876 INFO:tasks.workunit.client.0.smithi032.stdout:7/583: chown d16/d20/d26/d2d/faf 55465754 1 2024-04-06T14:17:05.878 INFO:tasks.workunit.client.0.smithi032.stdout:7/584: truncate d16/d20/d26/d70/f7f 1282983 0 2024-04-06T14:17:05.879 INFO:tasks.workunit.client.0.smithi032.stdout:7/585: chown d16/c2e 39412 1 2024-04-06T14:17:05.880 INFO:tasks.workunit.client.0.smithi032.stdout:6/539: rename d4/d5/d8/d1f to d4/d5/dbf 0 2024-04-06T14:17:05.880 INFO:tasks.workunit.client.0.smithi032.stdout:7/586: symlink d16/d20/d26/d5f/ld3 0 2024-04-06T14:17:05.881 INFO:tasks.workunit.client.0.smithi032.stdout:7/587: stat d16/d20/d54/f57 0 2024-04-06T14:17:05.885 INFO:tasks.workunit.client.0.smithi032.stdout:7/588: write d16/d20/f28 [60476,107376] 0 2024-04-06T14:17:05.886 INFO:tasks.workunit.client.0.smithi032.stdout:6/540: creat d4/d5/d8/d2f/d9b/db4/fc0 x:0 0 0 2024-04-06T14:17:05.887 INFO:tasks.workunit.client.0.smithi032.stdout:6/541: write d4/d5/d28/fbe [260770,173] 0 2024-04-06T14:17:05.888 INFO:tasks.workunit.client.0.smithi032.stdout:6/542: dread - d4/d5/d83/dad/fbb zero size 2024-04-06T14:17:05.889 INFO:tasks.workunit.client.0.smithi032.stdout:7/589: write d16/d20/d63/f72 [77477,103487] 0 2024-04-06T14:17:05.902 INFO:tasks.workunit.client.1.smithi086.stdout:6/310: creat d6/d9/db/f70 x:0 0 0 2024-04-06T14:17:05.903 INFO:tasks.workunit.client.0.smithi032.stdout:6/543: creat d4/d5/d14/fc1 x:0 0 0 2024-04-06T14:17:05.903 INFO:tasks.workunit.client.0.smithi032.stdout:7/590: write d16/f40 [3851654,109769] 0 2024-04-06T14:17:05.903 INFO:tasks.workunit.client.1.smithi086.stdout:2/314: mknod d0/d20/d27/d37/d52/c70 0 2024-04-06T14:17:05.905 INFO:tasks.workunit.client.0.smithi032.stdout:6/544: write d4/d5/dbf/f33 [2058040,64943] 0 2024-04-06T14:17:05.906 INFO:tasks.workunit.client.1.smithi086.stdout:9/298: rename d5/d18/c2c to d5/d13/d1a/c65 0 2024-04-06T14:17:05.907 INFO:tasks.workunit.client.1.smithi086.stdout:6/311: dread d6/d9/db/f2f [0,4194304] 0 2024-04-06T14:17:05.909 INFO:tasks.workunit.client.1.smithi086.stdout:3/360: truncate d5/d33/f32 1668262 0 2024-04-06T14:17:05.909 INFO:tasks.workunit.client.0.smithi032.stdout:1/610: sync 2024-04-06T14:17:05.909 INFO:tasks.workunit.client.0.smithi032.stdout:0/376: sync 2024-04-06T14:17:05.909 INFO:tasks.workunit.client.0.smithi032.stdout:2/527: sync 2024-04-06T14:17:05.910 INFO:tasks.workunit.client.1.smithi086.stdout:3/361: chown d5/d33/l39 57 1 2024-04-06T14:17:05.910 INFO:tasks.workunit.client.0.smithi032.stdout:6/545: link d4/d5/d19/d25/f87 d4/d5/d8/d2f/fc2 0 2024-04-06T14:17:05.910 INFO:tasks.workunit.client.0.smithi032.stdout:1/611: fdatasync d0/da/f6b 0 2024-04-06T14:17:05.910 INFO:tasks.workunit.client.0.smithi032.stdout:2/528: dread - d0/d5a/f9d zero size 2024-04-06T14:17:05.911 INFO:tasks.workunit.client.1.smithi086.stdout:4/242: sync 2024-04-06T14:17:05.912 INFO:tasks.workunit.client.0.smithi032.stdout:2/529: creat d0/d4e/d63/faf x:0 0 0 2024-04-06T14:17:05.914 INFO:tasks.workunit.client.0.smithi032.stdout:1/612: creat d0/d10/d16/d3f/d98/fc6 x:0 0 0 2024-04-06T14:17:05.917 INFO:tasks.workunit.client.0.smithi032.stdout:0/377: rename d3/da/d41/l6b to d3/da/d25/d2b/d4a/d79/d7c/l7e 0 2024-04-06T14:17:05.917 INFO:tasks.workunit.client.1.smithi086.stdout:9/299: unlink d5/d10/d14/d19/c31 0 2024-04-06T14:17:05.920 INFO:tasks.workunit.client.1.smithi086.stdout:4/243: rename d0/d2c/d30/d24/l41 to d0/d15/l47 0 2024-04-06T14:17:05.920 INFO:tasks.workunit.client.1.smithi086.stdout:1/302: write db/d13/f54 [1558701,106135] 0 2024-04-06T14:17:05.920 INFO:tasks.workunit.client.1.smithi086.stdout:2/315: link d0/c16 d0/df/d48/c71 0 2024-04-06T14:17:05.922 INFO:tasks.workunit.client.1.smithi086.stdout:6/312: creat d6/d13/d1a/d64/f71 x:0 0 0 2024-04-06T14:17:05.929 INFO:tasks.workunit.client.1.smithi086.stdout:1/303: creat db/d11/d1a/f63 x:0 0 0 2024-04-06T14:17:05.930 INFO:tasks.workunit.client.0.smithi032.stdout:0/378: dwrite d3/f10 [0,4194304] 0 2024-04-06T14:17:05.930 INFO:tasks.workunit.client.0.smithi032.stdout:1/613: dwrite d0/f90 [0,4194304] 0 2024-04-06T14:17:05.931 INFO:tasks.workunit.client.1.smithi086.stdout:6/313: creat d6/d13/d42/f72 x:0 0 0 2024-04-06T14:17:05.931 INFO:tasks.workunit.client.1.smithi086.stdout:6/314: fdatasync d6/f40 0 2024-04-06T14:17:05.931 INFO:tasks.workunit.client.1.smithi086.stdout:3/362: link d5/df/c23 d5/d33/d5d/c62 0 2024-04-06T14:17:05.934 INFO:tasks.workunit.client.0.smithi032.stdout:2/530: dwrite d0/d5a/f2c [0,4194304] 0 2024-04-06T14:17:05.935 INFO:tasks.workunit.client.1.smithi086.stdout:9/300: unlink d5/d18/f26 0 2024-04-06T14:17:05.935 INFO:tasks.workunit.client.1.smithi086.stdout:9/301: readlink d5/d18/l49 0 2024-04-06T14:17:05.940 INFO:tasks.workunit.client.0.smithi032.stdout:2/531: write d0/d30/f36 [4027314,28336] 0 2024-04-06T14:17:05.940 INFO:tasks.workunit.client.1.smithi086.stdout:1/304: rename db/f39 to db/d11/d42/d53/f64 0 2024-04-06T14:17:05.941 INFO:tasks.workunit.client.1.smithi086.stdout:1/305: readlink db/d11/d1f/l2d 0 2024-04-06T14:17:05.942 INFO:tasks.workunit.client.1.smithi086.stdout:1/306: chown db/d13/c3e 4626225 1 2024-04-06T14:17:05.945 INFO:tasks.workunit.client.0.smithi032.stdout:1/614: write d0/d9/f23 [1724337,128380] 0 2024-04-06T14:17:05.947 INFO:tasks.workunit.client.0.smithi032.stdout:2/532: rename d0/d4f to d0/d30/d39/d59/d67/d74/d78/d8c/db0 0 2024-04-06T14:17:05.948 INFO:tasks.workunit.client.1.smithi086.stdout:1/307: mknod db/d11/d1f/c65 0 2024-04-06T14:17:05.950 INFO:tasks.workunit.client.1.smithi086.stdout:3/363: creat d5/d36/d49/d58/f63 x:0 0 0 2024-04-06T14:17:05.950 INFO:tasks.workunit.client.0.smithi032.stdout:1/615: write d0/d10/d16/f6d [4438693,77268] 0 2024-04-06T14:17:05.950 INFO:tasks.workunit.client.0.smithi032.stdout:0/379: dwrite d3/f19 [0,4194304] 0 2024-04-06T14:17:05.952 INFO:tasks.workunit.client.0.smithi032.stdout:2/533: truncate d0/d5a/f1f 8943084 0 2024-04-06T14:17:05.953 INFO:tasks.workunit.client.1.smithi086.stdout:9/302: mknod d5/d10/d14/d19/c66 0 2024-04-06T14:17:05.953 INFO:tasks.workunit.client.1.smithi086.stdout:3/364: write d5/df/f45 [592999,82813] 0 2024-04-06T14:17:05.953 INFO:tasks.workunit.client.1.smithi086.stdout:9/303: write d5/d10/f53 [968263,102583] 0 2024-04-06T14:17:05.955 INFO:tasks.workunit.client.1.smithi086.stdout:9/304: write d5/d10/d14/f22 [992987,42143] 0 2024-04-06T14:17:05.955 INFO:tasks.workunit.client.1.smithi086.stdout:9/305: chown d5/d58 7933 1 2024-04-06T14:17:05.956 INFO:tasks.workunit.client.0.smithi032.stdout:0/380: symlink d3/da/d35/l7f 0 2024-04-06T14:17:05.957 INFO:tasks.workunit.client.1.smithi086.stdout:9/306: write d5/d13/f2d [41864,21513] 0 2024-04-06T14:17:05.957 INFO:tasks.workunit.client.1.smithi086.stdout:3/365: mkdir d5/d36/d49/d64 0 2024-04-06T14:17:05.958 INFO:tasks.workunit.client.0.smithi032.stdout:2/534: rename d0/ff to d0/d5a/fb1 0 2024-04-06T14:17:05.958 INFO:tasks.workunit.client.1.smithi086.stdout:3/366: fdatasync d5/f20 0 2024-04-06T14:17:05.958 INFO:tasks.workunit.client.1.smithi086.stdout:6/315: getdents d6/d13/d42/d52 0 2024-04-06T14:17:05.959 INFO:tasks.workunit.client.0.smithi032.stdout:2/535: fsync d0/d5a/f2c 0 2024-04-06T14:17:05.959 INFO:tasks.workunit.client.1.smithi086.stdout:6/316: write d6/d13/d1a/d22/f6e [956278,110388] 0 2024-04-06T14:17:05.963 INFO:tasks.workunit.client.0.smithi032.stdout:1/616: rename d0/d10/d16/d3f/d98/cc1 to d0/d9/d1f/d7c/d8f/cc7 0 2024-04-06T14:17:05.964 INFO:tasks.workunit.client.0.smithi032.stdout:2/536: rmdir d0/d30/d39/d59/d67/d74/d78/d65 39 2024-04-06T14:17:05.966 INFO:tasks.workunit.client.0.smithi032.stdout:0/381: mkdir d3/da/d25/d2b/d80 0 2024-04-06T14:17:05.966 INFO:tasks.workunit.client.1.smithi086.stdout:3/367: mkdir d5/d36/d65 0 2024-04-06T14:17:05.966 INFO:tasks.workunit.client.0.smithi032.stdout:2/537: write d0/d5a/f25 [983837,29241] 0 2024-04-06T14:17:05.968 INFO:tasks.workunit.client.0.smithi032.stdout:0/382: symlink d3/da/d25/d36/l81 0 2024-04-06T14:17:05.968 INFO:tasks.workunit.client.1.smithi086.stdout:6/317: mkdir d6/d13/d5f/d60/d45/d73 0 2024-04-06T14:17:05.970 INFO:tasks.workunit.client.1.smithi086.stdout:9/307: unlink d5/d13/f15 0 2024-04-06T14:17:05.971 INFO:tasks.workunit.client.1.smithi086.stdout:1/308: dread db/d11/d1f/f2b [0,4194304] 0 2024-04-06T14:17:05.972 INFO:tasks.workunit.client.1.smithi086.stdout:9/308: write d5/d13/f4f [276369,58100] 0 2024-04-06T14:17:05.973 INFO:tasks.workunit.client.0.smithi032.stdout:0/383: mkdir d3/da/d41/d82 0 2024-04-06T14:17:05.976 INFO:tasks.workunit.client.0.smithi032.stdout:0/384: mkdir d3/d83 0 2024-04-06T14:17:05.977 INFO:tasks.workunit.client.1.smithi086.stdout:9/309: chown c1 153028 1 2024-04-06T14:17:05.978 INFO:tasks.workunit.client.1.smithi086.stdout:9/310: chown d5/d13/d1a/l3a 278 1 2024-04-06T14:17:05.978 INFO:tasks.workunit.client.0.smithi032.stdout:2/538: dwrite d0/d5a/f4c [0,4194304] 0 2024-04-06T14:17:05.979 INFO:tasks.workunit.client.1.smithi086.stdout:9/311: write d5/d18/f32 [1021581,4219] 0 2024-04-06T14:17:05.979 INFO:tasks.workunit.client.1.smithi086.stdout:4/244: creat d0/d15/f48 x:0 0 0 2024-04-06T14:17:05.979 INFO:tasks.workunit.client.0.smithi032.stdout:1/617: dread d0/d9/d1f/f91 [0,4194304] 0 2024-04-06T14:17:05.980 INFO:tasks.workunit.client.0.smithi032.stdout:0/385: write d3/da/f26 [2235315,38617] 0 2024-04-06T14:17:05.982 INFO:tasks.workunit.client.1.smithi086.stdout:6/318: creat d6/d9/d43/f74 x:0 0 0 2024-04-06T14:17:05.982 INFO:tasks.workunit.client.1.smithi086.stdout:3/368: link d5/d33/l39 d5/d42/l66 0 2024-04-06T14:17:05.982 INFO:tasks.workunit.client.0.smithi032.stdout:1/618: write d0/d41/d59/f60 [2201146,112084] 0 2024-04-06T14:17:05.984 INFO:tasks.workunit.client.0.smithi032.stdout:1/619: chown d0/d9/d1f/c4f 46 1 2024-04-06T14:17:05.986 INFO:tasks.workunit.client.0.smithi032.stdout:1/620: chown d0/da/d3e/fad 4 1 2024-04-06T14:17:05.986 INFO:tasks.workunit.client.0.smithi032.stdout:1/621: chown d0/lae 2 1 2024-04-06T14:17:05.987 INFO:tasks.workunit.client.1.smithi086.stdout:9/312: creat d5/d10/d14/d19/d25/f67 x:0 0 0 2024-04-06T14:17:05.987 INFO:tasks.workunit.client.0.smithi032.stdout:0/386: unlink d3/da/d41/l5c 0 2024-04-06T14:17:05.987 INFO:tasks.workunit.client.0.smithi032.stdout:0/387: chown d3/fd 822 1 2024-04-06T14:17:05.988 INFO:tasks.workunit.client.1.smithi086.stdout:9/313: chown d5/d10/d14/d19/d1b 6697 1 2024-04-06T14:17:05.988 INFO:tasks.workunit.client.1.smithi086.stdout:4/245: symlink d0/d15/l49 0 2024-04-06T14:17:05.989 INFO:tasks.workunit.client.1.smithi086.stdout:3/369: creat d5/d36/f67 x:0 0 0 2024-04-06T14:17:05.990 INFO:tasks.workunit.client.0.smithi032.stdout:0/388: symlink d3/da/d25/d36/l84 0 2024-04-06T14:17:05.993 INFO:tasks.workunit.client.1.smithi086.stdout:4/246: rmdir d0/de 39 2024-04-06T14:17:05.995 INFO:tasks.workunit.client.0.smithi032.stdout:2/539: dwrite d0/d5a/d3a/f42 [0,4194304] 0 2024-04-06T14:17:05.995 INFO:tasks.workunit.client.1.smithi086.stdout:1/309: dread db/d13/f1e [0,4194304] 0 2024-04-06T14:17:05.997 INFO:tasks.workunit.client.0.smithi032.stdout:0/389: symlink d3/da/d25/d2b/d80/l85 0 2024-04-06T14:17:05.997 INFO:tasks.workunit.client.1.smithi086.stdout:6/319: dwrite d6/d20/f46 [0,4194304] 0 2024-04-06T14:17:05.998 INFO:tasks.workunit.client.1.smithi086.stdout:4/247: chown d0/d15/f3c 207 1 2024-04-06T14:17:06.000 INFO:tasks.workunit.client.0.smithi032.stdout:1/622: write d0/d41/f5c [1194081,9631] 0 2024-04-06T14:17:06.000 INFO:tasks.workunit.client.0.smithi032.stdout:0/390: mkdir d3/da/d25/d2b/d4a/d86 0 2024-04-06T14:17:06.001 INFO:tasks.workunit.client.1.smithi086.stdout:3/370: dwrite d5/d36/f67 [0,4194304] 0 2024-04-06T14:17:06.002 INFO:tasks.workunit.client.0.smithi032.stdout:2/540: getdents d0/d30/d39/d59/d67/d74/d78/d57 0 2024-04-06T14:17:06.002 INFO:tasks.workunit.client.1.smithi086.stdout:1/310: creat db/d11/d1f/f66 x:0 0 0 2024-04-06T14:17:06.003 INFO:tasks.workunit.client.0.smithi032.stdout:1/623: chown d0/d9/d1f/d72/l8b 601909 1 2024-04-06T14:17:06.005 INFO:tasks.workunit.client.0.smithi032.stdout:1/624: chown d0/d41/d59/f9a 665077926 1 2024-04-06T14:17:06.006 INFO:tasks.workunit.client.0.smithi032.stdout:2/541: mkdir d0/db2 0 2024-04-06T14:17:06.007 INFO:tasks.workunit.client.1.smithi086.stdout:6/320: rmdir d6/d9/d63 39 2024-04-06T14:17:06.008 INFO:tasks.workunit.client.1.smithi086.stdout:3/371: chown d5/f3e 1489247 1 2024-04-06T14:17:06.009 INFO:tasks.workunit.client.0.smithi032.stdout:1/625: mknod d0/d10/d78/cc8 0 2024-04-06T14:17:06.010 INFO:tasks.workunit.client.0.smithi032.stdout:1/626: dread - d0/d10/f8d zero size 2024-04-06T14:17:06.014 INFO:tasks.workunit.client.0.smithi032.stdout:1/627: mkdir d0/dc9 0 2024-04-06T14:17:06.015 INFO:tasks.workunit.client.0.smithi032.stdout:0/391: truncate d3/da/d25/f2a 1902405 0 2024-04-06T14:17:06.015 INFO:tasks.workunit.client.1.smithi086.stdout:3/372: creat d5/d36/d5c/f68 x:0 0 0 2024-04-06T14:17:06.016 INFO:tasks.workunit.client.0.smithi032.stdout:1/628: stat d0/da/d3e/f6a 0 2024-04-06T14:17:06.016 INFO:tasks.workunit.client.1.smithi086.stdout:7/275: sync 2024-04-06T14:17:06.017 INFO:tasks.workunit.client.1.smithi086.stdout:5/333: sync 2024-04-06T14:17:06.017 INFO:tasks.workunit.client.1.smithi086.stdout:6/321: getdents d6/d13/d42/d52 0 2024-04-06T14:17:06.018 INFO:tasks.workunit.client.1.smithi086.stdout:6/322: chown d6 1568 1 2024-04-06T14:17:06.018 INFO:tasks.workunit.client.0.smithi032.stdout:0/392: symlink d3/da/d41/l87 0 2024-04-06T14:17:06.019 INFO:tasks.workunit.client.0.smithi032.stdout:0/393: write d3/f19 [5107813,115628] 0 2024-04-06T14:17:06.023 INFO:tasks.workunit.client.0.smithi032.stdout:0/394: rename d3/c7 to d3/da/d25/d36/c88 0 2024-04-06T14:17:06.023 INFO:tasks.workunit.client.1.smithi086.stdout:6/323: fdatasync d6/d20/f51 0 2024-04-06T14:17:06.024 INFO:tasks.workunit.client.1.smithi086.stdout:3/373: mkdir d5/d26/d4f/d69 0 2024-04-06T14:17:06.028 INFO:tasks.workunit.client.1.smithi086.stdout:3/374: stat d5/d36/d5c 0 2024-04-06T14:17:06.029 INFO:tasks.workunit.client.1.smithi086.stdout:9/314: dwrite d5/d13/d1a/f63 [0,4194304] 0 2024-04-06T14:17:06.032 INFO:tasks.workunit.client.1.smithi086.stdout:9/315: read f2 [2600482,86301] 0 2024-04-06T14:17:06.033 INFO:tasks.workunit.client.1.smithi086.stdout:1/311: dwrite db/d11/f12 [0,4194304] 0 2024-04-06T14:17:06.034 INFO:tasks.workunit.client.1.smithi086.stdout:4/248: dwrite d0/d2c/d30/d24/f2d [0,4194304] 0 2024-04-06T14:17:06.035 INFO:tasks.workunit.client.0.smithi032.stdout:2/542: dread d0/f1b [0,4194304] 0 2024-04-06T14:17:06.037 INFO:tasks.workunit.client.1.smithi086.stdout:6/324: truncate d6/d9/f17 228656 0 2024-04-06T14:17:06.041 INFO:tasks.workunit.client.1.smithi086.stdout:6/325: stat d6/d13/d1a/c33 0 2024-04-06T14:17:06.043 INFO:tasks.workunit.client.1.smithi086.stdout:3/375: link d5/df/f1c d5/df/f6a 0 2024-04-06T14:17:06.043 INFO:tasks.workunit.client.1.smithi086.stdout:8/417: sync 2024-04-06T14:17:06.043 INFO:tasks.workunit.client.1.smithi086.stdout:0/336: sync 2024-04-06T14:17:06.044 INFO:tasks.workunit.client.0.smithi032.stdout:2/543: rename d0/d5a/f1f to d0/d30/d39/d59/d67/d74/d78/d8c/db0/fb3 0 2024-04-06T14:17:06.044 INFO:tasks.workunit.client.1.smithi086.stdout:7/276: dwrite f5 [0,4194304] 0 2024-04-06T14:17:06.051 INFO:tasks.workunit.client.0.smithi032.stdout:2/544: truncate d0/fb 4896204 0 2024-04-06T14:17:06.052 INFO:tasks.workunit.client.1.smithi086.stdout:4/249: mkdir d0/d15/d4a 0 2024-04-06T14:17:06.052 INFO:tasks.workunit.client.1.smithi086.stdout:6/326: chown d6/d9/db/c14 33268 1 2024-04-06T14:17:06.052 INFO:tasks.workunit.client.1.smithi086.stdout:5/334: dwrite d0/d16/d18/f1a [4194304,4194304] 0 2024-04-06T14:17:06.052 INFO:tasks.workunit.client.1.smithi086.stdout:7/277: chown d7/d10 225926 1 2024-04-06T14:17:06.052 INFO:tasks.workunit.client.1.smithi086.stdout:9/316: dread d5/d18/f48 [0,4194304] 0 2024-04-06T14:17:06.053 INFO:tasks.workunit.client.1.smithi086.stdout:1/312: mknod db/d11/d1a/d2a/d52/c67 0 2024-04-06T14:17:06.053 INFO:tasks.workunit.client.1.smithi086.stdout:3/376: dread - d5/d36/d5c/f68 zero size 2024-04-06T14:17:06.053 INFO:tasks.workunit.client.1.smithi086.stdout:0/337: dread - d0/d15/d26/d33/f66 zero size 2024-04-06T14:17:06.053 INFO:tasks.workunit.client.0.smithi032.stdout:2/545: mknod d0/d5a/d3a/da2/cb4 0 2024-04-06T14:17:06.054 INFO:tasks.workunit.client.1.smithi086.stdout:1/313: dread - db/d11/d1d/f5f zero size 2024-04-06T14:17:06.054 INFO:tasks.workunit.client.1.smithi086.stdout:0/338: readlink d0/d1/d4/d7/l36 0 2024-04-06T14:17:06.056 INFO:tasks.workunit.client.1.smithi086.stdout:0/339: chown d0/d1/d4/de 0 1 2024-04-06T14:17:06.057 INFO:tasks.workunit.client.1.smithi086.stdout:6/327: unlink d6/d13/d1a/f1b 0 2024-04-06T14:17:06.058 INFO:tasks.workunit.client.1.smithi086.stdout:6/328: fdatasync d6/d13/f5d 0 2024-04-06T14:17:06.060 INFO:tasks.workunit.client.0.smithi032.stdout:2/546: mknod d0/d5a/d3a/da2/cb5 0 2024-04-06T14:17:06.061 INFO:tasks.workunit.client.1.smithi086.stdout:6/329: write d6/d9/db/f19 [3787113,28539] 0 2024-04-06T14:17:06.061 INFO:tasks.workunit.client.1.smithi086.stdout:9/317: unlink d5/d13/c29 0 2024-04-06T14:17:06.062 INFO:tasks.workunit.client.1.smithi086.stdout:9/318: chown d5/d13/d47/d5e 0 1 2024-04-06T14:17:06.063 INFO:tasks.workunit.client.1.smithi086.stdout:6/330: fsync d6/d13/d42/f72 0 2024-04-06T14:17:06.063 INFO:tasks.workunit.client.1.smithi086.stdout:6/331: readlink d6/d9/db/l54 0 2024-04-06T14:17:06.067 INFO:tasks.workunit.client.0.smithi032.stdout:0/395: dread d3/da/d25/f54 [0,4194304] 0 2024-04-06T14:17:06.067 INFO:tasks.workunit.client.1.smithi086.stdout:7/278: mknod d7/d10/d11/d1c/d36/c61 0 2024-04-06T14:17:06.069 INFO:tasks.workunit.client.1.smithi086.stdout:1/314: mkdir db/d11/d1a/d2a/d52/d5b/d68 0 2024-04-06T14:17:06.069 INFO:tasks.workunit.client.1.smithi086.stdout:9/319: mknod d5/d10/c68 0 2024-04-06T14:17:06.069 INFO:tasks.workunit.client.1.smithi086.stdout:5/335: dread d0/fa [0,4194304] 0 2024-04-06T14:17:06.071 INFO:tasks.workunit.client.1.smithi086.stdout:5/336: chown d0/c30 10057529 1 2024-04-06T14:17:06.071 INFO:tasks.workunit.client.1.smithi086.stdout:0/340: dread d0/f16 [0,4194304] 0 2024-04-06T14:17:06.072 INFO:tasks.workunit.client.1.smithi086.stdout:6/332: creat d6/d13/f75 x:0 0 0 2024-04-06T14:17:06.074 INFO:tasks.workunit.client.0.smithi032.stdout:4/600: sync 2024-04-06T14:17:06.075 INFO:tasks.workunit.client.0.smithi032.stdout:9/575: sync 2024-04-06T14:17:06.076 INFO:tasks.workunit.client.0.smithi032.stdout:2/547: getdents d0/d5a/d3a/da2 0 2024-04-06T14:17:06.077 INFO:tasks.workunit.client.1.smithi086.stdout:3/377: write d5/d42/f60 [6024326,118322] 0 2024-04-06T14:17:06.077 INFO:tasks.workunit.client.1.smithi086.stdout:7/279: creat d7/d10/d11/d1c/d24/d4d/f62 x:0 0 0 2024-04-06T14:17:06.078 INFO:tasks.workunit.client.0.smithi032.stdout:0/396: rename d3/da/d41/c5b to d3/da/d25/d2b/c89 0 2024-04-06T14:17:06.078 INFO:tasks.workunit.client.0.smithi032.stdout:2/548: fsync d0/d30/d39/d59/d67/d74/d78/d8c/f9e 0 2024-04-06T14:17:06.079 INFO:tasks.workunit.client.1.smithi086.stdout:9/320: mknod d5/d13/d1a/d5f/c69 0 2024-04-06T14:17:06.080 INFO:tasks.workunit.client.0.smithi032.stdout:2/549: write d0/f13 [7819911,99141] 0 2024-04-06T14:17:06.081 INFO:tasks.workunit.client.1.smithi086.stdout:0/341: symlink d0/d1/d1b/d46/d52/l6c 0 2024-04-06T14:17:06.082 INFO:tasks.workunit.client.0.smithi032.stdout:9/576: creat d2/d15/d1e/d65/d9c/fbd x:0 0 0 2024-04-06T14:17:06.082 INFO:tasks.workunit.client.0.smithi032.stdout:4/601: creat d0/df/d23/d31/d50/da0/fd1 x:0 0 0 2024-04-06T14:17:06.083 INFO:tasks.workunit.client.1.smithi086.stdout:1/315: dread db/d11/f12 [0,4194304] 0 2024-04-06T14:17:06.083 INFO:tasks.workunit.client.0.smithi032.stdout:0/397: creat d3/da/d35/f8a x:0 0 0 2024-04-06T14:17:06.083 INFO:tasks.workunit.client.0.smithi032.stdout:9/577: stat d2/d15/d1e/d41/d6c 0 2024-04-06T14:17:06.084 INFO:tasks.workunit.client.0.smithi032.stdout:2/550: rename d0/d5a/d3a/f5b to d0/d5a/d6a/fb6 0 2024-04-06T14:17:06.086 INFO:tasks.workunit.client.0.smithi032.stdout:0/398: mkdir d3/da/d35/d8b 0 2024-04-06T14:17:06.087 INFO:tasks.workunit.client.1.smithi086.stdout:9/321: write d5/d13/d1a/f5a [621921,66306] 0 2024-04-06T14:17:06.087 INFO:tasks.workunit.client.1.smithi086.stdout:7/280: creat d7/d10/d11/d1c/d24/d2c/f63 x:0 0 0 2024-04-06T14:17:06.089 INFO:tasks.workunit.client.1.smithi086.stdout:3/378: truncate d5/d36/d5c/f5e 47671 0 2024-04-06T14:17:06.089 INFO:tasks.workunit.client.0.smithi032.stdout:2/551: unlink d0/d5a/f25 0 2024-04-06T14:17:06.089 INFO:tasks.workunit.client.0.smithi032.stdout:2/552: chown d0/f1b 2754158 1 2024-04-06T14:17:06.090 INFO:tasks.workunit.client.1.smithi086.stdout:0/342: write d0/d1/d4/d7/f27 [1743290,730] 0 2024-04-06T14:17:06.093 INFO:tasks.workunit.client.0.smithi032.stdout:2/553: chown d0/d30/d39/d59/d67/d74/d78/d57/d7c/c83 12573063 1 2024-04-06T14:17:06.094 INFO:tasks.workunit.client.1.smithi086.stdout:5/337: dwrite d0/d5e/f60 [0,4194304] 0 2024-04-06T14:17:06.094 INFO:tasks.workunit.client.0.smithi032.stdout:9/578: rename d2/d15/d55/d59/d93/d95/db8/d81/d4a/l60 to d2/d15/d55/d59/d93/d95/db8/d81/d77/lbe 0 2024-04-06T14:17:06.094 INFO:tasks.workunit.client.1.smithi086.stdout:6/333: dwrite d6/d13/d5f/d60/f5b [0,4194304] 0 2024-04-06T14:17:06.096 INFO:tasks.workunit.client.1.smithi086.stdout:3/379: creat d5/df/f6b x:0 0 0 2024-04-06T14:17:06.097 INFO:tasks.workunit.client.1.smithi086.stdout:6/334: dread - d6/d13/d42/f72 zero size 2024-04-06T14:17:06.098 INFO:tasks.workunit.client.0.smithi032.stdout:0/399: dwrite d3/da/d25/d2b/f32 [4194304,4194304] 0 2024-04-06T14:17:06.099 INFO:tasks.workunit.client.0.smithi032.stdout:4/602: link d0/df/d1f/d42/c78 d0/df/d1f/d5a/cd2 0 2024-04-06T14:17:06.099 INFO:tasks.workunit.client.1.smithi086.stdout:5/338: chown d0/d16/d65 0 1 2024-04-06T14:17:06.103 INFO:tasks.workunit.client.1.smithi086.stdout:5/339: chown d0/d19/f4b 5158 1 2024-04-06T14:17:06.104 INFO:tasks.workunit.client.1.smithi086.stdout:7/281: creat d7/d10/f64 x:0 0 0 2024-04-06T14:17:06.104 INFO:tasks.workunit.client.1.smithi086.stdout:9/322: truncate d5/d10/d14/d19/d25/f3e 987435 0 2024-04-06T14:17:06.106 INFO:tasks.workunit.client.1.smithi086.stdout:3/380: symlink d5/d36/d65/l6c 0 2024-04-06T14:17:06.106 INFO:tasks.workunit.client.1.smithi086.stdout:7/282: dread - d7/d10/d11/d1c/f60 zero size 2024-04-06T14:17:06.107 INFO:tasks.workunit.client.0.smithi032.stdout:0/400: getdents d3/da/d25/d2b/d4a/d86 0 2024-04-06T14:17:06.108 INFO:tasks.workunit.client.1.smithi086.stdout:0/343: dwrite d0/d15/f6a [0,4194304] 0 2024-04-06T14:17:06.110 INFO:tasks.workunit.client.0.smithi032.stdout:9/579: dwrite d2/d15/d1e/d65/d83/d7e/f64 [0,4194304] 0 2024-04-06T14:17:06.115 INFO:tasks.workunit.client.0.smithi032.stdout:0/401: mknod d3/da/d25/d2b/d4a/c8c 0 2024-04-06T14:17:06.115 INFO:tasks.workunit.client.0.smithi032.stdout:9/580: creat d2/d15/d1e/d65/d6b/d7f/fbf x:0 0 0 2024-04-06T14:17:06.117 INFO:tasks.workunit.client.0.smithi032.stdout:9/581: readlink d2/d15/d55/d59/d93/d95/db8/d81/d77/l99 0 2024-04-06T14:17:06.118 INFO:tasks.workunit.client.1.smithi086.stdout:1/316: dread db/fc [0,4194304] 0 2024-04-06T14:17:06.118 INFO:tasks.workunit.client.1.smithi086.stdout:9/323: dwrite d5/d13/d1a/f24 [0,4194304] 0 2024-04-06T14:17:06.119 INFO:tasks.workunit.client.1.smithi086.stdout:1/317: chown f2 2064154144 1 2024-04-06T14:17:06.120 INFO:tasks.workunit.client.1.smithi086.stdout:6/335: fdatasync d6/d9/db/f16 0 2024-04-06T14:17:06.122 INFO:tasks.workunit.client.1.smithi086.stdout:7/283: chown d7/d10/d11/c1a 3 1 2024-04-06T14:17:06.122 INFO:tasks.workunit.client.1.smithi086.stdout:1/318: write db/fe [3776673,97005] 0 2024-04-06T14:17:06.124 INFO:tasks.workunit.client.1.smithi086.stdout:3/381: mkdir d5/d36/d5a/d6d 0 2024-04-06T14:17:06.124 INFO:tasks.workunit.client.1.smithi086.stdout:0/344: creat d0/d1/d1b/d46/f6d x:0 0 0 2024-04-06T14:17:06.125 INFO:tasks.workunit.client.0.smithi032.stdout:2/554: truncate d0/f22 624804 0 2024-04-06T14:17:06.127 INFO:tasks.workunit.client.1.smithi086.stdout:6/336: readlink d6/d9/db/l54 0 2024-04-06T14:17:06.130 INFO:tasks.workunit.client.0.smithi032.stdout:2/555: chown d0/d30/d39/d59/d67/d74/d78/l71 5 1 2024-04-06T14:17:06.130 INFO:tasks.workunit.client.1.smithi086.stdout:6/337: write d6/d13/d1a/d22/f35 [4375555,73492] 0 2024-04-06T14:17:06.131 INFO:tasks.workunit.client.0.smithi032.stdout:2/556: dread - d0/d30/d39/faa zero size 2024-04-06T14:17:06.131 INFO:tasks.workunit.client.1.smithi086.stdout:9/324: dwrite d5/d10/d14/d19/d25/f67 [0,4194304] 0 2024-04-06T14:17:06.132 INFO:tasks.workunit.client.0.smithi032.stdout:9/582: dwrite d2/d15/d55/d5f/f62 [0,4194304] 0 2024-04-06T14:17:06.133 INFO:tasks.workunit.client.1.smithi086.stdout:3/382: mknod d5/d36/d65/c6e 0 2024-04-06T14:17:06.136 INFO:tasks.workunit.client.1.smithi086.stdout:7/284: rename d7/d10/c57 to d7/d10/d11/d1c/d24/d4d/d5c/c65 0 2024-04-06T14:17:06.136 INFO:tasks.workunit.client.1.smithi086.stdout:1/319: dwrite db/d11/d1f/f66 [0,4194304] 0 2024-04-06T14:17:06.137 INFO:tasks.workunit.client.1.smithi086.stdout:6/338: fdatasync f5 0 2024-04-06T14:17:06.137 INFO:tasks.workunit.client.0.smithi032.stdout:0/402: dread d3/da/d41/f42 [0,4194304] 0 2024-04-06T14:17:06.140 INFO:tasks.workunit.client.0.smithi032.stdout:0/403: chown d3/da/d25/d2b/f40 123784565 1 2024-04-06T14:17:06.141 INFO:tasks.workunit.client.1.smithi086.stdout:6/339: write d6/d13/d1a/d22/f6e [1403874,121597] 0 2024-04-06T14:17:06.143 INFO:tasks.workunit.client.1.smithi086.stdout:1/320: write db/d11/d1a/d2a/f3b [634659,31242] 0 2024-04-06T14:17:06.144 INFO:tasks.workunit.client.1.smithi086.stdout:3/383: unlink d5/d36/d56/c5b 0 2024-04-06T14:17:06.147 INFO:tasks.workunit.client.1.smithi086.stdout:6/340: creat d6/d9/db/f76 x:0 0 0 2024-04-06T14:17:06.148 INFO:tasks.workunit.client.0.smithi032.stdout:2/557: dread d0/d5a/d3a/f48 [0,4194304] 0 2024-04-06T14:17:06.151 INFO:tasks.workunit.client.0.smithi032.stdout:9/583: getdents d2/d15/d55/d59/d93/d95/db8/d81 0 2024-04-06T14:17:06.152 INFO:tasks.workunit.client.1.smithi086.stdout:1/321: mkdir db/d11/d1a/d2a/d52/d69 0 2024-04-06T14:17:06.153 INFO:tasks.workunit.client.1.smithi086.stdout:9/325: dwrite d5/d10/d14/d19/d25/f3e [0,4194304] 0 2024-04-06T14:17:06.154 INFO:tasks.workunit.client.0.smithi032.stdout:9/584: symlink d2/d15/d1e/d65/d83/d31/d39/lc0 0 2024-04-06T14:17:06.154 INFO:tasks.workunit.client.1.smithi086.stdout:3/384: creat d5/d33/d5d/f6f x:0 0 0 2024-04-06T14:17:06.156 INFO:tasks.workunit.client.1.smithi086.stdout:7/285: rename d7/d10/d11/d1c/d24/d2c/d3c/d52 to d7/d10/d11/d1c/d24/d4d/d5c/d66 0 2024-04-06T14:17:06.156 INFO:tasks.workunit.client.1.smithi086.stdout:9/326: stat d5/d10/d14/f4e 0 2024-04-06T14:17:06.158 INFO:tasks.workunit.client.1.smithi086.stdout:2/316: sync 2024-04-06T14:17:06.159 INFO:tasks.workunit.client.1.smithi086.stdout:1/322: write db/d11/d1d/f58 [775668,65375] 0 2024-04-06T14:17:06.160 INFO:tasks.workunit.client.1.smithi086.stdout:3/385: write d5/d36/d5c/f5e [775660,8808] 0 2024-04-06T14:17:06.160 INFO:tasks.workunit.client.0.smithi032.stdout:9/585: mknod d2/d15/d1e/d65/d83/d7e/da7/cc1 0 2024-04-06T14:17:06.164 INFO:tasks.workunit.client.1.smithi086.stdout:6/341: dwrite d6/d9/d5c/f58 [0,4194304] 0 2024-04-06T14:17:06.164 INFO:tasks.workunit.client.1.smithi086.stdout:1/323: write db/fe [5246365,68042] 0 2024-04-06T14:17:06.164 INFO:tasks.workunit.client.1.smithi086.stdout:3/386: stat d5/c1f 0 2024-04-06T14:17:06.166 INFO:tasks.workunit.client.1.smithi086.stdout:0/345: write d0/d1/d1b/f39 [4903311,83817] 0 2024-04-06T14:17:06.168 INFO:tasks.workunit.client.1.smithi086.stdout:1/324: write f2 [6125830,33701] 0 2024-04-06T14:17:06.171 INFO:tasks.workunit.client.1.smithi086.stdout:0/346: mknod d0/d1/d1b/c6e 0 2024-04-06T14:17:06.173 INFO:tasks.workunit.client.0.smithi032.stdout:0/404: truncate d3/f63 2435778 0 2024-04-06T14:17:06.175 INFO:tasks.workunit.client.1.smithi086.stdout:7/286: getdents d7/d10/d30 0 2024-04-06T14:17:06.177 INFO:tasks.workunit.client.1.smithi086.stdout:7/287: stat d7/d10/d30/f46 0 2024-04-06T14:17:06.180 INFO:tasks.workunit.client.1.smithi086.stdout:9/327: dread f2 [0,4194304] 0 2024-04-06T14:17:06.181 INFO:tasks.workunit.client.1.smithi086.stdout:0/347: mkdir d0/d1/d2/d6f 0 2024-04-06T14:17:06.182 INFO:tasks.workunit.client.0.smithi032.stdout:9/586: write d2/d15/d1e/d65/d83/d31/f3a [637911,114147] 0 2024-04-06T14:17:06.183 INFO:tasks.workunit.client.1.smithi086.stdout:1/325: dread db/d13/f23 [0,4194304] 0 2024-04-06T14:17:06.184 INFO:tasks.workunit.client.1.smithi086.stdout:7/288: creat d7/d10/d11/d1c/d24/d4d/d5c/d66/f67 x:0 0 0 2024-04-06T14:17:06.185 INFO:tasks.workunit.client.0.smithi032.stdout:9/587: stat d2/d15/d1e/d65/d83/d7e/c16 0 2024-04-06T14:17:06.186 INFO:tasks.workunit.client.1.smithi086.stdout:1/326: read - db/d11/d1d/f5f zero size 2024-04-06T14:17:06.186 INFO:tasks.workunit.client.1.smithi086.stdout:6/342: dwrite d6/d13/d1a/f21 [0,4194304] 0 2024-04-06T14:17:06.187 INFO:tasks.workunit.client.0.smithi032.stdout:2/558: dread d0/d6e/f72 [0,4194304] 0 2024-04-06T14:17:06.187 INFO:tasks.workunit.client.1.smithi086.stdout:0/348: creat d0/d1/d25/f70 x:0 0 0 2024-04-06T14:17:06.188 INFO:tasks.workunit.client.1.smithi086.stdout:9/328: dread d5/d13/d1a/f63 [0,4194304] 0 2024-04-06T14:17:06.188 INFO:tasks.workunit.client.1.smithi086.stdout:6/343: write d6/d20/f46 [5240805,101526] 0 2024-04-06T14:17:06.190 INFO:tasks.workunit.client.1.smithi086.stdout:9/329: chown d5/d13/d1a/l38 70105458 1 2024-04-06T14:17:06.192 INFO:tasks.workunit.client.1.smithi086.stdout:1/327: creat db/d13/d55/f6a x:0 0 0 2024-04-06T14:17:06.193 INFO:tasks.workunit.client.1.smithi086.stdout:7/289: dwrite d7/d10/d11/d1c/f60 [0,4194304] 0 2024-04-06T14:17:06.195 INFO:tasks.workunit.client.1.smithi086.stdout:7/290: write d7/d1d/f3e [1021155,124608] 0 2024-04-06T14:17:06.195 INFO:tasks.workunit.client.1.smithi086.stdout:0/349: symlink d0/l71 0 2024-04-06T14:17:06.199 INFO:tasks.workunit.client.1.smithi086.stdout:9/330: creat d5/d13/d1a/d5f/f6a x:0 0 0 2024-04-06T14:17:06.200 INFO:tasks.workunit.client.1.smithi086.stdout:7/291: write d7/d10/d30/f35 [2106923,130965] 0 2024-04-06T14:17:06.200 INFO:tasks.workunit.client.1.smithi086.stdout:0/350: write d0/d15/f6a [3249222,82475] 0 2024-04-06T14:17:06.201 INFO:tasks.workunit.client.1.smithi086.stdout:6/344: getdents d6/d13/d5f 0 2024-04-06T14:17:06.203 INFO:tasks.workunit.client.1.smithi086.stdout:9/331: dread - d5/d10/d14/d19/d25/d59/f60 zero size 2024-04-06T14:17:06.204 INFO:tasks.workunit.client.1.smithi086.stdout:1/328: dwrite db/d11/d1d/f58 [0,4194304] 0 2024-04-06T14:17:06.205 INFO:tasks.workunit.client.1.smithi086.stdout:7/292: unlink d7/d1d/f19 0 2024-04-06T14:17:06.205 INFO:tasks.workunit.client.1.smithi086.stdout:2/317: dwrite d0/d20/f2e [0,4194304] 0 2024-04-06T14:17:06.206 INFO:tasks.workunit.client.1.smithi086.stdout:0/351: mknod d0/d1/d1b/d46/d52/c72 0 2024-04-06T14:17:06.209 INFO:tasks.workunit.client.1.smithi086.stdout:1/329: mkdir db/d11/d1a/d2a/d52/d6b 0 2024-04-06T14:17:06.209 INFO:tasks.workunit.client.0.smithi032.stdout:9/588: fsync d2/d15/d1e/d65/d83/d31/d39/f5a 0 2024-04-06T14:17:06.213 INFO:tasks.workunit.client.0.smithi032.stdout:2/559: mkdir d0/d6e/dae/db7 0 2024-04-06T14:17:06.216 INFO:tasks.workunit.client.0.smithi032.stdout:2/560: stat d0/d5a/d3a/c76 0 2024-04-06T14:17:06.217 INFO:tasks.workunit.client.0.smithi032.stdout:9/589: read d2/d15/d1e/d65/d83/d7e/f37 [3358418,26275] 0 2024-04-06T14:17:06.218 INFO:tasks.workunit.client.1.smithi086.stdout:9/332: write d5/d18/f3f [3573168,97782] 0 2024-04-06T14:17:06.218 INFO:tasks.workunit.client.1.smithi086.stdout:0/352: mkdir d0/d1/d1b/d73 0 2024-04-06T14:17:06.218 INFO:tasks.workunit.client.1.smithi086.stdout:7/293: symlink d7/d10/d11/d1c/d24/d4d/l68 0 2024-04-06T14:17:06.218 INFO:tasks.workunit.client.1.smithi086.stdout:6/345: link d6/f40 d6/d13/f77 0 2024-04-06T14:17:06.218 INFO:tasks.workunit.client.1.smithi086.stdout:1/330: mknod db/d11/d1f/d5c/c6c 0 2024-04-06T14:17:06.222 INFO:tasks.workunit.client.0.smithi032.stdout:2/561: mknod d0/d30/d7f/cb8 0 2024-04-06T14:17:06.228 INFO:tasks.workunit.client.0.smithi032.stdout:9/590: rename d2/d15/d55/d59/d93/d95/db8/d81/d4a/d73/fa1 to d2/d15/d55/d59/fc2 0 2024-04-06T14:17:06.228 INFO:tasks.workunit.client.1.smithi086.stdout:2/318: mknod d0/d20/d27/d2b/d35/d36/d4d/d59/c72 0 2024-04-06T14:17:06.229 INFO:tasks.workunit.client.1.smithi086.stdout:2/319: chown d0/d20/d44 14 1 2024-04-06T14:17:06.231 INFO:tasks.workunit.client.0.smithi032.stdout:9/591: dread - d2/fb9 zero size 2024-04-06T14:17:06.231 INFO:tasks.workunit.client.1.smithi086.stdout:6/346: rename d6/d13/d42/l50 to d6/d9/d5c/l78 0 2024-04-06T14:17:06.232 INFO:tasks.workunit.client.0.smithi032.stdout:0/405: dwrite d3/da/d25/d36/f51 [0,4194304] 0 2024-04-06T14:17:06.234 INFO:tasks.workunit.client.0.smithi032.stdout:2/562: rename d0/d30/d39/f58 to d0/d30/d39/d59/d67/d74/d78/d57/d7c/fb9 0 2024-04-06T14:17:06.235 INFO:tasks.workunit.client.1.smithi086.stdout:1/331: dwrite f2 [0,4194304] 0 2024-04-06T14:17:06.235 INFO:tasks.workunit.client.1.smithi086.stdout:7/294: truncate d7/d10/d11/d1c/d24/d2c/f39 819288 0 2024-04-06T14:17:06.236 INFO:tasks.workunit.client.1.smithi086.stdout:6/347: write d6/d9/d5c/f58 [3186308,63900] 0 2024-04-06T14:17:06.238 INFO:tasks.workunit.client.1.smithi086.stdout:6/348: fsync d6/d9/d43/f74 0 2024-04-06T14:17:06.239 INFO:tasks.workunit.client.0.smithi032.stdout:9/592: dread d2/d15/d1e/d65/d6b/f9b [0,4194304] 0 2024-04-06T14:17:06.240 INFO:tasks.workunit.client.1.smithi086.stdout:9/333: link d5/d10/d14/d19/d25/l2e d5/d10/d14/d19/d25/d59/l6b 0 2024-04-06T14:17:06.241 INFO:tasks.workunit.client.0.smithi032.stdout:9/593: fdatasync d2/d15/d1e/d65/d6b/d92/fbc 0 2024-04-06T14:17:06.241 INFO:tasks.workunit.client.1.smithi086.stdout:7/295: stat d7/d10/d11/d21/l2a 0 2024-04-06T14:17:06.241 INFO:tasks.workunit.client.1.smithi086.stdout:1/332: write db/d11/d1a/d2a/f4c [373218,10411] 0 2024-04-06T14:17:06.245 INFO:tasks.workunit.client.1.smithi086.stdout:9/334: symlink d5/d13/d1a/d5f/l6c 0 2024-04-06T14:17:06.245 INFO:tasks.workunit.client.1.smithi086.stdout:1/333: chown db/d11/d1f/d5c/c6c 11343 1 2024-04-06T14:17:06.245 INFO:tasks.workunit.client.0.smithi032.stdout:9/594: dread d2/d15/d55/f9a [0,4194304] 0 2024-04-06T14:17:06.246 INFO:tasks.workunit.client.1.smithi086.stdout:1/334: chown db 1792329 1 2024-04-06T14:17:06.246 INFO:tasks.workunit.client.1.smithi086.stdout:7/296: chown d7/d10/d11/d1c/l37 0 1 2024-04-06T14:17:06.247 INFO:tasks.workunit.client.1.smithi086.stdout:2/320: dwrite d0/d20/f23 [0,4194304] 0 2024-04-06T14:17:06.247 INFO:tasks.workunit.client.0.smithi032.stdout:9/595: dread - d2/d15/d1e/d65/d83/d31/d39/f52 zero size 2024-04-06T14:17:06.248 INFO:tasks.workunit.client.1.smithi086.stdout:0/353: dread d0/f17 [0,4194304] 0 2024-04-06T14:17:06.256 INFO:tasks.workunit.client.1.smithi086.stdout:1/335: creat db/d11/d1a/d2a/d52/f6d x:0 0 0 2024-04-06T14:17:06.256 INFO:tasks.workunit.client.1.smithi086.stdout:7/297: truncate d7/d10/d11/d1c/d24/d4d/f62 101183 0 2024-04-06T14:17:06.256 INFO:tasks.workunit.client.1.smithi086.stdout:0/354: write d0/d1/d2/d1a/d51/f60 [603308,57696] 0 2024-04-06T14:17:06.256 INFO:tasks.workunit.client.1.smithi086.stdout:9/335: dread d5/d13/d47/d5e/f61 [0,4194304] 0 2024-04-06T14:17:06.257 INFO:tasks.workunit.client.1.smithi086.stdout:2/321: rename d0/df/f1a to d0/d20/d27/d2b/d35/d36/d4d/d6f/f73 0 2024-04-06T14:17:06.257 INFO:tasks.workunit.client.0.smithi032.stdout:9/596: creat d2/d15/d1e/d41/d6c/d74/fc3 x:0 0 0 2024-04-06T14:17:06.257 INFO:tasks.workunit.client.0.smithi032.stdout:0/406: dread d3/da/f2d [0,4194304] 0 2024-04-06T14:17:06.260 INFO:tasks.workunit.client.0.smithi032.stdout:0/407: symlink d3/da/d25/d2b/d4a/l8d 0 2024-04-06T14:17:06.262 INFO:tasks.workunit.client.1.smithi086.stdout:0/355: write d0/d1/d4/f6b [2531124,44899] 0 2024-04-06T14:17:06.264 INFO:tasks.workunit.client.1.smithi086.stdout:1/336: mkdir db/d11/d1a/d2a/d52/d5b/d68/d6e 0 2024-04-06T14:17:06.264 INFO:tasks.workunit.client.0.smithi032.stdout:9/597: mknod d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/cc4 0 2024-04-06T14:17:06.266 INFO:tasks.workunit.client.1.smithi086.stdout:9/336: rename d5/d10/d14/d19/d25/f3e to d5/d10/d14/f6d 0 2024-04-06T14:17:06.268 INFO:tasks.workunit.client.1.smithi086.stdout:7/298: dwrite d7/d10/d11/d1c/d24/d2c/f63 [0,4194304] 0 2024-04-06T14:17:06.270 INFO:tasks.workunit.client.1.smithi086.stdout:2/322: mknod d0/d20/d27/d2b/c74 0 2024-04-06T14:17:06.275 INFO:tasks.workunit.client.0.smithi032.stdout:2/563: symlink d0/d30/d39/d59/d67/d80/lba 0 2024-04-06T14:17:06.277 INFO:tasks.workunit.client.1.smithi086.stdout:6/349: dread d6/f7 [0,4194304] 0 2024-04-06T14:17:06.280 INFO:tasks.workunit.client.1.smithi086.stdout:2/323: mknod d0/d20/d27/d2b/d35/c75 0 2024-04-06T14:17:06.280 INFO:tasks.workunit.client.0.smithi032.stdout:0/408: dread d3/da/d25/d2b/f40 [0,4194304] 0 2024-04-06T14:17:06.280 INFO:tasks.workunit.client.0.smithi032.stdout:9/598: dwrite d2/d15/d1e/d65/d83/d31/f40 [0,4194304] 0 2024-04-06T14:17:06.280 INFO:tasks.workunit.client.0.smithi032.stdout:3/515: sync 2024-04-06T14:17:06.282 INFO:tasks.workunit.client.0.smithi032.stdout:9/599: read - d2/d15/d1e/d65/d6b/d7f/fbf zero size 2024-04-06T14:17:06.284 INFO:tasks.workunit.client.1.smithi086.stdout:2/324: mkdir d0/df/d48/d53/d76 0 2024-04-06T14:17:06.285 INFO:tasks.workunit.client.1.smithi086.stdout:7/299: dwrite d7/d10/f64 [0,4194304] 0 2024-04-06T14:17:06.285 INFO:tasks.workunit.client.0.smithi032.stdout:2/564: getdents d0/d5a 0 2024-04-06T14:17:06.286 INFO:tasks.workunit.client.0.smithi032.stdout:9/600: truncate d2/d15/d1e/f6d 1039139 0 2024-04-06T14:17:06.287 INFO:tasks.workunit.client.1.smithi086.stdout:6/350: chown d6/d13/d1a/l25 916933050 1 2024-04-06T14:17:06.293 INFO:tasks.workunit.client.0.smithi032.stdout:0/409: link d3/da/d25/d2b/d4a/l67 d3/da/d35/l8e 0 2024-04-06T14:17:06.294 INFO:tasks.workunit.client.0.smithi032.stdout:2/565: mknod d0/d5a/d3a/cbb 0 2024-04-06T14:17:06.296 INFO:tasks.workunit.client.1.smithi086.stdout:2/325: rmdir d0/d20/d27/d2b/d35 39 2024-04-06T14:17:06.296 INFO:tasks.workunit.client.1.smithi086.stdout:7/300: mkdir d7/d10/d11/d1c/d24/d2c/d69 0 2024-04-06T14:17:06.297 INFO:tasks.workunit.client.1.smithi086.stdout:6/351: rename d6/d9/db/f69 to d6/d13/d5f/d60/d45/d4a/f79 0 2024-04-06T14:17:06.297 INFO:tasks.workunit.client.1.smithi086.stdout:1/337: dread f1 [0,4194304] 0 2024-04-06T14:17:06.299 INFO:tasks.workunit.client.1.smithi086.stdout:1/338: chown f7 1 1 2024-04-06T14:17:06.300 INFO:tasks.workunit.client.1.smithi086.stdout:9/337: write d5/d10/d14/d19/d25/f67 [4797643,100784] 0 2024-04-06T14:17:06.301 INFO:tasks.workunit.client.1.smithi086.stdout:0/356: dread d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:06.302 INFO:tasks.workunit.client.1.smithi086.stdout:1/339: readlink db/d11/d1a/d2a/l5d 0 2024-04-06T14:17:06.302 INFO:tasks.workunit.client.1.smithi086.stdout:7/301: symlink d7/d10/d11/d1c/d24/d2c/l6a 0 2024-04-06T14:17:06.303 INFO:tasks.workunit.client.1.smithi086.stdout:1/340: chown db/d11/d1f/d5c 465067 1 2024-04-06T14:17:06.304 INFO:tasks.workunit.client.1.smithi086.stdout:0/357: write d0/d1/d4/d7/f27 [1386412,111123] 0 2024-04-06T14:17:06.304 INFO:tasks.workunit.client.1.smithi086.stdout:6/352: rename d6/d13/d5f/d60/d45/d4a/f4c to d6/d9/db/f7a 0 2024-04-06T14:17:06.305 INFO:tasks.workunit.client.1.smithi086.stdout:1/341: fdatasync db/ff 0 2024-04-06T14:17:06.306 INFO:tasks.workunit.client.1.smithi086.stdout:2/326: creat d0/d20/d27/d2b/d35/d36/d4d/d6f/f77 x:0 0 0 2024-04-06T14:17:06.308 INFO:tasks.workunit.client.1.smithi086.stdout:6/353: creat d6/d13/d42/f7b x:0 0 0 2024-04-06T14:17:06.308 INFO:tasks.workunit.client.1.smithi086.stdout:0/358: mkdir d0/d1/d1b/d44/d5d/d74 0 2024-04-06T14:17:06.310 INFO:tasks.workunit.client.1.smithi086.stdout:1/342: mkdir db/d11/d1a/d2a/d52/d5b/d68/d6f 0 2024-04-06T14:17:06.311 INFO:tasks.workunit.client.1.smithi086.stdout:0/359: stat d0/d1/d1b/d44/d5d/d74 0 2024-04-06T14:17:06.312 INFO:tasks.workunit.client.1.smithi086.stdout:2/327: mkdir d0/df/d48/d53/d76/d78 0 2024-04-06T14:17:06.313 INFO:tasks.workunit.client.1.smithi086.stdout:9/338: dwrite d5/d10/d14/d19/d25/d59/f60 [0,4194304] 0 2024-04-06T14:17:06.314 INFO:tasks.workunit.client.1.smithi086.stdout:1/343: chown db/d11/d1f/f5a 4933 1 2024-04-06T14:17:06.315 INFO:tasks.workunit.client.0.smithi032.stdout:3/516: truncate d3/d65/d7d/d9c/dad/db1/d3c/f79 1572860 0 2024-04-06T14:17:06.315 INFO:tasks.workunit.client.0.smithi032.stdout:3/517: chown l2 219 1 2024-04-06T14:17:06.319 INFO:tasks.workunit.client.1.smithi086.stdout:0/360: write d0/d15/d4e/f45 [1694524,117105] 0 2024-04-06T14:17:06.319 INFO:tasks.workunit.client.1.smithi086.stdout:1/344: rmdir db/d11/d42 39 2024-04-06T14:17:06.320 INFO:tasks.workunit.client.1.smithi086.stdout:0/361: readlink d0/d1/d1b/d44/l58 0 2024-04-06T14:17:06.320 INFO:tasks.workunit.client.1.smithi086.stdout:1/345: chown db/d11/d24/f45 3 1 2024-04-06T14:17:06.321 INFO:tasks.workunit.client.1.smithi086.stdout:0/362: chown d0/d1/d1b 64978106 1 2024-04-06T14:17:06.323 INFO:tasks.workunit.client.1.smithi086.stdout:0/363: read d0/d1/d25/f61 [2249503,2534] 0 2024-04-06T14:17:06.325 INFO:tasks.workunit.client.1.smithi086.stdout:0/364: write d0/d1/d4/d7/f11 [320935,85816] 0 2024-04-06T14:17:06.327 INFO:tasks.workunit.client.1.smithi086.stdout:1/346: unlink db/fc 0 2024-04-06T14:17:06.330 INFO:tasks.workunit.client.0.smithi032.stdout:2/566: dwrite d0/f8a [0,4194304] 0 2024-04-06T14:17:06.331 INFO:tasks.workunit.client.1.smithi086.stdout:7/302: dread d7/f9 [0,4194304] 0 2024-04-06T14:17:06.332 INFO:tasks.workunit.client.1.smithi086.stdout:1/347: mknod db/c70 0 2024-04-06T14:17:06.333 INFO:tasks.workunit.client.0.smithi032.stdout:3/518: dwrite d3/d16/f5c [0,4194304] 0 2024-04-06T14:17:06.334 INFO:tasks.workunit.client.0.smithi032.stdout:0/410: dwrite d3/f20 [0,4194304] 0 2024-04-06T14:17:06.336 INFO:tasks.workunit.client.0.smithi032.stdout:2/567: write d0/f6c [1208151,105255] 0 2024-04-06T14:17:06.337 INFO:tasks.workunit.client.1.smithi086.stdout:7/303: creat d7/d1d/f6b x:0 0 0 2024-04-06T14:17:06.337 INFO:tasks.workunit.client.1.smithi086.stdout:2/328: truncate d0/f2a 219834 0 2024-04-06T14:17:06.338 INFO:tasks.workunit.client.0.smithi032.stdout:5/534: sync 2024-04-06T14:17:06.338 INFO:tasks.workunit.client.0.smithi032.stdout:8/499: sync 2024-04-06T14:17:06.338 INFO:tasks.workunit.client.0.smithi032.stdout:7/591: sync 2024-04-06T14:17:06.338 INFO:tasks.workunit.client.0.smithi032.stdout:1/629: sync 2024-04-06T14:17:06.338 INFO:tasks.workunit.client.0.smithi032.stdout:6/546: sync 2024-04-06T14:17:06.338 INFO:tasks.workunit.client.0.smithi032.stdout:4/603: sync 2024-04-06T14:17:06.345 INFO:tasks.workunit.client.1.smithi086.stdout:8/418: sync 2024-04-06T14:17:06.348 INFO:tasks.workunit.client.0.smithi032.stdout:4/604: rmdir d0/d82/daf 39 2024-04-06T14:17:06.348 INFO:tasks.workunit.client.0.smithi032.stdout:7/592: mknod d16/d20/d63/d9d/cd4 0 2024-04-06T14:17:06.349 INFO:tasks.workunit.client.1.smithi086.stdout:9/339: truncate d5/d13/f4f 190062 0 2024-04-06T14:17:06.350 INFO:tasks.workunit.client.1.smithi086.stdout:7/304: symlink d7/d10/d11/l6c 0 2024-04-06T14:17:06.350 INFO:tasks.workunit.client.0.smithi032.stdout:9/601: dread d2/d15/f29 [0,4194304] 0 2024-04-06T14:17:06.350 INFO:tasks.workunit.client.0.smithi032.stdout:4/605: chown d0/fba 15698593 1 2024-04-06T14:17:06.351 INFO:tasks.workunit.client.1.smithi086.stdout:8/419: read d0/d3b/f3d [908884,42669] 0 2024-04-06T14:17:06.351 INFO:tasks.workunit.client.1.smithi086.stdout:0/365: dread d0/d1/d4/f6b [0,4194304] 0 2024-04-06T14:17:06.351 INFO:tasks.workunit.client.0.smithi032.stdout:9/602: truncate d2/fb9 816920 0 2024-04-06T14:17:06.351 INFO:tasks.workunit.client.0.smithi032.stdout:0/411: symlink d3/da/d25/l8f 0 2024-04-06T14:17:06.352 INFO:tasks.workunit.client.1.smithi086.stdout:2/329: creat d0/d20/d27/d37/d52/f79 x:0 0 0 2024-04-06T14:17:06.352 INFO:tasks.workunit.client.1.smithi086.stdout:7/305: write d7/d10/d11/d1c/f2e [2578931,6024] 0 2024-04-06T14:17:06.352 INFO:tasks.workunit.client.0.smithi032.stdout:3/519: creat d3/dcf/fd0 x:0 0 0 2024-04-06T14:17:06.353 INFO:tasks.workunit.client.0.smithi032.stdout:6/547: link d4/d5/d8/d2f/fb2 d4/d5/d19/fc3 0 2024-04-06T14:17:06.355 INFO:tasks.workunit.client.1.smithi086.stdout:7/306: write d7/d10/d11/d1c/d24/f4e [631393,105702] 0 2024-04-06T14:17:06.355 INFO:tasks.workunit.client.1.smithi086.stdout:9/340: rename d5/d10/l37 to d5/d58/l6e 0 2024-04-06T14:17:06.357 INFO:tasks.workunit.client.1.smithi086.stdout:8/420: unlink d0/d71/d22/d43/d6d/d75/l81 0 2024-04-06T14:17:06.358 INFO:tasks.workunit.client.0.smithi032.stdout:6/548: dread d4/d5/fa0 [0,4194304] 0 2024-04-06T14:17:06.358 INFO:tasks.workunit.client.1.smithi086.stdout:1/348: dread db/d13/f3d [0,4194304] 0 2024-04-06T14:17:06.358 INFO:tasks.workunit.client.0.smithi032.stdout:8/500: creat d3/d8/d12/d2f/d4a/d60/d90/fb6 x:0 0 0 2024-04-06T14:17:06.359 INFO:tasks.workunit.client.0.smithi032.stdout:4/606: mknod d0/df/d2b/d46/d58/cd3 0 2024-04-06T14:17:06.360 INFO:tasks.workunit.client.1.smithi086.stdout:9/341: creat d5/d10/d14/d19/d1b/f6f x:0 0 0 2024-04-06T14:17:06.361 INFO:tasks.workunit.client.0.smithi032.stdout:9/603: mknod d2/d15/d1e/d65/d6b/cc5 0 2024-04-06T14:17:06.361 INFO:tasks.workunit.client.1.smithi086.stdout:7/307: mknod d7/d10/d11/d1c/d24/d4d/d5c/c6d 0 2024-04-06T14:17:06.362 INFO:tasks.workunit.client.1.smithi086.stdout:2/330: dwrite d0/df/f14 [0,4194304] 0 2024-04-06T14:17:06.362 INFO:tasks.workunit.client.0.smithi032.stdout:1/630: creat d0/d41/fca x:0 0 0 2024-04-06T14:17:06.362 INFO:tasks.workunit.client.1.smithi086.stdout:8/421: mkdir d0/d71/d93 0 2024-04-06T14:17:06.364 INFO:tasks.workunit.client.1.smithi086.stdout:1/349: mkdir db/d13/d71 0 2024-04-06T14:17:06.364 INFO:tasks.workunit.client.0.smithi032.stdout:5/535: dread d16/d18/d33/d45/f9f [0,4194304] 0 2024-04-06T14:17:06.365 INFO:tasks.workunit.client.0.smithi032.stdout:1/631: dread - d0/da/f6b zero size 2024-04-06T14:17:06.365 INFO:tasks.workunit.client.1.smithi086.stdout:1/350: readlink db/l3c 0 2024-04-06T14:17:06.370 INFO:tasks.workunit.client.1.smithi086.stdout:9/342: dwrite d5/d10/d14/d19/d25/d59/f60 [0,4194304] 0 2024-04-06T14:17:06.372 INFO:tasks.workunit.client.0.smithi032.stdout:8/501: symlink d3/d8/d12/d2f/d4a/d60/d82/lb7 0 2024-04-06T14:17:06.372 INFO:tasks.workunit.client.1.smithi086.stdout:1/351: rmdir db/d11 39 2024-04-06T14:17:06.373 INFO:tasks.workunit.client.0.smithi032.stdout:1/632: write d0/d9/f24 [5003160,73757] 0 2024-04-06T14:17:06.373 INFO:tasks.workunit.client.1.smithi086.stdout:9/343: fdatasync d5/d10/d14/d19/d1b/f45 0 2024-04-06T14:17:06.373 INFO:tasks.workunit.client.1.smithi086.stdout:7/308: dwrite d7/fb [0,4194304] 0 2024-04-06T14:17:06.376 INFO:tasks.workunit.client.0.smithi032.stdout:0/412: link d3/da/f56 d3/da/d35/d8b/f90 0 2024-04-06T14:17:06.380 INFO:tasks.workunit.client.0.smithi032.stdout:3/520: truncate d3/d16/f2a 3334778 0 2024-04-06T14:17:06.380 INFO:tasks.workunit.client.1.smithi086.stdout:2/331: truncate d0/f1e 418010 0 2024-04-06T14:17:06.380 INFO:tasks.workunit.client.1.smithi086.stdout:8/422: creat d0/d71/d16/d39/d90/f94 x:0 0 0 2024-04-06T14:17:06.388 INFO:tasks.workunit.client.0.smithi032.stdout:6/549: dwrite d4/d5/d35/f59 [0,4194304] 0 2024-04-06T14:17:06.390 INFO:tasks.workunit.client.0.smithi032.stdout:8/502: creat d3/d8/d12/d83/d89/da0/fb8 x:0 0 0 2024-04-06T14:17:06.391 INFO:tasks.workunit.client.0.smithi032.stdout:3/521: fsync d3/d16/d63/fa1 0 2024-04-06T14:17:06.392 INFO:tasks.workunit.client.1.smithi086.stdout:7/309: creat d7/d10/d11/d1c/d24/d2c/d3c/f6e x:0 0 0 2024-04-06T14:17:06.393 INFO:tasks.workunit.client.1.smithi086.stdout:9/344: fsync f0 0 2024-04-06T14:17:06.394 INFO:tasks.workunit.client.1.smithi086.stdout:1/352: unlink db/d13/c4e 0 2024-04-06T14:17:06.395 INFO:tasks.workunit.client.0.smithi032.stdout:1/633: symlink d0/d9/d1f/d72/lcb 0 2024-04-06T14:17:06.396 INFO:tasks.workunit.client.1.smithi086.stdout:2/332: chown d0/d20/d27/d37/d52/f5c 1614074 1 2024-04-06T14:17:06.397 INFO:tasks.workunit.client.0.smithi032.stdout:3/522: mknod d3/d65/d7d/d9c/dbc/dcd/cd1 0 2024-04-06T14:17:06.399 INFO:tasks.workunit.client.1.smithi086.stdout:2/333: dread - d0/d20/d27/d2b/d35/d36/d4d/f4f zero size 2024-04-06T14:17:06.399 INFO:tasks.workunit.client.1.smithi086.stdout:9/345: symlink d5/d13/d47/d5e/l70 0 2024-04-06T14:17:06.400 INFO:tasks.workunit.client.1.smithi086.stdout:2/334: fdatasync d0/d20/f25 0 2024-04-06T14:17:06.400 INFO:tasks.workunit.client.1.smithi086.stdout:7/310: dwrite d7/d10/d11/d1c/f60 [0,4194304] 0 2024-04-06T14:17:06.401 INFO:tasks.workunit.client.0.smithi032.stdout:5/536: dwrite f9 [0,4194304] 0 2024-04-06T14:17:06.402 INFO:tasks.workunit.client.0.smithi032.stdout:8/503: stat d3/d8/d7/fa7 0 2024-04-06T14:17:06.403 INFO:tasks.workunit.client.1.smithi086.stdout:7/311: chown d7/d10/c1f 3409235 1 2024-04-06T14:17:06.404 INFO:tasks.workunit.client.1.smithi086.stdout:7/312: stat d7/d1d/l3b 0 2024-04-06T14:17:06.408 INFO:tasks.workunit.client.0.smithi032.stdout:1/634: read d0/f1e [1138970,99245] 0 2024-04-06T14:17:06.409 INFO:tasks.workunit.client.1.smithi086.stdout:9/346: rename d5/d13/f39 to d5/d13/f71 0 2024-04-06T14:17:06.411 INFO:tasks.workunit.client.0.smithi032.stdout:7/593: dread d16/d20/f24 [0,4194304] 0 2024-04-06T14:17:06.413 INFO:tasks.workunit.client.1.smithi086.stdout:1/353: dwrite db/d11/d24/f60 [0,4194304] 0 2024-04-06T14:17:06.417 INFO:tasks.workunit.client.1.smithi086.stdout:7/313: rename d7/f9 to d7/d10/d11/d1c/d24/d2c/f6f 0 2024-04-06T14:17:06.420 INFO:tasks.workunit.client.0.smithi032.stdout:2/568: dread d0/f24 [0,4194304] 0 2024-04-06T14:17:06.422 INFO:tasks.workunit.client.0.smithi032.stdout:8/504: mkdir d3/d8/d3a/d7b/db9 0 2024-04-06T14:17:06.422 INFO:tasks.workunit.client.0.smithi032.stdout:6/550: write d4/d5/d28/d54/f89 [22031,54521] 0 2024-04-06T14:17:06.424 INFO:tasks.workunit.client.1.smithi086.stdout:8/423: dread d0/d71/d16/f41 [0,4194304] 0 2024-04-06T14:17:06.426 INFO:tasks.workunit.client.0.smithi032.stdout:0/413: dwrite d3/da/d25/d36/f68 [0,4194304] 0 2024-04-06T14:17:06.428 INFO:tasks.workunit.client.1.smithi086.stdout:2/335: creat d0/f7a x:0 0 0 2024-04-06T14:17:06.428 INFO:tasks.workunit.client.1.smithi086.stdout:2/336: chown d0/l5 134 1 2024-04-06T14:17:06.429 INFO:tasks.workunit.client.1.smithi086.stdout:9/347: mknod d5/d10/d14/d19/c72 0 2024-04-06T14:17:06.429 INFO:tasks.workunit.client.1.smithi086.stdout:1/354: mknod db/d11/d1a/d2a/d52/d5b/d68/d6e/c72 0 2024-04-06T14:17:06.430 INFO:tasks.workunit.client.0.smithi032.stdout:3/523: write d3/d65/d7d/d9c/dad/db1/d3c/f57 [769854,114744] 0 2024-04-06T14:17:06.431 INFO:tasks.workunit.client.1.smithi086.stdout:1/355: stat db/d11/d1f/c21 0 2024-04-06T14:17:06.431 INFO:tasks.workunit.client.1.smithi086.stdout:8/424: mkdir d0/d71/d16/d95 0 2024-04-06T14:17:06.435 INFO:tasks.workunit.client.0.smithi032.stdout:4/607: dread d0/d82/d4b/f63 [0,4194304] 0 2024-04-06T14:17:06.435 INFO:tasks.workunit.client.0.smithi032.stdout:6/551: stat d4/d5/d8/d2f/d9b/db4 0 2024-04-06T14:17:06.436 INFO:tasks.workunit.client.0.smithi032.stdout:9/604: truncate d2/d15/d1e/d65/d83/d7e/f6 2062558 0 2024-04-06T14:17:06.439 INFO:tasks.workunit.client.1.smithi086.stdout:1/356: symlink db/l73 0 2024-04-06T14:17:06.439 INFO:tasks.workunit.client.0.smithi032.stdout:8/505: read d3/d8/d3a/f4f [128681,68755] 0 2024-04-06T14:17:06.440 INFO:tasks.workunit.client.0.smithi032.stdout:4/608: chown d0/df/d1f/d42/d45/f75 90 1 2024-04-06T14:17:06.440 INFO:tasks.workunit.client.0.smithi032.stdout:5/537: write d16/d58/f6d [1154563,123109] 0 2024-04-06T14:17:06.440 INFO:tasks.workunit.client.1.smithi086.stdout:8/425: mknod d0/d71/d22/d43/d6d/d75/c96 0 2024-04-06T14:17:06.440 INFO:tasks.workunit.client.0.smithi032.stdout:7/594: symlink d16/d20/d26/d2d/d30/dbb/ld5 0 2024-04-06T14:17:06.441 INFO:tasks.workunit.client.1.smithi086.stdout:7/314: creat d7/d10/d11/d1c/d24/d2c/f70 x:0 0 0 2024-04-06T14:17:06.443 INFO:tasks.workunit.client.0.smithi032.stdout:1/635: rename d0/d9/c6e to d0/d41/d5b/d96/ccc 0 2024-04-06T14:17:06.443 INFO:tasks.workunit.client.0.smithi032.stdout:3/524: chown d3/d65/d7d/d9c/dad/db1/l67 437 1 2024-04-06T14:17:06.443 INFO:tasks.workunit.client.0.smithi032.stdout:8/506: write d3/d8/d12/f5b [946533,81231] 0 2024-04-06T14:17:06.444 INFO:tasks.workunit.client.0.smithi032.stdout:6/552: symlink d4/d5/d8/d2f/lc4 0 2024-04-06T14:17:06.444 INFO:tasks.workunit.client.0.smithi032.stdout:2/569: symlink d0/d30/d39/d59/d67/d74/d78/d8c/db0/d95/lbc 0 2024-04-06T14:17:06.445 INFO:tasks.workunit.client.1.smithi086.stdout:2/337: dwrite d0/d20/d27/d2b/d35/f62 [0,4194304] 0 2024-04-06T14:17:06.445 INFO:tasks.workunit.client.1.smithi086.stdout:9/348: dwrite f2 [4194304,4194304] 0 2024-04-06T14:17:06.445 INFO:tasks.workunit.client.1.smithi086.stdout:8/426: creat d0/d71/d16/d39/d90/f97 x:0 0 0 2024-04-06T14:17:06.447 INFO:tasks.workunit.client.0.smithi032.stdout:9/605: mknod d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/cc6 0 2024-04-06T14:17:06.448 INFO:tasks.workunit.client.1.smithi086.stdout:2/338: chown d0/d20/d27/d2b/d35/d36/l58 823310006 1 2024-04-06T14:17:06.449 INFO:tasks.workunit.client.0.smithi032.stdout:7/595: creat d16/d20/d7a/da6/fd6 x:0 0 0 2024-04-06T14:17:06.449 INFO:tasks.workunit.client.0.smithi032.stdout:2/570: chown d0/d30/d7f/fac 13696 1 2024-04-06T14:17:06.449 INFO:tasks.workunit.client.0.smithi032.stdout:9/606: chown d2/d15/f2b 50763209 1 2024-04-06T14:17:06.450 INFO:tasks.workunit.client.0.smithi032.stdout:0/414: rename d3/da/d35/d75 to d3/da/d25/d2b/d4a/d91 0 2024-04-06T14:17:06.451 INFO:tasks.workunit.client.0.smithi032.stdout:5/538: mknod d16/d18/d33/d45/d52/d39/d86/cb9 0 2024-04-06T14:17:06.452 INFO:tasks.workunit.client.1.smithi086.stdout:9/349: symlink d5/d10/d14/d19/d25/d59/l73 0 2024-04-06T14:17:06.453 INFO:tasks.workunit.client.0.smithi032.stdout:8/507: creat d3/d8/d12/d83/fba x:0 0 0 2024-04-06T14:17:06.454 INFO:tasks.workunit.client.1.smithi086.stdout:8/427: mkdir d0/d71/d22/d65/d6e/d98 0 2024-04-06T14:17:06.454 INFO:tasks.workunit.client.0.smithi032.stdout:0/415: write d3/f78 [1043252,113656] 0 2024-04-06T14:17:06.458 INFO:tasks.workunit.client.0.smithi032.stdout:0/416: truncate d3/da/f26 4233782 0 2024-04-06T14:17:06.458 INFO:tasks.workunit.client.0.smithi032.stdout:7/596: creat d16/d92/db3/fd7 x:0 0 0 2024-04-06T14:17:06.459 INFO:tasks.workunit.client.1.smithi086.stdout:9/350: mkdir d5/d10/d14/d19/d25/d74 0 2024-04-06T14:17:06.459 INFO:tasks.workunit.client.1.smithi086.stdout:8/428: mknod d0/d71/d16/d52/d5f/c99 0 2024-04-06T14:17:06.460 INFO:tasks.workunit.client.1.smithi086.stdout:2/339: getdents d0/df/d48 0 2024-04-06T14:17:06.460 INFO:tasks.workunit.client.1.smithi086.stdout:1/357: dwrite f7 [0,4194304] 0 2024-04-06T14:17:06.460 INFO:tasks.workunit.client.0.smithi032.stdout:9/607: unlink d2/d15/d1e/d65/d83/d31/f57 0 2024-04-06T14:17:06.461 INFO:tasks.workunit.client.0.smithi032.stdout:5/539: symlink d16/d58/d54/d74/lba 0 2024-04-06T14:17:06.461 INFO:tasks.workunit.client.0.smithi032.stdout:2/571: getdents d0/d6e/dae/db7 0 2024-04-06T14:17:06.461 INFO:tasks.workunit.client.1.smithi086.stdout:9/351: chown d5/d10/d14/d19/d25/d59/l73 1964 1 2024-04-06T14:17:06.462 INFO:tasks.workunit.client.0.smithi032.stdout:9/608: stat d2/f4 0 2024-04-06T14:17:06.462 INFO:tasks.workunit.client.1.smithi086.stdout:9/352: fsync d5/d10/d14/d19/f5d 0 2024-04-06T14:17:06.466 INFO:tasks.workunit.client.1.smithi086.stdout:2/340: mknod d0/d20/d27/d2b/d35/d36/d4d/d59/c7b 0 2024-04-06T14:17:06.467 INFO:tasks.workunit.client.0.smithi032.stdout:5/540: mkdir d16/d18/dae/dbb 0 2024-04-06T14:17:06.468 INFO:tasks.workunit.client.1.smithi086.stdout:9/353: chown d5/d58/l6e 283932145 1 2024-04-06T14:17:06.470 INFO:tasks.workunit.client.1.smithi086.stdout:2/341: fsync d0/d20/d27/d2b/d35/f62 0 2024-04-06T14:17:06.470 INFO:tasks.workunit.client.1.smithi086.stdout:8/429: dread d0/d71/d16/f41 [0,4194304] 0 2024-04-06T14:17:06.472 INFO:tasks.workunit.client.1.smithi086.stdout:1/358: dwrite f1 [4194304,4194304] 0 2024-04-06T14:17:06.474 INFO:tasks.workunit.client.1.smithi086.stdout:1/359: write f7 [1352248,54054] 0 2024-04-06T14:17:06.474 INFO:tasks.workunit.client.1.smithi086.stdout:2/342: rename d0/df/f1b to d0/df/d48/f7c 0 2024-04-06T14:17:06.476 INFO:tasks.workunit.client.1.smithi086.stdout:1/360: read db/d11/f12 [2598584,20215] 0 2024-04-06T14:17:06.477 INFO:tasks.workunit.client.0.smithi032.stdout:6/553: rename d4/d5/d18/d23 to d4/d5/dc5 0 2024-04-06T14:17:06.479 INFO:tasks.workunit.client.1.smithi086.stdout:1/361: fdatasync db/d11/f41 0 2024-04-06T14:17:06.480 INFO:tasks.workunit.client.0.smithi032.stdout:2/572: dread d0/d30/d39/d59/d67/d74/d78/d57/f90 [0,4194304] 0 2024-04-06T14:17:06.480 INFO:tasks.workunit.client.0.smithi032.stdout:4/609: write d0/df/d23/f3c [1285866,129670] 0 2024-04-06T14:17:06.482 INFO:tasks.workunit.client.1.smithi086.stdout:8/430: dwrite d0/db/f92 [0,4194304] 0 2024-04-06T14:17:06.482 INFO:tasks.workunit.client.1.smithi086.stdout:1/362: chown db/d11/d1a/d2a/d52/d6b 112280 1 2024-04-06T14:17:06.482 INFO:tasks.workunit.client.0.smithi032.stdout:5/541: write d16/d58/d54/d94/db2/fb8 [673900,70860] 0 2024-04-06T14:17:06.483 INFO:tasks.workunit.client.1.smithi086.stdout:8/431: write d0/d71/d22/f74 [802557,84891] 0 2024-04-06T14:17:06.484 INFO:tasks.workunit.client.1.smithi086.stdout:8/432: fsync d0/d71/d16/d39/f66 0 2024-04-06T14:17:06.484 INFO:tasks.workunit.client.0.smithi032.stdout:8/508: fsync d3/d8/d12/d83/fba 0 2024-04-06T14:17:06.485 INFO:tasks.workunit.client.0.smithi032.stdout:2/573: dread - d0/d30/d7f/fac zero size 2024-04-06T14:17:06.487 INFO:tasks.workunit.client.0.smithi032.stdout:1/636: write d0/d10/f25 [697136,54129] 0 2024-04-06T14:17:06.487 INFO:tasks.workunit.client.0.smithi032.stdout:3/525: truncate d3/d65/d7d/d9c/dad/fb3 663104 0 2024-04-06T14:17:06.490 INFO:tasks.workunit.client.0.smithi032.stdout:0/417: dread d3/f5a [0,4194304] 0 2024-04-06T14:17:06.491 INFO:tasks.workunit.client.0.smithi032.stdout:6/554: creat d4/fc6 x:0 0 0 2024-04-06T14:17:06.493 INFO:tasks.workunit.client.0.smithi032.stdout:2/574: unlink d0/d30/d39/d59/f82 0 2024-04-06T14:17:06.499 INFO:tasks.workunit.client.1.smithi086.stdout:4/250: sync 2024-04-06T14:17:06.499 INFO:tasks.workunit.client.1.smithi086.stdout:3/387: sync 2024-04-06T14:17:06.499 INFO:tasks.workunit.client.1.smithi086.stdout:5/340: sync 2024-04-06T14:17:06.502 INFO:tasks.workunit.client.0.smithi032.stdout:3/526: dwrite d3/d16/d63/fa1 [0,4194304] 0 2024-04-06T14:17:06.504 INFO:tasks.workunit.client.1.smithi086.stdout:3/388: fdatasync d5/d33/d5d/f6f 0 2024-04-06T14:17:06.506 INFO:tasks.workunit.client.1.smithi086.stdout:3/389: readlink d5/d33/l30 0 2024-04-06T14:17:06.507 INFO:tasks.workunit.client.1.smithi086.stdout:4/251: dread d0/d2c/d30/d24/f2d [0,4194304] 0 2024-04-06T14:17:06.508 INFO:tasks.workunit.client.1.smithi086.stdout:5/341: mkdir d0/d16/d18/d2a/d50/d66 0 2024-04-06T14:17:06.509 INFO:tasks.workunit.client.1.smithi086.stdout:3/390: fdatasync d5/d42/f4c 0 2024-04-06T14:17:06.509 INFO:tasks.workunit.client.1.smithi086.stdout:5/342: fdatasync d0/d17/d2d/f4f 0 2024-04-06T14:17:06.509 INFO:tasks.workunit.client.0.smithi032.stdout:3/527: read - d3/d65/f93 zero size 2024-04-06T14:17:06.509 INFO:tasks.workunit.client.0.smithi032.stdout:5/542: dwrite d16/d18/d33/f8a [0,4194304] 0 2024-04-06T14:17:06.511 INFO:tasks.workunit.client.0.smithi032.stdout:0/418: rename d3/da/d25/d57/c6c to d3/da/d25/d2b/d4a/d86/c92 0 2024-04-06T14:17:06.514 INFO:tasks.workunit.client.1.smithi086.stdout:3/391: readlink d5/d42/l66 0 2024-04-06T14:17:06.516 INFO:tasks.workunit.client.0.smithi032.stdout:2/575: mknod d0/d30/d39/cbd 0 2024-04-06T14:17:06.517 INFO:tasks.workunit.client.0.smithi032.stdout:6/555: dwrite d4/d5/d28/d65/fa3 [0,4194304] 0 2024-04-06T14:17:06.521 INFO:tasks.workunit.client.0.smithi032.stdout:0/419: creat d3/da/d25/d36/f93 x:0 0 0 2024-04-06T14:17:06.522 INFO:tasks.workunit.client.0.smithi032.stdout:8/509: dread d3/d8/f87 [0,4194304] 0 2024-04-06T14:17:06.524 INFO:tasks.workunit.client.1.smithi086.stdout:5/343: dread d0/d16/f39 [0,4194304] 0 2024-04-06T14:17:06.525 INFO:tasks.workunit.client.1.smithi086.stdout:8/433: truncate d0/db/d32/f3e 2052786 0 2024-04-06T14:17:06.526 INFO:tasks.workunit.client.0.smithi032.stdout:5/543: rmdir d16/d18/d33/da4 0 2024-04-06T14:17:06.528 INFO:tasks.workunit.client.1.smithi086.stdout:2/343: dwrite d0/d20/d44/f49 [0,4194304] 0 2024-04-06T14:17:06.528 INFO:tasks.workunit.client.0.smithi032.stdout:6/556: write d4/d5/dbf/f8c [264297,43900] 0 2024-04-06T14:17:06.529 INFO:tasks.workunit.client.0.smithi032.stdout:4/610: dwrite d0/df/d1f/d2e/f49 [0,4194304] 0 2024-04-06T14:17:06.531 INFO:tasks.workunit.client.0.smithi032.stdout:5/544: chown d16/d58/f59 26478488 1 2024-04-06T14:17:06.531 INFO:tasks.workunit.client.1.smithi086.stdout:3/392: rename d5/d26/d4f/d69 to d5/d42/d59/d70 0 2024-04-06T14:17:06.531 INFO:tasks.workunit.client.1.smithi086.stdout:8/434: dread - d0/d13/f7c zero size 2024-04-06T14:17:06.532 INFO:tasks.workunit.client.1.smithi086.stdout:4/252: dwrite d0/de/f26 [0,4194304] 0 2024-04-06T14:17:06.532 INFO:tasks.workunit.client.0.smithi032.stdout:0/420: truncate d3/da/d25/d2b/f3d 623293 0 2024-04-06T14:17:06.538 INFO:tasks.workunit.client.1.smithi086.stdout:5/344: getdents d0/d5e 0 2024-04-06T14:17:06.539 INFO:tasks.workunit.client.1.smithi086.stdout:2/344: write d0/d20/d27/d2b/f65 [1025291,26587] 0 2024-04-06T14:17:06.542 INFO:tasks.workunit.client.0.smithi032.stdout:2/576: rename d0/d30/d39/d59/d67/d74/d78/d65 to d0/d4e/d63/dbe 0 2024-04-06T14:17:06.543 INFO:tasks.workunit.client.1.smithi086.stdout:2/345: truncate d0/f33 4817960 0 2024-04-06T14:17:06.543 INFO:tasks.workunit.client.1.smithi086.stdout:4/253: symlink d0/d15/d20/d34/l4b 0 2024-04-06T14:17:06.545 INFO:tasks.workunit.client.0.smithi032.stdout:4/611: creat d0/df/d2b/d66/fd4 x:0 0 0 2024-04-06T14:17:06.546 INFO:tasks.workunit.client.1.smithi086.stdout:5/345: write d0/d16/d18/d35/f4d [635312,71603] 0 2024-04-06T14:17:06.546 INFO:tasks.workunit.client.0.smithi032.stdout:5/545: symlink d16/d18/d33/d45/d5e/lbc 0 2024-04-06T14:17:06.547 INFO:tasks.workunit.client.1.smithi086.stdout:2/346: chown d0/d20 14 1 2024-04-06T14:17:06.548 INFO:tasks.workunit.client.1.smithi086.stdout:8/435: dwrite d0/d71/d22/d43/f47 [0,4194304] 0 2024-04-06T14:17:06.551 INFO:tasks.workunit.client.1.smithi086.stdout:4/254: mkdir d0/de/d4c 0 2024-04-06T14:17:06.552 INFO:tasks.workunit.client.0.smithi032.stdout:2/577: mknod d0/d30/d39/d59/d67/d80/cbf 0 2024-04-06T14:17:06.552 INFO:tasks.workunit.client.0.smithi032.stdout:4/612: dread - d0/f5c zero size 2024-04-06T14:17:06.554 INFO:tasks.workunit.client.1.smithi086.stdout:4/255: truncate d0/d15/d20/f44 1004126 0 2024-04-06T14:17:06.554 INFO:tasks.workunit.client.1.smithi086.stdout:3/393: getdents d5/d42/d59 0 2024-04-06T14:17:06.555 INFO:tasks.workunit.client.0.smithi032.stdout:3/528: write d3/d65/d7d/d9c/dad/db1/f40 [135364,114288] 0 2024-04-06T14:17:06.556 INFO:tasks.workunit.client.1.smithi086.stdout:4/256: write d0/f36 [851929,55204] 0 2024-04-06T14:17:06.557 INFO:tasks.workunit.client.1.smithi086.stdout:3/394: chown d5/d36/d5c/f5e 61501 1 2024-04-06T14:17:06.558 INFO:tasks.workunit.client.0.smithi032.stdout:6/557: dwrite d4/d5/f15 [0,4194304] 0 2024-04-06T14:17:06.558 INFO:tasks.workunit.client.1.smithi086.stdout:2/347: dwrite d0/d20/d27/d2b/f65 [0,4194304] 0 2024-04-06T14:17:06.559 INFO:tasks.workunit.client.1.smithi086.stdout:2/348: fsync d0/d20/d27/d2b/f45 0 2024-04-06T14:17:06.560 INFO:tasks.workunit.client.0.smithi032.stdout:8/510: dwrite d3/d8/f25 [4194304,4194304] 0 2024-04-06T14:17:06.560 INFO:tasks.workunit.client.0.smithi032.stdout:2/578: chown d0/d30/d7f/fac 23694 1 2024-04-06T14:17:06.562 INFO:tasks.workunit.client.0.smithi032.stdout:1/637: dread d0/d10/d16/f7a [0,4194304] 0 2024-04-06T14:17:06.564 INFO:tasks.workunit.client.0.smithi032.stdout:0/421: rename d3/f19 to d3/da/d25/f94 0 2024-04-06T14:17:06.565 INFO:tasks.workunit.client.1.smithi086.stdout:3/395: rename d5/d36/d5c/f5e to d5/d36/d49/d64/f71 0 2024-04-06T14:17:06.566 INFO:tasks.workunit.client.0.smithi032.stdout:0/422: fsync d3/f20 0 2024-04-06T14:17:06.568 INFO:tasks.workunit.client.0.smithi032.stdout:6/558: read - d4/d5/d8/d4b/fa6 zero size 2024-04-06T14:17:06.568 INFO:tasks.workunit.client.0.smithi032.stdout:8/511: creat d3/d8/d12/d83/fbb x:0 0 0 2024-04-06T14:17:06.570 INFO:tasks.workunit.client.1.smithi086.stdout:2/349: fsync d0/f2a 0 2024-04-06T14:17:06.573 INFO:tasks.workunit.client.0.smithi032.stdout:3/529: link d3/d65/d7d/d9c/db8/fc2 d3/d65/d7d/fd2 0 2024-04-06T14:17:06.574 INFO:tasks.workunit.client.0.smithi032.stdout:1/638: mknod d0/d10/ccd 0 2024-04-06T14:17:06.574 INFO:tasks.workunit.client.1.smithi086.stdout:2/350: mkdir d0/df/d48/d7d 0 2024-04-06T14:17:06.575 INFO:tasks.workunit.client.0.smithi032.stdout:5/546: getdents d16/d18/d33/d35/d88 0 2024-04-06T14:17:06.576 INFO:tasks.workunit.client.0.smithi032.stdout:8/512: fsync d3/d8/d12/f4d 0 2024-04-06T14:17:06.576 INFO:tasks.workunit.client.0.smithi032.stdout:1/639: readlink d0/d41/l9c 0 2024-04-06T14:17:06.577 INFO:tasks.workunit.client.0.smithi032.stdout:4/613: rename d0/df/d1f/d42/c4c to d0/df/d1f/d2e/cd5 0 2024-04-06T14:17:06.581 INFO:tasks.workunit.client.0.smithi032.stdout:6/559: rename d4/d5/d14/fc1 to d4/fc7 0 2024-04-06T14:17:06.582 INFO:tasks.workunit.client.0.smithi032.stdout:8/513: chown d3/d8/d3a/l8f 44302 1 2024-04-06T14:17:06.582 INFO:tasks.workunit.client.1.smithi086.stdout:5/346: write d0/d16/d18/d21/f45 [829949,104675] 0 2024-04-06T14:17:06.584 INFO:tasks.workunit.client.0.smithi032.stdout:5/547: mkdir d16/d18/d33/d35/d88/dbd 0 2024-04-06T14:17:06.587 INFO:tasks.workunit.client.1.smithi086.stdout:5/347: rename d0/f54 to d0/d16/f67 0 2024-04-06T14:17:06.588 INFO:tasks.workunit.client.0.smithi032.stdout:3/530: link d3/d65/l36 d3/d65/d7d/d9c/dad/db1/d3c/ld3 0 2024-04-06T14:17:06.588 INFO:tasks.workunit.client.1.smithi086.stdout:8/436: dread d0/d3b/f3d [0,4194304] 0 2024-04-06T14:17:06.588 INFO:tasks.workunit.client.0.smithi032.stdout:1/640: creat d0/d9/d94/fce x:0 0 0 2024-04-06T14:17:06.590 INFO:tasks.workunit.client.0.smithi032.stdout:6/560: readlink d4/d5/d8/lb7 0 2024-04-06T14:17:06.591 INFO:tasks.workunit.client.1.smithi086.stdout:8/437: mknod d0/d71/d22/d43/c9a 0 2024-04-06T14:17:06.591 INFO:tasks.workunit.client.1.smithi086.stdout:4/257: dread d0/d15/d20/f22 [0,4194304] 0 2024-04-06T14:17:06.594 INFO:tasks.workunit.client.0.smithi032.stdout:5/548: creat d16/d18/d33/d45/d52/d39/d86/fbe x:0 0 0 2024-04-06T14:17:06.595 INFO:tasks.workunit.client.1.smithi086.stdout:4/258: creat d0/d2c/f4d x:0 0 0 2024-04-06T14:17:06.596 INFO:tasks.workunit.client.0.smithi032.stdout:1/641: mkdir d0/d10/d16/dcf 0 2024-04-06T14:17:06.596 INFO:tasks.workunit.client.0.smithi032.stdout:2/579: dread d0/d6e/f86 [0,4194304] 0 2024-04-06T14:17:06.597 INFO:tasks.workunit.client.1.smithi086.stdout:8/438: creat d0/d71/d22/d43/d6d/d75/d7d/f9b x:0 0 0 2024-04-06T14:17:06.599 INFO:tasks.workunit.client.0.smithi032.stdout:5/549: chown d16/d58/f7a 19580855 1 2024-04-06T14:17:06.600 INFO:tasks.workunit.client.0.smithi032.stdout:1/642: fdatasync d0/d10/d16/d3f/fa3 0 2024-04-06T14:17:06.600 INFO:tasks.workunit.client.0.smithi032.stdout:8/514: getdents d3/d8 0 2024-04-06T14:17:06.601 INFO:tasks.workunit.client.1.smithi086.stdout:5/348: write d0/d16/f67 [608880,60294] 0 2024-04-06T14:17:06.602 INFO:tasks.workunit.client.0.smithi032.stdout:2/580: creat d0/d30/d39/d59/d67/fc0 x:0 0 0 2024-04-06T14:17:06.603 INFO:tasks.workunit.client.1.smithi086.stdout:4/259: rename d0/de/l23 to d0/l4e 0 2024-04-06T14:17:06.606 INFO:tasks.workunit.client.1.smithi086.stdout:2/351: dread d0/d20/d27/d2b/d35/f4e [0,4194304] 0 2024-04-06T14:17:06.606 INFO:tasks.workunit.client.0.smithi032.stdout:8/515: chown d3/l58 263484447 1 2024-04-06T14:17:06.607 INFO:tasks.workunit.client.1.smithi086.stdout:8/439: rename d0/d71/d22/d43/d6d/d75/d7d/f9b to d0/d3b/d79/f9c 0 2024-04-06T14:17:06.607 INFO:tasks.workunit.client.0.smithi032.stdout:1/643: chown d0/d41/f5c 0 1 2024-04-06T14:17:06.608 INFO:tasks.workunit.client.0.smithi032.stdout:2/581: dread - d0/d30/d39/d59/d67/fc0 zero size 2024-04-06T14:17:06.609 INFO:tasks.workunit.client.0.smithi032.stdout:5/550: creat d16/d18/d33/d45/d52/d39/fbf x:0 0 0 2024-04-06T14:17:06.609 INFO:tasks.workunit.client.0.smithi032.stdout:6/561: dwrite d4/d5/d18/f52 [0,4194304] 0 2024-04-06T14:17:06.611 INFO:tasks.workunit.client.1.smithi086.stdout:2/352: creat d0/d20/d27/d37/f7e x:0 0 0 2024-04-06T14:17:06.611 INFO:tasks.workunit.client.1.smithi086.stdout:8/440: fdatasync d0/d71/d22/d65/d6e/f8f 0 2024-04-06T14:17:06.611 INFO:tasks.workunit.client.0.smithi032.stdout:6/562: readlink d4/d5/d35/l93 0 2024-04-06T14:17:06.611 INFO:tasks.workunit.client.0.smithi032.stdout:3/531: dread d3/d16/f2a [0,4194304] 0 2024-04-06T14:17:06.613 INFO:tasks.workunit.client.1.smithi086.stdout:5/349: dwrite d0/f4 [0,4194304] 0 2024-04-06T14:17:06.613 INFO:tasks.workunit.client.1.smithi086.stdout:4/260: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:17:06.616 INFO:tasks.workunit.client.0.smithi032.stdout:6/563: mknod d4/d5/d28/d54/cc8 0 2024-04-06T14:17:06.618 INFO:tasks.workunit.client.0.smithi032.stdout:3/532: write d3/d16/f9a [2647917,115040] 0 2024-04-06T14:17:06.619 INFO:tasks.workunit.client.0.smithi032.stdout:4/614: dread d0/df/d23/d31/f68 [0,4194304] 0 2024-04-06T14:17:06.621 INFO:tasks.workunit.client.0.smithi032.stdout:5/551: creat d16/d18/d33/d45/fc0 x:0 0 0 2024-04-06T14:17:06.621 INFO:tasks.workunit.client.1.smithi086.stdout:4/261: symlink d0/d2c/d38/l4f 0 2024-04-06T14:17:06.622 INFO:tasks.workunit.client.1.smithi086.stdout:8/441: write d0/db/f12 [2435845,45029] 0 2024-04-06T14:17:06.630 INFO:tasks.workunit.client.0.smithi032.stdout:6/564: dread d4/d5/d14/f1a [0,4194304] 0 2024-04-06T14:17:06.632 INFO:tasks.workunit.client.1.smithi086.stdout:4/262: dwrite d0/d15/f3c [0,4194304] 0 2024-04-06T14:17:06.633 INFO:tasks.workunit.client.0.smithi032.stdout:6/565: symlink d4/d5/d14/lc9 0 2024-04-06T14:17:06.633 INFO:tasks.workunit.client.0.smithi032.stdout:4/615: dwrite d0/d82/d4b/f63 [0,4194304] 0 2024-04-06T14:17:06.637 INFO:tasks.workunit.client.1.smithi086.stdout:8/442: dwrite d0/db/d32/f5d [0,4194304] 0 2024-04-06T14:17:06.637 INFO:tasks.workunit.client.0.smithi032.stdout:3/533: rename d3/d65/d7d/d9c/dad/db1/d3c/lba to d3/d65/d7d/d9c/dad/db1/ld4 0 2024-04-06T14:17:06.639 INFO:tasks.workunit.client.0.smithi032.stdout:6/566: creat d4/d5/dbf/d61/db9/fca x:0 0 0 2024-04-06T14:17:06.642 INFO:tasks.workunit.client.0.smithi032.stdout:6/567: write d4/d5/d8/d2f/d9b/db4/fc0 [739857,49439] 0 2024-04-06T14:17:06.644 INFO:tasks.workunit.client.0.smithi032.stdout:8/516: truncate d3/d8/f6d 2834480 0 2024-04-06T14:17:06.644 INFO:tasks.workunit.client.0.smithi032.stdout:6/568: dread - d4/d5/dbf/d51/fa8 zero size 2024-04-06T14:17:06.645 INFO:tasks.workunit.client.1.smithi086.stdout:4/263: dread d0/f36 [0,4194304] 0 2024-04-06T14:17:06.647 INFO:tasks.workunit.client.0.smithi032.stdout:1/644: write d0/da/d3e/d75/f81 [5203288,102084] 0 2024-04-06T14:17:06.648 INFO:tasks.workunit.client.1.smithi086.stdout:4/264: mkdir d0/d2c/d30/d24/d50 0 2024-04-06T14:17:06.650 INFO:tasks.workunit.client.0.smithi032.stdout:8/517: creat d3/d8/d7/fbc x:0 0 0 2024-04-06T14:17:06.652 INFO:tasks.workunit.client.1.smithi086.stdout:4/265: symlink d0/d15/d20/d34/l51 0 2024-04-06T14:17:06.652 INFO:tasks.workunit.client.1.smithi086.stdout:8/443: dread d0/f26 [0,4194304] 0 2024-04-06T14:17:06.653 INFO:tasks.workunit.client.0.smithi032.stdout:4/616: dwrite d0/df/d23/d31/d50/d5e/fbf [0,4194304] 0 2024-04-06T14:17:06.653 INFO:tasks.workunit.client.1.smithi086.stdout:4/266: chown d0/d2c/d38/c3e 9 1 2024-04-06T14:17:06.654 INFO:tasks.workunit.client.0.smithi032.stdout:8/518: symlink d3/d8/d7/d8a/lbd 0 2024-04-06T14:17:06.654 INFO:tasks.workunit.client.1.smithi086.stdout:5/350: truncate d0/d19/f26 3301358 0 2024-04-06T14:17:06.655 INFO:tasks.workunit.client.0.smithi032.stdout:3/534: rename d3/d16/ca9 to d3/d65/d7d/d9c/dad/cd5 0 2024-04-06T14:17:06.655 INFO:tasks.workunit.client.1.smithi086.stdout:5/351: write d0/f1 [504427,104582] 0 2024-04-06T14:17:06.656 INFO:tasks.workunit.client.1.smithi086.stdout:8/444: chown d0/d71/d16/d39/c8d 3 1 2024-04-06T14:17:06.656 INFO:tasks.workunit.client.0.smithi032.stdout:6/569: rmdir d4/d5/dc5/d82/d9d/dac 0 2024-04-06T14:17:06.656 INFO:tasks.workunit.client.1.smithi086.stdout:2/353: truncate d0/d20/d44/f49 2010571 0 2024-04-06T14:17:06.657 INFO:tasks.workunit.client.0.smithi032.stdout:1/645: fsync d0/d9/d1f/d7c/f95 0 2024-04-06T14:17:06.663 INFO:tasks.workunit.client.1.smithi086.stdout:2/354: creat d0/d20/d27/f7f x:0 0 0 2024-04-06T14:17:06.663 INFO:tasks.workunit.client.1.smithi086.stdout:4/267: chown d0/d15/d2b/l33 8423 1 2024-04-06T14:17:06.663 INFO:tasks.workunit.client.1.smithi086.stdout:5/352: mknod d0/d16/d18/d2a/d50/d66/c68 0 2024-04-06T14:17:06.663 INFO:tasks.workunit.client.0.smithi032.stdout:8/519: truncate d3/d8/f99 391915 0 2024-04-06T14:17:06.663 INFO:tasks.workunit.client.0.smithi032.stdout:3/535: chown d3/d16/c5a 14 1 2024-04-06T14:17:06.663 INFO:tasks.workunit.client.0.smithi032.stdout:6/570: rmdir d4/d5/dbf/d51 39 2024-04-06T14:17:06.664 INFO:tasks.workunit.client.1.smithi086.stdout:8/445: mknod d0/d71/d22/d43/d4d/d62/c9d 0 2024-04-06T14:17:06.664 INFO:tasks.workunit.client.1.smithi086.stdout:4/268: write d0/d15/d20/f3a [519977,12875] 0 2024-04-06T14:17:06.665 INFO:tasks.workunit.client.0.smithi032.stdout:5/552: dwrite d16/d58/d54/f60 [0,4194304] 0 2024-04-06T14:17:06.665 INFO:tasks.workunit.client.0.smithi032.stdout:3/536: write d3/fae [852602,80309] 0 2024-04-06T14:17:06.665 INFO:tasks.workunit.client.1.smithi086.stdout:8/446: dread - d0/d71/d22/d43/d7f/f88 zero size 2024-04-06T14:17:06.665 INFO:tasks.workunit.client.1.smithi086.stdout:4/269: write d0/d15/f48 [651810,545] 0 2024-04-06T14:17:06.666 INFO:tasks.workunit.client.1.smithi086.stdout:8/447: chown d0/d71/d16/d52/l70 336 1 2024-04-06T14:17:06.668 INFO:tasks.workunit.client.0.smithi032.stdout:5/553: symlink d16/d18/d33/d45/d52/d39/d86/lc1 0 2024-04-06T14:17:06.669 INFO:tasks.workunit.client.0.smithi032.stdout:6/571: fdatasync d4/d5/d35/f39 0 2024-04-06T14:17:06.670 INFO:tasks.workunit.client.0.smithi032.stdout:3/537: write d3/d16/f55 [1280219,62808] 0 2024-04-06T14:17:06.672 INFO:tasks.workunit.client.0.smithi032.stdout:1/646: dwrite d0/f90 [0,4194304] 0 2024-04-06T14:17:06.675 INFO:tasks.workunit.client.1.smithi086.stdout:6/354: sync 2024-04-06T14:17:06.677 INFO:tasks.workunit.client.0.smithi032.stdout:6/572: creat d4/d5/d8/d4b/d7c/fcb x:0 0 0 2024-04-06T14:17:06.678 INFO:tasks.workunit.client.0.smithi032.stdout:1/647: creat d0/d9/d94/fd0 x:0 0 0 2024-04-06T14:17:06.684 INFO:tasks.workunit.client.0.smithi032.stdout:6/573: mkdir d4/d5/dc5/dcc 0 2024-04-06T14:17:06.684 INFO:tasks.workunit.client.0.smithi032.stdout:3/538: dwrite d3/d16/f9e [0,4194304] 0 2024-04-06T14:17:06.687 INFO:tasks.workunit.client.0.smithi032.stdout:3/539: rmdir d3/d65/d7d/d9c/dbc 39 2024-04-06T14:17:06.690 INFO:tasks.workunit.client.0.smithi032.stdout:3/540: fdatasync d3/fa 0 2024-04-06T14:17:06.691 INFO:tasks.workunit.client.0.smithi032.stdout:6/574: fdatasync d4/d5/d19/d25/f57 0 2024-04-06T14:17:06.693 INFO:tasks.workunit.client.0.smithi032.stdout:3/541: write d3/d65/f32 [2355842,118277] 0 2024-04-06T14:17:06.701 INFO:tasks.workunit.client.0.smithi032.stdout:4/617: truncate d0/df/d23/f32 2717372 0 2024-04-06T14:17:06.703 INFO:tasks.workunit.client.0.smithi032.stdout:1/648: write d0/f1e [5213410,94137] 0 2024-04-06T14:17:06.705 INFO:tasks.workunit.client.0.smithi032.stdout:4/618: creat d0/df/d23/d31/fd6 x:0 0 0 2024-04-06T14:17:06.707 INFO:tasks.workunit.client.0.smithi032.stdout:3/542: truncate d3/d16/f9a 1167054 0 2024-04-06T14:17:06.707 INFO:tasks.workunit.client.0.smithi032.stdout:6/575: truncate d4/d5/d18/f31 1642598 0 2024-04-06T14:17:06.708 INFO:tasks.workunit.client.0.smithi032.stdout:1/649: write d0/d10/f8d [119977,10224] 0 2024-04-06T14:17:06.709 INFO:tasks.workunit.client.0.smithi032.stdout:8/520: dwrite d3/d8/d3a/f4c [0,4194304] 0 2024-04-06T14:17:06.711 INFO:tasks.workunit.client.0.smithi032.stdout:4/619: write d0/d82/d4b/fbd [541034,121225] 0 2024-04-06T14:17:06.711 INFO:tasks.workunit.client.0.smithi032.stdout:5/554: dread fe [0,4194304] 0 2024-04-06T14:17:06.711 INFO:tasks.workunit.client.0.smithi032.stdout:3/543: readlink d3/d16/d63/l8a 0 2024-04-06T14:17:06.712 INFO:tasks.workunit.client.0.smithi032.stdout:3/544: stat d3/f37 0 2024-04-06T14:17:06.714 INFO:tasks.workunit.client.0.smithi032.stdout:5/555: truncate d16/fab 160364 0 2024-04-06T14:17:06.715 INFO:tasks.workunit.client.0.smithi032.stdout:3/545: rmdir d3/d16 39 2024-04-06T14:17:06.716 INFO:tasks.workunit.client.0.smithi032.stdout:8/521: write d3/d8/d12/d2f/d4a/d60/d82/fa6 [261349,2004] 0 2024-04-06T14:17:06.721 INFO:tasks.workunit.client.0.smithi032.stdout:8/522: write d3/d8/d7/d1a/f78 [1480622,39165] 0 2024-04-06T14:17:06.723 INFO:tasks.workunit.client.0.smithi032.stdout:4/620: truncate d0/df/d1f/d2e/fa5 1121514 0 2024-04-06T14:17:06.725 INFO:tasks.workunit.client.0.smithi032.stdout:6/576: dwrite d4/d5/d35/f39 [0,4194304] 0 2024-04-06T14:17:06.726 INFO:tasks.workunit.client.0.smithi032.stdout:8/523: write d3/d8/d3a/f56 [408160,2131] 0 2024-04-06T14:17:06.729 INFO:tasks.workunit.client.0.smithi032.stdout:1/650: dread d0/da/d3e/f6a [0,4194304] 0 2024-04-06T14:17:06.729 INFO:tasks.workunit.client.0.smithi032.stdout:8/524: readlink d3/d8/d3a/l92 0 2024-04-06T14:17:06.729 INFO:tasks.workunit.client.0.smithi032.stdout:3/546: rename d3/d65/d7d/d9c/dad/db1/d3c/f90 to d3/fd6 0 2024-04-06T14:17:06.730 INFO:tasks.workunit.client.0.smithi032.stdout:6/577: write d4/d5/dc5/d82/d9d/fbd [665240,84670] 0 2024-04-06T14:17:06.731 INFO:tasks.workunit.client.0.smithi032.stdout:5/556: link d16/d18/d33/d45/d52/d39/fbf d16/d58/d54/d74/d8f/d95/fc2 0 2024-04-06T14:17:06.732 INFO:tasks.workunit.client.0.smithi032.stdout:4/621: read d0/df/d2b/d43/fa6 [25888,72136] 0 2024-04-06T14:17:06.733 INFO:tasks.workunit.client.0.smithi032.stdout:4/622: stat d0/df/d23/f4a 0 2024-04-06T14:17:06.733 INFO:tasks.workunit.client.0.smithi032.stdout:5/557: chown d16/d18/d33/d45/d52/f9e 49467 1 2024-04-06T14:17:06.735 INFO:tasks.workunit.client.1.smithi086.stdout:8/448: fdatasync d0/d3b/d79/f9c 0 2024-04-06T14:17:06.737 INFO:tasks.workunit.client.1.smithi086.stdout:6/355: rmdir d6/d13/d5f 39 2024-04-06T14:17:06.737 INFO:tasks.workunit.client.1.smithi086.stdout:8/449: mkdir d0/d71/d16/d52/d5f/d9e 0 2024-04-06T14:17:06.738 INFO:tasks.workunit.client.1.smithi086.stdout:6/356: chown d6/d9/db/l2e 661 1 2024-04-06T14:17:06.739 INFO:tasks.workunit.client.1.smithi086.stdout:8/450: write d0/d71/d16/f6c [287503,126345] 0 2024-04-06T14:17:06.743 INFO:tasks.workunit.client.0.smithi032.stdout:1/651: mknod d0/d9/d1f/cd1 0 2024-04-06T14:17:06.743 INFO:tasks.workunit.client.0.smithi032.stdout:3/547: read d3/d16/d63/fa1 [2791739,82288] 0 2024-04-06T14:17:06.745 INFO:tasks.workunit.client.1.smithi086.stdout:6/357: mknod d6/d13/d5f/c7c 0 2024-04-06T14:17:06.745 INFO:tasks.workunit.client.0.smithi032.stdout:6/578: creat d4/d5/dbf/fcd x:0 0 0 2024-04-06T14:17:06.746 INFO:tasks.workunit.client.1.smithi086.stdout:2/355: write d0/d20/d27/d37/d52/f5c [2158719,100138] 0 2024-04-06T14:17:06.747 INFO:tasks.workunit.client.1.smithi086.stdout:2/356: fdatasync d0/fd 0 2024-04-06T14:17:06.748 INFO:tasks.workunit.client.0.smithi032.stdout:1/652: creat d0/d10/d16/fd2 x:0 0 0 2024-04-06T14:17:06.749 INFO:tasks.workunit.client.0.smithi032.stdout:4/623: creat d0/df/d23/d31/d50/d57/d8a/fd7 x:0 0 0 2024-04-06T14:17:06.750 INFO:tasks.workunit.client.0.smithi032.stdout:6/579: rmdir d4/d5/d28/d54 39 2024-04-06T14:17:06.751 INFO:tasks.workunit.client.0.smithi032.stdout:4/624: chown d0/df/d23/f3c 12085 1 2024-04-06T14:17:06.752 INFO:tasks.workunit.client.1.smithi086.stdout:2/357: mkdir d0/d20/d80 0 2024-04-06T14:17:06.752 INFO:tasks.workunit.client.1.smithi086.stdout:6/358: dread d6/d13/d1a/d22/f6e [0,4194304] 0 2024-04-06T14:17:06.754 INFO:tasks.workunit.client.0.smithi032.stdout:1/653: truncate d0/da/d3e/f6a 3261613 0 2024-04-06T14:17:06.755 INFO:tasks.workunit.client.0.smithi032.stdout:1/654: truncate d0/faf 56534 0 2024-04-06T14:17:06.755 INFO:tasks.workunit.client.1.smithi086.stdout:6/359: symlink d6/d13/d42/d52/l7d 0 2024-04-06T14:17:06.756 INFO:tasks.workunit.client.1.smithi086.stdout:2/358: getdents d0/df/d5e 0 2024-04-06T14:17:06.757 INFO:tasks.workunit.client.1.smithi086.stdout:5/353: dread d0/d19/f26 [0,4194304] 0 2024-04-06T14:17:06.758 INFO:tasks.workunit.client.0.smithi032.stdout:4/625: creat d0/df/d23/d31/fd8 x:0 0 0 2024-04-06T14:17:06.759 INFO:tasks.workunit.client.0.smithi032.stdout:1/655: creat d0/d41/d5b/d96/fd3 x:0 0 0 2024-04-06T14:17:06.759 INFO:tasks.workunit.client.1.smithi086.stdout:2/359: creat d0/d20/d27/d2b/d35/d36/d4d/d6f/f81 x:0 0 0 2024-04-06T14:17:06.761 INFO:tasks.workunit.client.1.smithi086.stdout:6/360: rename d6/d13/l3d to d6/d13/d1a/d22/l7e 0 2024-04-06T14:17:06.762 INFO:tasks.workunit.client.1.smithi086.stdout:2/360: mkdir d0/d20/d82 0 2024-04-06T14:17:06.763 INFO:tasks.workunit.client.1.smithi086.stdout:2/361: fsync d0/df/f21 0 2024-04-06T14:17:06.763 INFO:tasks.workunit.client.0.smithi032.stdout:5/558: dread d16/f2c [0,4194304] 0 2024-04-06T14:17:06.764 INFO:tasks.workunit.client.0.smithi032.stdout:1/656: link d0/d9/fbe d0/d41/d59/fd4 0 2024-04-06T14:17:06.765 INFO:tasks.workunit.client.1.smithi086.stdout:6/361: truncate d6/d13/d1a/d64/f71 898794 0 2024-04-06T14:17:06.765 INFO:tasks.workunit.client.1.smithi086.stdout:2/362: creat d0/d20/d27/d37/f83 x:0 0 0 2024-04-06T14:17:06.766 INFO:tasks.workunit.client.0.smithi032.stdout:4/626: dwrite d0/df/d1f/d42/d45/f8c [0,4194304] 0 2024-04-06T14:17:06.767 INFO:tasks.workunit.client.1.smithi086.stdout:8/451: dread d0/db/f5b [0,4194304] 0 2024-04-06T14:17:06.767 INFO:tasks.workunit.client.1.smithi086.stdout:5/354: dwrite d0/d16/d18/ff [0,4194304] 0 2024-04-06T14:17:06.767 INFO:tasks.workunit.client.1.smithi086.stdout:2/363: chown d0/df/d48/d53 41935 1 2024-04-06T14:17:06.768 INFO:tasks.workunit.client.1.smithi086.stdout:6/362: fdatasync d6/d13/d42/f7b 0 2024-04-06T14:17:06.768 INFO:tasks.workunit.client.0.smithi032.stdout:4/627: chown d0/df/d23/d31/l36 26279 1 2024-04-06T14:17:06.768 INFO:tasks.workunit.client.0.smithi032.stdout:5/559: truncate d16/d18/d33/d45/d52/f4e 1331560 0 2024-04-06T14:17:06.769 INFO:tasks.workunit.client.0.smithi032.stdout:1/657: mknod d0/d10/d16/d3f/d98/cd5 0 2024-04-06T14:17:06.769 INFO:tasks.workunit.client.1.smithi086.stdout:6/363: stat d6/f7 0 2024-04-06T14:17:06.769 INFO:tasks.workunit.client.1.smithi086.stdout:8/452: chown d0/db/f5e 2 1 2024-04-06T14:17:06.769 INFO:tasks.workunit.client.1.smithi086.stdout:2/364: chown d0/d20/d27/d2b/d35/d36/d4a 5146 1 2024-04-06T14:17:06.770 INFO:tasks.workunit.client.1.smithi086.stdout:5/355: stat d0/d16/d18/d21/d23/d28 0 2024-04-06T14:17:06.770 INFO:tasks.workunit.client.0.smithi032.stdout:4/628: write d0/df/d23/d31/d50/d57/d8a/f65 [4338094,60612] 0 2024-04-06T14:17:06.772 INFO:tasks.workunit.client.0.smithi032.stdout:5/560: mknod d16/d18/cc3 0 2024-04-06T14:17:06.774 INFO:tasks.workunit.client.0.smithi032.stdout:1/658: rename d0/da/f45 to d0/d9/d1f/d7c/d8f/daa/fd6 0 2024-04-06T14:17:06.777 INFO:tasks.workunit.client.1.smithi086.stdout:6/364: unlink d6/d9/c48 0 2024-04-06T14:17:06.778 INFO:tasks.workunit.client.1.smithi086.stdout:5/356: mknod d0/d16/d18/d21/c69 0 2024-04-06T14:17:06.779 INFO:tasks.workunit.client.0.smithi032.stdout:4/629: creat d0/df/d2b/fd9 x:0 0 0 2024-04-06T14:17:06.779 INFO:tasks.workunit.client.0.smithi032.stdout:1/659: rmdir d0/d10/d66 39 2024-04-06T14:17:06.779 INFO:tasks.workunit.client.0.smithi032.stdout:4/630: chown d0/d82/d4b 1825 1 2024-04-06T14:17:06.780 INFO:tasks.workunit.client.1.smithi086.stdout:5/357: write d0/d16/d18/d2a/d50/f5f [482845,126626] 0 2024-04-06T14:17:06.780 INFO:tasks.workunit.client.1.smithi086.stdout:8/453: creat d0/d71/f9f x:0 0 0 2024-04-06T14:17:06.781 INFO:tasks.workunit.client.0.smithi032.stdout:1/660: fdatasync d0/d10/d16/d3f/fb9 0 2024-04-06T14:17:06.782 INFO:tasks.workunit.client.0.smithi032.stdout:4/631: symlink d0/df/d1f/lda 0 2024-04-06T14:17:06.783 INFO:tasks.workunit.client.1.smithi086.stdout:8/454: creat d0/d71/d16/d39/d90/fa0 x:0 0 0 2024-04-06T14:17:06.783 INFO:tasks.workunit.client.0.smithi032.stdout:5/561: dwrite d16/d18/d33/d45/d52/d39/fb6 [0,4194304] 0 2024-04-06T14:17:06.785 INFO:tasks.workunit.client.0.smithi032.stdout:1/661: rmdir d0/d9/d94 39 2024-04-06T14:17:06.785 INFO:tasks.workunit.client.1.smithi086.stdout:8/455: rmdir d0/d71/d22/d43/d4d/d62 39 2024-04-06T14:17:06.786 INFO:tasks.workunit.client.1.smithi086.stdout:5/358: link d0/d17/d2d/c33 d0/d16/d18/c6a 0 2024-04-06T14:17:06.786 INFO:tasks.workunit.client.0.smithi032.stdout:5/562: write d16/d18/d33/d45/d52/d39/d86/f93 [758901,18479] 0 2024-04-06T14:17:06.789 INFO:tasks.workunit.client.0.smithi032.stdout:4/632: unlink d0/df/c2a 0 2024-04-06T14:17:06.791 INFO:tasks.workunit.client.0.smithi032.stdout:5/563: symlink d16/d58/d54/d94/lc4 0 2024-04-06T14:17:06.797 INFO:tasks.workunit.client.1.smithi086.stdout:8/456: link d0/d71/d16/f41 d0/d71/d22/d43/d6d/d75/d7d/fa1 0 2024-04-06T14:17:06.800 INFO:tasks.workunit.client.1.smithi086.stdout:5/359: dwrite d0/d16/f4e [0,4194304] 0 2024-04-06T14:17:06.802 INFO:tasks.workunit.client.0.smithi032.stdout:4/633: dread d0/df/d23/f4a [0,4194304] 0 2024-04-06T14:17:06.802 INFO:tasks.workunit.client.1.smithi086.stdout:5/360: mknod d0/d16/d18/d2a/d50/c6b 0 2024-04-06T14:17:06.803 INFO:tasks.workunit.client.0.smithi032.stdout:4/634: fsync d0/df/d23/f40 0 2024-04-06T14:17:06.803 INFO:tasks.workunit.client.1.smithi086.stdout:8/457: link d0/db/l45 d0/d71/la2 0 2024-04-06T14:17:06.805 INFO:tasks.workunit.client.1.smithi086.stdout:8/458: read - d0/d3b/f46 zero size 2024-04-06T14:17:06.806 INFO:tasks.workunit.client.1.smithi086.stdout:8/459: dread - d0/d13/f7c zero size 2024-04-06T14:17:06.807 INFO:tasks.workunit.client.0.smithi032.stdout:4/635: mknod d0/d82/daf/cdb 0 2024-04-06T14:17:06.808 INFO:tasks.workunit.client.1.smithi086.stdout:8/460: mkdir d0/d71/d22/d43/d4d/da3 0 2024-04-06T14:17:06.814 INFO:tasks.workunit.client.0.smithi032.stdout:4/636: dwrite d0/df/d2b/fa8 [0,4194304] 0 2024-04-06T14:17:06.816 INFO:tasks.workunit.client.0.smithi032.stdout:4/637: dread - d0/df/d23/fc7 zero size 2024-04-06T14:17:06.819 INFO:tasks.workunit.client.1.smithi086.stdout:0/366: sync 2024-04-06T14:17:06.819 INFO:tasks.workunit.client.1.smithi086.stdout:7/315: sync 2024-04-06T14:17:06.819 INFO:tasks.workunit.client.0.smithi032.stdout:4/638: unlink d0/df/fbc 0 2024-04-06T14:17:06.821 INFO:tasks.workunit.client.1.smithi086.stdout:7/316: fsync d7/d10/d11/d1c/d24/d4d/d5c/d66/f67 0 2024-04-06T14:17:06.822 INFO:tasks.workunit.client.1.smithi086.stdout:0/367: mknod d0/d1/d1b/d44/c75 0 2024-04-06T14:17:06.823 INFO:tasks.workunit.client.1.smithi086.stdout:0/368: fdatasync d0/d15/d4e/f42 0 2024-04-06T14:17:06.826 INFO:tasks.workunit.client.1.smithi086.stdout:7/317: getdents d7/d10/d11/d1c/d24/d4d/d5c 0 2024-04-06T14:17:06.827 INFO:tasks.workunit.client.0.smithi032.stdout:8/525: write d3/d8/f6d [2557485,115908] 0 2024-04-06T14:17:06.827 INFO:tasks.workunit.client.1.smithi086.stdout:0/369: creat d0/d1/f76 x:0 0 0 2024-04-06T14:17:06.829 INFO:tasks.workunit.client.1.smithi086.stdout:0/370: chown d0/d15/d4e/f45 1 1 2024-04-06T14:17:06.830 INFO:tasks.workunit.client.0.smithi032.stdout:5/564: dread d16/d18/d33/d35/d88/fad [0,4194304] 0 2024-04-06T14:17:06.830 INFO:tasks.workunit.client.0.smithi032.stdout:8/526: fdatasync d3/d8/f35 0 2024-04-06T14:17:06.830 INFO:tasks.workunit.client.1.smithi086.stdout:7/318: write d7/fb [586833,24365] 0 2024-04-06T14:17:06.831 INFO:tasks.workunit.client.1.smithi086.stdout:0/371: write d0/d15/f4f [627761,70080] 0 2024-04-06T14:17:06.832 INFO:tasks.workunit.client.0.smithi032.stdout:3/548: truncate d3/d16/f5c 1485945 0 2024-04-06T14:17:06.833 INFO:tasks.workunit.client.1.smithi086.stdout:7/319: stat d7/d10/l20 0 2024-04-06T14:17:06.835 INFO:tasks.workunit.client.0.smithi032.stdout:6/580: truncate d4/d5/f6a 6267084 0 2024-04-06T14:17:06.836 INFO:tasks.workunit.client.1.smithi086.stdout:0/372: dread d0/d15/f6a [0,4194304] 0 2024-04-06T14:17:06.838 INFO:tasks.workunit.client.1.smithi086.stdout:7/320: creat d7/d10/d11/d1c/d24/d2c/d69/f71 x:0 0 0 2024-04-06T14:17:06.839 INFO:tasks.workunit.client.0.smithi032.stdout:6/581: creat d4/d5/d35/fce x:0 0 0 2024-04-06T14:17:06.842 INFO:tasks.workunit.client.1.smithi086.stdout:8/461: rmdir d0/d71/d22/d43/d6d 39 2024-04-06T14:17:06.844 INFO:tasks.workunit.client.0.smithi032.stdout:8/527: dwrite d3/d8/d7/fbc [0,4194304] 0 2024-04-06T14:17:06.846 INFO:tasks.workunit.client.1.smithi086.stdout:8/462: rename d0/d3b/d79/f9c to d0/d3b/fa4 0 2024-04-06T14:17:06.847 INFO:tasks.workunit.client.0.smithi032.stdout:1/662: mkdir d0/da/d3e/d75/dd7 0 2024-04-06T14:17:06.849 INFO:tasks.workunit.client.1.smithi086.stdout:8/463: write d0/f5 [1424903,1060] 0 2024-04-06T14:17:06.851 INFO:tasks.workunit.client.1.smithi086.stdout:0/373: dread d0/d1/d4/f43 [0,4194304] 0 2024-04-06T14:17:06.853 INFO:tasks.workunit.client.0.smithi032.stdout:3/549: dread d3/d65/f47 [0,4194304] 0 2024-04-06T14:17:06.854 INFO:tasks.workunit.client.0.smithi032.stdout:6/582: dwrite d4/d5/d28/d65/fa3 [0,4194304] 0 2024-04-06T14:17:06.854 INFO:tasks.workunit.client.1.smithi086.stdout:8/464: write d0/d71/d22/d43/d7f/f88 [693739,44631] 0 2024-04-06T14:17:06.854 INFO:tasks.workunit.client.1.smithi086.stdout:0/374: chown d0/l71 3539 1 2024-04-06T14:17:06.854 INFO:tasks.workunit.client.1.smithi086.stdout:0/375: chown d0/d1 65 1 2024-04-06T14:17:06.855 INFO:tasks.workunit.client.0.smithi032.stdout:3/550: chown d3/d65/cb6 31 1 2024-04-06T14:17:06.855 INFO:tasks.workunit.client.0.smithi032.stdout:8/528: rmdir d3/d8/d7/d9e 0 2024-04-06T14:17:06.856 INFO:tasks.workunit.client.1.smithi086.stdout:8/465: symlink d0/d71/d16/d39/la5 0 2024-04-06T14:17:06.856 INFO:tasks.workunit.client.0.smithi032.stdout:3/551: fdatasync d3/f4 0 2024-04-06T14:17:06.858 INFO:tasks.workunit.client.0.smithi032.stdout:6/583: stat d4/d5/d14 0 2024-04-06T14:17:06.858 INFO:tasks.workunit.client.0.smithi032.stdout:8/529: write d3/d8/f99 [1255107,105311] 0 2024-04-06T14:17:06.859 INFO:tasks.workunit.client.1.smithi086.stdout:0/376: link d0/d1/d2/l3 d0/d1/d1b/l77 0 2024-04-06T14:17:06.860 INFO:tasks.workunit.client.0.smithi032.stdout:3/552: fsync d3/d16/f2c 0 2024-04-06T14:17:06.860 INFO:tasks.workunit.client.0.smithi032.stdout:3/553: stat d3/d65/d89/f97 0 2024-04-06T14:17:06.867 INFO:tasks.workunit.client.0.smithi032.stdout:8/530: creat d3/d8/d3a/d4b/fbe x:0 0 0 2024-04-06T14:17:06.867 INFO:tasks.workunit.client.0.smithi032.stdout:6/584: creat d4/d5/d8/d2b/d77/fcf x:0 0 0 2024-04-06T14:17:06.868 INFO:tasks.workunit.client.1.smithi086.stdout:0/377: rename d0/d1/d4/de/c10 to d0/d15/d4e/d48/c78 0 2024-04-06T14:17:06.868 INFO:tasks.workunit.client.1.smithi086.stdout:0/378: write d0/d1/d4/de/f2e [993949,90544] 0 2024-04-06T14:17:06.868 INFO:tasks.workunit.client.1.smithi086.stdout:8/466: truncate d0/d71/d22/d43/d7f/f88 563898 0 2024-04-06T14:17:06.869 INFO:tasks.workunit.client.1.smithi086.stdout:0/379: chown d0/d1/d25/f70 6235 1 2024-04-06T14:17:06.869 INFO:tasks.workunit.client.0.smithi032.stdout:6/585: unlink d4/d5/c12 0 2024-04-06T14:17:06.870 INFO:tasks.workunit.client.1.smithi086.stdout:0/380: write d0/d1/d4/d7/f14 [4794383,58283] 0 2024-04-06T14:17:06.872 INFO:tasks.workunit.client.1.smithi086.stdout:7/321: dread d7/d1d/f3e [0,4194304] 0 2024-04-06T14:17:06.872 INFO:tasks.workunit.client.0.smithi032.stdout:5/565: dread d16/d18/d33/d45/d52/d39/f47 [0,4194304] 0 2024-04-06T14:17:06.873 INFO:tasks.workunit.client.0.smithi032.stdout:8/531: symlink d3/d8/d12/d2f/lbf 0 2024-04-06T14:17:06.873 INFO:tasks.workunit.client.1.smithi086.stdout:0/381: rmdir d0/d1/d25 39 2024-04-06T14:17:06.875 INFO:tasks.workunit.client.0.smithi032.stdout:3/554: getdents d3/d65 0 2024-04-06T14:17:06.876 INFO:tasks.workunit.client.1.smithi086.stdout:0/382: mkdir d0/d1/d2/d1a/d79 0 2024-04-06T14:17:06.877 INFO:tasks.workunit.client.1.smithi086.stdout:0/383: chown d0/d15 1100 1 2024-04-06T14:17:06.878 INFO:tasks.workunit.client.1.smithi086.stdout:8/467: write d0/d71/d22/f83 [2469289,26186] 0 2024-04-06T14:17:06.878 INFO:tasks.workunit.client.0.smithi032.stdout:8/532: rename d3/d8/d7/d14/l21 to d3/d8/lc0 0 2024-04-06T14:17:06.878 INFO:tasks.workunit.client.1.smithi086.stdout:0/384: write d0/d15/f4c [43956,97482] 0 2024-04-06T14:17:06.881 INFO:tasks.workunit.client.0.smithi032.stdout:5/566: mkdir d16/d18/d33/d35/d88/dbd/dc5 0 2024-04-06T14:17:06.881 INFO:tasks.workunit.client.1.smithi086.stdout:7/322: getdents d7/d10/d11/d1c/d24/d4d 0 2024-04-06T14:17:06.882 INFO:tasks.workunit.client.0.smithi032.stdout:5/567: chown d16/d18/d33/d45/d52/d39/f3e 214864 1 2024-04-06T14:17:06.884 INFO:tasks.workunit.client.0.smithi032.stdout:8/533: creat d3/d8/d3a/d7b/fc1 x:0 0 0 2024-04-06T14:17:06.888 INFO:tasks.workunit.client.1.smithi086.stdout:0/385: dwrite d0/d1/d4/de/f62 [0,4194304] 0 2024-04-06T14:17:06.890 INFO:tasks.workunit.client.1.smithi086.stdout:7/323: getdents d7/d10 0 2024-04-06T14:17:06.891 INFO:tasks.workunit.client.0.smithi032.stdout:3/555: creat d3/fd7 x:0 0 0 2024-04-06T14:17:06.891 INFO:tasks.workunit.client.1.smithi086.stdout:0/386: truncate d0/d1/d2/d1a/f63 1874935 0 2024-04-06T14:17:06.891 INFO:tasks.workunit.client.0.smithi032.stdout:6/586: link d4/d5/d83/dad/cb8 d4/d5/d8/d4b/d7c/cd0 0 2024-04-06T14:17:06.893 INFO:tasks.workunit.client.0.smithi032.stdout:1/663: truncate d0/d10/d16/fa6 616629 0 2024-04-06T14:17:06.895 INFO:tasks.workunit.client.1.smithi086.stdout:0/387: mkdir d0/d15/d26/d33/d7a 0 2024-04-06T14:17:06.896 INFO:tasks.workunit.client.0.smithi032.stdout:8/534: dwrite d3/d8/d12/d2f/d4a/f8b [0,4194304] 0 2024-04-06T14:17:06.896 INFO:tasks.workunit.client.0.smithi032.stdout:6/587: write d4/d5/dbf/d61/f66 [688669,33072] 0 2024-04-06T14:17:06.896 INFO:tasks.workunit.client.1.smithi086.stdout:8/468: dwrite d0/f34 [0,4194304] 0 2024-04-06T14:17:06.897 INFO:tasks.workunit.client.1.smithi086.stdout:0/388: chown d0/d1/d4/c21 6 1 2024-04-06T14:17:06.897 INFO:tasks.workunit.client.0.smithi032.stdout:3/556: fdatasync d3/f7e 0 2024-04-06T14:17:06.898 INFO:tasks.workunit.client.1.smithi086.stdout:0/389: stat d0/d1/d1b/d46/d52 0 2024-04-06T14:17:06.899 INFO:tasks.workunit.client.1.smithi086.stdout:8/469: creat d0/d3b/fa6 x:0 0 0 2024-04-06T14:17:06.900 INFO:tasks.workunit.client.0.smithi032.stdout:5/568: dread f1 [4194304,4194304] 0 2024-04-06T14:17:06.900 INFO:tasks.workunit.client.0.smithi032.stdout:8/535: chown d3/d8/f87 88 1 2024-04-06T14:17:06.900 INFO:tasks.workunit.client.1.smithi086.stdout:0/390: write d0/d1/f13 [4953120,64606] 0 2024-04-06T14:17:06.908 INFO:tasks.workunit.client.1.smithi086.stdout:0/391: fsync d0/d1/d25/f70 0 2024-04-06T14:17:06.908 INFO:tasks.workunit.client.0.smithi032.stdout:5/569: readlink d16/d18/d33/d35/la7 0 2024-04-06T14:17:06.909 INFO:tasks.workunit.client.1.smithi086.stdout:8/470: dwrite d0/db/f5e [0,4194304] 0 2024-04-06T14:17:06.909 INFO:tasks.workunit.client.0.smithi032.stdout:3/557: symlink d3/d65/d7d/d9c/ld8 0 2024-04-06T14:17:06.909 INFO:tasks.workunit.client.0.smithi032.stdout:1/664: dwrite d0/d10/d16/f51 [0,4194304] 0 2024-04-06T14:17:06.911 INFO:tasks.workunit.client.0.smithi032.stdout:8/536: symlink d3/d8/d7/d8a/lc2 0 2024-04-06T14:17:06.912 INFO:tasks.workunit.client.0.smithi032.stdout:1/665: chown d0/d9/d1f/d7c/f95 4642 1 2024-04-06T14:17:06.914 INFO:tasks.workunit.client.0.smithi032.stdout:8/537: fdatasync d3/d8/d12/d2f/d4a/d60/d82/fa1 0 2024-04-06T14:17:06.917 INFO:tasks.workunit.client.0.smithi032.stdout:5/570: symlink d16/d58/d54/d94/lc6 0 2024-04-06T14:17:06.920 INFO:tasks.workunit.client.0.smithi032.stdout:8/538: chown d3/d8/d3a/d7b/d8c 0 1 2024-04-06T14:17:06.920 INFO:tasks.workunit.client.1.smithi086.stdout:0/392: link d0/f17 d0/d1/d2/d1a/f7b 0 2024-04-06T14:17:06.921 INFO:tasks.workunit.client.1.smithi086.stdout:0/393: stat d0/d1/d1b/d44/d5d/d74 0 2024-04-06T14:17:06.925 INFO:tasks.workunit.client.0.smithi032.stdout:1/666: dwrite d0/da/d3e/d75/f8a [0,4194304] 0 2024-04-06T14:17:06.926 INFO:tasks.workunit.client.0.smithi032.stdout:5/571: dread f9 [0,4194304] 0 2024-04-06T14:17:06.926 INFO:tasks.workunit.client.0.smithi032.stdout:8/539: mknod d3/d8/d12/d2f/d4a/d60/d75/cc3 0 2024-04-06T14:17:06.927 INFO:tasks.workunit.client.0.smithi032.stdout:1/667: dread d0/faf [0,4194304] 0 2024-04-06T14:17:06.928 INFO:tasks.workunit.client.0.smithi032.stdout:8/540: fsync d3/d8/d12/d2f/d4a/d60/d90/fb6 0 2024-04-06T14:17:06.929 INFO:tasks.workunit.client.0.smithi032.stdout:1/668: chown d0/da/d3e/d75/l88 231173982 1 2024-04-06T14:17:06.936 INFO:tasks.workunit.client.0.smithi032.stdout:5/572: dwrite d16/d18/d33/d35/d4d/f9c [0,4194304] 0 2024-04-06T14:17:06.942 INFO:tasks.workunit.client.0.smithi032.stdout:8/541: dread d3/d8/d12/d2f/f62 [0,4194304] 0 2024-04-06T14:17:06.945 INFO:tasks.workunit.client.0.smithi032.stdout:5/573: dwrite d16/d18/d33/d45/d52/d39/d86/fbe [0,4194304] 0 2024-04-06T14:17:06.953 INFO:tasks.workunit.client.1.smithi086.stdout:0/394: dread d0/d1/d1b/d44/f5a [0,4194304] 0 2024-04-06T14:17:06.953 INFO:tasks.workunit.client.0.smithi032.stdout:5/574: dwrite d16/d58/d54/d94/db2/fb8 [0,4194304] 0 2024-04-06T14:17:06.954 INFO:tasks.workunit.client.0.smithi032.stdout:3/558: dread d3/f21 [0,4194304] 0 2024-04-06T14:17:06.957 INFO:tasks.workunit.client.1.smithi086.stdout:0/395: read d0/ff [10148008,118113] 0 2024-04-06T14:17:06.960 INFO:tasks.workunit.client.1.smithi086.stdout:0/396: creat d0/d1/d1b/d73/f7c x:0 0 0 2024-04-06T14:17:06.962 INFO:tasks.workunit.client.1.smithi086.stdout:0/397: dread - d0/d1/d1b/d73/f7c zero size 2024-04-06T14:17:06.963 INFO:tasks.workunit.client.0.smithi032.stdout:3/559: dwrite d3/d65/d7d/d9c/dad/db1/d3c/f5f [4194304,4194304] 0 2024-04-06T14:17:06.965 INFO:tasks.workunit.client.1.smithi086.stdout:0/398: getdents d0/d1/d2/d1a/d51 0 2024-04-06T14:17:06.967 INFO:tasks.workunit.client.1.smithi086.stdout:0/399: mknod d0/d15/c7d 0 2024-04-06T14:17:06.968 INFO:tasks.workunit.client.1.smithi086.stdout:0/400: readlink d0/d1/d4/d7/l36 0 2024-04-06T14:17:06.970 INFO:tasks.workunit.client.1.smithi086.stdout:0/401: chown d0/d1/d1b/l77 609 1 2024-04-06T14:17:06.971 INFO:tasks.workunit.client.0.smithi032.stdout:8/542: dwrite d3/d8/d12/f4d [4194304,4194304] 0 2024-04-06T14:17:06.972 INFO:tasks.workunit.client.0.smithi032.stdout:8/543: chown d3/d8/d7/d8a/lbd 48279015 1 2024-04-06T14:17:06.973 INFO:tasks.workunit.client.1.smithi086.stdout:0/402: fsync d0/d15/d26/f2c 0 2024-04-06T14:17:06.974 INFO:tasks.workunit.client.0.smithi032.stdout:3/560: link d3/d65/l5b d3/d65/d89/ld9 0 2024-04-06T14:17:06.975 INFO:tasks.workunit.client.0.smithi032.stdout:8/544: mkdir d3/d8/d12/d83/dc4 0 2024-04-06T14:17:06.976 INFO:tasks.workunit.client.0.smithi032.stdout:3/561: stat d3/d65/d7d/d9c/db8/lce 0 2024-04-06T14:17:06.976 INFO:tasks.workunit.client.1.smithi086.stdout:0/403: symlink d0/d15/d4e/l7e 0 2024-04-06T14:17:06.977 INFO:tasks.workunit.client.0.smithi032.stdout:8/545: chown d3/d8/d12/l23 139803 1 2024-04-06T14:17:06.981 INFO:tasks.workunit.client.1.smithi086.stdout:0/404: link d0/f16 d0/d1/d1b/d46/d52/f7f 0 2024-04-06T14:17:06.981 INFO:tasks.workunit.client.1.smithi086.stdout:0/405: stat d0/d1/c53 0 2024-04-06T14:17:06.984 INFO:tasks.workunit.client.0.smithi032.stdout:3/562: write d3/d65/f52 [221235,105502] 0 2024-04-06T14:17:06.988 INFO:tasks.workunit.client.0.smithi032.stdout:7/597: sync 2024-04-06T14:17:06.989 INFO:tasks.workunit.client.0.smithi032.stdout:7/598: stat d16/d20/d63 0 2024-04-06T14:17:06.991 INFO:tasks.workunit.client.0.smithi032.stdout:7/599: dread - d16/d20/d26/d2d/d30/d97/fca zero size 2024-04-06T14:17:06.993 INFO:tasks.workunit.client.0.smithi032.stdout:3/563: dwrite d3/d65/d7d/d9c/dad/dc5/fcc [0,4194304] 0 2024-04-06T14:17:06.993 INFO:tasks.workunit.client.0.smithi032.stdout:8/546: dread d3/d8/d7/f16 [0,4194304] 0 2024-04-06T14:17:06.998 INFO:tasks.workunit.client.0.smithi032.stdout:3/564: rename d3/d65/d7d/lbe to d3/lda 0 2024-04-06T14:17:06.999 INFO:tasks.workunit.client.0.smithi032.stdout:8/547: rmdir d3/d8/d12/d83/dc4 0 2024-04-06T14:17:06.999 INFO:tasks.workunit.client.0.smithi032.stdout:7/600: dwrite fe [0,4194304] 0 2024-04-06T14:17:07.002 INFO:tasks.workunit.client.0.smithi032.stdout:8/548: write d3/d8/d7/d14/f9d [810065,110544] 0 2024-04-06T14:17:07.003 INFO:tasks.workunit.client.0.smithi032.stdout:3/565: creat d3/d65/d7d/dc6/fdb x:0 0 0 2024-04-06T14:17:07.004 INFO:tasks.workunit.client.0.smithi032.stdout:7/601: dread d16/d20/d63/f72 [0,4194304] 0 2024-04-06T14:17:07.006 INFO:tasks.workunit.client.0.smithi032.stdout:7/602: chown d16/d20/d26/d2d/l34 29 1 2024-04-06T14:17:07.007 INFO:tasks.workunit.client.0.smithi032.stdout:7/603: readlink d16/d20/d26/d5f/ld3 0 2024-04-06T14:17:07.007 INFO:tasks.workunit.client.0.smithi032.stdout:3/566: rename d3/fd6 to d3/d65/fdc 0 2024-04-06T14:17:07.009 INFO:tasks.workunit.client.0.smithi032.stdout:3/567: write d3/d65/d7d/d9c/dad/db1/d3c/f8f [957504,130851] 0 2024-04-06T14:17:07.011 INFO:tasks.workunit.client.0.smithi032.stdout:8/549: write d3/d8/d12/f19 [1668796,36107] 0 2024-04-06T14:17:07.013 INFO:tasks.workunit.client.0.smithi032.stdout:3/568: stat d3/d65/d89/c99 0 2024-04-06T14:17:07.017 INFO:tasks.workunit.client.0.smithi032.stdout:7/604: dwrite d16/d20/f28 [0,4194304] 0 2024-04-06T14:17:07.017 INFO:tasks.workunit.client.0.smithi032.stdout:3/569: mkdir d3/d16/ddd 0 2024-04-06T14:17:07.017 INFO:tasks.workunit.client.0.smithi032.stdout:8/550: mkdir d3/d8/d12/d2f/d4a/d60/d90/dc5 0 2024-04-06T14:17:07.019 INFO:tasks.workunit.client.0.smithi032.stdout:8/551: mknod d3/d8/d7/d8a/cc6 0 2024-04-06T14:17:07.019 INFO:tasks.workunit.client.0.smithi032.stdout:7/605: read d16/d20/d48/d6d/f89 [856480,123756] 0 2024-04-06T14:17:07.021 INFO:tasks.workunit.client.0.smithi032.stdout:8/552: mknod d3/cc7 0 2024-04-06T14:17:07.023 INFO:tasks.workunit.client.0.smithi032.stdout:7/606: creat d16/d20/d63/d9d/fd8 x:0 0 0 2024-04-06T14:17:07.024 INFO:tasks.workunit.client.0.smithi032.stdout:8/553: rmdir d3/d8/d3a/d7b 39 2024-04-06T14:17:07.026 INFO:tasks.workunit.client.0.smithi032.stdout:3/570: write d3/f7e [1079015,32019] 0 2024-04-06T14:17:07.032 INFO:tasks.workunit.client.1.smithi086.stdout:3/396: sync 2024-04-06T14:17:07.033 INFO:tasks.workunit.client.1.smithi086.stdout:1/363: sync 2024-04-06T14:17:07.033 INFO:tasks.workunit.client.1.smithi086.stdout:9/354: sync 2024-04-06T14:17:07.036 INFO:tasks.workunit.client.1.smithi086.stdout:9/355: mknod d5/c75 0 2024-04-06T14:17:07.038 INFO:tasks.workunit.client.1.smithi086.stdout:3/397: rename d5/d36/d49 to d5/d36/d5a/d6d/d72 0 2024-04-06T14:17:07.039 INFO:tasks.workunit.client.1.smithi086.stdout:3/398: write d5/d36/d5c/f68 [620127,127196] 0 2024-04-06T14:17:07.039 INFO:tasks.workunit.client.1.smithi086.stdout:1/364: link db/d11/d1f/f44 db/d13/f74 0 2024-04-06T14:17:07.040 INFO:tasks.workunit.client.1.smithi086.stdout:3/399: chown d5/f37 16277610 1 2024-04-06T14:17:07.041 INFO:tasks.workunit.client.1.smithi086.stdout:1/365: mkdir db/d11/d1a/d2a/d52/d75 0 2024-04-06T14:17:07.044 INFO:tasks.workunit.client.1.smithi086.stdout:3/400: symlink d5/d33/l73 0 2024-04-06T14:17:07.045 INFO:tasks.workunit.client.1.smithi086.stdout:1/366: mknod db/d11/d42/d53/c76 0 2024-04-06T14:17:07.046 INFO:tasks.workunit.client.1.smithi086.stdout:3/401: creat d5/d36/d56/f74 x:0 0 0 2024-04-06T14:17:07.046 INFO:tasks.workunit.client.1.smithi086.stdout:3/402: dread - d5/d36/d56/f74 zero size 2024-04-06T14:17:07.047 INFO:tasks.workunit.client.1.smithi086.stdout:1/367: rename db/l73 to db/d11/d1f/d5c/l77 0 2024-04-06T14:17:07.048 INFO:tasks.workunit.client.1.smithi086.stdout:3/403: creat d5/d36/d5a/d6d/d72/f75 x:0 0 0 2024-04-06T14:17:07.051 INFO:tasks.workunit.client.1.smithi086.stdout:1/368: rename db/d11/d1a/d2a/d52/d75 to db/d11/d1a/d2a/d52/d5b/d78 0 2024-04-06T14:17:07.051 INFO:tasks.workunit.client.1.smithi086.stdout:1/369: fdatasync db/d13/f54 0 2024-04-06T14:17:07.051 INFO:tasks.workunit.client.1.smithi086.stdout:3/404: mknod d5/d36/d5a/d6d/d72/c76 0 2024-04-06T14:17:07.053 INFO:tasks.workunit.client.1.smithi086.stdout:1/370: getdents db/d13/d71 0 2024-04-06T14:17:07.054 INFO:tasks.workunit.client.1.smithi086.stdout:3/405: creat d5/f77 x:0 0 0 2024-04-06T14:17:07.056 INFO:tasks.workunit.client.1.smithi086.stdout:3/406: creat d5/d33/f78 x:0 0 0 2024-04-06T14:17:07.056 INFO:tasks.workunit.client.1.smithi086.stdout:3/407: chown d5/d26/d4f/c41 0 1 2024-04-06T14:17:07.058 INFO:tasks.workunit.client.1.smithi086.stdout:3/408: creat d5/d36/d5a/d6d/d72/d58/f79 x:0 0 0 2024-04-06T14:17:07.058 INFO:tasks.workunit.client.1.smithi086.stdout:3/409: dread - d5/f77 zero size 2024-04-06T14:17:07.060 INFO:tasks.workunit.client.1.smithi086.stdout:3/410: unlink d5/d36/f54 0 2024-04-06T14:17:07.130 INFO:tasks.workunit.client.0.smithi032.stdout:3/571: symlink d3/d65/d7d/d9c/dbc/dcd/lde 0 2024-04-06T14:17:07.132 INFO:tasks.workunit.client.0.smithi032.stdout:7/607: dwrite fd [4194304,4194304] 0 2024-04-06T14:17:07.141 INFO:tasks.workunit.client.1.smithi086.stdout:3/411: dwrite d5/d33/f32 [0,4194304] 0 2024-04-06T14:17:07.146 INFO:tasks.workunit.client.1.smithi086.stdout:3/412: creat d5/d36/d5a/d6d/d72/f7a x:0 0 0 2024-04-06T14:17:07.146 INFO:tasks.workunit.client.1.smithi086.stdout:3/413: readlink d5/d26/l28 0 2024-04-06T14:17:07.146 INFO:tasks.workunit.client.1.smithi086.stdout:3/414: stat d5/df 0 2024-04-06T14:17:07.147 INFO:tasks.workunit.client.1.smithi086.stdout:3/415: stat d5/d36 0 2024-04-06T14:17:07.149 INFO:tasks.workunit.client.1.smithi086.stdout:3/416: creat d5/d36/d65/f7b x:0 0 0 2024-04-06T14:17:07.151 INFO:tasks.workunit.client.1.smithi086.stdout:3/417: truncate d5/d36/d5a/d6d/d72/d58/f79 611076 0 2024-04-06T14:17:07.152 INFO:tasks.workunit.client.1.smithi086.stdout:3/418: rmdir d5/d42/d59 39 2024-04-06T14:17:07.152 INFO:tasks.workunit.client.1.smithi086.stdout:4/270: sync 2024-04-06T14:17:07.152 INFO:tasks.workunit.client.1.smithi086.stdout:6/365: sync 2024-04-06T14:17:07.152 INFO:tasks.workunit.client.1.smithi086.stdout:2/365: sync 2024-04-06T14:17:07.152 INFO:tasks.workunit.client.1.smithi086.stdout:3/419: stat d5/d26/f3c 0 2024-04-06T14:17:07.153 INFO:tasks.workunit.client.1.smithi086.stdout:3/420: stat d5/cd 0 2024-04-06T14:17:07.153 INFO:tasks.workunit.client.1.smithi086.stdout:6/366: chown d6/f2b 532704515 1 2024-04-06T14:17:07.154 INFO:tasks.workunit.client.1.smithi086.stdout:2/366: stat d0/df/l32 0 2024-04-06T14:17:07.155 INFO:tasks.workunit.client.1.smithi086.stdout:3/421: fdatasync d5/d36/d5a/d6d/d72/f7a 0 2024-04-06T14:17:07.155 INFO:tasks.workunit.client.1.smithi086.stdout:4/271: chown d0/d15/d19/f1c 183601 1 2024-04-06T14:17:07.157 INFO:tasks.workunit.client.1.smithi086.stdout:4/272: write d0/d15/d19/f1c [2375871,44512] 0 2024-04-06T14:17:07.157 INFO:tasks.workunit.client.1.smithi086.stdout:6/367: creat d6/d13/d42/d52/f7f x:0 0 0 2024-04-06T14:17:07.159 INFO:tasks.workunit.client.1.smithi086.stdout:6/368: read d6/d13/d1a/f21 [1022040,109151] 0 2024-04-06T14:17:07.159 INFO:tasks.workunit.client.1.smithi086.stdout:3/422: mknod d5/d26/d4f/c7c 0 2024-04-06T14:17:07.162 INFO:tasks.workunit.client.1.smithi086.stdout:3/423: mkdir d5/d36/d7d 0 2024-04-06T14:17:07.162 INFO:tasks.workunit.client.1.smithi086.stdout:2/367: write d0/f1e [1229661,122681] 0 2024-04-06T14:17:07.164 INFO:tasks.workunit.client.1.smithi086.stdout:4/273: dwrite d0/f17 [0,4194304] 0 2024-04-06T14:17:07.166 INFO:tasks.workunit.client.1.smithi086.stdout:2/368: write d0/d20/d27/d2b/d35/d36/f40 [5159556,92035] 0 2024-04-06T14:17:07.166 INFO:tasks.workunit.client.1.smithi086.stdout:3/424: mknod d5/d36/d5c/c7e 0 2024-04-06T14:17:07.167 INFO:tasks.workunit.client.1.smithi086.stdout:4/274: read d0/f12 [128450,125158] 0 2024-04-06T14:17:07.174 INFO:tasks.workunit.client.1.smithi086.stdout:2/369: link d0/f12 d0/df/f84 0 2024-04-06T14:17:07.175 INFO:tasks.workunit.client.1.smithi086.stdout:2/370: stat d0/df/d48/c6c 0 2024-04-06T14:17:07.175 INFO:tasks.workunit.client.1.smithi086.stdout:2/371: readlink d0/l24 0 2024-04-06T14:17:07.177 INFO:tasks.workunit.client.1.smithi086.stdout:2/372: read d0/d20/d27/d2b/f65 [1930725,20506] 0 2024-04-06T14:17:07.178 INFO:tasks.workunit.client.1.smithi086.stdout:5/361: sync 2024-04-06T14:17:07.178 INFO:tasks.workunit.client.1.smithi086.stdout:0/406: sync 2024-04-06T14:17:07.178 INFO:tasks.workunit.client.1.smithi086.stdout:7/324: sync 2024-04-06T14:17:07.179 INFO:tasks.workunit.client.1.smithi086.stdout:8/471: sync 2024-04-06T14:17:07.179 INFO:tasks.workunit.client.1.smithi086.stdout:9/356: sync 2024-04-06T14:17:07.179 INFO:tasks.workunit.client.1.smithi086.stdout:4/275: dwrite d0/d2c/d30/d24/f31 [0,4194304] 0 2024-04-06T14:17:07.181 INFO:tasks.workunit.client.0.smithi032.stdout:3/572: mknod d3/d65/d7d/dc6/cdf 0 2024-04-06T14:17:07.182 INFO:tasks.workunit.client.1.smithi086.stdout:0/407: chown d0/d1/f76 6 1 2024-04-06T14:17:07.182 INFO:tasks.workunit.client.1.smithi086.stdout:4/276: write d0/f5 [1404563,18809] 0 2024-04-06T14:17:07.182 INFO:tasks.workunit.client.1.smithi086.stdout:6/369: dread d6/d20/f46 [0,4194304] 0 2024-04-06T14:17:07.183 INFO:tasks.workunit.client.0.smithi032.stdout:7/608: mknod d16/d20/d26/d2d/d30/d3e/cd9 0 2024-04-06T14:17:07.187 INFO:tasks.workunit.client.1.smithi086.stdout:7/325: dread - d7/d10/d11/d1c/d24/d4d/d5c/d66/f67 zero size 2024-04-06T14:17:07.188 INFO:tasks.workunit.client.1.smithi086.stdout:7/326: dread - d7/d10/d11/d1c/f58 zero size 2024-04-06T14:17:07.192 INFO:tasks.workunit.client.0.smithi032.stdout:7/609: dread d16/d20/d26/d5f/fc7 [0,4194304] 0 2024-04-06T14:17:07.201 INFO:tasks.workunit.client.0.smithi032.stdout:3/573: dread d3/faf [0,4194304] 0 2024-04-06T14:17:07.203 INFO:tasks.workunit.client.0.smithi032.stdout:7/610: link d16/d20/d26/d2d/l9e d16/d20/d26/d5f/lda 0 2024-04-06T14:17:07.206 INFO:tasks.workunit.client.0.smithi032.stdout:7/611: write d16/d20/d26/f31 [258379,442] 0 2024-04-06T14:17:07.206 INFO:tasks.workunit.client.0.smithi032.stdout:8/554: dread d3/f32 [4194304,4194304] 0 2024-04-06T14:17:07.207 INFO:tasks.workunit.client.0.smithi032.stdout:3/574: write d3/d65/d7d/d9c/dad/db1/fa4 [903707,128353] 0 2024-04-06T14:17:07.209 INFO:tasks.workunit.client.0.smithi032.stdout:8/555: creat d3/d8/d7/fc8 x:0 0 0 2024-04-06T14:17:07.211 INFO:tasks.workunit.client.1.smithi086.stdout:9/357: dread d5/d13/d1a/f63 [0,4194304] 0 2024-04-06T14:17:07.212 INFO:tasks.workunit.client.0.smithi032.stdout:7/612: unlink d16/d20/d26/d2d/d30/d97/fca 0 2024-04-06T14:17:07.212 INFO:tasks.workunit.client.0.smithi032.stdout:3/575: symlink d3/d65/d7d/d9c/dad/le0 0 2024-04-06T14:17:07.213 INFO:tasks.workunit.client.0.smithi032.stdout:7/613: write f10 [5154937,99417] 0 2024-04-06T14:17:07.216 INFO:tasks.workunit.client.0.smithi032.stdout:3/576: write d3/d65/f34 [853806,53218] 0 2024-04-06T14:17:07.218 INFO:tasks.workunit.client.1.smithi086.stdout:1/371: sync 2024-04-06T14:17:07.218 INFO:tasks.workunit.client.0.smithi032.stdout:7/614: unlink d16/l85 0 2024-04-06T14:17:07.223 INFO:tasks.workunit.client.0.smithi032.stdout:7/615: creat d16/d20/d26/d2d/d30/d97/fdb x:0 0 0 2024-04-06T14:17:07.224 INFO:tasks.workunit.client.0.smithi032.stdout:7/616: readlink l5 0 2024-04-06T14:17:07.226 INFO:tasks.workunit.client.0.smithi032.stdout:3/577: dread d3/d65/d7d/d9c/dad/db1/d3c/f8f [0,4194304] 0 2024-04-06T14:17:07.227 INFO:tasks.workunit.client.1.smithi086.stdout:4/277: fdatasync d0/d15/d20/f22 0 2024-04-06T14:17:07.228 INFO:tasks.workunit.client.0.smithi032.stdout:7/617: unlink d16/d20/d26/d2d/d30/d3e/f45 0 2024-04-06T14:17:07.228 INFO:tasks.workunit.client.0.smithi032.stdout:3/578: rmdir d3/d65/d7d/d9c/dbc 39 2024-04-06T14:17:07.229 INFO:tasks.workunit.client.1.smithi086.stdout:6/370: mknod d6/d13/d42/c80 0 2024-04-06T14:17:07.229 INFO:tasks.workunit.client.0.smithi032.stdout:7/618: creat d16/d20/d26/d70/fdc x:0 0 0 2024-04-06T14:17:07.230 INFO:tasks.workunit.client.1.smithi086.stdout:7/327: creat d7/d10/d11/d1c/d24/d2c/f72 x:0 0 0 2024-04-06T14:17:07.234 INFO:tasks.workunit.client.0.smithi032.stdout:3/579: link d3/d16/f9e d3/d65/d7d/d9c/fe1 0 2024-04-06T14:17:07.235 INFO:tasks.workunit.client.0.smithi032.stdout:8/556: dwrite d3/d8/d7/d14/f45 [0,4194304] 0 2024-04-06T14:17:07.237 INFO:tasks.workunit.client.0.smithi032.stdout:3/580: write d3/d65/f93 [642191,118049] 0 2024-04-06T14:17:07.237 INFO:tasks.workunit.client.0.smithi032.stdout:7/619: rename d16/d20/d26/d2d/d30/d3e/d96/dba to d16/d20/d26/d5f/ddd 0 2024-04-06T14:17:07.240 INFO:tasks.workunit.client.0.smithi032.stdout:8/557: write d3/d8/d7/d8a/fab [367656,49941] 0 2024-04-06T14:17:07.249 INFO:tasks.workunit.client.1.smithi086.stdout:6/371: dwrite d6/d13/f26 [4194304,4194304] 0 2024-04-06T14:17:07.252 INFO:tasks.workunit.client.0.smithi032.stdout:8/558: dwrite d3/d8/d12/d2f/d4a/f94 [0,4194304] 0 2024-04-06T14:17:07.255 INFO:tasks.workunit.client.0.smithi032.stdout:3/581: dread d3/d65/f95 [0,4194304] 0 2024-04-06T14:17:07.256 INFO:tasks.workunit.client.0.smithi032.stdout:8/559: mknod d3/d8/d7/cc9 0 2024-04-06T14:17:07.258 INFO:tasks.workunit.client.0.smithi032.stdout:3/582: fsync d3/f37 0 2024-04-06T14:17:07.258 INFO:tasks.workunit.client.1.smithi086.stdout:0/408: creat d0/d1/d2/f80 x:0 0 0 2024-04-06T14:17:07.259 INFO:tasks.workunit.client.0.smithi032.stdout:3/583: chown d3/dcf 799 1 2024-04-06T14:17:07.259 INFO:tasks.workunit.client.1.smithi086.stdout:0/409: chown d0/d15/c30 9 1 2024-04-06T14:17:07.263 INFO:tasks.workunit.client.1.smithi086.stdout:5/362: creat d0/d16/d18/d21/d23/f6c x:0 0 0 2024-04-06T14:17:07.264 INFO:tasks.workunit.client.0.smithi032.stdout:3/584: unlink d3/d65/d7d/d9c/dad/le0 0 2024-04-06T14:17:07.264 INFO:tasks.workunit.client.1.smithi086.stdout:1/372: rmdir db/d11/d1a 39 2024-04-06T14:17:07.267 INFO:tasks.workunit.client.1.smithi086.stdout:6/372: symlink d6/d13/d5f/d60/d45/d4a/l81 0 2024-04-06T14:17:07.268 INFO:tasks.workunit.client.0.smithi032.stdout:3/585: dread d3/f37 [0,4194304] 0 2024-04-06T14:17:07.269 INFO:tasks.workunit.client.1.smithi086.stdout:2/373: getdents d0/df/d48/d53 0 2024-04-06T14:17:07.269 INFO:tasks.workunit.client.0.smithi032.stdout:3/586: write d3/d65/f47 [2070749,16471] 0 2024-04-06T14:17:07.276 INFO:tasks.workunit.client.1.smithi086.stdout:8/472: link d0/c1e d0/d71/d22/d43/d6d/d75/ca7 0 2024-04-06T14:17:07.276 INFO:tasks.workunit.client.1.smithi086.stdout:8/473: dread - d0/d71/d22/f8e zero size 2024-04-06T14:17:07.276 INFO:tasks.workunit.client.0.smithi032.stdout:7/620: dread d16/d20/d54/f57 [0,4194304] 0 2024-04-06T14:17:07.276 INFO:tasks.workunit.client.0.smithi032.stdout:7/621: fdatasync d16/f49 0 2024-04-06T14:17:07.278 INFO:tasks.workunit.client.1.smithi086.stdout:6/373: symlink d6/d13/d42/d52/l82 0 2024-04-06T14:17:07.279 INFO:tasks.workunit.client.0.smithi032.stdout:7/622: chown d16/d20/d26/d2d/l9e 2 1 2024-04-06T14:17:07.279 INFO:tasks.workunit.client.1.smithi086.stdout:8/474: dwrite d0/d71/d16/f6c [0,4194304] 0 2024-04-06T14:17:07.284 INFO:tasks.workunit.client.1.smithi086.stdout:4/278: getdents d0/d15 0 2024-04-06T14:17:07.286 INFO:tasks.workunit.client.1.smithi086.stdout:2/374: dread d0/df/f18 [0,4194304] 0 2024-04-06T14:17:07.286 INFO:tasks.workunit.client.1.smithi086.stdout:9/358: creat d5/d10/d14/d19/f76 x:0 0 0 2024-04-06T14:17:07.288 INFO:tasks.workunit.client.1.smithi086.stdout:2/375: read d0/d20/f2e [4113526,56424] 0 2024-04-06T14:17:07.289 INFO:tasks.workunit.client.1.smithi086.stdout:6/374: mknod d6/d9/db/c83 0 2024-04-06T14:17:07.291 INFO:tasks.workunit.client.1.smithi086.stdout:8/475: symlink d0/db/la8 0 2024-04-06T14:17:07.292 INFO:tasks.workunit.client.1.smithi086.stdout:8/476: write d0/f2 [3983118,128694] 0 2024-04-06T14:17:07.293 INFO:tasks.workunit.client.1.smithi086.stdout:8/477: chown d0/f5 355 1 2024-04-06T14:17:07.293 INFO:tasks.workunit.client.1.smithi086.stdout:4/279: creat d0/d15/d19/f52 x:0 0 0 2024-04-06T14:17:07.294 INFO:tasks.workunit.client.1.smithi086.stdout:8/478: dread - d0/d13/f7c zero size 2024-04-06T14:17:07.294 INFO:tasks.workunit.client.1.smithi086.stdout:4/280: chown d0/d15/d2b 0 1 2024-04-06T14:17:07.295 INFO:tasks.workunit.client.0.smithi032.stdout:7/623: dwrite d16/d20/d26/d5f/f91 [0,4194304] 0 2024-04-06T14:17:07.298 INFO:tasks.workunit.client.1.smithi086.stdout:8/479: dread d0/d3b/f4c [0,4194304] 0 2024-04-06T14:17:07.298 INFO:tasks.workunit.client.0.smithi032.stdout:8/560: dwrite d3/d8/d12/d2f/d46/f76 [0,4194304] 0 2024-04-06T14:17:07.298 INFO:tasks.workunit.client.1.smithi086.stdout:8/480: chown d0/d71/d82 0 1 2024-04-06T14:17:07.299 INFO:tasks.workunit.client.1.smithi086.stdout:8/481: fsync d0/f72 0 2024-04-06T14:17:07.300 INFO:tasks.workunit.client.1.smithi086.stdout:8/482: chown d0/d71/d16/d39/la5 5511 1 2024-04-06T14:17:07.300 INFO:tasks.workunit.client.1.smithi086.stdout:8/483: stat d0/db/la8 0 2024-04-06T14:17:07.301 INFO:tasks.workunit.client.0.smithi032.stdout:3/587: dwrite d3/f4 [0,4194304] 0 2024-04-06T14:17:07.302 INFO:tasks.workunit.client.1.smithi086.stdout:6/375: mknod d6/d13/d5f/d60/c84 0 2024-04-06T14:17:07.305 INFO:tasks.workunit.client.0.smithi032.stdout:8/561: symlink d3/d8/d12/d2f/d4a/d60/d82/d84/lca 0 2024-04-06T14:17:07.305 INFO:tasks.workunit.client.0.smithi032.stdout:7/624: mkdir d16/d20/d26/d2d/d30/d3e/da7/db5/dde 0 2024-04-06T14:17:07.306 INFO:tasks.workunit.client.1.smithi086.stdout:4/281: mkdir d0/d15/d53 0 2024-04-06T14:17:07.307 INFO:tasks.workunit.client.0.smithi032.stdout:3/588: fdatasync d3/d65/d7d/d9c/dad/dc5/fcc 0 2024-04-06T14:17:07.307 INFO:tasks.workunit.client.0.smithi032.stdout:9/609: sync 2024-04-06T14:17:07.307 INFO:tasks.workunit.client.0.smithi032.stdout:2/582: sync 2024-04-06T14:17:07.308 INFO:tasks.workunit.client.0.smithi032.stdout:0/423: sync 2024-04-06T14:17:07.308 INFO:tasks.workunit.client.0.smithi032.stdout:3/589: chown d3/d16/l60 1 1 2024-04-06T14:17:07.309 INFO:tasks.workunit.client.0.smithi032.stdout:8/562: mknod d3/ccb 0 2024-04-06T14:17:07.311 INFO:tasks.workunit.client.1.smithi086.stdout:1/373: rmdir db/d11/d1a/d2a/d52/d69 0 2024-04-06T14:17:07.314 INFO:tasks.workunit.client.0.smithi032.stdout:2/583: rename d0/d5a/d3a/f42 to d0/d5a/d3a/da2/fc1 0 2024-04-06T14:17:07.315 INFO:tasks.workunit.client.1.smithi086.stdout:8/484: creat d0/d13/fa9 x:0 0 0 2024-04-06T14:17:07.316 INFO:tasks.workunit.client.0.smithi032.stdout:0/424: mknod d3/da/d25/d57/c95 0 2024-04-06T14:17:07.316 INFO:tasks.workunit.client.0.smithi032.stdout:2/584: chown d0/d5a/f97 345 1 2024-04-06T14:17:07.316 INFO:tasks.workunit.client.1.smithi086.stdout:2/376: symlink d0/d20/d80/l85 0 2024-04-06T14:17:07.317 INFO:tasks.workunit.client.0.smithi032.stdout:8/563: unlink d3/d69/c70 0 2024-04-06T14:17:07.317 INFO:tasks.workunit.client.0.smithi032.stdout:8/564: readlink l2 0 2024-04-06T14:17:07.317 INFO:tasks.workunit.client.1.smithi086.stdout:6/376: readlink d6/d13/l3a 0 2024-04-06T14:17:07.318 INFO:tasks.workunit.client.1.smithi086.stdout:2/377: write d0/d20/d27/d2b/d35/f47 [166669,82437] 0 2024-04-06T14:17:07.318 INFO:tasks.workunit.client.0.smithi032.stdout:9/610: mknod d2/d15/d1e/d65/d83/d31/cc7 0 2024-04-06T14:17:07.320 INFO:tasks.workunit.client.0.smithi032.stdout:3/590: mknod d3/d65/d7d/d9c/dbc/dcd/ce2 0 2024-04-06T14:17:07.320 INFO:tasks.workunit.client.0.smithi032.stdout:8/565: write d3/d8/d7/d1a/f78 [1894798,63621] 0 2024-04-06T14:17:07.321 INFO:tasks.workunit.client.0.smithi032.stdout:7/625: rename d16/d20/d63/c8e to d16/d20/d26/d70/cdf 0 2024-04-06T14:17:07.322 INFO:tasks.workunit.client.1.smithi086.stdout:1/374: fdatasync db/d13/f23 0 2024-04-06T14:17:07.323 INFO:tasks.workunit.client.1.smithi086.stdout:1/375: chown db/d11/l37 1379003528 1 2024-04-06T14:17:07.324 INFO:tasks.workunit.client.0.smithi032.stdout:9/611: creat d2/d15/d55/d5f/fc8 x:0 0 0 2024-04-06T14:17:07.326 INFO:tasks.workunit.client.0.smithi032.stdout:7/626: dread - d16/d20/d7a/fa5 zero size 2024-04-06T14:17:07.327 INFO:tasks.workunit.client.1.smithi086.stdout:4/282: dwrite d0/de/f1a [0,4194304] 0 2024-04-06T14:17:07.328 INFO:tasks.workunit.client.0.smithi032.stdout:8/566: truncate d3/d8/f5f 260554 0 2024-04-06T14:17:07.329 INFO:tasks.workunit.client.1.smithi086.stdout:6/377: mkdir d6/d13/d1a/d64/d85 0 2024-04-06T14:17:07.330 INFO:tasks.workunit.client.1.smithi086.stdout:6/378: read - d6/d13/f75 zero size 2024-04-06T14:17:07.331 INFO:tasks.workunit.client.1.smithi086.stdout:1/376: write db/d11/f17 [4778971,4773] 0 2024-04-06T14:17:07.331 INFO:tasks.workunit.client.0.smithi032.stdout:9/612: write d2/d15/d1e/d65/d83/d7e/fb5 [947096,61827] 0 2024-04-06T14:17:07.332 INFO:tasks.workunit.client.0.smithi032.stdout:2/585: dwrite d0/d30/d39/d59/d67/d74/d78/d8c/f9f [0,4194304] 0 2024-04-06T14:17:07.333 INFO:tasks.workunit.client.1.smithi086.stdout:8/485: mkdir d0/d71/d22/d43/daa 0 2024-04-06T14:17:07.334 INFO:tasks.workunit.client.1.smithi086.stdout:8/486: fdatasync d0/d71/d16/d39/d90/fa0 0 2024-04-06T14:17:07.335 INFO:tasks.workunit.client.1.smithi086.stdout:8/487: write d0/f2 [4308703,14824] 0 2024-04-06T14:17:07.335 INFO:tasks.workunit.client.0.smithi032.stdout:3/591: creat d3/d65/d7d/d9c/dbc/fe3 x:0 0 0 2024-04-06T14:17:07.336 INFO:tasks.workunit.client.0.smithi032.stdout:0/425: link d3/da/d25/c3e d3/c96 0 2024-04-06T14:17:07.336 INFO:tasks.workunit.client.1.smithi086.stdout:8/488: chown d0/d71/d16/d52/d5f/d9e 3987 1 2024-04-06T14:17:07.337 INFO:tasks.workunit.client.1.smithi086.stdout:6/379: mknod d6/d13/d1a/d22/c86 0 2024-04-06T14:17:07.338 INFO:tasks.workunit.client.1.smithi086.stdout:6/380: readlink d6/d13/d42/d52/l82 0 2024-04-06T14:17:07.339 INFO:tasks.workunit.client.0.smithi032.stdout:8/567: fdatasync d3/d8/d12/d2f/d46/f55 0 2024-04-06T14:17:07.339 INFO:tasks.workunit.client.1.smithi086.stdout:2/378: dread d0/d20/d27/d2b/d35/d36/d4d/d6f/f73 [0,4194304] 0 2024-04-06T14:17:07.341 INFO:tasks.workunit.client.0.smithi032.stdout:7/627: getdents d16/d20/d26/d2d/d30 0 2024-04-06T14:17:07.342 INFO:tasks.workunit.client.0.smithi032.stdout:0/426: mkdir d3/da/d25/d2b/d4a/d79/d97 0 2024-04-06T14:17:07.344 INFO:tasks.workunit.client.1.smithi086.stdout:2/379: dread d0/d20/d27/d37/d52/f5c [0,4194304] 0 2024-04-06T14:17:07.344 INFO:tasks.workunit.client.1.smithi086.stdout:2/380: readlink d0/l3f 0 2024-04-06T14:17:07.345 INFO:tasks.workunit.client.1.smithi086.stdout:2/381: readlink d0/d20/d80/l85 0 2024-04-06T14:17:07.345 INFO:tasks.workunit.client.1.smithi086.stdout:8/489: mkdir d0/db/d32/dab 0 2024-04-06T14:17:07.346 INFO:tasks.workunit.client.0.smithi032.stdout:9/613: creat d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8f/fc9 x:0 0 0 2024-04-06T14:17:07.347 INFO:tasks.workunit.client.0.smithi032.stdout:0/427: truncate d3/f45 986151 0 2024-04-06T14:17:07.347 INFO:tasks.workunit.client.0.smithi032.stdout:6/588: sync 2024-04-06T14:17:07.347 INFO:tasks.workunit.client.0.smithi032.stdout:1/669: sync 2024-04-06T14:17:07.347 INFO:tasks.workunit.client.0.smithi032.stdout:5/575: sync 2024-04-06T14:17:07.347 INFO:tasks.workunit.client.1.smithi086.stdout:8/490: write d0/db/f92 [4678829,56213] 0 2024-04-06T14:17:07.348 INFO:tasks.workunit.client.0.smithi032.stdout:7/628: mkdir d16/d20/d26/de0 0 2024-04-06T14:17:07.350 INFO:tasks.workunit.client.0.smithi032.stdout:4/639: sync 2024-04-06T14:17:07.353 INFO:tasks.workunit.client.1.smithi086.stdout:1/377: getdents db/d11/d1a/d2a/d52/d5b 0 2024-04-06T14:17:07.353 INFO:tasks.workunit.client.0.smithi032.stdout:9/614: creat d2/d15/d1e/d41/d6c/fca x:0 0 0 2024-04-06T14:17:07.353 INFO:tasks.workunit.client.1.smithi086.stdout:6/381: dwrite d6/d13/d5f/d60/d45/d4a/f79 [0,4194304] 0 2024-04-06T14:17:07.354 INFO:tasks.workunit.client.1.smithi086.stdout:7/328: truncate d7/d10/d11/d1c/f2e 2816091 0 2024-04-06T14:17:07.355 INFO:tasks.workunit.client.0.smithi032.stdout:8/568: creat d3/d8/d12/d2f/d4a/d60/d82/d84/fcc x:0 0 0 2024-04-06T14:17:07.356 INFO:tasks.workunit.client.1.smithi086.stdout:7/329: write d7/d1d/f38 [246509,18804] 0 2024-04-06T14:17:07.357 INFO:tasks.workunit.client.0.smithi032.stdout:0/428: getdents d3/da/d41/d82 0 2024-04-06T14:17:07.357 INFO:tasks.workunit.client.0.smithi032.stdout:7/629: dread d16/d20/d7a/da6/fc1 [0,4194304] 0 2024-04-06T14:17:07.357 INFO:tasks.workunit.client.1.smithi086.stdout:6/382: mknod d6/d13/d1a/d64/c87 0 2024-04-06T14:17:07.358 INFO:tasks.workunit.client.0.smithi032.stdout:2/586: dwrite d0/d30/d39/d59/d67/d74/d78/d57/d7c/fa1 [0,4194304] 0 2024-04-06T14:17:07.360 INFO:tasks.workunit.client.0.smithi032.stdout:1/670: write d0/d41/d59/f64 [1648377,105751] 0 2024-04-06T14:17:07.360 INFO:tasks.workunit.client.0.smithi032.stdout:5/576: symlink d16/d18/d33/d35/d88/dbd/lc7 0 2024-04-06T14:17:07.361 INFO:tasks.workunit.client.0.smithi032.stdout:4/640: creat d0/df/d2b/d46/d58/fdc x:0 0 0 2024-04-06T14:17:07.366 INFO:tasks.workunit.client.0.smithi032.stdout:8/569: write d3/d8/f93 [1162281,130138] 0 2024-04-06T14:17:07.366 INFO:tasks.workunit.client.0.smithi032.stdout:2/587: chown d0/d30/d39/d59/d67/d74/d78/d8c/f9e 0 1 2024-04-06T14:17:07.369 INFO:tasks.workunit.client.0.smithi032.stdout:9/615: read d2/d15/d55/d59/d93/d95/db8/d81/d77/f78 [687667,19985] 0 2024-04-06T14:17:07.371 INFO:tasks.workunit.client.0.smithi032.stdout:4/641: fdatasync d0/df/d1f/d42/d45/f8c 0 2024-04-06T14:17:07.371 INFO:tasks.workunit.client.0.smithi032.stdout:1/671: chown d0/d9/d94/dc2 77319 1 2024-04-06T14:17:07.374 INFO:tasks.workunit.client.0.smithi032.stdout:8/570: creat d3/d8/d7/d8a/fcd x:0 0 0 2024-04-06T14:17:07.375 INFO:tasks.workunit.client.0.smithi032.stdout:3/592: write d3/d65/d7d/d9c/fe1 [195409,26745] 0 2024-04-06T14:17:07.376 INFO:tasks.workunit.client.0.smithi032.stdout:4/642: fdatasync d0/df/d76/f61 0 2024-04-06T14:17:07.377 INFO:tasks.workunit.client.0.smithi032.stdout:1/672: write d0/f90 [2889380,54728] 0 2024-04-06T14:17:07.378 INFO:tasks.workunit.client.0.smithi032.stdout:6/589: dwrite d4/fc7 [0,4194304] 0 2024-04-06T14:17:07.382 INFO:tasks.workunit.client.0.smithi032.stdout:2/588: mkdir d0/d30/d39/d59/d67/d74/d78/dc2 0 2024-04-06T14:17:07.386 INFO:tasks.workunit.client.0.smithi032.stdout:8/571: creat d3/d8/d12/d2f/d46/fce x:0 0 0 2024-04-06T14:17:07.387 INFO:tasks.workunit.client.0.smithi032.stdout:7/630: rmdir d16/d20/d26 39 2024-04-06T14:17:07.389 INFO:tasks.workunit.client.0.smithi032.stdout:1/673: fdatasync d0/faf 0 2024-04-06T14:17:07.389 INFO:tasks.workunit.client.0.smithi032.stdout:9/616: dread - d2/d15/d1e/d65/d83/d7e/f49 zero size 2024-04-06T14:17:07.389 INFO:tasks.workunit.client.0.smithi032.stdout:4/643: mknod d0/d82/d4b/cdd 0 2024-04-06T14:17:07.390 INFO:tasks.workunit.client.0.smithi032.stdout:0/429: dwrite d3/da/d25/d36/f39 [0,4194304] 0 2024-04-06T14:17:07.394 INFO:tasks.workunit.client.1.smithi086.stdout:6/383: dread - d6/d9/d5c/f6b zero size 2024-04-06T14:17:07.394 INFO:tasks.workunit.client.1.smithi086.stdout:1/378: mkdir db/d11/d1a/d79 0 2024-04-06T14:17:07.394 INFO:tasks.workunit.client.1.smithi086.stdout:5/363: mknod d0/d16/c6d 0 2024-04-06T14:17:07.395 INFO:tasks.workunit.client.1.smithi086.stdout:0/410: truncate d0/ff 8252987 0 2024-04-06T14:17:07.395 INFO:tasks.workunit.client.0.smithi032.stdout:4/644: chown d0/fba 249 1 2024-04-06T14:17:07.395 INFO:tasks.workunit.client.0.smithi032.stdout:5/577: write d16/d58/d54/d94/db2/fb8 [4569192,96390] 0 2024-04-06T14:17:07.395 INFO:tasks.workunit.client.0.smithi032.stdout:6/590: chown d4/d5/dbf/d51/fa8 136332597 1 2024-04-06T14:17:07.395 INFO:tasks.workunit.client.0.smithi032.stdout:3/593: read d3/d65/d7d/d9c/dad/fb3 [387347,20822] 0 2024-04-06T14:17:07.396 INFO:tasks.workunit.client.1.smithi086.stdout:2/382: unlink d0/d20/f25 0 2024-04-06T14:17:07.396 INFO:tasks.workunit.client.0.smithi032.stdout:2/589: symlink d0/d4e/d63/dbe/lc3 0 2024-04-06T14:17:07.397 INFO:tasks.workunit.client.1.smithi086.stdout:3/425: rename d5/d42 to d5/d26/d7f 0 2024-04-06T14:17:07.397 INFO:tasks.workunit.client.1.smithi086.stdout:9/359: truncate d5/d10/d14/f22 2201596 0 2024-04-06T14:17:07.398 INFO:tasks.workunit.client.1.smithi086.stdout:4/283: truncate d0/f10 44876 0 2024-04-06T14:17:07.399 INFO:tasks.workunit.client.0.smithi032.stdout:6/591: fdatasync d4/d5/d35/fce 0 2024-04-06T14:17:07.399 INFO:tasks.workunit.client.0.smithi032.stdout:0/430: read d3/da/d25/d2b/f32 [5362615,3489] 0 2024-04-06T14:17:07.399 INFO:tasks.workunit.client.0.smithi032.stdout:9/617: readlink d2/d15/d1e/d65/d6b/d7f/la0 0 2024-04-06T14:17:07.400 INFO:tasks.workunit.client.1.smithi086.stdout:2/383: fsync d0/df/f13 0 2024-04-06T14:17:07.401 INFO:tasks.workunit.client.1.smithi086.stdout:4/284: mkdir d0/d2c/d38/d54 0 2024-04-06T14:17:07.405 INFO:tasks.workunit.client.1.smithi086.stdout:9/360: fsync d5/d18/f48 0 2024-04-06T14:17:07.405 INFO:tasks.workunit.client.0.smithi032.stdout:9/618: chown d2/d15/d55/d59/d93/d95 36459 1 2024-04-06T14:17:07.406 INFO:tasks.workunit.client.1.smithi086.stdout:6/384: mkdir d6/d13/d1a/d64/d85/d88 0 2024-04-06T14:17:07.407 INFO:tasks.workunit.client.0.smithi032.stdout:5/578: creat d16/d18/d33/d45/d5e/fc8 x:0 0 0 2024-04-06T14:17:07.408 INFO:tasks.workunit.client.1.smithi086.stdout:3/426: symlink d5/d26/l80 0 2024-04-06T14:17:07.410 INFO:tasks.workunit.client.1.smithi086.stdout:3/427: write d5/f20 [1325367,6820] 0 2024-04-06T14:17:07.410 INFO:tasks.workunit.client.1.smithi086.stdout:7/330: link d7/d10/d11/d21/c5b d7/d10/d11/d1c/d24/d2c/c73 0 2024-04-06T14:17:07.412 INFO:tasks.workunit.client.0.smithi032.stdout:8/572: getdents d3/d8/d7/d8a 0 2024-04-06T14:17:07.413 INFO:tasks.workunit.client.0.smithi032.stdout:7/631: mknod d16/d20/d26/de0/ce1 0 2024-04-06T14:17:07.413 INFO:tasks.workunit.client.1.smithi086.stdout:1/379: rmdir db/d11/d1a/d2a/d52/d5b/d68/d6e 39 2024-04-06T14:17:07.415 INFO:tasks.workunit.client.0.smithi032.stdout:0/431: read d3/da/f1f [1011919,74297] 0 2024-04-06T14:17:07.415 INFO:tasks.workunit.client.1.smithi086.stdout:3/428: creat d5/df/f81 x:0 0 0 2024-04-06T14:17:07.416 INFO:tasks.workunit.client.0.smithi032.stdout:9/619: unlink d2/f97 0 2024-04-06T14:17:07.416 INFO:tasks.workunit.client.0.smithi032.stdout:5/579: read - d16/d58/d54/d74/d8f/d95/fc2 zero size 2024-04-06T14:17:07.416 INFO:tasks.workunit.client.0.smithi032.stdout:6/592: write d4/d5/d19/f55 [35367,58146] 0 2024-04-06T14:17:07.417 INFO:tasks.workunit.client.1.smithi086.stdout:6/385: link d6/d13/d5f/d60/f5b d6/d13/d5f/d60/d45/d4a/d62/f89 0 2024-04-06T14:17:07.417 INFO:tasks.workunit.client.1.smithi086.stdout:3/429: read d5/d26/f3d [458800,90137] 0 2024-04-06T14:17:07.418 INFO:tasks.workunit.client.1.smithi086.stdout:7/331: dwrite f5 [0,4194304] 0 2024-04-06T14:17:07.419 INFO:tasks.workunit.client.0.smithi032.stdout:1/674: truncate d0/d10/d16/f51 2041216 0 2024-04-06T14:17:07.420 INFO:tasks.workunit.client.0.smithi032.stdout:4/645: rmdir d0/d82/daf/dc2 0 2024-04-06T14:17:07.421 INFO:tasks.workunit.client.1.smithi086.stdout:4/285: rename d0/f12 to d0/d15/d20/d34/f55 0 2024-04-06T14:17:07.422 INFO:tasks.workunit.client.0.smithi032.stdout:3/594: dread d3/d65/d7d/d9c/dad/db1/f9b [0,4194304] 0 2024-04-06T14:17:07.422 INFO:tasks.workunit.client.0.smithi032.stdout:9/620: fsync d2/d15/d1e/d65/d83/d7e/fb5 0 2024-04-06T14:17:07.423 INFO:tasks.workunit.client.1.smithi086.stdout:1/380: write db/d11/d24/f45 [1587796,34502] 0 2024-04-06T14:17:07.424 INFO:tasks.workunit.client.0.smithi032.stdout:0/432: fsync d3/da/d25/d2b/f32 0 2024-04-06T14:17:07.424 INFO:tasks.workunit.client.0.smithi032.stdout:5/580: creat d16/d58/fc9 x:0 0 0 2024-04-06T14:17:07.425 INFO:tasks.workunit.client.0.smithi032.stdout:2/590: getdents d0/d30/d39/d59/d67/d74/d78/d8c 0 2024-04-06T14:17:07.425 INFO:tasks.workunit.client.1.smithi086.stdout:3/430: write d5/d26/f3d [4454663,66184] 0 2024-04-06T14:17:07.426 INFO:tasks.workunit.client.0.smithi032.stdout:1/675: unlink d0/faf 0 2024-04-06T14:17:07.426 INFO:tasks.workunit.client.1.smithi086.stdout:7/332: stat d7/d10/d11/d1c/l26 0 2024-04-06T14:17:07.430 INFO:tasks.workunit.client.0.smithi032.stdout:9/621: chown d2/d15/d55/d59/d93/d95/db8/d81/d4a/d73/f87 184977657 1 2024-04-06T14:17:07.430 INFO:tasks.workunit.client.1.smithi086.stdout:7/333: write d7/d10/d11/d1c/f58 [59388,64461] 0 2024-04-06T14:17:07.431 INFO:tasks.workunit.client.0.smithi032.stdout:4/646: creat d0/d82/d8e/fde x:0 0 0 2024-04-06T14:17:07.433 INFO:tasks.workunit.client.1.smithi086.stdout:4/286: dwrite d0/d15/d20/f44 [0,4194304] 0 2024-04-06T14:17:07.434 INFO:tasks.workunit.client.0.smithi032.stdout:0/433: dread d3/da/d25/d2b/f3d [0,4194304] 0 2024-04-06T14:17:07.436 INFO:tasks.workunit.client.1.smithi086.stdout:4/287: truncate d0/d15/d20/f22 4432232 0 2024-04-06T14:17:07.436 INFO:tasks.workunit.client.0.smithi032.stdout:6/593: rename d4/d5/d14/l22 to d4/d5/d8/d2f/ld1 0 2024-04-06T14:17:07.437 INFO:tasks.workunit.client.1.smithi086.stdout:3/431: link d5/d36/c4b d5/df/c82 0 2024-04-06T14:17:07.439 INFO:tasks.workunit.client.1.smithi086.stdout:4/288: mknod d0/d15/d20/c56 0 2024-04-06T14:17:07.440 INFO:tasks.workunit.client.0.smithi032.stdout:3/595: creat d3/d65/d89/fe4 x:0 0 0 2024-04-06T14:17:07.441 INFO:tasks.workunit.client.1.smithi086.stdout:7/334: link d7/d10/d11/d1c/d24/f4e d7/d10/d11/d1c/d36/f74 0 2024-04-06T14:17:07.443 INFO:tasks.workunit.client.1.smithi086.stdout:6/386: dread d6/d13/f5d [0,4194304] 0 2024-04-06T14:17:07.444 INFO:tasks.workunit.client.1.smithi086.stdout:6/387: write d6/d13/f75 [507793,2691] 0 2024-04-06T14:17:07.446 INFO:tasks.workunit.client.0.smithi032.stdout:7/632: getdents d16/d20 0 2024-04-06T14:17:07.446 INFO:tasks.workunit.client.1.smithi086.stdout:1/381: creat db/d11/d1a/d2a/d52/d5b/d68/f7a x:0 0 0 2024-04-06T14:17:07.447 INFO:tasks.workunit.client.1.smithi086.stdout:1/382: readlink db/l3c 0 2024-04-06T14:17:07.447 INFO:tasks.workunit.client.0.smithi032.stdout:1/676: creat d0/d41/d59/fd8 x:0 0 0 2024-04-06T14:17:07.448 INFO:tasks.workunit.client.0.smithi032.stdout:2/591: write d0/d6e/f86 [42761,20716] 0 2024-04-06T14:17:07.448 INFO:tasks.workunit.client.1.smithi086.stdout:4/289: getdents d0/d2c/d38/d54 0 2024-04-06T14:17:07.449 INFO:tasks.workunit.client.0.smithi032.stdout:1/677: write d0/d9/d1f/f91 [2129336,30125] 0 2024-04-06T14:17:07.450 INFO:tasks.workunit.client.1.smithi086.stdout:6/388: mkdir d6/d13/d5f/d60/d45/d8a 0 2024-04-06T14:17:07.450 INFO:tasks.workunit.client.0.smithi032.stdout:6/594: mknod d4/cd2 0 2024-04-06T14:17:07.451 INFO:tasks.workunit.client.1.smithi086.stdout:7/335: fdatasync d7/d10/d11/f16 0 2024-04-06T14:17:07.452 INFO:tasks.workunit.client.0.smithi032.stdout:8/573: getdents d3/d8/d7/d1a 0 2024-04-06T14:17:07.453 INFO:tasks.workunit.client.0.smithi032.stdout:3/596: mkdir d3/d65/d7d/dc6/de5 0 2024-04-06T14:17:07.454 INFO:tasks.workunit.client.0.smithi032.stdout:7/633: symlink d16/d20/d26/d2d/d30/le2 0 2024-04-06T14:17:07.454 INFO:tasks.workunit.client.1.smithi086.stdout:3/432: getdents d5/d26/d4f 0 2024-04-06T14:17:07.455 INFO:tasks.workunit.client.0.smithi032.stdout:5/581: mknod d16/d58/d54/d74/d8f/d95/cca 0 2024-04-06T14:17:07.455 INFO:tasks.workunit.client.1.smithi086.stdout:1/383: creat db/f7b x:0 0 0 2024-04-06T14:17:07.456 INFO:tasks.workunit.client.0.smithi032.stdout:4/647: symlink d0/d82/ldf 0 2024-04-06T14:17:07.456 INFO:tasks.workunit.client.1.smithi086.stdout:1/384: readlink db/d11/d1f/l48 0 2024-04-06T14:17:07.459 INFO:tasks.workunit.client.0.smithi032.stdout:1/678: mkdir d0/d10/d78/dd9 0 2024-04-06T14:17:07.460 INFO:tasks.workunit.client.0.smithi032.stdout:9/622: write d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/fae [934523,13558] 0 2024-04-06T14:17:07.461 INFO:tasks.workunit.client.1.smithi086.stdout:3/433: dwrite d5/f44 [0,4194304] 0 2024-04-06T14:17:07.461 INFO:tasks.workunit.client.0.smithi032.stdout:6/595: creat d4/d5/d8/d4b/fd3 x:0 0 0 2024-04-06T14:17:07.462 INFO:tasks.workunit.client.1.smithi086.stdout:7/336: creat d7/d10/d11/d1c/d24/f75 x:0 0 0 2024-04-06T14:17:07.462 INFO:tasks.workunit.client.0.smithi032.stdout:3/597: fdatasync d3/d65/f47 0 2024-04-06T14:17:07.465 INFO:tasks.workunit.client.1.smithi086.stdout:7/337: chown d7/d10/d11/d1c/d24/d2c/f6f 46512 1 2024-04-06T14:17:07.467 INFO:tasks.workunit.client.0.smithi032.stdout:5/582: unlink d16/d18/d33/d35/c7b 0 2024-04-06T14:17:07.468 INFO:tasks.workunit.client.1.smithi086.stdout:1/385: dwrite db/d13/f1e [0,4194304] 0 2024-04-06T14:17:07.468 INFO:tasks.workunit.client.1.smithi086.stdout:7/338: dread - d7/d10/d11/d1c/d24/d2c/f48 zero size 2024-04-06T14:17:07.468 INFO:tasks.workunit.client.1.smithi086.stdout:6/389: dread d6/d9/f10 [0,4194304] 0 2024-04-06T14:17:07.470 INFO:tasks.workunit.client.1.smithi086.stdout:6/390: chown d6/d13/d5f 36505 1 2024-04-06T14:17:07.472 INFO:tasks.workunit.client.0.smithi032.stdout:1/679: creat d0/d9/d1f/d7c/d8f/daa/fda x:0 0 0 2024-04-06T14:17:07.472 INFO:tasks.workunit.client.1.smithi086.stdout:1/386: creat db/d11/d1f/f7c x:0 0 0 2024-04-06T14:17:07.472 INFO:tasks.workunit.client.0.smithi032.stdout:9/623: creat d2/d15/d1e/d41/fcb x:0 0 0 2024-04-06T14:17:07.473 INFO:tasks.workunit.client.1.smithi086.stdout:3/434: link d5/d26/d4f/c7c d5/d36/d5a/d6d/d72/d57/c83 0 2024-04-06T14:17:07.474 INFO:tasks.workunit.client.0.smithi032.stdout:6/596: creat d4/d5/dbf/d53/fd4 x:0 0 0 2024-04-06T14:17:07.479 INFO:tasks.workunit.client.1.smithi086.stdout:6/391: dread d6/d9/d43/f53 [0,4194304] 0 2024-04-06T14:17:07.479 INFO:tasks.workunit.client.1.smithi086.stdout:1/387: unlink db/d11/f41 0 2024-04-06T14:17:07.479 INFO:tasks.workunit.client.0.smithi032.stdout:3/598: mknod d3/d65/d89/ce6 0 2024-04-06T14:17:07.479 INFO:tasks.workunit.client.0.smithi032.stdout:5/583: dread d16/d18/f23 [0,4194304] 0 2024-04-06T14:17:07.480 INFO:tasks.workunit.client.0.smithi032.stdout:2/592: dwrite d0/d5a/d3a/f48 [0,4194304] 0 2024-04-06T14:17:07.480 INFO:tasks.workunit.client.1.smithi086.stdout:7/339: getdents d7/d10/d11/d1c/d36 0 2024-04-06T14:17:07.481 INFO:tasks.workunit.client.1.smithi086.stdout:6/392: symlink d6/d13/d5f/d60/l8b 0 2024-04-06T14:17:07.482 INFO:tasks.workunit.client.1.smithi086.stdout:1/388: symlink db/d11/d1a/d2a/d52/l7d 0 2024-04-06T14:17:07.483 INFO:tasks.workunit.client.1.smithi086.stdout:7/340: dread - d7/d10/d11/d1c/d24/d4d/d5c/d66/f67 zero size 2024-04-06T14:17:07.484 INFO:tasks.workunit.client.1.smithi086.stdout:3/435: getdents d5/d36/d65 0 2024-04-06T14:17:07.485 INFO:tasks.workunit.client.1.smithi086.stdout:3/436: stat d5/d26 0 2024-04-06T14:17:07.485 INFO:tasks.workunit.client.0.smithi032.stdout:4/648: truncate d0/df/d1f/f34 2336137 0 2024-04-06T14:17:07.486 INFO:tasks.workunit.client.0.smithi032.stdout:3/599: write d3/f37 [3049131,96808] 0 2024-04-06T14:17:07.486 INFO:tasks.workunit.client.0.smithi032.stdout:5/584: chown l2 1428 1 2024-04-06T14:17:07.490 INFO:tasks.workunit.client.0.smithi032.stdout:8/574: link d3/d8/d12/d2f/d4a/d60/d82/fa1 d3/d8/d3a/d7b/db9/fcf 0 2024-04-06T14:17:07.490 INFO:tasks.workunit.client.0.smithi032.stdout:1/680: write d0/d10/d16/d3f/f5d [2058887,83892] 0 2024-04-06T14:17:07.491 INFO:tasks.workunit.client.0.smithi032.stdout:2/593: write d0/d4e/d63/dbe/f93 [471016,77] 0 2024-04-06T14:17:07.497 INFO:tasks.workunit.client.0.smithi032.stdout:7/634: getdents d16/d20/d26/d5f 0 2024-04-06T14:17:07.497 INFO:tasks.workunit.client.0.smithi032.stdout:6/597: mknod d4/d5/d8/d4b/d7c/cd5 0 2024-04-06T14:17:07.498 INFO:tasks.workunit.client.0.smithi032.stdout:7/635: stat d16/d20/d26/d70/fd2 0 2024-04-06T14:17:07.500 INFO:tasks.workunit.client.0.smithi032.stdout:2/594: readlink d0/d30/d39/l3d 0 2024-04-06T14:17:07.501 INFO:tasks.workunit.client.0.smithi032.stdout:8/575: creat d3/d8/d12/d2f/d4a/d60/d82/d84/fd0 x:0 0 0 2024-04-06T14:17:07.503 INFO:tasks.workunit.client.0.smithi032.stdout:4/649: fdatasync d0/df/d1f/f25 0 2024-04-06T14:17:07.506 INFO:tasks.workunit.client.0.smithi032.stdout:6/598: read d4/d5/f6a [631680,86197] 0 2024-04-06T14:17:07.506 INFO:tasks.workunit.client.0.smithi032.stdout:1/681: creat d0/d9/d94/dc2/fdb x:0 0 0 2024-04-06T14:17:07.508 INFO:tasks.workunit.client.0.smithi032.stdout:2/595: symlink d0/d30/d39/d59/d67/lc4 0 2024-04-06T14:17:07.508 INFO:tasks.workunit.client.0.smithi032.stdout:1/682: read - d0/d41/d59/fd4 zero size 2024-04-06T14:17:07.509 INFO:tasks.workunit.client.0.smithi032.stdout:4/650: unlink d0/d82/l7f 0 2024-04-06T14:17:07.510 INFO:tasks.workunit.client.0.smithi032.stdout:9/624: link d2/c22 d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/ccc 0 2024-04-06T14:17:07.512 INFO:tasks.workunit.client.0.smithi032.stdout:8/576: dread d3/d8/d7/d1a/f30 [0,4194304] 0 2024-04-06T14:17:07.513 INFO:tasks.workunit.client.0.smithi032.stdout:4/651: write d0/df/d23/fc7 [596178,96267] 0 2024-04-06T14:17:07.513 INFO:tasks.workunit.client.0.smithi032.stdout:6/599: fdatasync d4/d5/d28/f41 0 2024-04-06T14:17:07.513 INFO:tasks.workunit.client.0.smithi032.stdout:1/683: dread d0/da/d3e/fad [0,4194304] 0 2024-04-06T14:17:07.513 INFO:tasks.workunit.client.0.smithi032.stdout:9/625: chown d2/d15/f29 1800513 1 2024-04-06T14:17:07.517 INFO:tasks.workunit.client.0.smithi032.stdout:4/652: fdatasync d0/df/d2b/fd9 0 2024-04-06T14:17:07.517 INFO:tasks.workunit.client.0.smithi032.stdout:9/626: chown d2/c51 19 1 2024-04-06T14:17:07.518 INFO:tasks.workunit.client.0.smithi032.stdout:2/596: mkdir d0/d5a/d3a/dc5 0 2024-04-06T14:17:07.518 INFO:tasks.workunit.client.0.smithi032.stdout:5/585: dread d16/d18/d33/f56 [0,4194304] 0 2024-04-06T14:17:07.519 INFO:tasks.workunit.client.0.smithi032.stdout:2/597: chown d0/d5a/f2a 17687 1 2024-04-06T14:17:07.520 INFO:tasks.workunit.client.0.smithi032.stdout:8/577: mkdir d3/d8/d12/d2f/d4a/d60/d75/dd1 0 2024-04-06T14:17:07.522 INFO:tasks.workunit.client.0.smithi032.stdout:8/578: chown d3/d8/d7/f16 79 1 2024-04-06T14:17:07.522 INFO:tasks.workunit.client.0.smithi032.stdout:2/598: mkdir d0/d4e/dc6 0 2024-04-06T14:17:07.524 INFO:tasks.workunit.client.0.smithi032.stdout:5/586: mkdir d16/d58/d54/d94/db2/dcb 0 2024-04-06T14:17:07.524 INFO:tasks.workunit.client.0.smithi032.stdout:3/600: dread d3/f3f [0,4194304] 0 2024-04-06T14:17:07.525 INFO:tasks.workunit.client.0.smithi032.stdout:1/684: creat d0/d9/d1f/fdc x:0 0 0 2024-04-06T14:17:07.526 INFO:tasks.workunit.client.0.smithi032.stdout:6/600: dread d4/d5/d18/f49 [0,4194304] 0 2024-04-06T14:17:07.527 INFO:tasks.workunit.client.0.smithi032.stdout:3/601: fsync d3/d65/d89/fe4 0 2024-04-06T14:17:07.530 INFO:tasks.workunit.client.0.smithi032.stdout:5/587: symlink d16/d18/d33/d35/d88/lcc 0 2024-04-06T14:17:07.531 INFO:tasks.workunit.client.0.smithi032.stdout:3/602: dread - d3/d65/d7d/d9c/db8/fc2 zero size 2024-04-06T14:17:07.532 INFO:tasks.workunit.client.0.smithi032.stdout:9/627: dwrite d2/d15/d1e/d65/d83/d7e/f14 [0,4194304] 0 2024-04-06T14:17:07.534 INFO:tasks.workunit.client.0.smithi032.stdout:5/588: dread - d16/d58/d54/d74/d8f/d95/fc2 zero size 2024-04-06T14:17:07.534 INFO:tasks.workunit.client.0.smithi032.stdout:2/599: rename d0/d30/d39/l3d to d0/d30/d39/d59/d67/d74/d78/d8c/db0/lc7 0 2024-04-06T14:17:07.535 INFO:tasks.workunit.client.0.smithi032.stdout:2/600: readlink d0/l9 0 2024-04-06T14:17:07.538 INFO:tasks.workunit.client.0.smithi032.stdout:3/603: symlink d3/d65/d7d/d9c/dad/le7 0 2024-04-06T14:17:07.539 INFO:tasks.workunit.client.0.smithi032.stdout:6/601: dread d4/d5/dbf/f33 [0,4194304] 0 2024-04-06T14:17:07.540 INFO:tasks.workunit.client.0.smithi032.stdout:9/628: rename d2/d15/d55/d59/d93/d95/db8/d81/d4a/d73 to d2/dcd 0 2024-04-06T14:17:07.544 INFO:tasks.workunit.client.0.smithi032.stdout:2/601: creat d0/d30/fc8 x:0 0 0 2024-04-06T14:17:07.545 INFO:tasks.workunit.client.0.smithi032.stdout:4/653: truncate d0/fc3 610926 0 2024-04-06T14:17:07.546 INFO:tasks.workunit.client.0.smithi032.stdout:3/604: rename d3/d65/d7d/d9c/dad/db1/d3c/fc4 to d3/d16/fe8 0 2024-04-06T14:17:07.547 INFO:tasks.workunit.client.0.smithi032.stdout:3/605: write d3/fae [837756,106718] 0 2024-04-06T14:17:07.548 INFO:tasks.workunit.client.0.smithi032.stdout:5/589: creat d16/d18/d33/d35/d88/dbd/dc5/fcd x:0 0 0 2024-04-06T14:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout:8/579: truncate d3/d8/d12/d2f/d4a/f8b 2849815 0 2024-04-06T14:17:07.549 INFO:tasks.workunit.client.0.smithi032.stdout:5/590: write d16/d18/d33/d45/d52/f9e [745755,53190] 0 2024-04-06T14:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout:2/602: unlink d0/d30/d39/d59/d67/lc4 0 2024-04-06T14:17:07.551 INFO:tasks.workunit.client.0.smithi032.stdout:2/603: chown d0/c5e 777191 1 2024-04-06T14:17:07.552 INFO:tasks.workunit.client.0.smithi032.stdout:4/654: mknod d0/d82/d4b/db3/ce0 0 2024-04-06T14:17:07.554 INFO:tasks.workunit.client.0.smithi032.stdout:9/629: symlink d2/d15/d1e/d65/d83/d31/d39/da6/lce 0 2024-04-06T14:17:07.555 INFO:tasks.workunit.client.0.smithi032.stdout:9/630: readlink d2/d15/d1e/d65/d83/l3c 0 2024-04-06T14:17:07.557 INFO:tasks.workunit.client.0.smithi032.stdout:8/580: mkdir d3/d8/d12/d83/d89/da0/dd2 0 2024-04-06T14:17:07.560 INFO:tasks.workunit.client.0.smithi032.stdout:9/631: truncate d2/d15/d1e/d65/d83/d7e/faf 944824 0 2024-04-06T14:17:07.562 INFO:tasks.workunit.client.0.smithi032.stdout:8/581: write d3/d8/d3a/f56 [115843,37408] 0 2024-04-06T14:17:07.566 INFO:tasks.workunit.client.0.smithi032.stdout:5/591: write d16/d18/f3c [687671,58365] 0 2024-04-06T14:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout:6/602: dwrite d4/d5/d19/fc3 [0,4194304] 0 2024-04-06T14:17:07.567 INFO:tasks.workunit.client.0.smithi032.stdout:8/582: write d3/d8/d7/f47 [1937220,93024] 0 2024-04-06T14:17:07.569 INFO:tasks.workunit.client.0.smithi032.stdout:4/655: dread d0/d82/d4b/f63 [0,4194304] 0 2024-04-06T14:17:07.571 INFO:tasks.workunit.client.0.smithi032.stdout:5/592: chown d16/d18/d33/f8a 13 1 2024-04-06T14:17:07.573 INFO:tasks.workunit.client.0.smithi032.stdout:6/603: write d4/d5/d8/d2f/d9b/fa5 [435781,109192] 0 2024-04-06T14:17:07.574 INFO:tasks.workunit.client.0.smithi032.stdout:4/656: symlink d0/df/d23/d31/d50/d57/le1 0 2024-04-06T14:17:07.579 INFO:tasks.workunit.client.1.smithi086.stdout:7/341: mknod d7/d1d/c76 0 2024-04-06T14:17:07.579 INFO:tasks.workunit.client.1.smithi086.stdout:6/393: creat d6/d9/f8c x:0 0 0 2024-04-06T14:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout:5/593: creat d16/d18/dae/dbb/fce x:0 0 0 2024-04-06T14:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout:8/583: mknod d3/d8/d12/d2f/cd3 0 2024-04-06T14:17:07.579 INFO:tasks.workunit.client.0.smithi032.stdout:9/632: dwrite d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/fae [0,4194304] 0 2024-04-06T14:17:07.580 INFO:tasks.workunit.client.0.smithi032.stdout:6/604: truncate d4/d5/dbf/d61/f66 952451 0 2024-04-06T14:17:07.581 INFO:tasks.workunit.client.1.smithi086.stdout:7/342: write d7/d1d/f38 [1039000,131029] 0 2024-04-06T14:17:07.581 INFO:tasks.workunit.client.1.smithi086.stdout:3/437: symlink d5/d26/l84 0 2024-04-06T14:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout:8/584: write d3/d8/d7/f81 [82926,59720] 0 2024-04-06T14:17:07.582 INFO:tasks.workunit.client.0.smithi032.stdout:4/657: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2 0 2024-04-06T14:17:07.582 INFO:tasks.workunit.client.1.smithi086.stdout:6/394: write d6/d13/d42/f72 [485183,3195] 0 2024-04-06T14:17:07.582 INFO:tasks.workunit.client.1.smithi086.stdout:3/438: chown d5/d26/d4f 402402 1 2024-04-06T14:17:07.583 INFO:tasks.workunit.client.1.smithi086.stdout:6/395: fsync d6/d13/d42/d52/f7f 0 2024-04-06T14:17:07.584 INFO:tasks.workunit.client.0.smithi032.stdout:5/594: symlink d16/d18/d33/d35/lcf 0 2024-04-06T14:17:07.585 INFO:tasks.workunit.client.0.smithi032.stdout:6/605: creat d4/d5/d8/d4b/fd6 x:0 0 0 2024-04-06T14:17:07.586 INFO:tasks.workunit.client.0.smithi032.stdout:8/585: unlink d3/d8/d3a/f3b 0 2024-04-06T14:17:07.588 INFO:tasks.workunit.client.0.smithi032.stdout:6/606: read d4/d5/d19/d25/d67/f56 [1517748,106386] 0 2024-04-06T14:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout:5/595: mknod d16/d18/d33/d45/d87/cd0 0 2024-04-06T14:17:07.590 INFO:tasks.workunit.client.0.smithi032.stdout:4/658: creat d0/d82/fe3 x:0 0 0 2024-04-06T14:17:07.593 INFO:tasks.workunit.client.0.smithi032.stdout:8/586: creat d3/d8/d12/d2f/d4a/d60/d75/d85/fd4 x:0 0 0 2024-04-06T14:17:07.594 INFO:tasks.workunit.client.0.smithi032.stdout:4/659: chown d0/fc3 4002194 1 2024-04-06T14:17:07.596 INFO:tasks.workunit.client.1.smithi086.stdout:7/343: dwrite d7/d10/d11/d1c/d24/d2c/f6f [4194304,4194304] 0 2024-04-06T14:17:07.596 INFO:tasks.workunit.client.0.smithi032.stdout:4/660: readlink d0/df/d23/d31/l91 0 2024-04-06T14:17:07.598 INFO:tasks.workunit.client.0.smithi032.stdout:5/596: dwrite d16/d18/f20 [4194304,4194304] 0 2024-04-06T14:17:07.600 INFO:tasks.workunit.client.1.smithi086.stdout:3/439: dwrite d5/d36/d5a/d6d/d72/d58/f79 [0,4194304] 0 2024-04-06T14:17:07.600 INFO:tasks.workunit.client.0.smithi032.stdout:8/587: dread d3/d8/d12/d2f/f39 [0,4194304] 0 2024-04-06T14:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout:5/597: write d16/d18/dae/dbb/fce [860485,15622] 0 2024-04-06T14:17:07.601 INFO:tasks.workunit.client.0.smithi032.stdout:4/661: creat d0/df/d2b/dbe/fe4 x:0 0 0 2024-04-06T14:17:07.602 INFO:tasks.workunit.client.1.smithi086.stdout:7/344: creat d7/f77 x:0 0 0 2024-04-06T14:17:07.604 INFO:tasks.workunit.client.1.smithi086.stdout:6/396: dwrite d6/d13/d1a/d22/f3f [0,4194304] 0 2024-04-06T14:17:07.605 INFO:tasks.workunit.client.1.smithi086.stdout:6/397: readlink d6/d9/db/l54 0 2024-04-06T14:17:07.606 INFO:tasks.workunit.client.1.smithi086.stdout:1/389: dread db/d11/d1d/f58 [0,4194304] 0 2024-04-06T14:17:07.606 INFO:tasks.workunit.client.1.smithi086.stdout:3/440: creat d5/df/f85 x:0 0 0 2024-04-06T14:17:07.609 INFO:tasks.workunit.client.0.smithi032.stdout:5/598: rename f5 to d16/d58/d54/d74/fd1 0 2024-04-06T14:17:07.610 INFO:tasks.workunit.client.0.smithi032.stdout:8/588: fdatasync d3/f96 0 2024-04-06T14:17:07.611 INFO:tasks.workunit.client.1.smithi086.stdout:3/441: stat d5/d36/d56/f74 0 2024-04-06T14:17:07.611 INFO:tasks.workunit.client.1.smithi086.stdout:7/345: write d7/d10/d30/f50 [7252445,48005] 0 2024-04-06T14:17:07.611 INFO:tasks.workunit.client.1.smithi086.stdout:6/398: fdatasync d6/d13/d1a/f21 0 2024-04-06T14:17:07.611 INFO:tasks.workunit.client.1.smithi086.stdout:3/442: chown d5/d36/d5a/d6d/d72/d58 5308 1 2024-04-06T14:17:07.612 INFO:tasks.workunit.client.1.smithi086.stdout:1/390: write db/d11/d1f/f5a [559376,96725] 0 2024-04-06T14:17:07.617 INFO:tasks.workunit.client.1.smithi086.stdout:3/443: read d5/df/f6a [5175080,105297] 0 2024-04-06T14:17:07.618 INFO:tasks.workunit.client.1.smithi086.stdout:7/346: getdents d7/d10/d11/d1c/d36 0 2024-04-06T14:17:07.620 INFO:tasks.workunit.client.1.smithi086.stdout:3/444: write d5/d36/f67 [4807622,61893] 0 2024-04-06T14:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout:5/599: chown d16/d18/c55 195871 1 2024-04-06T14:17:07.620 INFO:tasks.workunit.client.0.smithi032.stdout:4/662: dwrite d0/df/d2b/fd9 [0,4194304] 0 2024-04-06T14:17:07.621 INFO:tasks.workunit.client.0.smithi032.stdout:8/589: mkdir d3/d8/d7/dd5 0 2024-04-06T14:17:07.624 INFO:tasks.workunit.client.0.smithi032.stdout:4/663: fdatasync d0/df/d23/f40 0 2024-04-06T14:17:07.626 INFO:tasks.workunit.client.1.smithi086.stdout:7/347: creat d7/d10/d11/d1c/d24/d2c/d69/f78 x:0 0 0 2024-04-06T14:17:07.626 INFO:tasks.workunit.client.0.smithi032.stdout:4/664: write d0/df/d23/f4a [5887984,56798] 0 2024-04-06T14:17:07.628 INFO:tasks.workunit.client.1.smithi086.stdout:3/445: symlink d5/d36/d7d/l86 0 2024-04-06T14:17:07.629 INFO:tasks.workunit.client.0.smithi032.stdout:4/665: write d0/df/d1f/d2e/f9b [1329336,95837] 0 2024-04-06T14:17:07.630 INFO:tasks.workunit.client.0.smithi032.stdout:6/607: dread d4/d5/d19/d25/f87 [0,4194304] 0 2024-04-06T14:17:07.631 INFO:tasks.workunit.client.1.smithi086.stdout:3/446: chown d5/d26/d7f/l53 0 1 2024-04-06T14:17:07.631 INFO:tasks.workunit.client.0.smithi032.stdout:6/608: write d4/d5/d8/d4b/fa6 [514521,83553] 0 2024-04-06T14:17:07.632 INFO:tasks.workunit.client.0.smithi032.stdout:8/590: getdents d3/d8/d12/d2f/d4a/d60 0 2024-04-06T14:17:07.633 INFO:tasks.workunit.client.0.smithi032.stdout:8/591: chown d3/d8/d7/d14/f44 42637424 1 2024-04-06T14:17:07.634 INFO:tasks.workunit.client.0.smithi032.stdout:6/609: write d4/d5/d19/f55 [811400,121524] 0 2024-04-06T14:17:07.635 INFO:tasks.workunit.client.0.smithi032.stdout:6/610: stat d4/d5/l20 0 2024-04-06T14:17:07.638 INFO:tasks.workunit.client.0.smithi032.stdout:6/611: creat d4/d5/d8/d2f/fd7 x:0 0 0 2024-04-06T14:17:07.641 INFO:tasks.workunit.client.1.smithi086.stdout:7/348: dwrite d7/d10/d11/d1c/f58 [0,4194304] 0 2024-04-06T14:17:07.642 INFO:tasks.workunit.client.1.smithi086.stdout:6/399: dread d6/d13/f27 [4194304,4194304] 0 2024-04-06T14:17:07.644 INFO:tasks.workunit.client.0.smithi032.stdout:6/612: rename d4/d5/d83/dad to d4/d5/dd8 0 2024-04-06T14:17:07.645 INFO:tasks.workunit.client.0.smithi032.stdout:6/613: chown d4/d5/d28/d65/f92 2 1 2024-04-06T14:17:07.647 INFO:tasks.workunit.client.1.smithi086.stdout:6/400: rename d6/d13/d5f/d60/l8b to d6/d13/d1a/d64/d85/l8d 0 2024-04-06T14:17:07.649 INFO:tasks.workunit.client.1.smithi086.stdout:7/349: creat d7/d10/d11/d1c/f79 x:0 0 0 2024-04-06T14:17:07.649 INFO:tasks.workunit.client.0.smithi032.stdout:8/592: dread d3/d8/d7/d14/f1f [0,4194304] 0 2024-04-06T14:17:07.653 INFO:tasks.workunit.client.0.smithi032.stdout:6/614: rmdir d4/d5/d19/d21/dbc 0 2024-04-06T14:17:07.653 INFO:tasks.workunit.client.1.smithi086.stdout:6/401: dread d6/d9/db/f28 [0,4194304] 0 2024-04-06T14:17:07.653 INFO:tasks.workunit.client.1.smithi086.stdout:6/402: chown d6/d13/d5f/d60/c84 420 1 2024-04-06T14:17:07.654 INFO:tasks.workunit.client.1.smithi086.stdout:6/403: chown d6/d13/d42/l5a 3 1 2024-04-06T14:17:07.654 INFO:tasks.workunit.client.0.smithi032.stdout:6/615: write d4/d5/d28/d65/fb1 [800084,76228] 0 2024-04-06T14:17:07.654 INFO:tasks.workunit.client.0.smithi032.stdout:8/593: mknod d3/d8/d12/d2f/d4a/d60/d90/cd6 0 2024-04-06T14:17:07.654 INFO:tasks.workunit.client.1.smithi086.stdout:6/404: stat d6/d13/d5f/d60/d45/d8a 0 2024-04-06T14:17:07.655 INFO:tasks.workunit.client.0.smithi032.stdout:6/616: read - d4/d5/d8/d2b/f90 zero size 2024-04-06T14:17:07.655 INFO:tasks.workunit.client.1.smithi086.stdout:1/391: dread db/d11/d42/d53/f57 [0,4194304] 0 2024-04-06T14:17:07.656 INFO:tasks.workunit.client.1.smithi086.stdout:1/392: fsync db/d11/d1f/f51 0 2024-04-06T14:17:07.657 INFO:tasks.workunit.client.1.smithi086.stdout:6/405: mknod d6/d13/c8e 0 2024-04-06T14:17:07.657 INFO:tasks.workunit.client.0.smithi032.stdout:6/617: creat d4/d5/d18/fd9 x:0 0 0 2024-04-06T14:17:07.658 INFO:tasks.workunit.client.1.smithi086.stdout:6/406: truncate d6/d9/d5c/f6b 518833 0 2024-04-06T14:17:07.659 INFO:tasks.workunit.client.1.smithi086.stdout:6/407: chown d6/d13/d5f/d60/c4e 0 1 2024-04-06T14:17:07.659 INFO:tasks.workunit.client.1.smithi086.stdout:1/393: mknod db/d11/d1a/d2a/d52/d5b/d68/c7e 0 2024-04-06T14:17:07.659 INFO:tasks.workunit.client.0.smithi032.stdout:8/594: mkdir d3/d8/d12/d2f/d4a/d60/d90/dc5/dd7 0 2024-04-06T14:17:07.659 INFO:tasks.workunit.client.0.smithi032.stdout:6/618: dread - d4/d5/d19/f46 zero size 2024-04-06T14:17:07.661 INFO:tasks.workunit.client.1.smithi086.stdout:8/491: sync 2024-04-06T14:17:07.661 INFO:tasks.workunit.client.1.smithi086.stdout:1/394: write db/d11/d1a/f4b [332722,95894] 0 2024-04-06T14:17:07.662 INFO:tasks.workunit.client.0.smithi032.stdout:6/619: mkdir d4/d5/dbf/d61/da7/dda 0 2024-04-06T14:17:07.662 INFO:tasks.workunit.client.1.smithi086.stdout:1/395: readlink db/d11/d1f/l2d 0 2024-04-06T14:17:07.663 INFO:tasks.workunit.client.0.smithi032.stdout:6/620: fdatasync d4/d5/d14/fb3 0 2024-04-06T14:17:07.665 INFO:tasks.workunit.client.1.smithi086.stdout:8/492: creat d0/db/fac x:0 0 0 2024-04-06T14:17:07.665 INFO:tasks.workunit.client.1.smithi086.stdout:7/350: write d7/d10/d11/d1c/f2e [958317,67477] 0 2024-04-06T14:17:07.667 INFO:tasks.workunit.client.0.smithi032.stdout:6/621: mkdir d4/d5/d28/d65/d6e/ddb 0 2024-04-06T14:17:07.668 INFO:tasks.workunit.client.0.smithi032.stdout:8/595: dwrite d3/d8/d12/d2f/d46/f74 [0,4194304] 0 2024-04-06T14:17:07.668 INFO:tasks.workunit.client.1.smithi086.stdout:7/351: rmdir d7/d10/d11/d1c/d36 39 2024-04-06T14:17:07.669 INFO:tasks.workunit.client.1.smithi086.stdout:7/352: write f5 [2751633,977] 0 2024-04-06T14:17:07.670 INFO:tasks.workunit.client.1.smithi086.stdout:7/353: stat d7/d10/d30/c47 0 2024-04-06T14:17:07.671 INFO:tasks.workunit.client.0.smithi032.stdout:6/622: mknod d4/d5/d19/cdc 0 2024-04-06T14:17:07.673 INFO:tasks.workunit.client.1.smithi086.stdout:7/354: write d7/d10/d11/d1c/d24/d2c/d3c/f6e [912489,14572] 0 2024-04-06T14:17:07.673 INFO:tasks.workunit.client.1.smithi086.stdout:6/408: dwrite d6/d13/d5f/d60/f5b [0,4194304] 0 2024-04-06T14:17:07.674 INFO:tasks.workunit.client.0.smithi032.stdout:6/623: unlink d4/d5/dbf/d53/fd4 0 2024-04-06T14:17:07.681 INFO:tasks.workunit.client.1.smithi086.stdout:6/409: rmdir d6/d9/db 39 2024-04-06T14:17:07.681 INFO:tasks.workunit.client.0.smithi032.stdout:6/624: dread d4/d5/dbf/f63 [0,4194304] 0 2024-04-06T14:17:07.682 INFO:tasks.workunit.client.0.smithi032.stdout:6/625: fdatasync d4/d5/f15 0 2024-04-06T14:17:07.683 INFO:tasks.workunit.client.1.smithi086.stdout:7/355: getdents d7/d10/d11/d1c/d24/d2c/d3c 0 2024-04-06T14:17:07.684 INFO:tasks.workunit.client.0.smithi032.stdout:6/626: write d4/d5/d8/d2f/fd7 [442350,48721] 0 2024-04-06T14:17:07.685 INFO:tasks.workunit.client.1.smithi086.stdout:8/493: dwrite d0/d3b/f46 [0,4194304] 0 2024-04-06T14:17:07.686 INFO:tasks.workunit.client.1.smithi086.stdout:8/494: chown d0/db/f12 46827123 1 2024-04-06T14:17:07.687 INFO:tasks.workunit.client.1.smithi086.stdout:8/495: read - d0/d71/d16/d39/d90/f94 zero size 2024-04-06T14:17:07.690 INFO:tasks.workunit.client.1.smithi086.stdout:6/410: rename d6/d9/d5c/f58 to d6/d13/d1a/f8f 0 2024-04-06T14:17:07.690 INFO:tasks.workunit.client.0.smithi032.stdout:8/596: dwrite d3/d8/f35 [0,4194304] 0 2024-04-06T14:17:07.692 INFO:tasks.workunit.client.1.smithi086.stdout:6/411: write f5 [2981093,52665] 0 2024-04-06T14:17:07.693 INFO:tasks.workunit.client.1.smithi086.stdout:6/412: chown d6/d13/d42/d52/f65 388358332 1 2024-04-06T14:17:07.694 INFO:tasks.workunit.client.1.smithi086.stdout:8/496: mkdir d0/db/d32/dab/dad 0 2024-04-06T14:17:07.695 INFO:tasks.workunit.client.1.smithi086.stdout:6/413: write f5 [1624409,81252] 0 2024-04-06T14:17:07.695 INFO:tasks.workunit.client.1.smithi086.stdout:6/414: dread - d6/d9/d43/f74 zero size 2024-04-06T14:17:07.698 INFO:tasks.workunit.client.0.smithi032.stdout:6/627: dwrite d4/d5/d19/f7d [0,4194304] 0 2024-04-06T14:17:07.699 INFO:tasks.workunit.client.1.smithi086.stdout:8/497: mknod d0/d71/d22/d65/d6e/d98/cae 0 2024-04-06T14:17:07.701 INFO:tasks.workunit.client.0.smithi032.stdout:6/628: chown d4/d5/d14/d80/c84 185250676 1 2024-04-06T14:17:07.706 INFO:tasks.workunit.client.1.smithi086.stdout:8/498: rename d0/d71/d16/d52/d5f/d9e to d0/d71/d16/d39/d90/daf 0 2024-04-06T14:17:07.706 INFO:tasks.workunit.client.0.smithi032.stdout:8/597: dwrite d3/d8/d12/d2f/d4a/d60/d82/d84/fd0 [0,4194304] 0 2024-04-06T14:17:07.706 INFO:tasks.workunit.client.1.smithi086.stdout:8/499: fdatasync d0/f2 0 2024-04-06T14:17:07.707 INFO:tasks.workunit.client.1.smithi086.stdout:8/500: chown d0/d71/d22/d43 3139658 1 2024-04-06T14:17:07.711 INFO:tasks.workunit.client.0.smithi032.stdout:8/598: truncate d3/d8/d3a/f5c 1044479 0 2024-04-06T14:17:07.711 INFO:tasks.workunit.client.1.smithi086.stdout:8/501: write d0/f26 [1316690,66769] 0 2024-04-06T14:17:07.711 INFO:tasks.workunit.client.1.smithi086.stdout:8/502: fdatasync d0/db/f12 0 2024-04-06T14:17:07.714 INFO:tasks.workunit.client.1.smithi086.stdout:8/503: truncate d0/d71/d22/f8e 37150 0 2024-04-06T14:17:07.714 INFO:tasks.workunit.client.0.smithi032.stdout:8/599: symlink d3/d8/d12/d2f/d4a/d60/d82/d84/ld8 0 2024-04-06T14:17:07.715 INFO:tasks.workunit.client.0.smithi032.stdout:8/600: chown d3/d8/f93 200 1 2024-04-06T14:17:07.716 INFO:tasks.workunit.client.1.smithi086.stdout:8/504: unlink d0/db/f92 0 2024-04-06T14:17:07.717 INFO:tasks.workunit.client.0.smithi032.stdout:8/601: truncate d3/d8/d12/fb0 377961 0 2024-04-06T14:17:07.720 INFO:tasks.workunit.client.1.smithi086.stdout:8/505: truncate d0/d71/d22/d43/d7f/f88 251485 0 2024-04-06T14:17:07.720 INFO:tasks.workunit.client.1.smithi086.stdout:7/356: dread d7/d10/d11/d1c/f60 [0,4194304] 0 2024-04-06T14:17:07.721 INFO:tasks.workunit.client.0.smithi032.stdout:8/602: getdents d3/d8/d3a/d4b 0 2024-04-06T14:17:07.722 INFO:tasks.workunit.client.0.smithi032.stdout:8/603: write d3/d8/d7/d1a/f78 [192226,13489] 0 2024-04-06T14:17:07.727 INFO:tasks.workunit.client.1.smithi086.stdout:7/357: dwrite d7/d10/d11/d1c/d24/d2c/d69/f71 [0,4194304] 0 2024-04-06T14:17:07.729 INFO:tasks.workunit.client.0.smithi032.stdout:8/604: dwrite d3/d8/f35 [0,4194304] 0 2024-04-06T14:17:07.731 INFO:tasks.workunit.client.1.smithi086.stdout:7/358: mkdir d7/d10/d11/d1c/d24/d2c/d3c/d7a 0 2024-04-06T14:17:07.732 INFO:tasks.workunit.client.1.smithi086.stdout:7/359: dread - d7/d10/d30/f4b zero size 2024-04-06T14:17:07.734 INFO:tasks.workunit.client.0.smithi032.stdout:8/605: creat d3/d8/d12/d2f/d4a/d60/d90/fd9 x:0 0 0 2024-04-06T14:17:07.734 INFO:tasks.workunit.client.1.smithi086.stdout:8/506: rmdir d0/d71/d16/d52 39 2024-04-06T14:17:07.735 INFO:tasks.workunit.client.0.smithi032.stdout:8/606: dread - d3/d8/d3a/d7b/db9/fcf zero size 2024-04-06T14:17:07.741 INFO:tasks.workunit.client.0.smithi032.stdout:8/607: symlink d3/d8/d12/d2f/d4a/d60/d75/dd1/lda 0 2024-04-06T14:17:07.747 INFO:tasks.workunit.client.0.smithi032.stdout:8/608: dwrite d3/d8/d7/f16 [0,4194304] 0 2024-04-06T14:17:07.749 INFO:tasks.workunit.client.0.smithi032.stdout:8/609: symlink d3/d8/d12/d2f/d4a/d60/d90/ldb 0 2024-04-06T14:17:07.751 INFO:tasks.workunit.client.0.smithi032.stdout:8/610: read d3/d8/d12/d2f/d4a/f8b [2361811,104073] 0 2024-04-06T14:17:07.754 INFO:tasks.workunit.client.1.smithi086.stdout:7/360: truncate d7/d10/d11/d1c/f60 3171634 0 2024-04-06T14:17:07.754 INFO:tasks.workunit.client.0.smithi032.stdout:8/611: write d3/d8/fac [508092,123956] 0 2024-04-06T14:17:07.756 INFO:tasks.workunit.client.1.smithi086.stdout:7/361: getdents d7/d10/d30 0 2024-04-06T14:17:07.759 INFO:tasks.workunit.client.1.smithi086.stdout:7/362: symlink d7/d10/d11/d1c/d24/d2c/d3c/l7b 0 2024-04-06T14:17:07.760 INFO:tasks.workunit.client.0.smithi032.stdout:8/612: rmdir d3/d8/d12 39 2024-04-06T14:17:07.761 INFO:tasks.workunit.client.1.smithi086.stdout:7/363: unlink d7/d10/d11/d1c/d24/d4d/l68 0 2024-04-06T14:17:07.770 INFO:tasks.workunit.client.0.smithi032.stdout:8/613: dread d3/d8/f7a [0,4194304] 0 2024-04-06T14:17:07.770 INFO:tasks.workunit.client.0.smithi032.stdout:8/614: fdatasync d3/f79 0 2024-04-06T14:17:07.773 INFO:tasks.workunit.client.0.smithi032.stdout:8/615: rmdir d3/d8/d12/d2f/d4a/d60/d75 39 2024-04-06T14:17:07.775 INFO:tasks.workunit.client.0.smithi032.stdout:8/616: mknod d3/d8/d3a/d4b/cdc 0 2024-04-06T14:17:07.789 INFO:tasks.workunit.client.0.smithi032.stdout:8/617: dread d3/d8/d12/f38 [0,4194304] 0 2024-04-06T14:17:07.790 INFO:tasks.workunit.client.0.smithi032.stdout:8/618: write d3/d8/d12/f4d [6616684,57169] 0 2024-04-06T14:17:07.796 INFO:tasks.workunit.client.0.smithi032.stdout:8/619: rename d3/d8/d3a/f56 to d3/d8/d12/d2f/d4a/d60/d82/d84/fdd 0 2024-04-06T14:17:07.797 INFO:tasks.workunit.client.0.smithi032.stdout:8/620: mknod d3/d8/d3a/d4b/cde 0 2024-04-06T14:17:07.798 INFO:tasks.workunit.client.0.smithi032.stdout:8/621: read d3/d8/f25 [4557429,89163] 0 2024-04-06T14:17:07.798 INFO:tasks.workunit.client.0.smithi032.stdout:8/622: fsync d3/d8/d12/d2f/d4a/f94 0 2024-04-06T14:17:07.801 INFO:tasks.workunit.client.0.smithi032.stdout:8/623: mkdir d3/d8/d3a/d7b/db9/ddf 0 2024-04-06T14:17:07.801 INFO:tasks.workunit.client.0.smithi032.stdout:8/624: stat d3/d8/d12/d2f/d46/f88 0 2024-04-06T14:17:07.803 INFO:tasks.workunit.client.0.smithi032.stdout:8/625: mkdir d3/d8/d3a/d4b/de0 0 2024-04-06T14:17:07.805 INFO:tasks.workunit.client.0.smithi032.stdout:8/626: mknod d3/d8/d3a/ce1 0 2024-04-06T14:17:07.806 INFO:tasks.workunit.client.0.smithi032.stdout:8/627: symlink d3/d8/d3a/d4b/le2 0 2024-04-06T14:17:07.808 INFO:tasks.workunit.client.0.smithi032.stdout:8/628: mknod d3/d8/d7/d8a/ce3 0 2024-04-06T14:17:07.813 INFO:tasks.workunit.client.0.smithi032.stdout:8/629: dwrite d3/f79 [0,4194304] 0 2024-04-06T14:17:07.822 INFO:tasks.workunit.client.0.smithi032.stdout:8/630: dread d3/d8/f6d [0,4194304] 0 2024-04-06T14:17:07.825 INFO:tasks.workunit.client.0.smithi032.stdout:8/631: dread d3/d8/d12/d2f/d46/fa8 [4194304,4194304] 0 2024-04-06T14:17:07.828 INFO:tasks.workunit.client.0.smithi032.stdout:8/632: rename d3/d69/cb4 to d3/d8/d7/d1a/ce4 0 2024-04-06T14:17:07.828 INFO:tasks.workunit.client.1.smithi086.stdout:5/364: sync 2024-04-06T14:17:07.831 INFO:tasks.workunit.client.0.smithi032.stdout:8/633: unlink d3/d8/d12/f5b 0 2024-04-06T14:17:07.834 INFO:tasks.workunit.client.0.smithi032.stdout:8/634: creat d3/d8/d3a/d4b/de0/fe5 x:0 0 0 2024-04-06T14:17:07.836 INFO:tasks.workunit.client.1.smithi086.stdout:5/365: truncate d0/f3f 1048696 0 2024-04-06T14:17:07.836 INFO:tasks.workunit.client.0.smithi032.stdout:8/635: creat d3/d69/fe6 x:0 0 0 2024-04-06T14:17:07.837 INFO:tasks.workunit.client.0.smithi032.stdout:8/636: chown d3/d8/d7/fa7 102905096 1 2024-04-06T14:17:07.838 INFO:tasks.workunit.client.0.smithi032.stdout:8/637: read - d3/d8/d12/d83/fba zero size 2024-04-06T14:17:07.838 INFO:tasks.workunit.client.0.smithi032.stdout:8/638: fsync d3/d8/d7/d14/f44 0 2024-04-06T14:17:07.841 INFO:tasks.workunit.client.0.smithi032.stdout:8/639: mknod d3/d8/d7/d14/ce7 0 2024-04-06T14:17:07.842 INFO:tasks.workunit.client.1.smithi086.stdout:5/366: dwrite d0/fa [0,4194304] 0 2024-04-06T14:17:07.844 INFO:tasks.workunit.client.1.smithi086.stdout:5/367: rmdir d0/d16/d18/d21/d23 39 2024-04-06T14:17:07.846 INFO:tasks.workunit.client.1.smithi086.stdout:5/368: write d0/d16/f4e [336796,61324] 0 2024-04-06T14:17:07.853 INFO:tasks.workunit.client.0.smithi032.stdout:8/640: dwrite d3/d8/d12/d2f/d4a/f94 [0,4194304] 0 2024-04-06T14:17:07.866 INFO:tasks.workunit.client.0.smithi032.stdout:8/641: dwrite d3/d8/f25 [8388608,4194304] 0 2024-04-06T14:17:07.876 INFO:tasks.workunit.client.0.smithi032.stdout:8/642: dwrite d3/f96 [0,4194304] 0 2024-04-06T14:17:07.881 INFO:tasks.workunit.client.0.smithi032.stdout:8/643: dread d3/d8/d12/d2f/f62 [0,4194304] 0 2024-04-06T14:17:07.886 INFO:tasks.workunit.client.1.smithi086.stdout:5/369: dwrite d0/d5e/f60 [0,4194304] 0 2024-04-06T14:17:07.886 INFO:tasks.workunit.client.0.smithi032.stdout:8/644: dread - d3/d8/d12/d2f/d4a/d60/d82/fa1 zero size 2024-04-06T14:17:07.887 INFO:tasks.workunit.client.1.smithi086.stdout:5/370: symlink d0/d16/d18/d2a/d50/d66/l6e 0 2024-04-06T14:17:07.895 INFO:tasks.workunit.client.1.smithi086.stdout:5/371: link d0/d19/c1e d0/d16/d18/d2a/c6f 0 2024-04-06T14:17:07.897 INFO:tasks.workunit.client.1.smithi086.stdout:5/372: symlink d0/d43/l70 0 2024-04-06T14:17:07.912 INFO:tasks.workunit.client.1.smithi086.stdout:0/411: sync 2024-04-06T14:17:07.912 INFO:tasks.workunit.client.1.smithi086.stdout:2/384: sync 2024-04-06T14:17:07.918 INFO:tasks.workunit.client.1.smithi086.stdout:5/373: dwrite d0/d16/d18/d35/f4d [0,4194304] 0 2024-04-06T14:17:07.920 INFO:tasks.workunit.client.1.smithi086.stdout:0/412: link d0/d1/f13 d0/d15/d4e/f81 0 2024-04-06T14:17:07.922 INFO:tasks.workunit.client.1.smithi086.stdout:2/385: rename d0/d20/d27/d2b/d35/d36/f3e to d0/d20/d27/d2b/d35/d36/d4a/f86 0 2024-04-06T14:17:07.923 INFO:tasks.workunit.client.1.smithi086.stdout:2/386: fsync d0/d20/d27/d2b/d35/d36/d4d/f4f 0 2024-04-06T14:17:07.924 INFO:tasks.workunit.client.1.smithi086.stdout:0/413: unlink d0/d1/d1b/c6e 0 2024-04-06T14:17:07.924 INFO:tasks.workunit.client.1.smithi086.stdout:0/414: truncate d0/f40 5155473 0 2024-04-06T14:17:07.925 INFO:tasks.workunit.client.1.smithi086.stdout:5/374: getdents d0/d19 0 2024-04-06T14:17:07.926 INFO:tasks.workunit.client.1.smithi086.stdout:2/387: creat d0/df/d48/d7d/f87 x:0 0 0 2024-04-06T14:17:07.929 INFO:tasks.workunit.client.1.smithi086.stdout:2/388: creat d0/f88 x:0 0 0 2024-04-06T14:17:07.932 INFO:tasks.workunit.client.1.smithi086.stdout:5/375: dwrite d0/d43/f5c [0,4194304] 0 2024-04-06T14:17:07.935 INFO:tasks.workunit.client.1.smithi086.stdout:2/389: rename d0/d20/d27/d2b/d35/d36/d4d/d6f/f73 to d0/df/d48/d7d/f89 0 2024-04-06T14:17:07.937 INFO:tasks.workunit.client.1.smithi086.stdout:2/390: rmdir d0/d20/d27/d2b/d35/d36/d4d/d59 39 2024-04-06T14:17:07.940 INFO:tasks.workunit.client.1.smithi086.stdout:0/415: dread d0/d1/d25/f61 [0,4194304] 0 2024-04-06T14:17:07.949 INFO:tasks.workunit.client.1.smithi086.stdout:2/391: dread d0/d20/d27/d2b/d35/f62 [0,4194304] 0 2024-04-06T14:17:07.988 INFO:tasks.workunit.client.1.smithi086.stdout:2/392: truncate d0/df/f14 1839286 0 2024-04-06T14:17:07.991 INFO:tasks.workunit.client.1.smithi086.stdout:2/393: write d0/d20/d27/d2b/d35/d36/f3b [1660697,128259] 0 2024-04-06T14:17:07.993 INFO:tasks.workunit.client.1.smithi086.stdout:0/416: dwrite d0/d1/d2/d1a/f7b [0,4194304] 0 2024-04-06T14:17:07.995 INFO:tasks.workunit.client.1.smithi086.stdout:2/394: creat d0/d20/d27/f8a x:0 0 0 2024-04-06T14:17:07.997 INFO:tasks.workunit.client.1.smithi086.stdout:0/417: getdents d0/d1/d1b/d44/d5d/d74 0 2024-04-06T14:17:07.999 INFO:tasks.workunit.client.1.smithi086.stdout:2/395: write d0/d20/d27/f8a [989144,128149] 0 2024-04-06T14:17:08.005 INFO:tasks.workunit.client.1.smithi086.stdout:2/396: creat d0/d20/d27/d2b/d35/d36/d4d/d59/f8b x:0 0 0 2024-04-06T14:17:08.006 INFO:tasks.workunit.client.1.smithi086.stdout:2/397: write d0/d20/d27/d2b/d35/d36/f68 [48416,81674] 0 2024-04-06T14:17:08.010 INFO:tasks.workunit.client.1.smithi086.stdout:2/398: creat d0/df/d48/d53/d76/d78/f8c x:0 0 0 2024-04-06T14:17:08.012 INFO:tasks.workunit.client.1.smithi086.stdout:2/399: creat d0/df/d48/d53/d76/d78/f8d x:0 0 0 2024-04-06T14:17:08.016 INFO:tasks.workunit.client.1.smithi086.stdout:2/400: creat d0/f8e x:0 0 0 2024-04-06T14:17:08.018 INFO:tasks.workunit.client.1.smithi086.stdout:2/401: fdatasync d0/df/d48/d53/d76/d78/f8c 0 2024-04-06T14:17:08.018 INFO:tasks.workunit.client.1.smithi086.stdout:0/418: dwrite d0/d1/d1b/d46/d52/f7f [0,4194304] 0 2024-04-06T14:17:08.019 INFO:tasks.workunit.client.1.smithi086.stdout:2/402: chown d0/df/l32 238128 1 2024-04-06T14:17:08.021 INFO:tasks.workunit.client.0.smithi032.stdout:0/434: sync 2024-04-06T14:17:08.021 INFO:tasks.workunit.client.0.smithi032.stdout:7/636: sync 2024-04-06T14:17:08.024 INFO:tasks.workunit.client.1.smithi086.stdout:0/419: dwrite d0/f40 [0,4194304] 0 2024-04-06T14:17:08.025 INFO:tasks.workunit.client.0.smithi032.stdout:7/637: mknod d16/d20/d48/d6d/ce3 0 2024-04-06T14:17:08.025 INFO:tasks.workunit.client.0.smithi032.stdout:0/435: truncate d3/da/d25/d2b/f32 8378099 0 2024-04-06T14:17:08.025 INFO:tasks.workunit.client.1.smithi086.stdout:0/420: chown d0/d1/d1b/d46/l4d 2969 1 2024-04-06T14:17:08.027 INFO:tasks.workunit.client.1.smithi086.stdout:4/290: sync 2024-04-06T14:17:08.028 INFO:tasks.workunit.client.1.smithi086.stdout:9/361: sync 2024-04-06T14:17:08.028 INFO:tasks.workunit.client.1.smithi086.stdout:3/447: sync 2024-04-06T14:17:08.028 INFO:tasks.workunit.client.1.smithi086.stdout:2/403: rename d0/c16 to d0/c8f 0 2024-04-06T14:17:08.029 INFO:tasks.workunit.client.1.smithi086.stdout:9/362: chown d5/d13/f44 3922232 1 2024-04-06T14:17:08.030 INFO:tasks.workunit.client.1.smithi086.stdout:4/291: write d0/d15/d20/f22 [1942978,45724] 0 2024-04-06T14:17:08.031 INFO:tasks.workunit.client.1.smithi086.stdout:2/404: symlink d0/d20/d27/l90 0 2024-04-06T14:17:08.032 INFO:tasks.workunit.client.0.smithi032.stdout:0/436: getdents d3/da/d35/d8b 0 2024-04-06T14:17:08.033 INFO:tasks.workunit.client.1.smithi086.stdout:9/363: chown d5/d10/d14/f51 173493222 1 2024-04-06T14:17:08.033 INFO:tasks.workunit.client.1.smithi086.stdout:3/448: mknod d5/d26/c87 0 2024-04-06T14:17:08.034 INFO:tasks.workunit.client.1.smithi086.stdout:4/292: creat d0/d15/d20/f57 x:0 0 0 2024-04-06T14:17:08.037 INFO:tasks.workunit.client.1.smithi086.stdout:0/421: dwrite d0/d15/d4e/f5e [0,4194304] 0 2024-04-06T14:17:08.038 INFO:tasks.workunit.client.1.smithi086.stdout:4/293: write d0/d15/d20/f44 [2415110,76821] 0 2024-04-06T14:17:08.039 INFO:tasks.workunit.client.1.smithi086.stdout:0/422: write d0/d15/d4e/f5e [254245,92167] 0 2024-04-06T14:17:08.041 INFO:tasks.workunit.client.0.smithi032.stdout:7/638: write d16/d20/d26/d5f/f65 [1613632,59884] 0 2024-04-06T14:17:08.041 INFO:tasks.workunit.client.1.smithi086.stdout:3/449: rename d5/d33/f32 to d5/d26/d4f/f88 0 2024-04-06T14:17:08.043 INFO:tasks.workunit.client.0.smithi032.stdout:7/639: fdatasync d16/d20/d26/d2d/faf 0 2024-04-06T14:17:08.047 INFO:tasks.workunit.client.1.smithi086.stdout:2/405: getdents d0/d20/d27 0 2024-04-06T14:17:08.048 INFO:tasks.workunit.client.0.smithi032.stdout:0/437: dwrite d3/f63 [0,4194304] 0 2024-04-06T14:17:08.048 INFO:tasks.workunit.client.0.smithi032.stdout:7/640: rename d16/d20/d26 to d16/d20/d54/de4 0 2024-04-06T14:17:08.048 INFO:tasks.workunit.client.1.smithi086.stdout:3/450: mkdir d5/d36/d5a/d6d/d72/d58/d89 0 2024-04-06T14:17:08.049 INFO:tasks.workunit.client.0.smithi032.stdout:0/438: stat d3/da/f23 0 2024-04-06T14:17:08.050 INFO:tasks.workunit.client.1.smithi086.stdout:4/294: dwrite d0/d15/f48 [0,4194304] 0 2024-04-06T14:17:08.052 INFO:tasks.workunit.client.0.smithi032.stdout:0/439: write d3/f63 [2501107,39883] 0 2024-04-06T14:17:08.053 INFO:tasks.workunit.client.1.smithi086.stdout:2/406: creat d0/df/d48/d53/d76/f91 x:0 0 0 2024-04-06T14:17:08.056 INFO:tasks.workunit.client.1.smithi086.stdout:3/451: unlink d5/f37 0 2024-04-06T14:17:08.059 INFO:tasks.workunit.client.1.smithi086.stdout:4/295: chown d0/d15/d2b/l39 7 1 2024-04-06T14:17:08.060 INFO:tasks.workunit.client.1.smithi086.stdout:2/407: write d0/df/f18 [461690,55320] 0 2024-04-06T14:17:08.061 INFO:tasks.workunit.client.1.smithi086.stdout:3/452: rmdir d5/df 39 2024-04-06T14:17:08.063 INFO:tasks.workunit.client.0.smithi032.stdout:7/641: mknod d16/d20/d7a/da6/ce5 0 2024-04-06T14:17:08.063 INFO:tasks.workunit.client.1.smithi086.stdout:2/408: read d0/d20/d44/f46 [86324,71651] 0 2024-04-06T14:17:08.065 INFO:tasks.workunit.client.1.smithi086.stdout:0/423: truncate d0/d1/d1b/f39 2840433 0 2024-04-06T14:17:08.070 INFO:tasks.workunit.client.1.smithi086.stdout:3/453: dwrite d5/d36/f3b [4194304,4194304] 0 2024-04-06T14:17:08.072 INFO:tasks.workunit.client.0.smithi032.stdout:7/642: creat d16/d20/d54/de4/d2d/d30/d3e/da7/db5/fe6 x:0 0 0 2024-04-06T14:17:08.072 INFO:tasks.workunit.client.0.smithi032.stdout:0/440: dwrite d3/da/d25/d2b/f3a [0,4194304] 0 2024-04-06T14:17:08.072 INFO:tasks.workunit.client.1.smithi086.stdout:3/454: fdatasync d5/d36/d65/f7b 0 2024-04-06T14:17:08.074 INFO:tasks.workunit.client.1.smithi086.stdout:0/424: dwrite d0/f16 [0,4194304] 0 2024-04-06T14:17:08.076 INFO:tasks.workunit.client.1.smithi086.stdout:4/296: link d0/d15/d20/l43 d0/d15/d19/l58 0 2024-04-06T14:17:08.076 INFO:tasks.workunit.client.1.smithi086.stdout:0/425: stat d0/d1/d2/d1a/d79 0 2024-04-06T14:17:08.078 INFO:tasks.workunit.client.1.smithi086.stdout:2/409: creat d0/d20/d27/d2b/d35/d36/f92 x:0 0 0 2024-04-06T14:17:08.079 INFO:tasks.workunit.client.1.smithi086.stdout:2/410: dread - d0/d20/d27/f7f zero size 2024-04-06T14:17:08.082 INFO:tasks.workunit.client.0.smithi032.stdout:7/643: rename d16/d20/d54/de4/d2d/d30/f52 to d16/d20/d54/de4/d70/fe7 0 2024-04-06T14:17:08.091 INFO:tasks.workunit.client.0.smithi032.stdout:7/644: fsync d16/d20/d63/f72 0 2024-04-06T14:17:08.091 INFO:tasks.workunit.client.1.smithi086.stdout:3/455: creat d5/d26/d7f/d59/f8a x:0 0 0 2024-04-06T14:17:08.091 INFO:tasks.workunit.client.1.smithi086.stdout:0/426: truncate d0/d1/d4/de/f3c 78470 0 2024-04-06T14:17:08.091 INFO:tasks.workunit.client.1.smithi086.stdout:4/297: creat d0/f59 x:0 0 0 2024-04-06T14:17:08.091 INFO:tasks.workunit.client.1.smithi086.stdout:3/456: truncate d5/d26/d7f/d59/f8a 926058 0 2024-04-06T14:17:08.092 INFO:tasks.workunit.client.0.smithi032.stdout:0/441: write d3/da/ff [2979078,15303] 0 2024-04-06T14:17:08.092 INFO:tasks.workunit.client.0.smithi032.stdout:7/645: read - d16/d20/d54/f9c zero size 2024-04-06T14:17:08.092 INFO:tasks.workunit.client.1.smithi086.stdout:4/298: write d0/d15/f48 [3249764,59361] 0 2024-04-06T14:17:08.092 INFO:tasks.workunit.client.1.smithi086.stdout:0/427: creat d0/d15/d4e/f82 x:0 0 0 2024-04-06T14:17:08.093 INFO:tasks.workunit.client.1.smithi086.stdout:4/299: write d0/d15/f48 [530937,42351] 0 2024-04-06T14:17:08.094 INFO:tasks.workunit.client.1.smithi086.stdout:2/411: link d0/d20/d27/c60 d0/d20/d82/c93 0 2024-04-06T14:17:08.099 INFO:tasks.workunit.client.1.smithi086.stdout:0/428: unlink d0/d15/d4e/d48/c78 0 2024-04-06T14:17:08.101 INFO:tasks.workunit.client.1.smithi086.stdout:4/300: rename d0/d15/d20/f22 to d0/de/d4c/f5a 0 2024-04-06T14:17:08.105 INFO:tasks.workunit.client.1.smithi086.stdout:1/396: sync 2024-04-06T14:17:08.105 INFO:tasks.workunit.client.1.smithi086.stdout:8/507: sync 2024-04-06T14:17:08.105 INFO:tasks.workunit.client.1.smithi086.stdout:7/364: sync 2024-04-06T14:17:08.105 INFO:tasks.workunit.client.1.smithi086.stdout:6/415: sync 2024-04-06T14:17:08.106 INFO:tasks.workunit.client.1.smithi086.stdout:1/397: readlink db/d11/d1d/l32 0 2024-04-06T14:17:08.109 INFO:tasks.workunit.client.0.smithi032.stdout:7/646: dread d16/d20/d54/de4/d2d/f41 [0,4194304] 0 2024-04-06T14:17:08.109 INFO:tasks.workunit.client.1.smithi086.stdout:4/301: unlink d0/f17 0 2024-04-06T14:17:08.112 INFO:tasks.workunit.client.1.smithi086.stdout:2/412: dread d0/f33 [0,4194304] 0 2024-04-06T14:17:08.113 INFO:tasks.workunit.client.0.smithi032.stdout:7/647: creat d16/d20/d54/de4/d70/fe8 x:0 0 0 2024-04-06T14:17:08.113 INFO:tasks.workunit.client.1.smithi086.stdout:6/416: write d6/d9/f10 [3819639,59837] 0 2024-04-06T14:17:08.115 INFO:tasks.workunit.client.1.smithi086.stdout:6/417: chown d6/d13/d42/f72 56444309 1 2024-04-06T14:17:08.118 INFO:tasks.workunit.client.0.smithi032.stdout:0/442: dread d3/da/d25/d2b/f32 [0,4194304] 0 2024-04-06T14:17:08.120 INFO:tasks.workunit.client.1.smithi086.stdout:4/302: rename d0/de/f46 to d0/d2c/d38/f5b 0 2024-04-06T14:17:08.120 INFO:tasks.workunit.client.0.smithi032.stdout:0/443: mknod d3/da/c98 0 2024-04-06T14:17:08.122 INFO:tasks.workunit.client.1.smithi086.stdout:6/418: dwrite d6/d9/f6c [0,4194304] 0 2024-04-06T14:17:08.123 INFO:tasks.workunit.client.1.smithi086.stdout:3/457: dwrite d5/df/f2c [0,4194304] 0 2024-04-06T14:17:08.124 INFO:tasks.workunit.client.0.smithi032.stdout:0/444: symlink d3/da/d25/d2b/d4a/d91/l99 0 2024-04-06T14:17:08.127 INFO:tasks.workunit.client.1.smithi086.stdout:1/398: creat db/d13/d71/f7f x:0 0 0 2024-04-06T14:17:08.127 INFO:tasks.workunit.client.1.smithi086.stdout:7/365: mknod d7/d10/d11/d1c/d24/d2c/d3c/d7a/c7c 0 2024-04-06T14:17:08.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:07 smithi086 ceph-mon[47925]: pgmap v21: 65 pgs: 65 active+clean; 1.5 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 27 MiB/s rd, 168 MiB/s wr, 260 op/s 2024-04-06T14:17:08.133 INFO:tasks.workunit.client.1.smithi086.stdout:1/399: write db/d11/d1f/f7c [701533,37409] 0 2024-04-06T14:17:08.133 INFO:tasks.workunit.client.1.smithi086.stdout:0/429: write d0/d1/d4/de/f3c [178044,44284] 0 2024-04-06T14:17:08.133 INFO:tasks.workunit.client.1.smithi086.stdout:4/303: dwrite d0/f59 [0,4194304] 0 2024-04-06T14:17:08.136 INFO:tasks.workunit.client.1.smithi086.stdout:6/419: creat d6/d13/d42/d52/f90 x:0 0 0 2024-04-06T14:17:08.139 INFO:tasks.workunit.client.1.smithi086.stdout:2/413: write d0/d20/d27/d2b/d35/d36/f3a [354070,122480] 0 2024-04-06T14:17:08.140 INFO:tasks.workunit.client.1.smithi086.stdout:0/430: creat d0/d1/d2/d1a/d51/f83 x:0 0 0 2024-04-06T14:17:08.141 INFO:tasks.workunit.client.1.smithi086.stdout:1/400: truncate fa 3629511 0 2024-04-06T14:17:08.142 INFO:tasks.workunit.client.1.smithi086.stdout:4/304: symlink d0/d15/d53/l5c 0 2024-04-06T14:17:08.144 INFO:tasks.workunit.client.1.smithi086.stdout:3/458: mknod d5/d36/d5a/d6d/d72/d57/c8b 0 2024-04-06T14:17:08.144 INFO:tasks.workunit.client.1.smithi086.stdout:1/401: mknod db/d11/d42/c80 0 2024-04-06T14:17:08.146 INFO:tasks.workunit.client.1.smithi086.stdout:7/366: creat d7/d10/d11/d1c/d24/f7d x:0 0 0 2024-04-06T14:17:08.146 INFO:tasks.workunit.client.1.smithi086.stdout:3/459: write d5/d33/f78 [542678,101546] 0 2024-04-06T14:17:08.146 INFO:tasks.workunit.client.1.smithi086.stdout:3/460: rename d5/df to d5/df/d8c 22 2024-04-06T14:17:08.147 INFO:tasks.workunit.client.0.smithi032.stdout:3/606: sync 2024-04-06T14:17:08.147 INFO:tasks.workunit.client.0.smithi032.stdout:2/604: sync 2024-04-06T14:17:08.147 INFO:tasks.workunit.client.0.smithi032.stdout:1/685: sync 2024-04-06T14:17:08.149 INFO:tasks.workunit.client.0.smithi032.stdout:1/686: dread - d0/d10/d16/d3f/fa3 zero size 2024-04-06T14:17:08.150 INFO:tasks.workunit.client.0.smithi032.stdout:2/605: write d0/d5a/f9d [578085,10179] 0 2024-04-06T14:17:08.151 INFO:tasks.workunit.client.1.smithi086.stdout:0/431: mkdir d0/d1/d2/d84 0 2024-04-06T14:17:08.151 INFO:tasks.workunit.client.1.smithi086.stdout:1/402: chown db/d11/c20 60873135 1 2024-04-06T14:17:08.152 INFO:tasks.workunit.client.0.smithi032.stdout:1/687: chown d0/l9b 2543330 1 2024-04-06T14:17:08.152 INFO:tasks.workunit.client.1.smithi086.stdout:7/367: mkdir d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e 0 2024-04-06T14:17:08.152 INFO:tasks.workunit.client.1.smithi086.stdout:1/403: chown db/d11/d1a/d2a/d52 0 1 2024-04-06T14:17:08.155 INFO:tasks.workunit.client.0.smithi032.stdout:2/606: mkdir d0/d5a/dc9 0 2024-04-06T14:17:08.156 INFO:tasks.workunit.client.1.smithi086.stdout:8/508: truncate d0/d71/d22/f83 424922 0 2024-04-06T14:17:08.157 INFO:tasks.workunit.client.1.smithi086.stdout:0/432: readlink d0/d1/d25/l3b 0 2024-04-06T14:17:08.158 INFO:tasks.workunit.client.1.smithi086.stdout:7/368: rmdir d7/d10/d11 39 2024-04-06T14:17:08.158 INFO:tasks.workunit.client.1.smithi086.stdout:0/433: readlink d0/d1/d1b/d46/l4d 0 2024-04-06T14:17:08.161 INFO:tasks.workunit.client.0.smithi032.stdout:3/607: dwrite d3/fd7 [0,4194304] 0 2024-04-06T14:17:08.162 INFO:tasks.workunit.client.0.smithi032.stdout:1/688: getdents d0/d41/d5b 0 2024-04-06T14:17:08.164 INFO:tasks.workunit.client.1.smithi086.stdout:3/461: rename d5/f20 to d5/f8d 0 2024-04-06T14:17:08.164 INFO:tasks.workunit.client.0.smithi032.stdout:0/445: dwrite d3/da/d25/d2b/d4a/f65 [0,4194304] 0 2024-04-06T14:17:08.167 INFO:tasks.workunit.client.1.smithi086.stdout:3/462: write d5/d36/f67 [3931044,87522] 0 2024-04-06T14:17:08.169 INFO:tasks.workunit.client.0.smithi032.stdout:3/608: symlink d3/le9 0 2024-04-06T14:17:08.170 INFO:tasks.workunit.client.1.smithi086.stdout:4/305: dread d0/d15/d20/d34/f35 [0,4194304] 0 2024-04-06T14:17:08.171 INFO:tasks.workunit.client.0.smithi032.stdout:1/689: truncate d0/d9/d1f/d7c/da9/fb3 883975 0 2024-04-06T14:17:08.172 INFO:tasks.workunit.client.0.smithi032.stdout:3/609: stat d3/d65/d7d/d9c/dad/db1/d3c/f57 0 2024-04-06T14:17:08.173 INFO:tasks.workunit.client.1.smithi086.stdout:0/434: dread d0/d1/d2/d1a/f56 [0,4194304] 0 2024-04-06T14:17:08.173 INFO:tasks.workunit.client.1.smithi086.stdout:8/509: dwrite d0/d71/d22/d43/d6d/f80 [0,4194304] 0 2024-04-06T14:17:08.173 INFO:tasks.workunit.client.1.smithi086.stdout:4/306: write d0/d15/d20/f57 [963816,4045] 0 2024-04-06T14:17:08.173 INFO:tasks.workunit.client.0.smithi032.stdout:2/607: creat d0/d5a/dc9/fca x:0 0 0 2024-04-06T14:17:08.174 INFO:tasks.workunit.client.1.smithi086.stdout:1/404: dread db/fe [0,4194304] 0 2024-04-06T14:17:08.175 INFO:tasks.workunit.client.0.smithi032.stdout:0/446: read d3/da/d41/f43 [4086278,79172] 0 2024-04-06T14:17:08.177 INFO:tasks.workunit.client.1.smithi086.stdout:4/307: write d0/d2c/d30/f3b [1027163,85290] 0 2024-04-06T14:17:08.177 INFO:tasks.workunit.client.0.smithi032.stdout:3/610: unlink d3/d65/d7d/d9c/dad/db1/lbd 0 2024-04-06T14:17:08.179 INFO:tasks.workunit.client.1.smithi086.stdout:1/405: readlink db/d11/d1a/d2a/d52/l7d 0 2024-04-06T14:17:08.183 INFO:tasks.workunit.client.0.smithi032.stdout:3/611: dread d3/d65/d7d/d9c/dad/db1/fa4 [0,4194304] 0 2024-04-06T14:17:08.184 INFO:tasks.workunit.client.0.smithi032.stdout:3/612: fdatasync d3/f7e 0 2024-04-06T14:17:08.185 INFO:tasks.workunit.client.1.smithi086.stdout:3/463: mkdir d5/d36/d5c/d8e 0 2024-04-06T14:17:08.187 INFO:tasks.workunit.client.0.smithi032.stdout:3/613: unlink d3/la7 0 2024-04-06T14:17:08.188 INFO:tasks.workunit.client.1.smithi086.stdout:3/464: chown d5/d26/d7f/d59/d70 348457610 1 2024-04-06T14:17:08.188 INFO:tasks.workunit.client.1.smithi086.stdout:0/435: chown d0/d1/d1b/f39 20181995 1 2024-04-06T14:17:08.188 INFO:tasks.workunit.client.0.smithi032.stdout:3/614: chown d3/d16/f2c 86 1 2024-04-06T14:17:08.189 INFO:tasks.workunit.client.1.smithi086.stdout:2/414: dwrite d0/f12 [0,4194304] 0 2024-04-06T14:17:08.189 INFO:tasks.workunit.client.1.smithi086.stdout:7/369: rmdir d7/d10/d11/d1c/d24/d4d 39 2024-04-06T14:17:08.189 INFO:tasks.workunit.client.1.smithi086.stdout:8/510: dwrite d0/d71/d22/f74 [0,4194304] 0 2024-04-06T14:17:08.192 INFO:tasks.workunit.client.1.smithi086.stdout:8/511: dread - d0/d13/f68 zero size 2024-04-06T14:17:08.192 INFO:tasks.workunit.client.0.smithi032.stdout:3/615: creat d3/d65/d7d/d9c/dad/fea x:0 0 0 2024-04-06T14:17:08.196 INFO:tasks.workunit.client.0.smithi032.stdout:3/616: symlink d3/d65/d89/dcb/leb 0 2024-04-06T14:17:08.197 INFO:tasks.workunit.client.1.smithi086.stdout:4/308: mkdir d0/d2c/d30/d24/d5d 0 2024-04-06T14:17:08.198 INFO:tasks.workunit.client.0.smithi032.stdout:3/617: chown d3/d65/f95 10649 1 2024-04-06T14:17:08.201 INFO:tasks.workunit.client.0.smithi032.stdout:1/690: truncate d0/d9/d1f/d72/f7e 1054318 0 2024-04-06T14:17:08.204 INFO:tasks.workunit.client.1.smithi086.stdout:7/370: truncate d7/d1d/f13 1368597 0 2024-04-06T14:17:08.206 INFO:tasks.workunit.client.1.smithi086.stdout:8/512: creat d0/d71/d22/d43/d6d/d75/d78/fb0 x:0 0 0 2024-04-06T14:17:08.206 INFO:tasks.workunit.client.0.smithi032.stdout:1/691: mkdir d0/d9/d1f/d7c/da9/ddd 0 2024-04-06T14:17:08.208 INFO:tasks.workunit.client.1.smithi086.stdout:0/436: rmdir d0/d1/d2/d84 0 2024-04-06T14:17:08.208 INFO:tasks.workunit.client.1.smithi086.stdout:2/415: dwrite d0/f8e [0,4194304] 0 2024-04-06T14:17:08.210 INFO:tasks.workunit.client.0.smithi032.stdout:1/692: dread - d0/d41/d5b/d96/fd3 zero size 2024-04-06T14:17:08.211 INFO:tasks.workunit.client.1.smithi086.stdout:3/465: dread d5/d36/d5a/d6d/d72/d64/f71 [0,4194304] 0 2024-04-06T14:17:08.211 INFO:tasks.workunit.client.0.smithi032.stdout:3/618: link d3/d65/d89/c99 d3/d65/d7d/dc6/cec 0 2024-04-06T14:17:08.213 INFO:tasks.workunit.client.1.smithi086.stdout:7/371: mknod d7/d10/d11/d1c/d24/d4d/d5c/d66/c7f 0 2024-04-06T14:17:08.214 INFO:tasks.workunit.client.0.smithi032.stdout:0/447: dwrite d3/da/d35/d8b/f90 [0,4194304] 0 2024-04-06T14:17:08.214 INFO:tasks.workunit.client.0.smithi032.stdout:2/608: dwrite d0/d30/d39/d59/d67/d74/d78/d57/d7c/f87 [0,4194304] 0 2024-04-06T14:17:08.218 INFO:tasks.workunit.client.1.smithi086.stdout:4/309: dwrite d0/d15/f37 [0,4194304] 0 2024-04-06T14:17:08.222 INFO:tasks.workunit.client.1.smithi086.stdout:2/416: mkdir d0/d20/d27/d2b/d35/d36/d4d/d59/d94 0 2024-04-06T14:17:08.223 INFO:tasks.workunit.client.0.smithi032.stdout:3/619: mknod d3/d65/d7d/d9c/dbc/ced 0 2024-04-06T14:17:08.223 INFO:tasks.workunit.client.0.smithi032.stdout:1/693: dread d0/da/f52 [0,4194304] 0 2024-04-06T14:17:08.225 INFO:tasks.workunit.client.0.smithi032.stdout:0/448: truncate d3/f45 1460790 0 2024-04-06T14:17:08.225 INFO:tasks.workunit.client.0.smithi032.stdout:2/609: write d0/d30/d39/d59/d67/d74/d78/d57/f96 [1706344,120213] 0 2024-04-06T14:17:08.226 INFO:tasks.workunit.client.1.smithi086.stdout:8/513: dwrite d0/d3b/fa6 [0,4194304] 0 2024-04-06T14:17:08.226 INFO:tasks.workunit.client.1.smithi086.stdout:4/310: chown d0/d15/d4a 783611312 1 2024-04-06T14:17:08.226 INFO:tasks.workunit.client.1.smithi086.stdout:3/466: dwrite d5/d26/d7f/f4c [0,4194304] 0 2024-04-06T14:17:08.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:07 smithi032 ceph-mon[39730]: pgmap v21: 65 pgs: 65 active+clean; 1.5 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 27 MiB/s rd, 168 MiB/s wr, 260 op/s 2024-04-06T14:17:08.230 INFO:tasks.workunit.client.1.smithi086.stdout:4/311: stat d0/d15/l49 0 2024-04-06T14:17:08.231 INFO:tasks.workunit.client.0.smithi032.stdout:3/620: creat d3/d65/d7d/d9c/dad/db1/d3c/dc9/fee x:0 0 0 2024-04-06T14:17:08.232 INFO:tasks.workunit.client.0.smithi032.stdout:2/610: chown d0/d5a/f31 3 1 2024-04-06T14:17:08.233 INFO:tasks.workunit.client.0.smithi032.stdout:0/449: mknod d3/da/d25/d36/c9a 0 2024-04-06T14:17:08.235 INFO:tasks.workunit.client.0.smithi032.stdout:2/611: read - d0/d30/d39/d59/d67/fc0 zero size 2024-04-06T14:17:08.235 INFO:tasks.workunit.client.0.smithi032.stdout:0/450: chown d3/da/d41 8438 1 2024-04-06T14:17:08.239 INFO:tasks.workunit.client.0.smithi032.stdout:1/694: getdents d0/da 0 2024-04-06T14:17:08.239 INFO:tasks.workunit.client.0.smithi032.stdout:0/451: creat d3/da/d25/d36/f9b x:0 0 0 2024-04-06T14:17:08.241 INFO:tasks.workunit.client.0.smithi032.stdout:2/612: mkdir d0/d5a/d6a/dcb 0 2024-04-06T14:17:08.243 INFO:tasks.workunit.client.0.smithi032.stdout:0/452: read d3/da/d25/f94 [184456,14106] 0 2024-04-06T14:17:08.245 INFO:tasks.workunit.client.0.smithi032.stdout:2/613: creat d0/d30/d39/d59/fcc x:0 0 0 2024-04-06T14:17:08.246 INFO:tasks.workunit.client.0.smithi032.stdout:2/614: write d0/d30/d7f/fac [927472,66459] 0 2024-04-06T14:17:08.246 INFO:tasks.workunit.client.0.smithi032.stdout:1/695: truncate d0/d10/d16/f3c 8254456 0 2024-04-06T14:17:08.247 INFO:tasks.workunit.client.1.smithi086.stdout:5/376: sync 2024-04-06T14:17:08.247 INFO:tasks.workunit.client.0.smithi032.stdout:0/453: truncate d3/da/f23 1016943 0 2024-04-06T14:17:08.249 INFO:tasks.workunit.client.1.smithi086.stdout:1/406: truncate db/d11/f17 1138011 0 2024-04-06T14:17:08.249 INFO:tasks.workunit.client.0.smithi032.stdout:2/615: truncate d0/d5a/f97 428200 0 2024-04-06T14:17:08.251 INFO:tasks.workunit.client.1.smithi086.stdout:8/514: dread d0/d13/f40 [0,4194304] 0 2024-04-06T14:17:08.251 INFO:tasks.workunit.client.0.smithi032.stdout:1/696: rename d0/d41/d59/f60 to d0/fde 0 2024-04-06T14:17:08.253 INFO:tasks.workunit.client.1.smithi086.stdout:5/377: mkdir d0/d16/d71 0 2024-04-06T14:17:08.253 INFO:tasks.workunit.client.1.smithi086.stdout:7/372: dwrite d7/d10/d11/d1c/f1e [0,4194304] 0 2024-04-06T14:17:08.253 INFO:tasks.workunit.client.1.smithi086.stdout:3/467: rename d5/f2d to d5/d36/d5a/d6d/d72/d58/d89/f8f 0 2024-04-06T14:17:08.254 INFO:tasks.workunit.client.0.smithi032.stdout:2/616: readlink d0/d30/d39/d59/d67/d74/d78/d8c/db0/lc7 0 2024-04-06T14:17:08.255 INFO:tasks.workunit.client.0.smithi032.stdout:1/697: creat d0/da/d3e/d75/fdf x:0 0 0 2024-04-06T14:17:08.259 INFO:tasks.workunit.client.1.smithi086.stdout:8/515: write d0/db/fac [737361,93289] 0 2024-04-06T14:17:08.263 INFO:tasks.workunit.client.1.smithi086.stdout:4/312: rename d0/d15/d19/f1c to d0/d15/d53/f5e 0 2024-04-06T14:17:08.263 INFO:tasks.workunit.client.0.smithi032.stdout:0/454: getdents d3/da/d25/d2b/d4a/d79 0 2024-04-06T14:17:08.263 INFO:tasks.workunit.client.1.smithi086.stdout:1/407: mkdir db/d11/d1a/d2a/d52/d5b/d68/d6e/d81 0 2024-04-06T14:17:08.264 INFO:tasks.workunit.client.1.smithi086.stdout:3/468: symlink d5/d33/d5d/l90 0 2024-04-06T14:17:08.265 INFO:tasks.workunit.client.1.smithi086.stdout:1/408: chown db/d11/d1d 55 1 2024-04-06T14:17:08.265 INFO:tasks.workunit.client.1.smithi086.stdout:8/516: creat d0/d71/d22/d43/d7f/fb1 x:0 0 0 2024-04-06T14:17:08.265 INFO:tasks.workunit.client.0.smithi032.stdout:1/698: truncate d0/d10/d16/fa1 675773 0 2024-04-06T14:17:08.266 INFO:tasks.workunit.client.1.smithi086.stdout:3/469: fsync d5/f50 0 2024-04-06T14:17:08.266 INFO:tasks.workunit.client.1.smithi086.stdout:2/417: dread d0/f2d [0,4194304] 0 2024-04-06T14:17:08.267 INFO:tasks.workunit.client.0.smithi032.stdout:9/633: sync 2024-04-06T14:17:08.267 INFO:tasks.workunit.client.1.smithi086.stdout:4/313: truncate d0/d15/f48 4377500 0 2024-04-06T14:17:08.268 INFO:tasks.workunit.client.1.smithi086.stdout:7/373: rename d7/d10/d30 to d7/d10/d11/d80 0 2024-04-06T14:17:08.268 INFO:tasks.workunit.client.0.smithi032.stdout:1/699: write d0/d9/d1f/f91 [1170164,43573] 0 2024-04-06T14:17:08.269 INFO:tasks.workunit.client.1.smithi086.stdout:4/314: fsync d0/d15/f48 0 2024-04-06T14:17:08.270 INFO:tasks.workunit.client.1.smithi086.stdout:3/470: mknod d5/d36/d5a/d6d/d72/d58/c91 0 2024-04-06T14:17:08.272 INFO:tasks.workunit.client.1.smithi086.stdout:2/418: mknod d0/df/d48/d53/d76/d78/c95 0 2024-04-06T14:17:08.272 INFO:tasks.workunit.client.0.smithi032.stdout:0/455: dwrite d3/da/d41/f42 [0,4194304] 0 2024-04-06T14:17:08.273 INFO:tasks.workunit.client.1.smithi086.stdout:3/471: write d5/df/f81 [113390,35290] 0 2024-04-06T14:17:08.276 INFO:tasks.workunit.client.1.smithi086.stdout:7/374: creat d7/f81 x:0 0 0 2024-04-06T14:17:08.276 INFO:tasks.workunit.client.1.smithi086.stdout:3/472: readlink d5/d26/l84 0 2024-04-06T14:17:08.276 INFO:tasks.workunit.client.1.smithi086.stdout:1/409: dwrite db/d11/d1f/f51 [0,4194304] 0 2024-04-06T14:17:08.276 INFO:tasks.workunit.client.1.smithi086.stdout:4/315: rename d0/de/f1a to d0/d15/d2b/f5f 0 2024-04-06T14:17:08.277 INFO:tasks.workunit.client.0.smithi032.stdout:0/456: write d3/da/d25/d2b/f3d [1469558,85138] 0 2024-04-06T14:17:08.277 INFO:tasks.workunit.client.1.smithi086.stdout:7/375: rmdir d7/d10 39 2024-04-06T14:17:08.279 INFO:tasks.workunit.client.0.smithi032.stdout:0/457: chown d3/da/d25/d2b/d4a/f65 245303 1 2024-04-06T14:17:08.281 INFO:tasks.workunit.client.0.smithi032.stdout:3/621: write d3/d65/d7d/d9c/dad/db1/d3c/f85 [4887294,112729] 0 2024-04-06T14:17:08.285 INFO:tasks.workunit.client.0.smithi032.stdout:0/458: dread - d3/da/f64 zero size 2024-04-06T14:17:08.286 INFO:tasks.workunit.client.1.smithi086.stdout:2/419: write d0/df/d48/d7d/f87 [375992,19173] 0 2024-04-06T14:17:08.286 INFO:tasks.workunit.client.1.smithi086.stdout:4/316: write d0/f40 [855096,49618] 0 2024-04-06T14:17:08.289 INFO:tasks.workunit.client.1.smithi086.stdout:4/317: chown d0/d15/d20/l43 1017879 1 2024-04-06T14:17:08.289 INFO:tasks.workunit.client.1.smithi086.stdout:1/410: rmdir db/d11/d1f/d5c 39 2024-04-06T14:17:08.291 INFO:tasks.workunit.client.1.smithi086.stdout:7/376: creat d7/d10/d11/d1c/d24/d2c/d3c/f82 x:0 0 0 2024-04-06T14:17:08.294 INFO:tasks.workunit.client.0.smithi032.stdout:2/617: dread d0/d5a/f1c [0,4194304] 0 2024-04-06T14:17:08.296 INFO:tasks.workunit.client.1.smithi086.stdout:1/411: dwrite db/d11/d24/f45 [0,4194304] 0 2024-04-06T14:17:08.297 INFO:tasks.workunit.client.1.smithi086.stdout:1/412: fsync db/d11/d1f/f7c 0 2024-04-06T14:17:08.298 INFO:tasks.workunit.client.1.smithi086.stdout:1/413: readlink db/d11/d1f/l48 0 2024-04-06T14:17:08.299 INFO:tasks.workunit.client.1.smithi086.stdout:5/378: truncate d0/d16/d18/d21/f45 790620 0 2024-04-06T14:17:08.300 INFO:tasks.workunit.client.1.smithi086.stdout:1/414: read db/d11/d1f/f51 [289996,26580] 0 2024-04-06T14:17:08.300 INFO:tasks.workunit.client.0.smithi032.stdout:4/666: sync 2024-04-06T14:17:08.301 INFO:tasks.workunit.client.0.smithi032.stdout:8/645: sync 2024-04-06T14:17:08.301 INFO:tasks.workunit.client.0.smithi032.stdout:5/600: sync 2024-04-06T14:17:08.301 INFO:tasks.workunit.client.0.smithi032.stdout:6/629: sync 2024-04-06T14:17:08.301 INFO:tasks.workunit.client.0.smithi032.stdout:7/648: sync 2024-04-06T14:17:08.304 INFO:tasks.workunit.client.0.smithi032.stdout:6/630: read d4/d5/d8/d2f/fb2 [2898836,87272] 0 2024-04-06T14:17:08.304 INFO:tasks.workunit.client.0.smithi032.stdout:4/667: write d0/df/d1f/d2e/f49 [2965142,47606] 0 2024-04-06T14:17:08.306 INFO:tasks.workunit.client.1.smithi086.stdout:2/420: rmdir d0/df/d48/d53/d76 39 2024-04-06T14:17:08.307 INFO:tasks.workunit.client.1.smithi086.stdout:2/421: read - d0/d20/d27/d37/d52/f79 zero size 2024-04-06T14:17:08.307 INFO:tasks.workunit.client.1.smithi086.stdout:4/318: dread d0/d2c/d30/d24/f2d [0,4194304] 0 2024-04-06T14:17:08.308 INFO:tasks.workunit.client.1.smithi086.stdout:5/379: symlink d0/l72 0 2024-04-06T14:17:08.308 INFO:tasks.workunit.client.0.smithi032.stdout:4/668: truncate d0/df/d23/d31/f68 4502579 0 2024-04-06T14:17:08.309 INFO:tasks.workunit.client.0.smithi032.stdout:6/631: read d4/d5/f6a [1912261,102881] 0 2024-04-06T14:17:08.311 INFO:tasks.workunit.client.0.smithi032.stdout:6/632: dread - d4/d5/dbf/d61/db9/fca zero size 2024-04-06T14:17:08.313 INFO:tasks.workunit.client.1.smithi086.stdout:4/319: symlink d0/d15/l60 0 2024-04-06T14:17:08.313 INFO:tasks.workunit.client.0.smithi032.stdout:1/700: dread d0/d9/f83 [0,4194304] 0 2024-04-06T14:17:08.314 INFO:tasks.workunit.client.1.smithi086.stdout:7/377: rmdir d7/d10/d11/d1c/d24 39 2024-04-06T14:17:08.314 INFO:tasks.workunit.client.0.smithi032.stdout:1/701: write d0/f1e [3698616,20458] 0 2024-04-06T14:17:08.315 INFO:tasks.workunit.client.1.smithi086.stdout:2/422: dwrite d0/d20/d27/d37/f83 [0,4194304] 0 2024-04-06T14:17:08.316 INFO:tasks.workunit.client.0.smithi032.stdout:1/702: fsync d0/da/d3e/d75/f8a 0 2024-04-06T14:17:08.318 INFO:tasks.workunit.client.1.smithi086.stdout:4/320: dread d0/d2c/d30/d24/f2d [0,4194304] 0 2024-04-06T14:17:08.321 INFO:tasks.workunit.client.1.smithi086.stdout:1/415: write db/d11/d1f/f44 [580246,106138] 0 2024-04-06T14:17:08.322 INFO:tasks.workunit.client.1.smithi086.stdout:1/416: dread - db/d11/d1a/f63 zero size 2024-04-06T14:17:08.324 INFO:tasks.workunit.client.1.smithi086.stdout:7/378: mkdir d7/d10/d11/d1c/d24/d2c/d69/d83 0 2024-04-06T14:17:08.324 INFO:tasks.workunit.client.1.smithi086.stdout:1/417: read db/d13/f14 [181125,27851] 0 2024-04-06T14:17:08.327 INFO:tasks.workunit.client.1.smithi086.stdout:7/379: dread f5 [0,4194304] 0 2024-04-06T14:17:08.328 INFO:tasks.workunit.client.1.smithi086.stdout:4/321: mkdir d0/d2c/d38/d54/d61 0 2024-04-06T14:17:08.328 INFO:tasks.workunit.client.1.smithi086.stdout:1/418: stat db/d11/d1f/d5c/c6c 0 2024-04-06T14:17:08.329 INFO:tasks.workunit.client.1.smithi086.stdout:4/322: chown d0/l18 412 1 2024-04-06T14:17:08.329 INFO:tasks.workunit.client.1.smithi086.stdout:2/423: creat d0/d20/d27/d2b/d35/d36/f96 x:0 0 0 2024-04-06T14:17:08.331 INFO:tasks.workunit.client.1.smithi086.stdout:5/380: truncate d0/d16/d18/d21/d23/f55 949238 0 2024-04-06T14:17:08.333 INFO:tasks.workunit.client.1.smithi086.stdout:4/323: unlink d0/d15/d20/f3a 0 2024-04-06T14:17:08.334 INFO:tasks.workunit.client.1.smithi086.stdout:1/419: rmdir db/d11/d1a/d2a/d52/d5b/d68 39 2024-04-06T14:17:08.334 INFO:tasks.workunit.client.1.smithi086.stdout:4/324: dread d0/d15/d20/f27 [0,4194304] 0 2024-04-06T14:17:08.335 INFO:tasks.workunit.client.1.smithi086.stdout:2/424: creat d0/d20/d27/d2b/d35/d36/d4a/f97 x:0 0 0 2024-04-06T14:17:08.336 INFO:tasks.workunit.client.1.smithi086.stdout:1/420: dread - db/d11/d1a/d2a/d52/f6d zero size 2024-04-06T14:17:08.336 INFO:tasks.workunit.client.1.smithi086.stdout:7/380: mknod d7/d10/d11/d1c/d24/c84 0 2024-04-06T14:17:08.337 INFO:tasks.workunit.client.1.smithi086.stdout:5/381: creat d0/d16/d18/d2a/d50/f73 x:0 0 0 2024-04-06T14:17:08.339 INFO:tasks.workunit.client.1.smithi086.stdout:2/425: creat d0/d20/d27/d37/f98 x:0 0 0 2024-04-06T14:17:08.340 INFO:tasks.workunit.client.1.smithi086.stdout:4/325: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:17:08.341 INFO:tasks.workunit.client.1.smithi086.stdout:7/381: fsync d7/d10/d11/d1c/f60 0 2024-04-06T14:17:08.345 INFO:tasks.workunit.client.1.smithi086.stdout:1/421: mkdir db/d11/d1a/d2a/d52/d5b/d82 0 2024-04-06T14:17:08.349 INFO:tasks.workunit.client.1.smithi086.stdout:7/382: creat d7/d10/d11/d1c/d24/d4d/d5c/d66/f85 x:0 0 0 2024-04-06T14:17:08.353 INFO:tasks.workunit.client.1.smithi086.stdout:7/383: creat d7/d10/d11/d1c/d24/d4d/f86 x:0 0 0 2024-04-06T14:17:08.354 INFO:tasks.workunit.client.1.smithi086.stdout:7/384: dread - d7/d10/d11/d1c/d24/d4d/d5c/d66/f67 zero size 2024-04-06T14:17:08.355 INFO:tasks.workunit.client.1.smithi086.stdout:1/422: truncate db/ff 29445 0 2024-04-06T14:17:08.356 INFO:tasks.workunit.client.1.smithi086.stdout:2/426: write d0/d20/d44/f46 [2565864,100878] 0 2024-04-06T14:17:08.357 INFO:tasks.workunit.client.1.smithi086.stdout:1/423: write db/d13/d71/f7f [914306,113336] 0 2024-04-06T14:17:08.358 INFO:tasks.workunit.client.1.smithi086.stdout:4/326: getdents d0/de/d4c 0 2024-04-06T14:17:08.358 INFO:tasks.workunit.client.1.smithi086.stdout:2/427: write d0/fd [8272643,113140] 0 2024-04-06T14:17:08.359 INFO:tasks.workunit.client.1.smithi086.stdout:7/385: mkdir d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/d87 0 2024-04-06T14:17:08.361 INFO:tasks.workunit.client.1.smithi086.stdout:4/327: unlink d0/d15/d2b/l33 0 2024-04-06T14:17:08.365 INFO:tasks.workunit.client.0.smithi032.stdout:3/622: rename d3/c98 to d3/d65/d7d/d9c/db8/cef 0 2024-04-06T14:17:08.366 INFO:tasks.workunit.client.1.smithi086.stdout:4/328: fdatasync d0/f2 0 2024-04-06T14:17:08.367 INFO:tasks.workunit.client.1.smithi086.stdout:1/424: rename db/d11/d1f/f66 to db/d13/f83 0 2024-04-06T14:17:08.368 INFO:tasks.workunit.client.0.smithi032.stdout:0/459: rmdir d3/da/d35/d4b 39 2024-04-06T14:17:08.370 INFO:tasks.workunit.client.1.smithi086.stdout:2/428: write d0/f2a [1067278,59633] 0 2024-04-06T14:17:08.371 INFO:tasks.workunit.client.1.smithi086.stdout:7/386: dwrite d7/d10/d11/d80/f46 [4194304,4194304] 0 2024-04-06T14:17:08.371 INFO:tasks.workunit.client.0.smithi032.stdout:8/646: creat d3/d8/d7/d14/fe8 x:0 0 0 2024-04-06T14:17:08.372 INFO:tasks.workunit.client.1.smithi086.stdout:4/329: creat d0/de/f62 x:0 0 0 2024-04-06T14:17:08.375 INFO:tasks.workunit.client.1.smithi086.stdout:2/429: readlink d0/d20/d27/d2b/d35/d36/l6a 0 2024-04-06T14:17:08.376 INFO:tasks.workunit.client.0.smithi032.stdout:5/601: symlink d16/d18/d33/d35/d88/dbd/dc5/ld2 0 2024-04-06T14:17:08.379 INFO:tasks.workunit.client.0.smithi032.stdout:4/669: symlink d0/df/d2b/dbe/le5 0 2024-04-06T14:17:08.380 INFO:tasks.workunit.client.0.smithi032.stdout:6/633: mkdir d4/d5/d8/d2b/d77/ddd 0 2024-04-06T14:17:08.381 INFO:tasks.workunit.client.1.smithi086.stdout:7/387: chown d7/d10/d11/d1c/c2f 261575784 1 2024-04-06T14:17:08.382 INFO:tasks.workunit.client.0.smithi032.stdout:9/634: write d2/d15/d1e/d65/d83/d31/d39/da6/fb6 [981261,129004] 0 2024-04-06T14:17:08.383 INFO:tasks.workunit.client.0.smithi032.stdout:6/634: write d4/d5/dbf/d61/db9/fca [381335,14040] 0 2024-04-06T14:17:08.385 INFO:tasks.workunit.client.1.smithi086.stdout:7/388: chown d7/d10/d11/c5a 12105567 1 2024-04-06T14:17:08.387 INFO:tasks.workunit.client.0.smithi032.stdout:1/703: rename d0/lae to d0/d41/d5b/d96/le0 0 2024-04-06T14:17:08.387 INFO:tasks.workunit.client.0.smithi032.stdout:9/635: dread d2/d15/d1e/d65/d83/d7e/f56 [0,4194304] 0 2024-04-06T14:17:08.388 INFO:tasks.workunit.client.1.smithi086.stdout:1/425: dwrite db/d13/d55/f6a [0,4194304] 0 2024-04-06T14:17:08.388 INFO:tasks.workunit.client.1.smithi086.stdout:2/430: dwrite d0/f2a [0,4194304] 0 2024-04-06T14:17:08.389 INFO:tasks.workunit.client.0.smithi032.stdout:9/636: dread - d2/d15/d1e/d65/d83/d31/d39/f52 zero size 2024-04-06T14:17:08.389 INFO:tasks.workunit.client.0.smithi032.stdout:3/623: chown d3/d65/d7d/d9c/dad/db1/d3c/l58 292259 1 2024-04-06T14:17:08.391 INFO:tasks.workunit.client.1.smithi086.stdout:4/330: dwrite d0/f59 [0,4194304] 0 2024-04-06T14:17:08.391 INFO:tasks.workunit.client.0.smithi032.stdout:0/460: mkdir d3/da/d35/d8b/d9c 0 2024-04-06T14:17:08.392 INFO:tasks.workunit.client.1.smithi086.stdout:7/389: chown d7/d10/d11/d1c/d24/d2c/d3c/d7a/c7c 46347413 1 2024-04-06T14:17:08.392 INFO:tasks.workunit.client.1.smithi086.stdout:2/431: stat d0/d20/d44/f5b 0 2024-04-06T14:17:08.392 INFO:tasks.workunit.client.0.smithi032.stdout:9/637: chown d2/d15/d1e/d65/d83/d7e/da7/cc1 25735983 1 2024-04-06T14:17:08.393 INFO:tasks.workunit.client.0.smithi032.stdout:2/618: write d0/d5a/d3a/da2/fc1 [2229644,57929] 0 2024-04-06T14:17:08.393 INFO:tasks.workunit.client.0.smithi032.stdout:9/638: fdatasync d2/fb9 0 2024-04-06T14:17:08.394 INFO:tasks.workunit.client.1.smithi086.stdout:4/331: chown d0/de/f11 8313849 1 2024-04-06T14:17:08.397 INFO:tasks.workunit.client.1.smithi086.stdout:2/432: symlink d0/d20/d27/d2b/d35/d36/d4a/l99 0 2024-04-06T14:17:08.399 INFO:tasks.workunit.client.1.smithi086.stdout:7/390: dread d7/d10/d11/d1c/d24/d2c/f6f [4194304,4194304] 0 2024-04-06T14:17:08.402 INFO:tasks.workunit.client.1.smithi086.stdout:7/391: write d7/d10/d11/d1c/d24/d2c/f70 [514566,113573] 0 2024-04-06T14:17:08.403 INFO:tasks.workunit.client.1.smithi086.stdout:7/392: readlink d7/d10/d11/d1c/l26 0 2024-04-06T14:17:08.403 INFO:tasks.workunit.client.0.smithi032.stdout:3/624: mkdir d3/d65/d7d/dc6/df0 0 2024-04-06T14:17:08.403 INFO:tasks.workunit.client.1.smithi086.stdout:6/420: sync 2024-04-06T14:17:08.403 INFO:tasks.workunit.client.1.smithi086.stdout:9/364: sync 2024-04-06T14:17:08.404 INFO:tasks.workunit.client.1.smithi086.stdout:4/332: mknod d0/d2c/d30/d24/c63 0 2024-04-06T14:17:08.404 INFO:tasks.workunit.client.1.smithi086.stdout:2/433: unlink d0/d20/d27/d2b/d35/d36/d4d/f4f 0 2024-04-06T14:17:08.404 INFO:tasks.workunit.client.0.smithi032.stdout:0/461: fsync d3/da/f5e 0 2024-04-06T14:17:08.405 INFO:tasks.workunit.client.1.smithi086.stdout:4/333: chown d0/d15/l49 10 1 2024-04-06T14:17:08.407 INFO:tasks.workunit.client.0.smithi032.stdout:3/625: stat d3/d65/d7d/d9c/db8/fc2 0 2024-04-06T14:17:08.408 INFO:tasks.workunit.client.0.smithi032.stdout:2/619: mknod d0/d4e/ccd 0 2024-04-06T14:17:08.409 INFO:tasks.workunit.client.0.smithi032.stdout:8/647: rename d3/d8/d3a/f42 to d3/d8/d12/d2f/d4a/d60/d90/fe9 0 2024-04-06T14:17:08.409 INFO:tasks.workunit.client.1.smithi086.stdout:2/434: dread d0/f2d [0,4194304] 0 2024-04-06T14:17:08.411 INFO:tasks.workunit.client.0.smithi032.stdout:9/639: creat d2/d15/d1e/d65/d6b/fcf x:0 0 0 2024-04-06T14:17:08.411 INFO:tasks.workunit.client.1.smithi086.stdout:4/334: fdatasync d0/f59 0 2024-04-06T14:17:08.413 INFO:tasks.workunit.client.1.smithi086.stdout:4/335: chown d0/d2c/d30/f3b 53603 1 2024-04-06T14:17:08.413 INFO:tasks.workunit.client.0.smithi032.stdout:7/649: dwrite d16/d20/f37 [0,4194304] 0 2024-04-06T14:17:08.415 INFO:tasks.workunit.client.1.smithi086.stdout:4/336: truncate d0/de/f26 5049524 0 2024-04-06T14:17:08.415 INFO:tasks.workunit.client.0.smithi032.stdout:9/640: fdatasync d2/d15/d1e/d65/d83/d31/f40 0 2024-04-06T14:17:08.415 INFO:tasks.workunit.client.0.smithi032.stdout:5/602: link d16/d18/d33/d45/d52/d39/d86/lc1 d16/d18/dae/ld3 0 2024-04-06T14:17:08.416 INFO:tasks.workunit.client.1.smithi086.stdout:1/426: dread db/d11/d1f/f2b [0,4194304] 0 2024-04-06T14:17:08.419 INFO:tasks.workunit.client.1.smithi086.stdout:4/337: symlink d0/d2c/d30/l64 0 2024-04-06T14:17:08.419 INFO:tasks.workunit.client.1.smithi086.stdout:6/421: getdents d6/d9/d43 0 2024-04-06T14:17:08.420 INFO:tasks.workunit.client.0.smithi032.stdout:0/462: write d3/f5a [4161947,125617] 0 2024-04-06T14:17:08.421 INFO:tasks.workunit.client.0.smithi032.stdout:2/620: rmdir d0/d6e/d8d 39 2024-04-06T14:17:08.425 INFO:tasks.workunit.client.1.smithi086.stdout:2/435: getdents d0/df/d48/d53/d76 0 2024-04-06T14:17:08.426 INFO:tasks.workunit.client.1.smithi086.stdout:1/427: dwrite db/d11/d1a/f4b [0,4194304] 0 2024-04-06T14:17:08.426 INFO:tasks.workunit.client.0.smithi032.stdout:1/704: dread d0/d10/d16/f51 [0,4194304] 0 2024-04-06T14:17:08.426 INFO:tasks.workunit.client.0.smithi032.stdout:6/635: write d4/d5/d19/d25/f5d [747833,48608] 0 2024-04-06T14:17:08.427 INFO:tasks.workunit.client.0.smithi032.stdout:7/650: rename d16/d20/d54/de4/d2d/f32 to d16/d20/d54/de4/d70/fe9 0 2024-04-06T14:17:08.428 INFO:tasks.workunit.client.1.smithi086.stdout:6/422: symlink d6/d13/d42/d52/l91 0 2024-04-06T14:17:08.430 INFO:tasks.workunit.client.0.smithi032.stdout:7/651: chown l5 1020921 1 2024-04-06T14:17:08.431 INFO:tasks.workunit.client.1.smithi086.stdout:2/436: symlink d0/df/d48/d53/l9a 0 2024-04-06T14:17:08.432 INFO:tasks.workunit.client.1.smithi086.stdout:1/428: symlink db/d11/d1a/d2a/d52/d5b/l84 0 2024-04-06T14:17:08.432 INFO:tasks.workunit.client.0.smithi032.stdout:0/463: symlink d3/da/d41/d82/l9d 0 2024-04-06T14:17:08.434 INFO:tasks.workunit.client.0.smithi032.stdout:1/705: mkdir d0/d9/d1f/d7c/da9/de1 0 2024-04-06T14:17:08.438 INFO:tasks.workunit.client.1.smithi086.stdout:9/365: dread f2 [4194304,4194304] 0 2024-04-06T14:17:08.439 INFO:tasks.workunit.client.1.smithi086.stdout:4/338: truncate d0/d2c/d30/d24/f2d 3085622 0 2024-04-06T14:17:08.439 INFO:tasks.workunit.client.0.smithi032.stdout:4/670: truncate d0/df/d23/f3c 1109923 0 2024-04-06T14:17:08.439 INFO:tasks.workunit.client.0.smithi032.stdout:2/621: dwrite d0/f6c [0,4194304] 0 2024-04-06T14:17:08.440 INFO:tasks.workunit.client.0.smithi032.stdout:0/464: write d3/da/d25/d36/f39 [3822140,19414] 0 2024-04-06T14:17:08.441 INFO:tasks.workunit.client.0.smithi032.stdout:9/641: link d2/d15/d55/d59/d93/d95/db8/d81/d77/l99 d2/d15/d1e/d65/d6b/ld0 0 2024-04-06T14:17:08.441 INFO:tasks.workunit.client.0.smithi032.stdout:1/706: read d0/d10/d16/f6d [3484419,71305] 0 2024-04-06T14:17:08.441 INFO:tasks.workunit.client.1.smithi086.stdout:4/339: chown d0/d15/d20/d34 40 1 2024-04-06T14:17:08.442 INFO:tasks.workunit.client.1.smithi086.stdout:2/437: dwrite d0/d20/d27/d2b/d35/d36/f68 [0,4194304] 0 2024-04-06T14:17:08.442 INFO:tasks.workunit.client.1.smithi086.stdout:1/429: dread db/f16 [0,4194304] 0 2024-04-06T14:17:08.443 INFO:tasks.workunit.client.0.smithi032.stdout:3/626: dread d3/d16/d63/fa1 [0,4194304] 0 2024-04-06T14:17:08.445 INFO:tasks.workunit.client.0.smithi032.stdout:6/636: link d4/d5/dbf/f33 d4/d5/d8/d2f/d9b/fde 0 2024-04-06T14:17:08.447 INFO:tasks.workunit.client.1.smithi086.stdout:1/430: chown db/d11/d42/d53/f57 251381983 1 2024-04-06T14:17:08.448 INFO:tasks.workunit.client.0.smithi032.stdout:6/637: fdatasync d4/f13 0 2024-04-06T14:17:08.452 INFO:tasks.workunit.client.1.smithi086.stdout:2/438: creat d0/df/d48/d53/d76/f9b x:0 0 0 2024-04-06T14:17:08.453 INFO:tasks.workunit.client.0.smithi032.stdout:8/648: write d3/d8/d12/f1b [725854,85267] 0 2024-04-06T14:17:08.454 INFO:tasks.workunit.client.1.smithi086.stdout:2/439: mkdir d0/d20/d27/d37/d52/d9c 0 2024-04-06T14:17:08.455 INFO:tasks.workunit.client.1.smithi086.stdout:6/423: dread d6/d9/f10 [0,4194304] 0 2024-04-06T14:17:08.457 INFO:tasks.workunit.client.1.smithi086.stdout:2/440: chown d0/f12 38617869 1 2024-04-06T14:17:08.457 INFO:tasks.workunit.client.1.smithi086.stdout:1/431: getdents db/d13/d55 0 2024-04-06T14:17:08.459 INFO:tasks.workunit.client.0.smithi032.stdout:2/622: truncate d0/d5a/d3a/f61 1615941 0 2024-04-06T14:17:08.460 INFO:tasks.workunit.client.1.smithi086.stdout:6/424: mknod d6/d13/d1a/d22/c92 0 2024-04-06T14:17:08.461 INFO:tasks.workunit.client.1.smithi086.stdout:9/366: dwrite d5/d13/f4f [0,4194304] 0 2024-04-06T14:17:08.463 INFO:tasks.workunit.client.1.smithi086.stdout:6/425: chown d6/d13/d1a/d22/c86 40187 1 2024-04-06T14:17:08.464 INFO:tasks.workunit.client.1.smithi086.stdout:1/432: write db/d11/d1d/f58 [4585549,71352] 0 2024-04-06T14:17:08.464 INFO:tasks.workunit.client.1.smithi086.stdout:2/441: fdatasync d0/df/d48/d7d/f89 0 2024-04-06T14:17:08.466 INFO:tasks.workunit.client.1.smithi086.stdout:9/367: read d5/d13/d47/d5e/f61 [704619,37649] 0 2024-04-06T14:17:08.467 INFO:tasks.workunit.client.0.smithi032.stdout:6/638: symlink d4/d5/d8/d2b/d77/ddd/ldf 0 2024-04-06T14:17:08.467 INFO:tasks.workunit.client.1.smithi086.stdout:6/426: mkdir d6/d13/d1a/d64/d93 0 2024-04-06T14:17:08.467 INFO:tasks.workunit.client.1.smithi086.stdout:2/442: dread - d0/d20/d27/d2b/d35/d36/d4a/f97 zero size 2024-04-06T14:17:08.468 INFO:tasks.workunit.client.1.smithi086.stdout:9/368: symlink d5/d58/l77 0 2024-04-06T14:17:08.471 INFO:tasks.workunit.client.0.smithi032.stdout:8/649: chown d3/d8/d3a/d4b/c6c 114 1 2024-04-06T14:17:08.472 INFO:tasks.workunit.client.0.smithi032.stdout:3/627: link d3/d65/d7d/d9c/dbc/ced d3/d16/cf1 0 2024-04-06T14:17:08.472 INFO:tasks.workunit.client.0.smithi032.stdout:4/671: dread d0/df/d2b/fd9 [0,4194304] 0 2024-04-06T14:17:08.472 INFO:tasks.workunit.client.1.smithi086.stdout:6/427: rename d6/d13/d1a/d22/c86 to d6/d9/d43/c94 0 2024-04-06T14:17:08.474 INFO:tasks.workunit.client.1.smithi086.stdout:9/369: symlink d5/d10/l78 0 2024-04-06T14:17:08.474 INFO:tasks.workunit.client.1.smithi086.stdout:1/433: dwrite f4 [0,4194304] 0 2024-04-06T14:17:08.476 INFO:tasks.workunit.client.1.smithi086.stdout:6/428: creat d6/d13/d5f/f95 x:0 0 0 2024-04-06T14:17:08.477 INFO:tasks.workunit.client.1.smithi086.stdout:2/443: dwrite d0/d20/d27/f43 [0,4194304] 0 2024-04-06T14:17:08.478 INFO:tasks.workunit.client.0.smithi032.stdout:7/652: dread d16/d20/d54/de4/d2d/f64 [0,4194304] 0 2024-04-06T14:17:08.482 INFO:tasks.workunit.client.1.smithi086.stdout:9/370: symlink d5/d13/d1a/d5f/l79 0 2024-04-06T14:17:08.484 INFO:tasks.workunit.client.1.smithi086.stdout:1/434: dwrite db/d13/f23 [0,4194304] 0 2024-04-06T14:17:08.486 INFO:tasks.workunit.client.1.smithi086.stdout:6/429: symlink d6/l96 0 2024-04-06T14:17:08.486 INFO:tasks.workunit.client.0.smithi032.stdout:1/707: dwrite d0/d9/d1f/d7c/da9/fb3 [0,4194304] 0 2024-04-06T14:17:08.491 INFO:tasks.workunit.client.1.smithi086.stdout:9/371: mknod d5/d18/c7a 0 2024-04-06T14:17:08.493 INFO:tasks.workunit.client.1.smithi086.stdout:2/444: fdatasync d0/df/f14 0 2024-04-06T14:17:08.493 INFO:tasks.workunit.client.1.smithi086.stdout:9/372: write d5/d13/d1a/f24 [498219,23311] 0 2024-04-06T14:17:08.497 INFO:tasks.workunit.client.1.smithi086.stdout:2/445: chown d0/d20/d27/d2b/l64 140270088 1 2024-04-06T14:17:08.497 INFO:tasks.workunit.client.1.smithi086.stdout:2/446: chown d0/d20/d80 9911 1 2024-04-06T14:17:08.498 INFO:tasks.workunit.client.1.smithi086.stdout:2/447: stat d0/df/d48 0 2024-04-06T14:17:08.503 INFO:tasks.workunit.client.1.smithi086.stdout:6/430: fsync d6/d9/db/f28 0 2024-04-06T14:17:08.503 INFO:tasks.workunit.client.1.smithi086.stdout:6/431: fsync d6/d13/f26 0 2024-04-06T14:17:08.503 INFO:tasks.workunit.client.0.smithi032.stdout:3/628: write d3/d65/f66 [3141753,125814] 0 2024-04-06T14:17:08.503 INFO:tasks.workunit.client.0.smithi032.stdout:9/642: dwrite d2/d15/d55/f85 [0,4194304] 0 2024-04-06T14:17:08.503 INFO:tasks.workunit.client.0.smithi032.stdout:8/650: creat d3/d8/d12/d2f/d4a/d60/d90/dc5/dd7/fea x:0 0 0 2024-04-06T14:17:08.504 INFO:tasks.workunit.client.0.smithi032.stdout:5/603: dwrite d16/d18/d33/d45/d52/d39/f5b [0,4194304] 0 2024-04-06T14:17:08.505 INFO:tasks.workunit.client.0.smithi032.stdout:0/465: dwrite d3/da/d25/d36/f93 [0,4194304] 0 2024-04-06T14:17:08.509 INFO:tasks.workunit.client.0.smithi032.stdout:7/653: write d16/d20/d7a/da6/fd6 [559411,17874] 0 2024-04-06T14:17:08.515 INFO:tasks.workunit.client.0.smithi032.stdout:4/672: mkdir d0/d82/de6 0 2024-04-06T14:17:08.516 INFO:tasks.workunit.client.1.smithi086.stdout:9/373: mknod d5/d13/d47/d5e/c7b 0 2024-04-06T14:17:08.516 INFO:tasks.workunit.client.0.smithi032.stdout:5/604: rmdir d16/d58 39 2024-04-06T14:17:08.516 INFO:tasks.workunit.client.0.smithi032.stdout:7/654: truncate d16/d20/d54/de4/d2d/d30/d3e/da7/db5/fe6 358838 0 2024-04-06T14:17:08.517 INFO:tasks.workunit.client.1.smithi086.stdout:1/435: creat db/d11/d1a/f85 x:0 0 0 2024-04-06T14:17:08.518 INFO:tasks.workunit.client.0.smithi032.stdout:7/655: write d16/f40 [1398612,56460] 0 2024-04-06T14:17:08.519 INFO:tasks.workunit.client.1.smithi086.stdout:9/374: dread d5/d13/f2d [0,4194304] 0 2024-04-06T14:17:08.520 INFO:tasks.workunit.client.0.smithi032.stdout:1/708: mkdir d0/da/d3e/d75/dd7/de2 0 2024-04-06T14:17:08.521 INFO:tasks.workunit.client.1.smithi086.stdout:2/448: mkdir d0/df/d5e/d9d 0 2024-04-06T14:17:08.521 INFO:tasks.workunit.client.0.smithi032.stdout:4/673: stat d0/df/d1f/d2e/cd5 0 2024-04-06T14:17:08.522 INFO:tasks.workunit.client.1.smithi086.stdout:2/449: chown d0/df 9667 1 2024-04-06T14:17:08.522 INFO:tasks.workunit.client.1.smithi086.stdout:9/375: fsync f0 0 2024-04-06T14:17:08.522 INFO:tasks.workunit.client.1.smithi086.stdout:1/436: write db/d11/d1a/d2a/d52/d5b/d68/f7a [761234,127853] 0 2024-04-06T14:17:08.523 INFO:tasks.workunit.client.0.smithi032.stdout:5/605: mkdir d16/d18/d33/d45/d52/d39/d86/dd4 0 2024-04-06T14:17:08.524 INFO:tasks.workunit.client.1.smithi086.stdout:6/432: rmdir d6/d13/d1a/d64/d93 0 2024-04-06T14:17:08.525 INFO:tasks.workunit.client.0.smithi032.stdout:4/674: dread d0/fc3 [0,4194304] 0 2024-04-06T14:17:08.526 INFO:tasks.workunit.client.1.smithi086.stdout:6/433: stat d6/d13/d5f/d60/f5b 0 2024-04-06T14:17:08.526 INFO:tasks.workunit.client.0.smithi032.stdout:1/709: creat d0/d9/d94/fe3 x:0 0 0 2024-04-06T14:17:08.527 INFO:tasks.workunit.client.1.smithi086.stdout:6/434: write d6/d20/f51 [2785423,36384] 0 2024-04-06T14:17:08.527 INFO:tasks.workunit.client.1.smithi086.stdout:9/376: creat d5/d13/f7c x:0 0 0 2024-04-06T14:17:08.527 INFO:tasks.workunit.client.1.smithi086.stdout:1/437: rmdir db/d11/d1f/d5c 39 2024-04-06T14:17:08.528 INFO:tasks.workunit.client.0.smithi032.stdout:9/643: rename d2/d15/d1e/f2a to d2/d15/d1e/d65/d83/d7e/fd1 0 2024-04-06T14:17:08.529 INFO:tasks.workunit.client.1.smithi086.stdout:2/450: dread d0/f12 [0,4194304] 0 2024-04-06T14:17:08.529 INFO:tasks.workunit.client.1.smithi086.stdout:6/435: dread - d6/d9/f8c zero size 2024-04-06T14:17:08.530 INFO:tasks.workunit.client.1.smithi086.stdout:1/438: creat db/d13/f86 x:0 0 0 2024-04-06T14:17:08.530 INFO:tasks.workunit.client.1.smithi086.stdout:6/436: chown d6/d13/d5f/d60/d45/d8a 4390610 1 2024-04-06T14:17:08.531 INFO:tasks.workunit.client.0.smithi032.stdout:4/675: read d0/df/d23/d31/d50/d5e/fbf [1064666,88325] 0 2024-04-06T14:17:08.531 INFO:tasks.workunit.client.0.smithi032.stdout:1/710: creat d0/d10/d16/d3f/d98/fe4 x:0 0 0 2024-04-06T14:17:08.533 INFO:tasks.workunit.client.0.smithi032.stdout:8/651: getdents d3/d8/d12/d2f/d4a/d60 0 2024-04-06T14:17:08.534 INFO:tasks.workunit.client.0.smithi032.stdout:4/676: write d0/df/fb9 [466533,41793] 0 2024-04-06T14:17:08.534 INFO:tasks.workunit.client.1.smithi086.stdout:1/439: rmdir db/d11/d1a/d2a 39 2024-04-06T14:17:08.535 INFO:tasks.workunit.client.0.smithi032.stdout:7/656: rename d16/d92/db3/fcd to d16/d20/d54/de4/d70/fea 0 2024-04-06T14:17:08.535 INFO:tasks.workunit.client.0.smithi032.stdout:0/466: link d3/da/d25/d2b/d4a/l6e d3/da/d25/l9e 0 2024-04-06T14:17:08.535 INFO:tasks.workunit.client.0.smithi032.stdout:9/644: rename d2/d15/d1e to d2/d15/d1e/d65/d83/d7e/da7/dd2 22 2024-04-06T14:17:08.537 INFO:tasks.workunit.client.1.smithi086.stdout:1/440: write db/f10 [1681470,122129] 0 2024-04-06T14:17:08.538 INFO:tasks.workunit.client.0.smithi032.stdout:4/677: chown d0/df/d23/f60 57265 1 2024-04-06T14:17:08.538 INFO:tasks.workunit.client.0.smithi032.stdout:9/645: readlink d2/d15/d55/d59/l9d 0 2024-04-06T14:17:08.539 INFO:tasks.workunit.client.1.smithi086.stdout:6/437: mknod d6/d13/d5f/d60/d45/d8a/c97 0 2024-04-06T14:17:08.539 INFO:tasks.workunit.client.1.smithi086.stdout:9/377: dread d5/d10/d14/d19/d1b/f45 [0,4194304] 0 2024-04-06T14:17:08.539 INFO:tasks.workunit.client.0.smithi032.stdout:9/646: chown d2/f4 426067 1 2024-04-06T14:17:08.540 INFO:tasks.workunit.client.0.smithi032.stdout:1/711: write d0/d9/d1f/d7c/d8f/daa/fd6 [373368,111473] 0 2024-04-06T14:17:08.541 INFO:tasks.workunit.client.1.smithi086.stdout:2/451: dwrite d0/f1e [0,4194304] 0 2024-04-06T14:17:08.542 INFO:tasks.workunit.client.1.smithi086.stdout:9/378: chown d5/d13/d47/d5e/l70 463 1 2024-04-06T14:17:08.542 INFO:tasks.workunit.client.0.smithi032.stdout:0/467: creat d3/da/d25/d2b/d4a/d91/f9f x:0 0 0 2024-04-06T14:17:08.543 INFO:tasks.workunit.client.1.smithi086.stdout:6/438: symlink d6/d13/d42/l98 0 2024-04-06T14:17:08.544 INFO:tasks.workunit.client.1.smithi086.stdout:9/379: read d5/d10/d14/d19/d1b/f45 [947161,2560] 0 2024-04-06T14:17:08.545 INFO:tasks.workunit.client.1.smithi086.stdout:1/441: mkdir db/d11/d87 0 2024-04-06T14:17:08.547 INFO:tasks.workunit.client.1.smithi086.stdout:9/380: creat d5/d13/f7d x:0 0 0 2024-04-06T14:17:08.547 INFO:tasks.workunit.client.0.smithi032.stdout:7/657: unlink d16/d20/d54/de4/d2d/d30/d3e/d96/c67 0 2024-04-06T14:17:08.549 INFO:tasks.workunit.client.0.smithi032.stdout:9/647: mkdir d2/d15/d1e/dd3 0 2024-04-06T14:17:08.550 INFO:tasks.workunit.client.1.smithi086.stdout:6/439: rename d6/d20/l32 to d6/d13/d42/l99 0 2024-04-06T14:17:08.550 INFO:tasks.workunit.client.0.smithi032.stdout:8/652: dwrite d3/f31 [0,4194304] 0 2024-04-06T14:17:08.552 INFO:tasks.workunit.client.1.smithi086.stdout:1/442: read db/d11/d1a/d2a/d52/d5b/d68/f7a [593937,96916] 0 2024-04-06T14:17:08.553 INFO:tasks.workunit.client.0.smithi032.stdout:8/653: chown d3/d8/d12/d2f/f62 423628828 1 2024-04-06T14:17:08.554 INFO:tasks.workunit.client.1.smithi086.stdout:2/452: dwrite d0/df/d48/d7d/f87 [0,4194304] 0 2024-04-06T14:17:08.555 INFO:tasks.workunit.client.1.smithi086.stdout:1/443: write db/d11/d1f/f5a [4686471,621] 0 2024-04-06T14:17:08.556 INFO:tasks.workunit.client.1.smithi086.stdout:2/453: dread - d0/f7a zero size 2024-04-06T14:17:08.556 INFO:tasks.workunit.client.1.smithi086.stdout:9/381: rename d5/d13/d1a/l4c to d5/d13/d47/d5e/l7e 0 2024-04-06T14:17:08.560 INFO:tasks.workunit.client.1.smithi086.stdout:9/382: readlink d5/d10/l4a 0 2024-04-06T14:17:08.560 INFO:tasks.workunit.client.1.smithi086.stdout:0/437: sync 2024-04-06T14:17:08.561 INFO:tasks.workunit.client.1.smithi086.stdout:2/454: chown d0/l7 7737559 1 2024-04-06T14:17:08.562 INFO:tasks.workunit.client.0.smithi032.stdout:2/623: dwrite d0/d30/f43 [0,4194304] 0 2024-04-06T14:17:08.562 INFO:tasks.workunit.client.1.smithi086.stdout:2/455: fdatasync d0/f6 0 2024-04-06T14:17:08.562 INFO:tasks.workunit.client.1.smithi086.stdout:2/456: chown d0/fd 371333 1 2024-04-06T14:17:08.562 INFO:tasks.workunit.client.1.smithi086.stdout:2/457: fdatasync d0/df/f21 0 2024-04-06T14:17:08.564 INFO:tasks.workunit.client.0.smithi032.stdout:8/654: dwrite d3/f96 [4194304,4194304] 0 2024-04-06T14:17:08.565 INFO:tasks.workunit.client.0.smithi032.stdout:4/678: getdents d0/df/d2b/d43 0 2024-04-06T14:17:08.567 INFO:tasks.workunit.client.0.smithi032.stdout:8/655: dread - d3/d8/d12/d2f/d4a/d60/d90/fd9 zero size 2024-04-06T14:17:08.569 INFO:tasks.workunit.client.1.smithi086.stdout:0/438: mknod d0/d1/d2/d1a/d79/c85 0 2024-04-06T14:17:08.570 INFO:tasks.workunit.client.0.smithi032.stdout:8/656: chown d3/d8/d7/dd5 338645 1 2024-04-06T14:17:08.571 INFO:tasks.workunit.client.1.smithi086.stdout:2/458: symlink d0/df/d48/d53/l9e 0 2024-04-06T14:17:08.572 INFO:tasks.workunit.client.1.smithi086.stdout:0/439: creat d0/d1/d2/d1a/d51/f86 x:0 0 0 2024-04-06T14:17:08.573 INFO:tasks.workunit.client.1.smithi086.stdout:0/440: readlink d0/l35 0 2024-04-06T14:17:08.576 INFO:tasks.workunit.client.0.smithi032.stdout:8/657: unlink d3/d8/f6d 0 2024-04-06T14:17:08.577 INFO:tasks.workunit.client.1.smithi086.stdout:9/383: dread d5/d10/d14/d19/f4b [0,4194304] 0 2024-04-06T14:17:08.579 INFO:tasks.workunit.client.1.smithi086.stdout:0/441: rename d0/d15/f4c to d0/d15/d26/d33/f87 0 2024-04-06T14:17:08.579 INFO:tasks.workunit.client.0.smithi032.stdout:7/658: link d16/l25 d16/d20/d54/de4/d2d/d30/dbb/leb 0 2024-04-06T14:17:08.582 INFO:tasks.workunit.client.0.smithi032.stdout:8/658: getdents d3/d8/d3a/d7b/db9/ddf 0 2024-04-06T14:17:08.582 INFO:tasks.workunit.client.0.smithi032.stdout:1/712: dread d0/d10/d16/d3f/f5d [0,4194304] 0 2024-04-06T14:17:08.582 INFO:tasks.workunit.client.1.smithi086.stdout:2/459: dwrite d0/df/d48/d53/d76/f91 [0,4194304] 0 2024-04-06T14:17:08.584 INFO:tasks.workunit.client.1.smithi086.stdout:0/442: mkdir d0/d1/d2/d1a/d88 0 2024-04-06T14:17:08.585 INFO:tasks.workunit.client.1.smithi086.stdout:2/460: mkdir d0/d20/d44/d9f 0 2024-04-06T14:17:08.586 INFO:tasks.workunit.client.1.smithi086.stdout:0/443: fsync d0/d1/d4/de/f2e 0 2024-04-06T14:17:08.588 INFO:tasks.workunit.client.1.smithi086.stdout:2/461: symlink d0/d20/d27/la0 0 2024-04-06T14:17:08.589 INFO:tasks.workunit.client.1.smithi086.stdout:0/444: rename d0/d1/d4/d64 to d0/d15/d4e/d48/d89 0 2024-04-06T14:17:08.591 INFO:tasks.workunit.client.0.smithi032.stdout:1/713: link d0/d9/d1f/fdc d0/d9/fe5 0 2024-04-06T14:17:08.591 INFO:tasks.workunit.client.0.smithi032.stdout:8/659: getdents d3/d8/d12/d83/d89 0 2024-04-06T14:17:08.593 INFO:tasks.workunit.client.0.smithi032.stdout:3/629: write d3/d65/d7d/d9c/dad/db1/fc7 [238421,22232] 0 2024-04-06T14:17:08.594 INFO:tasks.workunit.client.0.smithi032.stdout:1/714: write d0/da/d3e/d75/f81 [3320803,62932] 0 2024-04-06T14:17:08.598 INFO:tasks.workunit.client.0.smithi032.stdout:9/648: unlink d2/d15/d1e/d65/d83/d7e/fd1 0 2024-04-06T14:17:08.601 INFO:tasks.workunit.client.0.smithi032.stdout:0/468: getdents d3/da/d25 0 2024-04-06T14:17:08.602 INFO:tasks.workunit.client.0.smithi032.stdout:1/715: rename d0/d10/d78/dd9 to d0/da/d3e/de6 0 2024-04-06T14:17:08.605 INFO:tasks.workunit.client.0.smithi032.stdout:0/469: dread - d3/da/f6f zero size 2024-04-06T14:17:08.606 INFO:tasks.workunit.client.0.smithi032.stdout:8/660: dwrite d3/d8/d7/f47 [4194304,4194304] 0 2024-04-06T14:17:08.609 INFO:tasks.workunit.client.0.smithi032.stdout:1/716: chown d0/d10/d16/fa1 844 1 2024-04-06T14:17:08.611 INFO:tasks.workunit.client.0.smithi032.stdout:4/679: write d0/df/d2b/f3b [2175386,99336] 0 2024-04-06T14:17:08.611 INFO:tasks.workunit.client.0.smithi032.stdout:2/624: truncate d0/d30/d39/d59/d67/d74/d78/d57/f90 606267 0 2024-04-06T14:17:08.615 INFO:tasks.workunit.client.0.smithi032.stdout:3/630: dwrite d3/d65/d7d/d9c/dad/db1/f9b [0,4194304] 0 2024-04-06T14:17:08.615 INFO:tasks.workunit.client.0.smithi032.stdout:0/470: write d3/da/d25/f94 [3433457,22446] 0 2024-04-06T14:17:08.619 INFO:tasks.workunit.client.0.smithi032.stdout:8/661: symlink d3/d8/d12/d83/d89/leb 0 2024-04-06T14:17:08.621 INFO:tasks.workunit.client.0.smithi032.stdout:7/659: dread d16/d20/d54/de4/d5f/f8a [0,4194304] 0 2024-04-06T14:17:08.623 INFO:tasks.workunit.client.0.smithi032.stdout:9/649: write d2/d15/d1e/d65/d83/d31/d39/f52 [41020,112890] 0 2024-04-06T14:17:08.623 INFO:tasks.workunit.client.0.smithi032.stdout:8/662: read - d3/d8/d12/d2f/d4a/d60/d90/fd9 zero size 2024-04-06T14:17:08.625 INFO:tasks.workunit.client.0.smithi032.stdout:0/471: symlink d3/da/d25/d57/la0 0 2024-04-06T14:17:08.632 INFO:tasks.workunit.client.0.smithi032.stdout:8/663: creat d3/d8/d7/fec x:0 0 0 2024-04-06T14:17:08.634 INFO:tasks.workunit.client.0.smithi032.stdout:9/650: creat d2/d15/d55/d59/d93/fd4 x:0 0 0 2024-04-06T14:17:08.634 INFO:tasks.workunit.client.0.smithi032.stdout:0/472: fsync d3/f14 0 2024-04-06T14:17:08.634 INFO:tasks.workunit.client.1.smithi086.stdout:2/462: dread d0/d20/d27/d37/f83 [0,4194304] 0 2024-04-06T14:17:08.634 INFO:tasks.workunit.client.1.smithi086.stdout:6/440: write d6/d13/f77 [2682882,86251] 0 2024-04-06T14:17:08.635 INFO:tasks.workunit.client.1.smithi086.stdout:2/463: stat d0/df/d48 0 2024-04-06T14:17:08.637 INFO:tasks.workunit.client.1.smithi086.stdout:0/445: write d0/d15/d4e/f81 [8018341,24241] 0 2024-04-06T14:17:08.639 INFO:tasks.workunit.client.0.smithi032.stdout:7/660: creat d16/d20/d54/de4/d2d/d30/d3e/d96/fec x:0 0 0 2024-04-06T14:17:08.640 INFO:tasks.workunit.client.0.smithi032.stdout:1/717: dwrite d0/d10/d16/fa5 [0,4194304] 0 2024-04-06T14:17:08.642 INFO:tasks.workunit.client.1.smithi086.stdout:0/446: mknod d0/d1/d2/d6f/c8a 0 2024-04-06T14:17:08.644 INFO:tasks.workunit.client.1.smithi086.stdout:1/444: dwrite db/d13/f14 [0,4194304] 0 2024-04-06T14:17:08.644 INFO:tasks.workunit.client.0.smithi032.stdout:3/631: dread d3/d16/f72 [0,4194304] 0 2024-04-06T14:17:08.644 INFO:tasks.workunit.client.0.smithi032.stdout:0/473: truncate d3/f55 703974 0 2024-04-06T14:17:08.644 INFO:tasks.workunit.client.0.smithi032.stdout:4/680: dwrite d0/df/d2b/f72 [0,4194304] 0 2024-04-06T14:17:08.644 INFO:tasks.workunit.client.0.smithi032.stdout:8/664: chown d3/d8/d12/d83/fbb 14525544 1 2024-04-06T14:17:08.645 INFO:tasks.workunit.client.1.smithi086.stdout:2/464: creat d0/d20/d27/d2b/d35/d36/fa1 x:0 0 0 2024-04-06T14:17:08.645 INFO:tasks.workunit.client.0.smithi032.stdout:2/625: dwrite d0/d30/d39/d59/d67/d80/fad [0,4194304] 0 2024-04-06T14:17:08.646 INFO:tasks.workunit.client.1.smithi086.stdout:1/445: write db/d11/d1f/f7c [85280,46232] 0 2024-04-06T14:17:08.646 INFO:tasks.workunit.client.1.smithi086.stdout:0/447: chown d0/d1/d4/c21 3447 1 2024-04-06T14:17:08.648 INFO:tasks.workunit.client.0.smithi032.stdout:9/651: write d2/d15/d1e/d65/d6b/d92/fb3 [532974,48733] 0 2024-04-06T14:17:08.650 INFO:tasks.workunit.client.0.smithi032.stdout:7/661: chown d16/d20/d6e/f93 180 1 2024-04-06T14:17:08.651 INFO:tasks.workunit.client.0.smithi032.stdout:1/718: dread d0/da/d3e/fad [0,4194304] 0 2024-04-06T14:17:08.652 INFO:tasks.workunit.client.1.smithi086.stdout:0/448: stat d0/d15/d4e/f82 0 2024-04-06T14:17:08.653 INFO:tasks.workunit.client.0.smithi032.stdout:2/626: stat d0/d30/d39/d59/d67/d74/d78/f77 0 2024-04-06T14:17:08.654 INFO:tasks.workunit.client.1.smithi086.stdout:1/446: creat db/d13/d71/f88 x:0 0 0 2024-04-06T14:17:08.659 INFO:tasks.workunit.client.1.smithi086.stdout:1/447: stat db/d11/d1f/d5c 0 2024-04-06T14:17:08.660 INFO:tasks.workunit.client.1.smithi086.stdout:2/465: dwrite d0/d20/d27/d2b/d35/d36/f92 [0,4194304] 0 2024-04-06T14:17:08.660 INFO:tasks.workunit.client.0.smithi032.stdout:9/652: read d2/d15/d1e/d65/d83/d31/f3a [837840,1608] 0 2024-04-06T14:17:08.662 INFO:tasks.workunit.client.0.smithi032.stdout:7/662: rename d16/d20/d63/d9d/la4 to d16/d20/d54/led 0 2024-04-06T14:17:08.662 INFO:tasks.workunit.client.0.smithi032.stdout:4/681: mknod d0/df/d23/d31/d50/d5e/da3/ce7 0 2024-04-06T14:17:08.663 INFO:tasks.workunit.client.1.smithi086.stdout:2/466: chown d0/f2a 635 1 2024-04-06T14:17:08.665 INFO:tasks.workunit.client.0.smithi032.stdout:3/632: unlink d3/d65/d7d/d9c/dad/db1/lb2 0 2024-04-06T14:17:08.667 INFO:tasks.workunit.client.0.smithi032.stdout:9/653: fdatasync d2/d15/d1e/d41/f63 0 2024-04-06T14:17:08.667 INFO:tasks.workunit.client.1.smithi086.stdout:1/448: rename db/d11/d1a/d2a/f3b to db/d11/d1a/d79/f89 0 2024-04-06T14:17:08.668 INFO:tasks.workunit.client.1.smithi086.stdout:0/449: dwrite d0/d1/d2/d1a/d51/f60 [0,4194304] 0 2024-04-06T14:17:08.669 INFO:tasks.workunit.client.0.smithi032.stdout:4/682: unlink d0/df/d1f/d2e/l83 0 2024-04-06T14:17:08.669 INFO:tasks.workunit.client.0.smithi032.stdout:9/654: fdatasync d2/d15/d1e/d65/d83/d7e/faf 0 2024-04-06T14:17:08.670 INFO:tasks.workunit.client.1.smithi086.stdout:0/450: dread - d0/d1/d2/d1a/d51/f86 zero size 2024-04-06T14:17:08.671 INFO:tasks.workunit.client.0.smithi032.stdout:8/665: dread d3/d8/d12/d2f/d4a/d60/d82/d84/fdd [0,4194304] 0 2024-04-06T14:17:08.674 INFO:tasks.workunit.client.0.smithi032.stdout:3/633: rename d3/d65/d7d/d9c/dad/fea to d3/d65/d7d/dc6/ff2 0 2024-04-06T14:17:08.675 INFO:tasks.workunit.client.1.smithi086.stdout:6/441: dread d6/f7 [0,4194304] 0 2024-04-06T14:17:08.675 INFO:tasks.workunit.client.0.smithi032.stdout:0/474: getdents d3/da/d25/d57 0 2024-04-06T14:17:08.677 INFO:tasks.workunit.client.0.smithi032.stdout:4/683: rmdir d0/df/d23/d31/d50/d57/d8a 39 2024-04-06T14:17:08.677 INFO:tasks.workunit.client.1.smithi086.stdout:2/467: link d0/f8e d0/d20/d27/d2b/d35/d36/fa2 0 2024-04-06T14:17:08.678 INFO:tasks.workunit.client.1.smithi086.stdout:1/449: dread db/d11/d1a/f4b [0,4194304] 0 2024-04-06T14:17:08.679 INFO:tasks.workunit.client.1.smithi086.stdout:6/442: truncate d6/d9/db/f7a 1695332 0 2024-04-06T14:17:08.680 INFO:tasks.workunit.client.0.smithi032.stdout:8/666: link d3/d8/d12/f51 d3/d8/d12/d2f/d4a/d60/fed 0 2024-04-06T14:17:08.682 INFO:tasks.workunit.client.1.smithi086.stdout:0/451: dwrite d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:08.682 INFO:tasks.workunit.client.1.smithi086.stdout:1/450: creat db/d11/d1d/f8a x:0 0 0 2024-04-06T14:17:08.682 INFO:tasks.workunit.client.1.smithi086.stdout:6/443: dread - d6/d13/d42/d52/f7f zero size 2024-04-06T14:17:08.683 INFO:tasks.workunit.client.0.smithi032.stdout:4/684: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/de8 0 2024-04-06T14:17:08.684 INFO:tasks.workunit.client.1.smithi086.stdout:1/451: chown db/d11/d42/d53/f64 338 1 2024-04-06T14:17:08.685 INFO:tasks.workunit.client.1.smithi086.stdout:1/452: fdatasync db/d13/f74 0 2024-04-06T14:17:08.686 INFO:tasks.workunit.client.1.smithi086.stdout:1/453: chown db/d13/f74 1621577 1 2024-04-06T14:17:08.687 INFO:tasks.workunit.client.0.smithi032.stdout:3/634: dwrite d3/d16/f72 [0,4194304] 0 2024-04-06T14:17:08.688 INFO:tasks.workunit.client.0.smithi032.stdout:4/685: unlink d0/d82/d4b/c5f 0 2024-04-06T14:17:08.689 INFO:tasks.workunit.client.1.smithi086.stdout:0/452: mknod d0/d1/d2/d5b/c8b 0 2024-04-06T14:17:08.689 INFO:tasks.workunit.client.1.smithi086.stdout:2/468: link d0/d20/d27/d37/l5f d0/df/la3 0 2024-04-06T14:17:08.690 INFO:tasks.workunit.client.1.smithi086.stdout:0/453: write d0/d1/f76 [879395,65915] 0 2024-04-06T14:17:08.693 INFO:tasks.workunit.client.0.smithi032.stdout:4/686: creat d0/fe9 x:0 0 0 2024-04-06T14:17:08.694 INFO:tasks.workunit.client.1.smithi086.stdout:1/454: rename db/c70 to db/d11/d1a/d2a/d52/d5b/d68/d6e/d81/c8b 0 2024-04-06T14:17:08.695 INFO:tasks.workunit.client.0.smithi032.stdout:4/687: chown d0/d82/d4b 29461 1 2024-04-06T14:17:08.695 INFO:tasks.workunit.client.0.smithi032.stdout:3/635: creat d3/d65/d7d/d9c/ff3 x:0 0 0 2024-04-06T14:17:08.696 INFO:tasks.workunit.client.1.smithi086.stdout:0/454: read d0/d15/f6a [4014540,108173] 0 2024-04-06T14:17:08.696 INFO:tasks.workunit.client.1.smithi086.stdout:6/444: dread d6/d9/db/f7a [0,4194304] 0 2024-04-06T14:17:08.697 INFO:tasks.workunit.client.1.smithi086.stdout:2/469: creat d0/d20/d27/d2b/fa4 x:0 0 0 2024-04-06T14:17:08.698 INFO:tasks.workunit.client.0.smithi032.stdout:4/688: write d0/df/d23/fc7 [1609465,61232] 0 2024-04-06T14:17:08.699 INFO:tasks.workunit.client.0.smithi032.stdout:4/689: dread - d0/df/d2b/d66/fd4 zero size 2024-04-06T14:17:08.700 INFO:tasks.workunit.client.0.smithi032.stdout:1/719: dread d0/da/d3e/f50 [0,4194304] 0 2024-04-06T14:17:08.701 INFO:tasks.workunit.client.0.smithi032.stdout:2/627: truncate d0/d4e/f53 1229006 0 2024-04-06T14:17:08.701 INFO:tasks.workunit.client.1.smithi086.stdout:1/455: write db/d13/f83 [3216409,26284] 0 2024-04-06T14:17:08.702 INFO:tasks.workunit.client.1.smithi086.stdout:6/445: unlink d6/d13/d42/d52/l7d 0 2024-04-06T14:17:08.703 INFO:tasks.workunit.client.1.smithi086.stdout:3/473: sync 2024-04-06T14:17:08.703 INFO:tasks.workunit.client.1.smithi086.stdout:8/517: sync 2024-04-06T14:17:08.704 INFO:tasks.workunit.client.0.smithi032.stdout:3/636: mknod d3/d65/d7d/d9c/dad/dc5/cf4 0 2024-04-06T14:17:08.705 INFO:tasks.workunit.client.0.smithi032.stdout:3/637: stat d3/d65/d89/l91 0 2024-04-06T14:17:08.707 INFO:tasks.workunit.client.0.smithi032.stdout:3/638: write d3/fac [1732619,24311] 0 2024-04-06T14:17:08.709 INFO:tasks.workunit.client.1.smithi086.stdout:6/446: write d6/d13/d42/d52/f65 [602383,95404] 0 2024-04-06T14:17:08.709 INFO:tasks.workunit.client.1.smithi086.stdout:1/456: mknod db/d11/d1a/d79/c8c 0 2024-04-06T14:17:08.711 INFO:tasks.workunit.client.1.smithi086.stdout:2/470: write d0/d20/d27/d2b/d35/d36/d4a/f97 [222647,6698] 0 2024-04-06T14:17:08.711 INFO:tasks.workunit.client.0.smithi032.stdout:4/690: chown d0/df/d1f/d5a/c69 468220265 1 2024-04-06T14:17:08.713 INFO:tasks.workunit.client.0.smithi032.stdout:9/655: dread d2/f53 [0,4194304] 0 2024-04-06T14:17:08.714 INFO:tasks.workunit.client.1.smithi086.stdout:8/518: mkdir d0/d71/d22/d43/d6d/d75/db2 0 2024-04-06T14:17:08.715 INFO:tasks.workunit.client.0.smithi032.stdout:7/663: dread d16/d20/f47 [0,4194304] 0 2024-04-06T14:17:08.716 INFO:tasks.workunit.client.0.smithi032.stdout:1/720: dread d0/f90 [0,4194304] 0 2024-04-06T14:17:08.717 INFO:tasks.workunit.client.0.smithi032.stdout:8/667: creat d3/d8/d12/d2f/d4a/d60/d75/d85/fee x:0 0 0 2024-04-06T14:17:08.717 INFO:tasks.workunit.client.0.smithi032.stdout:9/656: fsync d2/d15/d1e/d65/d6b/d7f/fbf 0 2024-04-06T14:17:08.718 INFO:tasks.workunit.client.1.smithi086.stdout:8/519: creat d0/d13/fb3 x:0 0 0 2024-04-06T14:17:08.718 INFO:tasks.workunit.client.0.smithi032.stdout:4/691: mknod d0/df/cea 0 2024-04-06T14:17:08.718 INFO:tasks.workunit.client.0.smithi032.stdout:7/664: dread - d16/d20/d54/f9c zero size 2024-04-06T14:17:08.719 INFO:tasks.workunit.client.0.smithi032.stdout:2/628: dread d0/f22 [0,4194304] 0 2024-04-06T14:17:08.719 INFO:tasks.workunit.client.0.smithi032.stdout:0/475: dread d3/da/f18 [0,4194304] 0 2024-04-06T14:17:08.721 INFO:tasks.workunit.client.0.smithi032.stdout:4/692: write d0/df/d23/fc7 [1789850,10602] 0 2024-04-06T14:17:08.722 INFO:tasks.workunit.client.1.smithi086.stdout:0/455: dwrite d0/f3f [0,4194304] 0 2024-04-06T14:17:08.723 INFO:tasks.workunit.client.1.smithi086.stdout:8/520: readlink d0/d71/d22/d43/d4d/l85 0 2024-04-06T14:17:08.726 INFO:tasks.workunit.client.1.smithi086.stdout:3/474: dwrite d5/d36/d5a/d6d/d72/d64/f71 [0,4194304] 0 2024-04-06T14:17:08.727 INFO:tasks.workunit.client.1.smithi086.stdout:3/475: write d5/f50 [318071,122074] 0 2024-04-06T14:17:08.727 INFO:tasks.workunit.client.1.smithi086.stdout:6/447: link d6/d9/db/c83 d6/d13/d1a/d64/d85/d88/c9a 0 2024-04-06T14:17:08.728 INFO:tasks.workunit.client.0.smithi032.stdout:4/693: read - d0/df/f8b zero size 2024-04-06T14:17:08.731 INFO:tasks.workunit.client.1.smithi086.stdout:2/471: mkdir d0/d20/d27/d2b/d35/d36/d4d/da5 0 2024-04-06T14:17:08.731 INFO:tasks.workunit.client.1.smithi086.stdout:0/456: rmdir d0/d15 39 2024-04-06T14:17:08.732 INFO:tasks.workunit.client.0.smithi032.stdout:2/629: rmdir d0/d30/d39/d59/d67/d74 39 2024-04-06T14:17:08.733 INFO:tasks.workunit.client.1.smithi086.stdout:6/448: fsync d6/d13/f75 0 2024-04-06T14:17:08.734 INFO:tasks.workunit.client.1.smithi086.stdout:8/521: chown d0/d71/d22/c6a 8293559 1 2024-04-06T14:17:08.734 INFO:tasks.workunit.client.1.smithi086.stdout:1/457: dread db/d11/d1f/f5a [0,4194304] 0 2024-04-06T14:17:08.735 INFO:tasks.workunit.client.1.smithi086.stdout:3/476: write d5/d36/d5a/d6d/d72/d58/f63 [956431,87302] 0 2024-04-06T14:17:08.737 INFO:tasks.workunit.client.1.smithi086.stdout:2/472: truncate d0/df/f21 5313577 0 2024-04-06T14:17:08.738 INFO:tasks.workunit.client.1.smithi086.stdout:0/457: creat d0/d1/d2/d1a/d79/f8c x:0 0 0 2024-04-06T14:17:08.740 INFO:tasks.workunit.client.1.smithi086.stdout:6/449: creat d6/d9/db/f9b x:0 0 0 2024-04-06T14:17:08.744 INFO:tasks.workunit.client.1.smithi086.stdout:2/473: dread d0/f8e [0,4194304] 0 2024-04-06T14:17:08.745 INFO:tasks.workunit.client.1.smithi086.stdout:1/458: creat db/d11/d1a/d2a/d52/d5b/d68/d6e/f8d x:0 0 0 2024-04-06T14:17:08.747 INFO:tasks.workunit.client.1.smithi086.stdout:8/522: creat d0/d71/d22/d65/d6e/d84/fb4 x:0 0 0 2024-04-06T14:17:08.748 INFO:tasks.workunit.client.1.smithi086.stdout:8/523: fsync d0/d13/fb3 0 2024-04-06T14:17:08.749 INFO:tasks.workunit.client.1.smithi086.stdout:0/458: creat d0/d15/f8d x:0 0 0 2024-04-06T14:17:08.751 INFO:tasks.workunit.client.1.smithi086.stdout:1/459: creat db/d11/d1d/f8e x:0 0 0 2024-04-06T14:17:08.752 INFO:tasks.workunit.client.1.smithi086.stdout:2/474: link d0/d20/d27/d2b/d35/d36/f3b d0/df/d48/d7d/fa6 0 2024-04-06T14:17:08.754 INFO:tasks.workunit.client.1.smithi086.stdout:1/460: creat db/d11/d1a/d2a/d52/d5b/d68/d6e/f8f x:0 0 0 2024-04-06T14:17:08.757 INFO:tasks.workunit.client.1.smithi086.stdout:3/477: write d5/d26/d7f/f60 [5343193,34232] 0 2024-04-06T14:17:08.757 INFO:tasks.workunit.client.1.smithi086.stdout:0/459: getdents d0/d1/d2/d1a/d88 0 2024-04-06T14:17:08.757 INFO:tasks.workunit.client.1.smithi086.stdout:1/461: readlink db/d11/l37 0 2024-04-06T14:17:08.757 INFO:tasks.workunit.client.1.smithi086.stdout:8/524: dwrite d0/db/f5e [0,4194304] 0 2024-04-06T14:17:08.758 INFO:tasks.workunit.client.1.smithi086.stdout:6/450: dread d6/d13/d1a/f8f [0,4194304] 0 2024-04-06T14:17:08.759 INFO:tasks.workunit.client.1.smithi086.stdout:0/460: fdatasync d0/d1/d2/d1a/d51/f86 0 2024-04-06T14:17:08.761 INFO:tasks.workunit.client.1.smithi086.stdout:0/461: chown d0/f40 3544552 1 2024-04-06T14:17:08.762 INFO:tasks.workunit.client.1.smithi086.stdout:3/478: rename d5/d36/d5a/d6d/d72/d64/f71 to d5/d33/d5d/f92 0 2024-04-06T14:17:08.766 INFO:tasks.workunit.client.1.smithi086.stdout:6/451: creat d6/d13/d1a/d22/f9c x:0 0 0 2024-04-06T14:17:08.771 INFO:tasks.workunit.client.1.smithi086.stdout:0/462: rename d0/d1/d2/d6f to d0/d1/d1b/d44/d8e 0 2024-04-06T14:17:08.772 INFO:tasks.workunit.client.1.smithi086.stdout:8/525: dwrite d0/d71/d16/d52/f69 [0,4194304] 0 2024-04-06T14:17:08.772 INFO:tasks.workunit.client.1.smithi086.stdout:5/382: sync 2024-04-06T14:17:08.772 INFO:tasks.workunit.client.1.smithi086.stdout:7/393: sync 2024-04-06T14:17:08.773 INFO:tasks.workunit.client.1.smithi086.stdout:2/475: dread d0/d20/d27/d2b/d35/f62 [0,4194304] 0 2024-04-06T14:17:08.774 INFO:tasks.workunit.client.1.smithi086.stdout:0/463: readlink d0/d1/d4/d7/l59 0 2024-04-06T14:17:08.778 INFO:tasks.workunit.client.1.smithi086.stdout:7/394: write d7/d10/d11/d80/f4a [295407,59283] 0 2024-04-06T14:17:08.778 INFO:tasks.workunit.client.1.smithi086.stdout:5/383: stat d0/l64 0 2024-04-06T14:17:08.779 INFO:tasks.workunit.client.1.smithi086.stdout:3/479: dwrite d5/d36/d56/f74 [0,4194304] 0 2024-04-06T14:17:08.779 INFO:tasks.workunit.client.1.smithi086.stdout:2/476: stat d0/df/c6b 0 2024-04-06T14:17:08.784 INFO:tasks.workunit.client.1.smithi086.stdout:0/464: rmdir d0/d1/d1b/d44/d8e 39 2024-04-06T14:17:08.785 INFO:tasks.workunit.client.1.smithi086.stdout:3/480: fdatasync d5/df/f81 0 2024-04-06T14:17:08.785 INFO:tasks.workunit.client.1.smithi086.stdout:6/452: symlink d6/d9/l9d 0 2024-04-06T14:17:08.787 INFO:tasks.workunit.client.1.smithi086.stdout:3/481: stat d5/df/f6b 0 2024-04-06T14:17:08.788 INFO:tasks.workunit.client.1.smithi086.stdout:1/462: truncate db/d13/d55/f59 230812 0 2024-04-06T14:17:08.792 INFO:tasks.workunit.client.1.smithi086.stdout:6/453: write d6/d13/d1a/d64/f71 [183058,35842] 0 2024-04-06T14:17:08.792 INFO:tasks.workunit.client.1.smithi086.stdout:8/526: dwrite d0/d13/fb3 [0,4194304] 0 2024-04-06T14:17:08.792 INFO:tasks.workunit.client.1.smithi086.stdout:0/465: mkdir d0/d1/d1b/d44/d8e/d8f 0 2024-04-06T14:17:08.794 INFO:tasks.workunit.client.1.smithi086.stdout:7/395: rename d7/d10/d11/d1c/d24/d2c/d69/f71 to d7/d10/d11/f88 0 2024-04-06T14:17:08.798 INFO:tasks.workunit.client.1.smithi086.stdout:8/527: write d0/d71/d22/f74 [3861831,20198] 0 2024-04-06T14:17:08.800 INFO:tasks.workunit.client.0.smithi032.stdout:6/639: sync 2024-04-06T14:17:08.800 INFO:tasks.workunit.client.1.smithi086.stdout:6/454: readlink d6/d13/d1a/d22/l7e 0 2024-04-06T14:17:08.801 INFO:tasks.workunit.client.1.smithi086.stdout:0/466: mknod d0/d15/d26/c90 0 2024-04-06T14:17:08.802 INFO:tasks.workunit.client.1.smithi086.stdout:0/467: chown d0/d1/d2/d1a/d79 15881 1 2024-04-06T14:17:08.803 INFO:tasks.workunit.client.1.smithi086.stdout:0/468: chown d0/d1/l19 2 1 2024-04-06T14:17:08.804 INFO:tasks.workunit.client.1.smithi086.stdout:1/463: rename db/d13/f14 to db/d11/d1a/d2a/d52/d5b/d68/d6e/d81/f90 0 2024-04-06T14:17:08.807 INFO:tasks.workunit.client.1.smithi086.stdout:6/455: creat d6/d9/d43/f9e x:0 0 0 2024-04-06T14:17:08.811 INFO:tasks.workunit.client.1.smithi086.stdout:3/482: getdents d5/d36/d5a/d6d/d72/d58 0 2024-04-06T14:17:08.811 INFO:tasks.workunit.client.1.smithi086.stdout:3/483: dread - d5/d36/d65/f7b zero size 2024-04-06T14:17:08.812 INFO:tasks.workunit.client.0.smithi032.stdout:1/721: creat d0/fe7 x:0 0 0 2024-04-06T14:17:08.813 INFO:tasks.workunit.client.0.smithi032.stdout:7/665: rename d16/d20/d48/d6d to d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee 0 2024-04-06T14:17:08.813 INFO:tasks.workunit.client.1.smithi086.stdout:1/464: write db/d11/d1a/d2a/d52/d5b/d68/f7a [1320157,127021] 0 2024-04-06T14:17:08.814 INFO:tasks.workunit.client.0.smithi032.stdout:3/639: dwrite d3/d65/d7d/fd2 [0,4194304] 0 2024-04-06T14:17:08.816 INFO:tasks.workunit.client.0.smithi032.stdout:9/657: write d2/d15/d55/d59/f67 [1407238,17600] 0 2024-04-06T14:17:08.816 INFO:tasks.workunit.client.0.smithi032.stdout:8/668: dread d3/d8/d7/fbc [0,4194304] 0 2024-04-06T14:17:08.817 INFO:tasks.workunit.client.0.smithi032.stdout:4/694: write d0/df/d1f/d42/f99 [306207,111428] 0 2024-04-06T14:17:08.818 INFO:tasks.workunit.client.1.smithi086.stdout:0/469: symlink d0/d1/d1b/d44/d5d/d74/l91 0 2024-04-06T14:17:08.819 INFO:tasks.workunit.client.1.smithi086.stdout:2/477: dread d0/d20/d44/f46 [0,4194304] 0 2024-04-06T14:17:08.819 INFO:tasks.workunit.client.0.smithi032.stdout:6/640: chown d4/d5/d19/d21 377406 1 2024-04-06T14:17:08.820 INFO:tasks.workunit.client.1.smithi086.stdout:8/528: link d0/d71/d22/c6a d0/d71/d22/d43/d6d/cb5 0 2024-04-06T14:17:08.820 INFO:tasks.workunit.client.1.smithi086.stdout:7/396: dread d7/d10/f64 [0,4194304] 0 2024-04-06T14:17:08.822 INFO:tasks.workunit.client.1.smithi086.stdout:0/470: truncate d0/d1/d4/f43 4328467 0 2024-04-06T14:17:08.822 INFO:tasks.workunit.client.0.smithi032.stdout:3/640: mknod d3/d65/d7d/cf5 0 2024-04-06T14:17:08.822 INFO:tasks.workunit.client.0.smithi032.stdout:1/722: chown d0/da/l18 35 1 2024-04-06T14:17:08.822 INFO:tasks.workunit.client.1.smithi086.stdout:2/478: stat d0/d20/d27/d2b/d35/d36/fa2 0 2024-04-06T14:17:08.823 INFO:tasks.workunit.client.0.smithi032.stdout:7/666: read d16/d20/d54/de4/d5f/f90 [42487,29979] 0 2024-04-06T14:17:08.825 INFO:tasks.workunit.client.0.smithi032.stdout:4/695: write d0/d82/d8e/fde [947844,126730] 0 2024-04-06T14:17:08.826 INFO:tasks.workunit.client.0.smithi032.stdout:6/641: creat d4/d5/d19/d25/d9c/fe0 x:0 0 0 2024-04-06T14:17:08.827 INFO:tasks.workunit.client.1.smithi086.stdout:3/484: link d5/d36/d5a/d6d/d72/d58/f63 d5/d36/d65/f93 0 2024-04-06T14:17:08.828 INFO:tasks.workunit.client.1.smithi086.stdout:8/529: mkdir d0/d13/db6 0 2024-04-06T14:17:08.829 INFO:tasks.workunit.client.0.smithi032.stdout:1/723: unlink d0/d10/l56 0 2024-04-06T14:17:08.830 INFO:tasks.workunit.client.1.smithi086.stdout:6/456: link d6/f7 d6/d13/d1a/d64/f9f 0 2024-04-06T14:17:08.830 INFO:tasks.workunit.client.1.smithi086.stdout:6/457: dread - d6/d9/db/f76 zero size 2024-04-06T14:17:08.832 INFO:tasks.workunit.client.0.smithi032.stdout:7/667: write d16/d20/f37 [2732723,2167] 0 2024-04-06T14:17:08.833 INFO:tasks.workunit.client.1.smithi086.stdout:0/471: creat d0/d15/d4e/f92 x:0 0 0 2024-04-06T14:17:08.834 INFO:tasks.workunit.client.1.smithi086.stdout:7/397: creat d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/f89 x:0 0 0 2024-04-06T14:17:08.836 INFO:tasks.workunit.client.0.smithi032.stdout:3/641: rename d3/d16/laa to d3/d65/d7d/d9c/dad/db1/d3c/lf6 0 2024-04-06T14:17:08.836 INFO:tasks.workunit.client.1.smithi086.stdout:6/458: mkdir d6/d13/d1a/da0 0 2024-04-06T14:17:08.837 INFO:tasks.workunit.client.0.smithi032.stdout:6/642: symlink d4/d5/d28/d65/d6e/le1 0 2024-04-06T14:17:08.839 INFO:tasks.workunit.client.0.smithi032.stdout:2/630: dwrite d0/d30/d39/d59/d67/d74/d78/d8c/db0/fb3 [8388608,4194304] 0 2024-04-06T14:17:08.839 INFO:tasks.workunit.client.0.smithi032.stdout:4/696: creat d0/df/d2b/d46/d58/feb x:0 0 0 2024-04-06T14:17:08.840 INFO:tasks.workunit.client.1.smithi086.stdout:1/465: rename db/d11/d1d/f5f to db/d11/d1a/d2a/d52/d5b/d68/f91 0 2024-04-06T14:17:08.841 INFO:tasks.workunit.client.0.smithi032.stdout:8/669: truncate d3/d8/d7/fbc 3631448 0 2024-04-06T14:17:08.842 INFO:tasks.workunit.client.1.smithi086.stdout:2/479: creat d0/df/d5e/d9d/fa7 x:0 0 0 2024-04-06T14:17:08.844 INFO:tasks.workunit.client.1.smithi086.stdout:7/398: unlink d7/d10/d11/d1c/d24/d2c/d3c/l51 0 2024-04-06T14:17:08.845 INFO:tasks.workunit.client.1.smithi086.stdout:2/480: readlink d0/l7 0 2024-04-06T14:17:08.846 INFO:tasks.workunit.client.0.smithi032.stdout:7/668: chown d16/d20/d54/de4/d2d/d30/d3e/d96/cc8 245807984 1 2024-04-06T14:17:08.846 INFO:tasks.workunit.client.0.smithi032.stdout:3/642: mkdir d3/d16/ddd/df7 0 2024-04-06T14:17:08.847 INFO:tasks.workunit.client.0.smithi032.stdout:6/643: write d4/d5/d28/f41 [860704,73567] 0 2024-04-06T14:17:08.847 INFO:tasks.workunit.client.0.smithi032.stdout:4/697: rename d0/df/d23 to d0/df/d23/d31/d50/d5e/da3/dec 22 2024-04-06T14:17:08.847 INFO:tasks.workunit.client.0.smithi032.stdout:7/669: stat d16/d20/d63/ca0 0 2024-04-06T14:17:08.848 INFO:tasks.workunit.client.1.smithi086.stdout:3/485: rename d5/d26/l80 to d5/d36/d5a/d6d/l94 0 2024-04-06T14:17:08.849 INFO:tasks.workunit.client.1.smithi086.stdout:6/459: dwrite d6/d13/d5f/d60/f5b [4194304,4194304] 0 2024-04-06T14:17:08.850 INFO:tasks.workunit.client.0.smithi032.stdout:7/670: chown d16/d20/d54/de4/d2d/d30/dbb 20429610 1 2024-04-06T14:17:08.850 INFO:tasks.workunit.client.0.smithi032.stdout:4/698: read d0/d82/d4b/fbd [479206,104126] 0 2024-04-06T14:17:08.853 INFO:tasks.workunit.client.0.smithi032.stdout:9/658: dread d2/d15/d1e/d65/d83/d31/f40 [0,4194304] 0 2024-04-06T14:17:08.854 INFO:tasks.workunit.client.1.smithi086.stdout:6/460: chown d6/d20/f46 8139986 1 2024-04-06T14:17:08.854 INFO:tasks.workunit.client.1.smithi086.stdout:2/481: chown d0/d20/d27/d2b/d35/d36/f40 0 1 2024-04-06T14:17:08.854 INFO:tasks.workunit.client.0.smithi032.stdout:3/643: write d3/d65/d7d/d9c/dad/db1/fc7 [1101989,2952] 0 2024-04-06T14:17:08.854 INFO:tasks.workunit.client.0.smithi032.stdout:6/644: fsync d4/d5/f2a 0 2024-04-06T14:17:08.856 INFO:tasks.workunit.client.1.smithi086.stdout:3/486: chown d5/d36/d5c/c7e 432 1 2024-04-06T14:17:08.857 INFO:tasks.workunit.client.1.smithi086.stdout:0/472: dwrite d0/d1/d2/d1a/f56 [0,4194304] 0 2024-04-06T14:17:08.857 INFO:tasks.workunit.client.1.smithi086.stdout:7/399: creat d7/d10/d11/d1c/d24/d2c/d3c/f8a x:0 0 0 2024-04-06T14:17:08.859 INFO:tasks.workunit.client.0.smithi032.stdout:4/699: write d0/df/d1f/d5a/f8f [328252,25026] 0 2024-04-06T14:17:08.860 INFO:tasks.workunit.client.0.smithi032.stdout:6/645: chown d4/f13 75007248 1 2024-04-06T14:17:08.860 INFO:tasks.workunit.client.1.smithi086.stdout:7/400: chown d7/d1d/f6b 16027 1 2024-04-06T14:17:08.860 INFO:tasks.workunit.client.0.smithi032.stdout:3/644: write d3/d65/f69 [5244080,13821] 0 2024-04-06T14:17:08.862 INFO:tasks.workunit.client.1.smithi086.stdout:1/466: dwrite db/d13/f23 [0,4194304] 0 2024-04-06T14:17:08.863 INFO:tasks.workunit.client.0.smithi032.stdout:8/670: getdents d3/d8/d3a/d4b 0 2024-04-06T14:17:08.864 INFO:tasks.workunit.client.1.smithi086.stdout:2/482: rmdir d0/d20/d80 39 2024-04-06T14:17:08.866 INFO:tasks.workunit.client.0.smithi032.stdout:3/645: fsync d3/d65/d7d/d9c/dad/db1/f62 0 2024-04-06T14:17:08.866 INFO:tasks.workunit.client.0.smithi032.stdout:7/671: rename d16/d20/d54/de4/d5f/ddd/cc0 to d16/d20/d54/de4/de0/cef 0 2024-04-06T14:17:08.868 INFO:tasks.workunit.client.0.smithi032.stdout:6/646: fsync d4/d5/dc5/f50 0 2024-04-06T14:17:08.869 INFO:tasks.workunit.client.0.smithi032.stdout:2/631: dread d0/d30/f4d [0,4194304] 0 2024-04-06T14:17:08.869 INFO:tasks.workunit.client.1.smithi086.stdout:6/461: creat d6/d13/d5f/d60/d45/fa1 x:0 0 0 2024-04-06T14:17:08.869 INFO:tasks.workunit.client.1.smithi086.stdout:0/473: mknod d0/d1/d2/d5b/c93 0 2024-04-06T14:17:08.870 INFO:tasks.workunit.client.0.smithi032.stdout:9/659: symlink d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8f/ld5 0 2024-04-06T14:17:08.872 INFO:tasks.workunit.client.1.smithi086.stdout:1/467: unlink db/d11/d1d/f8e 0 2024-04-06T14:17:08.872 INFO:tasks.workunit.client.1.smithi086.stdout:7/401: rmdir d7/d10/d11/d1c/d24/d2c/d3c/d7a 39 2024-04-06T14:17:08.872 INFO:tasks.workunit.client.0.smithi032.stdout:7/672: write d16/d20/d54/de4/d5f/fb0 [70319,91368] 0 2024-04-06T14:17:08.873 INFO:tasks.workunit.client.0.smithi032.stdout:8/671: mkdir d3/d8/d12/d2f/d46/def 0 2024-04-06T14:17:08.873 INFO:tasks.workunit.client.0.smithi032.stdout:2/632: chown d0/c29 3070787 1 2024-04-06T14:17:08.874 INFO:tasks.workunit.client.0.smithi032.stdout:9/660: chown d2/d15/d1e/d41/d6c/d74 15233 1 2024-04-06T14:17:08.874 INFO:tasks.workunit.client.1.smithi086.stdout:1/468: readlink db/d11/d1a/d2a/l5d 0 2024-04-06T14:17:08.875 INFO:tasks.workunit.client.1.smithi086.stdout:3/487: dwrite d5/df/f6b [0,4194304] 0 2024-04-06T14:17:08.876 INFO:tasks.workunit.client.1.smithi086.stdout:8/530: dread d0/d13/f4b [0,4194304] 0 2024-04-06T14:17:08.879 INFO:tasks.workunit.client.1.smithi086.stdout:7/402: mknod d7/d10/c8b 0 2024-04-06T14:17:08.882 INFO:tasks.workunit.client.1.smithi086.stdout:3/488: symlink d5/df/l95 0 2024-04-06T14:17:08.884 INFO:tasks.workunit.client.1.smithi086.stdout:1/469: dwrite db/d13/f54 [4194304,4194304] 0 2024-04-06T14:17:08.885 INFO:tasks.workunit.client.0.smithi032.stdout:6/647: dwrite d4/d5/d19/f7d [0,4194304] 0 2024-04-06T14:17:08.888 INFO:tasks.workunit.client.1.smithi086.stdout:1/470: write db/d13/d71/f88 [340098,53897] 0 2024-04-06T14:17:08.892 INFO:tasks.workunit.client.1.smithi086.stdout:8/531: mknod d0/d71/d22/d43/d4d/d62/cb7 0 2024-04-06T14:17:08.896 INFO:tasks.workunit.client.1.smithi086.stdout:3/489: creat d5/d36/d5a/d6d/f96 x:0 0 0 2024-04-06T14:17:08.896 INFO:tasks.workunit.client.1.smithi086.stdout:7/403: dwrite d7/d10/d11/d1c/f2e [0,4194304] 0 2024-04-06T14:17:08.896 INFO:tasks.workunit.client.1.smithi086.stdout:1/471: write db/d11/d1a/d2a/d52/d5b/d68/d6e/f8d [113262,83904] 0 2024-04-06T14:17:08.900 INFO:tasks.workunit.client.1.smithi086.stdout:6/462: truncate d6/d13/d1a/d64/f71 877212 0 2024-04-06T14:17:08.900 INFO:tasks.workunit.client.1.smithi086.stdout:3/490: stat d5/df/f45 0 2024-04-06T14:17:08.900 INFO:tasks.workunit.client.1.smithi086.stdout:1/472: readlink db/l3c 0 2024-04-06T14:17:08.900 INFO:tasks.workunit.client.1.smithi086.stdout:8/532: creat d0/d71/d22/d43/d6d/fb8 x:0 0 0 2024-04-06T14:17:08.901 INFO:tasks.workunit.client.1.smithi086.stdout:1/473: creat db/d13/f92 x:0 0 0 2024-04-06T14:17:08.902 INFO:tasks.workunit.client.1.smithi086.stdout:8/533: mkdir d0/d3b/db9 0 2024-04-06T14:17:08.903 INFO:tasks.workunit.client.1.smithi086.stdout:6/463: mknod d6/d9/d63/ca2 0 2024-04-06T14:17:08.904 INFO:tasks.workunit.client.1.smithi086.stdout:3/491: mkdir d5/d36/d5a/d6d/d72/d97 0 2024-04-06T14:17:08.904 INFO:tasks.workunit.client.1.smithi086.stdout:6/464: fdatasync d6/d13/d42/f7b 0 2024-04-06T14:17:08.906 INFO:tasks.workunit.client.1.smithi086.stdout:1/474: mknod db/d11/d42/d53/c93 0 2024-04-06T14:17:08.907 INFO:tasks.workunit.client.1.smithi086.stdout:9/384: sync 2024-04-06T14:17:08.908 INFO:tasks.workunit.client.1.smithi086.stdout:4/340: sync 2024-04-06T14:17:08.908 INFO:tasks.workunit.client.1.smithi086.stdout:7/404: mkdir d7/d10/d11/d1c/d24/d2c/d3c/d7a/d8c 0 2024-04-06T14:17:08.915 INFO:tasks.workunit.client.1.smithi086.stdout:9/385: write d5/d10/d14/d19/d1b/f6f [464162,81578] 0 2024-04-06T14:17:08.917 INFO:tasks.workunit.client.1.smithi086.stdout:8/534: dwrite d0/d13/f68 [0,4194304] 0 2024-04-06T14:17:08.917 INFO:tasks.workunit.client.1.smithi086.stdout:3/492: dread d5/d26/d7f/f60 [4194304,4194304] 0 2024-04-06T14:17:08.919 INFO:tasks.workunit.client.1.smithi086.stdout:7/405: chown d7/d10/d11/d1c/d24/d2c/d3c/d7a/d8c 644355 1 2024-04-06T14:17:08.919 INFO:tasks.workunit.client.1.smithi086.stdout:3/493: fsync d5/d26/f3d 0 2024-04-06T14:17:08.920 INFO:tasks.workunit.client.1.smithi086.stdout:0/474: dread d0/ff [4194304,4194304] 0 2024-04-06T14:17:08.920 INFO:tasks.workunit.client.1.smithi086.stdout:4/341: truncate d0/d2c/d38/f5b 14852 0 2024-04-06T14:17:08.921 INFO:tasks.workunit.client.1.smithi086.stdout:1/475: getdents db/d11/d42 0 2024-04-06T14:17:08.922 INFO:tasks.workunit.client.1.smithi086.stdout:6/465: truncate d6/d20/f51 3630912 0 2024-04-06T14:17:08.923 INFO:tasks.workunit.client.1.smithi086.stdout:1/476: write f4 [5283620,87276] 0 2024-04-06T14:17:08.923 INFO:tasks.workunit.client.1.smithi086.stdout:9/386: dread - d5/d10/d14/f4e zero size 2024-04-06T14:17:08.924 INFO:tasks.workunit.client.1.smithi086.stdout:6/466: write f5 [1708835,26323] 0 2024-04-06T14:17:08.925 INFO:tasks.workunit.client.1.smithi086.stdout:0/475: creat d0/d1/d2/d1a/f94 x:0 0 0 2024-04-06T14:17:08.926 INFO:tasks.workunit.client.1.smithi086.stdout:7/406: truncate d7/d10/d11/d1c/d24/f7d 840856 0 2024-04-06T14:17:08.926 INFO:tasks.workunit.client.1.smithi086.stdout:9/387: stat d5/d10/d14/d19/c5b 0 2024-04-06T14:17:08.927 INFO:tasks.workunit.client.1.smithi086.stdout:3/494: unlink d5/d36/d5a/d6d/d72/d57/c51 0 2024-04-06T14:17:08.927 INFO:tasks.workunit.client.1.smithi086.stdout:6/467: dread - d6/d13/d42/d52/f7f zero size 2024-04-06T14:17:08.927 INFO:tasks.workunit.client.1.smithi086.stdout:7/407: readlink d7/d1d/l3b 0 2024-04-06T14:17:08.928 INFO:tasks.workunit.client.1.smithi086.stdout:8/535: creat d0/d13/db6/fba x:0 0 0 2024-04-06T14:17:08.929 INFO:tasks.workunit.client.1.smithi086.stdout:4/342: mkdir d0/d2c/d38/d54/d61/d65 0 2024-04-06T14:17:08.930 INFO:tasks.workunit.client.1.smithi086.stdout:0/476: symlink d0/d1/d2/d1a/l95 0 2024-04-06T14:17:08.930 INFO:tasks.workunit.client.1.smithi086.stdout:6/468: fsync d6/d9/d43/f56 0 2024-04-06T14:17:08.932 INFO:tasks.workunit.client.1.smithi086.stdout:0/477: fsync d0/f40 0 2024-04-06T14:17:08.934 INFO:tasks.workunit.client.1.smithi086.stdout:4/343: mknod d0/d15/d20/d34/c66 0 2024-04-06T14:17:08.934 INFO:tasks.workunit.client.1.smithi086.stdout:6/469: symlink d6/d13/la3 0 2024-04-06T14:17:08.936 INFO:tasks.workunit.client.1.smithi086.stdout:7/408: dread - d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/f89 zero size 2024-04-06T14:17:08.936 INFO:tasks.workunit.client.1.smithi086.stdout:9/388: symlink d5/d10/d14/d19/d25/l7f 0 2024-04-06T14:17:08.937 INFO:tasks.workunit.client.1.smithi086.stdout:4/344: chown d0/f36 220 1 2024-04-06T14:17:08.939 INFO:tasks.workunit.client.1.smithi086.stdout:8/536: link d0/l25 d0/db/lbb 0 2024-04-06T14:17:08.941 INFO:tasks.workunit.client.1.smithi086.stdout:0/478: rmdir d0/d1/d2/d1a/d88 0 2024-04-06T14:17:08.941 INFO:tasks.workunit.client.1.smithi086.stdout:3/495: dwrite d5/d36/f3b [0,4194304] 0 2024-04-06T14:17:08.943 INFO:tasks.workunit.client.1.smithi086.stdout:7/409: mkdir d7/d10/d11/d1c/d24/d2c/d69/d8d 0 2024-04-06T14:17:08.943 INFO:tasks.workunit.client.1.smithi086.stdout:6/470: creat d6/d13/fa4 x:0 0 0 2024-04-06T14:17:08.946 INFO:tasks.workunit.client.1.smithi086.stdout:9/389: creat d5/d10/d14/f80 x:0 0 0 2024-04-06T14:17:08.946 INFO:tasks.workunit.client.1.smithi086.stdout:1/477: dread f1 [0,4194304] 0 2024-04-06T14:17:08.946 INFO:tasks.workunit.client.1.smithi086.stdout:7/410: rmdir d7/d10 39 2024-04-06T14:17:08.946 INFO:tasks.workunit.client.1.smithi086.stdout:0/479: symlink d0/d15/d4e/d48/d89/l96 0 2024-04-06T14:17:08.949 INFO:tasks.workunit.client.1.smithi086.stdout:3/496: creat d5/d36/d5a/d6d/d72/d64/f98 x:0 0 0 2024-04-06T14:17:08.950 INFO:tasks.workunit.client.1.smithi086.stdout:8/537: getdents d0/d71/d16 0 2024-04-06T14:17:08.950 INFO:tasks.workunit.client.1.smithi086.stdout:6/471: rename d6/d13/d1a/l25 to d6/d20/la5 0 2024-04-06T14:17:08.956 INFO:tasks.workunit.client.1.smithi086.stdout:9/390: rmdir d5/d13/d47/d5e 39 2024-04-06T14:17:08.956 INFO:tasks.workunit.client.1.smithi086.stdout:3/497: mknod d5/d26/d7f/d59/c99 0 2024-04-06T14:17:08.958 INFO:tasks.workunit.client.1.smithi086.stdout:1/478: dread db/d13/d71/f7f [0,4194304] 0 2024-04-06T14:17:08.960 INFO:tasks.workunit.client.1.smithi086.stdout:4/345: dwrite d0/d15/d20/f44 [4194304,4194304] 0 2024-04-06T14:17:08.960 INFO:tasks.workunit.client.1.smithi086.stdout:6/472: mkdir d6/d13/d5f/d60/d45/da6 0 2024-04-06T14:17:08.962 INFO:tasks.workunit.client.1.smithi086.stdout:8/538: write d0/d13/fa9 [2963,98149] 0 2024-04-06T14:17:08.965 INFO:tasks.workunit.client.1.smithi086.stdout:1/479: symlink db/d13/d55/l94 0 2024-04-06T14:17:08.968 INFO:tasks.workunit.client.1.smithi086.stdout:9/391: rename d5/d13/f44 to d5/d10/d14/d19/d25/d74/f81 0 2024-04-06T14:17:08.968 INFO:tasks.workunit.client.1.smithi086.stdout:6/473: chown d6/d13/d42/l98 14 1 2024-04-06T14:17:08.974 INFO:tasks.workunit.client.1.smithi086.stdout:0/480: dwrite d0/d1/d2/d1a/f63 [0,4194304] 0 2024-04-06T14:17:08.974 INFO:tasks.workunit.client.1.smithi086.stdout:6/474: read d6/f40 [1715432,21176] 0 2024-04-06T14:17:08.974 INFO:tasks.workunit.client.1.smithi086.stdout:8/539: truncate d0/d13/f40 1528655 0 2024-04-06T14:17:08.974 INFO:tasks.workunit.client.1.smithi086.stdout:9/392: chown d5/d10/l78 4 1 2024-04-06T14:17:08.979 INFO:tasks.workunit.client.1.smithi086.stdout:7/411: dread d7/d1d/fc [0,4194304] 0 2024-04-06T14:17:08.980 INFO:tasks.workunit.client.1.smithi086.stdout:3/498: dwrite d5/df/f2c [0,4194304] 0 2024-04-06T14:17:08.985 INFO:tasks.workunit.client.1.smithi086.stdout:1/480: mkdir db/d11/d1a/d2a/d52/d5b/d68/d6e/d95 0 2024-04-06T14:17:08.992 INFO:tasks.workunit.client.1.smithi086.stdout:4/346: truncate d0/f40 378343 0 2024-04-06T14:17:08.996 INFO:tasks.workunit.client.1.smithi086.stdout:1/481: mkdir db/d11/d1a/d2a/d52/d5b/d68/d6e/d81/d96 0 2024-04-06T14:17:08.998 INFO:tasks.workunit.client.1.smithi086.stdout:8/540: dwrite d0/d3b/f8b [0,4194304] 0 2024-04-06T14:17:08.999 INFO:tasks.workunit.client.1.smithi086.stdout:4/347: readlink d0/l4e 0 2024-04-06T14:17:09.004 INFO:tasks.workunit.client.1.smithi086.stdout:0/481: dwrite d0/d15/d26/f4b [0,4194304] 0 2024-04-06T14:17:09.004 INFO:tasks.workunit.client.1.smithi086.stdout:4/348: chown d0/f2 865971 1 2024-04-06T14:17:09.004 INFO:tasks.workunit.client.1.smithi086.stdout:9/393: dread d5/d13/f4f [0,4194304] 0 2024-04-06T14:17:09.004 INFO:tasks.workunit.client.1.smithi086.stdout:7/412: dread d7/d1d/f13 [0,4194304] 0 2024-04-06T14:17:09.005 INFO:tasks.workunit.client.1.smithi086.stdout:3/499: truncate d5/d26/d7f/d59/f8a 1071260 0 2024-04-06T14:17:09.008 INFO:tasks.workunit.client.1.smithi086.stdout:6/475: dwrite d6/d9/d43/f53 [0,4194304] 0 2024-04-06T14:17:09.010 INFO:tasks.workunit.client.1.smithi086.stdout:1/482: link db/d13/f74 db/d11/d1a/d2a/d52/d6b/f97 0 2024-04-06T14:17:09.017 INFO:tasks.workunit.client.1.smithi086.stdout:8/541: read d0/d3b/fa6 [541598,9917] 0 2024-04-06T14:17:09.017 INFO:tasks.workunit.client.1.smithi086.stdout:0/482: rmdir d0/d1/d1b/d73 39 2024-04-06T14:17:09.018 INFO:tasks.workunit.client.1.smithi086.stdout:7/413: creat d7/d10/d11/d21/f8e x:0 0 0 2024-04-06T14:17:09.020 INFO:tasks.workunit.client.1.smithi086.stdout:1/483: mknod db/d11/d1a/d2a/d52/d5b/d68/d6e/c98 0 2024-04-06T14:17:09.021 INFO:tasks.workunit.client.1.smithi086.stdout:3/500: dwrite d5/d33/f47 [0,4194304] 0 2024-04-06T14:17:09.022 INFO:tasks.workunit.client.1.smithi086.stdout:0/483: truncate d0/d15/d4e/f82 335530 0 2024-04-06T14:17:09.023 INFO:tasks.workunit.client.1.smithi086.stdout:6/476: creat d6/d13/d1a/d64/d85/d88/fa7 x:0 0 0 2024-04-06T14:17:09.024 INFO:tasks.workunit.client.1.smithi086.stdout:0/484: read - d0/d15/f8d zero size 2024-04-06T14:17:09.026 INFO:tasks.workunit.client.1.smithi086.stdout:7/414: creat d7/d10/d11/f8f x:0 0 0 2024-04-06T14:17:09.029 INFO:tasks.workunit.client.1.smithi086.stdout:9/394: rename d5/d13/d1a/l33 to d5/d13/d47/d5e/l82 0 2024-04-06T14:17:09.029 INFO:tasks.workunit.client.1.smithi086.stdout:8/542: mkdir d0/d71/d16/d95/dbc 0 2024-04-06T14:17:09.031 INFO:tasks.workunit.client.1.smithi086.stdout:4/349: getdents d0/d15/d20 0 2024-04-06T14:17:09.032 INFO:tasks.workunit.client.1.smithi086.stdout:3/501: dwrite d5/df/f2c [0,4194304] 0 2024-04-06T14:17:09.033 INFO:tasks.workunit.client.1.smithi086.stdout:0/485: mknod d0/d1/d1b/d44/d8e/d8f/c97 0 2024-04-06T14:17:09.036 INFO:tasks.workunit.client.1.smithi086.stdout:7/415: write d7/d1d/fc [7846962,44691] 0 2024-04-06T14:17:09.037 INFO:tasks.workunit.client.1.smithi086.stdout:1/484: rename db/d11/d1f/f51 to db/d11/d1a/d2a/d52/d5b/d68/d6e/d81/f99 0 2024-04-06T14:17:09.039 INFO:tasks.workunit.client.1.smithi086.stdout:0/486: read d0/d1/d4/d7/f27 [551872,48526] 0 2024-04-06T14:17:09.040 INFO:tasks.workunit.client.1.smithi086.stdout:1/485: chown db/d11/d1f 745 1 2024-04-06T14:17:09.040 INFO:tasks.workunit.client.1.smithi086.stdout:6/477: rename d6/f40 to d6/fa8 0 2024-04-06T14:17:09.042 INFO:tasks.workunit.client.1.smithi086.stdout:6/478: chown d6/d9/d43 15179 1 2024-04-06T14:17:09.044 INFO:tasks.workunit.client.1.smithi086.stdout:1/486: chown db/d11/d42/d53/c76 284 1 2024-04-06T14:17:09.046 INFO:tasks.workunit.client.1.smithi086.stdout:7/416: write d7/d10/f64 [98727,67315] 0 2024-04-06T14:17:09.046 INFO:tasks.workunit.client.1.smithi086.stdout:0/487: dread - d0/d1/d25/f70 zero size 2024-04-06T14:17:09.048 INFO:tasks.workunit.client.1.smithi086.stdout:8/543: link d0/d71/d22/d43/d7f/fb1 d0/d3b/db9/fbd 0 2024-04-06T14:17:09.050 INFO:tasks.workunit.client.1.smithi086.stdout:4/350: dwrite d0/de/d4c/f5a [0,4194304] 0 2024-04-06T14:17:09.050 INFO:tasks.workunit.client.1.smithi086.stdout:6/479: unlink d6/d9/db/l2e 0 2024-04-06T14:17:09.050 INFO:tasks.workunit.client.1.smithi086.stdout:7/417: readlink d7/d10/d11/d1c/d24/d2c/l6a 0 2024-04-06T14:17:09.052 INFO:tasks.workunit.client.1.smithi086.stdout:7/418: chown d7/f77 185 1 2024-04-06T14:17:09.052 INFO:tasks.workunit.client.1.smithi086.stdout:8/544: dread d0/d71/d22/f8e [0,4194304] 0 2024-04-06T14:17:09.052 INFO:tasks.workunit.client.1.smithi086.stdout:9/395: rename d5/d13/f2d to d5/d13/d47/f83 0 2024-04-06T14:17:09.054 INFO:tasks.workunit.client.1.smithi086.stdout:1/487: fdatasync db/d13/d55/f59 0 2024-04-06T14:17:09.056 INFO:tasks.workunit.client.1.smithi086.stdout:9/396: read d5/d10/f53 [906088,101357] 0 2024-04-06T14:17:09.057 INFO:tasks.workunit.client.1.smithi086.stdout:9/397: fdatasync d5/d10/d14/d19/d1b/f45 0 2024-04-06T14:17:09.058 INFO:tasks.workunit.client.1.smithi086.stdout:8/545: chown d0/c1e 42 1 2024-04-06T14:17:09.064 INFO:tasks.workunit.client.1.smithi086.stdout:7/419: symlink d7/d10/d11/d80/l90 0 2024-04-06T14:17:09.069 INFO:tasks.workunit.client.1.smithi086.stdout:1/488: rename db/d11/d1a/d2a/d52/d5b to db/d11/d1a/d2a/d9a 0 2024-04-06T14:17:09.070 INFO:tasks.workunit.client.1.smithi086.stdout:6/480: mkdir d6/d9/da9 0 2024-04-06T14:17:09.072 INFO:tasks.workunit.client.1.smithi086.stdout:9/398: symlink d5/d13/l84 0 2024-04-06T14:17:09.072 INFO:tasks.workunit.client.1.smithi086.stdout:5/384: sync 2024-04-06T14:17:09.073 INFO:tasks.workunit.client.1.smithi086.stdout:6/481: truncate d6/d9/db/f28 830147 0 2024-04-06T14:17:09.076 INFO:tasks.workunit.client.1.smithi086.stdout:8/546: chown d0/d71/d22/d43/d6d/cb5 342317642 1 2024-04-06T14:17:09.076 INFO:tasks.workunit.client.1.smithi086.stdout:3/502: dwrite d5/df/f6a [0,4194304] 0 2024-04-06T14:17:09.078 INFO:tasks.workunit.client.1.smithi086.stdout:1/489: unlink db/d11/d1d/l32 0 2024-04-06T14:17:09.078 INFO:tasks.workunit.client.1.smithi086.stdout:5/385: rmdir d0/d19 39 2024-04-06T14:17:09.079 INFO:tasks.workunit.client.1.smithi086.stdout:5/386: stat d0/d16 0 2024-04-06T14:17:09.080 INFO:tasks.workunit.client.1.smithi086.stdout:6/482: symlink d6/d13/d1a/d64/laa 0 2024-04-06T14:17:09.080 INFO:tasks.workunit.client.1.smithi086.stdout:5/387: write d0/f41 [407445,52081] 0 2024-04-06T14:17:09.081 INFO:tasks.workunit.client.1.smithi086.stdout:0/488: write d0/d1/d4/d7/f11 [3987792,101181] 0 2024-04-06T14:17:09.087 INFO:tasks.workunit.client.1.smithi086.stdout:0/489: chown d0/d1/d4/d7/l59 384232982 1 2024-04-06T14:17:09.088 INFO:tasks.workunit.client.1.smithi086.stdout:1/490: mkdir db/d13/d9b 0 2024-04-06T14:17:09.088 INFO:tasks.workunit.client.1.smithi086.stdout:8/547: dwrite d0/d71/d22/d43/f47 [0,4194304] 0 2024-04-06T14:17:09.093 INFO:tasks.workunit.client.1.smithi086.stdout:6/483: dwrite d6/d13/d42/f72 [0,4194304] 0 2024-04-06T14:17:09.093 INFO:tasks.workunit.client.1.smithi086.stdout:5/388: mkdir d0/d43/d74 0 2024-04-06T14:17:09.095 INFO:tasks.workunit.client.1.smithi086.stdout:3/503: mkdir d5/d36/d5a/d6d/d72/d9a 0 2024-04-06T14:17:09.098 INFO:tasks.workunit.client.1.smithi086.stdout:0/490: mknod d0/d1/d4/d7/c98 0 2024-04-06T14:17:09.098 INFO:tasks.workunit.client.1.smithi086.stdout:1/491: creat db/d11/d1a/d2a/f9c x:0 0 0 2024-04-06T14:17:09.100 INFO:tasks.workunit.client.1.smithi086.stdout:8/548: symlink d0/d71/d16/d52/d5f/lbe 0 2024-04-06T14:17:09.104 INFO:tasks.workunit.client.1.smithi086.stdout:3/504: write d5/d26/d7f/f55 [718553,113956] 0 2024-04-06T14:17:09.104 INFO:tasks.workunit.client.1.smithi086.stdout:1/492: unlink f4 0 2024-04-06T14:17:09.104 INFO:tasks.workunit.client.1.smithi086.stdout:7/420: write d7/d10/d11/d1c/d24/d2c/f39 [1325977,16935] 0 2024-04-06T14:17:09.105 INFO:tasks.workunit.client.1.smithi086.stdout:2/483: sync 2024-04-06T14:17:09.107 INFO:tasks.workunit.client.1.smithi086.stdout:4/351: dwrite d0/d15/d20/d34/f55 [0,4194304] 0 2024-04-06T14:17:09.107 INFO:tasks.workunit.client.1.smithi086.stdout:9/399: truncate d5/d13/d1a/f24 2838890 0 2024-04-06T14:17:09.110 INFO:tasks.workunit.client.1.smithi086.stdout:6/484: creat d6/d9/fab x:0 0 0 2024-04-06T14:17:09.113 INFO:tasks.workunit.client.1.smithi086.stdout:8/549: creat d0/d71/d16/d95/dbc/fbf x:0 0 0 2024-04-06T14:17:09.113 INFO:tasks.workunit.client.1.smithi086.stdout:7/421: fsync d7/d10/d11/d80/f50 0 2024-04-06T14:17:09.118 INFO:tasks.workunit.client.1.smithi086.stdout:2/484: creat d0/df/d48/d7d/fa8 x:0 0 0 2024-04-06T14:17:09.120 INFO:tasks.workunit.client.1.smithi086.stdout:9/400: symlink d5/d10/d14/d19/d25/d59/l85 0 2024-04-06T14:17:09.122 INFO:tasks.workunit.client.1.smithi086.stdout:5/389: dwrite d0/fa [0,4194304] 0 2024-04-06T14:17:09.123 INFO:tasks.workunit.client.1.smithi086.stdout:6/485: write d6/d9/d5c/f6b [810073,75021] 0 2024-04-06T14:17:09.124 INFO:tasks.workunit.client.1.smithi086.stdout:7/422: unlink d7/d10/d11/d80/c47 0 2024-04-06T14:17:09.126 INFO:tasks.workunit.client.1.smithi086.stdout:3/505: dread d5/d26/d7f/f60 [4194304,4194304] 0 2024-04-06T14:17:09.126 INFO:tasks.workunit.client.1.smithi086.stdout:9/401: chown d5/d13/d1a 1 1 2024-04-06T14:17:09.126 INFO:tasks.workunit.client.1.smithi086.stdout:0/491: dwrite d0/d1/d4/de/f2e [0,4194304] 0 2024-04-06T14:17:09.133 INFO:tasks.workunit.client.1.smithi086.stdout:8/550: truncate d0/d71/d16/d39/d90/f97 235529 0 2024-04-06T14:17:09.135 INFO:tasks.workunit.client.1.smithi086.stdout:4/352: getdents d0/d2c/d38 0 2024-04-06T14:17:09.136 INFO:tasks.workunit.client.1.smithi086.stdout:1/493: dwrite db/d11/d1a/d2a/f47 [0,4194304] 0 2024-04-06T14:17:09.137 INFO:tasks.workunit.client.1.smithi086.stdout:7/423: unlink d7/d10/d11/f88 0 2024-04-06T14:17:09.138 INFO:tasks.workunit.client.1.smithi086.stdout:5/390: symlink d0/d5e/l75 0 2024-04-06T14:17:09.143 INFO:tasks.workunit.client.1.smithi086.stdout:3/506: stat d5/d36/d5a/d6d/d72/d57/c83 0 2024-04-06T14:17:09.144 INFO:tasks.workunit.client.1.smithi086.stdout:4/353: rmdir d0/d2c 39 2024-04-06T14:17:09.146 INFO:tasks.workunit.client.1.smithi086.stdout:5/391: read d0/d16/f4e [3843607,27416] 0 2024-04-06T14:17:09.146 INFO:tasks.workunit.client.1.smithi086.stdout:2/485: dwrite d0/f2d [4194304,4194304] 0 2024-04-06T14:17:09.148 INFO:tasks.workunit.client.1.smithi086.stdout:8/551: symlink d0/d71/d82/lc0 0 2024-04-06T14:17:09.151 INFO:tasks.workunit.client.1.smithi086.stdout:9/402: creat d5/d10/d14/f86 x:0 0 0 2024-04-06T14:17:09.151 INFO:tasks.workunit.client.1.smithi086.stdout:6/486: link d6/l96 d6/d13/d5f/lac 0 2024-04-06T14:17:09.152 INFO:tasks.workunit.client.1.smithi086.stdout:4/354: write d0/d15/d20/f57 [7374,93445] 0 2024-04-06T14:17:09.153 INFO:tasks.workunit.client.1.smithi086.stdout:7/424: rename d7/d10/d11/d1c/d24/d4d/f5d to d7/d10/d11/f91 0 2024-04-06T14:17:09.153 INFO:tasks.workunit.client.1.smithi086.stdout:8/552: write d0/f5 [3184062,124456] 0 2024-04-06T14:17:09.155 INFO:tasks.workunit.client.1.smithi086.stdout:5/392: write d0/d16/f47 [2962699,66882] 0 2024-04-06T14:17:09.157 INFO:tasks.workunit.client.1.smithi086.stdout:0/492: getdents d0/d1/d1b/d44/d5d 0 2024-04-06T14:17:09.159 INFO:tasks.workunit.client.1.smithi086.stdout:2/486: mknod d0/d20/d27/d37/d52/ca9 0 2024-04-06T14:17:09.159 INFO:tasks.workunit.client.1.smithi086.stdout:1/494: dwrite db/d11/d1a/d2a/f9c [0,4194304] 0 2024-04-06T14:17:09.162 INFO:tasks.workunit.client.1.smithi086.stdout:9/403: write d5/d10/d14/d19/d25/f2f [385505,36978] 0 2024-04-06T14:17:09.162 INFO:tasks.workunit.client.1.smithi086.stdout:5/393: chown d0/d17/d2d/c58 1330461 1 2024-04-06T14:17:09.162 INFO:tasks.workunit.client.1.smithi086.stdout:0/493: rename d0 to d0/d15/d26/d33/d99 22 2024-04-06T14:17:09.163 INFO:tasks.workunit.client.1.smithi086.stdout:7/425: mkdir d7/d10/d92 0 2024-04-06T14:17:09.164 INFO:tasks.workunit.client.1.smithi086.stdout:4/355: dread d0/de/f26 [4194304,4194304] 0 2024-04-06T14:17:09.165 INFO:tasks.workunit.client.1.smithi086.stdout:4/356: stat d0 0 2024-04-06T14:17:09.168 INFO:tasks.workunit.client.1.smithi086.stdout:8/553: write d0/d71/d16/d39/d90/f97 [1202483,46220] 0 2024-04-06T14:17:09.169 INFO:tasks.workunit.client.0.smithi032.stdout:5/606: sync 2024-04-06T14:17:09.169 INFO:tasks.workunit.client.0.smithi032.stdout:0/476: sync 2024-04-06T14:17:09.169 INFO:tasks.workunit.client.0.smithi032.stdout:8/672: creat d3/d8/d12/d2f/d4a/ff0 x:0 0 0 2024-04-06T14:17:09.171 INFO:tasks.workunit.client.1.smithi086.stdout:1/495: dread - db/d11/d1a/d2a/d9a/d68/d6e/f8f zero size 2024-04-06T14:17:09.171 INFO:tasks.workunit.client.1.smithi086.stdout:2/487: mknod d0/d20/d44/d9f/caa 0 2024-04-06T14:17:09.171 INFO:tasks.workunit.client.0.smithi032.stdout:3/646: creat d3/d65/d7d/dc6/de5/ff8 x:0 0 0 2024-04-06T14:17:09.174 INFO:tasks.workunit.client.1.smithi086.stdout:2/488: read d0/d20/d27/d2b/d35/d36/fa2 [555784,7310] 0 2024-04-06T14:17:09.175 INFO:tasks.workunit.client.1.smithi086.stdout:4/357: mknod d0/d15/d19/c67 0 2024-04-06T14:17:09.176 INFO:tasks.workunit.client.1.smithi086.stdout:6/487: dread d6/d13/d5f/d60/d45/d4a/d62/f89 [4194304,4194304] 0 2024-04-06T14:17:09.176 INFO:tasks.workunit.client.1.smithi086.stdout:0/494: dread d0/d15/f4f [0,4194304] 0 2024-04-06T14:17:09.178 INFO:tasks.workunit.client.1.smithi086.stdout:4/358: truncate d0/f3 8697199 0 2024-04-06T14:17:09.179 INFO:tasks.workunit.client.1.smithi086.stdout:9/404: mkdir d5/d13/d47/d55/d87 0 2024-04-06T14:17:09.180 INFO:tasks.workunit.client.1.smithi086.stdout:5/394: dread d0/d16/f67 [0,4194304] 0 2024-04-06T14:17:09.180 INFO:tasks.workunit.client.1.smithi086.stdout:3/507: truncate d5/f44 425057 0 2024-04-06T14:17:09.181 INFO:tasks.workunit.client.1.smithi086.stdout:4/359: readlink d0/d15/d2b/l39 0 2024-04-06T14:17:09.182 INFO:tasks.workunit.client.1.smithi086.stdout:8/554: mkdir d0/d71/d16/d95/dbc/dc1 0 2024-04-06T14:17:09.184 INFO:tasks.workunit.client.1.smithi086.stdout:6/488: read d6/d13/f27 [7973829,106125] 0 2024-04-06T14:17:09.186 INFO:tasks.workunit.client.1.smithi086.stdout:4/360: write d0/de/f62 [726150,33800] 0 2024-04-06T14:17:09.187 INFO:tasks.workunit.client.1.smithi086.stdout:0/495: symlink d0/d15/d4e/d48/d89/l9a 0 2024-04-06T14:17:09.188 INFO:tasks.workunit.client.0.smithi032.stdout:7/673: stat d16/d20/d54/de4/d70/fe9 0 2024-04-06T14:17:09.190 INFO:tasks.workunit.client.0.smithi032.stdout:9/661: rename d2/d15/l4f to d2/d15/d1e/d41/d6c/d88/ld6 0 2024-04-06T14:17:09.191 INFO:tasks.workunit.client.1.smithi086.stdout:0/496: write d0/f17 [2260223,92137] 0 2024-04-06T14:17:09.193 INFO:tasks.workunit.client.1.smithi086.stdout:9/405: rename d5/d10 to d5/d13/d1a/d5f/d88 0 2024-04-06T14:17:09.193 INFO:tasks.workunit.client.1.smithi086.stdout:2/489: dwrite d0/f2d [0,4194304] 0 2024-04-06T14:17:09.193 INFO:tasks.workunit.client.0.smithi032.stdout:5/607: dread - d16/d18/f68 zero size 2024-04-06T14:17:09.195 INFO:tasks.workunit.client.0.smithi032.stdout:1/724: sync 2024-04-06T14:17:09.196 INFO:tasks.workunit.client.1.smithi086.stdout:7/426: dread d7/d10/d11/d80/f46 [4194304,4194304] 0 2024-04-06T14:17:09.196 INFO:tasks.workunit.client.1.smithi086.stdout:9/406: chown d5/d13/d1a/d5f/d88/l23 509577 1 2024-04-06T14:17:09.197 INFO:tasks.workunit.client.0.smithi032.stdout:0/477: mkdir d3/da/d25/d2b/d4a/da1 0 2024-04-06T14:17:09.199 INFO:tasks.workunit.client.0.smithi032.stdout:4/700: getdents d0/df/d2b/d46/d58 0 2024-04-06T14:17:09.203 INFO:tasks.workunit.client.0.smithi032.stdout:4/701: chown d0/df/c1a 335 1 2024-04-06T14:17:09.207 INFO:tasks.workunit.client.0.smithi032.stdout:5/608: dwrite d16/d18/d33/d35/d4d/f9c [0,4194304] 0 2024-04-06T14:17:09.209 INFO:tasks.workunit.client.1.smithi086.stdout:0/497: rmdir d0/d1/d1b/d46 39 2024-04-06T14:17:09.209 INFO:tasks.workunit.client.1.smithi086.stdout:0/498: chown d0/ff 1618 1 2024-04-06T14:17:09.209 INFO:tasks.workunit.client.1.smithi086.stdout:3/508: link d5/d36/f67 d5/d36/d5a/d6d/d72/d58/d89/f9b 0 2024-04-06T14:17:09.214 INFO:tasks.workunit.client.0.smithi032.stdout:6/648: write d4/d5/dc5/f50 [2276929,69028] 0 2024-04-06T14:17:09.214 INFO:tasks.workunit.client.0.smithi032.stdout:1/725: creat d0/d9/d1f/d7c/da9/fe8 x:0 0 0 2024-04-06T14:17:09.214 INFO:tasks.workunit.client.0.smithi032.stdout:3/647: symlink d3/d16/ddd/df7/lf9 0 2024-04-06T14:17:09.214 INFO:tasks.workunit.client.0.smithi032.stdout:9/662: chown d2/d15/c2c 9563330 1 2024-04-06T14:17:09.214 INFO:tasks.workunit.client.0.smithi032.stdout:4/702: symlink d0/df/d2b/d66/led 0 2024-04-06T14:17:09.216 INFO:tasks.workunit.client.1.smithi086.stdout:0/499: dread d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:09.217 INFO:tasks.workunit.client.0.smithi032.stdout:2/633: creat d0/d30/d39/d59/d67/d74/d78/fce x:0 0 0 2024-04-06T14:17:09.219 INFO:tasks.workunit.client.0.smithi032.stdout:0/478: dwrite d3/da/d25/d2b/d4a/f65 [4194304,4194304] 0 2024-04-06T14:17:09.219 INFO:tasks.workunit.client.0.smithi032.stdout:1/726: creat d0/d41/d59/fe9 x:0 0 0 2024-04-06T14:17:09.222 INFO:tasks.workunit.client.0.smithi032.stdout:5/609: creat d16/d18/d33/d45/d52/d39/fd5 x:0 0 0 2024-04-06T14:17:09.222 INFO:tasks.workunit.client.0.smithi032.stdout:7/674: link d16/d20/d54/de4/d2d/d30/dbb/ld5 d16/d20/d63/lf0 0 2024-04-06T14:17:09.225 INFO:tasks.workunit.client.1.smithi086.stdout:1/496: getdents db/d11/d1a/d79 0 2024-04-06T14:17:09.225 INFO:tasks.workunit.client.1.smithi086.stdout:1/497: readlink db/d11/d1a/d2a/d52/l7d 0 2024-04-06T14:17:09.226 INFO:tasks.workunit.client.1.smithi086.stdout:7/427: creat d7/d10/d11/d1c/d24/d2c/d3c/d7a/f93 x:0 0 0 2024-04-06T14:17:09.227 INFO:tasks.workunit.client.1.smithi086.stdout:6/489: mknod d6/d13/d5f/d60/d45/d4a/d62/cad 0 2024-04-06T14:17:09.227 INFO:tasks.workunit.client.1.smithi086.stdout:7/428: chown d7/d10/d11/d1c/d24/d4d 57219 1 2024-04-06T14:17:09.230 INFO:tasks.workunit.client.1.smithi086.stdout:5/395: getdents d0 0 2024-04-06T14:17:09.231 INFO:tasks.workunit.client.1.smithi086.stdout:3/509: symlink d5/d36/d5a/d6d/l9c 0 2024-04-06T14:17:09.232 INFO:tasks.workunit.client.1.smithi086.stdout:0/500: mknod d0/d1/d2/d1a/d79/c9b 0 2024-04-06T14:17:09.235 INFO:tasks.workunit.client.1.smithi086.stdout:5/396: dread d0/d16/f39 [0,4194304] 0 2024-04-06T14:17:09.239 INFO:tasks.workunit.client.1.smithi086.stdout:8/555: truncate d0/d71/d22/d43/d6d/f80 2010633 0 2024-04-06T14:17:09.242 INFO:tasks.workunit.client.1.smithi086.stdout:3/510: mkdir d5/d36/d56/d9d 0 2024-04-06T14:17:09.243 INFO:tasks.workunit.client.1.smithi086.stdout:3/511: truncate d5/f50 679823 0 2024-04-06T14:17:09.245 INFO:tasks.workunit.client.1.smithi086.stdout:2/490: rename d0/d20/d44/l67 to d0/d20/d27/d2b/d35/d36/d4d/lab 0 2024-04-06T14:17:09.246 INFO:tasks.workunit.client.1.smithi086.stdout:4/361: rename d0/d2c/d38/d54 to d0/d2c/d38/d54/d68 22 2024-04-06T14:17:09.247 INFO:tasks.workunit.client.1.smithi086.stdout:6/490: truncate d6/d13/f27 5570453 0 2024-04-06T14:17:09.247 INFO:tasks.workunit.client.1.smithi086.stdout:4/362: dread - d0/d2c/f4d zero size 2024-04-06T14:17:09.249 INFO:tasks.workunit.client.1.smithi086.stdout:0/501: symlink d0/d15/d26/d33/d7a/l9c 0 2024-04-06T14:17:09.251 INFO:tasks.workunit.client.1.smithi086.stdout:2/491: truncate d0/d20/d27/d2b/d35/f47 1587794 0 2024-04-06T14:17:09.252 INFO:tasks.workunit.client.1.smithi086.stdout:9/407: getdents d5/d13/d1a/d5f 0 2024-04-06T14:17:09.253 INFO:tasks.workunit.client.1.smithi086.stdout:5/397: creat d0/d16/d71/f76 x:0 0 0 2024-04-06T14:17:09.254 INFO:tasks.workunit.client.1.smithi086.stdout:3/512: dread d5/d33/f47 [0,4194304] 0 2024-04-06T14:17:09.255 INFO:tasks.workunit.client.1.smithi086.stdout:8/556: dread d0/d71/d22/f83 [0,4194304] 0 2024-04-06T14:17:09.256 INFO:tasks.workunit.client.1.smithi086.stdout:0/502: stat d0/d15/d26/d33/d7a 0 2024-04-06T14:17:09.256 INFO:tasks.workunit.client.1.smithi086.stdout:4/363: symlink d0/d2c/d30/d24/l69 0 2024-04-06T14:17:09.261 INFO:tasks.workunit.client.1.smithi086.stdout:7/429: rename d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/d87 to d7/d94 0 2024-04-06T14:17:09.262 INFO:tasks.workunit.client.1.smithi086.stdout:5/398: unlink d0/d16/d18/d2a/d50/d66/c68 0 2024-04-06T14:17:09.262 INFO:tasks.workunit.client.1.smithi086.stdout:0/503: fdatasync d0/d1/d4/d7/f11 0 2024-04-06T14:17:09.263 INFO:tasks.workunit.client.1.smithi086.stdout:1/498: dread db/d11/d1a/d2a/d9a/d68/d6e/d81/f99 [0,4194304] 0 2024-04-06T14:17:09.263 INFO:tasks.workunit.client.1.smithi086.stdout:9/408: write d5/d13/d1a/d5f/d88/d14/f86 [98347,19884] 0 2024-04-06T14:17:09.264 INFO:tasks.workunit.client.1.smithi086.stdout:6/491: dwrite d6/d13/fa4 [0,4194304] 0 2024-04-06T14:17:09.265 INFO:tasks.workunit.client.1.smithi086.stdout:7/430: chown d7/d10/d11/d1c/d24 3 1 2024-04-06T14:17:09.266 INFO:tasks.workunit.client.1.smithi086.stdout:4/364: truncate d0/d15/d20/f57 1298333 0 2024-04-06T14:17:09.266 INFO:tasks.workunit.client.1.smithi086.stdout:2/492: creat d0/df/d48/d53/d76/d78/fac x:0 0 0 2024-04-06T14:17:09.267 INFO:tasks.workunit.client.1.smithi086.stdout:8/557: creat d0/d71/d22/d65/d6e/d98/fc2 x:0 0 0 2024-04-06T14:17:09.269 INFO:tasks.workunit.client.1.smithi086.stdout:0/504: fsync d0/d1/d2/f67 0 2024-04-06T14:17:09.269 INFO:tasks.workunit.client.1.smithi086.stdout:1/499: fdatasync db/d13/f92 0 2024-04-06T14:17:09.271 INFO:tasks.workunit.client.1.smithi086.stdout:8/558: write d0/f72 [629123,34151] 0 2024-04-06T14:17:09.272 INFO:tasks.workunit.client.1.smithi086.stdout:7/431: creat d7/d10/d11/d1c/d24/d2c/d3c/d7a/f95 x:0 0 0 2024-04-06T14:17:09.273 INFO:tasks.workunit.client.1.smithi086.stdout:5/399: creat d0/d16/d18/d21/d23/f77 x:0 0 0 2024-04-06T14:17:09.273 INFO:tasks.workunit.client.1.smithi086.stdout:9/409: mknod d5/d13/d1a/d5f/d88/d14/c89 0 2024-04-06T14:17:09.274 INFO:tasks.workunit.client.1.smithi086.stdout:6/492: mknod d6/d13/d5f/d60/d45/d4a/d62/cae 0 2024-04-06T14:17:09.275 INFO:tasks.workunit.client.1.smithi086.stdout:2/493: creat d0/d20/d27/d2b/d35/d36/d4d/d59/fad x:0 0 0 2024-04-06T14:17:09.277 INFO:tasks.workunit.client.1.smithi086.stdout:1/500: write db/fe [2414276,49694] 0 2024-04-06T14:17:09.277 INFO:tasks.workunit.client.1.smithi086.stdout:5/400: read - d0/d16/d18/d2a/d50/f73 zero size 2024-04-06T14:17:09.279 INFO:tasks.workunit.client.1.smithi086.stdout:1/501: chown db 45543835 1 2024-04-06T14:17:09.280 INFO:tasks.workunit.client.1.smithi086.stdout:9/410: truncate f0 1015184 0 2024-04-06T14:17:09.281 INFO:tasks.workunit.client.1.smithi086.stdout:2/494: mkdir d0/d20/d27/dae 0 2024-04-06T14:17:09.283 INFO:tasks.workunit.client.1.smithi086.stdout:5/401: symlink d0/d17/l78 0 2024-04-06T14:17:09.284 INFO:tasks.workunit.client.1.smithi086.stdout:5/402: write d0/d16/d71/f76 [342714,123281] 0 2024-04-06T14:17:09.286 INFO:tasks.workunit.client.1.smithi086.stdout:6/493: write d6/d13/d5f/d60/d45/d4a/d62/f89 [839976,108913] 0 2024-04-06T14:17:09.287 INFO:tasks.workunit.client.1.smithi086.stdout:1/502: rmdir db/d11 39 2024-04-06T14:17:09.287 INFO:tasks.workunit.client.1.smithi086.stdout:6/494: dread - d6/d9/db/f70 zero size 2024-04-06T14:17:09.290 INFO:tasks.workunit.client.1.smithi086.stdout:9/411: symlink d5/d13/d1a/d5f/d88/d14/d19/d25/d74/l8a 0 2024-04-06T14:17:09.293 INFO:tasks.workunit.client.1.smithi086.stdout:7/432: dread d7/d10/d11/d1c/f58 [0,4194304] 0 2024-04-06T14:17:09.295 INFO:tasks.workunit.client.1.smithi086.stdout:0/505: truncate d0/d15/d4e/f5e 1465352 0 2024-04-06T14:17:09.298 INFO:tasks.workunit.client.1.smithi086.stdout:1/503: write db/d11/d1a/d2a/f43 [766392,87591] 0 2024-04-06T14:17:09.298 INFO:tasks.workunit.client.1.smithi086.stdout:5/403: dwrite d0/d16/f47 [0,4194304] 0 2024-04-06T14:17:09.302 INFO:tasks.workunit.client.1.smithi086.stdout:8/559: truncate d0/d13/fb3 2505578 0 2024-04-06T14:17:09.302 INFO:tasks.workunit.client.1.smithi086.stdout:0/506: truncate d0/d1/f76 1315216 0 2024-04-06T14:17:09.303 INFO:tasks.workunit.client.1.smithi086.stdout:7/433: write d7/d10/d11/d1c/d24/d2c/f63 [413533,3343] 0 2024-04-06T14:17:09.305 INFO:tasks.workunit.client.1.smithi086.stdout:2/495: dwrite d0/d20/d44/f46 [0,4194304] 0 2024-04-06T14:17:09.306 INFO:tasks.workunit.client.1.smithi086.stdout:9/412: write f2 [655976,108304] 0 2024-04-06T14:17:09.310 INFO:tasks.workunit.client.1.smithi086.stdout:8/560: mknod d0/d5a/cc3 0 2024-04-06T14:17:09.314 INFO:tasks.workunit.client.1.smithi086.stdout:5/404: creat d0/d17/f79 x:0 0 0 2024-04-06T14:17:09.322 INFO:tasks.workunit.client.1.smithi086.stdout:7/434: dread - d7/d10/d11/d1c/d24/d2c/d3c/d7a/f95 zero size 2024-04-06T14:17:09.322 INFO:tasks.workunit.client.1.smithi086.stdout:6/495: dread d6/d13/d1a/d64/f71 [0,4194304] 0 2024-04-06T14:17:09.322 INFO:tasks.workunit.client.1.smithi086.stdout:0/507: mknod d0/d1/d1b/d44/d8e/d8f/c9d 0 2024-04-06T14:17:09.322 INFO:tasks.workunit.client.1.smithi086.stdout:5/405: fsync d0/d43/f62 0 2024-04-06T14:17:09.323 INFO:tasks.workunit.client.1.smithi086.stdout:2/496: creat d0/d20/d27/d2b/faf x:0 0 0 2024-04-06T14:17:09.324 INFO:tasks.workunit.client.1.smithi086.stdout:6/496: dread - d6/d13/d5f/f95 zero size 2024-04-06T14:17:09.327 INFO:tasks.workunit.client.1.smithi086.stdout:5/406: creat d0/d17/d2d/f7a x:0 0 0 2024-04-06T14:17:09.327 INFO:tasks.workunit.client.1.smithi086.stdout:1/504: dwrite db/f7b [0,4194304] 0 2024-04-06T14:17:09.328 INFO:tasks.workunit.client.1.smithi086.stdout:0/508: symlink d0/d1/d1b/d44/d8e/d8f/l9e 0 2024-04-06T14:17:09.332 INFO:tasks.workunit.client.1.smithi086.stdout:8/561: dwrite d0/d71/d16/d39/d90/fa0 [0,4194304] 0 2024-04-06T14:17:09.332 INFO:tasks.workunit.client.1.smithi086.stdout:1/505: stat db/d11/f12 0 2024-04-06T14:17:09.333 INFO:tasks.workunit.client.1.smithi086.stdout:7/435: rename d7/d10/l20 to d7/d10/d11/d1c/l96 0 2024-04-06T14:17:09.337 INFO:tasks.workunit.client.1.smithi086.stdout:0/509: mkdir d0/d15/d26/d33/d9f 0 2024-04-06T14:17:09.345 INFO:tasks.workunit.client.1.smithi086.stdout:2/497: dwrite d0/df/d48/d53/d76/f91 [0,4194304] 0 2024-04-06T14:17:09.348 INFO:tasks.workunit.client.1.smithi086.stdout:7/436: rmdir d7/d10/d11/d1c/d24/d4d/d5c/d66 39 2024-04-06T14:17:09.352 INFO:tasks.workunit.client.1.smithi086.stdout:8/562: creat d0/d71/d22/d43/d4d/fc4 x:0 0 0 2024-04-06T14:17:09.354 INFO:tasks.workunit.client.1.smithi086.stdout:1/506: dwrite db/f7b [0,4194304] 0 2024-04-06T14:17:09.355 INFO:tasks.workunit.client.1.smithi086.stdout:2/498: symlink d0/df/d5e/d9d/lb0 0 2024-04-06T14:17:09.357 INFO:tasks.workunit.client.1.smithi086.stdout:5/407: getdents d0/d16/d18 0 2024-04-06T14:17:09.358 INFO:tasks.workunit.client.1.smithi086.stdout:0/510: dwrite d0/d1/d2/d1a/d51/f60 [0,4194304] 0 2024-04-06T14:17:09.359 INFO:tasks.workunit.client.1.smithi086.stdout:6/497: dwrite d6/d13/f77 [4194304,4194304] 0 2024-04-06T14:17:09.359 INFO:tasks.workunit.client.1.smithi086.stdout:8/563: unlink d0/d13/c4f 0 2024-04-06T14:17:09.364 INFO:tasks.workunit.client.1.smithi086.stdout:5/408: creat d0/d16/d18/d2a/d50/f7b x:0 0 0 2024-04-06T14:17:09.366 INFO:tasks.workunit.client.1.smithi086.stdout:2/499: mkdir d0/d20/d27/d2b/d35/d36/d4d/d59/db1 0 2024-04-06T14:17:09.368 INFO:tasks.workunit.client.1.smithi086.stdout:1/507: truncate db/d11/f17 847712 0 2024-04-06T14:17:09.369 INFO:tasks.workunit.client.1.smithi086.stdout:0/511: creat d0/d15/d26/fa0 x:0 0 0 2024-04-06T14:17:09.371 INFO:tasks.workunit.client.1.smithi086.stdout:0/512: chown d0/d1/d4/d7/l36 304 1 2024-04-06T14:17:09.372 INFO:tasks.workunit.client.1.smithi086.stdout:0/513: stat d0/d1/d2/d5b/c93 0 2024-04-06T14:17:09.373 INFO:tasks.workunit.client.1.smithi086.stdout:8/564: truncate d0/f2 1775713 0 2024-04-06T14:17:09.375 INFO:tasks.workunit.client.1.smithi086.stdout:2/500: symlink d0/d20/d27/dae/lb2 0 2024-04-06T14:17:09.376 INFO:tasks.workunit.client.1.smithi086.stdout:5/409: mkdir d0/d17/d2d/d52/d7c 0 2024-04-06T14:17:09.376 INFO:tasks.workunit.client.1.smithi086.stdout:7/437: link d7/d1d/l14 d7/d10/d11/d1c/d24/d2c/d3c/l97 0 2024-04-06T14:17:09.377 INFO:tasks.workunit.client.1.smithi086.stdout:0/514: rmdir d0/d1/d1b/d44/d5d 39 2024-04-06T14:17:09.379 INFO:tasks.workunit.client.1.smithi086.stdout:6/498: rename d6/d13/d42/d52/f7f to d6/d13/faf 0 2024-04-06T14:17:09.379 INFO:tasks.workunit.client.1.smithi086.stdout:7/438: write d7/d10/d11/d1c/d24/d2c/f39 [1718396,72893] 0 2024-04-06T14:17:09.380 INFO:tasks.workunit.client.1.smithi086.stdout:8/565: unlink d0/d71/d22/d43/d4d/l85 0 2024-04-06T14:17:09.382 INFO:tasks.workunit.client.1.smithi086.stdout:8/566: stat d0/d71/d22/d43/d4d/fc4 0 2024-04-06T14:17:09.382 INFO:tasks.workunit.client.1.smithi086.stdout:2/501: read d0/d20/d27/d2b/d35/d36/f3b [3003351,87539] 0 2024-04-06T14:17:09.386 INFO:tasks.workunit.client.1.smithi086.stdout:5/410: creat d0/d16/d18/d2a/d50/d66/f7d x:0 0 0 2024-04-06T14:17:09.388 INFO:tasks.workunit.client.1.smithi086.stdout:0/515: symlink d0/d1/d2/d1a/la1 0 2024-04-06T14:17:09.389 INFO:tasks.workunit.client.1.smithi086.stdout:7/439: dwrite d7/d10/d11/d1c/d24/d4d/f62 [0,4194304] 0 2024-04-06T14:17:09.395 INFO:tasks.workunit.client.1.smithi086.stdout:8/567: creat d0/d3b/d79/fc5 x:0 0 0 2024-04-06T14:17:09.396 INFO:tasks.workunit.client.1.smithi086.stdout:1/508: truncate db/d11/d1a/d2a/f9c 4167891 0 2024-04-06T14:17:09.396 INFO:tasks.workunit.client.1.smithi086.stdout:8/568: write d0/f72 [1051831,109686] 0 2024-04-06T14:17:09.397 INFO:tasks.workunit.client.1.smithi086.stdout:2/502: creat d0/df/fb3 x:0 0 0 2024-04-06T14:17:09.401 INFO:tasks.workunit.client.1.smithi086.stdout:5/411: unlink d0/fa 0 2024-04-06T14:17:09.404 INFO:tasks.workunit.client.1.smithi086.stdout:0/516: dread - d0/d1/d1b/d73/f7c zero size 2024-04-06T14:17:09.406 INFO:tasks.workunit.client.1.smithi086.stdout:1/509: dwrite db/f10 [0,4194304] 0 2024-04-06T14:17:09.413 INFO:tasks.workunit.client.0.smithi032.stdout:5/610: sync 2024-04-06T14:17:09.413 INFO:tasks.workunit.client.1.smithi086.stdout:6/499: link d6/d13/d42/f72 d6/d13/fb0 0 2024-04-06T14:17:09.416 INFO:tasks.workunit.client.1.smithi086.stdout:0/517: mknod d0/d1/d2/d1a/ca2 0 2024-04-06T14:17:09.417 INFO:tasks.workunit.client.1.smithi086.stdout:2/503: symlink d0/d20/d80/lb4 0 2024-04-06T14:17:09.419 INFO:tasks.workunit.client.1.smithi086.stdout:8/569: dread d0/d71/d16/d39/f66 [0,4194304] 0 2024-04-06T14:17:09.420 INFO:tasks.workunit.client.0.smithi032.stdout:5/611: dwrite d16/d18/d33/f8a [0,4194304] 0 2024-04-06T14:17:09.420 INFO:tasks.workunit.client.1.smithi086.stdout:1/510: creat db/d11/d1a/d79/f9d x:0 0 0 2024-04-06T14:17:09.424 INFO:tasks.workunit.client.1.smithi086.stdout:6/500: creat d6/d20/fb1 x:0 0 0 2024-04-06T14:17:09.426 INFO:tasks.workunit.client.1.smithi086.stdout:0/518: unlink d0/d1/d25/f70 0 2024-04-06T14:17:09.426 INFO:tasks.workunit.client.1.smithi086.stdout:1/511: dread db/f7b [0,4194304] 0 2024-04-06T14:17:09.430 INFO:tasks.workunit.client.1.smithi086.stdout:0/519: chown d0/d15/f8d 1059 1 2024-04-06T14:17:09.430 INFO:tasks.workunit.client.1.smithi086.stdout:7/440: getdents d7/d10/d11/d1c/d24/d4d 0 2024-04-06T14:17:09.430 INFO:tasks.workunit.client.1.smithi086.stdout:0/520: dread - d0/d1/d2/d1a/d51/f86 zero size 2024-04-06T14:17:09.434 INFO:tasks.workunit.client.1.smithi086.stdout:8/570: getdents d0/d71/d16/d39/d90/daf 0 2024-04-06T14:17:09.436 INFO:tasks.workunit.client.1.smithi086.stdout:1/512: creat db/d11/d1a/d2a/d9a/d68/d6e/f9e x:0 0 0 2024-04-06T14:17:09.440 INFO:tasks.workunit.client.1.smithi086.stdout:8/571: chown d0/l25 13147 1 2024-04-06T14:17:09.441 INFO:tasks.workunit.client.1.smithi086.stdout:2/504: truncate d0/d20/d27/d37/f83 4115423 0 2024-04-06T14:17:09.441 INFO:tasks.workunit.client.1.smithi086.stdout:8/572: fdatasync d0/db/f12 0 2024-04-06T14:17:09.445 INFO:tasks.workunit.client.1.smithi086.stdout:6/501: dwrite d6/d13/fb0 [0,4194304] 0 2024-04-06T14:17:09.446 INFO:tasks.workunit.client.1.smithi086.stdout:2/505: write d0/d20/d27/d2b/d35/d36/f96 [71043,5564] 0 2024-04-06T14:17:09.451 INFO:tasks.workunit.client.1.smithi086.stdout:8/573: chown d0/d13/l1f 989905 1 2024-04-06T14:17:09.452 INFO:tasks.workunit.client.1.smithi086.stdout:5/412: link d0/d17/d2d/l36 d0/d5e/l7e 0 2024-04-06T14:17:09.455 INFO:tasks.workunit.client.1.smithi086.stdout:2/506: symlink d0/d20/d82/lb5 0 2024-04-06T14:17:09.457 INFO:tasks.workunit.client.1.smithi086.stdout:5/413: readlink d0/d17/d2d/l32 0 2024-04-06T14:17:09.462 INFO:tasks.workunit.client.1.smithi086.stdout:0/521: dread d0/d15/f3e [0,4194304] 0 2024-04-06T14:17:09.464 INFO:tasks.workunit.client.1.smithi086.stdout:8/574: rename d0/d71/d16/d95 to d0/d3b/d79/dc6 0 2024-04-06T14:17:09.465 INFO:tasks.workunit.client.1.smithi086.stdout:7/441: dwrite d7/d10/d11/d1c/d36/f74 [0,4194304] 0 2024-04-06T14:17:09.467 INFO:tasks.workunit.client.1.smithi086.stdout:5/414: chown d0/d16/d18/d2a/d50/d66/f7d 51382 1 2024-04-06T14:17:09.471 INFO:tasks.workunit.client.1.smithi086.stdout:1/513: dwrite fa [4194304,4194304] 0 2024-04-06T14:17:09.474 INFO:tasks.workunit.client.1.smithi086.stdout:6/502: dread d6/d13/d5f/d60/d45/d4a/d62/f6a [0,4194304] 0 2024-04-06T14:17:09.476 INFO:tasks.workunit.client.1.smithi086.stdout:0/522: rename d0/d1/d1b to d0/d15/d4e/da3 0 2024-04-06T14:17:09.479 INFO:tasks.workunit.client.1.smithi086.stdout:1/514: dread db/d11/d1a/d2a/d9a/d68/d6e/f8d [0,4194304] 0 2024-04-06T14:17:09.479 INFO:tasks.workunit.client.1.smithi086.stdout:2/507: dwrite d0/d20/d44/f5b [4194304,4194304] 0 2024-04-06T14:17:09.482 INFO:tasks.workunit.client.1.smithi086.stdout:5/415: creat d0/d16/d18/f7f x:0 0 0 2024-04-06T14:17:09.487 INFO:tasks.workunit.client.1.smithi086.stdout:7/442: creat d7/d10/d11/d1c/d24/d2c/d3c/f98 x:0 0 0 2024-04-06T14:17:09.489 INFO:tasks.workunit.client.1.smithi086.stdout:0/523: rename d0/d1/d2/d1a/la1 to d0/d15/d4e/da3/la4 0 2024-04-06T14:17:09.492 INFO:tasks.workunit.client.1.smithi086.stdout:8/575: dread d0/d13/f4b [0,4194304] 0 2024-04-06T14:17:09.494 INFO:tasks.workunit.client.1.smithi086.stdout:6/503: rename d6/d13/d1a/d64/d85/d88 to d6/d13/d5f/d60/d45/d73/db2 0 2024-04-06T14:17:09.495 INFO:tasks.workunit.client.1.smithi086.stdout:6/504: dread - d6/d13/d5f/f95 zero size 2024-04-06T14:17:09.500 INFO:tasks.workunit.client.1.smithi086.stdout:1/515: truncate db/d11/d1a/d79/f89 479586 0 2024-04-06T14:17:09.500 INFO:tasks.workunit.client.1.smithi086.stdout:2/508: dread d0/f1e [0,4194304] 0 2024-04-06T14:17:09.501 INFO:tasks.workunit.client.1.smithi086.stdout:0/524: creat d0/d1/fa5 x:0 0 0 2024-04-06T14:17:09.502 INFO:tasks.workunit.client.1.smithi086.stdout:2/509: chown d0/d20/d82/lb5 77 1 2024-04-06T14:17:09.502 INFO:tasks.workunit.client.1.smithi086.stdout:0/525: dread - d0/d1/d2/f67 zero size 2024-04-06T14:17:09.503 INFO:tasks.workunit.client.1.smithi086.stdout:0/526: chown d0/d1/d2/d1a/d79 1004 1 2024-04-06T14:17:09.504 INFO:tasks.workunit.client.1.smithi086.stdout:0/527: write d0/d1/d4/d7/f11 [313385,101903] 0 2024-04-06T14:17:09.505 INFO:tasks.workunit.client.1.smithi086.stdout:6/505: rename d6/d13/d5f/d60/d45/d4a/d62/f6a to d6/d13/d1a/d64/fb3 0 2024-04-06T14:17:09.507 INFO:tasks.workunit.client.1.smithi086.stdout:8/576: mkdir d0/d71/d22/d43/d4d/da3/dc7 0 2024-04-06T14:17:09.511 INFO:tasks.workunit.client.1.smithi086.stdout:7/443: truncate d7/d1d/fc 2406944 0 2024-04-06T14:17:09.514 INFO:tasks.workunit.client.1.smithi086.stdout:6/506: symlink d6/d9/d43/lb4 0 2024-04-06T14:17:09.517 INFO:tasks.workunit.client.1.smithi086.stdout:1/516: link db/d13/d71/f88 db/d11/d1a/d2a/d9a/d78/f9f 0 2024-04-06T14:17:09.519 INFO:tasks.workunit.client.1.smithi086.stdout:0/528: mknod d0/d15/d4e/da3/d44/d5d/d74/ca6 0 2024-04-06T14:17:09.521 INFO:tasks.workunit.client.1.smithi086.stdout:7/444: symlink d7/d94/l99 0 2024-04-06T14:17:09.522 INFO:tasks.workunit.client.0.smithi032.stdout:8/673: getdents d3/d8/d3a 0 2024-04-06T14:17:09.526 INFO:tasks.workunit.client.1.smithi086.stdout:6/507: rename d6/d9/f8c to d6/d13/d5f/fb5 0 2024-04-06T14:17:09.526 INFO:tasks.workunit.client.1.smithi086.stdout:2/510: truncate d0/f2a 413360 0 2024-04-06T14:17:09.527 INFO:tasks.workunit.client.0.smithi032.stdout:3/648: link d3/fac d3/d16/ddd/ffa 0 2024-04-06T14:17:09.528 INFO:tasks.workunit.client.1.smithi086.stdout:5/416: getdents d0/d16/d18/d35 0 2024-04-06T14:17:09.532 INFO:tasks.workunit.client.1.smithi086.stdout:0/529: creat d0/d1/d25/fa7 x:0 0 0 2024-04-06T14:17:09.532 INFO:tasks.workunit.client.1.smithi086.stdout:8/577: creat d0/d71/d16/d52/fc8 x:0 0 0 2024-04-06T14:17:09.532 INFO:tasks.workunit.client.0.smithi032.stdout:6/649: symlink d4/d5/d28/le2 0 2024-04-06T14:17:09.532 INFO:tasks.workunit.client.0.smithi032.stdout:6/650: truncate d4/d5/d35/fce 826374 0 2024-04-06T14:17:09.533 INFO:tasks.workunit.client.0.smithi032.stdout:0/479: creat d3/da/d25/d2b/d4a/fa2 x:0 0 0 2024-04-06T14:17:09.534 INFO:tasks.workunit.client.1.smithi086.stdout:7/445: creat d7/d10/d11/d1c/d24/d4d/f9a x:0 0 0 2024-04-06T14:17:09.534 INFO:tasks.workunit.client.1.smithi086.stdout:7/446: fdatasync d7/f81 0 2024-04-06T14:17:09.535 INFO:tasks.workunit.client.0.smithi032.stdout:4/703: mknod d0/df/d23/d31/d50/d57/d5b/d9c/cee 0 2024-04-06T14:17:09.536 INFO:tasks.workunit.client.1.smithi086.stdout:6/508: dread d6/d9/f17 [0,4194304] 0 2024-04-06T14:17:09.536 INFO:tasks.workunit.client.1.smithi086.stdout:1/517: symlink db/d11/d1f/la0 0 2024-04-06T14:17:09.537 INFO:tasks.workunit.client.0.smithi032.stdout:4/704: dread - d0/df/d23/d31/d50/d57/d5b/fc9 zero size 2024-04-06T14:17:09.537 INFO:tasks.workunit.client.1.smithi086.stdout:5/417: mkdir d0/d17/d2d/d80 0 2024-04-06T14:17:09.539 INFO:tasks.workunit.client.0.smithi032.stdout:9/663: write d2/d15/d1e/d65/d83/f94 [151727,36680] 0 2024-04-06T14:17:09.540 INFO:tasks.workunit.client.1.smithi086.stdout:8/578: mknod d0/d71/d82/cc9 0 2024-04-06T14:17:09.541 INFO:tasks.workunit.client.0.smithi032.stdout:9/664: chown d2/d15/d1e/d41/d6c/d74 5607703 1 2024-04-06T14:17:09.545 INFO:tasks.workunit.client.1.smithi086.stdout:6/509: fsync d6/d13/d1a/d64/fb3 0 2024-04-06T14:17:09.547 INFO:tasks.workunit.client.1.smithi086.stdout:6/510: fdatasync d6/d13/d1a/d22/f35 0 2024-04-06T14:17:09.547 INFO:tasks.workunit.client.1.smithi086.stdout:5/418: symlink d0/d5e/l81 0 2024-04-06T14:17:09.548 INFO:tasks.workunit.client.1.smithi086.stdout:7/447: mknod d7/d10/d11/d1c/c9b 0 2024-04-06T14:17:09.548 INFO:tasks.workunit.client.1.smithi086.stdout:2/511: write d0/d20/d27/d2b/d35/d36/f3b [3620222,114592] 0 2024-04-06T14:17:09.549 INFO:tasks.workunit.client.1.smithi086.stdout:6/511: fdatasync d6/d13/d5f/d60/d45/d4a/f79 0 2024-04-06T14:17:09.549 INFO:tasks.workunit.client.1.smithi086.stdout:8/579: truncate d0/d3b/f3d 2312393 0 2024-04-06T14:17:09.553 INFO:tasks.workunit.client.1.smithi086.stdout:0/530: link d0/d15/l1e d0/d15/d26/la8 0 2024-04-06T14:17:09.555 INFO:tasks.workunit.client.1.smithi086.stdout:5/419: chown d0/d16/d18/d21/c69 360 1 2024-04-06T14:17:09.557 INFO:tasks.workunit.client.1.smithi086.stdout:8/580: unlink d0/d71/d22/f61 0 2024-04-06T14:17:09.560 INFO:tasks.workunit.client.1.smithi086.stdout:6/512: symlink d6/d13/d1a/d64/d85/lb6 0 2024-04-06T14:17:09.560 INFO:tasks.workunit.client.1.smithi086.stdout:2/512: write d0/d20/d27/d2b/d35/f4e [665032,127682] 0 2024-04-06T14:17:09.560 INFO:tasks.workunit.client.1.smithi086.stdout:8/581: write d0/d71/d22/d43/d4d/fc4 [380944,103386] 0 2024-04-06T14:17:09.561 INFO:tasks.workunit.client.1.smithi086.stdout:1/518: truncate db/d11/d24/f45 1465013 0 2024-04-06T14:17:09.563 INFO:tasks.workunit.client.1.smithi086.stdout:8/582: stat d0/d71/d22/d43/daa 0 2024-04-06T14:17:09.564 INFO:tasks.workunit.client.1.smithi086.stdout:6/513: dread - d6/d9/fab zero size 2024-04-06T14:17:09.566 INFO:tasks.workunit.client.1.smithi086.stdout:7/448: mknod d7/d10/d11/d1c/d24/d4d/d5c/d66/c9c 0 2024-04-06T14:17:09.568 INFO:tasks.workunit.client.1.smithi086.stdout:5/420: fsync d0/d16/d18/d2a/d50/f5f 0 2024-04-06T14:17:09.573 INFO:tasks.workunit.client.1.smithi086.stdout:8/583: creat d0/d71/d22/d43/d6d/d75/db2/fca x:0 0 0 2024-04-06T14:17:09.578 INFO:tasks.workunit.client.0.smithi032.stdout:1/727: dwrite d0/da/f69 [0,4194304] 0 2024-04-06T14:17:09.581 INFO:tasks.workunit.client.1.smithi086.stdout:0/531: dwrite d0/d15/f3e [0,4194304] 0 2024-04-06T14:17:09.583 INFO:tasks.workunit.client.1.smithi086.stdout:2/513: dwrite d0/d20/d27/d2b/d35/d36/fa1 [0,4194304] 0 2024-04-06T14:17:09.590 INFO:tasks.workunit.client.1.smithi086.stdout:6/514: dwrite d6/d9/fab [0,4194304] 0 2024-04-06T14:17:09.592 INFO:tasks.workunit.client.1.smithi086.stdout:6/515: write d6/d9/d5c/f6b [859252,71439] 0 2024-04-06T14:17:09.595 INFO:tasks.workunit.client.0.smithi032.stdout:8/674: chown d3/d8/d7/fbc 9200047 1 2024-04-06T14:17:09.597 INFO:tasks.workunit.client.0.smithi032.stdout:6/651: creat d4/d5/dd8/fe3 x:0 0 0 2024-04-06T14:17:09.605 INFO:tasks.workunit.client.0.smithi032.stdout:5/612: creat d16/d18/d33/d45/fd6 x:0 0 0 2024-04-06T14:17:09.605 INFO:tasks.workunit.client.0.smithi032.stdout:9/665: mknod d2/d15/d1e/d65/d83/d31/cd7 0 2024-04-06T14:17:09.609 INFO:tasks.workunit.client.0.smithi032.stdout:3/649: getdents d3/d16 0 2024-04-06T14:17:09.609 INFO:tasks.workunit.client.0.smithi032.stdout:8/675: link d3/d8/d12/d2f/d4a/d60/d75/d85/fd4 d3/d8/ff1 0 2024-04-06T14:17:09.610 INFO:tasks.workunit.client.0.smithi032.stdout:9/666: read d2/d15/d55/d59/d93/d95/db8/d81/f4c [470255,37778] 0 2024-04-06T14:17:09.614 INFO:tasks.workunit.client.0.smithi032.stdout:9/667: read d2/d15/d1e/f33 [419303,27162] 0 2024-04-06T14:17:09.614 INFO:tasks.workunit.client.0.smithi032.stdout:3/650: creat d3/d65/d7d/d9c/dad/db1/d3c/dc9/ffb x:0 0 0 2024-04-06T14:17:09.616 INFO:tasks.workunit.client.0.smithi032.stdout:9/668: truncate d2/fb9 822168 0 2024-04-06T14:17:09.617 INFO:tasks.workunit.client.0.smithi032.stdout:1/728: dread d0/da/d3e/f6a [0,4194304] 0 2024-04-06T14:17:09.618 INFO:tasks.workunit.client.0.smithi032.stdout:9/669: mkdir d2/dcd/dd8 0 2024-04-06T14:17:09.622 INFO:tasks.workunit.client.0.smithi032.stdout:9/670: mkdir d2/d15/dd9 0 2024-04-06T14:17:09.625 INFO:tasks.workunit.client.0.smithi032.stdout:1/729: link d0/da/d3e/c5e d0/da/d3e/d75/dd7/cea 0 2024-04-06T14:17:09.625 INFO:tasks.workunit.client.0.smithi032.stdout:3/651: dwrite d3/d65/d7d/d9c/dad/db1/d3c/dc9/fee [0,4194304] 0 2024-04-06T14:17:09.627 INFO:tasks.workunit.client.0.smithi032.stdout:9/671: write d2/d15/d1e/d65/d6b/f9b [459530,79519] 0 2024-04-06T14:17:09.627 INFO:tasks.workunit.client.0.smithi032.stdout:1/730: chown d0/d9/d1f/d7c 1979 1 2024-04-06T14:17:09.632 INFO:tasks.workunit.client.0.smithi032.stdout:5/613: dread d16/f2c [0,4194304] 0 2024-04-06T14:17:09.632 INFO:tasks.workunit.client.0.smithi032.stdout:1/731: creat d0/da/d3e/d75/dd7/feb x:0 0 0 2024-04-06T14:17:09.635 INFO:tasks.workunit.client.0.smithi032.stdout:3/652: rename d3/d65/d7d/d9c/dad/db1/d3c/lf6 to d3/d65/d7d/d9c/dad/lfc 0 2024-04-06T14:17:09.640 INFO:tasks.workunit.client.0.smithi032.stdout:9/672: link d2/d15/d1e/d65/d83/d31/d39/l6a d2/d15/d1e/dd3/lda 0 2024-04-06T14:17:09.643 INFO:tasks.workunit.client.0.smithi032.stdout:3/653: dwrite d3/d65/f34 [0,4194304] 0 2024-04-06T14:17:09.643 INFO:tasks.workunit.client.0.smithi032.stdout:2/634: truncate d0/d5a/d3a/da2/fc1 2408375 0 2024-04-06T14:17:09.646 INFO:tasks.workunit.client.0.smithi032.stdout:7/675: write d16/d20/d54/de4/d70/fe7 [67792,88277] 0 2024-04-06T14:17:09.647 INFO:tasks.workunit.client.0.smithi032.stdout:6/652: truncate d4/d5/dbf/f37 1054702 0 2024-04-06T14:17:09.647 INFO:tasks.workunit.client.0.smithi032.stdout:4/705: fsync d0/df/d23/d31/f68 0 2024-04-06T14:17:09.648 INFO:tasks.workunit.client.0.smithi032.stdout:3/654: mknod d3/d16/ddd/df7/cfd 0 2024-04-06T14:17:09.652 INFO:tasks.workunit.client.0.smithi032.stdout:8/676: truncate d3/d8/d3a/f4c 2258818 0 2024-04-06T14:17:09.653 INFO:tasks.workunit.client.0.smithi032.stdout:2/635: mknod d0/d6e/dae/ccf 0 2024-04-06T14:17:09.658 INFO:tasks.workunit.client.0.smithi032.stdout:0/480: dwrite d3/da/d25/d2b/f32 [0,4194304] 0 2024-04-06T14:17:09.660 INFO:tasks.workunit.client.0.smithi032.stdout:7/676: chown d16/l1b 90442 1 2024-04-06T14:17:09.666 INFO:tasks.workunit.client.0.smithi032.stdout:9/673: dwrite d2/d15/d1e/d65/d83/d7e/f75 [0,4194304] 0 2024-04-06T14:17:09.677 INFO:tasks.workunit.client.0.smithi032.stdout:5/614: dwrite d16/d18/f2b [0,4194304] 0 2024-04-06T14:17:09.677 INFO:tasks.workunit.client.0.smithi032.stdout:9/674: mknod d2/d15/d55/d59/d93/cdb 0 2024-04-06T14:17:09.681 INFO:tasks.workunit.client.0.smithi032.stdout:7/677: symlink d16/d20/d54/de4/lf1 0 2024-04-06T14:17:09.681 INFO:tasks.workunit.client.0.smithi032.stdout:1/732: dread d0/d10/d16/d3f/f85 [0,4194304] 0 2024-04-06T14:17:09.682 INFO:tasks.workunit.client.0.smithi032.stdout:0/481: symlink d3/da/d35/d8b/d9c/la3 0 2024-04-06T14:17:09.684 INFO:tasks.workunit.client.0.smithi032.stdout:0/482: fdatasync d3/f2e 0 2024-04-06T14:17:09.685 INFO:tasks.workunit.client.0.smithi032.stdout:7/678: mkdir d16/d20/d63/d9d/df2 0 2024-04-06T14:17:09.685 INFO:tasks.workunit.client.0.smithi032.stdout:8/677: getdents d3/d8/d7/d14 0 2024-04-06T14:17:09.687 INFO:tasks.workunit.client.0.smithi032.stdout:1/733: unlink d0/d10/d16/d3f/fb9 0 2024-04-06T14:17:09.687 INFO:tasks.workunit.client.0.smithi032.stdout:3/655: dread d3/d65/d7d/d9c/dad/db1/d3c/f79 [0,4194304] 0 2024-04-06T14:17:09.687 INFO:tasks.workunit.client.0.smithi032.stdout:4/706: truncate d0/d82/d8e/fde 304540 0 2024-04-06T14:17:09.688 INFO:tasks.workunit.client.0.smithi032.stdout:5/615: creat d16/d18/d33/d35/d88/fd7 x:0 0 0 2024-04-06T14:17:09.691 INFO:tasks.workunit.client.0.smithi032.stdout:3/656: readlink d3/lbb 0 2024-04-06T14:17:09.694 INFO:tasks.workunit.client.0.smithi032.stdout:5/616: stat d16/d18/d33/d45/d52/d39/f5b 0 2024-04-06T14:17:09.695 INFO:tasks.workunit.client.0.smithi032.stdout:6/653: dwrite d4/d5/d18/f31 [0,4194304] 0 2024-04-06T14:17:09.701 INFO:tasks.workunit.client.0.smithi032.stdout:4/707: mkdir d0/df/d23/d31/d50/d57/d8a/db5/def 0 2024-04-06T14:17:09.701 INFO:tasks.workunit.client.0.smithi032.stdout:8/678: truncate d3/d8/d12/d2f/d4a/d60/d75/d85/fee 337526 0 2024-04-06T14:17:09.705 INFO:tasks.workunit.client.0.smithi032.stdout:3/657: creat d3/d65/d7d/d9c/dbc/ffe x:0 0 0 2024-04-06T14:17:09.707 INFO:tasks.workunit.client.0.smithi032.stdout:2/636: dread d0/d30/f36 [0,4194304] 0 2024-04-06T14:17:09.707 INFO:tasks.workunit.client.0.smithi032.stdout:7/679: dwrite d16/d20/d54/de4/d2d/d30/f36 [4194304,4194304] 0 2024-04-06T14:17:09.709 INFO:tasks.workunit.client.0.smithi032.stdout:0/483: write d3/da/f23 [1152274,34371] 0 2024-04-06T14:17:09.709 INFO:tasks.workunit.client.0.smithi032.stdout:3/658: write d3/d65/d7d/d9c/dad/fc8 [647656,19075] 0 2024-04-06T14:17:09.710 INFO:tasks.workunit.client.0.smithi032.stdout:4/708: mkdir d0/df/d2b/d66/df0 0 2024-04-06T14:17:09.712 INFO:tasks.workunit.client.0.smithi032.stdout:5/617: unlink d16/d18/d33/d45/d52/d39/d86/lc1 0 2024-04-06T14:17:09.714 INFO:tasks.workunit.client.0.smithi032.stdout:1/734: link d0/d10/d16/l2d d0/d9/d1f/d7c/da9/de1/lec 0 2024-04-06T14:17:09.717 INFO:tasks.workunit.client.0.smithi032.stdout:2/637: unlink d0/l26 0 2024-04-06T14:17:09.719 INFO:tasks.workunit.client.0.smithi032.stdout:6/654: creat d4/d5/d8/fe4 x:0 0 0 2024-04-06T14:17:09.720 INFO:tasks.workunit.client.0.smithi032.stdout:7/680: write d16/d20/d54/de4/d5f/f90 [1394517,82924] 0 2024-04-06T14:17:09.721 INFO:tasks.workunit.client.0.smithi032.stdout:3/659: truncate d3/f3f 2881099 0 2024-04-06T14:17:09.721 INFO:tasks.workunit.client.0.smithi032.stdout:8/679: creat d3/d8/d12/d2f/d4a/d60/d90/ff2 x:0 0 0 2024-04-06T14:17:09.723 INFO:tasks.workunit.client.0.smithi032.stdout:5/618: mknod d16/d18/d33/d35/cd8 0 2024-04-06T14:17:09.724 INFO:tasks.workunit.client.0.smithi032.stdout:4/709: read d0/df/d1f/d42/d45/f75 [74924,1236] 0 2024-04-06T14:17:09.725 INFO:tasks.workunit.client.0.smithi032.stdout:6/655: symlink d4/d5/dbf/d61/da7/dda/le5 0 2024-04-06T14:17:09.731 INFO:tasks.workunit.client.0.smithi032.stdout:7/681: write d16/f27 [1881507,112075] 0 2024-04-06T14:17:09.731 INFO:tasks.workunit.client.0.smithi032.stdout:1/735: dwrite d0/d41/d5b/d96/fd3 [0,4194304] 0 2024-04-06T14:17:09.731 INFO:tasks.workunit.client.0.smithi032.stdout:8/680: dread d3/d69/fad [0,4194304] 0 2024-04-06T14:17:09.734 INFO:tasks.workunit.client.0.smithi032.stdout:6/656: readlink d4/d5/d8/ld 0 2024-04-06T14:17:09.734 INFO:tasks.workunit.client.0.smithi032.stdout:7/682: truncate d16/d20/d7a/fa5 921854 0 2024-04-06T14:17:09.740 INFO:tasks.workunit.client.0.smithi032.stdout:5/619: creat d16/d58/d54/d74/fd9 x:0 0 0 2024-04-06T14:17:09.741 INFO:tasks.workunit.client.0.smithi032.stdout:1/736: symlink d0/da/d3e/d75/dc4/led 0 2024-04-06T14:17:09.741 INFO:tasks.workunit.client.0.smithi032.stdout:3/660: rename d3/d65/d7d/d9c/dad/db1/c86 to d3/d16/cff 0 2024-04-06T14:17:09.742 INFO:tasks.workunit.client.0.smithi032.stdout:6/657: mkdir d4/d5/d19/d96/de6 0 2024-04-06T14:17:09.742 INFO:tasks.workunit.client.1.smithi086.stdout:4/365: sync 2024-04-06T14:17:09.743 INFO:tasks.workunit.client.1.smithi086.stdout:3/513: sync 2024-04-06T14:17:09.744 INFO:tasks.workunit.client.1.smithi086.stdout:4/366: chown d0/d15/d53/f5e 92790946 1 2024-04-06T14:17:09.744 INFO:tasks.workunit.client.0.smithi032.stdout:7/683: dwrite fd [0,4194304] 0 2024-04-06T14:17:09.745 INFO:tasks.workunit.client.0.smithi032.stdout:4/710: dwrite d0/df/d23/d31/f68 [0,4194304] 0 2024-04-06T14:17:09.746 INFO:tasks.workunit.client.0.smithi032.stdout:8/681: rename d3/d8/l2c to d3/d8/d12/d83/d89/da0/daf/lf3 0 2024-04-06T14:17:09.751 INFO:tasks.workunit.client.0.smithi032.stdout:5/620: mknod d16/d18/d33/d35/d4d/cda 0 2024-04-06T14:17:09.755 INFO:tasks.workunit.client.0.smithi032.stdout:1/737: truncate d0/d9/d1f/d7c/da9/fe8 723612 0 2024-04-06T14:17:09.763 INFO:tasks.workunit.client.0.smithi032.stdout:5/621: dread - d16/d18/d33/d45/d52/fa2 zero size 2024-04-06T14:17:09.763 INFO:tasks.workunit.client.0.smithi032.stdout:8/682: mknod d3/d8/d12/d2f/d4a/d60/d75/d85/cf4 0 2024-04-06T14:17:09.763 INFO:tasks.workunit.client.0.smithi032.stdout:4/711: symlink d0/df/d2b/d66/df0/lf1 0 2024-04-06T14:17:09.763 INFO:tasks.workunit.client.0.smithi032.stdout:3/661: dwrite d3/d65/d7d/d9c/dad/db1/f6c [0,4194304] 0 2024-04-06T14:17:09.763 INFO:tasks.workunit.client.0.smithi032.stdout:5/622: write d16/f90 [4235015,65202] 0 2024-04-06T14:17:09.765 INFO:tasks.workunit.client.0.smithi032.stdout:1/738: mkdir d0/da/d3e/d75/dee 0 2024-04-06T14:17:09.765 INFO:tasks.workunit.client.0.smithi032.stdout:8/683: rmdir d3/d8/d7/d14 39 2024-04-06T14:17:09.766 INFO:tasks.workunit.client.0.smithi032.stdout:3/662: write d3/d65/d7d/d9c/db8/fc2 [4521537,96158] 0 2024-04-06T14:17:09.769 INFO:tasks.workunit.client.0.smithi032.stdout:1/739: chown d0/da/fba 7232241 1 2024-04-06T14:17:09.773 INFO:tasks.workunit.client.0.smithi032.stdout:8/684: stat d3/l41 0 2024-04-06T14:17:09.775 INFO:tasks.workunit.client.0.smithi032.stdout:1/740: mkdir d0/d10/d16/d3f/def 0 2024-04-06T14:17:09.775 INFO:tasks.workunit.client.0.smithi032.stdout:3/663: fdatasync d3/d65/f8c 0 2024-04-06T14:17:09.777 INFO:tasks.workunit.client.0.smithi032.stdout:5/623: symlink d16/d18/d33/d45/ldb 0 2024-04-06T14:17:09.779 INFO:tasks.workunit.client.0.smithi032.stdout:4/712: dread d0/df/d23/d31/d50/d57/f64 [0,4194304] 0 2024-04-06T14:17:09.780 INFO:tasks.workunit.client.0.smithi032.stdout:6/658: dwrite d4/d5/d28/d54/f85 [0,4194304] 0 2024-04-06T14:17:09.782 INFO:tasks.workunit.client.0.smithi032.stdout:7/684: dread fd [4194304,4194304] 0 2024-04-06T14:17:09.782 INFO:tasks.workunit.client.0.smithi032.stdout:1/741: mknod d0/d10/d66/cf0 0 2024-04-06T14:17:09.783 INFO:tasks.workunit.client.1.smithi086.stdout:9/413: sync 2024-04-06T14:17:09.791 INFO:tasks.workunit.client.0.smithi032.stdout:1/742: mkdir d0/d9/d94/dc2/df1 0 2024-04-06T14:17:09.791 INFO:tasks.workunit.client.0.smithi032.stdout:6/659: symlink d4/d5/dbf/d61/da7/le7 0 2024-04-06T14:17:09.791 INFO:tasks.workunit.client.0.smithi032.stdout:5/624: mkdir d16/d18/dae/dbb/ddc 0 2024-04-06T14:17:09.794 INFO:tasks.workunit.client.0.smithi032.stdout:1/743: fsync d0/d9/d94/fbf 0 2024-04-06T14:17:09.794 INFO:tasks.workunit.client.0.smithi032.stdout:7/685: fdatasync d16/d20/d54/de4/d2d/f41 0 2024-04-06T14:17:09.795 INFO:tasks.workunit.client.1.smithi086.stdout:9/414: dwrite d5/d13/d47/d55/f64 [0,4194304] 0 2024-04-06T14:17:09.795 INFO:tasks.workunit.client.0.smithi032.stdout:8/685: dwrite d3/d8/d12/f51 [0,4194304] 0 2024-04-06T14:17:09.798 INFO:tasks.workunit.client.0.smithi032.stdout:4/713: dread d0/df/d1f/d2e/f9b [0,4194304] 0 2024-04-06T14:17:09.799 INFO:tasks.workunit.client.0.smithi032.stdout:3/664: link d3/d65/d7d/d9c/dad/db1/c4c d3/d65/d7d/dc6/de5/c100 0 2024-04-06T14:17:09.801 INFO:tasks.workunit.client.0.smithi032.stdout:6/660: symlink d4/d5/d19/d25/le8 0 2024-04-06T14:17:09.801 INFO:tasks.workunit.client.0.smithi032.stdout:8/686: chown d3/d8/d3a/d7b/db9 51 1 2024-04-06T14:17:09.806 INFO:tasks.workunit.client.0.smithi032.stdout:1/744: rmdir d0/d10/d16/d3f/d98 39 2024-04-06T14:17:09.806 INFO:tasks.workunit.client.0.smithi032.stdout:5/625: symlink d16/d58/ldd 0 2024-04-06T14:17:09.806 INFO:tasks.workunit.client.0.smithi032.stdout:7/686: dread - d16/d92/db3/fd7 zero size 2024-04-06T14:17:09.807 INFO:tasks.workunit.client.0.smithi032.stdout:3/665: creat d3/d65/d7d/d9c/dbc/dcd/f101 x:0 0 0 2024-04-06T14:17:09.807 INFO:tasks.workunit.client.0.smithi032.stdout:8/687: chown d3/d8/d12/d83/d89/da0/daf/cb1 0 1 2024-04-06T14:17:09.809 INFO:tasks.workunit.client.0.smithi032.stdout:5/626: readlink d16/d18/l5a 0 2024-04-06T14:17:09.810 INFO:tasks.workunit.client.0.smithi032.stdout:1/745: chown d0/da/d3e/d75/lb2 27 1 2024-04-06T14:17:09.810 INFO:tasks.workunit.client.0.smithi032.stdout:4/714: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/df2 0 2024-04-06T14:17:09.818 INFO:tasks.workunit.client.0.smithi032.stdout:6/661: rename d4/d5/d19/d25/f57 to d4/d5/dc5/dcc/fe9 0 2024-04-06T14:17:09.818 INFO:tasks.workunit.client.0.smithi032.stdout:3/666: mknod d3/d65/d7d/d9c/dbc/c102 0 2024-04-06T14:17:09.819 INFO:tasks.workunit.client.0.smithi032.stdout:4/715: chown d0/df/d23/fc7 356915474 1 2024-04-06T14:17:09.820 INFO:tasks.workunit.client.0.smithi032.stdout:3/667: readlink d3/d65/d7d/d9c/dad/le7 0 2024-04-06T14:17:09.820 INFO:tasks.workunit.client.0.smithi032.stdout:7/687: mkdir d16/d20/d54/de4/d2d/d30/d3e/df3 0 2024-04-06T14:17:09.822 INFO:tasks.workunit.client.0.smithi032.stdout:6/662: read d4/d5/d18/f52 [391605,21662] 0 2024-04-06T14:17:09.822 INFO:tasks.workunit.client.0.smithi032.stdout:4/716: write d0/df/d23/d31/d50/d57/d5b/fc9 [925687,119545] 0 2024-04-06T14:17:09.825 INFO:tasks.workunit.client.0.smithi032.stdout:3/668: dread d3/fae [0,4194304] 0 2024-04-06T14:17:09.825 INFO:tasks.workunit.client.0.smithi032.stdout:6/663: mknod d4/d5/dbf/d51/cea 0 2024-04-06T14:17:09.830 INFO:tasks.workunit.client.1.smithi086.stdout:6/516: sync 2024-04-06T14:17:09.831 INFO:tasks.workunit.client.0.smithi032.stdout:7/688: write d16/d20/d54/de4/d2d/d30/f53 [4909138,122613] 0 2024-04-06T14:17:09.831 INFO:tasks.workunit.client.0.smithi032.stdout:6/664: mkdir d4/d5/dd8/deb 0 2024-04-06T14:17:09.832 INFO:tasks.workunit.client.0.smithi032.stdout:8/688: dwrite d3/d8/d12/d2f/f65 [0,4194304] 0 2024-04-06T14:17:09.835 INFO:tasks.workunit.client.0.smithi032.stdout:5/627: dwrite d16/d18/d33/d45/d52/d39/f3e [0,4194304] 0 2024-04-06T14:17:09.841 INFO:tasks.workunit.client.0.smithi032.stdout:1/746: dread d0/d10/f3d [8388608,4194304] 0 2024-04-06T14:17:09.843 INFO:tasks.workunit.client.0.smithi032.stdout:5/628: symlink d16/d58/d54/d74/lde 0 2024-04-06T14:17:09.844 INFO:tasks.workunit.client.0.smithi032.stdout:8/689: creat d3/d8/ff5 x:0 0 0 2024-04-06T14:17:09.844 INFO:tasks.workunit.client.0.smithi032.stdout:6/665: truncate d4/d5/d18/f49 664661 0 2024-04-06T14:17:09.845 INFO:tasks.workunit.client.0.smithi032.stdout:6/666: readlink d4/d5/dbf/d51/l26 0 2024-04-06T14:17:09.941 INFO:tasks.workunit.client.0.smithi032.stdout:6/667: write d4/d5/d28/d65/fb1 [207588,85534] 0 2024-04-06T14:17:09.946 INFO:tasks.workunit.client.0.smithi032.stdout:1/747: dwrite d0/f90 [0,4194304] 0 2024-04-06T14:17:09.949 INFO:tasks.workunit.client.0.smithi032.stdout:1/748: fdatasync d0/d10/f8d 0 2024-04-06T14:17:09.949 INFO:tasks.workunit.client.0.smithi032.stdout:7/689: dread d16/f4b [0,4194304] 0 2024-04-06T14:17:09.950 INFO:tasks.workunit.client.0.smithi032.stdout:5/629: dwrite d16/d18/d33/f78 [0,4194304] 0 2024-04-06T14:17:09.953 INFO:tasks.workunit.client.0.smithi032.stdout:6/668: unlink d4/d5/d8/d4b/d7c/cd0 0 2024-04-06T14:17:09.958 INFO:tasks.workunit.client.0.smithi032.stdout:1/749: mkdir d0/d9/df2 0 2024-04-06T14:17:09.961 INFO:tasks.workunit.client.0.smithi032.stdout:8/690: dread d3/f32 [8388608,4194304] 0 2024-04-06T14:17:09.962 INFO:tasks.workunit.client.0.smithi032.stdout:6/669: creat d4/d5/dbf/d51/fec x:0 0 0 2024-04-06T14:17:09.964 INFO:tasks.workunit.client.0.smithi032.stdout:1/750: truncate d0/d9/d94/dc2/fdb 16435 0 2024-04-06T14:17:09.965 INFO:tasks.workunit.client.0.smithi032.stdout:8/691: fdatasync d3/d8/d12/d2f/f39 0 2024-04-06T14:17:09.968 INFO:tasks.workunit.client.0.smithi032.stdout:8/692: stat d3/d8/d12/d2f/d4a/d60/d75/dd1 0 2024-04-06T14:17:09.969 INFO:tasks.workunit.client.0.smithi032.stdout:1/751: creat d0/d10/d16/db8/ff3 x:0 0 0 2024-04-06T14:17:09.970 INFO:tasks.workunit.client.0.smithi032.stdout:5/630: write d16/f50 [1556396,96793] 0 2024-04-06T14:17:09.973 INFO:tasks.workunit.client.0.smithi032.stdout:7/690: write d16/d20/d48/f4a [3045147,5952] 0 2024-04-06T14:17:09.973 INFO:tasks.workunit.client.0.smithi032.stdout:5/631: write d16/d18/d33/d45/d52/d39/f5b [4768958,90444] 0 2024-04-06T14:17:09.978 INFO:tasks.workunit.client.0.smithi032.stdout:7/691: read - d16/d20/d7a/fb9 zero size 2024-04-06T14:17:09.979 INFO:tasks.workunit.client.0.smithi032.stdout:5/632: symlink d16/d58/d54/d94/db2/ldf 0 2024-04-06T14:17:09.984 INFO:tasks.workunit.client.0.smithi032.stdout:6/670: dwrite d4/d5/d19/d25/d67/f8a [0,4194304] 0 2024-04-06T14:17:09.984 INFO:tasks.workunit.client.0.smithi032.stdout:8/693: dwrite d3/d8/d7/d1a/f5a [0,4194304] 0 2024-04-06T14:17:09.987 INFO:tasks.workunit.client.0.smithi032.stdout:5/633: mkdir d16/d18/d33/d35/de0 0 2024-04-06T14:17:09.987 INFO:tasks.workunit.client.0.smithi032.stdout:7/692: write d16/d20/d7a/fb9 [937005,10072] 0 2024-04-06T14:17:09.991 INFO:tasks.workunit.client.0.smithi032.stdout:8/694: stat d3/d8/d12/d2f/d4a/cb2 0 2024-04-06T14:17:09.997 INFO:tasks.workunit.client.0.smithi032.stdout:6/671: link d4/d5/d19/c76 d4/d5/d8/d2b/d77/ced 0 2024-04-06T14:17:09.999 INFO:tasks.workunit.client.0.smithi032.stdout:8/695: rename d3/d8/d7/d14/f9d to d3/ff6 0 2024-04-06T14:17:10.001 INFO:tasks.workunit.client.0.smithi032.stdout:8/696: read - d3/d8/d7/fec zero size 2024-04-06T14:17:10.005 INFO:tasks.workunit.client.0.smithi032.stdout:5/634: dread d16/d58/f5f [0,4194304] 0 2024-04-06T14:17:10.005 INFO:tasks.workunit.client.0.smithi032.stdout:8/697: readlink d3/d8/d3a/l92 0 2024-04-06T14:17:10.007 INFO:tasks.workunit.client.0.smithi032.stdout:7/693: dwrite d16/d20/d7a/da6/fc1 [0,4194304] 0 2024-04-06T14:17:10.010 INFO:tasks.workunit.client.0.smithi032.stdout:7/694: read fa [303359,42642] 0 2024-04-06T14:17:10.010 INFO:tasks.workunit.client.0.smithi032.stdout:8/698: read - d3/d8/d7/fec zero size 2024-04-06T14:17:10.012 INFO:tasks.workunit.client.0.smithi032.stdout:6/672: dwrite d4/d5/dd8/fe3 [0,4194304] 0 2024-04-06T14:17:10.012 INFO:tasks.workunit.client.0.smithi032.stdout:5/635: creat d16/d58/d54/d74/d8f/fe1 x:0 0 0 2024-04-06T14:17:10.017 INFO:tasks.workunit.client.0.smithi032.stdout:8/699: readlink d3/l41 0 2024-04-06T14:17:10.022 INFO:tasks.workunit.client.0.smithi032.stdout:5/636: rmdir d16/d18/d33/d45/d52/d39/d86/dd4 0 2024-04-06T14:17:10.027 INFO:tasks.workunit.client.0.smithi032.stdout:7/695: dread d16/f27 [0,4194304] 0 2024-04-06T14:17:10.028 INFO:tasks.workunit.client.0.smithi032.stdout:5/637: creat d16/d18/d33/d45/d87/fe2 x:0 0 0 2024-04-06T14:17:10.028 INFO:tasks.workunit.client.0.smithi032.stdout:6/673: dwrite d4/d5/d8/d2f/fc2 [0,4194304] 0 2024-04-06T14:17:10.029 INFO:tasks.workunit.client.0.smithi032.stdout:8/700: dwrite d3/d8/d12/d2f/d46/f76 [4194304,4194304] 0 2024-04-06T14:17:10.034 INFO:tasks.workunit.client.0.smithi032.stdout:6/674: creat d4/d5/dbf/d61/da7/dda/fee x:0 0 0 2024-04-06T14:17:10.035 INFO:tasks.workunit.client.0.smithi032.stdout:6/675: readlink d4/l17 0 2024-04-06T14:17:10.036 INFO:tasks.workunit.client.0.smithi032.stdout:7/696: link fe d16/d20/d54/de4/d2d/d30/d3e/df3/ff4 0 2024-04-06T14:17:10.036 INFO:tasks.workunit.client.0.smithi032.stdout:6/676: chown d4/d5/d14/c1c 105 1 2024-04-06T14:17:10.037 INFO:tasks.workunit.client.0.smithi032.stdout:8/701: rmdir d3/d8/d3a/d7b/d8c 0 2024-04-06T14:17:10.039 INFO:tasks.workunit.client.0.smithi032.stdout:5/638: truncate d16/d18/d33/f41 516002 0 2024-04-06T14:17:10.040 INFO:tasks.workunit.client.0.smithi032.stdout:5/639: stat d16/d18/d33/f78 0 2024-04-06T14:17:10.042 INFO:tasks.workunit.client.0.smithi032.stdout:7/697: rename d16/d20/d54/de4/d70 to d16/d20/d63/d9d/df2/df5 0 2024-04-06T14:17:10.046 INFO:tasks.workunit.client.0.smithi032.stdout:8/702: creat d3/d8/d12/d2f/d4a/d60/d75/ff7 x:0 0 0 2024-04-06T14:17:10.049 INFO:tasks.workunit.client.0.smithi032.stdout:8/703: mkdir d3/d8/d3a/d7b/db9/df8 0 2024-04-06T14:17:10.058 INFO:tasks.workunit.client.0.smithi032.stdout:6/677: truncate d4/d5/d8/d2f/fb2 2986474 0 2024-04-06T14:17:10.058 INFO:tasks.workunit.client.0.smithi032.stdout:7/698: write d16/d20/d54/de4/d2d/f3f [10778,882] 0 2024-04-06T14:17:10.059 INFO:tasks.workunit.client.0.smithi032.stdout:6/678: chown d4/d5/dbf/d51/d5e/l9a 1543 1 2024-04-06T14:17:10.061 INFO:tasks.workunit.client.0.smithi032.stdout:7/699: symlink d16/d92/db3/lf6 0 2024-04-06T14:17:10.061 INFO:tasks.workunit.client.0.smithi032.stdout:6/679: fdatasync d4/d5/dbf/f33 0 2024-04-06T14:17:10.066 INFO:tasks.workunit.client.0.smithi032.stdout:6/680: rename d4/d5/d19/d21/c3f to d4/d5/d8/d2b/d77/cef 0 2024-04-06T14:17:10.067 INFO:tasks.workunit.client.0.smithi032.stdout:7/700: rmdir d16/d92/db3 39 2024-04-06T14:17:10.068 INFO:tasks.workunit.client.0.smithi032.stdout:6/681: write d4/d5/d35/f59 [4715062,88506] 0 2024-04-06T14:17:10.070 INFO:tasks.workunit.client.0.smithi032.stdout:6/682: write d4/d5/dd8/fe3 [5210928,97618] 0 2024-04-06T14:17:10.070 INFO:tasks.workunit.client.0.smithi032.stdout:8/704: dread d3/d8/d7/d1a/fb3 [0,4194304] 0 2024-04-06T14:17:10.071 INFO:tasks.workunit.client.0.smithi032.stdout:7/701: creat d16/d20/d63/d9d/df2/df5/ff7 x:0 0 0 2024-04-06T14:17:10.071 INFO:tasks.workunit.client.0.smithi032.stdout:6/683: chown d4/d5/dc5/f50 21102593 1 2024-04-06T14:17:10.073 INFO:tasks.workunit.client.0.smithi032.stdout:6/684: write d4/d5/d28/fbe [777790,54189] 0 2024-04-06T14:17:10.074 INFO:tasks.workunit.client.0.smithi032.stdout:6/685: read d4/d5/f4c [458474,49464] 0 2024-04-06T14:17:10.076 INFO:tasks.workunit.client.0.smithi032.stdout:6/686: truncate d4/d5/d19/d96/faa 562431 0 2024-04-06T14:17:10.078 INFO:tasks.workunit.client.0.smithi032.stdout:6/687: dread d4/d5/dbf/f8c [0,4194304] 0 2024-04-06T14:17:10.079 INFO:tasks.workunit.client.0.smithi032.stdout:6/688: fdatasync d4/d5/d8/d4b/fa6 0 2024-04-06T14:17:10.080 INFO:tasks.workunit.client.0.smithi032.stdout:6/689: rmdir d4/d5/dd8 39 2024-04-06T14:17:10.082 INFO:tasks.workunit.client.0.smithi032.stdout:6/690: unlink d4/d5/d19/d25/d9c/fe0 0 2024-04-06T14:17:10.083 INFO:tasks.workunit.client.0.smithi032.stdout:6/691: write d4/d5/d19/d25/f87 [1055299,46071] 0 2024-04-06T14:17:10.085 INFO:tasks.workunit.client.0.smithi032.stdout:6/692: symlink d4/d5/d14/d80/lf0 0 2024-04-06T14:17:10.091 INFO:tasks.workunit.client.0.smithi032.stdout:6/693: dwrite d4/d5/d28/d65/fb1 [0,4194304] 0 2024-04-06T14:17:10.097 INFO:tasks.workunit.client.0.smithi032.stdout:6/694: link d4/d5/d19/d21/l38 d4/d5/d34/lf1 0 2024-04-06T14:17:10.097 INFO:tasks.workunit.client.0.smithi032.stdout:6/695: chown d4/d5/d35/f39 68003236 1 2024-04-06T14:17:10.098 INFO:tasks.workunit.client.0.smithi032.stdout:6/696: chown d4/d5/d8/d4b/fd3 8475459 1 2024-04-06T14:17:10.102 INFO:tasks.workunit.client.0.smithi032.stdout:6/697: dread d4/d5/d14/f1a [0,4194304] 0 2024-04-06T14:17:10.103 INFO:tasks.workunit.client.0.smithi032.stdout:6/698: fdatasync d4/d5/d28/d65/f92 0 2024-04-06T14:17:10.106 INFO:tasks.workunit.client.0.smithi032.stdout:6/699: symlink d4/d5/d14/d80/lf2 0 2024-04-06T14:17:10.114 INFO:tasks.workunit.client.0.smithi032.stdout:6/700: dwrite d4/d5/d8/f72 [0,4194304] 0 2024-04-06T14:17:10.115 INFO:tasks.workunit.client.0.smithi032.stdout:6/701: chown d4/d5/d18/l71 0 1 2024-04-06T14:17:10.117 INFO:tasks.workunit.client.0.smithi032.stdout:6/702: write d4/d5/d19/f7d [4225242,34426] 0 2024-04-06T14:17:10.120 INFO:tasks.workunit.client.0.smithi032.stdout:6/703: unlink d4/d5/d18/c73 0 2024-04-06T14:17:10.121 INFO:tasks.workunit.client.0.smithi032.stdout:6/704: write d4/d5/d19/d96/faa [868651,43224] 0 2024-04-06T14:17:10.123 INFO:tasks.workunit.client.0.smithi032.stdout:6/705: mknod d4/d69/cf3 0 2024-04-06T14:17:10.124 INFO:tasks.workunit.client.0.smithi032.stdout:6/706: fdatasync d4/d5/dbf/d61/f66 0 2024-04-06T14:17:10.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:09 smithi086 ceph-mon[47925]: pgmap v22: 65 pgs: 65 active+clean; 2.1 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 65 MiB/s rd, 281 MiB/s wr, 488 op/s 2024-04-06T14:17:10.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:09 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:17:10.127 INFO:tasks.workunit.client.0.smithi032.stdout:6/707: unlink d4/d5/d28/f5b 0 2024-04-06T14:17:10.129 INFO:tasks.workunit.client.0.smithi032.stdout:6/708: getdents d4/d5/d19/d96/de6 0 2024-04-06T14:17:10.131 INFO:tasks.workunit.client.0.smithi032.stdout:6/709: symlink d4/d5/d28/lf4 0 2024-04-06T14:17:10.132 INFO:tasks.workunit.client.0.smithi032.stdout:6/710: dread - d4/d5/dc5/dcc/fe9 zero size 2024-04-06T14:17:10.134 INFO:tasks.workunit.client.0.smithi032.stdout:6/711: creat d4/d5/d28/d65/d6e/ff5 x:0 0 0 2024-04-06T14:17:10.212 INFO:tasks.workunit.client.0.smithi032.stdout:9/675: sync 2024-04-06T14:17:10.218 INFO:tasks.workunit.client.0.smithi032.stdout:9/676: dread d2/d15/d1e/d65/d83/d7e/f44 [8388608,4194304] 0 2024-04-06T14:17:10.225 INFO:tasks.workunit.client.0.smithi032.stdout:9/677: dwrite d2/d15/d1e/f6d [0,4194304] 0 2024-04-06T14:17:10.227 INFO:tasks.workunit.client.0.smithi032.stdout:9/678: read d2/d15/d1e/d65/d83/d7e/f14 [3935187,54631] 0 2024-04-06T14:17:10.229 INFO:tasks.workunit.client.0.smithi032.stdout:9/679: fdatasync d2/f21 0 2024-04-06T14:17:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:09 smithi032 ceph-mon[39730]: pgmap v22: 65 pgs: 65 active+clean; 2.1 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 65 MiB/s rd, 281 MiB/s wr, 488 op/s 2024-04-06T14:17:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:09 smithi032 ceph-mon[39730]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:17:10.230 INFO:tasks.workunit.client.0.smithi032.stdout:9/680: chown d2/d15/d1e/d65/d83/d31/d39/f52 14 1 2024-04-06T14:17:10.233 INFO:tasks.workunit.client.0.smithi032.stdout:9/681: mknod d2/d15/d55/d59/cdc 0 2024-04-06T14:17:10.235 INFO:tasks.workunit.client.0.smithi032.stdout:9/682: dread d2/d15/d1e/d65/d83/d31/d39/f5a [0,4194304] 0 2024-04-06T14:17:10.239 INFO:tasks.workunit.client.0.smithi032.stdout:9/683: dread d2/d15/d1e/d65/d83/d31/f40 [0,4194304] 0 2024-04-06T14:17:10.242 INFO:tasks.workunit.client.0.smithi032.stdout:9/684: dread d2/dcd/d7c/f80 [0,4194304] 0 2024-04-06T14:17:10.243 INFO:tasks.workunit.client.0.smithi032.stdout:9/685: fdatasync d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8f/fa2 0 2024-04-06T14:17:10.244 INFO:tasks.workunit.client.0.smithi032.stdout:9/686: fdatasync d2/d15/d55/d59/d93/d95/db8/d81/d4a/f8d 0 2024-04-06T14:17:10.248 INFO:tasks.workunit.client.0.smithi032.stdout:9/687: rename d2/lb to d2/d15/d55/d59/d93/ldd 0 2024-04-06T14:17:10.251 INFO:tasks.workunit.client.0.smithi032.stdout:9/688: getdents d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e 0 2024-04-06T14:17:10.252 INFO:tasks.workunit.client.0.smithi032.stdout:9/689: write d2/d15/d55/d59/d93/fd4 [15267,105981] 0 2024-04-06T14:17:10.254 INFO:tasks.workunit.client.0.smithi032.stdout:9/690: dread d2/fb9 [0,4194304] 0 2024-04-06T14:17:10.256 INFO:tasks.workunit.client.0.smithi032.stdout:9/691: rmdir d2/d15/d1e/d41/d6c/d74 39 2024-04-06T14:17:10.258 INFO:tasks.workunit.client.0.smithi032.stdout:9/692: truncate d2/d15/d55/d59/d93/d95/db8/d81/fb0 641095 0 2024-04-06T14:17:10.260 INFO:tasks.workunit.client.0.smithi032.stdout:9/693: unlink d2/dcd/fa9 0 2024-04-06T14:17:10.261 INFO:tasks.workunit.client.0.smithi032.stdout:9/694: chown d2/d15/d1e/d65/d83/l30 0 1 2024-04-06T14:17:10.261 INFO:tasks.workunit.client.0.smithi032.stdout:9/695: chown d2/d15/d1e/d65/d6b/d7f 123502 1 2024-04-06T14:17:10.268 INFO:tasks.workunit.client.0.smithi032.stdout:9/696: dwrite d2/d15/d1e/d41/d6c/fb7 [0,4194304] 0 2024-04-06T14:17:10.273 INFO:tasks.workunit.client.0.smithi032.stdout:9/697: dread d2/d15/d55/d59/d93/d95/db8/d81/d4a/f8d [0,4194304] 0 2024-04-06T14:17:10.277 INFO:tasks.workunit.client.0.smithi032.stdout:9/698: creat d2/d15/d1e/d65/d83/d31/fde x:0 0 0 2024-04-06T14:17:10.279 INFO:tasks.workunit.client.0.smithi032.stdout:9/699: creat d2/d15/d55/d5f/fdf x:0 0 0 2024-04-06T14:17:10.280 INFO:tasks.workunit.client.0.smithi032.stdout:9/700: write d2/d15/d55/d59/d93/d95/db8/d81/d4a/f8d [356211,29610] 0 2024-04-06T14:17:10.281 INFO:tasks.workunit.client.0.smithi032.stdout:9/701: dread - d2/d15/d1e/d65/d6b/fcf zero size 2024-04-06T14:17:10.284 INFO:tasks.workunit.client.0.smithi032.stdout:9/702: creat d2/d15/d1e/d65/d83/d31/fe0 x:0 0 0 2024-04-06T14:17:10.286 INFO:tasks.workunit.client.0.smithi032.stdout:9/703: mkdir d2/d15/d1e/d65/d83/d31/d39/de1 0 2024-04-06T14:17:10.287 INFO:tasks.workunit.client.0.smithi032.stdout:9/704: chown d2/d15/d1e/d65/d83/d7e/f3e 513 1 2024-04-06T14:17:10.293 INFO:tasks.workunit.client.0.smithi032.stdout:9/705: dwrite d2/f2d [4194304,4194304] 0 2024-04-06T14:17:10.306 INFO:tasks.workunit.client.0.smithi032.stdout:9/706: dread d2/d15/d1e/d65/d83/d7e/f12 [4194304,4194304] 0 2024-04-06T14:17:10.308 INFO:tasks.workunit.client.0.smithi032.stdout:9/707: creat d2/d15/d1e/d65/d83/d7e/da7/fe2 x:0 0 0 2024-04-06T14:17:10.315 INFO:tasks.workunit.client.0.smithi032.stdout:9/708: dwrite d2/dcd/d7c/f80 [0,4194304] 0 2024-04-06T14:17:10.322 INFO:tasks.workunit.client.0.smithi032.stdout:9/709: dwrite d2/d15/d1e/d65/d83/d7e/f64 [4194304,4194304] 0 2024-04-06T14:17:10.324 INFO:tasks.workunit.client.0.smithi032.stdout:9/710: write d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8f/fa2 [1536825,41234] 0 2024-04-06T14:17:10.338 INFO:tasks.workunit.client.0.smithi032.stdout:9/711: dwrite d2/d15/d1e/d65/d83/d7e/f1b [0,4194304] 0 2024-04-06T14:17:10.342 INFO:tasks.workunit.client.0.smithi032.stdout:9/712: symlink d2/d15/d1e/d65/d83/le3 0 2024-04-06T14:17:10.453 INFO:tasks.workunit.client.1.smithi086.stdout:8/584: creat d0/d5a/fcb x:0 0 0 2024-04-06T14:17:10.552 INFO:tasks.workunit.client.1.smithi086.stdout:0/532: rmdir d0/d1/d2/d1a/d79 39 2024-04-06T14:17:10.559 INFO:tasks.workunit.client.1.smithi086.stdout:4/367: dwrite d0/d2c/d38/f5b [0,4194304] 0 2024-04-06T14:17:10.563 INFO:tasks.workunit.client.1.smithi086.stdout:9/415: symlink d5/d13/d1a/d5f/l8b 0 2024-04-06T14:17:10.563 INFO:tasks.workunit.client.1.smithi086.stdout:6/517: rmdir d6 39 2024-04-06T14:17:10.566 INFO:tasks.workunit.client.1.smithi086.stdout:7/449: mknod d7/d10/d11/d1c/d24/c9d 0 2024-04-06T14:17:10.569 INFO:tasks.workunit.client.1.smithi086.stdout:9/416: dread d5/d13/d47/d55/f64 [0,4194304] 0 2024-04-06T14:17:10.569 INFO:tasks.workunit.client.1.smithi086.stdout:5/421: symlink d0/d17/d2d/d80/l82 0 2024-04-06T14:17:10.571 INFO:tasks.workunit.client.1.smithi086.stdout:8/585: creat d0/d71/d22/d43/d6d/d75/db2/fcc x:0 0 0 2024-04-06T14:17:10.574 INFO:tasks.workunit.client.0.smithi032.stdout:2/638: sync 2024-04-06T14:17:10.574 INFO:tasks.workunit.client.0.smithi032.stdout:1/752: sync 2024-04-06T14:17:10.574 INFO:tasks.workunit.client.0.smithi032.stdout:7/702: sync 2024-04-06T14:17:10.575 INFO:tasks.workunit.client.0.smithi032.stdout:8/705: sync 2024-04-06T14:17:10.575 INFO:tasks.workunit.client.0.smithi032.stdout:6/712: sync 2024-04-06T14:17:10.575 INFO:tasks.workunit.client.0.smithi032.stdout:4/717: sync 2024-04-06T14:17:10.575 INFO:tasks.workunit.client.0.smithi032.stdout:5/640: sync 2024-04-06T14:17:10.575 INFO:tasks.workunit.client.0.smithi032.stdout:0/484: sync 2024-04-06T14:17:10.575 INFO:tasks.workunit.client.0.smithi032.stdout:3/669: sync 2024-04-06T14:17:10.575 INFO:tasks.workunit.client.0.smithi032.stdout:9/713: sync 2024-04-06T14:17:10.579 INFO:tasks.workunit.client.1.smithi086.stdout:9/417: dread d5/d13/d1a/d5f/d88/d14/d19/f1e [0,4194304] 0 2024-04-06T14:17:10.580 INFO:tasks.workunit.client.0.smithi032.stdout:7/703: stat d16/d20/d54/f9c 0 2024-04-06T14:17:10.584 INFO:tasks.workunit.client.0.smithi032.stdout:0/485: chown d3/da/d25/d2b/d4a/d86 15775 1 2024-04-06T14:17:10.586 INFO:tasks.workunit.client.1.smithi086.stdout:7/450: dread d7/d10/d11/d80/f46 [0,4194304] 0 2024-04-06T14:17:10.591 INFO:tasks.workunit.client.0.smithi032.stdout:5/641: dwrite d16/d18/d33/d45/d52/d39/f5b [0,4194304] 0 2024-04-06T14:17:10.592 INFO:tasks.workunit.client.1.smithi086.stdout:7/451: dwrite d7/d10/d11/d1c/d36/f74 [0,4194304] 0 2024-04-06T14:17:11.615 INFO:tasks.workunit.client.0.smithi032.stdout:6/713: dread - d4/d5/dc5/fa1 zero size 2024-04-06T14:17:11.616 INFO:tasks.workunit.client.0.smithi032.stdout:8/706: read - d3/d8/d7/d1a/f72 zero size 2024-04-06T14:17:11.616 INFO:tasks.workunit.client.0.smithi032.stdout:4/718: creat d0/df/d2b/d46/ff3 x:0 0 0 2024-04-06T14:17:11.617 INFO:tasks.workunit.client.0.smithi032.stdout:3/670: rename d3/d65/d7d/d9c/ff3 to d3/d16/d63/f103 0 2024-04-06T14:17:11.617 INFO:tasks.workunit.client.0.smithi032.stdout:7/704: symlink d16/d20/d54/de4/d5f/lf8 0 2024-04-06T14:17:11.621 INFO:tasks.workunit.client.0.smithi032.stdout:1/753: creat d0/dc3/ff4 x:0 0 0 2024-04-06T14:17:11.622 INFO:tasks.workunit.client.0.smithi032.stdout:3/671: rmdir d3/d65/d7d/d9c/db8 39 2024-04-06T14:17:11.624 INFO:tasks.workunit.client.0.smithi032.stdout:7/705: unlink d16/d20/d63/d9d/cd4 0 2024-04-06T14:17:11.625 INFO:tasks.workunit.client.0.smithi032.stdout:9/714: link d2/d15/d1e/d65/d83/d7e/da7/cc1 d2/d15/d55/d5f/ce4 0 2024-04-06T14:17:11.631 INFO:tasks.workunit.client.0.smithi032.stdout:6/714: dwrite d4/d5/d19/f55 [0,4194304] 0 2024-04-06T14:17:11.635 INFO:tasks.workunit.client.0.smithi032.stdout:7/706: mkdir d16/d20/d54/de4/de0/df9 0 2024-04-06T14:17:11.642 INFO:tasks.workunit.client.0.smithi032.stdout:8/707: rename d3/d8/d12/d2f/d4a/d60/d75/cc3 to d3/d8/d7/cf9 0 2024-04-06T14:17:11.643 INFO:tasks.workunit.client.0.smithi032.stdout:2/639: dread d0/d4e/f69 [0,4194304] 0 2024-04-06T14:17:11.648 INFO:tasks.workunit.client.0.smithi032.stdout:8/708: write d3/d8/ff5 [436162,120729] 0 2024-04-06T14:17:11.649 INFO:tasks.workunit.client.0.smithi032.stdout:6/715: rename d4/d5/d8/d2b/d77/ddd to d4/d5/d35/df6 0 2024-04-06T14:17:11.650 INFO:tasks.workunit.client.0.smithi032.stdout:9/715: dwrite d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8f/fa2 [0,4194304] 0 2024-04-06T14:17:11.651 INFO:tasks.workunit.client.0.smithi032.stdout:5/642: dread d16/f6b [0,4194304] 0 2024-04-06T14:17:11.651 INFO:tasks.workunit.client.0.smithi032.stdout:2/640: rmdir d0/d30 39 2024-04-06T14:17:11.653 INFO:tasks.workunit.client.0.smithi032.stdout:2/641: stat d0/c75 0 2024-04-06T14:17:11.654 INFO:tasks.workunit.client.0.smithi032.stdout:4/719: write d0/d82/fc5 [2520502,10156] 0 2024-04-06T14:17:11.654 INFO:tasks.workunit.client.0.smithi032.stdout:1/754: dread d0/d10/d16/f3c [4194304,4194304] 0 2024-04-06T14:17:11.654 INFO:tasks.workunit.client.0.smithi032.stdout:0/486: truncate d3/da/d41/f43 3905726 0 2024-04-06T14:17:11.655 INFO:tasks.workunit.client.0.smithi032.stdout:7/707: creat d16/d92/ffa x:0 0 0 2024-04-06T14:17:11.655 INFO:tasks.workunit.client.0.smithi032.stdout:9/716: write d2/d15/d1e/d41/fcb [39888,3464] 0 2024-04-06T14:17:11.661 INFO:tasks.workunit.client.0.smithi032.stdout:6/716: readlink d4/d5/d8/d2f/lc4 0 2024-04-06T14:17:11.661 INFO:tasks.workunit.client.0.smithi032.stdout:5/643: dread - d16/d18/d33/d45/d5e/f89 zero size 2024-04-06T14:17:11.664 INFO:tasks.workunit.client.0.smithi032.stdout:9/717: read d2/f4 [4140523,81437] 0 2024-04-06T14:17:11.665 INFO:tasks.workunit.client.0.smithi032.stdout:5/644: write d16/d18/dae/dbb/fce [1433297,35203] 0 2024-04-06T14:17:11.666 INFO:tasks.workunit.client.0.smithi032.stdout:1/755: symlink d0/d41/d5b/d96/lf5 0 2024-04-06T14:17:11.667 INFO:tasks.workunit.client.0.smithi032.stdout:3/672: dread d3/d65/d89/f97 [0,4194304] 0 2024-04-06T14:17:11.667 INFO:tasks.workunit.client.0.smithi032.stdout:7/708: creat d16/d20/ffb x:0 0 0 2024-04-06T14:17:11.671 INFO:tasks.workunit.client.0.smithi032.stdout:0/487: rename d3/da/d25/d36/l84 to d3/da/d25/la4 0 2024-04-06T14:17:11.671 INFO:tasks.workunit.client.0.smithi032.stdout:5/645: dread - d16/d18/d33/d45/fc0 zero size 2024-04-06T14:17:11.673 INFO:tasks.workunit.client.0.smithi032.stdout:5/646: stat d16/d18/d33/d45/d52 0 2024-04-06T14:17:11.678 INFO:tasks.workunit.client.0.smithi032.stdout:2/642: rename d0/d5a/f97 to d0/d30/d39/d59/d67/d80/fd0 0 2024-04-06T14:17:11.679 INFO:tasks.workunit.client.0.smithi032.stdout:8/709: truncate d3/d8/d12/f51 1288270 0 2024-04-06T14:17:11.680 INFO:tasks.workunit.client.0.smithi032.stdout:3/673: mkdir d3/d65/d7d/d9c/dad/db1/d3c/dc9/d104 0 2024-04-06T14:17:11.681 INFO:tasks.workunit.client.0.smithi032.stdout:3/674: stat d3/f4 0 2024-04-06T14:17:11.682 INFO:tasks.workunit.client.0.smithi032.stdout:8/710: fsync d3/d8/d12/d2f/d4a/f94 0 2024-04-06T14:17:11.685 INFO:tasks.workunit.client.0.smithi032.stdout:5/647: mknod d16/d18/d33/d45/d52/d39/ce3 0 2024-04-06T14:17:11.686 INFO:tasks.workunit.client.0.smithi032.stdout:7/709: write d16/d20/d63/d9d/df2/df5/fe9 [3903711,76450] 0 2024-04-06T14:17:11.688 INFO:tasks.workunit.client.0.smithi032.stdout:9/718: rename d2/d15/d1e/d65/d83/d7e/f64 to d2/d15/d1e/d41/d6c/d74/fe5 0 2024-04-06T14:17:11.689 INFO:tasks.workunit.client.0.smithi032.stdout:4/720: dwrite d0/df/d1f/f25 [0,4194304] 0 2024-04-06T14:17:11.689 INFO:tasks.workunit.client.0.smithi032.stdout:5/648: readlink d16/d58/d54/d94/la9 0 2024-04-06T14:17:11.692 INFO:tasks.workunit.client.0.smithi032.stdout:0/488: creat d3/d83/fa5 x:0 0 0 2024-04-06T14:17:11.697 INFO:tasks.workunit.client.0.smithi032.stdout:9/719: rmdir d2/dcd/d7c 39 2024-04-06T14:17:11.698 INFO:tasks.workunit.client.0.smithi032.stdout:1/756: dwrite d0/f2c [4194304,4194304] 0 2024-04-06T14:17:11.699 INFO:tasks.workunit.client.0.smithi032.stdout:8/711: mkdir d3/d8/d12/d83/d89/da0/dd2/dfa 0 2024-04-06T14:17:11.701 INFO:tasks.workunit.client.0.smithi032.stdout:4/721: dread - d0/df/d2b/dbe/fe4 zero size 2024-04-06T14:17:11.704 INFO:tasks.workunit.client.0.smithi032.stdout:9/720: write d2/d15/d1e/d65/d6b/d92/fb3 [1125323,68048] 0 2024-04-06T14:17:11.704 INFO:tasks.workunit.client.0.smithi032.stdout:1/757: stat d0/d10/l39 0 2024-04-06T14:17:11.706 INFO:tasks.workunit.client.0.smithi032.stdout:5/649: creat d16/d18/d33/d35/d88/dbd/dc5/fe4 x:0 0 0 2024-04-06T14:17:11.707 INFO:tasks.workunit.client.0.smithi032.stdout:0/489: unlink d3/da/d25/d2b/d4a/fa2 0 2024-04-06T14:17:11.712 INFO:tasks.workunit.client.0.smithi032.stdout:8/712: symlink d3/d8/d12/d2f/d4a/d60/d90/lfb 0 2024-04-06T14:17:11.713 INFO:tasks.workunit.client.0.smithi032.stdout:3/675: dwrite d3/d65/d7d/d9c/dad/db1/f9b [0,4194304] 0 2024-04-06T14:17:11.714 INFO:tasks.workunit.client.0.smithi032.stdout:5/650: dread - d16/d18/f9a zero size 2024-04-06T14:17:11.722 INFO:tasks.workunit.client.0.smithi032.stdout:9/721: rename d2/d15/d1e/d65/d83/d31/d39/de1 to d2/d15/d1e/d41/d6c/d88/de6 0 2024-04-06T14:17:11.722 INFO:tasks.workunit.client.0.smithi032.stdout:0/490: write d3/da/d25/d2b/f61 [547506,84238] 0 2024-04-06T14:17:11.723 INFO:tasks.workunit.client.0.smithi032.stdout:7/710: link d16/d20/d54/fad d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee/ffc 0 2024-04-06T14:17:11.725 INFO:tasks.workunit.client.0.smithi032.stdout:4/722: dread d0/df/f92 [0,4194304] 0 2024-04-06T14:17:11.725 INFO:tasks.workunit.client.0.smithi032.stdout:1/758: link d0/d10/d66/l68 d0/lf6 0 2024-04-06T14:17:11.729 INFO:tasks.workunit.client.0.smithi032.stdout:2/643: dread d0/d6e/f72 [0,4194304] 0 2024-04-06T14:17:11.730 INFO:tasks.workunit.client.0.smithi032.stdout:8/713: dread - d3/d8/d12/d2f/d4a/d60/d82/d84/fcc zero size 2024-04-06T14:17:11.733 INFO:tasks.workunit.client.0.smithi032.stdout:0/491: symlink d3/da/d35/d4b/la6 0 2024-04-06T14:17:11.734 INFO:tasks.workunit.client.0.smithi032.stdout:4/723: symlink d0/d82/d4b/lf4 0 2024-04-06T14:17:11.736 INFO:tasks.workunit.client.0.smithi032.stdout:7/711: mkdir d16/d20/d63/d9d/df2/df5/dfd 0 2024-04-06T14:17:11.736 INFO:tasks.workunit.client.0.smithi032.stdout:2/644: chown d0/d30/c46 3 1 2024-04-06T14:17:11.738 INFO:tasks.workunit.client.0.smithi032.stdout:5/651: dwrite d16/fab [0,4194304] 0 2024-04-06T14:17:11.738 INFO:tasks.workunit.client.0.smithi032.stdout:3/676: dwrite d3/d65/d7d/fd2 [0,4194304] 0 2024-04-06T14:17:11.741 INFO:tasks.workunit.client.0.smithi032.stdout:8/714: write d3/d8/d12/d2f/f65 [1718029,83008] 0 2024-04-06T14:17:11.741 INFO:tasks.workunit.client.0.smithi032.stdout:9/722: truncate d2/d15/d1e/d65/d6b/d92/fb3 1397036 0 2024-04-06T14:17:11.747 INFO:tasks.workunit.client.0.smithi032.stdout:1/759: creat d0/d9/d1f/d7c/da9/de1/ff7 x:0 0 0 2024-04-06T14:17:11.748 INFO:tasks.workunit.client.0.smithi032.stdout:8/715: stat d3/d8/d12/d83/d89/da0/daf/cb1 0 2024-04-06T14:17:11.751 INFO:tasks.workunit.client.0.smithi032.stdout:1/760: read - d0/d41/d59/fd4 zero size 2024-04-06T14:17:11.752 INFO:tasks.workunit.client.0.smithi032.stdout:7/712: write d16/d20/d54/de4/d2d/faf [445192,130520] 0 2024-04-06T14:17:11.752 INFO:tasks.workunit.client.0.smithi032.stdout:8/716: stat d3/d8/d12/d2f/lbf 0 2024-04-06T14:17:11.753 INFO:tasks.workunit.client.0.smithi032.stdout:2/645: creat d0/d30/d39/d59/d67/d74/d78/d57/fd1 x:0 0 0 2024-04-06T14:17:11.758 INFO:tasks.workunit.client.0.smithi032.stdout:7/713: readlink d16/d20/d63/l77 0 2024-04-06T14:17:11.759 INFO:tasks.workunit.client.0.smithi032.stdout:1/761: fdatasync d0/d9/d94/dc2/fdb 0 2024-04-06T14:17:11.761 INFO:tasks.workunit.client.0.smithi032.stdout:9/723: creat d2/d15/d1e/d41/d6c/d74/fe7 x:0 0 0 2024-04-06T14:17:11.767 INFO:tasks.workunit.client.0.smithi032.stdout:5/652: dread d16/d18/d33/d45/d52/f4e [0,4194304] 0 2024-04-06T14:17:11.768 INFO:tasks.workunit.client.0.smithi032.stdout:7/714: mknod d16/d20/d54/de4/d2d/d30/d3e/d96/cfe 0 2024-04-06T14:17:11.770 INFO:tasks.workunit.client.0.smithi032.stdout:9/724: mknod d2/d15/d55/d5f/ce8 0 2024-04-06T14:17:11.771 INFO:tasks.workunit.client.0.smithi032.stdout:1/762: write d0/d10/d16/fd2 [616800,73837] 0 2024-04-06T14:17:11.772 INFO:tasks.workunit.client.0.smithi032.stdout:8/717: write d3/d8/d7/fa7 [381057,94308] 0 2024-04-06T14:17:11.779 INFO:tasks.workunit.client.0.smithi032.stdout:3/677: dread d3/f24 [0,4194304] 0 2024-04-06T14:17:11.779 INFO:tasks.workunit.client.0.smithi032.stdout:2/646: getdents d0/d5a/d3a/dc5 0 2024-04-06T14:17:11.779 INFO:tasks.workunit.client.0.smithi032.stdout:5/653: symlink d16/d18/d33/d45/d5e/le5 0 2024-04-06T14:17:11.782 INFO:tasks.workunit.client.0.smithi032.stdout:4/724: dwrite d0/df/d1f/f34 [0,4194304] 0 2024-04-06T14:17:11.783 INFO:tasks.workunit.client.0.smithi032.stdout:0/492: dwrite d3/da/d41/f43 [0,4194304] 0 2024-04-06T14:17:11.783 INFO:tasks.workunit.client.0.smithi032.stdout:7/715: chown d16/d20/d54/de4/d2d/d30/d3e/da7/db5/c99 39092 1 2024-04-06T14:17:11.789 INFO:tasks.workunit.client.0.smithi032.stdout:7/716: write f10 [2073869,75251] 0 2024-04-06T14:17:11.789 INFO:tasks.workunit.client.0.smithi032.stdout:2/647: mknod d0/d30/d39/d59/d67/d74/d78/cd2 0 2024-04-06T14:17:11.791 INFO:tasks.workunit.client.0.smithi032.stdout:5/654: write d16/d18/f9a [139744,55701] 0 2024-04-06T14:17:11.793 INFO:tasks.workunit.client.0.smithi032.stdout:0/493: dread - d3/f5d zero size 2024-04-06T14:17:11.794 INFO:tasks.workunit.client.0.smithi032.stdout:3/678: rename d3/f3f to d3/d65/d7d/d9c/dad/dc5/f105 0 2024-04-06T14:17:11.799 INFO:tasks.workunit.client.0.smithi032.stdout:9/725: dread d2/d15/d1e/d65/d83/d7e/f12 [4194304,4194304] 0 2024-04-06T14:17:11.799 INFO:tasks.workunit.client.0.smithi032.stdout:3/679: chown d3/d16/c1c 5616 1 2024-04-06T14:17:11.800 INFO:tasks.workunit.client.0.smithi032.stdout:1/763: rename d0/d9/d1f/fdc to d0/d10/d16/d3f/d98/ff8 0 2024-04-06T14:17:11.801 INFO:tasks.workunit.client.0.smithi032.stdout:7/717: creat d16/d20/d54/de4/d2d/d30/d3e/da7/fff x:0 0 0 2024-04-06T14:17:11.806 INFO:tasks.workunit.client.0.smithi032.stdout:3/680: creat d3/d65/f106 x:0 0 0 2024-04-06T14:17:11.814 INFO:tasks.workunit.client.0.smithi032.stdout:1/764: creat d0/d9/d1f/d7c/da9/ff9 x:0 0 0 2024-04-06T14:17:11.815 INFO:tasks.workunit.client.0.smithi032.stdout:8/718: dread d3/d8/f87 [0,4194304] 0 2024-04-06T14:17:11.816 INFO:tasks.workunit.client.0.smithi032.stdout:9/726: symlink d2/d15/d1e/d65/d83/d7e/le9 0 2024-04-06T14:17:11.819 INFO:tasks.workunit.client.0.smithi032.stdout:1/765: chown d0/d41/d5b/d96 12 1 2024-04-06T14:17:11.819 INFO:tasks.workunit.client.0.smithi032.stdout:2/648: dread d0/d30/f43 [0,4194304] 0 2024-04-06T14:17:11.820 INFO:tasks.workunit.client.0.smithi032.stdout:3/681: write d3/fa [3179702,122143] 0 2024-04-06T14:17:11.821 INFO:tasks.workunit.client.0.smithi032.stdout:0/494: dread d3/da/f2d [0,4194304] 0 2024-04-06T14:17:11.826 INFO:tasks.workunit.client.0.smithi032.stdout:8/719: creat d3/d8/d12/d2f/d4a/d60/d82/d84/ffc x:0 0 0 2024-04-06T14:17:11.827 INFO:tasks.workunit.client.0.smithi032.stdout:4/725: dwrite d0/df/d23/d31/f87 [0,4194304] 0 2024-04-06T14:17:11.828 INFO:tasks.workunit.client.0.smithi032.stdout:9/727: write d2/d15/d1e/d65/d83/d31/d39/f52 [41625,107361] 0 2024-04-06T14:17:11.829 INFO:tasks.workunit.client.0.smithi032.stdout:5/655: dread d16/d18/d33/d35/f44 [0,4194304] 0 2024-04-06T14:17:11.968 INFO:tasks.workunit.client.0.smithi032.stdout:1/766: mkdir d0/d9/d1f/d7c/dfa 0 2024-04-06T14:17:11.969 INFO:tasks.workunit.client.0.smithi032.stdout:3/682: rmdir d3/d65/d7d/d9c/dad/dc5 39 2024-04-06T14:17:11.970 INFO:tasks.workunit.client.0.smithi032.stdout:0/495: mknod d3/da/d25/d2b/d4a/d86/ca7 0 2024-04-06T14:17:11.971 INFO:tasks.workunit.client.0.smithi032.stdout:4/726: symlink d0/df/d2b/d66/df0/lf5 0 2024-04-06T14:17:11.972 INFO:tasks.workunit.client.0.smithi032.stdout:1/767: chown d0/d10/d16/f3c 27192 1 2024-04-06T14:17:11.972 INFO:tasks.workunit.client.0.smithi032.stdout:5/656: chown d16/f6b 222 1 2024-04-06T14:17:11.972 INFO:tasks.workunit.client.0.smithi032.stdout:2/649: rmdir d0/d5a/d3a/dc5 0 2024-04-06T14:17:11.973 INFO:tasks.workunit.client.0.smithi032.stdout:8/720: creat d3/d8/d12/d2f/d4a/d60/d75/dd1/ffd x:0 0 0 2024-04-06T14:17:11.973 INFO:tasks.workunit.client.0.smithi032.stdout:7/718: dread d16/d20/f28 [0,4194304] 0 2024-04-06T14:17:11.974 INFO:tasks.workunit.client.0.smithi032.stdout:3/683: rmdir d3/d16/ddd/df7 39 2024-04-06T14:17:11.975 INFO:tasks.workunit.client.0.smithi032.stdout:9/728: unlink d2/d15/d55/d59/d93/d95/db8/d81/d4a/f8d 0 2024-04-06T14:17:11.977 INFO:tasks.workunit.client.0.smithi032.stdout:0/496: fsync d3/da/f18 0 2024-04-06T14:17:11.982 INFO:tasks.workunit.client.0.smithi032.stdout:1/768: creat d0/d9/d1f/d72/ffb x:0 0 0 2024-04-06T14:17:11.982 INFO:tasks.workunit.client.0.smithi032.stdout:4/727: write d0/df/d23/fb8 [989082,115358] 0 2024-04-06T14:17:11.984 INFO:tasks.workunit.client.0.smithi032.stdout:2/650: dread d0/d30/d7f/fac [0,4194304] 0 2024-04-06T14:17:11.985 INFO:tasks.workunit.client.0.smithi032.stdout:5/657: creat d16/d18/d33/d35/de0/fe6 x:0 0 0 2024-04-06T14:17:11.987 INFO:tasks.workunit.client.0.smithi032.stdout:1/769: read d0/d10/f3d [10864617,108183] 0 2024-04-06T14:17:11.988 INFO:tasks.workunit.client.0.smithi032.stdout:8/721: chown d3/d8/d12/d83/d89/da0/daf/lf3 15411216 1 2024-04-06T14:17:11.988 INFO:tasks.workunit.client.0.smithi032.stdout:0/497: truncate d3/f76 822507 0 2024-04-06T14:17:11.988 INFO:tasks.workunit.client.0.smithi032.stdout:3/684: dread d3/d65/d7d/d9c/dad/db1/fc3 [0,4194304] 0 2024-04-06T14:17:11.997 INFO:tasks.workunit.client.0.smithi032.stdout:1/770: creat d0/da/ffc x:0 0 0 2024-04-06T14:17:12.000 INFO:tasks.workunit.client.0.smithi032.stdout:7/719: dwrite d16/d20/d54/de4/d2d/d30/d3e/d96/fec [0,4194304] 0 2024-04-06T14:17:12.001 INFO:tasks.workunit.client.0.smithi032.stdout:3/685: symlink d3/d65/d7d/dc6/l107 0 2024-04-06T14:17:12.002 INFO:tasks.workunit.client.0.smithi032.stdout:4/728: symlink d0/df/d23/d31/d50/d57/d5b/d9c/dc8/lf6 0 2024-04-06T14:17:12.006 INFO:tasks.workunit.client.0.smithi032.stdout:5/658: rename d16/d18/d33/d45/d5e/fc8 to d16/d18/dae/dbb/ddc/fe7 0 2024-04-06T14:17:12.006 INFO:tasks.workunit.client.0.smithi032.stdout:8/722: dread d3/d8/d12/d2f/d4a/d60/d82/d84/fdd [0,4194304] 0 2024-04-06T14:17:12.011 INFO:tasks.workunit.client.0.smithi032.stdout:3/686: rmdir d3 39 2024-04-06T14:17:12.015 INFO:tasks.workunit.client.0.smithi032.stdout:9/729: dwrite d2/d15/d1e/d65/d83/f45 [0,4194304] 0 2024-04-06T14:17:12.016 INFO:tasks.workunit.client.0.smithi032.stdout:7/720: creat d16/d20/d63/f100 x:0 0 0 2024-04-06T14:17:12.016 INFO:tasks.workunit.client.0.smithi032.stdout:0/498: rename d3/da/d25/d2b/d4a/d91 to d3/da/d25/d2b/d4a/d79/da8 0 2024-04-06T14:17:12.018 INFO:tasks.workunit.client.0.smithi032.stdout:2/651: dwrite d0/d4e/f69 [0,4194304] 0 2024-04-06T14:17:12.019 INFO:tasks.workunit.client.0.smithi032.stdout:3/687: stat d3/d65/f8c 0 2024-04-06T14:17:12.020 INFO:tasks.workunit.client.0.smithi032.stdout:4/729: write d0/df/d2b/fa8 [4400386,118319] 0 2024-04-06T14:17:12.023 INFO:tasks.workunit.client.0.smithi032.stdout:8/723: creat d3/d8/d12/d83/d89/da0/ffe x:0 0 0 2024-04-06T14:17:12.024 INFO:tasks.workunit.client.0.smithi032.stdout:5/659: truncate d16/d18/d33/d35/de0/fe6 383723 0 2024-04-06T14:17:12.032 INFO:tasks.workunit.client.0.smithi032.stdout:2/652: write d0/d5a/f31 [707563,52707] 0 2024-04-06T14:17:12.032 INFO:tasks.workunit.client.0.smithi032.stdout:7/721: write d16/d20/d63/d9d/df2/df5/fe9 [1934075,14062] 0 2024-04-06T14:17:12.036 INFO:tasks.workunit.client.0.smithi032.stdout:4/730: creat d0/d82/d4b/db3/ff7 x:0 0 0 2024-04-06T14:17:12.040 INFO:tasks.workunit.client.0.smithi032.stdout:7/722: chown d16/d20/d48/fbf 65321575 1 2024-04-06T14:17:12.042 INFO:tasks.workunit.client.0.smithi032.stdout:9/730: dread d2/d15/d1e/d65/d6b/d92/fb3 [0,4194304] 0 2024-04-06T14:17:12.045 INFO:tasks.workunit.client.0.smithi032.stdout:8/724: rename d3/d8/d12/d2f/d4a/ff0 to d3/d8/d12/d2f/d4a/d60/d75/dd1/fff 0 2024-04-06T14:17:12.046 INFO:tasks.workunit.client.0.smithi032.stdout:1/771: dwrite d0/d9/d1f/f93 [0,4194304] 0 2024-04-06T14:17:12.048 INFO:tasks.workunit.client.0.smithi032.stdout:7/723: read d16/d20/d48/f4a [2884849,11517] 0 2024-04-06T14:17:12.049 INFO:tasks.workunit.client.0.smithi032.stdout:5/660: dread d16/d18/d33/d35/f44 [0,4194304] 0 2024-04-06T14:17:12.050 INFO:tasks.workunit.client.0.smithi032.stdout:8/725: chown d3/d8/d7/d14/l15 3 1 2024-04-06T14:17:12.051 INFO:tasks.workunit.client.0.smithi032.stdout:0/499: dwrite d3/da/d25/d2b/d4a/d79/da8/f9f [0,4194304] 0 2024-04-06T14:17:12.051 INFO:tasks.workunit.client.0.smithi032.stdout:5/661: stat d16/d58/d54 0 2024-04-06T14:17:12.051 INFO:tasks.workunit.client.0.smithi032.stdout:2/653: mkdir d0/d30/d39/d59/d67/d74/d78/d57/dd3 0 2024-04-06T14:17:12.051 INFO:tasks.workunit.client.0.smithi032.stdout:4/731: symlink d0/df/d23/d31/d50/d5e/da3/lf8 0 2024-04-06T14:17:12.054 INFO:tasks.workunit.client.0.smithi032.stdout:4/732: chown d0/ca4 7 1 2024-04-06T14:17:12.057 INFO:tasks.workunit.client.0.smithi032.stdout:2/654: creat d0/db2/fd4 x:0 0 0 2024-04-06T14:17:12.061 INFO:tasks.workunit.client.0.smithi032.stdout:5/662: creat d16/d58/d54/d74/d8f/fe8 x:0 0 0 2024-04-06T14:17:12.061 INFO:tasks.workunit.client.0.smithi032.stdout:7/724: creat d16/d92/f101 x:0 0 0 2024-04-06T14:17:12.064 INFO:tasks.workunit.client.0.smithi032.stdout:2/655: rmdir d0/d5a/d3a 39 2024-04-06T14:17:12.067 INFO:tasks.workunit.client.0.smithi032.stdout:7/725: symlink d16/d20/d48/d8d/l102 0 2024-04-06T14:17:12.069 INFO:tasks.workunit.client.0.smithi032.stdout:9/731: dwrite d2/d15/d1e/d65/d83/d31/d39/da6/fb6 [0,4194304] 0 2024-04-06T14:17:12.076 INFO:tasks.workunit.client.0.smithi032.stdout:7/726: chown d16/d20/d54/de4/d2d/l34 204240420 1 2024-04-06T14:17:12.077 INFO:tasks.workunit.client.0.smithi032.stdout:2/656: rmdir d0/d30/d39/d59/d67/d74/d78/d8c/db0/d95 39 2024-04-06T14:17:12.077 INFO:tasks.workunit.client.0.smithi032.stdout:4/733: dread d0/df/d23/f3c [0,4194304] 0 2024-04-06T14:17:12.080 INFO:tasks.workunit.client.0.smithi032.stdout:9/732: chown d2/d15/d1e/d65/d83/d7e/l35 22845 1 2024-04-06T14:17:12.081 INFO:tasks.workunit.client.0.smithi032.stdout:1/772: dwrite d0/d10/f30 [0,4194304] 0 2024-04-06T14:17:12.085 INFO:tasks.workunit.client.0.smithi032.stdout:7/727: creat d16/d92/db3/f103 x:0 0 0 2024-04-06T14:17:12.085 INFO:tasks.workunit.client.0.smithi032.stdout:2/657: symlink d0/d30/d39/ld5 0 2024-04-06T14:17:12.085 INFO:tasks.workunit.client.0.smithi032.stdout:0/500: dread d3/da/d25/d36/f93 [0,4194304] 0 2024-04-06T14:17:12.088 INFO:tasks.workunit.client.0.smithi032.stdout:1/773: dread - d0/dc3/ff4 zero size 2024-04-06T14:17:12.091 INFO:tasks.workunit.client.0.smithi032.stdout:5/663: dwrite d16/d18/d33/d35/d4d/f82 [0,4194304] 0 2024-04-06T14:17:12.093 INFO:tasks.workunit.client.0.smithi032.stdout:4/734: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/df9 0 2024-04-06T14:17:12.095 INFO:tasks.workunit.client.0.smithi032.stdout:0/501: truncate d3/f10 4729142 0 2024-04-06T14:17:12.095 INFO:tasks.workunit.client.0.smithi032.stdout:8/726: dwrite d3/f79 [0,4194304] 0 2024-04-06T14:17:12.098 INFO:tasks.workunit.client.0.smithi032.stdout:5/664: fdatasync d16/f50 0 2024-04-06T14:17:12.099 INFO:tasks.workunit.client.0.smithi032.stdout:9/733: write d2/d15/d55/d59/d93/d95/db8/d81/d77/f78 [3886593,80962] 0 2024-04-06T14:17:12.106 INFO:tasks.workunit.client.0.smithi032.stdout:8/727: fsync d3/d8/d12/d2f/d4a/d60/d82/fa1 0 2024-04-06T14:17:12.110 INFO:tasks.workunit.client.0.smithi032.stdout:0/502: rename d3/da/d25/d2b/f3a to d3/da/d25/d2b/d4a/fa9 0 2024-04-06T14:17:12.110 INFO:tasks.workunit.client.0.smithi032.stdout:9/734: readlink d2/d15/d1e/d41/d6c/d88/ld6 0 2024-04-06T14:17:12.110 INFO:tasks.workunit.client.0.smithi032.stdout:4/735: truncate d0/df/d2b/d46/d58/f84 279059 0 2024-04-06T14:17:12.112 INFO:tasks.workunit.client.0.smithi032.stdout:5/665: creat d16/d18/d33/fe9 x:0 0 0 2024-04-06T14:17:12.113 INFO:tasks.workunit.client.0.smithi032.stdout:7/728: getdents d16/d20/d54/de4/d2d/d30/d3e/da7/db5 0 2024-04-06T14:17:12.114 INFO:tasks.workunit.client.0.smithi032.stdout:0/503: rename d3/da/d41/f42 to d3/da/faa 0 2024-04-06T14:17:12.116 INFO:tasks.workunit.client.0.smithi032.stdout:8/728: symlink d3/d8/d3a/d4b/de0/l100 0 2024-04-06T14:17:12.118 INFO:tasks.workunit.client.0.smithi032.stdout:7/729: fsync d16/f4b 0 2024-04-06T14:17:12.120 INFO:tasks.workunit.client.0.smithi032.stdout:4/736: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dfa 0 2024-04-06T14:17:12.123 INFO:tasks.workunit.client.0.smithi032.stdout:9/735: rename d2/d15/d1e/d65/d83/d31/fe0 to d2/d15/d55/d59/d93/d95/fea 0 2024-04-06T14:17:12.124 INFO:tasks.workunit.client.0.smithi032.stdout:2/658: truncate d0/d4e/f69 3670083 0 2024-04-06T14:17:12.125 INFO:tasks.workunit.client.0.smithi032.stdout:8/729: mkdir d3/d8/d12/d101 0 2024-04-06T14:17:12.125 INFO:tasks.workunit.client.0.smithi032.stdout:1/774: dwrite d0/d10/d16/d3f/d98/fa8 [0,4194304] 0 2024-04-06T14:17:12.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:11 smithi086 ceph-mon[47925]: pgmap v23: 65 pgs: 65 active+clean; 2.1 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 65 MiB/s rd, 256 MiB/s wr, 448 op/s 2024-04-06T14:17:12.130 INFO:tasks.workunit.client.0.smithi032.stdout:5/666: mkdir d16/d18/d33/dea 0 2024-04-06T14:17:12.130 INFO:tasks.workunit.client.0.smithi032.stdout:1/775: fdatasync d0/d9/d94/dc2/fdb 0 2024-04-06T14:17:12.132 INFO:tasks.workunit.client.0.smithi032.stdout:4/737: fdatasync d0/df/d23/d31/d50/d5e/fbf 0 2024-04-06T14:17:12.134 INFO:tasks.workunit.client.0.smithi032.stdout:0/504: dread d3/f10 [4194304,4194304] 0 2024-04-06T14:17:12.135 INFO:tasks.workunit.client.0.smithi032.stdout:9/736: mkdir d2/dcd/deb 0 2024-04-06T14:17:12.136 INFO:tasks.workunit.client.0.smithi032.stdout:8/730: rename d3/d8/d12/d83/d89/da0/daf/cb1 to d3/d8/d12/d83/d89/da0/dd2/c102 0 2024-04-06T14:17:12.139 INFO:tasks.workunit.client.0.smithi032.stdout:4/738: unlink d0/df/l17 0 2024-04-06T14:17:12.139 INFO:tasks.workunit.client.0.smithi032.stdout:5/667: dread f11 [0,4194304] 0 2024-04-06T14:17:12.140 INFO:tasks.workunit.client.0.smithi032.stdout:0/505: truncate d3/da/f2d 4115583 0 2024-04-06T14:17:12.141 INFO:tasks.workunit.client.0.smithi032.stdout:7/730: getdents d16/d20/d54/de4/d2d/d30/d3e/da7 0 2024-04-06T14:17:12.142 INFO:tasks.workunit.client.0.smithi032.stdout:9/737: symlink d2/d15/d1e/d65/d83/d7e/da7/lec 0 2024-04-06T14:17:12.143 INFO:tasks.workunit.client.0.smithi032.stdout:5/668: write d16/d18/d33/d45/d52/d39/d86/f93 [1315999,987] 0 2024-04-06T14:17:12.144 INFO:tasks.workunit.client.0.smithi032.stdout:1/776: dread d0/d10/f30 [0,4194304] 0 2024-04-06T14:17:12.146 INFO:tasks.workunit.client.0.smithi032.stdout:2/659: write d0/d30/d39/d59/d67/d74/d78/d57/f7d [471173,130378] 0 2024-04-06T14:17:12.147 INFO:tasks.workunit.client.0.smithi032.stdout:7/731: write d16/d20/d54/de4/d2d/d30/f53 [3765981,26232] 0 2024-04-06T14:17:12.150 INFO:tasks.workunit.client.0.smithi032.stdout:2/660: stat d0/la6 0 2024-04-06T14:17:12.154 INFO:tasks.workunit.client.0.smithi032.stdout:9/738: mkdir d2/d15/d55/d59/d93/d95/db8/d81/d4a/ded 0 2024-04-06T14:17:12.154 INFO:tasks.workunit.client.0.smithi032.stdout:0/506: dread d3/f78 [0,4194304] 0 2024-04-06T14:17:12.155 INFO:tasks.workunit.client.0.smithi032.stdout:5/669: mkdir d16/d18/d33/d35/d88/deb 0 2024-04-06T14:17:12.157 INFO:tasks.workunit.client.0.smithi032.stdout:7/732: mknod d16/d20/d63/d9d/df2/c104 0 2024-04-06T14:17:12.157 INFO:tasks.workunit.client.0.smithi032.stdout:2/661: chown d0/d30/d39/d59/d67/d74/d78/d57/d7c 1034631 1 2024-04-06T14:17:12.160 INFO:tasks.workunit.client.0.smithi032.stdout:1/777: truncate d0/f90 254904 0 2024-04-06T14:17:12.162 INFO:tasks.workunit.client.0.smithi032.stdout:5/670: mkdir d16/d18/d33/d35/d88/dbd/dc5/dec 0 2024-04-06T14:17:12.162 INFO:tasks.workunit.client.0.smithi032.stdout:8/731: dwrite d3/d8/d12/d2f/d4a/d60/d82/fa6 [0,4194304] 0 2024-04-06T14:17:12.165 INFO:tasks.workunit.client.0.smithi032.stdout:0/507: truncate d3/f4 1479539 0 2024-04-06T14:17:12.168 INFO:tasks.workunit.client.0.smithi032.stdout:2/662: symlink d0/d4e/dc6/ld6 0 2024-04-06T14:17:12.175 INFO:tasks.workunit.client.0.smithi032.stdout:7/733: read d16/d20/d54/de4/d2d/d30/f38 [3961850,107293] 0 2024-04-06T14:17:12.175 INFO:tasks.workunit.client.0.smithi032.stdout:5/671: fsync d16/d18/d33/d35/f6a 0 2024-04-06T14:17:12.178 INFO:tasks.workunit.client.0.smithi032.stdout:0/508: mkdir d3/d83/dab 0 2024-04-06T14:17:12.178 INFO:tasks.workunit.client.0.smithi032.stdout:9/739: dwrite d2/d15/d1e/d41/d6c/d74/fe7 [0,4194304] 0 2024-04-06T14:17:12.179 INFO:tasks.workunit.client.0.smithi032.stdout:4/739: dread d0/df/d23/d31/f68 [0,4194304] 0 2024-04-06T14:17:12.182 INFO:tasks.workunit.client.0.smithi032.stdout:2/663: truncate d0/d4e/f69 2285579 0 2024-04-06T14:17:12.184 INFO:tasks.workunit.client.0.smithi032.stdout:8/732: rename d3/d8/d12/d2f/d4a/d60/d90/lfb to d3/l103 0 2024-04-06T14:17:12.185 INFO:tasks.workunit.client.0.smithi032.stdout:1/778: rmdir d0/da/d3e/d75/dd7/de2 0 2024-04-06T14:17:12.186 INFO:tasks.workunit.client.0.smithi032.stdout:0/509: symlink d3/da/d25/d2b/d4a/da1/lac 0 2024-04-06T14:17:12.191 INFO:tasks.workunit.client.0.smithi032.stdout:4/740: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/df2/dfb 0 2024-04-06T14:17:12.192 INFO:tasks.workunit.client.0.smithi032.stdout:8/733: write d3/d8/d7/d8a/fcd [661263,106275] 0 2024-04-06T14:17:12.192 INFO:tasks.workunit.client.0.smithi032.stdout:2/664: rmdir d0/d30/d39/d59/d67/d74/d78/d8c/db0 39 2024-04-06T14:17:12.195 INFO:tasks.workunit.client.0.smithi032.stdout:0/510: write d3/da/d25/d2b/d4a/f65 [8069257,35917] 0 2024-04-06T14:17:12.203 INFO:tasks.workunit.client.0.smithi032.stdout:7/734: dwrite d16/d20/d54/de4/d2d/faf [0,4194304] 0 2024-04-06T14:17:12.206 INFO:tasks.workunit.client.0.smithi032.stdout:0/511: unlink d3/da/d25/d36/c9a 0 2024-04-06T14:17:12.206 INFO:tasks.workunit.client.0.smithi032.stdout:2/665: truncate d0/d5a/d3a/f61 1047203 0 2024-04-06T14:17:12.211 INFO:tasks.workunit.client.0.smithi032.stdout:2/666: stat d0/l9 0 2024-04-06T14:17:12.211 INFO:tasks.workunit.client.0.smithi032.stdout:5/672: dread d16/d18/f20 [4194304,4194304] 0 2024-04-06T14:17:12.212 INFO:tasks.workunit.client.0.smithi032.stdout:9/740: dread d2/d15/d1e/d65/d83/d7e/f37 [0,4194304] 0 2024-04-06T14:17:12.214 INFO:tasks.workunit.client.0.smithi032.stdout:1/779: dwrite d0/d9/d94/fd0 [0,4194304] 0 2024-04-06T14:17:12.215 INFO:tasks.workunit.client.0.smithi032.stdout:7/735: creat d16/d20/dbd/f105 x:0 0 0 2024-04-06T14:17:12.216 INFO:tasks.workunit.client.0.smithi032.stdout:5/673: chown d16/d18/d33/fa8 425301 1 2024-04-06T14:17:12.218 INFO:tasks.workunit.client.0.smithi032.stdout:0/512: symlink d3/da/d35/d8b/d9c/lad 0 2024-04-06T14:17:12.218 INFO:tasks.workunit.client.0.smithi032.stdout:8/734: link d3/d8/d12/d2f/d4a/ca2 d3/d8/d12/d2f/d4a/c104 0 2024-04-06T14:17:12.222 INFO:tasks.workunit.client.0.smithi032.stdout:7/736: creat d16/d20/d6e/f106 x:0 0 0 2024-04-06T14:17:12.223 INFO:tasks.workunit.client.0.smithi032.stdout:9/741: truncate d2/d15/d55/f86 885467 0 2024-04-06T14:17:12.224 INFO:tasks.workunit.client.0.smithi032.stdout:5/674: creat d16/d58/d54/d74/d8f/fed x:0 0 0 2024-04-06T14:17:12.225 INFO:tasks.workunit.client.0.smithi032.stdout:0/513: creat d3/da/d25/d2b/d4a/da1/fae x:0 0 0 2024-04-06T14:17:12.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:11 smithi032 ceph-mon[39730]: pgmap v23: 65 pgs: 65 active+clean; 2.1 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 65 MiB/s rd, 256 MiB/s wr, 448 op/s 2024-04-06T14:17:12.231 INFO:tasks.workunit.client.0.smithi032.stdout:2/667: rename d0/d4e/d63 to d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7 0 2024-04-06T14:17:12.232 INFO:tasks.workunit.client.0.smithi032.stdout:8/735: mknod d3/d8/d12/d83/d89/da0/dd2/dfa/c105 0 2024-04-06T14:17:12.232 INFO:tasks.workunit.client.0.smithi032.stdout:0/514: dread d3/f5a [4194304,4194304] 0 2024-04-06T14:17:12.233 INFO:tasks.workunit.client.0.smithi032.stdout:9/742: mkdir d2/d15/d1e/d65/d83/d31/d39/da6/dee 0 2024-04-06T14:17:12.234 INFO:tasks.workunit.client.0.smithi032.stdout:5/675: chown d16/d18/d33/d45/d52/d39/l8e 7820097 1 2024-04-06T14:17:12.235 INFO:tasks.workunit.client.0.smithi032.stdout:7/737: link d16/d20/d54/de4/d2d/d30/d3e/l6c d16/d20/d54/de4/d2d/d30/d3e/d96/l107 0 2024-04-06T14:17:12.237 INFO:tasks.workunit.client.0.smithi032.stdout:1/780: dwrite d0/d10/d16/f7a [0,4194304] 0 2024-04-06T14:17:12.240 INFO:tasks.workunit.client.0.smithi032.stdout:8/736: truncate d3/d8/d12/d2f/f62 1751703 0 2024-04-06T14:17:12.241 INFO:tasks.workunit.client.0.smithi032.stdout:5/676: creat d16/d18/d33/d45/d52/fee x:0 0 0 2024-04-06T14:17:12.242 INFO:tasks.workunit.client.0.smithi032.stdout:7/738: mknod d16/d20/d54/de4/de0/c108 0 2024-04-06T14:17:12.243 INFO:tasks.workunit.client.0.smithi032.stdout:8/737: dread - d3/d8/d12/d2f/d4a/d60/d90/ff2 zero size 2024-04-06T14:17:12.244 INFO:tasks.workunit.client.0.smithi032.stdout:7/739: fsync d16/d20/d54/de4/d5f/f90 0 2024-04-06T14:17:12.245 INFO:tasks.workunit.client.0.smithi032.stdout:0/515: creat d3/da/d25/d2b/d4a/d79/d97/faf x:0 0 0 2024-04-06T14:17:12.251 INFO:tasks.workunit.client.0.smithi032.stdout:1/781: rename d0/d9/d94/fbf to d0/d10/d16/ffd 0 2024-04-06T14:17:12.252 INFO:tasks.workunit.client.0.smithi032.stdout:9/743: write d2/d15/d55/d59/d93/d95/db8/d81/fab [430498,121784] 0 2024-04-06T14:17:12.252 INFO:tasks.workunit.client.0.smithi032.stdout:5/677: dwrite d16/d18/d33/fe9 [0,4194304] 0 2024-04-06T14:17:12.253 INFO:tasks.workunit.client.0.smithi032.stdout:0/516: symlink d3/da/d25/d36/lb0 0 2024-04-06T14:17:12.255 INFO:tasks.workunit.client.0.smithi032.stdout:7/740: link d16/d20/d54/de4/d2d/d30/d3e/df3/ff4 d16/d20/d54/de4/d2d/f109 0 2024-04-06T14:17:12.256 INFO:tasks.workunit.client.0.smithi032.stdout:9/744: chown d2/d15/d1e/d65/d83/d7e/da7 11 1 2024-04-06T14:17:12.257 INFO:tasks.workunit.client.0.smithi032.stdout:0/517: mknod d3/da/d25/d2b/d4a/d86/cb1 0 2024-04-06T14:17:12.258 INFO:tasks.workunit.client.0.smithi032.stdout:5/678: creat d16/d18/d33/d45/d52/d39/d86/fef x:0 0 0 2024-04-06T14:17:12.259 INFO:tasks.workunit.client.0.smithi032.stdout:0/518: write d3/f45 [1581425,47321] 0 2024-04-06T14:17:12.261 INFO:tasks.workunit.client.0.smithi032.stdout:1/782: rename d0/da/d3e/d75/dd7/feb to d0/d9/d1f/d7c/dfa/ffe 0 2024-04-06T14:17:12.268 INFO:tasks.workunit.client.0.smithi032.stdout:0/519: readlink d3/da/d25/la4 0 2024-04-06T14:17:12.270 INFO:tasks.workunit.client.0.smithi032.stdout:1/783: chown d0/d10/d16/cb0 30893381 1 2024-04-06T14:17:12.271 INFO:tasks.workunit.client.0.smithi032.stdout:7/741: getdents d16/d20/d54/de4/d2d/d30/d3e/da7 0 2024-04-06T14:17:12.272 INFO:tasks.workunit.client.0.smithi032.stdout:2/668: dread d0/d30/d39/d59/d67/d74/d78/d57/d7c/fb9 [0,4194304] 0 2024-04-06T14:17:12.278 INFO:tasks.workunit.client.0.smithi032.stdout:9/745: dwrite d2/d15/d55/d59/fc2 [0,4194304] 0 2024-04-06T14:17:12.279 INFO:tasks.workunit.client.0.smithi032.stdout:5/679: dwrite d16/d58/f6d [0,4194304] 0 2024-04-06T14:17:12.404 INFO:tasks.workunit.client.0.smithi032.stdout:8/738: dwrite d3/d69/fad [0,4194304] 0 2024-04-06T14:17:12.404 INFO:tasks.workunit.client.0.smithi032.stdout:1/784: write d0/da/d3e/d75/fdf [220214,124398] 0 2024-04-06T14:17:12.404 INFO:tasks.workunit.client.0.smithi032.stdout:7/742: readlink d16/d20/d54/de4/d2d/d30/laa 0 2024-04-06T14:17:12.411 INFO:tasks.workunit.client.0.smithi032.stdout:7/743: write d16/d20/d7a/fa5 [1179486,74027] 0 2024-04-06T14:17:12.422 INFO:tasks.workunit.client.1.smithi086.stdout:0/533: creat d0/d15/d4e/d48/d89/fa9 x:0 0 0 2024-04-06T14:17:12.423 INFO:tasks.workunit.client.1.smithi086.stdout:0/534: readlink d0/d15/d4e/da3/d44/l58 0 2024-04-06T14:17:12.442 INFO:tasks.workunit.client.0.smithi032.stdout:2/669: dread d0/d4e/f69 [0,4194304] 0 2024-04-06T14:17:12.445 INFO:tasks.workunit.client.0.smithi032.stdout:0/520: truncate d3/da/f1f 902592 0 2024-04-06T14:17:12.450 INFO:tasks.workunit.client.0.smithi032.stdout:7/744: mkdir d16/d20/d54/de4/d2d/d10a 0 2024-04-06T14:17:12.452 INFO:tasks.workunit.client.0.smithi032.stdout:0/521: readlink d3/da/lb 0 2024-04-06T14:17:12.453 INFO:tasks.workunit.client.0.smithi032.stdout:1/785: getdents d0/d9/d1f/d7c/d8f/daa 0 2024-04-06T14:17:12.453 INFO:tasks.workunit.client.1.smithi086.stdout:6/518: read - d6/d13/d5f/d60/d45/fa1 zero size 2024-04-06T14:17:12.454 INFO:tasks.workunit.client.1.smithi086.stdout:1/519: link db/d11/d1d/c56 db/d11/d1f/d5c/ca1 0 2024-04-06T14:17:12.455 INFO:tasks.workunit.client.0.smithi032.stdout:8/739: dwrite d3/d8/d12/d2f/d4a/d60/d75/dd1/ffd [0,4194304] 0 2024-04-06T14:17:12.458 INFO:tasks.workunit.client.0.smithi032.stdout:2/670: creat d0/d30/fd8 x:0 0 0 2024-04-06T14:17:12.460 INFO:tasks.workunit.client.0.smithi032.stdout:9/746: dwrite d2/d15/d1e/d65/d83/d7e/f12 [4194304,4194304] 0 2024-04-06T14:17:12.461 INFO:tasks.workunit.client.0.smithi032.stdout:1/786: mknod d0/d9/d1f/d72/cff 0 2024-04-06T14:17:12.461 INFO:tasks.workunit.client.0.smithi032.stdout:5/680: dread d16/d18/f30 [0,4194304] 0 2024-04-06T14:17:12.463 INFO:tasks.workunit.client.0.smithi032.stdout:7/745: mkdir d16/d20/d54/de4/d2d/d30/d3e/d96/d10b 0 2024-04-06T14:17:12.465 INFO:tasks.workunit.client.1.smithi086.stdout:9/418: creat d5/d13/f8c x:0 0 0 2024-04-06T14:17:12.465 INFO:tasks.workunit.client.0.smithi032.stdout:2/671: mkdir d0/d30/d7f/dd9 0 2024-04-06T14:17:12.466 INFO:tasks.workunit.client.0.smithi032.stdout:8/740: truncate d3/d8/d3a/f5c 1841866 0 2024-04-06T14:17:12.470 INFO:tasks.workunit.client.0.smithi032.stdout:1/787: mknod d0/d9/d94/dc2/df1/c100 0 2024-04-06T14:17:12.471 INFO:tasks.workunit.client.0.smithi032.stdout:0/522: link d3/f76 d3/da/d25/d2b/d4a/da1/fb2 0 2024-04-06T14:17:12.472 INFO:tasks.workunit.client.0.smithi032.stdout:9/747: dwrite d2/d15/d1e/d41/d6c/d74/fe7 [0,4194304] 0 2024-04-06T14:17:12.472 INFO:tasks.workunit.client.0.smithi032.stdout:8/741: mkdir d3/d8/d3a/d7b/db9/df8/d106 0 2024-04-06T14:17:12.474 INFO:tasks.workunit.client.0.smithi032.stdout:7/746: getdents d16/d20/d63/d9d/df2 0 2024-04-06T14:17:12.474 INFO:tasks.workunit.client.0.smithi032.stdout:1/788: stat d0/d10/d16/l43 0 2024-04-06T14:17:12.475 INFO:tasks.workunit.client.0.smithi032.stdout:9/748: chown d2/d15/d1e/d65/d83/d31/cd7 0 1 2024-04-06T14:17:12.475 INFO:tasks.workunit.client.0.smithi032.stdout:8/742: write d3/d8/d12/d2f/d46/f74 [5499662,12160] 0 2024-04-06T14:17:12.479 INFO:tasks.workunit.client.0.smithi032.stdout:7/747: fsync d16/d20/d54/de4/f3a 0 2024-04-06T14:17:12.480 INFO:tasks.workunit.client.0.smithi032.stdout:8/743: mkdir d3/d8/d12/d2f/d4a/d60/d82/d107 0 2024-04-06T14:17:12.485 INFO:tasks.workunit.client.0.smithi032.stdout:8/744: dread d3/f79 [0,4194304] 0 2024-04-06T14:17:12.485 INFO:tasks.workunit.client.0.smithi032.stdout:7/748: rmdir d16/d20/d7a 39 2024-04-06T14:17:12.486 INFO:tasks.workunit.client.0.smithi032.stdout:8/745: readlink d3/d8/d7/l6e 0 2024-04-06T14:17:12.493 INFO:tasks.workunit.client.0.smithi032.stdout:1/789: write d0/d41/fca [966264,58628] 0 2024-04-06T14:17:12.494 INFO:tasks.workunit.client.0.smithi032.stdout:7/749: link d16/d92/db3/lf6 d16/d20/d54/de4/d2d/d30/d3e/da7/db5/l10c 0 2024-04-06T14:17:12.494 INFO:tasks.workunit.client.0.smithi032.stdout:9/749: write d2/d15/d1e/d65/d83/d7e/f14 [3085860,103170] 0 2024-04-06T14:17:12.496 INFO:tasks.workunit.client.0.smithi032.stdout:0/523: dread d3/da/f18 [0,4194304] 0 2024-04-06T14:17:12.502 INFO:tasks.workunit.client.0.smithi032.stdout:1/790: rmdir d0/da/d3e/de6 0 2024-04-06T14:17:12.503 INFO:tasks.workunit.client.0.smithi032.stdout:9/750: write d2/d15/d1e/f33 [1635227,21907] 0 2024-04-06T14:17:12.504 INFO:tasks.workunit.client.0.smithi032.stdout:1/791: creat d0/d41/d5b/d96/f101 x:0 0 0 2024-04-06T14:17:12.507 INFO:tasks.workunit.client.0.smithi032.stdout:1/792: mkdir d0/d9/df2/d102 0 2024-04-06T14:17:12.508 INFO:tasks.workunit.client.0.smithi032.stdout:9/751: creat d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/fef x:0 0 0 2024-04-06T14:17:12.512 INFO:tasks.workunit.client.0.smithi032.stdout:1/793: link d0/da/d3e/f50 d0/d10/d16/d3f/d98/f103 0 2024-04-06T14:17:12.512 INFO:tasks.workunit.client.0.smithi032.stdout:7/750: dread d16/d20/d54/de4/f31 [0,4194304] 0 2024-04-06T14:17:12.524 INFO:tasks.workunit.client.0.smithi032.stdout:8/746: dread d3/d8/d7/d14/f1f [8388608,4194304] 0 2024-04-06T14:17:12.524 INFO:tasks.workunit.client.0.smithi032.stdout:0/524: dread d3/da/f56 [0,4194304] 0 2024-04-06T14:17:12.526 INFO:tasks.workunit.client.0.smithi032.stdout:1/794: dread d0/d9/d1f/f93 [0,4194304] 0 2024-04-06T14:17:12.528 INFO:tasks.workunit.client.0.smithi032.stdout:1/795: read d0/d10/d16/d3f/d98/f103 [2513333,53757] 0 2024-04-06T14:17:12.529 INFO:tasks.workunit.client.0.smithi032.stdout:1/796: readlink d0/d41/d5b/l87 0 2024-04-06T14:17:12.530 INFO:tasks.workunit.client.1.smithi086.stdout:3/514: truncate d5/d36/f3b 5915164 0 2024-04-06T14:17:12.530 INFO:tasks.workunit.client.1.smithi086.stdout:3/515: write d5/df/f1c [8991328,53764] 0 2024-04-06T14:17:12.538 INFO:tasks.workunit.client.1.smithi086.stdout:0/535: mkdir d0/d15/d26/d33/d7a/daa 0 2024-04-06T14:17:12.539 INFO:tasks.workunit.client.1.smithi086.stdout:4/368: symlink d0/d2c/d30/d24/d50/l6a 0 2024-04-06T14:17:12.539 INFO:tasks.workunit.client.1.smithi086.stdout:0/536: write d0/d15/d4e/f92 [170860,71148] 0 2024-04-06T14:17:12.541 INFO:tasks.workunit.client.1.smithi086.stdout:6/519: symlink d6/d13/d42/lb7 0 2024-04-06T14:17:12.547 INFO:tasks.workunit.client.1.smithi086.stdout:8/586: unlink d0/d71/d16/d39/d90/f97 0 2024-04-06T14:17:12.548 INFO:tasks.workunit.client.1.smithi086.stdout:8/587: dread - d0/d71/d16/d52/fc8 zero size 2024-04-06T14:17:12.550 INFO:tasks.workunit.client.1.smithi086.stdout:3/516: mkdir d5/df/d9e 0 2024-04-06T14:17:12.551 INFO:tasks.workunit.client.1.smithi086.stdout:3/517: chown d5/d36/f67 12880 1 2024-04-06T14:17:12.554 INFO:tasks.workunit.client.1.smithi086.stdout:2/514: getdents d0 0 2024-04-06T14:17:12.554 INFO:tasks.workunit.client.1.smithi086.stdout:4/369: symlink d0/d2c/d38/d54/d61/l6b 0 2024-04-06T14:17:12.555 INFO:tasks.workunit.client.1.smithi086.stdout:9/419: dwrite d5/d13/f71 [0,4194304] 0 2024-04-06T14:17:12.556 INFO:tasks.workunit.client.1.smithi086.stdout:9/420: fdatasync d5/d13/f8c 0 2024-04-06T14:17:12.557 INFO:tasks.workunit.client.1.smithi086.stdout:4/370: stat d0/d15/d20/d34/c66 0 2024-04-06T14:17:12.558 INFO:tasks.workunit.client.1.smithi086.stdout:4/371: read d0/d15/d20/f27 [58532,31507] 0 2024-04-06T14:17:12.559 INFO:tasks.workunit.client.1.smithi086.stdout:6/520: unlink d6/fa8 0 2024-04-06T14:17:12.560 INFO:tasks.workunit.client.1.smithi086.stdout:1/520: mknod db/d11/d87/ca2 0 2024-04-06T14:17:12.560 INFO:tasks.workunit.client.0.smithi032.stdout:9/752: link d2/d15/d55/d59/d93/d95/db8/c8a d2/d15/d1e/d65/d83/d7e/cf0 0 2024-04-06T14:17:12.562 INFO:tasks.workunit.client.1.smithi086.stdout:6/521: truncate d6/d20/fb1 193790 0 2024-04-06T14:17:12.562 INFO:tasks.workunit.client.1.smithi086.stdout:7/452: dwrite d7/d10/d11/d1c/f60 [0,4194304] 0 2024-04-06T14:17:12.562 INFO:tasks.workunit.client.0.smithi032.stdout:0/525: mkdir d3/da/d25/d2b/d80/db3 0 2024-04-06T14:17:12.563 INFO:tasks.workunit.client.0.smithi032.stdout:8/747: readlink d3/d8/d12/d2f/d4a/d60/l9b 0 2024-04-06T14:17:12.563 INFO:tasks.workunit.client.0.smithi032.stdout:1/797: mkdir d0/d10/d66/d104 0 2024-04-06T14:17:12.563 INFO:tasks.workunit.client.0.smithi032.stdout:9/753: read d2/d15/d1e/d65/d6b/fb2 [601304,27809] 0 2024-04-06T14:17:12.564 INFO:tasks.workunit.client.0.smithi032.stdout:0/526: read - d3/da/f53 zero size 2024-04-06T14:17:12.565 INFO:tasks.workunit.client.1.smithi086.stdout:6/522: write d6/d13/d5f/d60/d45/d4a/f79 [3370706,33637] 0 2024-04-06T14:17:12.567 INFO:tasks.workunit.client.0.smithi032.stdout:0/527: chown d3/d83/dab 11 1 2024-04-06T14:17:12.567 INFO:tasks.workunit.client.0.smithi032.stdout:9/754: stat d2/d15/d1e/d65/d6b/d7f/cba 0 2024-04-06T14:17:12.569 INFO:tasks.workunit.client.0.smithi032.stdout:1/798: symlink d0/d9/d94/dc2/df1/l105 0 2024-04-06T14:17:12.571 INFO:tasks.workunit.client.0.smithi032.stdout:8/748: dread d3/d8/fac [0,4194304] 0 2024-04-06T14:17:12.571 INFO:tasks.workunit.client.1.smithi086.stdout:0/537: creat d0/d15/d26/d33/d9f/fab x:0 0 0 2024-04-06T14:17:12.571 INFO:tasks.workunit.client.1.smithi086.stdout:3/518: symlink d5/d26/d7f/d59/l9f 0 2024-04-06T14:17:12.573 INFO:tasks.workunit.client.1.smithi086.stdout:1/521: dwrite db/d11/d1a/d2a/d9a/d68/f7a [0,4194304] 0 2024-04-06T14:17:12.574 INFO:tasks.workunit.client.1.smithi086.stdout:3/519: chown d5/f50 9718553 1 2024-04-06T14:17:12.574 INFO:tasks.workunit.client.0.smithi032.stdout:0/528: symlink d3/da/d25/d2b/d4a/d79/da8/lb4 0 2024-04-06T14:17:12.574 INFO:tasks.workunit.client.0.smithi032.stdout:8/749: dread d3/d8/ff5 [0,4194304] 0 2024-04-06T14:17:12.575 INFO:tasks.workunit.client.1.smithi086.stdout:2/515: mkdir d0/d20/d27/d37/d52/db6 0 2024-04-06T14:17:12.576 INFO:tasks.workunit.client.0.smithi032.stdout:1/799: fsync d0/d9/fbe 0 2024-04-06T14:17:12.578 INFO:tasks.workunit.client.1.smithi086.stdout:9/421: mkdir d5/d58/d8d 0 2024-04-06T14:17:12.578 INFO:tasks.workunit.client.1.smithi086.stdout:2/516: write d0/d20/d27/f43 [1041637,106143] 0 2024-04-06T14:17:12.579 INFO:tasks.workunit.client.1.smithi086.stdout:8/588: unlink d0/d71/d22/d43/l91 0 2024-04-06T14:17:12.582 INFO:tasks.workunit.client.0.smithi032.stdout:8/750: write d3/d8/d12/d2f/d46/fce [605893,97303] 0 2024-04-06T14:17:12.584 INFO:tasks.workunit.client.1.smithi086.stdout:1/522: readlink db/d11/d1f/d5c/l77 0 2024-04-06T14:17:12.584 INFO:tasks.workunit.client.1.smithi086.stdout:7/453: dwrite d7/d10/d11/d1c/d36/f74 [0,4194304] 0 2024-04-06T14:17:12.584 INFO:tasks.workunit.client.0.smithi032.stdout:1/800: mkdir d0/d9/d1f/d7c/d106 0 2024-04-06T14:17:12.585 INFO:tasks.workunit.client.0.smithi032.stdout:8/751: write d3/d8/d12/f4d [1580483,343] 0 2024-04-06T14:17:12.587 INFO:tasks.workunit.client.0.smithi032.stdout:0/529: mkdir d3/da/d25/d57/db5 0 2024-04-06T14:17:12.589 INFO:tasks.workunit.client.1.smithi086.stdout:2/517: creat d0/df/d48/fb7 x:0 0 0 2024-04-06T14:17:12.595 INFO:tasks.workunit.client.0.smithi032.stdout:8/752: read - d3/d8/d12/d2f/d4a/d60/d82/f95 zero size 2024-04-06T14:17:12.595 INFO:tasks.workunit.client.0.smithi032.stdout:0/530: fdatasync d3/da/d25/d57/f7d 0 2024-04-06T14:17:12.595 INFO:tasks.workunit.client.1.smithi086.stdout:1/523: write db/d13/f86 [496355,47851] 0 2024-04-06T14:17:12.596 INFO:tasks.workunit.client.0.smithi032.stdout:1/801: symlink d0/l107 0 2024-04-06T14:17:12.597 INFO:tasks.workunit.client.1.smithi086.stdout:3/520: dread d5/f3e [0,4194304] 0 2024-04-06T14:17:12.598 INFO:tasks.workunit.client.1.smithi086.stdout:9/422: symlink d5/d13/d1a/d5f/d88/d14/d19/d25/d59/l8e 0 2024-04-06T14:17:12.598 INFO:tasks.workunit.client.1.smithi086.stdout:6/523: dwrite d6/d13/d1a/f8f [0,4194304] 0 2024-04-06T14:17:12.598 INFO:tasks.workunit.client.0.smithi032.stdout:8/753: rename d3/d8/d12/d2f/f62 to d3/d8/d3a/d7b/f108 0 2024-04-06T14:17:12.599 INFO:tasks.workunit.client.1.smithi086.stdout:7/454: mknod d7/d10/d11/d1c/d24/d2c/d69/c9e 0 2024-04-06T14:17:12.600 INFO:tasks.workunit.client.0.smithi032.stdout:8/754: dread - d3/d69/fe6 zero size 2024-04-06T14:17:12.601 INFO:tasks.workunit.client.0.smithi032.stdout:0/531: mknod d3/da/d35/d8b/cb6 0 2024-04-06T14:17:12.601 INFO:tasks.workunit.client.0.smithi032.stdout:1/802: mknod d0/da/c108 0 2024-04-06T14:17:12.602 INFO:tasks.workunit.client.1.smithi086.stdout:9/423: dread - d5/d13/f7c zero size 2024-04-06T14:17:12.603 INFO:tasks.workunit.client.0.smithi032.stdout:1/803: stat d0/d41/d59/f8e 0 2024-04-06T14:17:12.604 INFO:tasks.workunit.client.1.smithi086.stdout:3/521: readlink d5/d36/d65/l6c 0 2024-04-06T14:17:12.604 INFO:tasks.workunit.client.1.smithi086.stdout:4/372: dwrite d0/f36 [0,4194304] 0 2024-04-06T14:17:12.604 INFO:tasks.workunit.client.1.smithi086.stdout:2/518: dread d0/f2a [0,4194304] 0 2024-04-06T14:17:12.606 INFO:tasks.workunit.client.0.smithi032.stdout:1/804: mkdir d0/dc3/d109 0 2024-04-06T14:17:12.606 INFO:tasks.workunit.client.1.smithi086.stdout:4/373: stat d0/d15/d19/f52 0 2024-04-06T14:17:12.607 INFO:tasks.workunit.client.0.smithi032.stdout:0/532: mkdir d3/da/d25/d2b/d80/db3/db7 0 2024-04-06T14:17:12.608 INFO:tasks.workunit.client.1.smithi086.stdout:9/424: chown d5/d13/d1a/d5f/d88/d14/d19/f1e 60336 1 2024-04-06T14:17:12.608 INFO:tasks.workunit.client.0.smithi032.stdout:8/755: truncate d3/d8/d12/d2f/d4a/d60/d82/d84/fcc 727173 0 2024-04-06T14:17:12.609 INFO:tasks.workunit.client.1.smithi086.stdout:3/522: read d5/d36/d56/f74 [2482694,107208] 0 2024-04-06T14:17:12.609 INFO:tasks.workunit.client.0.smithi032.stdout:0/533: truncate d3/f14 1886879 0 2024-04-06T14:17:12.610 INFO:tasks.workunit.client.0.smithi032.stdout:1/805: mknod d0/d10/d16/d3f/d98/c10a 0 2024-04-06T14:17:12.611 INFO:tasks.workunit.client.0.smithi032.stdout:1/806: chown d0/d9/c54 15728 1 2024-04-06T14:17:12.612 INFO:tasks.workunit.client.0.smithi032.stdout:1/807: dread - d0/dc3/ff4 zero size 2024-04-06T14:17:12.613 INFO:tasks.workunit.client.0.smithi032.stdout:8/756: symlink d3/d8/d12/d2f/d4a/d60/d75/d85/l109 0 2024-04-06T14:17:12.615 INFO:tasks.workunit.client.0.smithi032.stdout:0/534: fsync d3/da/f1d 0 2024-04-06T14:17:12.615 INFO:tasks.workunit.client.1.smithi086.stdout:2/519: dread - d0/d20/f61 zero size 2024-04-06T14:17:12.616 INFO:tasks.workunit.client.1.smithi086.stdout:2/520: write d0/d20/d27/d2b/faf [261099,67545] 0 2024-04-06T14:17:12.617 INFO:tasks.workunit.client.1.smithi086.stdout:4/374: creat d0/d2c/d30/f6c x:0 0 0 2024-04-06T14:17:12.618 INFO:tasks.workunit.client.1.smithi086.stdout:2/521: stat d0/df/d48/d53/d76/f91 0 2024-04-06T14:17:12.618 INFO:tasks.workunit.client.0.smithi032.stdout:0/535: readlink d3/l15 0 2024-04-06T14:17:12.620 INFO:tasks.workunit.client.1.smithi086.stdout:1/524: rmdir db/d13/d9b 0 2024-04-06T14:17:12.620 INFO:tasks.workunit.client.1.smithi086.stdout:1/525: readlink db/l3c 0 2024-04-06T14:17:12.620 INFO:tasks.workunit.client.0.smithi032.stdout:8/757: rename d3/d8/d12/d2f/d4a/d60/d82/d84/c9f to d3/d8/d3a/d7b/db9/ddf/c10a 0 2024-04-06T14:17:12.620 INFO:tasks.workunit.client.1.smithi086.stdout:1/526: chown db/d11/d1a/f85 28927831 1 2024-04-06T14:17:12.621 INFO:tasks.workunit.client.1.smithi086.stdout:7/455: creat d7/d10/d92/f9f x:0 0 0 2024-04-06T14:17:12.622 INFO:tasks.workunit.client.1.smithi086.stdout:7/456: readlink d7/d10/d11/l6c 0 2024-04-06T14:17:12.624 INFO:tasks.workunit.client.1.smithi086.stdout:7/457: write d7/d10/d11/d1c/d24/d2c/d3c/f82 [297113,97331] 0 2024-04-06T14:17:12.625 INFO:tasks.workunit.client.0.smithi032.stdout:8/758: getdents d3/d8/d12/d2f/d4a/d60/d90/dc5 0 2024-04-06T14:17:12.626 INFO:tasks.workunit.client.1.smithi086.stdout:7/458: dread - d7/d10/d11/d1c/d24/d2c/d69/f78 zero size 2024-04-06T14:17:12.626 INFO:tasks.workunit.client.1.smithi086.stdout:2/522: mkdir d0/d20/d27/d2b/d35/d36/d4a/db8 0 2024-04-06T14:17:12.627 INFO:tasks.workunit.client.1.smithi086.stdout:2/523: chown d0/d20/d82/lb5 0 1 2024-04-06T14:17:12.628 INFO:tasks.workunit.client.0.smithi032.stdout:8/759: dread - d3/d8/d12/d2f/d4a/d60/d75/ff7 zero size 2024-04-06T14:17:12.628 INFO:tasks.workunit.client.1.smithi086.stdout:1/527: mkdir db/d13/d55/da3 0 2024-04-06T14:17:12.628 INFO:tasks.workunit.client.0.smithi032.stdout:7/751: write d16/d20/d54/de4/d2d/f41 [2644977,22664] 0 2024-04-06T14:17:12.630 INFO:tasks.workunit.client.0.smithi032.stdout:8/760: write d3/d8/d7/f16 [1945831,53743] 0 2024-04-06T14:17:12.633 INFO:tasks.workunit.client.1.smithi086.stdout:7/459: mkdir d7/d10/d11/d21/da0 0 2024-04-06T14:17:12.633 INFO:tasks.workunit.client.0.smithi032.stdout:7/752: symlink d16/d20/d63/l10d 0 2024-04-06T14:17:12.635 INFO:tasks.workunit.client.1.smithi086.stdout:2/524: rename d0/df/d48/f51 to d0/df/d48/d53/d76/fb9 0 2024-04-06T14:17:12.636 INFO:tasks.workunit.client.0.smithi032.stdout:0/536: dread d3/da/f26 [0,4194304] 0 2024-04-06T14:17:12.636 INFO:tasks.workunit.client.1.smithi086.stdout:1/528: dwrite db/d11/d1a/d2a/d52/f6d [0,4194304] 0 2024-04-06T14:17:12.637 INFO:tasks.workunit.client.1.smithi086.stdout:3/523: link d5/df/l95 d5/d36/d5c/la0 0 2024-04-06T14:17:12.637 INFO:tasks.workunit.client.0.smithi032.stdout:0/537: dread - d3/da/d25/d57/f7d zero size 2024-04-06T14:17:12.638 INFO:tasks.workunit.client.1.smithi086.stdout:1/529: stat db/d11/d1f/c21 0 2024-04-06T14:17:12.639 INFO:tasks.workunit.client.0.smithi032.stdout:7/753: getdents d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dde 0 2024-04-06T14:17:12.639 INFO:tasks.workunit.client.0.smithi032.stdout:0/538: readlink d3/da/d35/d4b/l4d 0 2024-04-06T14:17:12.639 INFO:tasks.workunit.client.0.smithi032.stdout:8/761: dread d3/d8/d12/d2f/d4a/d60/d75/dd1/ffd [0,4194304] 0 2024-04-06T14:17:12.642 INFO:tasks.workunit.client.1.smithi086.stdout:7/460: unlink d7/d10/d11/d1c/d24/d4d/f9a 0 2024-04-06T14:17:12.643 INFO:tasks.workunit.client.0.smithi032.stdout:7/754: mkdir d16/d20/d6e/d10e 0 2024-04-06T14:17:12.643 INFO:tasks.workunit.client.1.smithi086.stdout:2/525: chown d0/d20/d27/d2b/d35/d36/d4d/l57 488 1 2024-04-06T14:17:12.644 INFO:tasks.workunit.client.0.smithi032.stdout:8/762: rmdir d3/d8/d3a/d7b/db9/df8 39 2024-04-06T14:17:12.644 INFO:tasks.workunit.client.1.smithi086.stdout:5/422: write d0/d16/d18/d35/f3a [292489,62909] 0 2024-04-06T14:17:12.646 INFO:tasks.workunit.client.1.smithi086.stdout:5/423: dread - d0/d16/d18/d21/d23/f77 zero size 2024-04-06T14:17:12.648 INFO:tasks.workunit.client.1.smithi086.stdout:7/461: fsync d7/d1d/f3e 0 2024-04-06T14:17:12.651 INFO:tasks.workunit.client.1.smithi086.stdout:9/425: mkdir d5/d13/d1a/d8f 0 2024-04-06T14:17:12.652 INFO:tasks.workunit.client.0.smithi032.stdout:8/763: write d3/d8/d12/d2f/d4a/d60/d90/fe9 [1617243,83033] 0 2024-04-06T14:17:12.653 INFO:tasks.workunit.client.1.smithi086.stdout:1/530: mknod db/d11/d1a/d2a/d9a/ca4 0 2024-04-06T14:17:12.654 INFO:tasks.workunit.client.0.smithi032.stdout:0/539: dwrite d3/d83/fa5 [0,4194304] 0 2024-04-06T14:17:12.656 INFO:tasks.workunit.client.0.smithi032.stdout:1/808: dwrite d0/d9/d1f/d7c/f95 [0,4194304] 0 2024-04-06T14:17:12.656 INFO:tasks.workunit.client.1.smithi086.stdout:3/524: rename d5/df/f1c to d5/fa1 0 2024-04-06T14:17:12.658 INFO:tasks.workunit.client.1.smithi086.stdout:9/426: dread d5/d13/d1a/d5f/d88/d14/d19/d1b/f45 [0,4194304] 0 2024-04-06T14:17:12.660 INFO:tasks.workunit.client.1.smithi086.stdout:9/427: write d5/d13/d1a/d5f/d88/d14/f80 [915468,98014] 0 2024-04-06T14:17:12.662 INFO:tasks.workunit.client.1.smithi086.stdout:1/531: truncate db/d11/f27 999270 0 2024-04-06T14:17:12.666 INFO:tasks.workunit.client.0.smithi032.stdout:1/809: unlink d0/d9/l19 0 2024-04-06T14:17:12.667 INFO:tasks.workunit.client.1.smithi086.stdout:3/525: creat d5/d33/d5d/fa2 x:0 0 0 2024-04-06T14:17:12.667 INFO:tasks.workunit.client.0.smithi032.stdout:7/755: dwrite d16/d20/d54/f82 [4194304,4194304] 0 2024-04-06T14:17:12.668 INFO:tasks.workunit.client.1.smithi086.stdout:7/462: mkdir d7/d10/d11/d1c/d24/d2c/d3c/da1 0 2024-04-06T14:17:12.669 INFO:tasks.workunit.client.0.smithi032.stdout:7/756: dread - d16/d20/d63/d9d/df2/df5/fe8 zero size 2024-04-06T14:17:12.670 INFO:tasks.workunit.client.1.smithi086.stdout:2/526: creat d0/d20/d27/d2b/d35/d36/fba x:0 0 0 2024-04-06T14:17:12.673 INFO:tasks.workunit.client.1.smithi086.stdout:7/463: dread d7/d10/d11/d80/f46 [0,4194304] 0 2024-04-06T14:17:12.676 INFO:tasks.workunit.client.1.smithi086.stdout:7/464: dread d7/d10/d11/d1c/d24/f7d [0,4194304] 0 2024-04-06T14:17:12.676 INFO:tasks.workunit.client.1.smithi086.stdout:5/424: rename d0/d16/d18/c12 to d0/d16/d65/c83 0 2024-04-06T14:17:12.677 INFO:tasks.workunit.client.1.smithi086.stdout:2/527: dread - d0/d20/d27/d2b/d35/d36/d4d/d6f/f81 zero size 2024-04-06T14:17:12.678 INFO:tasks.workunit.client.0.smithi032.stdout:7/757: rename d16/d20/d54/de4/d5f/ld3 to d16/d20/d86/l10f 0 2024-04-06T14:17:12.679 INFO:tasks.workunit.client.1.smithi086.stdout:9/428: mkdir d5/d13/d1a/d5f/d88/d14/d90 0 2024-04-06T14:17:12.679 INFO:tasks.workunit.client.1.smithi086.stdout:9/429: dread - d5/d13/f8c zero size 2024-04-06T14:17:12.681 INFO:tasks.workunit.client.0.smithi032.stdout:7/758: mkdir d16/d20/d54/de4/d2d/d30/d3e/df3/d110 0 2024-04-06T14:17:12.683 INFO:tasks.workunit.client.1.smithi086.stdout:1/532: rename db/d11/d1a/d2a/d9a/d68/d6e/d81/d96 to db/d11/d1a/d79/da5 0 2024-04-06T14:17:12.684 INFO:tasks.workunit.client.1.smithi086.stdout:0/538: truncate d0/d1/d2/d1a/f56 1455005 0 2024-04-06T14:17:12.684 INFO:tasks.workunit.client.1.smithi086.stdout:1/533: readlink db/d11/d1a/d2a/d52/l7d 0 2024-04-06T14:17:12.686 INFO:tasks.workunit.client.0.smithi032.stdout:7/759: getdents d16/d20/d7a/da6 0 2024-04-06T14:17:12.689 INFO:tasks.workunit.client.0.smithi032.stdout:7/760: mknod d16/d20/d54/de4/d2d/d30/d3e/c111 0 2024-04-06T14:17:12.689 INFO:tasks.workunit.client.1.smithi086.stdout:6/524: write d6/d9/db/f16 [301762,61370] 0 2024-04-06T14:17:12.691 INFO:tasks.workunit.client.0.smithi032.stdout:7/761: symlink d16/d20/d54/de4/de0/df9/l112 0 2024-04-06T14:17:12.692 INFO:tasks.workunit.client.1.smithi086.stdout:8/589: dwrite d0/d13/f4b [4194304,4194304] 0 2024-04-06T14:17:12.695 INFO:tasks.workunit.client.1.smithi086.stdout:0/539: mkdir d0/d1/d2/d1a/d79/dac 0 2024-04-06T14:17:12.696 INFO:tasks.workunit.client.0.smithi032.stdout:7/762: link f10 d16/d20/d54/de4/d5f/ddd/f113 0 2024-04-06T14:17:12.697 INFO:tasks.workunit.client.1.smithi086.stdout:1/534: symlink db/d11/d1a/d2a/d9a/la6 0 2024-04-06T14:17:12.700 INFO:tasks.workunit.client.1.smithi086.stdout:7/465: read d7/d10/d11/d1c/d24/d2c/d3c/f6e [634362,92118] 0 2024-04-06T14:17:12.700 INFO:tasks.workunit.client.0.smithi032.stdout:7/763: dread d16/d20/d7a/f83 [0,4194304] 0 2024-04-06T14:17:12.701 INFO:tasks.workunit.client.1.smithi086.stdout:7/466: stat d7/d10/c34 0 2024-04-06T14:17:12.704 INFO:tasks.workunit.client.1.smithi086.stdout:4/375: write d0/f10 [36118,110137] 0 2024-04-06T14:17:12.704 INFO:tasks.workunit.client.0.smithi032.stdout:0/540: write d3/f76 [1673682,21517] 0 2024-04-06T14:17:12.704 INFO:tasks.workunit.client.1.smithi086.stdout:7/467: write d7/d10/d11/d1c/d24/d2c/d3c/f8a [451116,96880] 0 2024-04-06T14:17:12.706 INFO:tasks.workunit.client.1.smithi086.stdout:4/376: write d0/f36 [2434800,129144] 0 2024-04-06T14:17:12.706 INFO:tasks.workunit.client.1.smithi086.stdout:8/590: mknod d0/d3b/d79/dc6/dbc/ccd 0 2024-04-06T14:17:12.706 INFO:tasks.workunit.client.0.smithi032.stdout:8/764: write d3/d8/d12/d2f/d4a/d60/d82/f95 [161831,3782] 0 2024-04-06T14:17:12.707 INFO:tasks.workunit.client.0.smithi032.stdout:8/765: chown d3/ff6 201832 1 2024-04-06T14:17:12.708 INFO:tasks.workunit.client.1.smithi086.stdout:0/540: symlink d0/d15/lad 0 2024-04-06T14:17:12.709 INFO:tasks.workunit.client.0.smithi032.stdout:1/810: write d0/d9/d94/fce [8435,32869] 0 2024-04-06T14:17:12.710 INFO:tasks.workunit.client.0.smithi032.stdout:8/766: chown d3/d8/d7/d1a/f72 24479772 1 2024-04-06T14:17:12.711 INFO:tasks.workunit.client.0.smithi032.stdout:0/541: getdents d3/da/d25/d57 0 2024-04-06T14:17:12.711 INFO:tasks.workunit.client.1.smithi086.stdout:7/468: chown d7/fb 108 1 2024-04-06T14:17:12.712 INFO:tasks.workunit.client.0.smithi032.stdout:7/764: readlink d16/d20/d86/l10f 0 2024-04-06T14:17:12.713 INFO:tasks.workunit.client.0.smithi032.stdout:0/542: write d3/da/d25/d2b/f32 [6018121,57947] 0 2024-04-06T14:17:12.714 INFO:tasks.workunit.client.1.smithi086.stdout:4/377: symlink d0/de/l6d 0 2024-04-06T14:17:12.714 INFO:tasks.workunit.client.0.smithi032.stdout:8/767: write d3/d8/d12/d2f/d4a/d60/d82/d84/fdd [414444,109351] 0 2024-04-06T14:17:12.716 INFO:tasks.workunit.client.0.smithi032.stdout:0/543: fsync d3/da/d25/d2b/f3d 0 2024-04-06T14:17:12.717 INFO:tasks.workunit.client.0.smithi032.stdout:7/765: symlink d16/d20/d54/de4/d5f/l114 0 2024-04-06T14:17:12.717 INFO:tasks.workunit.client.1.smithi086.stdout:0/541: unlink d0/d1/d2/f80 0 2024-04-06T14:17:12.720 INFO:tasks.workunit.client.0.smithi032.stdout:8/768: mkdir d3/d8/d12/d2f/d4a/d60/d10b 0 2024-04-06T14:17:12.720 INFO:tasks.workunit.client.0.smithi032.stdout:0/544: dread d3/da/f26 [0,4194304] 0 2024-04-06T14:17:12.721 INFO:tasks.workunit.client.0.smithi032.stdout:7/766: symlink d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dde/l115 0 2024-04-06T14:17:12.722 INFO:tasks.workunit.client.1.smithi086.stdout:4/378: symlink d0/d2c/d30/d24/d50/l6e 0 2024-04-06T14:17:12.725 INFO:tasks.workunit.client.0.smithi032.stdout:7/767: chown d16/d20/d86/l10f 78 1 2024-04-06T14:17:12.725 INFO:tasks.workunit.client.0.smithi032.stdout:8/769: symlink d3/d8/d12/d2f/d4a/d60/d75/l10c 0 2024-04-06T14:17:12.728 INFO:tasks.workunit.client.0.smithi032.stdout:7/768: dread d16/f49 [0,4194304] 0 2024-04-06T14:17:12.729 INFO:tasks.workunit.client.0.smithi032.stdout:8/770: creat d3/d8/d7/f10d x:0 0 0 2024-04-06T14:17:12.729 INFO:tasks.workunit.client.0.smithi032.stdout:0/545: truncate d3/f14 502665 0 2024-04-06T14:17:12.730 INFO:tasks.workunit.client.0.smithi032.stdout:7/769: creat d16/d20/d48/d8d/f116 x:0 0 0 2024-04-06T14:17:12.732 INFO:tasks.workunit.client.0.smithi032.stdout:8/771: mknod d3/d8/d7/c10e 0 2024-04-06T14:17:12.733 INFO:tasks.workunit.client.0.smithi032.stdout:0/546: mknod d3/da/d25/cb8 0 2024-04-06T14:17:12.733 INFO:tasks.workunit.client.1.smithi086.stdout:9/430: write d5/d13/d47/f83 [162805,92226] 0 2024-04-06T14:17:12.734 INFO:tasks.workunit.client.0.smithi032.stdout:8/772: creat d3/d8/d12/d83/d89/da0/f10f x:0 0 0 2024-04-06T14:17:12.735 INFO:tasks.workunit.client.0.smithi032.stdout:0/547: creat d3/da/d25/d2b/d4a/fb9 x:0 0 0 2024-04-06T14:17:12.736 INFO:tasks.workunit.client.1.smithi086.stdout:1/535: getdents db/d11/d1a/d2a/d9a 0 2024-04-06T14:17:12.737 INFO:tasks.workunit.client.1.smithi086.stdout:2/528: dwrite d0/d20/f23 [0,4194304] 0 2024-04-06T14:17:12.738 INFO:tasks.workunit.client.0.smithi032.stdout:0/548: truncate d3/da/d25/d2b/d4a/f65 8470147 0 2024-04-06T14:17:12.739 INFO:tasks.workunit.client.1.smithi086.stdout:7/469: rename d7/d10/d11/d1c/d24/d2c/f39 to d7/d10/d11/d1c/d24/fa2 0 2024-04-06T14:17:12.742 INFO:tasks.workunit.client.0.smithi032.stdout:0/549: creat d3/da/d35/fba x:0 0 0 2024-04-06T14:17:12.743 INFO:tasks.workunit.client.0.smithi032.stdout:8/773: dread d3/d8/d3a/d7b/f108 [0,4194304] 0 2024-04-06T14:17:12.747 INFO:tasks.workunit.client.1.smithi086.stdout:5/425: dwrite d0/d16/d18/d2a/d50/f7b [0,4194304] 0 2024-04-06T14:17:12.749 INFO:tasks.workunit.client.1.smithi086.stdout:2/529: mkdir d0/d20/d44/dbb 0 2024-04-06T14:17:12.750 INFO:tasks.workunit.client.1.smithi086.stdout:4/379: link d0/d2c/d30/c2e d0/d2c/d30/d24/d50/c6f 0 2024-04-06T14:17:12.751 INFO:tasks.workunit.client.1.smithi086.stdout:6/525: link d6/d20/f46 d6/d13/d5f/fb8 0 2024-04-06T14:17:12.753 INFO:tasks.workunit.client.0.smithi032.stdout:8/774: dwrite d3/d8/f35 [0,4194304] 0 2024-04-06T14:17:12.758 INFO:tasks.workunit.client.0.smithi032.stdout:8/775: unlink d3/d8/d7/f10d 0 2024-04-06T14:17:12.759 INFO:tasks.workunit.client.0.smithi032.stdout:0/550: write d3/da/f53 [540289,71648] 0 2024-04-06T14:17:12.760 INFO:tasks.workunit.client.0.smithi032.stdout:0/551: stat d3/l74 0 2024-04-06T14:17:12.761 INFO:tasks.workunit.client.0.smithi032.stdout:0/552: stat d3/da/d35/l7f 0 2024-04-06T14:17:12.764 INFO:tasks.workunit.client.0.smithi032.stdout:0/553: symlink d3/da/d25/d2b/d4a/d79/d7c/lbb 0 2024-04-06T14:17:12.764 INFO:tasks.workunit.client.0.smithi032.stdout:0/554: stat d3/da/d25/d2b/l3f 0 2024-04-06T14:17:12.767 INFO:tasks.workunit.client.1.smithi086.stdout:1/536: dwrite db/d11/d1a/d2a/d9a/d68/d6e/f8f [0,4194304] 0 2024-04-06T14:17:12.768 INFO:tasks.workunit.client.1.smithi086.stdout:7/470: creat d7/d10/d11/d1c/d24/d4d/d5c/d66/fa3 x:0 0 0 2024-04-06T14:17:12.768 INFO:tasks.workunit.client.1.smithi086.stdout:6/526: write d6/d9/d43/f74 [450628,83389] 0 2024-04-06T14:17:12.769 INFO:tasks.workunit.client.0.smithi032.stdout:0/555: creat d3/da/d35/fbc x:0 0 0 2024-04-06T14:17:12.769 INFO:tasks.workunit.client.1.smithi086.stdout:5/426: mknod d0/d16/c84 0 2024-04-06T14:17:12.769 INFO:tasks.workunit.client.1.smithi086.stdout:9/431: write d5/d18/f32 [692177,114887] 0 2024-04-06T14:17:12.772 INFO:tasks.workunit.client.0.smithi032.stdout:8/776: dwrite d3/d8/d12/d83/fbb [0,4194304] 0 2024-04-06T14:17:12.773 INFO:tasks.workunit.client.1.smithi086.stdout:7/471: chown d7/d10/d11/d1c/l26 9 1 2024-04-06T14:17:12.775 INFO:tasks.workunit.client.1.smithi086.stdout:1/537: creat db/d13/d71/fa7 x:0 0 0 2024-04-06T14:17:12.776 INFO:tasks.workunit.client.0.smithi032.stdout:0/556: rmdir d3/da/d25/d2b/d4a/d79/d97 39 2024-04-06T14:17:12.781 INFO:tasks.workunit.client.1.smithi086.stdout:9/432: mkdir d5/d13/d1a/d5f/d88/d91 0 2024-04-06T14:17:12.782 INFO:tasks.workunit.client.0.smithi032.stdout:0/557: fsync d3/da/d35/f4e 0 2024-04-06T14:17:12.783 INFO:tasks.workunit.client.1.smithi086.stdout:8/591: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:17:12.783 INFO:tasks.workunit.client.1.smithi086.stdout:2/530: getdents d0/d20/d27/d2b 0 2024-04-06T14:17:12.785 INFO:tasks.workunit.client.0.smithi032.stdout:0/558: chown d3/f55 419 1 2024-04-06T14:17:12.785 INFO:tasks.workunit.client.1.smithi086.stdout:7/472: truncate d7/d10/d11/d21/f8e 289136 0 2024-04-06T14:17:12.786 INFO:tasks.workunit.client.1.smithi086.stdout:1/538: read db/d11/d1a/d2a/f43 [2271827,96191] 0 2024-04-06T14:17:12.787 INFO:tasks.workunit.client.0.smithi032.stdout:8/777: dwrite d3/d8/d12/d2f/d4a/d60/d82/d84/fd0 [0,4194304] 0 2024-04-06T14:17:12.788 INFO:tasks.workunit.client.0.smithi032.stdout:8/778: chown d3/f79 22985 1 2024-04-06T14:17:12.788 INFO:tasks.workunit.client.0.smithi032.stdout:8/779: chown d3/d8/d3a 529270756 1 2024-04-06T14:17:12.790 INFO:tasks.workunit.client.1.smithi086.stdout:2/531: mkdir d0/df/dbc 0 2024-04-06T14:17:12.790 INFO:tasks.workunit.client.1.smithi086.stdout:6/527: mknod d6/d13/d5f/d60/d45/d4a/d62/cb9 0 2024-04-06T14:17:12.791 INFO:tasks.workunit.client.1.smithi086.stdout:4/380: dwrite d0/d15/d2b/f5f [0,4194304] 0 2024-04-06T14:17:12.793 INFO:tasks.workunit.client.1.smithi086.stdout:8/592: mknod d0/d71/d22/d43/d6d/d75/d78/cce 0 2024-04-06T14:17:12.793 INFO:tasks.workunit.client.1.smithi086.stdout:0/542: dread d0/d1/d2/d1a/f56 [0,4194304] 0 2024-04-06T14:17:12.794 INFO:tasks.workunit.client.1.smithi086.stdout:1/539: rename db/d11/d1d/f58 to db/d11/d42/fa8 0 2024-04-06T14:17:12.799 INFO:tasks.workunit.client.0.smithi032.stdout:8/780: fsync d3/d8/d12/f19 0 2024-04-06T14:17:12.803 INFO:tasks.workunit.client.1.smithi086.stdout:6/528: fdatasync d6/d9/db/f19 0 2024-04-06T14:17:12.803 INFO:tasks.workunit.client.1.smithi086.stdout:4/381: symlink d0/de/d4c/l70 0 2024-04-06T14:17:12.805 INFO:tasks.workunit.client.0.smithi032.stdout:8/781: rmdir d3/d8/d12/d2f/d4a/d60/d90/dc5 39 2024-04-06T14:17:12.805 INFO:tasks.workunit.client.1.smithi086.stdout:0/543: chown d0/d15/d4e/da3/d46/d52/c72 813 1 2024-04-06T14:17:12.806 INFO:tasks.workunit.client.1.smithi086.stdout:0/544: read d0/d1/f76 [381707,67074] 0 2024-04-06T14:17:12.810 INFO:tasks.workunit.client.0.smithi032.stdout:0/559: dwrite d3/da/d25/d2b/d4a/f62 [0,4194304] 0 2024-04-06T14:17:12.811 INFO:tasks.workunit.client.1.smithi086.stdout:5/427: dread d0/d43/f5c [0,4194304] 0 2024-04-06T14:17:12.811 INFO:tasks.workunit.client.0.smithi032.stdout:0/560: dread d3/f55 [0,4194304] 0 2024-04-06T14:17:12.814 INFO:tasks.workunit.client.0.smithi032.stdout:0/561: creat d3/da/d25/d36/fbd x:0 0 0 2024-04-06T14:17:12.818 INFO:tasks.workunit.client.0.smithi032.stdout:0/562: creat d3/da/d25/d2b/d4a/fbe x:0 0 0 2024-04-06T14:17:12.818 INFO:tasks.workunit.client.0.smithi032.stdout:8/782: dread d3/ff6 [0,4194304] 0 2024-04-06T14:17:12.819 INFO:tasks.workunit.client.0.smithi032.stdout:0/563: chown d3/da/d35/d8b/d9c/la3 416871 1 2024-04-06T14:17:12.819 INFO:tasks.workunit.client.0.smithi032.stdout:8/783: dread - d3/d8/d12/d83/d89/da0/ffe zero size 2024-04-06T14:17:12.822 INFO:tasks.workunit.client.0.smithi032.stdout:0/564: creat d3/da/d25/d2b/d4a/d79/d7c/fbf x:0 0 0 2024-04-06T14:17:12.825 INFO:tasks.workunit.client.0.smithi032.stdout:8/784: dwrite d3/d8/d12/d2f/d4a/f94 [4194304,4194304] 0 2024-04-06T14:17:12.826 INFO:tasks.workunit.client.1.smithi086.stdout:7/473: rmdir d7/d10/d11/d1c/d24/d4d/d5c 39 2024-04-06T14:17:12.826 INFO:tasks.workunit.client.0.smithi032.stdout:0/565: mkdir d3/da/d25/d2b/d80/db3/db7/dc0 0 2024-04-06T14:17:12.828 INFO:tasks.workunit.client.1.smithi086.stdout:1/540: write db/d11/d42/d53/f64 [357454,83941] 0 2024-04-06T14:17:12.830 INFO:tasks.workunit.client.1.smithi086.stdout:4/382: mkdir d0/d2c/d30/d24/d71 0 2024-04-06T14:17:12.831 INFO:tasks.workunit.client.0.smithi032.stdout:0/566: chown d3/da/d25/d2b/d4a/d86/cb1 64600 1 2024-04-06T14:17:12.833 INFO:tasks.workunit.client.1.smithi086.stdout:0/545: rmdir d0/d1 39 2024-04-06T14:17:12.836 INFO:tasks.workunit.client.1.smithi086.stdout:8/593: creat d0/d3b/d79/dc6/dbc/dc1/fcf x:0 0 0 2024-04-06T14:17:12.836 INFO:tasks.workunit.client.0.smithi032.stdout:8/785: dwrite d3/d8/d12/d2f/d4a/d60/d90/ff2 [0,4194304] 0 2024-04-06T14:17:12.836 INFO:tasks.workunit.client.1.smithi086.stdout:0/546: read d0/d15/f6a [2187397,65416] 0 2024-04-06T14:17:12.837 INFO:tasks.workunit.client.1.smithi086.stdout:7/474: mkdir d7/d10/d11/d1c/d24/d2c/d3c/d7a/da4 0 2024-04-06T14:17:12.838 INFO:tasks.workunit.client.1.smithi086.stdout:6/529: symlink d6/d13/d5f/d60/d45/da6/lba 0 2024-04-06T14:17:12.839 INFO:tasks.workunit.client.1.smithi086.stdout:1/541: read - db/d11/d42/f50 zero size 2024-04-06T14:17:12.839 INFO:tasks.workunit.client.0.smithi032.stdout:8/786: rmdir d3 39 2024-04-06T14:17:12.842 INFO:tasks.workunit.client.1.smithi086.stdout:5/428: dwrite d0/d43/f5c [0,4194304] 0 2024-04-06T14:17:12.845 INFO:tasks.workunit.client.1.smithi086.stdout:7/475: read d7/d10/d11/d1c/d24/d2c/d3c/f6e [86162,32985] 0 2024-04-06T14:17:12.845 INFO:tasks.workunit.client.0.smithi032.stdout:8/787: dread d3/d8/d3a/f5c [0,4194304] 0 2024-04-06T14:17:12.846 INFO:tasks.workunit.client.1.smithi086.stdout:4/383: fdatasync d0/f40 0 2024-04-06T14:17:12.848 INFO:tasks.workunit.client.0.smithi032.stdout:8/788: chown d3/d8/d7/d14/l73 633420 1 2024-04-06T14:17:12.849 INFO:tasks.workunit.client.1.smithi086.stdout:6/530: fsync d6/d13/d1a/f21 0 2024-04-06T14:17:12.849 INFO:tasks.workunit.client.1.smithi086.stdout:4/384: readlink d0/de/d4c/l70 0 2024-04-06T14:17:12.849 INFO:tasks.workunit.client.1.smithi086.stdout:5/429: mkdir d0/d17/d2d/d80/d85 0 2024-04-06T14:17:12.850 INFO:tasks.workunit.client.0.smithi032.stdout:0/567: dwrite d3/f10 [4194304,4194304] 0 2024-04-06T14:17:12.850 INFO:tasks.workunit.client.0.smithi032.stdout:8/789: fsync d3/f32 0 2024-04-06T14:17:12.855 INFO:tasks.workunit.client.1.smithi086.stdout:7/476: dread - d7/d10/d11/d80/f4b zero size 2024-04-06T14:17:12.858 INFO:tasks.workunit.client.0.smithi032.stdout:8/790: creat d3/d8/d12/d2f/d4a/d60/d90/dc5/f110 x:0 0 0 2024-04-06T14:17:12.858 INFO:tasks.workunit.client.1.smithi086.stdout:0/547: mkdir d0/d1/d4/de/dae 0 2024-04-06T14:17:12.859 INFO:tasks.workunit.client.0.smithi032.stdout:8/791: read d3/ff6 [775813,11187] 0 2024-04-06T14:17:12.861 INFO:tasks.workunit.client.1.smithi086.stdout:6/531: symlink d6/d20/lbb 0 2024-04-06T14:17:12.861 INFO:tasks.workunit.client.0.smithi032.stdout:8/792: chown d3/d8/d12/d2f/d46/c6a 11 1 2024-04-06T14:17:12.862 INFO:tasks.workunit.client.1.smithi086.stdout:5/430: readlink d0/d16/d18/d21/d23/d28/l38 0 2024-04-06T14:17:12.865 INFO:tasks.workunit.client.1.smithi086.stdout:6/532: creat d6/d9/d43/fbc x:0 0 0 2024-04-06T14:17:12.865 INFO:tasks.workunit.client.0.smithi032.stdout:0/568: truncate d3/da/d35/d8b/f90 1346191 0 2024-04-06T14:17:12.865 INFO:tasks.workunit.client.0.smithi032.stdout:8/793: symlink d3/d8/d3a/d7b/db9/df8/d106/l111 0 2024-04-06T14:17:12.867 INFO:tasks.workunit.client.0.smithi032.stdout:0/569: write d3/da/d25/d2b/f32 [690768,60590] 0 2024-04-06T14:17:12.868 INFO:tasks.workunit.client.1.smithi086.stdout:5/431: chown d0/d16/d18/c1d 50778 1 2024-04-06T14:17:12.868 INFO:tasks.workunit.client.1.smithi086.stdout:0/548: mknod d0/d1/d4/caf 0 2024-04-06T14:17:12.869 INFO:tasks.workunit.client.1.smithi086.stdout:7/477: link d7/d10/d11/d80/l90 d7/d94/la5 0 2024-04-06T14:17:12.869 INFO:tasks.workunit.client.1.smithi086.stdout:6/533: dread d6/d9/db/f28 [0,4194304] 0 2024-04-06T14:17:12.870 INFO:tasks.workunit.client.0.smithi032.stdout:0/570: creat d3/da/d35/fc1 x:0 0 0 2024-04-06T14:17:12.871 INFO:tasks.workunit.client.0.smithi032.stdout:8/794: rename d3/d8/d12/d2f/d4a/d60/d82/d84/lca to d3/d8/d3a/d7b/db9/ddf/l112 0 2024-04-06T14:17:12.872 INFO:tasks.workunit.client.0.smithi032.stdout:0/571: mkdir d3/dc2 0 2024-04-06T14:17:12.874 INFO:tasks.workunit.client.0.smithi032.stdout:8/795: truncate d3/d8/d12/d83/d89/da0/ffe 492500 0 2024-04-06T14:17:12.877 INFO:tasks.workunit.client.1.smithi086.stdout:1/542: dread db/d13/d55/f6a [0,4194304] 0 2024-04-06T14:17:12.881 INFO:tasks.workunit.client.1.smithi086.stdout:0/549: getdents d0/d15/d4e/da3 0 2024-04-06T14:17:12.882 INFO:tasks.workunit.client.1.smithi086.stdout:1/543: rmdir db/d13/d71 39 2024-04-06T14:17:12.884 INFO:tasks.workunit.client.1.smithi086.stdout:6/534: getdents d6/d9/d63 0 2024-04-06T14:17:12.884 INFO:tasks.workunit.client.1.smithi086.stdout:8/594: write d0/db/d32/f3e [981351,28707] 0 2024-04-06T14:17:12.887 INFO:tasks.workunit.client.1.smithi086.stdout:1/544: creat db/d11/d1a/d79/fa9 x:0 0 0 2024-04-06T14:17:12.889 INFO:tasks.workunit.client.1.smithi086.stdout:6/535: mkdir d6/d13/d5f/d60/d45/da6/dbd 0 2024-04-06T14:17:12.891 INFO:tasks.workunit.client.1.smithi086.stdout:6/536: stat d6/d9/d63/ca2 0 2024-04-06T14:17:12.892 INFO:tasks.workunit.client.1.smithi086.stdout:1/545: creat db/d11/d87/faa x:0 0 0 2024-04-06T14:17:12.894 INFO:tasks.workunit.client.1.smithi086.stdout:7/478: dread d7/d10/d11/d1c/d24/d4d/d5c/d66/f55 [0,4194304] 0 2024-04-06T14:17:12.895 INFO:tasks.workunit.client.1.smithi086.stdout:1/546: readlink db/d11/d1f/l2d 0 2024-04-06T14:17:12.895 INFO:tasks.workunit.client.1.smithi086.stdout:0/550: getdents d0/d15/d26/d33/d9f 0 2024-04-06T14:17:12.897 INFO:tasks.workunit.client.1.smithi086.stdout:1/547: stat db/d11/d87/ca2 0 2024-04-06T14:17:12.900 INFO:tasks.workunit.client.1.smithi086.stdout:1/548: write db/d11/d42/fa8 [5332453,122899] 0 2024-04-06T14:17:12.906 INFO:tasks.workunit.client.1.smithi086.stdout:7/479: dread - d7/d10/d11/d1c/d24/d2c/f48 zero size 2024-04-06T14:17:12.906 INFO:tasks.workunit.client.1.smithi086.stdout:5/432: dread d0/d16/d18/d35/f4d [0,4194304] 0 2024-04-06T14:17:12.906 INFO:tasks.workunit.client.1.smithi086.stdout:6/537: rename d6/d20/f51 to d6/d13/d5f/d60/d45/d73/db2/fbe 0 2024-04-06T14:17:12.908 INFO:tasks.workunit.client.1.smithi086.stdout:1/549: creat db/d11/d1a/d2a/d9a/d68/d6e/d81/fab x:0 0 0 2024-04-06T14:17:12.908 INFO:tasks.workunit.client.1.smithi086.stdout:0/551: write d0/d1/d4/d7/f14 [599248,64198] 0 2024-04-06T14:17:12.913 INFO:tasks.workunit.client.1.smithi086.stdout:7/480: write d7/d10/d11/d1c/d36/f74 [1129156,76357] 0 2024-04-06T14:17:12.914 INFO:tasks.workunit.client.1.smithi086.stdout:0/552: creat d0/d1/d4/de/fb0 x:0 0 0 2024-04-06T14:17:12.917 INFO:tasks.workunit.client.1.smithi086.stdout:8/595: dwrite d0/d13/f7c [0,4194304] 0 2024-04-06T14:17:12.918 INFO:tasks.workunit.client.1.smithi086.stdout:7/481: chown d7/d1d/l3b 286905596 1 2024-04-06T14:17:12.924 INFO:tasks.workunit.client.1.smithi086.stdout:5/433: mknod d0/d16/d18/d21/c86 0 2024-04-06T14:17:12.925 INFO:tasks.workunit.client.1.smithi086.stdout:8/596: fsync d0/d3b/f51 0 2024-04-06T14:17:12.925 INFO:tasks.workunit.client.1.smithi086.stdout:7/482: creat d7/d10/d11/d1c/d24/d2c/d69/fa6 x:0 0 0 2024-04-06T14:17:12.930 INFO:tasks.workunit.client.1.smithi086.stdout:8/597: truncate d0/d71/f3c 1627286 0 2024-04-06T14:17:12.931 INFO:tasks.workunit.client.1.smithi086.stdout:6/538: dwrite d6/d13/d1a/d64/fb3 [0,4194304] 0 2024-04-06T14:17:12.934 INFO:tasks.workunit.client.1.smithi086.stdout:0/553: write d0/d15/d4e/f45 [1216295,29614] 0 2024-04-06T14:17:12.935 INFO:tasks.workunit.client.1.smithi086.stdout:6/539: mknod d6/d13/d5f/d60/d45/d4a/cbf 0 2024-04-06T14:17:12.941 INFO:tasks.workunit.client.1.smithi086.stdout:8/598: rename d0/d71/d22/d65/d6e/d84 to d0/d71/d16/d52/dd0 0 2024-04-06T14:17:12.941 INFO:tasks.workunit.client.1.smithi086.stdout:7/483: link d7/d10/d11/d1c/d24/d4d/d5c/d66/c9c d7/ca7 0 2024-04-06T14:17:12.943 INFO:tasks.workunit.client.1.smithi086.stdout:1/550: dwrite db/d11/d1a/d2a/d9a/d78/f9f [0,4194304] 0 2024-04-06T14:17:12.950 INFO:tasks.workunit.client.1.smithi086.stdout:8/599: mknod d0/d5a/cd1 0 2024-04-06T14:17:12.951 INFO:tasks.workunit.client.1.smithi086.stdout:7/484: symlink d7/d1d/la8 0 2024-04-06T14:17:12.951 INFO:tasks.workunit.client.1.smithi086.stdout:0/554: creat d0/d1/d2/d5b/fb1 x:0 0 0 2024-04-06T14:17:12.952 INFO:tasks.workunit.client.1.smithi086.stdout:5/434: write d0/d19/f3c [2528404,44] 0 2024-04-06T14:17:12.954 INFO:tasks.workunit.client.1.smithi086.stdout:0/555: dread - d0/d1/d2/d1a/d51/f86 zero size 2024-04-06T14:17:12.956 INFO:tasks.workunit.client.1.smithi086.stdout:5/435: write d0/d16/f47 [2634403,66613] 0 2024-04-06T14:17:12.959 INFO:tasks.workunit.client.1.smithi086.stdout:0/556: fdatasync d0/d1/d4/de/f2e 0 2024-04-06T14:17:12.959 INFO:tasks.workunit.client.1.smithi086.stdout:7/485: truncate d7/d10/d11/d1c/d24/d2c/d69/f78 409364 0 2024-04-06T14:17:12.962 INFO:tasks.workunit.client.1.smithi086.stdout:5/436: dread - d0/d16/d18/d21/d23/f6c zero size 2024-04-06T14:17:12.966 INFO:tasks.workunit.client.1.smithi086.stdout:0/557: fdatasync d0/d15/d4e/f42 0 2024-04-06T14:17:12.967 INFO:tasks.workunit.client.1.smithi086.stdout:7/486: rename d7/d10/d11/d1c/d24/d2c/f63 to d7/d94/fa9 0 2024-04-06T14:17:12.969 INFO:tasks.workunit.client.1.smithi086.stdout:5/437: creat d0/d16/d18/d35/f87 x:0 0 0 2024-04-06T14:17:12.969 INFO:tasks.workunit.client.1.smithi086.stdout:0/558: write d0/d1/d2/d5b/fb1 [953214,21381] 0 2024-04-06T14:17:12.971 INFO:tasks.workunit.client.1.smithi086.stdout:1/551: write db/d13/f3d [2802279,40040] 0 2024-04-06T14:17:12.972 INFO:tasks.workunit.client.1.smithi086.stdout:6/540: dread d6/d13/d1a/d22/f35 [0,4194304] 0 2024-04-06T14:17:12.975 INFO:tasks.workunit.client.1.smithi086.stdout:7/487: truncate d7/fb 2373917 0 2024-04-06T14:17:12.976 INFO:tasks.workunit.client.1.smithi086.stdout:1/552: chown db/d11/d1a/d2a/d9a/d68 161 1 2024-04-06T14:17:12.978 INFO:tasks.workunit.client.1.smithi086.stdout:8/600: dread d0/d71/d22/d43/f28 [0,4194304] 0 2024-04-06T14:17:12.978 INFO:tasks.workunit.client.1.smithi086.stdout:7/488: chown d7/d10/d11/d1c/d24/f4e 59 1 2024-04-06T14:17:12.978 INFO:tasks.workunit.client.1.smithi086.stdout:0/559: creat d0/d1/d2/d1a/d79/fb2 x:0 0 0 2024-04-06T14:17:12.980 INFO:tasks.workunit.client.1.smithi086.stdout:1/553: symlink db/d11/d1a/d79/da5/lac 0 2024-04-06T14:17:12.983 INFO:tasks.workunit.client.1.smithi086.stdout:8/601: creat d0/d71/d22/d43/d6d/d75/db2/fd2 x:0 0 0 2024-04-06T14:17:12.985 INFO:tasks.workunit.client.1.smithi086.stdout:0/560: creat d0/d1/d2/d1a/d79/fb3 x:0 0 0 2024-04-06T14:17:12.987 INFO:tasks.workunit.client.1.smithi086.stdout:0/561: chown d0/d15/d4e/da3/d46/d52/f7f 3295818 1 2024-04-06T14:17:12.989 INFO:tasks.workunit.client.1.smithi086.stdout:7/489: creat d7/d10/d11/d1c/d24/d2c/faa x:0 0 0 2024-04-06T14:17:12.989 INFO:tasks.workunit.client.1.smithi086.stdout:5/438: write d0/f6 [4151677,51445] 0 2024-04-06T14:17:12.991 INFO:tasks.workunit.client.1.smithi086.stdout:6/541: link d6/d13/d5f/d60/d45/d4a/l81 d6/d9/da9/lc0 0 2024-04-06T14:17:12.992 INFO:tasks.workunit.client.1.smithi086.stdout:7/490: chown d7/d10/d11/d1c/d24/d4d/d5c/d66/f67 1404 1 2024-04-06T14:17:12.992 INFO:tasks.workunit.client.1.smithi086.stdout:0/562: mkdir d0/d1/d4/de/db4 0 2024-04-06T14:17:12.997 INFO:tasks.workunit.client.1.smithi086.stdout:6/542: read d6/f7 [2025626,64668] 0 2024-04-06T14:17:13.001 INFO:tasks.workunit.client.1.smithi086.stdout:0/563: fsync d0/d1/d2/d1a/f56 0 2024-04-06T14:17:13.002 INFO:tasks.workunit.client.1.smithi086.stdout:8/602: link d0/f26 d0/d71/d22/d43/d6d/d75/d7d/fd3 0 2024-04-06T14:17:13.006 INFO:tasks.workunit.client.1.smithi086.stdout:5/439: dread d0/d16/d18/d2a/f46 [0,4194304] 0 2024-04-06T14:17:13.008 INFO:tasks.workunit.client.0.smithi032.stdout:0/572: write d3/f20 [1181391,9096] 0 2024-04-06T14:17:13.008 INFO:tasks.workunit.client.1.smithi086.stdout:0/564: chown d0/d1/d2/d1a/d79/fb2 626 1 2024-04-06T14:17:13.009 INFO:tasks.workunit.client.1.smithi086.stdout:7/491: dwrite d7/d10/d11/d1c/d24/fa2 [0,4194304] 0 2024-04-06T14:17:13.009 INFO:tasks.workunit.client.1.smithi086.stdout:6/543: dread d6/d9/d5c/f6b [0,4194304] 0 2024-04-06T14:17:13.014 INFO:tasks.workunit.client.0.smithi032.stdout:0/573: mknod d3/da/d25/d2b/d4a/d79/d7c/cc3 0 2024-04-06T14:17:13.014 INFO:tasks.workunit.client.0.smithi032.stdout:8/796: truncate d3/d8/d7/d14/f44 851412 0 2024-04-06T14:17:13.016 INFO:tasks.workunit.client.1.smithi086.stdout:8/603: getdents d0/d71/d22/d43/d7f 0 2024-04-06T14:17:13.017 INFO:tasks.workunit.client.0.smithi032.stdout:0/574: fdatasync d3/da/d35/f4e 0 2024-04-06T14:17:13.017 INFO:tasks.workunit.client.1.smithi086.stdout:8/604: chown d0/d71/d22/d43/d4d/da3/dc7 57 1 2024-04-06T14:17:13.018 INFO:tasks.workunit.client.1.smithi086.stdout:0/565: creat d0/d1/d4/de/db4/fb5 x:0 0 0 2024-04-06T14:17:13.018 INFO:tasks.workunit.client.0.smithi032.stdout:0/575: dread - d3/da/d25/d2b/d4a/fbe zero size 2024-04-06T14:17:13.019 INFO:tasks.workunit.client.1.smithi086.stdout:6/544: creat d6/d13/d42/d52/fc1 x:0 0 0 2024-04-06T14:17:13.019 INFO:tasks.workunit.client.1.smithi086.stdout:0/566: chown d0/d1/d4/f43 35672 1 2024-04-06T14:17:13.020 INFO:tasks.workunit.client.1.smithi086.stdout:7/492: rmdir d7/d10/d11/d21/da0 0 2024-04-06T14:17:13.020 INFO:tasks.workunit.client.1.smithi086.stdout:6/545: fdatasync d6/d9/db/f70 0 2024-04-06T14:17:13.022 INFO:tasks.workunit.client.1.smithi086.stdout:0/567: write d0/d1/d2/d1a/d79/fb3 [233840,7152] 0 2024-04-06T14:17:13.023 INFO:tasks.workunit.client.1.smithi086.stdout:7/493: chown d7/d10/d11/d1c/d24/d2c/d69/f78 651797193 1 2024-04-06T14:17:13.024 INFO:tasks.workunit.client.0.smithi032.stdout:8/797: creat d3/d8/d12/d2f/d4a/d60/d90/f113 x:0 0 0 2024-04-06T14:17:13.025 INFO:tasks.workunit.client.1.smithi086.stdout:6/546: dread - d6/d13/faf zero size 2024-04-06T14:17:13.025 INFO:tasks.workunit.client.1.smithi086.stdout:7/494: fdatasync d7/d10/d11/f8f 0 2024-04-06T14:17:13.027 INFO:tasks.workunit.client.1.smithi086.stdout:6/547: read - d6/d13/d5f/d60/d45/fa1 zero size 2024-04-06T14:17:13.027 INFO:tasks.workunit.client.0.smithi032.stdout:8/798: stat d3/d8/d12/d2f/d4a/d60/d75/dd1/fff 0 2024-04-06T14:17:13.028 INFO:tasks.workunit.client.0.smithi032.stdout:0/576: dwrite d3/da/f6f [0,4194304] 0 2024-04-06T14:17:13.029 INFO:tasks.workunit.client.1.smithi086.stdout:0/568: creat d0/d15/d4e/da3/d46/d52/fb6 x:0 0 0 2024-04-06T14:17:13.030 INFO:tasks.workunit.client.0.smithi032.stdout:8/799: write d3/d8/d12/d2f/d4a/d60/d90/fd9 [158616,25945] 0 2024-04-06T14:17:13.034 INFO:tasks.workunit.client.1.smithi086.stdout:6/548: creat d6/d13/d5f/d60/d45/d8a/fc2 x:0 0 0 2024-04-06T14:17:13.035 INFO:tasks.workunit.client.0.smithi032.stdout:8/800: chown d3/d8/d3a/d7b/db9/ddf/l112 392802 1 2024-04-06T14:17:13.035 INFO:tasks.workunit.client.1.smithi086.stdout:6/549: chown d6/d13/d42/d52 9627728 1 2024-04-06T14:17:13.035 INFO:tasks.workunit.client.1.smithi086.stdout:7/495: creat d7/d10/d11/d1c/d24/d2c/d69/d83/fab x:0 0 0 2024-04-06T14:17:13.036 INFO:tasks.workunit.client.1.smithi086.stdout:0/569: mkdir d0/d1/d2/d1a/d79/dac/db7 0 2024-04-06T14:17:13.037 INFO:tasks.workunit.client.0.smithi032.stdout:0/577: dwrite d3/da/f6f [0,4194304] 0 2024-04-06T14:17:13.037 INFO:tasks.workunit.client.1.smithi086.stdout:7/496: truncate d7/f77 1044683 0 2024-04-06T14:17:13.042 INFO:tasks.workunit.client.1.smithi086.stdout:6/550: rename d6/d13/d5f/d60/c84 to d6/d9/d43/cc3 0 2024-04-06T14:17:13.043 INFO:tasks.workunit.client.0.smithi032.stdout:0/578: rmdir d3/da/d25/d57 39 2024-04-06T14:17:13.045 INFO:tasks.workunit.client.0.smithi032.stdout:0/579: chown d3/da/d35/fbc 11638 1 2024-04-06T14:17:13.046 INFO:tasks.workunit.client.1.smithi086.stdout:8/605: dwrite d0/d71/d22/d43/d6d/d75/d7d/fd3 [0,4194304] 0 2024-04-06T14:17:13.046 INFO:tasks.workunit.client.1.smithi086.stdout:6/551: readlink d6/d9/d63/l6f 0 2024-04-06T14:17:13.047 INFO:tasks.workunit.client.1.smithi086.stdout:0/570: mknod d0/d1/d2/d1a/d51/cb8 0 2024-04-06T14:17:13.047 INFO:tasks.workunit.client.1.smithi086.stdout:8/606: write d0/d13/db6/fba [567637,14498] 0 2024-04-06T14:17:13.052 INFO:tasks.workunit.client.1.smithi086.stdout:0/571: symlink d0/d15/d26/lb9 0 2024-04-06T14:17:13.053 INFO:tasks.workunit.client.1.smithi086.stdout:8/607: write d0/d71/d22/d43/d4d/fc4 [630061,51619] 0 2024-04-06T14:17:13.054 INFO:tasks.workunit.client.0.smithi032.stdout:8/801: write d3/d8/d7/fbc [778735,95024] 0 2024-04-06T14:17:13.057 INFO:tasks.workunit.client.1.smithi086.stdout:0/572: write d0/d1/d4/de/fb0 [318660,12373] 0 2024-04-06T14:17:13.057 INFO:tasks.workunit.client.0.smithi032.stdout:8/802: readlink d3/d8/d3a/d7b/db9/ddf/l112 0 2024-04-06T14:17:13.058 INFO:tasks.workunit.client.1.smithi086.stdout:0/573: read d0/d15/d26/d33/f87 [128473,72093] 0 2024-04-06T14:17:13.061 INFO:tasks.workunit.client.1.smithi086.stdout:6/552: dwrite d6/d13/d5f/d60/d45/d73/db2/fa7 [0,4194304] 0 2024-04-06T14:17:13.061 INFO:tasks.workunit.client.0.smithi032.stdout:8/803: truncate d3/d8/d3a/f4f 336436 0 2024-04-06T14:17:13.062 INFO:tasks.workunit.client.1.smithi086.stdout:0/574: readlink d0/d1/d4/d7/l59 0 2024-04-06T14:17:13.064 INFO:tasks.workunit.client.0.smithi032.stdout:0/580: dread d3/da/d25/f2a [0,4194304] 0 2024-04-06T14:17:13.064 INFO:tasks.workunit.client.0.smithi032.stdout:8/804: mknod d3/d8/d12/d2f/d4a/d60/d10b/c114 0 2024-04-06T14:17:13.065 INFO:tasks.workunit.client.0.smithi032.stdout:0/581: chown d3/da/f53 256097259 1 2024-04-06T14:17:13.066 INFO:tasks.workunit.client.0.smithi032.stdout:0/582: dread - d3/f5d zero size 2024-04-06T14:17:13.069 INFO:tasks.workunit.client.1.smithi086.stdout:0/575: write d0/d15/d26/d33/f87 [181171,105640] 0 2024-04-06T14:17:13.072 INFO:tasks.workunit.client.1.smithi086.stdout:0/576: creat d0/d15/d4e/d48/d89/fba x:0 0 0 2024-04-06T14:17:13.072 INFO:tasks.workunit.client.0.smithi032.stdout:0/583: dwrite d3/f63 [0,4194304] 0 2024-04-06T14:17:13.076 INFO:tasks.workunit.client.1.smithi086.stdout:0/577: symlink d0/d1/d4/lbb 0 2024-04-06T14:17:13.076 INFO:tasks.workunit.client.0.smithi032.stdout:0/584: creat d3/da/d25/d2b/d4a/d86/fc4 x:0 0 0 2024-04-06T14:17:13.077 INFO:tasks.workunit.client.1.smithi086.stdout:0/578: write d0/d1/d4/de/f3c [818032,58542] 0 2024-04-06T14:17:13.078 INFO:tasks.workunit.client.1.smithi086.stdout:0/579: dread - d0/d15/f8d zero size 2024-04-06T14:17:13.080 INFO:tasks.workunit.client.0.smithi032.stdout:0/585: dread - d3/da/d25/d2b/d4a/d79/d97/faf zero size 2024-04-06T14:17:13.082 INFO:tasks.workunit.client.0.smithi032.stdout:0/586: creat d3/d83/fc5 x:0 0 0 2024-04-06T14:17:13.086 INFO:tasks.workunit.client.0.smithi032.stdout:0/587: symlink d3/da/d41/lc6 0 2024-04-06T14:17:13.088 INFO:tasks.workunit.client.1.smithi086.stdout:7/497: dwrite d7/d10/d11/d1c/d24/d2c/f6f [0,4194304] 0 2024-04-06T14:17:13.088 INFO:tasks.workunit.client.0.smithi032.stdout:8/805: dwrite d3/d8/d12/d2f/f39 [0,4194304] 0 2024-04-06T14:17:13.090 INFO:tasks.workunit.client.1.smithi086.stdout:8/608: dwrite d0/d71/d22/d43/f28 [0,4194304] 0 2024-04-06T14:17:13.093 INFO:tasks.workunit.client.0.smithi032.stdout:8/806: truncate d3/d8/d12/d2f/d4a/d60/d90/fb6 755161 0 2024-04-06T14:17:13.095 INFO:tasks.workunit.client.0.smithi032.stdout:0/588: link d3/da/d35/f4e d3/da/d35/d8b/d9c/fc7 0 2024-04-06T14:17:13.096 INFO:tasks.workunit.client.1.smithi086.stdout:8/609: creat d0/d3b/d79/dc6/fd4 x:0 0 0 2024-04-06T14:17:13.097 INFO:tasks.workunit.client.0.smithi032.stdout:8/807: creat d3/d8/d12/d83/f115 x:0 0 0 2024-04-06T14:17:13.099 INFO:tasks.workunit.client.0.smithi032.stdout:0/589: read d3/f5a [56135,55031] 0 2024-04-06T14:17:13.100 INFO:tasks.workunit.client.0.smithi032.stdout:0/590: stat d3/da/d35/d8b/d9c/la3 0 2024-04-06T14:17:13.100 INFO:tasks.workunit.client.1.smithi086.stdout:7/498: dwrite d7/f81 [0,4194304] 0 2024-04-06T14:17:13.101 INFO:tasks.workunit.client.1.smithi086.stdout:8/610: write d0/d71/d22/d43/d6d/d75/d78/fb0 [320174,100626] 0 2024-04-06T14:17:13.102 INFO:tasks.workunit.client.1.smithi086.stdout:6/553: dread d6/d9/d43/f53 [0,4194304] 0 2024-04-06T14:17:13.107 INFO:tasks.workunit.client.1.smithi086.stdout:8/611: mknod d0/d71/d93/cd5 0 2024-04-06T14:17:13.108 INFO:tasks.workunit.client.1.smithi086.stdout:8/612: chown d0/d3b/db9 358473664 1 2024-04-06T14:17:13.112 INFO:tasks.workunit.client.0.smithi032.stdout:8/808: dwrite d3/d8/d12/d2f/d4a/d60/d75/d85/fee [0,4194304] 0 2024-04-06T14:17:13.115 INFO:tasks.workunit.client.0.smithi032.stdout:8/809: symlink d3/d8/d7/d1a/l116 0 2024-04-06T14:17:13.118 INFO:tasks.workunit.client.0.smithi032.stdout:8/810: creat d3/d8/d7/d14/f117 x:0 0 0 2024-04-06T14:17:13.123 INFO:tasks.workunit.client.0.smithi032.stdout:0/591: dread d3/da/f1d [0,4194304] 0 2024-04-06T14:17:13.126 INFO:tasks.workunit.client.1.smithi086.stdout:7/499: truncate d7/d10/d11/d80/f4a 2212214 0 2024-04-06T14:17:13.127 INFO:tasks.workunit.client.0.smithi032.stdout:0/592: mkdir d3/da/d25/d57/db5/dc8 0 2024-04-06T14:17:13.129 INFO:tasks.workunit.client.0.smithi032.stdout:0/593: mkdir d3/da/d25/d57/db5/dc9 0 2024-04-06T14:17:13.130 INFO:tasks.workunit.client.1.smithi086.stdout:7/500: symlink d7/d10/d11/d1c/d24/d2c/d3c/lac 0 2024-04-06T14:17:13.131 INFO:tasks.workunit.client.1.smithi086.stdout:6/554: dwrite d6/d13/f77 [0,4194304] 0 2024-04-06T14:17:13.131 INFO:tasks.workunit.client.0.smithi032.stdout:0/594: write d3/da/faa [2899803,112919] 0 2024-04-06T14:17:13.132 INFO:tasks.workunit.client.0.smithi032.stdout:8/811: write d3/d8/d12/d2f/d4a/d60/d90/fb6 [932672,70016] 0 2024-04-06T14:17:13.134 INFO:tasks.workunit.client.0.smithi032.stdout:0/595: stat d3/da/d25/c58 0 2024-04-06T14:17:13.135 INFO:tasks.workunit.client.1.smithi086.stdout:7/501: creat d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/fad x:0 0 0 2024-04-06T14:17:13.138 INFO:tasks.workunit.client.0.smithi032.stdout:0/596: symlink d3/da/d25/d57/db5/lca 0 2024-04-06T14:17:13.139 INFO:tasks.workunit.client.1.smithi086.stdout:6/555: dwrite d6/d13/f77 [4194304,4194304] 0 2024-04-06T14:17:13.143 INFO:tasks.workunit.client.1.smithi086.stdout:7/502: dread d7/d10/d11/d1c/d24/d2c/f70 [0,4194304] 0 2024-04-06T14:17:13.144 INFO:tasks.workunit.client.0.smithi032.stdout:8/812: truncate d3/d8/d7/f47 4342360 0 2024-04-06T14:17:13.146 INFO:tasks.workunit.client.1.smithi086.stdout:7/503: write d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/f89 [661775,90702] 0 2024-04-06T14:17:13.147 INFO:tasks.workunit.client.1.smithi086.stdout:6/556: write d6/d13/d1a/d22/f6e [1753554,35466] 0 2024-04-06T14:17:13.148 INFO:tasks.workunit.client.0.smithi032.stdout:8/813: dread - d3/d8/d7/d1a/f72 zero size 2024-04-06T14:17:13.150 INFO:tasks.workunit.client.1.smithi086.stdout:7/504: unlink d7/d10/d11/d1c/d24/f75 0 2024-04-06T14:17:13.151 INFO:tasks.workunit.client.1.smithi086.stdout:7/505: write d7/f81 [2474155,121492] 0 2024-04-06T14:17:13.151 INFO:tasks.workunit.client.1.smithi086.stdout:6/557: dread - d6/d13/d5f/d60/f6d zero size 2024-04-06T14:17:13.155 INFO:tasks.workunit.client.1.smithi086.stdout:6/558: mknod d6/d13/d5f/d60/d45/d8a/cc4 0 2024-04-06T14:17:13.156 INFO:tasks.workunit.client.1.smithi086.stdout:7/506: truncate d7/d10/d11/d1c/d24/d2c/d3c/d7a/f93 854840 0 2024-04-06T14:17:13.156 INFO:tasks.workunit.client.0.smithi032.stdout:8/814: link d3/d8/d7/d1a/l116 d3/d8/d12/d83/d89/da0/l118 0 2024-04-06T14:17:13.159 INFO:tasks.workunit.client.0.smithi032.stdout:8/815: symlink d3/d8/d12/d2f/d4a/d60/d90/dc5/dd7/l119 0 2024-04-06T14:17:13.160 INFO:tasks.workunit.client.1.smithi086.stdout:7/507: dread d7/d10/d11/d1c/d24/d2c/d3c/f6e [0,4194304] 0 2024-04-06T14:17:13.163 INFO:tasks.workunit.client.1.smithi086.stdout:6/559: dwrite d6/d13/d5f/fb5 [0,4194304] 0 2024-04-06T14:17:13.166 INFO:tasks.workunit.client.1.smithi086.stdout:6/560: rmdir d6/d13/d5f/d60/d45/d8a 39 2024-04-06T14:17:13.170 INFO:tasks.workunit.client.1.smithi086.stdout:6/561: dread d6/d13/d5f/d60/d45/d73/db2/fa7 [0,4194304] 0 2024-04-06T14:17:13.178 INFO:tasks.workunit.client.1.smithi086.stdout:6/562: dwrite d6/d13/d5f/d60/d45/d73/db2/fbe [0,4194304] 0 2024-04-06T14:17:13.183 INFO:tasks.workunit.client.1.smithi086.stdout:6/563: dread d6/d9/d43/f53 [0,4194304] 0 2024-04-06T14:17:13.187 INFO:tasks.workunit.client.1.smithi086.stdout:6/564: dread d6/d13/d5f/d60/d45/d73/db2/fa7 [0,4194304] 0 2024-04-06T14:17:13.189 INFO:tasks.workunit.client.1.smithi086.stdout:6/565: creat d6/d9/d43/fc5 x:0 0 0 2024-04-06T14:17:13.190 INFO:tasks.workunit.client.1.smithi086.stdout:6/566: getdents d6/d13/d5f/d60/d45/da6/dbd 0 2024-04-06T14:17:13.192 INFO:tasks.workunit.client.1.smithi086.stdout:6/567: mknod d6/d9/db/cc6 0 2024-04-06T14:17:13.193 INFO:tasks.workunit.client.1.smithi086.stdout:6/568: dread d6/d9/f17 [0,4194304] 0 2024-04-06T14:17:13.195 INFO:tasks.workunit.client.1.smithi086.stdout:6/569: write d6/d13/d5f/d60/d45/d73/db2/fa7 [4933755,81903] 0 2024-04-06T14:17:13.226 INFO:tasks.workunit.client.0.smithi032.stdout:6/717: sync 2024-04-06T14:17:13.230 INFO:tasks.workunit.client.0.smithi032.stdout:6/718: link d4/f98 d4/d5/d19/d25/d67/ff7 0 2024-04-06T14:17:13.253 INFO:tasks.workunit.client.0.smithi032.stdout:6/719: dwrite d4/fc7 [4194304,4194304] 0 2024-04-06T14:17:13.258 INFO:tasks.workunit.client.0.smithi032.stdout:6/720: write d4/d5/dbf/d51/fa8 [473197,31557] 0 2024-04-06T14:17:13.262 INFO:tasks.workunit.client.1.smithi086.stdout:6/570: dread d6/d13/fa4 [0,4194304] 0 2024-04-06T14:17:13.268 INFO:tasks.workunit.client.1.smithi086.stdout:6/571: dwrite d6/d13/f77 [0,4194304] 0 2024-04-06T14:17:13.268 INFO:tasks.workunit.client.0.smithi032.stdout:6/721: dwrite d4/d5/d28/f29 [0,4194304] 0 2024-04-06T14:17:13.270 INFO:tasks.workunit.client.0.smithi032.stdout:6/722: write d4/d5/d19/d96/faa [1330853,125907] 0 2024-04-06T14:17:13.273 INFO:tasks.workunit.client.1.smithi086.stdout:6/572: rmdir d6/d13/d5f/d60/d45/da6/dbd 0 2024-04-06T14:17:13.276 INFO:tasks.workunit.client.1.smithi086.stdout:6/573: mknod d6/d13/d42/d52/cc7 0 2024-04-06T14:17:13.277 INFO:tasks.workunit.client.1.smithi086.stdout:6/574: read d6/d13/d1a/d22/f6e [1772466,112885] 0 2024-04-06T14:17:13.280 INFO:tasks.workunit.client.1.smithi086.stdout:6/575: symlink d6/d9/d5c/lc8 0 2024-04-06T14:17:13.280 INFO:tasks.workunit.client.1.smithi086.stdout:6/576: readlink d6/d9/d5c/lc8 0 2024-04-06T14:17:13.284 INFO:tasks.workunit.client.1.smithi086.stdout:6/577: unlink d6/d9/d43/c94 0 2024-04-06T14:17:13.287 INFO:tasks.workunit.client.1.smithi086.stdout:6/578: getdents d6/d9 0 2024-04-06T14:17:13.291 INFO:tasks.workunit.client.1.smithi086.stdout:6/579: unlink d6/d13/f27 0 2024-04-06T14:17:13.293 INFO:tasks.workunit.client.1.smithi086.stdout:6/580: write d6/d13/fb0 [2932177,103047] 0 2024-04-06T14:17:13.296 INFO:tasks.workunit.client.1.smithi086.stdout:6/581: symlink d6/d13/d5f/d60/d45/da6/lc9 0 2024-04-06T14:17:13.296 INFO:tasks.workunit.client.0.smithi032.stdout:6/723: dread d4/d5/f15 [0,4194304] 0 2024-04-06T14:17:13.298 INFO:tasks.workunit.client.0.smithi032.stdout:6/724: write d4/d5/d28/d65/fb1 [245033,5973] 0 2024-04-06T14:17:13.299 INFO:tasks.workunit.client.1.smithi086.stdout:6/582: mknod d6/d9/da9/cca 0 2024-04-06T14:17:13.299 INFO:tasks.workunit.client.0.smithi032.stdout:6/725: chown d4/d5/dbf/d51/l26 34323935 1 2024-04-06T14:17:13.300 INFO:tasks.workunit.client.1.smithi086.stdout:6/583: rmdir d6/d13/d1a/d22 39 2024-04-06T14:17:13.301 INFO:tasks.workunit.client.1.smithi086.stdout:6/584: readlink d6/d20/la5 0 2024-04-06T14:17:13.303 INFO:tasks.workunit.client.0.smithi032.stdout:6/726: write d4/d5/dc5/d82/fb0 [59545,24991] 0 2024-04-06T14:17:13.305 INFO:tasks.workunit.client.1.smithi086.stdout:6/585: dread d6/d13/d1a/d64/fb3 [0,4194304] 0 2024-04-06T14:17:13.307 INFO:tasks.workunit.client.0.smithi032.stdout:6/727: rename d4/d5/dbf/d51/d5e to d4/d5/d28/d65/df8 0 2024-04-06T14:17:13.309 INFO:tasks.workunit.client.0.smithi032.stdout:6/728: truncate d4/d5/f4c 1824066 0 2024-04-06T14:17:13.309 INFO:tasks.workunit.client.1.smithi086.stdout:6/586: rename d6/d9/db/f19 to d6/d13/d5f/d60/d45/d4a/fcb 0 2024-04-06T14:17:13.310 INFO:tasks.workunit.client.1.smithi086.stdout:6/587: read d6/d13/d1a/d64/f71 [537907,46485] 0 2024-04-06T14:17:13.311 INFO:tasks.workunit.client.0.smithi032.stdout:6/729: creat d4/d5/d19/d21/ff9 x:0 0 0 2024-04-06T14:17:13.313 INFO:tasks.workunit.client.0.smithi032.stdout:6/730: write d4/d5/d14/f1a [2547262,115551] 0 2024-04-06T14:17:13.313 INFO:tasks.workunit.client.1.smithi086.stdout:6/588: creat d6/d13/d1a/da0/fcc x:0 0 0 2024-04-06T14:17:13.315 INFO:tasks.workunit.client.0.smithi032.stdout:6/731: mknod d4/d5/d28/d65/d6e/cfa 0 2024-04-06T14:17:13.315 INFO:tasks.workunit.client.1.smithi086.stdout:6/589: symlink d6/d13/d5f/d60/lcd 0 2024-04-06T14:17:13.316 INFO:tasks.workunit.client.0.smithi032.stdout:6/732: fdatasync d4/d5/d19/d96/faa 0 2024-04-06T14:17:13.316 INFO:tasks.workunit.client.1.smithi086.stdout:6/590: read d6/f7 [2610914,95071] 0 2024-04-06T14:17:13.322 INFO:tasks.workunit.client.0.smithi032.stdout:6/733: dread d4/d5/dbf/f33 [0,4194304] 0 2024-04-06T14:17:13.323 INFO:tasks.workunit.client.1.smithi086.stdout:6/591: dwrite d6/d13/d1a/da0/fcc [0,4194304] 0 2024-04-06T14:17:13.323 INFO:tasks.workunit.client.0.smithi032.stdout:6/734: write d4/d5/d19/d21/ff9 [861027,87225] 0 2024-04-06T14:17:13.329 INFO:tasks.workunit.client.1.smithi086.stdout:6/592: link d6/d9/da9/lc0 d6/d9/d63/lce 0 2024-04-06T14:17:13.338 INFO:tasks.workunit.client.0.smithi032.stdout:6/735: dwrite d4/d5/dbf/f33 [0,4194304] 0 2024-04-06T14:17:13.341 INFO:tasks.workunit.client.0.smithi032.stdout:6/736: truncate d4/d5/d28/f5c 4448186 0 2024-04-06T14:17:13.343 INFO:tasks.workunit.client.1.smithi086.stdout:6/593: dwrite d6/d9/db/f28 [0,4194304] 0 2024-04-06T14:17:13.343 INFO:tasks.workunit.client.0.smithi032.stdout:6/737: rmdir d4/d5/d28/d65 39 2024-04-06T14:17:13.344 INFO:tasks.workunit.client.0.smithi032.stdout:6/738: write d4/d5/d35/fce [73497,68599] 0 2024-04-06T14:17:13.345 INFO:tasks.workunit.client.0.smithi032.stdout:6/739: chown d4/d5/l1e 11 1 2024-04-06T14:17:13.348 INFO:tasks.workunit.client.0.smithi032.stdout:6/740: mkdir d4/d5/d8/d2b/dfb 0 2024-04-06T14:17:13.349 INFO:tasks.workunit.client.0.smithi032.stdout:6/741: stat d4/d5/d8/ld 0 2024-04-06T14:17:13.351 INFO:tasks.workunit.client.1.smithi086.stdout:6/594: dwrite d6/d13/f77 [0,4194304] 0 2024-04-06T14:17:13.355 INFO:tasks.workunit.client.0.smithi032.stdout:6/742: rename d4/d5/d18/fd9 to d4/d5/d28/d65/ffc 0 2024-04-06T14:17:13.359 INFO:tasks.workunit.client.1.smithi086.stdout:6/595: rename d6/d13/d5f/d60/d45/d73 to d6/d13/d1a/d22/dcf 0 2024-04-06T14:17:13.376 INFO:tasks.workunit.client.0.smithi032.stdout:6/743: dread d4/d5/d18/f52 [0,4194304] 0 2024-04-06T14:17:13.390 INFO:tasks.workunit.client.0.smithi032.stdout:6/744: dread d4/d5/dbf/f63 [0,4194304] 0 2024-04-06T14:17:13.395 INFO:tasks.workunit.client.0.smithi032.stdout:6/745: truncate d4/d5/d18/f52 1268646 0 2024-04-06T14:17:13.401 INFO:tasks.workunit.client.0.smithi032.stdout:6/746: dwrite d4/d5/d19/d25/d67/f8a [0,4194304] 0 2024-04-06T14:17:13.402 INFO:tasks.workunit.client.0.smithi032.stdout:6/747: read - d4/d5/d19/f46 zero size 2024-04-06T14:17:13.411 INFO:tasks.workunit.client.0.smithi032.stdout:6/748: dwrite d4/d5/dbf/fcd [0,4194304] 0 2024-04-06T14:17:13.418 INFO:tasks.workunit.client.0.smithi032.stdout:6/749: creat d4/d5/d8/d2b/ffd x:0 0 0 2024-04-06T14:17:13.418 INFO:tasks.workunit.client.0.smithi032.stdout:6/750: readlink d4/d5/dc5/l7f 0 2024-04-06T14:17:13.658 INFO:tasks.workunit.client.0.smithi032.stdout:7/770: sync 2024-04-06T14:17:13.658 INFO:tasks.workunit.client.0.smithi032.stdout:9/755: sync 2024-04-06T14:17:13.658 INFO:tasks.workunit.client.0.smithi032.stdout:2/672: sync 2024-04-06T14:17:13.658 INFO:tasks.workunit.client.0.smithi032.stdout:4/741: sync 2024-04-06T14:17:13.658 INFO:tasks.workunit.client.0.smithi032.stdout:3/688: sync 2024-04-06T14:17:13.658 INFO:tasks.workunit.client.0.smithi032.stdout:8/816: sync 2024-04-06T14:17:13.658 INFO:tasks.workunit.client.0.smithi032.stdout:6/751: sync 2024-04-06T14:17:13.658 INFO:tasks.workunit.client.0.smithi032.stdout:1/811: sync 2024-04-06T14:17:13.658 INFO:tasks.workunit.client.0.smithi032.stdout:5/681: sync 2024-04-06T14:17:13.659 INFO:tasks.workunit.client.0.smithi032.stdout:0/597: sync 2024-04-06T14:17:13.660 INFO:tasks.workunit.client.0.smithi032.stdout:1/812: rmdir d0/d41/d5b 39 2024-04-06T14:17:13.662 INFO:tasks.workunit.client.0.smithi032.stdout:7/771: chown d16/d20/d63/d9d/df2/df5/fe9 5014 1 2024-04-06T14:17:13.662 INFO:tasks.workunit.client.0.smithi032.stdout:3/689: mknod d3/d65/d89/c108 0 2024-04-06T14:17:13.663 INFO:tasks.workunit.client.0.smithi032.stdout:4/742: creat d0/d82/d8e/ffc x:0 0 0 2024-04-06T14:17:13.664 INFO:tasks.workunit.client.0.smithi032.stdout:5/682: creat d16/d18/d33/d45/d52/ff0 x:0 0 0 2024-04-06T14:17:13.664 INFO:tasks.workunit.client.0.smithi032.stdout:0/598: truncate d3/f78 803740 0 2024-04-06T14:17:13.665 INFO:tasks.workunit.client.0.smithi032.stdout:1/813: rmdir d0/da/d3e 39 2024-04-06T14:17:13.666 INFO:tasks.workunit.client.0.smithi032.stdout:2/673: getdents d0/d6e/d8d 0 2024-04-06T14:17:13.666 INFO:tasks.workunit.client.0.smithi032.stdout:8/817: symlink d3/d8/d3a/d7b/db9/df8/d106/l11a 0 2024-04-06T14:17:13.667 INFO:tasks.workunit.client.0.smithi032.stdout:7/772: mkdir d16/d20/d7a/da6/d117 0 2024-04-06T14:17:13.668 INFO:tasks.workunit.client.0.smithi032.stdout:5/683: rmdir d16/d58 39 2024-04-06T14:17:13.670 INFO:tasks.workunit.client.0.smithi032.stdout:0/599: fsync d3/da/f64 0 2024-04-06T14:17:13.672 INFO:tasks.workunit.client.0.smithi032.stdout:2/674: creat d0/d30/d39/d59/fda x:0 0 0 2024-04-06T14:17:13.674 INFO:tasks.workunit.client.0.smithi032.stdout:6/752: dwrite d4/d5/d28/d65/fb1 [0,4194304] 0 2024-04-06T14:17:13.674 INFO:tasks.workunit.client.0.smithi032.stdout:3/690: creat d3/d16/ddd/f109 x:0 0 0 2024-04-06T14:17:13.674 INFO:tasks.workunit.client.0.smithi032.stdout:4/743: truncate d0/df/d23/fac 1313864 0 2024-04-06T14:17:13.675 INFO:tasks.workunit.client.0.smithi032.stdout:0/600: write d3/da/d25/d2b/d4a/d79/d97/faf [799004,20092] 0 2024-04-06T14:17:13.677 INFO:tasks.workunit.client.0.smithi032.stdout:9/756: getdents d2/d15/d1e/d65/d83/d31/d39/da6 0 2024-04-06T14:17:13.677 INFO:tasks.workunit.client.0.smithi032.stdout:1/814: getdents d0/d10/d16/d3f/def 0 2024-04-06T14:17:13.678 INFO:tasks.workunit.client.0.smithi032.stdout:7/773: read - d16/d20/d48/fa3 zero size 2024-04-06T14:17:13.680 INFO:tasks.workunit.client.0.smithi032.stdout:5/684: creat d16/d58/d54/d94/ff1 x:0 0 0 2024-04-06T14:17:13.680 INFO:tasks.workunit.client.0.smithi032.stdout:6/753: unlink d4/d5/d28/d54/cc8 0 2024-04-06T14:17:13.683 INFO:tasks.workunit.client.0.smithi032.stdout:4/744: mknod d0/df/d23/d31/d50/d57/d5b/d9c/cfd 0 2024-04-06T14:17:13.683 INFO:tasks.workunit.client.0.smithi032.stdout:8/818: dread d3/d8/d12/f51 [0,4194304] 0 2024-04-06T14:17:13.683 INFO:tasks.workunit.client.0.smithi032.stdout:0/601: mknod d3/da/d25/d2b/d4a/da1/ccb 0 2024-04-06T14:17:13.684 INFO:tasks.workunit.client.0.smithi032.stdout:4/745: readlink d0/df/d1f/l56 0 2024-04-06T14:17:13.686 INFO:tasks.workunit.client.0.smithi032.stdout:4/746: truncate d0/df/d1f/d5a/f8f 1174156 0 2024-04-06T14:17:13.686 INFO:tasks.workunit.client.0.smithi032.stdout:2/675: creat d0/fdb x:0 0 0 2024-04-06T14:17:13.687 INFO:tasks.workunit.client.0.smithi032.stdout:4/747: fsync d0/df/d1f/d5a/f8f 0 2024-04-06T14:17:13.688 INFO:tasks.workunit.client.0.smithi032.stdout:1/815: creat d0/da/d3e/d75/f10b x:0 0 0 2024-04-06T14:17:13.689 INFO:tasks.workunit.client.0.smithi032.stdout:7/774: fsync d16/d20/d54/de4/d5f/f65 0 2024-04-06T14:17:13.692 INFO:tasks.workunit.client.0.smithi032.stdout:2/676: fsync d0/f8a 0 2024-04-06T14:17:13.695 INFO:tasks.workunit.client.0.smithi032.stdout:4/748: unlink d0/df/d23/d31/d50/d57/le1 0 2024-04-06T14:17:13.696 INFO:tasks.workunit.client.0.smithi032.stdout:5/685: creat d16/d18/d33/d35/d88/deb/ff2 x:0 0 0 2024-04-06T14:17:13.697 INFO:tasks.workunit.client.0.smithi032.stdout:4/749: readlink d0/df/d1f/d42/d45/l7b 0 2024-04-06T14:17:13.699 INFO:tasks.workunit.client.0.smithi032.stdout:8/819: dwrite d3/d8/f25 [0,4194304] 0 2024-04-06T14:17:13.699 INFO:tasks.workunit.client.0.smithi032.stdout:4/750: readlink d0/l9 0 2024-04-06T14:17:13.701 INFO:tasks.workunit.client.0.smithi032.stdout:1/816: mkdir d0/d9/d1f/d7c/d8f/dc5/d10c 0 2024-04-06T14:17:13.702 INFO:tasks.workunit.client.0.smithi032.stdout:8/820: readlink d3/d8/d7/l4e 0 2024-04-06T14:17:13.704 INFO:tasks.workunit.client.0.smithi032.stdout:6/754: link d4/d5/d18/caf d4/d69/cfe 0 2024-04-06T14:17:13.705 INFO:tasks.workunit.client.0.smithi032.stdout:1/817: chown d0/d9/d94/fd0 945 1 2024-04-06T14:17:13.706 INFO:tasks.workunit.client.0.smithi032.stdout:7/775: dread d16/d20/f37 [0,4194304] 0 2024-04-06T14:17:13.707 INFO:tasks.workunit.client.0.smithi032.stdout:5/686: truncate d16/d58/d54/d94/db2/fb8 3866563 0 2024-04-06T14:17:13.708 INFO:tasks.workunit.client.0.smithi032.stdout:6/755: write d4/d5/d28/d65/d6e/ff5 [283739,113665] 0 2024-04-06T14:17:13.708 INFO:tasks.workunit.client.0.smithi032.stdout:4/751: symlink d0/df/d23/d31/d50/d5e/lfe 0 2024-04-06T14:17:13.708 INFO:tasks.workunit.client.0.smithi032.stdout:6/756: readlink l3 0 2024-04-06T14:17:13.710 INFO:tasks.workunit.client.0.smithi032.stdout:8/821: creat d3/d8/d12/d83/d89/da0/dd2/f11b x:0 0 0 2024-04-06T14:17:13.710 INFO:tasks.workunit.client.0.smithi032.stdout:7/776: chown d16/d20/d54/de4/f31 116 1 2024-04-06T14:17:13.711 INFO:tasks.workunit.client.0.smithi032.stdout:5/687: chown d16/d18/d33/d45/f9f 2687 1 2024-04-06T14:17:13.712 INFO:tasks.workunit.client.0.smithi032.stdout:1/818: creat d0/d10/d16/d3f/f10d x:0 0 0 2024-04-06T14:17:13.713 INFO:tasks.workunit.client.0.smithi032.stdout:8/822: fsync d3/d8/d12/d2f/d4a/d60/d82/f95 0 2024-04-06T14:17:13.714 INFO:tasks.workunit.client.0.smithi032.stdout:1/819: chown d0/fe7 336778629 1 2024-04-06T14:17:13.717 INFO:tasks.workunit.client.0.smithi032.stdout:7/777: creat d16/d20/d54/de4/d2d/d30/d3e/df3/d110/f118 x:0 0 0 2024-04-06T14:17:13.719 INFO:tasks.workunit.client.0.smithi032.stdout:4/752: truncate d0/df/d23/d31/d50/d57/d5b/d9c/f9f 1344353 0 2024-04-06T14:17:13.720 INFO:tasks.workunit.client.0.smithi032.stdout:4/753: dread - d0/df/d2b/d46/ff3 zero size 2024-04-06T14:17:13.724 INFO:tasks.workunit.client.0.smithi032.stdout:5/688: rename d16/d18/d33/d35/d88/dbd/dc5/fcd to d16/d18/dae/ff3 0 2024-04-06T14:17:13.724 INFO:tasks.workunit.client.0.smithi032.stdout:1/820: rename d0/da/d3e/d75/dc4 to d0/da/d3e/d75/dc4/d10e 22 2024-04-06T14:17:13.726 INFO:tasks.workunit.client.0.smithi032.stdout:1/821: chown d0/d10/d16/d3f/d98/c10a 0 1 2024-04-06T14:17:13.728 INFO:tasks.workunit.client.0.smithi032.stdout:8/823: link d3/d8/d7/ld d3/d8/d12/d2f/d4a/d60/d75/d85/l11c 0 2024-04-06T14:17:13.729 INFO:tasks.workunit.client.0.smithi032.stdout:8/824: chown d3/d8/d7/d8a/fab 231 1 2024-04-06T14:17:13.730 INFO:tasks.workunit.client.0.smithi032.stdout:7/778: link d16/d20/d7a/f83 d16/d20/d63/d9d/df2/df5/f119 0 2024-04-06T14:17:13.732 INFO:tasks.workunit.client.0.smithi032.stdout:8/825: symlink d3/d8/d3a/l11d 0 2024-04-06T14:17:13.734 INFO:tasks.workunit.client.0.smithi032.stdout:1/822: creat d0/d9/d1f/d7c/d8f/dc5/d10c/f10f x:0 0 0 2024-04-06T14:17:13.741 INFO:tasks.workunit.client.0.smithi032.stdout:9/757: dwrite d2/d15/d1e/d65/d83/d7e/f18 [4194304,4194304] 0 2024-04-06T14:17:13.742 INFO:tasks.workunit.client.0.smithi032.stdout:6/757: write d4/d5/d8/d2f/fd7 [537775,104409] 0 2024-04-06T14:17:13.747 INFO:tasks.workunit.client.0.smithi032.stdout:1/823: truncate d0/d41/d59/fd8 246944 0 2024-04-06T14:17:13.748 INFO:tasks.workunit.client.0.smithi032.stdout:7/779: dwrite d16/d20/dbd/f105 [0,4194304] 0 2024-04-06T14:17:13.748 INFO:tasks.workunit.client.0.smithi032.stdout:2/677: dwrite d0/fb [0,4194304] 0 2024-04-06T14:17:13.751 INFO:tasks.workunit.client.0.smithi032.stdout:4/754: dread d0/df/d2b/f72 [0,4194304] 0 2024-04-06T14:17:13.751 INFO:tasks.workunit.client.0.smithi032.stdout:5/689: write d16/d18/d33/d45/d5e/f89 [627160,26081] 0 2024-04-06T14:17:13.753 INFO:tasks.workunit.client.0.smithi032.stdout:4/755: chown d0 16901 1 2024-04-06T14:17:13.755 INFO:tasks.workunit.client.0.smithi032.stdout:6/758: creat d4/d5/d28/d65/fff x:0 0 0 2024-04-06T14:17:13.755 INFO:tasks.workunit.client.0.smithi032.stdout:9/758: mknod d2/d15/d1e/d41/d6c/d88/cf1 0 2024-04-06T14:17:13.758 INFO:tasks.workunit.client.0.smithi032.stdout:5/690: fsync d16/f96 0 2024-04-06T14:17:13.760 INFO:tasks.workunit.client.0.smithi032.stdout:7/780: dread - d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee/ffc zero size 2024-04-06T14:17:13.760 INFO:tasks.workunit.client.0.smithi032.stdout:1/824: chown d0/d41/d5b/d96/ccc 1950 1 2024-04-06T14:17:13.763 INFO:tasks.workunit.client.0.smithi032.stdout:9/759: rename d2/dcd/deb to d2/d15/d55/d59/df2 0 2024-04-06T14:17:13.765 INFO:tasks.workunit.client.0.smithi032.stdout:2/678: link d0/d30/f36 d0/d30/d39/d59/d67/d74/d78/d57/dd3/fdc 0 2024-04-06T14:17:13.766 INFO:tasks.workunit.client.0.smithi032.stdout:4/756: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/df2/dff 0 2024-04-06T14:17:13.766 INFO:tasks.workunit.client.0.smithi032.stdout:6/759: mknod d4/d5/d19/d96/de6/c100 0 2024-04-06T14:17:13.769 INFO:tasks.workunit.client.0.smithi032.stdout:8/826: dread d3/d8/d12/d2f/d46/f53 [0,4194304] 0 2024-04-06T14:17:13.771 INFO:tasks.workunit.client.0.smithi032.stdout:9/760: readlink d2/d15/d55/d59/d93/d95/db8/d81/d4a/l5e 0 2024-04-06T14:17:13.771 INFO:tasks.workunit.client.0.smithi032.stdout:7/781: getdents d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee 0 2024-04-06T14:17:13.773 INFO:tasks.workunit.client.0.smithi032.stdout:2/679: write d0/d30/d39/d59/fda [336392,74283] 0 2024-04-06T14:17:13.775 INFO:tasks.workunit.client.0.smithi032.stdout:8/827: chown d3/d8/d12/d2f/d4a/d60 1 1 2024-04-06T14:17:13.776 INFO:tasks.workunit.client.0.smithi032.stdout:2/680: read - d0/d30/fd8 zero size 2024-04-06T14:17:13.776 INFO:tasks.workunit.client.0.smithi032.stdout:1/825: dwrite d0/d9/d1f/d7c/da9/fe8 [0,4194304] 0 2024-04-06T14:17:13.779 INFO:tasks.workunit.client.0.smithi032.stdout:7/782: truncate d16/f49 1124062 0 2024-04-06T14:17:13.779 INFO:tasks.workunit.client.0.smithi032.stdout:8/828: read - d3/d8/d12/d2f/d4a/d60/d75/dd1/fff zero size 2024-04-06T14:17:13.787 INFO:tasks.workunit.client.0.smithi032.stdout:6/760: truncate d4/d5/dbf/d61/f66 810059 0 2024-04-06T14:17:13.787 INFO:tasks.workunit.client.0.smithi032.stdout:2/681: dread d0/d5a/d3a/da2/fc1 [0,4194304] 0 2024-04-06T14:17:13.789 INFO:tasks.workunit.client.0.smithi032.stdout:8/829: dread d3/d8/d3a/f4f [0,4194304] 0 2024-04-06T14:17:13.789 INFO:tasks.workunit.client.0.smithi032.stdout:1/826: unlink d0/da/c27 0 2024-04-06T14:17:13.790 INFO:tasks.workunit.client.0.smithi032.stdout:7/783: mkdir d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dde/d11a 0 2024-04-06T14:17:13.791 INFO:tasks.workunit.client.0.smithi032.stdout:4/757: dread d0/df/d2b/fd9 [0,4194304] 0 2024-04-06T14:17:13.836 INFO:tasks.workunit.client.0.smithi032.stdout:2/682: chown d0/d5a/fb1 401292 1 2024-04-06T14:17:13.836 INFO:tasks.workunit.client.0.smithi032.stdout:9/761: write d2/dcd/f87 [397210,90051] 0 2024-04-06T14:17:13.839 INFO:tasks.workunit.client.0.smithi032.stdout:9/762: fdatasync d2/d15/d1e/d65/d83/d7e/f75 0 2024-04-06T14:17:13.839 INFO:tasks.workunit.client.0.smithi032.stdout:4/758: rmdir d0/d82/d8e 39 2024-04-06T14:17:13.841 INFO:tasks.workunit.client.0.smithi032.stdout:5/691: dwrite d16/d18/d33/d45/d87/f8c [0,4194304] 0 2024-04-06T14:17:13.842 INFO:tasks.workunit.client.0.smithi032.stdout:9/763: write d2/d15/d1e/f33 [1209502,58086] 0 2024-04-06T14:17:13.843 INFO:tasks.workunit.client.0.smithi032.stdout:2/683: mknod d0/d4e/dc6/cdd 0 2024-04-06T14:17:13.846 INFO:tasks.workunit.client.0.smithi032.stdout:6/761: link d4/d5/d28/le2 d4/d5/d8/d4b/l101 0 2024-04-06T14:17:13.850 INFO:tasks.workunit.client.0.smithi032.stdout:7/784: dwrite d16/d20/d54/de4/d2d/f75 [0,4194304] 0 2024-04-06T14:17:13.852 INFO:tasks.workunit.client.0.smithi032.stdout:4/759: symlink d0/df/d23/l100 0 2024-04-06T14:17:13.853 INFO:tasks.workunit.client.0.smithi032.stdout:5/692: symlink d16/d58/d54/d94/db2/dcb/lf4 0 2024-04-06T14:17:13.855 INFO:tasks.workunit.client.0.smithi032.stdout:4/760: dread - d0/df/d2b/d66/fd4 zero size 2024-04-06T14:17:13.859 INFO:tasks.workunit.client.0.smithi032.stdout:5/693: truncate d16/f24 2670547 0 2024-04-06T14:17:13.860 INFO:tasks.workunit.client.0.smithi032.stdout:1/827: dwrite d0/d9/d1f/d72/f7e [0,4194304] 0 2024-04-06T14:17:13.861 INFO:tasks.workunit.client.0.smithi032.stdout:7/785: mkdir d16/d20/d7a/da6/d117/d11b 0 2024-04-06T14:17:13.862 INFO:tasks.workunit.client.0.smithi032.stdout:9/764: getdents d2/d15/d55/d59/d93 0 2024-04-06T14:17:13.864 INFO:tasks.workunit.client.0.smithi032.stdout:1/828: write d0/d41/d5b/d96/fd3 [4155014,100692] 0 2024-04-06T14:17:13.865 INFO:tasks.workunit.client.0.smithi032.stdout:5/694: creat d16/d18/d33/d45/d52/ff5 x:0 0 0 2024-04-06T14:17:13.868 INFO:tasks.workunit.client.0.smithi032.stdout:9/765: chown d2/d15/d55/f86 51 1 2024-04-06T14:17:13.872 INFO:tasks.workunit.client.0.smithi032.stdout:6/762: dwrite d4/d5/d8/d4b/fd3 [0,4194304] 0 2024-04-06T14:17:13.874 INFO:tasks.workunit.client.0.smithi032.stdout:9/766: unlink d2/f53 0 2024-04-06T14:17:13.874 INFO:tasks.workunit.client.0.smithi032.stdout:7/786: getdents d16/d20/d54/de4/d2d/d30/d3e/da7 0 2024-04-06T14:17:13.875 INFO:tasks.workunit.client.0.smithi032.stdout:7/787: readlink d16/d92/lbe 0 2024-04-06T14:17:13.942 INFO:tasks.workunit.client.0.smithi032.stdout:6/763: fdatasync d4/d5/fa0 0 2024-04-06T14:17:13.945 INFO:tasks.workunit.client.0.smithi032.stdout:7/788: rmdir d16/d20/d54/de4/d2d/d30/d3e/da7/db5 39 2024-04-06T14:17:13.946 INFO:tasks.workunit.client.0.smithi032.stdout:6/764: chown d4/d5/dd8/fbb 238 1 2024-04-06T14:17:13.950 INFO:tasks.workunit.client.0.smithi032.stdout:1/829: dwrite d0/d41/d59/fd4 [0,4194304] 0 2024-04-06T14:17:13.951 INFO:tasks.workunit.client.0.smithi032.stdout:7/789: rmdir d16/d20/d54/de4/d2d/d30/d3e 39 2024-04-06T14:17:13.951 INFO:tasks.workunit.client.0.smithi032.stdout:6/765: creat d4/d5/d19/f102 x:0 0 0 2024-04-06T14:17:13.952 INFO:tasks.workunit.client.0.smithi032.stdout:9/767: dwrite d2/d15/d1e/d41/d6c/d74/fe7 [0,4194304] 0 2024-04-06T14:17:13.956 INFO:tasks.workunit.client.0.smithi032.stdout:9/768: fdatasync d2/d15/d1e/d65/d83/d31/f40 0 2024-04-06T14:17:13.956 INFO:tasks.workunit.client.0.smithi032.stdout:1/830: read d0/d9/d1f/d7c/da9/fb3 [2327319,118449] 0 2024-04-06T14:17:13.957 INFO:tasks.workunit.client.0.smithi032.stdout:5/695: dread d16/d18/d33/d45/d52/d39/fb6 [0,4194304] 0 2024-04-06T14:17:13.958 INFO:tasks.workunit.client.0.smithi032.stdout:6/766: rename d4/d5/d8/d2f/fc2 to d4/d5/d8/d4b/f103 0 2024-04-06T14:17:13.959 INFO:tasks.workunit.client.0.smithi032.stdout:7/790: creat d16/d20/d7a/da6/d117/f11c x:0 0 0 2024-04-06T14:17:13.960 INFO:tasks.workunit.client.0.smithi032.stdout:1/831: truncate d0/da/f28 4595619 0 2024-04-06T14:17:13.961 INFO:tasks.workunit.client.0.smithi032.stdout:9/769: rmdir d2/d15/d55/d59/d93/d95/db8/d81/d77 39 2024-04-06T14:17:13.962 INFO:tasks.workunit.client.0.smithi032.stdout:5/696: rmdir d16/d58/d54/d94/db2/dcb 39 2024-04-06T14:17:13.963 INFO:tasks.workunit.client.0.smithi032.stdout:6/767: mknod d4/d5/d8/d4b/d7c/c104 0 2024-04-06T14:17:13.966 INFO:tasks.workunit.client.0.smithi032.stdout:6/768: dread - d4/d5/dbf/d61/da7/dda/fee zero size 2024-04-06T14:17:13.967 INFO:tasks.workunit.client.0.smithi032.stdout:7/791: dread d16/f27 [0,4194304] 0 2024-04-06T14:17:13.970 INFO:tasks.workunit.client.0.smithi032.stdout:1/832: getdents d0/d9/d1f/d72 0 2024-04-06T14:17:13.970 INFO:tasks.workunit.client.0.smithi032.stdout:9/770: link d2/d15/d1e/d65/d83/d7e/f56 d2/d15/d55/d59/d93/d95/db8/ff3 0 2024-04-06T14:17:13.972 INFO:tasks.workunit.client.0.smithi032.stdout:6/769: creat d4/d5/dbf/f105 x:0 0 0 2024-04-06T14:17:13.973 INFO:tasks.workunit.client.0.smithi032.stdout:1/833: creat d0/d9/d1f/d7c/d8f/daa/f110 x:0 0 0 2024-04-06T14:17:13.974 INFO:tasks.workunit.client.0.smithi032.stdout:6/770: stat d4/d5/d8/c94 0 2024-04-06T14:17:13.975 INFO:tasks.workunit.client.0.smithi032.stdout:9/771: symlink d2/d15/d1e/d65/d83/d7e/lf4 0 2024-04-06T14:17:13.975 INFO:tasks.workunit.client.0.smithi032.stdout:1/834: mkdir d0/d41/d5b/d111 0 2024-04-06T14:17:13.978 INFO:tasks.workunit.client.0.smithi032.stdout:1/835: dread - d0/d9/d1f/d72/ffb zero size 2024-04-06T14:17:13.980 INFO:tasks.workunit.client.0.smithi032.stdout:7/792: link d16/d20/d54/led d16/d20/d54/de4/d2d/d30/d3e/d96/d10b/l11d 0 2024-04-06T14:17:13.982 INFO:tasks.workunit.client.0.smithi032.stdout:1/836: rmdir d0/d41/d5b 39 2024-04-06T14:17:13.983 INFO:tasks.workunit.client.0.smithi032.stdout:6/771: dwrite d4/d5/d8/fe4 [0,4194304] 0 2024-04-06T14:17:13.983 INFO:tasks.workunit.client.0.smithi032.stdout:7/793: chown d16/d20/d54/de4/d2d/d30/d3e/df3 537 1 2024-04-06T14:17:13.985 INFO:tasks.workunit.client.0.smithi032.stdout:1/837: write d0/d41/d59/fe9 [466491,54211] 0 2024-04-06T14:17:13.986 INFO:tasks.workunit.client.0.smithi032.stdout:9/772: write d2/fb9 [947105,16507] 0 2024-04-06T14:17:13.988 INFO:tasks.workunit.client.0.smithi032.stdout:1/838: write d0/d9/fbe [3771464,47197] 0 2024-04-06T14:17:13.993 INFO:tasks.workunit.client.0.smithi032.stdout:9/773: creat d2/d15/d55/d59/d93/d95/ff5 x:0 0 0 2024-04-06T14:17:13.994 INFO:tasks.workunit.client.0.smithi032.stdout:5/697: dread d16/d18/d33/d45/d52/d39/f5b [0,4194304] 0 2024-04-06T14:17:13.994 INFO:tasks.workunit.client.0.smithi032.stdout:7/794: write fb [784187,5238] 0 2024-04-06T14:17:13.997 INFO:tasks.workunit.client.0.smithi032.stdout:1/839: rename d0/da/d3e/d75/lbc to d0/d9/d1f/d7c/d8f/daa/l112 0 2024-04-06T14:17:13.998 INFO:tasks.workunit.client.0.smithi032.stdout:5/698: creat d16/d18/d33/d35/d88/deb/ff6 x:0 0 0 2024-04-06T14:17:13.999 INFO:tasks.workunit.client.0.smithi032.stdout:1/840: truncate d0/d9/d1f/d7c/da9/ff9 115178 0 2024-04-06T14:17:14.001 INFO:tasks.workunit.client.0.smithi032.stdout:1/841: write d0/f2c [4621965,10486] 0 2024-04-06T14:17:14.002 INFO:tasks.workunit.client.0.smithi032.stdout:9/774: mkdir d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/df6 0 2024-04-06T14:17:14.003 INFO:tasks.workunit.client.0.smithi032.stdout:7/795: link d16/f4b d16/d20/d54/de4/d2d/d30/d3e/f11e 0 2024-04-06T14:17:14.004 INFO:tasks.workunit.client.0.smithi032.stdout:5/699: mkdir d16/d18/d33/df7 0 2024-04-06T14:17:14.005 INFO:tasks.workunit.client.0.smithi032.stdout:5/700: chown d16/d18/c61 1 1 2024-04-06T14:17:14.005 INFO:tasks.workunit.client.0.smithi032.stdout:1/842: mkdir d0/d9/d1f/d7c/d8f/dc5/d10c/d113 0 2024-04-06T14:17:14.005 INFO:tasks.workunit.client.0.smithi032.stdout:6/772: dread d4/d5/d19/d21/f30 [0,4194304] 0 2024-04-06T14:17:14.006 INFO:tasks.workunit.client.0.smithi032.stdout:7/796: fsync d16/d20/d48/fbf 0 2024-04-06T14:17:14.007 INFO:tasks.workunit.client.0.smithi032.stdout:6/773: fdatasync d4/fc7 0 2024-04-06T14:17:14.010 INFO:tasks.workunit.client.0.smithi032.stdout:9/775: creat d2/d15/d1e/d65/d83/ff7 x:0 0 0 2024-04-06T14:17:14.011 INFO:tasks.workunit.client.0.smithi032.stdout:1/843: rmdir d0/d9/d1f/d7c/da9 39 2024-04-06T14:17:14.017 INFO:tasks.workunit.client.0.smithi032.stdout:5/701: dwrite d16/d18/d33/d35/de0/fe6 [0,4194304] 0 2024-04-06T14:17:14.018 INFO:tasks.workunit.client.0.smithi032.stdout:7/797: link d16/d92/db3/f103 d16/d20/d54/de4/d2d/d30/d3e/da7/db5/f11f 0 2024-04-06T14:17:14.019 INFO:tasks.workunit.client.0.smithi032.stdout:1/844: creat d0/d10/d16/d3f/d98/f114 x:0 0 0 2024-04-06T14:17:14.019 INFO:tasks.workunit.client.0.smithi032.stdout:6/774: dwrite d4/d5/d14/f1a [0,4194304] 0 2024-04-06T14:17:14.023 INFO:tasks.workunit.client.0.smithi032.stdout:9/776: link d2/d15/d55/d59/d93/d95/db8/d81/d4a/l4e d2/d15/d55/d59/d93/d95/db8/lf8 0 2024-04-06T14:17:14.023 INFO:tasks.workunit.client.0.smithi032.stdout:7/798: creat d16/d20/d54/de4/de0/df9/f120 x:0 0 0 2024-04-06T14:17:14.023 INFO:tasks.workunit.client.0.smithi032.stdout:1/845: mknod d0/d9/c115 0 2024-04-06T14:17:14.025 INFO:tasks.workunit.client.0.smithi032.stdout:6/775: rename d4/d5/d14/c1c to d4/d5/d8/d2f/c106 0 2024-04-06T14:17:14.025 INFO:tasks.workunit.client.0.smithi032.stdout:1/846: fdatasync d0/d41/d59/fe9 0 2024-04-06T14:17:14.031 INFO:tasks.workunit.client.0.smithi032.stdout:6/776: chown d4/d5/dbf/d61/db9 7 1 2024-04-06T14:17:14.031 INFO:tasks.workunit.client.0.smithi032.stdout:5/702: dread d16/d18/d33/d45/d52/d39/fb6 [0,4194304] 0 2024-04-06T14:17:14.032 INFO:tasks.workunit.client.0.smithi032.stdout:1/847: write d0/d9/d94/fe3 [107265,70921] 0 2024-04-06T14:17:14.032 INFO:tasks.workunit.client.0.smithi032.stdout:9/777: chown d2/d15/d1e/d65/d83/d7e/f6 0 1 2024-04-06T14:17:14.032 INFO:tasks.workunit.client.0.smithi032.stdout:6/777: symlink d4/d5/d83/l107 0 2024-04-06T14:17:14.032 INFO:tasks.workunit.client.0.smithi032.stdout:1/848: write d0/d41/d5b/d96/fd3 [4720140,120050] 0 2024-04-06T14:17:14.034 INFO:tasks.workunit.client.0.smithi032.stdout:5/703: mkdir d16/d18/d33/d35/d88/df8 0 2024-04-06T14:17:14.035 INFO:tasks.workunit.client.0.smithi032.stdout:9/778: symlink d2/d15/d55/d59/df2/lf9 0 2024-04-06T14:17:14.035 INFO:tasks.workunit.client.0.smithi032.stdout:1/849: write d0/da/ffc [993665,68067] 0 2024-04-06T14:17:14.040 INFO:tasks.workunit.client.0.smithi032.stdout:9/779: unlink d2/d15/c70 0 2024-04-06T14:17:14.040 INFO:tasks.workunit.client.0.smithi032.stdout:7/799: link d16/d20/d63/d9d/fd8 d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee/f121 0 2024-04-06T14:17:14.042 INFO:tasks.workunit.client.0.smithi032.stdout:7/800: write d16/d20/d48/d8d/f116 [946497,24523] 0 2024-04-06T14:17:14.043 INFO:tasks.workunit.client.0.smithi032.stdout:1/850: truncate d0/d10/d16/d3f/fa3 45646 0 2024-04-06T14:17:14.045 INFO:tasks.workunit.client.0.smithi032.stdout:5/704: link d16/d18/d33/d45/d52/fa2 d16/d18/dae/dbb/ff9 0 2024-04-06T14:17:14.047 INFO:tasks.workunit.client.0.smithi032.stdout:9/780: mkdir d2/dcd/d7c/dfa 0 2024-04-06T14:17:14.047 INFO:tasks.workunit.client.0.smithi032.stdout:5/705: chown d16/d18/d33/d35/d88/la0 46 1 2024-04-06T14:17:14.048 INFO:tasks.workunit.client.1.smithi086.stdout:3/526: sync 2024-04-06T14:17:14.049 INFO:tasks.workunit.client.0.smithi032.stdout:7/801: fsync d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee/fcb 0 2024-04-06T14:17:14.051 INFO:tasks.workunit.client.0.smithi032.stdout:7/802: write d16/d20/d63/d9d/df2/df5/fe8 [174126,11385] 0 2024-04-06T14:17:14.052 INFO:tasks.workunit.client.0.smithi032.stdout:7/803: chown d16/d20/d54/de4/de0 4419 1 2024-04-06T14:17:14.054 INFO:tasks.workunit.client.0.smithi032.stdout:9/781: mknod d2/dcd/d7c/dfa/cfb 0 2024-04-06T14:17:14.055 INFO:tasks.workunit.client.1.smithi086.stdout:3/527: link d5/d33/d5d/c62 d5/d36/d5a/d6d/d72/d9a/ca3 0 2024-04-06T14:17:14.056 INFO:tasks.workunit.client.0.smithi032.stdout:5/706: symlink d16/d18/d33/d35/d88/dbd/dc5/dec/lfa 0 2024-04-06T14:17:14.057 INFO:tasks.workunit.client.0.smithi032.stdout:7/804: symlink d16/d20/d7a/l122 0 2024-04-06T14:17:14.059 INFO:tasks.workunit.client.0.smithi032.stdout:9/782: unlink d2/d15/d1e/d65/d83/d7e/f18 0 2024-04-06T14:17:14.059 INFO:tasks.workunit.client.0.smithi032.stdout:6/778: write d4/d5/d18/f49 [466301,129038] 0 2024-04-06T14:17:14.061 INFO:tasks.workunit.client.0.smithi032.stdout:7/805: write d16/d20/f28 [4045376,116585] 0 2024-04-06T14:17:14.062 INFO:tasks.workunit.client.0.smithi032.stdout:7/806: chown d16/d20/d7a/da6/d117 211 1 2024-04-06T14:17:14.064 INFO:tasks.workunit.client.0.smithi032.stdout:5/707: creat d16/d18/d33/df7/ffb x:0 0 0 2024-04-06T14:17:14.065 INFO:tasks.workunit.client.0.smithi032.stdout:9/783: chown d2/d15/d1e/d65/d83/d31/d39/c42 5552 1 2024-04-06T14:17:14.067 INFO:tasks.workunit.client.0.smithi032.stdout:6/779: unlink d4/d5/d28/d65/d6e/le1 0 2024-04-06T14:17:14.068 INFO:tasks.workunit.client.0.smithi032.stdout:6/780: stat d4/d5/d8/d4b/d7c/cd5 0 2024-04-06T14:17:14.068 INFO:tasks.workunit.client.0.smithi032.stdout:7/807: creat d16/d20/d48/d8d/f123 x:0 0 0 2024-04-06T14:17:14.070 INFO:tasks.workunit.client.0.smithi032.stdout:3/691: sync 2024-04-06T14:17:14.070 INFO:tasks.workunit.client.0.smithi032.stdout:0/602: sync 2024-04-06T14:17:14.072 INFO:tasks.workunit.client.0.smithi032.stdout:5/708: creat d16/d58/d54/d94/db2/ffc x:0 0 0 2024-04-06T14:17:14.072 INFO:tasks.workunit.client.0.smithi032.stdout:6/781: chown d4/d5/d8/d2f/d9b/db4 1 1 2024-04-06T14:17:14.072 INFO:tasks.workunit.client.0.smithi032.stdout:1/851: write d0/d10/d16/d3f/fa3 [421707,39914] 0 2024-04-06T14:17:14.073 INFO:tasks.workunit.client.0.smithi032.stdout:0/603: readlink d3/da/lb 0 2024-04-06T14:17:14.076 INFO:tasks.workunit.client.0.smithi032.stdout:7/808: mknod d16/d20/d54/de4/d2d/d30/d3e/c124 0 2024-04-06T14:17:14.077 INFO:tasks.workunit.client.0.smithi032.stdout:9/784: dwrite d2/d15/d1e/d65/d83/d7e/da7/fe2 [0,4194304] 0 2024-04-06T14:17:14.077 INFO:tasks.workunit.client.0.smithi032.stdout:5/709: rmdir d16/d18/dae 39 2024-04-06T14:17:14.080 INFO:tasks.workunit.client.0.smithi032.stdout:9/785: fsync d2/d15/d55/d59/f67 0 2024-04-06T14:17:14.082 INFO:tasks.workunit.client.0.smithi032.stdout:6/782: fdatasync d4/d5/dc5/dcc/fe9 0 2024-04-06T14:17:14.084 INFO:tasks.workunit.client.0.smithi032.stdout:0/604: dread d3/da/f56 [0,4194304] 0 2024-04-06T14:17:14.085 INFO:tasks.workunit.client.0.smithi032.stdout:1/852: write d0/d10/d16/ffd [123230,94383] 0 2024-04-06T14:17:14.086 INFO:tasks.workunit.client.0.smithi032.stdout:5/710: creat d16/d18/d33/d45/d52/ffd x:0 0 0 2024-04-06T14:17:14.087 INFO:tasks.workunit.client.0.smithi032.stdout:3/692: creat d3/d65/d7d/d9c/dad/db1/d3c/dc9/d104/f10a x:0 0 0 2024-04-06T14:17:14.088 INFO:tasks.workunit.client.0.smithi032.stdout:9/786: chown d2/d15/d1e/d65/d83/d7e/c16 29557 1 2024-04-06T14:17:14.088 INFO:tasks.workunit.client.0.smithi032.stdout:7/809: dread d16/d20/d7a/fb9 [0,4194304] 0 2024-04-06T14:17:14.090 INFO:tasks.workunit.client.0.smithi032.stdout:6/783: rename d4/d5/d8/d2f/c81 to d4/d5/d28/c108 0 2024-04-06T14:17:14.091 INFO:tasks.workunit.client.0.smithi032.stdout:0/605: getdents d3/da/d25/d57 0 2024-04-06T14:17:14.093 INFO:tasks.workunit.client.0.smithi032.stdout:3/693: creat d3/d16/ddd/f10b x:0 0 0 2024-04-06T14:17:14.094 INFO:tasks.workunit.client.0.smithi032.stdout:0/606: mkdir d3/da/d35/d8b/d9c/dcc 0 2024-04-06T14:17:14.095 INFO:tasks.workunit.client.0.smithi032.stdout:1/853: rmdir d0/d10/d66/d104 0 2024-04-06T14:17:14.098 INFO:tasks.workunit.client.0.smithi032.stdout:1/854: fdatasync d0/d10/d16/d3f/d98/f114 0 2024-04-06T14:17:14.100 INFO:tasks.workunit.client.0.smithi032.stdout:6/784: dwrite d4/d5/d28/f29 [4194304,4194304] 0 2024-04-06T14:17:14.101 INFO:tasks.workunit.client.0.smithi032.stdout:1/855: write d0/d10/d16/db8/ff3 [325908,118456] 0 2024-04-06T14:17:14.105 INFO:tasks.workunit.client.0.smithi032.stdout:7/810: truncate d16/d20/d54/de4/d2d/d30/d3e/d96/fec 19745 0 2024-04-06T14:17:14.106 INFO:tasks.workunit.client.0.smithi032.stdout:6/785: stat d4/d5/d19/d21/l62 0 2024-04-06T14:17:14.107 INFO:tasks.workunit.client.0.smithi032.stdout:1/856: rmdir d0/da/d3e 39 2024-04-06T14:17:14.110 INFO:tasks.workunit.client.0.smithi032.stdout:0/607: dwrite d3/f5d [0,4194304] 0 2024-04-06T14:17:14.111 INFO:tasks.workunit.client.0.smithi032.stdout:3/694: getdents d3/d16/ddd 0 2024-04-06T14:17:14.112 INFO:tasks.workunit.client.0.smithi032.stdout:1/857: chown d0/d9/d1f/d7c/d8f/dc5/d10c/f10f 37771 1 2024-04-06T14:17:14.113 INFO:tasks.workunit.client.0.smithi032.stdout:6/786: mknod d4/d5/dbf/d61/db9/c109 0 2024-04-06T14:17:14.113 INFO:tasks.workunit.client.0.smithi032.stdout:9/787: truncate d2/d15/d55/d59/d93/d95/db8/d81/fab 291393 0 2024-04-06T14:17:14.115 INFO:tasks.workunit.client.0.smithi032.stdout:1/858: read d0/d9/d1f/d72/f7e [2813302,8445] 0 2024-04-06T14:17:14.115 INFO:tasks.workunit.client.0.smithi032.stdout:3/695: read d3/d65/f66 [1202842,11875] 0 2024-04-06T14:17:14.115 INFO:tasks.workunit.client.0.smithi032.stdout:0/608: mkdir d3/da/d25/d2b/d80/db3/dcd 0 2024-04-06T14:17:14.117 INFO:tasks.workunit.client.0.smithi032.stdout:3/696: chown d3/d65/d7d/d9c/dad/db1/d3c/dc9/d104 0 1 2024-04-06T14:17:14.122 INFO:tasks.workunit.client.0.smithi032.stdout:3/697: creat d3/d16/ddd/f10c x:0 0 0 2024-04-06T14:17:14.126 INFO:tasks.workunit.client.0.smithi032.stdout:3/698: unlink d3/d65/fdc 0 2024-04-06T14:17:14.127 INFO:tasks.workunit.client.0.smithi032.stdout:1/859: rename d0/da/d3e/fbd to d0/d9/d1f/d7c/d8f/dc5/d10c/d113/f116 0 2024-04-06T14:17:14.127 INFO:tasks.workunit.client.0.smithi032.stdout:0/609: dwrite d3/da/d25/d2b/d4a/da1/fb2 [0,4194304] 0 2024-04-06T14:17:14.130 INFO:tasks.workunit.client.0.smithi032.stdout:0/610: readlink d3/da/d25/d2b/d4a/d79/da8/l99 0 2024-04-06T14:17:14.133 INFO:tasks.workunit.client.0.smithi032.stdout:9/788: dread d2/d15/d1e/d41/f63 [4194304,4194304] 0 2024-04-06T14:17:14.135 INFO:tasks.workunit.client.0.smithi032.stdout:9/789: creat d2/ffc x:0 0 0 2024-04-06T14:17:14.137 INFO:tasks.workunit.client.0.smithi032.stdout:1/860: read d0/da/f52 [110060,30769] 0 2024-04-06T14:17:14.137 INFO:tasks.workunit.client.0.smithi032.stdout:9/790: fdatasync d2/d15/d1e/d65/d83/f94 0 2024-04-06T14:17:14.139 INFO:tasks.workunit.client.0.smithi032.stdout:0/611: link d3/da/d25/d2b/f32 d3/da/d35/fce 0 2024-04-06T14:17:14.141 INFO:tasks.workunit.client.0.smithi032.stdout:7/811: write d16/d20/d63/d9d/df2/df5/fdc [1038663,47882] 0 2024-04-06T14:17:14.145 INFO:tasks.workunit.client.0.smithi032.stdout:0/612: mknod d3/da/d25/d2b/d80/db3/ccf 0 2024-04-06T14:17:14.145 INFO:tasks.workunit.client.0.smithi032.stdout:3/699: link d3/d16/fb0 d3/d65/d7d/d9c/dad/dc5/f10d 0 2024-04-06T14:17:14.150 INFO:tasks.workunit.client.0.smithi032.stdout:7/812: creat d16/d20/d54/de4/d2d/d30/d3e/da7/db5/f125 x:0 0 0 2024-04-06T14:17:14.150 INFO:tasks.workunit.client.0.smithi032.stdout:9/791: mkdir d2/dcd/dfd 0 2024-04-06T14:17:14.150 INFO:tasks.workunit.client.0.smithi032.stdout:0/613: mknod d3/da/d25/d57/db5/cd0 0 2024-04-06T14:17:14.154 INFO:tasks.workunit.client.0.smithi032.stdout:6/787: dwrite d4/fc6 [0,4194304] 0 2024-04-06T14:17:14.155 INFO:tasks.workunit.client.0.smithi032.stdout:9/792: fdatasync d2/d15/d55/d59/d93/d95/db8/d81/fb0 0 2024-04-06T14:17:14.156 INFO:tasks.workunit.client.0.smithi032.stdout:7/813: dread d16/d20/d54/de4/d2d/d30/d3e/da7/db5/fe6 [0,4194304] 0 2024-04-06T14:17:14.156 INFO:tasks.workunit.client.0.smithi032.stdout:0/614: creat d3/da/d25/d2b/d4a/fd1 x:0 0 0 2024-04-06T14:17:14.158 INFO:tasks.workunit.client.0.smithi032.stdout:1/861: dwrite d0/d10/f30 [0,4194304] 0 2024-04-06T14:17:14.160 INFO:tasks.workunit.client.0.smithi032.stdout:1/862: dread - d0/d9/d1f/d7c/d8f/daa/f110 zero size 2024-04-06T14:17:14.161 INFO:tasks.workunit.client.0.smithi032.stdout:3/700: write d3/d65/d89/fa2 [3255694,7177] 0 2024-04-06T14:17:14.163 INFO:tasks.workunit.client.0.smithi032.stdout:7/814: mkdir d16/d20/d54/de4/d2d/d30/d3e/da7/db5/d126 0 2024-04-06T14:17:14.164 INFO:tasks.workunit.client.0.smithi032.stdout:6/788: truncate d4/d5/d18/f52 2311341 0 2024-04-06T14:17:14.164 INFO:tasks.workunit.client.0.smithi032.stdout:9/793: mknod d2/d15/dd9/cfe 0 2024-04-06T14:17:14.165 INFO:tasks.workunit.client.0.smithi032.stdout:7/815: write fe [8620027,110759] 0 2024-04-06T14:17:14.168 INFO:tasks.workunit.client.0.smithi032.stdout:7/816: dread - d16/d92/f101 zero size 2024-04-06T14:17:14.170 INFO:tasks.workunit.client.0.smithi032.stdout:3/701: creat d3/d65/d7d/d9c/dad/f10e x:0 0 0 2024-04-06T14:17:14.171 INFO:tasks.workunit.client.0.smithi032.stdout:6/789: rename d4/d5/d8/d2f/fd7 to d4/d5/d19/d96/de6/f10a 0 2024-04-06T14:17:14.175 INFO:tasks.workunit.client.0.smithi032.stdout:3/702: write d3/d65/d7d/d9c/dad/db1/f62 [5055508,51347] 0 2024-04-06T14:17:14.176 INFO:tasks.workunit.client.0.smithi032.stdout:6/790: creat d4/d5/d8/d4b/d7c/f10b x:0 0 0 2024-04-06T14:17:14.176 INFO:tasks.workunit.client.0.smithi032.stdout:9/794: dwrite d2/d15/d1e/d41/f63 [0,4194304] 0 2024-04-06T14:17:14.179 INFO:tasks.workunit.client.0.smithi032.stdout:3/703: read d3/fd7 [1098940,57942] 0 2024-04-06T14:17:14.181 INFO:tasks.workunit.client.0.smithi032.stdout:7/817: dwrite d16/d20/d63/f100 [0,4194304] 0 2024-04-06T14:17:14.183 INFO:tasks.workunit.client.0.smithi032.stdout:8/830: sync 2024-04-06T14:17:14.183 INFO:tasks.workunit.client.0.smithi032.stdout:2/684: sync 2024-04-06T14:17:14.183 INFO:tasks.workunit.client.0.smithi032.stdout:4/761: sync 2024-04-06T14:17:14.183 INFO:tasks.workunit.client.0.smithi032.stdout:9/795: chown d2/d15/d1e/d65/d83/d7e/f12 69168218 1 2024-04-06T14:17:14.184 INFO:tasks.workunit.client.0.smithi032.stdout:1/863: dwrite d0/d10/d16/d3f/d98/f103 [0,4194304] 0 2024-04-06T14:17:14.185 INFO:tasks.workunit.client.0.smithi032.stdout:9/796: stat d2/d15/dd9 0 2024-04-06T14:17:14.189 INFO:tasks.workunit.client.0.smithi032.stdout:6/791: dread d4/d5/dbf/d61/f66 [0,4194304] 0 2024-04-06T14:17:14.190 INFO:tasks.workunit.client.0.smithi032.stdout:7/818: mknod d16/d20/c127 0 2024-04-06T14:17:14.197 INFO:tasks.workunit.client.0.smithi032.stdout:2/685: dread - d0/d30/d39/d59/d67/d74/d78/f77 zero size 2024-04-06T14:17:14.197 INFO:tasks.workunit.client.0.smithi032.stdout:1/864: rename d0/da/d3e/l84 to d0/d9/d1f/d7c/d8f/dc5/l117 0 2024-04-06T14:17:14.200 INFO:tasks.workunit.client.0.smithi032.stdout:4/762: creat d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/df9/f101 x:0 0 0 2024-04-06T14:17:14.201 INFO:tasks.workunit.client.0.smithi032.stdout:9/797: dwrite d2/fb9 [0,4194304] 0 2024-04-06T14:17:14.202 INFO:tasks.workunit.client.0.smithi032.stdout:6/792: chown d4/f98 8749494 1 2024-04-06T14:17:14.203 INFO:tasks.workunit.client.0.smithi032.stdout:7/819: rmdir d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee 39 2024-04-06T14:17:14.203 INFO:tasks.workunit.client.0.smithi032.stdout:8/831: dwrite d3/d8/d12/d2f/d4a/d60/d90/fd9 [0,4194304] 0 2024-04-06T14:17:14.205 INFO:tasks.workunit.client.0.smithi032.stdout:4/763: chown d0/df/d76 1498897 1 2024-04-06T14:17:14.206 INFO:tasks.workunit.client.1.smithi086.stdout:2/532: sync 2024-04-06T14:17:14.206 INFO:tasks.workunit.client.1.smithi086.stdout:4/385: sync 2024-04-06T14:17:14.206 INFO:tasks.workunit.client.1.smithi086.stdout:8/613: sync 2024-04-06T14:17:14.206 INFO:tasks.workunit.client.1.smithi086.stdout:6/596: sync 2024-04-06T14:17:14.206 INFO:tasks.workunit.client.1.smithi086.stdout:5/440: sync 2024-04-06T14:17:14.206 INFO:tasks.workunit.client.1.smithi086.stdout:1/554: sync 2024-04-06T14:17:14.207 INFO:tasks.workunit.client.1.smithi086.stdout:0/580: sync 2024-04-06T14:17:14.207 INFO:tasks.workunit.client.1.smithi086.stdout:7/508: sync 2024-04-06T14:17:14.207 INFO:tasks.workunit.client.1.smithi086.stdout:9/433: sync 2024-04-06T14:17:14.207 INFO:tasks.workunit.client.1.smithi086.stdout:4/386: chown d0 0 1 2024-04-06T14:17:14.209 INFO:tasks.workunit.client.1.smithi086.stdout:1/555: chown db/d13/d55/da3 24 1 2024-04-06T14:17:14.210 INFO:tasks.workunit.client.0.smithi032.stdout:1/865: dread - d0/d9/d1f/d7c/da9/de1/ff7 zero size 2024-04-06T14:17:14.211 INFO:tasks.workunit.client.0.smithi032.stdout:3/704: dwrite d3/d16/f9e [0,4194304] 0 2024-04-06T14:17:14.212 INFO:tasks.workunit.client.1.smithi086.stdout:8/614: read d0/d3b/f8b [1006576,66386] 0 2024-04-06T14:17:14.213 INFO:tasks.workunit.client.0.smithi032.stdout:9/798: write d2/d15/d1e/d65/d6b/f9b [587050,111644] 0 2024-04-06T14:17:14.215 INFO:tasks.workunit.client.1.smithi086.stdout:5/441: mknod d0/c88 0 2024-04-06T14:17:14.216 INFO:tasks.workunit.client.1.smithi086.stdout:2/533: dread - d0/d20/d27/d37/f98 zero size 2024-04-06T14:17:14.219 INFO:tasks.workunit.client.1.smithi086.stdout:5/442: readlink d0/d17/d2d/d80/l82 0 2024-04-06T14:17:14.219 INFO:tasks.workunit.client.1.smithi086.stdout:8/615: chown d0/db/l45 0 1 2024-04-06T14:17:14.220 INFO:tasks.workunit.client.0.smithi032.stdout:9/799: creat d2/d15/d1e/fff x:0 0 0 2024-04-06T14:17:14.221 INFO:tasks.workunit.client.1.smithi086.stdout:6/597: creat d6/d13/d1a/fd0 x:0 0 0 2024-04-06T14:17:14.221 INFO:tasks.workunit.client.0.smithi032.stdout:7/820: fdatasync d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee/fcb 0 2024-04-06T14:17:14.222 INFO:tasks.workunit.client.0.smithi032.stdout:2/686: creat d0/d30/d39/d59/d67/d74/d78/d57/fde x:0 0 0 2024-04-06T14:17:14.223 INFO:tasks.workunit.client.1.smithi086.stdout:8/616: write d0/d71/d22/d43/d6d/fb8 [457319,32333] 0 2024-04-06T14:17:14.223 INFO:tasks.workunit.client.0.smithi032.stdout:9/800: readlink d2/le 0 2024-04-06T14:17:14.226 INFO:tasks.workunit.client.1.smithi086.stdout:4/387: dread d0/d2c/d30/f3b [0,4194304] 0 2024-04-06T14:17:14.226 INFO:tasks.workunit.client.0.smithi032.stdout:3/705: creat d3/d65/d7d/d9c/dad/db1/f10f x:0 0 0 2024-04-06T14:17:14.227 INFO:tasks.workunit.client.1.smithi086.stdout:4/388: chown d0/d2c/d30 146951182 1 2024-04-06T14:17:14.227 INFO:tasks.workunit.client.1.smithi086.stdout:9/434: getdents d5/d13/d47/d55 0 2024-04-06T14:17:14.227 INFO:tasks.workunit.client.0.smithi032.stdout:4/764: dread d0/d82/d8e/fde [0,4194304] 0 2024-04-06T14:17:14.229 INFO:tasks.workunit.client.0.smithi032.stdout:1/866: fsync d0/d41/d59/fd8 0 2024-04-06T14:17:14.229 INFO:tasks.workunit.client.0.smithi032.stdout:4/765: dread - d0/df/f8b zero size 2024-04-06T14:17:14.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:13 smithi032 ceph-mon[39730]: pgmap v24: 65 pgs: 65 active+clean; 2.6 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 101 MiB/s rd, 337 MiB/s wr, 630 op/s 2024-04-06T14:17:14.231 INFO:tasks.workunit.client.0.smithi032.stdout:3/706: dread - d3/d65/d7d/dc6/ff2 zero size 2024-04-06T14:17:14.231 INFO:tasks.workunit.client.0.smithi032.stdout:8/832: write d3/d8/f13 [3884557,32041] 0 2024-04-06T14:17:14.231 INFO:tasks.workunit.client.1.smithi086.stdout:2/534: rename d0/d20/d27/d37/f83 to d0/df/d48/d53/d76/d78/fbd 0 2024-04-06T14:17:14.232 INFO:tasks.workunit.client.0.smithi032.stdout:2/687: read - d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/fa7 zero size 2024-04-06T14:17:14.234 INFO:tasks.workunit.client.1.smithi086.stdout:0/581: dread d0/f3f [0,4194304] 0 2024-04-06T14:17:14.236 INFO:tasks.workunit.client.0.smithi032.stdout:2/688: stat d0/d30/d39/d59/d67/d74/d78 0 2024-04-06T14:17:14.237 INFO:tasks.workunit.client.1.smithi086.stdout:8/617: mkdir d0/db/d32/dd6 0 2024-04-06T14:17:14.237 INFO:tasks.workunit.client.0.smithi032.stdout:8/833: unlink d3/d8/d7/fbc 0 2024-04-06T14:17:14.237 INFO:tasks.workunit.client.1.smithi086.stdout:8/618: truncate d0/d3b/db9/fbd 886460 0 2024-04-06T14:17:14.238 INFO:tasks.workunit.client.0.smithi032.stdout:4/766: truncate d0/df/d23/d31/d50/d57/f64 228077 0 2024-04-06T14:17:14.240 INFO:tasks.workunit.client.1.smithi086.stdout:4/389: creat d0/d2c/d30/d24/d50/f72 x:0 0 0 2024-04-06T14:17:14.242 INFO:tasks.workunit.client.1.smithi086.stdout:9/435: dread d5/d13/d1a/f5a [0,4194304] 0 2024-04-06T14:17:14.245 INFO:tasks.workunit.client.1.smithi086.stdout:4/390: mknod d0/d15/d20/c73 0 2024-04-06T14:17:14.247 INFO:tasks.workunit.client.1.smithi086.stdout:6/598: dread d6/f3c [0,4194304] 0 2024-04-06T14:17:14.248 INFO:tasks.workunit.client.1.smithi086.stdout:6/599: chown d6/d9/db/f7a 1009 1 2024-04-06T14:17:14.249 INFO:tasks.workunit.client.1.smithi086.stdout:6/600: readlink d6/d13/d42/d52/l91 0 2024-04-06T14:17:14.250 INFO:tasks.workunit.client.0.smithi032.stdout:4/767: mknod d0/df/d2b/d46/d58/c102 0 2024-04-06T14:17:14.251 INFO:tasks.workunit.client.0.smithi032.stdout:7/821: dread d16/d20/d54/de4/f31 [0,4194304] 0 2024-04-06T14:17:14.251 INFO:tasks.workunit.client.0.smithi032.stdout:6/793: dwrite d4/d5/d19/d25/d67/fa4 [0,4194304] 0 2024-04-06T14:17:14.252 INFO:tasks.workunit.client.0.smithi032.stdout:2/689: symlink d0/d30/d39/d59/d67/d74/d78/dc2/ldf 0 2024-04-06T14:17:14.253 INFO:tasks.workunit.client.1.smithi086.stdout:2/535: rename d0/d20/d27/d37/d52/db6 to d0/d20/d27/d2b/d35/d36/dbe 0 2024-04-06T14:17:14.256 INFO:tasks.workunit.client.1.smithi086.stdout:0/582: creat d0/d15/d4e/da3/d44/fbc x:0 0 0 2024-04-06T14:17:14.257 INFO:tasks.workunit.client.0.smithi032.stdout:7/822: stat d16/d20/l3c 0 2024-04-06T14:17:14.258 INFO:tasks.workunit.client.0.smithi032.stdout:3/707: dwrite d3/d65/d7d/dc6/fdb [0,4194304] 0 2024-04-06T14:17:14.260 INFO:tasks.workunit.client.0.smithi032.stdout:4/768: rmdir d0/df/d76 39 2024-04-06T14:17:14.260 INFO:tasks.workunit.client.0.smithi032.stdout:9/801: dwrite d2/d15/d1e/d65/d83/d7e/f49 [0,4194304] 0 2024-04-06T14:17:14.262 INFO:tasks.workunit.client.1.smithi086.stdout:8/619: dread d0/db/d32/f5d [0,4194304] 0 2024-04-06T14:17:14.263 INFO:tasks.workunit.client.1.smithi086.stdout:4/391: rename d0/d2c/d38/d54/d61/d65 to d0/d15/d20/d74 0 2024-04-06T14:17:14.264 INFO:tasks.workunit.client.0.smithi032.stdout:6/794: mknod d4/d5/d28/d65/df8/c10c 0 2024-04-06T14:17:14.264 INFO:tasks.workunit.client.1.smithi086.stdout:4/392: fdatasync d0/f10 0 2024-04-06T14:17:14.265 INFO:tasks.workunit.client.1.smithi086.stdout:1/556: truncate db/d11/d1a/f63 79519 0 2024-04-06T14:17:14.267 INFO:tasks.workunit.client.0.smithi032.stdout:7/823: rmdir d16/d20/d54/de4/de0/df9 39 2024-04-06T14:17:14.269 INFO:tasks.workunit.client.1.smithi086.stdout:9/436: creat d5/d13/d47/f92 x:0 0 0 2024-04-06T14:17:14.270 INFO:tasks.workunit.client.0.smithi032.stdout:4/769: symlink d0/df/d23/d31/d50/d5e/l103 0 2024-04-06T14:17:14.271 INFO:tasks.workunit.client.0.smithi032.stdout:6/795: creat d4/d5/d14/d80/f10d x:0 0 0 2024-04-06T14:17:14.273 INFO:tasks.workunit.client.1.smithi086.stdout:6/601: link d6/d9/db/f70 d6/d13/d1a/d64/fd1 0 2024-04-06T14:17:14.273 INFO:tasks.workunit.client.0.smithi032.stdout:8/834: link d3/d8/d7/f47 d3/d8/d12/d2f/d4a/d60/d90/dc5/dd7/f11e 0 2024-04-06T14:17:14.275 INFO:tasks.workunit.client.0.smithi032.stdout:3/708: rename d3/ld to d3/d65/d7d/d9c/dbc/l110 0 2024-04-06T14:17:14.277 INFO:tasks.workunit.client.1.smithi086.stdout:4/393: dread d0/de/f62 [0,4194304] 0 2024-04-06T14:17:14.279 INFO:tasks.workunit.client.1.smithi086.stdout:4/394: rename d0/d15 to d0/d15/d2b/d75 22 2024-04-06T14:17:14.279 INFO:tasks.workunit.client.0.smithi032.stdout:4/770: mknod d0/df/d23/d31/d50/d57/d5b/d9c/c104 0 2024-04-06T14:17:14.281 INFO:tasks.workunit.client.1.smithi086.stdout:9/437: read d5/d18/f3f [8103564,4639] 0 2024-04-06T14:17:14.282 INFO:tasks.workunit.client.1.smithi086.stdout:0/583: creat d0/d1/d4/fbd x:0 0 0 2024-04-06T14:17:14.283 INFO:tasks.workunit.client.0.smithi032.stdout:8/835: dwrite d3/d8/d12/d2f/d4a/d60/d82/d84/fd0 [0,4194304] 0 2024-04-06T14:17:14.285 INFO:tasks.workunit.client.0.smithi032.stdout:2/690: link d0/d30/d39/d59/d67/fc0 d0/d5a/d6a/dcb/fe0 0 2024-04-06T14:17:14.286 INFO:tasks.workunit.client.1.smithi086.stdout:4/395: rename d0/d15/d20/d34/l51 to d0/d2c/d30/l76 0 2024-04-06T14:17:14.286 INFO:tasks.workunit.client.1.smithi086.stdout:1/557: dwrite db/d11/d1f/f5a [0,4194304] 0 2024-04-06T14:17:14.287 INFO:tasks.workunit.client.0.smithi032.stdout:3/709: truncate d3/d65/d7d/d9c/dad/db1/d3c/f79 1785606 0 2024-04-06T14:17:14.288 INFO:tasks.workunit.client.0.smithi032.stdout:9/802: getdents d2/d15/d1e/d65/d83/d31 0 2024-04-06T14:17:14.289 INFO:tasks.workunit.client.1.smithi086.stdout:9/438: creat d5/d13/d47/d5e/f93 x:0 0 0 2024-04-06T14:17:14.289 INFO:tasks.workunit.client.0.smithi032.stdout:6/796: link d4/d5/dbf/f37 d4/d5/d14/d80/f10e 0 2024-04-06T14:17:14.290 INFO:tasks.workunit.client.0.smithi032.stdout:8/836: dread d3/d8/d3a/f5c [0,4194304] 0 2024-04-06T14:17:14.292 INFO:tasks.workunit.client.0.smithi032.stdout:4/771: mkdir d0/df/d23/d31/d50/d57/d5b/d105 0 2024-04-06T14:17:14.294 INFO:tasks.workunit.client.0.smithi032.stdout:2/691: symlink d0/le1 0 2024-04-06T14:17:14.294 INFO:tasks.workunit.client.0.smithi032.stdout:8/837: stat d3/d8/d12/d2f/d4a/d60/d90/fe9 0 2024-04-06T14:17:14.294 INFO:tasks.workunit.client.1.smithi086.stdout:4/396: creat d0/d15/d4a/f77 x:0 0 0 2024-04-06T14:17:14.294 INFO:tasks.workunit.client.1.smithi086.stdout:9/439: truncate d5/d13/d1a/d5f/d88/d14/f80 1368793 0 2024-04-06T14:17:14.295 INFO:tasks.workunit.client.0.smithi032.stdout:7/824: dwrite d16/d92/db3/f103 [0,4194304] 0 2024-04-06T14:17:14.295 INFO:tasks.workunit.client.1.smithi086.stdout:8/620: dread d0/db/f5e [0,4194304] 0 2024-04-06T14:17:14.297 INFO:tasks.workunit.client.1.smithi086.stdout:9/440: fsync d5/d13/d1a/f63 0 2024-04-06T14:17:14.297 INFO:tasks.workunit.client.1.smithi086.stdout:8/621: write d0/d13/fa9 [964940,129377] 0 2024-04-06T14:17:14.301 INFO:tasks.workunit.client.1.smithi086.stdout:6/602: truncate d6/d9/db/f9b 1019284 0 2024-04-06T14:17:14.301 INFO:tasks.workunit.client.0.smithi032.stdout:4/772: chown d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/df9/f101 344 1 2024-04-06T14:17:14.301 INFO:tasks.workunit.client.1.smithi086.stdout:6/603: read d6/d9/db/f28 [3744040,104290] 0 2024-04-06T14:17:14.302 INFO:tasks.workunit.client.1.smithi086.stdout:4/397: mknod d0/d2c/d38/d54/d61/c78 0 2024-04-06T14:17:14.305 INFO:tasks.workunit.client.1.smithi086.stdout:8/622: write d0/d71/d22/d43/d6d/d75/db2/fd2 [691870,77120] 0 2024-04-06T14:17:14.305 INFO:tasks.workunit.client.1.smithi086.stdout:9/441: unlink d5/f1c 0 2024-04-06T14:17:14.306 INFO:tasks.workunit.client.0.smithi032.stdout:7/825: write d16/d20/d48/d8d/f116 [1920410,43340] 0 2024-04-06T14:17:14.308 INFO:tasks.workunit.client.1.smithi086.stdout:4/398: mkdir d0/d2c/d30/d24/d50/d79 0 2024-04-06T14:17:14.309 INFO:tasks.workunit.client.0.smithi032.stdout:2/692: symlink d0/d30/d39/d59/d67/d74/d78/d57/d7c/le2 0 2024-04-06T14:17:14.310 INFO:tasks.workunit.client.0.smithi032.stdout:3/710: dwrite d3/faf [0,4194304] 0 2024-04-06T14:17:14.313 INFO:tasks.workunit.client.0.smithi032.stdout:6/797: getdents d4/d5/d19/d25 0 2024-04-06T14:17:14.314 INFO:tasks.workunit.client.0.smithi032.stdout:9/803: mkdir d2/d15/d55/d59/d93/d95/db8/d81/d4a/ded/d100 0 2024-04-06T14:17:14.315 INFO:tasks.workunit.client.1.smithi086.stdout:4/399: dwrite d0/f36 [0,4194304] 0 2024-04-06T14:17:14.317 INFO:tasks.workunit.client.1.smithi086.stdout:7/509: write d7/d1d/f6b [642895,44932] 0 2024-04-06T14:17:14.319 INFO:tasks.workunit.client.1.smithi086.stdout:8/623: unlink d0/d71/d16/c30 0 2024-04-06T14:17:14.322 INFO:tasks.workunit.client.1.smithi086.stdout:9/442: mkdir d5/d13/d1a/d5f/d88/d14/d19/d94 0 2024-04-06T14:17:14.322 INFO:tasks.workunit.client.0.smithi032.stdout:3/711: mkdir d3/d65/d7d/d9c/dad/d111 0 2024-04-06T14:17:14.324 INFO:tasks.workunit.client.1.smithi086.stdout:4/400: mkdir d0/d2c/d38/d54/d61/d7a 0 2024-04-06T14:17:14.325 INFO:tasks.workunit.client.0.smithi032.stdout:2/693: write d0/f6c [2605538,64578] 0 2024-04-06T14:17:14.329 INFO:tasks.workunit.client.1.smithi086.stdout:2/536: getdents d0/d20/d27/d37 0 2024-04-06T14:17:14.329 INFO:tasks.workunit.client.1.smithi086.stdout:2/537: chown d0/df/d48/d53/d76/f9b 2902437 1 2024-04-06T14:17:14.329 INFO:tasks.workunit.client.0.smithi032.stdout:3/712: chown d3/d65/d7d/d9c/dad/db1/f40 326 1 2024-04-06T14:17:14.329 INFO:tasks.workunit.client.1.smithi086.stdout:2/538: chown d0/d20/d27/d2b/d35/d36/d4d 120694339 1 2024-04-06T14:17:14.330 INFO:tasks.workunit.client.1.smithi086.stdout:7/510: rename d7/d10/d11/d1c/c2f to d7/d10/d11/d1c/d24/d2c/d3c/d7a/cae 0 2024-04-06T14:17:14.331 INFO:tasks.workunit.client.0.smithi032.stdout:2/694: unlink d0/d4e/f69 0 2024-04-06T14:17:14.332 INFO:tasks.workunit.client.1.smithi086.stdout:3/528: sync 2024-04-06T14:17:14.333 INFO:tasks.workunit.client.0.smithi032.stdout:8/838: truncate d3/d8/d12/d2f/d4a/d60/d82/d84/fcc 485500 0 2024-04-06T14:17:14.334 INFO:tasks.workunit.client.0.smithi032.stdout:6/798: dwrite d4/d5/d8/d4b/f103 [0,4194304] 0 2024-04-06T14:17:14.335 INFO:tasks.workunit.client.1.smithi086.stdout:4/401: creat d0/d15/d20/d74/f7b x:0 0 0 2024-04-06T14:17:14.337 INFO:tasks.workunit.client.0.smithi032.stdout:2/695: mknod d0/d6e/dae/ce3 0 2024-04-06T14:17:14.337 INFO:tasks.workunit.client.1.smithi086.stdout:5/443: dwrite d0/d16/f63 [0,4194304] 0 2024-04-06T14:17:14.342 INFO:tasks.workunit.client.0.smithi032.stdout:7/826: dwrite d16/d20/d54/de4/d5f/ddd/f113 [0,4194304] 0 2024-04-06T14:17:14.342 INFO:tasks.workunit.client.0.smithi032.stdout:6/799: write d4/d5/d19/f46 [503863,44971] 0 2024-04-06T14:17:14.342 INFO:tasks.workunit.client.0.smithi032.stdout:9/804: dread d2/d15/d1e/d65/d83/d7e/f1d [0,4194304] 0 2024-04-06T14:17:14.344 INFO:tasks.workunit.client.0.smithi032.stdout:3/713: rename d3/d65/d7d/d9c/dad/db1/d3c/dc9/ffb to d3/d65/d7d/dc6/df0/f112 0 2024-04-06T14:17:14.345 INFO:tasks.workunit.client.0.smithi032.stdout:2/696: rmdir d0/d30/d39/d59/d67/d74 39 2024-04-06T14:17:14.345 INFO:tasks.workunit.client.1.smithi086.stdout:3/529: read d5/d26/f3c [3017283,5918] 0 2024-04-06T14:17:14.347 INFO:tasks.workunit.client.0.smithi032.stdout:6/800: creat d4/d5/d8/d2f/d9b/f10f x:0 0 0 2024-04-06T14:17:14.348 INFO:tasks.workunit.client.1.smithi086.stdout:8/624: rmdir d0/db/d32/dab/dad 0 2024-04-06T14:17:14.350 INFO:tasks.workunit.client.0.smithi032.stdout:3/714: rmdir d3/dcf 39 2024-04-06T14:17:14.350 INFO:tasks.workunit.client.1.smithi086.stdout:5/444: fsync d0/d43/f62 0 2024-04-06T14:17:14.351 INFO:tasks.workunit.client.1.smithi086.stdout:5/445: fdatasync d0/d16/d18/d2a/d50/f5f 0 2024-04-06T14:17:14.352 INFO:tasks.workunit.client.1.smithi086.stdout:5/446: stat d0/d17/f79 0 2024-04-06T14:17:14.352 INFO:tasks.workunit.client.1.smithi086.stdout:9/443: link d5/d13/d1a/d5f/d88/d14/d19/f4b d5/d13/d1a/d5f/f95 0 2024-04-06T14:17:14.352 INFO:tasks.workunit.client.0.smithi032.stdout:4/773: dread d0/df/d23/d31/f68 [0,4194304] 0 2024-04-06T14:17:14.353 INFO:tasks.workunit.client.0.smithi032.stdout:8/839: dwrite d3/d8/d12/d2f/d46/fa8 [8388608,4194304] 0 2024-04-06T14:17:14.354 INFO:tasks.workunit.client.1.smithi086.stdout:9/444: write d5/d13/d47/f92 [82468,107424] 0 2024-04-06T14:17:14.354 INFO:tasks.workunit.client.0.smithi032.stdout:4/774: chown d0/l9 42454 1 2024-04-06T14:17:14.355 INFO:tasks.workunit.client.0.smithi032.stdout:4/775: dread - d0/f5c zero size 2024-04-06T14:17:14.357 INFO:tasks.workunit.client.0.smithi032.stdout:4/776: write d0/d82/fc5 [3250515,84310] 0 2024-04-06T14:17:14.357 INFO:tasks.workunit.client.1.smithi086.stdout:8/625: rmdir d0/d71/d82 39 2024-04-06T14:17:14.359 INFO:tasks.workunit.client.0.smithi032.stdout:5/711: sync 2024-04-06T14:17:14.360 INFO:tasks.workunit.client.1.smithi086.stdout:4/402: creat d0/d2c/d30/d24/d50/d79/f7c x:0 0 0 2024-04-06T14:17:14.361 INFO:tasks.workunit.client.0.smithi032.stdout:7/827: symlink d16/d20/d54/de4/d2d/d30/l128 0 2024-04-06T14:17:14.363 INFO:tasks.workunit.client.0.smithi032.stdout:6/801: mkdir d4/d5/d28/d65/d6e/ddb/d110 0 2024-04-06T14:17:14.364 INFO:tasks.workunit.client.0.smithi032.stdout:4/777: mknod d0/df/d2b/c106 0 2024-04-06T14:17:14.365 INFO:tasks.workunit.client.0.smithi032.stdout:8/840: symlink d3/d8/l11f 0 2024-04-06T14:17:14.365 INFO:tasks.workunit.client.1.smithi086.stdout:7/511: rename d7/d10/d11/d1c/d24/d2c/d3c/d7a/da4 to d7/d10/d11/d1c/d24/d4d/daf 0 2024-04-06T14:17:14.366 INFO:tasks.workunit.client.0.smithi032.stdout:4/778: stat d0/d82/d8e 0 2024-04-06T14:17:14.366 INFO:tasks.workunit.client.0.smithi032.stdout:3/715: truncate d3/d65/d7d/d9c/db8/fc2 4745937 0 2024-04-06T14:17:14.369 INFO:tasks.workunit.client.0.smithi032.stdout:5/712: symlink d16/d58/d54/d74/d8f/lfe 0 2024-04-06T14:17:14.369 INFO:tasks.workunit.client.1.smithi086.stdout:4/403: dread d0/d15/d20/f27 [0,4194304] 0 2024-04-06T14:17:14.374 INFO:tasks.workunit.client.0.smithi032.stdout:8/841: symlink d3/d8/d3a/d7b/db9/df8/d106/l120 0 2024-04-06T14:17:14.376 INFO:tasks.workunit.client.1.smithi086.stdout:6/604: write d6/d9/f17 [1044128,60204] 0 2024-04-06T14:17:14.376 INFO:tasks.workunit.client.1.smithi086.stdout:0/584: dwrite d0/d15/d26/d33/f66 [0,4194304] 0 2024-04-06T14:17:14.376 INFO:tasks.workunit.client.0.smithi032.stdout:6/802: dread d4/d5/dc5/f50 [0,4194304] 0 2024-04-06T14:17:14.377 INFO:tasks.workunit.client.0.smithi032.stdout:8/842: rmdir d3/d8/d12/d2f/d4a/d60/d82 39 2024-04-06T14:17:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:13 smithi086 ceph-mon[47925]: pgmap v24: 65 pgs: 65 active+clean; 2.6 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 101 MiB/s rd, 337 MiB/s wr, 630 op/s 2024-04-06T14:17:14.378 INFO:tasks.workunit.client.1.smithi086.stdout:1/558: dwrite db/d11/d42/f50 [0,4194304] 0 2024-04-06T14:17:14.382 INFO:tasks.workunit.client.1.smithi086.stdout:5/447: getdents d0 0 2024-04-06T14:17:14.384 INFO:tasks.workunit.client.0.smithi032.stdout:6/803: creat d4/d5/d28/d65/df8/f111 x:0 0 0 2024-04-06T14:17:14.384 INFO:tasks.workunit.client.1.smithi086.stdout:4/404: fdatasync d0/d2c/d30/d24/f31 0 2024-04-06T14:17:14.387 INFO:tasks.workunit.client.1.smithi086.stdout:5/448: mkdir d0/d16/d71/d89 0 2024-04-06T14:17:14.388 INFO:tasks.workunit.client.1.smithi086.stdout:9/445: getdents d5/d13/d1a/d5f/d88/d14/d19 0 2024-04-06T14:17:14.388 INFO:tasks.workunit.client.1.smithi086.stdout:0/585: mkdir d0/d1/d25/dbe 0 2024-04-06T14:17:14.393 INFO:tasks.workunit.client.0.smithi032.stdout:6/804: dread d4/d5/d19/f46 [0,4194304] 0 2024-04-06T14:17:14.393 INFO:tasks.workunit.client.0.smithi032.stdout:2/697: write d0/d30/d39/d59/d67/d74/d78/d57/dd3/fdc [4783787,73743] 0 2024-04-06T14:17:14.394 INFO:tasks.workunit.client.0.smithi032.stdout:8/843: dread d3/d8/d7/fa7 [0,4194304] 0 2024-04-06T14:17:14.394 INFO:tasks.workunit.client.1.smithi086.stdout:3/530: dread d5/d36/d5a/d6d/d72/d58/f79 [0,4194304] 0 2024-04-06T14:17:14.394 INFO:tasks.workunit.client.1.smithi086.stdout:7/512: dread d7/d10/d11/d1c/d24/fa2 [0,4194304] 0 2024-04-06T14:17:14.397 INFO:tasks.workunit.client.1.smithi086.stdout:6/605: rename d6/d13/d1a/d22/dcf/db2/fa7 to d6/d9/db/fd2 0 2024-04-06T14:17:14.399 INFO:tasks.workunit.client.1.smithi086.stdout:7/513: stat d7/d10/d11/d1c/d24/d2c/f48 0 2024-04-06T14:17:14.399 INFO:tasks.workunit.client.0.smithi032.stdout:6/805: write d4/d5/d28/fbe [1411832,89646] 0 2024-04-06T14:17:14.400 INFO:tasks.workunit.client.1.smithi086.stdout:3/531: rename d5/d36 to d5/d36/d5a/d6d/d72/d9a/da4 22 2024-04-06T14:17:14.402 INFO:tasks.workunit.client.1.smithi086.stdout:2/539: dwrite d0/d20/d27/d2b/d35/d36/d4d/d6f/f77 [0,4194304] 0 2024-04-06T14:17:14.404 INFO:tasks.workunit.client.0.smithi032.stdout:9/805: dwrite d2/d15/d1e/d65/d83/d31/d39/f5a [0,4194304] 0 2024-04-06T14:17:14.405 INFO:tasks.workunit.client.1.smithi086.stdout:5/449: rmdir d0/d16/d65 39 2024-04-06T14:17:14.405 INFO:tasks.workunit.client.0.smithi032.stdout:2/698: symlink d0/d30/d39/d59/d67/d80/le4 0 2024-04-06T14:17:14.406 INFO:tasks.workunit.client.1.smithi086.stdout:0/586: fdatasync d0/d1/d2/d1a/d79/f8c 0 2024-04-06T14:17:14.407 INFO:tasks.workunit.client.1.smithi086.stdout:6/606: unlink d6/c68 0 2024-04-06T14:17:14.410 INFO:tasks.workunit.client.0.smithi032.stdout:6/806: write d4/d5/d19/d96/faa [33977,6223] 0 2024-04-06T14:17:14.411 INFO:tasks.workunit.client.1.smithi086.stdout:1/559: creat db/d11/d1a/d2a/d9a/d68/fad x:0 0 0 2024-04-06T14:17:14.414 INFO:tasks.workunit.client.1.smithi086.stdout:6/607: chown d6/d13/d1a/d22/c39 373 1 2024-04-06T14:17:14.415 INFO:tasks.workunit.client.1.smithi086.stdout:8/626: dwrite d0/db/d32/f5d [0,4194304] 0 2024-04-06T14:17:14.419 INFO:tasks.workunit.client.1.smithi086.stdout:3/532: fdatasync d5/d36/d5a/d6d/d72/f7a 0 2024-04-06T14:17:14.423 INFO:tasks.workunit.client.1.smithi086.stdout:5/450: rmdir d0/d16/d18/d21/d23/d28 39 2024-04-06T14:17:14.424 INFO:tasks.workunit.client.1.smithi086.stdout:8/627: chown d0/d71/d16/d52/fc8 1405718 1 2024-04-06T14:17:14.424 INFO:tasks.workunit.client.1.smithi086.stdout:4/405: dwrite d0/f59 [0,4194304] 0 2024-04-06T14:17:14.425 INFO:tasks.workunit.client.0.smithi032.stdout:7/828: dread fd [4194304,4194304] 0 2024-04-06T14:17:14.425 INFO:tasks.workunit.client.0.smithi032.stdout:9/806: unlink d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8f/laa 0 2024-04-06T14:17:14.425 INFO:tasks.workunit.client.0.smithi032.stdout:5/713: dwrite f11 [0,4194304] 0 2024-04-06T14:17:14.426 INFO:tasks.workunit.client.1.smithi086.stdout:5/451: stat d0/d16/f39 0 2024-04-06T14:17:14.427 INFO:tasks.workunit.client.1.smithi086.stdout:7/514: dwrite d7/d10/d11/d1c/f58 [4194304,4194304] 0 2024-04-06T14:17:14.428 INFO:tasks.workunit.client.0.smithi032.stdout:6/807: read d4/d5/d35/f39 [241022,12837] 0 2024-04-06T14:17:14.433 INFO:tasks.workunit.client.0.smithi032.stdout:4/779: dread d0/df/d23/d31/d50/d5e/fbf [0,4194304] 0 2024-04-06T14:17:14.434 INFO:tasks.workunit.client.0.smithi032.stdout:9/807: chown d2/d15/d55/d59/d93/d95/db8/d81/fb0 2 1 2024-04-06T14:17:14.438 INFO:tasks.workunit.client.0.smithi032.stdout:3/716: dread d3/d65/f69 [4194304,4194304] 0 2024-04-06T14:17:14.438 INFO:tasks.workunit.client.0.smithi032.stdout:6/808: creat d4/d5/d19/d96/f112 x:0 0 0 2024-04-06T14:17:14.438 INFO:tasks.workunit.client.0.smithi032.stdout:9/808: fsync d2/d15/d1e/d65/d83/f94 0 2024-04-06T14:17:14.439 INFO:tasks.workunit.client.0.smithi032.stdout:7/829: getdents d16/d20/d86 0 2024-04-06T14:17:14.439 INFO:tasks.workunit.client.1.smithi086.stdout:6/608: rename d6/d13/d42/c80 to d6/cd3 0 2024-04-06T14:17:14.443 INFO:tasks.workunit.client.0.smithi032.stdout:3/717: creat d3/d65/d7d/d9c/db8/f113 x:0 0 0 2024-04-06T14:17:14.443 INFO:tasks.workunit.client.0.smithi032.stdout:5/714: dwrite d16/d58/d54/d74/d8f/fe8 [0,4194304] 0 2024-04-06T14:17:14.446 INFO:tasks.workunit.client.0.smithi032.stdout:3/718: chown d3/d16/f72 149373 1 2024-04-06T14:17:14.446 INFO:tasks.workunit.client.1.smithi086.stdout:8/628: dwrite d0/db/f12 [4194304,4194304] 0 2024-04-06T14:17:14.448 INFO:tasks.workunit.client.0.smithi032.stdout:4/780: getdents d0/df/d1f/d42/d45 0 2024-04-06T14:17:14.449 INFO:tasks.workunit.client.1.smithi086.stdout:4/406: rename d0/l9 to d0/d2c/d30/d24/l7d 0 2024-04-06T14:17:14.449 INFO:tasks.workunit.client.0.smithi032.stdout:7/830: symlink d16/d20/d54/de4/d2d/d30/d3e/da7/db5/l129 0 2024-04-06T14:17:14.450 INFO:tasks.workunit.client.1.smithi086.stdout:9/446: dread d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f81 [0,4194304] 0 2024-04-06T14:17:14.453 INFO:tasks.workunit.client.1.smithi086.stdout:7/515: dwrite d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/f89 [0,4194304] 0 2024-04-06T14:17:14.454 INFO:tasks.workunit.client.1.smithi086.stdout:2/540: write d0/d20/d27/d2b/d35/d36/d4a/f86 [1460991,50540] 0 2024-04-06T14:17:14.458 INFO:tasks.workunit.client.0.smithi032.stdout:3/719: write d3/d65/d7d/d9c/dad/db1/fc3 [610660,58033] 0 2024-04-06T14:17:14.458 INFO:tasks.workunit.client.0.smithi032.stdout:5/715: chown d16/d18/d33/caf 17211 1 2024-04-06T14:17:14.459 INFO:tasks.workunit.client.1.smithi086.stdout:1/560: write db/d11/d1a/d2a/d52/d6b/f97 [573759,44243] 0 2024-04-06T14:17:14.460 INFO:tasks.workunit.client.1.smithi086.stdout:6/609: dwrite d6/d13/d42/d52/fc1 [0,4194304] 0 2024-04-06T14:17:14.460 INFO:tasks.workunit.client.0.smithi032.stdout:6/809: getdents d4/d5/dbf/d61/db9 0 2024-04-06T14:17:14.461 INFO:tasks.workunit.client.0.smithi032.stdout:7/831: creat d16/d20/d7a/f12a x:0 0 0 2024-04-06T14:17:14.462 INFO:tasks.workunit.client.0.smithi032.stdout:3/720: chown d3/d65/c2f 155543 1 2024-04-06T14:17:14.467 INFO:tasks.workunit.client.0.smithi032.stdout:2/699: dread d0/f23 [4194304,4194304] 0 2024-04-06T14:17:14.467 INFO:tasks.workunit.client.0.smithi032.stdout:4/781: dread d0/df/d23/d31/d50/d5e/fbf [0,4194304] 0 2024-04-06T14:17:14.468 INFO:tasks.workunit.client.0.smithi032.stdout:5/716: mknod d16/d18/cff 0 2024-04-06T14:17:14.468 INFO:tasks.workunit.client.1.smithi086.stdout:8/629: creat d0/d3b/db9/fd7 x:0 0 0 2024-04-06T14:17:14.468 INFO:tasks.workunit.client.1.smithi086.stdout:0/587: dwrite d0/d1/d25/f61 [0,4194304] 0 2024-04-06T14:17:14.469 INFO:tasks.workunit.client.0.smithi032.stdout:6/810: mknod d4/d5/dbf/d51/c113 0 2024-04-06T14:17:14.472 INFO:tasks.workunit.client.0.smithi032.stdout:3/721: mknod d3/d65/d7d/d9c/dad/db1/d3c/c114 0 2024-04-06T14:17:14.472 INFO:tasks.workunit.client.0.smithi032.stdout:7/832: stat d16/d20/d54/de4/d2d/d30/c43 0 2024-04-06T14:17:14.472 INFO:tasks.workunit.client.1.smithi086.stdout:7/516: mkdir d7/d10/d11/d21/db0 0 2024-04-06T14:17:14.472 INFO:tasks.workunit.client.1.smithi086.stdout:0/588: creat d0/d15/d26/d33/d9f/fbf x:0 0 0 2024-04-06T14:17:14.473 INFO:tasks.workunit.client.0.smithi032.stdout:4/782: unlink d0/df/d2b/d66/df0/lf1 0 2024-04-06T14:17:14.474 INFO:tasks.workunit.client.1.smithi086.stdout:1/561: dread db/d11/d42/d53/f64 [0,4194304] 0 2024-04-06T14:17:14.476 INFO:tasks.workunit.client.0.smithi032.stdout:6/811: creat d4/d69/f114 x:0 0 0 2024-04-06T14:17:14.479 INFO:tasks.workunit.client.0.smithi032.stdout:9/809: truncate d2/d15/d55/d59/d93/d95/db8/d81/d77/f78 565636 0 2024-04-06T14:17:14.479 INFO:tasks.workunit.client.0.smithi032.stdout:4/783: stat d0/df/d1f/d42/f99 0 2024-04-06T14:17:14.480 INFO:tasks.workunit.client.0.smithi032.stdout:3/722: truncate d3/d65/d7d/d9c/dad/db1/f6c 4838029 0 2024-04-06T14:17:14.480 INFO:tasks.workunit.client.1.smithi086.stdout:2/541: getdents d0/df/d48 0 2024-04-06T14:17:14.481 INFO:tasks.workunit.client.0.smithi032.stdout:5/717: rmdir d16/d18/dae/dbb/ddc 39 2024-04-06T14:17:14.482 INFO:tasks.workunit.client.0.smithi032.stdout:2/700: creat d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/fe5 x:0 0 0 2024-04-06T14:17:14.482 INFO:tasks.workunit.client.0.smithi032.stdout:6/812: creat d4/d5/d34/f115 x:0 0 0 2024-04-06T14:17:14.483 INFO:tasks.workunit.client.1.smithi086.stdout:2/542: fdatasync d0/d20/d27/d2b/d35/d36/d4d/d6f/f77 0 2024-04-06T14:17:14.483 INFO:tasks.workunit.client.1.smithi086.stdout:7/517: chown d7/d10/d11/d1c/l26 27489252 1 2024-04-06T14:17:14.483 INFO:tasks.workunit.client.0.smithi032.stdout:3/723: creat d3/d65/f115 x:0 0 0 2024-04-06T14:17:14.485 INFO:tasks.workunit.client.1.smithi086.stdout:8/630: dwrite d0/db/d32/f5d [0,4194304] 0 2024-04-06T14:17:14.485 INFO:tasks.workunit.client.1.smithi086.stdout:4/407: write d0/d15/f3c [4145004,76137] 0 2024-04-06T14:17:14.485 INFO:tasks.workunit.client.0.smithi032.stdout:7/833: mkdir d16/d20/d6e/d10e/d12b 0 2024-04-06T14:17:14.486 INFO:tasks.workunit.client.0.smithi032.stdout:6/813: write d4/d5/d19/d96/f112 [740260,30399] 0 2024-04-06T14:17:14.490 INFO:tasks.workunit.client.0.smithi032.stdout:2/701: mkdir d0/d30/d39/d59/d67/d74/d78/d57/d7c/de6 0 2024-04-06T14:17:14.492 INFO:tasks.workunit.client.1.smithi086.stdout:9/447: truncate d5/d13/f4f 3123128 0 2024-04-06T14:17:14.493 INFO:tasks.workunit.client.0.smithi032.stdout:6/814: symlink d4/d5/dc5/d82/d9d/l116 0 2024-04-06T14:17:14.493 INFO:tasks.workunit.client.1.smithi086.stdout:7/518: mknod d7/d10/d11/d1c/d24/d4d/d5c/d66/cb1 0 2024-04-06T14:17:14.495 INFO:tasks.workunit.client.0.smithi032.stdout:6/815: fdatasync d4/d5/dc5/d82/fb0 0 2024-04-06T14:17:14.498 INFO:tasks.workunit.client.0.smithi032.stdout:4/784: getdents d0/df/d2b/d46/d58 0 2024-04-06T14:17:14.499 INFO:tasks.workunit.client.0.smithi032.stdout:4/785: write d0/fe9 [467869,6723] 0 2024-04-06T14:17:14.500 INFO:tasks.workunit.client.1.smithi086.stdout:6/610: dwrite d6/f7 [4194304,4194304] 0 2024-04-06T14:17:14.501 INFO:tasks.workunit.client.1.smithi086.stdout:7/519: write d7/d10/d11/d1c/d24/d4d/d5c/d66/fa3 [1025152,31503] 0 2024-04-06T14:17:14.502 INFO:tasks.workunit.client.1.smithi086.stdout:2/543: write d0/d20/d27/f8a [40394,39437] 0 2024-04-06T14:17:14.502 INFO:tasks.workunit.client.0.smithi032.stdout:3/724: creat d3/d65/d7d/d9c/dad/f116 x:0 0 0 2024-04-06T14:17:14.503 INFO:tasks.workunit.client.1.smithi086.stdout:4/408: mknod d0/d2c/d38/d54/d61/d7a/c7e 0 2024-04-06T14:17:14.506 INFO:tasks.workunit.client.1.smithi086.stdout:4/409: chown d0/d15/d20/c56 12665202 1 2024-04-06T14:17:14.507 INFO:tasks.workunit.client.1.smithi086.stdout:6/611: chown d6/d20 1021709179 1 2024-04-06T14:17:14.508 INFO:tasks.workunit.client.1.smithi086.stdout:7/520: chown d7/d10/d11/d1c/d24/f4e 2 1 2024-04-06T14:17:14.508 INFO:tasks.workunit.client.1.smithi086.stdout:0/589: dwrite d0/d15/d4e/da3/d46/d52/f7f [0,4194304] 0 2024-04-06T14:17:14.508 INFO:tasks.workunit.client.0.smithi032.stdout:6/816: dread d4/d5/d14/d80/f10e [0,4194304] 0 2024-04-06T14:17:14.510 INFO:tasks.workunit.client.0.smithi032.stdout:6/817: truncate d4/d5/dbf/d51/fec 110346 0 2024-04-06T14:17:14.511 INFO:tasks.workunit.client.1.smithi086.stdout:7/521: readlink d7/d10/d11/d21/l43 0 2024-04-06T14:17:14.511 INFO:tasks.workunit.client.0.smithi032.stdout:6/818: stat d4/d5/d28/d65/d6e/ddb 0 2024-04-06T14:17:14.511 INFO:tasks.workunit.client.1.smithi086.stdout:9/448: rename d5/cd to d5/c96 0 2024-04-06T14:17:14.513 INFO:tasks.workunit.client.0.smithi032.stdout:7/834: getdents d16/d20/d54/de4/d2d/d30/dbb 0 2024-04-06T14:17:14.513 INFO:tasks.workunit.client.0.smithi032.stdout:6/819: fdatasync d4/d5/dbf/f33 0 2024-04-06T14:17:14.513 INFO:tasks.workunit.client.1.smithi086.stdout:8/631: truncate d0/f26 1682591 0 2024-04-06T14:17:14.514 INFO:tasks.workunit.client.0.smithi032.stdout:5/718: write d16/d58/f6f [991666,72479] 0 2024-04-06T14:17:14.515 INFO:tasks.workunit.client.1.smithi086.stdout:4/410: mknod d0/d2c/d30/c7f 0 2024-04-06T14:17:14.516 INFO:tasks.workunit.client.0.smithi032.stdout:4/786: mknod d0/df/d76/c107 0 2024-04-06T14:17:14.516 INFO:tasks.workunit.client.1.smithi086.stdout:1/562: dread db/d13/f3d [0,4194304] 0 2024-04-06T14:17:14.517 INFO:tasks.workunit.client.1.smithi086.stdout:0/590: creat d0/d1/d25/fc0 x:0 0 0 2024-04-06T14:17:14.518 INFO:tasks.workunit.client.1.smithi086.stdout:8/632: stat d0/db/cd 0 2024-04-06T14:17:14.518 INFO:tasks.workunit.client.1.smithi086.stdout:7/522: read d7/d10/d11/d1c/d24/f4e [3036191,79676] 0 2024-04-06T14:17:14.521 INFO:tasks.workunit.client.1.smithi086.stdout:6/612: mknod d6/d13/d5f/d60/d45/cd4 0 2024-04-06T14:17:14.521 INFO:tasks.workunit.client.0.smithi032.stdout:2/702: write d0/d30/f4d [942312,20998] 0 2024-04-06T14:17:14.521 INFO:tasks.workunit.client.1.smithi086.stdout:7/523: dread - d7/d10/d11/d1c/d24/d2c/d3c/d7a/f95 zero size 2024-04-06T14:17:14.522 INFO:tasks.workunit.client.1.smithi086.stdout:9/449: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f97 x:0 0 0 2024-04-06T14:17:14.523 INFO:tasks.workunit.client.1.smithi086.stdout:4/411: creat d0/d2c/d38/d54/f80 x:0 0 0 2024-04-06T14:17:14.523 INFO:tasks.workunit.client.1.smithi086.stdout:0/591: truncate d0/d1/d2/d1a/d51/f60 5140126 0 2024-04-06T14:17:14.524 INFO:tasks.workunit.client.0.smithi032.stdout:5/719: mkdir d16/d18/dae/dbb/ddc/d100 0 2024-04-06T14:17:14.527 INFO:tasks.workunit.client.1.smithi086.stdout:7/524: fdatasync d7/d94/fa9 0 2024-04-06T14:17:14.531 INFO:tasks.workunit.client.1.smithi086.stdout:4/412: creat d0/d2c/f81 x:0 0 0 2024-04-06T14:17:14.532 INFO:tasks.workunit.client.1.smithi086.stdout:9/450: mkdir d5/d13/d47/d98 0 2024-04-06T14:17:14.533 INFO:tasks.workunit.client.1.smithi086.stdout:2/544: dread d0/d20/d27/f43 [0,4194304] 0 2024-04-06T14:17:14.533 INFO:tasks.workunit.client.1.smithi086.stdout:1/563: dwrite db/d13/f23 [0,4194304] 0 2024-04-06T14:17:14.537 INFO:tasks.workunit.client.0.smithi032.stdout:6/820: link d4/d5/dc5/d82/fb0 d4/d5/d8/d2b/dfb/f117 0 2024-04-06T14:17:14.538 INFO:tasks.workunit.client.0.smithi032.stdout:5/720: chown d16/f90 139153 1 2024-04-06T14:17:14.542 INFO:tasks.workunit.client.1.smithi086.stdout:2/545: mkdir d0/d20/d80/dbf 0 2024-04-06T14:17:14.544 INFO:tasks.workunit.client.0.smithi032.stdout:9/810: dwrite d2/d15/d55/d59/d93/d95/db8/d81/d77/f78 [0,4194304] 0 2024-04-06T14:17:14.548 INFO:tasks.workunit.client.0.smithi032.stdout:4/787: truncate d0/df/f8b 693476 0 2024-04-06T14:17:14.548 INFO:tasks.workunit.client.0.smithi032.stdout:6/821: chown d4/d5/d8/d2b/dfb 23 1 2024-04-06T14:17:14.550 INFO:tasks.workunit.client.0.smithi032.stdout:0/615: sync 2024-04-06T14:17:14.551 INFO:tasks.workunit.client.1.smithi086.stdout:4/413: dwrite d0/d15/d20/d74/f7b [0,4194304] 0 2024-04-06T14:17:14.551 INFO:tasks.workunit.client.1.smithi086.stdout:9/451: rename d5/d13/d1a/f5a to d5/d13/d1a/d5f/d88/d14/d19/d25/f99 0 2024-04-06T14:17:14.552 INFO:tasks.workunit.client.0.smithi032.stdout:7/835: dwrite d16/d20/d54/de4/d2d/faf [0,4194304] 0 2024-04-06T14:17:14.552 INFO:tasks.workunit.client.0.smithi032.stdout:3/725: dwrite d3/d65/f32 [4194304,4194304] 0 2024-04-06T14:17:14.554 INFO:tasks.workunit.client.0.smithi032.stdout:6/822: readlink d4/d5/d8/d2f/l3a 0 2024-04-06T14:17:14.555 INFO:tasks.workunit.client.0.smithi032.stdout:2/703: dread d0/d5a/fb1 [4194304,4194304] 0 2024-04-06T14:17:14.556 INFO:tasks.workunit.client.1.smithi086.stdout:7/525: dwrite d7/d10/d11/d1c/d24/d2c/d3c/f8a [0,4194304] 0 2024-04-06T14:17:14.556 INFO:tasks.workunit.client.1.smithi086.stdout:6/613: truncate d6/d13/d1a/da0/fcc 3744931 0 2024-04-06T14:17:14.557 INFO:tasks.workunit.client.0.smithi032.stdout:5/721: mknod d16/d18/d33/d35/d4d/c101 0 2024-04-06T14:17:14.558 INFO:tasks.workunit.client.0.smithi032.stdout:6/823: truncate d4/d5/d19/d25/f88 273344 0 2024-04-06T14:17:14.561 INFO:tasks.workunit.client.1.smithi086.stdout:2/546: stat d0/d20/d27/d2b/d35/d36/d4d/d59/c69 0 2024-04-06T14:17:14.563 INFO:tasks.workunit.client.0.smithi032.stdout:9/811: unlink d2/d15/d1e/d65/d6b/d92/fbc 0 2024-04-06T14:17:14.563 INFO:tasks.workunit.client.0.smithi032.stdout:0/616: rmdir d3/da/d25 39 2024-04-06T14:17:14.564 INFO:tasks.workunit.client.1.smithi086.stdout:8/633: dwrite d0/d71/d22/f8e [0,4194304] 0 2024-04-06T14:17:14.565 INFO:tasks.workunit.client.1.smithi086.stdout:0/592: dread d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:14.565 INFO:tasks.workunit.client.0.smithi032.stdout:5/722: write d16/d58/d54/d74/d8f/fe8 [2629592,124821] 0 2024-04-06T14:17:14.565 INFO:tasks.workunit.client.0.smithi032.stdout:7/836: chown d16/cc9 5187 1 2024-04-06T14:17:14.565 INFO:tasks.workunit.client.0.smithi032.stdout:9/812: readlink d2/d15/d1e/d65/d83/l30 0 2024-04-06T14:17:14.566 INFO:tasks.workunit.client.1.smithi086.stdout:7/526: rmdir d7/d10/d11/d1c/d24/d4d 39 2024-04-06T14:17:14.568 INFO:tasks.workunit.client.0.smithi032.stdout:1/867: sync 2024-04-06T14:17:14.570 INFO:tasks.workunit.client.1.smithi086.stdout:6/614: chown d6/d13/d1a/d22/dcf/db2/c9a 1115 1 2024-04-06T14:17:14.572 INFO:tasks.workunit.client.0.smithi032.stdout:8/844: sync 2024-04-06T14:17:14.576 INFO:tasks.workunit.client.0.smithi032.stdout:7/837: mknod d16/d20/d54/c12c 0 2024-04-06T14:17:14.577 INFO:tasks.workunit.client.0.smithi032.stdout:5/723: fdatasync d16/d18/d33/d45/d52/d39/d86/fbe 0 2024-04-06T14:17:14.578 INFO:tasks.workunit.client.0.smithi032.stdout:0/617: mknod d3/da/d25/d2b/d4a/d79/da8/cd2 0 2024-04-06T14:17:14.580 INFO:tasks.workunit.client.1.smithi086.stdout:1/564: dwrite db/d13/d71/f7f [0,4194304] 0 2024-04-06T14:17:14.581 INFO:tasks.workunit.client.0.smithi032.stdout:3/726: mknod d3/d16/ddd/c117 0 2024-04-06T14:17:14.581 INFO:tasks.workunit.client.1.smithi086.stdout:2/547: rename d0/d20/d27/d2b/d35/f4e to d0/d20/d44/dbb/fc0 0 2024-04-06T14:17:14.585 INFO:tasks.workunit.client.0.smithi032.stdout:5/724: write d16/d18/d33/d45/d52/d39/d86/f93 [724183,33911] 0 2024-04-06T14:17:14.585 INFO:tasks.workunit.client.0.smithi032.stdout:8/845: dread - d3/d8/d3a/d4b/fbe zero size 2024-04-06T14:17:14.585 INFO:tasks.workunit.client.0.smithi032.stdout:1/868: mkdir d0/d10/d16/d118 0 2024-04-06T14:17:14.586 INFO:tasks.workunit.client.0.smithi032.stdout:7/838: mknod d16/d20/d7a/da6/d117/c12d 0 2024-04-06T14:17:14.589 INFO:tasks.workunit.client.1.smithi086.stdout:6/615: symlink d6/d13/d42/ld5 0 2024-04-06T14:17:14.590 INFO:tasks.workunit.client.0.smithi032.stdout:7/839: chown d16/d20/d48/lae 75 1 2024-04-06T14:17:14.591 INFO:tasks.workunit.client.1.smithi086.stdout:4/414: dwrite d0/d15/f3c [0,4194304] 0 2024-04-06T14:17:14.592 INFO:tasks.workunit.client.0.smithi032.stdout:9/813: dread d2/d15/f1a [0,4194304] 0 2024-04-06T14:17:14.592 INFO:tasks.workunit.client.0.smithi032.stdout:6/824: getdents d4/d5/d8/d2b/d77 0 2024-04-06T14:17:14.592 INFO:tasks.workunit.client.1.smithi086.stdout:1/565: dread db/d13/d55/f59 [0,4194304] 0 2024-04-06T14:17:14.592 INFO:tasks.workunit.client.1.smithi086.stdout:2/548: chown d0/d20/d27/d2b/d35/d36/d4a/f86 98547186 1 2024-04-06T14:17:14.593 INFO:tasks.workunit.client.1.smithi086.stdout:7/527: creat d7/d10/d11/d1c/d24/fb2 x:0 0 0 2024-04-06T14:17:14.594 INFO:tasks.workunit.client.0.smithi032.stdout:3/727: dwrite d3/d65/d89/fa2 [0,4194304] 0 2024-04-06T14:17:14.596 INFO:tasks.workunit.client.0.smithi032.stdout:6/825: chown d4/d5/d8/d2b/dfb/f117 886724 1 2024-04-06T14:17:14.596 INFO:tasks.workunit.client.1.smithi086.stdout:0/593: dwrite d0/d15/f3e [0,4194304] 0 2024-04-06T14:17:14.596 INFO:tasks.workunit.client.1.smithi086.stdout:8/634: link d0/d5a/fcb d0/d71/d22/d43/fd8 0 2024-04-06T14:17:14.597 INFO:tasks.workunit.client.0.smithi032.stdout:9/814: fdatasync d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8f/fa2 0 2024-04-06T14:17:14.599 INFO:tasks.workunit.client.0.smithi032.stdout:8/846: symlink d3/d8/d12/d2f/d46/l121 0 2024-04-06T14:17:14.599 INFO:tasks.workunit.client.0.smithi032.stdout:1/869: chown d0/d9/d1f/f91 28714 1 2024-04-06T14:17:14.603 INFO:tasks.workunit.client.0.smithi032.stdout:1/870: read d0/d41/fca [41012,80786] 0 2024-04-06T14:17:14.604 INFO:tasks.workunit.client.0.smithi032.stdout:5/725: symlink d16/d18/d33/d45/l102 0 2024-04-06T14:17:14.606 INFO:tasks.workunit.client.1.smithi086.stdout:4/415: fdatasync d0/d15/d20/d34/f35 0 2024-04-06T14:17:14.606 INFO:tasks.workunit.client.1.smithi086.stdout:6/616: truncate d6/d13/d5f/d60/f6d 704370 0 2024-04-06T14:17:14.607 INFO:tasks.workunit.client.0.smithi032.stdout:6/826: mknod d4/d5/d8/d2b/d77/c118 0 2024-04-06T14:17:14.607 INFO:tasks.workunit.client.0.smithi032.stdout:5/726: readlink d16/d18/lb1 0 2024-04-06T14:17:14.609 INFO:tasks.workunit.client.1.smithi086.stdout:0/594: mknod d0/d15/cc1 0 2024-04-06T14:17:14.609 INFO:tasks.workunit.client.1.smithi086.stdout:1/566: symlink db/d11/d42/d53/lae 0 2024-04-06T14:17:14.610 INFO:tasks.workunit.client.0.smithi032.stdout:6/827: write d4/d5/f15 [3756933,54157] 0 2024-04-06T14:17:14.612 INFO:tasks.workunit.client.0.smithi032.stdout:3/728: creat d3/d16/ddd/df7/f118 x:0 0 0 2024-04-06T14:17:14.613 INFO:tasks.workunit.client.1.smithi086.stdout:9/452: write d5/d13/d1a/f24 [1469419,36066] 0 2024-04-06T14:17:14.613 INFO:tasks.workunit.client.0.smithi032.stdout:9/815: dwrite d2/d15/d1e/d65/d83/d31/f40 [0,4194304] 0 2024-04-06T14:17:14.613 INFO:tasks.workunit.client.0.smithi032.stdout:5/727: unlink d16/d58/ldd 0 2024-04-06T14:17:14.613 INFO:tasks.workunit.client.0.smithi032.stdout:8/847: dread d3/d8/d12/f19 [0,4194304] 0 2024-04-06T14:17:14.614 INFO:tasks.workunit.client.1.smithi086.stdout:6/617: mkdir d6/d9/d5c/dd6 0 2024-04-06T14:17:14.616 INFO:tasks.workunit.client.1.smithi086.stdout:8/635: truncate d0/d13/fb3 2072205 0 2024-04-06T14:17:14.618 INFO:tasks.workunit.client.0.smithi032.stdout:1/871: creat d0/d41/d5b/d111/f119 x:0 0 0 2024-04-06T14:17:14.619 INFO:tasks.workunit.client.1.smithi086.stdout:7/528: mknod d7/d10/d11/d1c/d24/d2c/d69/d8d/cb3 0 2024-04-06T14:17:14.620 INFO:tasks.workunit.client.0.smithi032.stdout:9/816: fsync d2/d15/d1e/d65/d83/f45 0 2024-04-06T14:17:14.622 INFO:tasks.workunit.client.1.smithi086.stdout:0/595: truncate d0/d1/d2/d1a/f56 536902 0 2024-04-06T14:17:14.623 INFO:tasks.workunit.client.0.smithi032.stdout:2/704: write d0/d5a/f41 [2733940,79604] 0 2024-04-06T14:17:14.623 INFO:tasks.workunit.client.1.smithi086.stdout:4/416: link d0/de/f62 d0/d15/d2b/f82 0 2024-04-06T14:17:14.623 INFO:tasks.workunit.client.0.smithi032.stdout:6/828: mknod d4/d5/d28/d65/d6e/ddb/d110/c119 0 2024-04-06T14:17:14.625 INFO:tasks.workunit.client.1.smithi086.stdout:2/549: write d0/df/f14 [1915796,3466] 0 2024-04-06T14:17:14.628 INFO:tasks.workunit.client.0.smithi032.stdout:3/729: rename d3/l6b to d3/d65/d7d/dc6/l119 0 2024-04-06T14:17:14.630 INFO:tasks.workunit.client.0.smithi032.stdout:6/829: write d4/d5/dbf/d61/da7/dda/fee [34392,40950] 0 2024-04-06T14:17:14.631 INFO:tasks.workunit.client.0.smithi032.stdout:9/817: readlink d2/d15/d1e/d41/la3 0 2024-04-06T14:17:14.631 INFO:tasks.workunit.client.1.smithi086.stdout:1/567: mkdir db/d11/d1a/d2a/d9a/d68/daf 0 2024-04-06T14:17:14.631 INFO:tasks.workunit.client.1.smithi086.stdout:8/636: fsync d0/d71/d22/d43/d6d/d75/d7d/fa1 0 2024-04-06T14:17:14.631 INFO:tasks.workunit.client.1.smithi086.stdout:1/568: chown f2 943 1 2024-04-06T14:17:14.631 INFO:tasks.workunit.client.1.smithi086.stdout:0/596: mkdir d0/d1/d2/d1a/d79/dac/dc2 0 2024-04-06T14:17:14.633 INFO:tasks.workunit.client.1.smithi086.stdout:4/417: rename d0/d15/d4a to d0/d2c/d38/d54/d83 0 2024-04-06T14:17:14.633 INFO:tasks.workunit.client.0.smithi032.stdout:0/618: truncate d3/f76 1317122 0 2024-04-06T14:17:14.634 INFO:tasks.workunit.client.1.smithi086.stdout:6/618: rmdir d6/d9/d5c/dd6 0 2024-04-06T14:17:14.635 INFO:tasks.workunit.client.0.smithi032.stdout:4/788: dwrite d0/df/d23/f4a [4194304,4194304] 0 2024-04-06T14:17:14.637 INFO:tasks.workunit.client.0.smithi032.stdout:1/872: creat d0/d9/d1f/d7c/da9/ddd/f11a x:0 0 0 2024-04-06T14:17:14.637 INFO:tasks.workunit.client.1.smithi086.stdout:0/597: symlink d0/d1/d2/d1a/d79/lc3 0 2024-04-06T14:17:14.637 INFO:tasks.workunit.client.1.smithi086.stdout:0/598: chown d0 2 1 2024-04-06T14:17:14.638 INFO:tasks.workunit.client.0.smithi032.stdout:8/848: dwrite d3/d8/d7/d1a/f5a [0,4194304] 0 2024-04-06T14:17:14.638 INFO:tasks.workunit.client.1.smithi086.stdout:2/550: creat d0/d20/d27/d2b/d35/d36/d4a/db8/fc1 x:0 0 0 2024-04-06T14:17:14.639 INFO:tasks.workunit.client.1.smithi086.stdout:4/418: rmdir d0/d2c/d38/d54/d61/d7a 39 2024-04-06T14:17:14.642 INFO:tasks.workunit.client.0.smithi032.stdout:2/705: mknod d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/ce7 0 2024-04-06T14:17:14.642 INFO:tasks.workunit.client.1.smithi086.stdout:4/419: truncate d0/d15/d20/d74/f7b 5097471 0 2024-04-06T14:17:14.643 INFO:tasks.workunit.client.0.smithi032.stdout:8/849: truncate d3/ff6 1442671 0 2024-04-06T14:17:14.643 INFO:tasks.workunit.client.0.smithi032.stdout:5/728: rename d16/d18/d33/d45/d52/f92 to d16/d18/f103 0 2024-04-06T14:17:14.646 INFO:tasks.workunit.client.0.smithi032.stdout:7/840: dwrite d16/d20/d7a/da6/fd6 [0,4194304] 0 2024-04-06T14:17:14.649 INFO:tasks.workunit.client.0.smithi032.stdout:9/818: unlink d2/d15/d1e/d65/d83/d7e/faf 0 2024-04-06T14:17:14.650 INFO:tasks.workunit.client.0.smithi032.stdout:9/819: stat d2/d15/d55/d59/cdc 0 2024-04-06T14:17:14.651 INFO:tasks.workunit.client.0.smithi032.stdout:9/820: chown d2/d15/d1e/fff 3 1 2024-04-06T14:17:14.651 INFO:tasks.workunit.client.1.smithi086.stdout:2/551: symlink d0/df/d48/d53/d76/lc2 0 2024-04-06T14:17:14.651 INFO:tasks.workunit.client.1.smithi086.stdout:4/420: rmdir d0/d2c/d30 39 2024-04-06T14:17:14.652 INFO:tasks.workunit.client.1.smithi086.stdout:4/421: fdatasync d0/d15/d2b/f5f 0 2024-04-06T14:17:14.653 INFO:tasks.workunit.client.1.smithi086.stdout:7/529: write d7/d1d/f38 [1166898,9476] 0 2024-04-06T14:17:14.657 INFO:tasks.workunit.client.1.smithi086.stdout:4/422: write d0/d15/d20/d34/f55 [684923,42795] 0 2024-04-06T14:17:14.659 INFO:tasks.workunit.client.0.smithi032.stdout:0/619: mknod d3/da/d25/cd3 0 2024-04-06T14:17:14.659 INFO:tasks.workunit.client.0.smithi032.stdout:3/730: rename l2 to d3/d65/d7d/d9c/dad/dc5/l11a 0 2024-04-06T14:17:14.661 INFO:tasks.workunit.client.0.smithi032.stdout:6/830: getdents d4/d5/dd8/deb 0 2024-04-06T14:17:14.663 INFO:tasks.workunit.client.1.smithi086.stdout:1/569: write db/d11/f12 [3922623,2265] 0 2024-04-06T14:17:14.663 INFO:tasks.workunit.client.1.smithi086.stdout:4/423: chown d0/d15/d19/c25 260383940 1 2024-04-06T14:17:14.664 INFO:tasks.workunit.client.1.smithi086.stdout:2/552: getdents d0/d20/d27/dae 0 2024-04-06T14:17:14.665 INFO:tasks.workunit.client.1.smithi086.stdout:9/453: dwrite d5/d13/d1a/d5f/d88/d14/d19/f1e [0,4194304] 0 2024-04-06T14:17:14.666 INFO:tasks.workunit.client.0.smithi032.stdout:1/873: creat d0/d41/f11b x:0 0 0 2024-04-06T14:17:14.668 INFO:tasks.workunit.client.1.smithi086.stdout:1/570: creat db/d11/d42/d53/fb0 x:0 0 0 2024-04-06T14:17:14.669 INFO:tasks.workunit.client.1.smithi086.stdout:7/530: link d7/l44 d7/d10/d92/lb4 0 2024-04-06T14:17:14.669 INFO:tasks.workunit.client.0.smithi032.stdout:2/706: creat d0/d30/d39/d59/d67/d74/fe8 x:0 0 0 2024-04-06T14:17:14.671 INFO:tasks.workunit.client.0.smithi032.stdout:3/731: creat d3/d65/d7d/d9c/dad/dc5/f11b x:0 0 0 2024-04-06T14:17:14.672 INFO:tasks.workunit.client.0.smithi032.stdout:1/874: rmdir d0/da/d3e 39 2024-04-06T14:17:14.673 INFO:tasks.workunit.client.0.smithi032.stdout:5/729: getdents d16/d18/d33/df7 0 2024-04-06T14:17:14.674 INFO:tasks.workunit.client.0.smithi032.stdout:1/875: dread - d0/d41/f11b zero size 2024-04-06T14:17:14.675 INFO:tasks.workunit.client.1.smithi086.stdout:6/619: dread d6/d20/f46 [0,4194304] 0 2024-04-06T14:17:14.676 INFO:tasks.workunit.client.0.smithi032.stdout:0/620: creat d3/da/d25/fd4 x:0 0 0 2024-04-06T14:17:14.676 INFO:tasks.workunit.client.0.smithi032.stdout:2/707: mkdir d0/d4e/de9 0 2024-04-06T14:17:14.676 INFO:tasks.workunit.client.1.smithi086.stdout:6/620: creat d6/d13/d1a/d22/dcf/fd7 x:0 0 0 2024-04-06T14:17:14.678 INFO:tasks.workunit.client.0.smithi032.stdout:5/730: unlink d16/d18/d33/f78 0 2024-04-06T14:17:14.678 INFO:tasks.workunit.client.1.smithi086.stdout:1/571: link fa db/d13/d55/fb1 0 2024-04-06T14:17:14.680 INFO:tasks.workunit.client.0.smithi032.stdout:1/876: symlink d0/d9/d1f/d7c/da9/ddd/l11c 0 2024-04-06T14:17:14.681 INFO:tasks.workunit.client.1.smithi086.stdout:6/621: write d6/d13/d5f/fb8 [3992880,3734] 0 2024-04-06T14:17:14.682 INFO:tasks.workunit.client.1.smithi086.stdout:1/572: mknod db/d11/d1f/d5c/cb2 0 2024-04-06T14:17:14.682 INFO:tasks.workunit.client.0.smithi032.stdout:2/708: rmdir d0/d30/d39/d59/d67/d74/d78/d57/dd3 39 2024-04-06T14:17:14.683 INFO:tasks.workunit.client.0.smithi032.stdout:5/731: symlink d16/d18/d33/d35/l104 0 2024-04-06T14:17:14.684 INFO:tasks.workunit.client.0.smithi032.stdout:4/789: dread d0/f1 [0,4194304] 0 2024-04-06T14:17:14.685 INFO:tasks.workunit.client.0.smithi032.stdout:5/732: chown d16/d18/d33/d45/d52/d39/fd5 0 1 2024-04-06T14:17:14.687 INFO:tasks.workunit.client.1.smithi086.stdout:0/599: dwrite d0/d1/d4/de/f62 [0,4194304] 0 2024-04-06T14:17:14.688 INFO:tasks.workunit.client.1.smithi086.stdout:1/573: unlink db/d11/d1f/d5c/c6c 0 2024-04-06T14:17:14.690 INFO:tasks.workunit.client.1.smithi086.stdout:0/600: readlink d0/d1/d2/d1a/l50 0 2024-04-06T14:17:14.691 INFO:tasks.workunit.client.0.smithi032.stdout:4/790: chown d0/df/d1f/d42/c78 1244 1 2024-04-06T14:17:14.694 INFO:tasks.workunit.client.0.smithi032.stdout:2/709: mkdir d0/d4e/de9/dea 0 2024-04-06T14:17:14.695 INFO:tasks.workunit.client.1.smithi086.stdout:1/574: creat db/d11/d1a/d2a/d9a/fb3 x:0 0 0 2024-04-06T14:17:14.695 INFO:tasks.workunit.client.1.smithi086.stdout:9/454: write d5/d13/f4f [1066846,95177] 0 2024-04-06T14:17:14.695 INFO:tasks.workunit.client.1.smithi086.stdout:2/553: write d0/d20/d27/d2b/d35/d36/f3a [297864,40282] 0 2024-04-06T14:17:14.699 INFO:tasks.workunit.client.1.smithi086.stdout:0/601: rename d0/d15/d26/d33/f66 to d0/d15/d4e/da3/d44/d8e/d8f/fc4 0 2024-04-06T14:17:14.700 INFO:tasks.workunit.client.0.smithi032.stdout:9/821: write d2/f98 [1797691,38931] 0 2024-04-06T14:17:14.700 INFO:tasks.workunit.client.0.smithi032.stdout:4/791: getdents d0/df/d23/d31/d50/d57/d5b/d9c/dfa 0 2024-04-06T14:17:14.702 INFO:tasks.workunit.client.0.smithi032.stdout:3/732: getdents d3/d65/d7d/d9c/dad/dc5 0 2024-04-06T14:17:14.704 INFO:tasks.workunit.client.1.smithi086.stdout:6/622: getdents d6/d9 0 2024-04-06T14:17:14.705 INFO:tasks.workunit.client.0.smithi032.stdout:4/792: dread - d0/fba zero size 2024-04-06T14:17:14.706 INFO:tasks.workunit.client.0.smithi032.stdout:3/733: fdatasync d3/d65/f106 0 2024-04-06T14:17:14.707 INFO:tasks.workunit.client.1.smithi086.stdout:0/602: symlink d0/d1/d4/lc5 0 2024-04-06T14:17:14.708 INFO:tasks.workunit.client.0.smithi032.stdout:0/621: write d3/da/d25/d36/f9b [725197,22958] 0 2024-04-06T14:17:14.711 INFO:tasks.workunit.client.1.smithi086.stdout:1/575: symlink db/d11/d1a/d2a/d9a/d68/d6e/lb4 0 2024-04-06T14:17:14.711 INFO:tasks.workunit.client.1.smithi086.stdout:4/424: dwrite d0/de/f11 [0,4194304] 0 2024-04-06T14:17:14.714 INFO:tasks.workunit.client.0.smithi032.stdout:9/822: write d2/d15/d1e/d41/d6c/d74/fc3 [50790,13437] 0 2024-04-06T14:17:14.714 INFO:tasks.workunit.client.0.smithi032.stdout:7/841: dwrite f11 [0,4194304] 0 2024-04-06T14:17:14.714 INFO:tasks.workunit.client.1.smithi086.stdout:7/531: dwrite d7/d10/d11/d80/f4a [0,4194304] 0 2024-04-06T14:17:14.715 INFO:tasks.workunit.client.1.smithi086.stdout:6/623: creat d6/d13/d1a/d22/dcf/db2/fd8 x:0 0 0 2024-04-06T14:17:14.715 INFO:tasks.workunit.client.0.smithi032.stdout:8/850: dwrite d3/d8/d12/d2f/d4a/d60/d90/dc5/dd7/fea [0,4194304] 0 2024-04-06T14:17:14.716 INFO:tasks.workunit.client.0.smithi032.stdout:4/793: readlink d0/df/d23/d31/d50/d57/l7d 0 2024-04-06T14:17:14.717 INFO:tasks.workunit.client.1.smithi086.stdout:4/425: chown d0/d15/d2b/f82 0 1 2024-04-06T14:17:14.717 INFO:tasks.workunit.client.1.smithi086.stdout:0/603: mknod d0/d15/d4e/da3/d73/cc6 0 2024-04-06T14:17:14.720 INFO:tasks.workunit.client.0.smithi032.stdout:8/851: chown d3/d8/d12/l40 15722 1 2024-04-06T14:17:14.721 INFO:tasks.workunit.client.1.smithi086.stdout:7/532: fsync d7/fb 0 2024-04-06T14:17:14.721 INFO:tasks.workunit.client.1.smithi086.stdout:7/533: fdatasync d7/d1d/f6b 0 2024-04-06T14:17:14.721 INFO:tasks.workunit.client.1.smithi086.stdout:9/455: rename d5/d13/d1a/d5f/d88/d14/d19/d1b/f43 to d5/d13/d1a/d8f/f9a 0 2024-04-06T14:17:14.722 INFO:tasks.workunit.client.0.smithi032.stdout:5/733: dwrite f1 [4194304,4194304] 0 2024-04-06T14:17:14.722 INFO:tasks.workunit.client.1.smithi086.stdout:4/426: chown d0/d2c/d30 125 1 2024-04-06T14:17:14.723 INFO:tasks.workunit.client.1.smithi086.stdout:7/534: dread - d7/d10/d11/d1c/d24/d2c/d3c/f98 zero size 2024-04-06T14:17:14.726 INFO:tasks.workunit.client.1.smithi086.stdout:7/535: read - d7/d10/d11/d1c/d24/d2c/f48 zero size 2024-04-06T14:17:14.726 INFO:tasks.workunit.client.1.smithi086.stdout:6/624: chown d6/d13/d5f/d60/d45/d8a/cc4 15211364 1 2024-04-06T14:17:14.727 INFO:tasks.workunit.client.0.smithi032.stdout:7/842: read d16/d20/f28 [2349641,103154] 0 2024-04-06T14:17:14.728 INFO:tasks.workunit.client.1.smithi086.stdout:9/456: write d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f97 [335595,109343] 0 2024-04-06T14:17:14.729 INFO:tasks.workunit.client.0.smithi032.stdout:4/794: dread - d0/df/d2b/d46/d58/feb zero size 2024-04-06T14:17:14.729 INFO:tasks.workunit.client.0.smithi032.stdout:5/734: symlink d16/d18/d33/d35/d88/deb/l105 0 2024-04-06T14:17:14.729 INFO:tasks.workunit.client.1.smithi086.stdout:6/625: readlink d6/d9/d63/l6f 0 2024-04-06T14:17:14.729 INFO:tasks.workunit.client.1.smithi086.stdout:4/427: getdents d0/d2c/d30/d24/d5d 0 2024-04-06T14:17:14.730 INFO:tasks.workunit.client.1.smithi086.stdout:7/536: write d7/d10/d92/f9f [432230,84220] 0 2024-04-06T14:17:14.731 INFO:tasks.workunit.client.0.smithi032.stdout:8/852: dread d3/d8/d12/d83/d89/da0/ffe [0,4194304] 0 2024-04-06T14:17:14.732 INFO:tasks.workunit.client.1.smithi086.stdout:6/626: read - d6/d13/d1a/fd0 zero size 2024-04-06T14:17:14.732 INFO:tasks.workunit.client.1.smithi086.stdout:7/537: chown d7/d10/f64 2788061 1 2024-04-06T14:17:14.733 INFO:tasks.workunit.client.1.smithi086.stdout:6/627: chown d6/d9/d43 201362520 1 2024-04-06T14:17:14.733 INFO:tasks.workunit.client.0.smithi032.stdout:5/735: rmdir d16/d18/dae/dbb 39 2024-04-06T14:17:14.735 INFO:tasks.workunit.client.1.smithi086.stdout:9/457: truncate d5/d13/f7d 7274 0 2024-04-06T14:17:14.737 INFO:tasks.workunit.client.1.smithi086.stdout:6/628: creat d6/d13/d1a/d22/fd9 x:0 0 0 2024-04-06T14:17:14.739 INFO:tasks.workunit.client.0.smithi032.stdout:8/853: dread - d3/d8/fa5 zero size 2024-04-06T14:17:14.739 INFO:tasks.workunit.client.0.smithi032.stdout:2/710: dread d0/d30/d39/d59/d67/d74/d78/d57/d7c/fb9 [0,4194304] 0 2024-04-06T14:17:14.739 INFO:tasks.workunit.client.1.smithi086.stdout:4/428: creat d0/d2c/d30/d24/d71/f84 x:0 0 0 2024-04-06T14:17:14.743 INFO:tasks.workunit.client.1.smithi086.stdout:9/458: chown d5/d13/d1a/d5f/d88/d14/d19/d1b/l3d 1180377350 1 2024-04-06T14:17:14.743 INFO:tasks.workunit.client.1.smithi086.stdout:2/554: dread d0/df/d48/d7d/fa6 [0,4194304] 0 2024-04-06T14:17:14.743 INFO:tasks.workunit.client.1.smithi086.stdout:4/429: stat d0/d15/l60 0 2024-04-06T14:17:14.745 INFO:tasks.workunit.client.0.smithi032.stdout:4/795: write d0/df/d23/f3d [1876840,68358] 0 2024-04-06T14:17:14.747 INFO:tasks.workunit.client.1.smithi086.stdout:6/629: creat d6/d13/d1a/da0/fda x:0 0 0 2024-04-06T14:17:14.747 INFO:tasks.workunit.client.1.smithi086.stdout:7/538: dwrite d7/d1d/f6b [0,4194304] 0 2024-04-06T14:17:14.748 INFO:tasks.workunit.client.1.smithi086.stdout:2/555: mknod d0/d20/d27/d2b/cc3 0 2024-04-06T14:17:14.749 INFO:tasks.workunit.client.0.smithi032.stdout:4/796: fdatasync d0/f5c 0 2024-04-06T14:17:14.751 INFO:tasks.workunit.client.0.smithi032.stdout:0/622: dwrite d3/da/d25/f94 [0,4194304] 0 2024-04-06T14:17:14.753 INFO:tasks.workunit.client.0.smithi032.stdout:8/854: rename d3/d8/d7/d1a to d3/d8/d12/d2f/d4a/d60/d90/d122 0 2024-04-06T14:17:14.753 INFO:tasks.workunit.client.1.smithi086.stdout:4/430: rename d0/d2c/f4d to d0/d2c/d30/d24/f85 0 2024-04-06T14:17:14.754 INFO:tasks.workunit.client.1.smithi086.stdout:6/630: truncate d6/d9/f6c 665727 0 2024-04-06T14:17:14.754 INFO:tasks.workunit.client.1.smithi086.stdout:2/556: mkdir d0/d20/d44/dbb/dc4 0 2024-04-06T14:17:14.765 INFO:tasks.workunit.client.0.smithi032.stdout:5/736: dread d16/d18/d33/fa8 [0,4194304] 0 2024-04-06T14:17:14.765 INFO:tasks.workunit.client.0.smithi032.stdout:4/797: getdents d0/df/d23/d31/d50/d57/d5b/d9c/dfa 0 2024-04-06T14:17:14.766 INFO:tasks.workunit.client.1.smithi086.stdout:7/539: rename d7/d10/d11/d1c/d24/d2c/d3c to d7/d10/d11/d1c/d24/d4d/daf/db5 0 2024-04-06T14:17:14.767 INFO:tasks.workunit.client.1.smithi086.stdout:6/631: fsync d6/d9/db/f9b 0 2024-04-06T14:17:14.770 INFO:tasks.workunit.client.0.smithi032.stdout:3/734: dread d3/f4 [0,4194304] 0 2024-04-06T14:17:14.773 INFO:tasks.workunit.client.1.smithi086.stdout:6/632: truncate d6/f2b 5630578 0 2024-04-06T14:17:14.773 INFO:tasks.workunit.client.1.smithi086.stdout:9/459: write d5/d13/d1a/d5f/d88/d14/d19/d25/d59/f60 [62496,66485] 0 2024-04-06T14:17:14.773 INFO:tasks.workunit.client.0.smithi032.stdout:0/623: write d3/f4 [762583,49350] 0 2024-04-06T14:17:14.773 INFO:tasks.workunit.client.0.smithi032.stdout:5/737: chown d16/d18/f1e 521994629 1 2024-04-06T14:17:14.773 INFO:tasks.workunit.client.1.smithi086.stdout:6/633: chown d6/d13/d1a/c3b 5 1 2024-04-06T14:17:14.774 INFO:tasks.workunit.client.0.smithi032.stdout:3/735: chown d3/d65/d7d/d9c/dad/db1/fa4 612 1 2024-04-06T14:17:14.776 INFO:tasks.workunit.client.0.smithi032.stdout:7/843: dread d16/d20/d54/de4/d2d/f3f [0,4194304] 0 2024-04-06T14:17:14.781 INFO:tasks.workunit.client.0.smithi032.stdout:0/624: dwrite d3/da/d35/f4e [0,4194304] 0 2024-04-06T14:17:14.782 INFO:tasks.workunit.client.0.smithi032.stdout:8/855: dread d3/d69/fad [0,4194304] 0 2024-04-06T14:17:14.794 INFO:tasks.workunit.client.1.smithi086.stdout:3/533: sync 2024-04-06T14:17:14.795 INFO:tasks.workunit.client.0.smithi032.stdout:5/738: unlink d16/d18/d33/d35/f44 0 2024-04-06T14:17:14.797 INFO:tasks.workunit.client.0.smithi032.stdout:7/844: unlink d16/d20/d48/d8d/fd1 0 2024-04-06T14:17:14.798 INFO:tasks.workunit.client.0.smithi032.stdout:2/711: dread d0/f13 [0,4194304] 0 2024-04-06T14:17:14.799 INFO:tasks.workunit.client.0.smithi032.stdout:8/856: unlink d3/d8/c3e 0 2024-04-06T14:17:14.803 INFO:tasks.workunit.client.0.smithi032.stdout:2/712: symlink d0/d5a/d3a/da2/leb 0 2024-04-06T14:17:14.807 INFO:tasks.workunit.client.0.smithi032.stdout:5/739: rename d16/d58/d54/d74/l97 to d16/d58/d54/d94/db2/dcb/l106 0 2024-04-06T14:17:14.809 INFO:tasks.workunit.client.0.smithi032.stdout:0/625: fdatasync d3/f4 0 2024-04-06T14:17:14.809 INFO:tasks.workunit.client.0.smithi032.stdout:2/713: fsync d0/d5a/fb1 0 2024-04-06T14:17:14.810 INFO:tasks.workunit.client.0.smithi032.stdout:7/845: truncate d16/d20/d63/d9d/df2/df5/f119 41178 0 2024-04-06T14:17:14.810 INFO:tasks.workunit.client.0.smithi032.stdout:8/857: dwrite d3/d8/d12/d2f/d4a/d60/d90/ff2 [0,4194304] 0 2024-04-06T14:17:14.811 INFO:tasks.workunit.client.0.smithi032.stdout:3/736: dwrite d3/f43 [0,4194304] 0 2024-04-06T14:17:14.812 INFO:tasks.workunit.client.0.smithi032.stdout:8/858: stat d3 0 2024-04-06T14:17:14.815 INFO:tasks.workunit.client.0.smithi032.stdout:3/737: chown d3/d65/d7d/d9c/dad/db1/fc3 562894 1 2024-04-06T14:17:14.815 INFO:tasks.workunit.client.0.smithi032.stdout:8/859: write d3/d8/d12/d2f/f39 [486201,106860] 0 2024-04-06T14:17:14.815 INFO:tasks.workunit.client.0.smithi032.stdout:2/714: symlink d0/db2/lec 0 2024-04-06T14:17:14.816 INFO:tasks.workunit.client.0.smithi032.stdout:2/715: readlink d0/db2/lec 0 2024-04-06T14:17:14.818 INFO:tasks.workunit.client.0.smithi032.stdout:5/740: creat d16/d18/dae/dbb/ddc/d100/f107 x:0 0 0 2024-04-06T14:17:14.820 INFO:tasks.workunit.client.0.smithi032.stdout:8/860: chown d3/d8/d12/d2f/d4a/d60/d75/c7d 52025718 1 2024-04-06T14:17:14.821 INFO:tasks.workunit.client.0.smithi032.stdout:8/861: chown d3/d8/d12/d2f/d4a/d60/d90/d122 1633226 1 2024-04-06T14:17:14.822 INFO:tasks.workunit.client.0.smithi032.stdout:5/741: creat d16/d58/f108 x:0 0 0 2024-04-06T14:17:14.825 INFO:tasks.workunit.client.0.smithi032.stdout:4/798: dread d0/df/d23/d31/f87 [0,4194304] 0 2024-04-06T14:17:14.828 INFO:tasks.workunit.client.0.smithi032.stdout:5/742: mknod d16/d18/d33/d35/d88/c109 0 2024-04-06T14:17:14.829 INFO:tasks.workunit.client.0.smithi032.stdout:0/626: write d3/da/f18 [4834354,92443] 0 2024-04-06T14:17:14.830 INFO:tasks.workunit.client.0.smithi032.stdout:3/738: getdents d3/d16/ddd/df7 0 2024-04-06T14:17:14.831 INFO:tasks.workunit.client.0.smithi032.stdout:7/846: write d16/d20/d54/de4/d5f/fb0 [3406316,64704] 0 2024-04-06T14:17:14.833 INFO:tasks.workunit.client.0.smithi032.stdout:3/739: fdatasync d3/d65/d7d/d9c/dad/db1/d3c/f5f 0 2024-04-06T14:17:14.835 INFO:tasks.workunit.client.0.smithi032.stdout:7/847: creat d16/d20/d6e/d10e/f12e x:0 0 0 2024-04-06T14:17:14.856 INFO:tasks.workunit.client.0.smithi032.stdout:4/799: creat d0/d82/f108 x:0 0 0 2024-04-06T14:17:14.857 INFO:tasks.workunit.client.1.smithi086.stdout:6/634: unlink d6/d13/d1a/d64/f71 0 2024-04-06T14:17:14.857 INFO:tasks.workunit.client.0.smithi032.stdout:2/716: truncate d0/d5a/d6a/fb6 174281 0 2024-04-06T14:17:14.858 INFO:tasks.workunit.client.1.smithi086.stdout:2/557: dread - d0/d20/d27/d2b/d35/d36/d4d/d59/f8b zero size 2024-04-06T14:17:14.859 INFO:tasks.workunit.client.0.smithi032.stdout:5/743: write d16/d18/f68 [519913,83517] 0 2024-04-06T14:17:14.859 INFO:tasks.workunit.client.0.smithi032.stdout:3/740: rename d3/d65/d7d/d9c/db8/f113 to d3/d65/d89/f11c 0 2024-04-06T14:17:14.859 INFO:tasks.workunit.client.1.smithi086.stdout:3/534: write d5/f8d [1106463,47213] 0 2024-04-06T14:17:14.860 INFO:tasks.workunit.client.0.smithi032.stdout:2/717: write d0/d5a/fb1 [4897125,1344] 0 2024-04-06T14:17:14.863 INFO:tasks.workunit.client.0.smithi032.stdout:0/627: dread d3/da/d25/d2b/d4a/da1/fb2 [0,4194304] 0 2024-04-06T14:17:14.864 INFO:tasks.workunit.client.1.smithi086.stdout:4/431: dwrite d0/d15/f37 [0,4194304] 0 2024-04-06T14:17:14.864 INFO:tasks.workunit.client.0.smithi032.stdout:3/741: mkdir d3/d16/d63/d11d 0 2024-04-06T14:17:14.867 INFO:tasks.workunit.client.0.smithi032.stdout:0/628: rmdir d3/da/d35 39 2024-04-06T14:17:14.868 INFO:tasks.workunit.client.1.smithi086.stdout:7/540: dwrite d7/d10/d11/d1c/d24/d4d/d5c/d66/f85 [0,4194304] 0 2024-04-06T14:17:14.869 INFO:tasks.workunit.client.0.smithi032.stdout:8/862: dwrite d3/d8/d3a/f4f [0,4194304] 0 2024-04-06T14:17:14.871 INFO:tasks.workunit.client.0.smithi032.stdout:2/718: creat d0/d30/d39/fed x:0 0 0 2024-04-06T14:17:14.877 INFO:tasks.workunit.client.0.smithi032.stdout:3/742: truncate d3/d16/ddd/df7/f118 715983 0 2024-04-06T14:17:14.877 INFO:tasks.workunit.client.0.smithi032.stdout:5/744: dwrite d16/d18/f21 [0,4194304] 0 2024-04-06T14:17:14.878 INFO:tasks.workunit.client.0.smithi032.stdout:0/629: symlink d3/da/d25/d2b/d80/db3/dcd/ld5 0 2024-04-06T14:17:14.880 INFO:tasks.workunit.client.0.smithi032.stdout:4/800: dread d0/d82/fc5 [0,4194304] 0 2024-04-06T14:17:14.886 INFO:tasks.workunit.client.0.smithi032.stdout:3/743: creat d3/d65/f11e x:0 0 0 2024-04-06T14:17:14.887 INFO:tasks.workunit.client.0.smithi032.stdout:8/863: write d3/d8/d7/fa7 [1113471,124245] 0 2024-04-06T14:17:14.889 INFO:tasks.workunit.client.0.smithi032.stdout:4/801: rename d0/d82/de6 to d0/df/d23/d31/d50/d57/d5b/d9c/d109 0 2024-04-06T14:17:14.889 INFO:tasks.workunit.client.0.smithi032.stdout:3/744: chown d3/d65/d7d/d9c/fe1 7004 1 2024-04-06T14:17:14.890 INFO:tasks.workunit.client.0.smithi032.stdout:8/864: unlink d3/d8/d12/d2f/d46/f55 0 2024-04-06T14:17:14.892 INFO:tasks.workunit.client.0.smithi032.stdout:5/745: creat d16/d18/dae/dbb/f10a x:0 0 0 2024-04-06T14:17:14.893 INFO:tasks.workunit.client.0.smithi032.stdout:0/630: rename d3/da/d41/d82 to d3/da/d25/d2b/d4a/d86/dd6 0 2024-04-06T14:17:14.894 INFO:tasks.workunit.client.0.smithi032.stdout:4/802: mknod d0/d82/d4b/c10a 0 2024-04-06T14:17:14.895 INFO:tasks.workunit.client.0.smithi032.stdout:4/803: chown d0/df/d23/f60 2508 1 2024-04-06T14:17:14.895 INFO:tasks.workunit.client.0.smithi032.stdout:5/746: fdatasync d16/d18/d33/fa8 0 2024-04-06T14:17:14.896 INFO:tasks.workunit.client.0.smithi032.stdout:4/804: chown d0/df/d23/f3d 7788264 1 2024-04-06T14:17:14.900 INFO:tasks.workunit.client.0.smithi032.stdout:8/865: symlink d3/d8/d12/d2f/d46/def/l123 0 2024-04-06T14:17:14.900 INFO:tasks.workunit.client.0.smithi032.stdout:8/866: chown d3 1093 1 2024-04-06T14:17:14.905 INFO:tasks.workunit.client.0.smithi032.stdout:8/867: mkdir d3/d8/d3a/d7b/db9/df8/d106/d124 0 2024-04-06T14:17:14.905 INFO:tasks.workunit.client.0.smithi032.stdout:3/745: link d3/d65/d7d/d9c/dad/db1/d3c/c114 d3/d16/d63/d11d/c11f 0 2024-04-06T14:17:14.907 INFO:tasks.workunit.client.0.smithi032.stdout:8/868: stat d3/d8/d7/d14/f45 0 2024-04-06T14:17:14.909 INFO:tasks.workunit.client.0.smithi032.stdout:5/747: getdents d16/d18/d33 0 2024-04-06T14:17:14.910 INFO:tasks.workunit.client.0.smithi032.stdout:3/746: mknod d3/d65/d7d/dc6/de5/c120 0 2024-04-06T14:17:14.912 INFO:tasks.workunit.client.0.smithi032.stdout:0/631: getdents d3/da/d35/d8b/d9c 0 2024-04-06T14:17:14.915 INFO:tasks.workunit.client.0.smithi032.stdout:2/719: dwrite d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/faf [0,4194304] 0 2024-04-06T14:17:14.916 INFO:tasks.workunit.client.0.smithi032.stdout:3/747: dread - d3/d65/d7d/dc6/de5/ff8 zero size 2024-04-06T14:17:14.918 INFO:tasks.workunit.client.0.smithi032.stdout:8/869: rename d3/d8/d12/d2f/d4a/d60/d90/d122 to d3/d8/d12/d2f/d46/d125 0 2024-04-06T14:17:14.920 INFO:tasks.workunit.client.0.smithi032.stdout:0/632: rename d3/da/d35/fc1 to d3/da/d25/d57/fd7 0 2024-04-06T14:17:14.923 INFO:tasks.workunit.client.0.smithi032.stdout:5/748: write d16/d18/d33/d45/fa1 [2636507,45920] 0 2024-04-06T14:17:14.926 INFO:tasks.workunit.client.0.smithi032.stdout:4/805: dwrite d0/fba [0,4194304] 0 2024-04-06T14:17:14.927 INFO:tasks.workunit.client.0.smithi032.stdout:3/748: mknod d3/d65/d7d/d9c/dad/dc5/c121 0 2024-04-06T14:17:14.929 INFO:tasks.workunit.client.0.smithi032.stdout:2/720: mknod d0/d30/d39/d59/d67/d74/d78/d57/d7c/de6/cee 0 2024-04-06T14:17:14.931 INFO:tasks.workunit.client.0.smithi032.stdout:3/749: symlink d3/d65/l122 0 2024-04-06T14:17:14.934 INFO:tasks.workunit.client.0.smithi032.stdout:8/870: dwrite d3/d8/d12/f4d [0,4194304] 0 2024-04-06T14:17:14.935 INFO:tasks.workunit.client.0.smithi032.stdout:0/633: write d3/f78 [293453,42270] 0 2024-04-06T14:17:14.938 INFO:tasks.workunit.client.1.smithi086.stdout:6/635: chown d6/d13/f75 2 1 2024-04-06T14:17:14.938 INFO:tasks.workunit.client.0.smithi032.stdout:4/806: creat d0/df/d1f/d42/f10b x:0 0 0 2024-04-06T14:17:14.939 INFO:tasks.workunit.client.1.smithi086.stdout:9/460: creat d5/d13/d1a/d5f/d88/d14/d90/f9b x:0 0 0 2024-04-06T14:17:14.939 INFO:tasks.workunit.client.1.smithi086.stdout:6/636: readlink d6/d13/d1a/l2d 0 2024-04-06T14:17:14.940 INFO:tasks.workunit.client.0.smithi032.stdout:5/749: write d16/d58/f7a [902495,39496] 0 2024-04-06T14:17:14.941 INFO:tasks.workunit.client.0.smithi032.stdout:8/871: symlink d3/d8/d12/d2f/d46/d125/l126 0 2024-04-06T14:17:14.941 INFO:tasks.workunit.client.0.smithi032.stdout:3/750: write d3/d16/f72 [1288880,28575] 0 2024-04-06T14:17:14.942 INFO:tasks.workunit.client.1.smithi086.stdout:9/461: truncate d5/d13/d1a/d5f/d88/d14/d90/f9b 153601 0 2024-04-06T14:17:14.942 INFO:tasks.workunit.client.1.smithi086.stdout:6/637: truncate d6/d20/fb1 625403 0 2024-04-06T14:17:14.943 INFO:tasks.workunit.client.1.smithi086.stdout:4/432: rmdir d0/d15/d20/d34 39 2024-04-06T14:17:14.947 INFO:tasks.workunit.client.0.smithi032.stdout:2/721: write d0/d5a/f1c [1462857,65119] 0 2024-04-06T14:17:14.950 INFO:tasks.workunit.client.1.smithi086.stdout:6/638: unlink d6/d13/d1a/c41 0 2024-04-06T14:17:14.950 INFO:tasks.workunit.client.0.smithi032.stdout:4/807: creat d0/df/d23/d31/d50/d57/d8a/db5/f10c x:0 0 0 2024-04-06T14:17:14.953 INFO:tasks.workunit.client.0.smithi032.stdout:0/634: dwrite d3/da/d25/d2b/d4a/d79/d7c/fbf [0,4194304] 0 2024-04-06T14:17:14.954 INFO:tasks.workunit.client.1.smithi086.stdout:9/462: dread d5/d13/d1a/d5f/d88/d14/d19/d25/f2f [0,4194304] 0 2024-04-06T14:17:14.955 INFO:tasks.workunit.client.0.smithi032.stdout:0/635: write d3/f5d [4421513,26329] 0 2024-04-06T14:17:14.955 INFO:tasks.workunit.client.0.smithi032.stdout:3/751: rename d3/fd7 to d3/d65/d7d/d9c/dad/d111/f123 0 2024-04-06T14:17:14.957 INFO:tasks.workunit.client.1.smithi086.stdout:6/639: rename d6/d13/d1a/d64/fd1 to d6/d13/d1a/d64/d85/fdb 0 2024-04-06T14:17:14.957 INFO:tasks.workunit.client.1.smithi086.stdout:5/452: sync 2024-04-06T14:17:14.957 INFO:tasks.workunit.client.0.smithi032.stdout:8/872: creat d3/d8/d3a/d7b/db9/f127 x:0 0 0 2024-04-06T14:17:14.959 INFO:tasks.workunit.client.1.smithi086.stdout:4/433: dwrite d0/d15/d20/f44 [0,4194304] 0 2024-04-06T14:17:14.960 INFO:tasks.workunit.client.0.smithi032.stdout:3/752: fsync d3/d65/d7d/dc6/de5/ff8 0 2024-04-06T14:17:14.960 INFO:tasks.workunit.client.1.smithi086.stdout:5/453: write d0/d16/d18/d2a/d50/f7b [3307282,64872] 0 2024-04-06T14:17:14.961 INFO:tasks.workunit.client.1.smithi086.stdout:6/640: mkdir d6/d9/db/ddc 0 2024-04-06T14:17:14.963 INFO:tasks.workunit.client.0.smithi032.stdout:3/753: dread - d3/d65/d7d/d9c/dbc/dcd/f101 zero size 2024-04-06T14:17:14.964 INFO:tasks.workunit.client.1.smithi086.stdout:9/463: rename d5/d13/d47/d5e/c7b to d5/d13/d1a/d5f/d88/c9c 0 2024-04-06T14:17:14.965 INFO:tasks.workunit.client.0.smithi032.stdout:2/722: truncate d0/d5a/d3a/f48 3866130 0 2024-04-06T14:17:14.965 INFO:tasks.workunit.client.0.smithi032.stdout:5/750: write d16/d18/d33/d45/d52/d39/f47 [355077,30864] 0 2024-04-06T14:17:14.968 INFO:tasks.workunit.client.1.smithi086.stdout:4/434: dwrite d0/d15/d2b/f5f [0,4194304] 0 2024-04-06T14:17:14.968 INFO:tasks.workunit.client.1.smithi086.stdout:9/464: fdatasync d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f97 0 2024-04-06T14:17:14.972 INFO:tasks.workunit.client.0.smithi032.stdout:0/636: dread d3/da/d35/d8b/d9c/fc7 [0,4194304] 0 2024-04-06T14:17:14.974 INFO:tasks.workunit.client.0.smithi032.stdout:4/808: dread d0/df/d23/fc7 [0,4194304] 0 2024-04-06T14:17:14.974 INFO:tasks.workunit.client.0.smithi032.stdout:5/751: symlink d16/d58/d54/d74/d8f/d95/l10b 0 2024-04-06T14:17:14.975 INFO:tasks.workunit.client.0.smithi032.stdout:3/754: truncate d3/d16/f35 1274026 0 2024-04-06T14:17:14.980 INFO:tasks.workunit.client.1.smithi086.stdout:5/454: dread d0/d16/d18/d21/d23/f55 [0,4194304] 0 2024-04-06T14:17:14.980 INFO:tasks.workunit.client.0.smithi032.stdout:8/873: getdents d3/d8/d12/d2f/d4a 0 2024-04-06T14:17:14.981 INFO:tasks.workunit.client.1.smithi086.stdout:9/465: dread d5/d13/d1a/d5f/d88/d14/d19/d1b/f45 [0,4194304] 0 2024-04-06T14:17:14.982 INFO:tasks.workunit.client.0.smithi032.stdout:0/637: unlink d3/da/ff 0 2024-04-06T14:17:14.983 INFO:tasks.workunit.client.0.smithi032.stdout:5/752: mkdir d16/d18/d33/d45/d87/d10c 0 2024-04-06T14:17:14.984 INFO:tasks.workunit.client.1.smithi086.stdout:5/455: mkdir d0/d17/d2d/d8a 0 2024-04-06T14:17:14.984 INFO:tasks.workunit.client.0.smithi032.stdout:3/755: dwrite d3/d65/f34 [0,4194304] 0 2024-04-06T14:17:14.985 INFO:tasks.workunit.client.0.smithi032.stdout:4/809: symlink d0/df/d23/d31/d50/d57/d5b/d9c/dc1/l10d 0 2024-04-06T14:17:14.987 INFO:tasks.workunit.client.0.smithi032.stdout:0/638: mkdir d3/dc2/dd8 0 2024-04-06T14:17:14.988 INFO:tasks.workunit.client.1.smithi086.stdout:5/456: symlink d0/d43/d74/l8b 0 2024-04-06T14:17:14.989 INFO:tasks.workunit.client.0.smithi032.stdout:5/753: write d16/d18/d33/df7/ffb [56252,68613] 0 2024-04-06T14:17:14.992 INFO:tasks.workunit.client.0.smithi032.stdout:3/756: truncate d3/d65/d7d/d9c/dad/db1/fc3 1334553 0 2024-04-06T14:17:14.992 INFO:tasks.workunit.client.1.smithi086.stdout:5/457: unlink d0/c30 0 2024-04-06T14:17:14.994 INFO:tasks.workunit.client.1.smithi086.stdout:4/435: dread d0/de/f26 [0,4194304] 0 2024-04-06T14:17:14.995 INFO:tasks.workunit.client.0.smithi032.stdout:3/757: write d3/d65/d7d/d9c/db8/fc2 [2255521,9692] 0 2024-04-06T14:17:14.998 INFO:tasks.workunit.client.1.smithi086.stdout:4/436: truncate d0/d15/f48 3826436 0 2024-04-06T14:17:14.998 INFO:tasks.workunit.client.0.smithi032.stdout:8/874: rename d3/d8/d12/d2f/d4a/d60/d82 to d3/d8/d12/d2f/d46/def/d128 0 2024-04-06T14:17:14.999 INFO:tasks.workunit.client.0.smithi032.stdout:0/639: fsync d3/da/f64 0 2024-04-06T14:17:15.000 INFO:tasks.workunit.client.1.smithi086.stdout:4/437: symlink d0/d2c/d30/d24/d50/l86 0 2024-04-06T14:17:15.001 INFO:tasks.workunit.client.1.smithi086.stdout:4/438: read - d0/d2c/d30/d24/f85 zero size 2024-04-06T14:17:15.001 INFO:tasks.workunit.client.1.smithi086.stdout:4/439: chown d0/de/c2f 1 1 2024-04-06T14:17:15.002 INFO:tasks.workunit.client.0.smithi032.stdout:2/723: dread d0/d5a/f21 [0,4194304] 0 2024-04-06T14:17:15.003 INFO:tasks.workunit.client.0.smithi032.stdout:4/810: dwrite d0/df/d2b/dbe/fe4 [0,4194304] 0 2024-04-06T14:17:15.005 INFO:tasks.workunit.client.1.smithi086.stdout:4/440: symlink d0/d2c/d38/d54/d61/l87 0 2024-04-06T14:17:15.009 INFO:tasks.workunit.client.0.smithi032.stdout:3/758: creat d3/dcf/f124 x:0 0 0 2024-04-06T14:17:15.009 INFO:tasks.workunit.client.0.smithi032.stdout:0/640: rmdir d3/da/d25/d2b/d80/db3/db7 39 2024-04-06T14:17:15.011 INFO:tasks.workunit.client.1.smithi086.stdout:3/535: dwrite d5/f44 [0,4194304] 0 2024-04-06T14:17:15.015 INFO:tasks.workunit.client.0.smithi032.stdout:8/875: mknod d3/d8/d12/d2f/d4a/d60/d90/dc5/dd7/c129 0 2024-04-06T14:17:15.016 INFO:tasks.workunit.client.1.smithi086.stdout:9/466: write f0 [1131165,77176] 0 2024-04-06T14:17:15.018 INFO:tasks.workunit.client.0.smithi032.stdout:3/759: mknod d3/d65/d7d/d9c/dbc/dcd/c125 0 2024-04-06T14:17:15.018 INFO:tasks.workunit.client.0.smithi032.stdout:5/754: write d16/d18/d33/d45/d52/f43 [1066905,78076] 0 2024-04-06T14:17:15.018 INFO:tasks.workunit.client.0.smithi032.stdout:0/641: mkdir d3/da/d25/d57/dd9 0 2024-04-06T14:17:15.018 INFO:tasks.workunit.client.0.smithi032.stdout:4/811: write d0/f5c [180174,42824] 0 2024-04-06T14:17:15.019 INFO:tasks.workunit.client.1.smithi086.stdout:5/458: fsync d0/d16/f39 0 2024-04-06T14:17:15.019 INFO:tasks.workunit.client.1.smithi086.stdout:7/541: dwrite d7/f77 [0,4194304] 0 2024-04-06T14:17:15.020 INFO:tasks.workunit.client.0.smithi032.stdout:2/724: creat d0/d30/d39/d59/d67/d74/d78/fef x:0 0 0 2024-04-06T14:17:15.020 INFO:tasks.workunit.client.0.smithi032.stdout:6/831: sync 2024-04-06T14:17:15.023 INFO:tasks.workunit.client.0.smithi032.stdout:5/755: creat d16/d18/d33/d45/d87/f10d x:0 0 0 2024-04-06T14:17:15.024 INFO:tasks.workunit.client.0.smithi032.stdout:4/812: readlink d0/df/d2b/dbe/le5 0 2024-04-06T14:17:15.024 INFO:tasks.workunit.client.1.smithi086.stdout:6/641: dwrite d6/d13/d1a/d22/f9c [0,4194304] 0 2024-04-06T14:17:15.025 INFO:tasks.workunit.client.0.smithi032.stdout:0/642: creat d3/dc2/fda x:0 0 0 2024-04-06T14:17:15.026 INFO:tasks.workunit.client.0.smithi032.stdout:5/756: chown d16/d18/f23 35819 1 2024-04-06T14:17:15.028 INFO:tasks.workunit.client.1.smithi086.stdout:9/467: chown d5/d13/d1a/d5f/d88/d14/d19/d25/d59/l85 24804396 1 2024-04-06T14:17:15.028 INFO:tasks.workunit.client.0.smithi032.stdout:2/725: write d0/d30/d39/d59/d67/d74/d78/d57/fd1 [410763,93579] 0 2024-04-06T14:17:15.028 INFO:tasks.workunit.client.0.smithi032.stdout:3/760: creat d3/d65/d7d/dc6/df0/f126 x:0 0 0 2024-04-06T14:17:15.030 INFO:tasks.workunit.client.0.smithi032.stdout:4/813: mkdir d0/df/d2b/dbe/d10e 0 2024-04-06T14:17:15.033 INFO:tasks.workunit.client.1.smithi086.stdout:3/536: dwrite d5/f44 [0,4194304] 0 2024-04-06T14:17:15.033 INFO:tasks.workunit.client.0.smithi032.stdout:5/757: write d16/d18/d33/d35/d88/fad [2558138,115946] 0 2024-04-06T14:17:15.035 INFO:tasks.workunit.client.0.smithi032.stdout:3/761: creat d3/d65/f127 x:0 0 0 2024-04-06T14:17:15.035 INFO:tasks.workunit.client.0.smithi032.stdout:5/758: readlink l2 0 2024-04-06T14:17:15.036 INFO:tasks.workunit.client.0.smithi032.stdout:4/814: symlink d0/df/d76/l10f 0 2024-04-06T14:17:15.038 INFO:tasks.workunit.client.0.smithi032.stdout:3/762: chown d3/d16/d63/f83 64152407 1 2024-04-06T14:17:15.038 INFO:tasks.workunit.client.1.smithi086.stdout:7/542: creat d7/d10/d11/d1c/d24/d2c/fb6 x:0 0 0 2024-04-06T14:17:15.038 INFO:tasks.workunit.client.1.smithi086.stdout:7/543: chown d7/d1d 216234700 1 2024-04-06T14:17:15.039 INFO:tasks.workunit.client.1.smithi086.stdout:9/468: rmdir d5/d13/d1a/d5f/d88 39 2024-04-06T14:17:15.039 INFO:tasks.workunit.client.0.smithi032.stdout:2/726: fdatasync d0/d30/d39/d59/d67/d74/d78/fce 0 2024-04-06T14:17:15.040 INFO:tasks.workunit.client.1.smithi086.stdout:3/537: chown d5/d36/d5a/d6d/l94 238446935 1 2024-04-06T14:17:15.041 INFO:tasks.workunit.client.1.smithi086.stdout:9/469: readlink d5/d13/d1a/d5f/d88/d14/d19/d25/d74/l8a 0 2024-04-06T14:17:15.041 INFO:tasks.workunit.client.0.smithi032.stdout:0/643: link d3/da/d35/d8b/d9c/fc7 d3/da/d25/d2b/d4a/fdb 0 2024-04-06T14:17:15.043 INFO:tasks.workunit.client.0.smithi032.stdout:5/759: mkdir d16/d18/d33/d35/d88/dbd/dc5/d10e 0 2024-04-06T14:17:15.044 INFO:tasks.workunit.client.1.smithi086.stdout:9/470: read d5/d13/f4f [2916373,13026] 0 2024-04-06T14:17:15.045 INFO:tasks.workunit.client.1.smithi086.stdout:3/538: mkdir d5/df/d9e/da5 0 2024-04-06T14:17:15.051 INFO:tasks.workunit.client.1.smithi086.stdout:7/544: getdents d7/d10/d11/d1c/d24/d2c/d69/d8d 0 2024-04-06T14:17:15.051 INFO:tasks.workunit.client.1.smithi086.stdout:9/471: creat d5/d13/d1a/d5f/d88/d14/d19/d1b/f9d x:0 0 0 2024-04-06T14:17:15.051 INFO:tasks.workunit.client.0.smithi032.stdout:3/763: mkdir d3/d65/d7d/d9c/dbc/d128 0 2024-04-06T14:17:15.052 INFO:tasks.workunit.client.0.smithi032.stdout:2/727: read d0/d5a/f1a [3869720,76578] 0 2024-04-06T14:17:15.052 INFO:tasks.workunit.client.0.smithi032.stdout:4/815: rename d0/df/d23/d31/d50/d57/l7d to d0/df/d23/d31/d50/d57/d5b/d9c/dc1/l110 0 2024-04-06T14:17:15.052 INFO:tasks.workunit.client.0.smithi032.stdout:6/832: write d4/d5/d8/d4b/f4a [1823435,59135] 0 2024-04-06T14:17:15.052 INFO:tasks.workunit.client.0.smithi032.stdout:5/760: unlink d16/d58/d54/d74/d8f/cb0 0 2024-04-06T14:17:15.054 INFO:tasks.workunit.client.1.smithi086.stdout:7/545: rmdir d7/d10/d11/d1c/d24/d4d/d5c/d66 39 2024-04-06T14:17:15.057 INFO:tasks.workunit.client.1.smithi086.stdout:3/539: rename d5/d36/d5a/d6d/d72/f5f to d5/d36/d65/fa6 0 2024-04-06T14:17:15.058 INFO:tasks.workunit.client.0.smithi032.stdout:0/644: dwrite d3/f45 [0,4194304] 0 2024-04-06T14:17:15.059 INFO:tasks.workunit.client.0.smithi032.stdout:6/833: chown d4/d5/f15 121395032 1 2024-04-06T14:17:15.059 INFO:tasks.workunit.client.1.smithi086.stdout:6/642: dwrite d6/d9/db/f28 [0,4194304] 0 2024-04-06T14:17:15.062 INFO:tasks.workunit.client.1.smithi086.stdout:5/459: dwrite d0/d19/f4b [0,4194304] 0 2024-04-06T14:17:15.063 INFO:tasks.workunit.client.0.smithi032.stdout:8/876: dwrite d3/d8/d7/d14/fe8 [0,4194304] 0 2024-04-06T14:17:15.063 INFO:tasks.workunit.client.0.smithi032.stdout:4/816: symlink d0/df/d23/d31/d50/d57/d8a/l111 0 2024-04-06T14:17:15.065 INFO:tasks.workunit.client.0.smithi032.stdout:4/817: stat d0/df/d1f/d2e/c7a 0 2024-04-06T14:17:15.067 INFO:tasks.workunit.client.0.smithi032.stdout:3/764: dread d3/d65/f52 [0,4194304] 0 2024-04-06T14:17:15.068 INFO:tasks.workunit.client.1.smithi086.stdout:7/546: creat d7/d10/d11/d1c/d24/d4d/fb7 x:0 0 0 2024-04-06T14:17:15.073 INFO:tasks.workunit.client.0.smithi032.stdout:6/834: mkdir d4/d5/dbf/d11a 0 2024-04-06T14:17:15.076 INFO:tasks.workunit.client.1.smithi086.stdout:6/643: unlink d6/d13/d1a/c3b 0 2024-04-06T14:17:15.076 INFO:tasks.workunit.client.1.smithi086.stdout:7/547: truncate d7/d10/d11/d1c/d24/d4d/daf/db5/d7a/f95 609209 0 2024-04-06T14:17:15.076 INFO:tasks.workunit.client.1.smithi086.stdout:9/472: truncate d5/d13/d1a/d5f/d88/d14/f4e 594631 0 2024-04-06T14:17:15.076 INFO:tasks.workunit.client.1.smithi086.stdout:6/644: truncate d6/d13/faf 847453 0 2024-04-06T14:17:15.077 INFO:tasks.workunit.client.0.smithi032.stdout:3/765: mkdir d3/d16/d63/d129 0 2024-04-06T14:17:15.078 INFO:tasks.workunit.client.0.smithi032.stdout:6/835: symlink d4/d5/d8/d2f/d9b/db4/l11b 0 2024-04-06T14:17:15.078 INFO:tasks.workunit.client.0.smithi032.stdout:2/728: link d0/d30/d39/d59/d67/d74/f9a d0/d30/ff0 0 2024-04-06T14:17:15.079 INFO:tasks.workunit.client.1.smithi086.stdout:6/645: symlink d6/d9/db/ldd 0 2024-04-06T14:17:15.080 INFO:tasks.workunit.client.0.smithi032.stdout:0/645: creat d3/da/d25/d2b/fdc x:0 0 0 2024-04-06T14:17:15.081 INFO:tasks.workunit.client.0.smithi032.stdout:3/766: mknod d3/d65/d7d/d9c/dad/db1/d3c/c12a 0 2024-04-06T14:17:15.081 INFO:tasks.workunit.client.1.smithi086.stdout:6/646: chown d6/d9/d63/lce 30947 1 2024-04-06T14:17:15.082 INFO:tasks.workunit.client.0.smithi032.stdout:3/767: stat d3/d16/d63/l8a 0 2024-04-06T14:17:15.084 INFO:tasks.workunit.client.0.smithi032.stdout:2/729: getdents d0/d30/d39/d59/d67/d74/d78/d8c 0 2024-04-06T14:17:15.085 INFO:tasks.workunit.client.1.smithi086.stdout:6/647: mknod d6/d13/d1a/cde 0 2024-04-06T14:17:15.087 INFO:tasks.workunit.client.0.smithi032.stdout:6/836: dread d4/f10 [0,4194304] 0 2024-04-06T14:17:15.087 INFO:tasks.workunit.client.0.smithi032.stdout:5/761: dread d16/d18/f2b [0,4194304] 0 2024-04-06T14:17:15.092 INFO:tasks.workunit.client.1.smithi086.stdout:6/648: dwrite d6/d13/f77 [8388608,4194304] 0 2024-04-06T14:17:15.093 INFO:tasks.workunit.client.0.smithi032.stdout:5/762: unlink d16/d58/d54/l69 0 2024-04-06T14:17:15.095 INFO:tasks.workunit.client.0.smithi032.stdout:6/837: dwrite d4/d5/d8/d4b/fa6 [0,4194304] 0 2024-04-06T14:17:15.098 INFO:tasks.workunit.client.1.smithi086.stdout:3/540: dwrite d5/f77 [0,4194304] 0 2024-04-06T14:17:15.102 INFO:tasks.workunit.client.0.smithi032.stdout:6/838: mknod d4/d5/d28/d65/d6e/ddb/c11c 0 2024-04-06T14:17:15.107 INFO:tasks.workunit.client.0.smithi032.stdout:5/763: rename d16/d18/d33/d35/d88/dbd/dc5/fe4 to d16/d18/d33/d45/f10f 0 2024-04-06T14:17:15.107 INFO:tasks.workunit.client.1.smithi086.stdout:5/460: dwrite d0/d16/d18/d35/f5b [0,4194304] 0 2024-04-06T14:17:15.108 INFO:tasks.workunit.client.1.smithi086.stdout:3/541: getdents d5/df/d9e 0 2024-04-06T14:17:15.110 INFO:tasks.workunit.client.0.smithi032.stdout:3/768: truncate d3/d65/d7d/d9c/dad/db1/f6c 755617 0 2024-04-06T14:17:15.111 INFO:tasks.workunit.client.1.smithi086.stdout:8/637: sync 2024-04-06T14:17:15.115 INFO:tasks.workunit.client.0.smithi032.stdout:2/730: dwrite d0/d5a/d3a/f61 [0,4194304] 0 2024-04-06T14:17:15.117 INFO:tasks.workunit.client.1.smithi086.stdout:9/473: dread d5/d13/d1a/d5f/d88/d14/d19/d1b/f41 [0,4194304] 0 2024-04-06T14:17:15.117 INFO:tasks.workunit.client.1.smithi086.stdout:7/548: write d7/d10/d11/d1c/f79 [288098,39529] 0 2024-04-06T14:17:15.117 INFO:tasks.workunit.client.1.smithi086.stdout:3/542: rename d5/d36/d5a/d6d/d72/d58 to d5/d36/d56/d9d/da7 0 2024-04-06T14:17:15.118 INFO:tasks.workunit.client.0.smithi032.stdout:6/839: rename d4/d5/f2a to d4/d5/d28/d54/f11d 0 2024-04-06T14:17:15.118 INFO:tasks.workunit.client.0.smithi032.stdout:5/764: rmdir d16/d58 39 2024-04-06T14:17:15.120 INFO:tasks.workunit.client.1.smithi086.stdout:5/461: symlink d0/d17/d2d/d80/d85/l8c 0 2024-04-06T14:17:15.120 INFO:tasks.workunit.client.1.smithi086.stdout:6/649: write d6/d13/d1a/da0/fcc [2922850,1679] 0 2024-04-06T14:17:15.120 INFO:tasks.workunit.client.1.smithi086.stdout:7/549: fsync d7/d10/d11/d1c/d24/d4d/daf/db5/f8a 0 2024-04-06T14:17:15.121 INFO:tasks.workunit.client.0.smithi032.stdout:4/818: dread d0/df/d2b/fd9 [0,4194304] 0 2024-04-06T14:17:15.124 INFO:tasks.workunit.client.1.smithi086.stdout:6/650: readlink d6/d13/d42/l98 0 2024-04-06T14:17:15.124 INFO:tasks.workunit.client.1.smithi086.stdout:9/474: rename d5/d13/d1a/d5f/d88/l78 to d5/d13/l9e 0 2024-04-06T14:17:15.124 INFO:tasks.workunit.client.1.smithi086.stdout:5/462: rmdir d0/d16/d18/d2a/d50 39 2024-04-06T14:17:15.125 INFO:tasks.workunit.client.0.smithi032.stdout:3/769: creat d3/d65/d7d/d9c/dad/f12b x:0 0 0 2024-04-06T14:17:15.126 INFO:tasks.workunit.client.1.smithi086.stdout:3/543: rmdir d5/d36/d5c 39 2024-04-06T14:17:15.126 INFO:tasks.workunit.client.0.smithi032.stdout:5/765: creat d16/d18/d33/d35/d4d/f110 x:0 0 0 2024-04-06T14:17:15.127 INFO:tasks.workunit.client.0.smithi032.stdout:6/840: dread d4/d5/dc5/d82/fb0 [0,4194304] 0 2024-04-06T14:17:15.128 INFO:tasks.workunit.client.1.smithi086.stdout:6/651: symlink d6/d9/d63/ldf 0 2024-04-06T14:17:15.130 INFO:tasks.workunit.client.0.smithi032.stdout:4/819: rename d0/df/d1f/d2e/c2f to d0/df/d23/d31/d50/d57/d8a/db5/def/c112 0 2024-04-06T14:17:15.130 INFO:tasks.workunit.client.1.smithi086.stdout:9/475: chown d5/d13/d1a/d5f/d88/d14/f51 335 1 2024-04-06T14:17:15.130 INFO:tasks.workunit.client.0.smithi032.stdout:2/731: dwrite d0/d30/d39/d59/d67/d80/fd0 [0,4194304] 0 2024-04-06T14:17:15.134 INFO:tasks.workunit.client.0.smithi032.stdout:5/766: fdatasync d16/d18/d33/d45/d52/d39/fb7 0 2024-04-06T14:17:15.135 INFO:tasks.workunit.client.1.smithi086.stdout:3/544: mknod d5/d36/d5c/ca8 0 2024-04-06T14:17:15.136 INFO:tasks.workunit.client.0.smithi032.stdout:4/820: write d0/df/d1f/d5a/f8f [546379,43833] 0 2024-04-06T14:17:15.136 INFO:tasks.workunit.client.0.smithi032.stdout:3/770: mkdir d3/d65/d7d/d9c/dad/db1/d3c/dc9/d104/d12c 0 2024-04-06T14:17:15.137 INFO:tasks.workunit.client.0.smithi032.stdout:6/841: read d4/d5/f4c [760143,31472] 0 2024-04-06T14:17:15.137 INFO:tasks.workunit.client.0.smithi032.stdout:0/646: dread d3/da/d25/d36/f68 [0,4194304] 0 2024-04-06T14:17:15.139 INFO:tasks.workunit.client.0.smithi032.stdout:5/767: write d16/d18/d33/d45/d5e/f89 [549286,9838] 0 2024-04-06T14:17:15.140 INFO:tasks.workunit.client.1.smithi086.stdout:3/545: truncate d5/df/f2c 4683100 0 2024-04-06T14:17:15.142 INFO:tasks.workunit.client.1.smithi086.stdout:7/550: write d7/d10/d11/d80/f50 [6605498,35945] 0 2024-04-06T14:17:15.143 INFO:tasks.workunit.client.1.smithi086.stdout:6/652: rmdir d6 39 2024-04-06T14:17:15.143 INFO:tasks.workunit.client.1.smithi086.stdout:5/463: write d0/d17/f79 [716427,15034] 0 2024-04-06T14:17:15.145 INFO:tasks.workunit.client.0.smithi032.stdout:6/842: write d4/d5/d8/d2f/d9b/f10f [605527,77947] 0 2024-04-06T14:17:15.145 INFO:tasks.workunit.client.1.smithi086.stdout:3/546: read d5/df/f6b [3815322,56040] 0 2024-04-06T14:17:15.148 INFO:tasks.workunit.client.0.smithi032.stdout:4/821: mkdir d0/df/d23/d31/d50/d57/d113 0 2024-04-06T14:17:15.148 INFO:tasks.workunit.client.1.smithi086.stdout:7/551: rename d7/f81 to d7/d1d/fb8 0 2024-04-06T14:17:15.150 INFO:tasks.workunit.client.0.smithi032.stdout:5/768: creat d16/d58/d54/d74/f111 x:0 0 0 2024-04-06T14:17:15.151 INFO:tasks.workunit.client.1.smithi086.stdout:3/547: fdatasync d5/d33/f52 0 2024-04-06T14:17:15.152 INFO:tasks.workunit.client.0.smithi032.stdout:3/771: dwrite d3/d65/d7d/d9c/dad/db1/d3c/dc9/d104/f10a [0,4194304] 0 2024-04-06T14:17:15.152 INFO:tasks.workunit.client.1.smithi086.stdout:5/464: symlink d0/d16/d71/d89/l8d 0 2024-04-06T14:17:15.153 INFO:tasks.workunit.client.0.smithi032.stdout:6/843: write d4/d5/d8/f8e [1034549,47888] 0 2024-04-06T14:17:15.154 INFO:tasks.workunit.client.0.smithi032.stdout:4/822: dread - d0/d82/fe3 zero size 2024-04-06T14:17:15.155 INFO:tasks.workunit.client.0.smithi032.stdout:5/769: readlink d16/d58/d54/l7f 0 2024-04-06T14:17:15.155 INFO:tasks.workunit.client.1.smithi086.stdout:3/548: mkdir d5/d26/da9 0 2024-04-06T14:17:15.156 INFO:tasks.workunit.client.0.smithi032.stdout:0/647: write d3/da/f64 [443684,87531] 0 2024-04-06T14:17:15.157 INFO:tasks.workunit.client.1.smithi086.stdout:5/465: symlink d0/d16/d71/l8e 0 2024-04-06T14:17:15.158 INFO:tasks.workunit.client.0.smithi032.stdout:6/844: stat d4/d5/d19/d25/c58 0 2024-04-06T14:17:15.161 INFO:tasks.workunit.client.0.smithi032.stdout:4/823: creat d0/df/d23/d31/d50/d57/d5b/d9c/dc1/f114 x:0 0 0 2024-04-06T14:17:15.161 INFO:tasks.workunit.client.1.smithi086.stdout:3/549: rename d5/df/f46 to d5/d36/d5c/faa 0 2024-04-06T14:17:15.161 INFO:tasks.workunit.client.1.smithi086.stdout:6/653: write d6/d9/db/f9b [1195777,47377] 0 2024-04-06T14:17:15.162 INFO:tasks.workunit.client.0.smithi032.stdout:5/770: rename d16/d18/f1c to d16/d58/d54/d94/f112 0 2024-04-06T14:17:15.163 INFO:tasks.workunit.client.1.smithi086.stdout:5/466: creat d0/d17/d2d/d52/d7c/f8f x:0 0 0 2024-04-06T14:17:15.164 INFO:tasks.workunit.client.0.smithi032.stdout:6/845: mknod d4/d5/d28/d65/df8/c11e 0 2024-04-06T14:17:15.164 INFO:tasks.workunit.client.1.smithi086.stdout:6/654: dread - d6/d13/d1a/d64/d85/fdb zero size 2024-04-06T14:17:15.164 INFO:tasks.workunit.client.0.smithi032.stdout:4/824: symlink d0/df/d2b/dbe/l115 0 2024-04-06T14:17:15.166 INFO:tasks.workunit.client.0.smithi032.stdout:5/771: creat d16/d18/dae/dbb/ddc/f113 x:0 0 0 2024-04-06T14:17:15.166 INFO:tasks.workunit.client.0.smithi032.stdout:2/732: dread d0/d30/d39/d59/d67/d74/d78/d57/d7c/fa1 [0,4194304] 0 2024-04-06T14:17:15.167 INFO:tasks.workunit.client.0.smithi032.stdout:5/772: chown d16/d18/c27 2496 1 2024-04-06T14:17:15.167 INFO:tasks.workunit.client.1.smithi086.stdout:5/467: mknod d0/d19/c90 0 2024-04-06T14:17:15.169 INFO:tasks.workunit.client.0.smithi032.stdout:4/825: fsync d0/df/d23/fac 0 2024-04-06T14:17:15.169 INFO:tasks.workunit.client.0.smithi032.stdout:5/773: write d16/d18/d33/d45/d52/ff0 [629730,27784] 0 2024-04-06T14:17:15.170 INFO:tasks.workunit.client.1.smithi086.stdout:6/655: rename d6/d13/d5f/d60/d45/d4a/d62 to d6/d20/de0 0 2024-04-06T14:17:15.171 INFO:tasks.workunit.client.1.smithi086.stdout:5/468: mknod d0/d17/d2d/d52/d7c/c91 0 2024-04-06T14:17:15.172 INFO:tasks.workunit.client.1.smithi086.stdout:6/656: mknod d6/d13/d5f/ce1 0 2024-04-06T14:17:15.173 INFO:tasks.workunit.client.0.smithi032.stdout:5/774: write d16/d18/d33/d45/d52/d39/f5b [4513320,129330] 0 2024-04-06T14:17:15.174 INFO:tasks.workunit.client.1.smithi086.stdout:5/469: symlink d0/d16/d18/d21/l92 0 2024-04-06T14:17:15.178 INFO:tasks.workunit.client.0.smithi032.stdout:6/846: dread d4/d5/f6a [4194304,4194304] 0 2024-04-06T14:17:15.180 INFO:tasks.workunit.client.0.smithi032.stdout:2/733: dwrite d0/d6e/f72 [0,4194304] 0 2024-04-06T14:17:15.182 INFO:tasks.workunit.client.0.smithi032.stdout:6/847: creat d4/d5/d14/f11f x:0 0 0 2024-04-06T14:17:15.185 INFO:tasks.workunit.client.1.smithi086.stdout:3/550: dwrite d5/d36/f3b [4194304,4194304] 0 2024-04-06T14:17:15.187 INFO:tasks.workunit.client.0.smithi032.stdout:2/734: fdatasync d0/d30/d39/d59/d67/fc0 0 2024-04-06T14:17:15.189 INFO:tasks.workunit.client.1.smithi086.stdout:3/551: creat d5/d33/d5d/fab x:0 0 0 2024-04-06T14:17:15.194 INFO:tasks.workunit.client.0.smithi032.stdout:2/735: symlink d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/lf1 0 2024-04-06T14:17:15.195 INFO:tasks.workunit.client.1.smithi086.stdout:5/470: dwrite d0/d16/d18/d21/d23/f6c [0,4194304] 0 2024-04-06T14:17:15.196 INFO:tasks.workunit.client.1.smithi086.stdout:6/657: dwrite d6/d13/d5f/d60/d45/d4a/fcb [0,4194304] 0 2024-04-06T14:17:15.204 INFO:tasks.workunit.client.0.smithi032.stdout:2/736: rename d0/d5a/f31 to d0/d6e/ff2 0 2024-04-06T14:17:15.205 INFO:tasks.workunit.client.0.smithi032.stdout:6/848: dwrite d4/d5/d34/f5f [0,4194304] 0 2024-04-06T14:17:15.211 INFO:tasks.workunit.client.1.smithi086.stdout:3/552: dwrite d5/df/f45 [0,4194304] 0 2024-04-06T14:17:15.212 INFO:tasks.workunit.client.0.smithi032.stdout:2/737: creat d0/d30/d39/d59/d67/d74/ff3 x:0 0 0 2024-04-06T14:17:15.214 INFO:tasks.workunit.client.0.smithi032.stdout:2/738: creat d0/d5a/dc9/ff4 x:0 0 0 2024-04-06T14:17:15.224 INFO:tasks.workunit.client.1.smithi086.stdout:5/471: dread d0/d16/d18/f1a [4194304,4194304] 0 2024-04-06T14:17:15.226 INFO:tasks.workunit.client.1.smithi086.stdout:5/472: creat d0/d16/d18/d2a/d50/f93 x:0 0 0 2024-04-06T14:17:15.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:14 smithi032 ceph-mon[39730]: pgmap v25: 65 pgs: 65 active+clean; 2.6 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 83 MiB/s rd, 218 MiB/s wr, 443 op/s 2024-04-06T14:17:15.230 INFO:tasks.workunit.client.1.smithi086.stdout:5/473: getdents d0/d5e 0 2024-04-06T14:17:15.233 INFO:tasks.workunit.client.1.smithi086.stdout:5/474: dread d0/d16/d18/d21/f45 [0,4194304] 0 2024-04-06T14:17:15.235 INFO:tasks.workunit.client.1.smithi086.stdout:5/475: read - d0/d17/d2d/f7a zero size 2024-04-06T14:17:15.235 INFO:tasks.workunit.client.1.smithi086.stdout:6/658: truncate d6/d9/d5c/f6b 145918 0 2024-04-06T14:17:15.236 INFO:tasks.workunit.client.0.smithi032.stdout:6/849: write d4/d5/d8/d2b/f90 [19318,82921] 0 2024-04-06T14:17:15.236 INFO:tasks.workunit.client.1.smithi086.stdout:3/553: write d5/d26/f3d [3436088,124332] 0 2024-04-06T14:17:15.238 INFO:tasks.workunit.client.1.smithi086.stdout:5/476: truncate d0/d16/d18/d21/d23/f6c 5010231 0 2024-04-06T14:17:15.240 INFO:tasks.workunit.client.1.smithi086.stdout:5/477: chown d0/d43 1498 1 2024-04-06T14:17:15.240 INFO:tasks.workunit.client.1.smithi086.stdout:3/554: mknod d5/d26/d7f/cac 0 2024-04-06T14:17:15.241 INFO:tasks.workunit.client.0.smithi032.stdout:6/850: write d4/d5/d14/fb3 [410641,17152] 0 2024-04-06T14:17:15.243 INFO:tasks.workunit.client.0.smithi032.stdout:6/851: read d4/d5/d19/d96/f112 [92129,1987] 0 2024-04-06T14:17:15.244 INFO:tasks.workunit.client.1.smithi086.stdout:6/659: write d6/d9/d43/f9e [716511,23827] 0 2024-04-06T14:17:15.244 INFO:tasks.workunit.client.1.smithi086.stdout:6/660: readlink d6/d13/d42/l98 0 2024-04-06T14:17:15.245 INFO:tasks.workunit.client.1.smithi086.stdout:3/555: creat d5/df/d9e/da5/fad x:0 0 0 2024-04-06T14:17:15.246 INFO:tasks.workunit.client.0.smithi032.stdout:6/852: mkdir d4/d5/d8/d2b/d77/d120 0 2024-04-06T14:17:15.246 INFO:tasks.workunit.client.0.smithi032.stdout:6/853: fsync d4/d5/d8/f8e 0 2024-04-06T14:17:15.247 INFO:tasks.workunit.client.1.smithi086.stdout:5/478: rename d0/d17/d2d to d0/d16/d18/d21/d23/d28/d94 0 2024-04-06T14:17:15.255 INFO:tasks.workunit.client.1.smithi086.stdout:3/556: mkdir d5/d26/d7f/dae 0 2024-04-06T14:17:15.255 INFO:tasks.workunit.client.1.smithi086.stdout:6/661: write f5 [3361701,51087] 0 2024-04-06T14:17:15.255 INFO:tasks.workunit.client.1.smithi086.stdout:5/479: creat d0/d16/f95 x:0 0 0 2024-04-06T14:17:15.255 INFO:tasks.workunit.client.1.smithi086.stdout:6/662: dread - d6/d13/d1a/fd0 zero size 2024-04-06T14:17:15.255 INFO:tasks.workunit.client.1.smithi086.stdout:6/663: stat d6/d13/d1a/d64 0 2024-04-06T14:17:15.256 INFO:tasks.workunit.client.1.smithi086.stdout:3/557: creat d5/d36/d5a/d6d/d72/faf x:0 0 0 2024-04-06T14:17:15.256 INFO:tasks.workunit.client.1.smithi086.stdout:5/480: dread - d0/d16/d18/d21/d23/d28/d94/f56 zero size 2024-04-06T14:17:15.256 INFO:tasks.workunit.client.1.smithi086.stdout:6/664: mkdir d6/d13/d1a/d64/de2 0 2024-04-06T14:17:15.259 INFO:tasks.workunit.client.1.smithi086.stdout:3/558: mkdir d5/d36/d5a/d6d/d72/d97/db0 0 2024-04-06T14:17:15.259 INFO:tasks.workunit.client.1.smithi086.stdout:6/665: mkdir d6/d13/d5f/d60/d45/d8a/de3 0 2024-04-06T14:17:15.262 INFO:tasks.workunit.client.1.smithi086.stdout:6/666: fdatasync d6/d9/db/fd2 0 2024-04-06T14:17:15.262 INFO:tasks.workunit.client.1.smithi086.stdout:3/559: dread d5/f3e [0,4194304] 0 2024-04-06T14:17:15.263 INFO:tasks.workunit.client.1.smithi086.stdout:6/667: chown d6/d9/d43/f74 15584567 1 2024-04-06T14:17:15.264 INFO:tasks.workunit.client.1.smithi086.stdout:5/481: dwrite d0/d16/d18/d35/f4d [0,4194304] 0 2024-04-06T14:17:15.264 INFO:tasks.workunit.client.1.smithi086.stdout:6/668: chown d6/d13/d1a/d64 40648857 1 2024-04-06T14:17:15.265 INFO:tasks.workunit.client.0.smithi032.stdout:6/854: dread d4/d5/d19/d25/f87 [0,4194304] 0 2024-04-06T14:17:15.265 INFO:tasks.workunit.client.0.smithi032.stdout:6/855: chown d4 1 1 2024-04-06T14:17:15.265 INFO:tasks.workunit.client.1.smithi086.stdout:3/560: mknod d5/d36/d5a/d6d/d72/d57/cb1 0 2024-04-06T14:17:15.266 INFO:tasks.workunit.client.1.smithi086.stdout:5/482: mkdir d0/d16/d71/d96 0 2024-04-06T14:17:15.266 INFO:tasks.workunit.client.1.smithi086.stdout:3/561: write d5/d26/f3d [451070,50409] 0 2024-04-06T14:17:15.267 INFO:tasks.workunit.client.1.smithi086.stdout:6/669: mkdir d6/d13/d1a/d22/de4 0 2024-04-06T14:17:15.267 INFO:tasks.workunit.client.0.smithi032.stdout:6/856: mkdir d4/d5/d8/d4b/d121 0 2024-04-06T14:17:15.267 INFO:tasks.workunit.client.0.smithi032.stdout:6/857: readlink d4/d5/d8/lb7 0 2024-04-06T14:17:15.268 INFO:tasks.workunit.client.1.smithi086.stdout:3/562: chown d5/d33/f78 462 1 2024-04-06T14:17:15.270 INFO:tasks.workunit.client.1.smithi086.stdout:3/563: fdatasync d5/d26/f3c 0 2024-04-06T14:17:15.271 INFO:tasks.workunit.client.1.smithi086.stdout:5/483: symlink d0/d16/l97 0 2024-04-06T14:17:15.273 INFO:tasks.workunit.client.1.smithi086.stdout:6/670: getdents d6 0 2024-04-06T14:17:15.274 INFO:tasks.workunit.client.1.smithi086.stdout:3/564: fdatasync d5/d36/d65/f7b 0 2024-04-06T14:17:15.280 INFO:tasks.workunit.client.1.smithi086.stdout:6/671: dwrite d6/d13/d5f/d60/d45/d4a/f79 [0,4194304] 0 2024-04-06T14:17:15.281 INFO:tasks.workunit.client.1.smithi086.stdout:6/672: chown d6/f3c 24117 1 2024-04-06T14:17:15.284 INFO:tasks.workunit.client.1.smithi086.stdout:1/576: sync 2024-04-06T14:17:15.285 INFO:tasks.workunit.client.1.smithi086.stdout:1/577: fsync db/d11/d1f/f44 0 2024-04-06T14:17:15.287 INFO:tasks.workunit.client.1.smithi086.stdout:6/673: creat d6/d13/d1a/fe5 x:0 0 0 2024-04-06T14:17:15.287 INFO:tasks.workunit.client.1.smithi086.stdout:3/565: dwrite d5/df/f81 [0,4194304] 0 2024-04-06T14:17:15.291 INFO:tasks.workunit.client.1.smithi086.stdout:6/674: creat d6/d13/d1a/d64/d85/fe6 x:0 0 0 2024-04-06T14:17:15.293 INFO:tasks.workunit.client.1.smithi086.stdout:3/566: creat d5/d26/d7f/fb2 x:0 0 0 2024-04-06T14:17:15.297 INFO:tasks.workunit.client.1.smithi086.stdout:1/578: dwrite db/d11/d1f/f44 [0,4194304] 0 2024-04-06T14:17:15.299 INFO:tasks.workunit.client.1.smithi086.stdout:6/675: creat d6/d13/d5f/d60/d45/fe7 x:0 0 0 2024-04-06T14:17:15.304 INFO:tasks.workunit.client.1.smithi086.stdout:6/676: link d6/d13/d1a/d64/fb3 d6/d9/fe8 0 2024-04-06T14:17:15.307 INFO:tasks.workunit.client.1.smithi086.stdout:3/567: write d5/d33/f78 [697189,91873] 0 2024-04-06T14:17:15.307 INFO:tasks.workunit.client.1.smithi086.stdout:1/579: write db/d11/d24/f60 [4893355,16172] 0 2024-04-06T14:17:15.310 INFO:tasks.workunit.client.1.smithi086.stdout:1/580: unlink db/d11/d1f/c21 0 2024-04-06T14:17:15.311 INFO:tasks.workunit.client.1.smithi086.stdout:1/581: rmdir db/d11 39 2024-04-06T14:17:15.312 INFO:tasks.workunit.client.1.smithi086.stdout:6/677: write d6/d13/f75 [476024,101313] 0 2024-04-06T14:17:15.313 INFO:tasks.workunit.client.1.smithi086.stdout:3/568: dread d5/d26/d7f/f55 [0,4194304] 0 2024-04-06T14:17:15.315 INFO:tasks.workunit.client.1.smithi086.stdout:1/582: creat db/d11/d1a/d2a/fb5 x:0 0 0 2024-04-06T14:17:15.316 INFO:tasks.workunit.client.1.smithi086.stdout:3/569: fdatasync f4 0 2024-04-06T14:17:15.317 INFO:tasks.workunit.client.1.smithi086.stdout:1/583: creat db/d11/d87/fb6 x:0 0 0 2024-04-06T14:17:15.320 INFO:tasks.workunit.client.1.smithi086.stdout:3/570: symlink d5/df/d9e/lb3 0 2024-04-06T14:17:15.324 INFO:tasks.workunit.client.1.smithi086.stdout:1/584: getdents db/d11 0 2024-04-06T14:17:15.330 INFO:tasks.workunit.client.1.smithi086.stdout:3/571: dwrite d5/d36/d5a/d6d/d72/d64/f98 [0,4194304] 0 2024-04-06T14:17:15.334 INFO:tasks.workunit.client.1.smithi086.stdout:3/572: unlink d5/d33/d5d/fab 0 2024-04-06T14:17:15.339 INFO:tasks.workunit.client.1.smithi086.stdout:1/585: dread db/d11/d42/d53/f57 [0,4194304] 0 2024-04-06T14:17:15.339 INFO:tasks.workunit.client.1.smithi086.stdout:3/573: rename d5/d36/d56/d9d to d5/d26/d4f/db4 0 2024-04-06T14:17:15.344 INFO:tasks.workunit.client.1.smithi086.stdout:1/586: dread db/d11/d1a/d2a/f4c [0,4194304] 0 2024-04-06T14:17:15.346 INFO:tasks.workunit.client.1.smithi086.stdout:1/587: symlink db/d13/d55/lb7 0 2024-04-06T14:17:15.348 INFO:tasks.workunit.client.1.smithi086.stdout:1/588: mknod db/d11/d1d/cb8 0 2024-04-06T14:17:15.351 INFO:tasks.workunit.client.1.smithi086.stdout:3/574: dwrite d5/d33/f52 [0,4194304] 0 2024-04-06T14:17:15.352 INFO:tasks.workunit.client.1.smithi086.stdout:3/575: chown d5/d36/d65/c6e 6 1 2024-04-06T14:17:15.353 INFO:tasks.workunit.client.1.smithi086.stdout:3/576: chown d5/d26/d4f/c41 114 1 2024-04-06T14:17:15.353 INFO:tasks.workunit.client.1.smithi086.stdout:3/577: readlink d5/l38 0 2024-04-06T14:17:15.354 INFO:tasks.workunit.client.1.smithi086.stdout:1/589: creat db/d11/d1a/d2a/d9a/d68/d6f/fb9 x:0 0 0 2024-04-06T14:17:15.355 INFO:tasks.workunit.client.1.smithi086.stdout:3/578: mkdir d5/df/d9e/db5 0 2024-04-06T14:17:15.359 INFO:tasks.workunit.client.1.smithi086.stdout:1/590: link db/d11/d1a/d2a/fb5 db/d11/d24/fba 0 2024-04-06T14:17:15.360 INFO:tasks.workunit.client.1.smithi086.stdout:3/579: truncate d5/d36/d5a/d6d/d72/f75 314997 0 2024-04-06T14:17:15.360 INFO:tasks.workunit.client.1.smithi086.stdout:1/591: stat db/d11/d1a/d2a/d9a/d68/d6e/f8f 0 2024-04-06T14:17:15.360 INFO:tasks.workunit.client.1.smithi086.stdout:3/580: chown d5/d33/l39 4799 1 2024-04-06T14:17:15.362 INFO:tasks.workunit.client.1.smithi086.stdout:1/592: mknod db/d11/d1a/d2a/d52/cbb 0 2024-04-06T14:17:15.363 INFO:tasks.workunit.client.1.smithi086.stdout:3/581: creat d5/d36/d5a/d6d/d72/d97/fb6 x:0 0 0 2024-04-06T14:17:15.365 INFO:tasks.workunit.client.1.smithi086.stdout:1/593: creat db/d11/d1f/fbc x:0 0 0 2024-04-06T14:17:15.365 INFO:tasks.workunit.client.1.smithi086.stdout:3/582: mknod d5/d36/d7d/cb7 0 2024-04-06T14:17:15.365 INFO:tasks.workunit.client.1.smithi086.stdout:1/594: chown db/d11/d1d/cb8 223676 1 2024-04-06T14:17:15.372 INFO:tasks.workunit.client.1.smithi086.stdout:1/595: dwrite db/d13/d55/f6a [0,4194304] 0 2024-04-06T14:17:15.375 INFO:tasks.workunit.client.1.smithi086.stdout:3/583: write d5/d26/d7f/f4c [2890943,58862] 0 2024-04-06T14:17:15.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:14 smithi086 ceph-mon[47925]: pgmap v25: 65 pgs: 65 active+clean; 2.6 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 83 MiB/s rd, 218 MiB/s wr, 443 op/s 2024-04-06T14:17:15.377 INFO:tasks.workunit.client.1.smithi086.stdout:3/584: dread d5/f3e [0,4194304] 0 2024-04-06T14:17:15.379 INFO:tasks.workunit.client.1.smithi086.stdout:1/596: write db/d11/d1a/d2a/f47 [1367087,12577] 0 2024-04-06T14:17:15.382 INFO:tasks.workunit.client.1.smithi086.stdout:3/585: getdents d5/d26/d7f/d59 0 2024-04-06T14:17:15.382 INFO:tasks.workunit.client.1.smithi086.stdout:3/586: chown d5/d26/d7f/d59/d70 36 1 2024-04-06T14:17:15.384 INFO:tasks.workunit.client.1.smithi086.stdout:1/597: rename db/d11/d1a/d2a/d9a/d68/d6e/d95 to db/d11/d42/dbd 0 2024-04-06T14:17:15.386 INFO:tasks.workunit.client.1.smithi086.stdout:3/587: link d5/d33/d5d/f92 d5/d26/da9/fb8 0 2024-04-06T14:17:15.386 INFO:tasks.workunit.client.1.smithi086.stdout:3/588: chown d5/d33/f47 4 1 2024-04-06T14:17:15.430 INFO:tasks.workunit.client.1.smithi086.stdout:2/558: sync 2024-04-06T14:17:15.430 INFO:tasks.workunit.client.1.smithi086.stdout:8/638: sync 2024-04-06T14:17:15.430 INFO:tasks.workunit.client.1.smithi086.stdout:9/476: sync 2024-04-06T14:17:15.430 INFO:tasks.workunit.client.1.smithi086.stdout:0/604: sync 2024-04-06T14:17:15.430 INFO:tasks.workunit.client.1.smithi086.stdout:4/441: sync 2024-04-06T14:17:15.434 INFO:tasks.workunit.client.1.smithi086.stdout:2/559: unlink d0/d20/d27/d37/f7e 0 2024-04-06T14:17:15.437 INFO:tasks.workunit.client.1.smithi086.stdout:2/560: fsync d0/d20/d27/d2b/fa4 0 2024-04-06T14:17:15.438 INFO:tasks.workunit.client.1.smithi086.stdout:4/442: write d0/d2c/d30/d24/f31 [3129742,24228] 0 2024-04-06T14:17:15.440 INFO:tasks.workunit.client.1.smithi086.stdout:9/477: dread d5/d13/d1a/d5f/d88/f53 [0,4194304] 0 2024-04-06T14:17:15.440 INFO:tasks.workunit.client.1.smithi086.stdout:4/443: chown d0/d2c/d38/c3e 310 1 2024-04-06T14:17:15.441 INFO:tasks.workunit.client.1.smithi086.stdout:2/561: fdatasync d0/df/f18 0 2024-04-06T14:17:15.443 INFO:tasks.workunit.client.1.smithi086.stdout:4/444: mkdir d0/d15/d19/d88 0 2024-04-06T14:17:15.444 INFO:tasks.workunit.client.1.smithi086.stdout:9/478: creat d5/d58/f9f x:0 0 0 2024-04-06T14:17:15.445 INFO:tasks.workunit.client.0.smithi032.stdout:9/823: sync 2024-04-06T14:17:15.446 INFO:tasks.workunit.client.0.smithi032.stdout:8/877: sync 2024-04-06T14:17:15.446 INFO:tasks.workunit.client.0.smithi032.stdout:1/877: sync 2024-04-06T14:17:15.446 INFO:tasks.workunit.client.0.smithi032.stdout:7/848: sync 2024-04-06T14:17:15.446 INFO:tasks.workunit.client.1.smithi086.stdout:8/639: dread d0/d71/d22/f74 [0,4194304] 0 2024-04-06T14:17:15.447 INFO:tasks.workunit.client.0.smithi032.stdout:7/849: write d16/d20/d7a/da6/fd6 [4735357,19492] 0 2024-04-06T14:17:15.450 INFO:tasks.workunit.client.1.smithi086.stdout:8/640: stat d0/d71/l14 0 2024-04-06T14:17:15.451 INFO:tasks.workunit.client.1.smithi086.stdout:0/605: dread d0/d1/d2/d1a/f7b [0,4194304] 0 2024-04-06T14:17:15.452 INFO:tasks.workunit.client.1.smithi086.stdout:2/562: dwrite d0/d20/d27/d2b/d35/d36/d4a/f86 [0,4194304] 0 2024-04-06T14:17:15.452 INFO:tasks.workunit.client.0.smithi032.stdout:1/878: rename d0/d9/d1f/d7c/da9/ddd/l11c to d0/d41/d5b/l11d 0 2024-04-06T14:17:15.453 INFO:tasks.workunit.client.0.smithi032.stdout:7/850: symlink d16/d20/d54/de4/d5f/l12f 0 2024-04-06T14:17:15.454 INFO:tasks.workunit.client.0.smithi032.stdout:7/851: stat d16/cc9 0 2024-04-06T14:17:15.458 INFO:tasks.workunit.client.0.smithi032.stdout:9/824: truncate d2/d15/d55/f6e 893149 0 2024-04-06T14:17:15.458 INFO:tasks.workunit.client.0.smithi032.stdout:7/852: creat d16/d20/dbd/f130 x:0 0 0 2024-04-06T14:17:15.459 INFO:tasks.workunit.client.1.smithi086.stdout:8/641: getdents d0/d71/d22/d43/d7f 0 2024-04-06T14:17:15.459 INFO:tasks.workunit.client.1.smithi086.stdout:9/479: dwrite d5/f20 [0,4194304] 0 2024-04-06T14:17:15.461 INFO:tasks.workunit.client.1.smithi086.stdout:8/642: truncate d0/db/d32/f3e 1736881 0 2024-04-06T14:17:15.463 INFO:tasks.workunit.client.1.smithi086.stdout:9/480: write d5/d13/d1a/d5f/d88/d14/d19/d25/d59/f60 [4453891,30170] 0 2024-04-06T14:17:15.465 INFO:tasks.workunit.client.1.smithi086.stdout:0/606: write d0/d1/d4/f6b [1980764,106386] 0 2024-04-06T14:17:15.466 INFO:tasks.workunit.client.1.smithi086.stdout:2/563: write d0/df/d48/d53/d76/f91 [3424590,41610] 0 2024-04-06T14:17:15.470 INFO:tasks.workunit.client.1.smithi086.stdout:9/481: fdatasync d5/d13/d1a/d5f/d88/d14/f4e 0 2024-04-06T14:17:15.471 INFO:tasks.workunit.client.0.smithi032.stdout:8/878: dwrite d3/d8/d12/d83/d89/da0/ffe [0,4194304] 0 2024-04-06T14:17:15.474 INFO:tasks.workunit.client.0.smithi032.stdout:8/879: symlink d3/d69/l12a 0 2024-04-06T14:17:15.475 INFO:tasks.workunit.client.1.smithi086.stdout:0/607: symlink d0/d1/d2/lc7 0 2024-04-06T14:17:15.476 INFO:tasks.workunit.client.1.smithi086.stdout:8/643: getdents d0/d71/d93 0 2024-04-06T14:17:15.476 INFO:tasks.workunit.client.1.smithi086.stdout:9/482: symlink d5/d13/la0 0 2024-04-06T14:17:15.478 INFO:tasks.workunit.client.0.smithi032.stdout:8/880: mkdir d3/d8/d3a/d7b/db9/ddf/d12b 0 2024-04-06T14:17:15.478 INFO:tasks.workunit.client.0.smithi032.stdout:9/825: dread d2/d15/d1e/d65/d83/f32 [0,4194304] 0 2024-04-06T14:17:15.479 INFO:tasks.workunit.client.0.smithi032.stdout:7/853: write d16/f40 [4696324,613] 0 2024-04-06T14:17:15.481 INFO:tasks.workunit.client.0.smithi032.stdout:8/881: readlink d3/d8/d12/d2f/d46/d125/l116 0 2024-04-06T14:17:15.481 INFO:tasks.workunit.client.1.smithi086.stdout:0/608: read - d0/d1/d4/de/db4/fb5 zero size 2024-04-06T14:17:15.481 INFO:tasks.workunit.client.1.smithi086.stdout:8/644: read d0/d13/f40 [247340,128820] 0 2024-04-06T14:17:15.485 INFO:tasks.workunit.client.0.smithi032.stdout:4/826: sync 2024-04-06T14:17:15.485 INFO:tasks.workunit.client.0.smithi032.stdout:3/772: sync 2024-04-06T14:17:15.485 INFO:tasks.workunit.client.0.smithi032.stdout:0/648: sync 2024-04-06T14:17:15.485 INFO:tasks.workunit.client.0.smithi032.stdout:2/739: sync 2024-04-06T14:17:15.485 INFO:tasks.workunit.client.0.smithi032.stdout:6/858: sync 2024-04-06T14:17:15.485 INFO:tasks.workunit.client.0.smithi032.stdout:5/775: sync 2024-04-06T14:17:15.488 INFO:tasks.workunit.client.0.smithi032.stdout:2/740: fdatasync d0/d5a/f41 0 2024-04-06T14:17:15.489 INFO:tasks.workunit.client.0.smithi032.stdout:4/827: stat d0/df/d23/d31/d50/d5e/faa 0 2024-04-06T14:17:15.489 INFO:tasks.workunit.client.0.smithi032.stdout:3/773: chown d3/d65/d7d/d9c/dad/db1/d3c/l73 6 1 2024-04-06T14:17:15.493 INFO:tasks.workunit.client.0.smithi032.stdout:4/828: chown d0/d82/d4b/db3/ce0 78452818 1 2024-04-06T14:17:15.493 INFO:tasks.workunit.client.0.smithi032.stdout:0/649: getdents d3 0 2024-04-06T14:17:15.493 INFO:tasks.workunit.client.0.smithi032.stdout:5/776: rename d16/d18/d33/d35/f6a to d16/d18/d33/d35/d88/deb/f114 0 2024-04-06T14:17:15.494 INFO:tasks.workunit.client.1.smithi086.stdout:2/564: write d0/d20/d27/f7f [635880,57943] 0 2024-04-06T14:17:15.495 INFO:tasks.workunit.client.0.smithi032.stdout:6/859: dread d4/d5/d34/f5f [0,4194304] 0 2024-04-06T14:17:15.496 INFO:tasks.workunit.client.0.smithi032.stdout:7/854: write d16/d20/d48/fa3 [193655,130757] 0 2024-04-06T14:17:15.496 INFO:tasks.workunit.client.1.smithi086.stdout:9/483: getdents d5/d13 0 2024-04-06T14:17:15.497 INFO:tasks.workunit.client.0.smithi032.stdout:2/741: getdents d0/d30/d39/d59/d67/d74/d78/d57/dd3 0 2024-04-06T14:17:15.498 INFO:tasks.workunit.client.1.smithi086.stdout:8/645: write d0/f72 [1892802,76047] 0 2024-04-06T14:17:15.499 INFO:tasks.workunit.client.0.smithi032.stdout:1/879: dread d0/d9/f23 [0,4194304] 0 2024-04-06T14:17:15.500 INFO:tasks.workunit.client.0.smithi032.stdout:0/650: chown d3/da/d41/l47 1567 1 2024-04-06T14:17:15.501 INFO:tasks.workunit.client.1.smithi086.stdout:9/484: symlink d5/d13/d1a/d5f/d88/d14/d19/d25/d59/la1 0 2024-04-06T14:17:15.502 INFO:tasks.workunit.client.1.smithi086.stdout:8/646: rmdir d0/d71/d22/d65/d6e 39 2024-04-06T14:17:15.504 INFO:tasks.workunit.client.0.smithi032.stdout:4/829: rename d0/df/d2b/d66 to d0/df/d2b/dbe/d116 0 2024-04-06T14:17:15.509 INFO:tasks.workunit.client.1.smithi086.stdout:0/609: dwrite d0/d1/d2/d1a/f94 [0,4194304] 0 2024-04-06T14:17:15.510 INFO:tasks.workunit.client.1.smithi086.stdout:2/565: dwrite d0/d20/d27/d2b/d35/f47 [0,4194304] 0 2024-04-06T14:17:15.511 INFO:tasks.workunit.client.0.smithi032.stdout:2/742: dread d0/d5a/f9d [0,4194304] 0 2024-04-06T14:17:15.511 INFO:tasks.workunit.client.0.smithi032.stdout:1/880: stat d0/d10/f25 0 2024-04-06T14:17:15.511 INFO:tasks.workunit.client.1.smithi086.stdout:9/485: mknod d5/d13/d1a/d5f/d88/d14/d19/d25/ca2 0 2024-04-06T14:17:15.511 INFO:tasks.workunit.client.0.smithi032.stdout:8/882: dwrite d3/f8e [0,4194304] 0 2024-04-06T14:17:15.514 INFO:tasks.workunit.client.1.smithi086.stdout:6/678: sync 2024-04-06T14:17:15.514 INFO:tasks.workunit.client.1.smithi086.stdout:5/484: sync 2024-04-06T14:17:15.514 INFO:tasks.workunit.client.1.smithi086.stdout:3/589: sync 2024-04-06T14:17:15.514 INFO:tasks.workunit.client.1.smithi086.stdout:7/552: sync 2024-04-06T14:17:15.514 INFO:tasks.workunit.client.1.smithi086.stdout:1/598: sync 2024-04-06T14:17:15.515 INFO:tasks.workunit.client.1.smithi086.stdout:8/647: rename d0/d71/d22/d43/f28 to d0/db/d32/dd6/fd9 0 2024-04-06T14:17:15.515 INFO:tasks.workunit.client.1.smithi086.stdout:9/486: write d5/d58/f9f [28059,130449] 0 2024-04-06T14:17:15.516 INFO:tasks.workunit.client.0.smithi032.stdout:4/830: mknod d0/df/d1f/d5a/c117 0 2024-04-06T14:17:15.516 INFO:tasks.workunit.client.1.smithi086.stdout:2/566: rmdir d0/d20/d80 39 2024-04-06T14:17:15.518 INFO:tasks.workunit.client.0.smithi032.stdout:9/826: dread d2/d15/d1e/d41/d6c/d74/fe5 [4194304,4194304] 0 2024-04-06T14:17:15.519 INFO:tasks.workunit.client.1.smithi086.stdout:0/610: write d0/d15/d4e/f45 [943945,92478] 0 2024-04-06T14:17:15.520 INFO:tasks.workunit.client.0.smithi032.stdout:5/777: dwrite d16/d58/d54/d74/fd1 [0,4194304] 0 2024-04-06T14:17:15.521 INFO:tasks.workunit.client.1.smithi086.stdout:8/648: creat d0/d71/d16/d52/dd0/fda x:0 0 0 2024-04-06T14:17:15.522 INFO:tasks.workunit.client.0.smithi032.stdout:7/855: write d16/d20/d54/de4/d2d/d30/d97/fdb [560137,82384] 0 2024-04-06T14:17:15.524 INFO:tasks.workunit.client.0.smithi032.stdout:6/860: link d4/d5/d28/d54/f89 d4/d5/dbf/d53/f122 0 2024-04-06T14:17:15.525 INFO:tasks.workunit.client.0.smithi032.stdout:0/651: dread d3/da/d25/f94 [0,4194304] 0 2024-04-06T14:17:15.527 INFO:tasks.workunit.client.1.smithi086.stdout:8/649: chown d0/d71/d22/d43/d4d/c7e 18 1 2024-04-06T14:17:15.527 INFO:tasks.workunit.client.1.smithi086.stdout:7/553: fdatasync d7/d10/d11/d1c/d24/d2c/f48 0 2024-04-06T14:17:15.527 INFO:tasks.workunit.client.1.smithi086.stdout:1/599: fsync db/d11/d1a/d2a/f4c 0 2024-04-06T14:17:15.528 INFO:tasks.workunit.client.1.smithi086.stdout:8/650: stat d0/db/d32/l49 0 2024-04-06T14:17:15.528 INFO:tasks.workunit.client.1.smithi086.stdout:5/485: read d0/d16/d18/d21/d23/d28/d94/f4f [8238420,55790] 0 2024-04-06T14:17:15.529 INFO:tasks.workunit.client.0.smithi032.stdout:7/856: chown d16/d20/d63/d9d/df2/df5/dfd 253870565 1 2024-04-06T14:17:15.529 INFO:tasks.workunit.client.0.smithi032.stdout:4/831: truncate d0/d82/d4b/db3/ff7 378344 0 2024-04-06T14:17:15.529 INFO:tasks.workunit.client.0.smithi032.stdout:0/652: readlink d3/da/d25/d2b/d4a/da1/lac 0 2024-04-06T14:17:15.529 INFO:tasks.workunit.client.1.smithi086.stdout:0/611: read - d0/d1/d25/fa7 zero size 2024-04-06T14:17:15.533 INFO:tasks.workunit.client.0.smithi032.stdout:6/861: readlink d4/d5/d83/l107 0 2024-04-06T14:17:15.533 INFO:tasks.workunit.client.0.smithi032.stdout:2/743: creat d0/d30/d39/d59/d67/d74/d78/dc2/ff5 x:0 0 0 2024-04-06T14:17:15.534 INFO:tasks.workunit.client.1.smithi086.stdout:9/487: truncate d5/d13/d1a/d5f/d88/d14/d19/d25/f67 1143802 0 2024-04-06T14:17:15.534 INFO:tasks.workunit.client.0.smithi032.stdout:5/778: mknod d16/d18/d33/d35/d88/dbd/dc5/d10e/c115 0 2024-04-06T14:17:15.537 INFO:tasks.workunit.client.1.smithi086.stdout:5/486: mkdir d0/d5e/d98 0 2024-04-06T14:17:15.539 INFO:tasks.workunit.client.1.smithi086.stdout:5/487: write d0/d16/d18/d21/d23/f6c [3403452,5311] 0 2024-04-06T14:17:15.541 INFO:tasks.workunit.client.1.smithi086.stdout:0/612: creat d0/d15/d4e/da3/d44/d5d/fc8 x:0 0 0 2024-04-06T14:17:15.543 INFO:tasks.workunit.client.1.smithi086.stdout:6/679: write d6/d13/d42/d52/f90 [440276,10424] 0 2024-04-06T14:17:15.543 INFO:tasks.workunit.client.1.smithi086.stdout:7/554: mkdir d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9 0 2024-04-06T14:17:15.545 INFO:tasks.workunit.client.0.smithi032.stdout:2/744: read d0/f23 [5023720,108777] 0 2024-04-06T14:17:15.546 INFO:tasks.workunit.client.1.smithi086.stdout:8/651: mkdir d0/d71/d22/d65/d6e/d98/ddb 0 2024-04-06T14:17:15.548 INFO:tasks.workunit.client.1.smithi086.stdout:3/590: write d5/d26/d4f/db4/da7/f63 [645107,31468] 0 2024-04-06T14:17:15.549 INFO:tasks.workunit.client.0.smithi032.stdout:1/881: write d0/f90 [43642,45510] 0 2024-04-06T14:17:15.549 INFO:tasks.workunit.client.0.smithi032.stdout:5/779: stat d16/d18/d33/d45/d52/d39/d86/fef 0 2024-04-06T14:17:15.552 INFO:tasks.workunit.client.0.smithi032.stdout:6/862: truncate d4/d5/dbf/d51/fec 1142869 0 2024-04-06T14:17:15.552 INFO:tasks.workunit.client.1.smithi086.stdout:2/567: link d0/l24 d0/d20/d82/lc5 0 2024-04-06T14:17:15.553 INFO:tasks.workunit.client.0.smithi032.stdout:1/882: fdatasync d0/d10/f30 0 2024-04-06T14:17:15.553 INFO:tasks.workunit.client.1.smithi086.stdout:0/613: creat d0/d1/d4/de/db4/fc9 x:0 0 0 2024-04-06T14:17:15.555 INFO:tasks.workunit.client.0.smithi032.stdout:9/827: dwrite d2/d15/d1e/d65/d83/d31/fde [0,4194304] 0 2024-04-06T14:17:15.557 INFO:tasks.workunit.client.0.smithi032.stdout:8/883: dread d3/d8/d12/d2f/f39 [0,4194304] 0 2024-04-06T14:17:15.557 INFO:tasks.workunit.client.0.smithi032.stdout:1/883: truncate d0/d10/d16/fd2 793698 0 2024-04-06T14:17:15.558 INFO:tasks.workunit.client.1.smithi086.stdout:8/652: mkdir d0/d71/d16/d52/dd0/ddc 0 2024-04-06T14:17:15.559 INFO:tasks.workunit.client.1.smithi086.stdout:1/600: link db/d11/d1d/c5e db/d11/d1a/d79/da5/cbe 0 2024-04-06T14:17:15.560 INFO:tasks.workunit.client.1.smithi086.stdout:3/591: symlink d5/d26/d4f/lb9 0 2024-04-06T14:17:15.563 INFO:tasks.workunit.client.1.smithi086.stdout:5/488: rename d0/d16/d18/ff to d0/d17/f99 0 2024-04-06T14:17:15.563 INFO:tasks.workunit.client.1.smithi086.stdout:9/488: dwrite d5/d13/d1a/d5f/d88/d14/f6d [4194304,4194304] 0 2024-04-06T14:17:15.564 INFO:tasks.workunit.client.0.smithi032.stdout:0/653: symlink d3/da/d25/d2b/d80/db3/ldd 0 2024-04-06T14:17:15.565 INFO:tasks.workunit.client.1.smithi086.stdout:0/614: rmdir d0/d15/d26/d33 39 2024-04-06T14:17:15.567 INFO:tasks.workunit.client.0.smithi032.stdout:7/857: symlink d16/d20/d54/de4/d5f/l131 0 2024-04-06T14:17:15.567 INFO:tasks.workunit.client.1.smithi086.stdout:4/445: sync 2024-04-06T14:17:15.568 INFO:tasks.workunit.client.0.smithi032.stdout:2/745: symlink d0/d30/d39/d59/d67/d74/d78/d57/d7c/lf6 0 2024-04-06T14:17:15.570 INFO:tasks.workunit.client.0.smithi032.stdout:2/746: dread - d0/d30/d39/d59/d67/d74/d78/fce zero size 2024-04-06T14:17:15.571 INFO:tasks.workunit.client.1.smithi086.stdout:8/653: rmdir d0/d5a 39 2024-04-06T14:17:15.573 INFO:tasks.workunit.client.0.smithi032.stdout:8/884: creat d3/d8/d12/d83/d89/da0/dd2/dfa/f12c x:0 0 0 2024-04-06T14:17:15.575 INFO:tasks.workunit.client.1.smithi086.stdout:6/680: rename d6/d13/d5f/c7c to d6/d13/d5f/d60/ce9 0 2024-04-06T14:17:15.575 INFO:tasks.workunit.client.0.smithi032.stdout:0/654: unlink d3/da/d25/d2b/d4a/d86/ca7 0 2024-04-06T14:17:15.576 INFO:tasks.workunit.client.1.smithi086.stdout:4/446: mkdir d0/d2c/d30/d24/d50/d79/d89 0 2024-04-06T14:17:15.576 INFO:tasks.workunit.client.0.smithi032.stdout:7/858: mkdir d16/d20/d54/de4/d2d/d30/d3e/d96/d132 0 2024-04-06T14:17:15.577 INFO:tasks.workunit.client.1.smithi086.stdout:3/592: mknod d5/d33/cba 0 2024-04-06T14:17:15.578 INFO:tasks.workunit.client.1.smithi086.stdout:5/489: symlink d0/l9a 0 2024-04-06T14:17:15.578 INFO:tasks.workunit.client.0.smithi032.stdout:1/884: rmdir d0/da/d3e 39 2024-04-06T14:17:15.579 INFO:tasks.workunit.client.1.smithi086.stdout:1/601: rename db/d11/d1f/f7c to db/d11/d42/fbf 0 2024-04-06T14:17:15.579 INFO:tasks.workunit.client.1.smithi086.stdout:8/654: chown d0/d71/d16/f41 2193676 1 2024-04-06T14:17:15.579 INFO:tasks.workunit.client.0.smithi032.stdout:4/832: getdents d0/df/d23/d31 0 2024-04-06T14:17:15.580 INFO:tasks.workunit.client.1.smithi086.stdout:6/681: creat d6/d13/d42/fea x:0 0 0 2024-04-06T14:17:15.581 INFO:tasks.workunit.client.1.smithi086.stdout:3/593: dread - d5/d33/d5d/fa2 zero size 2024-04-06T14:17:15.582 INFO:tasks.workunit.client.0.smithi032.stdout:4/833: chown d0/df/d23/d31/d50/d5e 582490883 1 2024-04-06T14:17:15.582 INFO:tasks.workunit.client.1.smithi086.stdout:8/655: fdatasync d0/f72 0 2024-04-06T14:17:15.583 INFO:tasks.workunit.client.1.smithi086.stdout:0/615: creat d0/d15/d26/d33/d7a/daa/fca x:0 0 0 2024-04-06T14:17:15.583 INFO:tasks.workunit.client.1.smithi086.stdout:7/555: dwrite d7/d10/f64 [0,4194304] 0 2024-04-06T14:17:15.586 INFO:tasks.workunit.client.0.smithi032.stdout:5/780: getdents d16/d18/d33/d35/d4d 0 2024-04-06T14:17:15.587 INFO:tasks.workunit.client.1.smithi086.stdout:6/682: fdatasync d6/d13/d1a/d22/dcf/db2/fd8 0 2024-04-06T14:17:15.587 INFO:tasks.workunit.client.1.smithi086.stdout:9/489: rename d5/d13/d1a/d5f/d88/l11 to d5/d13/d1a/d5f/la3 0 2024-04-06T14:17:15.588 INFO:tasks.workunit.client.0.smithi032.stdout:2/747: creat d0/d30/d39/d59/d67/ff7 x:0 0 0 2024-04-06T14:17:15.588 INFO:tasks.workunit.client.0.smithi032.stdout:1/885: dwrite d0/d41/f11b [0,4194304] 0 2024-04-06T14:17:15.592 INFO:tasks.workunit.client.1.smithi086.stdout:7/556: dread - d7/d10/d11/d1c/d24/d2c/fb6 zero size 2024-04-06T14:17:15.593 INFO:tasks.workunit.client.0.smithi032.stdout:4/834: symlink d0/d82/daf/l118 0 2024-04-06T14:17:15.597 INFO:tasks.workunit.client.0.smithi032.stdout:2/748: mkdir d0/d4e/df8 0 2024-04-06T14:17:15.597 INFO:tasks.workunit.client.0.smithi032.stdout:4/835: dread d0/df/d23/f60 [0,4194304] 0 2024-04-06T14:17:15.599 INFO:tasks.workunit.client.1.smithi086.stdout:2/568: dwrite d0/d20/d27/d2b/f65 [0,4194304] 0 2024-04-06T14:17:15.600 INFO:tasks.workunit.client.1.smithi086.stdout:6/683: rmdir d6/d9/db 39 2024-04-06T14:17:15.600 INFO:tasks.workunit.client.0.smithi032.stdout:7/859: getdents d16/d20/d7a/da6 0 2024-04-06T14:17:15.601 INFO:tasks.workunit.client.0.smithi032.stdout:5/781: creat d16/d18/d33/d45/f116 x:0 0 0 2024-04-06T14:17:15.602 INFO:tasks.workunit.client.1.smithi086.stdout:5/490: symlink d0/d16/d18/l9b 0 2024-04-06T14:17:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:1/886: mknod d0/da/d3e/d75/dc4/c11e 0 2024-04-06T14:17:15.603 INFO:tasks.workunit.client.1.smithi086.stdout:1/602: write db/d11/d1a/d2a/fb5 [793834,90113] 0 2024-04-06T14:17:15.603 INFO:tasks.workunit.client.0.smithi032.stdout:5/782: stat d16/d18/d33/d45/d52/d39/c66 0 2024-04-06T14:17:15.604 INFO:tasks.workunit.client.0.smithi032.stdout:8/885: dread d3/d8/f25 [8388608,4194304] 0 2024-04-06T14:17:15.606 INFO:tasks.workunit.client.0.smithi032.stdout:4/836: symlink d0/l119 0 2024-04-06T14:17:15.607 INFO:tasks.workunit.client.1.smithi086.stdout:8/656: truncate d0/d71/d22/f8e 1005723 0 2024-04-06T14:17:15.609 INFO:tasks.workunit.client.0.smithi032.stdout:7/860: creat d16/d20/d7a/da6/d117/f133 x:0 0 0 2024-04-06T14:17:15.611 INFO:tasks.workunit.client.1.smithi086.stdout:7/557: symlink d7/d10/d11/d1c/lba 0 2024-04-06T14:17:15.611 INFO:tasks.workunit.client.0.smithi032.stdout:4/837: rmdir d0 39 2024-04-06T14:17:15.612 INFO:tasks.workunit.client.0.smithi032.stdout:7/861: rmdir d16/d20/d48 39 2024-04-06T14:17:15.613 INFO:tasks.workunit.client.1.smithi086.stdout:9/490: link d5/d13/c46 d5/d13/d1a/d5f/d88/d14/d19/d25/ca4 0 2024-04-06T14:17:15.616 INFO:tasks.workunit.client.0.smithi032.stdout:8/886: rmdir d3/d8/d3a/d7b/db9/df8 39 2024-04-06T14:17:15.618 INFO:tasks.workunit.client.0.smithi032.stdout:4/838: fdatasync d0/df/d23/fac 0 2024-04-06T14:17:15.619 INFO:tasks.workunit.client.1.smithi086.stdout:8/657: dread d0/d71/f3c [0,4194304] 0 2024-04-06T14:17:15.620 INFO:tasks.workunit.client.0.smithi032.stdout:9/828: write d2/d15/d1e/d65/d83/d31/f3a [359369,65394] 0 2024-04-06T14:17:15.620 INFO:tasks.workunit.client.0.smithi032.stdout:8/887: mkdir d3/d8/d7/d8a/d12d 0 2024-04-06T14:17:15.620 INFO:tasks.workunit.client.0.smithi032.stdout:6/863: write d4/d5/d19/d25/d67/f56 [2473916,98569] 0 2024-04-06T14:17:15.620 INFO:tasks.workunit.client.1.smithi086.stdout:3/594: link d5/d26/d4f/lb9 d5/d36/d5a/d6d/d72/d57/lbb 0 2024-04-06T14:17:15.621 INFO:tasks.workunit.client.1.smithi086.stdout:2/569: write d0/d20/d27/d37/d52/f5c [2286979,103339] 0 2024-04-06T14:17:15.621 INFO:tasks.workunit.client.1.smithi086.stdout:7/558: rmdir d7/d10/d11/d1c/d24/d4d/daf/db5/d7a 39 2024-04-06T14:17:15.623 INFO:tasks.workunit.client.1.smithi086.stdout:3/595: fdatasync d5/d26/d4f/db4/da7/f63 0 2024-04-06T14:17:15.624 INFO:tasks.workunit.client.1.smithi086.stdout:9/491: rmdir d5/d13 39 2024-04-06T14:17:15.624 INFO:tasks.workunit.client.0.smithi032.stdout:5/783: getdents d16/d58/d54/d94/db2/dcb 0 2024-04-06T14:17:15.626 INFO:tasks.workunit.client.0.smithi032.stdout:4/839: dread d0/df/d1f/d42/f99 [0,4194304] 0 2024-04-06T14:17:15.627 INFO:tasks.workunit.client.1.smithi086.stdout:0/616: getdents d0/d1/d4/de 0 2024-04-06T14:17:15.627 INFO:tasks.workunit.client.1.smithi086.stdout:3/596: truncate d5/f8d 2262737 0 2024-04-06T14:17:15.628 INFO:tasks.workunit.client.0.smithi032.stdout:9/829: creat d2/d15/d1e/d41/d6c/d88/de6/f101 x:0 0 0 2024-04-06T14:17:15.628 INFO:tasks.workunit.client.0.smithi032.stdout:0/655: truncate d3/da/d25/d2b/d4a/f62 1663329 0 2024-04-06T14:17:15.629 INFO:tasks.workunit.client.0.smithi032.stdout:6/864: mknod d4/d5/d35/c123 0 2024-04-06T14:17:15.630 INFO:tasks.workunit.client.0.smithi032.stdout:8/888: creat d3/d8/d3a/d4b/de0/f12e x:0 0 0 2024-04-06T14:17:15.631 INFO:tasks.workunit.client.0.smithi032.stdout:8/889: chown d3/d8/d7 11106 1 2024-04-06T14:17:15.632 INFO:tasks.workunit.client.0.smithi032.stdout:8/890: chown d3/d8/d12/d2f/d46/def/d128/lb7 177 1 2024-04-06T14:17:15.633 INFO:tasks.workunit.client.1.smithi086.stdout:2/570: creat d0/d20/d27/d2b/d35/d36/d4d/d6f/fc6 x:0 0 0 2024-04-06T14:17:15.635 INFO:tasks.workunit.client.1.smithi086.stdout:1/603: link fa db/d11/d1a/d2a/d9a/d68/fc0 0 2024-04-06T14:17:15.636 INFO:tasks.workunit.client.1.smithi086.stdout:7/559: mknod d7/d10/d11/d1c/d24/d4d/d5c/cbb 0 2024-04-06T14:17:15.636 INFO:tasks.workunit.client.1.smithi086.stdout:9/492: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/fa5 x:0 0 0 2024-04-06T14:17:15.636 INFO:tasks.workunit.client.0.smithi032.stdout:9/830: symlink d2/d15/d1e/d41/d6c/l102 0 2024-04-06T14:17:15.638 INFO:tasks.workunit.client.1.smithi086.stdout:7/560: chown d7/d10/d11/d80/l5f 8442 1 2024-04-06T14:17:15.639 INFO:tasks.workunit.client.0.smithi032.stdout:6/865: fsync d4/d5/d8/d2f/fb2 0 2024-04-06T14:17:15.641 INFO:tasks.workunit.client.1.smithi086.stdout:3/597: mknod d5/d26/d7f/cbc 0 2024-04-06T14:17:15.642 INFO:tasks.workunit.client.1.smithi086.stdout:6/684: dread d6/d20/de0/f89 [0,4194304] 0 2024-04-06T14:17:15.642 INFO:tasks.workunit.client.0.smithi032.stdout:9/831: mknod d2/dcd/d7c/c103 0 2024-04-06T14:17:15.642 INFO:tasks.workunit.client.1.smithi086.stdout:5/491: dread d0/d19/f26 [0,4194304] 0 2024-04-06T14:17:15.643 INFO:tasks.workunit.client.1.smithi086.stdout:3/598: stat d5/d36/d7d 0 2024-04-06T14:17:15.644 INFO:tasks.workunit.client.1.smithi086.stdout:7/561: mkdir d7/d10/d11/d1c/d36/dbc 0 2024-04-06T14:17:15.645 INFO:tasks.workunit.client.0.smithi032.stdout:5/784: dread d16/d18/d33/f56 [0,4194304] 0 2024-04-06T14:17:15.646 INFO:tasks.workunit.client.1.smithi086.stdout:8/658: dwrite d0/d71/d16/d39/f66 [0,4194304] 0 2024-04-06T14:17:15.647 INFO:tasks.workunit.client.1.smithi086.stdout:1/604: truncate db/d11/d87/fb6 879756 0 2024-04-06T14:17:15.648 INFO:tasks.workunit.client.1.smithi086.stdout:3/599: creat d5/d36/d5a/d6d/d72/d64/fbd x:0 0 0 2024-04-06T14:17:15.648 INFO:tasks.workunit.client.0.smithi032.stdout:1/887: dwrite d0/d10/d16/d3f/d98/fe4 [0,4194304] 0 2024-04-06T14:17:15.649 INFO:tasks.workunit.client.0.smithi032.stdout:9/832: mkdir d2/d15/d1e/d41/d6c/d74/d104 0 2024-04-06T14:17:15.649 INFO:tasks.workunit.client.0.smithi032.stdout:5/785: write d16/d18/dae/dbb/ddc/d100/f107 [909049,74545] 0 2024-04-06T14:17:15.649 INFO:tasks.workunit.client.1.smithi086.stdout:9/493: read d5/d13/d1a/d5f/d88/d14/d19/d25/d59/f60 [2074106,9911] 0 2024-04-06T14:17:15.663 INFO:tasks.workunit.client.0.smithi032.stdout:5/786: rename d16/d18/d33/d35/d88/deb/l105 to d16/d18/d33/d35/d88/dbd/l117 0 2024-04-06T14:17:15.663 INFO:tasks.workunit.client.0.smithi032.stdout:6/866: link d4/d5/d28/d65/d6e/cfa d4/d5/d19/c124 0 2024-04-06T14:17:15.663 INFO:tasks.workunit.client.0.smithi032.stdout:0/656: dwrite d3/da/f56 [0,4194304] 0 2024-04-06T14:17:15.663 INFO:tasks.workunit.client.0.smithi032.stdout:1/888: link d0/d10/d16/d3f/d98/ca7 d0/da/d3e/d75/dee/c11f 0 2024-04-06T14:17:15.663 INFO:tasks.workunit.client.0.smithi032.stdout:5/787: readlink d16/d58/d54/d94/db2/dcb/l106 0 2024-04-06T14:17:15.663 INFO:tasks.workunit.client.0.smithi032.stdout:9/833: rename d2/d15/d55/d59/d93/fd4 to d2/d15/d1e/d41/d6c/d74/f105 0 2024-04-06T14:17:15.668 INFO:tasks.workunit.client.0.smithi032.stdout:6/867: rmdir d4/d5/d8/d2f/d9b/db4 39 2024-04-06T14:17:15.668 INFO:tasks.workunit.client.0.smithi032.stdout:0/657: write d3/da/d25/d2b/d4a/d79/d97/faf [1690425,65229] 0 2024-04-06T14:17:15.668 INFO:tasks.workunit.client.0.smithi032.stdout:1/889: symlink d0/da/d3e/d75/l120 0 2024-04-06T14:17:15.669 INFO:tasks.workunit.client.0.smithi032.stdout:1/890: stat d0/d10/d16/d118 0 2024-04-06T14:17:15.671 INFO:tasks.workunit.client.0.smithi032.stdout:9/834: mknod d2/d15/d1e/d65/d83/d7e/da7/c106 0 2024-04-06T14:17:15.672 INFO:tasks.workunit.client.0.smithi032.stdout:6/868: creat d4/d5/dc5/d82/f125 x:0 0 0 2024-04-06T14:17:15.676 INFO:tasks.workunit.client.0.smithi032.stdout:5/788: dwrite d16/d18/d33/d45/f10f [0,4194304] 0 2024-04-06T14:17:15.678 INFO:tasks.workunit.client.0.smithi032.stdout:6/869: creat d4/d5/dc5/d82/f126 x:0 0 0 2024-04-06T14:17:15.679 INFO:tasks.workunit.client.0.smithi032.stdout:9/835: dread d2/d15/d55/f6e [0,4194304] 0 2024-04-06T14:17:15.679 INFO:tasks.workunit.client.0.smithi032.stdout:5/789: rmdir d16/d58/d54/d74/d8f 39 2024-04-06T14:17:15.681 INFO:tasks.workunit.client.0.smithi032.stdout:0/658: getdents d3/dc2 0 2024-04-06T14:17:15.682 INFO:tasks.workunit.client.0.smithi032.stdout:1/891: write d0/da/f63 [66325,48397] 0 2024-04-06T14:17:15.683 INFO:tasks.workunit.client.0.smithi032.stdout:5/790: mknod d16/d18/d33/d45/d87/c118 0 2024-04-06T14:17:15.684 INFO:tasks.workunit.client.0.smithi032.stdout:9/836: mknod d2/d15/d1e/d41/c107 0 2024-04-06T14:17:15.684 INFO:tasks.workunit.client.0.smithi032.stdout:1/892: chown d0/d41/d59/f9a 8777 1 2024-04-06T14:17:15.686 INFO:tasks.workunit.client.0.smithi032.stdout:0/659: unlink d3/da/f2d 0 2024-04-06T14:17:15.688 INFO:tasks.workunit.client.0.smithi032.stdout:5/791: creat d16/d18/d33/d35/d88/dbd/dc5/f119 x:0 0 0 2024-04-06T14:17:15.688 INFO:tasks.workunit.client.0.smithi032.stdout:5/792: chown d16/d18/f68 2 1 2024-04-06T14:17:15.690 INFO:tasks.workunit.client.0.smithi032.stdout:9/837: rename d2/d15/d1e/d65/d83/d7e/f1b to d2/dcd/dd8/f108 0 2024-04-06T14:17:15.695 INFO:tasks.workunit.client.0.smithi032.stdout:0/660: truncate d3/da/d35/f4e 2112167 0 2024-04-06T14:17:15.695 INFO:tasks.workunit.client.0.smithi032.stdout:9/838: read d2/d15/d1e/f1f [183485,85094] 0 2024-04-06T14:17:15.696 INFO:tasks.workunit.client.0.smithi032.stdout:6/870: truncate d4/d5/d35/fce 741856 0 2024-04-06T14:17:15.697 INFO:tasks.workunit.client.0.smithi032.stdout:1/893: write d0/d9/f83 [2252330,96085] 0 2024-04-06T14:17:15.699 INFO:tasks.workunit.client.0.smithi032.stdout:5/793: write d16/d18/d33/f41 [553085,21132] 0 2024-04-06T14:17:15.700 INFO:tasks.workunit.client.0.smithi032.stdout:5/794: write f1 [5351337,940] 0 2024-04-06T14:17:15.701 INFO:tasks.workunit.client.0.smithi032.stdout:1/894: creat d0/d10/d16/d3f/d98/f121 x:0 0 0 2024-04-06T14:17:15.701 INFO:tasks.workunit.client.0.smithi032.stdout:9/839: link d2/cb4 d2/c109 0 2024-04-06T14:17:15.704 INFO:tasks.workunit.client.0.smithi032.stdout:9/840: fdatasync d2/d15/d55/f9a 0 2024-04-06T14:17:15.707 INFO:tasks.workunit.client.0.smithi032.stdout:1/895: dread d0/d41/f11b [0,4194304] 0 2024-04-06T14:17:15.709 INFO:tasks.workunit.client.0.smithi032.stdout:9/841: mkdir d2/d15/d1e/d65/d83/d31/d39/da6/d10a 0 2024-04-06T14:17:15.710 INFO:tasks.workunit.client.0.smithi032.stdout:0/661: truncate d3/da/d25/d2b/d4a/d79/da8/f9f 380074 0 2024-04-06T14:17:15.715 INFO:tasks.workunit.client.0.smithi032.stdout:1/896: truncate d0/d9/d1f/d7c/d8f/daa/fda 290957 0 2024-04-06T14:17:15.716 INFO:tasks.workunit.client.0.smithi032.stdout:1/897: rmdir d0/d9/d1f/d72 39 2024-04-06T14:17:15.718 INFO:tasks.workunit.client.0.smithi032.stdout:1/898: rmdir d0/dc3 39 2024-04-06T14:17:15.720 INFO:tasks.workunit.client.0.smithi032.stdout:6/871: dwrite d4/d5/d28/d54/f89 [0,4194304] 0 2024-04-06T14:17:15.721 INFO:tasks.workunit.client.0.smithi032.stdout:9/842: rmdir d2 39 2024-04-06T14:17:15.726 INFO:tasks.workunit.client.0.smithi032.stdout:6/872: rename d4/d5/d19/d96/f112 to d4/d5/d8/d2f/f127 0 2024-04-06T14:17:15.737 INFO:tasks.workunit.client.1.smithi086.stdout:6/685: creat d6/d13/d42/feb x:0 0 0 2024-04-06T14:17:15.738 INFO:tasks.workunit.client.1.smithi086.stdout:0/617: link d0/d15/d26/l2b d0/d1/d4/d7/lcb 0 2024-04-06T14:17:15.741 INFO:tasks.workunit.client.1.smithi086.stdout:7/562: creat d7/d10/d11/d1c/d36/dbc/fbd x:0 0 0 2024-04-06T14:17:15.741 INFO:tasks.workunit.client.1.smithi086.stdout:8/659: read - d0/d71/d16/d52/dd0/fb4 zero size 2024-04-06T14:17:15.743 INFO:tasks.workunit.client.1.smithi086.stdout:6/686: dread - d6/d13/d1a/d22/dcf/db2/fd8 zero size 2024-04-06T14:17:15.744 INFO:tasks.workunit.client.1.smithi086.stdout:2/571: dwrite d0/fe [0,4194304] 0 2024-04-06T14:17:15.745 INFO:tasks.workunit.client.1.smithi086.stdout:7/563: dread - d7/d10/d11/f8f zero size 2024-04-06T14:17:15.746 INFO:tasks.workunit.client.1.smithi086.stdout:0/618: unlink d0/d15/c41 0 2024-04-06T14:17:15.746 INFO:tasks.workunit.client.1.smithi086.stdout:3/600: write d5/df/f6b [4950308,79709] 0 2024-04-06T14:17:15.749 INFO:tasks.workunit.client.1.smithi086.stdout:9/494: dread d5/d13/d1a/d5f/d88/d14/d19/d1b/f45 [0,4194304] 0 2024-04-06T14:17:15.749 INFO:tasks.workunit.client.1.smithi086.stdout:2/572: creat d0/df/d5e/fc7 x:0 0 0 2024-04-06T14:17:15.751 INFO:tasks.workunit.client.1.smithi086.stdout:7/564: creat d7/d10/d11/fbe x:0 0 0 2024-04-06T14:17:15.753 INFO:tasks.workunit.client.1.smithi086.stdout:0/619: creat d0/d15/d4e/da3/d73/fcc x:0 0 0 2024-04-06T14:17:15.754 INFO:tasks.workunit.client.1.smithi086.stdout:3/601: creat d5/d36/d5a/d6d/d72/d97/fbe x:0 0 0 2024-04-06T14:17:15.755 INFO:tasks.workunit.client.1.smithi086.stdout:8/660: link d0/d71/d16/d52/d5f/l86 d0/d71/d22/d43/d6d/ldd 0 2024-04-06T14:17:15.756 INFO:tasks.workunit.client.1.smithi086.stdout:9/495: chown d5/d13/d1a/d5f/d88/d14/c89 954 1 2024-04-06T14:17:15.757 INFO:tasks.workunit.client.1.smithi086.stdout:1/605: dwrite db/d11/d1a/d2a/d9a/d68/d6e/f9e [0,4194304] 0 2024-04-06T14:17:15.757 INFO:tasks.workunit.client.1.smithi086.stdout:0/620: chown d0/d1/d4/f6b 51 1 2024-04-06T14:17:15.759 INFO:tasks.workunit.client.1.smithi086.stdout:9/496: read - d5/d13/d47/d5e/f93 zero size 2024-04-06T14:17:15.762 INFO:tasks.workunit.client.1.smithi086.stdout:9/497: chown d5/d13/f4f 21153 1 2024-04-06T14:17:15.762 INFO:tasks.workunit.client.1.smithi086.stdout:2/573: rename d0/df/d5e/d9d/fa7 to d0/d20/d44/d9f/fc8 0 2024-04-06T14:17:15.764 INFO:tasks.workunit.client.1.smithi086.stdout:1/606: stat db/d11/d1a/d2a/d9a/d78 0 2024-04-06T14:17:15.765 INFO:tasks.workunit.client.1.smithi086.stdout:3/602: creat d5/d26/d4f/db4/da7/d89/fbf x:0 0 0 2024-04-06T14:17:15.766 INFO:tasks.workunit.client.1.smithi086.stdout:2/574: fdatasync d0/df/d48/f7c 0 2024-04-06T14:17:15.768 INFO:tasks.workunit.client.1.smithi086.stdout:7/565: getdents d7/d10/d11/d1c 0 2024-04-06T14:17:15.768 INFO:tasks.workunit.client.1.smithi086.stdout:1/607: creat db/d11/fc1 x:0 0 0 2024-04-06T14:17:15.771 INFO:tasks.workunit.client.1.smithi086.stdout:6/687: dread d6/f3c [4194304,4194304] 0 2024-04-06T14:17:15.772 INFO:tasks.workunit.client.1.smithi086.stdout:2/575: mknod d0/d20/d27/d2b/d35/d36/d4d/da5/cc9 0 2024-04-06T14:17:15.772 INFO:tasks.workunit.client.1.smithi086.stdout:7/566: write d7/d10/d11/f91 [832563,19399] 0 2024-04-06T14:17:15.772 INFO:tasks.workunit.client.1.smithi086.stdout:1/608: dread db/f16 [0,4194304] 0 2024-04-06T14:17:15.775 INFO:tasks.workunit.client.1.smithi086.stdout:6/688: link d6/d13/d1a/da0/fda d6/d9/da9/fec 0 2024-04-06T14:17:15.776 INFO:tasks.workunit.client.1.smithi086.stdout:2/576: symlink d0/d20/d27/d2b/d35/d36/d4d/lca 0 2024-04-06T14:17:15.778 INFO:tasks.workunit.client.1.smithi086.stdout:7/567: link d7/d10/d11/d1c/d24/d4d/d5c/cbb d7/d10/d11/d1c/d24/d4d/daf/db5/da1/cbf 0 2024-04-06T14:17:15.780 INFO:tasks.workunit.client.1.smithi086.stdout:4/447: sync 2024-04-06T14:17:15.780 INFO:tasks.workunit.client.1.smithi086.stdout:6/689: getdents d6/d13/d5f 0 2024-04-06T14:17:15.782 INFO:tasks.workunit.client.1.smithi086.stdout:7/568: fdatasync d7/d10/d11/d1c/d24/fa2 0 2024-04-06T14:17:15.783 INFO:tasks.workunit.client.1.smithi086.stdout:6/690: creat d6/d9/d43/fed x:0 0 0 2024-04-06T14:17:15.784 INFO:tasks.workunit.client.1.smithi086.stdout:0/621: write d0/d15/d4e/f81 [1027568,118973] 0 2024-04-06T14:17:15.784 INFO:tasks.workunit.client.1.smithi086.stdout:8/661: write d0/d13/fb3 [1151238,49825] 0 2024-04-06T14:17:15.787 INFO:tasks.workunit.client.1.smithi086.stdout:3/603: write d5/df/f2c [3000636,77862] 0 2024-04-06T14:17:15.787 INFO:tasks.workunit.client.1.smithi086.stdout:7/569: unlink d7/d10/d11/d1c/d24/d2c/l6a 0 2024-04-06T14:17:15.788 INFO:tasks.workunit.client.1.smithi086.stdout:6/691: chown d6/d13/d42/l99 8 1 2024-04-06T14:17:15.791 INFO:tasks.workunit.client.1.smithi086.stdout:0/622: symlink d0/d1/d2/d5b/lcd 0 2024-04-06T14:17:15.791 INFO:tasks.workunit.client.1.smithi086.stdout:6/692: read d6/d13/d1a/d22/f9c [2698391,93865] 0 2024-04-06T14:17:15.792 INFO:tasks.workunit.client.1.smithi086.stdout:4/448: dwrite d0/f10 [0,4194304] 0 2024-04-06T14:17:15.792 INFO:tasks.workunit.client.1.smithi086.stdout:2/577: write d0/d20/d27/d37/f42 [2892315,107018] 0 2024-04-06T14:17:15.795 INFO:tasks.workunit.client.1.smithi086.stdout:8/662: creat d0/d71/d22/d65/fde x:0 0 0 2024-04-06T14:17:15.797 INFO:tasks.workunit.client.1.smithi086.stdout:0/623: creat d0/d15/fce x:0 0 0 2024-04-06T14:17:15.797 INFO:tasks.workunit.client.1.smithi086.stdout:2/578: fdatasync d0/df/d48/fb7 0 2024-04-06T14:17:15.798 INFO:tasks.workunit.client.1.smithi086.stdout:3/604: dread d5/d36/d5a/d6d/d72/f75 [0,4194304] 0 2024-04-06T14:17:15.799 INFO:tasks.workunit.client.1.smithi086.stdout:1/609: rename db/d11/d1d to db/d11/d1a/d2a/d9a/d68/daf/dc2 0 2024-04-06T14:17:15.800 INFO:tasks.workunit.client.1.smithi086.stdout:7/570: fdatasync d7/d10/d11/d1c/d24/d4d/daf/db5/d7a/f93 0 2024-04-06T14:17:15.802 INFO:tasks.workunit.client.1.smithi086.stdout:6/693: mkdir d6/d13/d1a/d22/dcf/db2/dee 0 2024-04-06T14:17:15.802 INFO:tasks.workunit.client.1.smithi086.stdout:4/449: mkdir d0/d2c/d30/d24/d5d/d8a 0 2024-04-06T14:17:15.803 INFO:tasks.workunit.client.1.smithi086.stdout:4/450: readlink d0/d15/l60 0 2024-04-06T14:17:15.803 INFO:tasks.workunit.client.1.smithi086.stdout:0/624: readlink d0/d1/d4/d7/lcb 0 2024-04-06T14:17:15.803 INFO:tasks.workunit.client.1.smithi086.stdout:6/694: read d6/d13/d5f/d60/f5b [1624446,93035] 0 2024-04-06T14:17:15.805 INFO:tasks.workunit.client.1.smithi086.stdout:3/605: mknod d5/d36/d5a/cc0 0 2024-04-06T14:17:15.806 INFO:tasks.workunit.client.1.smithi086.stdout:7/571: unlink d7/d10/d11/d1c/d36/l4c 0 2024-04-06T14:17:15.807 INFO:tasks.workunit.client.1.smithi086.stdout:8/663: rename d0/d71/d16/d39/c33 to d0/d71/d22/d43/d6d/d75/d78/cdf 0 2024-04-06T14:17:15.808 INFO:tasks.workunit.client.1.smithi086.stdout:6/695: creat d6/d13/d5f/d60/d45/d8a/fef x:0 0 0 2024-04-06T14:17:15.811 INFO:tasks.workunit.client.1.smithi086.stdout:0/625: truncate d0/d1/d4/d7/f27 692418 0 2024-04-06T14:17:15.813 INFO:tasks.workunit.client.1.smithi086.stdout:8/664: chown d0/d71/d16/c4e 78522978 1 2024-04-06T14:17:15.816 INFO:tasks.workunit.client.0.smithi032.stdout:1/899: symlink d0/dc9/l122 0 2024-04-06T14:17:15.818 INFO:tasks.workunit.client.0.smithi032.stdout:1/900: stat d0/d10/d78 0 2024-04-06T14:17:15.818 INFO:tasks.workunit.client.0.smithi032.stdout:9/843: mknod d2/d15/dd9/c10b 0 2024-04-06T14:17:15.819 INFO:tasks.workunit.client.1.smithi086.stdout:7/572: dwrite d7/d10/d11/d1c/d24/d2c/faa [0,4194304] 0 2024-04-06T14:17:15.819 INFO:tasks.workunit.client.0.smithi032.stdout:6/873: creat d4/d5/d19/d25/d9c/f128 x:0 0 0 2024-04-06T14:17:15.824 INFO:tasks.workunit.client.0.smithi032.stdout:9/844: fsync d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/fef 0 2024-04-06T14:17:15.824 INFO:tasks.workunit.client.1.smithi086.stdout:8/665: creat d0/d71/d93/fe0 x:0 0 0 2024-04-06T14:17:15.825 INFO:tasks.workunit.client.0.smithi032.stdout:1/901: creat d0/da/d3e/d75/dee/f123 x:0 0 0 2024-04-06T14:17:15.826 INFO:tasks.workunit.client.0.smithi032.stdout:6/874: truncate d4/d5/dc5/fa1 85053 0 2024-04-06T14:17:15.826 INFO:tasks.workunit.client.1.smithi086.stdout:1/610: fsync db/d11/d87/faa 0 2024-04-06T14:17:15.827 INFO:tasks.workunit.client.0.smithi032.stdout:0/662: dwrite d3/da/d25/d2b/d4a/fdb [0,4194304] 0 2024-04-06T14:17:15.831 INFO:tasks.workunit.client.1.smithi086.stdout:2/579: dwrite d0/d20/d27/d2b/d35/d36/d4d/d59/f8b [0,4194304] 0 2024-04-06T14:17:15.831 INFO:tasks.workunit.client.0.smithi032.stdout:9/845: stat d2/d15/c23 0 2024-04-06T14:17:15.831 INFO:tasks.workunit.client.0.smithi032.stdout:6/875: chown d4/d5/d8/c94 10960 1 2024-04-06T14:17:15.832 INFO:tasks.workunit.client.1.smithi086.stdout:7/573: unlink d7/d10/d11/d21/c5b 0 2024-04-06T14:17:15.836 INFO:tasks.workunit.client.1.smithi086.stdout:0/626: getdents d0/d15/d4e/da3/d46/d52 0 2024-04-06T14:17:15.837 INFO:tasks.workunit.client.0.smithi032.stdout:0/663: creat d3/dc2/dd8/fde x:0 0 0 2024-04-06T14:17:15.837 INFO:tasks.workunit.client.0.smithi032.stdout:6/876: dread - d4/d5/dc5/dcc/fe9 zero size 2024-04-06T14:17:15.841 INFO:tasks.workunit.client.1.smithi086.stdout:3/606: dread d5/d36/d5c/faa [0,4194304] 0 2024-04-06T14:17:15.841 INFO:tasks.workunit.client.0.smithi032.stdout:6/877: fsync d4/d5/d28/d65/ffc 0 2024-04-06T14:17:15.842 INFO:tasks.workunit.client.1.smithi086.stdout:6/696: dread d6/d9/fe8 [0,4194304] 0 2024-04-06T14:17:15.842 INFO:tasks.workunit.client.1.smithi086.stdout:0/627: mkdir d0/d1/d2/dcf 0 2024-04-06T14:17:15.843 INFO:tasks.workunit.client.0.smithi032.stdout:0/664: write d3/da/d35/fba [342325,120378] 0 2024-04-06T14:17:15.843 INFO:tasks.workunit.client.1.smithi086.stdout:7/574: getdents d7/d10/d11/d1c/d24/d4d 0 2024-04-06T14:17:15.844 INFO:tasks.workunit.client.1.smithi086.stdout:6/697: fsync d6/d13/d42/d52/f90 0 2024-04-06T14:17:15.845 INFO:tasks.workunit.client.1.smithi086.stdout:1/611: truncate db/d11/d1a/d2a/d9a/d68/fc0 1182971 0 2024-04-06T14:17:15.845 INFO:tasks.workunit.client.1.smithi086.stdout:0/628: chown d0/d15/d4e/da3/la4 12809 1 2024-04-06T14:17:15.846 INFO:tasks.workunit.client.0.smithi032.stdout:0/665: stat d3/da/d25/d36/lb0 0 2024-04-06T14:17:15.847 INFO:tasks.workunit.client.1.smithi086.stdout:2/580: dread d0/d20/d27/d2b/d35/d36/f40 [4194304,4194304] 0 2024-04-06T14:17:15.847 INFO:tasks.workunit.client.1.smithi086.stdout:3/607: mkdir d5/d36/d5c/d8e/dc1 0 2024-04-06T14:17:15.847 INFO:tasks.workunit.client.0.smithi032.stdout:6/878: truncate d4/d5/d19/d25/d67/f56 5020417 0 2024-04-06T14:17:15.848 INFO:tasks.workunit.client.0.smithi032.stdout:1/902: dwrite d0/f1e [0,4194304] 0 2024-04-06T14:17:15.850 INFO:tasks.workunit.client.0.smithi032.stdout:6/879: symlink d4/d5/d19/l129 0 2024-04-06T14:17:15.853 INFO:tasks.workunit.client.1.smithi086.stdout:7/575: fdatasync d7/d10/d11/d1c/d24/d4d/daf/db5/f6e 0 2024-04-06T14:17:15.854 INFO:tasks.workunit.client.0.smithi032.stdout:6/880: mknod d4/d5/d28/d65/d6e/ddb/c12a 0 2024-04-06T14:17:15.855 INFO:tasks.workunit.client.0.smithi032.stdout:1/903: mknod d0/d10/d16/db8/c124 0 2024-04-06T14:17:15.858 INFO:tasks.workunit.client.0.smithi032.stdout:9/846: dread d2/d15/d1e/d41/f63 [4194304,4194304] 0 2024-04-06T14:17:15.858 INFO:tasks.workunit.client.1.smithi086.stdout:8/666: dread d0/d71/d22/d65/d6e/f8f [0,4194304] 0 2024-04-06T14:17:15.859 INFO:tasks.workunit.client.1.smithi086.stdout:7/576: creat d7/d10/d11/d1c/d24/d2c/fc0 x:0 0 0 2024-04-06T14:17:15.860 INFO:tasks.workunit.client.0.smithi032.stdout:6/881: rename d4/d5/d19/d21/l62 to d4/d5/d8/d2b/dfb/l12b 0 2024-04-06T14:17:15.860 INFO:tasks.workunit.client.1.smithi086.stdout:6/698: write d6/d13/d1a/f21 [1675323,101114] 0 2024-04-06T14:17:15.861 INFO:tasks.workunit.client.1.smithi086.stdout:1/612: dread db/d11/d42/d53/f64 [0,4194304] 0 2024-04-06T14:17:15.862 INFO:tasks.workunit.client.0.smithi032.stdout:1/904: unlink d0/d41/d5b/d96/ccc 0 2024-04-06T14:17:15.864 INFO:tasks.workunit.client.0.smithi032.stdout:9/847: creat d2/dcd/d7c/dfa/f10c x:0 0 0 2024-04-06T14:17:15.864 INFO:tasks.workunit.client.1.smithi086.stdout:1/613: chown db/d11/d1a/d2a/f4c 24914813 1 2024-04-06T14:17:15.864 INFO:tasks.workunit.client.1.smithi086.stdout:3/608: creat d5/d26/d7f/fc2 x:0 0 0 2024-04-06T14:17:15.865 INFO:tasks.workunit.client.1.smithi086.stdout:6/699: dread - d6/d13/d5f/d60/d45/fa1 zero size 2024-04-06T14:17:15.871 INFO:tasks.workunit.client.1.smithi086.stdout:3/609: mkdir d5/dc3 0 2024-04-06T14:17:15.871 INFO:tasks.workunit.client.1.smithi086.stdout:6/700: chown d6/d9/d43/f74 61 1 2024-04-06T14:17:15.871 INFO:tasks.workunit.client.1.smithi086.stdout:1/614: mknod db/d11/d1a/d2a/d9a/d68/d6e/d81/cc3 0 2024-04-06T14:17:15.871 INFO:tasks.workunit.client.0.smithi032.stdout:1/905: mkdir d0/d9/d1f/d7c/d8f/dc5/d10c/d125 0 2024-04-06T14:17:15.871 INFO:tasks.workunit.client.0.smithi032.stdout:6/882: dwrite d4/d5/d8/d4b/f4a [0,4194304] 0 2024-04-06T14:17:15.871 INFO:tasks.workunit.client.0.smithi032.stdout:6/883: stat d4/d5/d8 0 2024-04-06T14:17:15.872 INFO:tasks.workunit.client.0.smithi032.stdout:6/884: truncate d4/d5/dbf/f105 208995 0 2024-04-06T14:17:15.878 INFO:tasks.workunit.client.1.smithi086.stdout:2/581: dread d0/f1e [0,4194304] 0 2024-04-06T14:17:15.878 INFO:tasks.workunit.client.1.smithi086.stdout:8/667: dwrite d0/d71/f3c [0,4194304] 0 2024-04-06T14:17:15.879 INFO:tasks.workunit.client.1.smithi086.stdout:2/582: stat d0/l31 0 2024-04-06T14:17:15.898 INFO:tasks.workunit.client.1.smithi086.stdout:3/610: symlink d5/d36/d5a/d6d/d72/d64/lc4 0 2024-04-06T14:17:15.899 INFO:tasks.workunit.client.1.smithi086.stdout:6/701: mknod d6/d13/d1a/d22/dcf/db2/cf0 0 2024-04-06T14:17:15.901 INFO:tasks.workunit.client.1.smithi086.stdout:6/702: read - d6/d9/d43/fc5 zero size 2024-04-06T14:17:15.901 INFO:tasks.workunit.client.1.smithi086.stdout:1/615: truncate db/d11/d1a/d79/f89 5182 0 2024-04-06T14:17:15.902 INFO:tasks.workunit.client.0.smithi032.stdout:1/906: write d0/d10/d16/f6d [3014590,76621] 0 2024-04-06T14:17:15.904 INFO:tasks.workunit.client.0.smithi032.stdout:6/885: write d4/d5/d28/f41 [3274980,88026] 0 2024-04-06T14:17:15.904 INFO:tasks.workunit.client.0.smithi032.stdout:1/907: chown d0/d9/df2 27 1 2024-04-06T14:17:15.906 INFO:tasks.workunit.client.1.smithi086.stdout:2/583: mkdir d0/d20/d27/d2b/d35/d36/dbe/dcb 0 2024-04-06T14:17:15.907 INFO:tasks.workunit.client.0.smithi032.stdout:6/886: rmdir d4/d5/d35 39 2024-04-06T14:17:15.907 INFO:tasks.workunit.client.1.smithi086.stdout:7/577: dread d7/d10/d11/d1c/f2e [0,4194304] 0 2024-04-06T14:17:15.909 INFO:tasks.workunit.client.1.smithi086.stdout:6/703: mkdir d6/d13/d42/d52/df1 0 2024-04-06T14:17:15.910 INFO:tasks.workunit.client.0.smithi032.stdout:6/887: readlink d4/d5/d19/d21/l38 0 2024-04-06T14:17:15.910 INFO:tasks.workunit.client.1.smithi086.stdout:6/704: write d6/d13/d42/feb [171781,90279] 0 2024-04-06T14:17:15.911 INFO:tasks.workunit.client.1.smithi086.stdout:1/616: stat db/d11/d1a/d79/da5/cbe 0 2024-04-06T14:17:15.911 INFO:tasks.workunit.client.1.smithi086.stdout:6/705: fdatasync d6/d13/f77 0 2024-04-06T14:17:15.915 INFO:tasks.workunit.client.0.smithi032.stdout:9/848: dread d2/d15/d55/d59/d93/d95/db8/d81/d77/f78 [0,4194304] 0 2024-04-06T14:17:15.915 INFO:tasks.workunit.client.0.smithi032.stdout:1/908: dwrite d0/d9/d94/fe3 [0,4194304] 0 2024-04-06T14:17:15.918 INFO:tasks.workunit.client.1.smithi086.stdout:8/668: truncate d0/db/d32/f3e 1149112 0 2024-04-06T14:17:15.919 INFO:tasks.workunit.client.1.smithi086.stdout:6/706: dwrite d6/d9/d43/fed [0,4194304] 0 2024-04-06T14:17:15.919 INFO:tasks.workunit.client.1.smithi086.stdout:1/617: dread - db/d11/d1a/d2a/d9a/d68/daf/dc2/f8a zero size 2024-04-06T14:17:15.920 INFO:tasks.workunit.client.0.smithi032.stdout:1/909: creat d0/d10/d16/db8/f126 x:0 0 0 2024-04-06T14:17:15.922 INFO:tasks.workunit.client.0.smithi032.stdout:6/888: dwrite d4/d5/dbf/fcd [0,4194304] 0 2024-04-06T14:17:15.926 INFO:tasks.workunit.client.1.smithi086.stdout:2/584: link d0/d20/d27/d2b/d35/d36/d4d/l57 d0/df/lcc 0 2024-04-06T14:17:15.926 INFO:tasks.workunit.client.0.smithi032.stdout:1/910: chown d0/d10/d16 311097002 1 2024-04-06T14:17:15.927 INFO:tasks.workunit.client.0.smithi032.stdout:6/889: dread - d4/d5/d28/d65/f92 zero size 2024-04-06T14:17:15.933 INFO:tasks.workunit.client.0.smithi032.stdout:9/849: dwrite d2/d15/d55/d59/f67 [0,4194304] 0 2024-04-06T14:17:15.933 INFO:tasks.workunit.client.0.smithi032.stdout:1/911: mkdir d0/d10/d16/d3f/d98/d127 0 2024-04-06T14:17:15.934 INFO:tasks.workunit.client.0.smithi032.stdout:6/890: mknod d4/d5/d28/d54/c12c 0 2024-04-06T14:17:15.935 INFO:tasks.workunit.client.1.smithi086.stdout:6/707: creat d6/d13/d42/d52/df1/ff2 x:0 0 0 2024-04-06T14:17:15.935 INFO:tasks.workunit.client.1.smithi086.stdout:1/618: creat db/d13/d55/da3/fc4 x:0 0 0 2024-04-06T14:17:15.935 INFO:tasks.workunit.client.1.smithi086.stdout:7/578: truncate d7/d10/d11/d1c/d24/d2c/f6f 3388966 0 2024-04-06T14:17:15.936 INFO:tasks.workunit.client.1.smithi086.stdout:2/585: read - d0/d20/d44/d9f/fc8 zero size 2024-04-06T14:17:15.940 INFO:tasks.workunit.client.0.smithi032.stdout:6/891: symlink d4/d5/dbf/l12d 0 2024-04-06T14:17:15.942 INFO:tasks.workunit.client.1.smithi086.stdout:6/708: truncate d6/d13/d42/f7b 697063 0 2024-04-06T14:17:15.944 INFO:tasks.workunit.client.1.smithi086.stdout:8/669: truncate d0/db/f12 2556167 0 2024-04-06T14:17:15.944 INFO:tasks.workunit.client.0.smithi032.stdout:6/892: write d4/d5/d8/d2b/ffd [909831,81633] 0 2024-04-06T14:17:15.944 INFO:tasks.workunit.client.1.smithi086.stdout:8/670: stat d0/d71/d16/d39/d90/fa0 0 2024-04-06T14:17:15.945 INFO:tasks.workunit.client.1.smithi086.stdout:2/586: mknod d0/d20/d27/dae/ccd 0 2024-04-06T14:17:15.946 INFO:tasks.workunit.client.1.smithi086.stdout:7/579: mknod d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/cc1 0 2024-04-06T14:17:15.947 INFO:tasks.workunit.client.0.smithi032.stdout:1/912: dwrite d0/d9/d1f/d7c/da9/ddd/f11a [0,4194304] 0 2024-04-06T14:17:15.950 INFO:tasks.workunit.client.1.smithi086.stdout:6/709: rename d6/d13/d42/d52/l91 to d6/d13/d1a/d22/dcf/db2/lf3 0 2024-04-06T14:17:15.951 INFO:tasks.workunit.client.0.smithi032.stdout:1/913: mkdir d0/d10/d78/d128 0 2024-04-06T14:17:15.951 INFO:tasks.workunit.client.1.smithi086.stdout:2/587: mkdir d0/df/d48/dce 0 2024-04-06T14:17:15.953 INFO:tasks.workunit.client.1.smithi086.stdout:8/671: symlink d0/d71/d22/d43/d4d/d62/le1 0 2024-04-06T14:17:15.954 INFO:tasks.workunit.client.1.smithi086.stdout:7/580: mknod d7/d10/d11/d80/cc2 0 2024-04-06T14:17:15.954 INFO:tasks.workunit.client.1.smithi086.stdout:2/588: unlink d0/d20/c22 0 2024-04-06T14:17:15.956 INFO:tasks.workunit.client.1.smithi086.stdout:1/619: creat db/d11/d1f/fc5 x:0 0 0 2024-04-06T14:17:15.957 INFO:tasks.workunit.client.1.smithi086.stdout:8/672: fdatasync d0/db/f5b 0 2024-04-06T14:17:15.957 INFO:tasks.workunit.client.0.smithi032.stdout:1/914: dwrite d0/d9/fbe [0,4194304] 0 2024-04-06T14:17:15.957 INFO:tasks.workunit.client.1.smithi086.stdout:1/620: write db/d11/d87/fb6 [1325332,68943] 0 2024-04-06T14:17:15.959 INFO:tasks.workunit.client.0.smithi032.stdout:6/893: truncate d4/d5/dc5/f50 149608 0 2024-04-06T14:17:15.959 INFO:tasks.workunit.client.1.smithi086.stdout:2/589: stat d0/df/la3 0 2024-04-06T14:17:15.963 INFO:tasks.workunit.client.0.smithi032.stdout:1/915: truncate d0/d41/fca 980366 0 2024-04-06T14:17:15.964 INFO:tasks.workunit.client.1.smithi086.stdout:7/581: rename d7/d10/d11/d21/f8e to d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/fc3 0 2024-04-06T14:17:15.965 INFO:tasks.workunit.client.1.smithi086.stdout:6/710: link d6/d9/d43/lb4 d6/d13/d5f/d60/d45/d8a/lf4 0 2024-04-06T14:17:15.967 INFO:tasks.workunit.client.0.smithi032.stdout:1/916: mkdir d0/d129 0 2024-04-06T14:17:15.967 INFO:tasks.workunit.client.1.smithi086.stdout:2/590: dwrite d0/fe [4194304,4194304] 0 2024-04-06T14:17:15.968 INFO:tasks.workunit.client.1.smithi086.stdout:8/673: creat d0/d71/d22/d43/d4d/da3/fe2 x:0 0 0 2024-04-06T14:17:15.970 INFO:tasks.workunit.client.1.smithi086.stdout:2/591: read - d0/d20/f61 zero size 2024-04-06T14:17:15.970 INFO:tasks.workunit.client.1.smithi086.stdout:6/711: truncate d6/d9/fe8 4776388 0 2024-04-06T14:17:15.971 INFO:tasks.workunit.client.0.smithi032.stdout:1/917: rmdir d0/d10/d16/d3f 39 2024-04-06T14:17:15.975 INFO:tasks.workunit.client.0.smithi032.stdout:1/918: rename d0/d9/d1f/d7c/d8f/dc5/d10c to d0/d9/d1f/d7c/d8f/dc5/d10c/d113/d12a 22 2024-04-06T14:17:15.977 INFO:tasks.workunit.client.1.smithi086.stdout:7/582: mknod d7/d10/d11/cc4 0 2024-04-06T14:17:15.979 INFO:tasks.workunit.client.0.smithi032.stdout:1/919: mkdir d0/dc3/d109/d12b 0 2024-04-06T14:17:15.979 INFO:tasks.workunit.client.1.smithi086.stdout:6/712: dwrite d6/d9/f17 [0,4194304] 0 2024-04-06T14:17:15.980 INFO:tasks.workunit.client.0.smithi032.stdout:1/920: chown d0/d9/f24 89831538 1 2024-04-06T14:17:15.981 INFO:tasks.workunit.client.1.smithi086.stdout:8/674: creat d0/d71/d16/d39/d90/daf/fe3 x:0 0 0 2024-04-06T14:17:15.984 INFO:tasks.workunit.client.1.smithi086.stdout:2/592: creat d0/d20/d27/d2b/d35/d36/d4d/d59/fcf x:0 0 0 2024-04-06T14:17:15.986 INFO:tasks.workunit.client.0.smithi032.stdout:1/921: rename d0/d10/cab to d0/d10/d16/db8/c12c 0 2024-04-06T14:17:15.987 INFO:tasks.workunit.client.1.smithi086.stdout:1/621: dread db/d11/f12 [0,4194304] 0 2024-04-06T14:17:15.988 INFO:tasks.workunit.client.1.smithi086.stdout:7/583: rmdir d7/d10/d11/d1c/d24 39 2024-04-06T14:17:15.989 INFO:tasks.workunit.client.1.smithi086.stdout:6/713: creat d6/d13/d1a/d22/ff5 x:0 0 0 2024-04-06T14:17:15.990 INFO:tasks.workunit.client.1.smithi086.stdout:8/675: creat d0/d13/db6/fe4 x:0 0 0 2024-04-06T14:17:15.991 INFO:tasks.workunit.client.1.smithi086.stdout:1/622: chown db/d11/l37 1 1 2024-04-06T14:17:15.991 INFO:tasks.workunit.client.1.smithi086.stdout:5/492: sync 2024-04-06T14:17:15.991 INFO:tasks.workunit.client.1.smithi086.stdout:8/676: dread - d0/d71/d22/d43/d4d/da3/fe2 zero size 2024-04-06T14:17:15.992 INFO:tasks.workunit.client.1.smithi086.stdout:1/623: chown db/d11/d1a/d2a/f47 1959807 1 2024-04-06T14:17:15.997 INFO:tasks.workunit.client.1.smithi086.stdout:8/677: symlink d0/d3b/le5 0 2024-04-06T14:17:15.998 INFO:tasks.workunit.client.1.smithi086.stdout:2/593: getdents d0/d20/d27/d2b/d35/d36/d4d/d59/d94 0 2024-04-06T14:17:15.999 INFO:tasks.workunit.client.1.smithi086.stdout:1/624: rename db/d11/d1a/d2a/d9a/d68/daf/dc2/c4f to db/d11/d1a/d2a/d9a/d68/d6f/cc6 0 2024-04-06T14:17:16.000 INFO:tasks.workunit.client.0.smithi032.stdout:1/922: truncate d0/d41/d59/fd4 1260016 0 2024-04-06T14:17:16.001 INFO:tasks.workunit.client.1.smithi086.stdout:8/678: creat d0/db/d32/dd6/fe6 x:0 0 0 2024-04-06T14:17:16.001 INFO:tasks.workunit.client.0.smithi032.stdout:1/923: chown d0/d9/d1f/d7c/lbb 494 1 2024-04-06T14:17:16.002 INFO:tasks.workunit.client.0.smithi032.stdout:1/924: mknod d0/d10/d66/c12d 0 2024-04-06T14:17:16.004 INFO:tasks.workunit.client.1.smithi086.stdout:7/584: rename d7/d10/d11/d1c/d24/d4d/f86 to d7/d10/d92/fc5 0 2024-04-06T14:17:16.005 INFO:tasks.workunit.client.1.smithi086.stdout:1/625: mkdir db/d11/d1a/d2a/d9a/d68/daf/dc7 0 2024-04-06T14:17:16.005 INFO:tasks.workunit.client.0.smithi032.stdout:1/925: mknod d0/d10/d16/d3f/d98/c12e 0 2024-04-06T14:17:16.008 INFO:tasks.workunit.client.0.smithi032.stdout:1/926: creat d0/da/d3e/f12f x:0 0 0 2024-04-06T14:17:16.010 INFO:tasks.workunit.client.1.smithi086.stdout:2/594: write d0/d20/d27/d2b/d35/d36/fa1 [1908083,18026] 0 2024-04-06T14:17:16.011 INFO:tasks.workunit.client.0.smithi032.stdout:1/927: truncate d0/da/d3e/f6c 2374501 0 2024-04-06T14:17:16.011 INFO:tasks.workunit.client.0.smithi032.stdout:1/928: stat d0/d9/f83 0 2024-04-06T14:17:16.012 INFO:tasks.workunit.client.0.smithi032.stdout:1/929: readlink d0/d10/l39 0 2024-04-06T14:17:16.014 INFO:tasks.workunit.client.1.smithi086.stdout:5/493: dwrite d0/d16/d71/f76 [0,4194304] 0 2024-04-06T14:17:16.015 INFO:tasks.workunit.client.0.smithi032.stdout:1/930: creat d0/f130 x:0 0 0 2024-04-06T14:17:16.015 INFO:tasks.workunit.client.1.smithi086.stdout:8/679: write d0/d3b/f46 [1592715,84398] 0 2024-04-06T14:17:16.017 INFO:tasks.workunit.client.0.smithi032.stdout:1/931: getdents d0/d9/df2/d102 0 2024-04-06T14:17:16.019 INFO:tasks.workunit.client.1.smithi086.stdout:7/585: mknod d7/d10/d11/d1c/d24/d4d/d5c/d66/cc6 0 2024-04-06T14:17:16.021 INFO:tasks.workunit.client.1.smithi086.stdout:1/626: dwrite db/d11/d24/fba [0,4194304] 0 2024-04-06T14:17:16.028 INFO:tasks.workunit.client.1.smithi086.stdout:5/494: dread d0/d16/d18/d35/f5b [0,4194304] 0 2024-04-06T14:17:16.033 INFO:tasks.workunit.client.1.smithi086.stdout:2/595: dread d0/d20/d27/d2b/d35/d36/f92 [0,4194304] 0 2024-04-06T14:17:16.068 INFO:tasks.workunit.client.1.smithi086.stdout:9/498: sync 2024-04-06T14:17:16.069 INFO:tasks.workunit.client.1.smithi086.stdout:0/629: sync 2024-04-06T14:17:16.069 INFO:tasks.workunit.client.1.smithi086.stdout:3/611: sync 2024-04-06T14:17:16.069 INFO:tasks.workunit.client.1.smithi086.stdout:4/451: sync 2024-04-06T14:17:16.070 INFO:tasks.workunit.client.1.smithi086.stdout:0/630: chown d0/d1/d2/d1a/f94 1 1 2024-04-06T14:17:16.070 INFO:tasks.workunit.client.1.smithi086.stdout:0/631: dread - d0/d1/d2/d1a/d79/f8c zero size 2024-04-06T14:17:16.097 INFO:tasks.workunit.client.1.smithi086.stdout:8/680: rename d0/db/d32 to d0/d71/d16/d52/dd0/ddc/de7 0 2024-04-06T14:17:16.099 INFO:tasks.workunit.client.1.smithi086.stdout:7/586: write d7/d10/d11/d1c/d24/d4d/daf/db5/f8a [2522326,86579] 0 2024-04-06T14:17:16.100 INFO:tasks.workunit.client.1.smithi086.stdout:1/627: creat db/d11/d1a/d2a/d9a/d78/fc8 x:0 0 0 2024-04-06T14:17:16.102 INFO:tasks.workunit.client.1.smithi086.stdout:5/495: readlink d0/d16/d18/d21/d23/d28/d94/l31 0 2024-04-06T14:17:16.102 INFO:tasks.workunit.client.1.smithi086.stdout:1/628: read - db/d11/d1f/fbc zero size 2024-04-06T14:17:16.109 INFO:tasks.workunit.client.1.smithi086.stdout:6/714: sync 2024-04-06T14:17:16.109 INFO:tasks.workunit.client.0.smithi032.stdout:3/774: sync 2024-04-06T14:17:16.111 INFO:tasks.workunit.client.0.smithi032.stdout:3/775: dread - d3/d65/d7d/d9c/dad/db1/d3c/f96 zero size 2024-04-06T14:17:16.111 INFO:tasks.workunit.client.1.smithi086.stdout:2/596: fsync d0/d20/d27/d2b/d35/d36/d4d/d6f/f81 0 2024-04-06T14:17:16.113 INFO:tasks.workunit.client.0.smithi032.stdout:3/776: chown d3/d16/cff 24 1 2024-04-06T14:17:16.114 INFO:tasks.workunit.client.1.smithi086.stdout:4/452: readlink d0/d2c/d38/l45 0 2024-04-06T14:17:16.115 INFO:tasks.workunit.client.1.smithi086.stdout:9/499: creat d5/d13/d1a/d5f/d88/d14/d19/fa6 x:0 0 0 2024-04-06T14:17:16.119 INFO:tasks.workunit.client.1.smithi086.stdout:0/632: rename d0/d1/d25 to d0/d15/d4e/da3/d46/d52/dd0 0 2024-04-06T14:17:16.120 INFO:tasks.workunit.client.1.smithi086.stdout:8/681: truncate d0/d71/d16/d52/dd0/fb4 94781 0 2024-04-06T14:17:16.121 INFO:tasks.workunit.client.1.smithi086.stdout:9/500: dwrite f0 [0,4194304] 0 2024-04-06T14:17:16.122 INFO:tasks.workunit.client.1.smithi086.stdout:9/501: chown d5/d13/d1a/d5f/d88/d14/f4e 105447329 1 2024-04-06T14:17:16.123 INFO:tasks.workunit.client.1.smithi086.stdout:6/715: fsync d6/d13/f5d 0 2024-04-06T14:17:16.125 INFO:tasks.workunit.client.0.smithi032.stdout:3/777: dread d3/d16/f9e [0,4194304] 0 2024-04-06T14:17:16.125 INFO:tasks.workunit.client.1.smithi086.stdout:2/597: symlink d0/d20/d27/d2b/d35/d36/d4a/ld0 0 2024-04-06T14:17:16.125 INFO:tasks.workunit.client.1.smithi086.stdout:4/453: mknod d0/d15/d19/c8b 0 2024-04-06T14:17:16.127 INFO:tasks.workunit.client.1.smithi086.stdout:0/633: mknod d0/d15/d4e/da3/cd1 0 2024-04-06T14:17:16.128 INFO:tasks.workunit.client.1.smithi086.stdout:7/587: mknod d7/d10/d11/d1c/d24/cc7 0 2024-04-06T14:17:16.130 INFO:tasks.workunit.client.0.smithi032.stdout:3/778: rename d3/d16/d63/d11d to d3/d16/ddd/d12d 0 2024-04-06T14:17:16.130 INFO:tasks.workunit.client.1.smithi086.stdout:5/496: creat d0/d43/f9c x:0 0 0 2024-04-06T14:17:16.133 INFO:tasks.workunit.client.0.smithi032.stdout:3/779: mkdir d3/d12e 0 2024-04-06T14:17:16.134 INFO:tasks.workunit.client.1.smithi086.stdout:3/612: dwrite d5/df/f85 [0,4194304] 0 2024-04-06T14:17:16.135 INFO:tasks.workunit.client.1.smithi086.stdout:8/682: creat d0/d71/d22/d43/daa/fe8 x:0 0 0 2024-04-06T14:17:16.136 INFO:tasks.workunit.client.0.smithi032.stdout:3/780: creat d3/d65/d7d/d9c/dad/dc5/f12f x:0 0 0 2024-04-06T14:17:16.139 INFO:tasks.workunit.client.1.smithi086.stdout:2/598: fdatasync d0/d20/d27/d2b/d35/d36/fa2 0 2024-04-06T14:17:16.139 INFO:tasks.workunit.client.0.smithi032.stdout:3/781: creat d3/d65/d7d/d9c/dbc/d128/f130 x:0 0 0 2024-04-06T14:17:16.140 INFO:tasks.workunit.client.1.smithi086.stdout:4/454: creat d0/d2c/d38/d54/f8c x:0 0 0 2024-04-06T14:17:16.141 INFO:tasks.workunit.client.1.smithi086.stdout:1/629: write db/d11/d1a/f63 [463426,23621] 0 2024-04-06T14:17:16.142 INFO:tasks.workunit.client.1.smithi086.stdout:4/455: read d0/f40 [90370,3884] 0 2024-04-06T14:17:16.142 INFO:tasks.workunit.client.0.smithi032.stdout:3/782: stat d3/d65/d7d/d9c/dad/db1/c53 0 2024-04-06T14:17:16.142 INFO:tasks.workunit.client.1.smithi086.stdout:0/634: fdatasync d0/d1/d2/d1a/f63 0 2024-04-06T14:17:16.144 INFO:tasks.workunit.client.1.smithi086.stdout:8/683: mknod d0/d71/d22/d65/d6e/ce9 0 2024-04-06T14:17:16.149 INFO:tasks.workunit.client.1.smithi086.stdout:5/497: creat d0/d16/d18/f9d x:0 0 0 2024-04-06T14:17:16.156 INFO:tasks.workunit.client.1.smithi086.stdout:6/716: getdents d6/d13 0 2024-04-06T14:17:16.157 INFO:tasks.workunit.client.1.smithi086.stdout:7/588: rename d7/d10/d11/d1c/d24/d4d/daf/db5/d7a/d8c to d7/d10/d11/d1c/d24/dc8 0 2024-04-06T14:17:16.157 INFO:tasks.workunit.client.1.smithi086.stdout:4/456: dread d0/de/f62 [0,4194304] 0 2024-04-06T14:17:16.159 INFO:tasks.workunit.client.1.smithi086.stdout:8/684: symlink d0/d71/d22/d43/d6d/d75/db2/lea 0 2024-04-06T14:17:16.161 INFO:tasks.workunit.client.1.smithi086.stdout:9/502: dwrite d5/d13/d1a/d5f/d88/d14/f4e [0,4194304] 0 2024-04-06T14:17:16.162 INFO:tasks.workunit.client.0.smithi032.stdout:3/783: dwrite d3/d16/f35 [0,4194304] 0 2024-04-06T14:17:16.164 INFO:tasks.workunit.client.1.smithi086.stdout:3/613: dread d5/d26/d4f/db4/da7/d89/f8f [0,4194304] 0 2024-04-06T14:17:16.165 INFO:tasks.workunit.client.1.smithi086.stdout:1/630: dwrite db/d11/d1a/d2a/d9a/d68/d6e/f9e [0,4194304] 0 2024-04-06T14:17:16.167 INFO:tasks.workunit.client.1.smithi086.stdout:5/498: mkdir d0/d16/d18/d35/d9e 0 2024-04-06T14:17:16.167 INFO:tasks.workunit.client.1.smithi086.stdout:0/635: mkdir d0/d15/d4e/da3/d44/d8e/dd2 0 2024-04-06T14:17:16.167 INFO:tasks.workunit.client.0.smithi032.stdout:3/784: truncate d3/d65/d7d/d9c/dad/db1/d3c/f85 2105356 0 2024-04-06T14:17:16.168 INFO:tasks.workunit.client.1.smithi086.stdout:4/457: mkdir d0/d15/d20/d74/d8d 0 2024-04-06T14:17:16.172 INFO:tasks.workunit.client.1.smithi086.stdout:8/685: symlink d0/db/leb 0 2024-04-06T14:17:16.181 INFO:tasks.workunit.client.1.smithi086.stdout:2/599: dwrite d0/f8e [0,4194304] 0 2024-04-06T14:17:16.182 INFO:tasks.workunit.client.0.smithi032.stdout:3/785: dread d3/d65/d7d/d9c/dad/dc5/fcc [0,4194304] 0 2024-04-06T14:17:16.183 INFO:tasks.workunit.client.1.smithi086.stdout:4/458: symlink d0/de/d4c/l8e 0 2024-04-06T14:17:16.183 INFO:tasks.workunit.client.1.smithi086.stdout:9/503: write d5/d18/f3f [2093371,63427] 0 2024-04-06T14:17:16.184 INFO:tasks.workunit.client.1.smithi086.stdout:7/589: dwrite d7/d10/d11/fbe [0,4194304] 0 2024-04-06T14:17:16.186 INFO:tasks.workunit.client.1.smithi086.stdout:0/636: rename d0/d1/d2/d5b/fb1 to d0/d1/fd3 0 2024-04-06T14:17:16.187 INFO:tasks.workunit.client.1.smithi086.stdout:4/459: write d0/d2c/d30/f6c [368613,4816] 0 2024-04-06T14:17:16.187 INFO:tasks.workunit.client.1.smithi086.stdout:1/631: mknod db/d11/d1a/d2a/d9a/d82/cc9 0 2024-04-06T14:17:16.188 INFO:tasks.workunit.client.0.smithi032.stdout:3/786: creat d3/d65/d7d/d9c/dad/db1/f131 x:0 0 0 2024-04-06T14:17:16.189 INFO:tasks.workunit.client.0.smithi032.stdout:3/787: fdatasync d3/d65/d7d/dc6/fdb 0 2024-04-06T14:17:16.191 INFO:tasks.workunit.client.1.smithi086.stdout:4/460: read d0/d2c/d30/f3b [371082,45333] 0 2024-04-06T14:17:16.193 INFO:tasks.workunit.client.0.smithi032.stdout:3/788: symlink d3/d65/d89/l132 0 2024-04-06T14:17:16.195 INFO:tasks.workunit.client.1.smithi086.stdout:0/637: rmdir d0/d1/d2 39 2024-04-06T14:17:16.197 INFO:tasks.workunit.client.1.smithi086.stdout:6/717: dread d6/d13/d42/f72 [0,4194304] 0 2024-04-06T14:17:16.197 INFO:tasks.workunit.client.0.smithi032.stdout:3/789: mknod d3/d16/d63/d129/c133 0 2024-04-06T14:17:16.197 INFO:tasks.workunit.client.1.smithi086.stdout:0/638: dread - d0/d15/d26/d33/d9f/fab zero size 2024-04-06T14:17:16.198 INFO:tasks.workunit.client.1.smithi086.stdout:5/499: dread d0/f3 [0,4194304] 0 2024-04-06T14:17:16.200 INFO:tasks.workunit.client.1.smithi086.stdout:3/614: getdents d5/d36/d65 0 2024-04-06T14:17:16.201 INFO:tasks.workunit.client.0.smithi032.stdout:3/790: dread d3/d65/d7d/d9c/dad/db1/f54 [0,4194304] 0 2024-04-06T14:17:16.201 INFO:tasks.workunit.client.1.smithi086.stdout:0/639: unlink d0/d15/d26/d33/f87 0 2024-04-06T14:17:16.202 INFO:tasks.workunit.client.1.smithi086.stdout:5/500: readlink d0/d16/d18/d21/d23/d28/l59 0 2024-04-06T14:17:16.202 INFO:tasks.workunit.client.0.smithi032.stdout:3/791: fsync d3/d65/f106 0 2024-04-06T14:17:16.203 INFO:tasks.workunit.client.1.smithi086.stdout:1/632: creat db/fca x:0 0 0 2024-04-06T14:17:16.203 INFO:tasks.workunit.client.1.smithi086.stdout:6/718: chown d6/d13/d1a/d64/d85/l8d 3 1 2024-04-06T14:17:16.204 INFO:tasks.workunit.client.1.smithi086.stdout:5/501: chown d0/d16/c6d 340 1 2024-04-06T14:17:16.204 INFO:tasks.workunit.client.0.smithi032.stdout:3/792: creat d3/d65/d7d/d9c/db8/f134 x:0 0 0 2024-04-06T14:17:16.204 INFO:tasks.workunit.client.1.smithi086.stdout:0/640: symlink d0/d1/d4/d7/ld4 0 2024-04-06T14:17:16.206 INFO:tasks.workunit.client.1.smithi086.stdout:3/615: link d5/d36/d5a/d6d/d72/faf d5/d26/d7f/d59/fc5 0 2024-04-06T14:17:16.207 INFO:tasks.workunit.client.1.smithi086.stdout:1/633: truncate db/d11/d1a/d2a/d9a/d68/f7a 4382677 0 2024-04-06T14:17:16.207 INFO:tasks.workunit.client.0.smithi032.stdout:3/793: mknod d3/d65/d7d/d9c/dbc/c135 0 2024-04-06T14:17:16.208 INFO:tasks.workunit.client.1.smithi086.stdout:6/719: readlink d6/d13/d5f/d60/d45/d8a/lf4 0 2024-04-06T14:17:16.209 INFO:tasks.workunit.client.1.smithi086.stdout:3/616: fdatasync d5/f3e 0 2024-04-06T14:17:16.211 INFO:tasks.workunit.client.1.smithi086.stdout:6/720: readlink d6/d9/da9/lc0 0 2024-04-06T14:17:16.212 INFO:tasks.workunit.client.0.smithi032.stdout:3/794: symlink d3/d65/d7d/d9c/dad/db1/d3c/dc9/d104/l136 0 2024-04-06T14:17:16.213 INFO:tasks.workunit.client.0.smithi032.stdout:3/795: stat d3/d65/d7d/d9c/dbc/fe3 0 2024-04-06T14:17:16.214 INFO:tasks.workunit.client.1.smithi086.stdout:6/721: chown d6/d13/d42/d52/f90 1327 1 2024-04-06T14:17:16.215 INFO:tasks.workunit.client.1.smithi086.stdout:8/686: write d0/d71/d22/d43/d6d/f80 [2717698,16366] 0 2024-04-06T14:17:16.215 INFO:tasks.workunit.client.1.smithi086.stdout:7/590: truncate d7/d10/d92/f9f 100248 0 2024-04-06T14:17:16.217 INFO:tasks.workunit.client.1.smithi086.stdout:0/641: rename d0/d1/d2/d1a/d51/f60 to d0/d15/d4e/da3/d44/d8e/dd2/fd5 0 2024-04-06T14:17:16.218 INFO:tasks.workunit.client.0.smithi032.stdout:3/796: write d3/d16/f70 [1650789,100391] 0 2024-04-06T14:17:16.221 INFO:tasks.workunit.client.1.smithi086.stdout:3/617: symlink d5/d36/d5a/d6d/d72/d57/lc6 0 2024-04-06T14:17:16.223 INFO:tasks.workunit.client.1.smithi086.stdout:9/504: dread d5/d13/d1a/d5f/d88/d14/d19/d1b/f41 [0,4194304] 0 2024-04-06T14:17:16.224 INFO:tasks.workunit.client.0.smithi032.stdout:3/797: rename d3/d16/f9e to d3/d65/d7d/d9c/dad/db1/f137 0 2024-04-06T14:17:16.225 INFO:tasks.workunit.client.1.smithi086.stdout:3/618: write d5/d33/f78 [1820087,40031] 0 2024-04-06T14:17:16.226 INFO:tasks.workunit.client.1.smithi086.stdout:5/502: link d0/d16/d18/d2a/d50/f5f d0/d16/d18/d21/d23/d28/d94/d80/d85/f9f 0 2024-04-06T14:17:16.227 INFO:tasks.workunit.client.0.smithi032.stdout:3/798: creat d3/d12e/f138 x:0 0 0 2024-04-06T14:17:16.227 INFO:tasks.workunit.client.1.smithi086.stdout:2/600: dwrite d0/df/d48/d53/f54 [0,4194304] 0 2024-04-06T14:17:16.227 INFO:tasks.workunit.client.1.smithi086.stdout:0/642: fsync d0/d1/d4/f8 0 2024-04-06T14:17:16.229 INFO:tasks.workunit.client.1.smithi086.stdout:1/634: write db/d11/d42/fbf [733929,38304] 0 2024-04-06T14:17:16.232 INFO:tasks.workunit.client.1.smithi086.stdout:4/461: dread d0/f36 [0,4194304] 0 2024-04-06T14:17:16.233 INFO:tasks.workunit.client.0.smithi032.stdout:2/749: sync 2024-04-06T14:17:16.233 INFO:tasks.workunit.client.0.smithi032.stdout:7/862: sync 2024-04-06T14:17:16.236 INFO:tasks.workunit.client.1.smithi086.stdout:3/619: creat d5/df/fc7 x:0 0 0 2024-04-06T14:17:16.236 INFO:tasks.workunit.client.0.smithi032.stdout:3/799: dwrite d3/d16/ddd/f10b [0,4194304] 0 2024-04-06T14:17:16.237 INFO:tasks.workunit.client.1.smithi086.stdout:2/601: truncate d0/df/d5e/fc7 160074 0 2024-04-06T14:17:16.238 INFO:tasks.workunit.client.0.smithi032.stdout:3/800: chown d3/d65/f32 29369948 1 2024-04-06T14:17:16.240 INFO:tasks.workunit.client.1.smithi086.stdout:1/635: fdatasync db/fe 0 2024-04-06T14:17:16.240 INFO:tasks.workunit.client.1.smithi086.stdout:3/620: dread - d5/d33/d5d/fa2 zero size 2024-04-06T14:17:16.240 INFO:tasks.workunit.client.0.smithi032.stdout:2/750: mkdir d0/d6e/dae/db7/df9 0 2024-04-06T14:17:16.241 INFO:tasks.workunit.client.0.smithi032.stdout:7/863: dread d16/f49 [0,4194304] 0 2024-04-06T14:17:16.242 INFO:tasks.workunit.client.1.smithi086.stdout:0/643: readlink d0/d15/d4e/da3/d46/l4d 0 2024-04-06T14:17:16.244 INFO:tasks.workunit.client.1.smithi086.stdout:6/722: dwrite d6/d13/d1a/d22/dcf/db2/fd8 [0,4194304] 0 2024-04-06T14:17:16.246 INFO:tasks.workunit.client.1.smithi086.stdout:7/591: getdents d7/d10/d11/d1c/d24/d2c 0 2024-04-06T14:17:16.247 INFO:tasks.workunit.client.1.smithi086.stdout:9/505: chown d5/d13/d1a/d5f/d88/d14/d19/d25/l2e 3 1 2024-04-06T14:17:16.248 INFO:tasks.workunit.client.0.smithi032.stdout:3/801: write d3/d65/d7d/dc6/de5/ff8 [1032756,9370] 0 2024-04-06T14:17:16.250 INFO:tasks.workunit.client.1.smithi086.stdout:4/462: creat d0/d2c/d30/d24/d5d/f8f x:0 0 0 2024-04-06T14:17:16.253 INFO:tasks.workunit.client.0.smithi032.stdout:3/802: mknod d3/d65/c139 0 2024-04-06T14:17:16.253 INFO:tasks.workunit.client.1.smithi086.stdout:9/506: chown d5/d13/l9e 47217022 1 2024-04-06T14:17:16.255 INFO:tasks.workunit.client.0.smithi032.stdout:2/751: rename d0/d30/d39/d59/d67/d74/d78/d8c/db0/d95/lbc to d0/d4e/de9/lfa 0 2024-04-06T14:17:16.255 INFO:tasks.workunit.client.1.smithi086.stdout:3/621: read d5/df/f6b [3079765,71633] 0 2024-04-06T14:17:16.255 INFO:tasks.workunit.client.1.smithi086.stdout:6/723: symlink d6/d20/lf6 0 2024-04-06T14:17:16.256 INFO:tasks.workunit.client.0.smithi032.stdout:3/803: fdatasync d3/d65/d7d/d9c/dad/fb3 0 2024-04-06T14:17:16.256 INFO:tasks.workunit.client.1.smithi086.stdout:0/644: mknod d0/d1/d4/de/db4/cd6 0 2024-04-06T14:17:16.257 INFO:tasks.workunit.client.1.smithi086.stdout:4/463: mkdir d0/d2c/d38/d54/d90 0 2024-04-06T14:17:16.260 INFO:tasks.workunit.client.0.smithi032.stdout:7/864: write d16/d20/d54/de4/d5f/fc7 [1067753,30832] 0 2024-04-06T14:17:16.261 INFO:tasks.workunit.client.1.smithi086.stdout:3/622: unlink d5/d26/f3c 0 2024-04-06T14:17:16.261 INFO:tasks.workunit.client.1.smithi086.stdout:8/687: dwrite d0/d71/d22/d43/d6d/d75/d7d/fa1 [4194304,4194304] 0 2024-04-06T14:17:16.262 INFO:tasks.workunit.client.1.smithi086.stdout:6/724: mknod d6/d13/d1a/cf7 0 2024-04-06T14:17:16.263 INFO:tasks.workunit.client.1.smithi086.stdout:9/507: dread d5/d13/d47/f92 [0,4194304] 0 2024-04-06T14:17:16.267 INFO:tasks.workunit.client.0.smithi032.stdout:2/752: write d0/d30/d39/d59/d67/d74/d78/d8c/f9f [957218,96040] 0 2024-04-06T14:17:16.270 INFO:tasks.workunit.client.1.smithi086.stdout:2/602: write d0/d20/d27/d2b/d35/d36/d4d/d59/fad [586879,65233] 0 2024-04-06T14:17:16.270 INFO:tasks.workunit.client.0.smithi032.stdout:7/865: dwrite d16/d20/d54/de4/d5f/fc7 [0,4194304] 0 2024-04-06T14:17:16.271 INFO:tasks.workunit.client.1.smithi086.stdout:1/636: write db/d11/f27 [68300,35488] 0 2024-04-06T14:17:16.273 INFO:tasks.workunit.client.0.smithi032.stdout:7/866: fsync d16/d20/d54/de4/d5f/ddd/f113 0 2024-04-06T14:17:16.273 INFO:tasks.workunit.client.0.smithi032.stdout:3/804: dread d3/d16/d63/f83 [0,4194304] 0 2024-04-06T14:17:16.275 INFO:tasks.workunit.client.1.smithi086.stdout:4/464: creat d0/d15/f91 x:0 0 0 2024-04-06T14:17:16.277 INFO:tasks.workunit.client.1.smithi086.stdout:7/592: dwrite d7/d10/d11/d1c/d24/d4d/d5c/d66/f85 [0,4194304] 0 2024-04-06T14:17:16.278 INFO:tasks.workunit.client.0.smithi032.stdout:2/753: dread d0/d6e/ff2 [0,4194304] 0 2024-04-06T14:17:16.278 INFO:tasks.workunit.client.0.smithi032.stdout:3/805: creat d3/d65/d7d/d9c/dbc/dcd/f13a x:0 0 0 2024-04-06T14:17:16.278 INFO:tasks.workunit.client.0.smithi032.stdout:7/867: symlink d16/d20/d48/l134 0 2024-04-06T14:17:16.282 INFO:tasks.workunit.client.1.smithi086.stdout:0/645: dwrite d0/d15/fce [0,4194304] 0 2024-04-06T14:17:16.283 INFO:tasks.workunit.client.0.smithi032.stdout:2/754: chown d0/d30/d39/d59/d67/d74/ff3 37369 1 2024-04-06T14:17:16.283 INFO:tasks.workunit.client.0.smithi032.stdout:7/868: fsync d16/d20/d54/de4/d2d/d30/d3e/da7/fff 0 2024-04-06T14:17:16.286 INFO:tasks.workunit.client.0.smithi032.stdout:2/755: rename d0/db2/fd4 to d0/d6e/dae/db7/ffb 0 2024-04-06T14:17:16.288 INFO:tasks.workunit.client.0.smithi032.stdout:3/806: link d3/c76 d3/d65/d7d/d9c/dad/db1/d3c/c13b 0 2024-04-06T14:17:16.290 INFO:tasks.workunit.client.1.smithi086.stdout:6/725: getdents d6/d13/d5f/d60/d45/d8a/de3 0 2024-04-06T14:17:16.291 INFO:tasks.workunit.client.0.smithi032.stdout:3/807: chown d3/dcf 1111668 1 2024-04-06T14:17:16.291 INFO:tasks.workunit.client.0.smithi032.stdout:2/756: rename d0/d5a/d3a to d0/db2/dfc 0 2024-04-06T14:17:16.292 INFO:tasks.workunit.client.1.smithi086.stdout:7/593: creat d7/d10/d11/d80/fc9 x:0 0 0 2024-04-06T14:17:16.294 INFO:tasks.workunit.client.0.smithi032.stdout:3/808: symlink d3/d16/d63/l13c 0 2024-04-06T14:17:16.295 INFO:tasks.workunit.client.1.smithi086.stdout:8/688: truncate d0/d3b/f8b 450205 0 2024-04-06T14:17:16.295 INFO:tasks.workunit.client.1.smithi086.stdout:1/637: write db/d13/f92 [746937,71050] 0 2024-04-06T14:17:16.295 INFO:tasks.workunit.client.0.smithi032.stdout:3/809: write d3/d65/d7d/dc6/fdb [867858,95589] 0 2024-04-06T14:17:16.296 INFO:tasks.workunit.client.1.smithi086.stdout:0/646: dread - d0/d15/d4e/da3/d46/d52/dd0/fc0 zero size 2024-04-06T14:17:16.297 INFO:tasks.workunit.client.0.smithi032.stdout:3/810: read - d3/d65/d7d/d9c/dbc/dcd/f13a zero size 2024-04-06T14:17:16.298 INFO:tasks.workunit.client.1.smithi086.stdout:9/508: creat d5/d13/d1a/d5f/d88/d14/fa7 x:0 0 0 2024-04-06T14:17:16.300 INFO:tasks.workunit.client.0.smithi032.stdout:3/811: readlink d3/d65/d7d/dc6/l107 0 2024-04-06T14:17:16.300 INFO:tasks.workunit.client.0.smithi032.stdout:7/869: dread fb [0,4194304] 0 2024-04-06T14:17:16.301 INFO:tasks.workunit.client.1.smithi086.stdout:6/726: dread - d6/d13/d1a/d64/d85/fdb zero size 2024-04-06T14:17:16.301 INFO:tasks.workunit.client.1.smithi086.stdout:4/465: dwrite d0/d2c/d30/f6c [0,4194304] 0 2024-04-06T14:17:16.301 INFO:tasks.workunit.client.1.smithi086.stdout:7/594: creat d7/d10/d11/d80/fca x:0 0 0 2024-04-06T14:17:16.301 INFO:tasks.workunit.client.1.smithi086.stdout:2/603: dwrite d0/d20/d27/d2b/d35/d36/fa2 [0,4194304] 0 2024-04-06T14:17:16.305 INFO:tasks.workunit.client.1.smithi086.stdout:1/638: creat db/d11/d1a/d2a/fcb x:0 0 0 2024-04-06T14:17:16.305 INFO:tasks.workunit.client.1.smithi086.stdout:3/623: dwrite d5/d36/d65/f7b [0,4194304] 0 2024-04-06T14:17:16.306 INFO:tasks.workunit.client.1.smithi086.stdout:4/466: write d0/d15/d20/f44 [6016832,94837] 0 2024-04-06T14:17:16.306 INFO:tasks.workunit.client.0.smithi032.stdout:7/870: creat d16/d20/d54/de4/d2d/d30/d3e/d96/d132/f135 x:0 0 0 2024-04-06T14:17:16.307 INFO:tasks.workunit.client.1.smithi086.stdout:4/467: fdatasync d0/d2c/d38/d54/f80 0 2024-04-06T14:17:16.308 INFO:tasks.workunit.client.1.smithi086.stdout:0/647: mkdir d0/d1/d2/d1a/d51/dd7 0 2024-04-06T14:17:16.314 INFO:tasks.workunit.client.1.smithi086.stdout:6/727: fsync d6/d13/d1a/d64/d85/fdb 0 2024-04-06T14:17:16.319 INFO:tasks.workunit.client.1.smithi086.stdout:3/624: mknod d5/df/d9e/cc8 0 2024-04-06T14:17:16.320 INFO:tasks.workunit.client.1.smithi086.stdout:8/689: creat d0/d71/d16/d39/fec x:0 0 0 2024-04-06T14:17:16.326 INFO:tasks.workunit.client.1.smithi086.stdout:7/595: mknod d7/d10/d11/d21/db0/ccb 0 2024-04-06T14:17:16.326 INFO:tasks.workunit.client.1.smithi086.stdout:9/509: creat d5/d13/d47/fa8 x:0 0 0 2024-04-06T14:17:16.326 INFO:tasks.workunit.client.1.smithi086.stdout:1/639: dread db/d11/f17 [0,4194304] 0 2024-04-06T14:17:16.327 INFO:tasks.workunit.client.1.smithi086.stdout:3/625: mknod d5/df/d9e/da5/cc9 0 2024-04-06T14:17:16.329 INFO:tasks.workunit.client.1.smithi086.stdout:4/468: fsync d0/d2c/d30/d24/f2d 0 2024-04-06T14:17:16.329 INFO:tasks.workunit.client.1.smithi086.stdout:2/604: getdents d0/d20/d80/dbf 0 2024-04-06T14:17:16.330 INFO:tasks.workunit.client.1.smithi086.stdout:8/690: read d0/d3b/f4c [497523,86323] 0 2024-04-06T14:17:16.330 INFO:tasks.workunit.client.0.smithi032.stdout:7/871: write d16/d20/d54/de4/d2d/d30/f46 [1003857,80104] 0 2024-04-06T14:17:16.331 INFO:tasks.workunit.client.1.smithi086.stdout:9/510: creat d5/d13/fa9 x:0 0 0 2024-04-06T14:17:16.333 INFO:tasks.workunit.client.1.smithi086.stdout:7/596: rmdir d7/d10/d11/d1c/d36 39 2024-04-06T14:17:16.333 INFO:tasks.workunit.client.0.smithi032.stdout:7/872: creat d16/d20/d86/f136 x:0 0 0 2024-04-06T14:17:16.336 INFO:tasks.workunit.client.1.smithi086.stdout:2/605: mkdir d0/d20/d27/dd1 0 2024-04-06T14:17:16.336 INFO:tasks.workunit.client.1.smithi086.stdout:4/469: mknod d0/d2c/d30/d24/d50/c92 0 2024-04-06T14:17:16.338 INFO:tasks.workunit.client.1.smithi086.stdout:8/691: mkdir d0/d71/d93/ded 0 2024-04-06T14:17:16.338 INFO:tasks.workunit.client.0.smithi032.stdout:7/873: dread d16/d20/d54/de4/d5f/f90 [0,4194304] 0 2024-04-06T14:17:16.339 INFO:tasks.workunit.client.1.smithi086.stdout:3/626: dwrite d5/df/f2c [0,4194304] 0 2024-04-06T14:17:16.339 INFO:tasks.workunit.client.1.smithi086.stdout:3/627: chown d5/cd 260343876 1 2024-04-06T14:17:16.340 INFO:tasks.workunit.client.1.smithi086.stdout:6/728: link d6/d13/d5f/d60/ce9 d6/d13/d5f/d60/d45/da6/cf8 0 2024-04-06T14:17:16.342 INFO:tasks.workunit.client.1.smithi086.stdout:9/511: symlink d5/d13/d1a/d5f/d88/d14/d19/d1b/laa 0 2024-04-06T14:17:16.342 INFO:tasks.workunit.client.0.smithi032.stdout:7/874: unlink d16/d20/d54/de4/d2d/f3f 0 2024-04-06T14:17:16.343 INFO:tasks.workunit.client.1.smithi086.stdout:2/606: mkdir d0/df/dd2 0 2024-04-06T14:17:16.343 INFO:tasks.workunit.client.0.smithi032.stdout:7/875: dread - d16/d20/d7a/da6/d117/f133 zero size 2024-04-06T14:17:16.344 INFO:tasks.workunit.client.1.smithi086.stdout:6/729: dread - d6/d13/d5f/d60/d45/d8a/fef zero size 2024-04-06T14:17:16.345 INFO:tasks.workunit.client.0.smithi032.stdout:7/876: readlink d16/d20/d63/lf0 0 2024-04-06T14:17:16.345 INFO:tasks.workunit.client.1.smithi086.stdout:7/597: unlink d7/d10/d11/d1c/d24/d4d/d5c/c6d 0 2024-04-06T14:17:16.346 INFO:tasks.workunit.client.0.smithi032.stdout:7/877: chown d16/d20/dbd/f105 18439653 1 2024-04-06T14:17:16.347 INFO:tasks.workunit.client.1.smithi086.stdout:6/730: dread - d6/d13/d5f/d60/d45/fe7 zero size 2024-04-06T14:17:16.348 INFO:tasks.workunit.client.0.smithi032.stdout:7/878: truncate d16/d20/f47 2570929 0 2024-04-06T14:17:16.348 INFO:tasks.workunit.client.1.smithi086.stdout:6/731: fsync d6/d13/d42/d52/df1/ff2 0 2024-04-06T14:17:16.349 INFO:tasks.workunit.client.1.smithi086.stdout:4/470: mkdir d0/d2c/d30/d24/d93 0 2024-04-06T14:17:16.349 INFO:tasks.workunit.client.1.smithi086.stdout:7/598: dread d7/d10/d11/d1c/d24/d4d/daf/db5/d7a/f93 [0,4194304] 0 2024-04-06T14:17:16.349 INFO:tasks.workunit.client.1.smithi086.stdout:0/648: truncate d0/d15/d4e/f45 3029845 0 2024-04-06T14:17:16.349 INFO:tasks.workunit.client.1.smithi086.stdout:6/732: chown d6/d13/d1a/cde 112 1 2024-04-06T14:17:16.350 INFO:tasks.workunit.client.0.smithi032.stdout:7/879: dread d16/d20/d54/de4/d2d/d30/d3e/da7/db5/fe6 [0,4194304] 0 2024-04-06T14:17:16.351 INFO:tasks.workunit.client.1.smithi086.stdout:4/471: truncate d0/d2c/f81 983703 0 2024-04-06T14:17:16.351 INFO:tasks.workunit.client.1.smithi086.stdout:8/692: mkdir d0/d71/d22/d43/d6d/d75/dee 0 2024-04-06T14:17:16.353 INFO:tasks.workunit.client.1.smithi086.stdout:9/512: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab 0 2024-04-06T14:17:16.353 INFO:tasks.workunit.client.0.smithi032.stdout:7/880: mkdir d16/d20/d6e/d137 0 2024-04-06T14:17:16.354 INFO:tasks.workunit.client.1.smithi086.stdout:6/733: fdatasync d6/d13/d1a/d64/fb3 0 2024-04-06T14:17:16.354 INFO:tasks.workunit.client.1.smithi086.stdout:3/628: getdents d5/df/d9e/db5 0 2024-04-06T14:17:16.355 INFO:tasks.workunit.client.0.smithi032.stdout:7/881: truncate d16/d20/d7a/da6/d117/f133 11541 0 2024-04-06T14:17:16.355 INFO:tasks.workunit.client.1.smithi086.stdout:8/693: read d0/d71/d16/d52/dd0/ddc/de7/f5d [3512261,4969] 0 2024-04-06T14:17:16.355 INFO:tasks.workunit.client.1.smithi086.stdout:0/649: mknod d0/d15/d4e/da3/d44/d8e/dd2/cd8 0 2024-04-06T14:17:16.355 INFO:tasks.workunit.client.1.smithi086.stdout:1/640: dread db/d11/d1a/d2a/d52/d6b/f97 [0,4194304] 0 2024-04-06T14:17:16.356 INFO:tasks.workunit.client.1.smithi086.stdout:4/472: symlink d0/d2c/d30/l94 0 2024-04-06T14:17:16.357 INFO:tasks.workunit.client.1.smithi086.stdout:0/650: chown d0/d1/c53 3900 1 2024-04-06T14:17:16.357 INFO:tasks.workunit.client.1.smithi086.stdout:2/607: rename d0/d20/d27/d2b/d35/d36/d4d/lab to d0/df/dbc/ld3 0 2024-04-06T14:17:16.358 INFO:tasks.workunit.client.0.smithi032.stdout:7/882: truncate d16/d20/d63/d9d/df2/df5/fe7 502010 0 2024-04-06T14:17:16.358 INFO:tasks.workunit.client.1.smithi086.stdout:4/473: write d0/d15/f3c [394901,75134] 0 2024-04-06T14:17:16.359 INFO:tasks.workunit.client.1.smithi086.stdout:9/513: rmdir d5/d13/d47/d5e 39 2024-04-06T14:17:16.359 INFO:tasks.workunit.client.1.smithi086.stdout:3/629: write d5/d36/d5a/d6d/d72/d64/fbd [461688,74577] 0 2024-04-06T14:17:16.360 INFO:tasks.workunit.client.0.smithi032.stdout:7/883: mkdir d16/d20/d6e/d138 0 2024-04-06T14:17:16.364 INFO:tasks.workunit.client.1.smithi086.stdout:8/694: symlink d0/d71/d16/d52/lef 0 2024-04-06T14:17:16.369 INFO:tasks.workunit.client.0.smithi032.stdout:7/884: write d16/d20/d63/d9d/df2/df5/ff7 [873897,73540] 0 2024-04-06T14:17:16.370 INFO:tasks.workunit.client.1.smithi086.stdout:4/474: mknod d0/d2c/d30/d24/c95 0 2024-04-06T14:17:16.371 INFO:tasks.workunit.client.1.smithi086.stdout:4/475: write d0/d15/d20/d74/f7b [467049,11618] 0 2024-04-06T14:17:16.373 INFO:tasks.workunit.client.1.smithi086.stdout:3/630: rename d5/df/fc7 to d5/df/fca 0 2024-04-06T14:17:16.374 INFO:tasks.workunit.client.1.smithi086.stdout:4/476: truncate d0/d2c/d38/d54/d83/f77 879948 0 2024-04-06T14:17:16.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:16 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:17:16.377 INFO:tasks.workunit.client.1.smithi086.stdout:0/651: dread d0/d15/d4e/da3/d44/d8e/d8f/fc4 [0,4194304] 0 2024-04-06T14:17:16.379 INFO:tasks.workunit.client.1.smithi086.stdout:9/514: fsync d5/d13/d1a/d5f/d88/d14/d19/d25/f99 0 2024-04-06T14:17:16.380 INFO:tasks.workunit.client.1.smithi086.stdout:4/477: truncate d0/d2c/d38/f5b 2570874 0 2024-04-06T14:17:16.382 INFO:tasks.workunit.client.1.smithi086.stdout:2/608: creat d0/d20/d27/d2b/d35/d36/fd4 x:0 0 0 2024-04-06T14:17:16.383 INFO:tasks.workunit.client.0.smithi032.stdout:7/885: dread d16/d20/d54/de4/d2d/f69 [0,4194304] 0 2024-04-06T14:17:16.384 INFO:tasks.workunit.client.1.smithi086.stdout:4/478: write d0/d2c/f81 [54216,63122] 0 2024-04-06T14:17:16.384 INFO:tasks.workunit.client.1.smithi086.stdout:0/652: truncate d0/d1/d2/d1a/d51/f83 984557 0 2024-04-06T14:17:16.385 INFO:tasks.workunit.client.1.smithi086.stdout:1/641: truncate db/d13/d71/f7f 3844859 0 2024-04-06T14:17:16.385 INFO:tasks.workunit.client.0.smithi032.stdout:7/886: mknod d16/d20/d48/d8d/c139 0 2024-04-06T14:17:16.386 INFO:tasks.workunit.client.1.smithi086.stdout:1/642: chown db/d13/d55/da3 868506 1 2024-04-06T14:17:16.388 INFO:tasks.workunit.client.1.smithi086.stdout:9/515: rmdir d5/d13/d1a/d5f/d88/d14/d19/d1b 39 2024-04-06T14:17:16.390 INFO:tasks.workunit.client.1.smithi086.stdout:6/734: dwrite d6/d9/db/f70 [0,4194304] 0 2024-04-06T14:17:16.390 INFO:tasks.workunit.client.0.smithi032.stdout:7/887: unlink d16/d20/d7a/da6/ce5 0 2024-04-06T14:17:16.390 INFO:tasks.workunit.client.1.smithi086.stdout:1/643: chown db/d11/d1a/f85 288073376 1 2024-04-06T14:17:16.391 INFO:tasks.workunit.client.0.smithi032.stdout:7/888: chown d16/d20/d54/de4/d2d/d30/d3e/d96/cfe 263024142 1 2024-04-06T14:17:16.392 INFO:tasks.workunit.client.1.smithi086.stdout:3/631: truncate d5/df/f6b 1192367 0 2024-04-06T14:17:16.394 INFO:tasks.workunit.client.1.smithi086.stdout:0/653: unlink d0/d15/f8d 0 2024-04-06T14:17:16.396 INFO:tasks.workunit.client.1.smithi086.stdout:0/654: chown d0/d1/d2/d1a/d79/c85 146617 1 2024-04-06T14:17:16.402 INFO:tasks.workunit.client.0.smithi032.stdout:7/889: creat d16/d20/d54/de4/d2d/d30/f13a x:0 0 0 2024-04-06T14:17:16.403 INFO:tasks.workunit.client.0.smithi032.stdout:7/890: stat d16/d20/d86/c8c 0 2024-04-06T14:17:16.405 INFO:tasks.workunit.client.1.smithi086.stdout:3/632: mknod d5/d26/d4f/db4/da7/d89/ccb 0 2024-04-06T14:17:16.408 INFO:tasks.workunit.client.1.smithi086.stdout:2/609: getdents d0/d20/d27/d2b 0 2024-04-06T14:17:16.410 INFO:tasks.workunit.client.0.smithi032.stdout:7/891: mkdir d16/d20/d54/de4/d2d/d30/d97/d13b 0 2024-04-06T14:17:16.411 INFO:tasks.workunit.client.1.smithi086.stdout:2/610: readlink d0/l31 0 2024-04-06T14:17:16.411 INFO:tasks.workunit.client.1.smithi086.stdout:9/516: dread d5/d13/d1a/d8f/f9a [0,4194304] 0 2024-04-06T14:17:16.412 INFO:tasks.workunit.client.1.smithi086.stdout:6/735: write d6/d13/d1a/da0/fda [425676,84584] 0 2024-04-06T14:17:16.412 INFO:tasks.workunit.client.1.smithi086.stdout:4/479: dread d0/d15/f37 [0,4194304] 0 2024-04-06T14:17:16.412 INFO:tasks.workunit.client.1.smithi086.stdout:1/644: link db/f10 db/d11/d1a/d2a/d9a/d78/fcc 0 2024-04-06T14:17:16.413 INFO:tasks.workunit.client.1.smithi086.stdout:2/611: stat d0/df/d48/d53/l9a 0 2024-04-06T14:17:16.413 INFO:tasks.workunit.client.0.smithi032.stdout:7/892: fsync d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee/f79 0 2024-04-06T14:17:16.415 INFO:tasks.workunit.client.0.smithi032.stdout:7/893: dread - d16/d20/d54/de4/d2d/d30/d3e/d96/d132/f135 zero size 2024-04-06T14:17:16.416 INFO:tasks.workunit.client.1.smithi086.stdout:2/612: chown d0/d20/d27/d2b/d35/d36/fa2 2 1 2024-04-06T14:17:16.418 INFO:tasks.workunit.client.0.smithi032.stdout:7/894: creat d16/d20/d54/de4/d2d/d30/d3e/d96/d132/f13c x:0 0 0 2024-04-06T14:17:16.418 INFO:tasks.workunit.client.1.smithi086.stdout:3/633: rename d5/d26/d7f/fb2 to d5/d36/d5a/d6d/d72/fcc 0 2024-04-06T14:17:16.420 INFO:tasks.workunit.client.1.smithi086.stdout:6/736: unlink d6/d13/d1a/d22/c2c 0 2024-04-06T14:17:16.421 INFO:tasks.workunit.client.1.smithi086.stdout:0/655: dwrite d0/d1/d2/d1a/f7b [4194304,4194304] 0 2024-04-06T14:17:16.430 INFO:tasks.workunit.client.0.smithi032.stdout:7/895: dwrite d16/d20/d54/de4/f44 [0,4194304] 0 2024-04-06T14:17:16.437 INFO:tasks.workunit.client.0.smithi032.stdout:7/896: dwrite f11 [0,4194304] 0 2024-04-06T14:17:16.444 INFO:tasks.workunit.client.0.smithi032.stdout:7/897: truncate d16/d20/d54/de4/d2d/d30/d3e/df3/ff4 6870943 0 2024-04-06T14:17:16.457 INFO:tasks.workunit.client.0.smithi032.stdout:7/898: dread d16/d20/d54/de4/d2d/d30/d3e/f11e [0,4194304] 0 2024-04-06T14:17:16.458 INFO:tasks.workunit.client.0.smithi032.stdout:7/899: write d16/d20/d54/de4/d5f/fb0 [885873,17939] 0 2024-04-06T14:17:16.460 INFO:tasks.workunit.client.0.smithi032.stdout:7/900: unlink d16/d20/d63/d9d/df2/df5/ff7 0 2024-04-06T14:17:16.467 INFO:tasks.workunit.client.0.smithi032.stdout:7/901: dread fe [0,4194304] 0 2024-04-06T14:17:16.470 INFO:tasks.workunit.client.0.smithi032.stdout:7/902: mkdir d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dde/d11a/d13d 0 2024-04-06T14:17:16.472 INFO:tasks.workunit.client.0.smithi032.stdout:7/903: dread - d16/d20/d48/d8d/fb6 zero size 2024-04-06T14:17:16.477 INFO:tasks.workunit.client.0.smithi032.stdout:7/904: write d16/d20/d63/d9d/df2/df5/fe7 [696036,64272] 0 2024-04-06T14:17:16.477 INFO:tasks.workunit.client.1.smithi086.stdout:5/503: sync 2024-04-06T14:17:16.479 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:16 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:17:16.483 INFO:tasks.workunit.client.1.smithi086.stdout:6/737: write d6/d9/db/f16 [1959752,15160] 0 2024-04-06T14:17:16.485 INFO:tasks.workunit.client.1.smithi086.stdout:4/480: getdents d0/d2c/d38/d54/d90 0 2024-04-06T14:17:16.485 INFO:tasks.workunit.client.1.smithi086.stdout:1/645: link db/d11/f27 db/d11/d1a/d2a/d9a/d68/daf/dc2/fcd 0 2024-04-06T14:17:16.487 INFO:tasks.workunit.client.1.smithi086.stdout:9/517: write d5/d13/d1a/d5f/d88/d14/d19/d1b/f45 [1804227,75546] 0 2024-04-06T14:17:16.488 INFO:tasks.workunit.client.1.smithi086.stdout:6/738: truncate d6/d9/db/fd2 5593493 0 2024-04-06T14:17:16.490 INFO:tasks.workunit.client.1.smithi086.stdout:2/613: fsync d0/df/d48/d53/d76/d78/fbd 0 2024-04-06T14:17:16.493 INFO:tasks.workunit.client.0.smithi032.stdout:7/905: dread d16/d92/db3/f103 [0,4194304] 0 2024-04-06T14:17:16.494 INFO:tasks.workunit.client.1.smithi086.stdout:0/656: write d0/d1/f76 [2220011,61992] 0 2024-04-06T14:17:16.496 INFO:tasks.workunit.client.0.smithi032.stdout:7/906: creat d16/d20/d6e/f13e x:0 0 0 2024-04-06T14:17:16.498 INFO:tasks.workunit.client.1.smithi086.stdout:9/518: write d5/d13/f7d [49523,54759] 0 2024-04-06T14:17:16.498 INFO:tasks.workunit.client.1.smithi086.stdout:5/504: rmdir d0/d5e/d98 0 2024-04-06T14:17:16.498 INFO:tasks.workunit.client.1.smithi086.stdout:6/739: write d6/d13/f5d [2571479,61246] 0 2024-04-06T14:17:16.499 INFO:tasks.workunit.client.0.smithi032.stdout:7/907: fdatasync d16/d20/d54/de4/f39 0 2024-04-06T14:17:16.502 INFO:tasks.workunit.client.0.smithi032.stdout:7/908: creat d16/d20/d54/de4/d5f/ddd/f13f x:0 0 0 2024-04-06T14:17:16.502 INFO:tasks.workunit.client.0.smithi032.stdout:7/909: stat d16/f27 0 2024-04-06T14:17:16.503 INFO:tasks.workunit.client.1.smithi086.stdout:3/634: dwrite d5/d36/f67 [0,4194304] 0 2024-04-06T14:17:16.504 INFO:tasks.workunit.client.0.smithi032.stdout:7/910: symlink d16/d20/d7a/da6/d117/d11b/l140 0 2024-04-06T14:17:16.504 INFO:tasks.workunit.client.1.smithi086.stdout:1/646: dwrite db/d13/d71/fa7 [0,4194304] 0 2024-04-06T14:17:16.505 INFO:tasks.workunit.client.1.smithi086.stdout:6/740: dread - d6/d13/d1a/d22/dcf/fd7 zero size 2024-04-06T14:17:16.507 INFO:tasks.workunit.client.0.smithi032.stdout:7/911: mknod d16/d92/c141 0 2024-04-06T14:17:16.507 INFO:tasks.workunit.client.1.smithi086.stdout:5/505: truncate d0/f4 1892009 0 2024-04-06T14:17:16.508 INFO:tasks.workunit.client.1.smithi086.stdout:1/647: write db/d13/f86 [311278,20459] 0 2024-04-06T14:17:16.511 INFO:tasks.workunit.client.1.smithi086.stdout:6/741: creat d6/d9/d43/ff9 x:0 0 0 2024-04-06T14:17:16.513 INFO:tasks.workunit.client.0.smithi032.stdout:7/912: rmdir d16/d20/d63/d9d 39 2024-04-06T14:17:16.515 INFO:tasks.workunit.client.1.smithi086.stdout:2/614: link d0/df/dbc/ld3 d0/d20/d27/d2b/d35/d36/d4d/d59/d94/ld5 0 2024-04-06T14:17:16.515 INFO:tasks.workunit.client.1.smithi086.stdout:9/519: dread d5/d13/d1a/d5f/d88/d14/f86 [0,4194304] 0 2024-04-06T14:17:16.517 INFO:tasks.workunit.client.0.smithi032.stdout:7/913: unlink d16/d20/d54/de4/d5f/ddd/f13f 0 2024-04-06T14:17:16.517 INFO:tasks.workunit.client.1.smithi086.stdout:3/635: creat d5/d26/d7f/dae/fcd x:0 0 0 2024-04-06T14:17:16.518 INFO:tasks.workunit.client.1.smithi086.stdout:1/648: mkdir db/d11/d1a/d79/dce 0 2024-04-06T14:17:16.519 INFO:tasks.workunit.client.1.smithi086.stdout:0/657: dwrite d0/d1/d4/f6b [4194304,4194304] 0 2024-04-06T14:17:16.519 INFO:tasks.workunit.client.0.smithi032.stdout:7/914: write d16/d20/d63/d9d/df2/df5/fe9 [901824,98648] 0 2024-04-06T14:17:16.522 INFO:tasks.workunit.client.1.smithi086.stdout:3/636: creat d5/d26/d4f/db4/fce x:0 0 0 2024-04-06T14:17:16.523 INFO:tasks.workunit.client.1.smithi086.stdout:0/658: creat d0/d15/d4e/da3/d44/d8e/fd9 x:0 0 0 2024-04-06T14:17:16.524 INFO:tasks.workunit.client.1.smithi086.stdout:0/659: readlink d0/l71 0 2024-04-06T14:17:16.527 INFO:tasks.workunit.client.1.smithi086.stdout:2/615: link d0/d20/d27/d37/f98 d0/d20/d27/d37/d52/d9c/fd6 0 2024-04-06T14:17:16.527 INFO:tasks.workunit.client.1.smithi086.stdout:3/637: write d5/d36/d65/f7b [4751457,4639] 0 2024-04-06T14:17:16.530 INFO:tasks.workunit.client.1.smithi086.stdout:9/520: dread d5/d13/d1a/d5f/d88/d14/f80 [0,4194304] 0 2024-04-06T14:17:16.530 INFO:tasks.workunit.client.1.smithi086.stdout:2/616: mknod d0/d20/d44/cd7 0 2024-04-06T14:17:16.533 INFO:tasks.workunit.client.1.smithi086.stdout:2/617: chown d0/df/dbc 823286 1 2024-04-06T14:17:16.533 INFO:tasks.workunit.client.0.smithi032.stdout:7/915: dwrite d16/d20/d54/de4/d2d/f69 [0,4194304] 0 2024-04-06T14:17:16.534 INFO:tasks.workunit.client.1.smithi086.stdout:3/638: mknod d5/d26/ccf 0 2024-04-06T14:17:16.534 INFO:tasks.workunit.client.1.smithi086.stdout:1/649: dwrite db/d13/f3d [0,4194304] 0 2024-04-06T14:17:16.535 INFO:tasks.workunit.client.1.smithi086.stdout:0/660: dread d0/d1/d4/de/fb0 [0,4194304] 0 2024-04-06T14:17:16.535 INFO:tasks.workunit.client.0.smithi032.stdout:7/916: write d16/d20/d48/d8d/f123 [559264,23738] 0 2024-04-06T14:17:16.544 INFO:tasks.workunit.client.1.smithi086.stdout:0/661: read d0/d15/f6a [1634249,73448] 0 2024-04-06T14:17:16.545 INFO:tasks.workunit.client.1.smithi086.stdout:5/506: dwrite d0/d5e/f60 [0,4194304] 0 2024-04-06T14:17:16.547 INFO:tasks.workunit.client.1.smithi086.stdout:9/521: rename d5/d13/d47/l57 to d5/d13/d1a/d5f/lac 0 2024-04-06T14:17:16.547 INFO:tasks.workunit.client.1.smithi086.stdout:5/507: write d0/d16/f63 [4209606,121110] 0 2024-04-06T14:17:16.548 INFO:tasks.workunit.client.1.smithi086.stdout:1/650: creat db/d11/d42/d53/fcf x:0 0 0 2024-04-06T14:17:16.550 INFO:tasks.workunit.client.0.smithi032.stdout:7/917: dwrite d16/d20/d54/de4/f3a [0,4194304] 0 2024-04-06T14:17:16.550 INFO:tasks.workunit.client.1.smithi086.stdout:1/651: chown db/d11/d1a/d2a/f47 73671 1 2024-04-06T14:17:16.552 INFO:tasks.workunit.client.1.smithi086.stdout:0/662: rename d0/d15/d4e/d48/d89/l96 to d0/d1/d2/dcf/lda 0 2024-04-06T14:17:16.554 INFO:tasks.workunit.client.1.smithi086.stdout:0/663: fsync d0/d15/d4e/da3/d44/fbc 0 2024-04-06T14:17:16.559 INFO:tasks.workunit.client.0.smithi032.stdout:7/918: creat d16/d20/f142 x:0 0 0 2024-04-06T14:17:16.561 INFO:tasks.workunit.client.1.smithi086.stdout:1/652: rename db/d11/d1a/d2a/d52/f6d to db/d13/d71/fd0 0 2024-04-06T14:17:16.565 INFO:tasks.workunit.client.1.smithi086.stdout:9/522: getdents d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab 0 2024-04-06T14:17:16.566 INFO:tasks.workunit.client.1.smithi086.stdout:3/639: write d5/d26/d7f/f55 [641991,73995] 0 2024-04-06T14:17:16.567 INFO:tasks.workunit.client.1.smithi086.stdout:1/653: creat db/d11/d1a/d2a/d9a/d78/fd1 x:0 0 0 2024-04-06T14:17:16.569 INFO:tasks.workunit.client.1.smithi086.stdout:3/640: dread - d5/d36/d5a/d6d/d72/d97/fbe zero size 2024-04-06T14:17:16.569 INFO:tasks.workunit.client.1.smithi086.stdout:2/618: dread d0/f6 [0,4194304] 0 2024-04-06T14:17:16.571 INFO:tasks.workunit.client.1.smithi086.stdout:9/523: rename d5/d13/d1a/d5f/d88/l23 to d5/d18/lad 0 2024-04-06T14:17:16.572 INFO:tasks.workunit.client.1.smithi086.stdout:9/524: readlink d5/d13/d1a/d5f/d88/d14/d19/d25/d59/la1 0 2024-04-06T14:17:16.572 INFO:tasks.workunit.client.1.smithi086.stdout:1/654: mkdir db/d11/d1a/d79/dce/dd2 0 2024-04-06T14:17:16.574 INFO:tasks.workunit.client.0.smithi032.stdout:7/919: dread d16/d20/d54/de4/f39 [0,4194304] 0 2024-04-06T14:17:16.574 INFO:tasks.workunit.client.1.smithi086.stdout:3/641: creat d5/d33/fd0 x:0 0 0 2024-04-06T14:17:16.575 INFO:tasks.workunit.client.1.smithi086.stdout:9/525: write d5/d18/f3f [8184990,92519] 0 2024-04-06T14:17:16.576 INFO:tasks.workunit.client.0.smithi032.stdout:7/920: symlink d16/d92/l143 0 2024-04-06T14:17:16.579 INFO:tasks.workunit.client.1.smithi086.stdout:1/655: creat db/d11/d1a/d2a/d52/d6b/fd3 x:0 0 0 2024-04-06T14:17:16.579 INFO:tasks.workunit.client.0.smithi032.stdout:7/921: rmdir d16/d20/d7a/da6 39 2024-04-06T14:17:16.581 INFO:tasks.workunit.client.1.smithi086.stdout:2/619: dwrite d0/d20/d27/d2b/d35/d36/f3b [0,4194304] 0 2024-04-06T14:17:16.581 INFO:tasks.workunit.client.0.smithi032.stdout:7/922: fsync d16/d20/f28 0 2024-04-06T14:17:16.587 INFO:tasks.workunit.client.1.smithi086.stdout:1/656: creat db/d11/d42/d53/fd4 x:0 0 0 2024-04-06T14:17:16.589 INFO:tasks.workunit.client.0.smithi032.stdout:7/923: truncate d16/d20/f5e 3331324 0 2024-04-06T14:17:16.591 INFO:tasks.workunit.client.1.smithi086.stdout:3/642: dwrite d5/f44 [0,4194304] 0 2024-04-06T14:17:16.592 INFO:tasks.workunit.client.0.smithi032.stdout:7/924: write d16/d20/f142 [992983,93927] 0 2024-04-06T14:17:16.592 INFO:tasks.workunit.client.1.smithi086.stdout:0/664: dwrite d0/d15/d4e/da3/d46/d52/dd0/fa7 [0,4194304] 0 2024-04-06T14:17:16.597 INFO:tasks.workunit.client.1.smithi086.stdout:2/620: link d0/df/d48/d53/l9e d0/d20/d44/d9f/ld8 0 2024-04-06T14:17:16.597 INFO:tasks.workunit.client.1.smithi086.stdout:1/657: creat db/fd5 x:0 0 0 2024-04-06T14:17:16.597 INFO:tasks.workunit.client.0.smithi032.stdout:7/925: creat d16/d20/d7a/f144 x:0 0 0 2024-04-06T14:17:16.598 INFO:tasks.workunit.client.1.smithi086.stdout:0/665: truncate d0/d15/d4e/f42 577961 0 2024-04-06T14:17:16.599 INFO:tasks.workunit.client.1.smithi086.stdout:3/643: creat d5/d36/d5a/d6d/d72/d57/fd1 x:0 0 0 2024-04-06T14:17:16.600 INFO:tasks.workunit.client.1.smithi086.stdout:9/526: dwrite d5/d13/d1a/d5f/d88/d14/f51 [0,4194304] 0 2024-04-06T14:17:16.600 INFO:tasks.workunit.client.1.smithi086.stdout:7/599: sync 2024-04-06T14:17:16.602 INFO:tasks.workunit.client.0.smithi032.stdout:7/926: symlink d16/d20/d6e/d10e/l145 0 2024-04-06T14:17:16.603 INFO:tasks.workunit.client.1.smithi086.stdout:2/621: dread - d0/df/d48/d53/d76/f9b zero size 2024-04-06T14:17:16.603 INFO:tasks.workunit.client.1.smithi086.stdout:3/644: write d5/d33/f52 [4796405,101848] 0 2024-04-06T14:17:16.607 INFO:tasks.workunit.client.1.smithi086.stdout:1/658: fdatasync db/d13/d71/f7f 0 2024-04-06T14:17:16.608 INFO:tasks.workunit.client.1.smithi086.stdout:0/666: creat d0/d15/d26/d33/d9f/fdb x:0 0 0 2024-04-06T14:17:16.608 INFO:tasks.workunit.client.1.smithi086.stdout:7/600: chown d7/d10/d11/d1c/d24/d4d/d5c/d66/f67 32756 1 2024-04-06T14:17:16.608 INFO:tasks.workunit.client.1.smithi086.stdout:9/527: dread - d5/d13/d1a/d5f/d88/d14/d19/d25/d74/fa5 zero size 2024-04-06T14:17:16.611 INFO:tasks.workunit.client.0.smithi032.stdout:7/927: creat d16/d20/d54/de4/d2d/d30/d3e/d96/d10b/f146 x:0 0 0 2024-04-06T14:17:16.612 INFO:tasks.workunit.client.1.smithi086.stdout:7/601: fdatasync d7/d10/d11/d1c/d24/d2c/d69/fa6 0 2024-04-06T14:17:16.612 INFO:tasks.workunit.client.1.smithi086.stdout:2/622: creat d0/d20/d27/d37/d52/d9c/fd9 x:0 0 0 2024-04-06T14:17:16.612 INFO:tasks.workunit.client.1.smithi086.stdout:1/659: symlink db/d13/d71/ld6 0 2024-04-06T14:17:16.613 INFO:tasks.workunit.client.0.smithi032.stdout:7/928: truncate d16/d20/d63/d9d/df2/df5/fea 1087692 0 2024-04-06T14:17:16.613 INFO:tasks.workunit.client.0.smithi032.stdout:8/891: sync 2024-04-06T14:17:16.613 INFO:tasks.workunit.client.0.smithi032.stdout:4/840: sync 2024-04-06T14:17:16.614 INFO:tasks.workunit.client.1.smithi086.stdout:2/623: stat d0/df/d48 0 2024-04-06T14:17:16.614 INFO:tasks.workunit.client.1.smithi086.stdout:1/660: chown db/d11/d42/d53 1 1 2024-04-06T14:17:16.616 INFO:tasks.workunit.client.1.smithi086.stdout:3/645: dwrite d5/d33/f47 [4194304,4194304] 0 2024-04-06T14:17:16.618 INFO:tasks.workunit.client.0.smithi032.stdout:4/841: creat d0/df/d23/d31/d50/d57/d8a/db5/def/f11a x:0 0 0 2024-04-06T14:17:16.621 INFO:tasks.workunit.client.0.smithi032.stdout:7/929: write d16/f49 [1355356,29362] 0 2024-04-06T14:17:16.624 INFO:tasks.workunit.client.1.smithi086.stdout:7/602: mknod d7/d10/d11/d1c/d24/d4d/d5c/ccc 0 2024-04-06T14:17:16.626 INFO:tasks.workunit.client.0.smithi032.stdout:7/930: symlink d16/d20/d54/de4/d2d/d30/d3e/df3/d110/l147 0 2024-04-06T14:17:16.626 INFO:tasks.workunit.client.1.smithi086.stdout:2/624: symlink d0/df/d48/dce/lda 0 2024-04-06T14:17:16.627 INFO:tasks.workunit.client.1.smithi086.stdout:9/528: dwrite d5/d13/d1a/d5f/d88/d14/d19/fa6 [0,4194304] 0 2024-04-06T14:17:16.627 INFO:tasks.workunit.client.1.smithi086.stdout:7/603: write d7/d10/d11/d1c/f58 [3533137,12352] 0 2024-04-06T14:17:16.632 INFO:tasks.workunit.client.1.smithi086.stdout:2/625: dread - d0/d20/d27/d2b/d35/d36/d4d/d6f/fc6 zero size 2024-04-06T14:17:16.638 INFO:tasks.workunit.client.1.smithi086.stdout:2/626: creat d0/d20/d44/dbb/fdb x:0 0 0 2024-04-06T14:17:16.639 INFO:tasks.workunit.client.1.smithi086.stdout:9/529: dwrite d5/d13/d47/f92 [0,4194304] 0 2024-04-06T14:17:16.641 INFO:tasks.workunit.client.1.smithi086.stdout:3/646: write d5/f3e [852653,109032] 0 2024-04-06T14:17:16.641 INFO:tasks.workunit.client.1.smithi086.stdout:1/661: truncate db/d13/f92 273572 0 2024-04-06T14:17:16.642 INFO:tasks.workunit.client.1.smithi086.stdout:9/530: write d5/f20 [2610466,75552] 0 2024-04-06T14:17:16.644 INFO:tasks.workunit.client.1.smithi086.stdout:3/647: chown d5/d26/d7f/l53 7691 1 2024-04-06T14:17:16.645 INFO:tasks.workunit.client.1.smithi086.stdout:2/627: read d0/df/f84 [1629822,31125] 0 2024-04-06T14:17:16.645 INFO:tasks.workunit.client.1.smithi086.stdout:7/604: truncate d7/d10/d11/d80/f50 8373678 0 2024-04-06T14:17:16.646 INFO:tasks.workunit.client.1.smithi086.stdout:1/662: rmdir db/d11/d1a/d2a/d9a/d78 39 2024-04-06T14:17:16.648 INFO:tasks.workunit.client.1.smithi086.stdout:1/663: stat db/d11/d1a/d2a/d9a 0 2024-04-06T14:17:16.648 INFO:tasks.workunit.client.0.smithi032.stdout:8/892: dread d3/d8/d12/d2f/d4a/f8b [0,4194304] 0 2024-04-06T14:17:16.649 INFO:tasks.workunit.client.1.smithi086.stdout:7/605: mkdir d7/d10/d11/d1c/d24/dc8/dcd 0 2024-04-06T14:17:16.651 INFO:tasks.workunit.client.0.smithi032.stdout:8/893: rmdir d3/d8/d12/d2f/d4a/d60/d10b 39 2024-04-06T14:17:16.651 INFO:tasks.workunit.client.1.smithi086.stdout:1/664: symlink db/d11/d1a/d2a/d52/d6b/ld7 0 2024-04-06T14:17:16.652 INFO:tasks.workunit.client.1.smithi086.stdout:2/628: truncate d0/d20/d44/d9f/fc8 541288 0 2024-04-06T14:17:16.652 INFO:tasks.workunit.client.1.smithi086.stdout:3/648: getdents d5/d26/da9 0 2024-04-06T14:17:16.654 INFO:tasks.workunit.client.1.smithi086.stdout:7/606: truncate d7/d10/d11/d1c/d24/d2c/d69/d83/fab 465058 0 2024-04-06T14:17:16.655 INFO:tasks.workunit.client.1.smithi086.stdout:2/629: write d0/df/d5e/fc7 [297879,34202] 0 2024-04-06T14:17:16.656 INFO:tasks.workunit.client.1.smithi086.stdout:3/649: mkdir d5/d26/d4f/dd2 0 2024-04-06T14:17:16.658 INFO:tasks.workunit.client.1.smithi086.stdout:1/665: mkdir db/d11/d1a/d2a/d9a/d68/daf/dc2/dd8 0 2024-04-06T14:17:16.658 INFO:tasks.workunit.client.0.smithi032.stdout:7/931: dread d16/d20/dbd/f105 [0,4194304] 0 2024-04-06T14:17:16.662 INFO:tasks.workunit.client.1.smithi086.stdout:3/650: rename d5/dc3 to d5/d36/d56/dd3 0 2024-04-06T14:17:16.665 INFO:tasks.workunit.client.0.smithi032.stdout:8/894: dwrite d3/d8/d12/d2f/d46/def/d128/fa1 [0,4194304] 0 2024-04-06T14:17:16.666 INFO:tasks.workunit.client.1.smithi086.stdout:7/607: dwrite d7/d1d/f3e [0,4194304] 0 2024-04-06T14:17:16.667 INFO:tasks.workunit.client.1.smithi086.stdout:3/651: chown d5/d36/d5a/d6d/f96 103 1 2024-04-06T14:17:16.668 INFO:tasks.workunit.client.1.smithi086.stdout:2/630: dwrite d0/d20/d27/d37/d52/f5c [0,4194304] 0 2024-04-06T14:17:16.673 INFO:tasks.workunit.client.1.smithi086.stdout:3/652: creat d5/d36/d5a/fd4 x:0 0 0 2024-04-06T14:17:16.673 INFO:tasks.workunit.client.0.smithi032.stdout:7/932: dwrite d16/d20/d54/de4/d2d/f9b [0,4194304] 0 2024-04-06T14:17:16.675 INFO:tasks.workunit.client.1.smithi086.stdout:1/666: getdents db/d11/d1a/d2a/d9a/d68/d6e 0 2024-04-06T14:17:16.677 INFO:tasks.workunit.client.1.smithi086.stdout:3/653: write d5/f8d [2714299,130317] 0 2024-04-06T14:17:16.677 INFO:tasks.workunit.client.1.smithi086.stdout:9/531: dwrite d5/d13/d1a/d5f/d88/d14/f86 [0,4194304] 0 2024-04-06T14:17:16.680 INFO:tasks.workunit.client.0.smithi032.stdout:8/895: dwrite d3/d8/d12/d2f/d4a/d60/d90/f113 [0,4194304] 0 2024-04-06T14:17:16.685 INFO:tasks.workunit.client.1.smithi086.stdout:1/667: fsync db/d11/d1a/d2a/d9a/d68/daf/dc2/f8a 0 2024-04-06T14:17:16.693 INFO:tasks.workunit.client.1.smithi086.stdout:2/631: mkdir d0/d20/d80/dbf/ddc 0 2024-04-06T14:17:16.696 INFO:tasks.workunit.client.1.smithi086.stdout:9/532: dwrite d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f97 [0,4194304] 0 2024-04-06T14:17:16.696 INFO:tasks.workunit.client.1.smithi086.stdout:3/654: creat d5/d36/d5a/d6d/d72/d97/db0/fd5 x:0 0 0 2024-04-06T14:17:16.697 INFO:tasks.workunit.client.1.smithi086.stdout:1/668: dread db/d11/d1a/d2a/d9a/d68/d6e/f9e [0,4194304] 0 2024-04-06T14:17:16.697 INFO:tasks.workunit.client.0.smithi032.stdout:8/896: dread d3/d8/f2e [0,4194304] 0 2024-04-06T14:17:16.700 INFO:tasks.workunit.client.1.smithi086.stdout:3/655: dread - d5/d26/d4f/db4/fce zero size 2024-04-06T14:17:16.700 INFO:tasks.workunit.client.1.smithi086.stdout:2/632: read d0/d20/d27/d2b/d35/d36/fa2 [1925394,96415] 0 2024-04-06T14:17:16.701 INFO:tasks.workunit.client.1.smithi086.stdout:3/656: fsync d5/d26/d4f/db4/da7/f63 0 2024-04-06T14:17:16.701 INFO:tasks.workunit.client.0.smithi032.stdout:7/933: write d16/d20/d54/de4/d2d/d30/d3e/da7/db5/fe6 [779550,99048] 0 2024-04-06T14:17:16.702 INFO:tasks.workunit.client.1.smithi086.stdout:3/657: chown f4 9050296 1 2024-04-06T14:17:16.704 INFO:tasks.workunit.client.0.smithi032.stdout:7/934: creat d16/d92/db3/f148 x:0 0 0 2024-04-06T14:17:16.706 INFO:tasks.workunit.client.1.smithi086.stdout:9/533: rename d5/d58/f9f to d5/d13/d1a/d5f/d88/d14/d19/d25/d74/fae 0 2024-04-06T14:17:16.708 INFO:tasks.workunit.client.1.smithi086.stdout:3/658: unlink d5/d33/l39 0 2024-04-06T14:17:16.710 INFO:tasks.workunit.client.1.smithi086.stdout:9/534: readlink d5/d13/d1a/d5f/d88/d14/d19/d1b/l1f 0 2024-04-06T14:17:16.711 INFO:tasks.workunit.client.1.smithi086.stdout:2/633: dwrite d0/d20/d27/d37/d52/f5c [0,4194304] 0 2024-04-06T14:17:16.712 INFO:tasks.workunit.client.1.smithi086.stdout:3/659: creat d5/d36/d5c/fd6 x:0 0 0 2024-04-06T14:17:16.713 INFO:tasks.workunit.client.1.smithi086.stdout:1/669: creat db/d11/d1f/fd9 x:0 0 0 2024-04-06T14:17:16.714 INFO:tasks.workunit.client.1.smithi086.stdout:9/535: write d5/d13/d47/fa8 [1009696,84696] 0 2024-04-06T14:17:16.715 INFO:tasks.workunit.client.0.smithi032.stdout:0/666: sync 2024-04-06T14:17:16.715 INFO:tasks.workunit.client.0.smithi032.stdout:9/850: sync 2024-04-06T14:17:16.715 INFO:tasks.workunit.client.0.smithi032.stdout:5/795: sync 2024-04-06T14:17:16.715 INFO:tasks.workunit.client.0.smithi032.stdout:6/894: sync 2024-04-06T14:17:16.715 INFO:tasks.workunit.client.0.smithi032.stdout:2/757: sync 2024-04-06T14:17:16.715 INFO:tasks.workunit.client.0.smithi032.stdout:3/812: sync 2024-04-06T14:17:16.715 INFO:tasks.workunit.client.0.smithi032.stdout:1/932: sync 2024-04-06T14:17:16.717 INFO:tasks.workunit.client.1.smithi086.stdout:7/608: dwrite d7/d10/d11/d1c/d24/d4d/d5c/d66/f67 [0,4194304] 0 2024-04-06T14:17:16.718 INFO:tasks.workunit.client.1.smithi086.stdout:2/634: unlink d0/df/d48/dce/lda 0 2024-04-06T14:17:16.722 INFO:tasks.workunit.client.0.smithi032.stdout:0/667: chown d3/da/d25/d2b/d4a/fdb 112 1 2024-04-06T14:17:16.722 INFO:tasks.workunit.client.0.smithi032.stdout:5/796: chown d16/d18/f30 174051964 1 2024-04-06T14:17:16.723 INFO:tasks.workunit.client.1.smithi086.stdout:1/670: dread db/d13/f74 [0,4194304] 0 2024-04-06T14:17:16.724 INFO:tasks.workunit.client.0.smithi032.stdout:8/897: write d3/d8/d12/f3f [1746697,6761] 0 2024-04-06T14:17:16.724 INFO:tasks.workunit.client.0.smithi032.stdout:1/933: write d0/d41/d5b/d111/f119 [46341,5192] 0 2024-04-06T14:17:16.726 INFO:tasks.workunit.client.0.smithi032.stdout:7/935: dread d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee/f79 [0,4194304] 0 2024-04-06T14:17:16.726 INFO:tasks.workunit.client.0.smithi032.stdout:9/851: mkdir d2/d15/d55/d59/d93/d95/db8/d81/d4a/d10d 0 2024-04-06T14:17:16.727 INFO:tasks.workunit.client.0.smithi032.stdout:8/898: stat d3/d8/d7 0 2024-04-06T14:17:16.727 INFO:tasks.workunit.client.1.smithi086.stdout:2/635: fsync d0/d20/d27/d37/f98 0 2024-04-06T14:17:16.728 INFO:tasks.workunit.client.0.smithi032.stdout:3/813: creat d3/d65/d7d/d9c/dbc/f13d x:0 0 0 2024-04-06T14:17:16.729 INFO:tasks.workunit.client.0.smithi032.stdout:6/895: link d4/d5/f15 d4/d5/dc5/f12e 0 2024-04-06T14:17:16.731 INFO:tasks.workunit.client.0.smithi032.stdout:0/668: rename d3/da/d41/f43 to d3/da/d25/d57/fdf 0 2024-04-06T14:17:16.731 INFO:tasks.workunit.client.0.smithi032.stdout:3/814: chown d3/le9 16026298 1 2024-04-06T14:17:16.731 INFO:tasks.workunit.client.1.smithi086.stdout:7/609: creat d7/d10/d11/d1c/d24/d4d/daf/db5/da1/fce x:0 0 0 2024-04-06T14:17:16.732 INFO:tasks.workunit.client.1.smithi086.stdout:3/660: rmdir d5/d36/d5a/d6d/d72/d97 39 2024-04-06T14:17:16.733 INFO:tasks.workunit.client.1.smithi086.stdout:3/661: readlink d5/d36/d5a/d6d/d72/d64/lc4 0 2024-04-06T14:17:16.734 INFO:tasks.workunit.client.0.smithi032.stdout:5/797: truncate d16/d18/dae/dbb/ddc/f113 400528 0 2024-04-06T14:17:16.736 INFO:tasks.workunit.client.0.smithi032.stdout:6/896: dread d4/d5/d19/d25/f87 [4194304,4194304] 0 2024-04-06T14:17:16.737 INFO:tasks.workunit.client.1.smithi086.stdout:8/695: sync 2024-04-06T14:17:16.737 INFO:tasks.workunit.client.0.smithi032.stdout:1/934: symlink d0/d9/d1f/d72/l131 0 2024-04-06T14:17:16.740 INFO:tasks.workunit.client.1.smithi086.stdout:7/610: read d7/d10/d11/d80/f35 [4167553,94918] 0 2024-04-06T14:17:16.740 INFO:tasks.workunit.client.0.smithi032.stdout:0/669: mkdir d3/da/d25/d2b/d4a/d79/d97/de0 0 2024-04-06T14:17:16.740 INFO:tasks.workunit.client.1.smithi086.stdout:7/611: chown d7/d1d/c76 875543 1 2024-04-06T14:17:16.741 INFO:tasks.workunit.client.1.smithi086.stdout:3/662: rename d5/d36/d56/f74 to d5/d36/d5a/d6d/d72/d9a/fd7 0 2024-04-06T14:17:16.742 INFO:tasks.workunit.client.0.smithi032.stdout:0/670: readlink d3/da/d35/d4b/l4d 0 2024-04-06T14:17:16.743 INFO:tasks.workunit.client.1.smithi086.stdout:1/671: rmdir db/d11/d1a/d79/dce/dd2 0 2024-04-06T14:17:16.745 INFO:tasks.workunit.client.1.smithi086.stdout:8/696: mkdir d0/d71/d16/d39/d90/daf/df0 0 2024-04-06T14:17:16.745 INFO:tasks.workunit.client.0.smithi032.stdout:2/758: rename d0/fdb to d0/d30/d39/d59/d67/d74/d78/d57/d7c/de6/ffd 0 2024-04-06T14:17:16.746 INFO:tasks.workunit.client.0.smithi032.stdout:1/935: dwrite d0/d41/d59/fe9 [0,4194304] 0 2024-04-06T14:17:16.747 INFO:tasks.workunit.client.0.smithi032.stdout:1/936: fdatasync d0/d10/d16/db8/ff3 0 2024-04-06T14:17:16.750 INFO:tasks.workunit.client.0.smithi032.stdout:5/798: mknod d16/d58/d54/d94/db2/dcb/c11a 0 2024-04-06T14:17:16.750 INFO:tasks.workunit.client.0.smithi032.stdout:3/815: write d3/f7a [5591,101831] 0 2024-04-06T14:17:16.751 INFO:tasks.workunit.client.1.smithi086.stdout:2/636: write d0/d20/d27/d2b/d35/d36/f40 [4631353,83371] 0 2024-04-06T14:17:16.752 INFO:tasks.workunit.client.1.smithi086.stdout:9/536: dwrite d5/d13/d1a/d5f/d88/d14/d19/f5d [0,4194304] 0 2024-04-06T14:17:16.755 INFO:tasks.workunit.client.0.smithi032.stdout:8/899: dwrite d3/d8/f99 [0,4194304] 0 2024-04-06T14:17:16.757 INFO:tasks.workunit.client.0.smithi032.stdout:8/900: chown d3/d8/d3a/l92 44627 1 2024-04-06T14:17:16.760 INFO:tasks.workunit.client.0.smithi032.stdout:8/901: chown d3/d8/d12/d2f/d46/d125/f78 202215806 1 2024-04-06T14:17:16.765 INFO:tasks.workunit.client.0.smithi032.stdout:9/852: dwrite d2/d15/d55/d59/d93/d95/db8/d81/fab [0,4194304] 0 2024-04-06T14:17:16.767 INFO:tasks.workunit.client.0.smithi032.stdout:2/759: mknod d0/d4e/dc6/cfe 0 2024-04-06T14:17:16.767 INFO:tasks.workunit.client.1.smithi086.stdout:8/697: creat d0/d71/d16/ff1 x:0 0 0 2024-04-06T14:17:16.768 INFO:tasks.workunit.client.0.smithi032.stdout:6/897: mkdir d4/d5/dbf/d11a/d12f 0 2024-04-06T14:17:16.769 INFO:tasks.workunit.client.1.smithi086.stdout:1/672: rename db/d11/d1a/d2a/d9a/l84 to db/d11/d1f/lda 0 2024-04-06T14:17:16.770 INFO:tasks.workunit.client.0.smithi032.stdout:3/816: mkdir d3/d65/d7d/d9c/dbc/d13e 0 2024-04-06T14:17:16.771 INFO:tasks.workunit.client.0.smithi032.stdout:6/898: stat d4/d5/dbf/d61/da7/lab 0 2024-04-06T14:17:16.772 INFO:tasks.workunit.client.0.smithi032.stdout:2/760: chown d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/fe5 37 1 2024-04-06T14:17:16.773 INFO:tasks.workunit.client.0.smithi032.stdout:2/761: truncate d0/f24 9334085 0 2024-04-06T14:17:16.775 INFO:tasks.workunit.client.0.smithi032.stdout:6/899: write d4/d5/d19/d25/d67/fa4 [1523981,10371] 0 2024-04-06T14:17:16.775 INFO:tasks.workunit.client.0.smithi032.stdout:7/936: link d16/d20/d54/de4/d2d/d30/d3e/da7/db5/l10c d16/d20/d6e/d10e/d12b/l149 0 2024-04-06T14:17:16.776 INFO:tasks.workunit.client.0.smithi032.stdout:9/853: write d2/d15/d55/d59/d93/d95/fea [353598,17985] 0 2024-04-06T14:17:16.777 INFO:tasks.workunit.client.1.smithi086.stdout:2/637: rename d0/d20/d27/d2b/cc3 to d0/df/d48/d53/cdd 0 2024-04-06T14:17:16.778 INFO:tasks.workunit.client.0.smithi032.stdout:6/900: write d4/d5/dbf/f105 [1211062,16962] 0 2024-04-06T14:17:16.778 INFO:tasks.workunit.client.0.smithi032.stdout:0/671: rename d3/da/d25/d2b/d4a/fa9 to d3/da/fe1 0 2024-04-06T14:17:16.779 INFO:tasks.workunit.client.1.smithi086.stdout:1/673: dwrite db/d13/f86 [0,4194304] 0 2024-04-06T14:17:16.782 INFO:tasks.workunit.client.1.smithi086.stdout:8/698: rename d0/d71/d22/d43/d7f/fb1 to d0/d71/d22/d43/d6d/d75/d78/ff2 0 2024-04-06T14:17:16.784 INFO:tasks.workunit.client.1.smithi086.stdout:3/663: write d5/d26/d7f/d59/f8a [1101064,5502] 0 2024-04-06T14:17:16.784 INFO:tasks.workunit.client.0.smithi032.stdout:0/672: dread d3/da/f53 [0,4194304] 0 2024-04-06T14:17:16.786 INFO:tasks.workunit.client.1.smithi086.stdout:7/612: write d7/d10/d11/d1c/f2e [3522878,69350] 0 2024-04-06T14:17:16.787 INFO:tasks.workunit.client.0.smithi032.stdout:6/901: read d4/d5/d19/f7d [3733716,44978] 0 2024-04-06T14:17:16.789 INFO:tasks.workunit.client.0.smithi032.stdout:0/673: mkdir d3/da/d25/d2b/d4a/d79/d97/de2 0 2024-04-06T14:17:16.789 INFO:tasks.workunit.client.1.smithi086.stdout:1/674: mkdir db/d11/d1f/d5c/ddb 0 2024-04-06T14:17:16.790 INFO:tasks.workunit.client.0.smithi032.stdout:6/902: chown d4/d5/d14/d8d 371041857 1 2024-04-06T14:17:16.791 INFO:tasks.workunit.client.0.smithi032.stdout:7/937: dwrite d16/d20/d7a/da6/d117/f133 [0,4194304] 0 2024-04-06T14:17:16.793 INFO:tasks.workunit.client.1.smithi086.stdout:2/638: dwrite d0/df/d48/d53/f54 [4194304,4194304] 0 2024-04-06T14:17:16.794 INFO:tasks.workunit.client.0.smithi032.stdout:1/937: truncate d0/f2c 3047731 0 2024-04-06T14:17:16.795 INFO:tasks.workunit.client.0.smithi032.stdout:0/674: symlink d3/da/d35/le3 0 2024-04-06T14:17:16.795 INFO:tasks.workunit.client.1.smithi086.stdout:1/675: chown db/d11/d1a/d2a/d9a/d68/daf/dc2 4 1 2024-04-06T14:17:16.799 INFO:tasks.workunit.client.1.smithi086.stdout:1/676: fsync db/f7b 0 2024-04-06T14:17:16.801 INFO:tasks.workunit.client.0.smithi032.stdout:7/938: stat d16/d20/d54/de4/d2d/d30/d3e/l87 0 2024-04-06T14:17:16.801 INFO:tasks.workunit.client.1.smithi086.stdout:3/664: readlink d5/d26/d7f/l66 0 2024-04-06T14:17:16.803 INFO:tasks.workunit.client.0.smithi032.stdout:1/938: dread d0/d41/d59/fd8 [0,4194304] 0 2024-04-06T14:17:16.804 INFO:tasks.workunit.client.0.smithi032.stdout:0/675: mknod d3/da/d35/d8b/d9c/ce4 0 2024-04-06T14:17:16.805 INFO:tasks.workunit.client.1.smithi086.stdout:1/677: stat db/d11/d42/d53/fcf 0 2024-04-06T14:17:16.806 INFO:tasks.workunit.client.1.smithi086.stdout:7/613: link d7/d1d/f38 d7/d10/d11/d1c/d24/d4d/daf/db5/fcf 0 2024-04-06T14:17:16.807 INFO:tasks.workunit.client.1.smithi086.stdout:3/665: creat d5/d26/d4f/db4/fd8 x:0 0 0 2024-04-06T14:17:16.809 INFO:tasks.workunit.client.0.smithi032.stdout:0/676: write d3/f20 [4050295,118663] 0 2024-04-06T14:17:16.810 INFO:tasks.workunit.client.1.smithi086.stdout:2/639: dwrite d0/d20/d27/d37/d52/f5c [4194304,4194304] 0 2024-04-06T14:17:16.810 INFO:tasks.workunit.client.0.smithi032.stdout:8/902: dwrite d3/d8/d12/d83/d89/da0/fb8 [0,4194304] 0 2024-04-06T14:17:16.810 INFO:tasks.workunit.client.1.smithi086.stdout:1/678: fdatasync db/d11/d42/d53/f64 0 2024-04-06T14:17:16.811 INFO:tasks.workunit.client.1.smithi086.stdout:8/699: dwrite d0/f5 [0,4194304] 0 2024-04-06T14:17:16.811 INFO:tasks.workunit.client.1.smithi086.stdout:1/679: stat db/c49 0 2024-04-06T14:17:16.813 INFO:tasks.workunit.client.1.smithi086.stdout:7/614: dread - d7/d10/d11/d1c/d24/d4d/fb7 zero size 2024-04-06T14:17:16.813 INFO:tasks.workunit.client.0.smithi032.stdout:6/903: dwrite d4/d69/f114 [0,4194304] 0 2024-04-06T14:17:16.815 INFO:tasks.workunit.client.0.smithi032.stdout:1/939: read d0/d10/d16/d3f/f5d [323168,117783] 0 2024-04-06T14:17:16.816 INFO:tasks.workunit.client.1.smithi086.stdout:7/615: chown d7/d10/d11/d1c/d24 1966027 1 2024-04-06T14:17:16.818 INFO:tasks.workunit.client.1.smithi086.stdout:1/680: creat db/d11/d87/fdc x:0 0 0 2024-04-06T14:17:16.818 INFO:tasks.workunit.client.1.smithi086.stdout:8/700: creat d0/d71/d22/d43/d6d/ff3 x:0 0 0 2024-04-06T14:17:16.818 INFO:tasks.workunit.client.0.smithi032.stdout:2/762: dwrite d0/db2/dfc/f48 [0,4194304] 0 2024-04-06T14:17:16.825 INFO:tasks.workunit.client.0.smithi032.stdout:1/940: symlink d0/da/d3e/l132 0 2024-04-06T14:17:16.826 INFO:tasks.workunit.client.0.smithi032.stdout:2/763: symlink d0/d30/lff 0 2024-04-06T14:17:16.830 INFO:tasks.workunit.client.0.smithi032.stdout:1/941: dread d0/d10/d16/ffd [0,4194304] 0 2024-04-06T14:17:16.830 INFO:tasks.workunit.client.1.smithi086.stdout:3/666: rmdir d5/d26/d7f/dae 39 2024-04-06T14:17:16.831 INFO:tasks.workunit.client.0.smithi032.stdout:1/942: readlink d0/d41/d5b/l97 0 2024-04-06T14:17:16.832 INFO:tasks.workunit.client.0.smithi032.stdout:8/903: link d3/d8/d7/f16 d3/d8/d12/d2f/d4a/d60/d90/dc5/dd7/f12f 0 2024-04-06T14:17:16.833 INFO:tasks.workunit.client.0.smithi032.stdout:1/943: dread - d0/d9/d1f/d7c/dfa/ffe zero size 2024-04-06T14:17:16.834 INFO:tasks.workunit.client.1.smithi086.stdout:3/667: readlink d5/l38 0 2024-04-06T14:17:16.835 INFO:tasks.workunit.client.0.smithi032.stdout:0/677: link d3/da/d25/d2b/d4a/d79/d7c/l7e d3/da/d41/le5 0 2024-04-06T14:17:16.836 INFO:tasks.workunit.client.0.smithi032.stdout:6/904: dread d4/d5/d8/d2b/dfb/f117 [0,4194304] 0 2024-04-06T14:17:16.836 INFO:tasks.workunit.client.0.smithi032.stdout:1/944: chown d0/d9/d1f/d7c/d8f/dc5 9379 1 2024-04-06T14:17:16.837 INFO:tasks.workunit.client.1.smithi086.stdout:2/640: dwrite d0/df/f18 [4194304,4194304] 0 2024-04-06T14:17:16.837 INFO:tasks.workunit.client.1.smithi086.stdout:8/701: readlink d0/d71/d16/d52/d5f/lbe 0 2024-04-06T14:17:16.839 INFO:tasks.workunit.client.0.smithi032.stdout:8/904: creat d3/d8/d12/d83/d89/da0/dd2/f130 x:0 0 0 2024-04-06T14:17:16.839 INFO:tasks.workunit.client.0.smithi032.stdout:0/678: readlink d3/da/d25/d2b/d4a/d79/d7c/lbb 0 2024-04-06T14:17:16.841 INFO:tasks.workunit.client.0.smithi032.stdout:0/679: chown d3/da/l17 366 1 2024-04-06T14:17:16.841 INFO:tasks.workunit.client.0.smithi032.stdout:6/905: read - d4/d5/dc5/d82/f126 zero size 2024-04-06T14:17:16.842 INFO:tasks.workunit.client.1.smithi086.stdout:2/641: creat d0/df/d48/d7d/fde x:0 0 0 2024-04-06T14:17:16.843 INFO:tasks.workunit.client.0.smithi032.stdout:1/945: readlink d0/d10/d16/l21 0 2024-04-06T14:17:16.844 INFO:tasks.workunit.client.1.smithi086.stdout:3/668: dread d5/d33/f52 [4194304,4194304] 0 2024-04-06T14:17:16.845 INFO:tasks.workunit.client.0.smithi032.stdout:8/905: dread - d3/d8/d12/d2f/d4a/d60/d90/dc5/f110 zero size 2024-04-06T14:17:16.849 INFO:tasks.workunit.client.0.smithi032.stdout:0/680: symlink d3/da/d25/d2b/d80/db3/db7/le6 0 2024-04-06T14:17:16.852 INFO:tasks.workunit.client.0.smithi032.stdout:2/764: dread d0/fb [0,4194304] 0 2024-04-06T14:17:16.854 INFO:tasks.workunit.client.0.smithi032.stdout:0/681: read d3/f5a [840471,87965] 0 2024-04-06T14:17:16.855 INFO:tasks.workunit.client.0.smithi032.stdout:1/946: getdents d0 0 2024-04-06T14:17:16.856 INFO:tasks.workunit.client.1.smithi086.stdout:1/681: dread db/d11/d1a/d2a/f47 [0,4194304] 0 2024-04-06T14:17:16.857 INFO:tasks.workunit.client.0.smithi032.stdout:0/682: stat d3/da/d35/l60 0 2024-04-06T14:17:16.858 INFO:tasks.workunit.client.0.smithi032.stdout:6/906: dwrite d4/d5/dbf/f105 [0,4194304] 0 2024-04-06T14:17:16.858 INFO:tasks.workunit.client.0.smithi032.stdout:1/947: stat d0/d10/d16/d3f/def 0 2024-04-06T14:17:16.859 INFO:tasks.workunit.client.0.smithi032.stdout:7/939: dread d16/d20/d54/de4/d2d/d30/f36 [0,4194304] 0 2024-04-06T14:17:16.860 INFO:tasks.workunit.client.0.smithi032.stdout:0/683: read - d3/da/d35/fbc zero size 2024-04-06T14:17:16.860 INFO:tasks.workunit.client.0.smithi032.stdout:2/765: write d0/d30/d39/d59/d67/d74/d78/d57/fde [866520,128423] 0 2024-04-06T14:17:16.864 INFO:tasks.workunit.client.0.smithi032.stdout:6/907: mkdir d4/d5/d130 0 2024-04-06T14:17:16.865 INFO:tasks.workunit.client.0.smithi032.stdout:0/684: stat d3/da/d25/d2b/d4a/l67 0 2024-04-06T14:17:16.867 INFO:tasks.workunit.client.0.smithi032.stdout:7/940: creat d16/d20/d6e/d138/f14a x:0 0 0 2024-04-06T14:17:16.868 INFO:tasks.workunit.client.0.smithi032.stdout:8/906: dwrite d3/d8/d12/d2f/d4a/f8b [0,4194304] 0 2024-04-06T14:17:16.876 INFO:tasks.workunit.client.0.smithi032.stdout:6/908: read d4/d5/d19/f7d [1960932,84866] 0 2024-04-06T14:17:16.876 INFO:tasks.workunit.client.0.smithi032.stdout:1/948: link d0/d9/d94/fce d0/d10/d78/f133 0 2024-04-06T14:17:16.877 INFO:tasks.workunit.client.1.smithi086.stdout:4/481: sync 2024-04-06T14:17:16.877 INFO:tasks.workunit.client.1.smithi086.stdout:6/742: sync 2024-04-06T14:17:16.877 INFO:tasks.workunit.client.1.smithi086.stdout:8/702: symlink d0/d71/d22/d65/d6e/d98/ddb/lf4 0 2024-04-06T14:17:16.878 INFO:tasks.workunit.client.0.smithi032.stdout:8/907: readlink d3/l6 0 2024-04-06T14:17:16.879 INFO:tasks.workunit.client.1.smithi086.stdout:6/743: chown d6/d9/d43/fc5 5611 1 2024-04-06T14:17:16.879 INFO:tasks.workunit.client.1.smithi086.stdout:3/669: stat d5/d26/d7f/dae 0 2024-04-06T14:17:16.880 INFO:tasks.workunit.client.1.smithi086.stdout:1/682: creat db/d13/d55/da3/fdd x:0 0 0 2024-04-06T14:17:16.880 INFO:tasks.workunit.client.0.smithi032.stdout:0/685: creat d3/da/d25/d2b/d4a/d79/d97/de0/fe7 x:0 0 0 2024-04-06T14:17:16.881 INFO:tasks.workunit.client.0.smithi032.stdout:6/909: read d4/d5/d19/d25/d67/fa4 [697005,111803] 0 2024-04-06T14:17:16.882 INFO:tasks.workunit.client.0.smithi032.stdout:2/766: dread d0/d30/d39/d59/d67/d74/d78/d57/fd1 [0,4194304] 0 2024-04-06T14:17:16.882 INFO:tasks.workunit.client.1.smithi086.stdout:4/482: unlink d0/d15/l60 0 2024-04-06T14:17:16.883 INFO:tasks.workunit.client.0.smithi032.stdout:7/941: creat d16/d20/d54/de4/de0/df9/f14b x:0 0 0 2024-04-06T14:17:16.884 INFO:tasks.workunit.client.1.smithi086.stdout:2/642: truncate d0/d20/d27/d2b/faf 687143 0 2024-04-06T14:17:16.884 INFO:tasks.workunit.client.1.smithi086.stdout:3/670: mkdir d5/d33/d5d/dd9 0 2024-04-06T14:17:16.884 INFO:tasks.workunit.client.0.smithi032.stdout:8/908: mknod d3/d8/d12/d2f/d46/def/d128/d107/c131 0 2024-04-06T14:17:16.885 INFO:tasks.workunit.client.1.smithi086.stdout:6/744: mknod d6/d13/d1a/d64/cfa 0 2024-04-06T14:17:16.886 INFO:tasks.workunit.client.0.smithi032.stdout:6/910: mknod d4/d5/dd8/c131 0 2024-04-06T14:17:16.886 INFO:tasks.workunit.client.1.smithi086.stdout:7/616: getdents d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9 0 2024-04-06T14:17:16.887 INFO:tasks.workunit.client.1.smithi086.stdout:4/483: symlink d0/d15/d2b/l96 0 2024-04-06T14:17:16.889 INFO:tasks.workunit.client.0.smithi032.stdout:0/686: rename d3/da/d41/lc6 to d3/dc2/dd8/le8 0 2024-04-06T14:17:16.890 INFO:tasks.workunit.client.1.smithi086.stdout:1/683: mknod db/d13/cde 0 2024-04-06T14:17:16.890 INFO:tasks.workunit.client.0.smithi032.stdout:2/767: dread d0/fb [0,4194304] 0 2024-04-06T14:17:16.890 INFO:tasks.workunit.client.0.smithi032.stdout:6/911: unlink d4/d5/d18/f49 0 2024-04-06T14:17:16.891 INFO:tasks.workunit.client.0.smithi032.stdout:2/768: write d0/d5a/f41 [1287519,81449] 0 2024-04-06T14:17:16.892 INFO:tasks.workunit.client.1.smithi086.stdout:7/617: creat d7/d10/d11/d21/db0/fd0 x:0 0 0 2024-04-06T14:17:16.892 INFO:tasks.workunit.client.0.smithi032.stdout:7/942: dread d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee/f79 [0,4194304] 0 2024-04-06T14:17:16.892 INFO:tasks.workunit.client.1.smithi086.stdout:3/671: mknod d5/df/d9e/db5/cda 0 2024-04-06T14:17:16.893 INFO:tasks.workunit.client.0.smithi032.stdout:6/912: write d4/d5/d19/f102 [126385,81476] 0 2024-04-06T14:17:16.894 INFO:tasks.workunit.client.1.smithi086.stdout:2/643: truncate d0/df/d48/d53/d76/fb9 396203 0 2024-04-06T14:17:16.894 INFO:tasks.workunit.client.0.smithi032.stdout:0/687: creat d3/da/d35/d8b/fe9 x:0 0 0 2024-04-06T14:17:16.895 INFO:tasks.workunit.client.1.smithi086.stdout:4/484: mknod d0/d15/c97 0 2024-04-06T14:17:16.897 INFO:tasks.workunit.client.0.smithi032.stdout:8/909: symlink d3/d8/d3a/d7b/db9/df8/d106/l132 0 2024-04-06T14:17:16.898 INFO:tasks.workunit.client.1.smithi086.stdout:6/745: creat d6/d13/d1a/ffb x:0 0 0 2024-04-06T14:17:16.898 INFO:tasks.workunit.client.1.smithi086.stdout:7/618: chown d7/d10/d11/d1c/d24/c4f 15270636 1 2024-04-06T14:17:16.898 INFO:tasks.workunit.client.0.smithi032.stdout:6/913: creat d4/d5/d19/d25/d67/f132 x:0 0 0 2024-04-06T14:17:16.899 INFO:tasks.workunit.client.1.smithi086.stdout:7/619: readlink d7/d10/d11/d80/l5f 0 2024-04-06T14:17:16.901 INFO:tasks.workunit.client.1.smithi086.stdout:6/746: unlink d6/d13/d42/d52/c66 0 2024-04-06T14:17:16.902 INFO:tasks.workunit.client.0.smithi032.stdout:2/769: rename d0/d5a/f1c to d0/d30/d39/d59/d67/d74/d78/d57/d7c/f100 0 2024-04-06T14:17:16.903 INFO:tasks.workunit.client.0.smithi032.stdout:0/688: creat d3/da/d35/d8b/fea x:0 0 0 2024-04-06T14:17:16.903 INFO:tasks.workunit.client.1.smithi086.stdout:2/644: dread d0/f8e [0,4194304] 0 2024-04-06T14:17:16.903 INFO:tasks.workunit.client.1.smithi086.stdout:8/703: dread d0/d71/d16/d52/f69 [0,4194304] 0 2024-04-06T14:17:16.904 INFO:tasks.workunit.client.1.smithi086.stdout:0/667: sync 2024-04-06T14:17:16.904 INFO:tasks.workunit.client.1.smithi086.stdout:5/508: sync 2024-04-06T14:17:16.904 INFO:tasks.workunit.client.0.smithi032.stdout:1/949: dwrite d0/d41/d59/f8e [0,4194304] 0 2024-04-06T14:17:16.905 INFO:tasks.workunit.client.0.smithi032.stdout:6/914: creat d4/d5/d8/d2b/f133 x:0 0 0 2024-04-06T14:17:16.906 INFO:tasks.workunit.client.0.smithi032.stdout:1/950: dread - d0/d9/fe5 zero size 2024-04-06T14:17:16.907 INFO:tasks.workunit.client.1.smithi086.stdout:3/672: getdents d5/d26/d7f/d59 0 2024-04-06T14:17:16.907 INFO:tasks.workunit.client.0.smithi032.stdout:2/770: creat d0/d4e/f101 x:0 0 0 2024-04-06T14:17:16.907 INFO:tasks.workunit.client.1.smithi086.stdout:8/704: read d0/d3b/f46 [1912732,74653] 0 2024-04-06T14:17:16.910 INFO:tasks.workunit.client.1.smithi086.stdout:7/620: symlink d7/d10/d11/d1c/d24/d2c/d69/d83/ld1 0 2024-04-06T14:17:16.912 INFO:tasks.workunit.client.0.smithi032.stdout:7/943: rename d16/d20/d63/d9d/df2/df5/cdf to d16/d20/d63/d9d/df2/df5/dfd/c14c 0 2024-04-06T14:17:16.912 INFO:tasks.workunit.client.1.smithi086.stdout:0/668: symlink d0/d15/ldc 0 2024-04-06T14:17:16.913 INFO:tasks.workunit.client.1.smithi086.stdout:6/747: dwrite d6/d9/f17 [0,4194304] 0 2024-04-06T14:17:16.914 INFO:tasks.workunit.client.0.smithi032.stdout:6/915: symlink d4/d5/dc5/l134 0 2024-04-06T14:17:16.916 INFO:tasks.workunit.client.1.smithi086.stdout:5/509: mknod d0/d16/d18/d21/d23/d28/ca0 0 2024-04-06T14:17:16.916 INFO:tasks.workunit.client.0.smithi032.stdout:1/951: mkdir d0/d9/df2/d134 0 2024-04-06T14:17:16.917 INFO:tasks.workunit.client.1.smithi086.stdout:1/684: write db/d11/d1a/d2a/d9a/d68/d6e/f8d [1013780,71988] 0 2024-04-06T14:17:16.920 INFO:tasks.workunit.client.0.smithi032.stdout:8/910: dread d3/d8/f87 [0,4194304] 0 2024-04-06T14:17:16.921 INFO:tasks.workunit.client.1.smithi086.stdout:0/669: symlink d0/d15/d4e/da3/d44/d5d/ldd 0 2024-04-06T14:17:16.923 INFO:tasks.workunit.client.1.smithi086.stdout:6/748: read d6/d13/d1a/f8f [1715200,115449] 0 2024-04-06T14:17:16.924 INFO:tasks.workunit.client.1.smithi086.stdout:0/670: stat d0/d15/ldc 0 2024-04-06T14:17:16.924 INFO:tasks.workunit.client.0.smithi032.stdout:1/952: write d0/d10/d16/d3f/d98/f121 [437695,95224] 0 2024-04-06T14:17:16.924 INFO:tasks.workunit.client.1.smithi086.stdout:2/645: creat d0/d20/d27/d2b/d35/d36/dbe/fdf x:0 0 0 2024-04-06T14:17:16.926 INFO:tasks.workunit.client.1.smithi086.stdout:3/673: dread d5/f44 [0,4194304] 0 2024-04-06T14:17:16.926 INFO:tasks.workunit.client.1.smithi086.stdout:7/621: mkdir d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/dd2 0 2024-04-06T14:17:16.926 INFO:tasks.workunit.client.0.smithi032.stdout:7/944: unlink d16/l98 0 2024-04-06T14:17:16.927 INFO:tasks.workunit.client.1.smithi086.stdout:5/510: symlink d0/d16/d18/d21/d23/d28/d94/d80/la1 0 2024-04-06T14:17:16.927 INFO:tasks.workunit.client.0.smithi032.stdout:8/911: creat d3/d8/d12/d83/d89/da0/dd2/dfa/f133 x:0 0 0 2024-04-06T14:17:16.927 INFO:tasks.workunit.client.0.smithi032.stdout:6/916: dwrite d4/d5/d19/f102 [0,4194304] 0 2024-04-06T14:17:16.930 INFO:tasks.workunit.client.1.smithi086.stdout:7/622: dread - d7/d10/d11/f8f zero size 2024-04-06T14:17:16.930 INFO:tasks.workunit.client.0.smithi032.stdout:1/953: readlink d0/d10/d66/l76 0 2024-04-06T14:17:16.930 INFO:tasks.workunit.client.0.smithi032.stdout:8/912: chown d3/d8/d12/d2f/l36 10 1 2024-04-06T14:17:16.931 INFO:tasks.workunit.client.0.smithi032.stdout:2/771: fsync d0/d4e/f101 0 2024-04-06T14:17:16.932 INFO:tasks.workunit.client.1.smithi086.stdout:0/671: dread - d0/d1/d2/d1a/d51/f86 zero size 2024-04-06T14:17:16.934 INFO:tasks.workunit.client.0.smithi032.stdout:6/917: rmdir d4/d5/d19/d25/d67 39 2024-04-06T14:17:16.936 INFO:tasks.workunit.client.1.smithi086.stdout:2/646: fdatasync d0/d20/d27/d2b/d35/d36/d4d/d6f/f77 0 2024-04-06T14:17:16.936 INFO:tasks.workunit.client.0.smithi032.stdout:1/954: creat d0/d10/d78/f135 x:0 0 0 2024-04-06T14:17:16.938 INFO:tasks.workunit.client.1.smithi086.stdout:4/485: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:17:16.938 INFO:tasks.workunit.client.1.smithi086.stdout:7/623: symlink d7/d10/d11/d80/ld3 0 2024-04-06T14:17:16.939 INFO:tasks.workunit.client.0.smithi032.stdout:6/918: fsync d4/d5/d19/d25/d9c/f128 0 2024-04-06T14:17:16.941 INFO:tasks.workunit.client.0.smithi032.stdout:1/955: write d0/f130 [289210,93047] 0 2024-04-06T14:17:16.941 INFO:tasks.workunit.client.1.smithi086.stdout:6/749: symlink d6/d13/d1a/d22/dcf/db2/dee/lfc 0 2024-04-06T14:17:16.942 INFO:tasks.workunit.client.0.smithi032.stdout:0/689: dwrite d3/f5a [0,4194304] 0 2024-04-06T14:17:16.943 INFO:tasks.workunit.client.1.smithi086.stdout:5/511: fsync d0/d16/d18/d35/f5b 0 2024-04-06T14:17:16.943 INFO:tasks.workunit.client.1.smithi086.stdout:9/537: sync 2024-04-06T14:17:16.946 INFO:tasks.workunit.client.0.smithi032.stdout:1/956: rename d0 to d0/d9/d94/dc2/d136 22 2024-04-06T14:17:16.948 INFO:tasks.workunit.client.0.smithi032.stdout:6/919: fsync d4/d5/dbf/f37 0 2024-04-06T14:17:16.948 INFO:tasks.workunit.client.1.smithi086.stdout:1/685: dwrite db/d11/d42/d53/fcf [0,4194304] 0 2024-04-06T14:17:16.950 INFO:tasks.workunit.client.1.smithi086.stdout:0/672: symlink d0/d15/d4e/lde 0 2024-04-06T14:17:16.950 INFO:tasks.workunit.client.0.smithi032.stdout:2/772: mkdir d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102 0 2024-04-06T14:17:16.950 INFO:tasks.workunit.client.0.smithi032.stdout:0/690: symlink d3/da/d25/d57/leb 0 2024-04-06T14:17:16.952 INFO:tasks.workunit.client.0.smithi032.stdout:6/920: fdatasync d4/d5/d14/fb3 0 2024-04-06T14:17:16.953 INFO:tasks.workunit.client.1.smithi086.stdout:7/624: fsync d7/d1d/fc 0 2024-04-06T14:17:16.955 INFO:tasks.workunit.client.0.smithi032.stdout:1/957: creat d0/da/d3e/d75/dd7/f137 x:0 0 0 2024-04-06T14:17:16.958 INFO:tasks.workunit.client.0.smithi032.stdout:7/945: dread d16/d20/d54/de4/d2d/f75 [0,4194304] 0 2024-04-06T14:17:16.960 INFO:tasks.workunit.client.0.smithi032.stdout:1/958: chown d0/d10/d16/f6d 853860 1 2024-04-06T14:17:16.960 INFO:tasks.workunit.client.0.smithi032.stdout:6/921: fdatasync d4/d5/dbf/f8c 0 2024-04-06T14:17:16.961 INFO:tasks.workunit.client.0.smithi032.stdout:2/773: rename d0/d4e/df8 to d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/d103 0 2024-04-06T14:17:16.961 INFO:tasks.workunit.client.1.smithi086.stdout:7/625: chown d7/d10/c8b 13198 1 2024-04-06T14:17:16.961 INFO:tasks.workunit.client.1.smithi086.stdout:1/686: rename db/fd5 to db/d11/d42/d53/fdf 0 2024-04-06T14:17:16.961 INFO:tasks.workunit.client.1.smithi086.stdout:7/626: chown d7/d10/d11/d1c/d24/d4d/fb7 0 1 2024-04-06T14:17:16.962 INFO:tasks.workunit.client.1.smithi086.stdout:9/538: getdents d5/d13/d1a/d5f 0 2024-04-06T14:17:16.963 INFO:tasks.workunit.client.0.smithi032.stdout:7/946: rmdir d16/d92 39 2024-04-06T14:17:16.963 INFO:tasks.workunit.client.1.smithi086.stdout:4/486: creat d0/d2c/d38/d54/d83/f98 x:0 0 0 2024-04-06T14:17:16.964 INFO:tasks.workunit.client.0.smithi032.stdout:0/691: dread d3/da/d25/f94 [4194304,4194304] 0 2024-04-06T14:17:16.966 INFO:tasks.workunit.client.1.smithi086.stdout:9/539: write f0 [5191290,120308] 0 2024-04-06T14:17:16.966 INFO:tasks.workunit.client.1.smithi086.stdout:8/705: dwrite d0/d71/d16/d52/dd0/ddc/de7/f5d [0,4194304] 0 2024-04-06T14:17:16.967 INFO:tasks.workunit.client.1.smithi086.stdout:0/673: getdents d0/d15/d4e/da3/d73 0 2024-04-06T14:17:16.969 INFO:tasks.workunit.client.0.smithi032.stdout:7/947: dread - d16/d20/d54/de4/fb2 zero size 2024-04-06T14:17:16.969 INFO:tasks.workunit.client.0.smithi032.stdout:1/959: dwrite d0/d41/d59/f8e [0,4194304] 0 2024-04-06T14:17:16.969 INFO:tasks.workunit.client.1.smithi086.stdout:7/627: readlink d7/d10/d11/d1c/l31 0 2024-04-06T14:17:16.970 INFO:tasks.workunit.client.1.smithi086.stdout:3/674: dread d5/f77 [0,4194304] 0 2024-04-06T14:17:16.971 INFO:tasks.workunit.client.0.smithi032.stdout:6/922: link d4/d5/dbf/d51/f7b d4/d5/dbf/d61/db9/f135 0 2024-04-06T14:17:16.971 INFO:tasks.workunit.client.0.smithi032.stdout:0/692: symlink d3/da/d25/lec 0 2024-04-06T14:17:16.972 INFO:tasks.workunit.client.1.smithi086.stdout:1/687: link db/d11/d42/d53/fd4 db/d11/fe0 0 2024-04-06T14:17:16.973 INFO:tasks.workunit.client.1.smithi086.stdout:7/628: stat d7/d10/d11/d1c/d24/d4d/daf/db5/d7a/c7c 0 2024-04-06T14:17:16.974 INFO:tasks.workunit.client.1.smithi086.stdout:7/629: stat d7/d1d 0 2024-04-06T14:17:16.974 INFO:tasks.workunit.client.0.smithi032.stdout:6/923: write d4/d5/d28/d65/fff [486292,77964] 0 2024-04-06T14:17:16.975 INFO:tasks.workunit.client.1.smithi086.stdout:3/675: fsync d5/f3e 0 2024-04-06T14:17:16.975 INFO:tasks.workunit.client.1.smithi086.stdout:0/674: write d0/d1/d4/de/fb0 [1371924,124804] 0 2024-04-06T14:17:16.976 INFO:tasks.workunit.client.0.smithi032.stdout:7/948: mknod d16/d20/d6e/d137/c14d 0 2024-04-06T14:17:16.976 INFO:tasks.workunit.client.1.smithi086.stdout:9/540: symlink d5/d58/laf 0 2024-04-06T14:17:16.977 INFO:tasks.workunit.client.1.smithi086.stdout:4/487: truncate d0/d2c/d30/f3b 1444965 0 2024-04-06T14:17:16.977 INFO:tasks.workunit.client.1.smithi086.stdout:3/676: stat d5/d33/f47 0 2024-04-06T14:17:16.978 INFO:tasks.workunit.client.1.smithi086.stdout:1/688: truncate db/d13/d55/da3/fc4 395453 0 2024-04-06T14:17:16.979 INFO:tasks.workunit.client.0.smithi032.stdout:2/774: dread d0/d30/f4d [0,4194304] 0 2024-04-06T14:17:16.979 INFO:tasks.workunit.client.1.smithi086.stdout:0/675: readlink d0/d15/d4e/da3/l77 0 2024-04-06T14:17:16.981 INFO:tasks.workunit.client.1.smithi086.stdout:7/630: symlink d7/d10/ld4 0 2024-04-06T14:17:16.982 INFO:tasks.workunit.client.0.smithi032.stdout:6/924: fsync d4/d5/d8/d4b/d7c/fcb 0 2024-04-06T14:17:16.982 INFO:tasks.workunit.client.1.smithi086.stdout:9/541: symlink d5/d13/lb0 0 2024-04-06T14:17:16.985 INFO:tasks.workunit.client.0.smithi032.stdout:0/693: link d3/da/d25/d2b/d80/db3/ldd d3/da/d35/d8b/led 0 2024-04-06T14:17:16.986 INFO:tasks.workunit.client.1.smithi086.stdout:1/689: dread - db/d11/d1f/fbc zero size 2024-04-06T14:17:16.988 INFO:tasks.workunit.client.1.smithi086.stdout:0/676: fsync d0/d15/d26/d33/f57 0 2024-04-06T14:17:16.991 INFO:tasks.workunit.client.1.smithi086.stdout:9/542: creat d5/d13/d1a/d8f/fb1 x:0 0 0 2024-04-06T14:17:16.993 INFO:tasks.workunit.client.0.smithi032.stdout:0/694: read d3/da/f56 [1629945,62000] 0 2024-04-06T14:17:16.993 INFO:tasks.workunit.client.1.smithi086.stdout:6/750: truncate d6/d13/f5d 320095 0 2024-04-06T14:17:16.994 INFO:tasks.workunit.client.1.smithi086.stdout:6/751: chown d6/d9/c4f 120837 1 2024-04-06T14:17:16.995 INFO:tasks.workunit.client.0.smithi032.stdout:0/695: chown d3/l72 19 1 2024-04-06T14:17:16.996 INFO:tasks.workunit.client.1.smithi086.stdout:5/512: dread d0/d19/f4b [0,4194304] 0 2024-04-06T14:17:16.996 INFO:tasks.workunit.client.1.smithi086.stdout:9/543: stat d5/d13/d1a/c62 0 2024-04-06T14:17:16.998 INFO:tasks.workunit.client.0.smithi032.stdout:2/775: dwrite d0/d30/d39/d59/d67/d74/fe8 [0,4194304] 0 2024-04-06T14:17:16.999 INFO:tasks.workunit.client.1.smithi086.stdout:1/690: write db/d11/d24/f60 [1629230,113817] 0 2024-04-06T14:17:17.004 INFO:tasks.workunit.client.0.smithi032.stdout:7/949: dwrite d16/d20/d6e/d10e/f12e [0,4194304] 0 2024-04-06T14:17:17.004 INFO:tasks.workunit.client.1.smithi086.stdout:8/706: dread d0/f34 [0,4194304] 0 2024-04-06T14:17:17.005 INFO:tasks.workunit.client.1.smithi086.stdout:7/631: dwrite d7/d10/d11/fbe [0,4194304] 0 2024-04-06T14:17:17.006 INFO:tasks.workunit.client.1.smithi086.stdout:6/752: mknod d6/d13/d1a/d64/de2/cfd 0 2024-04-06T14:17:17.008 INFO:tasks.workunit.client.0.smithi032.stdout:0/696: symlink d3/da/d25/d57/db5/dc9/lee 0 2024-04-06T14:17:17.009 INFO:tasks.workunit.client.0.smithi032.stdout:6/925: dread d4/d5/d28/d54/f11d [0,4194304] 0 2024-04-06T14:17:17.010 INFO:tasks.workunit.client.1.smithi086.stdout:2/647: dwrite d0/f1e [4194304,4194304] 0 2024-04-06T14:17:17.011 INFO:tasks.workunit.client.0.smithi032.stdout:6/926: chown d4/d5/dbf/c7e 218 1 2024-04-06T14:17:17.012 INFO:tasks.workunit.client.1.smithi086.stdout:0/677: truncate d0/d1/d4/d7/f27 924984 0 2024-04-06T14:17:17.012 INFO:tasks.workunit.client.1.smithi086.stdout:5/513: rmdir d0/d16/d18/d21/d23/d28/d94 39 2024-04-06T14:17:17.013 INFO:tasks.workunit.client.0.smithi032.stdout:2/776: creat d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102/f104 x:0 0 0 2024-04-06T14:17:17.014 INFO:tasks.workunit.client.1.smithi086.stdout:1/691: unlink db/d11/d1a/d2a/d9a/d68/d6e/c98 0 2024-04-06T14:17:17.014 INFO:tasks.workunit.client.0.smithi032.stdout:0/697: chown d3/da/d25/d2b/d4a/d79/d97/de0/fe7 9188061 1 2024-04-06T14:17:17.015 INFO:tasks.workunit.client.0.smithi032.stdout:1/960: dwrite d0/fe7 [0,4194304] 0 2024-04-06T14:17:17.015 INFO:tasks.workunit.client.1.smithi086.stdout:0/678: chown d0/d1/d2/d1a/d79/c85 6 1 2024-04-06T14:17:17.018 INFO:tasks.workunit.client.1.smithi086.stdout:7/632: symlink d7/d10/d11/d21/ld5 0 2024-04-06T14:17:17.018 INFO:tasks.workunit.client.1.smithi086.stdout:0/679: chown d0/d1/d4/d7/f14 507507 1 2024-04-06T14:17:17.021 INFO:tasks.workunit.client.1.smithi086.stdout:8/707: fdatasync d0/d71/d22/d43/d6d/d75/d7d/fd3 0 2024-04-06T14:17:17.025 INFO:tasks.workunit.client.1.smithi086.stdout:5/514: creat d0/d16/d18/d35/fa2 x:0 0 0 2024-04-06T14:17:17.025 INFO:tasks.workunit.client.1.smithi086.stdout:2/648: rename d0/df/d48/d7d/fa6 to d0/df/d5e/d9d/fe0 0 2024-04-06T14:17:17.026 INFO:tasks.workunit.client.0.smithi032.stdout:6/927: creat d4/d5/dbf/d53/f136 x:0 0 0 2024-04-06T14:17:17.027 INFO:tasks.workunit.client.1.smithi086.stdout:7/633: chown d7/d10/d11/d1c/d24/d4d/d5c/d66 2560863 1 2024-04-06T14:17:17.028 INFO:tasks.workunit.client.1.smithi086.stdout:0/680: rmdir d0/d1/d2/d1a/d79 39 2024-04-06T14:17:17.028 INFO:tasks.workunit.client.1.smithi086.stdout:3/677: dread d5/d36/d5a/d6d/d72/d9a/fd7 [0,4194304] 0 2024-04-06T14:17:17.029 INFO:tasks.workunit.client.0.smithi032.stdout:0/698: symlink d3/da/d25/d57/db5/lef 0 2024-04-06T14:17:17.029 INFO:tasks.workunit.client.0.smithi032.stdout:6/928: read d4/d5/d8/d2b/ffd [621219,42762] 0 2024-04-06T14:17:17.029 INFO:tasks.workunit.client.1.smithi086.stdout:8/708: symlink d0/d71/d22/d65/d6e/d98/ddb/lf5 0 2024-04-06T14:17:17.032 INFO:tasks.workunit.client.0.smithi032.stdout:0/699: truncate d3/f55 1513206 0 2024-04-06T14:17:17.034 INFO:tasks.workunit.client.0.smithi032.stdout:6/929: rename d4/d5/d8/c94 to d4/d69/c137 0 2024-04-06T14:17:17.036 INFO:tasks.workunit.client.0.smithi032.stdout:0/700: symlink d3/da/d25/d2b/d4a/d79/da8/lf0 0 2024-04-06T14:17:17.037 INFO:tasks.workunit.client.0.smithi032.stdout:6/930: chown d4/d5/dbf/d61/db9/f135 15074660 1 2024-04-06T14:17:17.038 INFO:tasks.workunit.client.0.smithi032.stdout:6/931: stat d4/d5/d8/d4b/fd3 0 2024-04-06T14:17:17.039 INFO:tasks.workunit.client.1.smithi086.stdout:3/678: write d5/fa1 [8778723,114812] 0 2024-04-06T14:17:17.039 INFO:tasks.workunit.client.1.smithi086.stdout:1/692: dwrite db/fca [0,4194304] 0 2024-04-06T14:17:17.039 INFO:tasks.workunit.client.0.smithi032.stdout:6/932: chown d4/d5/dbf/d61/f66 0 1 2024-04-06T14:17:17.040 INFO:tasks.workunit.client.0.smithi032.stdout:0/701: mkdir d3/da/d25/d2b/d80/db3/df1 0 2024-04-06T14:17:17.043 INFO:tasks.workunit.client.1.smithi086.stdout:8/709: read - d0/d71/d22/d43/d6d/d75/db2/fca zero size 2024-04-06T14:17:17.044 INFO:tasks.workunit.client.1.smithi086.stdout:1/693: fdatasync db/d11/d1f/f5a 0 2024-04-06T14:17:17.044 INFO:tasks.workunit.client.1.smithi086.stdout:2/649: creat d0/d20/d27/d2b/d35/d36/d4a/db8/fe1 x:0 0 0 2024-04-06T14:17:17.045 INFO:tasks.workunit.client.1.smithi086.stdout:3/679: mkdir d5/d26/d4f/dd2/ddb 0 2024-04-06T14:17:17.045 INFO:tasks.workunit.client.0.smithi032.stdout:6/933: creat d4/d5/d19/d25/f138 x:0 0 0 2024-04-06T14:17:17.045 INFO:tasks.workunit.client.1.smithi086.stdout:9/544: dread d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f81 [0,4194304] 0 2024-04-06T14:17:17.046 INFO:tasks.workunit.client.1.smithi086.stdout:2/650: rmdir d0/d20/d27/d2b 39 2024-04-06T14:17:17.047 INFO:tasks.workunit.client.1.smithi086.stdout:3/680: fdatasync d5/d33/f78 0 2024-04-06T14:17:17.048 INFO:tasks.workunit.client.1.smithi086.stdout:3/681: readlink d5/d36/d7d/l86 0 2024-04-06T14:17:17.048 INFO:tasks.workunit.client.0.smithi032.stdout:6/934: chown d4/d5/d35/f59 59 1 2024-04-06T14:17:17.049 INFO:tasks.workunit.client.1.smithi086.stdout:5/515: truncate d0/d16/d18/d21/d23/d28/d94/f56 115737 0 2024-04-06T14:17:17.049 INFO:tasks.workunit.client.0.smithi032.stdout:0/702: dwrite d3/f45 [0,4194304] 0 2024-04-06T14:17:17.050 INFO:tasks.workunit.client.1.smithi086.stdout:5/516: chown d0/d43/c48 63 1 2024-04-06T14:17:17.052 INFO:tasks.workunit.client.1.smithi086.stdout:9/545: symlink d5/d13/d47/lb2 0 2024-04-06T14:17:17.053 INFO:tasks.workunit.client.0.smithi032.stdout:0/703: creat d3/dc2/dd8/ff2 x:0 0 0 2024-04-06T14:17:17.055 INFO:tasks.workunit.client.1.smithi086.stdout:2/651: dread - d0/d20/d27/d2b/d35/d36/dbe/fdf zero size 2024-04-06T14:17:17.055 INFO:tasks.workunit.client.1.smithi086.stdout:5/517: write d0/d16/d18/d21/d23/f6c [1154552,62925] 0 2024-04-06T14:17:17.056 INFO:tasks.workunit.client.0.smithi032.stdout:6/935: symlink d4/d5/dbf/d51/l139 0 2024-04-06T14:17:17.056 INFO:tasks.workunit.client.0.smithi032.stdout:2/777: rmdir d0 39 2024-04-06T14:17:17.057 INFO:tasks.workunit.client.0.smithi032.stdout:7/950: dread d16/d20/d54/f57 [0,4194304] 0 2024-04-06T14:17:17.058 INFO:tasks.workunit.client.0.smithi032.stdout:7/951: stat d16/f40 0 2024-04-06T14:17:17.060 INFO:tasks.workunit.client.0.smithi032.stdout:1/961: write d0/f4d [2292165,95833] 0 2024-04-06T14:17:17.060 INFO:tasks.workunit.client.1.smithi086.stdout:3/682: rename d5/d33/d5d/f92 to d5/d36/d5a/d6d/d72/d64/fdc 0 2024-04-06T14:17:17.063 INFO:tasks.workunit.client.0.smithi032.stdout:4/842: sync 2024-04-06T14:17:17.063 INFO:tasks.workunit.client.0.smithi032.stdout:5/799: sync 2024-04-06T14:17:17.064 INFO:tasks.workunit.client.0.smithi032.stdout:0/704: symlink d3/da/d25/d2b/d4a/d79/d97/de2/lf3 0 2024-04-06T14:17:17.066 INFO:tasks.workunit.client.1.smithi086.stdout:9/546: mkdir d5/d13/d1a/d5f/d88/db3 0 2024-04-06T14:17:17.066 INFO:tasks.workunit.client.1.smithi086.stdout:6/753: dread d6/d13/fb0 [0,4194304] 0 2024-04-06T14:17:17.066 INFO:tasks.workunit.client.1.smithi086.stdout:3/683: write d5/d36/d5a/d6d/d72/d57/fd1 [276943,83965] 0 2024-04-06T14:17:17.067 INFO:tasks.workunit.client.0.smithi032.stdout:1/962: dread d0/da/d3e/fad [0,4194304] 0 2024-04-06T14:17:17.069 INFO:tasks.workunit.client.1.smithi086.stdout:5/518: dread d0/d5e/f60 [0,4194304] 0 2024-04-06T14:17:17.071 INFO:tasks.workunit.client.0.smithi032.stdout:0/705: write d3/da/d25/d36/fbd [1036119,7465] 0 2024-04-06T14:17:17.071 INFO:tasks.workunit.client.0.smithi032.stdout:5/800: unlink d16/d18/d33/d45/d5e/c73 0 2024-04-06T14:17:17.072 INFO:tasks.workunit.client.1.smithi086.stdout:3/684: dread - d5/df/fca zero size 2024-04-06T14:17:17.073 INFO:tasks.workunit.client.1.smithi086.stdout:7/634: dread d7/d10/d11/d80/f35 [0,4194304] 0 2024-04-06T14:17:17.073 INFO:tasks.workunit.client.1.smithi086.stdout:6/754: truncate d6/f7 8862990 0 2024-04-06T14:17:17.073 INFO:tasks.workunit.client.0.smithi032.stdout:1/963: rmdir d0/d9/d94/dc2 39 2024-04-06T14:17:17.078 INFO:tasks.workunit.client.0.smithi032.stdout:0/706: fsync d3/da/d35/fce 0 2024-04-06T14:17:17.079 INFO:tasks.workunit.client.1.smithi086.stdout:3/685: creat d5/d26/d7f/d59/fdd x:0 0 0 2024-04-06T14:17:17.080 INFO:tasks.workunit.client.0.smithi032.stdout:7/952: dwrite d16/d20/d54/de4/d2d/d30/d3e/da7/fff [0,4194304] 0 2024-04-06T14:17:17.081 INFO:tasks.workunit.client.0.smithi032.stdout:6/936: dread d4/d5/d8/d4b/fd3 [0,4194304] 0 2024-04-06T14:17:17.082 INFO:tasks.workunit.client.1.smithi086.stdout:8/710: dwrite d0/d71/d16/d39/d90/f94 [0,4194304] 0 2024-04-06T14:17:17.083 INFO:tasks.workunit.client.0.smithi032.stdout:2/778: write d0/d5a/d6a/dcb/fe0 [1042631,44046] 0 2024-04-06T14:17:17.085 INFO:tasks.workunit.client.0.smithi032.stdout:0/707: write d3/da/d25/d2b/f61 [601346,124785] 0 2024-04-06T14:17:17.087 INFO:tasks.workunit.client.0.smithi032.stdout:4/843: write d0/df/d23/d31/d50/d5e/faa [1283189,17110] 0 2024-04-06T14:17:17.089 INFO:tasks.workunit.client.0.smithi032.stdout:7/953: write d16/d20/d54/de4/de0/df9/f14b [754478,68260] 0 2024-04-06T14:17:17.089 INFO:tasks.workunit.client.0.smithi032.stdout:6/937: write d4/d5/d28/d65/df8/f111 [669065,44322] 0 2024-04-06T14:17:17.090 INFO:tasks.workunit.client.1.smithi086.stdout:7/635: rmdir d7/d10/d11/d1c/d36/dbc 39 2024-04-06T14:17:17.090 INFO:tasks.workunit.client.1.smithi086.stdout:8/711: write d0/d71/d22/d43/d6d/f80 [2866472,46631] 0 2024-04-06T14:17:17.093 INFO:tasks.workunit.client.0.smithi032.stdout:5/801: dread d16/d18/d33/d45/f10f [0,4194304] 0 2024-04-06T14:17:17.093 INFO:tasks.workunit.client.1.smithi086.stdout:9/547: dread d5/d13/d1a/d5f/d88/d14/d19/f5d [0,4194304] 0 2024-04-06T14:17:17.094 INFO:tasks.workunit.client.0.smithi032.stdout:7/954: chown d16/d20/d54/de4/d5f/fc7 4 1 2024-04-06T14:17:17.094 INFO:tasks.workunit.client.1.smithi086.stdout:9/548: symlink d5/d58/lb4 0 2024-04-06T14:17:17.098 INFO:tasks.workunit.client.1.smithi086.stdout:6/755: dwrite d6/d13/d5f/d60/d45/d4a/fcb [4194304,4194304] 0 2024-04-06T14:17:17.098 INFO:tasks.workunit.client.1.smithi086.stdout:7/636: creat d7/fd6 x:0 0 0 2024-04-06T14:17:17.101 INFO:tasks.workunit.client.1.smithi086.stdout:9/549: dread d5/d13/d1a/d8f/f9a [0,4194304] 0 2024-04-06T14:17:17.102 INFO:tasks.workunit.client.0.smithi032.stdout:5/802: mkdir d16/d58/d54/d94/db2/dcb/d11b 0 2024-04-06T14:17:17.103 INFO:tasks.workunit.client.1.smithi086.stdout:2/652: dwrite d0/d20/d44/f5b [4194304,4194304] 0 2024-04-06T14:17:17.105 INFO:tasks.workunit.client.0.smithi032.stdout:7/955: dwrite d16/d20/ffb [0,4194304] 0 2024-04-06T14:17:17.107 INFO:tasks.workunit.client.1.smithi086.stdout:7/637: write d7/d10/d11/d1c/d24/d4d/daf/db5/f6e [396078,1400] 0 2024-04-06T14:17:17.108 INFO:tasks.workunit.client.0.smithi032.stdout:7/956: chown d16/d20/d54/de4/d2d/f69 91480 1 2024-04-06T14:17:17.113 INFO:tasks.workunit.client.1.smithi086.stdout:2/653: dread - d0/d20/d27/d2b/d35/d36/d4a/db8/fe1 zero size 2024-04-06T14:17:17.114 INFO:tasks.workunit.client.1.smithi086.stdout:7/638: symlink d7/d94/ld7 0 2024-04-06T14:17:17.115 INFO:tasks.workunit.client.1.smithi086.stdout:9/550: unlink d5/d13/d1a/d5f/d88/d14/d19/d25/d59/la1 0 2024-04-06T14:17:17.115 INFO:tasks.workunit.client.0.smithi032.stdout:5/803: rename d16/d18/d33/d35/d88/dbd/dc5/dec/lfa to d16/d18/d33/d45/d52/l11c 0 2024-04-06T14:17:17.115 INFO:tasks.workunit.client.0.smithi032.stdout:4/844: link d0/df/d1f/lda d0/l11b 0 2024-04-06T14:17:17.115 INFO:tasks.workunit.client.0.smithi032.stdout:1/964: truncate d0/d10/d66/f70 3056347 0 2024-04-06T14:17:17.116 INFO:tasks.workunit.client.0.smithi032.stdout:2/779: link d0/d4e/ccd d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/c105 0 2024-04-06T14:17:17.116 INFO:tasks.workunit.client.0.smithi032.stdout:7/957: mknod d16/d20/d54/de4/d2d/d30/d97/c14e 0 2024-04-06T14:17:17.117 INFO:tasks.workunit.client.1.smithi086.stdout:6/756: rename d6/d13/d1a/c49 to d6/d9/d63/cfe 0 2024-04-06T14:17:17.118 INFO:tasks.workunit.client.0.smithi032.stdout:1/965: stat d0/d9/d1f/cd1 0 2024-04-06T14:17:17.118 INFO:tasks.workunit.client.0.smithi032.stdout:2/780: read - d0/d4e/f101 zero size 2024-04-06T14:17:17.118 INFO:tasks.workunit.client.0.smithi032.stdout:4/845: rmdir d0/df/d1f 39 2024-04-06T14:17:17.122 INFO:tasks.workunit.client.1.smithi086.stdout:3/686: dread d5/d26/d4f/db4/da7/f79 [0,4194304] 0 2024-04-06T14:17:17.122 INFO:tasks.workunit.client.1.smithi086.stdout:6/757: write d6/d13/f75 [1437711,34479] 0 2024-04-06T14:17:17.122 INFO:tasks.workunit.client.1.smithi086.stdout:8/712: dwrite d0/d71/d16/d39/f66 [0,4194304] 0 2024-04-06T14:17:17.125 INFO:tasks.workunit.client.1.smithi086.stdout:7/639: fdatasync d7/d10/d11/d1c/f1e 0 2024-04-06T14:17:17.128 INFO:tasks.workunit.client.1.smithi086.stdout:7/640: dread - d7/fd6 zero size 2024-04-06T14:17:17.130 INFO:tasks.workunit.client.1.smithi086.stdout:8/713: symlink d0/d71/d16/d52/dd0/lf6 0 2024-04-06T14:17:17.131 INFO:tasks.workunit.client.0.smithi032.stdout:2/781: dread - d0/d30/d39/d59/d67/d74/d78/dc2/ff5 zero size 2024-04-06T14:17:17.132 INFO:tasks.workunit.client.1.smithi086.stdout:2/654: dwrite d0/d20/d27/d2b/d35/d36/d4a/db8/fe1 [0,4194304] 0 2024-04-06T14:17:17.135 INFO:tasks.workunit.client.0.smithi032.stdout:0/708: dwrite d3/da/d25/d36/f68 [0,4194304] 0 2024-04-06T14:17:17.140 INFO:tasks.workunit.client.0.smithi032.stdout:4/846: dwrite d0/df/d23/fb8 [0,4194304] 0 2024-04-06T14:17:17.141 INFO:tasks.workunit.client.0.smithi032.stdout:0/709: fsync d3/da/f1f 0 2024-04-06T14:17:17.141 INFO:tasks.workunit.client.1.smithi086.stdout:7/641: creat d7/d10/d11/d1c/d24/dc8/dcd/fd8 x:0 0 0 2024-04-06T14:17:17.147 INFO:tasks.workunit.client.0.smithi032.stdout:7/958: rename d16/d20/d54/de4/d2d/d30/d97/lac to d16/d20/d54/de4/de0/l14f 0 2024-04-06T14:17:17.153 INFO:tasks.workunit.client.0.smithi032.stdout:6/938: dwrite d4/d5/d8/d4b/fd6 [0,4194304] 0 2024-04-06T14:17:17.153 INFO:tasks.workunit.client.1.smithi086.stdout:9/551: dwrite d5/d13/d1a/d5f/f6a [0,4194304] 0 2024-04-06T14:17:17.154 INFO:tasks.workunit.client.0.smithi032.stdout:0/710: dread - d3/da/f5e zero size 2024-04-06T14:17:17.154 INFO:tasks.workunit.client.1.smithi086.stdout:7/642: chown d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/dd2 12851107 1 2024-04-06T14:17:17.155 INFO:tasks.workunit.client.0.smithi032.stdout:7/959: dread - d16/d20/d7a/f12a zero size 2024-04-06T14:17:17.155 INFO:tasks.workunit.client.1.smithi086.stdout:9/552: readlink d5/d58/l6e 0 2024-04-06T14:17:17.157 INFO:tasks.workunit.client.1.smithi086.stdout:3/687: dread d5/d36/d65/f7b [0,4194304] 0 2024-04-06T14:17:17.159 INFO:tasks.workunit.client.0.smithi032.stdout:0/711: fdatasync d3/da/f1d 0 2024-04-06T14:17:17.161 INFO:tasks.workunit.client.1.smithi086.stdout:6/758: dwrite d6/d9/f6c [0,4194304] 0 2024-04-06T14:17:17.161 INFO:tasks.workunit.client.0.smithi032.stdout:1/966: dwrite d0/da/f69 [0,4194304] 0 2024-04-06T14:17:17.165 INFO:tasks.workunit.client.1.smithi086.stdout:9/553: fsync d5/d13/d1a/f63 0 2024-04-06T14:17:17.165 INFO:tasks.workunit.client.1.smithi086.stdout:7/643: write d7/d10/d11/d1c/d24/d2c/f6f [2260542,45862] 0 2024-04-06T14:17:17.165 INFO:tasks.workunit.client.0.smithi032.stdout:6/939: creat d4/d5/d19/f13a x:0 0 0 2024-04-06T14:17:17.166 INFO:tasks.workunit.client.0.smithi032.stdout:5/804: dwrite d16/d18/d33/d35/d88/deb/f114 [0,4194304] 0 2024-04-06T14:17:17.167 INFO:tasks.workunit.client.0.smithi032.stdout:4/847: truncate d0/df/d23/d31/fd8 219724 0 2024-04-06T14:17:17.169 INFO:tasks.workunit.client.0.smithi032.stdout:2/782: dwrite d0/d30/d39/d59/d67/d74/d78/d57/fde [0,4194304] 0 2024-04-06T14:17:17.169 INFO:tasks.workunit.client.1.smithi086.stdout:3/688: truncate f4 4317832 0 2024-04-06T14:17:17.172 INFO:tasks.workunit.client.0.smithi032.stdout:6/940: write d4/d5/d19/d25/f138 [977283,113192] 0 2024-04-06T14:17:17.175 INFO:tasks.workunit.client.1.smithi086.stdout:7/644: creat d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/fd9 x:0 0 0 2024-04-06T14:17:17.175 INFO:tasks.workunit.client.1.smithi086.stdout:9/554: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5 0 2024-04-06T14:17:17.176 INFO:tasks.workunit.client.0.smithi032.stdout:2/783: dread - d0/d30/d39/d59/d67/d74/ff3 zero size 2024-04-06T14:17:17.176 INFO:tasks.workunit.client.0.smithi032.stdout:2/784: readlink d0/l5 0 2024-04-06T14:17:17.177 INFO:tasks.workunit.client.0.smithi032.stdout:0/712: read d3/da/d25/d2b/d4a/d79/d97/faf [1202004,59011] 0 2024-04-06T14:17:17.181 INFO:tasks.workunit.client.1.smithi086.stdout:6/759: dwrite d6/d20/fb1 [0,4194304] 0 2024-04-06T14:17:17.183 INFO:tasks.workunit.client.0.smithi032.stdout:5/805: stat d16/d58/d54/d74/d8f 0 2024-04-06T14:17:17.183 INFO:tasks.workunit.client.0.smithi032.stdout:6/941: mkdir d4/d5/d14/d80/d13b 0 2024-04-06T14:17:17.183 INFO:tasks.workunit.client.0.smithi032.stdout:7/960: rename d16/d20/d54/de4/d2d/d30/d3e/c111 to d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dde/c150 0 2024-04-06T14:17:17.185 INFO:tasks.workunit.client.1.smithi086.stdout:6/760: mkdir d6/d9/db/dff 0 2024-04-06T14:17:17.186 INFO:tasks.workunit.client.0.smithi032.stdout:2/785: mknod d0/d30/d39/d59/d67/d74/d78/d57/d7c/c106 0 2024-04-06T14:17:17.187 INFO:tasks.workunit.client.0.smithi032.stdout:6/942: truncate d4/d5/dbf/d61/da7/dda/fee 267650 0 2024-04-06T14:17:17.188 INFO:tasks.workunit.client.0.smithi032.stdout:7/961: rename d16/d20/d54/de4/l7d to d16/d20/dbd/l151 0 2024-04-06T14:17:17.188 INFO:tasks.workunit.client.1.smithi086.stdout:3/689: getdents d5/d36/d5a/d6d/d72/d57 0 2024-04-06T14:17:17.190 INFO:tasks.workunit.client.1.smithi086.stdout:6/761: creat d6/d9/d5c/f100 x:0 0 0 2024-04-06T14:17:17.191 INFO:tasks.workunit.client.1.smithi086.stdout:7/645: getdents d7/d10/d11/d1c/d24/d4d/d5c/d66 0 2024-04-06T14:17:17.193 INFO:tasks.workunit.client.1.smithi086.stdout:7/646: fsync d7/d10/d11/d80/f4a 0 2024-04-06T14:17:17.195 INFO:tasks.workunit.client.0.smithi032.stdout:0/713: dread d3/da/f64 [0,4194304] 0 2024-04-06T14:17:17.200 INFO:tasks.workunit.client.0.smithi032.stdout:7/962: rename d16/c1c to d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dde/c152 0 2024-04-06T14:17:17.202 INFO:tasks.workunit.client.0.smithi032.stdout:5/806: dwrite d16/d18/f23 [0,4194304] 0 2024-04-06T14:17:17.205 INFO:tasks.workunit.client.0.smithi032.stdout:6/943: dwrite d4/d5/dbf/d53/f136 [0,4194304] 0 2024-04-06T14:17:17.207 INFO:tasks.workunit.client.0.smithi032.stdout:1/967: write d0/da/f28 [4884385,108258] 0 2024-04-06T14:17:17.209 INFO:tasks.workunit.client.0.smithi032.stdout:2/786: rename d0/d30/d39/d59/d67/d74/d78/d8c/db0/c91 to d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/d103/c107 0 2024-04-06T14:17:17.211 INFO:tasks.workunit.client.0.smithi032.stdout:7/963: mknod d16/d20/d54/de4/d2d/d30/c153 0 2024-04-06T14:17:17.214 INFO:tasks.workunit.client.1.smithi086.stdout:9/555: truncate d5/d13/d47/f92 2489421 0 2024-04-06T14:17:17.214 INFO:tasks.workunit.client.0.smithi032.stdout:7/964: dread - d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dee/ffc zero size 2024-04-06T14:17:17.214 INFO:tasks.workunit.client.1.smithi086.stdout:6/762: write d6/d9/d43/f56 [508790,110383] 0 2024-04-06T14:17:17.215 INFO:tasks.workunit.client.0.smithi032.stdout:5/807: creat d16/d18/d33/d35/d4d/f11d x:0 0 0 2024-04-06T14:17:17.215 INFO:tasks.workunit.client.1.smithi086.stdout:3/690: truncate d5/d36/f3b 7449429 0 2024-04-06T14:17:17.217 INFO:tasks.workunit.client.0.smithi032.stdout:4/848: dwrite d0/df/d1f/d2e/f49 [0,4194304] 0 2024-04-06T14:17:17.219 INFO:tasks.workunit.client.1.smithi086.stdout:6/763: mknod d6/d13/d5f/c101 0 2024-04-06T14:17:17.221 INFO:tasks.workunit.client.0.smithi032.stdout:6/944: creat d4/d5/d35/df6/f13c x:0 0 0 2024-04-06T14:17:17.221 INFO:tasks.workunit.client.1.smithi086.stdout:6/764: chown d6/d13/d1a/cde 264582627 1 2024-04-06T14:17:17.223 INFO:tasks.workunit.client.0.smithi032.stdout:8/913: sync 2024-04-06T14:17:17.224 INFO:tasks.workunit.client.0.smithi032.stdout:9/854: sync 2024-04-06T14:17:17.224 INFO:tasks.workunit.client.0.smithi032.stdout:3/817: sync 2024-04-06T14:17:17.224 INFO:tasks.workunit.client.1.smithi086.stdout:7/647: dwrite d7/d10/d11/d1c/f1e [0,4194304] 0 2024-04-06T14:17:17.225 INFO:tasks.workunit.client.0.smithi032.stdout:1/968: getdents d0/d10/d16/d118 0 2024-04-06T14:17:17.227 INFO:tasks.workunit.client.0.smithi032.stdout:6/945: truncate d4/f10 1793494 0 2024-04-06T14:17:17.227 INFO:tasks.workunit.client.0.smithi032.stdout:2/787: chown d0/d6e/d8d/ca5 1327230 1 2024-04-06T14:17:17.228 INFO:tasks.workunit.client.1.smithi086.stdout:6/765: creat d6/d13/d1a/d22/dcf/db2/f102 x:0 0 0 2024-04-06T14:17:17.230 INFO:tasks.workunit.client.0.smithi032.stdout:1/969: symlink d0/d41/d5b/l138 0 2024-04-06T14:17:17.231 INFO:tasks.workunit.client.0.smithi032.stdout:8/914: mkdir d3/d8/d12/d2f/d46/d134 0 2024-04-06T14:17:17.233 INFO:tasks.workunit.client.0.smithi032.stdout:2/788: mkdir d0/d6e/dae/db7/d108 0 2024-04-06T14:17:17.233 INFO:tasks.workunit.client.0.smithi032.stdout:6/946: read d4/d5/d8/d2f/f127 [38384,74638] 0 2024-04-06T14:17:17.234 INFO:tasks.workunit.client.1.smithi086.stdout:6/766: creat d6/d13/d1a/da0/f103 x:0 0 0 2024-04-06T14:17:17.234 INFO:tasks.workunit.client.0.smithi032.stdout:2/789: stat d0/d6e/d8d/ca9 0 2024-04-06T14:17:17.235 INFO:tasks.workunit.client.0.smithi032.stdout:7/965: creat d16/f154 x:0 0 0 2024-04-06T14:17:17.236 INFO:tasks.workunit.client.1.smithi086.stdout:6/767: mknod d6/d20/c104 0 2024-04-06T14:17:17.237 INFO:tasks.workunit.client.1.smithi086.stdout:6/768: stat d6/d13/fb0 0 2024-04-06T14:17:17.238 INFO:tasks.workunit.client.0.smithi032.stdout:0/714: dwrite d3/da/d25/d2b/d4a/da1/fb2 [0,4194304] 0 2024-04-06T14:17:17.239 INFO:tasks.workunit.client.1.smithi086.stdout:6/769: symlink d6/d13/d1a/da0/l105 0 2024-04-06T14:17:17.239 INFO:tasks.workunit.client.0.smithi032.stdout:4/849: rename d0/df/d2b/dbe/d116/df0/lf5 to d0/df/d23/d31/d50/d57/d8a/l11c 0 2024-04-06T14:17:17.240 INFO:tasks.workunit.client.0.smithi032.stdout:8/915: symlink d3/d8/l135 0 2024-04-06T14:17:17.241 INFO:tasks.workunit.client.0.smithi032.stdout:0/715: stat d3/da/d25/d2b/d4a 0 2024-04-06T14:17:17.241 INFO:tasks.workunit.client.0.smithi032.stdout:7/966: mknod d16/d20/d63/d9d/c155 0 2024-04-06T14:17:17.242 INFO:tasks.workunit.client.0.smithi032.stdout:1/970: truncate d0/d10/d16/d3f/d98/fc6 507159 0 2024-04-06T14:17:17.245 INFO:tasks.workunit.client.0.smithi032.stdout:6/947: creat d4/d5/dbf/d61/db9/f13d x:0 0 0 2024-04-06T14:17:17.245 INFO:tasks.workunit.client.1.smithi086.stdout:3/691: dwrite d5/d26/da9/fb8 [0,4194304] 0 2024-04-06T14:17:17.246 INFO:tasks.workunit.client.0.smithi032.stdout:3/818: rename d3/d65/l30 to d3/d65/d7d/d9c/dbc/dcd/l13f 0 2024-04-06T14:17:17.246 INFO:tasks.workunit.client.0.smithi032.stdout:2/790: mknod d0/d30/d39/d59/d67/d74/d78/c109 0 2024-04-06T14:17:17.246 INFO:tasks.workunit.client.0.smithi032.stdout:6/948: readlink d4/l17 0 2024-04-06T14:17:17.248 INFO:tasks.workunit.client.0.smithi032.stdout:0/716: dread d3/da/f64 [0,4194304] 0 2024-04-06T14:17:17.249 INFO:tasks.workunit.client.0.smithi032.stdout:8/916: mkdir d3/d8/d3a/d4b/d136 0 2024-04-06T14:17:17.250 INFO:tasks.workunit.client.1.smithi086.stdout:9/556: truncate d5/d13/d1a/d5f/d88/d14/d19/f1e 1637709 0 2024-04-06T14:17:17.252 INFO:tasks.workunit.client.0.smithi032.stdout:5/808: dwrite d16/d18/d33/f8b [4194304,4194304] 0 2024-04-06T14:17:17.252 INFO:tasks.workunit.client.1.smithi086.stdout:7/648: truncate d7/d10/d11/f91 769586 0 2024-04-06T14:17:17.253 INFO:tasks.workunit.client.0.smithi032.stdout:4/850: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/d11d 0 2024-04-06T14:17:17.253 INFO:tasks.workunit.client.0.smithi032.stdout:8/917: write d3/d8/d12/d2f/d4a/d60/d90/fe9 [3340457,1388] 0 2024-04-06T14:17:17.257 INFO:tasks.workunit.client.1.smithi086.stdout:7/649: creat d7/d10/d11/d1c/d24/dc8/fda x:0 0 0 2024-04-06T14:17:17.257 INFO:tasks.workunit.client.0.smithi032.stdout:0/717: read d3/da/d35/d8b/f90 [2579088,82114] 0 2024-04-06T14:17:17.258 INFO:tasks.workunit.client.0.smithi032.stdout:9/855: rename d2/d15/dd9 to d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/d10e 0 2024-04-06T14:17:17.260 INFO:tasks.workunit.client.0.smithi032.stdout:6/949: unlink d4/d5/dbf/d51/fa8 0 2024-04-06T14:17:17.263 INFO:tasks.workunit.client.0.smithi032.stdout:5/809: dread - d16/d18/dae/dbb/ff9 zero size 2024-04-06T14:17:17.263 INFO:tasks.workunit.client.0.smithi032.stdout:4/851: creat d0/df/d2b/d43/f11e x:0 0 0 2024-04-06T14:17:17.264 INFO:tasks.workunit.client.1.smithi086.stdout:7/650: dread d7/d10/d11/d1c/d24/d4d/daf/db5/f6e [0,4194304] 0 2024-04-06T14:17:17.264 INFO:tasks.workunit.client.0.smithi032.stdout:7/967: creat d16/d20/d54/de4/d2d/d30/f156 x:0 0 0 2024-04-06T14:17:17.266 INFO:tasks.workunit.client.0.smithi032.stdout:7/968: readlink d16/d20/d63/d9d/df2/df5/la8 0 2024-04-06T14:17:17.266 INFO:tasks.workunit.client.0.smithi032.stdout:3/819: rename d3/d65/d7d/d9c/db8/fc2 to d3/d65/d7d/d9c/dad/d111/f140 0 2024-04-06T14:17:17.267 INFO:tasks.workunit.client.0.smithi032.stdout:1/971: rmdir d0/d9/df2/d134 0 2024-04-06T14:17:17.267 INFO:tasks.workunit.client.1.smithi086.stdout:7/651: creat d7/d10/d11/d1c/d24/d4d/fdb x:0 0 0 2024-04-06T14:17:17.268 INFO:tasks.workunit.client.0.smithi032.stdout:9/856: mknod d2/d15/d1e/d65/d6b/d92/c10f 0 2024-04-06T14:17:17.269 INFO:tasks.workunit.client.0.smithi032.stdout:6/950: unlink d4/d5/l1e 0 2024-04-06T14:17:17.273 INFO:tasks.workunit.client.0.smithi032.stdout:1/972: fdatasync d0/d9/d94/dc2/fdb 0 2024-04-06T14:17:17.277 INFO:tasks.workunit.client.0.smithi032.stdout:9/857: mknod d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/d10e/c110 0 2024-04-06T14:17:17.279 INFO:tasks.workunit.client.0.smithi032.stdout:4/852: symlink d0/df/d23/d31/d50/d57/l11f 0 2024-04-06T14:17:17.280 INFO:tasks.workunit.client.0.smithi032.stdout:3/820: dwrite d3/d65/d7d/d9c/dad/db1/f62 [4194304,4194304] 0 2024-04-06T14:17:17.283 INFO:tasks.workunit.client.0.smithi032.stdout:1/973: symlink d0/da/d3e/d75/dd7/l139 0 2024-04-06T14:17:17.286 INFO:tasks.workunit.client.0.smithi032.stdout:8/918: rename d3/d8/d12/d83/d89/da0/ffe to d3/d8/d3a/d7b/db9/f137 0 2024-04-06T14:17:17.288 INFO:tasks.workunit.client.0.smithi032.stdout:4/853: fdatasync d0/df/f92 0 2024-04-06T14:17:17.295 INFO:tasks.workunit.client.0.smithi032.stdout:5/810: dread d16/d18/d33/d35/d88/fad [0,4194304] 0 2024-04-06T14:17:17.296 INFO:tasks.workunit.client.0.smithi032.stdout:9/858: truncate d2/d15/d55/d5f/f62 2430374 0 2024-04-06T14:17:17.300 INFO:tasks.workunit.client.0.smithi032.stdout:1/974: mkdir d0/d41/d5b/d13a 0 2024-04-06T14:17:17.303 INFO:tasks.workunit.client.0.smithi032.stdout:1/975: read d0/d10/d78/f133 [993,45627] 0 2024-04-06T14:17:17.304 INFO:tasks.workunit.client.0.smithi032.stdout:0/718: rename d3/da/d25/d2b/d4a/f65 to d3/da/d25/d2b/d4a/d79/d97/de2/ff4 0 2024-04-06T14:17:17.307 INFO:tasks.workunit.client.0.smithi032.stdout:8/919: creat d3/d8/d3a/d7b/db9/ddf/f138 x:0 0 0 2024-04-06T14:17:17.308 INFO:tasks.workunit.client.0.smithi032.stdout:8/920: chown d3/d8/d3a/l11d 0 1 2024-04-06T14:17:17.311 INFO:tasks.workunit.client.1.smithi086.stdout:7/652: creat d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/dd2/fdc x:0 0 0 2024-04-06T14:17:17.313 INFO:tasks.workunit.client.0.smithi032.stdout:5/811: unlink d16/d18/d33/d35/d88/dbd/dc5/d10e/c115 0 2024-04-06T14:17:17.316 INFO:tasks.workunit.client.1.smithi086.stdout:7/653: mknod d7/d10/d11/d1c/d24/d2c/d69/cdd 0 2024-04-06T14:17:17.317 INFO:tasks.workunit.client.0.smithi032.stdout:9/859: creat d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8f/f111 x:0 0 0 2024-04-06T14:17:17.321 INFO:tasks.workunit.client.0.smithi032.stdout:1/976: creat d0/d9/d1f/d72/f13b x:0 0 0 2024-04-06T14:17:17.322 INFO:tasks.workunit.client.0.smithi032.stdout:6/951: dwrite d4/d5/d18/f74 [0,4194304] 0 2024-04-06T14:17:17.323 INFO:tasks.workunit.client.0.smithi032.stdout:8/921: mknod d3/d8/d12/d2f/d4a/d60/d75/d85/c139 0 2024-04-06T14:17:17.325 INFO:tasks.workunit.client.1.smithi086.stdout:7/654: creat d7/d10/d11/d1c/d24/dc8/fde x:0 0 0 2024-04-06T14:17:17.325 INFO:tasks.workunit.client.0.smithi032.stdout:3/821: link d3/d65/d7d/d9c/ld8 d3/d65/d7d/d9c/dad/d111/l141 0 2024-04-06T14:17:17.327 INFO:tasks.workunit.client.0.smithi032.stdout:5/812: truncate d16/d18/d33/d45/d52/d39/d86/fbe 1337023 0 2024-04-06T14:17:17.327 INFO:tasks.workunit.client.0.smithi032.stdout:3/822: write d3/d65/d7d/d9c/dbc/dcd/f101 [820296,3003] 0 2024-04-06T14:17:17.328 INFO:tasks.workunit.client.0.smithi032.stdout:9/860: creat d2/f112 x:0 0 0 2024-04-06T14:17:17.329 INFO:tasks.workunit.client.1.smithi086.stdout:7/655: fsync d7/d1d/fb8 0 2024-04-06T14:17:17.330 INFO:tasks.workunit.client.0.smithi032.stdout:5/813: truncate d16/d18/d33/df7/ffb 568601 0 2024-04-06T14:17:17.333 INFO:tasks.workunit.client.0.smithi032.stdout:8/922: chown d3/d8/d12/d83/d89/da0/dd2/c102 2838 1 2024-04-06T14:17:17.333 INFO:tasks.workunit.client.1.smithi086.stdout:4/488: sync 2024-04-06T14:17:17.334 INFO:tasks.workunit.client.0.smithi032.stdout:1/977: mkdir d0/d10/d78/d128/d13c 0 2024-04-06T14:17:17.334 INFO:tasks.workunit.client.1.smithi086.stdout:7/656: mknod d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/dd2/cdf 0 2024-04-06T14:17:17.339 INFO:tasks.workunit.client.0.smithi032.stdout:5/814: creat d16/d18/d33/d35/d88/f11e x:0 0 0 2024-04-06T14:17:17.340 INFO:tasks.workunit.client.0.smithi032.stdout:4/854: dwrite d0/df/d23/d31/d50/d57/d8a/fd7 [0,4194304] 0 2024-04-06T14:17:17.341 INFO:tasks.workunit.client.0.smithi032.stdout:5/815: dread - d16/d58/f108 zero size 2024-04-06T14:17:17.341 INFO:tasks.workunit.client.0.smithi032.stdout:4/855: fsync d0/d82/f108 0 2024-04-06T14:17:17.348 INFO:tasks.workunit.client.0.smithi032.stdout:4/856: creat d0/d82/d8e/f120 x:0 0 0 2024-04-06T14:17:17.351 INFO:tasks.workunit.client.0.smithi032.stdout:5/816: symlink d16/d18/d33/d35/d88/dbd/dc5/dec/l11f 0 2024-04-06T14:17:17.354 INFO:tasks.workunit.client.0.smithi032.stdout:3/823: link d3/d65/d89/dcb/leb d3/d65/d89/dcb/l142 0 2024-04-06T14:17:17.360 INFO:tasks.workunit.client.0.smithi032.stdout:4/857: symlink d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/d11d/l121 0 2024-04-06T14:17:17.360 INFO:tasks.workunit.client.0.smithi032.stdout:8/923: dwrite d3/d8/d12/fb0 [0,4194304] 0 2024-04-06T14:17:17.363 INFO:tasks.workunit.client.0.smithi032.stdout:9/861: truncate d2/d15/d1e/d41/d6c/d74/fe7 2321739 0 2024-04-06T14:17:17.364 INFO:tasks.workunit.client.1.smithi086.stdout:4/489: dread d0/f10 [0,4194304] 0 2024-04-06T14:17:17.366 INFO:tasks.workunit.client.0.smithi032.stdout:4/858: mkdir d0/df/d1f/d2e/d122 0 2024-04-06T14:17:17.367 INFO:tasks.workunit.client.1.smithi086.stdout:7/657: write f5 [1162412,31338] 0 2024-04-06T14:17:17.368 INFO:tasks.workunit.client.0.smithi032.stdout:8/924: mkdir d3/d8/d7/d13a 0 2024-04-06T14:17:17.368 INFO:tasks.workunit.client.1.smithi086.stdout:7/658: stat d7/d10/d11/d1c/d24/dc8/fda 0 2024-04-06T14:17:17.369 INFO:tasks.workunit.client.0.smithi032.stdout:9/862: chown d2/d15/d1e/d65/d83/l30 8440 1 2024-04-06T14:17:17.369 INFO:tasks.workunit.client.0.smithi032.stdout:6/952: getdents d4/d5/d8/d2f/d9b/db4 0 2024-04-06T14:17:17.370 INFO:tasks.workunit.client.0.smithi032.stdout:1/978: dwrite d0/d9/d1f/d72/f7e [0,4194304] 0 2024-04-06T14:17:17.373 INFO:tasks.workunit.client.1.smithi086.stdout:4/490: rmdir d0/d15 39 2024-04-06T14:17:17.373 INFO:tasks.workunit.client.0.smithi032.stdout:8/925: mkdir d3/d8/d13b 0 2024-04-06T14:17:17.373 INFO:tasks.workunit.client.1.smithi086.stdout:4/491: chown d0/d2c/d38/d54/d83/f98 2 1 2024-04-06T14:17:17.374 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:17 smithi032 ceph-mon[39730]: pgmap v26: 65 pgs: 65 active+clean; 2.6 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 83 MiB/s rd, 218 MiB/s wr, 443 op/s 2024-04-06T14:17:17.374 INFO:tasks.workunit.client.0.smithi032.stdout:6/953: write d4/d5/d8/d4b/fa6 [520234,115851] 0 2024-04-06T14:17:17.375 INFO:tasks.workunit.client.1.smithi086.stdout:7/659: creat d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/dd2/fe0 x:0 0 0 2024-04-06T14:17:17.375 INFO:tasks.workunit.client.0.smithi032.stdout:5/817: write d16/d18/d33/d45/d52/d39/fb7 [372967,78637] 0 2024-04-06T14:17:17.376 INFO:tasks.workunit.client.1.smithi086.stdout:7/660: read d7/d1d/f3e [2454445,90794] 0 2024-04-06T14:17:17.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:17 smithi086 ceph-mon[47925]: pgmap v26: 65 pgs: 65 active+clean; 2.6 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 83 MiB/s rd, 218 MiB/s wr, 443 op/s 2024-04-06T14:17:17.378 INFO:tasks.workunit.client.1.smithi086.stdout:7/661: truncate d7/d10/d11/d80/fc9 284295 0 2024-04-06T14:17:17.382 INFO:tasks.workunit.client.0.smithi032.stdout:9/863: symlink d2/d15/d55/d59/d93/d95/l113 0 2024-04-06T14:17:17.382 INFO:tasks.workunit.client.0.smithi032.stdout:3/824: dwrite d3/fa [0,4194304] 0 2024-04-06T14:17:17.382 INFO:tasks.workunit.client.0.smithi032.stdout:8/926: write d3/d8/d12/f3f [89144,66149] 0 2024-04-06T14:17:17.382 INFO:tasks.workunit.client.1.smithi086.stdout:4/492: creat d0/d2c/d30/d24/d93/f99 x:0 0 0 2024-04-06T14:17:17.383 INFO:tasks.workunit.client.0.smithi032.stdout:1/979: mknod d0/d10/d16/d3f/def/c13d 0 2024-04-06T14:17:17.383 INFO:tasks.workunit.client.0.smithi032.stdout:3/825: chown d3/d65/l4e 857 1 2024-04-06T14:17:17.384 INFO:tasks.workunit.client.1.smithi086.stdout:7/662: fdatasync d7/d10/d11/d1c/d24/d4d/daf/db5/d7a/f95 0 2024-04-06T14:17:17.385 INFO:tasks.workunit.client.0.smithi032.stdout:1/980: chown d0/fde 385 1 2024-04-06T14:17:17.387 INFO:tasks.workunit.client.1.smithi086.stdout:7/663: stat d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/fc3 0 2024-04-06T14:17:17.390 INFO:tasks.workunit.client.0.smithi032.stdout:8/927: dread d3/d8/d12/d2f/d46/def/d128/f95 [0,4194304] 0 2024-04-06T14:17:17.396 INFO:tasks.workunit.client.1.smithi086.stdout:7/664: dwrite d7/d10/d11/d1c/d24/dc8/fda [0,4194304] 0 2024-04-06T14:17:17.397 INFO:tasks.workunit.client.1.smithi086.stdout:7/665: stat d7/d10/d11/f8f 0 2024-04-06T14:17:17.398 INFO:tasks.workunit.client.0.smithi032.stdout:6/954: mkdir d4/d5/d8/d2f/d9b/db4/d13e 0 2024-04-06T14:17:17.398 INFO:tasks.workunit.client.0.smithi032.stdout:5/818: mkdir d16/d18/d33/d35/d88/dbd/d120 0 2024-04-06T14:17:17.400 INFO:tasks.workunit.client.1.smithi086.stdout:4/493: write d0/d15/d19/f52 [847946,45347] 0 2024-04-06T14:17:17.400 INFO:tasks.workunit.client.0.smithi032.stdout:1/981: mknod d0/dc3/c13e 0 2024-04-06T14:17:17.402 INFO:tasks.workunit.client.0.smithi032.stdout:8/928: creat d3/d8/d7/dd5/f13c x:0 0 0 2024-04-06T14:17:17.403 INFO:tasks.workunit.client.1.smithi086.stdout:7/666: fdatasync d7/d10/d11/f16 0 2024-04-06T14:17:17.403 INFO:tasks.workunit.client.0.smithi032.stdout:6/955: dread d4/d5/d8/d2b/dfb/f117 [0,4194304] 0 2024-04-06T14:17:17.405 INFO:tasks.workunit.client.0.smithi032.stdout:5/819: fsync d16/d18/d33/d45/d52/d39/fd5 0 2024-04-06T14:17:17.407 INFO:tasks.workunit.client.1.smithi086.stdout:7/667: truncate d7/d10/d11/d1c/d24/d2c/f48 717421 0 2024-04-06T14:17:17.408 INFO:tasks.workunit.client.1.smithi086.stdout:4/494: creat d0/d2c/d30/d24/d5d/d8a/f9a x:0 0 0 2024-04-06T14:17:17.416 INFO:tasks.workunit.client.0.smithi032.stdout:6/956: read d4/d5/d19/d25/d67/ff7 [1336274,79081] 0 2024-04-06T14:17:17.419 INFO:tasks.workunit.client.1.smithi086.stdout:4/495: rename d0/lb to d0/d15/d20/d74/l9b 0 2024-04-06T14:17:17.421 INFO:tasks.workunit.client.0.smithi032.stdout:6/957: mkdir d4/d69/d13f 0 2024-04-06T14:17:17.423 INFO:tasks.workunit.client.1.smithi086.stdout:4/496: link d0/c1e d0/d15/d19/d88/c9c 0 2024-04-06T14:17:17.424 INFO:tasks.workunit.client.1.smithi086.stdout:4/497: fsync d0/f2 0 2024-04-06T14:17:17.425 INFO:tasks.workunit.client.0.smithi032.stdout:9/864: dwrite d2/d15/d1e/d65/d83/d7e/f6 [0,4194304] 0 2024-04-06T14:17:17.427 INFO:tasks.workunit.client.0.smithi032.stdout:3/826: dwrite d3/f5e [0,4194304] 0 2024-04-06T14:17:17.427 INFO:tasks.workunit.client.1.smithi086.stdout:4/498: symlink d0/d2c/d30/l9d 0 2024-04-06T14:17:17.427 INFO:tasks.workunit.client.0.smithi032.stdout:8/929: creat d3/d8/d3a/d7b/db9/df8/d106/d124/f13d x:0 0 0 2024-04-06T14:17:17.428 INFO:tasks.workunit.client.1.smithi086.stdout:4/499: chown d0/f10 703 1 2024-04-06T14:17:17.429 INFO:tasks.workunit.client.0.smithi032.stdout:5/820: dwrite d16/d58/f6f [0,4194304] 0 2024-04-06T14:17:17.433 INFO:tasks.workunit.client.0.smithi032.stdout:5/821: write d16/d18/d33/d45/d5e/f89 [1284425,64665] 0 2024-04-06T14:17:17.433 INFO:tasks.workunit.client.0.smithi032.stdout:6/958: mknod d4/d5/d28/c140 0 2024-04-06T14:17:17.435 INFO:tasks.workunit.client.0.smithi032.stdout:6/959: dread - d4/d5/d28/d65/ffc zero size 2024-04-06T14:17:17.436 INFO:tasks.workunit.client.0.smithi032.stdout:3/827: readlink d3/d65/d7d/d9c/dbc/l110 0 2024-04-06T14:17:17.436 INFO:tasks.workunit.client.0.smithi032.stdout:8/930: symlink d3/d8/d3a/d4b/de0/l13e 0 2024-04-06T14:17:17.437 INFO:tasks.workunit.client.0.smithi032.stdout:5/822: unlink d16/d18/cc3 0 2024-04-06T14:17:17.440 INFO:tasks.workunit.client.0.smithi032.stdout:9/865: symlink d2/d15/l114 0 2024-04-06T14:17:17.443 INFO:tasks.workunit.client.1.smithi086.stdout:4/500: rmdir d0 39 2024-04-06T14:17:17.446 INFO:tasks.workunit.client.0.smithi032.stdout:8/931: dread - d3/d8/d3a/d4b/de0/fe5 zero size 2024-04-06T14:17:17.446 INFO:tasks.workunit.client.0.smithi032.stdout:3/828: write d3/d65/d7d/dc6/fdb [4272456,5912] 0 2024-04-06T14:17:17.452 INFO:tasks.workunit.client.0.smithi032.stdout:3/829: write d3/f4 [3258104,120149] 0 2024-04-06T14:17:17.453 INFO:tasks.workunit.client.0.smithi032.stdout:8/932: truncate d3/d8/d7/dd5/f13c 746354 0 2024-04-06T14:17:17.456 INFO:tasks.workunit.client.1.smithi086.stdout:4/501: dwrite d0/d2c/d30/d24/d93/f99 [0,4194304] 0 2024-04-06T14:17:17.457 INFO:tasks.workunit.client.0.smithi032.stdout:3/830: read d3/d65/f8c [162243,106579] 0 2024-04-06T14:17:17.457 INFO:tasks.workunit.client.0.smithi032.stdout:8/933: write d3/f32 [6273870,50283] 0 2024-04-06T14:17:17.460 INFO:tasks.workunit.client.0.smithi032.stdout:9/866: dwrite d2/d15/d1e/d65/d83/d31/fde [4194304,4194304] 0 2024-04-06T14:17:17.463 INFO:tasks.workunit.client.0.smithi032.stdout:9/867: chown d2/d15/d1e/d41 123635 1 2024-04-06T14:17:17.464 INFO:tasks.workunit.client.0.smithi032.stdout:3/831: unlink d3/d16/ca3 0 2024-04-06T14:17:17.466 INFO:tasks.workunit.client.0.smithi032.stdout:5/823: dwrite d16/d18/d33/d45/d52/f4e [0,4194304] 0 2024-04-06T14:17:17.469 INFO:tasks.workunit.client.0.smithi032.stdout:8/934: mknod d3/d8/d12/d83/d89/da0/dd2/dfa/c13f 0 2024-04-06T14:17:17.472 INFO:tasks.workunit.client.0.smithi032.stdout:3/832: unlink d3/d16/f9a 0 2024-04-06T14:17:17.473 INFO:tasks.workunit.client.0.smithi032.stdout:6/960: dwrite d4/d5/d35/f39 [0,4194304] 0 2024-04-06T14:17:17.474 INFO:tasks.workunit.client.0.smithi032.stdout:5/824: fdatasync d16/d58/fc9 0 2024-04-06T14:17:17.480 INFO:tasks.workunit.client.0.smithi032.stdout:6/961: read d4/d5/d28/d65/fff [486879,64217] 0 2024-04-06T14:17:17.483 INFO:tasks.workunit.client.0.smithi032.stdout:9/868: write d2/d15/d1e/d65/d6b/d7f/fbf [758642,131067] 0 2024-04-06T14:17:17.484 INFO:tasks.workunit.client.0.smithi032.stdout:8/935: chown d3/d8/d12/d2f/d4a/d60/d10b/c114 126 1 2024-04-06T14:17:17.487 INFO:tasks.workunit.client.0.smithi032.stdout:5/825: write d16/d18/f23 [4021326,22509] 0 2024-04-06T14:17:17.491 INFO:tasks.workunit.client.0.smithi032.stdout:6/962: mknod d4/d5/dbf/d51/c141 0 2024-04-06T14:17:17.493 INFO:tasks.workunit.client.0.smithi032.stdout:3/833: link d3/d65/d7d/d9c/dad/db1/d3c/f78 d3/d65/d7d/d9c/f143 0 2024-04-06T14:17:17.494 INFO:tasks.workunit.client.0.smithi032.stdout:8/936: mknod d3/d8/d12/d101/c140 0 2024-04-06T14:17:17.495 INFO:tasks.workunit.client.0.smithi032.stdout:3/834: dread - d3/d65/f115 zero size 2024-04-06T14:17:17.495 INFO:tasks.workunit.client.0.smithi032.stdout:6/963: mknod d4/d5/d8/d4b/d121/c142 0 2024-04-06T14:17:17.497 INFO:tasks.workunit.client.0.smithi032.stdout:5/826: creat d16/d18/f121 x:0 0 0 2024-04-06T14:17:17.499 INFO:tasks.workunit.client.0.smithi032.stdout:3/835: read d3/d16/d63/fa1 [98827,27132] 0 2024-04-06T14:17:17.499 INFO:tasks.workunit.client.0.smithi032.stdout:6/964: mkdir d4/d5/dbf/d53/d143 0 2024-04-06T14:17:17.500 INFO:tasks.workunit.client.0.smithi032.stdout:8/937: mknod d3/d8/d12/d2f/d4a/d60/d90/dc5/dd7/c141 0 2024-04-06T14:17:17.501 INFO:tasks.workunit.client.0.smithi032.stdout:3/836: write d3/d16/f35 [3860779,115431] 0 2024-04-06T14:17:17.503 INFO:tasks.workunit.client.0.smithi032.stdout:6/965: symlink d4/d5/d8/d2f/l144 0 2024-04-06T14:17:17.507 INFO:tasks.workunit.client.0.smithi032.stdout:9/869: dread d2/d15/d1e/d65/d83/d7e/f12 [4194304,4194304] 0 2024-04-06T14:17:17.509 INFO:tasks.workunit.client.0.smithi032.stdout:5/827: link d16/d18/d33/d35/la7 d16/d18/d33/d35/d88/dbd/dc5/dec/l122 0 2024-04-06T14:17:17.512 INFO:tasks.workunit.client.0.smithi032.stdout:3/837: truncate d3/d65/d7d/d9c/dad/db1/d3c/f85 844117 0 2024-04-06T14:17:17.513 INFO:tasks.workunit.client.0.smithi032.stdout:5/828: unlink d16/d18/d33/d35/d88/fd7 0 2024-04-06T14:17:17.514 INFO:tasks.workunit.client.0.smithi032.stdout:3/838: chown d3/d16/f5c 8041 1 2024-04-06T14:17:17.517 INFO:tasks.workunit.client.0.smithi032.stdout:5/829: chown d16/d18/d33/d35/d4d/f82 471 1 2024-04-06T14:17:17.518 INFO:tasks.workunit.client.0.smithi032.stdout:9/870: dwrite d2/dcd/d7c/dfa/f10c [0,4194304] 0 2024-04-06T14:17:17.520 INFO:tasks.workunit.client.0.smithi032.stdout:5/830: chown d16/d18/d33/d45/d87/l9b 5 1 2024-04-06T14:17:17.520 INFO:tasks.workunit.client.0.smithi032.stdout:8/938: dread d3/d8/d12/d2f/d4a/d60/d90/dc5/dd7/fea [0,4194304] 0 2024-04-06T14:17:17.521 INFO:tasks.workunit.client.0.smithi032.stdout:3/839: creat d3/d65/d7d/d9c/dad/db1/d3c/dc9/d104/f144 x:0 0 0 2024-04-06T14:17:17.522 INFO:tasks.workunit.client.0.smithi032.stdout:5/831: symlink d16/d58/l123 0 2024-04-06T14:17:17.523 INFO:tasks.workunit.client.0.smithi032.stdout:8/939: dread - d3/d8/d12/d2f/d4a/d60/d90/f91 zero size 2024-04-06T14:17:17.523 INFO:tasks.workunit.client.0.smithi032.stdout:5/832: write d16/d18/f23 [887943,79712] 0 2024-04-06T14:17:17.524 INFO:tasks.workunit.client.0.smithi032.stdout:3/840: mknod d3/d16/ddd/d12d/c145 0 2024-04-06T14:17:17.527 INFO:tasks.workunit.client.0.smithi032.stdout:5/833: creat d16/d58/d54/d74/d8f/d95/f124 x:0 0 0 2024-04-06T14:17:17.533 INFO:tasks.workunit.client.0.smithi032.stdout:8/940: dwrite d3/d8/f13 [0,4194304] 0 2024-04-06T14:17:17.536 INFO:tasks.workunit.client.0.smithi032.stdout:5/834: dread d16/d18/d33/d45/d5e/f89 [0,4194304] 0 2024-04-06T14:17:17.536 INFO:tasks.workunit.client.0.smithi032.stdout:3/841: dwrite d3/dcf/fd0 [0,4194304] 0 2024-04-06T14:17:17.537 INFO:tasks.workunit.client.0.smithi032.stdout:5/835: dread - d16/d58/f108 zero size 2024-04-06T14:17:17.539 INFO:tasks.workunit.client.0.smithi032.stdout:8/941: symlink d3/d8/d7/dd5/l142 0 2024-04-06T14:17:17.539 INFO:tasks.workunit.client.0.smithi032.stdout:3/842: fsync d3/d65/f66 0 2024-04-06T14:17:17.543 INFO:tasks.workunit.client.0.smithi032.stdout:8/942: rename d3/d8/d3a/d4b/de0/l100 to d3/d8/d3a/d7b/db9/df8/l143 0 2024-04-06T14:17:17.544 INFO:tasks.workunit.client.0.smithi032.stdout:3/843: unlink d3/d65/d89/c99 0 2024-04-06T14:17:17.545 INFO:tasks.workunit.client.0.smithi032.stdout:8/943: mkdir d3/d8/d3a/d7b/db9/df8/d144 0 2024-04-06T14:17:17.545 INFO:tasks.workunit.client.0.smithi032.stdout:3/844: write d3/fa [1260355,93842] 0 2024-04-06T14:17:17.547 INFO:tasks.workunit.client.0.smithi032.stdout:8/944: truncate d3/d8/d12/f38 4761293 0 2024-04-06T14:17:17.548 INFO:tasks.workunit.client.0.smithi032.stdout:8/945: stat d3/d8/d3a/d4b/d136 0 2024-04-06T14:17:17.549 INFO:tasks.workunit.client.0.smithi032.stdout:8/946: dread - d3/d8/d12/d2f/f77 zero size 2024-04-06T14:17:17.558 INFO:tasks.workunit.client.0.smithi032.stdout:3/845: truncate d3/f5e 2871497 0 2024-04-06T14:17:17.578 INFO:tasks.workunit.client.1.smithi086.stdout:4/502: dread d0/d15/d20/d34/f35 [0,4194304] 0 2024-04-06T14:17:17.581 INFO:tasks.workunit.client.1.smithi086.stdout:4/503: symlink d0/d2c/d38/d54/d90/l9e 0 2024-04-06T14:17:17.582 INFO:tasks.workunit.client.0.smithi032.stdout:3/846: mknod d3/d65/d7d/d9c/db8/c146 0 2024-04-06T14:17:17.582 INFO:tasks.workunit.client.1.smithi086.stdout:4/504: stat d0/d2c/d30/d24/d5d/d8a 0 2024-04-06T14:17:17.586 INFO:tasks.workunit.client.1.smithi086.stdout:4/505: unlink d0/d2c/d30/d24/d93/f99 0 2024-04-06T14:17:17.589 INFO:tasks.workunit.client.1.smithi086.stdout:4/506: dread d0/d2c/f81 [0,4194304] 0 2024-04-06T14:17:17.591 INFO:tasks.workunit.client.0.smithi032.stdout:3/847: dwrite d3/d65/d7d/d9c/dad/f12b [0,4194304] 0 2024-04-06T14:17:17.591 INFO:tasks.workunit.client.1.smithi086.stdout:4/507: mknod d0/d15/d20/d34/c9f 0 2024-04-06T14:17:17.592 INFO:tasks.workunit.client.0.smithi032.stdout:3/848: fdatasync d3/faf 0 2024-04-06T14:17:17.592 INFO:tasks.workunit.client.1.smithi086.stdout:4/508: chown d0/d2c/d30/l64 306779720 1 2024-04-06T14:17:17.593 INFO:tasks.workunit.client.1.smithi086.stdout:4/509: fdatasync d0/d15/d2b/f5f 0 2024-04-06T14:17:17.594 INFO:tasks.workunit.client.1.smithi086.stdout:4/510: write d0/d15/d2b/f5f [3595722,103348] 0 2024-04-06T14:17:17.597 INFO:tasks.workunit.client.1.smithi086.stdout:4/511: creat d0/d2c/d30/d24/d71/fa0 x:0 0 0 2024-04-06T14:17:17.600 INFO:tasks.workunit.client.1.smithi086.stdout:4/512: creat d0/d2c/d38/d54/fa1 x:0 0 0 2024-04-06T14:17:17.601 INFO:tasks.workunit.client.0.smithi032.stdout:8/947: write d3/d8/d12/f1b [192508,64330] 0 2024-04-06T14:17:17.603 INFO:tasks.workunit.client.0.smithi032.stdout:3/849: rename d3/d65 to d3/d16/ddd/df7/d147 0 2024-04-06T14:17:17.605 INFO:tasks.workunit.client.0.smithi032.stdout:8/948: mknod d3/d8/d13b/c145 0 2024-04-06T14:17:17.608 INFO:tasks.workunit.client.1.smithi086.stdout:4/513: dwrite d0/d2c/d30/d24/f85 [0,4194304] 0 2024-04-06T14:17:17.610 INFO:tasks.workunit.client.0.smithi032.stdout:3/850: unlink d3/d16/ddd/d12d/c145 0 2024-04-06T14:17:17.611 INFO:tasks.workunit.client.1.smithi086.stdout:4/514: readlink d0/d15/d20/d74/l9b 0 2024-04-06T14:17:17.613 INFO:tasks.workunit.client.0.smithi032.stdout:3/851: readlink d3/d16/ddd/df7/d147/d7d/d9c/db8/lce 0 2024-04-06T14:17:17.614 INFO:tasks.workunit.client.0.smithi032.stdout:8/949: getdents d3/d8/d12/d83 0 2024-04-06T14:17:17.615 INFO:tasks.workunit.client.1.smithi086.stdout:4/515: rename d0/d15/f3c to d0/d2c/d30/d24/fa2 0 2024-04-06T14:17:17.616 INFO:tasks.workunit.client.1.smithi086.stdout:4/516: chown d0/d2c/d30/d24/d50/f72 238671 1 2024-04-06T14:17:17.617 INFO:tasks.workunit.client.1.smithi086.stdout:1/694: sync 2024-04-06T14:17:17.617 INFO:tasks.workunit.client.1.smithi086.stdout:0/681: sync 2024-04-06T14:17:17.617 INFO:tasks.workunit.client.1.smithi086.stdout:5/519: sync 2024-04-06T14:17:17.617 INFO:tasks.workunit.client.0.smithi032.stdout:8/950: symlink d3/d8/d12/l146 0 2024-04-06T14:17:17.622 INFO:tasks.workunit.client.0.smithi032.stdout:3/852: write d3/d16/ddd/df7/d147/d7d/d9c/dad/fc8 [1368969,18071] 0 2024-04-06T14:17:17.623 INFO:tasks.workunit.client.1.smithi086.stdout:5/520: symlink d0/d19/la3 0 2024-04-06T14:17:17.624 INFO:tasks.workunit.client.1.smithi086.stdout:1/695: symlink db/d11/d1a/le1 0 2024-04-06T14:17:17.624 INFO:tasks.workunit.client.0.smithi032.stdout:8/951: rmdir d3/d8/d3a/d4b/d136 0 2024-04-06T14:17:17.624 INFO:tasks.workunit.client.1.smithi086.stdout:5/521: fdatasync d0/d16/d18/d35/f4d 0 2024-04-06T14:17:17.625 INFO:tasks.workunit.client.1.smithi086.stdout:5/522: stat d0/d16/d18/d2a/d50/d66/l6e 0 2024-04-06T14:17:17.626 INFO:tasks.workunit.client.1.smithi086.stdout:0/682: symlink d0/d1/d2/d1a/d79/dac/dc2/ldf 0 2024-04-06T14:17:17.627 INFO:tasks.workunit.client.0.smithi032.stdout:3/853: rmdir d3/d16/ddd/df7/d147/d7d 39 2024-04-06T14:17:17.628 INFO:tasks.workunit.client.1.smithi086.stdout:4/517: write d0/de/d4c/f5a [2167462,6861] 0 2024-04-06T14:17:17.629 INFO:tasks.workunit.client.1.smithi086.stdout:1/696: fsync db/d11/d42/d53/f64 0 2024-04-06T14:17:17.629 INFO:tasks.workunit.client.1.smithi086.stdout:4/518: write d0/d15/f91 [1017934,76867] 0 2024-04-06T14:17:17.630 INFO:tasks.workunit.client.0.smithi032.stdout:8/952: write d3/d8/ff1 [482336,115991] 0 2024-04-06T14:17:17.631 INFO:tasks.workunit.client.1.smithi086.stdout:0/683: unlink d0/d15/cc1 0 2024-04-06T14:17:17.631 INFO:tasks.workunit.client.0.smithi032.stdout:3/854: rename d3/d16/ddd/df7/d147/c2f to d3/d16/ddd/df7/d147/d89/dcb/c148 0 2024-04-06T14:17:17.632 INFO:tasks.workunit.client.0.smithi032.stdout:8/953: write d3/d8/d12/f4d [1874158,49757] 0 2024-04-06T14:17:17.633 INFO:tasks.workunit.client.1.smithi086.stdout:5/523: mknod d0/d16/ca4 0 2024-04-06T14:17:17.635 INFO:tasks.workunit.client.1.smithi086.stdout:0/684: unlink d0/d1/d4/d7/lcb 0 2024-04-06T14:17:17.636 INFO:tasks.workunit.client.0.smithi032.stdout:8/954: creat d3/d8/d3a/d7b/db9/df8/f147 x:0 0 0 2024-04-06T14:17:17.637 INFO:tasks.workunit.client.0.smithi032.stdout:3/855: symlink d3/d16/ddd/df7/d147/d7d/d9c/dbc/d13e/l149 0 2024-04-06T14:17:17.637 INFO:tasks.workunit.client.1.smithi086.stdout:4/519: dwrite d0/f59 [0,4194304] 0 2024-04-06T14:17:17.639 INFO:tasks.workunit.client.1.smithi086.stdout:5/524: rename d0/d16/d18/d21/d23/d28/d94/c33 to d0/d16/d71/d89/ca5 0 2024-04-06T14:17:17.642 INFO:tasks.workunit.client.1.smithi086.stdout:5/525: readlink d0/l2c 0 2024-04-06T14:17:17.644 INFO:tasks.workunit.client.1.smithi086.stdout:5/526: chown d0/d16/c3b 12605401 1 2024-04-06T14:17:17.644 INFO:tasks.workunit.client.0.smithi032.stdout:8/955: dread d3/d8/d3a/d7b/f108 [0,4194304] 0 2024-04-06T14:17:17.645 INFO:tasks.workunit.client.1.smithi086.stdout:0/685: dwrite d0/d15/d4e/da3/d44/d8e/fd9 [0,4194304] 0 2024-04-06T14:17:17.646 INFO:tasks.workunit.client.1.smithi086.stdout:4/520: rename d0/d15/d19 to d0/d2c/d30/d24/da3 0 2024-04-06T14:17:17.647 INFO:tasks.workunit.client.1.smithi086.stdout:5/527: mkdir d0/d43/da6 0 2024-04-06T14:17:17.649 INFO:tasks.workunit.client.1.smithi086.stdout:0/686: creat d0/d15/d26/d33/d7a/fe0 x:0 0 0 2024-04-06T14:17:17.650 INFO:tasks.workunit.client.0.smithi032.stdout:3/856: dread d3/d16/ddd/df7/d147/d7d/d9c/fe1 [0,4194304] 0 2024-04-06T14:17:17.651 INFO:tasks.workunit.client.1.smithi086.stdout:5/528: mknod d0/d16/d71/d89/ca7 0 2024-04-06T14:17:17.651 INFO:tasks.workunit.client.1.smithi086.stdout:0/687: dread - d0/d1/d2/d1a/d51/f86 zero size 2024-04-06T14:17:17.655 INFO:tasks.workunit.client.1.smithi086.stdout:5/529: symlink d0/d19/la8 0 2024-04-06T14:17:17.656 INFO:tasks.workunit.client.1.smithi086.stdout:4/521: link d0/d2c/d38/d54/f8c d0/d2c/d30/d24/d93/fa4 0 2024-04-06T14:17:17.657 INFO:tasks.workunit.client.1.smithi086.stdout:0/688: fdatasync d0/d1/d2/f67 0 2024-04-06T14:17:17.658 INFO:tasks.workunit.client.1.smithi086.stdout:5/530: readlink d0/d5e/l7e 0 2024-04-06T14:17:17.658 INFO:tasks.workunit.client.0.smithi032.stdout:3/857: rename d3/d16/ddd/df7/d147/d89/cc0 to d3/d16/ddd/df7/d147/d7d/dc6/de5/c14a 0 2024-04-06T14:17:17.661 INFO:tasks.workunit.client.0.smithi032.stdout:3/858: dread - d3/d16/ddd/df7/d147/d7d/d9c/db8/f134 zero size 2024-04-06T14:17:17.665 INFO:tasks.workunit.client.1.smithi086.stdout:0/689: rename d0/d15/d4e/da3/d46/d52/dd0/l3b to d0/d15/d26/d33/d7a/le1 0 2024-04-06T14:17:17.665 INFO:tasks.workunit.client.0.smithi032.stdout:3/859: creat d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/dc9/f14b x:0 0 0 2024-04-06T14:17:17.666 INFO:tasks.workunit.client.1.smithi086.stdout:0/690: dread - d0/d15/d26/d33/d7a/daa/fca zero size 2024-04-06T14:17:17.667 INFO:tasks.workunit.client.0.smithi032.stdout:3/860: mknod d3/d16/d63/c14c 0 2024-04-06T14:17:17.670 INFO:tasks.workunit.client.0.smithi032.stdout:3/861: rmdir d3/d16/d63/d129 39 2024-04-06T14:17:17.671 INFO:tasks.workunit.client.1.smithi086.stdout:0/691: rename d0/d1/d4/d7 to d0/d1/d2/d1a/d51/dd7/de2 0 2024-04-06T14:17:17.671 INFO:tasks.workunit.client.1.smithi086.stdout:5/531: link d0/d16/d18/d21/d23/l37 d0/d16/d65/la9 0 2024-04-06T14:17:17.675 INFO:tasks.workunit.client.1.smithi086.stdout:0/692: symlink d0/d15/d4e/da3/d44/d5d/le3 0 2024-04-06T14:17:17.676 INFO:tasks.workunit.client.1.smithi086.stdout:4/522: dread d0/d2c/d30/d24/f31 [0,4194304] 0 2024-04-06T14:17:17.677 INFO:tasks.workunit.client.0.smithi032.stdout:3/862: dwrite d3/d16/ddd/df7/d147/d7d/d9c/dad/f12b [0,4194304] 0 2024-04-06T14:17:17.679 INFO:tasks.workunit.client.0.smithi032.stdout:8/956: dread d3/f31 [0,4194304] 0 2024-04-06T14:17:17.680 INFO:tasks.workunit.client.1.smithi086.stdout:5/532: creat d0/d16/d18/d21/d23/d28/faa x:0 0 0 2024-04-06T14:17:17.681 INFO:tasks.workunit.client.1.smithi086.stdout:0/693: mknod d0/d1/d4/de/dae/ce4 0 2024-04-06T14:17:17.683 INFO:tasks.workunit.client.0.smithi032.stdout:3/863: rmdir d3/d16/ddd/df7/d147/d7d/d9c/db8 39 2024-04-06T14:17:17.683 INFO:tasks.workunit.client.1.smithi086.stdout:4/523: unlink d0/d15/l47 0 2024-04-06T14:17:17.684 INFO:tasks.workunit.client.1.smithi086.stdout:5/533: mknod d0/d16/d18/d21/d23/d28/d94/d52/d7c/cab 0 2024-04-06T14:17:17.685 INFO:tasks.workunit.client.1.smithi086.stdout:5/534: truncate d0/d16/d18/f9d 63038 0 2024-04-06T14:17:17.686 INFO:tasks.workunit.client.0.smithi032.stdout:8/957: dwrite d3/d8/d12/f4d [0,4194304] 0 2024-04-06T14:17:17.686 INFO:tasks.workunit.client.1.smithi086.stdout:0/694: unlink d0/d15/d4e/da3/d73/f7c 0 2024-04-06T14:17:17.687 INFO:tasks.workunit.client.1.smithi086.stdout:4/524: write d0/f36 [4583516,105926] 0 2024-04-06T14:17:17.689 INFO:tasks.workunit.client.1.smithi086.stdout:0/695: symlink d0/d15/d4e/da3/d46/le5 0 2024-04-06T14:17:17.690 INFO:tasks.workunit.client.0.smithi032.stdout:8/958: fsync d3/d8/d3a/d4b/fbe 0 2024-04-06T14:17:17.691 INFO:tasks.workunit.client.1.smithi086.stdout:5/535: rename d0/d16/d18/d2a/f46 to d0/d17/fac 0 2024-04-06T14:17:17.692 INFO:tasks.workunit.client.0.smithi032.stdout:8/959: read - d3/d8/d12/d2f/d4a/d60/d90/dc5/f110 zero size 2024-04-06T14:17:17.693 INFO:tasks.workunit.client.0.smithi032.stdout:3/864: rename d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/dc9/d104 to d3/d14d 0 2024-04-06T14:17:17.696 INFO:tasks.workunit.client.1.smithi086.stdout:0/696: rename d0/d15/d4e/da3/d44/d8e/dd2 to d0/d1/d2/d1a/d79/dac/db7/de6 0 2024-04-06T14:17:17.697 INFO:tasks.workunit.client.1.smithi086.stdout:5/536: mkdir d0/d16/d18/d35/d9e/dad 0 2024-04-06T14:17:17.699 INFO:tasks.workunit.client.0.smithi032.stdout:8/960: rename d3/d8/d12/d2f to d3/d8/d7/dd5/d148 0 2024-04-06T14:17:17.701 INFO:tasks.workunit.client.1.smithi086.stdout:0/697: creat d0/d15/d4e/da3/fe7 x:0 0 0 2024-04-06T14:17:17.703 INFO:tasks.workunit.client.1.smithi086.stdout:0/698: dread - d0/d15/d26/d33/d9f/fbf zero size 2024-04-06T14:17:17.706 INFO:tasks.workunit.client.0.smithi032.stdout:3/865: write d3/d16/ddd/df7/d147/d7d/d9c/dbc/fe3 [251090,108007] 0 2024-04-06T14:17:17.706 INFO:tasks.workunit.client.0.smithi032.stdout:8/961: write d3/d8/d7/d8a/fab [543774,19152] 0 2024-04-06T14:17:17.709 INFO:tasks.workunit.client.0.smithi032.stdout:3/866: fdatasync d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/fc3 0 2024-04-06T14:17:17.711 INFO:tasks.workunit.client.0.smithi032.stdout:8/962: write d3/d8/d3a/d4b/de0/fe5 [80297,104619] 0 2024-04-06T14:17:17.711 INFO:tasks.workunit.client.1.smithi086.stdout:4/525: dread d0/d15/d20/d74/f7b [0,4194304] 0 2024-04-06T14:17:17.713 INFO:tasks.workunit.client.0.smithi032.stdout:3/867: mkdir d3/d16/ddd/df7/d147/d7d/d9c/dbc/d128/d14e 0 2024-04-06T14:17:17.713 INFO:tasks.workunit.client.1.smithi086.stdout:4/526: rmdir d0/d15/d20/d74 39 2024-04-06T14:17:17.715 INFO:tasks.workunit.client.1.smithi086.stdout:4/527: symlink d0/d15/d53/la5 0 2024-04-06T14:17:17.716 INFO:tasks.workunit.client.1.smithi086.stdout:0/699: write d0/d15/d26/d33/f57 [2172957,106221] 0 2024-04-06T14:17:17.716 INFO:tasks.workunit.client.0.smithi032.stdout:3/868: getdents d3/d16/ddd/df7/d147/d7d 0 2024-04-06T14:17:17.717 INFO:tasks.workunit.client.0.smithi032.stdout:3/869: dread - d3/d12e/f138 zero size 2024-04-06T14:17:17.719 INFO:tasks.workunit.client.0.smithi032.stdout:3/870: chown d3/f4 0 1 2024-04-06T14:17:17.720 INFO:tasks.workunit.client.1.smithi086.stdout:5/537: dwrite d0/d19/f3e [0,4194304] 0 2024-04-06T14:17:17.720 INFO:tasks.workunit.client.0.smithi032.stdout:8/963: dwrite d3/d8/d3a/d7b/db9/df8/d106/d124/f13d [0,4194304] 0 2024-04-06T14:17:17.722 INFO:tasks.workunit.client.0.smithi032.stdout:2/791: sync 2024-04-06T14:17:17.724 INFO:tasks.workunit.client.1.smithi086.stdout:5/538: truncate d0/d16/d18/d21/d23/d28/faa 484033 0 2024-04-06T14:17:17.725 INFO:tasks.workunit.client.0.smithi032.stdout:8/964: symlink d3/d8/d7/dd5/d148/d46/def/d128/d84/l149 0 2024-04-06T14:17:17.725 INFO:tasks.workunit.client.1.smithi086.stdout:4/528: dwrite d0/d2c/f81 [0,4194304] 0 2024-04-06T14:17:17.726 INFO:tasks.workunit.client.0.smithi032.stdout:2/792: dread - d0/d30/d39/d59/d67/d74/d78/fef zero size 2024-04-06T14:17:17.729 INFO:tasks.workunit.client.1.smithi086.stdout:0/700: link d0/d1/d4/f6b d0/d15/d4e/da3/d46/d52/fe8 0 2024-04-06T14:17:17.732 INFO:tasks.workunit.client.1.smithi086.stdout:5/539: creat d0/d16/d18/d21/d23/d28/d94/fae x:0 0 0 2024-04-06T14:17:17.732 INFO:tasks.workunit.client.0.smithi032.stdout:2/793: creat d0/d30/d39/d59/d67/d80/f10a x:0 0 0 2024-04-06T14:17:17.734 INFO:tasks.workunit.client.1.smithi086.stdout:4/529: creat d0/d2c/d30/d24/d50/d79/d89/fa6 x:0 0 0 2024-04-06T14:17:17.735 INFO:tasks.workunit.client.0.smithi032.stdout:2/794: symlink d0/d6e/dae/l10b 0 2024-04-06T14:17:17.736 INFO:tasks.workunit.client.1.smithi086.stdout:5/540: symlink d0/d16/d18/d21/laf 0 2024-04-06T14:17:17.738 INFO:tasks.workunit.client.1.smithi086.stdout:4/530: write d0/f10 [410888,7645] 0 2024-04-06T14:17:17.739 INFO:tasks.workunit.client.1.smithi086.stdout:0/701: dwrite d0/d1/d2/d1a/d79/fb2 [0,4194304] 0 2024-04-06T14:17:17.740 INFO:tasks.workunit.client.0.smithi032.stdout:3/871: dread d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/f9b [0,4194304] 0 2024-04-06T14:17:17.740 INFO:tasks.workunit.client.0.smithi032.stdout:2/795: mknod d0/d30/d39/d59/d67/d74/d78/d57/c10c 0 2024-04-06T14:17:17.742 INFO:tasks.workunit.client.1.smithi086.stdout:5/541: symlink d0/d16/d18/d21/d23/d28/lb0 0 2024-04-06T14:17:17.743 INFO:tasks.workunit.client.1.smithi086.stdout:0/702: fsync d0/d15/d4e/da3/d46/d52/dd0/f69 0 2024-04-06T14:17:17.744 INFO:tasks.workunit.client.0.smithi032.stdout:3/872: fdatasync d3/d16/d63/f83 0 2024-04-06T14:17:17.745 INFO:tasks.workunit.client.1.smithi086.stdout:4/531: link d0/d2c/d38/d54/d83/f77 d0/d2c/d38/fa7 0 2024-04-06T14:17:17.747 INFO:tasks.workunit.client.0.smithi032.stdout:8/965: dwrite d3/d8/d7/f16 [0,4194304] 0 2024-04-06T14:17:17.749 INFO:tasks.workunit.client.0.smithi032.stdout:2/796: rename d0/d30/d39/d59/d67/d74/d78/d8c/db0/d95 to d0/d6e/d8d/d10d 0 2024-04-06T14:17:17.749 INFO:tasks.workunit.client.1.smithi086.stdout:4/532: unlink d0/d15/d53/f5e 0 2024-04-06T14:17:17.750 INFO:tasks.workunit.client.0.smithi032.stdout:8/966: chown d3/d8/d7/fc8 10543648 1 2024-04-06T14:17:17.750 INFO:tasks.workunit.client.0.smithi032.stdout:3/873: creat d3/d16/ddd/df7/d147/d7d/d9c/dad/d111/f14f x:0 0 0 2024-04-06T14:17:17.751 INFO:tasks.workunit.client.1.smithi086.stdout:5/542: dread d0/d16/f39 [0,4194304] 0 2024-04-06T14:17:17.751 INFO:tasks.workunit.client.1.smithi086.stdout:0/703: rename d0/d15/d4e/da3/d44/d8e/c8a to d0/d15/d4e/da3/d44/d5d/ce9 0 2024-04-06T14:17:17.751 INFO:tasks.workunit.client.1.smithi086.stdout:5/543: stat d0/d43/c48 0 2024-04-06T14:17:17.752 INFO:tasks.workunit.client.1.smithi086.stdout:0/704: chown d0/d15/d4e/da3/d44/d5d/le3 87079 1 2024-04-06T14:17:17.754 INFO:tasks.workunit.client.1.smithi086.stdout:2/655: sync 2024-04-06T14:17:17.754 INFO:tasks.workunit.client.1.smithi086.stdout:8/714: sync 2024-04-06T14:17:17.755 INFO:tasks.workunit.client.1.smithi086.stdout:4/533: creat d0/d2c/d30/d24/d50/d79/fa8 x:0 0 0 2024-04-06T14:17:17.756 INFO:tasks.workunit.client.0.smithi032.stdout:8/967: mknod d3/d8/d3a/c14a 0 2024-04-06T14:17:17.758 INFO:tasks.workunit.client.0.smithi032.stdout:3/874: rename d3/d16/ddd/df7/f118 to d3/d16/ddd/df7/d147/d7d/dc6/df0/f150 0 2024-04-06T14:17:17.759 INFO:tasks.workunit.client.0.smithi032.stdout:2/797: dwrite d0/d6e/f86 [4194304,4194304] 0 2024-04-06T14:17:17.761 INFO:tasks.workunit.client.0.smithi032.stdout:2/798: chown d0/d6e/dae 608122 1 2024-04-06T14:17:17.761 INFO:tasks.workunit.client.0.smithi032.stdout:8/968: unlink d3/f79 0 2024-04-06T14:17:17.763 INFO:tasks.workunit.client.1.smithi086.stdout:4/534: mknod d0/d2c/d38/d54/d83/ca9 0 2024-04-06T14:17:17.763 INFO:tasks.workunit.client.0.smithi032.stdout:3/875: truncate d3/d16/f5c 2335490 0 2024-04-06T14:17:17.763 INFO:tasks.workunit.client.0.smithi032.stdout:2/799: rmdir d0/d4e/de9 39 2024-04-06T14:17:17.769 INFO:tasks.workunit.client.0.smithi032.stdout:3/876: symlink d3/d16/ddd/df7/d147/d89/dcb/l151 0 2024-04-06T14:17:17.772 INFO:tasks.workunit.client.1.smithi086.stdout:5/544: dwrite d0/d16/d18/d21/d23/d28/d94/d52/d7c/f8f [0,4194304] 0 2024-04-06T14:17:17.772 INFO:tasks.workunit.client.1.smithi086.stdout:8/715: dwrite d0/d13/fa9 [0,4194304] 0 2024-04-06T14:17:17.772 INFO:tasks.workunit.client.1.smithi086.stdout:4/535: dread d0/d2c/d38/fa7 [0,4194304] 0 2024-04-06T14:17:17.772 INFO:tasks.workunit.client.1.smithi086.stdout:2/656: rename d0/d20/d27/d2b/d35/d36/l41 to d0/d20/d27/le2 0 2024-04-06T14:17:17.773 INFO:tasks.workunit.client.1.smithi086.stdout:4/536: chown d0/d2c/d38/d54/f8c 436 1 2024-04-06T14:17:17.773 INFO:tasks.workunit.client.1.smithi086.stdout:8/716: mknod d0/d71/d22/d43/daa/cf7 0 2024-04-06T14:17:17.773 INFO:tasks.workunit.client.0.smithi032.stdout:3/877: dread - d3/d16/ddd/df7/d147/d7d/d9c/dbc/d128/f130 zero size 2024-04-06T14:17:17.775 INFO:tasks.workunit.client.0.smithi032.stdout:8/969: write d3/d8/d7/dd5/d148/d46/def/d128/fa6 [4748123,32251] 0 2024-04-06T14:17:17.775 INFO:tasks.workunit.client.0.smithi032.stdout:2/800: dwrite d0/d5a/fb1 [4194304,4194304] 0 2024-04-06T14:17:17.776 INFO:tasks.workunit.client.1.smithi086.stdout:2/657: write d0/df/d48/d53/d76/f91 [3297789,4787] 0 2024-04-06T14:17:17.781 INFO:tasks.workunit.client.1.smithi086.stdout:4/537: link d0/d15/d2b/f82 d0/d2c/d38/faa 0 2024-04-06T14:17:17.781 INFO:tasks.workunit.client.1.smithi086.stdout:0/705: dwrite d0/d15/d26/f4b [0,4194304] 0 2024-04-06T14:17:17.781 INFO:tasks.workunit.client.0.smithi032.stdout:8/970: readlink d3/l103 0 2024-04-06T14:17:17.783 INFO:tasks.workunit.client.0.smithi032.stdout:2/801: fdatasync d0/d30/d39/faa 0 2024-04-06T14:17:17.784 INFO:tasks.workunit.client.1.smithi086.stdout:0/706: chown d0/d1/d2/d1a/d51/cb8 725 1 2024-04-06T14:17:17.792 INFO:tasks.workunit.client.1.smithi086.stdout:8/717: rename d0/d71/d16/d39/c8d to d0/d71/d22/d65/d6e/cf8 0 2024-04-06T14:17:17.794 INFO:tasks.workunit.client.1.smithi086.stdout:5/545: truncate d0/d17/f79 477734 0 2024-04-06T14:17:17.795 INFO:tasks.workunit.client.0.smithi032.stdout:8/971: rename d3/d8/d7/dd5/d148/d4a/d60/d90/fe9 to d3/d8/d3a/d7b/db9/f14b 0 2024-04-06T14:17:17.795 INFO:tasks.workunit.client.1.smithi086.stdout:7/668: sync 2024-04-06T14:17:17.795 INFO:tasks.workunit.client.1.smithi086.stdout:3/692: sync 2024-04-06T14:17:17.796 INFO:tasks.workunit.client.1.smithi086.stdout:1/697: sync 2024-04-06T14:17:17.796 INFO:tasks.workunit.client.1.smithi086.stdout:6/770: sync 2024-04-06T14:17:17.796 INFO:tasks.workunit.client.1.smithi086.stdout:9/557: sync 2024-04-06T14:17:17.797 INFO:tasks.workunit.client.1.smithi086.stdout:2/658: rename d0/df/d48/d53 to d0/d20/d44/dbb/dc4/de3 0 2024-04-06T14:17:17.799 INFO:tasks.workunit.client.0.smithi032.stdout:2/802: dwrite d0/d30/d39/d59/d67/d74/d78/d57/d7c/de6/ffd [0,4194304] 0 2024-04-06T14:17:17.799 INFO:tasks.workunit.client.0.smithi032.stdout:8/972: chown d3/d8/d7/dd5/d148/d46/def/l123 274 1 2024-04-06T14:17:17.800 INFO:tasks.workunit.client.1.smithi086.stdout:0/707: truncate d0/d15/d4e/da3/d44/d5d/f5f 1134142 0 2024-04-06T14:17:17.800 INFO:tasks.workunit.client.1.smithi086.stdout:9/558: read - d5/d13/d1a/d5f/d88/d14/d19/d1b/f9d zero size 2024-04-06T14:17:17.801 INFO:tasks.workunit.client.0.smithi032.stdout:3/878: dread d3/d16/ddd/df7/d147/f39 [0,4194304] 0 2024-04-06T14:17:17.801 INFO:tasks.workunit.client.0.smithi032.stdout:8/973: fsync d3/d8/d3a/f4f 0 2024-04-06T14:17:17.803 INFO:tasks.workunit.client.1.smithi086.stdout:6/771: write d6/d13/d1a/ffb [504630,39806] 0 2024-04-06T14:17:17.803 INFO:tasks.workunit.client.0.smithi032.stdout:2/803: chown d0/d30/d39/d59/d67/d74/d78/d57/fde 422 1 2024-04-06T14:17:17.805 INFO:tasks.workunit.client.1.smithi086.stdout:3/693: chown d5/d36/f3b 8 1 2024-04-06T14:17:17.805 INFO:tasks.workunit.client.1.smithi086.stdout:0/708: dread - d0/d1/d4/fbd zero size 2024-04-06T14:17:17.807 INFO:tasks.workunit.client.1.smithi086.stdout:2/659: symlink d0/df/d5e/le4 0 2024-04-06T14:17:17.807 INFO:tasks.workunit.client.1.smithi086.stdout:6/772: stat d6/d20/la5 0 2024-04-06T14:17:17.807 INFO:tasks.workunit.client.0.smithi032.stdout:1/982: sync 2024-04-06T14:17:17.808 INFO:tasks.workunit.client.0.smithi032.stdout:7/969: sync 2024-04-06T14:17:17.808 INFO:tasks.workunit.client.0.smithi032.stdout:0/719: sync 2024-04-06T14:17:17.808 INFO:tasks.workunit.client.0.smithi032.stdout:4/859: sync 2024-04-06T14:17:17.809 INFO:tasks.workunit.client.1.smithi086.stdout:8/718: dwrite d0/d13/db6/fba [0,4194304] 0 2024-04-06T14:17:17.809 INFO:tasks.workunit.client.0.smithi032.stdout:8/974: rename d3/d8/d7/dd5/d148/d4a/ca2 to d3/d8/d7/dd5/d148/d4a/d60/d75/d85/c14c 0 2024-04-06T14:17:17.811 INFO:tasks.workunit.client.0.smithi032.stdout:0/720: chown d3/da/d25/d2b/d4a/d86 3746759 1 2024-04-06T14:17:17.811 INFO:tasks.workunit.client.1.smithi086.stdout:7/669: link d7/fd6 d7/d10/d11/d1c/d24/fe1 0 2024-04-06T14:17:17.813 INFO:tasks.workunit.client.0.smithi032.stdout:7/970: symlink d16/d20/d7a/da6/l157 0 2024-04-06T14:17:17.814 INFO:tasks.workunit.client.0.smithi032.stdout:2/804: getdents d0/d5a/dc9 0 2024-04-06T14:17:17.814 INFO:tasks.workunit.client.1.smithi086.stdout:9/559: creat d5/d13/d1a/fb6 x:0 0 0 2024-04-06T14:17:17.814 INFO:tasks.workunit.client.0.smithi032.stdout:1/983: mknod d0/d10/d16/d3f/c13f 0 2024-04-06T14:17:17.815 INFO:tasks.workunit.client.1.smithi086.stdout:2/660: dread - d0/d20/d27/d37/f98 zero size 2024-04-06T14:17:17.816 INFO:tasks.workunit.client.1.smithi086.stdout:0/709: rename d0/d15 to d0/d15/d4e/da3/d46/d52/dd0/dea 22 2024-04-06T14:17:17.816 INFO:tasks.workunit.client.0.smithi032.stdout:0/721: dread - d3/da/d35/f8a zero size 2024-04-06T14:17:17.819 INFO:tasks.workunit.client.1.smithi086.stdout:0/710: write d0/d15/d4e/da3/d44/fbc [969668,54198] 0 2024-04-06T14:17:17.820 INFO:tasks.workunit.client.1.smithi086.stdout:8/719: creat d0/d71/d16/d52/dd0/ddc/de7/dd6/ff9 x:0 0 0 2024-04-06T14:17:17.821 INFO:tasks.workunit.client.1.smithi086.stdout:3/694: dwrite d5/f77 [0,4194304] 0 2024-04-06T14:17:17.823 INFO:tasks.workunit.client.0.smithi032.stdout:7/971: fsync d16/d20/d54/de4/d2d/f41 0 2024-04-06T14:17:17.824 INFO:tasks.workunit.client.0.smithi032.stdout:3/879: dwrite d3/d16/ddd/df7/d147/d7d/d9c/db8/f134 [0,4194304] 0 2024-04-06T14:17:17.824 INFO:tasks.workunit.client.1.smithi086.stdout:7/670: chown d7/d10/d11/d1c/d36/dbc/fbd 53 1 2024-04-06T14:17:17.825 INFO:tasks.workunit.client.0.smithi032.stdout:0/722: readlink d3/da/d25/d2b/d4a/da1/lac 0 2024-04-06T14:17:17.826 INFO:tasks.workunit.client.1.smithi086.stdout:8/720: stat d0/d71/f3c 0 2024-04-06T14:17:17.826 INFO:tasks.workunit.client.1.smithi086.stdout:0/711: fsync d0/d15/f6a 0 2024-04-06T14:17:17.826 INFO:tasks.workunit.client.0.smithi032.stdout:4/860: rename d0/df/d1f/d2e/cd5 to d0/df/d23/d31/d50/d57/d5b/d9c/d109/c123 0 2024-04-06T14:17:17.828 INFO:tasks.workunit.client.0.smithi032.stdout:3/880: dread - d3/d16/ddd/df7/d147/d7d/d9c/dad/dc5/f11b zero size 2024-04-06T14:17:17.834 INFO:tasks.workunit.client.1.smithi086.stdout:8/721: chown d0/d3b/l63 3302480 1 2024-04-06T14:17:17.834 INFO:tasks.workunit.client.1.smithi086.stdout:4/538: dread d0/d15/d20/f44 [4194304,4194304] 0 2024-04-06T14:17:17.835 INFO:tasks.workunit.client.0.smithi032.stdout:8/975: rename d3/d8/l11f to d3/d8/d7/dd5/d148/d46/l14d 0 2024-04-06T14:17:17.840 INFO:tasks.workunit.client.0.smithi032.stdout:2/805: dwrite d0/d30/d39/d59/d67/d74/d78/d57/fd1 [0,4194304] 0 2024-04-06T14:17:17.842 INFO:tasks.workunit.client.0.smithi032.stdout:1/984: creat d0/d9/d94/f140 x:0 0 0 2024-04-06T14:17:17.842 INFO:tasks.workunit.client.1.smithi086.stdout:3/695: rename d5/df/c82 to d5/d36/d7d/cde 0 2024-04-06T14:17:17.842 INFO:tasks.workunit.client.0.smithi032.stdout:2/806: fsync d0/d30/d39/d59/d67/d80/fd0 0 2024-04-06T14:17:17.845 INFO:tasks.workunit.client.0.smithi032.stdout:3/881: chown d3/d16/ddd/df7/d147/l49 4015659 1 2024-04-06T14:17:17.851 INFO:tasks.workunit.client.0.smithi032.stdout:1/985: fdatasync d0/f2c 0 2024-04-06T14:17:17.855 INFO:tasks.workunit.client.0.smithi032.stdout:3/882: creat d3/d14d/d12c/f152 x:0 0 0 2024-04-06T14:17:17.857 INFO:tasks.workunit.client.1.smithi086.stdout:7/671: rmdir d7/d10/d11 39 2024-04-06T14:17:17.858 INFO:tasks.workunit.client.0.smithi032.stdout:1/986: mkdir d0/d9/d1f/d7c/da9/d141 0 2024-04-06T14:17:17.860 INFO:tasks.workunit.client.0.smithi032.stdout:0/723: write d3/da/d25/d2b/d4a/f62 [2323740,65547] 0 2024-04-06T14:17:17.861 INFO:tasks.workunit.client.1.smithi086.stdout:5/546: dwrite d0/d16/d18/f7f [0,4194304] 0 2024-04-06T14:17:17.862 INFO:tasks.workunit.client.1.smithi086.stdout:6/773: dread d6/d9/d43/fed [0,4194304] 0 2024-04-06T14:17:17.863 INFO:tasks.workunit.client.1.smithi086.stdout:9/560: write d5/d13/d47/f92 [208343,77764] 0 2024-04-06T14:17:17.863 INFO:tasks.workunit.client.0.smithi032.stdout:4/861: dwrite d0/fc3 [0,4194304] 0 2024-04-06T14:17:17.866 INFO:tasks.workunit.client.1.smithi086.stdout:8/722: rename d0/d71/d22/d43/d4d/c7e to d0/d71/d16/d52/cfa 0 2024-04-06T14:17:17.866 INFO:tasks.workunit.client.1.smithi086.stdout:9/561: write f0 [3149805,19465] 0 2024-04-06T14:17:17.868 INFO:tasks.workunit.client.0.smithi032.stdout:6/966: sync 2024-04-06T14:17:17.868 INFO:tasks.workunit.client.1.smithi086.stdout:3/696: symlink d5/d36/d56/dd3/ldf 0 2024-04-06T14:17:17.870 INFO:tasks.workunit.client.1.smithi086.stdout:0/712: dread d0/d15/fce [0,4194304] 0 2024-04-06T14:17:17.871 INFO:tasks.workunit.client.0.smithi032.stdout:3/883: dread - d3/d16/ddd/df7/d147/d7d/d9c/dbc/d128/f130 zero size 2024-04-06T14:17:17.871 INFO:tasks.workunit.client.1.smithi086.stdout:4/539: fsync d0/d15/d2b/f82 0 2024-04-06T14:17:17.872 INFO:tasks.workunit.client.1.smithi086.stdout:1/698: dwrite db/d13/d55/f59 [0,4194304] 0 2024-04-06T14:17:17.872 INFO:tasks.workunit.client.0.smithi032.stdout:0/724: fsync d3/da/d35/f4e 0 2024-04-06T14:17:17.875 INFO:tasks.workunit.client.0.smithi032.stdout:7/972: dwrite d16/d20/d54/de4/fb2 [0,4194304] 0 2024-04-06T14:17:17.877 INFO:tasks.workunit.client.0.smithi032.stdout:4/862: stat d0/df/d23/d31/d50/d57/d5b/d9c/dc8/lf6 0 2024-04-06T14:17:17.882 INFO:tasks.workunit.client.1.smithi086.stdout:9/562: chown d5/d13/d1a/d5f/d88/d14/f4e 43399862 1 2024-04-06T14:17:17.882 INFO:tasks.workunit.client.0.smithi032.stdout:1/987: dwrite d0/d10/d16/d3f/d98/f103 [0,4194304] 0 2024-04-06T14:17:17.883 INFO:tasks.workunit.client.1.smithi086.stdout:2/661: dwrite d0/d20/d44/dbb/dc4/de3/d76/f9b [0,4194304] 0 2024-04-06T14:17:17.886 INFO:tasks.workunit.client.0.smithi032.stdout:1/988: read - d0/d9/d1f/d72/f13b zero size 2024-04-06T14:17:17.888 INFO:tasks.workunit.client.1.smithi086.stdout:7/672: write d7/d10/d11/d1c/d24/d4d/daf/db5/f8a [1894288,13248] 0 2024-04-06T14:17:17.890 INFO:tasks.workunit.client.1.smithi086.stdout:2/662: chown d0/d20/d44/dbb/dc4/de3 2 1 2024-04-06T14:17:17.892 INFO:tasks.workunit.client.1.smithi086.stdout:6/774: symlink d6/d13/d42/l106 0 2024-04-06T14:17:17.894 INFO:tasks.workunit.client.1.smithi086.stdout:0/713: dread d0/d15/d26/f4b [0,4194304] 0 2024-04-06T14:17:17.895 INFO:tasks.workunit.client.0.smithi032.stdout:8/976: dread d3/d8/d7/dd5/d148/d46/d125/fb3 [0,4194304] 0 2024-04-06T14:17:17.895 INFO:tasks.workunit.client.0.smithi032.stdout:8/977: read - d3/d69/fe6 zero size 2024-04-06T14:17:17.899 INFO:tasks.workunit.client.1.smithi086.stdout:4/540: creat d0/de/fab x:0 0 0 2024-04-06T14:17:17.899 INFO:tasks.workunit.client.1.smithi086.stdout:3/697: creat d5/d36/d56/dd3/fe0 x:0 0 0 2024-04-06T14:17:17.901 INFO:tasks.workunit.client.1.smithi086.stdout:1/699: creat db/d11/d1a/d2a/d9a/d68/daf/dc2/fe2 x:0 0 0 2024-04-06T14:17:17.902 INFO:tasks.workunit.client.0.smithi032.stdout:8/978: chown d3/d8/d3a/d7b/db9/ddf/f138 22 1 2024-04-06T14:17:17.902 INFO:tasks.workunit.client.0.smithi032.stdout:2/807: dread d0/d5a/f8e [0,4194304] 0 2024-04-06T14:17:17.906 INFO:tasks.workunit.client.0.smithi032.stdout:1/989: dread d0/d10/d16/f6d [0,4194304] 0 2024-04-06T14:17:17.907 INFO:tasks.workunit.client.0.smithi032.stdout:1/990: fdatasync d0/d10/d78/f135 0 2024-04-06T14:17:17.912 INFO:tasks.workunit.client.1.smithi086.stdout:1/700: mknod db/d11/d1a/d79/dce/ce3 0 2024-04-06T14:17:17.914 INFO:tasks.workunit.client.1.smithi086.stdout:8/723: getdents d0/d71/d16/d39 0 2024-04-06T14:17:17.915 INFO:tasks.workunit.client.1.smithi086.stdout:5/547: dread d0/d16/d71/f76 [0,4194304] 0 2024-04-06T14:17:17.916 INFO:tasks.workunit.client.1.smithi086.stdout:3/698: rename d5/d36/d5a/d6d/l9c to d5/d36/d5a/d6d/d72/d97/le1 0 2024-04-06T14:17:17.920 INFO:tasks.workunit.client.1.smithi086.stdout:4/541: getdents d0/d2c/d30/d24/d50 0 2024-04-06T14:17:17.921 INFO:tasks.workunit.client.1.smithi086.stdout:3/699: mkdir d5/d36/d5a/d6d/d72/d9a/de2 0 2024-04-06T14:17:17.921 INFO:tasks.workunit.client.1.smithi086.stdout:5/548: chown d0/l72 1 1 2024-04-06T14:17:17.922 INFO:tasks.workunit.client.1.smithi086.stdout:4/542: chown d0/l1b 947201 1 2024-04-06T14:17:17.922 INFO:tasks.workunit.client.1.smithi086.stdout:8/724: creat d0/d71/d22/d43/d6d/d75/dee/ffb x:0 0 0 2024-04-06T14:17:17.924 INFO:tasks.workunit.client.1.smithi086.stdout:5/549: read - d0/d43/f9c zero size 2024-04-06T14:17:17.926 INFO:tasks.workunit.client.1.smithi086.stdout:7/673: write d7/d10/d11/f91 [1063459,117365] 0 2024-04-06T14:17:17.926 INFO:tasks.workunit.client.1.smithi086.stdout:9/563: write d5/d13/d1a/d5f/d88/d14/d19/d1b/f6f [1309776,77017] 0 2024-04-06T14:17:17.926 INFO:tasks.workunit.client.1.smithi086.stdout:0/714: write d0/d15/d26/f2c [3008528,2497] 0 2024-04-06T14:17:17.927 INFO:tasks.workunit.client.1.smithi086.stdout:3/700: fsync d5/d36/f3b 0 2024-04-06T14:17:17.930 INFO:tasks.workunit.client.1.smithi086.stdout:2/663: write d0/d20/d27/d2b/f45 [231590,58230] 0 2024-04-06T14:17:17.932 INFO:tasks.workunit.client.1.smithi086.stdout:6/775: dread d6/d9/db/f16 [0,4194304] 0 2024-04-06T14:17:17.932 INFO:tasks.workunit.client.0.smithi032.stdout:4/863: fdatasync d0/df/d2b/fd9 0 2024-04-06T14:17:17.933 INFO:tasks.workunit.client.1.smithi086.stdout:1/701: write db/d11/d1a/d2a/f4c [61265,77007] 0 2024-04-06T14:17:17.934 INFO:tasks.workunit.client.1.smithi086.stdout:8/725: chown d0/d71/d22/d43/d6d/d75/db2/fcc 352672 1 2024-04-06T14:17:17.934 INFO:tasks.workunit.client.0.smithi032.stdout:7/973: mkdir d16/d20/d54/de4/de0/df9/d158 0 2024-04-06T14:17:17.934 INFO:tasks.workunit.client.1.smithi086.stdout:3/701: creat d5/df/fe3 x:0 0 0 2024-04-06T14:17:17.935 INFO:tasks.workunit.client.1.smithi086.stdout:9/564: stat d5/d13/d1a/d5f/d88/d14/d19/d1b/f9d 0 2024-04-06T14:17:17.937 INFO:tasks.workunit.client.1.smithi086.stdout:4/543: creat d0/d15/d20/d74/d8d/fac x:0 0 0 2024-04-06T14:17:17.938 INFO:tasks.workunit.client.1.smithi086.stdout:8/726: chown d0/d13/f68 74469273 1 2024-04-06T14:17:17.943 INFO:tasks.workunit.client.1.smithi086.stdout:6/776: dread d6/d9/da9/fec [0,4194304] 0 2024-04-06T14:17:17.945 INFO:tasks.workunit.client.1.smithi086.stdout:5/550: creat d0/d16/d18/d21/d23/d28/d94/d52/fb1 x:0 0 0 2024-04-06T14:17:17.949 INFO:tasks.workunit.client.1.smithi086.stdout:6/777: write d6/f7 [3009520,34262] 0 2024-04-06T14:17:17.949 INFO:tasks.workunit.client.1.smithi086.stdout:0/715: dread d0/d15/d4e/f42 [0,4194304] 0 2024-04-06T14:17:17.951 INFO:tasks.workunit.client.1.smithi086.stdout:1/702: link db/d13/d55/f59 db/d11/d1a/d79/da5/fe4 0 2024-04-06T14:17:17.952 INFO:tasks.workunit.client.1.smithi086.stdout:0/716: dread - d0/d1/d4/fbd zero size 2024-04-06T14:17:17.953 INFO:tasks.workunit.client.1.smithi086.stdout:3/702: rename d5/d26/d4f/db4/da7/d89/f9b to d5/d36/d56/dd3/fe4 0 2024-04-06T14:17:17.957 INFO:tasks.workunit.client.1.smithi086.stdout:4/544: dwrite d0/d2c/d38/d54/d83/f98 [0,4194304] 0 2024-04-06T14:17:17.960 INFO:tasks.workunit.client.1.smithi086.stdout:9/565: write d5/d13/d1a/d5f/d88/d14/d19/d25/f67 [1896183,90672] 0 2024-04-06T14:17:17.960 INFO:tasks.workunit.client.1.smithi086.stdout:6/778: creat d6/d13/d1a/d22/dcf/db2/f107 x:0 0 0 2024-04-06T14:17:17.960 INFO:tasks.workunit.client.1.smithi086.stdout:5/551: symlink d0/d16/d18/d21/d23/lb2 0 2024-04-06T14:17:17.961 INFO:tasks.workunit.client.1.smithi086.stdout:0/717: mknod d0/d1/d2/d1a/d79/dac/db7/ceb 0 2024-04-06T14:17:17.962 INFO:tasks.workunit.client.1.smithi086.stdout:1/703: creat db/d11/d1a/d79/dce/fe5 x:0 0 0 2024-04-06T14:17:17.963 INFO:tasks.workunit.client.1.smithi086.stdout:8/727: link d0/d71/d16/d52/d5f/l86 d0/d71/d22/d65/d6e/d98/ddb/lfc 0 2024-04-06T14:17:17.964 INFO:tasks.workunit.client.1.smithi086.stdout:2/664: dwrite d0/d20/d27/d2b/d35/d36/fa2 [0,4194304] 0 2024-04-06T14:17:17.966 INFO:tasks.workunit.client.1.smithi086.stdout:4/545: fsync d0/d2c/d30/d24/d93/fa4 0 2024-04-06T14:17:17.966 INFO:tasks.workunit.client.1.smithi086.stdout:6/779: symlink d6/d13/d5f/d60/d45/d8a/l108 0 2024-04-06T14:17:17.970 INFO:tasks.workunit.client.1.smithi086.stdout:3/703: getdents d5/d33/d5d 0 2024-04-06T14:17:17.971 INFO:tasks.workunit.client.1.smithi086.stdout:4/546: chown d0/d2c/d30/d24/da3/d88 361037 1 2024-04-06T14:17:17.972 INFO:tasks.workunit.client.1.smithi086.stdout:6/780: write d6/d13/d1a/ffb [285451,14246] 0 2024-04-06T14:17:17.974 INFO:tasks.workunit.client.1.smithi086.stdout:0/718: rename d0/d15/d4e/da3/l77 to d0/d15/d26/d33/d7a/lec 0 2024-04-06T14:17:17.977 INFO:tasks.workunit.client.1.smithi086.stdout:5/552: getdents d0 0 2024-04-06T14:17:17.979 INFO:tasks.workunit.client.1.smithi086.stdout:3/704: rename d5/d33/l30 to d5/d36/d56/le5 0 2024-04-06T14:17:17.980 INFO:tasks.workunit.client.1.smithi086.stdout:8/728: truncate d0/d71/d22/d43/d6d/d75/db2/fcc 652834 0 2024-04-06T14:17:17.982 INFO:tasks.workunit.client.1.smithi086.stdout:1/704: dwrite db/d11/d42/fbf [0,4194304] 0 2024-04-06T14:17:17.982 INFO:tasks.workunit.client.1.smithi086.stdout:0/719: creat d0/d1/d2/d5b/fed x:0 0 0 2024-04-06T14:17:17.983 INFO:tasks.workunit.client.1.smithi086.stdout:9/566: dread d5/d13/d1a/d5f/d88/d14/d19/d25/f67 [0,4194304] 0 2024-04-06T14:17:17.985 INFO:tasks.workunit.client.1.smithi086.stdout:1/705: stat db/d13/f54 0 2024-04-06T14:17:17.991 INFO:tasks.workunit.client.0.smithi032.stdout:0/725: rename d3/da/d35 to d3/da/d25/d2b/d4a/d79/d97/de0/df5 0 2024-04-06T14:17:17.993 INFO:tasks.workunit.client.1.smithi086.stdout:4/547: dwrite d0/d15/d20/d34/f55 [4194304,4194304] 0 2024-04-06T14:17:17.995 INFO:tasks.workunit.client.0.smithi032.stdout:9/871: sync 2024-04-06T14:17:17.995 INFO:tasks.workunit.client.0.smithi032.stdout:5/836: sync 2024-04-06T14:17:17.997 INFO:tasks.workunit.client.0.smithi032.stdout:1/991: mknod d0/d41/d5b/d111/c142 0 2024-04-06T14:17:17.999 INFO:tasks.workunit.client.0.smithi032.stdout:0/726: dread d3/da/d25/d2b/f3d [0,4194304] 0 2024-04-06T14:17:18.000 INFO:tasks.workunit.client.0.smithi032.stdout:1/992: chown d0/da/d3e/d75/dd7/l139 29 1 2024-04-06T14:17:18.003 INFO:tasks.workunit.client.1.smithi086.stdout:2/665: dread d0/d20/d27/d2b/d35/d36/d4d/d6f/f77 [0,4194304] 0 2024-04-06T14:17:18.003 INFO:tasks.workunit.client.1.smithi086.stdout:5/553: dwrite d0/d5e/f60 [0,4194304] 0 2024-04-06T14:17:18.004 INFO:tasks.workunit.client.1.smithi086.stdout:4/548: unlink d0/d2c/d30/d24/l7d 0 2024-04-06T14:17:18.007 INFO:tasks.workunit.client.1.smithi086.stdout:2/666: truncate d0/d20/f2e 2271995 0 2024-04-06T14:17:18.008 INFO:tasks.workunit.client.0.smithi032.stdout:8/979: mknod d3/d8/d7/d13a/c14e 0 2024-04-06T14:17:18.009 INFO:tasks.workunit.client.1.smithi086.stdout:5/554: fsync d0/d16/d18/d2a/d50/f7b 0 2024-04-06T14:17:18.009 INFO:tasks.workunit.client.1.smithi086.stdout:4/549: fdatasync d0/d2c/d30/d24/f2d 0 2024-04-06T14:17:18.010 INFO:tasks.workunit.client.1.smithi086.stdout:2/667: creat d0/df/d48/dce/fe5 x:0 0 0 2024-04-06T14:17:18.013 INFO:tasks.workunit.client.0.smithi032.stdout:9/872: mkdir d2/d15/d55/d59/d115 0 2024-04-06T14:17:18.013 INFO:tasks.workunit.client.1.smithi086.stdout:4/550: truncate d0/de/fab 869407 0 2024-04-06T14:17:18.014 INFO:tasks.workunit.client.1.smithi086.stdout:3/705: truncate d5/d36/f67 3811619 0 2024-04-06T14:17:18.017 INFO:tasks.workunit.client.1.smithi086.stdout:0/720: write d0/d15/d4e/da3/f39 [1938393,120617] 0 2024-04-06T14:17:18.023 INFO:tasks.workunit.client.1.smithi086.stdout:2/668: rename d0/df/d48/d7d/fde to d0/d20/d80/dbf/ddc/fe6 0 2024-04-06T14:17:18.023 INFO:tasks.workunit.client.0.smithi032.stdout:8/980: creat d3/d8/d7/dd5/d148/d46/f14f x:0 0 0 2024-04-06T14:17:18.023 INFO:tasks.workunit.client.0.smithi032.stdout:6/967: getdents d4/d5/d14/d80 0 2024-04-06T14:17:18.025 INFO:tasks.workunit.client.0.smithi032.stdout:8/981: chown d3/d8/d7/dd5/d148/d4a/d60/d75/d85/cf4 355307 1 2024-04-06T14:17:18.025 INFO:tasks.workunit.client.1.smithi086.stdout:8/729: dwrite d0/d71/f9f [0,4194304] 0 2024-04-06T14:17:18.025 INFO:tasks.workunit.client.0.smithi032.stdout:3/884: getdents d3/d16/ddd/df7/d147/d7d/dc6 0 2024-04-06T14:17:18.027 INFO:tasks.workunit.client.1.smithi086.stdout:9/567: dread d5/d13/d1a/d5f/d88/d14/f51 [0,4194304] 0 2024-04-06T14:17:18.028 INFO:tasks.workunit.client.0.smithi032.stdout:5/837: symlink d16/d58/d54/l125 0 2024-04-06T14:17:18.029 INFO:tasks.workunit.client.0.smithi032.stdout:5/838: truncate d16/d18/d33/f41 870759 0 2024-04-06T14:17:18.032 INFO:tasks.workunit.client.1.smithi086.stdout:2/669: write d0/d20/d27/d2b/faf [286542,16650] 0 2024-04-06T14:17:18.037 INFO:tasks.workunit.client.1.smithi086.stdout:4/551: dread d0/d2c/d30/d24/f85 [0,4194304] 0 2024-04-06T14:17:18.037 INFO:tasks.workunit.client.1.smithi086.stdout:1/706: dwrite db/d11/f17 [0,4194304] 0 2024-04-06T14:17:18.038 INFO:tasks.workunit.client.1.smithi086.stdout:8/730: mknod d0/d71/d16/cfd 0 2024-04-06T14:17:18.040 INFO:tasks.workunit.client.0.smithi032.stdout:0/727: rmdir d3/da/d25/d57/dd9 0 2024-04-06T14:17:18.042 INFO:tasks.workunit.client.1.smithi086.stdout:3/706: dread d5/d26/d4f/db4/da7/f63 [0,4194304] 0 2024-04-06T14:17:18.043 INFO:tasks.workunit.client.1.smithi086.stdout:2/670: mkdir d0/df/d5e/d9d/de7 0 2024-04-06T14:17:18.043 INFO:tasks.workunit.client.1.smithi086.stdout:1/707: dread - db/d11/d1f/fd9 zero size 2024-04-06T14:17:18.048 INFO:tasks.workunit.client.1.smithi086.stdout:5/555: link d0/d17/f99 d0/d16/d18/d35/d9e/fb3 0 2024-04-06T14:17:18.050 INFO:tasks.workunit.client.0.smithi032.stdout:1/993: getdents d0/da/d3e/d75 0 2024-04-06T14:17:18.053 INFO:tasks.workunit.client.1.smithi086.stdout:8/731: rmdir d0/d71/d22/d65/d6e 39 2024-04-06T14:17:18.054 INFO:tasks.workunit.client.1.smithi086.stdout:1/708: unlink db/d11/d1a/d79/da5/cbe 0 2024-04-06T14:17:18.055 INFO:tasks.workunit.client.0.smithi032.stdout:0/728: dwrite d3/da/f53 [0,4194304] 0 2024-04-06T14:17:18.055 INFO:tasks.workunit.client.1.smithi086.stdout:4/552: dwrite d0/d2c/d30/f6c [0,4194304] 0 2024-04-06T14:17:18.057 INFO:tasks.workunit.client.1.smithi086.stdout:0/721: write d0/d1/d2/d1a/d51/dd7/de2/f27 [1973506,56032] 0 2024-04-06T14:17:18.060 INFO:tasks.workunit.client.1.smithi086.stdout:5/556: write d0/d16/d71/f76 [3851173,61735] 0 2024-04-06T14:17:18.061 INFO:tasks.workunit.client.0.smithi032.stdout:5/839: getdents d16/d18/d33/d35/d88/dbd/dc5 0 2024-04-06T14:17:18.062 INFO:tasks.workunit.client.1.smithi086.stdout:3/707: dread d5/df/f13 [0,4194304] 0 2024-04-06T14:17:18.064 INFO:tasks.workunit.client.1.smithi086.stdout:1/709: fsync db/d11/d1f/f44 0 2024-04-06T14:17:18.065 INFO:tasks.workunit.client.1.smithi086.stdout:2/671: dwrite d0/fe [0,4194304] 0 2024-04-06T14:17:18.067 INFO:tasks.workunit.client.0.smithi032.stdout:6/968: creat d4/d5/d28/f145 x:0 0 0 2024-04-06T14:17:18.067 INFO:tasks.workunit.client.1.smithi086.stdout:5/557: chown d0/d16/d18/d21/d23/d28/d94/d52/d7c/c91 1 1 2024-04-06T14:17:18.073 INFO:tasks.workunit.client.0.smithi032.stdout:5/840: getdents d16/d18/d33/d35/d88/dbd/d120 0 2024-04-06T14:17:18.075 INFO:tasks.workunit.client.1.smithi086.stdout:8/732: dread d0/d13/f40 [0,4194304] 0 2024-04-06T14:17:18.076 INFO:tasks.workunit.client.1.smithi086.stdout:9/568: dread d5/d13/d1a/d5f/d88/d14/f86 [0,4194304] 0 2024-04-06T14:17:18.076 INFO:tasks.workunit.client.1.smithi086.stdout:4/553: truncate d0/d2c/d30/d24/d71/f84 1046899 0 2024-04-06T14:17:18.078 INFO:tasks.workunit.client.0.smithi032.stdout:6/969: creat d4/d5/d28/f146 x:0 0 0 2024-04-06T14:17:18.079 INFO:tasks.workunit.client.0.smithi032.stdout:5/841: fsync d16/d18/d33/f8a 0 2024-04-06T14:17:18.081 INFO:tasks.workunit.client.1.smithi086.stdout:3/708: rename d5/df to d5/d26/d4f/dd2/ddb/de6 0 2024-04-06T14:17:18.082 INFO:tasks.workunit.client.0.smithi032.stdout:0/729: dread d3/da/d25/d2b/d4a/d79/d97/de2/ff4 [0,4194304] 0 2024-04-06T14:17:18.084 INFO:tasks.workunit.client.1.smithi086.stdout:0/722: dread d0/d1/d2/d1a/d79/fb2 [0,4194304] 0 2024-04-06T14:17:18.086 INFO:tasks.workunit.client.0.smithi032.stdout:6/970: creat d4/d5/dc5/d82/d9d/f147 x:0 0 0 2024-04-06T14:17:18.086 INFO:tasks.workunit.client.0.smithi032.stdout:2/808: symlink d0/d30/d39/d59/d67/d74/d78/l10e 0 2024-04-06T14:17:18.086 INFO:tasks.workunit.client.0.smithi032.stdout:6/971: stat d4/d5/d19/d25/l32 0 2024-04-06T14:17:18.087 INFO:tasks.workunit.client.1.smithi086.stdout:2/672: truncate d0/d20/d27/d2b/d35/d36/f96 664409 0 2024-04-06T14:17:18.088 INFO:tasks.workunit.client.0.smithi032.stdout:6/972: dread - d4/d5/d8/d2b/d77/fa2 zero size 2024-04-06T14:17:18.088 INFO:tasks.workunit.client.1.smithi086.stdout:3/709: unlink d5/d36/d5a/d6d/d72/d57/cb1 0 2024-04-06T14:17:18.089 INFO:tasks.workunit.client.0.smithi032.stdout:0/730: chown d3/da/d25/f54 4882564 1 2024-04-06T14:17:18.090 INFO:tasks.workunit.client.1.smithi086.stdout:5/558: getdents d0/d43 0 2024-04-06T14:17:18.091 INFO:tasks.workunit.client.0.smithi032.stdout:2/809: symlink d0/d30/d39/d59/l10f 0 2024-04-06T14:17:18.091 INFO:tasks.workunit.client.0.smithi032.stdout:2/810: dread - d0/d4e/f101 zero size 2024-04-06T14:17:18.092 INFO:tasks.workunit.client.1.smithi086.stdout:0/723: chown d0/d15/d4e/da3/d44/d5d/d74 2776 1 2024-04-06T14:17:18.092 INFO:tasks.workunit.client.1.smithi086.stdout:3/710: fdatasync d5/d36/f3b 0 2024-04-06T14:17:18.093 INFO:tasks.workunit.client.1.smithi086.stdout:9/569: dwrite d5/d13/d1a/fb6 [0,4194304] 0 2024-04-06T14:17:18.094 INFO:tasks.workunit.client.0.smithi032.stdout:4/864: write d0/df/d23/d31/f68 [1021291,82430] 0 2024-04-06T14:17:18.094 INFO:tasks.workunit.client.1.smithi086.stdout:4/554: rename d0/d2c/d30/d24/c63 to d0/de/cad 0 2024-04-06T14:17:18.096 INFO:tasks.workunit.client.0.smithi032.stdout:0/731: fsync d3/da/d25/d2b/d4a/d79/d97/de2/ff4 0 2024-04-06T14:17:18.099 INFO:tasks.workunit.client.0.smithi032.stdout:7/974: write d16/d20/d7a/fa5 [1515199,7635] 0 2024-04-06T14:17:18.099 INFO:tasks.workunit.client.1.smithi086.stdout:2/673: symlink d0/d20/d27/le8 0 2024-04-06T14:17:18.100 INFO:tasks.workunit.client.1.smithi086.stdout:1/710: write db/d11/d1a/d2a/d9a/d78/fcc [3548982,22671] 0 2024-04-06T14:17:18.101 INFO:tasks.workunit.client.1.smithi086.stdout:5/559: mknod d0/cb4 0 2024-04-06T14:17:18.103 INFO:tasks.workunit.client.0.smithi032.stdout:0/732: rename d3/f10 to d3/d83/ff6 0 2024-04-06T14:17:18.103 INFO:tasks.workunit.client.1.smithi086.stdout:4/555: creat d0/d2c/d30/d24/d50/d79/d89/fae x:0 0 0 2024-04-06T14:17:18.104 INFO:tasks.workunit.client.1.smithi086.stdout:0/724: mknod d0/cee 0 2024-04-06T14:17:18.104 INFO:tasks.workunit.client.0.smithi032.stdout:0/733: chown d3/da/d25/d2b/f40 217200926 1 2024-04-06T14:17:18.106 INFO:tasks.workunit.client.1.smithi086.stdout:3/711: mkdir d5/d26/d7f/de7 0 2024-04-06T14:17:18.107 INFO:tasks.workunit.client.0.smithi032.stdout:4/865: symlink d0/l124 0 2024-04-06T14:17:18.107 INFO:tasks.workunit.client.1.smithi086.stdout:2/674: unlink d0/d20/d27/d2b/d35/f47 0 2024-04-06T14:17:18.109 INFO:tasks.workunit.client.1.smithi086.stdout:1/711: write db/d11/d1a/d2a/f47 [2776937,122779] 0 2024-04-06T14:17:18.109 INFO:tasks.workunit.client.0.smithi032.stdout:0/734: rmdir d3/da/d25/d2b/d4a/d79/d97 39 2024-04-06T14:17:18.114 INFO:tasks.workunit.client.1.smithi086.stdout:3/712: rmdir d5/d26/d4f/dd2/ddb/de6/d9e/da5 39 2024-04-06T14:17:18.114 INFO:tasks.workunit.client.1.smithi086.stdout:8/733: dread d0/db/f5b [0,4194304] 0 2024-04-06T14:17:18.114 INFO:tasks.workunit.client.1.smithi086.stdout:5/560: dwrite d0/d16/d18/d35/f87 [0,4194304] 0 2024-04-06T14:17:18.114 INFO:tasks.workunit.client.1.smithi086.stdout:2/675: creat d0/df/d48/d7d/fe9 x:0 0 0 2024-04-06T14:17:18.117 INFO:tasks.workunit.client.1.smithi086.stdout:8/734: chown d0/d3b/db9/fd7 4628282 1 2024-04-06T14:17:18.118 INFO:tasks.workunit.client.1.smithi086.stdout:2/676: fdatasync d0/df/d5e/d9d/fe0 0 2024-04-06T14:17:18.121 INFO:tasks.workunit.client.0.smithi032.stdout:1/994: write d0/d9/f83 [936145,83748] 0 2024-04-06T14:17:18.122 INFO:tasks.workunit.client.1.smithi086.stdout:3/713: creat d5/d26/d7f/de7/fe8 x:0 0 0 2024-04-06T14:17:18.123 INFO:tasks.workunit.client.1.smithi086.stdout:8/735: unlink d0/d71/d22/d43/fd8 0 2024-04-06T14:17:18.124 INFO:tasks.workunit.client.0.smithi032.stdout:4/866: dread d0/df/d23/d31/d50/d57/d5b/d9c/f9f [0,4194304] 0 2024-04-06T14:17:18.125 INFO:tasks.workunit.client.1.smithi086.stdout:1/712: truncate db/d11/d1a/d2a/d9a/d68/daf/dc2/f8a 169738 0 2024-04-06T14:17:18.125 INFO:tasks.workunit.client.1.smithi086.stdout:3/714: mknod d5/d26/d7f/de7/ce9 0 2024-04-06T14:17:18.125 INFO:tasks.workunit.client.0.smithi032.stdout:5/842: truncate d16/d18/d33/d35/de0/fe6 1665987 0 2024-04-06T14:17:18.126 INFO:tasks.workunit.client.1.smithi086.stdout:5/561: getdents d0/d16/d71 0 2024-04-06T14:17:18.127 INFO:tasks.workunit.client.0.smithi032.stdout:0/735: dwrite d3/da/d25/d2b/d4a/d79/d97/de0/df5/fbc [0,4194304] 0 2024-04-06T14:17:18.127 INFO:tasks.workunit.client.1.smithi086.stdout:2/677: mknod d0/d20/d27/d2b/cea 0 2024-04-06T14:17:18.131 INFO:tasks.workunit.client.1.smithi086.stdout:2/678: creat d0/d20/d27/d37/d52/feb x:0 0 0 2024-04-06T14:17:18.132 INFO:tasks.workunit.client.1.smithi086.stdout:8/736: creat d0/d71/d22/d65/d6e/d98/ffe x:0 0 0 2024-04-06T14:17:18.132 INFO:tasks.workunit.client.0.smithi032.stdout:8/982: dwrite d3/f96 [4194304,4194304] 0 2024-04-06T14:17:18.135 INFO:tasks.workunit.client.1.smithi086.stdout:2/679: fsync d0/d20/d44/dbb/dc4/de3/d76/d78/f8c 0 2024-04-06T14:17:18.135 INFO:tasks.workunit.client.1.smithi086.stdout:1/713: dread db/d11/d87/fb6 [0,4194304] 0 2024-04-06T14:17:18.138 INFO:tasks.workunit.client.1.smithi086.stdout:5/562: creat d0/d16/d18/fb5 x:0 0 0 2024-04-06T14:17:18.139 INFO:tasks.workunit.client.1.smithi086.stdout:2/680: truncate d0/df/fb3 644906 0 2024-04-06T14:17:18.142 INFO:tasks.workunit.client.1.smithi086.stdout:3/715: getdents d5/d36/d5c/d8e 0 2024-04-06T14:17:18.144 INFO:tasks.workunit.client.0.smithi032.stdout:5/843: chown d16/d18/d33/d45/d52/d39/d86/cb9 34 1 2024-04-06T14:17:18.144 INFO:tasks.workunit.client.1.smithi086.stdout:5/563: symlink d0/d16/d71/d89/lb6 0 2024-04-06T14:17:18.146 INFO:tasks.workunit.client.0.smithi032.stdout:4/867: creat d0/df/d23/d31/f125 x:0 0 0 2024-04-06T14:17:18.148 INFO:tasks.workunit.client.1.smithi086.stdout:2/681: symlink d0/d20/d44/dbb/dc4/de3/d76/d78/lec 0 2024-04-06T14:17:18.149 INFO:tasks.workunit.client.0.smithi032.stdout:4/868: mkdir d0/d82/d8e/d126 0 2024-04-06T14:17:18.149 INFO:tasks.workunit.client.1.smithi086.stdout:5/564: creat d0/d19/fb7 x:0 0 0 2024-04-06T14:17:18.150 INFO:tasks.workunit.client.1.smithi086.stdout:2/682: chown d0/d20/d44/dbb/dc4 184 1 2024-04-06T14:17:18.151 INFO:tasks.workunit.client.0.smithi032.stdout:3/885: dwrite d3/d16/fb0 [0,4194304] 0 2024-04-06T14:17:18.151 INFO:tasks.workunit.client.1.smithi086.stdout:5/565: read d0/d16/d18/d35/f3a [412199,83786] 0 2024-04-06T14:17:18.153 INFO:tasks.workunit.client.0.smithi032.stdout:4/869: mkdir d0/df/d23/d127 0 2024-04-06T14:17:18.154 INFO:tasks.workunit.client.0.smithi032.stdout:8/983: truncate d3/d8/d7/f81 21181 0 2024-04-06T14:17:18.155 INFO:tasks.workunit.client.1.smithi086.stdout:2/683: mknod d0/d20/d27/d2b/d35/d36/d4d/d59/d94/ced 0 2024-04-06T14:17:18.155 INFO:tasks.workunit.client.0.smithi032.stdout:0/736: mknod d3/da/d25/d2b/d4a/d79/d97/de0/df5/cf7 0 2024-04-06T14:17:18.156 INFO:tasks.workunit.client.0.smithi032.stdout:8/984: dread - d3/d8/d3a/d7b/db9/f127 zero size 2024-04-06T14:17:18.157 INFO:tasks.workunit.client.1.smithi086.stdout:2/684: mkdir d0/d20/d44/dee 0 2024-04-06T14:17:18.158 INFO:tasks.workunit.client.0.smithi032.stdout:3/886: getdents d3/d16/ddd/df7 0 2024-04-06T14:17:18.159 INFO:tasks.workunit.client.0.smithi032.stdout:0/737: getdents d3/dc2 0 2024-04-06T14:17:18.160 INFO:tasks.workunit.client.0.smithi032.stdout:8/985: mknod d3/d8/d7/dd5/d148/d46/def/d128/c150 0 2024-04-06T14:17:18.161 INFO:tasks.workunit.client.0.smithi032.stdout:3/887: fsync d3/d16/ddd/df7/d147/f93 0 2024-04-06T14:17:18.162 INFO:tasks.workunit.client.1.smithi086.stdout:1/714: dread db/d11/d1a/d2a/f9c [0,4194304] 0 2024-04-06T14:17:18.162 INFO:tasks.workunit.client.1.smithi086.stdout:2/685: truncate d0/df/f21 5545411 0 2024-04-06T14:17:18.162 INFO:tasks.workunit.client.1.smithi086.stdout:2/686: mknod d0/d20/d44/dbb/cef 0 2024-04-06T14:17:18.162 INFO:tasks.workunit.client.1.smithi086.stdout:2/687: dread - d0/d20/d27/d37/f98 zero size 2024-04-06T14:17:18.163 INFO:tasks.workunit.client.0.smithi032.stdout:0/738: unlink d3/da/d25/d36/f93 0 2024-04-06T14:17:18.164 INFO:tasks.workunit.client.0.smithi032.stdout:8/986: mkdir d3/d8/d12/d101/d151 0 2024-04-06T14:17:18.165 INFO:tasks.workunit.client.1.smithi086.stdout:1/715: rename db/d11/d1f/l2d to db/d11/d1a/d2a/d52/d6b/le6 0 2024-04-06T14:17:18.165 INFO:tasks.workunit.client.1.smithi086.stdout:2/688: symlink d0/df/d48/d7d/lf0 0 2024-04-06T14:17:18.166 INFO:tasks.workunit.client.0.smithi032.stdout:8/987: read d3/d8/d7/dd5/d148/d46/f74 [4019811,73062] 0 2024-04-06T14:17:18.167 INFO:tasks.workunit.client.0.smithi032.stdout:3/888: symlink d3/l153 0 2024-04-06T14:17:18.168 INFO:tasks.workunit.client.1.smithi086.stdout:2/689: chown d0/d20/d44/dbb/dc4/de3/d76/fb9 3 1 2024-04-06T14:17:18.169 INFO:tasks.workunit.client.1.smithi086.stdout:2/690: read d0/d20/d27/f7f [281984,33886] 0 2024-04-06T14:17:18.170 INFO:tasks.workunit.client.0.smithi032.stdout:3/889: write d3/d16/ddd/df7/d147/d7d/d9c/dad/d111/f140 [5461566,113297] 0 2024-04-06T14:17:18.172 INFO:tasks.workunit.client.1.smithi086.stdout:1/716: dread fa [0,4194304] 0 2024-04-06T14:17:18.173 INFO:tasks.workunit.client.0.smithi032.stdout:3/890: mknod d3/d16/ddd/df7/d147/d7d/d9c/dbc/d128/d14e/c154 0 2024-04-06T14:17:18.175 INFO:tasks.workunit.client.1.smithi086.stdout:1/717: dread db/f16 [0,4194304] 0 2024-04-06T14:17:18.179 INFO:tasks.workunit.client.0.smithi032.stdout:3/891: dwrite d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/f10f [0,4194304] 0 2024-04-06T14:17:18.180 INFO:tasks.workunit.client.1.smithi086.stdout:1/718: dwrite db/d11/d42/f50 [0,4194304] 0 2024-04-06T14:17:18.185 INFO:tasks.workunit.client.1.smithi086.stdout:1/719: truncate db/d11/d1a/d2a/d9a/d68/daf/dc2/fcd 344763 0 2024-04-06T14:17:18.185 INFO:tasks.workunit.client.0.smithi032.stdout:3/892: truncate d3/d16/ddd/df7/d147/d89/f97 5736809 0 2024-04-06T14:17:18.187 INFO:tasks.workunit.client.1.smithi086.stdout:1/720: chown db/d11/d1a/d2a/d9a/d68/daf/dc2/fcd 5 1 2024-04-06T14:17:18.188 INFO:tasks.workunit.client.1.smithi086.stdout:1/721: truncate db/d11/d42/d53/fd4 681085 0 2024-04-06T14:17:18.190 INFO:tasks.workunit.client.0.smithi032.stdout:3/893: mkdir d3/d16/ddd/df7/d147/d89/dcb/d155 0 2024-04-06T14:17:18.191 INFO:tasks.workunit.client.1.smithi086.stdout:1/722: creat db/d11/d1a/d2a/d9a/d78/fe7 x:0 0 0 2024-04-06T14:17:18.194 INFO:tasks.workunit.client.0.smithi032.stdout:3/894: rename d3/d16/ddd/df7/d147/d7d/d9c/dad/dc5/cf4 to d3/d16/ddd/df7/d147/c156 0 2024-04-06T14:17:18.195 INFO:tasks.workunit.client.0.smithi032.stdout:3/895: read - d3/d16/d63/f103 zero size 2024-04-06T14:17:18.196 INFO:tasks.workunit.client.0.smithi032.stdout:3/896: chown d3/d16/ddd/df7/d147/d7d/d9c/dad/dc5/f10d 185827652 1 2024-04-06T14:17:18.200 INFO:tasks.workunit.client.0.smithi032.stdout:3/897: rename d3/d16/ddd/df7/d147/l36 to d3/d16/ddd/df7/d147/d7d/dc6/l157 0 2024-04-06T14:17:18.201 INFO:tasks.workunit.client.1.smithi086.stdout:1/723: dread db/fe [0,4194304] 0 2024-04-06T14:17:18.207 INFO:tasks.workunit.client.1.smithi086.stdout:1/724: dwrite db/d11/d1f/fd9 [0,4194304] 0 2024-04-06T14:17:18.209 INFO:tasks.workunit.client.1.smithi086.stdout:1/725: chown db/d11/d1a/d2a/d9a/d68/d6e/d81/c8b 1071057617 1 2024-04-06T14:17:18.216 INFO:tasks.workunit.client.1.smithi086.stdout:0/725: read d0/d15/d4e/da3/f39 [206400,122850] 0 2024-04-06T14:17:18.216 INFO:tasks.workunit.client.1.smithi086.stdout:1/726: chown db/d11/d1a/d2a/d9a/d68/daf/dc2/c46 27 1 2024-04-06T14:17:18.225 INFO:tasks.workunit.client.0.smithi032.stdout:3/898: dread d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/f5f [4194304,4194304] 0 2024-04-06T14:17:18.227 INFO:tasks.workunit.client.1.smithi086.stdout:0/726: mkdir d0/d15/d26/d33/def 0 2024-04-06T14:17:18.227 INFO:tasks.workunit.client.1.smithi086.stdout:0/727: dread - d0/d15/d26/d33/d7a/fe0 zero size 2024-04-06T14:17:18.227 INFO:tasks.workunit.client.1.smithi086.stdout:1/727: symlink db/d11/d1a/le8 0 2024-04-06T14:17:18.227 INFO:tasks.workunit.client.0.smithi032.stdout:3/899: truncate d3/d16/ddd/df7/d147/d7d/d9c/dad/dc5/f12f 395883 0 2024-04-06T14:17:18.228 INFO:tasks.workunit.client.0.smithi032.stdout:2/811: write d0/d30/d39/d59/d67/d74/d78/d57/d7c/fa1 [3247277,73375] 0 2024-04-06T14:17:18.229 INFO:tasks.workunit.client.0.smithi032.stdout:6/973: write d4/d5/dc5/fa1 [254205,101061] 0 2024-04-06T14:17:18.232 INFO:tasks.workunit.client.0.smithi032.stdout:6/974: dread - d4/d5/d28/f146 zero size 2024-04-06T14:17:18.232 INFO:tasks.workunit.client.0.smithi032.stdout:9/873: mknod d2/d15/d55/d59/d93/d95/db8/c116 0 2024-04-06T14:17:18.237 INFO:tasks.workunit.client.0.smithi032.stdout:3/900: dread d3/d16/ddd/df7/d147/d7d/d9c/dbc/dcd/f101 [0,4194304] 0 2024-04-06T14:17:18.238 INFO:tasks.workunit.client.0.smithi032.stdout:6/975: creat d4/d5/d8/d2f/d9b/f148 x:0 0 0 2024-04-06T14:17:18.238 INFO:tasks.workunit.client.0.smithi032.stdout:7/975: dwrite d16/d20/d54/de4/d5f/f91 [0,4194304] 0 2024-04-06T14:17:18.241 INFO:tasks.workunit.client.0.smithi032.stdout:3/901: chown d3/d16/ddd/df7/d147/d7d/d9c/dad/d111/f140 78917 1 2024-04-06T14:17:18.241 INFO:tasks.workunit.client.0.smithi032.stdout:7/976: read d16/d20/d54/de4/f31 [177311,11932] 0 2024-04-06T14:17:18.242 INFO:tasks.workunit.client.0.smithi032.stdout:7/977: chown fd 47652683 1 2024-04-06T14:17:18.250 INFO:tasks.workunit.client.0.smithi032.stdout:6/976: dwrite d4/d5/dbf/d61/da7/dda/fee [0,4194304] 0 2024-04-06T14:17:18.254 INFO:tasks.workunit.client.0.smithi032.stdout:6/977: mknod d4/c149 0 2024-04-06T14:17:18.255 INFO:tasks.workunit.client.0.smithi032.stdout:3/902: dwrite d3/d16/ddd/df7/d147/d7d/d9c/dad/f10e [0,4194304] 0 2024-04-06T14:17:18.256 INFO:tasks.workunit.client.0.smithi032.stdout:7/978: dwrite d16/d20/d54/de4/d2d/f109 [0,4194304] 0 2024-04-06T14:17:18.260 INFO:tasks.workunit.client.0.smithi032.stdout:3/903: fdatasync d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/fc3 0 2024-04-06T14:17:18.264 INFO:tasks.workunit.client.0.smithi032.stdout:7/979: creat d16/d20/d54/de4/d2d/d30/d3e/f159 x:0 0 0 2024-04-06T14:17:18.266 INFO:tasks.workunit.client.0.smithi032.stdout:3/904: mknod d3/d16/ddd/df7/d147/d7d/d9c/dbc/dcd/c158 0 2024-04-06T14:17:18.267 INFO:tasks.workunit.client.0.smithi032.stdout:5/844: write d16/d58/d54/d74/d8f/fe1 [783595,51788] 0 2024-04-06T14:17:18.268 INFO:tasks.workunit.client.0.smithi032.stdout:3/905: read - d3/d16/ddd/df7/d147/d7d/dc6/ff2 zero size 2024-04-06T14:17:18.269 INFO:tasks.workunit.client.0.smithi032.stdout:6/978: rename d4/d5/d8/ld to d4/d5/d28/l14a 0 2024-04-06T14:17:18.270 INFO:tasks.workunit.client.0.smithi032.stdout:3/906: stat d3/d16/ddd/df7/d147/d7d/d9c/dad/d111/f140 0 2024-04-06T14:17:18.272 INFO:tasks.workunit.client.0.smithi032.stdout:4/870: dread - d0/df/d23/d31/d50/d57/d5b/d9c/dc1/f114 zero size 2024-04-06T14:17:18.274 INFO:tasks.workunit.client.0.smithi032.stdout:9/874: dread d2/d15/d1e/d41/d6c/d74/fe7 [0,4194304] 0 2024-04-06T14:17:18.274 INFO:tasks.workunit.client.0.smithi032.stdout:7/980: mkdir d16/d20/d63/d15a 0 2024-04-06T14:17:18.275 INFO:tasks.workunit.client.0.smithi032.stdout:6/979: creat d4/d5/d8/d2f/f14b x:0 0 0 2024-04-06T14:17:18.278 INFO:tasks.workunit.client.0.smithi032.stdout:5/845: mkdir d16/d18/d126 0 2024-04-06T14:17:18.280 INFO:tasks.workunit.client.0.smithi032.stdout:4/871: chown d0/df/lcf 326 1 2024-04-06T14:17:18.280 INFO:tasks.workunit.client.0.smithi032.stdout:4/872: truncate d0/df/d2b/d46/ff3 557061 0 2024-04-06T14:17:18.282 INFO:tasks.workunit.client.0.smithi032.stdout:8/988: truncate d3/d8/d3a/d7b/db9/f137 2643434 0 2024-04-06T14:17:18.283 INFO:tasks.workunit.client.0.smithi032.stdout:6/980: symlink d4/d5/dbf/d61/da7/dda/l14c 0 2024-04-06T14:17:18.284 INFO:tasks.workunit.client.0.smithi032.stdout:5/846: stat d16/c1a 0 2024-04-06T14:17:18.284 INFO:tasks.workunit.client.0.smithi032.stdout:6/981: read - d4/d5/dc5/d82/f125 zero size 2024-04-06T14:17:18.286 INFO:tasks.workunit.client.0.smithi032.stdout:9/875: unlink d2/d15/d55/d59/d93/d95/db8/ff3 0 2024-04-06T14:17:18.288 INFO:tasks.workunit.client.0.smithi032.stdout:4/873: read d0/df/d23/f32 [1992361,108594] 0 2024-04-06T14:17:18.288 INFO:tasks.workunit.client.0.smithi032.stdout:3/907: creat d3/d16/ddd/df7/d147/f159 x:0 0 0 2024-04-06T14:17:18.291 INFO:tasks.workunit.client.0.smithi032.stdout:8/989: creat d3/d8/d7/dd5/d148/d4a/d60/d75/dd1/f152 x:0 0 0 2024-04-06T14:17:18.296 INFO:tasks.workunit.client.0.smithi032.stdout:5/847: mkdir d16/d127 0 2024-04-06T14:17:18.297 INFO:tasks.workunit.client.0.smithi032.stdout:9/876: dread d2/d15/d1e/d65/d83/d7e/f3e [0,4194304] 0 2024-04-06T14:17:18.300 INFO:tasks.workunit.client.1.smithi086.stdout:1/728: mkdir db/d11/d1f/d5c/ddb/de9 0 2024-04-06T14:17:18.300 INFO:tasks.workunit.client.0.smithi032.stdout:4/874: rename d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/df2/dff to d0/df/d23/d128 0 2024-04-06T14:17:18.300 INFO:tasks.workunit.client.0.smithi032.stdout:8/990: dwrite d3/d8/d7/dd5/d148/d4a/d60/d90/fb6 [0,4194304] 0 2024-04-06T14:17:18.302 INFO:tasks.workunit.client.0.smithi032.stdout:5/848: mknod d16/d18/d33/d45/d5e/c128 0 2024-04-06T14:17:18.302 INFO:tasks.workunit.client.1.smithi086.stdout:1/729: read db/d13/d71/f88 [1985762,45695] 0 2024-04-06T14:17:18.303 INFO:tasks.workunit.client.1.smithi086.stdout:6/781: sync 2024-04-06T14:17:18.303 INFO:tasks.workunit.client.1.smithi086.stdout:7/674: sync 2024-04-06T14:17:18.304 INFO:tasks.workunit.client.1.smithi086.stdout:0/728: unlink d0/d15/d4e/da3/d44/f5a 0 2024-04-06T14:17:18.310 INFO:tasks.workunit.client.1.smithi086.stdout:0/729: fsync d0/d1/d4/de/f62 0 2024-04-06T14:17:18.310 INFO:tasks.workunit.client.1.smithi086.stdout:1/730: creat db/d11/d1a/d2a/d9a/d68/d6e/fea x:0 0 0 2024-04-06T14:17:18.311 INFO:tasks.workunit.client.0.smithi032.stdout:8/991: creat d3/d8/d3a/d7b/db9/ddf/f153 x:0 0 0 2024-04-06T14:17:18.312 INFO:tasks.workunit.client.0.smithi032.stdout:9/877: fdatasync d2/dcd/dd8/f108 0 2024-04-06T14:17:18.313 INFO:tasks.workunit.client.1.smithi086.stdout:4/556: truncate d0/d2c/d30/f6c 491120 0 2024-04-06T14:17:18.315 INFO:tasks.workunit.client.0.smithi032.stdout:5/849: creat d16/d58/d54/d74/d8f/f129 x:0 0 0 2024-04-06T14:17:18.317 INFO:tasks.workunit.client.0.smithi032.stdout:4/875: fsync d0/df/f8b 0 2024-04-06T14:17:18.317 INFO:tasks.workunit.client.0.smithi032.stdout:9/878: chown d2/d15/d1e/d41/d6c/d74/fe7 1750 1 2024-04-06T14:17:18.318 INFO:tasks.workunit.client.1.smithi086.stdout:7/675: dwrite d7/d10/d11/d80/fca [0,4194304] 0 2024-04-06T14:17:18.320 INFO:tasks.workunit.client.0.smithi032.stdout:7/981: dwrite d16/d20/d54/f57 [0,4194304] 0 2024-04-06T14:17:18.320 INFO:tasks.workunit.client.0.smithi032.stdout:5/850: read d16/f2c [980804,28128] 0 2024-04-06T14:17:18.321 INFO:tasks.workunit.client.0.smithi032.stdout:8/992: unlink d3/d8/d7/dd5/d148/d46/d125/f5a 0 2024-04-06T14:17:18.321 INFO:tasks.workunit.client.0.smithi032.stdout:4/876: fdatasync d0/d82/d8e/ffc 0 2024-04-06T14:17:18.321 INFO:tasks.workunit.client.1.smithi086.stdout:4/557: unlink d0/d15/d20/d34/c9f 0 2024-04-06T14:17:18.323 INFO:tasks.workunit.client.1.smithi086.stdout:1/731: dwrite db/d13/d55/da3/fc4 [0,4194304] 0 2024-04-06T14:17:18.324 INFO:tasks.workunit.client.0.smithi032.stdout:7/982: dread - d16/d20/d7a/f144 zero size 2024-04-06T14:17:18.328 INFO:tasks.workunit.client.0.smithi032.stdout:3/908: write d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/f54 [893487,38484] 0 2024-04-06T14:17:18.331 INFO:tasks.workunit.client.0.smithi032.stdout:8/993: fsync d3/d8/d7/dd5/d148/d46/f74 0 2024-04-06T14:17:18.333 INFO:tasks.workunit.client.0.smithi032.stdout:9/879: truncate d2/d15/d1e/d65/d6b/d92/fb3 1127396 0 2024-04-06T14:17:18.334 INFO:tasks.workunit.client.1.smithi086.stdout:5/566: write d0/d16/f39 [1545655,21374] 0 2024-04-06T14:17:18.337 INFO:tasks.workunit.client.0.smithi032.stdout:7/983: symlink d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dde/d11a/l15b 0 2024-04-06T14:17:18.337 INFO:tasks.workunit.client.1.smithi086.stdout:2/691: write d0/d20/d27/d37/d52/f79 [238219,77675] 0 2024-04-06T14:17:18.340 INFO:tasks.workunit.client.0.smithi032.stdout:4/877: rename d0/df/d23/d31/fd8 to d0/df/d23/d128/f129 0 2024-04-06T14:17:18.341 INFO:tasks.workunit.client.0.smithi032.stdout:8/994: fdatasync d3/d8/d3a/d7b/db9/f14b 0 2024-04-06T14:17:18.342 INFO:tasks.workunit.client.0.smithi032.stdout:5/851: dwrite d16/d18/d33/d45/d52/ffd [0,4194304] 0 2024-04-06T14:17:18.344 INFO:tasks.workunit.client.0.smithi032.stdout:9/880: dread - d2/d15/d1e/d65/d9c/fbd zero size 2024-04-06T14:17:18.344 INFO:tasks.workunit.client.1.smithi086.stdout:2/692: chown d0/df/la3 4 1 2024-04-06T14:17:18.345 INFO:tasks.workunit.client.1.smithi086.stdout:8/737: dwrite d0/d3b/f46 [0,4194304] 0 2024-04-06T14:17:18.345 INFO:tasks.workunit.client.1.smithi086.stdout:3/716: dwrite d5/d36/d5a/d6d/d72/f75 [0,4194304] 0 2024-04-06T14:17:18.345 INFO:tasks.workunit.client.1.smithi086.stdout:0/730: write d0/d15/d26/fa0 [685993,107176] 0 2024-04-06T14:17:18.347 INFO:tasks.workunit.client.0.smithi032.stdout:3/909: dwrite d3/d16/f70 [0,4194304] 0 2024-04-06T14:17:18.356 INFO:tasks.workunit.client.1.smithi086.stdout:8/738: mknod d0/d71/d16/d39/d90/cff 0 2024-04-06T14:17:18.357 INFO:tasks.workunit.client.1.smithi086.stdout:3/717: dread d5/d26/d4f/db4/da7/f63 [0,4194304] 0 2024-04-06T14:17:18.357 INFO:tasks.workunit.client.1.smithi086.stdout:6/782: truncate d6/d13/d1a/d64/fb3 1273230 0 2024-04-06T14:17:18.358 INFO:tasks.workunit.client.1.smithi086.stdout:3/718: chown d5/f8d 13 1 2024-04-06T14:17:18.361 INFO:tasks.workunit.client.0.smithi032.stdout:4/878: mknod d0/d82/d8e/d126/c12a 0 2024-04-06T14:17:18.362 INFO:tasks.workunit.client.0.smithi032.stdout:7/984: rename d16/d20/d54/led to d16/d20/d6e/d137/l15c 0 2024-04-06T14:17:18.362 INFO:tasks.workunit.client.1.smithi086.stdout:8/739: readlink d0/d71/d16/d52/d5f/lbe 0 2024-04-06T14:17:18.362 INFO:tasks.workunit.client.1.smithi086.stdout:2/693: symlink d0/d20/d27/dae/lf1 0 2024-04-06T14:17:18.362 INFO:tasks.workunit.client.1.smithi086.stdout:1/732: truncate db/d11/d1a/d2a/d9a/d68/f91 548333 0 2024-04-06T14:17:18.362 INFO:tasks.workunit.client.1.smithi086.stdout:6/783: mkdir d6/d13/d1a/da0/d109 0 2024-04-06T14:17:18.366 INFO:tasks.workunit.client.0.smithi032.stdout:5/852: dwrite d16/d18/d33/f41 [0,4194304] 0 2024-04-06T14:17:18.367 INFO:tasks.workunit.client.0.smithi032.stdout:8/995: dwrite d3/d8/f5f [0,4194304] 0 2024-04-06T14:17:18.371 INFO:tasks.workunit.client.0.smithi032.stdout:8/996: chown d3/d8/f93 285 1 2024-04-06T14:17:18.372 INFO:tasks.workunit.client.0.smithi032.stdout:7/985: write d16/d20/d54/de4/de0/df9/f14b [1266527,72304] 0 2024-04-06T14:17:18.373 INFO:tasks.workunit.client.1.smithi086.stdout:4/558: dread d0/f36 [0,4194304] 0 2024-04-06T14:17:18.373 INFO:tasks.workunit.client.1.smithi086.stdout:3/719: rename d5/d26/d4f to d5/dea 0 2024-04-06T14:17:18.376 INFO:tasks.workunit.client.1.smithi086.stdout:0/731: dwrite d0/d15/d4e/da3/d44/d8e/d8f/fc4 [0,4194304] 0 2024-04-06T14:17:18.379 INFO:tasks.workunit.client.0.smithi032.stdout:7/986: fdatasync d16/d20/d6e/f106 0 2024-04-06T14:17:18.379 INFO:tasks.workunit.client.0.smithi032.stdout:8/997: mkdir d3/d69/d154 0 2024-04-06T14:17:18.379 INFO:tasks.workunit.client.1.smithi086.stdout:2/694: getdents d0/d20/d27/d37/d52/d9c 0 2024-04-06T14:17:18.380 INFO:tasks.workunit.client.1.smithi086.stdout:0/732: chown d0/d15/f3e 15051 1 2024-04-06T14:17:18.381 INFO:tasks.workunit.client.0.smithi032.stdout:5/853: symlink d16/d58/d54/d94/db2/dcb/d11b/l12a 0 2024-04-06T14:17:18.383 INFO:tasks.workunit.client.1.smithi086.stdout:8/740: rename d0/d3b/d79/dc6/dbc/dc1 to d0/d5a/d100 0 2024-04-06T14:17:18.383 INFO:tasks.workunit.client.0.smithi032.stdout:4/879: creat d0/d82/daf/f12b x:0 0 0 2024-04-06T14:17:18.386 INFO:tasks.workunit.client.0.smithi032.stdout:7/987: write d16/d20/d54/de4/d2d/d30/d3e/d96/d10b/f146 [107341,11763] 0 2024-04-06T14:17:18.388 INFO:tasks.workunit.client.1.smithi086.stdout:0/733: fsync d0/f16 0 2024-04-06T14:17:18.390 INFO:tasks.workunit.client.1.smithi086.stdout:6/784: getdents d6/d13/d1a/d64/de2 0 2024-04-06T14:17:18.390 INFO:tasks.workunit.client.1.smithi086.stdout:7/676: dwrite d7/d1d/f38 [0,4194304] 0 2024-04-06T14:17:18.391 INFO:tasks.workunit.client.1.smithi086.stdout:3/720: dread d5/dea/dd2/ddb/de6/f13 [0,4194304] 0 2024-04-06T14:17:18.392 INFO:tasks.workunit.client.0.smithi032.stdout:9/881: write d2/d15/d1e/d65/d83/d31/d39/f52 [1084966,102766] 0 2024-04-06T14:17:18.393 INFO:tasks.workunit.client.1.smithi086.stdout:8/741: dread - d0/d71/d22/d43/daa/fe8 zero size 2024-04-06T14:17:18.394 INFO:tasks.workunit.client.1.smithi086.stdout:2/695: mknod d0/d20/d80/dbf/ddc/cf2 0 2024-04-06T14:17:18.394 INFO:tasks.workunit.client.0.smithi032.stdout:8/998: creat d3/d8/d7/dd5/d148/f155 x:0 0 0 2024-04-06T14:17:18.395 INFO:tasks.workunit.client.1.smithi086.stdout:1/733: dwrite db/d11/d42/d53/fb0 [0,4194304] 0 2024-04-06T14:17:18.397 INFO:tasks.workunit.client.1.smithi086.stdout:9/570: sync 2024-04-06T14:17:18.397 INFO:tasks.workunit.client.1.smithi086.stdout:4/559: read d0/de/f62 [283464,97996] 0 2024-04-06T14:17:18.397 INFO:tasks.workunit.client.0.smithi032.stdout:7/988: rename d16/d20/d54/de4/d2d/d30/c43 to d16/d20/d7a/da6/d117/d11b/c15d 0 2024-04-06T14:17:18.398 INFO:tasks.workunit.client.1.smithi086.stdout:0/734: rmdir d0/d1/d2/d5b 39 2024-04-06T14:17:18.399 INFO:tasks.workunit.client.1.smithi086.stdout:2/696: fsync d0/df/d5e/fc7 0 2024-04-06T14:17:18.400 INFO:tasks.workunit.client.0.smithi032.stdout:4/880: symlink d0/df/d23/d31/d50/d57/d8a/db5/def/l12c 0 2024-04-06T14:17:18.400 INFO:tasks.workunit.client.1.smithi086.stdout:5/567: dread d0/d17/f99 [0,4194304] 0 2024-04-06T14:17:18.401 INFO:tasks.workunit.client.0.smithi032.stdout:7/989: fdatasync d16/d20/d7a/da6/d117/f133 0 2024-04-06T14:17:18.402 INFO:tasks.workunit.client.1.smithi086.stdout:7/677: read d7/d10/d11/d80/f35 [3550879,110409] 0 2024-04-06T14:17:18.402 INFO:tasks.workunit.client.0.smithi032.stdout:8/999: mknod d3/d8/d3a/d4b/c156 0 2024-04-06T14:17:18.403 INFO:tasks.workunit.client.1.smithi086.stdout:1/734: unlink db/l3c 0 2024-04-06T14:17:18.403 INFO:tasks.workunit.client.1.smithi086.stdout:4/560: read d0/d15/d2b/f82 [490450,107245] 0 2024-04-06T14:17:18.404 INFO:tasks.workunit.client.0.smithi032.stdout:9/882: creat d2/d15/d1e/d65/d83/d31/d39/da6/f117 x:0 0 0 2024-04-06T14:17:18.406 INFO:tasks.workunit.client.1.smithi086.stdout:8/742: rename d0/d3b/l63 to d0/d3b/db9/l101 0 2024-04-06T14:17:18.407 INFO:tasks.workunit.client.0.smithi032.stdout:7/990: readlink d16/d20/d54/de4/d5f/l12f 0 2024-04-06T14:17:18.409 INFO:tasks.workunit.client.1.smithi086.stdout:5/568: chown d0/d16/d18/d2a/d50/f5f 2064313 1 2024-04-06T14:17:18.409 INFO:tasks.workunit.client.0.smithi032.stdout:3/910: dwrite d3/d16/ddd/df7/d147/d89/fe4 [0,4194304] 0 2024-04-06T14:17:18.409 INFO:tasks.workunit.client.1.smithi086.stdout:1/735: readlink db/d11/d42/d53/lae 0 2024-04-06T14:17:18.410 INFO:tasks.workunit.client.1.smithi086.stdout:8/743: stat d0/d13/f7c 0 2024-04-06T14:17:18.410 INFO:tasks.workunit.client.1.smithi086.stdout:3/721: fsync d5/d36/f3b 0 2024-04-06T14:17:18.412 INFO:tasks.workunit.client.1.smithi086.stdout:9/571: fdatasync d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f81 0 2024-04-06T14:17:18.413 INFO:tasks.workunit.client.1.smithi086.stdout:2/697: symlink d0/df/d5e/d9d/lf3 0 2024-04-06T14:17:18.413 INFO:tasks.workunit.client.0.smithi032.stdout:4/881: creat d0/df/d2b/d46/d58/f12d x:0 0 0 2024-04-06T14:17:18.414 INFO:tasks.workunit.client.1.smithi086.stdout:7/678: creat d7/d10/d11/d21/fe2 x:0 0 0 2024-04-06T14:17:18.417 INFO:tasks.workunit.client.1.smithi086.stdout:6/785: dwrite d6/d13/d5f/d60/d45/d8a/fc2 [0,4194304] 0 2024-04-06T14:17:18.418 INFO:tasks.workunit.client.1.smithi086.stdout:5/569: readlink d0/d16/d71/l8e 0 2024-04-06T14:17:18.418 INFO:tasks.workunit.client.1.smithi086.stdout:2/698: chown d0/d20/d27/d2b/d35/c75 59 1 2024-04-06T14:17:18.421 INFO:tasks.workunit.client.1.smithi086.stdout:6/786: write d6/d13/d1a/d22/dcf/fd7 [74093,31505] 0 2024-04-06T14:17:18.422 INFO:tasks.workunit.client.0.smithi032.stdout:3/911: fsync d3/d16/ddd/df7/d147/d7d/d9c/dad/d111/f14f 0 2024-04-06T14:17:18.422 INFO:tasks.workunit.client.1.smithi086.stdout:4/561: creat d0/d2c/d30/faf x:0 0 0 2024-04-06T14:17:18.422 INFO:tasks.workunit.client.1.smithi086.stdout:0/735: creat d0/d15/d4e/da3/d44/d8e/d8f/ff0 x:0 0 0 2024-04-06T14:17:18.424 INFO:tasks.workunit.client.1.smithi086.stdout:6/787: dread - d6/d9/d43/fbc zero size 2024-04-06T14:17:18.425 INFO:tasks.workunit.client.1.smithi086.stdout:1/736: rmdir db/d11/d42/d53 39 2024-04-06T14:17:18.425 INFO:tasks.workunit.client.0.smithi032.stdout:7/991: creat d16/d20/d6e/d10e/d12b/f15e x:0 0 0 2024-04-06T14:17:18.426 INFO:tasks.workunit.client.1.smithi086.stdout:6/788: readlink d6/d13/d5f/d60/lcd 0 2024-04-06T14:17:18.426 INFO:tasks.workunit.client.1.smithi086.stdout:4/562: chown d0/d2c/d30/d24/d5d/f8f 214115344 1 2024-04-06T14:17:18.427 INFO:tasks.workunit.client.1.smithi086.stdout:9/572: truncate d5/d13/d47/f83 121339 0 2024-04-06T14:17:18.427 INFO:tasks.workunit.client.1.smithi086.stdout:0/736: chown d0/d15/d4e/da3/d44/d8e/fd9 2 1 2024-04-06T14:17:18.429 INFO:tasks.workunit.client.0.smithi032.stdout:4/882: mknod d0/df/d23/d31/d50/d57/d5b/d9c/dc8/c12e 0 2024-04-06T14:17:18.429 INFO:tasks.workunit.client.0.smithi032.stdout:4/883: read d0/df/f8b [553913,84767] 0 2024-04-06T14:17:18.429 INFO:tasks.workunit.client.0.smithi032.stdout:9/883: rename d2/f21 to d2/d15/d1e/d65/d83/d7e/f118 0 2024-04-06T14:17:18.430 INFO:tasks.workunit.client.1.smithi086.stdout:7/679: rename d7/d10/d11/d1c/d24/d2c/f6f to d7/d10/d11/d1c/d24/d2c/fe3 0 2024-04-06T14:17:18.430 INFO:tasks.workunit.client.1.smithi086.stdout:4/563: chown d0/d2c/d30/d24/d93 543 1 2024-04-06T14:17:18.435 INFO:tasks.workunit.client.1.smithi086.stdout:8/744: fsync d0/d71/d16/f6c 0 2024-04-06T14:17:18.435 INFO:tasks.workunit.client.1.smithi086.stdout:2/699: symlink d0/d20/d82/lf4 0 2024-04-06T14:17:18.435 INFO:tasks.workunit.client.1.smithi086.stdout:1/737: symlink db/d11/d24/leb 0 2024-04-06T14:17:18.436 INFO:tasks.workunit.client.0.smithi032.stdout:3/912: rename d3/d16/ddd/df7/d147/d7d/dc6/de5/c100 to d3/d16/ddd/d12d/c15a 0 2024-04-06T14:17:18.436 INFO:tasks.workunit.client.0.smithi032.stdout:7/992: dread d16/d20/d54/de4/d2d/d30/d97/fdb [0,4194304] 0 2024-04-06T14:17:18.439 INFO:tasks.workunit.client.1.smithi086.stdout:1/738: fdatasync db/d11/d24/f60 0 2024-04-06T14:17:18.440 INFO:tasks.workunit.client.1.smithi086.stdout:9/573: unlink d5/d13/d1a/d5f/d88/d14/d19/d25/l2e 0 2024-04-06T14:17:18.440 INFO:tasks.workunit.client.0.smithi032.stdout:9/884: dwrite d2/f112 [0,4194304] 0 2024-04-06T14:17:18.441 INFO:tasks.workunit.client.1.smithi086.stdout:0/737: chown d0/d15/d4e/da3/la4 1 1 2024-04-06T14:17:18.441 INFO:tasks.workunit.client.1.smithi086.stdout:2/700: write d0/df/d48/d7d/fe9 [422753,39075] 0 2024-04-06T14:17:18.444 INFO:tasks.workunit.client.1.smithi086.stdout:7/680: dwrite d7/d10/d11/f8f [0,4194304] 0 2024-04-06T14:17:18.449 INFO:tasks.workunit.client.0.smithi032.stdout:9/885: read d2/d15/d1e/d65/d83/d31/f40 [2109078,29786] 0 2024-04-06T14:17:18.449 INFO:tasks.workunit.client.1.smithi086.stdout:5/570: dread d0/f6 [0,4194304] 0 2024-04-06T14:17:18.449 INFO:tasks.workunit.client.1.smithi086.stdout:1/739: read db/d11/d1f/f44 [3940119,70098] 0 2024-04-06T14:17:18.449 INFO:tasks.workunit.client.1.smithi086.stdout:3/722: creat d5/dea/dd2/ddb/de6/d9e/feb x:0 0 0 2024-04-06T14:17:18.451 INFO:tasks.workunit.client.1.smithi086.stdout:6/789: creat d6/d13/d1a/d22/de4/f10a x:0 0 0 2024-04-06T14:17:18.451 INFO:tasks.workunit.client.0.smithi032.stdout:3/913: link d3/d16/f18 d3/d16/ddd/df7/d147/d7d/f15b 0 2024-04-06T14:17:18.453 INFO:tasks.workunit.client.1.smithi086.stdout:8/745: creat d0/d71/d16/d52/dd0/ddc/f102 x:0 0 0 2024-04-06T14:17:18.454 INFO:tasks.workunit.client.1.smithi086.stdout:8/746: readlink d0/d71/l14 0 2024-04-06T14:17:18.454 INFO:tasks.workunit.client.1.smithi086.stdout:0/738: rmdir d0 39 2024-04-06T14:17:18.455 INFO:tasks.workunit.client.0.smithi032.stdout:9/886: creat d2/d15/d55/d59/d93/d95/f119 x:0 0 0 2024-04-06T14:17:18.455 INFO:tasks.workunit.client.0.smithi032.stdout:3/914: symlink d3/d16/ddd/df7/d147/d7d/dc6/df0/l15c 0 2024-04-06T14:17:18.456 INFO:tasks.workunit.client.1.smithi086.stdout:4/564: mkdir d0/db0 0 2024-04-06T14:17:18.458 INFO:tasks.workunit.client.1.smithi086.stdout:7/681: write d7/fb [1895891,25695] 0 2024-04-06T14:17:18.458 INFO:tasks.workunit.client.0.smithi032.stdout:3/915: stat d3/d16/ddd/df7/d147/d7d/dc6/de5/c14a 0 2024-04-06T14:17:18.459 INFO:tasks.workunit.client.1.smithi086.stdout:2/701: symlink d0/df/dbc/lf5 0 2024-04-06T14:17:18.460 INFO:tasks.workunit.client.1.smithi086.stdout:5/571: symlink d0/d16/d18/d21/d23/d28/d94/d52/d7c/lb8 0 2024-04-06T14:17:18.461 INFO:tasks.workunit.client.0.smithi032.stdout:9/887: rmdir d2/dcd/dfd 0 2024-04-06T14:17:18.461 INFO:tasks.workunit.client.1.smithi086.stdout:1/740: dwrite db/d11/d1f/fbc [0,4194304] 0 2024-04-06T14:17:18.461 INFO:tasks.workunit.client.1.smithi086.stdout:3/723: creat d5/d36/d7d/fec x:0 0 0 2024-04-06T14:17:18.462 INFO:tasks.workunit.client.1.smithi086.stdout:6/790: fsync d6/d9/da9/fec 0 2024-04-06T14:17:18.463 INFO:tasks.workunit.client.0.smithi032.stdout:3/916: rmdir d3/d16/ddd/df7/d147/d7d/d9c/dad/d111 39 2024-04-06T14:17:18.463 INFO:tasks.workunit.client.1.smithi086.stdout:3/724: stat d5 0 2024-04-06T14:17:18.463 INFO:tasks.workunit.client.1.smithi086.stdout:5/572: readlink d0/d5e/l7e 0 2024-04-06T14:17:18.466 INFO:tasks.workunit.client.1.smithi086.stdout:6/791: chown d6/d13/d1a/d22/c92 5310466 1 2024-04-06T14:17:18.467 INFO:tasks.workunit.client.1.smithi086.stdout:3/725: write d5/dea/dd2/ddb/de6/f6a [7382738,125518] 0 2024-04-06T14:17:18.469 INFO:tasks.workunit.client.1.smithi086.stdout:5/573: fsync d0/d16/d18/d21/d23/d28/d94/d52/fb1 0 2024-04-06T14:17:18.469 INFO:tasks.workunit.client.1.smithi086.stdout:0/739: read - d0/d15/d4e/da3/d73/fcc zero size 2024-04-06T14:17:18.470 INFO:tasks.workunit.client.0.smithi032.stdout:7/993: dread d16/d20/dbd/f105 [0,4194304] 0 2024-04-06T14:17:18.471 INFO:tasks.workunit.client.1.smithi086.stdout:1/741: rmdir db/d11/d1f 39 2024-04-06T14:17:18.471 INFO:tasks.workunit.client.0.smithi032.stdout:9/888: unlink d2/d15/d55/d59/d93/d95/db8/d81/c96 0 2024-04-06T14:17:18.472 INFO:tasks.workunit.client.0.smithi032.stdout:3/917: dread - d3/d16/ddd/df7/d147/d7d/d9c/dbc/ffe zero size 2024-04-06T14:17:18.473 INFO:tasks.workunit.client.1.smithi086.stdout:8/747: dread d0/d3b/f8b [0,4194304] 0 2024-04-06T14:17:18.473 INFO:tasks.workunit.client.1.smithi086.stdout:2/702: symlink d0/d20/d27/dae/lf6 0 2024-04-06T14:17:18.474 INFO:tasks.workunit.client.0.smithi032.stdout:7/994: symlink d16/d20/d6e/d137/l15f 0 2024-04-06T14:17:18.475 INFO:tasks.workunit.client.1.smithi086.stdout:2/703: chown d0/df/d48/d7d/lf0 9 1 2024-04-06T14:17:18.476 INFO:tasks.workunit.client.0.smithi032.stdout:3/918: fdatasync d3/d16/ddd/df7/d147/f93 0 2024-04-06T14:17:18.478 INFO:tasks.workunit.client.1.smithi086.stdout:7/682: mknod d7/d10/d11/d1c/d24/d4d/ce4 0 2024-04-06T14:17:18.479 INFO:tasks.workunit.client.0.smithi032.stdout:9/889: mknod d2/d15/d55/d59/d93/d95/db8/d81/d4a/d8e/df6/c11a 0 2024-04-06T14:17:18.480 INFO:tasks.workunit.client.1.smithi086.stdout:7/683: chown d7/d10/d11/d1c/d24/d4d/daf/db5/d7a 284494 1 2024-04-06T14:17:18.483 INFO:tasks.workunit.client.0.smithi032.stdout:9/890: creat d2/d15/d1e/d65/d83/d31/d39/f11b x:0 0 0 2024-04-06T14:17:18.485 INFO:tasks.workunit.client.1.smithi086.stdout:2/704: mknod d0/cf7 0 2024-04-06T14:17:18.486 INFO:tasks.workunit.client.0.smithi032.stdout:7/995: creat d16/d20/d54/de4/f160 x:0 0 0 2024-04-06T14:17:18.487 INFO:tasks.workunit.client.1.smithi086.stdout:1/742: dwrite f1 [0,4194304] 0 2024-04-06T14:17:18.490 INFO:tasks.workunit.client.0.smithi032.stdout:7/996: symlink d16/d20/d54/de4/d5f/ddd/l161 0 2024-04-06T14:17:18.491 INFO:tasks.workunit.client.0.smithi032.stdout:9/891: dwrite d2/fb9 [0,4194304] 0 2024-04-06T14:17:18.494 INFO:tasks.workunit.client.1.smithi086.stdout:6/792: dread d6/d13/f77 [4194304,4194304] 0 2024-04-06T14:17:18.494 INFO:tasks.workunit.client.1.smithi086.stdout:0/740: creat d0/d1/d2/d1a/ff1 x:0 0 0 2024-04-06T14:17:18.494 INFO:tasks.workunit.client.1.smithi086.stdout:3/726: write d5/d36/d5a/d6d/d72/faf [487092,84647] 0 2024-04-06T14:17:18.495 INFO:tasks.workunit.client.1.smithi086.stdout:7/684: symlink d7/d10/d11/d1c/d24/d2c/d69/d83/le5 0 2024-04-06T14:17:18.495 INFO:tasks.workunit.client.0.smithi032.stdout:9/892: symlink d2/d15/d1e/d41/d6c/d74/l11c 0 2024-04-06T14:17:18.495 INFO:tasks.workunit.client.1.smithi086.stdout:8/748: write d0/f26 [964132,51108] 0 2024-04-06T14:17:18.496 INFO:tasks.workunit.client.1.smithi086.stdout:1/743: creat db/d11/d87/fec x:0 0 0 2024-04-06T14:17:18.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.498+0000 7f1c5719f700 1 -- 172.21.15.32:0/915964353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c500fb1e0 msgr2=0x7f1c500fb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:18.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.498+0000 7f1c5719f700 1 --2- 172.21.15.32:0/915964353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c500fb1e0 0x7f1c500fb610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c44004940 tx=0x7f1c44009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:18.500 INFO:tasks.workunit.client.0.smithi032.stdout:7/997: creat d16/d20/d54/de4/d2d/d30/d3e/da7/db5/dde/d11a/f162 x:0 0 0 2024-04-06T14:17:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.500+0000 7f1c5719f700 1 -- 172.21.15.32:0/915964353 shutdown_connections 2024-04-06T14:17:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.500+0000 7f1c5719f700 1 --2- 172.21.15.32:0/915964353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c500fbb50 0x7f1c50102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.500+0000 7f1c5719f700 1 --2- 172.21.15.32:0/915964353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c500fb1e0 0x7f1c500fb610 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.500+0000 7f1c5719f700 1 -- 172.21.15.32:0/915964353 >> 172.21.15.32:0/915964353 conn(0x7f1c500f6ba0 msgr2=0x7f1c500f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.500+0000 7f1c5719f700 1 -- 172.21.15.32:0/915964353 shutdown_connections 2024-04-06T14:17:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.500+0000 7f1c5719f700 1 -- 172.21.15.32:0/915964353 wait complete. 2024-04-06T14:17:18.501 INFO:tasks.workunit.client.0.smithi032.stdout:9/893: fsync d2/d15/d1e/d65/d83/d7e/f44 0 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.500+0000 7f1c5719f700 1 Processor -- start 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.501+0000 7f1c5719f700 1 -- start start 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.501+0000 7f1c5719f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c500fb1e0 0x7f1c500fdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.501+0000 7f1c5719f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c500fbb50 0x7f1c500fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.501+0000 7f1c5719f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50111590 con 0x7f1c500fbb50 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.501+0000 7f1c5719f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50111700 con 0x7f1c500fb1e0 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.501+0000 7f1c4ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c500fbb50 0x7f1c500fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.501+0000 7f1c4ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c500fbb50 0x7f1c500fc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55078/0 (socket says 172.21.15.32:55078) 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.501+0000 7f1c4ffff700 1 -- 172.21.15.32:0/2721467914 learned_addr learned my addr 172.21.15.32:0/2721467914 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.502+0000 7f1c4ffff700 1 -- 172.21.15.32:0/2721467914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c500fb1e0 msgr2=0x7f1c500fdcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:17:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.502+0000 7f1c4ffff700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c500fb1e0 0x7f1c500fdcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.502+0000 7f1c4ffff700 1 -- 172.21.15.32:0/2721467914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c44005040 con 0x7f1c500fbb50 2024-04-06T14:17:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.502+0000 7f1c4ffff700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c500fbb50 0x7f1c500fc1c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00eb40 tx=0x7f1c3c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.502+0000 7f1c4d7fa700 1 -- 172.21.15.32:0/2721467914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c3c007cc0 con 0x7f1c500fbb50 2024-04-06T14:17:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.502+0000 7f1c4d7fa700 1 -- 172.21.15.32:0/2721467914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c3c00f040 con 0x7f1c500fbb50 2024-04-06T14:17:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.502+0000 7f1c4d7fa700 1 -- 172.21.15.32:0/2721467914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c3c009570 con 0x7f1c500fbb50 2024-04-06T14:17:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.502+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c50111980 con 0x7f1c500fbb50 2024-04-06T14:17:18.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.503+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c50111f40 con 0x7f1c500fbb50 2024-04-06T14:17:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.504+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c5005f5f0 con 0x7f1c500fbb50 2024-04-06T14:17:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.505+0000 7f1c4d7fa700 1 -- 172.21.15.32:0/2721467914 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f1c3c0096d0 con 0x7f1c500fbb50 2024-04-06T14:17:18.507 INFO:tasks.workunit.client.1.smithi086.stdout:6/793: read d6/d13/d42/f7b [409001,95807] 0 2024-04-06T14:17:18.508 INFO:tasks.workunit.client.1.smithi086.stdout:3/727: dwrite d5/d36/d7d/fec [0,4194304] 0 2024-04-06T14:17:18.508 INFO:tasks.workunit.client.1.smithi086.stdout:1/744: rmdir db/d13/d71 39 2024-04-06T14:17:18.509 INFO:tasks.workunit.client.0.smithi032.stdout:9/894: rename d2/d15/d55 to d2/d15/d1e/d65/d6b/d92/d11d 0 2024-04-06T14:17:18.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.506+0000 7f1c4d7fa700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f1c4006aec0 0x7f1c4006d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:18.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.506+0000 7f1c4d7fa700 1 -- 172.21.15.32:0/2721467914 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f1c3c089550 con 0x7f1c500fbb50 2024-04-06T14:17:18.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.506+0000 7f1c54f3b700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f1c4006aec0 0x7f1c4006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:18.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.508+0000 7f1c54f3b700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f1c4006aec0 0x7f1c4006d370 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009fd0 tx=0x7f1c44009e30 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:18.511 INFO:tasks.workunit.client.0.smithi032.stdout:7/998: link d16/c4e d16/d20/d7a/da6/c163 0 2024-04-06T14:17:18.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.513+0000 7f1c4d7fa700 1 -- 172.21.15.32:0/2721467914 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f1c3c04f560 con 0x7f1c500fbb50 2024-04-06T14:17:18.514 INFO:tasks.workunit.client.1.smithi086.stdout:7/685: unlink d7/d10/c1f 0 2024-04-06T14:17:18.514 INFO:tasks.workunit.client.1.smithi086.stdout:3/728: dread - d5/d36/d5c/fd6 zero size 2024-04-06T14:17:18.518 INFO:tasks.workunit.client.1.smithi086.stdout:8/749: dwrite d0/d3b/fa4 [0,4194304] 0 2024-04-06T14:17:18.519 INFO:tasks.workunit.client.1.smithi086.stdout:9/574: rename d5/d13/d1a/d5f/d88/d14/d19/d25/d59/l6b to d5/lb7 0 2024-04-06T14:17:18.519 INFO:tasks.workunit.client.0.smithi032.stdout:9/895: mknod d2/d15/d1e/d65/d83/d31/d39/da6/d10a/c11e 0 2024-04-06T14:17:18.520 INFO:tasks.workunit.client.1.smithi086.stdout:6/794: unlink d6/d13/d1a/ffb 0 2024-04-06T14:17:18.522 INFO:tasks.workunit.client.0.smithi032.stdout:7/999: read d16/d20/d54/de4/d5f/f8a [2858536,61661] 0 2024-04-06T14:17:18.524 INFO:tasks.workunit.client.1.smithi086.stdout:6/795: read d6/d13/d5f/fb8 [2935887,114599] 0 2024-04-06T14:17:18.525 INFO:tasks.workunit.client.1.smithi086.stdout:8/750: unlink d0/d71/d16/d52/lef 0 2024-04-06T14:17:18.529 INFO:tasks.workunit.client.1.smithi086.stdout:5/574: rename d0/d16/d18/d21/f45 to d0/d16/d71/fb9 0 2024-04-06T14:17:18.529 INFO:tasks.workunit.client.1.smithi086.stdout:0/741: rename d0/d15/d4e/da3 to d0/d15/d4e/da3/d44/d8e/d8f/df2 22 2024-04-06T14:17:18.530 INFO:tasks.workunit.client.1.smithi086.stdout:1/745: dread db/d11/d1a/d2a/f47 [0,4194304] 0 2024-04-06T14:17:18.530 INFO:tasks.workunit.client.1.smithi086.stdout:3/729: symlink d5/led 0 2024-04-06T14:17:18.530 INFO:tasks.workunit.client.1.smithi086.stdout:0/742: write d0/d15/d4e/f81 [6363868,84898] 0 2024-04-06T14:17:18.532 INFO:tasks.workunit.client.1.smithi086.stdout:9/575: write d5/d13/d1a/d5f/d88/d14/d19/f5d [4948577,112678] 0 2024-04-06T14:17:18.533 INFO:tasks.workunit.client.1.smithi086.stdout:8/751: symlink d0/d5a/d100/l103 0 2024-04-06T14:17:18.537 INFO:tasks.workunit.client.1.smithi086.stdout:2/705: rename d0/d20/d44/dbb/dc4/de3/d76/d78/f8d to d0/d20/d27/d2b/d35/d36/d4a/ff8 0 2024-04-06T14:17:18.537 INFO:tasks.workunit.client.0.smithi032.stdout:9/896: write d2/d15/d1e/f6d [4379623,126445] 0 2024-04-06T14:17:18.538 INFO:tasks.workunit.client.1.smithi086.stdout:2/706: dread - d0/d20/d27/d37/d52/feb zero size 2024-04-06T14:17:18.538 INFO:tasks.workunit.client.1.smithi086.stdout:2/707: chown d0/d20/d27/d37 891741 1 2024-04-06T14:17:18.540 INFO:tasks.workunit.client.0.smithi032.stdout:9/897: mkdir d2/d15/d1e/d41/d6c/d88/de6/d11f 0 2024-04-06T14:17:18.540 INFO:tasks.workunit.client.1.smithi086.stdout:7/686: creat d7/d10/d11/d1c/fe6 x:0 0 0 2024-04-06T14:17:18.541 INFO:tasks.workunit.client.1.smithi086.stdout:7/687: chown d7/d10/d11/d1c/d36/dbc 86029906 1 2024-04-06T14:17:18.541 INFO:tasks.workunit.client.1.smithi086.stdout:7/688: chown d7/d1d/l3a 28 1 2024-04-06T14:17:18.542 INFO:tasks.workunit.client.1.smithi086.stdout:1/746: creat db/d11/d1a/d2a/d9a/d78/fed x:0 0 0 2024-04-06T14:17:18.543 INFO:tasks.workunit.client.1.smithi086.stdout:7/689: write d7/fb [2274671,23743] 0 2024-04-06T14:17:18.543 INFO:tasks.workunit.client.0.smithi032.stdout:9/898: unlink d2/d15/d1e/d65/d6b/d92/d11d/d59/l7a 0 2024-04-06T14:17:18.545 INFO:tasks.workunit.client.1.smithi086.stdout:9/576: creat d5/d13/d1a/d8f/fb8 x:0 0 0 2024-04-06T14:17:18.545 INFO:tasks.workunit.client.1.smithi086.stdout:8/752: creat d0/d71/d16/d52/dd0/ddc/de7/f104 x:0 0 0 2024-04-06T14:17:18.547 INFO:tasks.workunit.client.1.smithi086.stdout:2/708: symlink d0/d20/d27/d2b/d35/d36/d4d/d6f/lf9 0 2024-04-06T14:17:18.548 INFO:tasks.workunit.client.1.smithi086.stdout:2/709: fsync d0/d20/d44/dbb/dc4/de3/f54 0 2024-04-06T14:17:18.549 INFO:tasks.workunit.client.1.smithi086.stdout:3/730: dwrite d5/dea/db4/da7/d89/f8f [0,4194304] 0 2024-04-06T14:17:18.550 INFO:tasks.workunit.client.1.smithi086.stdout:2/710: fdatasync d0/d20/d27/d37/d52/d9c/fd9 0 2024-04-06T14:17:18.551 INFO:tasks.workunit.client.1.smithi086.stdout:8/753: readlink d0/d71/d82/lc0 0 2024-04-06T14:17:18.552 INFO:tasks.workunit.client.1.smithi086.stdout:2/711: chown d0/d20/d44/dbb/dc4/de3/d76/d78/c95 528097297 1 2024-04-06T14:17:18.553 INFO:tasks.workunit.client.1.smithi086.stdout:7/690: mknod d7/d10/d11/d1c/d24/dc8/ce7 0 2024-04-06T14:17:18.554 INFO:tasks.workunit.client.1.smithi086.stdout:1/747: mkdir db/d11/d42/dee 0 2024-04-06T14:17:18.556 INFO:tasks.workunit.client.1.smithi086.stdout:6/796: write d6/d13/d5f/d60/f6d [888283,128738] 0 2024-04-06T14:17:18.558 INFO:tasks.workunit.client.1.smithi086.stdout:5/575: rename d0/d16/ca4 to d0/d16/d18/d21/d23/d28/cba 0 2024-04-06T14:17:18.561 INFO:tasks.workunit.client.1.smithi086.stdout:1/748: symlink db/d11/d1a/d2a/d9a/d68/daf/lef 0 2024-04-06T14:17:18.562 INFO:tasks.workunit.client.1.smithi086.stdout:0/743: truncate d0/d15/fce 1938996 0 2024-04-06T14:17:18.566 INFO:tasks.workunit.client.1.smithi086.stdout:0/744: write d0/d15/d4e/da3/fe7 [522307,75508] 0 2024-04-06T14:17:18.567 INFO:tasks.workunit.client.1.smithi086.stdout:5/576: mknod d0/d16/d18/d2a/d50/d66/cbb 0 2024-04-06T14:17:18.568 INFO:tasks.workunit.client.1.smithi086.stdout:2/712: creat d0/df/ffa x:0 0 0 2024-04-06T14:17:18.569 INFO:tasks.workunit.client.1.smithi086.stdout:3/731: write d5/f50 [282813,45525] 0 2024-04-06T14:17:18.569 INFO:tasks.workunit.client.1.smithi086.stdout:7/691: symlink d7/le8 0 2024-04-06T14:17:18.573 INFO:tasks.workunit.client.1.smithi086.stdout:7/692: chown d7/d10/d11/d80/f4a 284277524 1 2024-04-06T14:17:18.574 INFO:tasks.workunit.client.1.smithi086.stdout:0/745: creat d0/d15/d4e/da3/d46/d52/ff3 x:0 0 0 2024-04-06T14:17:18.575 INFO:tasks.workunit.client.1.smithi086.stdout:3/732: chown d5/d36/d5a/d6d/d72/d57/c8b 34516421 1 2024-04-06T14:17:18.576 INFO:tasks.workunit.client.1.smithi086.stdout:5/577: symlink d0/d16/d18/d21/d23/d28/d94/d80/d85/lbc 0 2024-04-06T14:17:18.576 INFO:tasks.workunit.client.1.smithi086.stdout:6/797: write d6/d9/db/f7a [2737092,29876] 0 2024-04-06T14:17:18.579 INFO:tasks.workunit.client.1.smithi086.stdout:1/749: write db/d11/d1a/d2a/d9a/d78/f9f [480822,130599] 0 2024-04-06T14:17:18.582 INFO:tasks.workunit.client.1.smithi086.stdout:8/754: dwrite d0/d71/d22/f83 [0,4194304] 0 2024-04-06T14:17:18.583 INFO:tasks.workunit.client.1.smithi086.stdout:6/798: dread - d6/d13/d1a/d22/ff5 zero size 2024-04-06T14:17:18.583 INFO:tasks.workunit.client.1.smithi086.stdout:7/693: write d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/f89 [2772370,68472] 0 2024-04-06T14:17:18.587 INFO:tasks.workunit.client.1.smithi086.stdout:0/746: mkdir d0/d15/d4e/da3/d44/df4 0 2024-04-06T14:17:18.589 INFO:tasks.workunit.client.1.smithi086.stdout:3/733: write d5/d36/d65/fa6 [4424015,121796] 0 2024-04-06T14:17:18.592 INFO:tasks.workunit.client.1.smithi086.stdout:5/578: creat d0/d16/d18/d21/d23/d28/d94/d8a/fbd x:0 0 0 2024-04-06T14:17:18.594 INFO:tasks.workunit.client.1.smithi086.stdout:8/755: symlink d0/d71/d22/d43/d4d/da3/dc7/l105 0 2024-04-06T14:17:18.594 INFO:tasks.workunit.client.1.smithi086.stdout:2/713: dread d0/df/d5e/d9d/fe0 [0,4194304] 0 2024-04-06T14:17:18.599 INFO:tasks.workunit.client.1.smithi086.stdout:6/799: dwrite d6/d13/d1a/da0/f103 [0,4194304] 0 2024-04-06T14:17:18.599 INFO:tasks.workunit.client.1.smithi086.stdout:7/694: creat d7/d10/d11/d1c/d36/fe9 x:0 0 0 2024-04-06T14:17:18.600 INFO:tasks.workunit.client.1.smithi086.stdout:8/756: write d0/f26 [1633377,48463] 0 2024-04-06T14:17:18.603 INFO:tasks.workunit.client.1.smithi086.stdout:5/579: creat d0/d16/d18/d2a/d50/d66/fbe x:0 0 0 2024-04-06T14:17:18.604 INFO:tasks.workunit.client.1.smithi086.stdout:0/747: link d0/d15/d26/f4b d0/d15/d4e/da3/d46/d52/ff5 0 2024-04-06T14:17:18.604 INFO:tasks.workunit.client.1.smithi086.stdout:6/800: truncate d6/d13/fb0 4518558 0 2024-04-06T14:17:18.607 INFO:tasks.workunit.client.1.smithi086.stdout:3/734: dwrite d5/d36/d5a/d6d/d72/d64/fdc [0,4194304] 0 2024-04-06T14:17:18.607 INFO:tasks.workunit.client.1.smithi086.stdout:7/695: write d7/d10/d11/d1c/d24/dc8/fda [1166909,17393] 0 2024-04-06T14:17:18.609 INFO:tasks.workunit.client.1.smithi086.stdout:8/757: mkdir d0/d71/d22/d43/d4d/da3/dc7/d106 0 2024-04-06T14:17:18.614 INFO:tasks.workunit.client.1.smithi086.stdout:2/714: dwrite d0/d20/d27/d37/d52/d9c/fd9 [0,4194304] 0 2024-04-06T14:17:18.618 INFO:tasks.workunit.client.1.smithi086.stdout:5/580: dwrite d0/d16/d18/d21/d23/f6c [0,4194304] 0 2024-04-06T14:17:18.652 INFO:tasks.workunit.client.1.smithi086.stdout:0/748: creat d0/d1/d2/d1a/d79/ff6 x:0 0 0 2024-04-06T14:17:18.653 INFO:tasks.workunit.client.1.smithi086.stdout:6/801: creat d6/d13/d5f/d60/d45/d8a/de3/f10b x:0 0 0 2024-04-06T14:17:18.654 INFO:tasks.workunit.client.1.smithi086.stdout:3/735: mkdir d5/dea/dd2/ddb/de6/dee 0 2024-04-06T14:17:18.678 INFO:tasks.workunit.client.1.smithi086.stdout:4/565: sync 2024-04-06T14:17:18.679 INFO:tasks.workunit.client.1.smithi086.stdout:4/566: dread - d0/d2c/d30/d24/d93/fa4 zero size 2024-04-06T14:17:18.684 INFO:tasks.workunit.client.1.smithi086.stdout:4/567: dwrite d0/d15/d20/d34/f55 [4194304,4194304] 0 2024-04-06T14:17:18.744 INFO:tasks.workunit.client.1.smithi086.stdout:5/581: fdatasync d0/d16/d18/d21/d23/d28/d94/f4f 0 2024-04-06T14:17:18.745 INFO:tasks.workunit.client.1.smithi086.stdout:6/802: fdatasync d6/d9/fab 0 2024-04-06T14:17:18.746 INFO:tasks.workunit.client.1.smithi086.stdout:3/736: fsync d5/d36/d5a/d6d/f96 0 2024-04-06T14:17:18.749 INFO:tasks.workunit.client.1.smithi086.stdout:4/568: write d0/de/f11 [2245680,11455] 0 2024-04-06T14:17:18.750 INFO:tasks.workunit.client.1.smithi086.stdout:8/758: write d0/d71/d16/f6c [3103573,11396] 0 2024-04-06T14:17:18.750 INFO:tasks.workunit.client.1.smithi086.stdout:3/737: stat d5/dea/dd2/ddb/de6/f45 0 2024-04-06T14:17:18.750 INFO:tasks.workunit.client.1.smithi086.stdout:5/582: mkdir d0/d16/d18/d21/d23/d28/d94/d8a/dbf 0 2024-04-06T14:17:18.751 INFO:tasks.workunit.client.1.smithi086.stdout:0/749: write d0/d15/d4e/f42 [1596589,25614] 0 2024-04-06T14:17:18.754 INFO:tasks.workunit.client.1.smithi086.stdout:6/803: creat d6/d13/d1a/d22/dcf/db2/f10c x:0 0 0 2024-04-06T14:17:18.755 INFO:tasks.workunit.client.1.smithi086.stdout:3/738: mknod d5/d26/d7f/dae/cef 0 2024-04-06T14:17:18.757 INFO:tasks.workunit.client.1.smithi086.stdout:0/750: creat d0/d15/d26/d33/d7a/daa/ff7 x:0 0 0 2024-04-06T14:17:18.758 INFO:tasks.workunit.client.1.smithi086.stdout:3/739: fdatasync d5/d36/d65/f93 0 2024-04-06T14:17:18.759 INFO:tasks.workunit.client.1.smithi086.stdout:8/759: rename d0/d71/d22/d43/d6d/d75/d78/cdf to d0/d71/d22/d65/d6e/d98/c107 0 2024-04-06T14:17:18.762 INFO:tasks.workunit.client.1.smithi086.stdout:5/583: dwrite d0/d16/d18/f9d [0,4194304] 0 2024-04-06T14:17:18.764 INFO:tasks.workunit.client.1.smithi086.stdout:2/715: dwrite d0/d20/d44/d9f/fc8 [0,4194304] 0 2024-04-06T14:17:18.764 INFO:tasks.workunit.client.1.smithi086.stdout:6/804: rename d6/d9/db/c14 to d6/d20/de0/c10d 0 2024-04-06T14:17:18.765 INFO:tasks.workunit.client.1.smithi086.stdout:0/751: truncate d0/d1/d4/de/db4/fc9 697597 0 2024-04-06T14:17:18.765 INFO:tasks.workunit.client.1.smithi086.stdout:3/740: creat d5/dea/dd2/ddb/de6/ff0 x:0 0 0 2024-04-06T14:17:18.766 INFO:tasks.workunit.client.1.smithi086.stdout:5/584: write d0/d16/d18/d2a/d50/d66/fbe [153351,43860] 0 2024-04-06T14:17:18.768 INFO:tasks.workunit.client.0.smithi032.stdout:1/995: sync 2024-04-06T14:17:18.768 INFO:tasks.workunit.client.1.smithi086.stdout:2/716: creat d0/d20/d80/dbf/ffb x:0 0 0 2024-04-06T14:17:18.769 INFO:tasks.workunit.client.0.smithi032.stdout:1/996: truncate d0/d41/d59/fe9 5056362 0 2024-04-06T14:17:18.770 INFO:tasks.workunit.client.1.smithi086.stdout:8/760: link d0/d71/d16/d52/dd0/ddc/de7/f3e d0/d71/f108 0 2024-04-06T14:17:18.772 INFO:tasks.workunit.client.1.smithi086.stdout:8/761: chown d0/d71/d16/d39/d90/cff 325 1 2024-04-06T14:17:18.772 INFO:tasks.workunit.client.1.smithi086.stdout:6/805: rename d6/d13/d5f/d60/d45/da6/lba to d6/d13/d42/d52/df1/l10e 0 2024-04-06T14:17:18.773 INFO:tasks.workunit.client.1.smithi086.stdout:0/752: dread d0/d1/d2/d1a/d79/fb2 [0,4194304] 0 2024-04-06T14:17:18.774 INFO:tasks.workunit.client.1.smithi086.stdout:3/741: creat d5/d36/d5c/d8e/dc1/ff1 x:0 0 0 2024-04-06T14:17:18.775 INFO:tasks.workunit.client.1.smithi086.stdout:2/717: getdents d0/df/d5e/d9d/de7 0 2024-04-06T14:17:18.775 INFO:tasks.workunit.client.0.smithi032.stdout:1/997: creat d0/d41/d5b/f143 x:0 0 0 2024-04-06T14:17:18.777 INFO:tasks.workunit.client.1.smithi086.stdout:6/806: unlink d6/d9/lf 0 2024-04-06T14:17:18.777 INFO:tasks.workunit.client.0.smithi032.stdout:1/998: rmdir d0/d10/d16 39 2024-04-06T14:17:18.778 INFO:tasks.workunit.client.1.smithi086.stdout:6/807: chown d6/d9/db/dff 109 1 2024-04-06T14:17:18.780 INFO:tasks.workunit.client.1.smithi086.stdout:5/585: link d0/d16/d18/d35/fa2 d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc0 0 2024-04-06T14:17:18.782 INFO:tasks.workunit.client.1.smithi086.stdout:3/742: rename d5/dea/c7c to d5/d26/d7f/d59/d70/cf2 0 2024-04-06T14:17:18.782 INFO:tasks.workunit.client.0.smithi032.stdout:1/999: rename d0/d10/d16/d3f/f5d to d0/d10/d78/d128/f144 0 2024-04-06T14:17:18.783 INFO:tasks.workunit.client.1.smithi086.stdout:5/586: symlink d0/d16/d71/d96/lc1 0 2024-04-06T14:17:18.784 INFO:tasks.workunit.client.1.smithi086.stdout:6/808: rename d6/d13/f75 to d6/d13/d42/d52/f10f 0 2024-04-06T14:17:18.787 INFO:tasks.workunit.client.1.smithi086.stdout:3/743: rename d5/led to d5/d26/d7f/d59/lf3 0 2024-04-06T14:17:18.789 INFO:tasks.workunit.client.1.smithi086.stdout:5/587: getdents d0 0 2024-04-06T14:17:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.791+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c50101c60 con 0x7f1c4006aec0 2024-04-06T14:17:18.792 INFO:tasks.workunit.client.1.smithi086.stdout:5/588: chown d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc0 651236 1 2024-04-06T14:17:18.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.793+0000 7f1c4d7fa700 1 -- 172.21.15.32:0/2721467914 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f1c50101c60 con 0x7f1c4006aec0 2024-04-06T14:17:18.793 INFO:tasks.workunit.client.1.smithi086.stdout:5/589: rmdir d0 39 2024-04-06T14:17:18.794 INFO:tasks.workunit.client.1.smithi086.stdout:3/744: dwrite d5/d36/d5a/fd4 [0,4194304] 0 2024-04-06T14:17:18.794 INFO:tasks.workunit.client.1.smithi086.stdout:5/590: readlink d0/d16/d71/d96/lc1 0 2024-04-06T14:17:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f1c4006aec0 msgr2=0x7f1c4006d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f1c4006aec0 0x7f1c4006d370 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009fd0 tx=0x7f1c44009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c500fbb50 msgr2=0x7f1c500fc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c500fbb50 0x7f1c500fc1c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00eb40 tx=0x7f1c3c00b040 comp rx=0 tx=0).stop 2024-04-06T14:17:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 shutdown_connections 2024-04-06T14:17:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f1c4006aec0 0x7f1c4006d370 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c500fb1e0 0x7f1c500fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 --2- 172.21.15.32:0/2721467914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c500fbb50 0x7f1c500fc1c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 >> 172.21.15.32:0/2721467914 conn(0x7f1c500f6ba0 msgr2=0x7f1c50100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 shutdown_connections 2024-04-06T14:17:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.795+0000 7f1c5719f700 1 -- 172.21.15.32:0/2721467914 wait complete. 2024-04-06T14:17:18.796 INFO:tasks.workunit.client.1.smithi086.stdout:5/591: truncate d0/d16/d18/d2a/d50/f93 71892 0 2024-04-06T14:17:18.797 INFO:tasks.workunit.client.1.smithi086.stdout:3/745: truncate d5/d36/d56/dd3/fe4 2977398 0 2024-04-06T14:17:18.800 INFO:tasks.workunit.client.1.smithi086.stdout:5/592: rename d0/d19 to d0/d16/d18/d21/d23/d28/dc2 0 2024-04-06T14:17:18.802 INFO:tasks.workunit.client.1.smithi086.stdout:5/593: creat d0/d43/da6/fc3 x:0 0 0 2024-04-06T14:17:18.804 INFO:tasks.workunit.client.1.smithi086.stdout:5/594: mknod d0/d16/d71/d89/cc4 0 2024-04-06T14:17:18.805 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:17:18.806 INFO:tasks.workunit.client.1.smithi086.stdout:5/595: creat d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc5 x:0 0 0 2024-04-06T14:17:18.815 INFO:tasks.workunit.client.1.smithi086.stdout:9/577: sync 2024-04-06T14:17:18.829 INFO:tasks.workunit.client.1.smithi086.stdout:0/753: dread d0/d15/d26/f2c [0,4194304] 0 2024-04-06T14:17:18.832 INFO:tasks.workunit.client.1.smithi086.stdout:6/809: symlink d6/d13/d5f/d60/d45/l110 0 2024-04-06T14:17:18.833 INFO:tasks.workunit.client.1.smithi086.stdout:0/754: rename d0/ff to d0/d1/d2/d1a/d79/ff8 0 2024-04-06T14:17:18.834 INFO:tasks.workunit.client.1.smithi086.stdout:6/810: creat d6/d9/d5c/f111 x:0 0 0 2024-04-06T14:17:18.835 INFO:tasks.workunit.client.1.smithi086.stdout:6/811: fdatasync d6/d9/da9/fec 0 2024-04-06T14:17:18.837 INFO:tasks.workunit.client.1.smithi086.stdout:0/755: dread d0/d15/d26/fa0 [0,4194304] 0 2024-04-06T14:17:18.838 INFO:tasks.workunit.client.1.smithi086.stdout:6/812: symlink d6/d9/d63/l112 0 2024-04-06T14:17:18.839 INFO:tasks.workunit.client.1.smithi086.stdout:6/813: write d6/d13/d1a/d22/dcf/fd7 [906094,3616] 0 2024-04-06T14:17:18.854 INFO:tasks.workunit.client.1.smithi086.stdout:0/756: dread d0/d15/d4e/da3/d46/d52/dd0/fa7 [0,4194304] 0 2024-04-06T14:17:18.858 INFO:tasks.workunit.client.0.smithi032.stdout:2/812: sync 2024-04-06T14:17:18.858 INFO:tasks.workunit.client.0.smithi032.stdout:0/739: sync 2024-04-06T14:17:18.859 INFO:tasks.workunit.client.1.smithi086.stdout:0/757: dwrite d0/d1/d2/d1a/d51/dd7/de2/f27 [0,4194304] 0 2024-04-06T14:17:18.861 INFO:tasks.workunit.client.1.smithi086.stdout:0/758: chown d0/d1/d4/de/f2e 1 1 2024-04-06T14:17:18.861 INFO:tasks.workunit.client.0.smithi032.stdout:0/740: creat d3/da/d25/d2b/d4a/d79/d97/ff8 x:0 0 0 2024-04-06T14:17:18.862 INFO:tasks.workunit.client.1.smithi086.stdout:0/759: chown d0/d1/d2/f67 3550 1 2024-04-06T14:17:18.862 INFO:tasks.workunit.client.1.smithi086.stdout:6/814: link d6/d13/d1a/d64/fb3 d6/d13/d1a/d22/f113 0 2024-04-06T14:17:18.863 INFO:tasks.workunit.client.0.smithi032.stdout:2/813: creat d0/d6e/dae/db7/df9/f110 x:0 0 0 2024-04-06T14:17:18.863 INFO:tasks.workunit.client.1.smithi086.stdout:0/760: dread - d0/d15/d4e/da3/d73/fcc zero size 2024-04-06T14:17:18.864 INFO:tasks.workunit.client.1.smithi086.stdout:0/761: dread - d0/d1/d4/fbd zero size 2024-04-06T14:17:18.865 INFO:tasks.workunit.client.1.smithi086.stdout:0/762: dread - d0/d1/d2/d1a/ff1 zero size 2024-04-06T14:17:18.867 INFO:tasks.workunit.client.1.smithi086.stdout:6/815: rename d6/d13/d1a/d22/dcf to d6/d13/d1a/da0/d114 0 2024-04-06T14:17:18.868 INFO:tasks.workunit.client.0.smithi032.stdout:0/741: truncate d3/da/f1d 2800626 0 2024-04-06T14:17:18.868 INFO:tasks.workunit.client.0.smithi032.stdout:2/814: creat d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102/f111 x:0 0 0 2024-04-06T14:17:18.869 INFO:tasks.workunit.client.0.smithi032.stdout:2/815: dread - d0/d6e/dae/db7/ffb zero size 2024-04-06T14:17:18.870 INFO:tasks.workunit.client.1.smithi086.stdout:0/763: write d0/d1/d4/de/f3c [1441889,50440] 0 2024-04-06T14:17:18.872 INFO:tasks.workunit.client.1.smithi086.stdout:6/816: rename d6/d13/d1a/da0/fcc to d6/d13/d5f/d60/d45/d4a/f115 0 2024-04-06T14:17:18.875 INFO:tasks.workunit.client.1.smithi086.stdout:6/817: stat d6/d13/d1a/da0/d114/db2/lf3 0 2024-04-06T14:17:18.877 INFO:tasks.workunit.client.1.smithi086.stdout:0/764: dwrite d0/f16 [0,4194304] 0 2024-04-06T14:17:18.879 INFO:tasks.workunit.client.1.smithi086.stdout:0/765: dread - d0/d1/d2/d1a/ff1 zero size 2024-04-06T14:17:18.882 INFO:tasks.workunit.client.1.smithi086.stdout:0/766: mkdir d0/d1/d4/de/df9 0 2024-04-06T14:17:18.885 INFO:tasks.workunit.client.1.smithi086.stdout:6/818: creat d6/d13/d1a/f116 x:0 0 0 2024-04-06T14:17:18.888 INFO:tasks.workunit.client.1.smithi086.stdout:6/819: creat d6/d13/d5f/f117 x:0 0 0 2024-04-06T14:17:18.890 INFO:tasks.workunit.client.0.smithi032.stdout:2/816: dwrite d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/fa7 [0,4194304] 0 2024-04-06T14:17:18.891 INFO:tasks.workunit.client.1.smithi086.stdout:6/820: fdatasync d6/d13/d5f/fb5 0 2024-04-06T14:17:18.894 INFO:tasks.workunit.client.0.smithi032.stdout:2/817: write d0/d30/d39/d59/d67/d74/d78/d57/d7c/fb9 [3402617,105691] 0 2024-04-06T14:17:18.895 INFO:tasks.workunit.client.0.smithi032.stdout:2/818: chown d0/d5a/f8e 0 1 2024-04-06T14:17:18.895 INFO:tasks.workunit.client.1.smithi086.stdout:6/821: chown d6/d13/d42/d52/df1 1155 1 2024-04-06T14:17:18.900 INFO:tasks.workunit.client.1.smithi086.stdout:0/767: dread d0/d15/d4e/f45 [0,4194304] 0 2024-04-06T14:17:18.901 INFO:tasks.workunit.client.0.smithi032.stdout:2/819: link d0/c75 d0/d30/d39/d59/d67/d74/d78/dc2/c112 0 2024-04-06T14:17:18.902 INFO:tasks.workunit.client.1.smithi086.stdout:0/768: truncate d0/d1/d2/d1a/ff1 129930 0 2024-04-06T14:17:18.903 INFO:tasks.workunit.client.1.smithi086.stdout:6/822: dwrite d6/d9/db/f7a [0,4194304] 0 2024-04-06T14:17:18.906 INFO:tasks.workunit.client.1.smithi086.stdout:0/769: mkdir d0/d1/d2/d1a/d79/dac/dc2/dfa 0 2024-04-06T14:17:18.908 INFO:tasks.workunit.client.1.smithi086.stdout:6/823: unlink d6/d9/d43/f74 0 2024-04-06T14:17:18.908 INFO:tasks.workunit.client.0.smithi032.stdout:2/820: rename d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/d103 to d0/d30/d39/d59/d67/d74/d78/d57/d113 0 2024-04-06T14:17:18.912 INFO:tasks.workunit.client.0.smithi032.stdout:2/821: creat d0/d30/d39/d59/f114 x:0 0 0 2024-04-06T14:17:18.915 INFO:tasks.workunit.client.1.smithi086.stdout:0/770: write d0/d15/d4e/da3/d46/f6d [131952,56934] 0 2024-04-06T14:17:18.918 INFO:tasks.workunit.client.1.smithi086.stdout:0/771: truncate d0/d1/fd3 1811756 0 2024-04-06T14:17:18.919 INFO:tasks.workunit.client.0.smithi032.stdout:2/822: dwrite d0/d30/d39/d59/d67/d74/d78/d57/d7c/fa1 [0,4194304] 0 2024-04-06T14:17:18.920 INFO:tasks.workunit.client.1.smithi086.stdout:6/824: dwrite d6/d9/d43/f53 [0,4194304] 0 2024-04-06T14:17:18.921 INFO:tasks.workunit.client.0.smithi032.stdout:2/823: write d0/d30/d39/d59/d67/d80/fd0 [1110994,67466] 0 2024-04-06T14:17:18.923 INFO:tasks.workunit.client.1.smithi086.stdout:6/825: symlink d6/d13/d5f/d60/d45/d4a/l118 0 2024-04-06T14:17:18.925 INFO:tasks.workunit.client.1.smithi086.stdout:0/772: link d0/d1/d2/d1a/d51/dd7/de2/ld4 d0/d1/d4/de/dae/lfb 0 2024-04-06T14:17:18.928 INFO:tasks.workunit.client.1.smithi086.stdout:6/826: symlink d6/d13/d5f/d60/d45/da6/l119 0 2024-04-06T14:17:18.932 INFO:tasks.workunit.client.1.smithi086.stdout:0/773: rename d0/d15/d4e/d48/d89/l65 to d0/d15/d4e/lfc 0 2024-04-06T14:17:18.935 INFO:tasks.workunit.client.1.smithi086.stdout:6/827: rename d6/d9/d63/l6f to d6/d13/d42/d52/df1/l11a 0 2024-04-06T14:17:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.937+0000 7f62bb8ac700 1 -- 172.21.15.32:0/2251951578 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4102360 msgr2=0x7f62b4104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.937+0000 7f62bb8ac700 1 --2- 172.21.15.32:0/2251951578 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4102360 0x7f62b4104740 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f62a4004880 tx=0x7f62a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.937+0000 7f62bb8ac700 1 -- 172.21.15.32:0/2251951578 shutdown_connections 2024-04-06T14:17:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.937+0000 7f62bb8ac700 1 --2- 172.21.15.32:0/2251951578 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4102360 0x7f62b4104740 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.937+0000 7f62bb8ac700 1 --2- 172.21.15.32:0/2251951578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b4066280 0x7f62b4101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.937+0000 7f62bb8ac700 1 -- 172.21.15.32:0/2251951578 >> 172.21.15.32:0/2251951578 conn(0x7f62b40f7430 msgr2=0x7f62b40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.937+0000 7f62bb8ac700 1 -- 172.21.15.32:0/2251951578 shutdown_connections 2024-04-06T14:17:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.937+0000 7f62bb8ac700 1 -- 172.21.15.32:0/2251951578 wait complete. 2024-04-06T14:17:18.941 INFO:tasks.workunit.client.1.smithi086.stdout:6/828: truncate d6/d13/faf 1858406 0 2024-04-06T14:17:18.941 INFO:tasks.workunit.client.1.smithi086.stdout:6/829: fdatasync d6/d13/d5f/d60/d45/d8a/fef 0 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62bb8ac700 1 Processor -- start 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62bb8ac700 1 -- start start 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62bb8ac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4066280 0x7f62b410fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62bb8ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b4102360 0x7f62b4110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62bb8ac700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4110600 con 0x7f62b4102360 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62bb8ac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4110e50 con 0x7f62b4066280 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b8e47700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b4102360 0x7f62b4110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b9648700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4066280 0x7f62b410fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b9648700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4066280 0x7f62b410fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:42274/0 (socket says 172.21.15.32:42274) 2024-04-06T14:17:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b8e47700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b4102360 0x7f62b4110000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55100/0 (socket says 172.21.15.32:55100) 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b9648700 1 -- 172.21.15.32:0/180021393 learned_addr learned my addr 172.21.15.32:0/180021393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b9648700 1 -- 172.21.15.32:0/180021393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b4102360 msgr2=0x7f62b4110000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b9648700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b4102360 0x7f62b4110000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b9648700 1 -- 172.21.15.32:0/180021393 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a4005040 con 0x7f62b4066280 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b8e47700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b4102360 0x7f62b4110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.938+0000 7f62b9648700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4066280 0x7f62b410fac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f62b000eb40 tx=0x7f62b000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.939+0000 7f62a9ffb700 1 -- 172.21.15.32:0/180021393 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b0007cc0 con 0x7f62b4066280 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.939+0000 7f62a9ffb700 1 -- 172.21.15.32:0/180021393 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62b000f040 con 0x7f62b4066280 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.939+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b4111130 con 0x7f62b4066280 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.939+0000 7f62a9ffb700 1 -- 172.21.15.32:0/180021393 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b000ca60 con 0x7f62b4066280 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.939+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b4111670 con 0x7f62b4066280 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.940+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b405f5f0 con 0x7f62b4066280 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.940+0000 7f62a9ffb700 1 -- 172.21.15.32:0/180021393 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f62b000cbc0 con 0x7f62b4066280 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.941+0000 7f62a9ffb700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f62a006aec0 0x7f62a006d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.941+0000 7f62a9ffb700 1 -- 172.21.15.32:0/180021393 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f62b00893f0 con 0x7f62b4066280 2024-04-06T14:17:18.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.941+0000 7f62b8e47700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f62a006aec0 0x7f62a006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:18.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.941+0000 7f62b8e47700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f62a006aec0 0x7f62a006d370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f62a40135d0 tx=0x7f62a4009e30 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:18.945 INFO:tasks.workunit.client.1.smithi086.stdout:6/830: truncate d6/d13/d1a/da0/d114/db2/f10c 764332 0 2024-04-06T14:17:18.945 INFO:tasks.workunit.client.1.smithi086.stdout:6/831: stat d6/d9/d5c 0 2024-04-06T14:17:18.945 INFO:tasks.workunit.client.1.smithi086.stdout:6/832: chown d6/d13/d42/l106 120949 1 2024-04-06T14:17:18.946 INFO:tasks.workunit.client.1.smithi086.stdout:6/833: truncate d6/d13/d1a/d22/f6e 902333 0 2024-04-06T14:17:18.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:18.946+0000 7f62a9ffb700 1 -- 172.21.15.32:0/180021393 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f62b004f560 con 0x7f62b4066280 2024-04-06T14:17:18.950 INFO:tasks.workunit.client.0.smithi032.stdout:2/824: write d0/f22 [159795,94582] 0 2024-04-06T14:17:18.951 INFO:tasks.workunit.client.0.smithi032.stdout:2/825: readlink d0/d30/d39/d59/d67/d80/l9b 0 2024-04-06T14:17:18.954 INFO:tasks.workunit.client.1.smithi086.stdout:0/774: truncate d0/d1/d2/d1a/d79/fb2 1941062 0 2024-04-06T14:17:18.955 INFO:tasks.workunit.client.1.smithi086.stdout:1/750: sync 2024-04-06T14:17:18.955 INFO:tasks.workunit.client.1.smithi086.stdout:6/834: fsync d6/d13/d1a/d22/f6e 0 2024-04-06T14:17:18.956 INFO:tasks.workunit.client.1.smithi086.stdout:6/835: chown d6/c5e 1 1 2024-04-06T14:17:18.958 INFO:tasks.workunit.client.0.smithi032.stdout:2/826: dwrite d0/d30/d39/d59/d67/d74/d78/d57/d7c/fb9 [0,4194304] 0 2024-04-06T14:17:18.961 INFO:tasks.workunit.client.0.smithi032.stdout:2/827: fsync d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/fa7 0 2024-04-06T14:17:18.964 INFO:tasks.workunit.client.0.smithi032.stdout:2/828: mknod d0/d30/d7f/dd9/c115 0 2024-04-06T14:17:18.965 INFO:tasks.workunit.client.1.smithi086.stdout:6/836: rename d6/d13/d5f/d60 to d6/d13/d1a/da0/d11b 0 2024-04-06T14:17:18.966 INFO:tasks.workunit.client.0.smithi032.stdout:2/829: write d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102/f111 [915260,28519] 0 2024-04-06T14:17:18.969 INFO:tasks.workunit.client.0.smithi032.stdout:2/830: symlink d0/d30/d7f/dd9/l116 0 2024-04-06T14:17:18.969 INFO:tasks.workunit.client.1.smithi086.stdout:6/837: chown d6/d13/la3 1 1 2024-04-06T14:17:18.970 INFO:tasks.workunit.client.1.smithi086.stdout:0/775: getdents d0/d1/d4/de/db4 0 2024-04-06T14:17:18.974 INFO:tasks.workunit.client.1.smithi086.stdout:6/838: mknod d6/d13/d1a/da0/d114/db2/dee/c11c 0 2024-04-06T14:17:18.975 INFO:tasks.workunit.client.1.smithi086.stdout:0/776: mknod d0/d15/d4e/da3/d46/d52/cfd 0 2024-04-06T14:17:18.975 INFO:tasks.workunit.client.1.smithi086.stdout:1/751: write db/d11/d1a/d2a/d9a/d68/daf/dc2/fcd [622487,71846] 0 2024-04-06T14:17:18.975 INFO:tasks.workunit.client.0.smithi032.stdout:5/854: sync 2024-04-06T14:17:18.976 INFO:tasks.workunit.client.0.smithi032.stdout:4/884: sync 2024-04-06T14:17:18.976 INFO:tasks.workunit.client.0.smithi032.stdout:6/982: sync 2024-04-06T14:17:18.976 INFO:tasks.workunit.client.0.smithi032.stdout:3/919: sync 2024-04-06T14:17:18.976 INFO:tasks.workunit.client.0.smithi032.stdout:9/899: sync 2024-04-06T14:17:18.978 INFO:tasks.workunit.client.1.smithi086.stdout:6/839: creat d6/d9/f11d x:0 0 0 2024-04-06T14:17:18.978 INFO:tasks.workunit.client.1.smithi086.stdout:0/777: fdatasync d0/d15/d4e/da3/d46/d52/dd0/fa7 0 2024-04-06T14:17:18.978 INFO:tasks.workunit.client.0.smithi032.stdout:4/885: chown d0/df/d1f/l56 690662 1 2024-04-06T14:17:18.979 INFO:tasks.workunit.client.0.smithi032.stdout:3/920: stat d3/d16/ddd/df7/d147/d7d/dc6/de5/c14a 0 2024-04-06T14:17:18.979 INFO:tasks.workunit.client.1.smithi086.stdout:1/752: symlink db/d11/d1a/d2a/d9a/d68/daf/lf0 0 2024-04-06T14:17:18.979 INFO:tasks.workunit.client.0.smithi032.stdout:5/855: mkdir d16/d18/d126/d12b 0 2024-04-06T14:17:18.980 INFO:tasks.workunit.client.1.smithi086.stdout:1/753: readlink db/d11/d1a/d2a/d52/d6b/ld7 0 2024-04-06T14:17:18.980 INFO:tasks.workunit.client.1.smithi086.stdout:6/840: rmdir d6/d9/d5c 39 2024-04-06T14:17:18.982 INFO:tasks.workunit.client.0.smithi032.stdout:6/983: getdents d4/d5/d28/d65/d6e/ddb/d110 0 2024-04-06T14:17:18.982 INFO:tasks.workunit.client.1.smithi086.stdout:1/754: stat db/d11/d1a/d2a/d9a/d78/fe7 0 2024-04-06T14:17:18.982 INFO:tasks.workunit.client.1.smithi086.stdout:0/778: truncate d0/d15/d4e/f92 429633 0 2024-04-06T14:17:18.984 INFO:tasks.workunit.client.0.smithi032.stdout:5/856: creat d16/d18/d126/f12c x:0 0 0 2024-04-06T14:17:18.984 INFO:tasks.workunit.client.0.smithi032.stdout:3/921: rename d3/d16/ddd/df7 to d3/d16/ddd/df7/d147/d89/dcb/d155/d15d 22 2024-04-06T14:17:18.986 INFO:tasks.workunit.client.0.smithi032.stdout:4/886: unlink d0/df/d1f/d5a/c69 0 2024-04-06T14:17:18.986 INFO:tasks.workunit.client.1.smithi086.stdout:6/841: creat d6/d13/d1a/da0/d11b/d45/d8a/f11e x:0 0 0 2024-04-06T14:17:18.986 INFO:tasks.workunit.client.1.smithi086.stdout:6/842: write d6/d13/d42/fea [143340,49227] 0 2024-04-06T14:17:18.987 INFO:tasks.workunit.client.1.smithi086.stdout:6/843: write d6/d13/d42/d52/df1/ff2 [297959,112213] 0 2024-04-06T14:17:18.988 INFO:tasks.workunit.client.0.smithi032.stdout:6/984: chown d4/d5/d8/d4b/d7c/fcb 10746467 1 2024-04-06T14:17:18.988 INFO:tasks.workunit.client.0.smithi032.stdout:3/922: chown d3/d16/d63/cab 4830 1 2024-04-06T14:17:18.988 INFO:tasks.workunit.client.0.smithi032.stdout:5/857: chown d16/d18/d33/d45/d52/d39/d86 401020 1 2024-04-06T14:17:18.989 INFO:tasks.workunit.client.1.smithi086.stdout:1/755: read - db/d11/d42/d53/fdf zero size 2024-04-06T14:17:18.990 INFO:tasks.workunit.client.0.smithi032.stdout:4/887: creat d0/df/d1f/d5a/f12f x:0 0 0 2024-04-06T14:17:18.991 INFO:tasks.workunit.client.1.smithi086.stdout:0/779: mkdir d0/d1/dfe 0 2024-04-06T14:17:18.992 INFO:tasks.workunit.client.0.smithi032.stdout:5/858: chown d16/d18/d33/d45/d52/d39/cb4 360446 1 2024-04-06T14:17:18.992 INFO:tasks.workunit.client.0.smithi032.stdout:4/888: readlink d0/d82/ldf 0 2024-04-06T14:17:18.993 INFO:tasks.workunit.client.1.smithi086.stdout:6/844: creat d6/d13/d1a/d22/f11f x:0 0 0 2024-04-06T14:17:18.993 INFO:tasks.workunit.client.1.smithi086.stdout:6/845: truncate d6/d9/db/f9b 1353697 0 2024-04-06T14:17:18.993 INFO:tasks.workunit.client.1.smithi086.stdout:1/756: getdents db/d11/d1a/d2a/d9a/d68/daf/dc2/dd8 0 2024-04-06T14:17:18.995 INFO:tasks.workunit.client.1.smithi086.stdout:6/846: truncate d6/d9/d5c/f111 1004932 0 2024-04-06T14:17:18.995 INFO:tasks.workunit.client.0.smithi032.stdout:2/831: dread d0/d5a/f4c [0,4194304] 0 2024-04-06T14:17:18.997 INFO:tasks.workunit.client.1.smithi086.stdout:1/757: rename db/d11/d1a/d2a/d9a/d82 to db/d13/d55/df1 0 2024-04-06T14:17:18.997 INFO:tasks.workunit.client.0.smithi032.stdout:9/900: dread d2/d15/d1e/d65/d83/d7e/f37 [0,4194304] 0 2024-04-06T14:17:18.998 INFO:tasks.workunit.client.0.smithi032.stdout:4/889: creat d0/df/d1f/d42/d45/f130 x:0 0 0 2024-04-06T14:17:18.999 INFO:tasks.workunit.client.1.smithi086.stdout:2/718: sync 2024-04-06T14:17:18.999 INFO:tasks.workunit.client.1.smithi086.stdout:5/596: sync 2024-04-06T14:17:18.999 INFO:tasks.workunit.client.1.smithi086.stdout:7/696: sync 2024-04-06T14:17:18.999 INFO:tasks.workunit.client.1.smithi086.stdout:3/746: sync 2024-04-06T14:17:18.999 INFO:tasks.workunit.client.1.smithi086.stdout:8/762: sync 2024-04-06T14:17:18.999 INFO:tasks.workunit.client.1.smithi086.stdout:4/569: sync 2024-04-06T14:17:18.999 INFO:tasks.workunit.client.1.smithi086.stdout:9/578: sync 2024-04-06T14:17:18.999 INFO:tasks.workunit.client.1.smithi086.stdout:6/847: mknod d6/d20/c120 0 2024-04-06T14:17:19.000 INFO:tasks.workunit.client.0.smithi032.stdout:6/985: dwrite d4/d5/d35/f39 [0,4194304] 0 2024-04-06T14:17:19.001 INFO:tasks.workunit.client.0.smithi032.stdout:2/832: truncate d0/d30/d39/d59/d67/d80/f10a 161691 0 2024-04-06T14:17:19.004 INFO:tasks.workunit.client.1.smithi086.stdout:3/747: rmdir d5/dea/db4 39 2024-04-06T14:17:19.005 INFO:tasks.workunit.client.1.smithi086.stdout:2/719: rmdir d0/d20/d82 39 2024-04-06T14:17:19.006 INFO:tasks.workunit.client.0.smithi032.stdout:2/833: unlink d0/d4e/f101 0 2024-04-06T14:17:19.006 INFO:tasks.workunit.client.1.smithi086.stdout:0/780: dread d0/d15/d4e/da3/f39 [0,4194304] 0 2024-04-06T14:17:19.007 INFO:tasks.workunit.client.0.smithi032.stdout:4/890: creat d0/df/d23/d31/d50/d57/d5b/d9c/dfa/f131 x:0 0 0 2024-04-06T14:17:19.009 INFO:tasks.workunit.client.0.smithi032.stdout:5/859: dwrite d16/f50 [0,4194304] 0 2024-04-06T14:17:19.012 INFO:tasks.workunit.client.1.smithi086.stdout:7/697: truncate d7/d10/d11/d21/db0/fd0 239804 0 2024-04-06T14:17:19.012 INFO:tasks.workunit.client.1.smithi086.stdout:6/848: dread d6/d13/d1a/da0/d114/db2/f10c [0,4194304] 0 2024-04-06T14:17:19.012 INFO:tasks.workunit.client.1.smithi086.stdout:2/720: stat d0/d20/d27/c60 0 2024-04-06T14:17:19.013 INFO:tasks.workunit.client.0.smithi032.stdout:9/901: rename d2/d15/d1e/d65/d83/f32 to d2/d15/d1e/d65/d83/f120 0 2024-04-06T14:17:19.013 INFO:tasks.workunit.client.1.smithi086.stdout:9/579: mkdir d5/d13/d1a/d5f/d88/d91/db9 0 2024-04-06T14:17:19.015 INFO:tasks.workunit.client.1.smithi086.stdout:8/763: fsync d0/d71/d16/d52/dd0/ddc/de7/dd6/fd9 0 2024-04-06T14:17:19.016 INFO:tasks.workunit.client.1.smithi086.stdout:3/748: symlink d5/d36/d56/dd3/lf4 0 2024-04-06T14:17:19.018 INFO:tasks.workunit.client.1.smithi086.stdout:6/849: truncate d6/d9/d43/fbc 944719 0 2024-04-06T14:17:19.018 INFO:tasks.workunit.client.0.smithi032.stdout:6/986: rename d4/d5/d19/c124 to d4/d5/d28/d54/c14d 0 2024-04-06T14:17:19.018 INFO:tasks.workunit.client.0.smithi032.stdout:2/834: chown d0/d30/d39/d59/d67/d74/d78/dc2/c112 44210380 1 2024-04-06T14:17:19.018 INFO:tasks.workunit.client.0.smithi032.stdout:4/891: creat d0/df/d23/d31/d50/d57/d5b/d9c/dc1/f132 x:0 0 0 2024-04-06T14:17:19.019 INFO:tasks.workunit.client.1.smithi086.stdout:4/570: link d0/d2c/d38/fa7 d0/d2c/d38/d54/fb1 0 2024-04-06T14:17:19.019 INFO:tasks.workunit.client.1.smithi086.stdout:8/764: read d0/f34 [4209101,35749] 0 2024-04-06T14:17:19.021 INFO:tasks.workunit.client.1.smithi086.stdout:4/571: readlink d0/d2c/d30/d24/d50/l6a 0 2024-04-06T14:17:19.021 INFO:tasks.workunit.client.0.smithi032.stdout:9/902: mknod d2/d15/d1e/d65/d83/d31/d39/c121 0 2024-04-06T14:17:19.023 INFO:tasks.workunit.client.1.smithi086.stdout:6/850: dread d6/d9/d43/fbc [0,4194304] 0 2024-04-06T14:17:19.025 INFO:tasks.workunit.client.0.smithi032.stdout:6/987: getdents d4/d5/dbf/d53/d143 0 2024-04-06T14:17:19.025 INFO:tasks.workunit.client.1.smithi086.stdout:0/781: fdatasync d0/d1/d2/d1a/d79/fb2 0 2024-04-06T14:17:19.027 INFO:tasks.workunit.client.1.smithi086.stdout:7/698: symlink d7/lea 0 2024-04-06T14:17:19.028 INFO:tasks.workunit.client.0.smithi032.stdout:9/903: stat d2/d15/d1e/d65/d83/d31/d39/da6/fb6 0 2024-04-06T14:17:19.028 INFO:tasks.workunit.client.1.smithi086.stdout:1/758: dread db/d11/d42/fbf [0,4194304] 0 2024-04-06T14:17:19.028 INFO:tasks.workunit.client.0.smithi032.stdout:2/835: read d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/faf [371839,4269] 0 2024-04-06T14:17:19.029 INFO:tasks.workunit.client.0.smithi032.stdout:5/860: dwrite d16/d18/d33/d45/d5e/f89 [0,4194304] 0 2024-04-06T14:17:19.031 INFO:tasks.workunit.client.1.smithi086.stdout:8/765: creat d0/d71/d22/d43/d6d/f109 x:0 0 0 2024-04-06T14:17:19.031 INFO:tasks.workunit.client.1.smithi086.stdout:9/580: creat d5/d13/d47/d55/d87/fba x:0 0 0 2024-04-06T14:17:19.036 INFO:tasks.workunit.client.0.smithi032.stdout:4/892: getdents d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/de8 0 2024-04-06T14:17:19.040 INFO:tasks.workunit.client.0.smithi032.stdout:2/836: creat d0/d30/d39/f117 x:0 0 0 2024-04-06T14:17:19.040 INFO:tasks.workunit.client.1.smithi086.stdout:3/749: dwrite d5/d36/d56/dd3/fe0 [0,4194304] 0 2024-04-06T14:17:19.040 INFO:tasks.workunit.client.1.smithi086.stdout:5/597: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:17:19.041 INFO:tasks.workunit.client.1.smithi086.stdout:1/759: chown db/d11/d1a/d2a/d9a/d68/daf/dc2/fcd 4156 1 2024-04-06T14:17:19.041 INFO:tasks.workunit.client.0.smithi032.stdout:6/988: dwrite d4/d5/d28/d65/d6e/ff5 [0,4194304] 0 2024-04-06T14:17:19.042 INFO:tasks.workunit.client.0.smithi032.stdout:5/861: creat d16/d58/d54/d74/f12d x:0 0 0 2024-04-06T14:17:19.042 INFO:tasks.workunit.client.1.smithi086.stdout:7/699: creat d7/d10/d11/d21/feb x:0 0 0 2024-04-06T14:17:19.043 INFO:tasks.workunit.client.0.smithi032.stdout:2/837: chown d0/fb 678392649 1 2024-04-06T14:17:19.043 INFO:tasks.workunit.client.0.smithi032.stdout:5/862: dread - d16/d58/f108 zero size 2024-04-06T14:17:19.045 INFO:tasks.workunit.client.0.smithi032.stdout:2/838: write d0/f22 [836086,125747] 0 2024-04-06T14:17:19.049 INFO:tasks.workunit.client.1.smithi086.stdout:7/700: write d7/d10/d11/d1c/d36/fe9 [371911,14389] 0 2024-04-06T14:17:19.051 INFO:tasks.workunit.client.1.smithi086.stdout:6/851: dread d6/d9/db/f9b [0,4194304] 0 2024-04-06T14:17:19.052 INFO:tasks.workunit.client.1.smithi086.stdout:5/598: write d0/d16/d18/d35/f4d [4298428,128781] 0 2024-04-06T14:17:19.052 INFO:tasks.workunit.client.1.smithi086.stdout:9/581: mkdir d5/d13/d47/d55/d87/dbb 0 2024-04-06T14:17:19.052 INFO:tasks.workunit.client.0.smithi032.stdout:5/863: symlink d16/d18/l12e 0 2024-04-06T14:17:19.054 INFO:tasks.workunit.client.0.smithi032.stdout:6/989: mkdir d4/d5/dbf/d53/d143/d14e 0 2024-04-06T14:17:19.059 INFO:tasks.workunit.client.0.smithi032.stdout:2/839: link d0/d4e/dc6/cdd d0/d30/d39/d59/d67/d74/d78/dc2/c118 0 2024-04-06T14:17:19.059 INFO:tasks.workunit.client.1.smithi086.stdout:6/852: stat d6/d13/d42/l106 0 2024-04-06T14:17:19.060 INFO:tasks.workunit.client.0.smithi032.stdout:9/904: dwrite d2/d15/d1e/d65/d6b/d92/d11d/f85 [4194304,4194304] 0 2024-04-06T14:17:19.061 INFO:tasks.workunit.client.1.smithi086.stdout:0/782: dwrite d0/d1/f76 [0,4194304] 0 2024-04-06T14:17:19.061 INFO:tasks.workunit.client.1.smithi086.stdout:4/572: dwrite d0/de/f62 [0,4194304] 0 2024-04-06T14:17:19.062 INFO:tasks.workunit.client.1.smithi086.stdout:5/599: truncate d0/f41 746194 0 2024-04-06T14:17:19.063 INFO:tasks.workunit.client.0.smithi032.stdout:6/990: truncate d4/d5/dc5/fa1 1262504 0 2024-04-06T14:17:19.063 INFO:tasks.workunit.client.1.smithi086.stdout:3/750: chown d5/d36/c4b 17 1 2024-04-06T14:17:19.064 INFO:tasks.workunit.client.1.smithi086.stdout:7/701: write d7/d10/d11/d1c/d24/d4d/fdb [885033,73382] 0 2024-04-06T14:17:19.064 INFO:tasks.workunit.client.1.smithi086.stdout:2/721: dread d0/d20/d44/dbb/dc4/de3/d76/f91 [0,4194304] 0 2024-04-06T14:17:19.064 INFO:tasks.workunit.client.0.smithi032.stdout:2/840: dread - d0/d30/d39/fed zero size 2024-04-06T14:17:19.065 INFO:tasks.workunit.client.1.smithi086.stdout:9/582: creat d5/d13/d1a/d5f/d88/d91/fbc x:0 0 0 2024-04-06T14:17:19.068 INFO:tasks.workunit.client.1.smithi086.stdout:4/573: chown d0/d15 743016 1 2024-04-06T14:17:19.068 INFO:tasks.workunit.client.0.smithi032.stdout:5/864: dwrite d16/d18/d33/df7/ffb [0,4194304] 0 2024-04-06T14:17:19.071 INFO:tasks.workunit.client.0.smithi032.stdout:9/905: read - d2/d15/d1e/d65/d6b/d92/d11d/d5f/fdf zero size 2024-04-06T14:17:19.076 INFO:tasks.workunit.client.0.smithi032.stdout:6/991: dwrite d4/d5/dbf/d53/f122 [0,4194304] 0 2024-04-06T14:17:19.077 INFO:tasks.workunit.client.1.smithi086.stdout:6/853: mknod d6/d9/db/c121 0 2024-04-06T14:17:19.077 INFO:tasks.workunit.client.1.smithi086.stdout:3/751: chown d5/d36/d65/c6e 9138 1 2024-04-06T14:17:19.078 INFO:tasks.workunit.client.0.smithi032.stdout:5/865: creat d16/d18/d33/d35/f12f x:0 0 0 2024-04-06T14:17:19.079 INFO:tasks.workunit.client.1.smithi086.stdout:4/574: creat d0/d2c/d30/d24/d5d/d8a/fb2 x:0 0 0 2024-04-06T14:17:19.079 INFO:tasks.workunit.client.1.smithi086.stdout:1/760: creat db/d11/d1f/d5c/ff2 x:0 0 0 2024-04-06T14:17:19.079 INFO:tasks.workunit.client.0.smithi032.stdout:2/841: mkdir d0/d30/d39/d59/d67/d74/d78/d119 0 2024-04-06T14:17:19.080 INFO:tasks.workunit.client.1.smithi086.stdout:9/583: symlink d5/d13/d1a/d5f/d88/d14/d19/d25/lbd 0 2024-04-06T14:17:19.081 INFO:tasks.workunit.client.1.smithi086.stdout:8/766: dread d0/d13/f21 [0,4194304] 0 2024-04-06T14:17:19.082 INFO:tasks.workunit.client.0.smithi032.stdout:6/992: rmdir d4/d5/d8/d2b/d77 39 2024-04-06T14:17:19.082 INFO:tasks.workunit.client.1.smithi086.stdout:0/783: write d0/d15/d26/d33/d7a/daa/ff7 [396692,55751] 0 2024-04-06T14:17:19.084 INFO:tasks.workunit.client.1.smithi086.stdout:1/761: truncate db/d11/fc1 743091 0 2024-04-06T14:17:19.085 INFO:tasks.workunit.client.0.smithi032.stdout:9/906: symlink d2/d15/d1e/d65/d83/d7e/l122 0 2024-04-06T14:17:19.087 INFO:tasks.workunit.client.1.smithi086.stdout:1/762: stat db/d11/d1a/d2a/f43 0 2024-04-06T14:17:19.088 INFO:tasks.workunit.client.1.smithi086.stdout:1/763: read - db/d11/d1a/d2a/d9a/d68/d6e/fea zero size 2024-04-06T14:17:19.090 INFO:tasks.workunit.client.1.smithi086.stdout:6/854: mknod d6/d9/db/c122 0 2024-04-06T14:17:19.090 INFO:tasks.workunit.client.1.smithi086.stdout:7/702: rename d7/d10/d11/d80/f46 to d7/d10/d11/d1c/d24/dc8/fec 0 2024-04-06T14:17:19.092 INFO:tasks.workunit.client.1.smithi086.stdout:8/767: stat d0/d5a/fcb 0 2024-04-06T14:17:19.092 INFO:tasks.workunit.client.1.smithi086.stdout:1/764: unlink db/d11/d1a/d2a/d52/c67 0 2024-04-06T14:17:19.093 INFO:tasks.workunit.client.1.smithi086.stdout:6/855: fsync d6/d9/d43/f53 0 2024-04-06T14:17:19.093 INFO:tasks.workunit.client.1.smithi086.stdout:3/752: dread d5/d36/d5c/f68 [0,4194304] 0 2024-04-06T14:17:19.094 INFO:tasks.workunit.client.0.smithi032.stdout:2/842: dwrite d0/db2/dfc/f61 [0,4194304] 0 2024-04-06T14:17:19.094 INFO:tasks.workunit.client.1.smithi086.stdout:9/584: rename d5/d13/d1a/d5f/d88/d14/fa7 to d5/d58/fbe 0 2024-04-06T14:17:19.098 INFO:tasks.workunit.client.1.smithi086.stdout:8/768: creat d0/d71/d22/d43/d6d/d75/d78/f10a x:0 0 0 2024-04-06T14:17:19.101 INFO:tasks.workunit.client.1.smithi086.stdout:4/575: dwrite d0/d15/f37 [0,4194304] 0 2024-04-06T14:17:19.102 INFO:tasks.workunit.client.0.smithi032.stdout:2/843: link d0/d30/d39/d59/d67/fc0 d0/d4e/de9/f11a 0 2024-04-06T14:17:19.103 INFO:tasks.workunit.client.1.smithi086.stdout:0/784: rename d0/d1/d4/c21 to d0/d15/d4e/da3/d73/cff 0 2024-04-06T14:17:19.104 INFO:tasks.workunit.client.1.smithi086.stdout:7/703: mkdir d7/d10/d11/d1c/d24/d4d/d5c/ded 0 2024-04-06T14:17:19.104 INFO:tasks.workunit.client.1.smithi086.stdout:1/765: symlink db/d11/d1a/d2a/d9a/d68/daf/lf3 0 2024-04-06T14:17:19.107 INFO:tasks.workunit.client.1.smithi086.stdout:8/769: symlink d0/db/l10b 0 2024-04-06T14:17:19.108 INFO:tasks.workunit.client.0.smithi032.stdout:2/844: symlink d0/d30/d39/d59/d67/d74/d78/d57/dd3/l11b 0 2024-04-06T14:17:19.108 INFO:tasks.workunit.client.1.smithi086.stdout:9/585: chown d5/d13/d47/d5e/l82 11 1 2024-04-06T14:17:19.113 INFO:tasks.workunit.client.1.smithi086.stdout:0/785: creat d0/d1/d2/d1a/d51/dd7/de2/f100 x:0 0 0 2024-04-06T14:17:19.114 INFO:tasks.workunit.client.1.smithi086.stdout:3/753: rename d5/d36/d56/dd3/fe4 to d5/d36/d56/dd3/ff5 0 2024-04-06T14:17:19.114 INFO:tasks.workunit.client.1.smithi086.stdout:1/766: dwrite db/d11/d1a/d2a/d9a/d78/f9f [0,4194304] 0 2024-04-06T14:17:19.116 INFO:tasks.workunit.client.1.smithi086.stdout:0/786: write d0/d1/d4/de/fb0 [2167010,101490] 0 2024-04-06T14:17:19.117 INFO:tasks.workunit.client.1.smithi086.stdout:4/576: rename d0/c1e to d0/d2c/d30/d24/d71/cb3 0 2024-04-06T14:17:19.119 INFO:tasks.workunit.client.1.smithi086.stdout:4/577: read - d0/d2c/d30/d24/d5d/d8a/f9a zero size 2024-04-06T14:17:19.122 INFO:tasks.workunit.client.1.smithi086.stdout:3/754: write d5/d36/d5a/d6d/d72/d64/fdc [3012408,90211] 0 2024-04-06T14:17:19.126 INFO:tasks.workunit.client.1.smithi086.stdout:6/856: link d6/d9/d43/f53 d6/d13/d1a/da0/d11b/d45/f123 0 2024-04-06T14:17:19.127 INFO:tasks.workunit.client.1.smithi086.stdout:0/787: write d0/d1/d2/d1a/d79/dac/db7/de6/fd5 [1638539,23214] 0 2024-04-06T14:17:19.127 INFO:tasks.workunit.client.0.smithi032.stdout:6/993: dread d4/d5/d19/f102 [0,4194304] 0 2024-04-06T14:17:19.127 INFO:tasks.workunit.client.1.smithi086.stdout:3/755: mkdir d5/d36/d7d/df6 0 2024-04-06T14:17:19.129 INFO:tasks.workunit.client.0.smithi032.stdout:9/907: write d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/d4a/d8f/fc9 [358929,73179] 0 2024-04-06T14:17:19.129 INFO:tasks.workunit.client.1.smithi086.stdout:8/770: creat d0/d71/d22/d43/f10c x:0 0 0 2024-04-06T14:17:19.130 INFO:tasks.workunit.client.1.smithi086.stdout:8/771: fsync d0/d13/db6/fba 0 2024-04-06T14:17:19.131 INFO:tasks.workunit.client.1.smithi086.stdout:9/586: creat d5/d13/d1a/d5f/d88/d14/d19/fbf x:0 0 0 2024-04-06T14:17:19.131 INFO:tasks.workunit.client.0.smithi032.stdout:5/866: dread d16/d58/d54/f60 [0,4194304] 0 2024-04-06T14:17:19.132 INFO:tasks.workunit.client.0.smithi032.stdout:9/908: write d2/d15/d1e/d41/d6c/d74/fc3 [598250,13263] 0 2024-04-06T14:17:19.133 INFO:tasks.workunit.client.1.smithi086.stdout:8/772: chown d0/d3b/fa6 58 1 2024-04-06T14:17:19.135 INFO:tasks.workunit.client.1.smithi086.stdout:0/788: creat d0/d15/d26/d33/d9f/f101 x:0 0 0 2024-04-06T14:17:19.136 INFO:tasks.workunit.client.0.smithi032.stdout:9/909: fdatasync d2/d15/d1e/d65/d83/d31/d39/f58 0 2024-04-06T14:17:19.137 INFO:tasks.workunit.client.1.smithi086.stdout:1/767: getdents db/d13/d55/da3 0 2024-04-06T14:17:19.139 INFO:tasks.workunit.client.1.smithi086.stdout:9/587: readlink d5/lb7 0 2024-04-06T14:17:19.140 INFO:tasks.workunit.client.1.smithi086.stdout:6/857: symlink d6/d13/d1a/l124 0 2024-04-06T14:17:19.142 INFO:tasks.workunit.client.1.smithi086.stdout:9/588: fdatasync d5/d13/d47/d5e/f93 0 2024-04-06T14:17:19.142 INFO:tasks.workunit.client.0.smithi032.stdout:2/845: dwrite d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/f93 [0,4194304] 0 2024-04-06T14:17:19.142 INFO:tasks.workunit.client.1.smithi086.stdout:3/756: symlink d5/dea/dd2/ddb/de6/d9e/lf7 0 2024-04-06T14:17:19.143 INFO:tasks.workunit.client.1.smithi086.stdout:6/858: write d6/d13/d42/feb [1125025,56621] 0 2024-04-06T14:17:19.143 INFO:tasks.workunit.client.0.smithi032.stdout:5/867: write d16/d18/d33/d45/d52/d39/fd5 [564124,113800] 0 2024-04-06T14:17:19.145 INFO:tasks.workunit.client.0.smithi032.stdout:5/868: chown d16/d18/f31 1157385 1 2024-04-06T14:17:19.150 INFO:tasks.workunit.client.0.smithi032.stdout:5/869: symlink d16/d18/d33/d45/d87/l130 0 2024-04-06T14:17:19.150 INFO:tasks.workunit.client.0.smithi032.stdout:2/846: mknod d0/d4e/de9/dea/c11c 0 2024-04-06T14:17:19.151 INFO:tasks.workunit.client.0.smithi032.stdout:9/910: link d2/d15/d1e/d65/d83/d7e/l1c d2/dcd/l123 0 2024-04-06T14:17:19.152 INFO:tasks.workunit.client.0.smithi032.stdout:2/847: fdatasync d0/db2/dfc/f61 0 2024-04-06T14:17:19.155 INFO:tasks.workunit.client.0.smithi032.stdout:9/911: fdatasync d2/f2d 0 2024-04-06T14:17:19.156 INFO:tasks.workunit.client.0.smithi032.stdout:2/848: chown d0/db2/dfc/c49 223652492 1 2024-04-06T14:17:19.156 INFO:tasks.workunit.client.1.smithi086.stdout:7/704: dread d7/d1d/f6b [0,4194304] 0 2024-04-06T14:17:19.158 INFO:tasks.workunit.client.1.smithi086.stdout:6/859: mknod d6/d13/d42/d52/df1/c125 0 2024-04-06T14:17:19.159 INFO:tasks.workunit.client.1.smithi086.stdout:7/705: stat d7/d10/c34 0 2024-04-06T14:17:19.160 INFO:tasks.workunit.client.0.smithi032.stdout:2/849: truncate d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102/f104 1048541 0 2024-04-06T14:17:19.161 INFO:tasks.workunit.client.0.smithi032.stdout:5/870: write d16/d58/d54/d74/d8f/d95/fc2 [591123,96288] 0 2024-04-06T14:17:19.161 INFO:tasks.workunit.client.0.smithi032.stdout:9/912: read d2/d15/d1e/d41/d6c/d74/f105 [83186,81878] 0 2024-04-06T14:17:19.161 INFO:tasks.workunit.client.1.smithi086.stdout:2/722: dwrite d0/df/f84 [0,4194304] 0 2024-04-06T14:17:19.161 INFO:tasks.workunit.client.1.smithi086.stdout:5/600: dwrite d0/d16/d18/d21/d23/f77 [0,4194304] 0 2024-04-06T14:17:19.161 INFO:tasks.workunit.client.1.smithi086.stdout:6/860: readlink d6/d13/d42/lb7 0 2024-04-06T14:17:19.163 INFO:tasks.workunit.client.0.smithi032.stdout:9/913: write d2/d15/d1e/d65/d83/d31/d39/da6/fb6 [7084515,92391] 0 2024-04-06T14:17:19.164 INFO:tasks.workunit.client.1.smithi086.stdout:3/757: creat d5/dea/dd2/ddb/de6/d9e/da5/ff8 x:0 0 0 2024-04-06T14:17:19.164 INFO:tasks.workunit.client.1.smithi086.stdout:7/706: fsync d7/d10/d11/d1c/d24/d4d/daf/db5/d7a/f93 0 2024-04-06T14:17:19.167 INFO:tasks.workunit.client.1.smithi086.stdout:6/861: truncate d6/d13/d1a/fe5 63424 0 2024-04-06T14:17:19.167 INFO:tasks.workunit.client.1.smithi086.stdout:8/773: getdents d0 0 2024-04-06T14:17:19.171 INFO:tasks.workunit.client.1.smithi086.stdout:3/758: chown f4 131 1 2024-04-06T14:17:19.179 INFO:tasks.workunit.client.0.smithi032.stdout:9/914: dwrite d2/d15/d1e/f6d [4194304,4194304] 0 2024-04-06T14:17:19.179 INFO:tasks.workunit.client.0.smithi032.stdout:5/871: dread d16/d58/f6f [0,4194304] 0 2024-04-06T14:17:19.179 INFO:tasks.workunit.client.1.smithi086.stdout:2/723: mkdir d0/d20/d27/d2b/d35/d36/dbe/dfc 0 2024-04-06T14:17:19.179 INFO:tasks.workunit.client.1.smithi086.stdout:5/601: dread d0/d16/d18/d21/d23/d28/d94/d80/d85/f9f [0,4194304] 0 2024-04-06T14:17:19.180 INFO:tasks.workunit.client.0.smithi032.stdout:5/872: fsync d16/d18/f68 0 2024-04-06T14:17:19.185 INFO:tasks.workunit.client.0.smithi032.stdout:9/915: rename d2/d15/d1e/d65/d83/d31/d39/da6/f117 to d2/dcd/d7c/dfa/f124 0 2024-04-06T14:17:19.185 INFO:tasks.workunit.client.1.smithi086.stdout:3/759: symlink d5/d36/d5a/lf9 0 2024-04-06T14:17:19.186 INFO:tasks.workunit.client.1.smithi086.stdout:4/578: write d0/d2c/d30/f3b [1875225,96597] 0 2024-04-06T14:17:19.186 INFO:tasks.workunit.client.0.smithi032.stdout:5/873: creat d16/d18/d33/d45/d52/f131 x:0 0 0 2024-04-06T14:17:19.188 INFO:tasks.workunit.client.1.smithi086.stdout:8/774: mkdir d0/d71/d16/d39/d90/daf/df0/d10d 0 2024-04-06T14:17:19.189 INFO:tasks.workunit.client.0.smithi032.stdout:9/916: fsync d2/d15/d1e/d65/d6b/fb2 0 2024-04-06T14:17:19.189 INFO:tasks.workunit.client.1.smithi086.stdout:1/768: write db/d11/d42/d53/fd4 [1609582,101385] 0 2024-04-06T14:17:19.190 INFO:tasks.workunit.client.1.smithi086.stdout:7/707: creat d7/d10/d11/d1c/d24/fee x:0 0 0 2024-04-06T14:17:19.191 INFO:tasks.workunit.client.1.smithi086.stdout:7/708: fdatasync d7/d10/d11/d80/f4a 0 2024-04-06T14:17:19.192 INFO:tasks.workunit.client.0.smithi032.stdout:9/917: chown d2/d15/d1e/d65/d6b/d7f/cb1 77191881 1 2024-04-06T14:17:19.194 INFO:tasks.workunit.client.0.smithi032.stdout:5/874: symlink d16/d58/d54/d74/d8f/l132 0 2024-04-06T14:17:19.194 INFO:tasks.workunit.client.1.smithi086.stdout:0/789: dwrite d0/d15/d4e/d48/f31 [4194304,4194304] 0 2024-04-06T14:17:19.194 INFO:tasks.workunit.client.1.smithi086.stdout:2/724: link d0/df/ffa d0/d20/d27/d2b/d35/d36/d4a/ffd 0 2024-04-06T14:17:19.195 INFO:tasks.workunit.client.0.smithi032.stdout:5/875: chown d16/d18/d33/d35/d88/df8 6897 1 2024-04-06T14:17:19.195 INFO:tasks.workunit.client.0.smithi032.stdout:9/918: creat d2/d15/d1e/d65/d83/d31/d39/da6/d10a/f125 x:0 0 0 2024-04-06T14:17:19.196 INFO:tasks.workunit.client.1.smithi086.stdout:9/589: write d5/d13/d1a/d5f/d88/d14/d19/d25/d59/f60 [2734614,25996] 0 2024-04-06T14:17:19.196 INFO:tasks.workunit.client.0.smithi032.stdout:5/876: chown d16/d18/d33/d45/d52/d39/fd5 8181 1 2024-04-06T14:17:19.197 INFO:tasks.workunit.client.1.smithi086.stdout:6/862: link d6/d9/d43/cc3 d6/d13/d1a/da0/d11b/d45/da6/c126 0 2024-04-06T14:17:19.197 INFO:tasks.workunit.client.0.smithi032.stdout:9/919: readlink d2/d15/d1e/d65/d83/l3c 0 2024-04-06T14:17:19.199 INFO:tasks.workunit.client.1.smithi086.stdout:8/775: truncate d0/d71/d22/d43/d4d/fc4 28422 0 2024-04-06T14:17:19.201 INFO:tasks.workunit.client.0.smithi032.stdout:9/920: mknod d2/d15/d1e/d65/d83/d7e/da7/c126 0 2024-04-06T14:17:19.203 INFO:tasks.workunit.client.1.smithi086.stdout:3/760: mknod d5/d36/d5a/d6d/d72/cfa 0 2024-04-06T14:17:19.204 INFO:tasks.workunit.client.0.smithi032.stdout:9/921: mknod d2/d15/d1e/c127 0 2024-04-06T14:17:19.206 INFO:tasks.workunit.client.1.smithi086.stdout:4/579: creat d0/fb4 x:0 0 0 2024-04-06T14:17:19.207 INFO:tasks.workunit.client.0.smithi032.stdout:9/922: mkdir d2/d15/d1e/d41/d6c/d74/d104/d128 0 2024-04-06T14:17:19.210 INFO:tasks.workunit.client.0.smithi032.stdout:5/877: truncate d16/d18/d33/df7/ffb 1909529 0 2024-04-06T14:17:19.211 INFO:tasks.workunit.client.0.smithi032.stdout:9/923: symlink d2/d15/d1e/d41/d6c/d74/d104/l129 0 2024-04-06T14:17:19.211 INFO:tasks.workunit.client.1.smithi086.stdout:5/602: dwrite d0/d16/d18/d35/fa2 [0,4194304] 0 2024-04-06T14:17:19.212 INFO:tasks.workunit.client.1.smithi086.stdout:5/603: dread - d0/d16/f95 zero size 2024-04-06T14:17:19.212 INFO:tasks.workunit.client.0.smithi032.stdout:9/924: chown d2/d15/d1e/d65/d83/d7e/f37 322459 1 2024-04-06T14:17:19.212 INFO:tasks.workunit.client.1.smithi086.stdout:3/761: symlink d5/d36/d5a/d6d/d72/d97/db0/lfb 0 2024-04-06T14:17:19.213 INFO:tasks.workunit.client.0.smithi032.stdout:5/878: creat d16/d18/d33/d45/f133 x:0 0 0 2024-04-06T14:17:19.215 INFO:tasks.workunit.client.0.smithi032.stdout:9/925: write d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/fab [3778502,117564] 0 2024-04-06T14:17:19.216 INFO:tasks.workunit.client.1.smithi086.stdout:6/863: link d6/d13/d1a/da0/d114/db2/lf3 d6/d13/d1a/da0/d11b/d45/d8a/l127 0 2024-04-06T14:17:19.217 INFO:tasks.workunit.client.0.smithi032.stdout:5/879: dread - d16/d18/d33/d45/d52/d39/d86/f98 zero size 2024-04-06T14:17:19.218 INFO:tasks.workunit.client.1.smithi086.stdout:1/769: link db/d11/d1a/d2a/d9a/d68/d6e/d81/cc3 db/d11/d1a/d2a/d9a/cf4 0 2024-04-06T14:17:19.218 INFO:tasks.workunit.client.1.smithi086.stdout:9/590: dread d5/d13/d1a/d5f/d88/d14/d19/d1b/f41 [0,4194304] 0 2024-04-06T14:17:19.218 INFO:tasks.workunit.client.1.smithi086.stdout:6/864: chown d6/d13/d42/l106 26414 1 2024-04-06T14:17:19.220 INFO:tasks.workunit.client.1.smithi086.stdout:4/580: mknod d0/cb5 0 2024-04-06T14:17:19.220 INFO:tasks.workunit.client.1.smithi086.stdout:1/770: dread - db/d11/d87/fdc zero size 2024-04-06T14:17:19.220 INFO:tasks.workunit.client.1.smithi086.stdout:5/604: rmdir d0/d16/d71/d96 39 2024-04-06T14:17:19.221 INFO:tasks.workunit.client.1.smithi086.stdout:2/725: write d0/fd [5307963,69443] 0 2024-04-06T14:17:19.222 INFO:tasks.workunit.client.0.smithi032.stdout:5/880: creat d16/d18/d33/d45/d52/f134 x:0 0 0 2024-04-06T14:17:19.222 INFO:tasks.workunit.client.1.smithi086.stdout:3/762: mkdir d5/d36/d56/dd3/dfc 0 2024-04-06T14:17:19.222 INFO:tasks.workunit.client.0.smithi032.stdout:5/881: chown d16/d58/f6d 892 1 2024-04-06T14:17:19.223 INFO:tasks.workunit.client.1.smithi086.stdout:2/726: write d0/df/f84 [337953,63082] 0 2024-04-06T14:17:19.225 INFO:tasks.workunit.client.1.smithi086.stdout:9/591: creat d5/d13/d1a/d5f/d88/d91/db9/fc0 x:0 0 0 2024-04-06T14:17:19.225 INFO:tasks.workunit.client.1.smithi086.stdout:2/727: chown d0/d20/d27/d2b/d35/d36/d4d/lca 60227 1 2024-04-06T14:17:19.229 INFO:tasks.workunit.client.1.smithi086.stdout:3/763: symlink d5/d33/d5d/lfd 0 2024-04-06T14:17:19.230 INFO:tasks.workunit.client.0.smithi032.stdout:9/926: write d2/d15/d1e/d65/d6b/fcf [5092,32606] 0 2024-04-06T14:17:19.232 INFO:tasks.workunit.client.1.smithi086.stdout:3/764: write d5/dea/dd2/ddb/de6/d9e/da5/ff8 [181707,33265] 0 2024-04-06T14:17:19.232 INFO:tasks.workunit.client.1.smithi086.stdout:6/865: creat d6/d9/db/dff/f128 x:0 0 0 2024-04-06T14:17:19.233 INFO:tasks.workunit.client.1.smithi086.stdout:8/776: truncate d0/d3b/db9/fbd 596797 0 2024-04-06T14:17:19.236 INFO:tasks.workunit.client.1.smithi086.stdout:2/728: creat d0/d20/d44/dbb/dc4/ffe x:0 0 0 2024-04-06T14:17:19.238 INFO:tasks.workunit.client.1.smithi086.stdout:6/866: creat d6/d13/d1a/da0/f129 x:0 0 0 2024-04-06T14:17:19.238 INFO:tasks.workunit.client.0.smithi032.stdout:9/927: rename d2/d15/d1e/d65/d6b/d92/d11d/f85 to d2/dcd/f12a 0 2024-04-06T14:17:19.238 INFO:tasks.workunit.client.0.smithi032.stdout:5/882: dwrite d16/d18/dae/dbb/fce [0,4194304] 0 2024-04-06T14:17:19.239 INFO:tasks.workunit.client.1.smithi086.stdout:1/771: creat db/d11/d1a/d2a/d9a/ff5 x:0 0 0 2024-04-06T14:17:19.239 INFO:tasks.workunit.client.1.smithi086.stdout:6/867: chown d6/d13/d1a/da0/d114 3 1 2024-04-06T14:17:19.240 INFO:tasks.workunit.client.1.smithi086.stdout:8/777: creat d0/d13/db6/f10e x:0 0 0 2024-04-06T14:17:19.241 INFO:tasks.workunit.client.1.smithi086.stdout:9/592: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/fc1 x:0 0 0 2024-04-06T14:17:19.242 INFO:tasks.workunit.client.1.smithi086.stdout:5/605: dread d0/d16/f4e [0,4194304] 0 2024-04-06T14:17:19.243 INFO:tasks.workunit.client.1.smithi086.stdout:2/729: creat d0/df/d5e/d9d/fff x:0 0 0 2024-04-06T14:17:19.244 INFO:tasks.workunit.client.0.smithi032.stdout:5/883: getdents d16/d18/dae/dbb/ddc 0 2024-04-06T14:17:19.245 INFO:tasks.workunit.client.0.smithi032.stdout:5/884: dread - d16/d18/d33/d45/d52/fa2 zero size 2024-04-06T14:17:19.247 INFO:tasks.workunit.client.1.smithi086.stdout:8/778: unlink d0/c1e 0 2024-04-06T14:17:19.249 INFO:tasks.workunit.client.1.smithi086.stdout:9/593: readlink d5/l6 0 2024-04-06T14:17:19.250 INFO:tasks.workunit.client.0.smithi032.stdout:5/885: fdatasync d16/d18/f103 0 2024-04-06T14:17:19.251 INFO:tasks.workunit.client.1.smithi086.stdout:5/606: creat d0/d16/d18/d2a/d50/fc6 x:0 0 0 2024-04-06T14:17:19.252 INFO:tasks.workunit.client.0.smithi032.stdout:5/886: dread - d16/d18/d33/d45/f85 zero size 2024-04-06T14:17:19.254 INFO:tasks.workunit.client.1.smithi086.stdout:9/594: fdatasync d5/d13/d1a/d5f/d88/d14/d19/d1b/f9d 0 2024-04-06T14:17:19.254 INFO:tasks.workunit.client.1.smithi086.stdout:3/765: dread d5/d36/d65/f7b [0,4194304] 0 2024-04-06T14:17:19.256 INFO:tasks.workunit.client.1.smithi086.stdout:6/868: dread d6/d13/d1a/d64/fb3 [0,4194304] 0 2024-04-06T14:17:19.257 INFO:tasks.workunit.client.1.smithi086.stdout:9/595: mkdir d5/d13/d47/d5e/dc2 0 2024-04-06T14:17:19.258 INFO:tasks.workunit.client.1.smithi086.stdout:3/766: fsync d5/d36/d65/f93 0 2024-04-06T14:17:19.260 INFO:tasks.workunit.client.1.smithi086.stdout:2/730: dread d0/d20/d27/d2b/d35/d36/f55 [0,4194304] 0 2024-04-06T14:17:19.260 INFO:tasks.workunit.client.1.smithi086.stdout:1/772: dread db/d13/f86 [0,4194304] 0 2024-04-06T14:17:19.262 INFO:tasks.workunit.client.1.smithi086.stdout:6/869: creat d6/d13/d1a/d64/de2/f12a x:0 0 0 2024-04-06T14:17:19.263 INFO:tasks.workunit.client.1.smithi086.stdout:5/607: dread d0/f3f [0,4194304] 0 2024-04-06T14:17:19.263 INFO:tasks.workunit.client.1.smithi086.stdout:2/731: dread - d0/d20/d27/d2b/d35/d36/d4a/db8/fc1 zero size 2024-04-06T14:17:19.265 INFO:tasks.workunit.client.1.smithi086.stdout:1/773: dread db/d11/d1a/f63 [0,4194304] 0 2024-04-06T14:17:19.266 INFO:tasks.workunit.client.1.smithi086.stdout:6/870: write d6/d13/d1a/da0/d11b/d45/fa1 [179926,79448] 0 2024-04-06T14:17:19.266 INFO:tasks.workunit.client.1.smithi086.stdout:3/767: creat d5/d36/d5a/d6d/d72/d57/ffe x:0 0 0 2024-04-06T14:17:19.267 INFO:tasks.workunit.client.1.smithi086.stdout:1/774: dread - db/d11/d87/faa zero size 2024-04-06T14:17:19.270 INFO:tasks.workunit.client.1.smithi086.stdout:9/596: dread d5/d13/d1a/d5f/d88/d14/d19/f4b [0,4194304] 0 2024-04-06T14:17:19.270 INFO:tasks.workunit.client.1.smithi086.stdout:3/768: write d5/d36/d65/f93 [950779,20990] 0 2024-04-06T14:17:19.270 INFO:tasks.workunit.client.1.smithi086.stdout:2/732: mkdir d0/df/dbc/d100 0 2024-04-06T14:17:19.271 INFO:tasks.workunit.client.1.smithi086.stdout:6/871: mkdir d6/d9/db/d12b 0 2024-04-06T14:17:19.273 INFO:tasks.workunit.client.1.smithi086.stdout:5/608: symlink d0/d16/d18/d2a/lc7 0 2024-04-06T14:17:19.273 INFO:tasks.workunit.client.1.smithi086.stdout:9/597: readlink d5/d13/d1a/d5f/l79 0 2024-04-06T14:17:19.274 INFO:tasks.workunit.client.0.smithi032.stdout:5/887: dread d16/d58/d54/d94/db2/fb8 [0,4194304] 0 2024-04-06T14:17:19.274 INFO:tasks.workunit.client.1.smithi086.stdout:6/872: write d6/d13/d1a/da0/d11b/d45/d8a/fef [975152,50953] 0 2024-04-06T14:17:19.275 INFO:tasks.workunit.client.1.smithi086.stdout:2/733: mkdir d0/df/d48/dce/d101 0 2024-04-06T14:17:19.276 INFO:tasks.workunit.client.1.smithi086.stdout:3/769: rmdir d5/d36/d5a/d6d/d72/d97/db0 39 2024-04-06T14:17:19.276 INFO:tasks.workunit.client.1.smithi086.stdout:2/734: chown d0/df/d5e/d9d 378 1 2024-04-06T14:17:19.277 INFO:tasks.workunit.client.0.smithi032.stdout:5/888: mkdir d16/d135 0 2024-04-06T14:17:19.277 INFO:tasks.workunit.client.1.smithi086.stdout:6/873: write d6/d9/db/fd2 [2192188,84363] 0 2024-04-06T14:17:19.281 INFO:tasks.workunit.client.1.smithi086.stdout:8/779: write d0/d71/f3c [4275838,34172] 0 2024-04-06T14:17:19.283 INFO:tasks.workunit.client.1.smithi086.stdout:2/735: unlink d0/d20/d27/d37/d52/d9c/fd6 0 2024-04-06T14:17:19.285 INFO:tasks.workunit.client.1.smithi086.stdout:3/770: dread d5/d36/d56/dd3/fe0 [0,4194304] 0 2024-04-06T14:17:19.285 INFO:tasks.workunit.client.1.smithi086.stdout:6/874: truncate d6/d9/db/f2f 520005 0 2024-04-06T14:17:19.286 INFO:tasks.workunit.client.1.smithi086.stdout:1/775: write db/d11/d1a/d2a/d9a/d68/d6e/d81/fab [65181,8893] 0 2024-04-06T14:17:19.288 INFO:tasks.workunit.client.1.smithi086.stdout:6/875: write d6/d9/da9/fec [829663,63820] 0 2024-04-06T14:17:19.289 INFO:tasks.workunit.client.1.smithi086.stdout:2/736: rmdir d0/d20/d44/dbb/dc4/de3 39 2024-04-06T14:17:19.289 INFO:tasks.workunit.client.1.smithi086.stdout:3/771: symlink d5/d36/d5c/lff 0 2024-04-06T14:17:19.290 INFO:tasks.workunit.client.1.smithi086.stdout:2/737: readlink d0/d20/d27/le8 0 2024-04-06T14:17:19.290 INFO:tasks.workunit.client.1.smithi086.stdout:6/876: dread - d6/d9/d43/fc5 zero size 2024-04-06T14:17:19.294 INFO:tasks.workunit.client.1.smithi086.stdout:2/738: fdatasync d0/d20/d27/d2b/d35/d36/d4d/d59/fad 0 2024-04-06T14:17:19.295 INFO:tasks.workunit.client.1.smithi086.stdout:9/598: dread d5/d13/d1a/d5f/d88/d14/d19/d25/d59/f60 [0,4194304] 0 2024-04-06T14:17:19.295 INFO:tasks.workunit.client.1.smithi086.stdout:1/776: rename db/d11/d1f/fc5 to db/d11/d1f/d5c/ff6 0 2024-04-06T14:17:19.296 INFO:tasks.workunit.client.1.smithi086.stdout:5/609: truncate d0/d16/d18/d21/d23/d28/d94/d52/d7c/f8f 1090704 0 2024-04-06T14:17:19.297 INFO:tasks.workunit.client.1.smithi086.stdout:6/877: dread d6/d13/d42/d52/df1/ff2 [0,4194304] 0 2024-04-06T14:17:19.300 INFO:tasks.workunit.client.1.smithi086.stdout:8/780: rename d0/d71/d22/d43/d6d/d75/db2/lea to d0/d5a/l10f 0 2024-04-06T14:17:19.301 INFO:tasks.workunit.client.1.smithi086.stdout:5/610: write d0/d16/d18/f7f [3302805,60579] 0 2024-04-06T14:17:19.304 INFO:tasks.workunit.client.1.smithi086.stdout:9/599: rmdir d5/d13/d1a/d5f/d88/d14/d19/d25/d59 39 2024-04-06T14:17:19.304 INFO:tasks.workunit.client.1.smithi086.stdout:3/772: dwrite d5/d33/d5d/fa2 [0,4194304] 0 2024-04-06T14:17:19.305 INFO:tasks.workunit.client.1.smithi086.stdout:3/773: chown d5/d36/d56 11800 1 2024-04-06T14:17:19.306 INFO:tasks.workunit.client.1.smithi086.stdout:3/774: read d5/d36/d65/f7b [2725929,85286] 0 2024-04-06T14:17:19.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.325+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f62b4111a00 con 0x7f62a006aec0 2024-04-06T14:17:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.327+0000 7f62a9ffb700 1 -- 172.21.15.32:0/180021393 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f62b4111a00 con 0x7f62a006aec0 2024-04-06T14:17:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f62a006aec0 msgr2=0x7f62a006d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f62a006aec0 0x7f62a006d370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f62a40135d0 tx=0x7f62a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4066280 msgr2=0x7f62b410fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4066280 0x7f62b410fac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f62b000eb40 tx=0x7f62b000b040 comp rx=0 tx=0).stop 2024-04-06T14:17:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 shutdown_connections 2024-04-06T14:17:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f62a006aec0 0x7f62a006d370 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b4066280 0x7f62b410fac0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 --2- 172.21.15.32:0/180021393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b4102360 0x7f62b4110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 >> 172.21.15.32:0/180021393 conn(0x7f62b40f7430 msgr2=0x7f62b40f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 shutdown_connections 2024-04-06T14:17:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.329+0000 7f62bb8ac700 1 -- 172.21.15.32:0/180021393 wait complete. 2024-04-06T14:17:19.336 INFO:tasks.workunit.client.0.smithi032.stdout:4/893: sync 2024-04-06T14:17:19.336 INFO:tasks.workunit.client.0.smithi032.stdout:0/742: sync 2024-04-06T14:17:19.337 INFO:tasks.workunit.client.0.smithi032.stdout:3/923: sync 2024-04-06T14:17:19.338 INFO:tasks.workunit.client.0.smithi032.stdout:4/894: chown d0/df/d2b/dbe/d116/led 21360 1 2024-04-06T14:17:19.339 INFO:tasks.workunit.client.0.smithi032.stdout:4/895: chown d0/d82/d8e/fde 45818695 1 2024-04-06T14:17:19.340 INFO:tasks.workunit.client.0.smithi032.stdout:0/743: mknod d3/da/d25/d2b/d80/db3/dcd/cf9 0 2024-04-06T14:17:19.340 INFO:tasks.workunit.client.0.smithi032.stdout:0/744: chown d3/l74 18 1 2024-04-06T14:17:19.342 INFO:tasks.workunit.client.0.smithi032.stdout:4/896: mkdir d0/df/d23/d31/d50/d57/d5b/d133 0 2024-04-06T14:17:19.343 INFO:tasks.workunit.client.0.smithi032.stdout:0/745: mknod d3/da/d25/d2b/d80/db3/dcd/cfa 0 2024-04-06T14:17:19.344 INFO:tasks.workunit.client.0.smithi032.stdout:3/924: dread d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/f57 [0,4194304] 0 2024-04-06T14:17:19.344 INFO:tasks.workunit.client.0.smithi032.stdout:4/897: creat d0/df/d23/d31/d50/d57/d8a/db5/f134 x:0 0 0 2024-04-06T14:17:19.345 INFO:tasks.workunit.client.0.smithi032.stdout:0/746: mkdir d3/da/d25/d2b/d4a/d79/da8/dfb 0 2024-04-06T14:17:19.349 INFO:tasks.workunit.client.0.smithi032.stdout:0/747: stat d3/da/d25/d2b/d4a/d79/d97/de2/ff4 0 2024-04-06T14:17:19.351 INFO:tasks.workunit.client.0.smithi032.stdout:3/925: fsync d3/d16/ddd/df7/d147/d7d/d9c/dad/d111/f123 0 2024-04-06T14:17:19.354 INFO:tasks.workunit.client.0.smithi032.stdout:3/926: fsync d3/fc1 0 2024-04-06T14:17:19.358 INFO:tasks.workunit.client.0.smithi032.stdout:0/748: getdents d3/da/d25/d2b/d4a/da1 0 2024-04-06T14:17:19.362 INFO:tasks.workunit.client.1.smithi086.stdout:2/739: rename d0/df/dbc to d0/df/d5e/d9d/d102 0 2024-04-06T14:17:19.362 INFO:tasks.workunit.client.0.smithi032.stdout:4/898: dwrite d0/df/d23/fac [0,4194304] 0 2024-04-06T14:17:19.364 INFO:tasks.workunit.client.0.smithi032.stdout:4/899: chown d0/df/d76/c107 1829673692 1 2024-04-06T14:17:19.365 INFO:tasks.workunit.client.0.smithi032.stdout:3/927: truncate d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/f40 1310337 0 2024-04-06T14:17:19.367 INFO:tasks.workunit.client.1.smithi086.stdout:9/600: write d5/d13/f4f [2582012,23712] 0 2024-04-06T14:17:19.368 INFO:tasks.workunit.client.0.smithi032.stdout:3/928: rmdir d3/d12e 39 2024-04-06T14:17:19.369 INFO:tasks.workunit.client.1.smithi086.stdout:6/878: creat d6/d9/f12c x:0 0 0 2024-04-06T14:17:19.370 INFO:tasks.workunit.client.0.smithi032.stdout:4/900: creat d0/df/d23/f135 x:0 0 0 2024-04-06T14:17:19.371 INFO:tasks.workunit.client.1.smithi086.stdout:8/781: rename d0/d71/d82/cc9 to d0/d13/db6/c110 0 2024-04-06T14:17:19.372 INFO:tasks.workunit.client.0.smithi032.stdout:0/749: dwrite d3/da/d25/d2b/d4a/da1/fb2 [0,4194304] 0 2024-04-06T14:17:19.376 INFO:tasks.workunit.client.0.smithi032.stdout:4/901: mknod d0/df/d1f/d42/d45/c136 0 2024-04-06T14:17:19.377 INFO:tasks.workunit.client.1.smithi086.stdout:1/777: dwrite db/d11/d1a/d2a/f9c [4194304,4194304] 0 2024-04-06T14:17:19.378 INFO:tasks.workunit.client.0.smithi032.stdout:0/750: symlink d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/d9c/dcc/lfc 0 2024-04-06T14:17:19.379 INFO:tasks.workunit.client.1.smithi086.stdout:2/740: truncate d0/d20/d27/d2b/d35/d36/d4d/d6f/f77 2914914 0 2024-04-06T14:17:19.383 INFO:tasks.workunit.client.1.smithi086.stdout:8/782: fdatasync d0/d13/f7c 0 2024-04-06T14:17:19.383 INFO:tasks.workunit.client.1.smithi086.stdout:9/601: dread - d5/d13/f8c zero size 2024-04-06T14:17:19.384 INFO:tasks.workunit.client.1.smithi086.stdout:3/775: link d5/d26/d7f/d59/fc5 d5/dea/db4/f100 0 2024-04-06T14:17:19.387 INFO:tasks.workunit.client.1.smithi086.stdout:6/879: mkdir d6/d13/d1a/da0/d11b/d45/d12d 0 2024-04-06T14:17:19.388 INFO:tasks.workunit.client.0.smithi032.stdout:4/902: creat d0/df/d23/d31/d50/d57/d8a/f137 x:0 0 0 2024-04-06T14:17:19.388 INFO:tasks.workunit.client.0.smithi032.stdout:0/751: rename d3/d83/dab to d3/d83/dfd 0 2024-04-06T14:17:19.388 INFO:tasks.workunit.client.0.smithi032.stdout:3/929: write d3/d16/f2a [880690,2471] 0 2024-04-06T14:17:19.389 INFO:tasks.workunit.client.1.smithi086.stdout:3/776: write d5/dea/db4/fd8 [73839,126197] 0 2024-04-06T14:17:19.389 INFO:tasks.workunit.client.1.smithi086.stdout:5/611: dwrite d0/f3f [0,4194304] 0 2024-04-06T14:17:19.390 INFO:tasks.workunit.client.1.smithi086.stdout:3/777: chown d5/d26/d7f/de7 35 1 2024-04-06T14:17:19.391 INFO:tasks.workunit.client.1.smithi086.stdout:5/612: chown d0/d16/d18/d21/d23/d28/d94/f4f 1 1 2024-04-06T14:17:19.392 INFO:tasks.workunit.client.0.smithi032.stdout:4/903: write d0/df/d23/d31/d50/d57/d8a/f137 [874011,52687] 0 2024-04-06T14:17:19.393 INFO:tasks.workunit.client.1.smithi086.stdout:1/778: mknod db/d11/d42/cf7 0 2024-04-06T14:17:19.395 INFO:tasks.workunit.client.1.smithi086.stdout:1/779: write db/d11/d1a/d2a/d9a/d78/fe7 [412406,18143] 0 2024-04-06T14:17:19.398 INFO:tasks.workunit.client.1.smithi086.stdout:3/778: symlink d5/d36/d5c/d8e/dc1/l101 0 2024-04-06T14:17:19.399 INFO:tasks.workunit.client.1.smithi086.stdout:5/613: mknod d0/d16/d18/d21/d23/d28/d94/cc8 0 2024-04-06T14:17:19.400 INFO:tasks.workunit.client.1.smithi086.stdout:8/783: rmdir d0/d3b/d79/dc6 39 2024-04-06T14:17:19.401 INFO:tasks.workunit.client.0.smithi032.stdout:0/752: mkdir d3/da/d25/d2b/d4a/d79/da8/dfb/dfe 0 2024-04-06T14:17:19.401 INFO:tasks.workunit.client.1.smithi086.stdout:8/784: chown d0/d71/d93/cd5 466 1 2024-04-06T14:17:19.402 INFO:tasks.workunit.client.0.smithi032.stdout:4/904: rmdir d0/df/d1f/d42 39 2024-04-06T14:17:19.404 INFO:tasks.workunit.client.0.smithi032.stdout:3/930: dwrite d3/faf [0,4194304] 0 2024-04-06T14:17:19.406 INFO:tasks.workunit.client.0.smithi032.stdout:3/931: read d3/d16/f6f [26634,33950] 0 2024-04-06T14:17:19.407 INFO:tasks.workunit.client.0.smithi032.stdout:4/905: mkdir d0/df/d23/d31/d50/d5e/da3/d138 0 2024-04-06T14:17:19.412 INFO:tasks.workunit.client.0.smithi032.stdout:0/753: link d3/da/d25/d57/c95 d3/da/d25/d2b/d80/db3/cff 0 2024-04-06T14:17:19.419 INFO:tasks.workunit.client.0.smithi032.stdout:0/754: rename d3/da/d25/d57/db5/dc8 to d3/da/d25/d2b/d80/d100 0 2024-04-06T14:17:19.419 INFO:tasks.workunit.client.0.smithi032.stdout:0/755: fsync d3/da/f53 0 2024-04-06T14:17:19.423 INFO:tasks.workunit.client.0.smithi032.stdout:4/906: write d0/df/d23/d31/d50/d57/d5b/fc9 [1930381,101803] 0 2024-04-06T14:17:19.424 INFO:tasks.workunit.client.0.smithi032.stdout:0/756: getdents d3/da/d25/d2b/d4a/d86/dd6 0 2024-04-06T14:17:19.428 INFO:tasks.workunit.client.0.smithi032.stdout:4/907: rmdir d0/df/d2b/dbe/d116/df0 0 2024-04-06T14:17:19.430 INFO:tasks.workunit.client.0.smithi032.stdout:4/908: dread d0/df/d2b/d43/fa6 [0,4194304] 0 2024-04-06T14:17:19.432 INFO:tasks.workunit.client.0.smithi032.stdout:0/757: write d3/da/d25/d2b/d4a/d79/d97/de0/df5/f8a [509459,5701] 0 2024-04-06T14:17:19.433 INFO:tasks.workunit.client.0.smithi032.stdout:4/909: dread - d0/df/d23/d31/d50/d57/d5b/d9c/dfa/f131 zero size 2024-04-06T14:17:19.433 INFO:tasks.workunit.client.0.smithi032.stdout:0/758: chown d3/da/d25/d2b/f3d 725201 1 2024-04-06T14:17:19.437 INFO:tasks.workunit.client.0.smithi032.stdout:3/932: dread d3/d16/ddd/ffa [0,4194304] 0 2024-04-06T14:17:19.439 INFO:tasks.workunit.client.0.smithi032.stdout:0/759: rename d3/da/d25/d2b/d4a/da1 to d3/da/d25/d101 0 2024-04-06T14:17:19.440 INFO:tasks.workunit.client.0.smithi032.stdout:3/933: symlink d3/dcf/l15e 0 2024-04-06T14:17:19.442 INFO:tasks.workunit.client.0.smithi032.stdout:3/934: unlink d3/l153 0 2024-04-06T14:17:19.444 INFO:tasks.workunit.client.0.smithi032.stdout:0/760: write d3/da/f26 [1543354,115446] 0 2024-04-06T14:17:19.444 INFO:tasks.workunit.client.0.smithi032.stdout:3/935: read d3/d16/f2c [5409,47327] 0 2024-04-06T14:17:19.456 INFO:tasks.workunit.client.0.smithi032.stdout:3/936: write d3/d16/ddd/df7/d147/d7d/d9c/dad/f12b [1793781,109631] 0 2024-04-06T14:17:19.456 INFO:tasks.workunit.client.0.smithi032.stdout:2/850: sync 2024-04-06T14:17:19.456 INFO:tasks.workunit.client.0.smithi032.stdout:6/994: sync 2024-04-06T14:17:19.459 INFO:tasks.workunit.client.0.smithi032.stdout:6/995: read d4/d5/dc5/f50 [146203,94273] 0 2024-04-06T14:17:19.459 INFO:tasks.workunit.client.0.smithi032.stdout:0/761: link d3/da/d25/d36/fbd d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/d9c/f102 0 2024-04-06T14:17:19.461 INFO:tasks.workunit.client.0.smithi032.stdout:2/851: creat d0/d30/d39/f11d x:0 0 0 2024-04-06T14:17:19.461 INFO:tasks.workunit.client.0.smithi032.stdout:3/937: dread d3/d16/ddd/df7/d147/d7d/d9c/dbc/fe3 [0,4194304] 0 2024-04-06T14:17:19.462 INFO:tasks.workunit.client.0.smithi032.stdout:0/762: symlink d3/da/d25/d2b/d4a/d86/dd6/l103 0 2024-04-06T14:17:19.462 INFO:tasks.workunit.client.0.smithi032.stdout:6/996: dread d4/d5/dbf/d51/fec [0,4194304] 0 2024-04-06T14:17:19.464 INFO:tasks.workunit.client.0.smithi032.stdout:3/938: symlink d3/d16/d63/d129/l15f 0 2024-04-06T14:17:19.469 INFO:tasks.workunit.client.0.smithi032.stdout:6/997: getdents d4/d5/d28/d65/df8 0 2024-04-06T14:17:19.469 INFO:tasks.workunit.client.0.smithi032.stdout:6/998: readlink d4/d5/d14/d80/lf0 0 2024-04-06T14:17:19.472 INFO:tasks.workunit.client.0.smithi032.stdout:3/939: getdents d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/dc9 0 2024-04-06T14:17:19.474 INFO:tasks.workunit.client.0.smithi032.stdout:6/999: truncate d4/d5/dbf/d51/f7b 570517 0 2024-04-06T14:17:19.476 INFO:tasks.workunit.client.0.smithi032.stdout:2/852: dwrite d0/d5a/f9d [0,4194304] 0 2024-04-06T14:17:19.478 INFO:tasks.workunit.client.0.smithi032.stdout:3/940: creat d3/d16/ddd/df7/d147/d7d/d9c/dbc/d128/d14e/f160 x:0 0 0 2024-04-06T14:17:19.478 INFO:tasks.workunit.client.0.smithi032.stdout:2/853: dread - d0/d30/d39/d59/d67/d74/d78/dc2/ff5 zero size 2024-04-06T14:17:19.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.481+0000 7fbc5d951700 1 -- 172.21.15.32:0/1080949132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc581002a0 msgr2=0x7fbc58100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:19.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.481+0000 7fbc5d951700 1 --2- 172.21.15.32:0/1080949132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc581002a0 0x7fbc58100710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c004880 tx=0x7fbc4c009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:19.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.481+0000 7fbc5d951700 1 -- 172.21.15.32:0/1080949132 shutdown_connections 2024-04-06T14:17:19.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.481+0000 7fbc5d951700 1 --2- 172.21.15.32:0/1080949132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc581002a0 0x7fbc58100710 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:19.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.481+0000 7fbc5d951700 1 --2- 172.21.15.32:0/1080949132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc580feff0 0x7fbc580ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:19.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.481+0000 7fbc5d951700 1 -- 172.21.15.32:0/1080949132 >> 172.21.15.32:0/1080949132 conn(0x7fbc580fa950 msgr2=0x7fbc580fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:19.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.482+0000 7fbc5d951700 1 -- 172.21.15.32:0/1080949132 shutdown_connections 2024-04-06T14:17:19.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.482+0000 7fbc5d951700 1 -- 172.21.15.32:0/1080949132 wait complete. 2024-04-06T14:17:19.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.482+0000 7fbc5d951700 1 Processor -- start 2024-04-06T14:17:19.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.482+0000 7fbc5d951700 1 -- start start 2024-04-06T14:17:19.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc5d951700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580feff0 0x7fbc5806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:19.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc5d951700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc581002a0 0x7fbc5806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc5d951700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc580700b0 con 0x7fbc581002a0 2024-04-06T14:17:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc5d951700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5806eef0 con 0x7fbc580feff0 2024-04-06T14:17:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc56ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580feff0 0x7fbc5806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc56ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580feff0 0x7fbc5806e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:42288/0 (socket says 172.21.15.32:42288) 2024-04-06T14:17:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc56ffd700 1 -- 172.21.15.32:0/1920893070 learned_addr learned my addr 172.21.15.32:0/1920893070 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc56ffd700 1 -- 172.21.15.32:0/1920893070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc581002a0 msgr2=0x7fbc5806e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc56ffd700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc581002a0 0x7fbc5806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.483+0000 7fbc56ffd700 1 -- 172.21.15.32:0/1920893070 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc48005040 con 0x7fbc580feff0 2024-04-06T14:17:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.484+0000 7fbc56ffd700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580feff0 0x7fbc5806e470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc480096b0 tx=0x7fbc48009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.484+0000 7fbc3f7fe700 1 -- 172.21.15.32:0/1920893070 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc48017070 con 0x7fbc580feff0 2024-04-06T14:17:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.484+0000 7fbc3f7fe700 1 -- 172.21.15.32:0/1920893070 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc4800f040 con 0x7fbc580feff0 2024-04-06T14:17:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.484+0000 7fbc3f7fe700 1 -- 172.21.15.32:0/1920893070 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc48013b70 con 0x7fbc580feff0 2024-04-06T14:17:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.484+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c005040 con 0x7fbc580feff0 2024-04-06T14:17:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.484+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5810cfd0 con 0x7fbc580feff0 2024-04-06T14:17:19.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.485+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5805f5f0 con 0x7fbc580feff0 2024-04-06T14:17:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.485+0000 7fbc3f7fe700 1 -- 172.21.15.32:0/1920893070 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7fbc4800e840 con 0x7fbc580feff0 2024-04-06T14:17:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.486+0000 7fbc3f7fe700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fbc4006aec0 0x7fbc4006d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.486+0000 7fbc3f7fe700 1 -- 172.21.15.32:0/1920893070 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fbc480896f0 con 0x7fbc580feff0 2024-04-06T14:17:19.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.486+0000 7fbc567fc700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fbc4006aec0 0x7fbc4006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:19.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.487+0000 7fbc567fc700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fbc4006aec0 0x7fbc4006d370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c000c00 tx=0x7fbc4c009e30 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:19.488 INFO:tasks.workunit.client.0.smithi032.stdout:2/854: dwrite d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102/f104 [0,4194304] 0 2024-04-06T14:17:19.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.491+0000 7fbc3f7fe700 1 -- 172.21.15.32:0/1920893070 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fbc48053ac0 con 0x7fbc580feff0 2024-04-06T14:17:19.493 INFO:tasks.workunit.client.0.smithi032.stdout:2/855: rename d0/f8a to d0/d30/d39/d59/d67/d74/d78/d57/f11e 0 2024-04-06T14:17:19.498 INFO:tasks.workunit.client.1.smithi086.stdout:3/779: creat d5/dea/dd2/ddb/de6/d9e/db5/f102 x:0 0 0 2024-04-06T14:17:19.498 INFO:tasks.workunit.client.1.smithi086.stdout:5/614: creat d0/d16/d18/d2a/d50/d66/fc9 x:0 0 0 2024-04-06T14:17:19.499 INFO:tasks.workunit.client.1.smithi086.stdout:6/880: rmdir d6/d9/db/d12b 0 2024-04-06T14:17:19.501 INFO:tasks.workunit.client.1.smithi086.stdout:2/741: dwrite d0/df/ffa [0,4194304] 0 2024-04-06T14:17:19.504 INFO:tasks.workunit.client.1.smithi086.stdout:8/785: symlink d0/db/l111 0 2024-04-06T14:17:19.504 INFO:tasks.workunit.client.1.smithi086.stdout:9/602: link d5/d13/d1a/d5f/d88/d14/d19/c72 d5/d13/d1a/d5f/d88/d91/cc3 0 2024-04-06T14:17:19.504 INFO:tasks.workunit.client.1.smithi086.stdout:5/615: fsync d0/d16/d18/d21/d23/d28/d94/d52/fb1 0 2024-04-06T14:17:19.504 INFO:tasks.workunit.client.1.smithi086.stdout:3/780: symlink d5/dea/db4/da7/l103 0 2024-04-06T14:17:19.505 INFO:tasks.workunit.client.1.smithi086.stdout:6/881: mknod d6/d13/d1a/da0/d11b/d45/d8a/de3/c12e 0 2024-04-06T14:17:19.505 INFO:tasks.workunit.client.1.smithi086.stdout:2/742: rename d0/d20 to d0/d20/d27/d2b/d35/d36/d4a/db8/d103 22 2024-04-06T14:17:19.505 INFO:tasks.workunit.client.0.smithi032.stdout:2/856: dread d0/d30/d39/d59/d67/d74/fe8 [0,4194304] 0 2024-04-06T14:17:19.506 INFO:tasks.workunit.client.1.smithi086.stdout:8/786: mknod d0/d71/d93/c112 0 2024-04-06T14:17:19.508 INFO:tasks.workunit.client.1.smithi086.stdout:3/781: truncate d5/d36/d5a/d6d/d72/d57/ffe 198491 0 2024-04-06T14:17:19.509 INFO:tasks.workunit.client.1.smithi086.stdout:9/603: fdatasync d5/d13/d1a/d5f/d88/d14/f22 0 2024-04-06T14:17:19.511 INFO:tasks.workunit.client.1.smithi086.stdout:8/787: creat d0/d71/d22/d43/daa/f113 x:0 0 0 2024-04-06T14:17:19.511 INFO:tasks.workunit.client.1.smithi086.stdout:2/743: creat d0/f104 x:0 0 0 2024-04-06T14:17:19.513 INFO:tasks.workunit.client.0.smithi032.stdout:2/857: getdents d0/d30/d39/d59/d67/d80 0 2024-04-06T14:17:19.515 INFO:tasks.workunit.client.0.smithi032.stdout:2/858: symlink d0/d30/d7f/dd9/l11f 0 2024-04-06T14:17:19.516 INFO:tasks.workunit.client.0.smithi032.stdout:2/859: dread - d0/d5a/dc9/ff4 zero size 2024-04-06T14:17:19.517 INFO:tasks.workunit.client.1.smithi086.stdout:5/616: dwrite d0/d16/d18/d35/f5b [0,4194304] 0 2024-04-06T14:17:19.518 INFO:tasks.workunit.client.1.smithi086.stdout:2/744: rmdir d0/df/d48/d7d 39 2024-04-06T14:17:19.518 INFO:tasks.workunit.client.1.smithi086.stdout:9/604: write d5/d13/d1a/d5f/d88/d14/d19/d1b/f6f [305087,99945] 0 2024-04-06T14:17:19.524 INFO:tasks.workunit.client.1.smithi086.stdout:9/605: readlink d5/d13/d1a/d5f/la3 0 2024-04-06T14:17:19.527 INFO:tasks.workunit.client.1.smithi086.stdout:5/617: creat d0/d16/d18/d2a/d50/d66/fca x:0 0 0 2024-04-06T14:17:19.527 INFO:tasks.workunit.client.1.smithi086.stdout:6/882: dwrite d6/d9/db/f16 [0,4194304] 0 2024-04-06T14:17:19.529 INFO:tasks.workunit.client.1.smithi086.stdout:8/788: dwrite d0/d13/fb3 [0,4194304] 0 2024-04-06T14:17:19.529 INFO:tasks.workunit.client.1.smithi086.stdout:3/782: dwrite d5/d33/f47 [0,4194304] 0 2024-04-06T14:17:19.534 INFO:tasks.workunit.client.1.smithi086.stdout:2/745: creat d0/d20/d27/d2b/d35/d36/f105 x:0 0 0 2024-04-06T14:17:19.538 INFO:tasks.workunit.client.1.smithi086.stdout:9/606: truncate d5/d13/d1a/d5f/d88/d14/d19/d1b/f6f 1796353 0 2024-04-06T14:17:19.540 INFO:tasks.workunit.client.1.smithi086.stdout:3/783: creat d5/d36/d5a/d6d/d72/f104 x:0 0 0 2024-04-06T14:17:19.556 INFO:tasks.workunit.client.0.smithi032.stdout:3/941: dread d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/f40 [0,4194304] 0 2024-04-06T14:17:19.559 INFO:tasks.workunit.client.0.smithi032.stdout:3/942: fdatasync d3/f24 0 2024-04-06T14:17:19.560 INFO:tasks.workunit.client.0.smithi032.stdout:3/943: unlink d3/d16/ddd/df7/d147/d7d/dc6/de5/ff8 0 2024-04-06T14:17:19.560 INFO:tasks.workunit.client.1.smithi086.stdout:7/709: sync 2024-04-06T14:17:19.560 INFO:tasks.workunit.client.0.smithi032.stdout:5/889: sync 2024-04-06T14:17:19.561 INFO:tasks.workunit.client.0.smithi032.stdout:9/928: sync 2024-04-06T14:17:19.561 INFO:tasks.workunit.client.0.smithi032.stdout:3/944: chown d3/d16/ddd/df7/d147/f106 925 1 2024-04-06T14:17:19.563 INFO:tasks.workunit.client.1.smithi086.stdout:8/789: dwrite d0/d71/d16/d39/d90/daf/fe3 [0,4194304] 0 2024-04-06T14:17:19.563 INFO:tasks.workunit.client.0.smithi032.stdout:5/890: write d16/d18/d33/d45/d52/d39/fd5 [1648668,117779] 0 2024-04-06T14:17:19.563 INFO:tasks.workunit.client.1.smithi086.stdout:2/746: creat d0/d20/d27/f106 x:0 0 0 2024-04-06T14:17:19.566 INFO:tasks.workunit.client.1.smithi086.stdout:3/784: creat d5/d36/d56/f105 x:0 0 0 2024-04-06T14:17:19.567 INFO:tasks.workunit.client.1.smithi086.stdout:9/607: dread d5/d13/d47/f83 [0,4194304] 0 2024-04-06T14:17:19.571 INFO:tasks.workunit.client.1.smithi086.stdout:3/785: write d5/dea/dd2/ddb/de6/f45 [3538092,9177] 0 2024-04-06T14:17:19.572 INFO:tasks.workunit.client.1.smithi086.stdout:3/786: chown d5/c12 13412838 1 2024-04-06T14:17:19.574 INFO:tasks.workunit.client.1.smithi086.stdout:3/787: readlink d5/d36/d56/dd3/lf4 0 2024-04-06T14:17:19.576 INFO:tasks.workunit.client.1.smithi086.stdout:2/747: chown d0/d20/d44/dbb/dc4/de3/l9a 41 1 2024-04-06T14:17:19.576 INFO:tasks.workunit.client.1.smithi086.stdout:3/788: stat d5/d33/d5d/f6f 0 2024-04-06T14:17:19.579 INFO:tasks.workunit.client.1.smithi086.stdout:7/710: dwrite d7/d10/d11/d1c/d24/d2c/fb6 [0,4194304] 0 2024-04-06T14:17:19.580 INFO:tasks.workunit.client.1.smithi086.stdout:3/789: read d5/dea/db4/da7/f63 [893673,48408] 0 2024-04-06T14:17:19.583 INFO:tasks.workunit.client.1.smithi086.stdout:7/711: write d7/d10/d11/d1c/f1e [3814067,25399] 0 2024-04-06T14:17:19.586 INFO:tasks.workunit.client.1.smithi086.stdout:5/618: truncate d0/f3 4024783 0 2024-04-06T14:17:19.587 INFO:tasks.workunit.client.0.smithi032.stdout:2/860: dread d0/d30/d39/d59/d67/d74/d78/d8c/db0/fb3 [8388608,4194304] 0 2024-04-06T14:17:19.587 INFO:tasks.workunit.client.1.smithi086.stdout:5/619: chown d0/d16/f39 197440 1 2024-04-06T14:17:19.588 INFO:tasks.workunit.client.1.smithi086.stdout:2/748: getdents d0/d20/d27/d2b/d35 0 2024-04-06T14:17:19.591 INFO:tasks.workunit.client.1.smithi086.stdout:3/790: symlink d5/d33/l106 0 2024-04-06T14:17:19.591 INFO:tasks.workunit.client.1.smithi086.stdout:8/790: write d0/d3b/d79/fc5 [332084,98016] 0 2024-04-06T14:17:19.592 INFO:tasks.workunit.client.1.smithi086.stdout:5/620: creat d0/d43/fcb x:0 0 0 2024-04-06T14:17:19.594 INFO:tasks.workunit.client.1.smithi086.stdout:2/749: rename d0/d20/d44/d9f/fc8 to d0/d20/d44/d9f/f107 0 2024-04-06T14:17:19.597 INFO:tasks.workunit.client.1.smithi086.stdout:7/712: creat d7/fef x:0 0 0 2024-04-06T14:17:19.597 INFO:tasks.workunit.client.1.smithi086.stdout:9/608: dread d5/f20 [0,4194304] 0 2024-04-06T14:17:19.598 INFO:tasks.workunit.client.1.smithi086.stdout:3/791: mknod d5/dea/db4/da7/c107 0 2024-04-06T14:17:19.600 INFO:tasks.workunit.client.1.smithi086.stdout:3/792: readlink d5/d36/d65/l6c 0 2024-04-06T14:17:19.601 INFO:tasks.workunit.client.1.smithi086.stdout:8/791: link d0/d71/d16/d39/f66 d0/d71/d22/d43/d6d/d75/db2/f114 0 2024-04-06T14:17:19.603 INFO:tasks.workunit.client.1.smithi086.stdout:7/713: write d7/d10/d11/d1c/d24/d4d/d5c/d66/f67 [3717128,15502] 0 2024-04-06T14:17:19.608 INFO:tasks.workunit.client.1.smithi086.stdout:7/714: rmdir d7/d10/d11/d1c 39 2024-04-06T14:17:19.610 INFO:tasks.workunit.client.1.smithi086.stdout:3/793: mkdir d5/d36/d5a/d6d/d72/d108 0 2024-04-06T14:17:19.613 INFO:tasks.workunit.client.1.smithi086.stdout:8/792: creat d0/d71/d22/d43/d6d/d75/dee/f115 x:0 0 0 2024-04-06T14:17:19.616 INFO:tasks.workunit.client.1.smithi086.stdout:2/750: link d0/d20/d44/dbb/dc4/de3/d76/d78/f8c d0/d20/d27/d2b/d35/d36/f108 0 2024-04-06T14:17:19.617 INFO:tasks.workunit.client.1.smithi086.stdout:3/794: truncate d5/d26/d7f/f60 6655580 0 2024-04-06T14:17:19.617 INFO:tasks.workunit.client.1.smithi086.stdout:5/621: dread d0/d16/d18/d21/d23/d28/dc2/f3c [0,4194304] 0 2024-04-06T14:17:19.618 INFO:tasks.workunit.client.1.smithi086.stdout:7/715: mkdir d7/d10/d11/d1c/d24/df0 0 2024-04-06T14:17:19.619 INFO:tasks.workunit.client.1.smithi086.stdout:2/751: creat d0/d20/d27/d37/d52/f109 x:0 0 0 2024-04-06T14:17:19.619 INFO:tasks.workunit.client.1.smithi086.stdout:5/622: chown d0/d43/f9c 9 1 2024-04-06T14:17:19.620 INFO:tasks.workunit.client.1.smithi086.stdout:3/795: creat d5/d36/d7d/f109 x:0 0 0 2024-04-06T14:17:19.621 INFO:tasks.workunit.client.1.smithi086.stdout:9/609: dread d5/d13/d1a/d5f/f6a [0,4194304] 0 2024-04-06T14:17:19.623 INFO:tasks.workunit.client.1.smithi086.stdout:5/623: symlink d0/d16/d71/d96/lcc 0 2024-04-06T14:17:19.628 INFO:tasks.workunit.client.1.smithi086.stdout:3/796: creat d5/d36/d5a/d6d/d72/f10a x:0 0 0 2024-04-06T14:17:19.630 INFO:tasks.workunit.client.1.smithi086.stdout:2/752: rename d0/d20/d27/d2b/d35/d36/d4a/ffd to d0/d20/d27/f10a 0 2024-04-06T14:17:19.631 INFO:tasks.workunit.client.0.smithi032.stdout:9/929: rename d2/d15/d1e/d65/d6b/d92/d11d/f6e to d2/dcd/dd8/f12b 0 2024-04-06T14:17:19.631 INFO:tasks.workunit.client.1.smithi086.stdout:9/610: dwrite d5/d13/d1a/d5f/d88/d14/d19/fa6 [0,4194304] 0 2024-04-06T14:17:19.632 INFO:tasks.workunit.client.0.smithi032.stdout:3/945: symlink d3/d16/ddd/df7/d147/d89/dcb/l161 0 2024-04-06T14:17:19.633 INFO:tasks.workunit.client.0.smithi032.stdout:9/930: write d2/d15/d1e/d65/d83/f45 [3771900,17835] 0 2024-04-06T14:17:19.634 INFO:tasks.workunit.client.0.smithi032.stdout:5/891: mkdir d16/d18/d33/d45/d5e/d136 0 2024-04-06T14:17:19.638 INFO:tasks.workunit.client.0.smithi032.stdout:5/892: mknod d16/d18/dae/c137 0 2024-04-06T14:17:19.638 INFO:tasks.workunit.client.0.smithi032.stdout:3/946: mkdir d3/d16/ddd/df7/d147/d7d/d9c/dbc/d13e/d162 0 2024-04-06T14:17:19.640 INFO:tasks.workunit.client.0.smithi032.stdout:2/861: dread d0/d30/d7f/fac [0,4194304] 0 2024-04-06T14:17:19.642 INFO:tasks.workunit.client.1.smithi086.stdout:7/716: write d7/d10/d11/d1c/d24/d2c/f70 [1439537,12069] 0 2024-04-06T14:17:19.643 INFO:tasks.workunit.client.1.smithi086.stdout:8/793: truncate d0/d71/d22/d43/d6d/d75/d7d/fd3 727765 0 2024-04-06T14:17:19.643 INFO:tasks.workunit.client.0.smithi032.stdout:3/947: readlink d3/d16/ddd/df7/d147/d7d/d9c/dad/d111/l141 0 2024-04-06T14:17:19.643 INFO:tasks.workunit.client.0.smithi032.stdout:5/893: chown d16/d18/d33/d45/d87/l9b 99 1 2024-04-06T14:17:19.646 INFO:tasks.workunit.client.1.smithi086.stdout:5/624: write d0/f41 [494849,43948] 0 2024-04-06T14:17:19.647 INFO:tasks.workunit.client.0.smithi032.stdout:9/931: symlink d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/l12c 0 2024-04-06T14:17:19.655 INFO:tasks.workunit.client.1.smithi086.stdout:9/611: truncate d5/d13/d1a/d5f/d88/d14/d19/f5d 3388369 0 2024-04-06T14:17:19.657 INFO:tasks.workunit.client.1.smithi086.stdout:2/753: link d0/d20/d27/d2b/d35/d36/d4d/l57 d0/d20/d82/l10b 0 2024-04-06T14:17:19.658 INFO:tasks.workunit.client.1.smithi086.stdout:7/717: rename d7/d10/d11/d1c/d24/fee to d7/ff1 0 2024-04-06T14:17:19.659 INFO:tasks.workunit.client.1.smithi086.stdout:3/797: dwrite d5/d36/d5c/faa [0,4194304] 0 2024-04-06T14:17:19.662 INFO:tasks.workunit.client.1.smithi086.stdout:2/754: creat d0/d20/d27/d2b/d35/d36/dbe/f10c x:0 0 0 2024-04-06T14:17:19.664 INFO:tasks.workunit.client.1.smithi086.stdout:2/755: fsync d0/d20/d27/f8a 0 2024-04-06T14:17:19.667 INFO:tasks.workunit.client.1.smithi086.stdout:9/612: dwrite d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f97 [0,4194304] 0 2024-04-06T14:17:19.671 INFO:tasks.workunit.client.0.smithi032.stdout:5/894: dread d16/d18/d33/d45/f9f [0,4194304] 0 2024-04-06T14:17:19.671 INFO:tasks.workunit.client.0.smithi032.stdout:3/948: dread d3/d16/ddd/df7/d147/f66 [4194304,4194304] 0 2024-04-06T14:17:19.672 INFO:tasks.workunit.client.1.smithi086.stdout:8/794: truncate d0/d71/d16/d52/dd0/ddc/de7/f3e 2275532 0 2024-04-06T14:17:19.672 INFO:tasks.workunit.client.0.smithi032.stdout:5/895: readlink d16/d58/d54/l7f 0 2024-04-06T14:17:19.672 INFO:tasks.workunit.client.1.smithi086.stdout:7/718: creat d7/d10/d11/d1c/d24/ff2 x:0 0 0 2024-04-06T14:17:19.675 INFO:tasks.workunit.client.0.smithi032.stdout:3/949: creat d3/d16/ddd/d12d/f163 x:0 0 0 2024-04-06T14:17:19.677 INFO:tasks.workunit.client.0.smithi032.stdout:9/932: dread d2/dcd/dd8/f108 [0,4194304] 0 2024-04-06T14:17:19.678 INFO:tasks.workunit.client.0.smithi032.stdout:5/896: creat d16/d18/d33/dea/f138 x:0 0 0 2024-04-06T14:17:19.680 INFO:tasks.workunit.client.0.smithi032.stdout:3/950: symlink d3/d14d/d12c/l164 0 2024-04-06T14:17:19.680 INFO:tasks.workunit.client.1.smithi086.stdout:3/798: rename d5/d26/d7f/d59 to d5/dea/dd2/ddb/de6/d10b 0 2024-04-06T14:17:19.681 INFO:tasks.workunit.client.1.smithi086.stdout:3/799: chown d5/d36/d5a/d6d/d72/d64/lc4 2082130 1 2024-04-06T14:17:19.681 INFO:tasks.workunit.client.1.smithi086.stdout:8/795: symlink d0/d71/d22/d43/d4d/d62/l116 0 2024-04-06T14:17:19.681 INFO:tasks.workunit.client.1.smithi086.stdout:2/756: creat d0/d20/d27/dd1/f10d x:0 0 0 2024-04-06T14:17:19.681 INFO:tasks.workunit.client.1.smithi086.stdout:3/800: creat d5/dea/dd2/ddb/f10c x:0 0 0 2024-04-06T14:17:19.684 INFO:tasks.workunit.client.0.smithi032.stdout:5/897: unlink d16/d18/d33/d35/d88/deb/ff2 0 2024-04-06T14:17:19.684 INFO:tasks.workunit.client.1.smithi086.stdout:3/801: rename d5 to d5/d36/d56/dd3/d10d 22 2024-04-06T14:17:19.685 INFO:tasks.workunit.client.1.smithi086.stdout:7/719: creat d7/d10/d11/d1c/d24/d4d/daf/db5/ff3 x:0 0 0 2024-04-06T14:17:19.685 INFO:tasks.workunit.client.0.smithi032.stdout:3/951: creat d3/dcf/f165 x:0 0 0 2024-04-06T14:17:19.688 INFO:tasks.workunit.client.0.smithi032.stdout:5/898: creat d16/d18/d33/dea/f139 x:0 0 0 2024-04-06T14:17:19.688 INFO:tasks.workunit.client.1.smithi086.stdout:8/796: truncate d0/d13/db6/fba 482551 0 2024-04-06T14:17:19.689 INFO:tasks.workunit.client.1.smithi086.stdout:3/802: dread - d5/dea/dd2/ddb/de6/d9e/da5/fad zero size 2024-04-06T14:17:19.689 INFO:tasks.workunit.client.1.smithi086.stdout:2/757: mkdir d0/df/dd2/d10e 0 2024-04-06T14:17:19.689 INFO:tasks.workunit.client.1.smithi086.stdout:7/720: rmdir d7/d10/d11 39 2024-04-06T14:17:19.689 INFO:tasks.workunit.client.1.smithi086.stdout:2/758: dread - d0/df/d48/fb7 zero size 2024-04-06T14:17:19.690 INFO:tasks.workunit.client.0.smithi032.stdout:5/899: readlink d16/d58/d54/d74/d8f/d95/l10b 0 2024-04-06T14:17:19.690 INFO:tasks.workunit.client.1.smithi086.stdout:8/797: creat d0/d71/d22/d43/d7f/f117 x:0 0 0 2024-04-06T14:17:19.693 INFO:tasks.workunit.client.0.smithi032.stdout:3/952: dwrite d3/d16/ddd/f10b [4194304,4194304] 0 2024-04-06T14:17:19.695 INFO:tasks.workunit.client.1.smithi086.stdout:7/721: getdents d7 0 2024-04-06T14:17:19.696 INFO:tasks.workunit.client.0.smithi032.stdout:4/910: sync 2024-04-06T14:17:19.698 INFO:tasks.workunit.client.0.smithi032.stdout:5/900: rename d16/d18/f121 to d16/d18/f13a 0 2024-04-06T14:17:19.699 INFO:tasks.workunit.client.1.smithi086.stdout:3/803: dwrite d5/dea/db4/fd8 [0,4194304] 0 2024-04-06T14:17:19.705 INFO:tasks.workunit.client.1.smithi086.stdout:7/722: mkdir d7/d10/d11/d1c/df4 0 2024-04-06T14:17:19.709 INFO:tasks.workunit.client.1.smithi086.stdout:9/613: dwrite d5/d13/d1a/d5f/d88/d14/d19/d25/f99 [0,4194304] 0 2024-04-06T14:17:19.710 INFO:tasks.workunit.client.1.smithi086.stdout:3/804: fdatasync d5/dea/db4/da7/f79 0 2024-04-06T14:17:19.713 INFO:tasks.workunit.client.0.smithi032.stdout:3/953: creat d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/f166 x:0 0 0 2024-04-06T14:17:19.713 INFO:tasks.workunit.client.0.smithi032.stdout:4/911: rmdir d0/df/d1f/d2e/d122 0 2024-04-06T14:17:19.713 INFO:tasks.workunit.client.1.smithi086.stdout:3/805: read - d5/d36/d5c/d8e/dc1/ff1 zero size 2024-04-06T14:17:19.713 INFO:tasks.workunit.client.1.smithi086.stdout:2/759: link d0/d20/d27/l39 d0/d20/d27/d2b/d35/d36/l10f 0 2024-04-06T14:17:19.714 INFO:tasks.workunit.client.1.smithi086.stdout:3/806: stat d5/d33/d5d/fa2 0 2024-04-06T14:17:19.714 INFO:tasks.workunit.client.0.smithi032.stdout:2/862: write d0/d30/d39/d59/d67/d80/fad [1618131,121723] 0 2024-04-06T14:17:19.715 INFO:tasks.workunit.client.1.smithi086.stdout:8/798: write d0/d13/f21 [1472088,71698] 0 2024-04-06T14:17:19.716 INFO:tasks.workunit.client.1.smithi086.stdout:3/807: write d5/dea/dd2/ddb/de6/f85 [299583,64025] 0 2024-04-06T14:17:19.717 INFO:tasks.workunit.client.0.smithi032.stdout:9/933: write d2/d15/d1e/f1f [406236,58495] 0 2024-04-06T14:17:19.719 INFO:tasks.workunit.client.0.smithi032.stdout:3/954: rename d3/d16/ddd/df7/d147/d89 to d3/d16/ddd/df7/d147/d89/dcb/d155/d167 22 2024-04-06T14:17:19.723 INFO:tasks.workunit.client.1.smithi086.stdout:2/760: rename d0/d20/d44/d9f to d0/df/d110 0 2024-04-06T14:17:19.724 INFO:tasks.workunit.client.0.smithi032.stdout:9/934: chown d2/d15/d1e/d41/d6c/d74 57122 1 2024-04-06T14:17:19.725 INFO:tasks.workunit.client.0.smithi032.stdout:2/863: mkdir d0/d6e/dae/d120 0 2024-04-06T14:17:19.725 INFO:tasks.workunit.client.0.smithi032.stdout:4/912: write d0/df/d76/f61 [4674172,62433] 0 2024-04-06T14:17:19.725 INFO:tasks.workunit.client.1.smithi086.stdout:2/761: mkdir d0/d20/d27/d2b/d35/d36/d4a/d111 0 2024-04-06T14:17:19.726 INFO:tasks.workunit.client.1.smithi086.stdout:9/614: write d5/d13/f7c [436323,128235] 0 2024-04-06T14:17:19.727 INFO:tasks.workunit.client.1.smithi086.stdout:8/799: mkdir d0/d3b/d79/dc6/d118 0 2024-04-06T14:17:19.729 INFO:tasks.workunit.client.0.smithi032.stdout:5/901: dwrite d16/d18/d33/fa8 [0,4194304] 0 2024-04-06T14:17:19.729 INFO:tasks.workunit.client.1.smithi086.stdout:2/762: write d0/d20/d27/d37/d52/f5c [1166486,89441] 0 2024-04-06T14:17:19.731 INFO:tasks.workunit.client.1.smithi086.stdout:8/800: stat d0/d71/d16/c2e 0 2024-04-06T14:17:19.732 INFO:tasks.workunit.client.1.smithi086.stdout:8/801: dread - d0/d71/d16/ff1 zero size 2024-04-06T14:17:19.734 INFO:tasks.workunit.client.0.smithi032.stdout:2/864: chown d0/d30/d39/d59/d67/d74/d78/d57/d7c/de6/ffd 1656411527 1 2024-04-06T14:17:19.738 INFO:tasks.workunit.client.1.smithi086.stdout:9/615: creat d5/d13/d47/fc4 x:0 0 0 2024-04-06T14:17:19.739 INFO:tasks.workunit.client.0.smithi032.stdout:4/913: creat d0/df/d23/d31/d50/d57/d5b/d9c/dc8/f139 x:0 0 0 2024-04-06T14:17:19.739 INFO:tasks.workunit.client.0.smithi032.stdout:5/902: chown d16/d18/d33/d45/ldb 6025676 1 2024-04-06T14:17:19.740 INFO:tasks.workunit.client.0.smithi032.stdout:9/935: creat d2/d15/d1e/d41/d6c/f12d x:0 0 0 2024-04-06T14:17:19.740 INFO:tasks.workunit.client.0.smithi032.stdout:5/903: chown d16/d18/d33/d45 3947871 1 2024-04-06T14:17:19.740 INFO:tasks.workunit.client.1.smithi086.stdout:9/616: fsync d5/d13/d1a/f63 0 2024-04-06T14:17:19.742 INFO:tasks.workunit.client.0.smithi032.stdout:3/955: rename d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/c6e to d3/d16/ddd/df7/d147/d7d/d9c/dbc/c168 0 2024-04-06T14:17:19.743 INFO:tasks.workunit.client.1.smithi086.stdout:9/617: chown d5/d13/d1a/d5f/f6a 3205 1 2024-04-06T14:17:19.743 INFO:tasks.workunit.client.0.smithi032.stdout:9/936: mknod d2/d15/d1e/d65/d6b/d92/d11d/d59/df2/c12e 0 2024-04-06T14:17:19.746 INFO:tasks.workunit.client.1.smithi086.stdout:9/618: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5 0 2024-04-06T14:17:19.746 INFO:tasks.workunit.client.0.smithi032.stdout:4/914: dwrite d0/f5c [0,4194304] 0 2024-04-06T14:17:19.751 INFO:tasks.workunit.client.1.smithi086.stdout:9/619: write d5/d58/fbe [11370,4972] 0 2024-04-06T14:17:19.755 INFO:tasks.workunit.client.1.smithi086.stdout:9/620: fsync d5/d13/d47/d55/f64 0 2024-04-06T14:17:19.755 INFO:tasks.workunit.client.1.smithi086.stdout:2/763: dread d0/df/f21 [0,4194304] 0 2024-04-06T14:17:19.755 INFO:tasks.workunit.client.1.smithi086.stdout:9/621: dread d5/d13/d1a/d5f/d88/d14/d90/f9b [0,4194304] 0 2024-04-06T14:17:19.757 INFO:tasks.workunit.client.0.smithi032.stdout:4/915: dread d0/d82/d8e/fde [0,4194304] 0 2024-04-06T14:17:19.757 INFO:tasks.workunit.client.0.smithi032.stdout:9/937: dread d2/d15/d1e/d65/d83/f3b [0,4194304] 0 2024-04-06T14:17:19.757 INFO:tasks.workunit.client.0.smithi032.stdout:3/956: rename d3/d16/f35 to d3/d16/ddd/df7/d147/d7d/d9c/dbc/d128/d14e/f169 0 2024-04-06T14:17:19.761 INFO:tasks.workunit.client.1.smithi086.stdout:8/802: dwrite d0/f26 [0,4194304] 0 2024-04-06T14:17:19.763 INFO:tasks.workunit.client.0.smithi032.stdout:4/916: creat d0/df/d23/d31/d50/d5e/f13a x:0 0 0 2024-04-06T14:17:19.763 INFO:tasks.workunit.client.1.smithi086.stdout:4/581: sync 2024-04-06T14:17:19.763 INFO:tasks.workunit.client.1.smithi086.stdout:0/790: sync 2024-04-06T14:17:19.764 INFO:tasks.workunit.client.0.smithi032.stdout:9/938: mknod d2/d15/d1e/d65/d6b/c12f 0 2024-04-06T14:17:19.764 INFO:tasks.workunit.client.0.smithi032.stdout:2/865: write d0/d5a/f8e [4128776,12066] 0 2024-04-06T14:17:19.764 INFO:tasks.workunit.client.0.smithi032.stdout:3/957: write d3/d16/d63/f8e [350055,86856] 0 2024-04-06T14:17:19.765 INFO:tasks.workunit.client.1.smithi086.stdout:2/764: unlink d0/df/d110/ld8 0 2024-04-06T14:17:19.766 INFO:tasks.workunit.client.1.smithi086.stdout:8/803: mknod d0/d5a/d100/c119 0 2024-04-06T14:17:19.766 INFO:tasks.workunit.client.0.smithi032.stdout:4/917: mkdir d0/d82/d4b/d13b 0 2024-04-06T14:17:19.766 INFO:tasks.workunit.client.0.smithi032.stdout:9/939: write d2/d15/d1e/d41/d6c/d74/fc3 [1581041,18098] 0 2024-04-06T14:17:19.766 INFO:tasks.workunit.client.0.smithi032.stdout:5/904: dwrite d16/d18/dae/dbb/ff9 [0,4194304] 0 2024-04-06T14:17:19.767 INFO:tasks.workunit.client.0.smithi032.stdout:4/918: unlink d0/l7 0 2024-04-06T14:17:19.770 INFO:tasks.workunit.client.0.smithi032.stdout:0/763: sync 2024-04-06T14:17:19.770 INFO:tasks.workunit.client.1.smithi086.stdout:0/791: symlink d0/d1/d2/d5b/l102 0 2024-04-06T14:17:19.772 INFO:tasks.workunit.client.0.smithi032.stdout:4/919: dread - d0/df/d23/f135 zero size 2024-04-06T14:17:19.775 INFO:tasks.workunit.client.1.smithi086.stdout:2/765: dwrite d0/d20/d27/d2b/d35/d36/d4d/d59/fad [0,4194304] 0 2024-04-06T14:17:19.776 INFO:tasks.workunit.client.0.smithi032.stdout:2/866: rename d0/d30/d39/d59/d67/d74/d78/d57/d7c/lf6 to d0/d30/d39/d59/d67/d74/d78/d8c/l121 0 2024-04-06T14:17:19.777 INFO:tasks.workunit.client.1.smithi086.stdout:0/792: mknod d0/d1/d2/d5b/c103 0 2024-04-06T14:17:19.780 INFO:tasks.workunit.client.0.smithi032.stdout:5/905: dread d16/d18/d33/fa8 [0,4194304] 0 2024-04-06T14:17:19.781 INFO:tasks.workunit.client.0.smithi032.stdout:4/920: dread d0/df/d1f/d5a/f8f [0,4194304] 0 2024-04-06T14:17:19.781 INFO:tasks.workunit.client.1.smithi086.stdout:8/804: symlink d0/d71/d93/ded/l11a 0 2024-04-06T14:17:19.783 INFO:tasks.workunit.client.1.smithi086.stdout:4/582: truncate d0/f5 3922915 0 2024-04-06T14:17:19.783 INFO:tasks.workunit.client.0.smithi032.stdout:2/867: unlink d0/f22 0 2024-04-06T14:17:19.784 INFO:tasks.workunit.client.1.smithi086.stdout:8/805: readlink d0/d5a/d100/l103 0 2024-04-06T14:17:19.786 INFO:tasks.workunit.client.1.smithi086.stdout:2/766: symlink d0/l112 0 2024-04-06T14:17:19.787 INFO:tasks.workunit.client.0.smithi032.stdout:9/940: creat d2/d15/d1e/d65/d6b/d92/d11d/d59/d115/f130 x:0 0 0 2024-04-06T14:17:19.787 INFO:tasks.workunit.client.1.smithi086.stdout:0/793: rename d0/d15/d4e/da3/cd1 to d0/d15/d4e/d48/d89/c104 0 2024-04-06T14:17:19.788 INFO:tasks.workunit.client.1.smithi086.stdout:9/622: dwrite d5/d13/d1a/d5f/d88/d14/d19/f76 [0,4194304] 0 2024-04-06T14:17:19.790 INFO:tasks.workunit.client.1.smithi086.stdout:8/806: chown d0/d71/d22/d43/d4d/d62/c9d 24135044 1 2024-04-06T14:17:19.790 INFO:tasks.workunit.client.1.smithi086.stdout:9/623: stat d5/d13/d47/d55/d87/dbb 0 2024-04-06T14:17:19.791 INFO:tasks.workunit.client.1.smithi086.stdout:2/767: stat d0/d20/d27/d2b/d35/d36/d4a/ff8 0 2024-04-06T14:17:19.791 INFO:tasks.workunit.client.1.smithi086.stdout:8/807: dread - d0/d71/d22/d65/fde zero size 2024-04-06T14:17:19.791 INFO:tasks.workunit.client.1.smithi086.stdout:0/794: mknod d0/d15/d26/c105 0 2024-04-06T14:17:19.792 INFO:tasks.workunit.client.1.smithi086.stdout:9/624: dread - d5/d13/d1a/d8f/fb8 zero size 2024-04-06T14:17:19.792 INFO:tasks.workunit.client.0.smithi032.stdout:5/906: symlink d16/d18/d33/dea/l13b 0 2024-04-06T14:17:19.796 INFO:tasks.workunit.client.1.smithi086.stdout:9/625: dread - d5/d13/d1a/d5f/d88/d14/d19/d25/d74/fa5 zero size 2024-04-06T14:17:19.796 INFO:tasks.workunit.client.0.smithi032.stdout:2/868: rename d0/d5a/f1a to d0/d30/d39/d59/d67/d74/d78/d57/d7c/de6/f122 0 2024-04-06T14:17:19.801 INFO:tasks.workunit.client.1.smithi086.stdout:4/583: write d0/d15/f48 [2996864,4063] 0 2024-04-06T14:17:19.802 INFO:tasks.workunit.client.0.smithi032.stdout:0/764: write d3/f4 [1061180,12393] 0 2024-04-06T14:17:19.803 INFO:tasks.workunit.client.1.smithi086.stdout:0/795: creat d0/d15/d26/d33/d9f/f106 x:0 0 0 2024-04-06T14:17:19.804 INFO:tasks.workunit.client.1.smithi086.stdout:4/584: fdatasync d0/d2c/d30/d24/d71/f84 0 2024-04-06T14:17:19.807 INFO:tasks.workunit.client.1.smithi086.stdout:8/808: symlink d0/d71/d16/d39/d90/daf/df0/d10d/l11b 0 2024-04-06T14:17:19.808 INFO:tasks.workunit.client.0.smithi032.stdout:5/907: dwrite d16/d18/d33/f8b [0,4194304] 0 2024-04-06T14:17:19.810 INFO:tasks.workunit.client.0.smithi032.stdout:3/958: dwrite d3/d16/ddd/df7/d147/d89/f97 [4194304,4194304] 0 2024-04-06T14:17:19.810 INFO:tasks.workunit.client.0.smithi032.stdout:2/869: symlink d0/d6e/d8d/d10d/l123 0 2024-04-06T14:17:19.810 INFO:tasks.workunit.client.0.smithi032.stdout:4/921: write d0/df/d1f/f95 [3511075,82334] 0 2024-04-06T14:17:19.812 INFO:tasks.workunit.client.1.smithi086.stdout:2/768: creat d0/d20/d27/d2b/d35/d36/f113 x:0 0 0 2024-04-06T14:17:19.813 INFO:tasks.workunit.client.1.smithi086.stdout:4/585: symlink d0/d2c/d30/d24/da3/d88/lb6 0 2024-04-06T14:17:19.815 INFO:tasks.workunit.client.0.smithi032.stdout:9/941: write d2/d15/d1e/d65/d6b/d92/d11d/d5f/fc8 [117159,105962] 0 2024-04-06T14:17:19.817 INFO:tasks.workunit.client.1.smithi086.stdout:2/769: stat d0/d20/d27/d2b/d35/d36/d4d/d59/fcf 0 2024-04-06T14:17:19.820 INFO:tasks.workunit.client.1.smithi086.stdout:9/626: dread d5/d13/d1a/d5f/d88/d14/f80 [0,4194304] 0 2024-04-06T14:17:19.821 INFO:tasks.workunit.client.0.smithi032.stdout:3/959: unlink d3/d16/d63/l13c 0 2024-04-06T14:17:19.821 INFO:tasks.workunit.client.1.smithi086.stdout:4/586: mknod d0/d2c/d30/d24/d5d/cb7 0 2024-04-06T14:17:19.822 INFO:tasks.workunit.client.0.smithi032.stdout:0/765: write d3/da/d25/d2b/d4a/fb9 [949205,53289] 0 2024-04-06T14:17:19.822 INFO:tasks.workunit.client.0.smithi032.stdout:3/960: chown d3/d16/f70 1453 1 2024-04-06T14:17:19.824 INFO:tasks.workunit.client.0.smithi032.stdout:5/908: symlink d16/d18/d33/d35/d4d/l13c 0 2024-04-06T14:17:19.826 INFO:tasks.workunit.client.0.smithi032.stdout:5/909: chown d16/d18/d33/d45/d5e/lac 260793964 1 2024-04-06T14:17:19.826 INFO:tasks.workunit.client.0.smithi032.stdout:4/922: chown d0/df/d1f/d42/d45/f75 110812 1 2024-04-06T14:17:19.827 INFO:tasks.workunit.client.1.smithi086.stdout:0/796: dread d0/d15/d4e/f92 [0,4194304] 0 2024-04-06T14:17:19.828 INFO:tasks.workunit.client.1.smithi086.stdout:9/627: dwrite d5/d13/fa9 [0,4194304] 0 2024-04-06T14:17:19.828 INFO:tasks.workunit.client.0.smithi032.stdout:9/942: creat d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/d4a/ded/d100/f131 x:0 0 0 2024-04-06T14:17:19.829 INFO:tasks.workunit.client.1.smithi086.stdout:0/797: chown d0/f17 46081252 1 2024-04-06T14:17:19.830 INFO:tasks.workunit.client.1.smithi086.stdout:0/798: readlink d0/d15/lad 0 2024-04-06T14:17:19.830 INFO:tasks.workunit.client.0.smithi032.stdout:3/961: creat d3/d16/ddd/df7/d147/d7d/dc6/df0/f16a x:0 0 0 2024-04-06T14:17:19.832 INFO:tasks.workunit.client.0.smithi032.stdout:5/910: write fe [2739115,16328] 0 2024-04-06T14:17:19.832 INFO:tasks.workunit.client.1.smithi086.stdout:0/799: chown d0/d1/d2/d1a/d51/dd7 171377 1 2024-04-06T14:17:19.832 INFO:tasks.workunit.client.1.smithi086.stdout:8/809: write d0/d71/d22/d43/d6d/d75/db2/fcc [1010943,59244] 0 2024-04-06T14:17:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.832+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc581044e0 con 0x7fbc4006aec0 2024-04-06T14:17:19.833 INFO:tasks.workunit.client.0.smithi032.stdout:0/766: symlink d3/d83/dfd/l104 0 2024-04-06T14:17:19.834 INFO:tasks.workunit.client.0.smithi032.stdout:4/923: unlink d0/df/d23/d31/d50/d57/d5b/l9e 0 2024-04-06T14:17:19.835 INFO:tasks.workunit.client.1.smithi086.stdout:2/770: truncate d0/d20/d27/f10a 1564033 0 2024-04-06T14:17:19.835 INFO:tasks.workunit.client.1.smithi086.stdout:8/810: write d0/d13/db6/f10e [138605,89057] 0 2024-04-06T14:17:19.836 INFO:tasks.workunit.client.1.smithi086.stdout:0/800: mkdir d0/d15/d4e/da3/d46/d52/dd0/dbe/d107 0 2024-04-06T14:17:19.836 INFO:tasks.workunit.client.0.smithi032.stdout:4/924: write d0/df/d1f/d42/f10b [771131,35917] 0 2024-04-06T14:17:19.837 INFO:tasks.workunit.client.0.smithi032.stdout:9/943: rmdir d2/d15/d1e/d65/d6b/d92/d11d/d59/df2 39 2024-04-06T14:17:19.841 INFO:tasks.workunit.client.1.smithi086.stdout:8/811: mknod d0/d71/d22/d43/d7f/c11c 0 2024-04-06T14:17:19.841 INFO:tasks.workunit.client.1.smithi086.stdout:2/771: symlink d0/d20/d27/l114 0 2024-04-06T14:17:19.842 INFO:tasks.workunit.client.0.smithi032.stdout:4/925: truncate d0/d82/daf/f12b 848760 0 2024-04-06T14:17:19.842 INFO:tasks.workunit.client.1.smithi086.stdout:0/801: readlink d0/d1/d4/de/dae/lfb 0 2024-04-06T14:17:19.844 INFO:tasks.workunit.client.0.smithi032.stdout:2/870: getdents d0/d30/d39/d59/d67/d74/d78/d8c 0 2024-04-06T14:17:19.844 INFO:tasks.workunit.client.1.smithi086.stdout:4/587: dwrite d0/d15/d20/f57 [0,4194304] 0 2024-04-06T14:17:19.844 INFO:tasks.workunit.client.0.smithi032.stdout:5/911: dwrite d16/d18/d33/d35/d88/deb/f114 [0,4194304] 0 2024-04-06T14:17:19.846 INFO:tasks.workunit.client.1.smithi086.stdout:4/588: stat d0/d2c/d30/d24/d50/d79 0 2024-04-06T14:17:19.846 INFO:tasks.workunit.client.0.smithi032.stdout:9/944: mknod d2/d15/d1e/d41/d6c/d88/de6/d11f/c132 0 2024-04-06T14:17:19.846 INFO:tasks.workunit.client.0.smithi032.stdout:5/912: chown d16/d18/d33/d35/cd8 97000 1 2024-04-06T14:17:19.849 INFO:tasks.workunit.client.0.smithi032.stdout:2/871: unlink d0/c35 0 2024-04-06T14:17:19.852 INFO:tasks.workunit.client.0.smithi032.stdout:2/872: write d0/d30/d39/d59/d67/d74/d78/d57/fde [3598459,32938] 0 2024-04-06T14:17:19.852 INFO:tasks.workunit.client.0.smithi032.stdout:9/945: readlink d2/d15/d1e/d65/d6b/ld0 0 2024-04-06T14:17:19.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.853+0000 7fbc3f7fe700 1 -- 172.21.15.32:0/1920893070 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fbc581044e0 con 0x7fbc4006aec0 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (7m) 40s ago 8m 26.8M - 0.25.0 c8568f914cd2 d6f57d3d10f9 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 40s ago 9m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (8m) 43s ago 8m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 40s ago 8m 86.7M - 9.4.7 954c08fa6188 1cc1d256b791 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (3m) 40s ago 3m 11.4M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (3m) 40s ago 3m 108M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (3m) 43s ago 3m 108M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (3m) 43s ago 3m 14.2M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:9283 running (10m) 40s ago 10m 192M - 17.2.7-819-g6699495c ec300eaa768b 7737c4037eba 2024-04-06T14:17:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283 running (61s) 43s ago 8m 486M - 18.2.2-1086-g052adcad 87261d5ece5a f68af56edf8d 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 40s ago 10m 47.3M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (7m) 43s ago 7m 38.6M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 starting - - - - 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (8m) 43s ago 8m 18.9M - 1.5.0 0da6a335fe13 21ac74498650 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 40s ago 7m 76.0M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 40s ago 6m 83.0M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 40s ago 6m 76.6M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (5m) 43s ago 5m 90.2M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (5m) 43s ago 5m 83.9M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (4m) 43s ago 4m 75.4M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:17:19.855 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 40s ago 8m 57.4M - 2.43.0 a07b618ecd1d 6742cb6c38a9 2024-04-06T14:17:19.856 INFO:tasks.workunit.client.1.smithi086.stdout:0/802: mkdir d0/d1/d2/d1a/d51/d108 0 2024-04-06T14:17:19.857 INFO:tasks.workunit.client.1.smithi086.stdout:2/772: truncate d0/df/f21 5118071 0 2024-04-06T14:17:19.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.857+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fbc4006aec0 msgr2=0x7fbc4006d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.857+0000 7fbc5d951700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fbc4006aec0 0x7fbc4006d370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c000c00 tx=0x7fbc4c009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.857+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580feff0 msgr2=0x7fbc5806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.857+0000 7fbc5d951700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580feff0 0x7fbc5806e470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc480096b0 tx=0x7fbc48009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.858+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 shutdown_connections 2024-04-06T14:17:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.858+0000 7fbc5d951700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fbc4006aec0 0x7fbc4006d370 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.858+0000 7fbc5d951700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580feff0 0x7fbc5806e470 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc480096b0 tx=0x7fbc48009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.858+0000 7fbc5d951700 1 --2- 172.21.15.32:0/1920893070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc581002a0 0x7fbc5806e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.858+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 >> 172.21.15.32:0/1920893070 conn(0x7fbc580fa950 msgr2=0x7fbc580fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:19.859 INFO:tasks.workunit.client.0.smithi032.stdout:5/913: rename d16/d18/d33/fe9 to d16/d18/dae/dbb/ddc/f13d 0 2024-04-06T14:17:19.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.859+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 shutdown_connections 2024-04-06T14:17:19.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:19.859+0000 7fbc5d951700 1 -- 172.21.15.32:0/1920893070 wait complete. 2024-04-06T14:17:19.860 INFO:tasks.workunit.client.0.smithi032.stdout:4/926: link d0/df/d2b/dbe/d116/lcc d0/df/d23/d31/d50/d57/d8a/db5/l13c 0 2024-04-06T14:17:19.863 INFO:tasks.workunit.client.0.smithi032.stdout:2/873: rename d0/d30/d39/d59/d67/d74/d78/d57/fd1 to d0/d30/d39/d59/d67/d74/d78/d57/f124 0 2024-04-06T14:17:19.864 INFO:tasks.workunit.client.0.smithi032.stdout:4/927: dread - d0/d82/d8e/fa7 zero size 2024-04-06T14:17:19.866 INFO:tasks.workunit.client.0.smithi032.stdout:9/946: fsync d2/d15/f43 0 2024-04-06T14:17:19.868 INFO:tasks.workunit.client.0.smithi032.stdout:2/874: truncate d0/f6c 3786077 0 2024-04-06T14:17:19.868 INFO:tasks.workunit.client.0.smithi032.stdout:0/767: dwrite d3/d83/fc5 [0,4194304] 0 2024-04-06T14:17:19.871 INFO:tasks.workunit.client.0.smithi032.stdout:2/875: stat d0/d30/c47 0 2024-04-06T14:17:19.871 INFO:tasks.workunit.client.0.smithi032.stdout:9/947: read d2/d15/d1e/f24 [188380,7170] 0 2024-04-06T14:17:19.872 INFO:tasks.workunit.client.1.smithi086.stdout:4/589: mknod d0/d2c/cb8 0 2024-04-06T14:17:19.875 INFO:tasks.workunit.client.1.smithi086.stdout:8/812: dwrite d0/d71/d22/d43/d6d/d75/db2/f114 [0,4194304] 0 2024-04-06T14:17:19.877 INFO:tasks.workunit.client.0.smithi032.stdout:4/928: rename d0/df/d1f/d42/d45/l7b to d0/d82/d8e/d126/l13d 0 2024-04-06T14:17:19.878 INFO:tasks.workunit.client.1.smithi086.stdout:2/773: mknod d0/d20/d27/dae/c115 0 2024-04-06T14:17:19.879 INFO:tasks.workunit.client.0.smithi032.stdout:4/929: chown d0/d82/d8e/fa7 87 1 2024-04-06T14:17:19.879 INFO:tasks.workunit.client.1.smithi086.stdout:2/774: dread - d0/d20/d44/dbb/fdb zero size 2024-04-06T14:17:19.879 INFO:tasks.workunit.client.1.smithi086.stdout:4/590: mknod d0/d2c/d30/d24/da3/d88/cb9 0 2024-04-06T14:17:19.880 INFO:tasks.workunit.client.1.smithi086.stdout:2/775: stat d0/df/d5e/d9d/d102 0 2024-04-06T14:17:19.881 INFO:tasks.workunit.client.1.smithi086.stdout:8/813: mknod d0/d71/d22/d65/d6e/c11d 0 2024-04-06T14:17:19.882 INFO:tasks.workunit.client.0.smithi032.stdout:0/768: dwrite d3/f5a [0,4194304] 0 2024-04-06T14:17:19.884 INFO:tasks.workunit.client.1.smithi086.stdout:1/780: sync 2024-04-06T14:17:19.884 INFO:tasks.workunit.client.1.smithi086.stdout:6/883: sync 2024-04-06T14:17:19.884 INFO:tasks.workunit.client.1.smithi086.stdout:0/803: mkdir d0/d1/d2/d109 0 2024-04-06T14:17:19.886 INFO:tasks.workunit.client.0.smithi032.stdout:5/914: mkdir d16/d18/d33/d35/de0/d13e 0 2024-04-06T14:17:19.886 INFO:tasks.workunit.client.0.smithi032.stdout:2/876: mknod d0/d30/d39/d59/d67/d74/d78/d57/c125 0 2024-04-06T14:17:19.886 INFO:tasks.workunit.client.1.smithi086.stdout:8/814: dread - d0/d71/d22/d43/d4d/da3/fe2 zero size 2024-04-06T14:17:19.899 INFO:tasks.workunit.client.0.smithi032.stdout:4/930: chown d0/df/d23/d31/d50/d57/d5b/d9c/d109/c123 66859 1 2024-04-06T14:17:19.905 INFO:tasks.workunit.client.0.smithi032.stdout:0/769: dwrite d3/da/f1d [0,4194304] 0 2024-04-06T14:17:19.910 INFO:tasks.workunit.client.0.smithi032.stdout:2/877: dwrite d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102/f111 [0,4194304] 0 2024-04-06T14:17:19.910 INFO:tasks.workunit.client.0.smithi032.stdout:9/948: dread d2/d15/d1e/d65/d83/f120 [0,4194304] 0 2024-04-06T14:17:19.913 INFO:tasks.workunit.client.1.smithi086.stdout:1/781: fsync db/d13/f54 0 2024-04-06T14:17:19.913 INFO:tasks.workunit.client.1.smithi086.stdout:4/591: dread d0/d15/d2b/f82 [0,4194304] 0 2024-04-06T14:17:19.914 INFO:tasks.workunit.client.0.smithi032.stdout:0/770: mknod d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/d9c/dcc/c105 0 2024-04-06T14:17:19.922 INFO:tasks.workunit.client.1.smithi086.stdout:6/884: mknod d6/d13/d1a/d64/c12f 0 2024-04-06T14:17:19.922 INFO:tasks.workunit.client.0.smithi032.stdout:0/771: write d3/f5d [379977,101458] 0 2024-04-06T14:17:19.924 INFO:tasks.workunit.client.1.smithi086.stdout:8/815: readlink d0/l25 0 2024-04-06T14:17:19.924 INFO:tasks.workunit.client.1.smithi086.stdout:2/776: dwrite d0/d20/d27/d37/d52/feb [0,4194304] 0 2024-04-06T14:17:19.925 INFO:tasks.workunit.client.0.smithi032.stdout:2/878: creat d0/d30/d7f/f126 x:0 0 0 2024-04-06T14:17:19.926 INFO:tasks.workunit.client.1.smithi086.stdout:1/782: mknod db/d13/cf8 0 2024-04-06T14:17:19.926 INFO:tasks.workunit.client.1.smithi086.stdout:6/885: creat d6/d13/d1a/da0/d114/db2/f130 x:0 0 0 2024-04-06T14:17:19.926 INFO:tasks.workunit.client.0.smithi032.stdout:0/772: creat d3/da/d25/d2b/d4a/d86/f106 x:0 0 0 2024-04-06T14:17:19.926 INFO:tasks.workunit.client.0.smithi032.stdout:5/915: write d16/d18/dae/dbb/ddc/f13d [180389,107833] 0 2024-04-06T14:17:19.927 INFO:tasks.workunit.client.1.smithi086.stdout:8/816: rename d0/d71/d16/d39/d90/cff to d0/d3b/db9/c11e 0 2024-04-06T14:17:19.928 INFO:tasks.workunit.client.1.smithi086.stdout:6/886: symlink d6/d9/d63/l131 0 2024-04-06T14:17:19.929 INFO:tasks.workunit.client.0.smithi032.stdout:0/773: truncate d3/da/d25/d2b/d4a/f62 2642619 0 2024-04-06T14:17:19.929 INFO:tasks.workunit.client.0.smithi032.stdout:2/879: chown d0/d4e/de9/lfa 985097 1 2024-04-06T14:17:19.931 INFO:tasks.workunit.client.1.smithi086.stdout:6/887: readlink d6/l96 0 2024-04-06T14:17:19.932 INFO:tasks.workunit.client.0.smithi032.stdout:4/931: getdents d0/df/d1f/d2e 0 2024-04-06T14:17:19.932 INFO:tasks.workunit.client.0.smithi032.stdout:9/949: write d2/d15/d1e/fff [917481,127459] 0 2024-04-06T14:17:19.933 INFO:tasks.workunit.client.0.smithi032.stdout:0/774: truncate d3/f2e 3516059 0 2024-04-06T14:17:19.934 INFO:tasks.workunit.client.1.smithi086.stdout:6/888: mknod d6/d9/d43/c132 0 2024-04-06T14:17:19.934 INFO:tasks.workunit.client.0.smithi032.stdout:4/932: stat d0/d82/l6c 0 2024-04-06T14:17:19.935 INFO:tasks.workunit.client.0.smithi032.stdout:9/950: read - d2/ffc zero size 2024-04-06T14:17:19.939 INFO:tasks.workunit.client.1.smithi086.stdout:6/889: rename d6/d13/d1a/d64/f9f to d6/d9/db/f133 0 2024-04-06T14:17:19.945 INFO:tasks.workunit.client.0.smithi032.stdout:2/880: creat d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/f127 x:0 0 0 2024-04-06T14:17:19.946 INFO:tasks.workunit.client.1.smithi086.stdout:1/783: truncate db/d11/d1a/d2a/d9a/d78/f9f 3017838 0 2024-04-06T14:17:19.946 INFO:tasks.workunit.client.1.smithi086.stdout:8/817: write d0/d5a/d100/fcf [13769,96184] 0 2024-04-06T14:17:19.946 INFO:tasks.workunit.client.0.smithi032.stdout:5/916: dwrite f1 [4194304,4194304] 0 2024-04-06T14:17:19.946 INFO:tasks.workunit.client.0.smithi032.stdout:9/951: chown d2/f4 1 1 2024-04-06T14:17:19.947 INFO:tasks.workunit.client.1.smithi086.stdout:6/890: rename d6/d9/d63/ldf to d6/d20/de0/l134 0 2024-04-06T14:17:19.948 INFO:tasks.workunit.client.0.smithi032.stdout:2/881: dread - d0/d30/d39/faa zero size 2024-04-06T14:17:19.949 INFO:tasks.workunit.client.0.smithi032.stdout:9/952: chown d2/d15/d1e/d65/d6b/d92/d11d/f9a 4965 1 2024-04-06T14:17:19.949 INFO:tasks.workunit.client.0.smithi032.stdout:2/882: symlink d0/d30/d39/d59/d67/d74/d78/d57/l128 0 2024-04-06T14:17:19.950 INFO:tasks.workunit.client.0.smithi032.stdout:5/917: rename d16/d18/d33/d35/d88/fad to d16/d18/d33/d45/d52/f13f 0 2024-04-06T14:17:19.951 INFO:tasks.workunit.client.1.smithi086.stdout:1/784: stat db/d13/d55/df1/cc9 0 2024-04-06T14:17:19.951 INFO:tasks.workunit.client.1.smithi086.stdout:6/891: read - d6/d9/d43/ff9 zero size 2024-04-06T14:17:19.952 INFO:tasks.workunit.client.0.smithi032.stdout:2/883: mkdir d0/d30/d7f/d129 0 2024-04-06T14:17:19.954 INFO:tasks.workunit.client.0.smithi032.stdout:2/884: chown d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102 2341313 1 2024-04-06T14:17:19.959 INFO:tasks.workunit.client.1.smithi086.stdout:2/777: dwrite d0/f33 [0,4194304] 0 2024-04-06T14:17:19.959 INFO:tasks.workunit.client.0.smithi032.stdout:2/885: chown d0/d6e/d8d/ca9 6611 1 2024-04-06T14:17:19.959 INFO:tasks.workunit.client.0.smithi032.stdout:5/918: unlink d16/d18/d33/d35/d88/dbd/dc5/dec/l11f 0 2024-04-06T14:17:19.961 INFO:tasks.workunit.client.1.smithi086.stdout:4/592: dwrite d0/d2c/d38/d54/d83/f77 [0,4194304] 0 2024-04-06T14:17:19.962 INFO:tasks.workunit.client.1.smithi086.stdout:6/892: write d6/d13/f77 [11509246,76189] 0 2024-04-06T14:17:19.963 INFO:tasks.workunit.client.0.smithi032.stdout:9/953: creat d2/d15/d1e/d65/d83/d7e/f133 x:0 0 0 2024-04-06T14:17:19.963 INFO:tasks.workunit.client.0.smithi032.stdout:0/775: truncate d3/da/f26 3546943 0 2024-04-06T14:17:19.965 INFO:tasks.workunit.client.1.smithi086.stdout:8/818: dwrite d0/d71/d22/d65/d6e/d98/ffe [0,4194304] 0 2024-04-06T14:17:19.966 INFO:tasks.workunit.client.0.smithi032.stdout:9/954: rmdir d2/d15/d1e 39 2024-04-06T14:17:19.973 INFO:tasks.workunit.client.0.smithi032.stdout:5/919: dwrite d16/d18/d33/d45/d87/f10d [0,4194304] 0 2024-04-06T14:17:19.974 INFO:tasks.workunit.client.1.smithi086.stdout:1/785: creat db/d11/d1f/d5c/ddb/de9/ff9 x:0 0 0 2024-04-06T14:17:19.975 INFO:tasks.workunit.client.1.smithi086.stdout:4/593: creat d0/de/d4c/fba x:0 0 0 2024-04-06T14:17:19.976 INFO:tasks.workunit.client.1.smithi086.stdout:2/778: dwrite d0/df/d48/fb7 [0,4194304] 0 2024-04-06T14:17:19.976 INFO:tasks.workunit.client.1.smithi086.stdout:6/893: mkdir d6/d13/d42/d52/df1/d135 0 2024-04-06T14:17:19.977 INFO:tasks.workunit.client.0.smithi032.stdout:4/933: dwrite d0/df/d2b/d46/d58/f7e [4194304,4194304] 0 2024-04-06T14:17:19.978 INFO:tasks.workunit.client.1.smithi086.stdout:8/819: symlink d0/d3b/d79/l11f 0 2024-04-06T14:17:19.984 INFO:tasks.workunit.client.1.smithi086.stdout:2/779: unlink d0/d20/d27/d2b/d35/l6d 0 2024-04-06T14:17:19.985 INFO:tasks.workunit.client.0.smithi032.stdout:0/776: dread d3/da/d25/d2b/f61 [0,4194304] 0 2024-04-06T14:17:20.011 INFO:tasks.workunit.client.1.smithi086.stdout:4/594: unlink d0/d2c/d30/d24/d50/l6a 0 2024-04-06T14:17:20.012 INFO:tasks.workunit.client.1.smithi086.stdout:4/595: chown d0/d2c/d30/d24/da3 5 1 2024-04-06T14:17:20.013 INFO:tasks.workunit.client.0.smithi032.stdout:5/920: rmdir d16/d18/d33/d45/d52 39 2024-04-06T14:17:20.013 INFO:tasks.workunit.client.1.smithi086.stdout:8/820: symlink d0/d3b/l120 0 2024-04-06T14:17:20.013 INFO:tasks.workunit.client.0.smithi032.stdout:2/886: write d0/d30/d39/d59/d67/d74/d78/d57/f90 [1099984,115687] 0 2024-04-06T14:17:20.013 INFO:tasks.workunit.client.1.smithi086.stdout:4/596: write d0/d2c/d30/d24/d50/d79/d89/fae [891170,80882] 0 2024-04-06T14:17:20.018 INFO:tasks.workunit.client.1.smithi086.stdout:6/894: unlink d6/d13/d1a/da0/d11b/ce9 0 2024-04-06T14:17:20.019 INFO:tasks.workunit.client.0.smithi032.stdout:4/934: mknod d0/df/d23/d31/d50/d57/d5b/d9c/c13e 0 2024-04-06T14:17:20.019 INFO:tasks.workunit.client.0.smithi032.stdout:0/777: mkdir d3/da/d41/d107 0 2024-04-06T14:17:20.020 INFO:tasks.workunit.client.1.smithi086.stdout:4/597: write d0/d2c/d30/d24/d50/d79/d89/fa6 [723559,56593] 0 2024-04-06T14:17:20.022 INFO:tasks.workunit.client.0.smithi032.stdout:4/935: mknod d0/c13f 0 2024-04-06T14:17:20.023 INFO:tasks.workunit.client.1.smithi086.stdout:2/780: mkdir d0/d20/d27/d37/d116 0 2024-04-06T14:17:20.023 INFO:tasks.workunit.client.1.smithi086.stdout:4/598: mkdir d0/d15/d53/dbb 0 2024-04-06T14:17:20.024 INFO:tasks.workunit.client.1.smithi086.stdout:4/599: chown d0/d2c/d38/d54 317 1 2024-04-06T14:17:20.025 INFO:tasks.workunit.client.1.smithi086.stdout:6/895: creat d6/d13/d1a/da0/d11b/f136 x:0 0 0 2024-04-06T14:17:20.028 INFO:tasks.workunit.client.0.smithi032.stdout:0/778: creat d3/da/d25/d2b/d80/db3/df1/f108 x:0 0 0 2024-04-06T14:17:20.029 INFO:tasks.workunit.client.0.smithi032.stdout:9/955: truncate d2/f13 454128 0 2024-04-06T14:17:20.030 INFO:tasks.workunit.client.1.smithi086.stdout:4/600: symlink d0/d2c/d38/d54/lbc 0 2024-04-06T14:17:20.030 INFO:tasks.workunit.client.0.smithi032.stdout:5/921: dwrite d16/d18/d33/d35/d4d/f11d [0,4194304] 0 2024-04-06T14:17:20.031 INFO:tasks.workunit.client.0.smithi032.stdout:2/887: rename d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/fa7 to d0/d30/d39/d59/d67/d74/d78/dc2/f12a 0 2024-04-06T14:17:20.031 INFO:tasks.workunit.client.0.smithi032.stdout:2/888: truncate d0/d5a/dc9/ff4 787739 0 2024-04-06T14:17:20.037 INFO:tasks.workunit.client.1.smithi086.stdout:8/821: truncate d0/d71/d16/d52/dd0/ddc/de7/dd6/fd9 3138466 0 2024-04-06T14:17:20.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.031+0000 7f8fe1270700 1 -- 172.21.15.32:0/4006356543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fdc0fd410 msgr2=0x7f8fdc0fd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:20.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.031+0000 7f8fe1270700 1 --2- 172.21.15.32:0/4006356543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fdc0fd410 0x7f8fdc0fd820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc004940 tx=0x7f8fcc009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:20.040 INFO:tasks.workunit.client.0.smithi032.stdout:4/936: fsync d0/df/d2b/f3b 0 2024-04-06T14:17:20.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.032+0000 7f8fe1270700 1 -- 172.21.15.32:0/4006356543 shutdown_connections 2024-04-06T14:17:20.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.032+0000 7f8fe1270700 1 --2- 172.21.15.32:0/4006356543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fdc0fdd60 0x7f8fdc1002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:20.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.032+0000 7f8fe1270700 1 --2- 172.21.15.32:0/4006356543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fdc0fd410 0x7f8fdc0fd820 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:20.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.032+0000 7f8fe1270700 1 -- 172.21.15.32:0/4006356543 >> 172.21.15.32:0/4006356543 conn(0x7f8fdc0f6be0 msgr2=0x7f8fdc0f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.033+0000 7f8fe1270700 1 -- 172.21.15.32:0/4006356543 shutdown_connections 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.033+0000 7f8fe1270700 1 -- 172.21.15.32:0/4006356543 wait complete. 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.033+0000 7f8fe1270700 1 Processor -- start 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.033+0000 7f8fe1270700 1 -- start start 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fe1270700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fdc0fd410 0x7f8fdc113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fe1270700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fdc0fdd60 0x7f8fdc114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fe1270700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc114920 con 0x7f8fdc0fd410 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fe1270700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc114a90 con 0x7f8fdc0fdd60 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fdaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fdc0fd410 0x7f8fdc113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fdaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fdc0fd410 0x7f8fdc113de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55126/0 (socket says 172.21.15.32:55126) 2024-04-06T14:17:20.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fdaffd700 1 -- 172.21.15.32:0/504397798 learned_addr learned my addr 172.21.15.32:0/504397798 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:20.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fdaffd700 1 -- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fdc0fdd60 msgr2=0x7f8fdc114320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:20.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fda7fc700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fdc0fdd60 0x7f8fdc114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:20.044 INFO:tasks.workunit.client.0.smithi032.stdout:0/779: unlink d3/da/d25/d2b/d4a/l67 0 2024-04-06T14:17:20.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fdaffd700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fdc0fdd60 0x7f8fdc114320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.034+0000 7f8fdaffd700 1 -- 172.21.15.32:0/504397798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc005040 con 0x7f8fdc0fd410 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.035+0000 7f8fda7fc700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fdc0fdd60 0x7f8fdc114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.035+0000 7f8fdaffd700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fdc0fd410 0x7f8fdc113de0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00b0d0 tx=0x7f8fcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.035+0000 7f8fc37fe700 1 -- 172.21.15.32:0/504397798 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fcc00e070 con 0x7f8fdc0fd410 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.035+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fdc067b20 con 0x7f8fdc0fd410 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.035+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fdc0680e0 con 0x7f8fdc0fd410 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.035+0000 7f8fc37fe700 1 -- 172.21.15.32:0/504397798 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8fcc0093f0 con 0x7f8fdc0fd410 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.035+0000 7f8fc37fe700 1 -- 172.21.15.32:0/504397798 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fcc01b360 con 0x7f8fdc0fd410 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.037+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc80052e0 con 0x7f8fdc0fd410 2024-04-06T14:17:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.037+0000 7f8fc37fe700 1 -- 172.21.15.32:0/504397798 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f8fcc0123c0 con 0x7f8fdc0fd410 2024-04-06T14:17:20.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.038+0000 7f8fc37fe700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f8fc406aed0 0x7f8fc406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:20.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.038+0000 7f8fc37fe700 1 -- 172.21.15.32:0/504397798 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f8fcc08a5f0 con 0x7f8fdc0fd410 2024-04-06T14:17:20.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.039+0000 7f8fda7fc700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f8fc406aed0 0x7f8fc406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:20.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.039+0000 7f8fda7fc700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f8fc406aed0 0x7f8fc406d380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0004600 tx=0x7f8fd000b040 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:20.047 INFO:tasks.workunit.client.0.smithi032.stdout:2/889: mkdir d0/d30/d39/d59/d67/d74/d78/d8c/d12b 0 2024-04-06T14:17:20.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.044+0000 7f8fc37fe700 1 -- 172.21.15.32:0/504397798 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f8fcc058590 con 0x7f8fdc0fd410 2024-04-06T14:17:20.048 INFO:tasks.workunit.client.0.smithi032.stdout:0/780: creat d3/da/d25/d101/f109 x:0 0 0 2024-04-06T14:17:20.048 INFO:tasks.workunit.client.1.smithi086.stdout:8/822: mknod d0/d71/d22/d65/d6e/d98/c121 0 2024-04-06T14:17:20.049 INFO:tasks.workunit.client.1.smithi086.stdout:1/786: dwrite db/d11/d1a/d2a/f43 [4194304,4194304] 0 2024-04-06T14:17:20.049 INFO:tasks.workunit.client.1.smithi086.stdout:6/896: dwrite d6/d9/d5c/f111 [0,4194304] 0 2024-04-06T14:17:20.049 INFO:tasks.workunit.client.1.smithi086.stdout:2/781: dwrite d0/df/d110/f107 [0,4194304] 0 2024-04-06T14:17:20.049 INFO:tasks.workunit.client.1.smithi086.stdout:6/897: write d6/d20/fb1 [4191470,114566] 0 2024-04-06T14:17:20.053 INFO:tasks.workunit.client.0.smithi032.stdout:9/956: rename d2/d15/f2b to d2/d15/d1e/d41/d6c/d88/de6/d11f/f134 0 2024-04-06T14:17:20.057 INFO:tasks.workunit.client.0.smithi032.stdout:0/781: write d3/f4 [822311,71443] 0 2024-04-06T14:17:20.058 INFO:tasks.workunit.client.0.smithi032.stdout:5/922: unlink d16/d18/d33/d45/d52/d39/l91 0 2024-04-06T14:17:20.059 INFO:tasks.workunit.client.1.smithi086.stdout:1/787: truncate fa 498311 0 2024-04-06T14:17:20.059 INFO:tasks.workunit.client.1.smithi086.stdout:8/823: unlink d0/d5a/l10f 0 2024-04-06T14:17:20.060 INFO:tasks.workunit.client.0.smithi032.stdout:4/937: symlink d0/df/d23/d31/d50/d57/l140 0 2024-04-06T14:17:20.061 INFO:tasks.workunit.client.0.smithi032.stdout:0/782: dread - d3/dc2/dd8/ff2 zero size 2024-04-06T14:17:20.065 INFO:tasks.workunit.client.0.smithi032.stdout:4/938: write d0/d82/daf/f12b [639085,5395] 0 2024-04-06T14:17:20.065 INFO:tasks.workunit.client.1.smithi086.stdout:4/601: dread d0/f10 [0,4194304] 0 2024-04-06T14:17:20.066 INFO:tasks.workunit.client.1.smithi086.stdout:1/788: truncate db/d11/d1a/d2a/d9a/d68/fad 597128 0 2024-04-06T14:17:20.069 INFO:tasks.workunit.client.1.smithi086.stdout:1/789: mkdir db/d13/d55/dfa 0 2024-04-06T14:17:20.069 INFO:tasks.workunit.client.1.smithi086.stdout:4/602: creat d0/d2c/d38/d54/d90/fbd x:0 0 0 2024-04-06T14:17:20.070 INFO:tasks.workunit.client.0.smithi032.stdout:5/923: rename d16/d18/d33/d45/d52/d39/c66 to d16/c140 0 2024-04-06T14:17:20.070 INFO:tasks.workunit.client.0.smithi032.stdout:5/924: readlink l8 0 2024-04-06T14:17:20.072 INFO:tasks.workunit.client.0.smithi032.stdout:4/939: creat d0/df/d23/d31/d50/d57/d5b/d9c/dc1/f141 x:0 0 0 2024-04-06T14:17:20.073 INFO:tasks.workunit.client.1.smithi086.stdout:4/603: creat d0/d15/d2b/fbe x:0 0 0 2024-04-06T14:17:20.074 INFO:tasks.workunit.client.0.smithi032.stdout:9/957: rename d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/d4a/d8f/fa2 to d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/f135 0 2024-04-06T14:17:20.076 INFO:tasks.workunit.client.1.smithi086.stdout:4/604: write d0/d15/d20/f44 [3712074,12545] 0 2024-04-06T14:17:20.077 INFO:tasks.workunit.client.0.smithi032.stdout:4/940: truncate d0/df/d2b/d46/ff3 1231299 0 2024-04-06T14:17:20.077 INFO:tasks.workunit.client.1.smithi086.stdout:8/824: write d0/db/fac [1419343,7776] 0 2024-04-06T14:17:20.080 INFO:tasks.workunit.client.1.smithi086.stdout:4/605: chown d0/d2c/d30/d24/da3/c67 1266 1 2024-04-06T14:17:20.081 INFO:tasks.workunit.client.0.smithi032.stdout:2/890: dwrite d0/d30/d39/d59/d67/d74/d78/d57/f7d [0,4194304] 0 2024-04-06T14:17:20.081 INFO:tasks.workunit.client.0.smithi032.stdout:0/783: rename d3/da/d25/d2b/d4a/d79/da8/dfb to d3/da/d25/d2b/d4a/d86/d10a 0 2024-04-06T14:17:20.081 INFO:tasks.workunit.client.1.smithi086.stdout:6/898: dwrite d6/d20/f46 [0,4194304] 0 2024-04-06T14:17:20.085 INFO:tasks.workunit.client.1.smithi086.stdout:8/825: creat d0/d71/d16/d39/d90/f122 x:0 0 0 2024-04-06T14:17:20.089 INFO:tasks.workunit.client.0.smithi032.stdout:5/925: write d16/fab [2783210,126272] 0 2024-04-06T14:17:20.091 INFO:tasks.workunit.client.1.smithi086.stdout:4/606: dwrite d0/d2c/d30/d24/d71/fa0 [0,4194304] 0 2024-04-06T14:17:20.096 INFO:tasks.workunit.client.1.smithi086.stdout:6/899: unlink d6/d9/db/f2f 0 2024-04-06T14:17:20.096 INFO:tasks.workunit.client.1.smithi086.stdout:4/607: chown d0/d2c/d30/d24/d71/fa0 1 1 2024-04-06T14:17:20.100 INFO:tasks.workunit.client.1.smithi086.stdout:5/625: sync 2024-04-06T14:17:20.104 INFO:tasks.workunit.client.0.smithi032.stdout:4/941: write d0/df/d2b/d46/d58/fdc [731487,51050] 0 2024-04-06T14:17:20.104 INFO:tasks.workunit.client.0.smithi032.stdout:5/926: write d16/d58/f5f [556011,6770] 0 2024-04-06T14:17:20.106 INFO:tasks.workunit.client.0.smithi032.stdout:2/891: getdents d0/d30/d39/d59/d67/d74/d78/d57 0 2024-04-06T14:17:20.106 INFO:tasks.workunit.client.1.smithi086.stdout:2/782: dread d0/d20/d27/d2b/d35/d36/fa2 [0,4194304] 0 2024-04-06T14:17:20.108 INFO:tasks.workunit.client.1.smithi086.stdout:4/608: read d0/d2c/f81 [353239,80818] 0 2024-04-06T14:17:20.108 INFO:tasks.workunit.client.0.smithi032.stdout:2/892: readlink d0/d6e/l92 0 2024-04-06T14:17:20.109 INFO:tasks.workunit.client.1.smithi086.stdout:5/626: creat d0/d16/d18/d2a/d50/d66/fcd x:0 0 0 2024-04-06T14:17:20.109 INFO:tasks.workunit.client.1.smithi086.stdout:4/609: chown d0/d2c/d38/l4f 18830 1 2024-04-06T14:17:20.109 INFO:tasks.workunit.client.1.smithi086.stdout:8/826: dwrite d0/d71/d16/d52/fc8 [0,4194304] 0 2024-04-06T14:17:20.110 INFO:tasks.workunit.client.0.smithi032.stdout:0/784: dread d3/da/d25/f2a [0,4194304] 0 2024-04-06T14:17:20.111 INFO:tasks.workunit.client.1.smithi086.stdout:2/783: symlink d0/d20/d27/l117 0 2024-04-06T14:17:20.111 INFO:tasks.workunit.client.0.smithi032.stdout:5/927: mkdir d16/d58/d54/d94/d141 0 2024-04-06T14:17:20.114 INFO:tasks.workunit.client.0.smithi032.stdout:9/958: dwrite d2/d15/d1e/d65/d83/f120 [4194304,4194304] 0 2024-04-06T14:17:20.115 INFO:tasks.workunit.client.1.smithi086.stdout:6/900: dwrite d6/d13/d1a/f116 [0,4194304] 0 2024-04-06T14:17:20.116 INFO:tasks.workunit.client.0.smithi032.stdout:0/785: write d3/dc2/fda [171276,81873] 0 2024-04-06T14:17:20.121 INFO:tasks.workunit.client.1.smithi086.stdout:1/790: dread db/d11/d24/fba [0,4194304] 0 2024-04-06T14:17:20.122 INFO:tasks.workunit.client.1.smithi086.stdout:4/610: chown d0/de/c29 801 1 2024-04-06T14:17:20.124 INFO:tasks.workunit.client.0.smithi032.stdout:5/928: creat d16/d18/d33/d35/d88/dbd/dc5/dec/f142 x:0 0 0 2024-04-06T14:17:20.124 INFO:tasks.workunit.client.1.smithi086.stdout:6/901: truncate d6/d13/d42/fea 938947 0 2024-04-06T14:17:20.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:19 smithi086 ceph-mon[47925]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:20.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:19 smithi086 ceph-mon[47925]: pgmap v27: 65 pgs: 65 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 117 MiB/s rd, 289 MiB/s wr, 645 op/s 2024-04-06T14:17:20.128 INFO:tasks.workunit.client.1.smithi086.stdout:8/827: mkdir d0/d3b/d79/dc6/d123 0 2024-04-06T14:17:20.133 INFO:tasks.workunit.client.1.smithi086.stdout:2/784: dread d0/d20/d27/d2b/d35/d36/fa2 [0,4194304] 0 2024-04-06T14:17:20.133 INFO:tasks.workunit.client.0.smithi032.stdout:3/962: sync 2024-04-06T14:17:20.133 INFO:tasks.workunit.client.0.smithi032.stdout:5/929: link d16/d18/d33/d45/d52/fa2 d16/d18/d33/d45/d52/f143 0 2024-04-06T14:17:20.134 INFO:tasks.workunit.client.1.smithi086.stdout:4/611: dwrite d0/d2c/d30/d24/d50/d79/d89/fa6 [0,4194304] 0 2024-04-06T14:17:20.135 INFO:tasks.workunit.client.0.smithi032.stdout:4/942: write d0/df/d1f/d42/d45/f8c [4458179,78102] 0 2024-04-06T14:17:20.135 INFO:tasks.workunit.client.1.smithi086.stdout:8/828: mknod d0/d71/c124 0 2024-04-06T14:17:20.135 INFO:tasks.workunit.client.1.smithi086.stdout:1/791: link db/d11/d87/ca2 db/cfb 0 2024-04-06T14:17:20.137 INFO:tasks.workunit.client.0.smithi032.stdout:2/893: dread d0/d30/f4d [0,4194304] 0 2024-04-06T14:17:20.141 INFO:tasks.workunit.client.1.smithi086.stdout:8/829: truncate d0/db/f5e 601246 0 2024-04-06T14:17:20.142 INFO:tasks.workunit.client.1.smithi086.stdout:6/902: write d6/d9/fe8 [999975,18876] 0 2024-04-06T14:17:20.143 INFO:tasks.workunit.client.0.smithi032.stdout:0/786: dread d3/da/d25/d2b/f3d [0,4194304] 0 2024-04-06T14:17:20.144 INFO:tasks.workunit.client.1.smithi086.stdout:2/785: creat d0/d20/d27/d2b/d35/d36/d4d/d59/f118 x:0 0 0 2024-04-06T14:17:20.144 INFO:tasks.workunit.client.1.smithi086.stdout:6/903: readlink d6/d9/d43/lb4 0 2024-04-06T14:17:20.145 INFO:tasks.workunit.client.0.smithi032.stdout:5/930: unlink d16/d18/d33/d35/la7 0 2024-04-06T14:17:20.146 INFO:tasks.workunit.client.1.smithi086.stdout:5/627: dwrite d0/d16/d18/d2a/d50/f73 [0,4194304] 0 2024-04-06T14:17:20.148 INFO:tasks.workunit.client.1.smithi086.stdout:6/904: chown d6/d13/d1a/d22/l7e 15416 1 2024-04-06T14:17:20.153 INFO:tasks.workunit.client.1.smithi086.stdout:1/792: dwrite db/d11/d87/fb6 [0,4194304] 0 2024-04-06T14:17:20.153 INFO:tasks.workunit.client.0.smithi032.stdout:4/943: write d0/df/d1f/d2e/f94 [970667,75662] 0 2024-04-06T14:17:20.155 INFO:tasks.workunit.client.0.smithi032.stdout:5/931: write d16/d58/d54/d74/f12d [988195,103444] 0 2024-04-06T14:17:20.158 INFO:tasks.workunit.client.1.smithi086.stdout:5/628: chown d0/d16/d18/d21/d23/l37 16777395 1 2024-04-06T14:17:20.159 INFO:tasks.workunit.client.0.smithi032.stdout:3/963: dwrite d3/d16/ddd/df7/d147/f106 [0,4194304] 0 2024-04-06T14:17:20.162 INFO:tasks.workunit.client.0.smithi032.stdout:9/959: dwrite d2/d15/d1e/d65/d83/f3b [0,4194304] 0 2024-04-06T14:17:20.164 INFO:tasks.workunit.client.0.smithi032.stdout:4/944: creat d0/df/d1f/d5a/f142 x:0 0 0 2024-04-06T14:17:20.165 INFO:tasks.workunit.client.1.smithi086.stdout:1/793: unlink db/d13/d55/df1/cc9 0 2024-04-06T14:17:20.167 INFO:tasks.workunit.client.0.smithi032.stdout:0/787: dread d3/da/d25/d2b/d4a/d79/d97/de0/df5/f4e [0,4194304] 0 2024-04-06T14:17:20.178 INFO:tasks.workunit.client.0.smithi032.stdout:9/960: chown d2/d15/d1e/d65/d83/d31/c34 181290 1 2024-04-06T14:17:20.179 INFO:tasks.workunit.client.1.smithi086.stdout:5/629: write d0/d17/fac [906477,82202] 0 2024-04-06T14:17:20.180 INFO:tasks.workunit.client.0.smithi032.stdout:3/964: rmdir d3/d16/ddd/df7/d147/d7d/d9c/dad/dc5 39 2024-04-06T14:17:20.184 INFO:tasks.workunit.client.0.smithi032.stdout:2/894: dwrite d0/d30/d39/d59/fcc [0,4194304] 0 2024-04-06T14:17:20.185 INFO:tasks.workunit.client.1.smithi086.stdout:1/794: dwrite db/d13/d71/fd0 [0,4194304] 0 2024-04-06T14:17:20.185 INFO:tasks.workunit.client.0.smithi032.stdout:5/932: getdents d16/d58/d54/d94/db2/dcb/d11b 0 2024-04-06T14:17:20.186 INFO:tasks.workunit.client.1.smithi086.stdout:5/630: symlink d0/d16/d18/d21/d23/d28/d94/d52/d7c/lce 0 2024-04-06T14:17:20.188 INFO:tasks.workunit.client.0.smithi032.stdout:0/788: mknod d3/da/c10b 0 2024-04-06T14:17:20.188 INFO:tasks.workunit.client.0.smithi032.stdout:4/945: write d0/d82/fe3 [1043142,90727] 0 2024-04-06T14:17:20.189 INFO:tasks.workunit.client.1.smithi086.stdout:1/795: fdatasync db/d11/d1a/f4b 0 2024-04-06T14:17:20.190 INFO:tasks.workunit.client.0.smithi032.stdout:9/961: mknod d2/d15/d1e/d65/d9c/c136 0 2024-04-06T14:17:20.191 INFO:tasks.workunit.client.0.smithi032.stdout:3/965: mkdir d3/d16/ddd/df7/d147/d7d/dc6/df0/d16b 0 2024-04-06T14:17:20.191 INFO:tasks.workunit.client.1.smithi086.stdout:5/631: rename d0/d16/d18/d21/d23/f77 to d0/d16/d18/d35/fcf 0 2024-04-06T14:17:20.193 INFO:tasks.workunit.client.1.smithi086.stdout:6/905: dread d6/d9/db/f70 [0,4194304] 0 2024-04-06T14:17:20.198 INFO:tasks.workunit.client.1.smithi086.stdout:1/796: creat db/d11/d1a/d2a/d9a/d68/d6e/d81/ffc x:0 0 0 2024-04-06T14:17:20.199 INFO:tasks.workunit.client.1.smithi086.stdout:6/906: mknod d6/d20/de0/c137 0 2024-04-06T14:17:20.200 INFO:tasks.workunit.client.0.smithi032.stdout:2/895: chown d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/lc3 51562437 1 2024-04-06T14:17:20.200 INFO:tasks.workunit.client.0.smithi032.stdout:4/946: rmdir d0/df/d23/d31/d50/d57/d8a/db5 39 2024-04-06T14:17:20.202 INFO:tasks.workunit.client.0.smithi032.stdout:5/933: link d16/d58/f6d d16/d18/d33/d35/d4d/f144 0 2024-04-06T14:17:20.204 INFO:tasks.workunit.client.1.smithi086.stdout:1/797: dread db/d11/d87/fb6 [0,4194304] 0 2024-04-06T14:17:20.206 INFO:tasks.workunit.client.0.smithi032.stdout:3/966: dwrite d3/d16/ddd/df7/d147/d89/fa2 [0,4194304] 0 2024-04-06T14:17:20.206 INFO:tasks.workunit.client.0.smithi032.stdout:0/789: getdents d3/da/d25/d101 0 2024-04-06T14:17:20.207 INFO:tasks.workunit.client.1.smithi086.stdout:6/907: rename d6/d13/d1a/da0/d114/db2/dee/lfc to d6/d13/d42/d52/l138 0 2024-04-06T14:17:20.207 INFO:tasks.workunit.client.0.smithi032.stdout:0/790: stat d3/da/d25/d36/f51 0 2024-04-06T14:17:20.209 INFO:tasks.workunit.client.1.smithi086.stdout:5/632: getdents d0/d16/d18/d21/d23/d28/d94/d52 0 2024-04-06T14:17:20.211 INFO:tasks.workunit.client.1.smithi086.stdout:1/798: creat db/d11/d1a/d2a/d9a/d78/ffd x:0 0 0 2024-04-06T14:17:20.211 INFO:tasks.workunit.client.0.smithi032.stdout:3/967: unlink d3/d16/ddd/df7/d147/d7d/c9d 0 2024-04-06T14:17:20.212 INFO:tasks.workunit.client.0.smithi032.stdout:2/896: creat d0/d4e/f12c x:0 0 0 2024-04-06T14:17:20.213 INFO:tasks.workunit.client.1.smithi086.stdout:6/908: write d6/d9/db/f133 [9507747,20796] 0 2024-04-06T14:17:20.213 INFO:tasks.workunit.client.0.smithi032.stdout:4/947: write d0/df/d23/d31/fd6 [245729,46465] 0 2024-04-06T14:17:20.215 INFO:tasks.workunit.client.0.smithi032.stdout:0/791: mkdir d3/da/d25/d2b/d4a/d86/d10c 0 2024-04-06T14:17:20.215 INFO:tasks.workunit.client.0.smithi032.stdout:9/962: dread d2/d15/d1e/d65/d83/d31/d39/f58 [0,4194304] 0 2024-04-06T14:17:20.216 INFO:tasks.workunit.client.1.smithi086.stdout:6/909: unlink d6/d20/de0/c137 0 2024-04-06T14:17:20.219 INFO:tasks.workunit.client.1.smithi086.stdout:5/633: dwrite d0/d16/f39 [0,4194304] 0 2024-04-06T14:17:20.220 INFO:tasks.workunit.client.0.smithi032.stdout:4/948: dread - d0/df/d23/d31/d50/d57/d5b/d9c/dc1/f132 zero size 2024-04-06T14:17:20.220 INFO:tasks.workunit.client.0.smithi032.stdout:3/968: creat d3/d16/ddd/df7/d147/d89/dcb/d155/f16c x:0 0 0 2024-04-06T14:17:20.220 INFO:tasks.workunit.client.0.smithi032.stdout:0/792: creat d3/da/d25/d36/f10d x:0 0 0 2024-04-06T14:17:20.222 INFO:tasks.workunit.client.0.smithi032.stdout:4/949: readlink d0/d82/ldf 0 2024-04-06T14:17:20.223 INFO:tasks.workunit.client.1.smithi086.stdout:6/910: creat d6/d13/d1a/da0/d11b/d45/d8a/de3/f139 x:0 0 0 2024-04-06T14:17:20.224 INFO:tasks.workunit.client.1.smithi086.stdout:6/911: truncate d6/d13/d5f/f117 741305 0 2024-04-06T14:17:20.225 INFO:tasks.workunit.client.0.smithi032.stdout:0/793: mknod d3/da/d25/d2b/d80/db3/dcd/c10e 0 2024-04-06T14:17:20.227 INFO:tasks.workunit.client.0.smithi032.stdout:4/950: truncate d0/df/d1f/d42/d45/f8c 4698139 0 2024-04-06T14:17:20.228 INFO:tasks.workunit.client.1.smithi086.stdout:6/912: mkdir d6/d13/d1a/da0/d11b/d45/d8a/de3/d13a 0 2024-04-06T14:17:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:19 smithi032 ceph-mon[39730]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:19 smithi032 ceph-mon[39730]: pgmap v27: 65 pgs: 65 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 117 MiB/s rd, 289 MiB/s wr, 645 op/s 2024-04-06T14:17:20.231 INFO:tasks.workunit.client.0.smithi032.stdout:2/897: dread d0/d6e/f86 [0,4194304] 0 2024-04-06T14:17:20.232 INFO:tasks.workunit.client.1.smithi086.stdout:6/913: mkdir d6/d13/d1a/da0/d11b/d45/d4a/d13b 0 2024-04-06T14:17:20.233 INFO:tasks.workunit.client.0.smithi032.stdout:0/794: creat d3/da/d25/d2b/d4a/d86/d10a/dfe/f10f x:0 0 0 2024-04-06T14:17:20.234 INFO:tasks.workunit.client.1.smithi086.stdout:6/914: read - d6/d13/d1a/da0/d11b/d45/d8a/f11e zero size 2024-04-06T14:17:20.235 INFO:tasks.workunit.client.1.smithi086.stdout:5/634: truncate d0/d16/d18/d35/f87 2792335 0 2024-04-06T14:17:20.237 INFO:tasks.workunit.client.0.smithi032.stdout:2/898: fsync d0/d30/d39/d59/d67/d74/d78/d8c/f9e 0 2024-04-06T14:17:20.239 INFO:tasks.workunit.client.1.smithi086.stdout:1/799: dwrite db/d13/d71/f7f [0,4194304] 0 2024-04-06T14:17:20.242 INFO:tasks.workunit.client.0.smithi032.stdout:4/951: dwrite d0/df/d23/d31/d50/d57/d5b/d9c/dc8/f139 [0,4194304] 0 2024-04-06T14:17:20.242 INFO:tasks.workunit.client.1.smithi086.stdout:5/635: rename d0/d16/l97 to d0/d16/d18/d35/ld0 0 2024-04-06T14:17:20.244 INFO:tasks.workunit.client.0.smithi032.stdout:2/899: rename d0/f24 to d0/d6e/dae/db7/f12d 0 2024-04-06T14:17:20.247 INFO:tasks.workunit.client.1.smithi086.stdout:1/800: creat db/d11/ffe x:0 0 0 2024-04-06T14:17:20.249 INFO:tasks.workunit.client.1.smithi086.stdout:6/915: rename d6/d13/d42/f7b to d6/d13/d1a/da0/d11b/d45/da6/f13c 0 2024-04-06T14:17:20.250 INFO:tasks.workunit.client.1.smithi086.stdout:5/636: dwrite d0/d16/d18/d2a/d50/d66/fca [0,4194304] 0 2024-04-06T14:17:20.251 INFO:tasks.workunit.client.0.smithi032.stdout:9/963: dread d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/fab [0,4194304] 0 2024-04-06T14:17:20.252 INFO:tasks.workunit.client.1.smithi086.stdout:1/801: mknod db/d13/d71/cff 0 2024-04-06T14:17:20.252 INFO:tasks.workunit.client.0.smithi032.stdout:9/964: chown d2/d15/d1e/d65/d83/d31/d39/da6/fb6 44 1 2024-04-06T14:17:20.255 INFO:tasks.workunit.client.1.smithi086.stdout:6/916: creat d6/d13/d1a/da0/d11b/d45/f13d x:0 0 0 2024-04-06T14:17:20.256 INFO:tasks.workunit.client.1.smithi086.stdout:5/637: dwrite d0/d16/d18/fb5 [0,4194304] 0 2024-04-06T14:17:20.257 INFO:tasks.workunit.client.1.smithi086.stdout:5/638: read - d0/d16/d18/d2a/d50/fc6 zero size 2024-04-06T14:17:20.259 INFO:tasks.workunit.client.1.smithi086.stdout:7/723: sync 2024-04-06T14:17:20.266 INFO:tasks.workunit.client.1.smithi086.stdout:7/724: truncate d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/dd2/fe0 681211 0 2024-04-06T14:17:20.268 INFO:tasks.workunit.client.0.smithi032.stdout:0/795: link d3/da/d25/d2b/d4a/d79/d97/de0/df5/f4e d3/da/d25/d2b/d4a/d86/d10a/dfe/f110 0 2024-04-06T14:17:20.271 INFO:tasks.workunit.client.0.smithi032.stdout:2/900: symlink d0/d30/d39/d59/d67/d74/d78/d8c/l12e 0 2024-04-06T14:17:20.273 INFO:tasks.workunit.client.0.smithi032.stdout:4/952: rename d0/d82/d4b/d13b to d0/df/d1f/d42/d143 0 2024-04-06T14:17:20.274 INFO:tasks.workunit.client.0.smithi032.stdout:0/796: creat d3/dc2/dd8/f111 x:0 0 0 2024-04-06T14:17:20.275 INFO:tasks.workunit.client.1.smithi086.stdout:7/725: mknod d7/d10/d11/d1c/d24/d4d/cf5 0 2024-04-06T14:17:20.275 INFO:tasks.workunit.client.0.smithi032.stdout:4/953: read - d0/df/d1f/d5a/f142 zero size 2024-04-06T14:17:20.275 INFO:tasks.workunit.client.0.smithi032.stdout:2/901: creat d0/d6e/d8d/f12f x:0 0 0 2024-04-06T14:17:20.280 INFO:tasks.workunit.client.0.smithi032.stdout:2/902: rename d0/d30/d39/d59/d67/d74/d78/dc2/f12a to d0/d4e/dc6/f130 0 2024-04-06T14:17:20.281 INFO:tasks.workunit.client.0.smithi032.stdout:4/954: read d0/f1 [3800928,107295] 0 2024-04-06T14:17:20.283 INFO:tasks.workunit.client.0.smithi032.stdout:0/797: dwrite d3/da/d25/d36/f10d [0,4194304] 0 2024-04-06T14:17:20.283 INFO:tasks.workunit.client.0.smithi032.stdout:2/903: fsync d0/d4e/de9/f11a 0 2024-04-06T14:17:20.286 INFO:tasks.workunit.client.0.smithi032.stdout:2/904: write d0/d30/d39/d59/d67/d74/d78/fef [548458,130040] 0 2024-04-06T14:17:20.289 INFO:tasks.workunit.client.0.smithi032.stdout:0/798: dread d3/da/d25/d2b/d4a/fb9 [0,4194304] 0 2024-04-06T14:17:20.291 INFO:tasks.workunit.client.0.smithi032.stdout:2/905: truncate d0/d5a/f2a 562288 0 2024-04-06T14:17:20.294 INFO:tasks.workunit.client.0.smithi032.stdout:2/906: creat d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102/f131 x:0 0 0 2024-04-06T14:17:20.296 INFO:tasks.workunit.client.0.smithi032.stdout:2/907: fsync d0/d30/d39/d59/d67/d74/d78/d57/f96 0 2024-04-06T14:17:20.297 INFO:tasks.workunit.client.0.smithi032.stdout:2/908: fsync d0/d30/d39/d59/d67/d74/d78/dc2/ff5 0 2024-04-06T14:17:20.298 INFO:tasks.workunit.client.0.smithi032.stdout:2/909: chown d0/d30/d39/d59/d67/ca4 0 1 2024-04-06T14:17:20.299 INFO:tasks.workunit.client.0.smithi032.stdout:2/910: chown d0/d30/d39/d59/d67/d74/d78/d57/c10c 6901553 1 2024-04-06T14:17:20.300 INFO:tasks.workunit.client.0.smithi032.stdout:2/911: readlink d0/ld 0 2024-04-06T14:17:20.302 INFO:tasks.workunit.client.0.smithi032.stdout:2/912: truncate d0/db2/dfc/f40 1066321 0 2024-04-06T14:17:20.304 INFO:tasks.workunit.client.0.smithi032.stdout:2/913: fsync d0/d30/fc8 0 2024-04-06T14:17:20.305 INFO:tasks.workunit.client.0.smithi032.stdout:2/914: fsync d0/d4e/f12c 0 2024-04-06T14:17:20.306 INFO:tasks.workunit.client.0.smithi032.stdout:9/965: truncate d2/d15/d1e/f6d 5276396 0 2024-04-06T14:17:20.307 INFO:tasks.workunit.client.0.smithi032.stdout:2/915: readlink d0/d5a/l3b 0 2024-04-06T14:17:20.308 INFO:tasks.workunit.client.1.smithi086.stdout:5/639: dread d0/d16/d18/d35/f4d [0,4194304] 0 2024-04-06T14:17:20.309 INFO:tasks.workunit.client.0.smithi032.stdout:9/966: symlink d2/d15/d1e/d65/d83/d7e/da7/l137 0 2024-04-06T14:17:20.310 INFO:tasks.workunit.client.0.smithi032.stdout:2/916: fdatasync d0/d30/d39/d59/d67/d74/d78/d57/d7c/de6/f122 0 2024-04-06T14:17:20.312 INFO:tasks.workunit.client.0.smithi032.stdout:9/967: creat d2/dcd/d7c/f138 x:0 0 0 2024-04-06T14:17:20.312 INFO:tasks.workunit.client.1.smithi086.stdout:5/640: dread d0/d16/d18/d35/f4d [0,4194304] 0 2024-04-06T14:17:20.317 INFO:tasks.workunit.client.1.smithi086.stdout:5/641: symlink d0/d16/d18/d35/d9e/dad/ld1 0 2024-04-06T14:17:20.318 INFO:tasks.workunit.client.0.smithi032.stdout:2/917: getdents d0/d5a/d6a/dcb 0 2024-04-06T14:17:20.319 INFO:tasks.workunit.client.1.smithi086.stdout:5/642: chown d0/d16/d71/d89/lb6 0 1 2024-04-06T14:17:20.319 INFO:tasks.workunit.client.1.smithi086.stdout:7/726: write d7/d10/d11/d1c/d24/dc8/fec [3025914,87670] 0 2024-04-06T14:17:20.323 INFO:tasks.workunit.client.0.smithi032.stdout:9/968: write d2/f4 [1258297,113315] 0 2024-04-06T14:17:20.324 INFO:tasks.workunit.client.1.smithi086.stdout:5/643: creat d0/d16/d18/d35/d9e/fd2 x:0 0 0 2024-04-06T14:17:20.327 INFO:tasks.workunit.client.1.smithi086.stdout:6/917: dwrite d6/f2b [4194304,4194304] 0 2024-04-06T14:17:20.327 INFO:tasks.workunit.client.1.smithi086.stdout:5/644: stat d0/d16/d18/d2a/d50/f93 0 2024-04-06T14:17:20.329 INFO:tasks.workunit.client.0.smithi032.stdout:9/969: truncate d2/d15/d1e/d65/d6b/d92/d11d/d5f/f62 3242241 0 2024-04-06T14:17:20.332 INFO:tasks.workunit.client.1.smithi086.stdout:7/727: rename d7/d10/d11/d1c/d24/d4d/daf/db5/d7a/c7c to d7/d10/d11/d1c/d24/d4d/d5c/cf6 0 2024-04-06T14:17:20.334 INFO:tasks.workunit.client.0.smithi032.stdout:2/918: dwrite d0/d30/d39/d59/d67/d74/d78/fce [0,4194304] 0 2024-04-06T14:17:20.334 INFO:tasks.workunit.client.1.smithi086.stdout:7/728: write d7/d10/d11/d80/fc9 [474933,86986] 0 2024-04-06T14:17:20.334 INFO:tasks.workunit.client.1.smithi086.stdout:5/645: creat d0/d16/fd3 x:0 0 0 2024-04-06T14:17:20.336 INFO:tasks.workunit.client.1.smithi086.stdout:5/646: read - d0/d16/d18/d21/d23/d28/d94/fae zero size 2024-04-06T14:17:20.338 INFO:tasks.workunit.client.1.smithi086.stdout:7/729: symlink d7/d10/d11/d21/db0/lf7 0 2024-04-06T14:17:20.339 INFO:tasks.workunit.client.0.smithi032.stdout:2/919: fsync d0/d6e/ff2 0 2024-04-06T14:17:20.340 INFO:tasks.workunit.client.1.smithi086.stdout:5/647: mknod d0/d16/d18/d2a/d50/cd4 0 2024-04-06T14:17:20.342 INFO:tasks.workunit.client.1.smithi086.stdout:6/918: rename d6/d13/d1a/da0/d114/db2/f10c to d6/d13/f13e 0 2024-04-06T14:17:20.343 INFO:tasks.workunit.client.1.smithi086.stdout:7/730: creat d7/d10/d92/ff8 x:0 0 0 2024-04-06T14:17:20.344 INFO:tasks.workunit.client.1.smithi086.stdout:5/648: symlink d0/d16/d18/d35/d9e/dad/ld5 0 2024-04-06T14:17:20.344 INFO:tasks.workunit.client.0.smithi032.stdout:9/970: dwrite d2/d15/d1e/d65/d6b/d92/d11d/d5f/fc8 [0,4194304] 0 2024-04-06T14:17:20.346 INFO:tasks.workunit.client.1.smithi086.stdout:6/919: chown d6/d13/d1a/da0/d114/db2/dee/c11c 2691888 1 2024-04-06T14:17:20.349 INFO:tasks.workunit.client.1.smithi086.stdout:6/920: rmdir d6/d9/d43 39 2024-04-06T14:17:20.355 INFO:tasks.workunit.client.0.smithi032.stdout:9/971: write d2/f2d [2967661,33425] 0 2024-04-06T14:17:20.356 INFO:tasks.workunit.client.0.smithi032.stdout:9/972: chown d2/d15/d1e/d65/d6b/d92/d11d/d59/l9d 260624279 1 2024-04-06T14:17:20.358 INFO:tasks.workunit.client.1.smithi086.stdout:6/921: dwrite d6/d13/d1a/d64/d85/fe6 [0,4194304] 0 2024-04-06T14:17:20.358 INFO:tasks.workunit.client.0.smithi032.stdout:9/973: stat d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/d4a/d10d 0 2024-04-06T14:17:20.359 INFO:tasks.workunit.client.1.smithi086.stdout:6/922: dread - d6/d13/d1a/d22/f11f zero size 2024-04-06T14:17:20.361 INFO:tasks.workunit.client.0.smithi032.stdout:9/974: truncate d2/d15/d1e/d65/d83/d7e/f1d 4876495 0 2024-04-06T14:17:20.364 INFO:tasks.workunit.client.1.smithi086.stdout:7/731: dwrite d7/d10/d11/d1c/d24/f7d [0,4194304] 0 2024-04-06T14:17:20.366 INFO:tasks.workunit.client.1.smithi086.stdout:6/923: link d6/d9/da9/fec d6/d13/d1a/d22/de4/f13f 0 2024-04-06T14:17:20.367 INFO:tasks.workunit.client.1.smithi086.stdout:7/732: chown d7/d10/d11/d1c/d24/d2c/c56 1321 1 2024-04-06T14:17:20.368 INFO:tasks.workunit.client.0.smithi032.stdout:9/975: readlink d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/d4a/l4e 0 2024-04-06T14:17:20.369 INFO:tasks.workunit.client.1.smithi086.stdout:5/649: write d0/d16/d18/d21/d23/d28/d94/d52/d7c/f8f [290625,125342] 0 2024-04-06T14:17:20.370 INFO:tasks.workunit.client.1.smithi086.stdout:5/650: chown d0/d16/d18/d2a/d50/d66 2117 1 2024-04-06T14:17:20.373 INFO:tasks.workunit.client.1.smithi086.stdout:6/924: symlink d6/d9/db/ddc/l140 0 2024-04-06T14:17:20.374 INFO:tasks.workunit.client.1.smithi086.stdout:5/651: write d0/f6 [1832379,112391] 0 2024-04-06T14:17:20.374 INFO:tasks.workunit.client.0.smithi032.stdout:9/976: creat d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/d4a/f139 x:0 0 0 2024-04-06T14:17:20.374 INFO:tasks.workunit.client.1.smithi086.stdout:5/652: readlink d0/d5e/l81 0 2024-04-06T14:17:20.375 INFO:tasks.workunit.client.0.smithi032.stdout:2/920: dread d0/db2/dfc/f48 [0,4194304] 0 2024-04-06T14:17:20.378 INFO:tasks.workunit.client.0.smithi032.stdout:9/977: mkdir d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d13a 0 2024-04-06T14:17:20.380 INFO:tasks.workunit.client.1.smithi086.stdout:5/653: rename d0/d16/d18/fb5 to d0/d16/d18/d21/fd6 0 2024-04-06T14:17:20.380 INFO:tasks.workunit.client.0.smithi032.stdout:2/921: mknod d0/d30/c132 0 2024-04-06T14:17:20.381 INFO:tasks.workunit.client.1.smithi086.stdout:5/654: chown d0/d16/f63 37 1 2024-04-06T14:17:20.381 INFO:tasks.workunit.client.1.smithi086.stdout:5/655: readlink d0/d16/l2b 0 2024-04-06T14:17:20.384 INFO:tasks.workunit.client.1.smithi086.stdout:5/656: write d0/d16/d71/fb9 [1328395,64704] 0 2024-04-06T14:17:20.386 INFO:tasks.workunit.client.1.smithi086.stdout:7/733: write d7/d10/d11/d1c/d24/d2c/f48 [1149015,76718] 0 2024-04-06T14:17:20.388 INFO:tasks.workunit.client.1.smithi086.stdout:7/734: write d7/d10/d11/d1c/d24/dc8/fde [237641,85750] 0 2024-04-06T14:17:20.390 INFO:tasks.workunit.client.1.smithi086.stdout:5/657: truncate d0/d16/f39 262177 0 2024-04-06T14:17:20.391 INFO:tasks.workunit.client.0.smithi032.stdout:2/922: dwrite d0/d30/d39/d59/d67/d74/d78/d57/f96 [0,4194304] 0 2024-04-06T14:17:20.392 INFO:tasks.workunit.client.1.smithi086.stdout:5/658: chown d0/d16/d18/d21/d23/d28/d94/c58 47099104 1 2024-04-06T14:17:20.392 INFO:tasks.workunit.client.1.smithi086.stdout:5/659: write d0/d16/d18/d35/fa2 [3268556,128082] 0 2024-04-06T14:17:20.393 INFO:tasks.workunit.client.1.smithi086.stdout:5/660: chown d0/l2c 37893929 1 2024-04-06T14:17:20.398 INFO:tasks.workunit.client.1.smithi086.stdout:0/804: sync 2024-04-06T14:17:20.398 INFO:tasks.workunit.client.1.smithi086.stdout:9/628: sync 2024-04-06T14:17:20.398 INFO:tasks.workunit.client.1.smithi086.stdout:3/808: sync 2024-04-06T14:17:20.405 INFO:tasks.workunit.client.1.smithi086.stdout:0/805: unlink d0/d15/d4e/da3/d44/d8e/d8f/c9d 0 2024-04-06T14:17:20.408 INFO:tasks.workunit.client.1.smithi086.stdout:9/629: fsync d5/d13/d1a/d5f/d88/d14/d90/f9b 0 2024-04-06T14:17:20.408 INFO:tasks.workunit.client.1.smithi086.stdout:3/809: write d5/dea/dd2/ddb/de6/d10b/fc5 [207244,88074] 0 2024-04-06T14:17:20.411 INFO:tasks.workunit.client.0.smithi032.stdout:9/978: dread d2/f2d [0,4194304] 0 2024-04-06T14:17:20.413 INFO:tasks.workunit.client.1.smithi086.stdout:9/630: mknod d5/cc6 0 2024-04-06T14:17:20.413 INFO:tasks.workunit.client.1.smithi086.stdout:0/806: getdents d0/d1/d2/d1a/d79/dac/db7/de6 0 2024-04-06T14:17:20.414 INFO:tasks.workunit.client.0.smithi032.stdout:2/923: dwrite d0/d30/d39/d59/d67/d74/d78/d57/d7c/f87 [0,4194304] 0 2024-04-06T14:17:20.415 INFO:tasks.workunit.client.0.smithi032.stdout:9/979: chown d2/d15/d1e/d65/d83/le3 0 1 2024-04-06T14:17:20.416 INFO:tasks.workunit.client.0.smithi032.stdout:2/924: fsync d0/d6e/d8d/f12f 0 2024-04-06T14:17:20.417 INFO:tasks.workunit.client.1.smithi086.stdout:3/810: dwrite d5/dea/dd2/ddb/de6/d10b/fc5 [0,4194304] 0 2024-04-06T14:17:20.418 INFO:tasks.workunit.client.1.smithi086.stdout:9/631: symlink d5/d13/d47/d55/d87/lc7 0 2024-04-06T14:17:20.421 INFO:tasks.workunit.client.1.smithi086.stdout:0/807: unlink d0/d1/d2/l3 0 2024-04-06T14:17:20.423 INFO:tasks.workunit.client.0.smithi032.stdout:2/925: creat d0/d6e/dae/db7/f133 x:0 0 0 2024-04-06T14:17:20.427 INFO:tasks.workunit.client.1.smithi086.stdout:3/811: dwrite d5/d36/d5a/d6d/d72/d64/fdc [0,4194304] 0 2024-04-06T14:17:20.429 INFO:tasks.workunit.client.1.smithi086.stdout:9/632: dwrite d5/d13/d47/d55/d87/fba [0,4194304] 0 2024-04-06T14:17:20.429 INFO:tasks.workunit.client.0.smithi032.stdout:9/980: truncate d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/f4b 1327438 0 2024-04-06T14:17:20.430 INFO:tasks.workunit.client.1.smithi086.stdout:3/812: chown d5/dea/dd2/ddb/de6/dee 65 1 2024-04-06T14:17:20.432 INFO:tasks.workunit.client.0.smithi032.stdout:2/926: dread d0/d4e/f53 [0,4194304] 0 2024-04-06T14:17:20.434 INFO:tasks.workunit.client.0.smithi032.stdout:9/981: mknod d2/d15/d1e/d65/d83/d31/d39/c13b 0 2024-04-06T14:17:20.435 INFO:tasks.workunit.client.1.smithi086.stdout:0/808: read d0/d15/d4e/da3/d46/f6d [130155,87423] 0 2024-04-06T14:17:20.435 INFO:tasks.workunit.client.1.smithi086.stdout:9/633: creat d5/d13/d1a/d5f/fc8 x:0 0 0 2024-04-06T14:17:20.437 INFO:tasks.workunit.client.1.smithi086.stdout:9/634: truncate d5/d13/d1a/d8f/fb1 211882 0 2024-04-06T14:17:20.437 INFO:tasks.workunit.client.0.smithi032.stdout:9/982: mknod d2/d15/d1e/d65/d6b/d92/d11d/d5f/c13c 0 2024-04-06T14:17:20.437 INFO:tasks.workunit.client.0.smithi032.stdout:2/927: dread d0/d30/d39/d59/d67/d74/d78/d57/f96 [4194304,4194304] 0 2024-04-06T14:17:20.438 INFO:tasks.workunit.client.1.smithi086.stdout:0/809: mknod d0/d15/d4e/da3/d46/c10a 0 2024-04-06T14:17:20.442 INFO:tasks.workunit.client.1.smithi086.stdout:0/810: creat d0/d15/d4e/d48/d89/f10b x:0 0 0 2024-04-06T14:17:20.445 INFO:tasks.workunit.client.0.smithi032.stdout:9/983: getdents d2/d15 0 2024-04-06T14:17:20.446 INFO:tasks.workunit.client.1.smithi086.stdout:0/811: dread d0/d15/d4e/da3/fe7 [0,4194304] 0 2024-04-06T14:17:20.447 INFO:tasks.workunit.client.1.smithi086.stdout:0/812: chown d0/d15/d4e/da3/d44/d8e 349728 1 2024-04-06T14:17:20.451 INFO:tasks.workunit.client.0.smithi032.stdout:9/984: dwrite d2/f4 [0,4194304] 0 2024-04-06T14:17:20.457 INFO:tasks.workunit.client.0.smithi032.stdout:9/985: dread d2/d15/d1e/d65/d83/d7e/f3e [0,4194304] 0 2024-04-06T14:17:20.462 INFO:tasks.workunit.client.0.smithi032.stdout:2/928: dwrite d0/d30/f43 [0,4194304] 0 2024-04-06T14:17:20.463 INFO:tasks.workunit.client.1.smithi086.stdout:9/635: unlink d5/d13/d1a/d5f/l8b 0 2024-04-06T14:17:20.469 INFO:tasks.workunit.client.0.smithi032.stdout:2/929: creat d0/d30/d39/d59/d67/d74/d78/d8c/f134 x:0 0 0 2024-04-06T14:17:20.471 INFO:tasks.workunit.client.1.smithi086.stdout:9/636: mknod d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/cc9 0 2024-04-06T14:17:20.477 INFO:tasks.workunit.client.0.smithi032.stdout:9/986: dwrite d2/ffc [0,4194304] 0 2024-04-06T14:17:20.479 INFO:tasks.workunit.client.1.smithi086.stdout:9/637: write d5/d13/d1a/d5f/d88/d14/f80 [2119856,128791] 0 2024-04-06T14:17:20.479 INFO:tasks.workunit.client.0.smithi032.stdout:2/930: dwrite d0/f23 [4194304,4194304] 0 2024-04-06T14:17:20.483 INFO:tasks.workunit.client.0.smithi032.stdout:2/931: read d0/d30/d39/d59/d67/d74/d78/d57/d7c/f87 [2005038,56030] 0 2024-04-06T14:17:20.487 INFO:tasks.workunit.client.1.smithi086.stdout:9/638: dwrite d5/d13/d1a/d5f/d88/d14/d19/d1b/f45 [0,4194304] 0 2024-04-06T14:17:20.487 INFO:tasks.workunit.client.0.smithi032.stdout:9/987: rename d2/d15/d1e/d41/fcb to d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/d4a/d8f/f13d 0 2024-04-06T14:17:20.488 INFO:tasks.workunit.client.0.smithi032.stdout:2/932: fdatasync d0/d5a/f4c 0 2024-04-06T14:17:20.492 INFO:tasks.workunit.client.0.smithi032.stdout:2/933: write d0/f13 [8755857,22221] 0 2024-04-06T14:17:20.492 INFO:tasks.workunit.client.1.smithi086.stdout:9/639: creat d5/d13/d47/d55/fca x:0 0 0 2024-04-06T14:17:20.494 INFO:tasks.workunit.client.0.smithi032.stdout:9/988: getdents d2/d15/d1e/d41/d6c/d74/d104 0 2024-04-06T14:17:20.495 INFO:tasks.workunit.client.0.smithi032.stdout:2/934: getdents d0/d6e/dae/d120 0 2024-04-06T14:17:20.496 INFO:tasks.workunit.client.1.smithi086.stdout:9/640: unlink d5/d13/d1a/d5f/d88/d14/d19/d1b/l3d 0 2024-04-06T14:17:20.496 INFO:tasks.workunit.client.0.smithi032.stdout:9/989: chown d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/d4a/d8e/d10e/cfe 0 1 2024-04-06T14:17:20.497 INFO:tasks.workunit.client.0.smithi032.stdout:2/935: creat d0/d5a/d6a/f135 x:0 0 0 2024-04-06T14:17:20.499 INFO:tasks.workunit.client.0.smithi032.stdout:9/990: creat d2/d15/d1e/f13e x:0 0 0 2024-04-06T14:17:20.499 INFO:tasks.workunit.client.0.smithi032.stdout:2/936: fsync d0/d30/d39/d59/d67/d74/d78/d57/d7c/de6/f122 0 2024-04-06T14:17:20.501 INFO:tasks.workunit.client.0.smithi032.stdout:2/937: creat d0/d5a/d6a/f136 x:0 0 0 2024-04-06T14:17:20.501 INFO:tasks.workunit.client.0.smithi032.stdout:9/991: stat d2/d15/d1e/d65/d6b/d92/fb3 0 2024-04-06T14:17:20.501 INFO:tasks.workunit.client.0.smithi032.stdout:2/938: chown d0/l9 499 1 2024-04-06T14:17:20.504 INFO:tasks.workunit.client.0.smithi032.stdout:2/939: read d0/d30/d39/d59/d67/d74/d78/d57/f11e [3086894,128577] 0 2024-04-06T14:17:20.506 INFO:tasks.workunit.client.0.smithi032.stdout:2/940: mknod d0/db2/c137 0 2024-04-06T14:17:20.509 INFO:tasks.workunit.client.0.smithi032.stdout:2/941: getdents d0/d30/d39/d59/d67/d74/d78/d57/dd3 0 2024-04-06T14:17:20.512 INFO:tasks.workunit.client.0.smithi032.stdout:2/942: dread d0/d30/d39/d59/d67/d74/d78/fce [0,4194304] 0 2024-04-06T14:17:20.521 INFO:tasks.workunit.client.0.smithi032.stdout:5/934: sync 2024-04-06T14:17:20.521 INFO:tasks.workunit.client.1.smithi086.stdout:4/612: sync 2024-04-06T14:17:20.525 INFO:tasks.workunit.client.1.smithi086.stdout:4/613: link d0/d2c/d38/d54/d83/ca9 d0/d15/d53/dbb/cbf 0 2024-04-06T14:17:20.526 INFO:tasks.workunit.client.0.smithi032.stdout:2/943: dread d0/d4e/dc6/f130 [0,4194304] 0 2024-04-06T14:17:20.526 INFO:tasks.workunit.client.1.smithi086.stdout:4/614: truncate d0/d15/d20/f27 1169965 0 2024-04-06T14:17:20.527 INFO:tasks.workunit.client.0.smithi032.stdout:2/944: chown d0/l5 4044142 1 2024-04-06T14:17:20.527 INFO:tasks.workunit.client.0.smithi032.stdout:2/945: readlink d0/db2/dfc/l98 0 2024-04-06T14:17:20.529 INFO:tasks.workunit.client.0.smithi032.stdout:2/946: creat d0/d6e/dae/db7/f138 x:0 0 0 2024-04-06T14:17:20.531 INFO:tasks.workunit.client.0.smithi032.stdout:2/947: creat d0/d30/d39/d59/d67/d74/f139 x:0 0 0 2024-04-06T14:17:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.535+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8fc8005860 con 0x7f8fdc0fd410 2024-04-06T14:17:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.535+0000 7f8fc37fe700 1 -- 172.21.15.32:0/504397798 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7f8fcc058120 con 0x7f8fdc0fd410 2024-04-06T14:17:20.536 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:17:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:17:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:17:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:17:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:17:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:17:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 1 2024-04-06T14:17:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 13, 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 1 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:17:20.537 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:17:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f8fc406aed0 msgr2=0x7f8fc406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f8fc406aed0 0x7f8fc406d380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0004600 tx=0x7f8fd000b040 comp rx=0 tx=0).stop 2024-04-06T14:17:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fdc0fd410 msgr2=0x7f8fdc113de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fdc0fd410 0x7f8fdc113de0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00b0d0 tx=0x7f8fcc009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 shutdown_connections 2024-04-06T14:17:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fdc0fd410 0x7f8fdc113de0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f8fc406aed0 0x7f8fc406d380 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 --2- 172.21.15.32:0/504397798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fdc0fdd60 0x7f8fdc114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:20.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 >> 172.21.15.32:0/504397798 conn(0x7f8fdc0f6be0 msgr2=0x7f8fdc0f7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:20.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 shutdown_connections 2024-04-06T14:17:20.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.539+0000 7f8fe1270700 1 -- 172.21.15.32:0/504397798 wait complete. 2024-04-06T14:17:20.544 INFO:tasks.workunit.client.0.smithi032.stdout:2/948: dread d0/d5a/f41 [0,4194304] 0 2024-04-06T14:17:20.544 INFO:tasks.workunit.client.0.smithi032.stdout:9/992: truncate d2/dcd/f12a 7582742 0 2024-04-06T14:17:20.546 INFO:tasks.workunit.client.0.smithi032.stdout:2/949: mknod d0/d30/d39/d59/d67/d74/d78/d57/c13a 0 2024-04-06T14:17:20.546 INFO:tasks.workunit.client.0.smithi032.stdout:9/993: mkdir d2/d15/d1e/d41/d6c/d74/d104/d13f 0 2024-04-06T14:17:20.554 INFO:tasks.workunit.client.0.smithi032.stdout:2/950: truncate d0/d6e/f72 1189847 0 2024-04-06T14:17:20.555 INFO:tasks.workunit.client.1.smithi086.stdout:4/615: rename d0/d15/d53 to d0/dc0 0 2024-04-06T14:17:20.557 INFO:tasks.workunit.client.1.smithi086.stdout:9/641: dwrite d5/d13/d47/d55/f64 [0,4194304] 0 2024-04-06T14:17:20.561 INFO:tasks.workunit.client.1.smithi086.stdout:9/642: fsync d5/d13/d1a/f63 0 2024-04-06T14:17:20.564 INFO:tasks.workunit.client.0.smithi032.stdout:2/951: truncate d0/d30/d39/d59/d67/d74/d78/d8c/db0/d102/f111 2363873 0 2024-04-06T14:17:20.568 INFO:tasks.workunit.client.1.smithi086.stdout:4/616: dwrite d0/d15/d20/d34/f35 [0,4194304] 0 2024-04-06T14:17:20.570 INFO:tasks.workunit.client.1.smithi086.stdout:9/643: rename d5/d13/d1a/d5f/d88/d14/d19/d25/f99 to d5/d13/d1a/d5f/d88/d14/d19/d25/fcb 0 2024-04-06T14:17:20.572 INFO:tasks.workunit.client.0.smithi032.stdout:2/952: link d0/d30/lff d0/d30/d39/d59/d67/d74/d78/dc2/l13b 0 2024-04-06T14:17:20.575 INFO:tasks.workunit.client.1.smithi086.stdout:9/644: fsync d5/d13/d1a/d5f/d88/d14/d19/d25/fcb 0 2024-04-06T14:17:20.579 INFO:tasks.workunit.client.1.smithi086.stdout:4/617: link d0/de/l3f d0/d15/lc1 0 2024-04-06T14:17:20.581 INFO:tasks.workunit.client.1.smithi086.stdout:9/645: rmdir d5/d13/d47/d55/d87/dbb 0 2024-04-06T14:17:20.583 INFO:tasks.workunit.client.1.smithi086.stdout:9/646: chown d5/d13/l34 1596 1 2024-04-06T14:17:20.585 INFO:tasks.workunit.client.1.smithi086.stdout:9/647: mknod d5/d13/d1a/d5f/d88/d14/d19/d25/d74/ccc 0 2024-04-06T14:17:20.587 INFO:tasks.workunit.client.1.smithi086.stdout:9/648: creat d5/d13/d47/d5e/fcd x:0 0 0 2024-04-06T14:17:20.589 INFO:tasks.workunit.client.1.smithi086.stdout:9/649: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dce 0 2024-04-06T14:17:20.590 INFO:tasks.workunit.client.1.smithi086.stdout:4/618: dread d0/d2c/d30/d24/f2d [0,4194304] 0 2024-04-06T14:17:20.591 INFO:tasks.workunit.client.1.smithi086.stdout:9/650: creat d5/d13/d47/d55/fcf x:0 0 0 2024-04-06T14:17:20.593 INFO:tasks.workunit.client.1.smithi086.stdout:9/651: creat d5/d58/fd0 x:0 0 0 2024-04-06T14:17:20.594 INFO:tasks.workunit.client.1.smithi086.stdout:4/619: getdents d0/d2c/d30/d24/d5d/d8a 0 2024-04-06T14:17:20.597 INFO:tasks.workunit.client.1.smithi086.stdout:4/620: mkdir d0/d2c/d38/d54/d61/d7a/dc2 0 2024-04-06T14:17:20.600 INFO:tasks.workunit.client.1.smithi086.stdout:9/652: dwrite f2 [4194304,4194304] 0 2024-04-06T14:17:20.600 INFO:tasks.workunit.client.1.smithi086.stdout:4/621: unlink d0/d2c/d38/l45 0 2024-04-06T14:17:20.601 INFO:tasks.workunit.client.1.smithi086.stdout:4/622: fsync d0/d2c/d38/d54/d90/fbd 0 2024-04-06T14:17:20.610 INFO:tasks.workunit.client.1.smithi086.stdout:9/653: getdents d5/d13/d1a/d5f/d88/d14/d19 0 2024-04-06T14:17:20.610 INFO:tasks.workunit.client.0.smithi032.stdout:2/953: creat d0/d6e/d8d/f13c x:0 0 0 2024-04-06T14:17:20.611 INFO:tasks.workunit.client.1.smithi086.stdout:4/623: dwrite d0/d2c/d30/d24/d71/fa0 [0,4194304] 0 2024-04-06T14:17:20.612 INFO:tasks.workunit.client.1.smithi086.stdout:4/624: rename d0 to d0/d2c/d30/d24/d50/d79/dc3 22 2024-04-06T14:17:20.613 INFO:tasks.workunit.client.0.smithi032.stdout:2/954: unlink d0/db2/dfc/f48 0 2024-04-06T14:17:20.614 INFO:tasks.workunit.client.0.smithi032.stdout:2/955: read - d0/d5a/d6a/f135 zero size 2024-04-06T14:17:20.615 INFO:tasks.workunit.client.1.smithi086.stdout:9/654: rmdir d5/d13/d47/d98 0 2024-04-06T14:17:20.618 INFO:tasks.workunit.client.0.smithi032.stdout:2/956: truncate d0/d30/d39/d59/d67/fc0 1331835 0 2024-04-06T14:17:20.622 INFO:tasks.workunit.client.1.smithi086.stdout:4/625: dwrite d0/d2c/d38/d54/f80 [0,4194304] 0 2024-04-06T14:17:20.630 INFO:tasks.workunit.client.1.smithi086.stdout:9/655: rename d5/d13/d1a/d5f/d88/d91/cc3 to d5/d13/d1a/d5f/d88/d14/d19/d25/d59/cd1 0 2024-04-06T14:17:20.633 INFO:tasks.workunit.client.1.smithi086.stdout:4/626: rename d0/d2c/d30/d24/c95 to d0/d15/d20/d34/cc4 0 2024-04-06T14:17:20.635 INFO:tasks.workunit.client.1.smithi086.stdout:9/656: mknod d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/cd2 0 2024-04-06T14:17:20.636 INFO:tasks.workunit.client.1.smithi086.stdout:4/627: rename d0/d2c/d30/d24/f2d to d0/de/d4c/fc5 0 2024-04-06T14:17:20.637 INFO:tasks.workunit.client.1.smithi086.stdout:9/657: write d5/d13/d1a/d5f/d88/d14/f4e [2676229,105510] 0 2024-04-06T14:17:20.643 INFO:tasks.workunit.client.1.smithi086.stdout:9/658: mknod d5/d13/d1a/d5f/d88/d14/d90/cd3 0 2024-04-06T14:17:20.644 INFO:tasks.workunit.client.1.smithi086.stdout:4/628: dwrite d0/d2c/d30/d24/f31 [0,4194304] 0 2024-04-06T14:17:20.645 INFO:tasks.workunit.client.0.smithi032.stdout:2/957: dwrite d0/d30/d39/d59/d67/d74/d78/d57/dd3/fdc [4194304,4194304] 0 2024-04-06T14:17:20.648 INFO:tasks.workunit.client.1.smithi086.stdout:1/802: sync 2024-04-06T14:17:20.648 INFO:tasks.workunit.client.1.smithi086.stdout:2/786: sync 2024-04-06T14:17:20.648 INFO:tasks.workunit.client.1.smithi086.stdout:8/830: sync 2024-04-06T14:17:20.649 INFO:tasks.workunit.client.0.smithi032.stdout:2/958: mkdir d0/d4e/de9/dea/d13d 0 2024-04-06T14:17:20.650 INFO:tasks.workunit.client.1.smithi086.stdout:1/803: chown db/d13/d71/fd0 11327 1 2024-04-06T14:17:20.650 INFO:tasks.workunit.client.0.smithi032.stdout:2/959: chown d0/d30/d7f/d129 1 1 2024-04-06T14:17:20.650 INFO:tasks.workunit.client.1.smithi086.stdout:4/629: mknod d0/d2c/d38/d54/d61/cc6 0 2024-04-06T14:17:20.655 INFO:tasks.workunit.client.1.smithi086.stdout:8/831: rmdir d0/d71/d22/d43/d6d 39 2024-04-06T14:17:20.657 INFO:tasks.workunit.client.1.smithi086.stdout:2/787: creat d0/d20/d27/d2b/d35/d36/d4d/d59/f119 x:0 0 0 2024-04-06T14:17:20.657 INFO:tasks.workunit.client.1.smithi086.stdout:1/804: dread - db/d11/d1a/d79/f9d zero size 2024-04-06T14:17:20.658 INFO:tasks.workunit.client.0.smithi032.stdout:2/960: rename d0/d5a/d6a/fb6 to d0/d30/d39/d59/d67/d74/d78/d57/d7c/f13e 0 2024-04-06T14:17:20.660 INFO:tasks.workunit.client.1.smithi086.stdout:1/805: mknod db/d11/d1a/d2a/c100 0 2024-04-06T14:17:20.663 INFO:tasks.workunit.client.0.smithi032.stdout:2/961: mkdir d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/d13f 0 2024-04-06T14:17:20.665 INFO:tasks.workunit.client.1.smithi086.stdout:1/806: read db/d11/d1a/d2a/f9c [1447284,128057] 0 2024-04-06T14:17:20.668 INFO:tasks.workunit.client.1.smithi086.stdout:9/659: dread d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f81 [0,4194304] 0 2024-04-06T14:17:20.669 INFO:tasks.workunit.client.1.smithi086.stdout:2/788: dwrite d0/d20/d27/d2b/d35/d36/dbe/f10c [0,4194304] 0 2024-04-06T14:17:20.673 INFO:tasks.workunit.client.1.smithi086.stdout:8/832: dwrite d0/d71/d22/d43/f47 [0,4194304] 0 2024-04-06T14:17:20.674 INFO:tasks.workunit.client.1.smithi086.stdout:4/630: dread d0/d2c/d30/d24/fa2 [0,4194304] 0 2024-04-06T14:17:20.674 INFO:tasks.workunit.client.1.smithi086.stdout:9/660: symlink d5/d58/d8d/ld4 0 2024-04-06T14:17:20.675 INFO:tasks.workunit.client.1.smithi086.stdout:3/813: sync 2024-04-06T14:17:20.676 INFO:tasks.workunit.client.1.smithi086.stdout:5/661: sync 2024-04-06T14:17:20.676 INFO:tasks.workunit.client.1.smithi086.stdout:7/735: sync 2024-04-06T14:17:20.676 INFO:tasks.workunit.client.1.smithi086.stdout:6/925: sync 2024-04-06T14:17:20.676 INFO:tasks.workunit.client.1.smithi086.stdout:0/813: sync 2024-04-06T14:17:20.676 INFO:tasks.workunit.client.1.smithi086.stdout:8/833: write d0/d13/fa9 [2176407,94387] 0 2024-04-06T14:17:20.676 INFO:tasks.workunit.client.1.smithi086.stdout:2/789: symlink d0/df/d5e/d9d/d102/d100/l11a 0 2024-04-06T14:17:20.676 INFO:tasks.workunit.client.1.smithi086.stdout:1/807: truncate db/d11/d1a/d2a/d9a/d68/d6e/d81/f99 2145846 0 2024-04-06T14:17:20.678 INFO:tasks.workunit.client.1.smithi086.stdout:2/790: stat d0/d20/d27/la0 0 2024-04-06T14:17:20.680 INFO:tasks.workunit.client.1.smithi086.stdout:9/661: truncate d5/d13/d47/fc4 1048444 0 2024-04-06T14:17:20.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.681+0000 7f40fd22d700 1 -- 172.21.15.32:0/1055297860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f8066b70 msgr2=0x7f40f80fd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:20.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.681+0000 7f40fd22d700 1 --2- 172.21.15.32:0/1055297860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f8066b70 0x7f40f80fd7c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0049c0 tx=0x7f40ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.681+0000 7f40fd22d700 1 -- 172.21.15.32:0/1055297860 shutdown_connections 2024-04-06T14:17:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.681+0000 7f40fd22d700 1 --2- 172.21.15.32:0/1055297860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40f80fdd00 0x7f40f8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.681+0000 7f40fd22d700 1 --2- 172.21.15.32:0/1055297860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f8066b70 0x7f40f80fd7c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.681+0000 7f40fd22d700 1 -- 172.21.15.32:0/1055297860 >> 172.21.15.32:0/1055297860 conn(0x7f40f80f7480 msgr2=0x7f40f80f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.681+0000 7f40fd22d700 1 -- 172.21.15.32:0/1055297860 shutdown_connections 2024-04-06T14:17:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.681+0000 7f40fd22d700 1 -- 172.21.15.32:0/1055297860 wait complete. 2024-04-06T14:17:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.682+0000 7f40fd22d700 1 Processor -- start 2024-04-06T14:17:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.682+0000 7f40fd22d700 1 -- start start 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.682+0000 7f40fd22d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40f8066b70 0x7f40f806e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.682+0000 7f40fd22d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f80fdd00 0x7f40f806e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.682+0000 7f40fd22d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8070090 con 0x7f40f80fdd00 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.682+0000 7f40fd22d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f806eed0 con 0x7f40f8066b70 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.683+0000 7f40f659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f80fdd00 0x7f40f806e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.683+0000 7f40f659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f80fdd00 0x7f40f806e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55146/0 (socket says 172.21.15.32:55146) 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.683+0000 7f40f659c700 1 -- 172.21.15.32:0/12494977 learned_addr learned my addr 172.21.15.32:0/12494977 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.683+0000 7f40f6d9d700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40f8066b70 0x7f40f806e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.683+0000 7f40f659c700 1 -- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40f8066b70 msgr2=0x7f40f806e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.683+0000 7f40f659c700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40f8066b70 0x7f40f806e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.683+0000 7f40f659c700 1 -- 172.21.15.32:0/12494977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec005040 con 0x7f40f80fdd00 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.684+0000 7f40f659c700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f80fdd00 0x7f40f806e990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f40e8002a50 tx=0x7f40e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.684+0000 7f40df7fe700 1 -- 172.21.15.32:0/12494977 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40e8007cc0 con 0x7f40f80fdd00 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.684+0000 7f40df7fe700 1 -- 172.21.15.32:0/12494977 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40e800f040 con 0x7f40f80fdd00 2024-04-06T14:17:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.684+0000 7f40df7fe700 1 -- 172.21.15.32:0/12494977 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40e800d9c0 con 0x7f40f80fdd00 2024-04-06T14:17:20.686 INFO:tasks.workunit.client.1.smithi086.stdout:3/814: rmdir d5/d36/d5a/d6d/d72 39 2024-04-06T14:17:20.686 INFO:tasks.workunit.client.1.smithi086.stdout:4/631: dread d0/d15/d2b/f82 [0,4194304] 0 2024-04-06T14:17:20.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.684+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f806f1b0 con 0x7f40f80fdd00 2024-04-06T14:17:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.684+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f8108cb0 con 0x7f40f80fdd00 2024-04-06T14:17:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.685+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f805f5f0 con 0x7f40f80fdd00 2024-04-06T14:17:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.686+0000 7f40df7fe700 1 -- 172.21.15.32:0/12494977 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f40e800db40 con 0x7f40f80fdd00 2024-04-06T14:17:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.687+0000 7f40df7fe700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f40e006af90 0x7f40e006d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.687+0000 7f40df7fe700 1 -- 172.21.15.32:0/12494977 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f40e80895d0 con 0x7f40f80fdd00 2024-04-06T14:17:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.688+0000 7f40f6d9d700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f40e006af90 0x7f40e006d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:20.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.689+0000 7f40f6d9d700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f40e006af90 0x7f40e006d440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f40f806fd10 tx=0x7f40ec009e30 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:20.690 INFO:tasks.workunit.client.1.smithi086.stdout:4/632: fdatasync d0/d2c/d30/d24/d50/d79/d89/fa6 0 2024-04-06T14:17:20.690 INFO:tasks.workunit.client.1.smithi086.stdout:4/633: stat d0/d15/d20/c56 0 2024-04-06T14:17:20.691 INFO:tasks.workunit.client.1.smithi086.stdout:9/662: fsync d5/d13/d47/f83 0 2024-04-06T14:17:20.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:20.693+0000 7f40df7fe700 1 -- 172.21.15.32:0/12494977 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f40e80539a0 con 0x7f40f80fdd00 2024-04-06T14:17:20.693 INFO:tasks.workunit.client.1.smithi086.stdout:5/662: dwrite d0/d16/d18/d2a/d50/d66/fca [0,4194304] 0 2024-04-06T14:17:20.694 INFO:tasks.workunit.client.1.smithi086.stdout:2/791: rmdir d0/d20/d80/dbf 39 2024-04-06T14:17:20.695 INFO:tasks.workunit.client.1.smithi086.stdout:0/814: mkdir d0/d15/d4e/da3/d73/d10c 0 2024-04-06T14:17:20.699 INFO:tasks.workunit.client.1.smithi086.stdout:1/808: mkdir db/d11/d1a/d2a/d9a/d78/d101 0 2024-04-06T14:17:20.700 INFO:tasks.workunit.client.1.smithi086.stdout:8/834: symlink d0/d71/d22/d65/d6e/d98/l125 0 2024-04-06T14:17:20.700 INFO:tasks.workunit.client.1.smithi086.stdout:7/736: truncate d7/d10/d11/d1c/d36/f74 1675359 0 2024-04-06T14:17:20.700 INFO:tasks.workunit.client.1.smithi086.stdout:1/809: readlink db/d13/d71/ld6 0 2024-04-06T14:17:20.703 INFO:tasks.workunit.client.1.smithi086.stdout:3/815: read d5/d36/d5a/d6d/d72/d9a/fd7 [3043976,43411] 0 2024-04-06T14:17:20.703 INFO:tasks.workunit.client.1.smithi086.stdout:1/810: truncate db/d11/fc1 858317 0 2024-04-06T14:17:20.704 INFO:tasks.workunit.client.1.smithi086.stdout:4/634: unlink d0/d15/d20/l43 0 2024-04-06T14:17:20.705 INFO:tasks.workunit.client.1.smithi086.stdout:9/663: fsync d5/d13/d1a/d5f/d88/d14/f22 0 2024-04-06T14:17:20.706 INFO:tasks.workunit.client.1.smithi086.stdout:5/663: mkdir d0/d16/d18/d35/d9e/dd7 0 2024-04-06T14:17:20.710 INFO:tasks.workunit.client.1.smithi086.stdout:2/792: symlink d0/d20/l11b 0 2024-04-06T14:17:20.710 INFO:tasks.workunit.client.1.smithi086.stdout:1/811: dread - db/d11/d1a/d2a/d52/d6b/fd3 zero size 2024-04-06T14:17:20.711 INFO:tasks.workunit.client.1.smithi086.stdout:9/664: readlink d5/d13/d47/d55/d87/lc7 0 2024-04-06T14:17:20.712 INFO:tasks.workunit.client.1.smithi086.stdout:6/926: dread d6/d13/f77 [0,4194304] 0 2024-04-06T14:17:20.713 INFO:tasks.workunit.client.1.smithi086.stdout:4/635: rmdir d0/d2c/d30/d24 39 2024-04-06T14:17:20.713 INFO:tasks.workunit.client.1.smithi086.stdout:8/835: dwrite d0/d71/d16/d39/f66 [0,4194304] 0 2024-04-06T14:17:20.715 INFO:tasks.workunit.client.1.smithi086.stdout:9/665: stat d5/d13/d1a/d5f/d88/d14/d19/f1e 0 2024-04-06T14:17:20.716 INFO:tasks.workunit.client.1.smithi086.stdout:8/836: write d0/d13/fa9 [4300651,54246] 0 2024-04-06T14:17:20.717 INFO:tasks.workunit.client.1.smithi086.stdout:5/664: chown d0/d16/d18/c25 42413016 1 2024-04-06T14:17:20.718 INFO:tasks.workunit.client.1.smithi086.stdout:5/665: stat d0/d16/d18/f9d 0 2024-04-06T14:17:20.719 INFO:tasks.workunit.client.1.smithi086.stdout:0/815: dread d0/d15/d4e/da3/d46/d52/f7f [0,4194304] 0 2024-04-06T14:17:20.720 INFO:tasks.workunit.client.1.smithi086.stdout:0/816: fsync d0/d1/d4/de/db4/fb5 0 2024-04-06T14:17:20.721 INFO:tasks.workunit.client.1.smithi086.stdout:5/666: truncate d0/d16/d71/fb9 1951414 0 2024-04-06T14:17:20.723 INFO:tasks.workunit.client.0.smithi032.stdout:3/969: sync 2024-04-06T14:17:20.725 INFO:tasks.workunit.client.0.smithi032.stdout:3/970: creat d3/d14d/d12c/f16d x:0 0 0 2024-04-06T14:17:20.726 INFO:tasks.workunit.client.1.smithi086.stdout:2/793: rename d0/d20/d27/dae/lf6 to d0/d20/d82/l11c 0 2024-04-06T14:17:20.730 INFO:tasks.workunit.client.0.smithi032.stdout:3/971: dread d3/d16/ddd/df7/d147/d7d/d9c/dad/fb3 [0,4194304] 0 2024-04-06T14:17:20.731 INFO:tasks.workunit.client.1.smithi086.stdout:9/666: unlink d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/cd2 0 2024-04-06T14:17:20.733 INFO:tasks.workunit.client.1.smithi086.stdout:3/816: write d5/dea/f61 [927277,66762] 0 2024-04-06T14:17:20.739 INFO:tasks.workunit.client.0.smithi032.stdout:3/972: link d3/d16/ddd/df7/d147/d89/dcb/l142 d3/d16/ddd/l16e 0 2024-04-06T14:17:20.739 INFO:tasks.workunit.client.1.smithi086.stdout:0/817: rename d0/d1/d2/d1a/d79/c85 to d0/d1/d2/d1a/d79/dac/dc2/c10d 0 2024-04-06T14:17:20.739 INFO:tasks.workunit.client.1.smithi086.stdout:2/794: mkdir d0/d20/d27/d2b/d35/d36/d4a/d11d 0 2024-04-06T14:17:20.741 INFO:tasks.workunit.client.1.smithi086.stdout:0/818: readlink d0/d1/d2/lc7 0 2024-04-06T14:17:20.741 INFO:tasks.workunit.client.1.smithi086.stdout:3/817: dwrite d5/d26/d7f/dae/fcd [0,4194304] 0 2024-04-06T14:17:20.742 INFO:tasks.workunit.client.0.smithi032.stdout:3/973: creat d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/dc9/f16f x:0 0 0 2024-04-06T14:17:20.743 INFO:tasks.workunit.client.1.smithi086.stdout:3/818: dread - d5/d36/d7d/f109 zero size 2024-04-06T14:17:20.745 INFO:tasks.workunit.client.0.smithi032.stdout:3/974: truncate d3/d16/ddd/df7/d147/f32 3016642 0 2024-04-06T14:17:20.749 INFO:tasks.workunit.client.1.smithi086.stdout:8/837: creat d0/d71/d22/d43/d6d/d75/dee/f126 x:0 0 0 2024-04-06T14:17:20.750 INFO:tasks.workunit.client.1.smithi086.stdout:1/812: creat db/d11/d1f/f102 x:0 0 0 2024-04-06T14:17:20.750 INFO:tasks.workunit.client.0.smithi032.stdout:3/975: mknod d3/d16/ddd/df7/d147/d7d/d9c/c170 0 2024-04-06T14:17:20.751 INFO:tasks.workunit.client.1.smithi086.stdout:6/927: dread d6/d13/d42/d52/fc1 [0,4194304] 0 2024-04-06T14:17:20.751 INFO:tasks.workunit.client.1.smithi086.stdout:5/667: truncate d0/d16/d18/d21/d23/d28/d94/f56 524423 0 2024-04-06T14:17:20.752 INFO:tasks.workunit.client.1.smithi086.stdout:6/928: readlink d6/d13/d5f/lac 0 2024-04-06T14:17:20.755 INFO:tasks.workunit.client.0.smithi032.stdout:3/976: truncate d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/f57 1633388 0 2024-04-06T14:17:20.755 INFO:tasks.workunit.client.1.smithi086.stdout:2/795: creat d0/df/f11e x:0 0 0 2024-04-06T14:17:20.756 INFO:tasks.workunit.client.1.smithi086.stdout:7/737: link d7/d10/d11/d1c/d24/d4d/ce4 d7/d10/d11/d1c/d36/cf9 0 2024-04-06T14:17:20.757 INFO:tasks.workunit.client.1.smithi086.stdout:0/819: mknod d0/d15/d4e/d48/c10e 0 2024-04-06T14:17:20.759 INFO:tasks.workunit.client.1.smithi086.stdout:8/838: readlink d0/l7b 0 2024-04-06T14:17:20.759 INFO:tasks.workunit.client.1.smithi086.stdout:5/668: symlink d0/d16/d18/d21/d23/d28/d94/ld8 0 2024-04-06T14:17:20.761 INFO:tasks.workunit.client.1.smithi086.stdout:6/929: mknod d6/d9/db/dff/c141 0 2024-04-06T14:17:20.763 INFO:tasks.workunit.client.1.smithi086.stdout:1/813: dwrite db/d11/d1a/d2a/d9a/d68/d6e/fea [0,4194304] 0 2024-04-06T14:17:20.764 INFO:tasks.workunit.client.1.smithi086.stdout:9/667: dwrite d5/d13/f71 [0,4194304] 0 2024-04-06T14:17:20.766 INFO:tasks.workunit.client.1.smithi086.stdout:4/636: getdents d0/d2c/d30/d24/d71 0 2024-04-06T14:17:20.769 INFO:tasks.workunit.client.1.smithi086.stdout:6/930: fdatasync d6/d13/d1a/da0/d114/db2/f130 0 2024-04-06T14:17:20.770 INFO:tasks.workunit.client.1.smithi086.stdout:8/839: fdatasync d0/d3b/fa6 0 2024-04-06T14:17:20.772 INFO:tasks.workunit.client.1.smithi086.stdout:5/669: symlink d0/d43/ld9 0 2024-04-06T14:17:20.773 INFO:tasks.workunit.client.1.smithi086.stdout:5/670: fdatasync d0/d16/d18/d2a/d50/d66/fca 0 2024-04-06T14:17:20.774 INFO:tasks.workunit.client.1.smithi086.stdout:7/738: symlink d7/d10/d11/lfa 0 2024-04-06T14:17:20.774 INFO:tasks.workunit.client.1.smithi086.stdout:5/671: chown d0/d16/d18 1 1 2024-04-06T14:17:20.778 INFO:tasks.workunit.client.0.smithi032.stdout:3/977: dwrite d3/f5e [0,4194304] 0 2024-04-06T14:17:20.778 INFO:tasks.workunit.client.1.smithi086.stdout:3/819: getdents d5/d36/d5c 0 2024-04-06T14:17:20.780 INFO:tasks.workunit.client.1.smithi086.stdout:0/820: dwrite d0/d1/d4/f43 [4194304,4194304] 0 2024-04-06T14:17:20.783 INFO:tasks.workunit.client.0.smithi032.stdout:3/978: write d3/d14d/d12c/f16d [288098,29878] 0 2024-04-06T14:17:20.786 INFO:tasks.workunit.client.1.smithi086.stdout:1/814: rename db/f16 to db/d11/d1a/d2a/d9a/d78/d101/f103 0 2024-04-06T14:17:20.787 INFO:tasks.workunit.client.1.smithi086.stdout:6/931: symlink d6/d13/d1a/da0/d11b/l142 0 2024-04-06T14:17:20.787 INFO:tasks.workunit.client.1.smithi086.stdout:2/796: creat d0/d20/d44/dbb/dc4/f11f x:0 0 0 2024-04-06T14:17:20.788 INFO:tasks.workunit.client.0.smithi032.stdout:3/979: creat d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/dc9/f171 x:0 0 0 2024-04-06T14:17:20.788 INFO:tasks.workunit.client.1.smithi086.stdout:1/815: dread - db/d11/d87/fec zero size 2024-04-06T14:17:20.789 INFO:tasks.workunit.client.1.smithi086.stdout:3/820: mknod d5/d26/d7f/de7/c10e 0 2024-04-06T14:17:20.789 INFO:tasks.workunit.client.1.smithi086.stdout:7/739: chown d7/d10/d11/d80/l90 23 1 2024-04-06T14:17:20.790 INFO:tasks.workunit.client.1.smithi086.stdout:4/637: read d0/d2c/d30/d24/da3/f52 [743040,28564] 0 2024-04-06T14:17:20.792 INFO:tasks.workunit.client.1.smithi086.stdout:2/797: stat d0/d20/d27/dae/lb2 0 2024-04-06T14:17:20.793 INFO:tasks.workunit.client.1.smithi086.stdout:3/821: write d5/dea/dd2/ddb/de6/f6a [1196766,5765] 0 2024-04-06T14:17:20.793 INFO:tasks.workunit.client.1.smithi086.stdout:9/668: rename d5/d13/l84 to d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/ld5 0 2024-04-06T14:17:20.793 INFO:tasks.workunit.client.0.smithi032.stdout:3/980: symlink d3/d16/ddd/df7/d147/d7d/d9c/dbc/dcd/l172 0 2024-04-06T14:17:20.794 INFO:tasks.workunit.client.1.smithi086.stdout:6/932: mknod d6/d9/d63/c143 0 2024-04-06T14:17:20.795 INFO:tasks.workunit.client.1.smithi086.stdout:0/821: mkdir d0/d15/d4e/da3/d44/d10f 0 2024-04-06T14:17:20.795 INFO:tasks.workunit.client.1.smithi086.stdout:8/840: creat d0/d3b/d79/f127 x:0 0 0 2024-04-06T14:17:20.796 INFO:tasks.workunit.client.1.smithi086.stdout:4/638: rmdir d0/d15/d20/d74 39 2024-04-06T14:17:20.797 INFO:tasks.workunit.client.1.smithi086.stdout:2/798: write d0/d20/d27/d2b/d35/d36/fd4 [208223,125190] 0 2024-04-06T14:17:20.798 INFO:tasks.workunit.client.1.smithi086.stdout:1/816: mkdir db/d11/d1a/d2a/d9a/d68/daf/d104 0 2024-04-06T14:17:20.800 INFO:tasks.workunit.client.0.smithi032.stdout:3/981: rename d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/dc9/f16f to d3/d16/ddd/df7/d147/d7d/d9c/db8/f173 0 2024-04-06T14:17:20.800 INFO:tasks.workunit.client.1.smithi086.stdout:5/672: rename d0/d16/d18/d2a/d50/d66/fc9 to d0/d16/d18/d21/d23/d28/d94/fda 0 2024-04-06T14:17:20.801 INFO:tasks.workunit.client.1.smithi086.stdout:9/669: symlink d5/d13/d47/d55/ld6 0 2024-04-06T14:17:20.803 INFO:tasks.workunit.client.1.smithi086.stdout:8/841: fdatasync d0/d3b/d79/dc6/fd4 0 2024-04-06T14:17:20.803 INFO:tasks.workunit.client.0.smithi032.stdout:3/982: creat d3/d16/d63/d129/f174 x:0 0 0 2024-04-06T14:17:20.806 INFO:tasks.workunit.client.1.smithi086.stdout:7/740: rename d7/d10/d11/f16 to d7/d10/d11/d1c/d24/d2c/d69/d83/ffb 0 2024-04-06T14:17:20.807 INFO:tasks.workunit.client.1.smithi086.stdout:1/817: write db/d11/d1a/d2a/f9c [7202282,99562] 0 2024-04-06T14:17:20.811 INFO:tasks.workunit.client.0.smithi032.stdout:3/983: dwrite d3/f7a [0,4194304] 0 2024-04-06T14:17:20.811 INFO:tasks.workunit.client.1.smithi086.stdout:2/799: dwrite d0/df/d5e/fc7 [0,4194304] 0 2024-04-06T14:17:20.811 INFO:tasks.workunit.client.1.smithi086.stdout:5/673: creat d0/d16/d18/d35/fdb x:0 0 0 2024-04-06T14:17:20.811 INFO:tasks.workunit.client.1.smithi086.stdout:0/822: dread d0/d1/d2/d1a/d79/fb2 [0,4194304] 0 2024-04-06T14:17:20.812 INFO:tasks.workunit.client.1.smithi086.stdout:6/933: symlink d6/d13/d1a/da0/d11b/d45/da6/l144 0 2024-04-06T14:17:20.812 INFO:tasks.workunit.client.1.smithi086.stdout:5/674: stat d0/d43/c48 0 2024-04-06T14:17:20.818 INFO:tasks.workunit.client.1.smithi086.stdout:7/741: mknod d7/d10/d11/d1c/d24/d4d/d5c/d66/d7e/db9/dd2/cfc 0 2024-04-06T14:17:20.820 INFO:tasks.workunit.client.1.smithi086.stdout:4/639: dread d0/de/f11 [0,4194304] 0 2024-04-06T14:17:20.825 INFO:tasks.workunit.client.0.smithi032.stdout:3/984: truncate d3/faf 4132616 0 2024-04-06T14:17:20.825 INFO:tasks.workunit.client.1.smithi086.stdout:6/934: mkdir d6/d13/d1a/da0/d114/db2/dee/d145 0 2024-04-06T14:17:20.826 INFO:tasks.workunit.client.1.smithi086.stdout:5/675: truncate d0/d16/d18/d35/d9e/fb3 4351604 0 2024-04-06T14:17:20.827 INFO:tasks.workunit.client.1.smithi086.stdout:7/742: mknod d7/d10/d11/d80/cfd 0 2024-04-06T14:17:20.829 INFO:tasks.workunit.client.1.smithi086.stdout:4/640: symlink d0/d2c/d30/d24/d93/lc7 0 2024-04-06T14:17:20.838 INFO:tasks.workunit.client.1.smithi086.stdout:2/800: dread d0/d20/d27/d2b/d35/d36/fa1 [0,4194304] 0 2024-04-06T14:17:20.861 INFO:tasks.workunit.client.1.smithi086.stdout:5/676: mkdir d0/d16/d71/d96/ddc 0 2024-04-06T14:17:20.861 INFO:tasks.workunit.client.1.smithi086.stdout:0/823: mkdir d0/d15/d26/d33/d7a/daa/d110 0 2024-04-06T14:17:20.862 INFO:tasks.workunit.client.1.smithi086.stdout:7/743: creat d7/d10/d11/d1c/d24/dc8/ffe x:0 0 0 2024-04-06T14:17:20.862 INFO:tasks.workunit.client.1.smithi086.stdout:0/824: stat d0/d1/d2/d1a/d79/dac/db7 0 2024-04-06T14:17:20.863 INFO:tasks.workunit.client.1.smithi086.stdout:7/744: chown d7/ff1 3 1 2024-04-06T14:17:20.863 INFO:tasks.workunit.client.1.smithi086.stdout:6/935: creat d6/d13/d1a/da0/d11b/d45/d12d/f146 x:0 0 0 2024-04-06T14:17:20.864 INFO:tasks.workunit.client.1.smithi086.stdout:6/936: fdatasync d6/d9/f10 0 2024-04-06T14:17:20.865 INFO:tasks.workunit.client.1.smithi086.stdout:4/641: symlink d0/d2c/d30/d24/d50/d79/lc8 0 2024-04-06T14:17:20.865 INFO:tasks.workunit.client.1.smithi086.stdout:0/825: write d0/d15/d4e/da3/d46/d52/ff3 [414839,106782] 0 2024-04-06T14:17:20.865 INFO:tasks.workunit.client.1.smithi086.stdout:2/801: mkdir d0/d20/d44/d120 0 2024-04-06T14:17:20.868 INFO:tasks.workunit.client.1.smithi086.stdout:1/818: write db/d11/d1a/d79/f9d [904286,109439] 0 2024-04-06T14:17:20.869 INFO:tasks.workunit.client.1.smithi086.stdout:6/937: chown d6/d9/db/fd2 11 1 2024-04-06T14:17:20.869 INFO:tasks.workunit.client.1.smithi086.stdout:0/826: read d0/d1/f13 [301987,15052] 0 2024-04-06T14:17:20.871 INFO:tasks.workunit.client.1.smithi086.stdout:2/802: fdatasync d0/d20/d27/d37/d52/feb 0 2024-04-06T14:17:20.873 INFO:tasks.workunit.client.1.smithi086.stdout:5/677: dwrite d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc5 [0,4194304] 0 2024-04-06T14:17:20.874 INFO:tasks.workunit.client.1.smithi086.stdout:9/670: dwrite d5/d13/d1a/f24 [0,4194304] 0 2024-04-06T14:17:20.891 INFO:tasks.workunit.client.1.smithi086.stdout:9/671: rmdir d5/d13/d1a/d8f 39 2024-04-06T14:17:20.892 INFO:tasks.workunit.client.1.smithi086.stdout:5/678: truncate d0/d16/d18/d2a/d50/d66/f7d 778332 0 2024-04-06T14:17:20.894 INFO:tasks.workunit.client.1.smithi086.stdout:2/803: dwrite d0/d20/d44/dbb/fc0 [0,4194304] 0 2024-04-06T14:17:20.895 INFO:tasks.workunit.client.1.smithi086.stdout:2/804: dread - d0/df/f11e zero size 2024-04-06T14:17:20.896 INFO:tasks.workunit.client.1.smithi086.stdout:1/819: mkdir db/d11/d105 0 2024-04-06T14:17:20.899 INFO:tasks.workunit.client.1.smithi086.stdout:0/827: creat d0/d1/dfe/f111 x:0 0 0 2024-04-06T14:17:20.900 INFO:tasks.workunit.client.1.smithi086.stdout:9/672: dwrite f0 [4194304,4194304] 0 2024-04-06T14:17:20.904 INFO:tasks.workunit.client.1.smithi086.stdout:7/745: creat d7/d10/d11/d1c/d24/d4d/daf/db5/fff x:0 0 0 2024-04-06T14:17:20.904 INFO:tasks.workunit.client.1.smithi086.stdout:5/679: mknod d0/d43/da6/cdd 0 2024-04-06T14:17:20.904 INFO:tasks.workunit.client.1.smithi086.stdout:2/805: mkdir d0/d20/d82/d121 0 2024-04-06T14:17:20.905 INFO:tasks.workunit.client.1.smithi086.stdout:6/938: rename d6/d9/db/ldd to d6/d9/l147 0 2024-04-06T14:17:20.905 INFO:tasks.workunit.client.1.smithi086.stdout:1/820: dread - db/d11/d1a/f85 zero size 2024-04-06T14:17:20.906 INFO:tasks.workunit.client.1.smithi086.stdout:9/673: mknod d5/d18/cd7 0 2024-04-06T14:17:20.908 INFO:tasks.workunit.client.1.smithi086.stdout:4/642: getdents d0/d2c/d30/d24/d50/d79/d89 0 2024-04-06T14:17:20.909 INFO:tasks.workunit.client.1.smithi086.stdout:5/680: write d0/d16/d18/d21/d23/d28/dc2/f3e [4142861,38278] 0 2024-04-06T14:17:20.911 INFO:tasks.workunit.client.1.smithi086.stdout:6/939: creat d6/f148 x:0 0 0 2024-04-06T14:17:20.912 INFO:tasks.workunit.client.1.smithi086.stdout:6/940: chown d6/d13/d1a/da0/d11b/d45/f13d 80285 1 2024-04-06T14:17:20.916 INFO:tasks.workunit.client.1.smithi086.stdout:1/821: symlink db/d11/d1a/d2a/d9a/d68/l106 0 2024-04-06T14:17:20.917 INFO:tasks.workunit.client.1.smithi086.stdout:9/674: read d5/d13/d1a/d5f/d88/d14/f6d [3636750,119718] 0 2024-04-06T14:17:20.919 INFO:tasks.workunit.client.1.smithi086.stdout:9/675: stat d5/d13/d1a/d5f/d88/d14/d19/f76 0 2024-04-06T14:17:20.920 INFO:tasks.workunit.client.1.smithi086.stdout:6/941: creat d6/d13/d1a/da0/d11b/d45/d4a/d13b/f149 x:0 0 0 2024-04-06T14:17:20.921 INFO:tasks.workunit.client.1.smithi086.stdout:4/643: creat d0/d15/d20/fc9 x:0 0 0 2024-04-06T14:17:20.922 INFO:tasks.workunit.client.1.smithi086.stdout:1/822: unlink db/d11/d1a/d2a/d9a/d68/daf/dc2/cb8 0 2024-04-06T14:17:20.923 INFO:tasks.workunit.client.1.smithi086.stdout:7/746: dread d7/d10/d11/d1c/d24/dc8/fec [0,4194304] 0 2024-04-06T14:17:20.923 INFO:tasks.workunit.client.1.smithi086.stdout:6/942: chown d6/d9/d63/l112 1 1 2024-04-06T14:17:20.923 INFO:tasks.workunit.client.1.smithi086.stdout:9/676: chown d5/d13/d1a/d5f/d88/d14/c89 323039 1 2024-04-06T14:17:20.924 INFO:tasks.workunit.client.1.smithi086.stdout:9/677: stat d5/lb7 0 2024-04-06T14:17:20.925 INFO:tasks.workunit.client.1.smithi086.stdout:6/943: stat d6/d9/d43/fed 0 2024-04-06T14:17:20.926 INFO:tasks.workunit.client.1.smithi086.stdout:2/806: dread d0/df/d48/f7c [0,4194304] 0 2024-04-06T14:17:20.928 INFO:tasks.workunit.client.1.smithi086.stdout:9/678: symlink d5/d13/d47/d5e/ld8 0 2024-04-06T14:17:20.929 INFO:tasks.workunit.client.1.smithi086.stdout:9/679: readlink d5/d18/l49 0 2024-04-06T14:17:20.929 INFO:tasks.workunit.client.1.smithi086.stdout:1/823: dread db/d11/d24/fba [0,4194304] 0 2024-04-06T14:17:20.930 INFO:tasks.workunit.client.1.smithi086.stdout:9/680: stat d5/d58/fd0 0 2024-04-06T14:17:20.930 INFO:tasks.workunit.client.1.smithi086.stdout:9/681: read d5/d13/f71 [1059321,7569] 0 2024-04-06T14:17:20.955 INFO:tasks.workunit.client.0.smithi032.stdout:4/955: sync 2024-04-06T14:17:20.956 INFO:tasks.workunit.client.1.smithi086.stdout:7/747: mkdir d7/d10/d11/d1c/d24/d4d/daf/db5/d7a/d100 0 2024-04-06T14:17:20.957 INFO:tasks.workunit.client.1.smithi086.stdout:4/644: link d0/d15/d20/d34/l4b d0/db0/lca 0 2024-04-06T14:17:20.958 INFO:tasks.workunit.client.1.smithi086.stdout:7/748: write d7/d10/d11/d1c/d24/d2c/fc0 [770898,21968] 0 2024-04-06T14:17:20.958 INFO:tasks.workunit.client.1.smithi086.stdout:6/944: creat d6/d13/d1a/da0/d11b/d45/d12d/f14a x:0 0 0 2024-04-06T14:17:20.958 INFO:tasks.workunit.client.0.smithi032.stdout:4/956: dread d0/df/d1f/d42/f99 [0,4194304] 0 2024-04-06T14:17:20.959 INFO:tasks.workunit.client.1.smithi086.stdout:0/828: write d0/d1/d2/d1a/d51/dd7/de2/f11 [8497313,75472] 0 2024-04-06T14:17:20.960 INFO:tasks.workunit.client.1.smithi086.stdout:1/824: truncate f7 1129282 0 2024-04-06T14:17:20.961 INFO:tasks.workunit.client.0.smithi032.stdout:4/957: truncate d0/df/d2b/d46/d58/fdc 895157 0 2024-04-06T14:17:20.962 INFO:tasks.workunit.client.1.smithi086.stdout:0/829: write d0/d15/d4e/d48/d89/f10b [585183,50656] 0 2024-04-06T14:17:20.962 INFO:tasks.workunit.client.0.smithi032.stdout:4/958: chown d0/d82/d8e/f120 193974 1 2024-04-06T14:17:20.962 INFO:tasks.workunit.client.1.smithi086.stdout:2/807: symlink d0/d20/d27/d2b/d35/d36/d4a/d11d/l122 0 2024-04-06T14:17:20.963 INFO:tasks.workunit.client.0.smithi032.stdout:4/959: write d0/df/d23/f32 [3420665,36838] 0 2024-04-06T14:17:20.964 INFO:tasks.workunit.client.1.smithi086.stdout:6/945: mknod d6/d9/d5c/c14b 0 2024-04-06T14:17:20.964 INFO:tasks.workunit.client.1.smithi086.stdout:1/825: rmdir db/d11/d24 39 2024-04-06T14:17:20.967 INFO:tasks.workunit.client.0.smithi032.stdout:4/960: mknod d0/df/d23/d31/d50/d57/d113/c144 0 2024-04-06T14:17:20.972 INFO:tasks.workunit.client.1.smithi086.stdout:6/946: rmdir d6/d13/d1a/da0/d11b/d45/d12d 39 2024-04-06T14:17:20.972 INFO:tasks.workunit.client.1.smithi086.stdout:5/681: dwrite d0/d16/d18/d21/d23/d28/d94/f56 [0,4194304] 0 2024-04-06T14:17:20.975 INFO:tasks.workunit.client.1.smithi086.stdout:2/808: symlink d0/d20/d27/d2b/d35/d36/d4a/l123 0 2024-04-06T14:17:20.975 INFO:tasks.workunit.client.0.smithi032.stdout:4/961: dwrite d0/df/d1f/d5a/f8f [0,4194304] 0 2024-04-06T14:17:20.976 INFO:tasks.workunit.client.1.smithi086.stdout:0/830: truncate d0/d15/d4e/da3/d46/d52/ff3 1196771 0 2024-04-06T14:17:20.976 INFO:tasks.workunit.client.1.smithi086.stdout:4/645: dwrite d0/d15/d2b/f82 [0,4194304] 0 2024-04-06T14:17:20.978 INFO:tasks.workunit.client.1.smithi086.stdout:2/809: write d0/df/d110/f107 [2879142,126744] 0 2024-04-06T14:17:20.978 INFO:tasks.workunit.client.1.smithi086.stdout:4/646: dread - d0/d2c/d30/faf zero size 2024-04-06T14:17:20.980 INFO:tasks.workunit.client.0.smithi032.stdout:4/962: unlink d0/df/d23/f4a 0 2024-04-06T14:17:20.992 INFO:tasks.workunit.client.0.smithi032.stdout:4/963: symlink d0/d82/d4b/db3/l145 0 2024-04-06T14:17:20.993 INFO:tasks.workunit.client.0.smithi032.stdout:4/964: fsync d0/df/d76/f61 0 2024-04-06T14:17:20.993 INFO:tasks.workunit.client.1.smithi086.stdout:8/842: sync 2024-04-06T14:17:20.993 INFO:tasks.workunit.client.1.smithi086.stdout:3/822: sync 2024-04-06T14:17:20.993 INFO:tasks.workunit.client.1.smithi086.stdout:9/682: truncate d5/d13/f71 3634306 0 2024-04-06T14:17:20.993 INFO:tasks.workunit.client.1.smithi086.stdout:4/647: read d0/de/d4c/f5a [2810176,18039] 0 2024-04-06T14:17:20.994 INFO:tasks.workunit.client.1.smithi086.stdout:7/749: link d7/d10/d11/d1c/c9b d7/d10/d11/d1c/d24/d2c/c101 0 2024-04-06T14:17:20.994 INFO:tasks.workunit.client.1.smithi086.stdout:1/826: dwrite db/d11/d1a/d2a/d9a/d68/d6e/d81/f90 [0,4194304] 0 2024-04-06T14:17:20.994 INFO:tasks.workunit.client.1.smithi086.stdout:6/947: link d6/d20/f46 d6/d13/d1a/da0/d114/f14c 0 2024-04-06T14:17:20.995 INFO:tasks.workunit.client.0.smithi032.stdout:4/965: fdatasync d0/df/d23/d31/d50/d57/d8a/db5/def/f11a 0 2024-04-06T14:17:20.995 INFO:tasks.workunit.client.1.smithi086.stdout:7/750: chown d7/d10/d11/d1c/d24/d4d/cf5 5 1 2024-04-06T14:17:20.995 INFO:tasks.workunit.client.1.smithi086.stdout:6/948: chown d6/d13/d1a/da0/d11b/l142 3762 1 2024-04-06T14:17:20.995 INFO:tasks.workunit.client.1.smithi086.stdout:0/831: link d0/d15/d4e/da3/d46/d52/f7f d0/d1/d2/d1a/d79/dac/f112 0 2024-04-06T14:17:20.995 INFO:tasks.workunit.client.1.smithi086.stdout:2/810: getdents d0/df/d5e/d9d/de7 0 2024-04-06T14:17:20.999 INFO:tasks.workunit.client.1.smithi086.stdout:9/683: fdatasync d5/d13/d1a/d5f/d88/d14/d19/d25/f2f 0 2024-04-06T14:17:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.001+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f40f80f8f70 con 0x7f40e006af90 2024-04-06T14:17:21.002 INFO:tasks.workunit.client.0.smithi032.stdout:4/966: rename d0/df/d23/d31/d50/d57/d5b/fc9 to d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/de8/f146 0 2024-04-06T14:17:21.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.003+0000 7f40df7fe700 1 -- 172.21.15.32:0/12494977 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f40f80f8f70 con 0x7f40e006af90 2024-04-06T14:17:21.004 INFO:tasks.workunit.client.0.smithi032.stdout:4/967: chown d0/df/d1f/d42/c78 3372269 1 2024-04-06T14:17:21.005 INFO:tasks.workunit.client.1.smithi086.stdout:2/811: chown d0/l3f 41987641 1 2024-04-06T14:17:21.005 INFO:tasks.workunit.client.1.smithi086.stdout:6/949: truncate d6/d13/d1a/da0/d11b/d45/d4a/f79 2728038 0 2024-04-06T14:17:21.006 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:17:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:17:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:17:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T14:17:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-06T14:17:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "1/2 daemons upgraded", 2024-04-06T14:17:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-04-06T14:17:21.006 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:17:21.006 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:17:21.007 INFO:tasks.workunit.client.1.smithi086.stdout:2/812: fdatasync d0/d20/d44/dbb/fdb 0 2024-04-06T14:17:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.005+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f40e006af90 msgr2=0x7f40e006d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.005+0000 7f40fd22d700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f40e006af90 0x7f40e006d440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f40f806fd10 tx=0x7f40ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.005+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f80fdd00 msgr2=0x7f40f806e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.005+0000 7f40fd22d700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f80fdd00 0x7f40f806e990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f40e8002a50 tx=0x7f40e80046b0 comp rx=0 tx=0).stop 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.006+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 shutdown_connections 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.006+0000 7f40fd22d700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f40e006af90 0x7f40e006d440 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f40f806fd10 tx=0x7f40ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.006+0000 7f40fd22d700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40f8066b70 0x7f40f806e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.006+0000 7f40fd22d700 1 --2- 172.21.15.32:0/12494977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40f80fdd00 0x7f40f806e990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.006+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 >> 172.21.15.32:0/12494977 conn(0x7f40f80f7480 msgr2=0x7f40f80f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.006+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 shutdown_connections 2024-04-06T14:17:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:21.006+0000 7f40fd22d700 1 -- 172.21.15.32:0/12494977 wait complete. 2024-04-06T14:17:21.009 INFO:tasks.workunit.client.1.smithi086.stdout:5/682: write d0/d17/f99 [1081337,104572] 0 2024-04-06T14:17:21.011 INFO:tasks.workunit.client.1.smithi086.stdout:4/648: truncate d0/d2c/d30/d24/d71/f84 152563 0 2024-04-06T14:17:21.011 INFO:tasks.workunit.client.0.smithi032.stdout:4/968: dwrite d0/df/d76/f61 [0,4194304] 0 2024-04-06T14:17:21.012 INFO:tasks.workunit.client.1.smithi086.stdout:4/649: chown d0/d2c/d38/fa7 47 1 2024-04-06T14:17:21.015 INFO:tasks.workunit.client.0.smithi032.stdout:5/935: sync 2024-04-06T14:17:21.016 INFO:tasks.workunit.client.0.smithi032.stdout:0/799: sync 2024-04-06T14:17:21.016 INFO:tasks.workunit.client.0.smithi032.stdout:9/994: sync 2024-04-06T14:17:21.016 INFO:tasks.workunit.client.1.smithi086.stdout:9/684: creat d5/d13/d1a/d5f/d88/d91/db9/fd9 x:0 0 0 2024-04-06T14:17:21.017 INFO:tasks.workunit.client.0.smithi032.stdout:4/969: mkdir d0/d82/d4b/d147 0 2024-04-06T14:17:21.020 INFO:tasks.workunit.client.0.smithi032.stdout:5/936: fdatasync d16/d18/dae/dbb/f10a 0 2024-04-06T14:17:21.020 INFO:tasks.workunit.client.0.smithi032.stdout:0/800: dread d3/da/d25/d2b/d4a/fb9 [0,4194304] 0 2024-04-06T14:17:21.023 INFO:tasks.workunit.client.0.smithi032.stdout:9/995: fsync d2/d15/d1e/d65/d6b/d92/d11d/d5f/fdf 0 2024-04-06T14:17:21.024 INFO:tasks.workunit.client.1.smithi086.stdout:0/832: dread d0/d1/f13 [4194304,4194304] 0 2024-04-06T14:17:21.025 INFO:tasks.workunit.client.1.smithi086.stdout:5/683: rmdir d0/d16/d18/d21/d23/d28 39 2024-04-06T14:17:21.026 INFO:tasks.workunit.client.1.smithi086.stdout:0/833: stat d0/d15/d26/d33/d9f/fbf 0 2024-04-06T14:17:21.028 INFO:tasks.workunit.client.0.smithi032.stdout:9/996: creat d2/d15/d1e/d65/d6b/d92/f140 x:0 0 0 2024-04-06T14:17:21.029 INFO:tasks.workunit.client.1.smithi086.stdout:7/751: rename d7/d10/d11/d1c/d24/d4d to d7/d10/d102 0 2024-04-06T14:17:21.029 INFO:tasks.workunit.client.0.smithi032.stdout:5/937: read d16/d18/d33/f8a [1295926,110215] 0 2024-04-06T14:17:21.031 INFO:tasks.workunit.client.1.smithi086.stdout:1/827: truncate db/fca 1010840 0 2024-04-06T14:17:21.032 INFO:tasks.workunit.client.1.smithi086.stdout:3/823: getdents d5/d33/d5d 0 2024-04-06T14:17:21.032 INFO:tasks.workunit.client.0.smithi032.stdout:0/801: dwrite d3/f45 [0,4194304] 0 2024-04-06T14:17:21.033 INFO:tasks.workunit.client.1.smithi086.stdout:3/824: chown d5/dea/db4/da7/f79 88233256 1 2024-04-06T14:17:21.033 INFO:tasks.workunit.client.0.smithi032.stdout:9/997: creat d2/d15/d1e/d65/d83/d31/d39/f141 x:0 0 0 2024-04-06T14:17:21.033 INFO:tasks.workunit.client.1.smithi086.stdout:9/685: chown d5/d13/d1a/d5f/d88/d14/d19/d25/ca4 3 1 2024-04-06T14:17:21.035 INFO:tasks.workunit.client.1.smithi086.stdout:8/843: getdents d0/d3b/d79/dc6/dbc 0 2024-04-06T14:17:21.036 INFO:tasks.workunit.client.1.smithi086.stdout:8/844: write d0/d3b/fa4 [1870384,59974] 0 2024-04-06T14:17:21.036 INFO:tasks.workunit.client.0.smithi032.stdout:0/802: mkdir d3/da/d25/d2b/d4a/d79/da8/d112 0 2024-04-06T14:17:21.038 INFO:tasks.workunit.client.0.smithi032.stdout:5/938: dread d16/d18/f3c [0,4194304] 0 2024-04-06T14:17:21.038 INFO:tasks.workunit.client.0.smithi032.stdout:0/803: write d3/f45 [698170,16564] 0 2024-04-06T14:17:21.040 INFO:tasks.workunit.client.1.smithi086.stdout:2/813: unlink d0/d20/d27/d2b/fa4 0 2024-04-06T14:17:21.041 INFO:tasks.workunit.client.0.smithi032.stdout:9/998: mknod d2/d15/d1e/d65/d6b/d92/d11d/d59/d93/d95/db8/d81/d4a/d8e/df6/c142 0 2024-04-06T14:17:21.043 INFO:tasks.workunit.client.0.smithi032.stdout:9/999: dread - d2/d15/d1e/d65/d83/d31/d39/da6/d10a/f125 zero size 2024-04-06T14:17:21.045 INFO:tasks.workunit.client.0.smithi032.stdout:0/804: truncate d3/da/d25/d2b/f3d 184535 0 2024-04-06T14:17:21.048 INFO:tasks.workunit.client.0.smithi032.stdout:4/970: dread d0/df/d2b/fd9 [0,4194304] 0 2024-04-06T14:17:21.048 INFO:tasks.workunit.client.1.smithi086.stdout:8/845: dwrite d0/d71/d16/d52/dd0/ddc/de7/dd6/fe6 [0,4194304] 0 2024-04-06T14:17:21.050 INFO:tasks.workunit.client.1.smithi086.stdout:7/752: unlink d7/d10/d102/daf/db5/d7a/f95 0 2024-04-06T14:17:21.050 INFO:tasks.workunit.client.1.smithi086.stdout:6/950: truncate d6/d13/d1a/f116 769484 0 2024-04-06T14:17:21.051 INFO:tasks.workunit.client.1.smithi086.stdout:6/951: write d6/d9/f12c [684435,59254] 0 2024-04-06T14:17:21.052 INFO:tasks.workunit.client.1.smithi086.stdout:4/650: rename d0/d2c/d30/d24/d71/fa0 to d0/d15/d20/d74/d8d/fcb 0 2024-04-06T14:17:21.053 INFO:tasks.workunit.client.1.smithi086.stdout:3/825: creat d5/d36/d5a/f10f x:0 0 0 2024-04-06T14:17:21.054 INFO:tasks.workunit.client.1.smithi086.stdout:4/651: chown d0/d15/f37 7353 1 2024-04-06T14:17:21.056 INFO:tasks.workunit.client.0.smithi032.stdout:4/971: dread d0/df/d23/d31/d50/d57/f64 [0,4194304] 0 2024-04-06T14:17:21.057 INFO:tasks.workunit.client.1.smithi086.stdout:4/652: read d0/f5 [1684103,121673] 0 2024-04-06T14:17:21.057 INFO:tasks.workunit.client.0.smithi032.stdout:0/805: write d3/da/f5e [846032,1596] 0 2024-04-06T14:17:21.059 INFO:tasks.workunit.client.0.smithi032.stdout:0/806: stat d3/da/d25/d2b/d4a/f62 0 2024-04-06T14:17:21.061 INFO:tasks.workunit.client.1.smithi086.stdout:0/834: write d0/d15/f3e [2537234,44563] 0 2024-04-06T14:17:21.063 INFO:tasks.workunit.client.1.smithi086.stdout:7/753: mkdir d7/d10/d102/d5c/d66/d7e/d103 0 2024-04-06T14:17:21.064 INFO:tasks.workunit.client.0.smithi032.stdout:5/939: dwrite d16/d18/d33/f8a [0,4194304] 0 2024-04-06T14:17:21.066 INFO:tasks.workunit.client.1.smithi086.stdout:8/846: symlink d0/d71/d22/d43/d6d/d75/d7d/l128 0 2024-04-06T14:17:21.069 INFO:tasks.workunit.client.0.smithi032.stdout:4/972: dwrite d0/d82/fc5 [0,4194304] 0 2024-04-06T14:17:21.071 INFO:tasks.workunit.client.0.smithi032.stdout:4/973: write d0/d82/fe3 [186794,90691] 0 2024-04-06T14:17:21.071 INFO:tasks.workunit.client.0.smithi032.stdout:5/940: rename d16/d18/f103 to d16/d18/d33/d45/d5e/d136/f145 0 2024-04-06T14:17:21.072 INFO:tasks.workunit.client.1.smithi086.stdout:5/684: dread d0/d16/d18/d21/d23/d28/dc2/f26 [0,4194304] 0 2024-04-06T14:17:21.074 INFO:tasks.workunit.client.1.smithi086.stdout:4/653: rmdir d0/d15 39 2024-04-06T14:17:21.077 INFO:tasks.workunit.client.0.smithi032.stdout:5/941: mkdir d16/d18/d33/d45/d52/d39/d86/d146 0 2024-04-06T14:17:21.078 INFO:tasks.workunit.client.0.smithi032.stdout:4/974: mkdir d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/de8/d148 0 2024-04-06T14:17:21.079 INFO:tasks.workunit.client.0.smithi032.stdout:5/942: chown d16/f24 25795680 1 2024-04-06T14:17:21.083 INFO:tasks.workunit.client.0.smithi032.stdout:4/975: symlink d0/df/d23/d31/d50/d57/d5b/d9c/dc1/de2/df9/l149 0 2024-04-06T14:17:21.084 INFO:tasks.workunit.client.1.smithi086.stdout:5/685: symlink d0/d17/lde 0 2024-04-06T14:17:21.085 INFO:tasks.workunit.client.0.smithi032.stdout:4/976: truncate d0/df/d23/d31/d50/d57/d5b/d9c/dc8/f139 4970170 0 2024-04-06T14:17:21.087 INFO:tasks.workunit.client.1.smithi086.stdout:2/814: creat d0/d20/d27/d2b/d35/d36/f124 x:0 0 0 2024-04-06T14:17:21.087 INFO:tasks.workunit.client.1.smithi086.stdout:7/754: truncate d7/d10/d92/f9f 1108043 0 2024-04-06T14:17:21.088 INFO:tasks.workunit.client.1.smithi086.stdout:6/952: dwrite d6/d13/d1a/da0/d114/db2/fbe [0,4194304] 0 2024-04-06T14:17:21.089 INFO:tasks.workunit.client.1.smithi086.stdout:9/686: getdents d5/d13/d1a/d5f/d88/d91/db9 0 2024-04-06T14:17:21.089 INFO:tasks.workunit.client.1.smithi086.stdout:8/847: mknod d0/d71/d22/d43/c129 0 2024-04-06T14:17:21.094 INFO:tasks.workunit.client.1.smithi086.stdout:3/826: rename d5/d33/d5d/dd9 to d5/d26/d110 0 2024-04-06T14:17:21.095 INFO:tasks.workunit.client.1.smithi086.stdout:3/827: dread - d5/d36/d5c/d8e/dc1/ff1 zero size 2024-04-06T14:17:21.095 INFO:tasks.workunit.client.1.smithi086.stdout:3/828: chown d5/c14 344771 1 2024-04-06T14:17:21.096 INFO:tasks.workunit.client.1.smithi086.stdout:4/654: unlink d0/de/lf 0 2024-04-06T14:17:21.097 INFO:tasks.workunit.client.1.smithi086.stdout:5/686: mknod d0/d16/d18/d35/d9e/dad/cdf 0 2024-04-06T14:17:21.100 INFO:tasks.workunit.client.0.smithi032.stdout:4/977: dwrite d0/df/d23/d31/d50/d5e/fbf [0,4194304] 0 2024-04-06T14:17:21.103 INFO:tasks.workunit.client.1.smithi086.stdout:6/953: rmdir d6/d13/d1a 39 2024-04-06T14:17:21.105 INFO:tasks.workunit.client.1.smithi086.stdout:8/848: unlink d0/d3b/db9/fd7 0 2024-04-06T14:17:21.106 INFO:tasks.workunit.client.1.smithi086.stdout:9/687: truncate d5/d13/d1a/d5f/d88/d14/d19/d25/f2f 488554 0 2024-04-06T14:17:21.108 INFO:tasks.workunit.client.1.smithi086.stdout:9/688: chown d5/d13/d1a/d5f/d88/d14/d90 102459098 1 2024-04-06T14:17:21.110 INFO:tasks.workunit.client.0.smithi032.stdout:4/978: rename d0/df/d23/d31/d50/d5e/da3/d138 to d0/df/d1f/d14a 0 2024-04-06T14:17:21.111 INFO:tasks.workunit.client.1.smithi086.stdout:9/689: write d5/d13/d1a/d5f/d88/d91/db9/fd9 [408921,106440] 0 2024-04-06T14:17:21.111 INFO:tasks.workunit.client.1.smithi086.stdout:4/655: fsync d0/d15/f37 0 2024-04-06T14:17:21.113 INFO:tasks.workunit.client.1.smithi086.stdout:7/755: mkdir d7/d10/d11/d1c/d104 0 2024-04-06T14:17:21.115 INFO:tasks.workunit.client.1.smithi086.stdout:0/835: dread d0/d1/d2/d1a/f94 [0,4194304] 0 2024-04-06T14:17:21.117 INFO:tasks.workunit.client.0.smithi032.stdout:5/943: dread d16/f2c [0,4194304] 0 2024-04-06T14:17:21.118 INFO:tasks.workunit.client.0.smithi032.stdout:4/979: dwrite d0/df/d1f/d2e/f94 [0,4194304] 0 2024-04-06T14:17:21.120 INFO:tasks.workunit.client.1.smithi086.stdout:2/815: dread d0/f8e [0,4194304] 0 2024-04-06T14:17:21.120 INFO:tasks.workunit.client.0.smithi032.stdout:4/980: read - d0/df/d23/d31/f125 zero size 2024-04-06T14:17:21.125 INFO:tasks.workunit.client.0.smithi032.stdout:4/981: creat d0/df/d2b/dbe/d10e/f14b x:0 0 0 2024-04-06T14:17:21.125 INFO:tasks.workunit.client.1.smithi086.stdout:8/849: truncate d0/d71/f108 2467089 0 2024-04-06T14:17:21.127 INFO:tasks.workunit.client.1.smithi086.stdout:7/756: dwrite d7/d10/d11/d1c/d24/d2c/d69/d83/fab [0,4194304] 0 2024-04-06T14:17:21.127 INFO:tasks.workunit.client.0.smithi032.stdout:4/982: dread - d0/df/d23/d31/f125 zero size 2024-04-06T14:17:21.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:20 smithi086 ceph-mon[47925]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:21.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:20 smithi086 ceph-mon[47925]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:21.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:20 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/504397798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:17:21.129 INFO:tasks.workunit.client.0.smithi032.stdout:5/944: dwrite d16/d18/d33/d35/f12f [0,4194304] 0 2024-04-06T14:17:21.129 INFO:tasks.workunit.client.0.smithi032.stdout:4/983: write d0/df/d76/f61 [3677296,69278] 0 2024-04-06T14:17:21.130 INFO:tasks.workunit.client.1.smithi086.stdout:9/690: unlink d5/d13/d1a/c65 0 2024-04-06T14:17:21.132 INFO:tasks.workunit.client.1.smithi086.stdout:0/836: dread d0/d1/d2/d1a/d79/dac/f112 [0,4194304] 0 2024-04-06T14:17:21.132 INFO:tasks.workunit.client.1.smithi086.stdout:5/687: link d0/d16/d18/d21/laf d0/d16/d18/d21/d23/d28/d94/le0 0 2024-04-06T14:17:21.133 INFO:tasks.workunit.client.1.smithi086.stdout:4/656: creat d0/d2c/d30/d24/da3/d88/fcc x:0 0 0 2024-04-06T14:17:21.134 INFO:tasks.workunit.client.1.smithi086.stdout:7/757: chown d7/d10/d11/d1c/d36/cf9 92127 1 2024-04-06T14:17:21.139 INFO:tasks.workunit.client.1.smithi086.stdout:9/691: read d5/d13/d1a/f24 [3756385,105122] 0 2024-04-06T14:17:21.139 INFO:tasks.workunit.client.1.smithi086.stdout:6/954: dread d6/d9/d43/fed [0,4194304] 0 2024-04-06T14:17:21.143 INFO:tasks.workunit.client.1.smithi086.stdout:3/829: dwrite d5/d36/d5a/d6d/f96 [0,4194304] 0 2024-04-06T14:17:21.144 INFO:tasks.workunit.client.0.smithi032.stdout:4/984: dwrite d0/d82/d8e/f120 [0,4194304] 0 2024-04-06T14:17:21.145 INFO:tasks.workunit.client.1.smithi086.stdout:6/955: truncate d6/f7 10336787 0 2024-04-06T14:17:21.145 INFO:tasks.workunit.client.1.smithi086.stdout:2/816: rename d0/d20/d27/d37/d52/d9c to d0/d20/d44/dbb/dc4/de3/d76/d78/d125 0 2024-04-06T14:17:21.149 INFO:tasks.workunit.client.1.smithi086.stdout:0/837: symlink d0/d15/d4e/da3/d44/d8e/l113 0 2024-04-06T14:17:21.151 INFO:tasks.workunit.client.1.smithi086.stdout:6/956: creat d6/f14d x:0 0 0 2024-04-06T14:17:21.152 INFO:tasks.workunit.client.1.smithi086.stdout:0/838: readlink d0/d15/d4e/da3/la4 0 2024-04-06T14:17:21.154 INFO:tasks.workunit.client.1.smithi086.stdout:7/758: rename d7/d10/d11/d1c/d24/d2c/d69/d83/fab to d7/d10/d11/d1c/d24/d2c/d69/d8d/f105 0 2024-04-06T14:17:21.155 INFO:tasks.workunit.client.0.smithi032.stdout:5/945: dread d16/d18/d33/f56 [0,4194304] 0 2024-04-06T14:17:21.157 INFO:tasks.workunit.client.0.smithi032.stdout:5/946: dread - d16/d58/fc9 zero size 2024-04-06T14:17:21.157 INFO:tasks.workunit.client.1.smithi086.stdout:9/692: dwrite d5/d13/d1a/f24 [0,4194304] 0 2024-04-06T14:17:21.157 INFO:tasks.workunit.client.1.smithi086.stdout:0/839: write d0/d15/d4e/f42 [444267,40481] 0 2024-04-06T14:17:21.158 INFO:tasks.workunit.client.0.smithi032.stdout:4/985: write d0/df/d2b/f3b [3236906,105817] 0 2024-04-06T14:17:21.161 INFO:tasks.workunit.client.0.smithi032.stdout:4/986: mkdir d0/d82/d4b/d14c 0 2024-04-06T14:17:21.166 INFO:tasks.workunit.client.1.smithi086.stdout:3/830: dwrite d5/f3e [0,4194304] 0 2024-04-06T14:17:21.169 INFO:tasks.workunit.client.1.smithi086.stdout:2/817: creat d0/df/d5e/d9d/de7/f126 x:0 0 0 2024-04-06T14:17:21.172 INFO:tasks.workunit.client.1.smithi086.stdout:6/957: readlink d6/d13/d1a/d64/d85/lb6 0 2024-04-06T14:17:21.175 INFO:tasks.workunit.client.0.smithi032.stdout:4/987: write d0/df/d1f/d2e/f9b [2236245,35174] 0 2024-04-06T14:17:21.176 INFO:tasks.workunit.client.0.smithi032.stdout:4/988: chown d0/df/d1f/d2e/fa5 54 1 2024-04-06T14:17:21.177 INFO:tasks.workunit.client.0.smithi032.stdout:5/947: dread d16/d18/f3d [0,4194304] 0 2024-04-06T14:17:21.178 INFO:tasks.workunit.client.0.smithi032.stdout:4/989: dread - d0/df/d23/f135 zero size 2024-04-06T14:17:21.178 INFO:tasks.workunit.client.1.smithi086.stdout:9/693: truncate d5/d13/f8c 144590 0 2024-04-06T14:17:21.179 INFO:tasks.workunit.client.0.smithi032.stdout:4/990: truncate d0/df/d2b/d43/f11e 18742 0 2024-04-06T14:17:21.180 INFO:tasks.workunit.client.0.smithi032.stdout:5/948: symlink d16/d58/d54/d74/l147 0 2024-04-06T14:17:21.181 INFO:tasks.workunit.client.0.smithi032.stdout:4/991: creat d0/df/d2b/dbe/d10e/f14d x:0 0 0 2024-04-06T14:17:21.182 INFO:tasks.workunit.client.1.smithi086.stdout:8/850: dwrite d0/d3b/f3d [0,4194304] 0 2024-04-06T14:17:21.182 INFO:tasks.workunit.client.1.smithi086.stdout:5/688: dread d0/d16/d18/d2a/d50/f73 [0,4194304] 0 2024-04-06T14:17:21.183 INFO:tasks.workunit.client.0.smithi032.stdout:5/949: dread d16/d18/f1e [0,4194304] 0 2024-04-06T14:17:21.186 INFO:tasks.workunit.client.1.smithi086.stdout:7/759: dwrite d7/d10/d102/d5c/d66/f85 [0,4194304] 0 2024-04-06T14:17:21.190 INFO:tasks.workunit.client.1.smithi086.stdout:0/840: write d0/d1/d4/de/f2e [3194618,48423] 0 2024-04-06T14:17:21.190 INFO:tasks.workunit.client.1.smithi086.stdout:3/831: truncate d5/d26/d7f/fc2 536584 0 2024-04-06T14:17:21.193 INFO:tasks.workunit.client.1.smithi086.stdout:4/657: dwrite d0/de/d4c/fc5 [0,4194304] 0 2024-04-06T14:17:21.194 INFO:tasks.workunit.client.0.smithi032.stdout:4/992: write d0/df/d23/f60 [261158,124559] 0 2024-04-06T14:17:21.197 INFO:tasks.workunit.client.1.smithi086.stdout:5/689: creat d0/d16/d18/d21/d23/d28/dc2/fe1 x:0 0 0 2024-04-06T14:17:21.199 INFO:tasks.workunit.client.1.smithi086.stdout:9/694: getdents d5/d13/d47/d5e 0 2024-04-06T14:17:21.199 INFO:tasks.workunit.client.1.smithi086.stdout:4/658: write d0/d15/d20/f57 [1514557,35592] 0 2024-04-06T14:17:21.199 INFO:tasks.workunit.client.1.smithi086.stdout:8/851: read d0/d71/d16/d52/dd0/ddc/de7/dd6/fe6 [1077973,114837] 0 2024-04-06T14:17:21.199 INFO:tasks.workunit.client.1.smithi086.stdout:2/818: dwrite d0/f33 [0,4194304] 0 2024-04-06T14:17:21.201 INFO:tasks.workunit.client.1.smithi086.stdout:7/760: rename d7/d10/d102/daf to d7/d10/d102/daf/db5/d7a/d106 22 2024-04-06T14:17:21.201 INFO:tasks.workunit.client.0.smithi032.stdout:4/993: symlink d0/df/d2b/d43/l14e 0 2024-04-06T14:17:21.202 INFO:tasks.workunit.client.1.smithi086.stdout:5/690: stat d0/d43/f9c 0 2024-04-06T14:17:21.202 INFO:tasks.workunit.client.0.smithi032.stdout:4/994: chown d0/df/d2b/dbe/d10e 975373581 1 2024-04-06T14:17:21.204 INFO:tasks.workunit.client.1.smithi086.stdout:0/841: creat d0/d1/dfe/f114 x:0 0 0 2024-04-06T14:17:21.204 INFO:tasks.workunit.client.0.smithi032.stdout:5/950: dwrite d16/d18/d33/d45/fc0 [0,4194304] 0 2024-04-06T14:17:21.206 INFO:tasks.workunit.client.0.smithi032.stdout:5/951: stat d16/d18/dae/dbb/ddc/fe7 0 2024-04-06T14:17:21.207 INFO:tasks.workunit.client.1.smithi086.stdout:9/695: write d5/d13/f7c [744023,56628] 0 2024-04-06T14:17:21.210 INFO:tasks.workunit.client.0.smithi032.stdout:4/995: mknod d0/df/c14f 0 2024-04-06T14:17:21.213 INFO:tasks.workunit.client.1.smithi086.stdout:8/852: symlink d0/d13/db6/l12a 0 2024-04-06T14:17:21.213 INFO:tasks.workunit.client.1.smithi086.stdout:5/691: chown d0/d16/d18/d21/d23/d28/dc2/c90 3692 1 2024-04-06T14:17:21.213 INFO:tasks.workunit.client.1.smithi086.stdout:2/819: dread - d0/d20/d44/dbb/dc4/de3/d76/d78/f8c zero size 2024-04-06T14:17:21.214 INFO:tasks.workunit.client.1.smithi086.stdout:0/842: symlink d0/d15/d4e/l115 0 2024-04-06T14:17:21.215 INFO:tasks.workunit.client.1.smithi086.stdout:7/761: creat d7/d10/d11/d1c/d24/d2c/d69/f107 x:0 0 0 2024-04-06T14:17:21.217 INFO:tasks.workunit.client.0.smithi032.stdout:2/962: sync 2024-04-06T14:17:21.218 INFO:tasks.workunit.client.0.smithi032.stdout:3/985: sync 2024-04-06T14:17:21.219 INFO:tasks.workunit.client.1.smithi086.stdout:4/659: link d0/d2c/d38/faa d0/d15/d20/d34/fcd 0 2024-04-06T14:17:21.219 INFO:tasks.workunit.client.0.smithi032.stdout:4/996: dwrite d0/df/d23/d31/fd6 [0,4194304] 0 2024-04-06T14:17:21.220 INFO:tasks.workunit.client.0.smithi032.stdout:3/986: dread - d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/f131 zero size 2024-04-06T14:17:21.220 INFO:tasks.workunit.client.0.smithi032.stdout:4/997: dread - d0/d82/d8e/ffc zero size 2024-04-06T14:17:21.221 INFO:tasks.workunit.client.1.smithi086.stdout:3/832: dwrite d5/dea/dd2/ddb/de6/f6a [8388608,4194304] 0 2024-04-06T14:17:21.221 INFO:tasks.workunit.client.0.smithi032.stdout:2/963: mkdir d0/d30/d39/d59/d140 0 2024-04-06T14:17:21.222 INFO:tasks.workunit.client.1.smithi086.stdout:7/762: creat d7/d10/d102/d5c/d66/d7e/f108 x:0 0 0 2024-04-06T14:17:21.224 INFO:tasks.workunit.client.1.smithi086.stdout:5/692: truncate d0/d16/d18/d21/d23/d28/d94/f4f 6331490 0 2024-04-06T14:17:21.227 INFO:tasks.workunit.client.1.smithi086.stdout:0/843: fsync d0/d15/d4e/da3/d44/d5d/f5f 0 2024-04-06T14:17:21.227 INFO:tasks.workunit.client.1.smithi086.stdout:5/693: readlink d0/d16/d18/d21/d23/d28/d94/l32 0 2024-04-06T14:17:21.229 INFO:tasks.workunit.client.0.smithi032.stdout:3/987: rename d3/fc1 to d3/d16/ddd/df7/d147/d7d/d9c/dbc/dcd/f175 0 2024-04-06T14:17:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:20 smithi032 ceph-mon[39730]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:20 smithi032 ceph-mon[39730]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:20 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/504397798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:17:21.231 INFO:tasks.workunit.client.0.smithi032.stdout:2/964: creat d0/d4e/f141 x:0 0 0 2024-04-06T14:17:21.231 INFO:tasks.workunit.client.1.smithi086.stdout:3/833: creat d5/d36/d65/f111 x:0 0 0 2024-04-06T14:17:21.231 INFO:tasks.workunit.client.1.smithi086.stdout:2/820: rename d0/d20/d27/d2b/d35/d36/d4a/db8/fc1 to d0/df/dd2/d10e/f127 0 2024-04-06T14:17:21.231 INFO:tasks.workunit.client.1.smithi086.stdout:8/853: dwrite d0/d71/d22/d43/d6d/f80 [0,4194304] 0 2024-04-06T14:17:21.232 INFO:tasks.workunit.client.1.smithi086.stdout:7/763: chown d7/d10/d102/d5c/d66/c9c 468 1 2024-04-06T14:17:21.234 INFO:tasks.workunit.client.1.smithi086.stdout:9/696: getdents d5/d13/d1a/d5f/d88/d14/d19/d25/d74 0 2024-04-06T14:17:21.235 INFO:tasks.workunit.client.0.smithi032.stdout:5/952: dwrite d16/d58/d54/d74/f111 [0,4194304] 0 2024-04-06T14:17:21.242 INFO:tasks.workunit.client.1.smithi086.stdout:3/834: fsync d5/dea/db4/da7/f63 0 2024-04-06T14:17:21.244 INFO:tasks.workunit.client.1.smithi086.stdout:8/854: read d0/d71/d16/d52/dd0/fb4 [28062,49528] 0 2024-04-06T14:17:21.244 INFO:tasks.workunit.client.0.smithi032.stdout:3/988: dwrite d3/d16/ddd/df7/d147/d7d/d9c/dad/dc5/fcc [4194304,4194304] 0 2024-04-06T14:17:21.244 INFO:tasks.workunit.client.1.smithi086.stdout:5/694: symlink d0/d16/le2 0 2024-04-06T14:17:21.246 INFO:tasks.workunit.client.1.smithi086.stdout:2/821: fdatasync d0/d20/d44/dbb/dc4/de3/d76/f91 0 2024-04-06T14:17:21.247 INFO:tasks.workunit.client.0.smithi032.stdout:3/989: dread - d3/d16/ddd/df7/d147/d7d/dc6/df0/f16a zero size 2024-04-06T14:17:21.247 INFO:tasks.workunit.client.0.smithi032.stdout:4/998: truncate d0/df/d23/d31/d50/d57/f6a 746319 0 2024-04-06T14:17:21.247 INFO:tasks.workunit.client.1.smithi086.stdout:0/844: dwrite d0/d1/d4/de/fb0 [0,4194304] 0 2024-04-06T14:17:21.248 INFO:tasks.workunit.client.0.smithi032.stdout:3/990: fsync d3/d16/f70 0 2024-04-06T14:17:21.248 INFO:tasks.workunit.client.0.smithi032.stdout:4/999: dread - d0/df/d1f/d5a/f142 zero size 2024-04-06T14:17:21.250 INFO:tasks.workunit.client.1.smithi086.stdout:7/764: rename d7/d10/d92/f9f to d7/d10/d11/d1c/d24/d2c/d69/d8d/f109 0 2024-04-06T14:17:21.254 INFO:tasks.workunit.client.1.smithi086.stdout:5/695: symlink d0/d16/d18/d21/d23/d28/d94/d8a/le3 0 2024-04-06T14:17:21.256 INFO:tasks.workunit.client.1.smithi086.stdout:7/765: write d7/d10/d11/d1c/d24/d2c/f70 [2115606,97533] 0 2024-04-06T14:17:21.258 INFO:tasks.workunit.client.0.smithi032.stdout:3/991: creat d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/f176 x:0 0 0 2024-04-06T14:17:21.260 INFO:tasks.workunit.client.1.smithi086.stdout:5/696: stat d0/d17/fac 0 2024-04-06T14:17:21.263 INFO:tasks.workunit.client.1.smithi086.stdout:7/766: rename d7/d10/d102/d5c/d66/f67 to d7/d10/d102/daf/db5/d7a/d100/f10a 0 2024-04-06T14:17:21.264 INFO:tasks.workunit.client.0.smithi032.stdout:2/965: write d0/d6e/f72 [313455,413] 0 2024-04-06T14:17:21.266 INFO:tasks.workunit.client.1.smithi086.stdout:9/697: dread d5/d18/f32 [0,4194304] 0 2024-04-06T14:17:21.271 INFO:tasks.workunit.client.1.smithi086.stdout:8/855: rename d0/d71/d16/d52/dd0/ddc/de7/l49 to d0/d71/d16/d52/dd0/ddc/de7/dd6/l12b 0 2024-04-06T14:17:21.271 INFO:tasks.workunit.client.0.smithi032.stdout:2/966: dread d0/d30/d39/d59/d67/d74/d78/d57/f90 [0,4194304] 0 2024-04-06T14:17:21.272 INFO:tasks.workunit.client.0.smithi032.stdout:2/967: readlink d0/la6 0 2024-04-06T14:17:21.273 INFO:tasks.workunit.client.1.smithi086.stdout:9/698: fdatasync d5/d13/d1a/d5f/d88/d14/d19/f4b 0 2024-04-06T14:17:21.275 INFO:tasks.workunit.client.0.smithi032.stdout:2/968: symlink d0/d30/d39/d59/d67/d74/d78/d57/dd3/l142 0 2024-04-06T14:17:21.275 INFO:tasks.workunit.client.1.smithi086.stdout:7/767: creat d7/d10/d102/daf/db5/f10b x:0 0 0 2024-04-06T14:17:21.277 INFO:tasks.workunit.client.1.smithi086.stdout:4/660: dread d0/f3 [4194304,4194304] 0 2024-04-06T14:17:21.277 INFO:tasks.workunit.client.1.smithi086.stdout:8/856: dread - d0/d71/d22/d43/d6d/d75/dee/f115 zero size 2024-04-06T14:17:21.280 INFO:tasks.workunit.client.1.smithi086.stdout:5/697: dwrite d0/d16/d18/d2a/d50/d66/fbe [0,4194304] 0 2024-04-06T14:17:21.285 INFO:tasks.workunit.client.0.smithi032.stdout:2/969: write d0/d30/d39/d59/d67/fc0 [922586,94204] 0 2024-04-06T14:17:21.287 INFO:tasks.workunit.client.1.smithi086.stdout:0/845: dread d0/f3f [0,4194304] 0 2024-04-06T14:17:21.289 INFO:tasks.workunit.client.1.smithi086.stdout:5/698: creat d0/d16/d18/d21/d23/d28/d94/fe4 x:0 0 0 2024-04-06T14:17:21.290 INFO:tasks.workunit.client.0.smithi032.stdout:3/992: dread d3/d16/ddd/df7/d147/d7d/d9c/dad/d111/f140 [0,4194304] 0 2024-04-06T14:17:21.291 INFO:tasks.workunit.client.1.smithi086.stdout:8/857: getdents d0/d3b/d79/dc6/dbc 0 2024-04-06T14:17:21.291 INFO:tasks.workunit.client.0.smithi032.stdout:2/970: getdents d0/d30/d39/d59/d67/d74/d78/d57 0 2024-04-06T14:17:21.292 INFO:tasks.workunit.client.1.smithi086.stdout:4/661: write d0/de/f26 [4657662,2387] 0 2024-04-06T14:17:21.293 INFO:tasks.workunit.client.1.smithi086.stdout:7/768: dwrite d7/d10/d11/d1c/d24/dc8/ffe [0,4194304] 0 2024-04-06T14:17:21.295 INFO:tasks.workunit.client.0.smithi032.stdout:3/993: mkdir d3/d16/ddd/df7/d147/d7d/d9c/dbc/dcd/d177 0 2024-04-06T14:17:21.295 INFO:tasks.workunit.client.1.smithi086.stdout:0/846: creat d0/d15/d4e/da3/d44/df4/f116 x:0 0 0 2024-04-06T14:17:21.297 INFO:tasks.workunit.client.0.smithi032.stdout:2/971: creat d0/d5a/f143 x:0 0 0 2024-04-06T14:17:21.297 INFO:tasks.workunit.client.1.smithi086.stdout:2/822: dread d0/d20/d27/d37/d52/feb [0,4194304] 0 2024-04-06T14:17:21.299 INFO:tasks.workunit.client.0.smithi032.stdout:3/994: read d3/d16/ddd/df7/d147/d7d/d9c/dad/db1/d3c/f57 [1241339,54056] 0 2024-04-06T14:17:21.300 INFO:tasks.workunit.client.1.smithi086.stdout:5/699: rename d0/d16/d18/d21/d23/d28/dc2 to d0/d43/de5 0 2024-04-06T14:17:21.302 INFO:tasks.workunit.client.0.smithi032.stdout:2/972: symlink d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/l144 0 2024-04-06T14:17:21.303 INFO:tasks.workunit.client.0.smithi032.stdout:3/995: mkdir d3/d16/ddd/df7/d147/d7d/d9c/dbc/dcd/d178 0 2024-04-06T14:17:21.303 INFO:tasks.workunit.client.1.smithi086.stdout:4/662: truncate d0/d2c/d30/f6c 1131122 0 2024-04-06T14:17:21.305 INFO:tasks.workunit.client.1.smithi086.stdout:2/823: mkdir d0/d20/d27/d2b/d35/d36/d4d/da5/d128 0 2024-04-06T14:17:21.305 INFO:tasks.workunit.client.1.smithi086.stdout:0/847: write d0/d15/d4e/da3/d46/f6d [403432,68658] 0 2024-04-06T14:17:21.308 INFO:tasks.workunit.client.0.smithi032.stdout:3/996: creat d3/d16/ddd/df7/d147/d7d/dc6/df0/d16b/f179 x:0 0 0 2024-04-06T14:17:21.309 INFO:tasks.workunit.client.1.smithi086.stdout:8/858: getdents d0/d71/d22/d43/daa 0 2024-04-06T14:17:21.310 INFO:tasks.workunit.client.1.smithi086.stdout:7/769: dwrite d7/d10/d11/d1c/f1e [4194304,4194304] 0 2024-04-06T14:17:21.311 INFO:tasks.workunit.client.0.smithi032.stdout:3/997: creat d3/d16/ddd/df7/d147/d7d/d9c/dbc/d128/f17a x:0 0 0 2024-04-06T14:17:21.314 INFO:tasks.workunit.client.1.smithi086.stdout:7/770: stat d7/d10/d102/d5c 0 2024-04-06T14:17:21.317 INFO:tasks.workunit.client.1.smithi086.stdout:2/824: dread - d0/d20/d27/d2b/d35/d36/d4d/d6f/f81 zero size 2024-04-06T14:17:21.318 INFO:tasks.workunit.client.1.smithi086.stdout:0/848: dread - d0/d15/d26/d33/d9f/fbf zero size 2024-04-06T14:17:21.319 INFO:tasks.workunit.client.0.smithi032.stdout:2/973: dread d0/d30/d39/d59/d67/d74/f9a [0,4194304] 0 2024-04-06T14:17:21.320 INFO:tasks.workunit.client.1.smithi086.stdout:9/699: dread d5/d13/d47/d5e/f61 [0,4194304] 0 2024-04-06T14:17:21.320 INFO:tasks.workunit.client.1.smithi086.stdout:8/859: symlink d0/d71/d22/d65/d6e/d98/ddb/l12c 0 2024-04-06T14:17:21.321 INFO:tasks.workunit.client.1.smithi086.stdout:5/700: dwrite d0/d16/d18/d2a/d50/d66/fcd [0,4194304] 0 2024-04-06T14:17:21.322 INFO:tasks.workunit.client.1.smithi086.stdout:7/771: write d7/d10/d92/ff8 [404959,9347] 0 2024-04-06T14:17:21.325 INFO:tasks.workunit.client.1.smithi086.stdout:2/825: write d0/d20/d27/d2b/d35/d36/d4d/d59/f118 [124756,65038] 0 2024-04-06T14:17:21.327 INFO:tasks.workunit.client.0.smithi032.stdout:2/974: rename d0/d30/d39/d59/d67/d74/d78/d57/d7c/de6/cee to d0/d30/d39/d59/d67/d74/d78/d119/c145 0 2024-04-06T14:17:21.329 INFO:tasks.workunit.client.1.smithi086.stdout:5/701: mkdir d0/d16/d18/de6 0 2024-04-06T14:17:21.330 INFO:tasks.workunit.client.0.smithi032.stdout:3/998: dwrite d3/d16/ddd/df7/d147/d7d/d9c/dad/f116 [0,4194304] 0 2024-04-06T14:17:21.330 INFO:tasks.workunit.client.1.smithi086.stdout:0/849: getdents d0/d15/d4e/da3/d73 0 2024-04-06T14:17:21.331 INFO:tasks.workunit.client.1.smithi086.stdout:7/772: unlink d7/d10/d11/d1c/d36/c61 0 2024-04-06T14:17:21.331 INFO:tasks.workunit.client.0.smithi032.stdout:2/975: write d0/d6e/dae/db7/df9/f110 [415519,103404] 0 2024-04-06T14:17:21.332 INFO:tasks.workunit.client.1.smithi086.stdout:5/702: rmdir d0/d16/d18/d2a 39 2024-04-06T14:17:21.333 INFO:tasks.workunit.client.1.smithi086.stdout:2/826: mknod d0/d20/d44/d120/c129 0 2024-04-06T14:17:21.335 INFO:tasks.workunit.client.1.smithi086.stdout:1/828: sync 2024-04-06T14:17:21.336 INFO:tasks.workunit.client.0.smithi032.stdout:2/976: read d0/d5a/f2a [228090,98579] 0 2024-04-06T14:17:21.338 INFO:tasks.workunit.client.1.smithi086.stdout:0/850: creat d0/d1/d2/d1a/d79/dac/db7/f117 x:0 0 0 2024-04-06T14:17:21.339 INFO:tasks.workunit.client.1.smithi086.stdout:0/851: dread - d0/d15/d26/d33/d9f/f101 zero size 2024-04-06T14:17:21.340 INFO:tasks.workunit.client.0.smithi032.stdout:2/977: creat d0/d30/d39/f146 x:0 0 0 2024-04-06T14:17:21.341 INFO:tasks.workunit.client.1.smithi086.stdout:7/773: rmdir d7/d10/d11/d1c/df4 0 2024-04-06T14:17:21.342 INFO:tasks.workunit.client.1.smithi086.stdout:4/663: dread d0/d15/d20/f44 [4194304,4194304] 0 2024-04-06T14:17:21.343 INFO:tasks.workunit.client.1.smithi086.stdout:0/852: chown d0/d15/d4e/da3/d44/d10f 466229 1 2024-04-06T14:17:21.344 INFO:tasks.workunit.client.0.smithi032.stdout:2/978: creat d0/d30/d39/d59/d67/d74/d78/d8c/db0/dd7/dbe/f147 x:0 0 0 2024-04-06T14:17:21.344 INFO:tasks.workunit.client.1.smithi086.stdout:2/827: mknod d0/d20/d44/dbb/dc4/de3/c12a 0 2024-04-06T14:17:21.345 INFO:tasks.workunit.client.1.smithi086.stdout:4/664: fsync d0/d2c/d38/d54/d83/f77 0 2024-04-06T14:17:21.347 INFO:tasks.workunit.client.1.smithi086.stdout:7/774: chown d7/d10/d102/d5c/d66/cc6 66973591 1 2024-04-06T14:17:21.348 INFO:tasks.workunit.client.0.smithi032.stdout:2/979: mkdir d0/d148 0 2024-04-06T14:17:21.348 INFO:tasks.workunit.client.1.smithi086.stdout:0/853: creat d0/d1/d2/d1a/d51/dd7/f118 x:0 0 0 2024-04-06T14:17:21.349 INFO:tasks.workunit.client.1.smithi086.stdout:9/700: dread d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f81 [0,4194304] 0 2024-04-06T14:17:21.350 INFO:tasks.workunit.client.1.smithi086.stdout:4/665: write d0/d2c/d30/d24/d50/d79/fa8 [608678,34424] 0 2024-04-06T14:17:21.351 INFO:tasks.workunit.client.0.smithi032.stdout:2/980: symlink d0/db2/dfc/l149 0 2024-04-06T14:17:21.352 INFO:tasks.workunit.client.1.smithi086.stdout:0/854: write d0/d1/d2/d1a/d51/f86 [926949,123505] 0 2024-04-06T14:17:21.354 INFO:tasks.workunit.client.1.smithi086.stdout:9/701: unlink d5/d18/cd7 0 2024-04-06T14:17:21.355 INFO:tasks.workunit.client.0.smithi032.stdout:2/981: creat d0/d30/d39/d59/d67/d74/d78/d8c/f14a x:0 0 0 2024-04-06T14:17:21.357 INFO:tasks.workunit.client.1.smithi086.stdout:5/703: dread d0/d43/de5/f3e [0,4194304] 0 2024-04-06T14:17:21.358 INFO:tasks.workunit.client.1.smithi086.stdout:4/666: creat d0/d2c/d38/d54/d61/d7a/dc2/fce x:0 0 0 2024-04-06T14:17:21.358 INFO:tasks.workunit.client.0.smithi032.stdout:2/982: creat d0/d6e/f14b x:0 0 0 2024-04-06T14:17:21.359 INFO:tasks.workunit.client.0.smithi032.stdout:2/983: chown d0/l2e 3347 1 2024-04-06T14:17:21.359 INFO:tasks.workunit.client.1.smithi086.stdout:0/855: rmdir d0/d15/d4e/da3/d44 39 2024-04-06T14:17:21.361 INFO:tasks.workunit.client.0.smithi032.stdout:2/984: rmdir d0/d6e/d8d/d10d 39 2024-04-06T14:17:21.361 INFO:tasks.workunit.client.1.smithi086.stdout:2/828: dread d0/d20/d27/f10a [0,4194304] 0 2024-04-06T14:17:21.361 INFO:tasks.workunit.client.1.smithi086.stdout:1/829: dread db/d13/d71/f7f [0,4194304] 0 2024-04-06T14:17:21.361 INFO:tasks.workunit.client.1.smithi086.stdout:9/702: unlink d5/d13/d1a/d5f/d88/d14/f80 0 2024-04-06T14:17:21.363 INFO:tasks.workunit.client.0.smithi032.stdout:2/985: mkdir d0/d30/d39/d59/d14c 0 2024-04-06T14:17:21.363 INFO:tasks.workunit.client.1.smithi086.stdout:4/667: fdatasync d0/d2c/f81 0 2024-04-06T14:17:21.365 INFO:tasks.workunit.client.0.smithi032.stdout:2/986: truncate d0/d30/d39/faa 680269 0 2024-04-06T14:17:21.365 INFO:tasks.workunit.client.1.smithi086.stdout:5/704: rename d0/d17 to d0/d16/d18/d35/d9e/de7 0 2024-04-06T14:17:21.365 INFO:tasks.workunit.client.1.smithi086.stdout:7/775: dwrite d7/d10/d11/d80/f35 [0,4194304] 0 2024-04-06T14:17:21.368 INFO:tasks.workunit.client.1.smithi086.stdout:2/829: unlink d0/d20/d27/dae/c115 0 2024-04-06T14:17:21.368 INFO:tasks.workunit.client.1.smithi086.stdout:1/830: creat db/d11/d1f/d5c/ddb/de9/f107 x:0 0 0 2024-04-06T14:17:21.369 INFO:tasks.workunit.client.0.smithi032.stdout:2/987: truncate d0/d30/d39/d59/d67/d74/d78/d57/f11e 3670349 0 2024-04-06T14:17:21.371 INFO:tasks.workunit.client.1.smithi086.stdout:7/776: creat d7/d10/d102/daf/db5/d7a/d100/f10c x:0 0 0 2024-04-06T14:17:21.372 INFO:tasks.workunit.client.1.smithi086.stdout:5/705: creat d0/d16/d18/d21/d23/d28/d94/d80/d85/fe8 x:0 0 0 2024-04-06T14:17:21.372 INFO:tasks.workunit.client.0.smithi032.stdout:2/988: truncate d0/d30/d39/d59/d67/d74/d78/d57/f90 1433407 0 2024-04-06T14:17:21.373 INFO:tasks.workunit.client.1.smithi086.stdout:2/830: write d0/d20/d44/dbb/dc4/f11f [986529,99026] 0 2024-04-06T14:17:21.374 INFO:tasks.workunit.client.1.smithi086.stdout:0/856: link d0/d15/d4e/da3/fe7 d0/d1/d4/de/dae/f119 0 2024-04-06T14:17:21.376 INFO:tasks.workunit.client.0.smithi032.stdout:2/989: rename d0/l7e to d0/d4e/de9/dea/d13d/l14d 0 2024-04-06T14:17:21.377 INFO:tasks.workunit.client.1.smithi086.stdout:4/668: dwrite d0/d2c/d30/d24/d5d/f8f [0,4194304] 0 2024-04-06T14:17:21.379 INFO:tasks.workunit.client.1.smithi086.stdout:7/777: symlink d7/d10/d11/d1c/d24/dc8/dcd/l10d 0 2024-04-06T14:17:21.381 INFO:tasks.workunit.client.0.smithi032.stdout:2/990: rename d0/le1 to d0/d6e/dae/d120/l14e 0 2024-04-06T14:17:21.384 INFO:tasks.workunit.client.0.smithi032.stdout:2/991: mkdir d0/d6e/dae/db7/df9/d14f 0 2024-04-06T14:17:21.384 INFO:tasks.workunit.client.1.smithi086.stdout:0/857: readlink d0/d15/d4e/da3/d44/d8e/l113 0 2024-04-06T14:17:21.385 INFO:tasks.workunit.client.0.smithi032.stdout:2/992: read - d0/d5a/dc9/fca zero size 2024-04-06T14:17:21.385 INFO:tasks.workunit.client.0.smithi032.stdout:0/807: sync 2024-04-06T14:17:21.386 INFO:tasks.workunit.client.0.smithi032.stdout:0/808: chown d3/da/d25/d2b/l3f 172590 1 2024-04-06T14:17:21.387 INFO:tasks.workunit.client.1.smithi086.stdout:2/831: dwrite d0/d20/d44/dbb/fdb [0,4194304] 0 2024-04-06T14:17:21.388 INFO:tasks.workunit.client.0.smithi032.stdout:2/993: truncate d0/d6e/dae/db7/ffb 626337 0 2024-04-06T14:17:21.388 INFO:tasks.workunit.client.1.smithi086.stdout:5/706: rename d0/d16/d18/d21/d23/d28/cba to d0/d43/d74/ce9 0 2024-04-06T14:17:21.389 INFO:tasks.workunit.client.1.smithi086.stdout:1/831: rmdir db/d11/d1f/d5c/ddb/de9 39 2024-04-06T14:17:21.392 INFO:tasks.workunit.client.1.smithi086.stdout:9/703: truncate d5/f20 7233099 0 2024-04-06T14:17:21.393 INFO:tasks.workunit.client.1.smithi086.stdout:2/832: rmdir d0/d20/d44/dbb/dc4/de3/d76/d78 39 2024-04-06T14:17:21.393 INFO:tasks.workunit.client.0.smithi032.stdout:2/994: dread d0/d30/ff0 [0,4194304] 0 2024-04-06T14:17:21.393 INFO:tasks.workunit.client.0.smithi032.stdout:0/809: dread d3/f4 [0,4194304] 0 2024-04-06T14:17:21.401 INFO:tasks.workunit.client.1.smithi086.stdout:5/707: mknod d0/d16/d18/d35/d9e/de7/cea 0 2024-04-06T14:17:21.402 INFO:tasks.workunit.client.1.smithi086.stdout:7/778: write d7/d10/d102/daf/db5/f98 [359521,1248] 0 2024-04-06T14:17:21.403 INFO:tasks.workunit.client.1.smithi086.stdout:5/708: chown d0/d43/de5/c1e 1639827 1 2024-04-06T14:17:21.408 INFO:tasks.workunit.client.1.smithi086.stdout:9/704: mkdir d5/d13/d1a/d5f/d88/d14/dda 0 2024-04-06T14:17:21.408 INFO:tasks.workunit.client.0.smithi032.stdout:2/995: dwrite d0/d30/d39/d59/d67/d74/d78/dc2/ff5 [0,4194304] 0 2024-04-06T14:17:21.408 INFO:tasks.workunit.client.1.smithi086.stdout:2/833: rmdir d0/d20/d80/dbf/ddc 39 2024-04-06T14:17:21.408 INFO:tasks.workunit.client.1.smithi086.stdout:9/705: fsync d5/d13/d1a/d5f/fc8 0 2024-04-06T14:17:21.409 INFO:tasks.workunit.client.1.smithi086.stdout:4/669: truncate d0/d15/d20/d74/f7b 714212 0 2024-04-06T14:17:21.410 INFO:tasks.workunit.client.1.smithi086.stdout:4/670: write d0/d2c/d30/f3b [2860072,5623] 0 2024-04-06T14:17:21.413 INFO:tasks.workunit.client.0.smithi032.stdout:0/810: dread d3/f5a [0,4194304] 0 2024-04-06T14:17:21.413 INFO:tasks.workunit.client.0.smithi032.stdout:2/996: write d0/d30/d39/d59/d67/d74/d78/d8c/f14a [74722,58484] 0 2024-04-06T14:17:21.413 INFO:tasks.workunit.client.1.smithi086.stdout:5/709: chown d0/d16/d18/d21/d23/d28/d94/f4f 5601 1 2024-04-06T14:17:21.418 INFO:tasks.workunit.client.1.smithi086.stdout:9/706: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb 0 2024-04-06T14:17:21.419 INFO:tasks.workunit.client.1.smithi086.stdout:4/671: creat d0/d15/d20/d34/fcf x:0 0 0 2024-04-06T14:17:21.421 INFO:tasks.workunit.client.1.smithi086.stdout:2/834: fsync d0/f2a 0 2024-04-06T14:17:21.422 INFO:tasks.workunit.client.1.smithi086.stdout:5/710: mkdir d0/d16/d71/d96/deb 0 2024-04-06T14:17:21.422 INFO:tasks.workunit.client.0.smithi032.stdout:0/811: getdents d3/da/d25/d2b/d4a/d79/d97/de2 0 2024-04-06T14:17:21.423 INFO:tasks.workunit.client.1.smithi086.stdout:1/832: dwrite db/d11/d1a/d2a/d9a/d68/fc0 [0,4194304] 0 2024-04-06T14:17:21.424 INFO:tasks.workunit.client.0.smithi032.stdout:2/997: dwrite d0/d30/d39/d59/d67/d74/d78/d57/f124 [0,4194304] 0 2024-04-06T14:17:21.425 INFO:tasks.workunit.client.1.smithi086.stdout:9/707: mkdir d5/d13/d47/d55/d87/ddc 0 2024-04-06T14:17:21.428 INFO:tasks.workunit.client.1.smithi086.stdout:1/833: write db/d11/d1f/fbc [1514258,108873] 0 2024-04-06T14:17:21.428 INFO:tasks.workunit.client.1.smithi086.stdout:2/835: chown d0/f26 180876 1 2024-04-06T14:17:21.432 INFO:tasks.workunit.client.0.smithi032.stdout:0/812: creat d3/da/d41/f113 x:0 0 0 2024-04-06T14:17:21.432 INFO:tasks.workunit.client.0.smithi032.stdout:2/998: dwrite d0/d6e/dae/db7/f12d [4194304,4194304] 0 2024-04-06T14:17:21.432 INFO:tasks.workunit.client.1.smithi086.stdout:7/779: dwrite d7/d1d/f13 [0,4194304] 0 2024-04-06T14:17:21.434 INFO:tasks.workunit.client.1.smithi086.stdout:1/834: fsync db/d13/f86 0 2024-04-06T14:17:21.435 INFO:tasks.workunit.client.0.smithi032.stdout:0/813: chown d3/da/d25/d2b/d4a/d86/d10a/dfe/f110 23542 1 2024-04-06T14:17:21.437 INFO:tasks.workunit.client.0.smithi032.stdout:0/814: write d3/da/d25/d2b/d4a/f62 [1349427,97169] 0 2024-04-06T14:17:21.440 INFO:tasks.workunit.client.1.smithi086.stdout:7/780: write d7/d10/d11/d80/fc9 [605294,121525] 0 2024-04-06T14:17:21.443 INFO:tasks.workunit.client.1.smithi086.stdout:7/781: readlink d7/d94/la5 0 2024-04-06T14:17:21.443 INFO:tasks.workunit.client.1.smithi086.stdout:5/711: dwrite d0/d16/d18/d2a/d50/f73 [0,4194304] 0 2024-04-06T14:17:21.444 INFO:tasks.workunit.client.0.smithi032.stdout:0/815: creat d3/da/d25/d2b/d4a/d86/f114 x:0 0 0 2024-04-06T14:17:21.444 INFO:tasks.workunit.client.1.smithi086.stdout:2/836: truncate d0/f7a 364222 0 2024-04-06T14:17:21.447 INFO:tasks.workunit.client.1.smithi086.stdout:9/708: read d5/d13/d1a/d5f/d88/d14/d19/d25/f67 [130210,105242] 0 2024-04-06T14:17:21.453 INFO:tasks.workunit.client.1.smithi086.stdout:7/782: stat d7/d10/d11/d1c/d24/d2c/fe3 0 2024-04-06T14:17:21.453 INFO:tasks.workunit.client.1.smithi086.stdout:9/709: rename d5/d13/d47/f83 to d5/d58/fdd 0 2024-04-06T14:17:21.454 INFO:tasks.workunit.client.1.smithi086.stdout:1/835: dwrite db/d11/ffe [0,4194304] 0 2024-04-06T14:17:21.455 INFO:tasks.workunit.client.1.smithi086.stdout:5/712: link d0/d16/d18/d35/f5b d0/d16/d18/fec 0 2024-04-06T14:17:21.461 INFO:tasks.workunit.client.1.smithi086.stdout:9/710: rename d5/d13/d1a/f24 to d5/d13/d47/d55/d87/ddc/fde 0 2024-04-06T14:17:21.463 INFO:tasks.workunit.client.1.smithi086.stdout:5/713: creat d0/d16/d18/d35/d9e/dad/fed x:0 0 0 2024-04-06T14:17:21.464 INFO:tasks.workunit.client.1.smithi086.stdout:1/836: mknod db/d13/c108 0 2024-04-06T14:17:21.465 INFO:tasks.workunit.client.1.smithi086.stdout:1/837: truncate db/d11/d1f/d5c/ff6 857892 0 2024-04-06T14:17:21.466 INFO:tasks.workunit.client.1.smithi086.stdout:9/711: fdatasync d5/d13/d1a/d5f/d88/d14/d19/f5d 0 2024-04-06T14:17:21.467 INFO:tasks.workunit.client.1.smithi086.stdout:5/714: mkdir d0/d43/d74/dee 0 2024-04-06T14:17:21.469 INFO:tasks.workunit.client.1.smithi086.stdout:1/838: creat db/d11/d1a/d79/da5/f109 x:0 0 0 2024-04-06T14:17:21.469 INFO:tasks.workunit.client.0.smithi032.stdout:0/816: dread d3/f63 [0,4194304] 0 2024-04-06T14:17:21.472 INFO:tasks.workunit.client.1.smithi086.stdout:9/712: creat d5/d13/d1a/d5f/d88/d14/d19/fdf x:0 0 0 2024-04-06T14:17:21.472 INFO:tasks.workunit.client.1.smithi086.stdout:4/672: dread d0/d15/d20/d34/f35 [0,4194304] 0 2024-04-06T14:17:21.473 INFO:tasks.workunit.client.1.smithi086.stdout:5/715: getdents d0/d16/d18/d21/d23/d28/d94/d8a/dbf 0 2024-04-06T14:17:21.475 INFO:tasks.workunit.client.1.smithi086.stdout:1/839: dread db/fe [4194304,4194304] 0 2024-04-06T14:17:21.477 INFO:tasks.workunit.client.1.smithi086.stdout:9/713: chown d5/d13/d1a/d5f/lac 1917258 1 2024-04-06T14:17:21.477 INFO:tasks.workunit.client.1.smithi086.stdout:4/673: dread d0/d15/d20/f27 [0,4194304] 0 2024-04-06T14:17:21.479 INFO:tasks.workunit.client.0.smithi032.stdout:2/999: dwrite d0/d30/d39/d59/d67/d74/d78/d57/d7c/f13e [0,4194304] 0 2024-04-06T14:17:21.480 INFO:tasks.workunit.client.1.smithi086.stdout:4/674: readlink d0/d2c/d38/d54/d90/l9e 0 2024-04-06T14:17:21.481 INFO:tasks.workunit.client.1.smithi086.stdout:1/840: rmdir db/d11/d1f/d5c/ddb/de9 39 2024-04-06T14:17:21.482 INFO:tasks.workunit.client.1.smithi086.stdout:2/837: dread d0/d20/d27/f43 [0,4194304] 0 2024-04-06T14:17:21.484 INFO:tasks.workunit.client.1.smithi086.stdout:4/675: chown d0/d2c/d30/d24/da3/f52 2891601 1 2024-04-06T14:17:21.484 INFO:tasks.workunit.client.1.smithi086.stdout:7/783: write d7/d10/d92/fc5 [224645,85265] 0 2024-04-06T14:17:21.485 INFO:tasks.workunit.client.1.smithi086.stdout:5/716: dwrite d0/d16/d18/d2a/d50/fc6 [0,4194304] 0 2024-04-06T14:17:21.486 INFO:tasks.workunit.client.1.smithi086.stdout:2/838: fsync d0/d20/d27/dd1/f10d 0 2024-04-06T14:17:21.488 INFO:tasks.workunit.client.1.smithi086.stdout:9/714: link d5/d13/l9e d5/d13/d1a/d5f/d88/d14/d90/le0 0 2024-04-06T14:17:21.491 INFO:tasks.workunit.client.1.smithi086.stdout:6/958: sync 2024-04-06T14:17:21.496 INFO:tasks.workunit.client.1.smithi086.stdout:6/959: write d6/d9/d5c/f111 [730358,3834] 0 2024-04-06T14:17:21.499 INFO:tasks.workunit.client.1.smithi086.stdout:7/784: creat d7/d10/d102/daf/db5/da1/f10e x:0 0 0 2024-04-06T14:17:21.499 INFO:tasks.workunit.client.1.smithi086.stdout:4/676: dwrite d0/f2 [0,4194304] 0 2024-04-06T14:17:21.502 INFO:tasks.workunit.client.1.smithi086.stdout:1/841: dread db/d11/f27 [0,4194304] 0 2024-04-06T14:17:21.505 INFO:tasks.workunit.client.1.smithi086.stdout:2/839: dwrite d0/d20/d27/d2b/d35/d36/d4d/d59/f119 [0,4194304] 0 2024-04-06T14:17:21.505 INFO:tasks.workunit.client.1.smithi086.stdout:9/715: rename d5/d13/d1a/d5f/d88/d14/d19/d25/c54 to d5/d58/d8d/ce1 0 2024-04-06T14:17:21.509 INFO:tasks.workunit.client.1.smithi086.stdout:4/677: mkdir d0/d2c/d30/d24/d50/dd0 0 2024-04-06T14:17:21.512 INFO:tasks.workunit.client.1.smithi086.stdout:1/842: creat db/d11/d1a/d2a/d9a/d68/d6e/f10a x:0 0 0 2024-04-06T14:17:21.516 INFO:tasks.workunit.client.1.smithi086.stdout:4/678: creat d0/d2c/d38/d54/d83/fd1 x:0 0 0 2024-04-06T14:17:21.517 INFO:tasks.workunit.client.1.smithi086.stdout:9/716: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dce/fe2 x:0 0 0 2024-04-06T14:17:21.519 INFO:tasks.workunit.client.1.smithi086.stdout:6/960: link d6/d13/d1a/da0/d11b/d45/d8a/l127 d6/d13/d1a/da0/d114/l14e 0 2024-04-06T14:17:21.519 INFO:tasks.workunit.client.1.smithi086.stdout:5/717: dwrite d0/d16/f39 [0,4194304] 0 2024-04-06T14:17:21.519 INFO:tasks.workunit.client.1.smithi086.stdout:1/843: fsync db/d11/d1a/d2a/d9a/d68/d6e/d81/ffc 0 2024-04-06T14:17:21.522 INFO:tasks.workunit.client.1.smithi086.stdout:1/844: write db/d11/d1a/d2a/d9a/d78/fe7 [1200446,39380] 0 2024-04-06T14:17:21.522 INFO:tasks.workunit.client.1.smithi086.stdout:2/840: creat d0/d20/d80/f12b x:0 0 0 2024-04-06T14:17:21.524 INFO:tasks.workunit.client.1.smithi086.stdout:4/679: chown d0/d2c/d30/d24/da3/l58 2210 1 2024-04-06T14:17:21.526 INFO:tasks.workunit.client.1.smithi086.stdout:5/718: symlink d0/d16/d18/d35/lef 0 2024-04-06T14:17:21.529 INFO:tasks.workunit.client.1.smithi086.stdout:9/717: creat d5/d13/d1a/d5f/d88/db3/fe3 x:0 0 0 2024-04-06T14:17:21.531 INFO:tasks.workunit.client.1.smithi086.stdout:2/841: unlink d0/d20/d27/d2b/d35/d36/f3a 0 2024-04-06T14:17:21.533 INFO:tasks.workunit.client.1.smithi086.stdout:7/785: dread d7/d10/d11/d1c/f60 [0,4194304] 0 2024-04-06T14:17:21.533 INFO:tasks.workunit.client.1.smithi086.stdout:4/680: mkdir d0/d2c/d38/d54/d83/dd2 0 2024-04-06T14:17:21.534 INFO:tasks.workunit.client.1.smithi086.stdout:4/681: fdatasync d0/d15/f48 0 2024-04-06T14:17:21.535 INFO:tasks.workunit.client.1.smithi086.stdout:1/845: fdatasync db/d13/f92 0 2024-04-06T14:17:21.537 INFO:tasks.workunit.client.1.smithi086.stdout:9/718: fdatasync d5/d13/d1a/d8f/fb8 0 2024-04-06T14:17:21.537 INFO:tasks.workunit.client.1.smithi086.stdout:1/846: write db/d13/d71/fd0 [539373,113314] 0 2024-04-06T14:17:21.542 INFO:tasks.workunit.client.1.smithi086.stdout:4/682: truncate d0/d2c/d30/d24/da3/f52 507663 0 2024-04-06T14:17:21.544 INFO:tasks.workunit.client.1.smithi086.stdout:1/847: creat db/d11/d87/f10b x:0 0 0 2024-04-06T14:17:21.546 INFO:tasks.workunit.client.1.smithi086.stdout:1/848: readlink db/d13/d55/l94 0 2024-04-06T14:17:21.546 INFO:tasks.workunit.client.1.smithi086.stdout:7/786: link d7/d10/d11/d21/db0/lf7 d7/d10/d11/d1c/d24/l10f 0 2024-04-06T14:17:21.547 INFO:tasks.workunit.client.1.smithi086.stdout:1/849: stat db/d13/d55 0 2024-04-06T14:17:21.548 INFO:tasks.workunit.client.1.smithi086.stdout:2/842: link d0/d20/d44/dbb/dc4/de3/d76/d78/d125/fd9 d0/df/d48/f12c 0 2024-04-06T14:17:21.549 INFO:tasks.workunit.client.1.smithi086.stdout:7/787: stat d7/d10/d11/d1c/d36 0 2024-04-06T14:17:21.552 INFO:tasks.workunit.client.1.smithi086.stdout:4/683: dwrite d0/d2c/d38/d54/d90/fbd [0,4194304] 0 2024-04-06T14:17:21.556 INFO:tasks.workunit.client.1.smithi086.stdout:4/684: mkdir d0/d2c/d38/d54/d61/d7a/dc2/dd3 0 2024-04-06T14:17:21.556 INFO:tasks.workunit.client.1.smithi086.stdout:9/719: dread f2 [4194304,4194304] 0 2024-04-06T14:17:21.557 INFO:tasks.workunit.client.1.smithi086.stdout:7/788: creat d7/d10/d102/f110 x:0 0 0 2024-04-06T14:17:21.558 INFO:tasks.workunit.client.1.smithi086.stdout:1/850: rename db/cfb to db/d11/d1a/d79/da5/c10c 0 2024-04-06T14:17:21.561 INFO:tasks.workunit.client.1.smithi086.stdout:7/789: dread d7/d10/d11/d1c/d24/d2c/d69/f78 [0,4194304] 0 2024-04-06T14:17:21.562 INFO:tasks.workunit.client.1.smithi086.stdout:9/720: mknod d5/d13/d1a/d5f/d88/d14/d19/d1b/ce4 0 2024-04-06T14:17:21.562 INFO:tasks.workunit.client.1.smithi086.stdout:1/851: write db/d11/d1f/d5c/ddb/de9/f107 [775612,73785] 0 2024-04-06T14:17:21.562 INFO:tasks.workunit.client.1.smithi086.stdout:9/721: stat d5/d58 0 2024-04-06T14:17:21.566 INFO:tasks.workunit.client.1.smithi086.stdout:7/790: creat d7/d10/d102/d5c/ded/f111 x:0 0 0 2024-04-06T14:17:21.567 INFO:tasks.workunit.client.1.smithi086.stdout:4/685: dwrite d0/d15/d20/f44 [4194304,4194304] 0 2024-04-06T14:17:21.569 INFO:tasks.workunit.client.1.smithi086.stdout:7/791: fdatasync d7/d10/d11/d1c/f58 0 2024-04-06T14:17:21.569 INFO:tasks.workunit.client.1.smithi086.stdout:1/852: truncate db/d13/f74 3748131 0 2024-04-06T14:17:21.578 INFO:tasks.workunit.client.1.smithi086.stdout:7/792: rename d7/d10/d11/d1c/d24/dc8/dcd/fd8 to d7/d10/d11/d1c/f112 0 2024-04-06T14:17:21.581 INFO:tasks.workunit.client.1.smithi086.stdout:7/793: mknod d7/d10/d11/d21/c113 0 2024-04-06T14:17:21.582 INFO:tasks.workunit.client.1.smithi086.stdout:2/843: dread d0/df/d5e/d9d/fe0 [0,4194304] 0 2024-04-06T14:17:21.583 INFO:tasks.workunit.client.1.smithi086.stdout:9/722: truncate d5/d13/d1a/d8f/fb1 19534 0 2024-04-06T14:17:21.584 INFO:tasks.workunit.client.1.smithi086.stdout:2/844: readlink d0/d20/d44/dbb/dc4/de3/d76/d78/lec 0 2024-04-06T14:17:21.585 INFO:tasks.workunit.client.1.smithi086.stdout:9/723: readlink d5/d18/lad 0 2024-04-06T14:17:21.586 INFO:tasks.workunit.client.1.smithi086.stdout:9/724: chown d5/d13/d47/d55 1 1 2024-04-06T14:17:21.586 INFO:tasks.workunit.client.1.smithi086.stdout:4/686: dread d0/d2c/d38/d54/d83/f98 [0,4194304] 0 2024-04-06T14:17:21.587 INFO:tasks.workunit.client.1.smithi086.stdout:2/845: symlink d0/d20/d44/dbb/dc4/de3/d76/d78/l12d 0 2024-04-06T14:17:21.589 INFO:tasks.workunit.client.1.smithi086.stdout:4/687: unlink d0/d15/d20/d34/f35 0 2024-04-06T14:17:21.591 INFO:tasks.workunit.client.1.smithi086.stdout:2/846: creat d0/d20/d44/dee/f12e x:0 0 0 2024-04-06T14:17:21.592 INFO:tasks.workunit.client.1.smithi086.stdout:9/725: creat d5/d13/d1a/d5f/d88/fe5 x:0 0 0 2024-04-06T14:17:21.593 INFO:tasks.workunit.client.1.smithi086.stdout:4/688: creat d0/d2c/d38/d54/d61/fd4 x:0 0 0 2024-04-06T14:17:21.594 INFO:tasks.workunit.client.1.smithi086.stdout:1/853: dread db/d11/d1a/d79/da5/fe4 [0,4194304] 0 2024-04-06T14:17:21.595 INFO:tasks.workunit.client.1.smithi086.stdout:4/689: stat d0/d2c/d30/d24/d5d/d8a 0 2024-04-06T14:17:21.597 INFO:tasks.workunit.client.1.smithi086.stdout:9/726: symlink d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/le6 0 2024-04-06T14:17:21.598 INFO:tasks.workunit.client.1.smithi086.stdout:9/727: fdatasync d5/d13/d1a/fb6 0 2024-04-06T14:17:21.598 INFO:tasks.workunit.client.1.smithi086.stdout:1/854: mkdir db/d13/d55/dfa/d10d 0 2024-04-06T14:17:21.599 INFO:tasks.workunit.client.1.smithi086.stdout:4/690: mknod d0/cd5 0 2024-04-06T14:17:21.600 INFO:tasks.workunit.client.0.smithi032.stdout:5/953: sync 2024-04-06T14:17:21.601 INFO:tasks.workunit.client.1.smithi086.stdout:4/691: stat d0/d2c/d30/d24/da3/l58 0 2024-04-06T14:17:21.601 INFO:tasks.workunit.client.1.smithi086.stdout:4/692: chown d0/l18 30499600 1 2024-04-06T14:17:21.603 INFO:tasks.workunit.client.1.smithi086.stdout:4/693: write d0/d15/d20/f57 [14880,51620] 0 2024-04-06T14:17:21.606 INFO:tasks.workunit.client.1.smithi086.stdout:1/855: symlink db/d11/d24/l10e 0 2024-04-06T14:17:21.607 INFO:tasks.workunit.client.1.smithi086.stdout:1/856: chown db/d13/d55/fb1 61 1 2024-04-06T14:17:21.607 INFO:tasks.workunit.client.1.smithi086.stdout:1/857: write db/d11/d1a/d2a/d9a/d78/fcc [3385088,78187] 0 2024-04-06T14:17:21.608 INFO:tasks.workunit.client.1.smithi086.stdout:4/694: symlink d0/d15/d20/d74/ld6 0 2024-04-06T14:17:21.608 INFO:tasks.workunit.client.0.smithi032.stdout:5/954: dwrite d16/d58/f5f [0,4194304] 0 2024-04-06T14:17:21.608 INFO:tasks.workunit.client.1.smithi086.stdout:2/847: truncate d0/fd 3842178 0 2024-04-06T14:17:21.613 INFO:tasks.workunit.client.1.smithi086.stdout:1/858: unlink db/d11/d1a/d2a/d9a/ff5 0 2024-04-06T14:17:21.615 INFO:tasks.workunit.client.1.smithi086.stdout:4/695: unlink d0/d2c/d38/d54/d61/d7a/dc2/fce 0 2024-04-06T14:17:21.616 INFO:tasks.workunit.client.1.smithi086.stdout:9/728: getdents d5/d13/d1a/d5f/d88/d91 0 2024-04-06T14:17:21.621 INFO:tasks.workunit.client.1.smithi086.stdout:3/835: sync 2024-04-06T14:17:21.621 INFO:tasks.workunit.client.1.smithi086.stdout:8/860: sync 2024-04-06T14:17:21.623 INFO:tasks.workunit.client.1.smithi086.stdout:1/859: read db/d11/d1a/d2a/d9a/d68/d6e/d81/f99 [2015557,116223] 0 2024-04-06T14:17:21.624 INFO:tasks.workunit.client.1.smithi086.stdout:8/861: creat d0/db/f12d x:0 0 0 2024-04-06T14:17:21.625 INFO:tasks.workunit.client.0.smithi032.stdout:5/955: write d16/d18/d33/d35/d88/dbd/dc5/dec/f142 [567740,91005] 0 2024-04-06T14:17:21.625 INFO:tasks.workunit.client.1.smithi086.stdout:2/848: link d0/d20/d27/f43 d0/d20/d27/d2b/d35/f12f 0 2024-04-06T14:17:21.625 INFO:tasks.workunit.client.1.smithi086.stdout:8/862: readlink d0/d71/d16/d52/l59 0 2024-04-06T14:17:21.626 INFO:tasks.workunit.client.1.smithi086.stdout:3/836: fdatasync d5/d36/d56/dd3/ff5 0 2024-04-06T14:17:21.628 INFO:tasks.workunit.client.1.smithi086.stdout:1/860: symlink db/d11/d1a/d2a/d9a/d68/d6e/d81/l10f 0 2024-04-06T14:17:21.631 INFO:tasks.workunit.client.1.smithi086.stdout:3/837: symlink d5/dea/db4/da7/d89/l112 0 2024-04-06T14:17:21.632 INFO:tasks.workunit.client.1.smithi086.stdout:2/849: symlink d0/d20/d27/l130 0 2024-04-06T14:17:21.637 INFO:tasks.workunit.client.1.smithi086.stdout:3/838: write d5/d36/d5a/d6d/d72/f75 [4702904,55943] 0 2024-04-06T14:17:21.640 INFO:tasks.workunit.client.1.smithi086.stdout:9/729: dread d5/d13/d1a/d8f/fb1 [0,4194304] 0 2024-04-06T14:17:21.642 INFO:tasks.workunit.client.1.smithi086.stdout:8/863: dwrite d0/d71/d22/d43/daa/f113 [0,4194304] 0 2024-04-06T14:17:21.646 INFO:tasks.workunit.client.1.smithi086.stdout:8/864: readlink d0/d3b/d79/l11f 0 2024-04-06T14:17:21.650 INFO:tasks.workunit.client.1.smithi086.stdout:1/861: dwrite db/d11/d1a/d2a/d9a/d68/d6e/f8d [0,4194304] 0 2024-04-06T14:17:21.650 INFO:tasks.workunit.client.1.smithi086.stdout:4/696: dwrite d0/d15/d20/f27 [0,4194304] 0 2024-04-06T14:17:21.650 INFO:tasks.workunit.client.1.smithi086.stdout:2/850: rmdir d0/d20/d27/d2b/d35/d36/d4d/d59 39 2024-04-06T14:17:21.651 INFO:tasks.workunit.client.1.smithi086.stdout:9/730: mknod d5/d13/d47/d55/d87/ce7 0 2024-04-06T14:17:21.652 INFO:tasks.workunit.client.1.smithi086.stdout:4/697: fdatasync d0/d15/d2b/f82 0 2024-04-06T14:17:21.653 INFO:tasks.workunit.client.1.smithi086.stdout:2/851: chown d0/d20/d27/d2b/d35/d36/d4a/l99 3 1 2024-04-06T14:17:21.655 INFO:tasks.workunit.client.1.smithi086.stdout:8/865: mknod d0/d71/d16/d52/d5f/c12e 0 2024-04-06T14:17:21.656 INFO:tasks.workunit.client.1.smithi086.stdout:8/866: write d0/d3b/f3d [909692,36157] 0 2024-04-06T14:17:21.658 INFO:tasks.workunit.client.0.smithi032.stdout:5/956: dwrite d16/d18/d33/d45/fd6 [0,4194304] 0 2024-04-06T14:17:21.662 INFO:tasks.workunit.client.1.smithi086.stdout:1/862: unlink db/d11/d1a/f85 0 2024-04-06T14:17:21.663 INFO:tasks.workunit.client.1.smithi086.stdout:4/698: mkdir d0/d2c/d38/dd7 0 2024-04-06T14:17:21.665 INFO:tasks.workunit.client.1.smithi086.stdout:2/852: fsync d0/d20/d27/d2b/d35/d36/d4d/d59/f118 0 2024-04-06T14:17:21.665 INFO:tasks.workunit.client.0.smithi032.stdout:5/957: rmdir d16/d135 0 2024-04-06T14:17:21.666 INFO:tasks.workunit.client.1.smithi086.stdout:4/699: write d0/d2c/d38/d54/fa1 [550483,17570] 0 2024-04-06T14:17:21.666 INFO:tasks.workunit.client.1.smithi086.stdout:3/839: link d5/d36/d56/dd3/fe0 d5/d36/d5a/d6d/d72/d108/f113 0 2024-04-06T14:17:21.668 INFO:tasks.workunit.client.1.smithi086.stdout:8/867: rmdir d0/d3b/d79 39 2024-04-06T14:17:21.671 INFO:tasks.workunit.client.1.smithi086.stdout:1/863: write db/d13/d55/f59 [3443624,2996] 0 2024-04-06T14:17:21.672 INFO:tasks.workunit.client.1.smithi086.stdout:4/700: write d0/de/fab [1003955,85014] 0 2024-04-06T14:17:21.674 INFO:tasks.workunit.client.0.smithi032.stdout:3/999: sync 2024-04-06T14:17:21.675 INFO:tasks.workunit.client.1.smithi086.stdout:8/868: dread d0/d71/d22/d43/d7f/f88 [0,4194304] 0 2024-04-06T14:17:21.675 INFO:tasks.workunit.client.1.smithi086.stdout:1/864: write db/d11/d42/fbf [973202,116864] 0 2024-04-06T14:17:21.676 INFO:tasks.workunit.client.1.smithi086.stdout:9/731: rename d5/d13/d1a/d5f/d88/d91/db9/fd9 to d5/d13/d47/fe8 0 2024-04-06T14:17:21.677 INFO:tasks.workunit.client.0.smithi032.stdout:5/958: link d16/d18/d33/d45/d52/f131 d16/d18/f148 0 2024-04-06T14:17:21.681 INFO:tasks.workunit.client.1.smithi086.stdout:2/853: rename d0/d20/d44/d120/c129 to d0/d20/d82/d121/c131 0 2024-04-06T14:17:21.681 INFO:tasks.workunit.client.0.smithi032.stdout:5/959: symlink d16/d18/d33/d45/l149 0 2024-04-06T14:17:21.683 INFO:tasks.workunit.client.1.smithi086.stdout:3/840: dwrite d5/dea/f88 [0,4194304] 0 2024-04-06T14:17:21.685 INFO:tasks.workunit.client.1.smithi086.stdout:2/854: chown d0/d20/d80/l85 2 1 2024-04-06T14:17:21.685 INFO:tasks.workunit.client.1.smithi086.stdout:9/732: creat d5/d13/d1a/d5f/d88/d14/d19/d1b/fe9 x:0 0 0 2024-04-06T14:17:21.686 INFO:tasks.workunit.client.0.smithi032.stdout:5/960: creat d16/d18/d33/df7/f14a x:0 0 0 2024-04-06T14:17:21.687 INFO:tasks.workunit.client.1.smithi086.stdout:1/865: dread db/d11/fe0 [0,4194304] 0 2024-04-06T14:17:21.688 INFO:tasks.workunit.client.0.smithi032.stdout:5/961: creat d16/d18/d33/d35/f14b x:0 0 0 2024-04-06T14:17:21.688 INFO:tasks.workunit.client.1.smithi086.stdout:4/701: dwrite d0/d15/d20/f57 [0,4194304] 0 2024-04-06T14:17:21.689 INFO:tasks.workunit.client.1.smithi086.stdout:3/841: unlink d5/c1f 0 2024-04-06T14:17:21.691 INFO:tasks.workunit.client.1.smithi086.stdout:3/842: truncate d5/dea/dd2/ddb/de6/ff0 209618 0 2024-04-06T14:17:21.692 INFO:tasks.workunit.client.1.smithi086.stdout:2/855: creat d0/d20/d27/d2b/d35/d36/dbe/f132 x:0 0 0 2024-04-06T14:17:21.693 INFO:tasks.workunit.client.1.smithi086.stdout:3/843: dread - d5/dea/dd2/ddb/de6/d9e/da5/fad zero size 2024-04-06T14:17:21.694 INFO:tasks.workunit.client.0.smithi032.stdout:5/962: symlink d16/d18/d33/l14c 0 2024-04-06T14:17:21.694 INFO:tasks.workunit.client.1.smithi086.stdout:1/866: mkdir db/d13/d55/df1/d110 0 2024-04-06T14:17:21.695 INFO:tasks.workunit.client.1.smithi086.stdout:4/702: rmdir d0/d15/d20/d74 39 2024-04-06T14:17:21.695 INFO:tasks.workunit.client.1.smithi086.stdout:8/869: link d0/d71/d93/ded/l11a d0/d71/d16/d39/d90/daf/df0/l12f 0 2024-04-06T14:17:21.698 INFO:tasks.workunit.client.1.smithi086.stdout:3/844: unlink d5/d36/d5c/fd6 0 2024-04-06T14:17:21.700 INFO:tasks.workunit.client.1.smithi086.stdout:4/703: mknod d0/d2c/d30/d24/da3/d88/cd8 0 2024-04-06T14:17:21.701 INFO:tasks.workunit.client.1.smithi086.stdout:4/704: chown d0/d2c/d30/d24/d93/fa4 28 1 2024-04-06T14:17:21.702 INFO:tasks.workunit.client.1.smithi086.stdout:2/856: mknod d0/d20/d27/d2b/c133 0 2024-04-06T14:17:21.703 INFO:tasks.workunit.client.1.smithi086.stdout:8/870: symlink d0/d71/d22/d43/d6d/d75/dee/l130 0 2024-04-06T14:17:21.703 INFO:tasks.workunit.client.1.smithi086.stdout:2/857: dread - d0/d20/d27/f106 zero size 2024-04-06T14:17:21.704 INFO:tasks.workunit.client.1.smithi086.stdout:9/733: rename d5/d13/l9e to d5/d13/d1a/d5f/d88/d14/d19/d94/lea 0 2024-04-06T14:17:21.705 INFO:tasks.workunit.client.1.smithi086.stdout:1/867: symlink db/d11/d1a/l111 0 2024-04-06T14:17:21.705 INFO:tasks.workunit.client.1.smithi086.stdout:8/871: readlink d0/d71/d22/d65/d6e/d98/ddb/l12c 0 2024-04-06T14:17:21.709 INFO:tasks.workunit.client.1.smithi086.stdout:2/858: read d0/d20/d44/dbb/dc4/de3/d76/f91 [4051216,6892] 0 2024-04-06T14:17:21.710 INFO:tasks.workunit.client.1.smithi086.stdout:1/868: truncate db/f7b 2429362 0 2024-04-06T14:17:21.712 INFO:tasks.workunit.client.1.smithi086.stdout:4/705: dwrite d0/d15/d20/f27 [4194304,4194304] 0 2024-04-06T14:17:21.713 INFO:tasks.workunit.client.1.smithi086.stdout:8/872: symlink d0/d3b/d79/dc6/d123/l131 0 2024-04-06T14:17:21.714 INFO:tasks.workunit.client.1.smithi086.stdout:1/869: creat db/d11/d1f/d5c/f112 x:0 0 0 2024-04-06T14:17:21.715 INFO:tasks.workunit.client.0.smithi032.stdout:5/963: dwrite d16/d58/f6d [0,4194304] 0 2024-04-06T14:17:21.715 INFO:tasks.workunit.client.1.smithi086.stdout:4/706: chown d0/d2c/d30/d24/d5d/f8f 15 1 2024-04-06T14:17:21.716 INFO:tasks.workunit.client.1.smithi086.stdout:2/859: rename d0/df/ffa to d0/d20/d44/d120/f134 0 2024-04-06T14:17:21.716 INFO:tasks.workunit.client.1.smithi086.stdout:4/707: write d0/d2c/d30/d24/f31 [3231525,27394] 0 2024-04-06T14:17:21.717 INFO:tasks.workunit.client.1.smithi086.stdout:2/860: stat d0/d20/d44/f46 0 2024-04-06T14:17:21.717 INFO:tasks.workunit.client.1.smithi086.stdout:1/870: unlink db/d11/d42/c80 0 2024-04-06T14:17:21.718 INFO:tasks.workunit.client.0.smithi032.stdout:0/817: sync 2024-04-06T14:17:21.718 INFO:tasks.workunit.client.1.smithi086.stdout:2/861: chown d0/d20 0 1 2024-04-06T14:17:21.720 INFO:tasks.workunit.client.1.smithi086.stdout:8/873: rename d0/d71/d16/d52/dd0/fda to d0/d71/d16/d39/f132 0 2024-04-06T14:17:21.722 INFO:tasks.workunit.client.1.smithi086.stdout:4/708: symlink d0/d2c/d30/d24/d93/ld9 0 2024-04-06T14:17:21.724 INFO:tasks.workunit.client.1.smithi086.stdout:4/709: chown d0/d2c/d30/d24/d50/f72 1471193 1 2024-04-06T14:17:21.724 INFO:tasks.workunit.client.1.smithi086.stdout:1/871: dread db/d11/d1a/d79/da5/fe4 [0,4194304] 0 2024-04-06T14:17:21.725 INFO:tasks.workunit.client.1.smithi086.stdout:4/710: write d0/d2c/d38/d54/fa1 [556509,50295] 0 2024-04-06T14:17:21.726 INFO:tasks.workunit.client.0.smithi032.stdout:5/964: dread d16/d18/d33/d45/d52/f43 [0,4194304] 0 2024-04-06T14:17:21.726 INFO:tasks.workunit.client.1.smithi086.stdout:2/862: mkdir d0/d20/d27/d2b/d35/d36/d4d/d59/d135 0 2024-04-06T14:17:21.727 INFO:tasks.workunit.client.1.smithi086.stdout:8/874: mknod d0/d71/d16/d52/c133 0 2024-04-06T14:17:21.729 INFO:tasks.workunit.client.0.smithi032.stdout:5/965: dread - d16/d18/d33/d45/d52/d39/d86/f98 zero size 2024-04-06T14:17:21.731 INFO:tasks.workunit.client.1.smithi086.stdout:1/872: mknod db/d11/d1a/d2a/d9a/d68/d6e/c113 0 2024-04-06T14:17:21.731 INFO:tasks.workunit.client.0.smithi032.stdout:0/818: dwrite d3/d83/ff6 [0,4194304] 0 2024-04-06T14:17:21.734 INFO:tasks.workunit.client.0.smithi032.stdout:0/819: mknod d3/dc2/dd8/c115 0 2024-04-06T14:17:21.735 INFO:tasks.workunit.client.1.smithi086.stdout:9/734: readlink d5/d13/d1a/d5f/d88/d14/d19/d94/lea 0 2024-04-06T14:17:21.737 INFO:tasks.workunit.client.0.smithi032.stdout:0/820: creat d3/da/d25/d2b/d4a/d79/d97/f116 x:0 0 0 2024-04-06T14:17:21.741 INFO:tasks.workunit.client.1.smithi086.stdout:0/858: sync 2024-04-06T14:17:21.741 INFO:tasks.workunit.client.0.smithi032.stdout:0/821: readlink d3/da/d25/l48 0 2024-04-06T14:17:21.742 INFO:tasks.workunit.client.1.smithi086.stdout:2/863: creat d0/f136 x:0 0 0 2024-04-06T14:17:21.743 INFO:tasks.workunit.client.1.smithi086.stdout:1/873: creat db/d11/d1a/d2a/d9a/f114 x:0 0 0 2024-04-06T14:17:21.744 INFO:tasks.workunit.client.0.smithi032.stdout:5/966: dwrite d16/d18/dae/dbb/f10a [0,4194304] 0 2024-04-06T14:17:21.745 INFO:tasks.workunit.client.1.smithi086.stdout:1/874: truncate db/d11/d87/fec 319940 0 2024-04-06T14:17:21.746 INFO:tasks.workunit.client.1.smithi086.stdout:9/735: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/deb 0 2024-04-06T14:17:21.746 INFO:tasks.workunit.client.1.smithi086.stdout:9/736: readlink d5/d18/l52 0 2024-04-06T14:17:21.747 INFO:tasks.workunit.client.0.smithi032.stdout:5/967: dread - d16/d58/d54/d74/d8f/fed zero size 2024-04-06T14:17:21.748 INFO:tasks.workunit.client.1.smithi086.stdout:0/859: rmdir d0/d15/d4e/da3 39 2024-04-06T14:17:21.749 INFO:tasks.workunit.client.0.smithi032.stdout:0/822: creat d3/da/d25/d2b/d4a/d79/d97/de0/df5/f117 x:0 0 0 2024-04-06T14:17:21.750 INFO:tasks.workunit.client.0.smithi032.stdout:0/823: chown d3/fd 6 1 2024-04-06T14:17:21.750 INFO:tasks.workunit.client.1.smithi086.stdout:2/864: stat d0/df/d48/c71 0 2024-04-06T14:17:21.753 INFO:tasks.workunit.client.1.smithi086.stdout:9/737: symlink d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/lec 0 2024-04-06T14:17:21.753 INFO:tasks.workunit.client.0.smithi032.stdout:5/968: rename d16/d18/d33/d45/d52/d39/d86/cb9 to d16/d18/d33/d35/d4d/c14d 0 2024-04-06T14:17:21.754 INFO:tasks.workunit.client.0.smithi032.stdout:0/824: stat d3/da/d25/d2b/d80/db3/cff 0 2024-04-06T14:17:21.755 INFO:tasks.workunit.client.1.smithi086.stdout:2/865: creat d0/d20/d27/d2b/d35/d36/d4a/f137 x:0 0 0 2024-04-06T14:17:21.757 INFO:tasks.workunit.client.0.smithi032.stdout:0/825: chown d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/d9c/dcc 218987 1 2024-04-06T14:17:21.759 INFO:tasks.workunit.client.1.smithi086.stdout:9/738: dread d5/d13/d1a/d8f/f9a [0,4194304] 0 2024-04-06T14:17:21.759 INFO:tasks.workunit.client.0.smithi032.stdout:5/969: link d16/d18/d33/d45/l149 d16/d18/d126/l14e 0 2024-04-06T14:17:21.761 INFO:tasks.workunit.client.0.smithi032.stdout:5/970: write d16/d18/d33/d45/d52/f134 [117832,91286] 0 2024-04-06T14:17:21.761 INFO:tasks.workunit.client.1.smithi086.stdout:0/860: rename d0/d15/d4e/c32 to d0/d15/d4e/da3/d44/c11a 0 2024-04-06T14:17:21.764 INFO:tasks.workunit.client.0.smithi032.stdout:5/971: getdents d16/d127 0 2024-04-06T14:17:21.764 INFO:tasks.workunit.client.1.smithi086.stdout:2/866: dwrite d0/d20/d44/f46 [0,4194304] 0 2024-04-06T14:17:21.766 INFO:tasks.workunit.client.1.smithi086.stdout:8/875: dwrite d0/d71/d22/d43/d6d/d75/db2/fca [0,4194304] 0 2024-04-06T14:17:21.768 INFO:tasks.workunit.client.1.smithi086.stdout:9/739: creat d5/d13/d1a/d5f/d88/d91/fed x:0 0 0 2024-04-06T14:17:21.771 INFO:tasks.workunit.client.0.smithi032.stdout:0/826: dread d3/da/d25/d101/fb2 [0,4194304] 0 2024-04-06T14:17:21.773 INFO:tasks.workunit.client.1.smithi086.stdout:8/876: fsync d0/d71/d16/d52/dd0/ddc/f102 0 2024-04-06T14:17:21.775 INFO:tasks.workunit.client.1.smithi086.stdout:1/875: dwrite db/d13/f1e [0,4194304] 0 2024-04-06T14:17:21.775 INFO:tasks.workunit.client.1.smithi086.stdout:2/867: chown d0/d20/d44/dbb/dc4/de3/d76/d78/d125 508443308 1 2024-04-06T14:17:21.776 INFO:tasks.workunit.client.0.smithi032.stdout:0/827: symlink d3/da/d25/d2b/d80/l118 0 2024-04-06T14:17:21.776 INFO:tasks.workunit.client.1.smithi086.stdout:9/740: symlink d5/d13/d47/d55/d87/lee 0 2024-04-06T14:17:21.778 INFO:tasks.workunit.client.1.smithi086.stdout:2/868: readlink d0/d20/d27/d2b/d35/d36/d4d/lca 0 2024-04-06T14:17:21.778 INFO:tasks.workunit.client.1.smithi086.stdout:9/741: write d5/d13/f7c [724883,15363] 0 2024-04-06T14:17:21.778 INFO:tasks.workunit.client.1.smithi086.stdout:9/742: chown d5/d13 39584 1 2024-04-06T14:17:21.779 INFO:tasks.workunit.client.1.smithi086.stdout:8/877: mkdir d0/d71/d16/d52/dd0/d134 0 2024-04-06T14:17:21.780 INFO:tasks.workunit.client.1.smithi086.stdout:2/869: dread - d0/d20/d27/d2b/d35/d36/f105 zero size 2024-04-06T14:17:21.781 INFO:tasks.workunit.client.0.smithi032.stdout:0/828: symlink d3/da/d41/l119 0 2024-04-06T14:17:21.782 INFO:tasks.workunit.client.1.smithi086.stdout:2/870: chown d0/d20/d44/dbb/cef 345181235 1 2024-04-06T14:17:21.782 INFO:tasks.workunit.client.1.smithi086.stdout:8/878: dread - d0/d5a/fcb zero size 2024-04-06T14:17:21.782 INFO:tasks.workunit.client.0.smithi032.stdout:5/972: dwrite d16/d18/d33/d45/d52/f43 [0,4194304] 0 2024-04-06T14:17:21.783 INFO:tasks.workunit.client.0.smithi032.stdout:0/829: symlink d3/da/d41/d107/l11a 0 2024-04-06T14:17:21.784 INFO:tasks.workunit.client.1.smithi086.stdout:9/743: mknod d5/d13/cef 0 2024-04-06T14:17:21.785 INFO:tasks.workunit.client.1.smithi086.stdout:9/744: stat d5/d13/d1a/d5f/d88/db3/fe3 0 2024-04-06T14:17:21.787 INFO:tasks.workunit.client.1.smithi086.stdout:1/876: link db/d11/d87/fdc db/d11/d1a/d79/da5/f115 0 2024-04-06T14:17:21.790 INFO:tasks.workunit.client.0.smithi032.stdout:0/830: rename d3/da/d25/c58 to d3/c11b 0 2024-04-06T14:17:21.790 INFO:tasks.workunit.client.1.smithi086.stdout:8/879: rmdir d0/d71/d16/d39/d90 39 2024-04-06T14:17:21.791 INFO:tasks.workunit.client.1.smithi086.stdout:9/745: symlink d5/d13/d47/d5e/lf0 0 2024-04-06T14:17:21.792 INFO:tasks.workunit.client.0.smithi032.stdout:5/973: dwrite d16/d18/d33/d35/d88/dbd/dc5/dec/f142 [0,4194304] 0 2024-04-06T14:17:21.793 INFO:tasks.workunit.client.1.smithi086.stdout:1/877: mkdir db/d11/d1a/d2a/d9a/d68/daf/dc2/d116 0 2024-04-06T14:17:21.793 INFO:tasks.workunit.client.0.smithi032.stdout:5/974: chown d16/d18/d33/d35/d4d/c5c 47919759 1 2024-04-06T14:17:21.794 INFO:tasks.workunit.client.1.smithi086.stdout:2/871: dwrite d0/d20/d27/dd1/f10d [0,4194304] 0 2024-04-06T14:17:21.794 INFO:tasks.workunit.client.0.smithi032.stdout:5/975: chown d16/d18/c61 280 1 2024-04-06T14:17:21.795 INFO:tasks.workunit.client.0.smithi032.stdout:5/976: readlink d16/d58/d54/d74/d8f/d95/l10b 0 2024-04-06T14:17:21.796 INFO:tasks.workunit.client.1.smithi086.stdout:8/880: truncate d0/d3b/f51 2712605 0 2024-04-06T14:17:21.797 INFO:tasks.workunit.client.1.smithi086.stdout:9/746: fdatasync d5/d13/d1a/d5f/d88/d14/f51 0 2024-04-06T14:17:21.798 INFO:tasks.workunit.client.1.smithi086.stdout:9/747: chown d5/d13/d1a/d8f/fb8 2694892 1 2024-04-06T14:17:21.800 INFO:tasks.workunit.client.1.smithi086.stdout:2/872: symlink d0/d20/d27/d2b/d35/d36/d4a/d11d/l138 0 2024-04-06T14:17:21.804 INFO:tasks.workunit.client.1.smithi086.stdout:9/748: creat d5/d13/d1a/d8f/ff1 x:0 0 0 2024-04-06T14:17:21.805 INFO:tasks.workunit.client.1.smithi086.stdout:2/873: mknod d0/d20/d27/dae/c139 0 2024-04-06T14:17:21.805 INFO:tasks.workunit.client.0.smithi032.stdout:5/977: dwrite d16/d18/d33/d35/d88/deb/f114 [0,4194304] 0 2024-04-06T14:17:21.806 INFO:tasks.workunit.client.0.smithi032.stdout:0/831: dread d3/f14 [0,4194304] 0 2024-04-06T14:17:21.809 INFO:tasks.workunit.client.0.smithi032.stdout:5/978: creat d16/d18/d33/d45/d87/f14f x:0 0 0 2024-04-06T14:17:21.813 INFO:tasks.workunit.client.0.smithi032.stdout:5/979: creat d16/d18/d33/d35/d88/f150 x:0 0 0 2024-04-06T14:17:21.815 INFO:tasks.workunit.client.1.smithi086.stdout:9/749: rename d5/d13/l16 to d5/d13/d1a/d5f/d88/lf2 0 2024-04-06T14:17:21.816 INFO:tasks.workunit.client.0.smithi032.stdout:5/980: write d16/d18/d33/d35/d88/f11e [333331,112523] 0 2024-04-06T14:17:21.816 INFO:tasks.workunit.client.0.smithi032.stdout:0/832: dwrite d3/dc2/dd8/ff2 [0,4194304] 0 2024-04-06T14:17:21.817 INFO:tasks.workunit.client.1.smithi086.stdout:2/874: dwrite d0/df/d5e/d9d/fff [0,4194304] 0 2024-04-06T14:17:21.820 INFO:tasks.workunit.client.0.smithi032.stdout:5/981: chown d16/d18/d33/d45/d52/d39/fb7 7977759 1 2024-04-06T14:17:21.822 INFO:tasks.workunit.client.0.smithi032.stdout:0/833: mkdir d3/da/d25/d2b/d4a/d86/d10a/dfe/d11c 0 2024-04-06T14:17:21.823 INFO:tasks.workunit.client.0.smithi032.stdout:5/982: mknod d16/d18/dae/dbb/ddc/c151 0 2024-04-06T14:17:21.824 INFO:tasks.workunit.client.1.smithi086.stdout:8/881: dwrite d0/d71/d16/d39/d90/fa0 [0,4194304] 0 2024-04-06T14:17:21.825 INFO:tasks.workunit.client.0.smithi032.stdout:5/983: stat d16/d18/d33/d45/d52/d39/f3e 0 2024-04-06T14:17:21.825 INFO:tasks.workunit.client.1.smithi086.stdout:9/750: mknod d5/d13/d1a/d5f/d88/d14/d19/d25/d59/cf3 0 2024-04-06T14:17:21.827 INFO:tasks.workunit.client.0.smithi032.stdout:0/834: creat d3/da/d25/d2b/d4a/d86/dd6/f11d x:0 0 0 2024-04-06T14:17:21.828 INFO:tasks.workunit.client.1.smithi086.stdout:2/875: getdents d0/d20/d80/dbf 0 2024-04-06T14:17:21.829 INFO:tasks.workunit.client.1.smithi086.stdout:9/751: readlink d5/d13/d1a/d5f/d88/d14/d19/d25/d59/l85 0 2024-04-06T14:17:21.831 INFO:tasks.workunit.client.1.smithi086.stdout:2/876: write d0/d20/d27/d2b/d35/d36/fa2 [529108,40276] 0 2024-04-06T14:17:21.834 INFO:tasks.workunit.client.1.smithi086.stdout:8/882: rename d0/d71/d16/d52/dd0/ddc/de7/dab to d0/d71/d16/d135 0 2024-04-06T14:17:21.835 INFO:tasks.workunit.client.1.smithi086.stdout:1/878: dread db/d11/d87/fb6 [0,4194304] 0 2024-04-06T14:17:21.838 INFO:tasks.workunit.client.1.smithi086.stdout:2/877: chown d0/la 6252 1 2024-04-06T14:17:21.838 INFO:tasks.workunit.client.1.smithi086.stdout:8/883: readlink d0/d71/d16/l50 0 2024-04-06T14:17:21.838 INFO:tasks.workunit.client.1.smithi086.stdout:2/878: fdatasync d0/fe 0 2024-04-06T14:17:21.839 INFO:tasks.workunit.client.1.smithi086.stdout:9/752: dwrite d5/d18/f3f [0,4194304] 0 2024-04-06T14:17:21.840 INFO:tasks.workunit.client.1.smithi086.stdout:8/884: chown d0/d71/d22/d43/d6d/d75/dee/f115 1 1 2024-04-06T14:17:21.842 INFO:tasks.workunit.client.1.smithi086.stdout:8/885: chown d0/d71/d22/d43/l48 14071 1 2024-04-06T14:17:21.843 INFO:tasks.workunit.client.0.smithi032.stdout:0/835: dwrite d3/da/d25/d36/f39 [0,4194304] 0 2024-04-06T14:17:21.844 INFO:tasks.workunit.client.0.smithi032.stdout:0/836: chown d3/da/c50 130020777 1 2024-04-06T14:17:21.847 INFO:tasks.workunit.client.0.smithi032.stdout:0/837: write d3/da/f53 [1410665,40923] 0 2024-04-06T14:17:21.848 INFO:tasks.workunit.client.1.smithi086.stdout:8/886: mkdir d0/d71/d16/d52/dd0/ddc/de7/d136 0 2024-04-06T14:17:21.848 INFO:tasks.workunit.client.1.smithi086.stdout:9/753: rmdir d5/d13/d1a 39 2024-04-06T14:17:21.854 INFO:tasks.workunit.client.1.smithi086.stdout:8/887: fsync d0/d13/f40 0 2024-04-06T14:17:21.855 INFO:tasks.workunit.client.1.smithi086.stdout:8/888: chown d0/d3b/l77 37537 1 2024-04-06T14:17:21.855 INFO:tasks.workunit.client.1.smithi086.stdout:9/754: unlink d5/d13/d1a/d5f/d88/d14/d19/d1b/fe9 0 2024-04-06T14:17:21.857 INFO:tasks.workunit.client.1.smithi086.stdout:2/879: rename d0/f6 to d0/d20/d27/d2b/d35/d36/f13a 0 2024-04-06T14:17:21.859 INFO:tasks.workunit.client.1.smithi086.stdout:2/880: chown d0/df/d48/c6c 32 1 2024-04-06T14:17:21.860 INFO:tasks.workunit.client.1.smithi086.stdout:2/881: fdatasync d0/f8e 0 2024-04-06T14:17:21.861 INFO:tasks.workunit.client.0.smithi032.stdout:0/838: write d3/da/d25/f2a [2323144,19996] 0 2024-04-06T14:17:21.865 INFO:tasks.workunit.client.0.smithi032.stdout:0/839: mkdir d3/da/d25/d2b/d4a/d86/d11e 0 2024-04-06T14:17:21.872 INFO:tasks.workunit.client.1.smithi086.stdout:9/755: dwrite d5/d13/f7d [0,4194304] 0 2024-04-06T14:17:21.874 INFO:tasks.workunit.client.0.smithi032.stdout:0/840: dread d3/da/f53 [0,4194304] 0 2024-04-06T14:17:21.877 INFO:tasks.workunit.client.1.smithi086.stdout:8/889: dwrite d0/d71/d16/d52/dd0/ddc/de7/dd6/fd9 [0,4194304] 0 2024-04-06T14:17:21.879 INFO:tasks.workunit.client.1.smithi086.stdout:2/882: dwrite d0/f136 [0,4194304] 0 2024-04-06T14:17:21.879 INFO:tasks.workunit.client.0.smithi032.stdout:0/841: rename d3/da/f6f to d3/da/d25/d57/db5/dc9/f11f 0 2024-04-06T14:17:21.880 INFO:tasks.workunit.client.1.smithi086.stdout:8/890: truncate d0/d3b/f46 4919153 0 2024-04-06T14:17:21.897 INFO:tasks.workunit.client.1.smithi086.stdout:9/756: rename d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/le6 to d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/lf4 0 2024-04-06T14:17:21.897 INFO:tasks.workunit.client.1.smithi086.stdout:9/757: chown d5/d13/d1a 45184162 1 2024-04-06T14:17:21.900 INFO:tasks.workunit.client.1.smithi086.stdout:5/719: sync 2024-04-06T14:17:21.900 INFO:tasks.workunit.client.1.smithi086.stdout:6/961: sync 2024-04-06T14:17:21.901 INFO:tasks.workunit.client.1.smithi086.stdout:5/720: dread - d0/d43/da6/fc3 zero size 2024-04-06T14:17:21.902 INFO:tasks.workunit.client.1.smithi086.stdout:5/721: write d0/d16/f95 [646054,100749] 0 2024-04-06T14:17:21.916 INFO:tasks.workunit.client.1.smithi086.stdout:6/962: dread d6/f7 [4194304,4194304] 0 2024-04-06T14:17:21.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:21 smithi032 ceph-mon[39730]: pgmap v28: 65 pgs: 65 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 79 MiB/s rd, 177 MiB/s wr, 417 op/s 2024-04-06T14:17:21.994 INFO:tasks.workunit.client.1.smithi086.stdout:5/722: creat d0/d16/d18/d21/d23/d28/d94/ff0 x:0 0 0 2024-04-06T14:17:21.996 INFO:tasks.workunit.client.1.smithi086.stdout:6/963: creat d6/d13/d1a/da0/d114/db2/f14f x:0 0 0 2024-04-06T14:17:21.996 INFO:tasks.workunit.client.1.smithi086.stdout:5/723: write d0/d16/d18/d2a/d50/d66/fcd [757146,43245] 0 2024-04-06T14:17:21.997 INFO:tasks.workunit.client.1.smithi086.stdout:6/964: readlink d6/d13/d1a/da0/d11b/d45/l110 0 2024-04-06T14:17:21.999 INFO:tasks.workunit.client.1.smithi086.stdout:9/758: creat d5/d13/d1a/d5f/d88/d14/ff5 x:0 0 0 2024-04-06T14:17:22.001 INFO:tasks.workunit.client.1.smithi086.stdout:6/965: unlink d6/d13/d1a/da0/d11b/f6d 0 2024-04-06T14:17:22.002 INFO:tasks.workunit.client.1.smithi086.stdout:6/966: dread - d6/d13/d1a/da0/d114/db2/f130 zero size 2024-04-06T14:17:22.003 INFO:tasks.workunit.client.1.smithi086.stdout:9/759: symlink d5/d58/lf6 0 2024-04-06T14:17:22.003 INFO:tasks.workunit.client.1.smithi086.stdout:9/760: chown d5/d13/d47/d55/ld6 5660200 1 2024-04-06T14:17:22.005 INFO:tasks.workunit.client.1.smithi086.stdout:5/724: rename d0/d43/de5/la3 to d0/d16/d18/d2a/lf1 0 2024-04-06T14:17:22.011 INFO:tasks.workunit.client.1.smithi086.stdout:5/725: dread d0/d16/d18/d35/f3a [0,4194304] 0 2024-04-06T14:17:22.013 INFO:tasks.workunit.client.1.smithi086.stdout:8/891: write d0/d3b/f4c [615388,66820] 0 2024-04-06T14:17:22.016 INFO:tasks.workunit.client.1.smithi086.stdout:9/761: dread d5/d13/d1a/d5f/d88/d14/f86 [0,4194304] 0 2024-04-06T14:17:22.016 INFO:tasks.workunit.client.1.smithi086.stdout:5/726: creat d0/d16/d71/ff2 x:0 0 0 2024-04-06T14:17:22.016 INFO:tasks.workunit.client.1.smithi086.stdout:2/883: write d0/d20/d27/d2b/d35/d36/f13a [4348910,8852] 0 2024-04-06T14:17:22.018 INFO:tasks.workunit.client.1.smithi086.stdout:5/727: chown d0/d16/d18/d21/d23/d28/d94/fae 0 1 2024-04-06T14:17:22.020 INFO:tasks.workunit.client.1.smithi086.stdout:2/884: mkdir d0/d20/d27/d2b/d35/d36/d4a/d11d/d13b 0 2024-04-06T14:17:22.022 INFO:tasks.workunit.client.1.smithi086.stdout:9/762: rename d5/d13/fa9 to d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/ff7 0 2024-04-06T14:17:22.022 INFO:tasks.workunit.client.1.smithi086.stdout:9/763: dread - d5/d13/d47/d5e/f93 zero size 2024-04-06T14:17:22.023 INFO:tasks.workunit.client.1.smithi086.stdout:9/764: readlink d5/d18/l49 0 2024-04-06T14:17:22.026 INFO:tasks.workunit.client.1.smithi086.stdout:5/728: mknod d0/d16/d18/d2a/cf3 0 2024-04-06T14:17:22.027 INFO:tasks.workunit.client.1.smithi086.stdout:7/794: sync 2024-04-06T14:17:22.030 INFO:tasks.workunit.client.1.smithi086.stdout:7/795: read d7/d10/d11/d80/f35 [3991716,54135] 0 2024-04-06T14:17:22.032 INFO:tasks.workunit.client.1.smithi086.stdout:9/765: rmdir d5/d13/d1a/d5f/d88/d14/dda 0 2024-04-06T14:17:22.034 INFO:tasks.workunit.client.1.smithi086.stdout:6/967: dwrite d6/d13/d1a/f116 [0,4194304] 0 2024-04-06T14:17:22.034 INFO:tasks.workunit.client.1.smithi086.stdout:8/892: dread d0/d13/f68 [0,4194304] 0 2024-04-06T14:17:22.035 INFO:tasks.workunit.client.1.smithi086.stdout:5/729: fdatasync d0/d16/d18/d21/d23/d28/d94/d80/d85/f9f 0 2024-04-06T14:17:22.039 INFO:tasks.workunit.client.1.smithi086.stdout:7/796: mknod d7/d10/d102/d5c/c114 0 2024-04-06T14:17:22.039 INFO:tasks.workunit.client.1.smithi086.stdout:9/766: chown d5/d13/d47/d55/d87/lc7 893621 1 2024-04-06T14:17:22.041 INFO:tasks.workunit.client.1.smithi086.stdout:2/885: dwrite d0/d20/d44/d120/f134 [0,4194304] 0 2024-04-06T14:17:22.042 INFO:tasks.workunit.client.1.smithi086.stdout:8/893: symlink d0/d71/d16/d52/dd0/l137 0 2024-04-06T14:17:22.043 INFO:tasks.workunit.client.1.smithi086.stdout:6/968: write d6/d13/d1a/da0/d11b/d45/d4a/f115 [1553056,59623] 0 2024-04-06T14:17:22.046 INFO:tasks.workunit.client.1.smithi086.stdout:2/886: symlink d0/df/d5e/d9d/l13c 0 2024-04-06T14:17:22.049 INFO:tasks.workunit.client.1.smithi086.stdout:9/767: creat d5/d13/d1a/d5f/d88/d14/ff8 x:0 0 0 2024-04-06T14:17:22.051 INFO:tasks.workunit.client.1.smithi086.stdout:6/969: chown d6/d13/d1a/da0/d11b/d45/da6/lc9 95 1 2024-04-06T14:17:22.052 INFO:tasks.workunit.client.1.smithi086.stdout:2/887: creat d0/d20/d27/d37/d52/f13d x:0 0 0 2024-04-06T14:17:22.053 INFO:tasks.workunit.client.1.smithi086.stdout:8/894: truncate d0/d71/d22/d43/d7f/f88 930885 0 2024-04-06T14:17:22.053 INFO:tasks.workunit.client.1.smithi086.stdout:7/797: dwrite d7/d10/d11/d1c/d24/d2c/f70 [0,4194304] 0 2024-04-06T14:17:22.059 INFO:tasks.workunit.client.1.smithi086.stdout:9/768: chown d5/d13/d1a/d5f/d88/d14/d19/d25/d59/l8e 8 1 2024-04-06T14:17:22.060 INFO:tasks.workunit.client.1.smithi086.stdout:6/970: read d6/d9/f10 [980846,123688] 0 2024-04-06T14:17:22.063 INFO:tasks.workunit.client.1.smithi086.stdout:9/769: creat d5/d13/d47/ff9 x:0 0 0 2024-04-06T14:17:22.064 INFO:tasks.workunit.client.1.smithi086.stdout:7/798: dwrite d7/d10/d102/daf/db5/f8a [0,4194304] 0 2024-04-06T14:17:22.066 INFO:tasks.workunit.client.1.smithi086.stdout:9/770: chown d5/d13/d1a/d5f/d88/fe5 947969 1 2024-04-06T14:17:22.067 INFO:tasks.workunit.client.1.smithi086.stdout:6/971: link d6/d13/d42/f72 d6/d9/da9/f150 0 2024-04-06T14:17:22.069 INFO:tasks.workunit.client.1.smithi086.stdout:7/799: creat d7/d10/d11/d1c/d24/dc8/dcd/f115 x:0 0 0 2024-04-06T14:17:22.071 INFO:tasks.workunit.client.1.smithi086.stdout:8/895: dread d0/d71/d22/f8e [0,4194304] 0 2024-04-06T14:17:22.071 INFO:tasks.workunit.client.1.smithi086.stdout:6/972: chown d6/d13/d1a/da0 2138224736 1 2024-04-06T14:17:22.072 INFO:tasks.workunit.client.1.smithi086.stdout:9/771: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dfa 0 2024-04-06T14:17:22.077 INFO:tasks.workunit.client.1.smithi086.stdout:7/800: rename d7/d10/d102/d5c/ded to d7/d10/d11/d1c/d24/df0/d116 0 2024-04-06T14:17:22.077 INFO:tasks.workunit.client.1.smithi086.stdout:6/973: dread d6/d9/d43/f56 [0,4194304] 0 2024-04-06T14:17:22.080 INFO:tasks.workunit.client.1.smithi086.stdout:2/888: dread d0/d20/d44/dbb/dc4/de3/d76/f91 [0,4194304] 0 2024-04-06T14:17:22.080 INFO:tasks.workunit.client.1.smithi086.stdout:9/772: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/ffb x:0 0 0 2024-04-06T14:17:22.081 INFO:tasks.workunit.client.1.smithi086.stdout:9/773: stat c1 0 2024-04-06T14:17:22.081 INFO:tasks.workunit.client.1.smithi086.stdout:7/801: symlink d7/d10/d11/d21/l117 0 2024-04-06T14:17:22.083 INFO:tasks.workunit.client.1.smithi086.stdout:7/802: read d7/d10/d11/d1c/d24/dc8/fde [265141,128502] 0 2024-04-06T14:17:22.087 INFO:tasks.workunit.client.1.smithi086.stdout:2/889: dread d0/d20/d27/f43 [0,4194304] 0 2024-04-06T14:17:22.090 INFO:tasks.workunit.client.1.smithi086.stdout:7/803: symlink d7/d10/d11/l118 0 2024-04-06T14:17:22.090 INFO:tasks.workunit.client.1.smithi086.stdout:9/774: dwrite d5/d13/d1a/d5f/d88/d14/f86 [0,4194304] 0 2024-04-06T14:17:22.090 INFO:tasks.workunit.client.1.smithi086.stdout:6/974: rmdir d6/d13/d42 39 2024-04-06T14:17:22.092 INFO:tasks.workunit.client.1.smithi086.stdout:6/975: creat d6/d13/d5f/f151 x:0 0 0 2024-04-06T14:17:22.100 INFO:tasks.workunit.client.1.smithi086.stdout:8/896: dwrite d0/d3b/d79/dc6/dbc/fbf [0,4194304] 0 2024-04-06T14:17:22.100 INFO:tasks.workunit.client.1.smithi086.stdout:9/775: creat d5/d13/d1a/d5f/d88/d14/d19/ffc x:0 0 0 2024-04-06T14:17:22.101 INFO:tasks.workunit.client.1.smithi086.stdout:8/897: dread - d0/d71/d22/d43/d6d/f109 zero size 2024-04-06T14:17:22.102 INFO:tasks.workunit.client.1.smithi086.stdout:6/976: unlink d6/d13/d1a/da0/d11b/d45/d12d/f14a 0 2024-04-06T14:17:22.103 INFO:tasks.workunit.client.1.smithi086.stdout:4/711: sync 2024-04-06T14:17:22.103 INFO:tasks.workunit.client.1.smithi086.stdout:0/861: sync 2024-04-06T14:17:22.103 INFO:tasks.workunit.client.1.smithi086.stdout:1/879: sync 2024-04-06T14:17:22.103 INFO:tasks.workunit.client.1.smithi086.stdout:3/845: sync 2024-04-06T14:17:22.104 INFO:tasks.workunit.client.1.smithi086.stdout:2/890: write d0/df/f14 [463751,88679] 0 2024-04-06T14:17:22.107 INFO:tasks.workunit.client.1.smithi086.stdout:8/898: symlink d0/d71/d16/d52/d5f/l138 0 2024-04-06T14:17:22.109 INFO:tasks.workunit.client.1.smithi086.stdout:7/804: truncate d7/d10/d11/d1c/d24/dc8/ffe 3328686 0 2024-04-06T14:17:22.109 INFO:tasks.workunit.client.1.smithi086.stdout:4/712: creat d0/d15/d2b/fda x:0 0 0 2024-04-06T14:17:22.112 INFO:tasks.workunit.client.1.smithi086.stdout:1/880: mknod db/d11/d87/c117 0 2024-04-06T14:17:22.114 INFO:tasks.workunit.client.1.smithi086.stdout:9/776: symlink d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/lfd 0 2024-04-06T14:17:22.116 INFO:tasks.workunit.client.1.smithi086.stdout:0/862: rename d0/d1/d2/d1a/d79/ff8 to d0/d1/d2/d1a/d51/d108/f11b 0 2024-04-06T14:17:22.117 INFO:tasks.workunit.client.1.smithi086.stdout:2/891: truncate d0/d20/d27/d2b/d35/d36/d4d/d59/f8b 4062788 0 2024-04-06T14:17:22.118 INFO:tasks.workunit.client.1.smithi086.stdout:7/805: dwrite d7/d10/d11/d1c/d24/d2c/f70 [4194304,4194304] 0 2024-04-06T14:17:22.122 INFO:tasks.workunit.client.1.smithi086.stdout:3/846: link d5/d26/d7f/de7/fe8 d5/d36/d5a/d6d/d72/d9a/de2/f114 0 2024-04-06T14:17:22.122 INFO:tasks.workunit.client.1.smithi086.stdout:9/777: truncate d5/d18/f1d 242371 0 2024-04-06T14:17:22.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:21 smithi086 ceph-mon[47925]: pgmap v28: 65 pgs: 65 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 79 MiB/s rd, 177 MiB/s wr, 417 op/s 2024-04-06T14:17:22.131 INFO:tasks.workunit.client.1.smithi086.stdout:1/881: link db/d13/f54 db/d11/d1a/d2a/d9a/d78/f118 0 2024-04-06T14:17:22.131 INFO:tasks.workunit.client.1.smithi086.stdout:0/863: dwrite d0/d1/d4/f6b [4194304,4194304] 0 2024-04-06T14:17:22.131 INFO:tasks.workunit.client.1.smithi086.stdout:3/847: write d5/d36/d7d/f109 [134137,8061] 0 2024-04-06T14:17:22.136 INFO:tasks.workunit.client.1.smithi086.stdout:9/778: dread - d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/ffb zero size 2024-04-06T14:17:22.138 INFO:tasks.workunit.client.1.smithi086.stdout:9/779: chown d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/lfd 420692040 1 2024-04-06T14:17:22.138 INFO:tasks.workunit.client.1.smithi086.stdout:8/899: dread d0/d71/d16/f41 [0,4194304] 0 2024-04-06T14:17:22.140 INFO:tasks.workunit.client.1.smithi086.stdout:9/780: chown d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dfa 13 1 2024-04-06T14:17:22.141 INFO:tasks.workunit.client.1.smithi086.stdout:7/806: rename d7/d10/d92/ff8 to d7/d10/d102/daf/db5/da1/f119 0 2024-04-06T14:17:22.145 INFO:tasks.workunit.client.1.smithi086.stdout:1/882: creat db/d11/d1a/d2a/d9a/d78/d101/f119 x:0 0 0 2024-04-06T14:17:22.146 INFO:tasks.workunit.client.1.smithi086.stdout:0/864: creat d0/d15/f11c x:0 0 0 2024-04-06T14:17:22.149 INFO:tasks.workunit.client.1.smithi086.stdout:9/781: truncate d5/d13/d1a/d5f/d88/d14/ff5 188689 0 2024-04-06T14:17:22.149 INFO:tasks.workunit.client.1.smithi086.stdout:7/807: mknod d7/d10/d11/d1c/d24/dc8/dcd/c11a 0 2024-04-06T14:17:22.150 INFO:tasks.workunit.client.1.smithi086.stdout:1/883: stat db/d11/d1a/d2a/d9a/d68/daf/d104 0 2024-04-06T14:17:22.151 INFO:tasks.workunit.client.1.smithi086.stdout:8/900: dread d0/d3b/d79/fc5 [0,4194304] 0 2024-04-06T14:17:22.154 INFO:tasks.workunit.client.1.smithi086.stdout:0/865: creat d0/d1/d2/d1a/d79/dac/db7/f11d x:0 0 0 2024-04-06T14:17:22.154 INFO:tasks.workunit.client.1.smithi086.stdout:4/713: dwrite d0/d2c/d30/d24/d93/fa4 [0,4194304] 0 2024-04-06T14:17:22.155 INFO:tasks.workunit.client.1.smithi086.stdout:2/892: dread d0/df/d5e/fc7 [0,4194304] 0 2024-04-06T14:17:22.161 INFO:tasks.workunit.client.1.smithi086.stdout:9/782: dread d5/d58/fdd [0,4194304] 0 2024-04-06T14:17:22.161 INFO:tasks.workunit.client.1.smithi086.stdout:7/808: write d7/d10/d11/d1c/d24/dc8/dcd/f115 [524716,120040] 0 2024-04-06T14:17:22.162 INFO:tasks.workunit.client.1.smithi086.stdout:0/866: dread - d0/d15/d26/d33/d9f/fdb zero size 2024-04-06T14:17:22.162 INFO:tasks.workunit.client.1.smithi086.stdout:3/848: dread d5/d36/d5c/faa [0,4194304] 0 2024-04-06T14:17:22.162 INFO:tasks.workunit.client.1.smithi086.stdout:2/893: write d0/d20/d44/dbb/dc4/de3/f54 [2072840,96280] 0 2024-04-06T14:17:22.163 INFO:tasks.workunit.client.1.smithi086.stdout:1/884: rename db/d11/d1a/d2a/d9a/d78/ffd to db/d11/d1a/d2a/d9a/d68/f11a 0 2024-04-06T14:17:22.164 INFO:tasks.workunit.client.1.smithi086.stdout:7/809: write d7/d10/d11/d1c/d24/dc8/fda [5081137,108847] 0 2024-04-06T14:17:22.165 INFO:tasks.workunit.client.1.smithi086.stdout:9/783: fdatasync d5/d13/d1a/d5f/d88/f53 0 2024-04-06T14:17:22.165 INFO:tasks.workunit.client.1.smithi086.stdout:0/867: rmdir d0/d1/d2/d1a/d79/dac/dc2 39 2024-04-06T14:17:22.168 INFO:tasks.workunit.client.1.smithi086.stdout:9/784: chown d5/d13/lb0 1 1 2024-04-06T14:17:22.168 INFO:tasks.workunit.client.1.smithi086.stdout:0/868: readlink d0/d1/d2/d1a/d79/lc3 0 2024-04-06T14:17:22.169 INFO:tasks.workunit.client.1.smithi086.stdout:3/849: dread - d5/d36/d5a/d6d/d72/d97/db0/fd5 zero size 2024-04-06T14:17:22.170 INFO:tasks.workunit.client.1.smithi086.stdout:9/785: write d5/d13/f7c [1667074,126172] 0 2024-04-06T14:17:22.170 INFO:tasks.workunit.client.1.smithi086.stdout:2/894: dread d0/d20/d27/d2b/d35/d36/d4a/f97 [0,4194304] 0 2024-04-06T14:17:22.170 INFO:tasks.workunit.client.1.smithi086.stdout:4/714: rename d0/l1b to d0/d2c/d30/ldb 0 2024-04-06T14:17:22.172 INFO:tasks.workunit.client.1.smithi086.stdout:7/810: symlink d7/d10/d102/d5c/l11b 0 2024-04-06T14:17:22.173 INFO:tasks.workunit.client.1.smithi086.stdout:5/730: sync 2024-04-06T14:17:22.174 INFO:tasks.workunit.client.1.smithi086.stdout:9/786: stat d5/c96 0 2024-04-06T14:17:22.174 INFO:tasks.workunit.client.1.smithi086.stdout:9/787: stat d5/d58/fbe 0 2024-04-06T14:17:22.176 INFO:tasks.workunit.client.1.smithi086.stdout:2/895: creat d0/d20/d27/d2b/d35/d36/d4a/db8/f13e x:0 0 0 2024-04-06T14:17:22.176 INFO:tasks.workunit.client.1.smithi086.stdout:4/715: read d0/d2c/d38/f5b [2345901,92807] 0 2024-04-06T14:17:22.177 INFO:tasks.workunit.client.1.smithi086.stdout:5/731: unlink d0/d16/c3b 0 2024-04-06T14:17:22.178 INFO:tasks.workunit.client.1.smithi086.stdout:1/885: rename db/d11/d1a/d2a/d52/d6b/le6 to db/d11/d1a/d2a/d9a/d78/l11b 0 2024-04-06T14:17:22.179 INFO:tasks.workunit.client.1.smithi086.stdout:0/869: symlink d0/d15/d26/d33/d7a/daa/d110/l11e 0 2024-04-06T14:17:22.180 INFO:tasks.workunit.client.1.smithi086.stdout:0/870: chown d0/d15/d4e/d48/f31 7 1 2024-04-06T14:17:22.183 INFO:tasks.workunit.client.1.smithi086.stdout:3/850: dread d5/dea/dd2/ddb/de6/f85 [0,4194304] 0 2024-04-06T14:17:22.183 INFO:tasks.workunit.client.1.smithi086.stdout:9/788: rename d5/d58/l6e to d5/d13/d1a/d5f/d88/d14/d90/lfe 0 2024-04-06T14:17:22.183 INFO:tasks.workunit.client.1.smithi086.stdout:2/896: mknod d0/d20/d27/d2b/d35/d36/d4d/d59/db1/c13f 0 2024-04-06T14:17:22.184 INFO:tasks.workunit.client.1.smithi086.stdout:7/811: creat d7/d10/d11/d1c/d24/d2c/f11c x:0 0 0 2024-04-06T14:17:22.185 INFO:tasks.workunit.client.1.smithi086.stdout:0/871: read d0/d15/d26/d33/d7a/daa/ff7 [265459,8165] 0 2024-04-06T14:17:22.185 INFO:tasks.workunit.client.1.smithi086.stdout:3/851: chown d5/d36/d5a/d6d/d72/d108/f113 251591856 1 2024-04-06T14:17:22.187 INFO:tasks.workunit.client.1.smithi086.stdout:3/852: fdatasync d5/dea/dd2/ddb/de6/fe3 0 2024-04-06T14:17:22.188 INFO:tasks.workunit.client.1.smithi086.stdout:9/789: rename d5/d13/d1a/d8f to d5/d18/dff 0 2024-04-06T14:17:22.189 INFO:tasks.workunit.client.1.smithi086.stdout:2/897: symlink d0/d20/d27/d2b/d35/d36/d4a/l140 0 2024-04-06T14:17:22.191 INFO:tasks.workunit.client.1.smithi086.stdout:0/872: truncate d0/d15/d4e/da3/d46/d52/fb6 658353 0 2024-04-06T14:17:22.194 INFO:tasks.workunit.client.1.smithi086.stdout:9/790: dwrite d5/d13/d1a/d5f/d88/d14/d19/ffc [0,4194304] 0 2024-04-06T14:17:22.195 INFO:tasks.workunit.client.1.smithi086.stdout:7/812: rename d7/d1d/c2d to d7/d10/d11/d1c/d104/c11d 0 2024-04-06T14:17:22.199 INFO:tasks.workunit.client.1.smithi086.stdout:3/853: unlink d5/dea/dd2/ddb/de6/d10b/lf3 0 2024-04-06T14:17:22.201 INFO:tasks.workunit.client.1.smithi086.stdout:0/873: creat d0/d1/d4/de/df9/f11f x:0 0 0 2024-04-06T14:17:22.202 INFO:tasks.workunit.client.1.smithi086.stdout:9/791: fsync d5/d13/d1a/d5f/f6a 0 2024-04-06T14:17:22.203 INFO:tasks.workunit.client.1.smithi086.stdout:3/854: readlink d5/dea/dd2/ddb/de6/l16 0 2024-04-06T14:17:22.204 INFO:tasks.workunit.client.1.smithi086.stdout:2/898: getdents d0/df 0 2024-04-06T14:17:22.205 INFO:tasks.workunit.client.1.smithi086.stdout:7/813: creat d7/d10/d102/daf/db5/f11e x:0 0 0 2024-04-06T14:17:22.206 INFO:tasks.workunit.client.1.smithi086.stdout:9/792: stat d5/d13/d1a/fb6 0 2024-04-06T14:17:22.209 INFO:tasks.workunit.client.1.smithi086.stdout:4/716: dwrite d0/f10 [0,4194304] 0 2024-04-06T14:17:22.209 INFO:tasks.workunit.client.1.smithi086.stdout:3/855: fdatasync d5/dea/dd2/ddb/de6/d10b/fc5 0 2024-04-06T14:17:22.215 INFO:tasks.workunit.client.1.smithi086.stdout:1/886: dwrite db/fe [4194304,4194304] 0 2024-04-06T14:17:22.216 INFO:tasks.workunit.client.1.smithi086.stdout:7/814: mknod d7/d10/d102/daf/db5/d7a/c11f 0 2024-04-06T14:17:22.216 INFO:tasks.workunit.client.1.smithi086.stdout:9/793: write d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/ffb [995582,54970] 0 2024-04-06T14:17:22.219 INFO:tasks.workunit.client.1.smithi086.stdout:4/717: creat d0/d2c/d30/d24/d93/fdc x:0 0 0 2024-04-06T14:17:22.222 INFO:tasks.workunit.client.1.smithi086.stdout:4/718: chown d0/d2c/d38/l4f 418080 1 2024-04-06T14:17:22.224 INFO:tasks.workunit.client.1.smithi086.stdout:7/815: creat d7/d10/d11/d21/f120 x:0 0 0 2024-04-06T14:17:22.225 INFO:tasks.workunit.client.0.smithi032.stdout:5/984: sync 2024-04-06T14:17:22.225 INFO:tasks.workunit.client.0.smithi032.stdout:0/842: sync 2024-04-06T14:17:22.226 INFO:tasks.workunit.client.1.smithi086.stdout:3/856: dwrite d5/d33/f78 [0,4194304] 0 2024-04-06T14:17:22.227 INFO:tasks.workunit.client.1.smithi086.stdout:3/857: chown d5/d33/cba 101322818 1 2024-04-06T14:17:22.227 INFO:tasks.workunit.client.0.smithi032.stdout:0/843: fsync d3/f78 0 2024-04-06T14:17:22.228 INFO:tasks.workunit.client.0.smithi032.stdout:0/844: stat d3/da/d25/d2b/d80/db3/dcd 0 2024-04-06T14:17:22.230 INFO:tasks.workunit.client.1.smithi086.stdout:4/719: mkdir d0/db0/ddd 0 2024-04-06T14:17:22.233 INFO:tasks.workunit.client.0.smithi032.stdout:0/845: mkdir d3/da/d25/d2b/d80/db3/df1/d120 0 2024-04-06T14:17:22.234 INFO:tasks.workunit.client.1.smithi086.stdout:1/887: write db/d11/d1a/d2a/d9a/d78/fc8 [779696,98476] 0 2024-04-06T14:17:22.236 INFO:tasks.workunit.client.1.smithi086.stdout:4/720: creat d0/d2c/d38/d54/d61/d7a/dc2/fde x:0 0 0 2024-04-06T14:17:22.240 INFO:tasks.workunit.client.1.smithi086.stdout:2/899: dread d0/df/d48/d7d/f87 [0,4194304] 0 2024-04-06T14:17:22.240 INFO:tasks.workunit.client.0.smithi032.stdout:0/846: link d3/da/d25/d2b/l3f d3/da/d25/d101/l121 0 2024-04-06T14:17:22.241 INFO:tasks.workunit.client.1.smithi086.stdout:4/721: fsync d0/d15/d20/f57 0 2024-04-06T14:17:22.242 INFO:tasks.workunit.client.1.smithi086.stdout:7/816: dwrite d7/d1d/f38 [4194304,4194304] 0 2024-04-06T14:17:22.242 INFO:tasks.workunit.client.0.smithi032.stdout:5/985: write d16/d18/d33/df7/ffb [1980117,127302] 0 2024-04-06T14:17:22.244 INFO:tasks.workunit.client.0.smithi032.stdout:0/847: creat d3/da/d25/d2b/d80/db3/df1/d120/f122 x:0 0 0 2024-04-06T14:17:22.245 INFO:tasks.workunit.client.0.smithi032.stdout:5/986: truncate d16/d18/dae/dbb/ddc/fe7 2114 0 2024-04-06T14:17:22.247 INFO:tasks.workunit.client.0.smithi032.stdout:0/848: mkdir d3/dc2/d123 0 2024-04-06T14:17:22.250 INFO:tasks.workunit.client.0.smithi032.stdout:5/987: unlink d16/d18/d33/d45/c7d 0 2024-04-06T14:17:22.252 INFO:tasks.workunit.client.1.smithi086.stdout:9/794: dread d5/f20 [0,4194304] 0 2024-04-06T14:17:22.252 INFO:tasks.workunit.client.1.smithi086.stdout:7/817: rename d7/d10/d102/daf/db5/f8a to d7/d10/d92/f121 0 2024-04-06T14:17:22.253 INFO:tasks.workunit.client.1.smithi086.stdout:9/795: readlink d5/d18/l52 0 2024-04-06T14:17:22.254 INFO:tasks.workunit.client.0.smithi032.stdout:0/849: link d3/da/d25/d2b/d4a/c52 d3/dc2/dd8/c124 0 2024-04-06T14:17:22.256 INFO:tasks.workunit.client.0.smithi032.stdout:5/988: creat d16/d58/d54/d74/f152 x:0 0 0 2024-04-06T14:17:22.256 INFO:tasks.workunit.client.1.smithi086.stdout:4/722: getdents d0/d2c/d38/d54/d61/d7a/dc2/dd3 0 2024-04-06T14:17:22.257 INFO:tasks.workunit.client.1.smithi086.stdout:7/818: creat d7/d10/d11/d21/f122 x:0 0 0 2024-04-06T14:17:22.257 INFO:tasks.workunit.client.1.smithi086.stdout:3/858: dread d5/dea/dd2/ddb/de6/f45 [0,4194304] 0 2024-04-06T14:17:22.258 INFO:tasks.workunit.client.0.smithi032.stdout:5/989: symlink d16/d58/d54/l153 0 2024-04-06T14:17:22.259 INFO:tasks.workunit.client.0.smithi032.stdout:0/850: rmdir d3/da/d25/d2b/d4a/d86/d10c 0 2024-04-06T14:17:22.260 INFO:tasks.workunit.client.1.smithi086.stdout:2/900: truncate d0/d20/d44/dbb/fc0 277564 0 2024-04-06T14:17:22.260 INFO:tasks.workunit.client.1.smithi086.stdout:4/723: chown d0/d2c/d30/d24/d71/cb3 452530581 1 2024-04-06T14:17:22.260 INFO:tasks.workunit.client.0.smithi032.stdout:5/990: symlink d16/d18/d33/d35/d88/dbd/d120/l154 0 2024-04-06T14:17:22.261 INFO:tasks.workunit.client.1.smithi086.stdout:3/859: chown d5/dea/dd2/ddb/de6/d10b 913 1 2024-04-06T14:17:22.261 INFO:tasks.workunit.client.1.smithi086.stdout:9/796: rename d5/d13/d1a/d5f/d88/d91/db9 to d5/d58/d100 0 2024-04-06T14:17:22.263 INFO:tasks.workunit.client.0.smithi032.stdout:0/851: truncate d3/f55 2494916 0 2024-04-06T14:17:22.263 INFO:tasks.workunit.client.1.smithi086.stdout:1/888: dwrite db/d11/d1a/d2a/d9a/d68/d6e/f9e [4194304,4194304] 0 2024-04-06T14:17:22.264 INFO:tasks.workunit.client.1.smithi086.stdout:2/901: read d0/d20/d44/d120/f134 [2593792,81750] 0 2024-04-06T14:17:22.265 INFO:tasks.workunit.client.0.smithi032.stdout:5/991: mkdir d16/d18/d33/dea/d155 0 2024-04-06T14:17:22.265 INFO:tasks.workunit.client.0.smithi032.stdout:0/852: mkdir d3/da/d41/d107/d125 0 2024-04-06T14:17:22.265 INFO:tasks.workunit.client.1.smithi086.stdout:3/860: stat d5/d36/d5a/d6d/d72/d64 0 2024-04-06T14:17:22.267 INFO:tasks.workunit.client.1.smithi086.stdout:7/819: truncate d7/d10/d11/d80/f4b 763395 0 2024-04-06T14:17:22.268 INFO:tasks.workunit.client.1.smithi086.stdout:1/889: chown db/d11/d1a/d2a/d9a/d68/daf/lf3 1866024 1 2024-04-06T14:17:22.269 INFO:tasks.workunit.client.0.smithi032.stdout:5/992: dwrite d16/fab [0,4194304] 0 2024-04-06T14:17:22.270 INFO:tasks.workunit.client.1.smithi086.stdout:2/902: stat d0/df/d5e/d9d/fe0 0 2024-04-06T14:17:22.271 INFO:tasks.workunit.client.0.smithi032.stdout:5/993: fsync d16/d58/d54/d74/d8f/fe1 0 2024-04-06T14:17:22.273 INFO:tasks.workunit.client.0.smithi032.stdout:5/994: chown d16/d18/d33/d45/d52/d39/fd5 6789 1 2024-04-06T14:17:22.283 INFO:tasks.workunit.client.0.smithi032.stdout:0/853: getdents d3/dc2/dd8 0 2024-04-06T14:17:22.288 INFO:tasks.workunit.client.0.smithi032.stdout:0/854: mknod d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/d9c/c126 0 2024-04-06T14:17:22.288 INFO:tasks.workunit.client.1.smithi086.stdout:6/977: sync 2024-04-06T14:17:22.289 INFO:tasks.workunit.client.0.smithi032.stdout:0/855: chown d3/da/d25/d2b/d4a/d79/d97/de0/df5/f117 0 1 2024-04-06T14:17:22.291 INFO:tasks.workunit.client.0.smithi032.stdout:0/856: fdatasync d3/da/d25/d57/fd7 0 2024-04-06T14:17:22.292 INFO:tasks.workunit.client.0.smithi032.stdout:0/857: chown d3/da/d25/d2b/f32 3694904 1 2024-04-06T14:17:22.294 INFO:tasks.workunit.client.1.smithi086.stdout:3/861: dread d5/d36/d65/fa6 [0,4194304] 0 2024-04-06T14:17:22.297 INFO:tasks.workunit.client.0.smithi032.stdout:0/858: write d3/da/d25/d57/f7d [115242,91823] 0 2024-04-06T14:17:22.301 INFO:tasks.workunit.client.0.smithi032.stdout:0/859: rmdir d3/da/d25/d2b/d4a/d86 39 2024-04-06T14:17:22.304 INFO:tasks.workunit.client.0.smithi032.stdout:0/860: truncate d3/da/d25/d2b/f3d 634619 0 2024-04-06T14:17:22.312 INFO:tasks.workunit.client.0.smithi032.stdout:0/861: write d3/da/d25/f54 [3973860,123953] 0 2024-04-06T14:17:22.315 INFO:tasks.workunit.client.1.smithi086.stdout:3/862: dread d5/d33/d5d/fa2 [0,4194304] 0 2024-04-06T14:17:22.315 INFO:tasks.workunit.client.0.smithi032.stdout:0/862: unlink d3/da/d25/l9e 0 2024-04-06T14:17:22.317 INFO:tasks.workunit.client.0.smithi032.stdout:0/863: rmdir d3 39 2024-04-06T14:17:22.325 INFO:tasks.workunit.client.0.smithi032.stdout:0/864: getdents d3/da/d25/d2b/d4a/d79/d7c 0 2024-04-06T14:17:22.326 INFO:tasks.workunit.client.1.smithi086.stdout:3/863: dwrite d5/dea/dd2/ddb/de6/ff0 [0,4194304] 0 2024-04-06T14:17:22.330 INFO:tasks.workunit.client.0.smithi032.stdout:0/865: link d3/da/d25/d2b/f61 d3/da/d41/d107/f127 0 2024-04-06T14:17:22.337 INFO:tasks.workunit.client.1.smithi086.stdout:1/890: dwrite db/d11/d87/fb6 [0,4194304] 0 2024-04-06T14:17:22.339 INFO:tasks.workunit.client.0.smithi032.stdout:0/866: unlink d3/da/d25/d2b/d4a/d86/f106 0 2024-04-06T14:17:22.340 INFO:tasks.workunit.client.1.smithi086.stdout:2/903: fsync d0/f12 0 2024-04-06T14:17:22.343 INFO:tasks.workunit.client.1.smithi086.stdout:4/724: truncate d0/d15/d20/d74/f7b 412425 0 2024-04-06T14:17:22.346 INFO:tasks.workunit.client.1.smithi086.stdout:6/978: rename d6/d20/c120 to d6/d9/db/c152 0 2024-04-06T14:17:22.351 INFO:tasks.workunit.client.0.smithi032.stdout:0/867: write d3/f4 [4171975,57055] 0 2024-04-06T14:17:22.354 INFO:tasks.workunit.client.0.smithi032.stdout:0/868: rmdir d3/da/d25/d2b/d80/db3/df1/d120 39 2024-04-06T14:17:22.358 INFO:tasks.workunit.client.1.smithi086.stdout:7/820: fdatasync d7/d10/d11/d1c/d24/d2c/faa 0 2024-04-06T14:17:22.359 INFO:tasks.workunit.client.1.smithi086.stdout:7/821: readlink d7/d1d/l3b 0 2024-04-06T14:17:22.360 INFO:tasks.workunit.client.1.smithi086.stdout:9/797: creat d5/d13/d1a/d5f/d88/d14/f101 x:0 0 0 2024-04-06T14:17:22.361 INFO:tasks.workunit.client.1.smithi086.stdout:7/822: stat d7/d10/d11/d1c/d24/d2c/fe3 0 2024-04-06T14:17:22.361 INFO:tasks.workunit.client.1.smithi086.stdout:2/904: dread - d0/d20/d27/d2b/d35/d36/fba zero size 2024-04-06T14:17:22.364 INFO:tasks.workunit.client.1.smithi086.stdout:4/725: chown d0/d2c/d30/d24/da3/f52 139384 1 2024-04-06T14:17:22.365 INFO:tasks.workunit.client.0.smithi032.stdout:0/869: dread d3/da/d25/d36/f51 [0,4194304] 0 2024-04-06T14:17:22.368 INFO:tasks.workunit.client.0.smithi032.stdout:0/870: fdatasync d3/da/d25/d2b/d4a/d86/fc4 0 2024-04-06T14:17:22.368 INFO:tasks.workunit.client.1.smithi086.stdout:6/979: creat d6/d13/d1a/da0/d11b/d45/d8a/f153 x:0 0 0 2024-04-06T14:17:22.370 INFO:tasks.workunit.client.1.smithi086.stdout:1/891: mknod db/d11/d1a/d2a/d9a/d68/daf/dc2/d116/c11c 0 2024-04-06T14:17:22.370 INFO:tasks.workunit.client.0.smithi032.stdout:0/871: mknod d3/da/d25/d2b/d4a/d86/dd6/c128 0 2024-04-06T14:17:22.371 INFO:tasks.workunit.client.0.smithi032.stdout:0/872: write d3/f5d [3426732,23817] 0 2024-04-06T14:17:22.372 INFO:tasks.workunit.client.1.smithi086.stdout:9/798: truncate d5/d18/f32 1282979 0 2024-04-06T14:17:22.374 INFO:tasks.workunit.client.1.smithi086.stdout:7/823: write d7/d10/d102/d5c/d66/d7e/db9/fc3 [1076353,124639] 0 2024-04-06T14:17:22.376 INFO:tasks.workunit.client.0.smithi032.stdout:0/873: mkdir d3/da/d25/d2b/d129 0 2024-04-06T14:17:22.376 INFO:tasks.workunit.client.1.smithi086.stdout:2/905: mknod d0/d20/d80/dbf/ddc/c141 0 2024-04-06T14:17:22.376 INFO:tasks.workunit.client.1.smithi086.stdout:6/980: mknod d6/d13/d1a/da0/d114/db2/dee/c154 0 2024-04-06T14:17:22.377 INFO:tasks.workunit.client.1.smithi086.stdout:2/906: readlink d0/d20/d27/dae/lf1 0 2024-04-06T14:17:22.379 INFO:tasks.workunit.client.1.smithi086.stdout:1/892: creat db/d11/d1a/d79/da5/f11d x:0 0 0 2024-04-06T14:17:22.382 INFO:tasks.workunit.client.1.smithi086.stdout:1/893: read db/d11/d42/f50 [1255061,31048] 0 2024-04-06T14:17:22.382 INFO:tasks.workunit.client.0.smithi032.stdout:0/874: dwrite d3/f4 [0,4194304] 0 2024-04-06T14:17:22.384 INFO:tasks.workunit.client.1.smithi086.stdout:1/894: readlink db/d11/d1a/d2a/d52/l7d 0 2024-04-06T14:17:22.387 INFO:tasks.workunit.client.1.smithi086.stdout:6/981: dwrite d6/d13/d1a/da0/d11b/d45/d8a/fc2 [0,4194304] 0 2024-04-06T14:17:22.389 INFO:tasks.workunit.client.1.smithi086.stdout:2/907: truncate d0/d20/d27/d2b/d35/d36/d4d/d6f/fc6 862837 0 2024-04-06T14:17:22.389 INFO:tasks.workunit.client.0.smithi032.stdout:0/875: dwrite d3/f4 [0,4194304] 0 2024-04-06T14:17:22.392 INFO:tasks.workunit.client.0.smithi032.stdout:0/876: fsync d3/da/faa 0 2024-04-06T14:17:22.392 INFO:tasks.workunit.client.0.smithi032.stdout:0/877: chown d3/d83/dfd 1 1 2024-04-06T14:17:22.396 INFO:tasks.workunit.client.0.smithi032.stdout:0/878: rename d3/da/d41/l87 to d3/d83/dfd/l12a 0 2024-04-06T14:17:22.400 INFO:tasks.workunit.client.0.smithi032.stdout:0/879: creat d3/da/f12b x:0 0 0 2024-04-06T14:17:22.408 INFO:tasks.workunit.client.0.smithi032.stdout:0/880: truncate d3/f76 2602436 0 2024-04-06T14:17:22.422 INFO:tasks.workunit.client.1.smithi086.stdout:9/799: getdents d5/d13 0 2024-04-06T14:17:22.424 INFO:tasks.workunit.client.1.smithi086.stdout:9/800: read - d5/d13/d1a/d5f/d88/db3/fe3 zero size 2024-04-06T14:17:22.425 INFO:tasks.workunit.client.1.smithi086.stdout:9/801: read - d5/d13/d1a/d5f/d88/db3/fe3 zero size 2024-04-06T14:17:22.426 INFO:tasks.workunit.client.1.smithi086.stdout:2/908: mkdir d0/d20/d27/d2b/d35/d36/dbe/dfc/d142 0 2024-04-06T14:17:22.427 INFO:tasks.workunit.client.1.smithi086.stdout:2/909: write d0/df/f84 [2447998,52067] 0 2024-04-06T14:17:22.428 INFO:tasks.workunit.client.1.smithi086.stdout:6/982: creat d6/d13/d1a/da0/d114/db2/dee/d145/f155 x:0 0 0 2024-04-06T14:17:22.428 INFO:tasks.workunit.client.0.smithi032.stdout:0/881: rename d3/da/d25/d2b/d4a/d79/d7c to d3/da/d25/d2b/d4a/d79/d97/de0/df5/d12c 0 2024-04-06T14:17:22.431 INFO:tasks.workunit.client.0.smithi032.stdout:0/882: mknod d3/dc2/c12d 0 2024-04-06T14:17:22.431 INFO:tasks.workunit.client.0.smithi032.stdout:0/883: read d3/f4 [3570950,15257] 0 2024-04-06T14:17:22.433 INFO:tasks.workunit.client.0.smithi032.stdout:0/884: fsync d3/da/d25/d2b/d4a/d79/d97/f116 0 2024-04-06T14:17:22.436 INFO:tasks.workunit.client.0.smithi032.stdout:0/885: rmdir d3/da/d25/d2b/d4a/d79/da8 39 2024-04-06T14:17:22.436 INFO:tasks.workunit.client.1.smithi086.stdout:2/910: mkdir d0/df/d5e/d9d/d102/d143 0 2024-04-06T14:17:22.436 INFO:tasks.workunit.client.1.smithi086.stdout:9/802: symlink d5/d13/d1a/d5f/d88/d14/d19/d25/deb/l102 0 2024-04-06T14:17:22.436 INFO:tasks.workunit.client.1.smithi086.stdout:2/911: chown d0/d20/d27/d2b/d35/d36/d4d/d59 71 1 2024-04-06T14:17:22.437 INFO:tasks.workunit.client.0.smithi032.stdout:0/886: truncate d3/da/d25/d2b/d4a/fb9 1326029 0 2024-04-06T14:17:22.440 INFO:tasks.workunit.client.1.smithi086.stdout:9/803: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/d103 0 2024-04-06T14:17:22.440 INFO:tasks.workunit.client.1.smithi086.stdout:9/804: chown d5/d18/dff/f9a 821824 1 2024-04-06T14:17:22.444 INFO:tasks.workunit.client.0.smithi032.stdout:0/887: dwrite d3/da/d25/d2b/d4a/d79/d97/f116 [0,4194304] 0 2024-04-06T14:17:22.445 INFO:tasks.workunit.client.1.smithi086.stdout:9/805: symlink d5/d13/d47/l104 0 2024-04-06T14:17:22.450 INFO:tasks.workunit.client.1.smithi086.stdout:6/983: mkdir d6/d13/d5f/d156 0 2024-04-06T14:17:22.450 INFO:tasks.workunit.client.1.smithi086.stdout:9/806: mkdir d5/d13/d47/d5e/dc2/d105 0 2024-04-06T14:17:22.452 INFO:tasks.workunit.client.1.smithi086.stdout:6/984: mkdir d6/d13/d1a/da0/d11b/d45/d12d/d157 0 2024-04-06T14:17:22.454 INFO:tasks.workunit.client.1.smithi086.stdout:6/985: dread - d6/d13/d1a/da0/d114/db2/f107 zero size 2024-04-06T14:17:22.460 INFO:tasks.workunit.client.1.smithi086.stdout:6/986: dread d6/d13/d42/d52/df1/ff2 [0,4194304] 0 2024-04-06T14:17:22.461 INFO:tasks.workunit.client.1.smithi086.stdout:9/807: dwrite d5/d13/d1a/d5f/d88/d14/d19/d25/f67 [0,4194304] 0 2024-04-06T14:17:22.464 INFO:tasks.workunit.client.1.smithi086.stdout:6/987: write d6/d13/d1a/da0/d114/db2/f107 [95136,32262] 0 2024-04-06T14:17:22.468 INFO:tasks.workunit.client.1.smithi086.stdout:6/988: symlink d6/d13/d1a/da0/d11b/d45/d8a/l158 0 2024-04-06T14:17:22.470 INFO:tasks.workunit.client.1.smithi086.stdout:6/989: truncate f5 4858315 0 2024-04-06T14:17:22.472 INFO:tasks.workunit.client.1.smithi086.stdout:6/990: truncate d6/f148 116585 0 2024-04-06T14:17:22.486 INFO:tasks.workunit.client.1.smithi086.stdout:8/901: sync 2024-04-06T14:17:22.488 INFO:tasks.workunit.client.1.smithi086.stdout:8/902: rename d0/d71/d22/d43/d6d/d75/db2/fca to d0/d71/f139 0 2024-04-06T14:17:22.491 INFO:tasks.workunit.client.1.smithi086.stdout:8/903: read d0/d3b/fa6 [1321729,118095] 0 2024-04-06T14:17:22.491 INFO:tasks.workunit.client.1.smithi086.stdout:7/824: truncate d7/d10/d11/d1c/f1e 1261485 0 2024-04-06T14:17:22.491 INFO:tasks.workunit.client.1.smithi086.stdout:7/825: readlink d7/le8 0 2024-04-06T14:17:22.492 INFO:tasks.workunit.client.1.smithi086.stdout:8/904: chown d0/d71/d16/ff1 495549829 1 2024-04-06T14:17:22.496 INFO:tasks.workunit.client.1.smithi086.stdout:8/905: fsync d0/d71/d22/d43/d6d/d75/d78/fb0 0 2024-04-06T14:17:22.498 INFO:tasks.workunit.client.1.smithi086.stdout:1/895: dwrite db/d13/d55/f6a [4194304,4194304] 0 2024-04-06T14:17:22.501 INFO:tasks.workunit.client.1.smithi086.stdout:2/912: truncate d0/d20/d27/d2b/faf 343907 0 2024-04-06T14:17:22.502 INFO:tasks.workunit.client.1.smithi086.stdout:8/906: rename d0/d71/d16/d39/d90/daf/fe3 to d0/d71/d93/ded/f13a 0 2024-04-06T14:17:22.503 INFO:tasks.workunit.client.1.smithi086.stdout:1/896: creat db/d11/d1a/d2a/d9a/d68/d6e/f11e x:0 0 0 2024-04-06T14:17:22.505 INFO:tasks.workunit.client.1.smithi086.stdout:2/913: stat d0/d20/d27/d37/c38 0 2024-04-06T14:17:22.506 INFO:tasks.workunit.client.1.smithi086.stdout:8/907: write d0/d3b/d79/fc5 [728297,55442] 0 2024-04-06T14:17:22.507 INFO:tasks.workunit.client.1.smithi086.stdout:1/897: chown db/d11/d87/ca2 42 1 2024-04-06T14:17:22.509 INFO:tasks.workunit.client.1.smithi086.stdout:8/908: truncate d0/d71/d22/d43/d6d/d75/dee/ffb 1003223 0 2024-04-06T14:17:22.510 INFO:tasks.workunit.client.1.smithi086.stdout:1/898: unlink db/d11/d1f/d5c/l77 0 2024-04-06T14:17:22.510 INFO:tasks.workunit.client.1.smithi086.stdout:8/909: stat d0/d13/fa9 0 2024-04-06T14:17:22.510 INFO:tasks.workunit.client.1.smithi086.stdout:2/914: dread d0/f136 [0,4194304] 0 2024-04-06T14:17:22.511 INFO:tasks.workunit.client.1.smithi086.stdout:3/864: mknod d5/dea/dd2/ddb/de6/c115 0 2024-04-06T14:17:22.513 INFO:tasks.workunit.client.1.smithi086.stdout:2/915: creat d0/df/d5e/d9d/f144 x:0 0 0 2024-04-06T14:17:22.515 INFO:tasks.workunit.client.1.smithi086.stdout:3/865: fdatasync d5/d33/d5d/f6f 0 2024-04-06T14:17:22.516 INFO:tasks.workunit.client.1.smithi086.stdout:6/991: getdents d6/d13/d1a/da0/d11b/d45/d8a 0 2024-04-06T14:17:22.521 INFO:tasks.workunit.client.1.smithi086.stdout:2/916: rename d0/df/d5e/d9d/l13c to d0/d20/d80/l145 0 2024-04-06T14:17:22.522 INFO:tasks.workunit.client.1.smithi086.stdout:3/866: truncate d5/dea/dd2/ddb/de6/f2c 5237079 0 2024-04-06T14:17:22.528 INFO:tasks.workunit.client.1.smithi086.stdout:3/867: link d5/d33/f47 d5/d36/d5a/d6d/d72/d108/f116 0 2024-04-06T14:17:22.534 INFO:tasks.workunit.client.1.smithi086.stdout:1/899: write db/d11/d42/f50 [3653490,44344] 0 2024-04-06T14:17:22.534 INFO:tasks.workunit.client.1.smithi086.stdout:7/826: dwrite d7/d10/d11/d1c/f79 [0,4194304] 0 2024-04-06T14:17:22.535 INFO:tasks.workunit.client.1.smithi086.stdout:8/910: write d0/f72 [2632102,112456] 0 2024-04-06T14:17:22.537 INFO:tasks.workunit.client.1.smithi086.stdout:3/868: link d5/d36/d56/dd3/fe0 d5/dea/dd2/ddb/de6/d9e/db5/f117 0 2024-04-06T14:17:22.538 INFO:tasks.workunit.client.1.smithi086.stdout:6/992: dread d6/d13/d1a/da0/d11b/d45/f123 [0,4194304] 0 2024-04-06T14:17:22.539 INFO:tasks.workunit.client.1.smithi086.stdout:3/869: dread - d5/dea/dd2/ddb/de6/d10b/fdd zero size 2024-04-06T14:17:22.541 INFO:tasks.workunit.client.1.smithi086.stdout:2/917: truncate d0/df/d5e/fc7 2194702 0 2024-04-06T14:17:22.543 INFO:tasks.workunit.client.1.smithi086.stdout:6/993: fdatasync d6/d13/d42/d52/f90 0 2024-04-06T14:17:22.543 INFO:tasks.workunit.client.1.smithi086.stdout:1/900: chown db/d13/f74 442 1 2024-04-06T14:17:22.543 INFO:tasks.workunit.client.1.smithi086.stdout:3/870: chown d5/dea/dd2/ddb/de6/d9e/da5/fad 201821 1 2024-04-06T14:17:22.545 INFO:tasks.workunit.client.1.smithi086.stdout:3/871: chown d5/d36/d5a/d6d/d72/d9a 13269993 1 2024-04-06T14:17:22.547 INFO:tasks.workunit.client.1.smithi086.stdout:8/911: creat d0/d71/d16/f13b x:0 0 0 2024-04-06T14:17:22.551 INFO:tasks.workunit.client.1.smithi086.stdout:3/872: creat d5/d26/d7f/dae/f118 x:0 0 0 2024-04-06T14:17:22.555 INFO:tasks.workunit.client.1.smithi086.stdout:2/918: fdatasync d0/d20/d27/d2b/faf 0 2024-04-06T14:17:22.557 INFO:tasks.workunit.client.1.smithi086.stdout:3/873: mknod d5/d36/d5a/d6d/c119 0 2024-04-06T14:17:22.558 INFO:tasks.workunit.client.1.smithi086.stdout:8/912: truncate d0/d71/d16/d52/fc8 468274 0 2024-04-06T14:17:22.559 INFO:tasks.workunit.client.1.smithi086.stdout:7/827: dread d7/d94/fa9 [0,4194304] 0 2024-04-06T14:17:22.559 INFO:tasks.workunit.client.1.smithi086.stdout:3/874: chown d5/d36/d65/c6e 79 1 2024-04-06T14:17:22.562 INFO:tasks.workunit.client.1.smithi086.stdout:6/994: dwrite d6/f3c [4194304,4194304] 0 2024-04-06T14:17:22.563 INFO:tasks.workunit.client.1.smithi086.stdout:7/828: write d7/d10/d11/d21/fe2 [895726,79297] 0 2024-04-06T14:17:22.563 INFO:tasks.workunit.client.1.smithi086.stdout:8/913: symlink d0/d71/d22/d43/d6d/d75/l13c 0 2024-04-06T14:17:22.569 INFO:tasks.workunit.client.1.smithi086.stdout:3/875: write d5/d36/f3b [469288,64612] 0 2024-04-06T14:17:22.569 INFO:tasks.workunit.client.1.smithi086.stdout:2/919: dwrite d0/d20/d27/d2b/d35/d36/f40 [0,4194304] 0 2024-04-06T14:17:22.570 INFO:tasks.workunit.client.1.smithi086.stdout:8/914: dread d0/d71/d16/d52/dd0/fb4 [0,4194304] 0 2024-04-06T14:17:22.571 INFO:tasks.workunit.client.1.smithi086.stdout:3/876: write d5/d36/f3b [7522180,69953] 0 2024-04-06T14:17:22.573 INFO:tasks.workunit.client.1.smithi086.stdout:2/920: write d0/d20/d27/d2b/d35/d36/d4a/f137 [267252,104622] 0 2024-04-06T14:17:22.575 INFO:tasks.workunit.client.1.smithi086.stdout:8/915: read d0/d71/d22/d43/f47 [1561754,41962] 0 2024-04-06T14:17:22.575 INFO:tasks.workunit.client.1.smithi086.stdout:6/995: creat d6/d13/d1a/da0/d11b/d45/d4a/d13b/f159 x:0 0 0 2024-04-06T14:17:22.582 INFO:tasks.workunit.client.1.smithi086.stdout:3/877: mkdir d5/d11a 0 2024-04-06T14:17:22.582 INFO:tasks.workunit.client.1.smithi086.stdout:2/921: mknod d0/d20/d27/d2b/d35/d36/d4a/d111/c146 0 2024-04-06T14:17:22.585 INFO:tasks.workunit.client.1.smithi086.stdout:7/829: rename d7/d10/d11/d80 to d7/d10/d11/d123 0 2024-04-06T14:17:22.587 INFO:tasks.workunit.client.1.smithi086.stdout:8/916: fdatasync d0/d71/d16/d39/f132 0 2024-04-06T14:17:22.588 INFO:tasks.workunit.client.1.smithi086.stdout:6/996: mkdir d6/d13/d42/d52/df1/d135/d15a 0 2024-04-06T14:17:22.589 INFO:tasks.workunit.client.1.smithi086.stdout:3/878: mkdir d5/dea/db4/da7/d11b 0 2024-04-06T14:17:22.591 INFO:tasks.workunit.client.1.smithi086.stdout:2/922: rename d0/d20/d27/d2b/d35/d36/d4a/d11d/d13b to d0/d20/d44/d120/d147 0 2024-04-06T14:17:22.591 INFO:tasks.workunit.client.1.smithi086.stdout:2/923: readlink d0/d20/d27/la0 0 2024-04-06T14:17:22.592 INFO:tasks.workunit.client.1.smithi086.stdout:7/830: creat d7/d1d/f124 x:0 0 0 2024-04-06T14:17:22.592 INFO:tasks.workunit.client.1.smithi086.stdout:3/879: dread d5/f50 [0,4194304] 0 2024-04-06T14:17:22.594 INFO:tasks.workunit.client.1.smithi086.stdout:3/880: write d5/d26/d7f/dae/f118 [576132,120311] 0 2024-04-06T14:17:22.595 INFO:tasks.workunit.client.1.smithi086.stdout:6/997: mkdir d6/d13/d1a/d64/d15b 0 2024-04-06T14:17:22.596 INFO:tasks.workunit.client.1.smithi086.stdout:8/917: symlink d0/d71/d16/d39/d90/daf/df0/l13d 0 2024-04-06T14:17:22.600 INFO:tasks.workunit.client.1.smithi086.stdout:2/924: dwrite d0/df/f11e [0,4194304] 0 2024-04-06T14:17:22.601 INFO:tasks.workunit.client.1.smithi086.stdout:3/881: unlink d5/l38 0 2024-04-06T14:17:22.612 INFO:tasks.workunit.client.1.smithi086.stdout:6/998: creat d6/d13/d1a/da0/d11b/d45/d8a/de3/f15c x:0 0 0 2024-04-06T14:17:22.612 INFO:tasks.workunit.client.1.smithi086.stdout:7/831: rename d7/d10/d102/fb7 to d7/d1d/f125 0 2024-04-06T14:17:22.614 INFO:tasks.workunit.client.1.smithi086.stdout:3/882: fdatasync d5/dea/db4/da7/d89/fbf 0 2024-04-06T14:17:22.616 INFO:tasks.workunit.client.1.smithi086.stdout:6/999: rename d6/d9/d43 to d6/d13/d5f/d15d 0 2024-04-06T14:17:22.619 INFO:tasks.workunit.client.0.smithi032.stdout:5/995: sync 2024-04-06T14:17:22.619 INFO:tasks.workunit.client.0.smithi032.stdout:0/888: sync 2024-04-06T14:17:22.619 INFO:tasks.workunit.client.1.smithi086.stdout:0/874: sync 2024-04-06T14:17:22.619 INFO:tasks.workunit.client.1.smithi086.stdout:4/726: sync 2024-04-06T14:17:22.619 INFO:tasks.workunit.client.1.smithi086.stdout:5/732: sync 2024-04-06T14:17:22.620 INFO:tasks.workunit.client.1.smithi086.stdout:8/918: dread d0/d71/d16/f6c [0,4194304] 0 2024-04-06T14:17:22.620 INFO:tasks.workunit.client.1.smithi086.stdout:3/883: mknod d5/d36/d5c/d8e/dc1/c11c 0 2024-04-06T14:17:22.621 INFO:tasks.workunit.client.1.smithi086.stdout:2/925: link d0/df/d5e/d9d/lb0 d0/d20/d80/dbf/ddc/l148 0 2024-04-06T14:17:22.621 INFO:tasks.workunit.client.1.smithi086.stdout:0/875: fsync d0/d15/d4e/da3/d44/fbc 0 2024-04-06T14:17:22.622 INFO:tasks.workunit.client.1.smithi086.stdout:4/727: chown d0/d2c/d38/d54/d61/d7a 289 1 2024-04-06T14:17:22.623 INFO:tasks.workunit.client.1.smithi086.stdout:5/733: dread - d0/d16/d18/d21/d23/d28/d94/fae zero size 2024-04-06T14:17:22.623 INFO:tasks.workunit.client.0.smithi032.stdout:0/889: dread d3/f45 [0,4194304] 0 2024-04-06T14:17:22.623 INFO:tasks.workunit.client.1.smithi086.stdout:8/919: write d0/d13/f21 [2327368,25296] 0 2024-04-06T14:17:22.624 INFO:tasks.workunit.client.1.smithi086.stdout:4/728: fsync d0/f10 0 2024-04-06T14:17:22.624 INFO:tasks.workunit.client.0.smithi032.stdout:0/890: write d3/d83/fc5 [1832135,25736] 0 2024-04-06T14:17:22.625 INFO:tasks.workunit.client.1.smithi086.stdout:7/832: link d7/d10/d11/d1c/d24/c9d d7/d10/d102/d5c/d66/c126 0 2024-04-06T14:17:22.629 INFO:tasks.workunit.client.1.smithi086.stdout:5/734: rmdir d0/d16/d18/d35/d9e/de7 39 2024-04-06T14:17:22.630 INFO:tasks.workunit.client.1.smithi086.stdout:8/920: dread d0/d13/db6/f10e [0,4194304] 0 2024-04-06T14:17:22.630 INFO:tasks.workunit.client.1.smithi086.stdout:3/884: creat d5/d36/d5a/d6d/d72/d97/f11d x:0 0 0 2024-04-06T14:17:22.632 INFO:tasks.workunit.client.1.smithi086.stdout:0/876: rename d0/d15/d26/d33/d7a/daa to d0/d15/d4e/da3/d44/d5d/d74/d120 0 2024-04-06T14:17:22.633 INFO:tasks.workunit.client.1.smithi086.stdout:3/885: write d5/f77 [3935556,35854] 0 2024-04-06T14:17:22.634 INFO:tasks.workunit.client.1.smithi086.stdout:8/921: readlink d0/d71/d82/lc0 0 2024-04-06T14:17:22.635 INFO:tasks.workunit.client.1.smithi086.stdout:5/735: dread - d0/d16/d18/d21/d23/d28/d94/d8a/fbd zero size 2024-04-06T14:17:22.636 INFO:tasks.workunit.client.1.smithi086.stdout:2/926: dwrite d0/d20/d44/dbb/dc4/de3/d76/f9b [0,4194304] 0 2024-04-06T14:17:22.638 INFO:tasks.workunit.client.1.smithi086.stdout:7/833: rename d7/d10/d11/d21/f122 to d7/d10/f127 0 2024-04-06T14:17:22.640 INFO:tasks.workunit.client.1.smithi086.stdout:0/877: unlink d0/d15/d4e/da3/d73/cc6 0 2024-04-06T14:17:22.641 INFO:tasks.workunit.client.1.smithi086.stdout:0/878: readlink d0/d15/d4e/da3/d46/d52/l6c 0 2024-04-06T14:17:22.643 INFO:tasks.workunit.client.1.smithi086.stdout:2/927: creat d0/d20/d80/dbf/ddc/f149 x:0 0 0 2024-04-06T14:17:22.645 INFO:tasks.workunit.client.1.smithi086.stdout:3/886: truncate d5/d33/f47 3699912 0 2024-04-06T14:17:22.645 INFO:tasks.workunit.client.1.smithi086.stdout:8/922: link d0/d71/d22/d43/daa/fe8 d0/d71/d16/d52/dd0/ddc/de7/d136/f13e 0 2024-04-06T14:17:22.650 INFO:tasks.workunit.client.1.smithi086.stdout:8/923: chown d0/d71/d22/d43/d6d/ff3 182 1 2024-04-06T14:17:22.652 INFO:tasks.workunit.client.1.smithi086.stdout:0/879: creat d0/d15/d4e/da3/f121 x:0 0 0 2024-04-06T14:17:22.655 INFO:tasks.workunit.client.1.smithi086.stdout:0/880: write d0/d15/d4e/da3/d46/f6d [1234106,7382] 0 2024-04-06T14:17:22.662 INFO:tasks.workunit.client.1.smithi086.stdout:5/736: dread d0/d16/d18/fec [0,4194304] 0 2024-04-06T14:17:22.662 INFO:tasks.workunit.client.1.smithi086.stdout:0/881: mknod d0/d15/d4e/da3/d46/d52/c122 0 2024-04-06T14:17:22.666 INFO:tasks.workunit.client.1.smithi086.stdout:7/834: dwrite d7/d10/d11/d1c/d24/d2c/faa [4194304,4194304] 0 2024-04-06T14:17:22.666 INFO:tasks.workunit.client.1.smithi086.stdout:5/737: write d0/d16/f95 [1110406,34434] 0 2024-04-06T14:17:22.666 INFO:tasks.workunit.client.1.smithi086.stdout:2/928: dread d0/df/d5e/d9d/fff [0,4194304] 0 2024-04-06T14:17:22.668 INFO:tasks.workunit.client.0.smithi032.stdout:5/996: dread d16/d18/d33/d45/d52/d39/fb7 [0,4194304] 0 2024-04-06T14:17:22.671 INFO:tasks.workunit.client.1.smithi086.stdout:0/882: creat d0/d15/d4e/da3/d73/f123 x:0 0 0 2024-04-06T14:17:22.673 INFO:tasks.workunit.client.1.smithi086.stdout:5/738: write d0/d16/f39 [1721121,115971] 0 2024-04-06T14:17:22.674 INFO:tasks.workunit.client.0.smithi032.stdout:5/997: creat d16/d18/d33/d45/d87/d10c/f156 x:0 0 0 2024-04-06T14:17:22.677 INFO:tasks.workunit.client.1.smithi086.stdout:5/739: stat d0/d16/d71/d89 0 2024-04-06T14:17:22.677 INFO:tasks.workunit.client.1.smithi086.stdout:0/883: mknod d0/d15/d4e/c124 0 2024-04-06T14:17:22.679 INFO:tasks.workunit.client.0.smithi032.stdout:5/998: rmdir d16/d18/d33/dea/d155 0 2024-04-06T14:17:22.679 INFO:tasks.workunit.client.1.smithi086.stdout:3/887: dwrite d5/d26/d7f/f4c [0,4194304] 0 2024-04-06T14:17:22.680 INFO:tasks.workunit.client.1.smithi086.stdout:8/924: dwrite d0/d71/d22/f8e [0,4194304] 0 2024-04-06T14:17:22.683 INFO:tasks.workunit.client.1.smithi086.stdout:2/929: write d0/d20/d27/d2b/d35/d36/f105 [152393,86608] 0 2024-04-06T14:17:22.692 INFO:tasks.workunit.client.1.smithi086.stdout:5/740: write d0/d16/d18/d2a/d50/f7b [5228119,63900] 0 2024-04-06T14:17:22.692 INFO:tasks.workunit.client.1.smithi086.stdout:8/925: creat d0/d71/d82/f13f x:0 0 0 2024-04-06T14:17:22.692 INFO:tasks.workunit.client.1.smithi086.stdout:0/884: dwrite d0/d1/d4/f43 [4194304,4194304] 0 2024-04-06T14:17:22.696 INFO:tasks.workunit.client.1.smithi086.stdout:3/888: rename d5/d36/d5a/d6d/d72/d97/db0 to d5/d26/d7f/dae/d11e 0 2024-04-06T14:17:22.697 INFO:tasks.workunit.client.1.smithi086.stdout:2/930: dread d0/d20/d27/d2b/d35/d36/f55 [0,4194304] 0 2024-04-06T14:17:22.700 INFO:tasks.workunit.client.1.smithi086.stdout:5/741: mkdir d0/d16/d18/d21/d23/d28/df4 0 2024-04-06T14:17:22.701 INFO:tasks.workunit.client.1.smithi086.stdout:3/889: stat d5/d36/d5a/d6d/d72/d57/ffe 0 2024-04-06T14:17:22.703 INFO:tasks.workunit.client.1.smithi086.stdout:2/931: symlink d0/l14a 0 2024-04-06T14:17:22.704 INFO:tasks.workunit.client.1.smithi086.stdout:3/890: mknod d5/dea/dd2/ddb/c11f 0 2024-04-06T14:17:22.705 INFO:tasks.workunit.client.1.smithi086.stdout:5/742: read d0/d43/de5/f3c [2240161,81088] 0 2024-04-06T14:17:22.707 INFO:tasks.workunit.client.1.smithi086.stdout:0/885: dwrite d0/d1/d4/de/db4/fc9 [0,4194304] 0 2024-04-06T14:17:22.708 INFO:tasks.workunit.client.1.smithi086.stdout:2/932: mknod d0/df/dd2/d10e/c14b 0 2024-04-06T14:17:22.709 INFO:tasks.workunit.client.1.smithi086.stdout:7/835: truncate d7/d1d/f13 3222033 0 2024-04-06T14:17:22.710 INFO:tasks.workunit.client.1.smithi086.stdout:5/743: rmdir d0/d16/d18/d2a 39 2024-04-06T14:17:22.711 INFO:tasks.workunit.client.1.smithi086.stdout:5/744: readlink d0/d16/d18/d35/lef 0 2024-04-06T14:17:22.714 INFO:tasks.workunit.client.1.smithi086.stdout:9/808: sync 2024-04-06T14:17:22.715 INFO:tasks.workunit.client.1.smithi086.stdout:8/926: write d0/d71/d16/d39/fec [170048,30732] 0 2024-04-06T14:17:22.716 INFO:tasks.workunit.client.0.smithi032.stdout:0/891: write d3/da/d25/d57/db5/dc9/f11f [4536561,15412] 0 2024-04-06T14:17:22.719 INFO:tasks.workunit.client.0.smithi032.stdout:5/999: write d16/d18/d33/f34 [2984647,91489] 0 2024-04-06T14:17:22.720 INFO:tasks.workunit.client.0.smithi032.stdout:0/892: creat d3/da/d41/d107/d125/f12e x:0 0 0 2024-04-06T14:17:22.721 INFO:tasks.workunit.client.0.smithi032.stdout:0/893: chown d3/dc2/d123 105585 1 2024-04-06T14:17:22.724 INFO:tasks.workunit.client.0.smithi032.stdout:0/894: truncate d3/da/d25/d36/f9b 1122984 0 2024-04-06T14:17:22.725 INFO:tasks.workunit.client.1.smithi086.stdout:5/745: getdents d0/d16/d18/d21 0 2024-04-06T14:17:22.725 INFO:tasks.workunit.client.1.smithi086.stdout:3/891: truncate d5/d36/d5a/fd4 2845851 0 2024-04-06T14:17:22.725 INFO:tasks.workunit.client.0.smithi032.stdout:0/895: fsync d3/da/d25/f54 0 2024-04-06T14:17:22.726 INFO:tasks.workunit.client.1.smithi086.stdout:9/809: symlink d5/d13/d1a/d5f/d88/d14/d19/d25/l106 0 2024-04-06T14:17:22.726 INFO:tasks.workunit.client.0.smithi032.stdout:0/896: dread - d3/da/f6a zero size 2024-04-06T14:17:22.727 INFO:tasks.workunit.client.1.smithi086.stdout:7/836: link d7/d1d/c5e d7/d10/d11/d1c/d24/d2c/d69/c128 0 2024-04-06T14:17:22.727 INFO:tasks.workunit.client.1.smithi086.stdout:8/927: getdents d0/d71/d16/d52/dd0/d134 0 2024-04-06T14:17:22.728 INFO:tasks.workunit.client.1.smithi086.stdout:9/810: dread - d5/d13/d47/d55/fca zero size 2024-04-06T14:17:22.728 INFO:tasks.workunit.client.0.smithi032.stdout:0/897: creat d3/dc2/d123/f12f x:0 0 0 2024-04-06T14:17:22.730 INFO:tasks.workunit.client.1.smithi086.stdout:0/886: dwrite d0/d1/d4/f8 [0,4194304] 0 2024-04-06T14:17:22.731 INFO:tasks.workunit.client.1.smithi086.stdout:8/928: mkdir d0/d71/d22/d43/d6d/d140 0 2024-04-06T14:17:22.732 INFO:tasks.workunit.client.0.smithi032.stdout:0/898: truncate d3/da/d25/d36/f51 1293896 0 2024-04-06T14:17:22.732 INFO:tasks.workunit.client.1.smithi086.stdout:9/811: chown d5/d13/d1a/d5f/d88/db3/fe3 129115 1 2024-04-06T14:17:22.733 INFO:tasks.workunit.client.1.smithi086.stdout:0/887: chown d0/d15/d4e/da3/d44/d5d/d74/ca6 169888 1 2024-04-06T14:17:22.734 INFO:tasks.workunit.client.1.smithi086.stdout:5/746: rename d0/d16/d18/d35/d9e/de7/lde to d0/d16/d18/de6/lf5 0 2024-04-06T14:17:22.736 INFO:tasks.workunit.client.1.smithi086.stdout:9/812: write d5/d58/fbe [54513,79568] 0 2024-04-06T14:17:22.738 INFO:tasks.workunit.client.1.smithi086.stdout:7/837: getdents d7/d10/d92 0 2024-04-06T14:17:22.739 INFO:tasks.workunit.client.1.smithi086.stdout:0/888: symlink d0/l125 0 2024-04-06T14:17:22.744 INFO:tasks.workunit.client.0.smithi032.stdout:0/899: dread d3/da/faa [0,4194304] 0 2024-04-06T14:17:22.744 INFO:tasks.workunit.client.1.smithi086.stdout:5/747: read d0/d16/d18/d2a/d50/d66/f7d [172321,109479] 0 2024-04-06T14:17:22.746 INFO:tasks.workunit.client.1.smithi086.stdout:3/892: dread d5/dea/dd2/ddb/de6/d10b/fc5 [0,4194304] 0 2024-04-06T14:17:22.750 INFO:tasks.workunit.client.1.smithi086.stdout:3/893: creat d5/d36/d65/f120 x:0 0 0 2024-04-06T14:17:22.751 INFO:tasks.workunit.client.0.smithi032.stdout:0/900: getdents d3/da/d25 0 2024-04-06T14:17:22.752 INFO:tasks.workunit.client.0.smithi032.stdout:0/901: dread - d3/da/d41/d107/d125/f12e zero size 2024-04-06T14:17:22.753 INFO:tasks.workunit.client.1.smithi086.stdout:0/889: write d0/d15/d4e/d48/d89/fba [124951,97600] 0 2024-04-06T14:17:22.756 INFO:tasks.workunit.client.0.smithi032.stdout:0/902: dread d3/da/d25/d2b/d4a/d79/d97/de0/df5/f8a [0,4194304] 0 2024-04-06T14:17:22.758 INFO:tasks.workunit.client.0.smithi032.stdout:0/903: dread - d3/da/d25/d2b/d4a/d79/d97/de0/df5/f117 zero size 2024-04-06T14:17:22.759 INFO:tasks.workunit.client.1.smithi086.stdout:0/890: creat d0/d15/d4e/da3/d46/f126 x:0 0 0 2024-04-06T14:17:22.761 INFO:tasks.workunit.client.1.smithi086.stdout:7/838: dread d7/d1d/f3e [0,4194304] 0 2024-04-06T14:17:22.762 INFO:tasks.workunit.client.1.smithi086.stdout:8/929: dwrite d0/d71/f3c [0,4194304] 0 2024-04-06T14:17:22.764 INFO:tasks.workunit.client.1.smithi086.stdout:9/813: dwrite f0 [4194304,4194304] 0 2024-04-06T14:17:22.764 INFO:tasks.workunit.client.0.smithi032.stdout:0/904: dwrite d3/dc2/dd8/ff2 [4194304,4194304] 0 2024-04-06T14:17:22.766 INFO:tasks.workunit.client.0.smithi032.stdout:0/905: read d3/da/d25/f2a [2045870,42713] 0 2024-04-06T14:17:22.767 INFO:tasks.workunit.client.0.smithi032.stdout:0/906: fdatasync d3/d83/ff6 0 2024-04-06T14:17:22.768 INFO:tasks.workunit.client.1.smithi086.stdout:7/839: chown d7/d10/d11/d1c/d24/fe1 0 1 2024-04-06T14:17:22.769 INFO:tasks.workunit.client.1.smithi086.stdout:5/748: dread d0/d16/d18/f9d [0,4194304] 0 2024-04-06T14:17:22.770 INFO:tasks.workunit.client.1.smithi086.stdout:8/930: stat d0/d71/d22/d43/d6d/d75/ca7 0 2024-04-06T14:17:22.773 INFO:tasks.workunit.client.1.smithi086.stdout:3/894: truncate d5/d26/d7f/dae/fcd 4041598 0 2024-04-06T14:17:22.774 INFO:tasks.workunit.client.1.smithi086.stdout:9/814: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dce/f107 x:0 0 0 2024-04-06T14:17:22.777 INFO:tasks.workunit.client.1.smithi086.stdout:9/815: write d5/d13/d1a/d5f/d88/d14/d19/d25/fcb [5162357,103577] 0 2024-04-06T14:17:22.777 INFO:tasks.workunit.client.1.smithi086.stdout:3/895: readlink d5/d26/d7f/l53 0 2024-04-06T14:17:22.778 INFO:tasks.workunit.client.1.smithi086.stdout:0/891: getdents d0/d1/d2/d1a/d79/dac/dc2 0 2024-04-06T14:17:22.780 INFO:tasks.workunit.client.1.smithi086.stdout:9/816: stat d5/d13/d1a/d5f/d88/d14/d19/d25/l5c 0 2024-04-06T14:17:22.784 INFO:tasks.workunit.client.1.smithi086.stdout:9/817: write d5/d13/d1a/d5f/d88/d14/f86 [3252296,79866] 0 2024-04-06T14:17:22.785 INFO:tasks.workunit.client.1.smithi086.stdout:0/892: mknod d0/d15/d26/d33/d9f/c127 0 2024-04-06T14:17:22.786 INFO:tasks.workunit.client.1.smithi086.stdout:3/896: dread d5/d26/d7f/f4c [0,4194304] 0 2024-04-06T14:17:22.788 INFO:tasks.workunit.client.1.smithi086.stdout:9/818: mkdir d5/d13/d1a/d5f/d108 0 2024-04-06T14:17:22.790 INFO:tasks.workunit.client.1.smithi086.stdout:0/893: mkdir d0/d15/d4e/da3/d128 0 2024-04-06T14:17:22.791 INFO:tasks.workunit.client.1.smithi086.stdout:3/897: dread d5/d33/f78 [0,4194304] 0 2024-04-06T14:17:22.792 INFO:tasks.workunit.client.1.smithi086.stdout:9/819: chown d5/d13/d1a/d5f/d88/d14/d19/d1b/c56 207651506 1 2024-04-06T14:17:22.792 INFO:tasks.workunit.client.1.smithi086.stdout:3/898: stat f4 0 2024-04-06T14:17:22.792 INFO:tasks.workunit.client.1.smithi086.stdout:0/894: dread - d0/d1/fa5 zero size 2024-04-06T14:17:22.796 INFO:tasks.workunit.client.1.smithi086.stdout:9/820: dread f0 [4194304,4194304] 0 2024-04-06T14:17:22.796 INFO:tasks.workunit.client.1.smithi086.stdout:0/895: mkdir d0/d1/d2/d1a/d79/dac/db7/de6/d129 0 2024-04-06T14:17:22.797 INFO:tasks.workunit.client.1.smithi086.stdout:3/899: symlink d5/d36/d5a/d6d/d72/d97/l121 0 2024-04-06T14:17:22.797 INFO:tasks.workunit.client.1.smithi086.stdout:0/896: dread - d0/d1/d2/d5b/fed zero size 2024-04-06T14:17:22.799 INFO:tasks.workunit.client.1.smithi086.stdout:3/900: write d5/d36/d56/f105 [591394,123450] 0 2024-04-06T14:17:22.802 INFO:tasks.workunit.client.1.smithi086.stdout:7/840: truncate d7/d10/d11/f8f 1639783 0 2024-04-06T14:17:22.803 INFO:tasks.workunit.client.1.smithi086.stdout:8/931: write d0/d13/db6/fba [890096,67928] 0 2024-04-06T14:17:22.804 INFO:tasks.workunit.client.0.smithi032.stdout:0/907: dread d3/da/d25/d2b/d4a/f62 [0,4194304] 0 2024-04-06T14:17:22.808 INFO:tasks.workunit.client.1.smithi086.stdout:9/821: truncate d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f81 604837 0 2024-04-06T14:17:22.808 INFO:tasks.workunit.client.1.smithi086.stdout:5/749: dread d0/d16/f4e [0,4194304] 0 2024-04-06T14:17:22.808 INFO:tasks.workunit.client.0.smithi032.stdout:0/908: mkdir d3/da/d25/d2b/d4a/d130 0 2024-04-06T14:17:22.810 INFO:tasks.workunit.client.1.smithi086.stdout:7/841: truncate d7/d10/d102/d5c/d66/fa3 1376611 0 2024-04-06T14:17:22.811 INFO:tasks.workunit.client.1.smithi086.stdout:5/750: fdatasync d0/d16/d18/d2a/d50/d66/fbe 0 2024-04-06T14:17:22.812 INFO:tasks.workunit.client.1.smithi086.stdout:9/822: write d5/d13/d1a/d5f/d88/d14/d19/d25/d74/fae [758164,24453] 0 2024-04-06T14:17:22.812 INFO:tasks.workunit.client.1.smithi086.stdout:0/897: write d0/d15/d4e/f82 [1220098,64237] 0 2024-04-06T14:17:22.813 INFO:tasks.workunit.client.1.smithi086.stdout:9/823: stat d5/d13/d1a/d5f/d88/d14/d19 0 2024-04-06T14:17:22.817 INFO:tasks.workunit.client.1.smithi086.stdout:3/901: write d5/d36/d5a/fd4 [681635,78018] 0 2024-04-06T14:17:22.817 INFO:tasks.workunit.client.1.smithi086.stdout:3/902: stat d5/d36/d65/l6c 0 2024-04-06T14:17:22.818 INFO:tasks.workunit.client.1.smithi086.stdout:0/898: stat d0/d15/d4e/da3/d44/d5d/ce9 0 2024-04-06T14:17:22.820 INFO:tasks.workunit.client.1.smithi086.stdout:7/842: mknod d7/d10/d11/d1c/d24/c129 0 2024-04-06T14:17:22.821 INFO:tasks.workunit.client.1.smithi086.stdout:8/932: write d0/d71/d22/d65/d6e/d98/fc2 [411439,86992] 0 2024-04-06T14:17:22.821 INFO:tasks.workunit.client.0.smithi032.stdout:0/909: dwrite d3/f63 [0,4194304] 0 2024-04-06T14:17:22.823 INFO:tasks.workunit.client.0.smithi032.stdout:0/910: fdatasync d3/da/d25/d57/fd7 0 2024-04-06T14:17:22.823 INFO:tasks.workunit.client.1.smithi086.stdout:9/824: rename d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/ffb to d5/d13/d47/d55/d87/f109 0 2024-04-06T14:17:22.826 INFO:tasks.workunit.client.1.smithi086.stdout:3/903: fdatasync d5/d33/fd0 0 2024-04-06T14:17:22.829 INFO:tasks.workunit.client.1.smithi086.stdout:7/843: creat d7/d10/d11/d1c/d24/d2c/d69/f12a x:0 0 0 2024-04-06T14:17:22.830 INFO:tasks.workunit.client.1.smithi086.stdout:8/933: mkdir d0/d71/d22/d43/d6d/d75/dee/d141 0 2024-04-06T14:17:22.832 INFO:tasks.workunit.client.1.smithi086.stdout:9/825: unlink d5/d13/d1a/l38 0 2024-04-06T14:17:22.836 INFO:tasks.workunit.client.1.smithi086.stdout:5/751: dwrite d0/d16/f63 [0,4194304] 0 2024-04-06T14:17:22.838 INFO:tasks.workunit.client.1.smithi086.stdout:8/934: creat d0/d71/d22/d43/d4d/d62/f142 x:0 0 0 2024-04-06T14:17:22.841 INFO:tasks.workunit.client.1.smithi086.stdout:9/826: mknod d5/d13/d1a/d5f/d88/d14/d19/d25/d74/c10a 0 2024-04-06T14:17:22.841 INFO:tasks.workunit.client.1.smithi086.stdout:0/899: dread d0/d1/d2/d1a/d79/dac/f112 [0,4194304] 0 2024-04-06T14:17:22.842 INFO:tasks.workunit.client.1.smithi086.stdout:0/900: readlink d0/d15/d4e/l7e 0 2024-04-06T14:17:22.842 INFO:tasks.workunit.client.1.smithi086.stdout:9/827: dread - d5/d13/d1a/d5f/d88/d91/fed zero size 2024-04-06T14:17:22.843 INFO:tasks.workunit.client.1.smithi086.stdout:5/752: rename d0/d16/d18/d21/d23/d28/d94/fae to d0/d43/de5/ff6 0 2024-04-06T14:17:22.844 INFO:tasks.workunit.client.0.smithi032.stdout:0/911: dwrite d3/da/d25/d2b/d4a/d79/da8/f9f [0,4194304] 0 2024-04-06T14:17:22.846 INFO:tasks.workunit.client.1.smithi086.stdout:7/844: rename d7/d10/d11/d1c/d24/d2c/d69/c128 to d7/d10/d11/d1c/d24/d2c/d69/c12b 0 2024-04-06T14:17:22.846 INFO:tasks.workunit.client.1.smithi086.stdout:5/753: write d0/d16/d18/d21/d23/d28/d94/d8a/fbd [913191,13188] 0 2024-04-06T14:17:22.847 INFO:tasks.workunit.client.1.smithi086.stdout:7/845: stat d7/d10/d102/daf/db5/d7a 0 2024-04-06T14:17:22.848 INFO:tasks.workunit.client.1.smithi086.stdout:3/904: creat d5/d36/d5a/d6d/d72/f122 x:0 0 0 2024-04-06T14:17:22.850 INFO:tasks.workunit.client.0.smithi032.stdout:0/912: creat d3/da/d25/d2b/d80/db3/df1/d120/f131 x:0 0 0 2024-04-06T14:17:22.850 INFO:tasks.workunit.client.1.smithi086.stdout:0/901: chown d0/d1/d2/d1a/d51/f83 39065 1 2024-04-06T14:17:22.852 INFO:tasks.workunit.client.1.smithi086.stdout:9/828: rename d5/d13/d47/d5e/ld8 to d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/l10b 0 2024-04-06T14:17:22.853 INFO:tasks.workunit.client.0.smithi032.stdout:0/913: mknod d3/da/d25/d2b/d80/c132 0 2024-04-06T14:17:22.853 INFO:tasks.workunit.client.1.smithi086.stdout:5/754: chown d0/d16/d18/d21/d23/d28/d94/le0 22535 1 2024-04-06T14:17:22.854 INFO:tasks.workunit.client.1.smithi086.stdout:3/905: rmdir d5/d36/d56/dd3 39 2024-04-06T14:17:22.854 INFO:tasks.workunit.client.1.smithi086.stdout:9/829: dread - d5/d13/d1a/d5f/d88/d14/ff8 zero size 2024-04-06T14:17:22.855 INFO:tasks.workunit.client.1.smithi086.stdout:5/755: write d0/d16/f95 [1058778,95320] 0 2024-04-06T14:17:22.857 INFO:tasks.workunit.client.0.smithi032.stdout:0/914: fdatasync d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/d9c/fc7 0 2024-04-06T14:17:22.857 INFO:tasks.workunit.client.1.smithi086.stdout:5/756: chown d0/d16/d18/d21/d23/d28/lb0 13 1 2024-04-06T14:17:22.857 INFO:tasks.workunit.client.1.smithi086.stdout:7/846: getdents d7/d94 0 2024-04-06T14:17:22.858 INFO:tasks.workunit.client.1.smithi086.stdout:5/757: chown d0/d43/ld9 11130 1 2024-04-06T14:17:22.858 INFO:tasks.workunit.client.0.smithi032.stdout:0/915: chown d3/da/d25/d2b/d4a/d79/d97/de0/df5/cf7 1 1 2024-04-06T14:17:22.861 INFO:tasks.workunit.client.0.smithi032.stdout:0/916: creat d3/da/d25/d2b/d4a/d79/da8/f133 x:0 0 0 2024-04-06T14:17:22.864 INFO:tasks.workunit.client.1.smithi086.stdout:1/901: sync 2024-04-06T14:17:22.864 INFO:tasks.workunit.client.1.smithi086.stdout:3/906: rmdir d5/d11a 0 2024-04-06T14:17:22.865 INFO:tasks.workunit.client.0.smithi032.stdout:0/917: getdents d3/dc2 0 2024-04-06T14:17:22.869 INFO:tasks.workunit.client.1.smithi086.stdout:7/847: mknod d7/d10/d11/c12c 0 2024-04-06T14:17:22.869 INFO:tasks.workunit.client.1.smithi086.stdout:8/935: write d0/d71/d16/d52/dd0/ddc/de7/d136/f13e [423168,53859] 0 2024-04-06T14:17:22.869 INFO:tasks.workunit.client.1.smithi086.stdout:3/907: write d5/d36/d5a/d6d/d72/d57/ffe [174657,46618] 0 2024-04-06T14:17:22.880 INFO:tasks.workunit.client.1.smithi086.stdout:1/902: mknod db/d11/d42/c11f 0 2024-04-06T14:17:22.883 INFO:tasks.workunit.client.1.smithi086.stdout:1/903: stat db/d11/d1a/d2a/d9a/d68/daf/lef 0 2024-04-06T14:17:22.884 INFO:tasks.workunit.client.1.smithi086.stdout:9/830: dwrite d5/d13/d1a/d5f/d88/db3/fe3 [0,4194304] 0 2024-04-06T14:17:22.884 INFO:tasks.workunit.client.1.smithi086.stdout:8/936: truncate d0/d71/d82/f13f 913150 0 2024-04-06T14:17:22.886 INFO:tasks.workunit.client.1.smithi086.stdout:3/908: chown d5/d26/d7f/dae/d11e/fd5 2399 1 2024-04-06T14:17:22.886 INFO:tasks.workunit.client.1.smithi086.stdout:8/937: write d0/d71/f108 [2889204,2436] 0 2024-04-06T14:17:22.901 INFO:tasks.workunit.client.1.smithi086.stdout:1/904: dwrite db/d11/d1a/d2a/d9a/d68/daf/dc2/fe2 [0,4194304] 0 2024-04-06T14:17:22.902 INFO:tasks.workunit.client.1.smithi086.stdout:8/938: mkdir d0/d71/d22/d43/d6d/d75/d7d/d143 0 2024-04-06T14:17:22.905 INFO:tasks.workunit.client.1.smithi086.stdout:3/909: symlink d5/d36/l123 0 2024-04-06T14:17:22.907 INFO:tasks.workunit.client.1.smithi086.stdout:5/758: rename d0/d16/d18/d2a/d50/d66 to d0/d16/d18/d21/d23/d28/df7 0 2024-04-06T14:17:22.907 INFO:tasks.workunit.client.1.smithi086.stdout:5/759: readlink d0/d16/l2b 0 2024-04-06T14:17:22.911 INFO:tasks.workunit.client.1.smithi086.stdout:3/910: creat d5/d36/d5a/d6d/d72/d9a/f124 x:0 0 0 2024-04-06T14:17:22.914 INFO:tasks.workunit.client.1.smithi086.stdout:9/831: rename d5/d18/f48 to d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/f10c 0 2024-04-06T14:17:22.918 INFO:tasks.workunit.client.1.smithi086.stdout:5/760: write d0/d43/de5/ff6 [234149,99412] 0 2024-04-06T14:17:22.926 INFO:tasks.workunit.client.1.smithi086.stdout:5/761: symlink d0/d16/d18/d21/d23/d28/d94/d80/d85/lf8 0 2024-04-06T14:17:22.927 INFO:tasks.workunit.client.1.smithi086.stdout:5/762: dread - d0/d43/f9c zero size 2024-04-06T14:17:22.938 INFO:tasks.workunit.client.1.smithi086.stdout:8/939: write d0/d3b/db9/fbd [759579,127248] 0 2024-04-06T14:17:22.940 INFO:tasks.workunit.client.1.smithi086.stdout:1/905: dwrite db/d11/d1f/f5a [4194304,4194304] 0 2024-04-06T14:17:22.940 INFO:tasks.workunit.client.1.smithi086.stdout:5/763: write d0/d16/d18/d21/d23/d28/d94/f4f [577271,29308] 0 2024-04-06T14:17:22.943 INFO:tasks.workunit.client.1.smithi086.stdout:7/848: dwrite d7/d1d/f3e [0,4194304] 0 2024-04-06T14:17:22.947 INFO:tasks.workunit.client.1.smithi086.stdout:1/906: chown db/d13/c3e 1580777563 1 2024-04-06T14:17:22.947 INFO:tasks.workunit.client.1.smithi086.stdout:8/940: rmdir d0/d71/d16/d39/d90 39 2024-04-06T14:17:22.950 INFO:tasks.workunit.client.1.smithi086.stdout:5/764: read - d0/d43/de5/fe1 zero size 2024-04-06T14:17:22.955 INFO:tasks.workunit.client.1.smithi086.stdout:4/729: sync 2024-04-06T14:17:22.955 INFO:tasks.workunit.client.1.smithi086.stdout:2/933: sync 2024-04-06T14:17:22.955 INFO:tasks.workunit.client.1.smithi086.stdout:1/907: write db/d11/d87/fec [384458,75932] 0 2024-04-06T14:17:22.955 INFO:tasks.workunit.client.0.smithi032.stdout:0/918: write d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/fe9 [739351,28404] 0 2024-04-06T14:17:22.956 INFO:tasks.workunit.client.1.smithi086.stdout:4/730: readlink d0/l4e 0 2024-04-06T14:17:22.957 INFO:tasks.workunit.client.1.smithi086.stdout:3/911: dwrite d5/dea/dd2/ddb/de6/f2c [4194304,4194304] 0 2024-04-06T14:17:22.957 INFO:tasks.workunit.client.1.smithi086.stdout:7/849: creat d7/d10/d11/d1c/d24/df0/d116/f12d x:0 0 0 2024-04-06T14:17:22.959 INFO:tasks.workunit.client.0.smithi032.stdout:0/919: dread - d3/da/d25/d2b/d4a/d86/fc4 zero size 2024-04-06T14:17:22.960 INFO:tasks.workunit.client.1.smithi086.stdout:8/941: mknod d0/d71/d16/d52/dd0/d134/c144 0 2024-04-06T14:17:22.964 INFO:tasks.workunit.client.1.smithi086.stdout:9/832: dwrite d5/d13/d1a/d5f/d88/f53 [0,4194304] 0 2024-04-06T14:17:22.967 INFO:tasks.workunit.client.1.smithi086.stdout:3/912: unlink d5/d26/d7f/dae/d11e/fd5 0 2024-04-06T14:17:22.976 INFO:tasks.workunit.client.1.smithi086.stdout:3/913: unlink d5/dea/dd2/ddb/de6/f6a 0 2024-04-06T14:17:22.978 INFO:tasks.workunit.client.1.smithi086.stdout:1/908: dwrite db/d11/d1f/d5c/f112 [0,4194304] 0 2024-04-06T14:17:22.979 INFO:tasks.workunit.client.1.smithi086.stdout:1/909: stat db/d11/d1a/d2a/f4c 0 2024-04-06T14:17:22.980 INFO:tasks.workunit.client.1.smithi086.stdout:3/914: readlink d5/d36/d65/l6c 0 2024-04-06T14:17:22.980 INFO:tasks.workunit.client.1.smithi086.stdout:2/934: truncate d0/d20/d44/dbb/fc0 1118137 0 2024-04-06T14:17:22.986 INFO:tasks.workunit.client.1.smithi086.stdout:3/915: mknod d5/dea/dd2/ddb/de6/d10b/c125 0 2024-04-06T14:17:22.988 INFO:tasks.workunit.client.1.smithi086.stdout:3/916: chown d5/dea/dd2/ddb/de6/fca 6629332 1 2024-04-06T14:17:22.991 INFO:tasks.workunit.client.1.smithi086.stdout:2/935: rename d0/df/f11e to d0/df/d5e/d9d/d102/d100/f14c 0 2024-04-06T14:17:22.993 INFO:tasks.workunit.client.1.smithi086.stdout:9/833: dread d5/d18/dff/f9a [0,4194304] 0 2024-04-06T14:17:22.999 INFO:tasks.workunit.client.1.smithi086.stdout:5/765: dread d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc0 [0,4194304] 0 2024-04-06T14:17:22.999 INFO:tasks.workunit.client.1.smithi086.stdout:8/942: truncate d0/f26 1725452 0 2024-04-06T14:17:23.001 INFO:tasks.workunit.client.1.smithi086.stdout:4/731: truncate d0/de/d4c/fc5 681074 0 2024-04-06T14:17:23.002 INFO:tasks.workunit.client.1.smithi086.stdout:1/910: write db/d11/d1a/d2a/d9a/fb3 [760740,78084] 0 2024-04-06T14:17:23.004 INFO:tasks.workunit.client.1.smithi086.stdout:9/834: mknod d5/d13/d47/d55/c10d 0 2024-04-06T14:17:23.006 INFO:tasks.workunit.client.1.smithi086.stdout:3/917: write d5/d36/f67 [1251484,67414] 0 2024-04-06T14:17:23.007 INFO:tasks.workunit.client.1.smithi086.stdout:5/766: write d0/d16/d18/f9d [500305,109197] 0 2024-04-06T14:17:23.013 INFO:tasks.workunit.client.1.smithi086.stdout:9/835: unlink d5/d13/d1a/d5f/d88/d14/d19/d1b/l21 0 2024-04-06T14:17:23.019 INFO:tasks.workunit.client.1.smithi086.stdout:5/767: unlink d0/d16/d18/d21/d23/d28/lb0 0 2024-04-06T14:17:23.026 INFO:tasks.workunit.client.1.smithi086.stdout:8/943: rename d0/d71/d16/d52/dd0/ddc/de7/dd6 to d0/d71/d16/d145 0 2024-04-06T14:17:23.026 INFO:tasks.workunit.client.1.smithi086.stdout:5/768: rename d0/d16/d18/d2a to d0/d16/d18/d2a/d50/df9 22 2024-04-06T14:17:23.027 INFO:tasks.workunit.client.1.smithi086.stdout:5/769: readlink d0/d16/d71/d96/lcc 0 2024-04-06T14:17:23.030 INFO:tasks.workunit.client.1.smithi086.stdout:2/936: truncate d0/d20/d27/d2b/d35/d36/f13a 2439330 0 2024-04-06T14:17:23.031 INFO:tasks.workunit.client.1.smithi086.stdout:3/918: dread d5/f44 [0,4194304] 0 2024-04-06T14:17:23.034 INFO:tasks.workunit.client.1.smithi086.stdout:3/919: dread - d5/d36/d5a/f10f zero size 2024-04-06T14:17:23.038 INFO:tasks.workunit.client.1.smithi086.stdout:9/836: write d5/d13/d1a/d5f/d88/d14/f22 [95323,48905] 0 2024-04-06T14:17:23.040 INFO:tasks.workunit.client.1.smithi086.stdout:2/937: symlink d0/df/d5e/d9d/d102/l14d 0 2024-04-06T14:17:23.041 INFO:tasks.workunit.client.0.smithi032.stdout:0/920: dwrite d3/da/d25/f94 [0,4194304] 0 2024-04-06T14:17:23.042 INFO:tasks.workunit.client.1.smithi086.stdout:8/944: getdents d0/d71/d22/d43/d6d/d75/d7d 0 2024-04-06T14:17:23.043 INFO:tasks.workunit.client.1.smithi086.stdout:4/732: dwrite d0/f40 [0,4194304] 0 2024-04-06T14:17:23.043 INFO:tasks.workunit.client.1.smithi086.stdout:3/920: creat d5/d36/d7d/f126 x:0 0 0 2024-04-06T14:17:23.047 INFO:tasks.workunit.client.1.smithi086.stdout:1/911: dwrite db/d11/d1a/d79/f89 [0,4194304] 0 2024-04-06T14:17:23.048 INFO:tasks.workunit.client.1.smithi086.stdout:3/921: write d5/d26/d7f/dae/f118 [423590,23760] 0 2024-04-06T14:17:23.053 INFO:tasks.workunit.client.0.smithi032.stdout:0/921: mkdir d3/da/d25/d2b/d80/d100/d134 0 2024-04-06T14:17:23.056 INFO:tasks.workunit.client.1.smithi086.stdout:5/770: dread d0/d5e/f60 [0,4194304] 0 2024-04-06T14:17:23.060 INFO:tasks.workunit.client.1.smithi086.stdout:3/922: creat d5/dea/db4/f127 x:0 0 0 2024-04-06T14:17:23.061 INFO:tasks.workunit.client.1.smithi086.stdout:5/771: mknod d0/d16/d18/d35/d9e/de7/cfa 0 2024-04-06T14:17:23.063 INFO:tasks.workunit.client.1.smithi086.stdout:3/923: fsync d5/d36/d5a/d6d/d72/d9a/f124 0 2024-04-06T14:17:23.068 INFO:tasks.workunit.client.1.smithi086.stdout:9/837: dwrite d5/d18/dff/fb1 [0,4194304] 0 2024-04-06T14:17:23.072 INFO:tasks.workunit.client.1.smithi086.stdout:2/938: link d0/d20/d27/d2b/d35/d36/d4d/d6f/f81 d0/d20/d44/dee/f14e 0 2024-04-06T14:17:23.075 INFO:tasks.workunit.client.1.smithi086.stdout:8/945: getdents d0/d71/d22/d43/d6d 0 2024-04-06T14:17:23.075 INFO:tasks.workunit.client.1.smithi086.stdout:2/939: write d0/f104 [588183,34787] 0 2024-04-06T14:17:23.076 INFO:tasks.workunit.client.1.smithi086.stdout:1/912: getdents db/d11/d1a/d79 0 2024-04-06T14:17:23.078 INFO:tasks.workunit.client.1.smithi086.stdout:4/733: dwrite d0/d2c/d30/d24/d50/d79/f7c [0,4194304] 0 2024-04-06T14:17:23.081 INFO:tasks.workunit.client.1.smithi086.stdout:3/924: getdents d5/d36/d56 0 2024-04-06T14:17:23.082 INFO:tasks.workunit.client.0.smithi032.stdout:0/922: dwrite d3/da/d25/d36/fbd [0,4194304] 0 2024-04-06T14:17:23.085 INFO:tasks.workunit.client.1.smithi086.stdout:2/940: write d0/d20/d44/d120/f134 [1849748,61754] 0 2024-04-06T14:17:23.088 INFO:tasks.workunit.client.1.smithi086.stdout:9/838: dwrite d5/d13/d47/fe8 [0,4194304] 0 2024-04-06T14:17:23.088 INFO:tasks.workunit.client.1.smithi086.stdout:2/941: fdatasync d0/fe 0 2024-04-06T14:17:23.088 INFO:tasks.workunit.client.1.smithi086.stdout:5/772: dread d0/d16/d18/d35/f4d [0,4194304] 0 2024-04-06T14:17:23.090 INFO:tasks.workunit.client.1.smithi086.stdout:8/946: symlink d0/d3b/d79/dc6/d118/l146 0 2024-04-06T14:17:23.094 INFO:tasks.workunit.client.1.smithi086.stdout:5/773: readlink d0/d16/d18/d21/d23/d28/d94/d52/d7c/lce 0 2024-04-06T14:17:23.097 INFO:tasks.workunit.client.0.smithi032.stdout:0/923: dwrite d3/dc2/fda [0,4194304] 0 2024-04-06T14:17:23.098 INFO:tasks.workunit.client.1.smithi086.stdout:3/925: symlink d5/d26/l128 0 2024-04-06T14:17:23.098 INFO:tasks.workunit.client.1.smithi086.stdout:8/947: mkdir d0/d71/d16/d52/dd0/d134/d147 0 2024-04-06T14:17:23.098 INFO:tasks.workunit.client.1.smithi086.stdout:9/839: fsync d5/d13/d1a/d5f/d88/d14/d19/d25/d74/fa5 0 2024-04-06T14:17:23.100 INFO:tasks.workunit.client.1.smithi086.stdout:2/942: symlink d0/d20/d27/d2b/d35/d36/dbe/l14f 0 2024-04-06T14:17:23.100 INFO:tasks.workunit.client.0.smithi032.stdout:0/924: fdatasync d3/da/d25/d2b/d4a/fbe 0 2024-04-06T14:17:23.101 INFO:tasks.workunit.client.1.smithi086.stdout:3/926: fdatasync d5/d36/d5a/d6d/d72/d64/fdc 0 2024-04-06T14:17:23.101 INFO:tasks.workunit.client.1.smithi086.stdout:5/774: symlink d0/d16/d71/lfb 0 2024-04-06T14:17:23.107 INFO:tasks.workunit.client.1.smithi086.stdout:2/943: creat d0/df/d5e/d9d/de7/f150 x:0 0 0 2024-04-06T14:17:23.110 INFO:tasks.workunit.client.1.smithi086.stdout:3/927: creat d5/d36/d56/dd3/dfc/f129 x:0 0 0 2024-04-06T14:17:23.111 INFO:tasks.workunit.client.1.smithi086.stdout:2/944: stat d0/df/d48/d7d/fe9 0 2024-04-06T14:17:23.112 INFO:tasks.workunit.client.1.smithi086.stdout:4/734: dwrite d0/d2c/d38/faa [0,4194304] 0 2024-04-06T14:17:23.112 INFO:tasks.workunit.client.1.smithi086.stdout:2/945: chown d0/df/d5e 6525 1 2024-04-06T14:17:23.114 INFO:tasks.workunit.client.1.smithi086.stdout:1/913: dwrite db/d13/d71/f88 [0,4194304] 0 2024-04-06T14:17:23.114 INFO:tasks.workunit.client.1.smithi086.stdout:9/840: link d5/d13/d1a/c2a d5/d13/d1a/c10e 0 2024-04-06T14:17:23.117 INFO:tasks.workunit.client.1.smithi086.stdout:2/946: chown d0/d20/d27/d2b/cea 180384 1 2024-04-06T14:17:23.124 INFO:tasks.workunit.client.1.smithi086.stdout:0/902: sync 2024-04-06T14:17:23.124 INFO:tasks.workunit.client.1.smithi086.stdout:5/775: dread d0/d16/d18/d21/d23/d28/d94/d80/d85/f9f [0,4194304] 0 2024-04-06T14:17:23.125 INFO:tasks.workunit.client.1.smithi086.stdout:2/947: symlink d0/df/dd2/l151 0 2024-04-06T14:17:23.127 INFO:tasks.workunit.client.1.smithi086.stdout:9/841: creat d5/d13/d1a/d5f/d88/d14/d19/d94/f10f x:0 0 0 2024-04-06T14:17:23.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:22 smithi086 ceph-mon[47925]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:23.127 INFO:tasks.workunit.client.1.smithi086.stdout:4/735: creat d0/fdf x:0 0 0 2024-04-06T14:17:23.129 INFO:tasks.workunit.client.1.smithi086.stdout:5/776: mknod d0/d16/d18/de6/cfc 0 2024-04-06T14:17:23.130 INFO:tasks.workunit.client.1.smithi086.stdout:9/842: write d5/d13/d47/fc4 [1914481,86106] 0 2024-04-06T14:17:23.132 INFO:tasks.workunit.client.1.smithi086.stdout:2/948: creat d0/df/d5e/d9d/d102/d143/f152 x:0 0 0 2024-04-06T14:17:23.132 INFO:tasks.workunit.client.1.smithi086.stdout:1/914: dwrite db/d11/d1a/d2a/d9a/d68/d6e/d81/ffc [0,4194304] 0 2024-04-06T14:17:23.134 INFO:tasks.workunit.client.1.smithi086.stdout:5/777: chown d0/d16/d18/fec 42490179 1 2024-04-06T14:17:23.136 INFO:tasks.workunit.client.1.smithi086.stdout:0/903: getdents d0/d1/d2/d1a/d79/dac/db7 0 2024-04-06T14:17:23.138 INFO:tasks.workunit.client.1.smithi086.stdout:9/843: write d5/d13/d47/d55/d87/f109 [1877060,44715] 0 2024-04-06T14:17:23.142 INFO:tasks.workunit.client.1.smithi086.stdout:8/948: write d0/d71/d22/d43/d7f/f88 [181619,26806] 0 2024-04-06T14:17:23.142 INFO:tasks.workunit.client.1.smithi086.stdout:4/736: dwrite d0/d2c/d38/d54/f80 [0,4194304] 0 2024-04-06T14:17:23.145 INFO:tasks.workunit.client.1.smithi086.stdout:2/949: rename d0/d20/d80/dbf/ddc/fe6 to d0/d20/d80/dbf/ddc/f153 0 2024-04-06T14:17:23.145 INFO:tasks.workunit.client.1.smithi086.stdout:8/949: truncate d0/d13/db6/fba 1232543 0 2024-04-06T14:17:23.148 INFO:tasks.workunit.client.1.smithi086.stdout:1/915: creat db/d11/d1f/d5c/ddb/f120 x:0 0 0 2024-04-06T14:17:23.148 INFO:tasks.workunit.client.1.smithi086.stdout:0/904: creat d0/d1/d2/d1a/d79/f12a x:0 0 0 2024-04-06T14:17:23.152 INFO:tasks.workunit.client.1.smithi086.stdout:9/844: creat d5/d13/d47/d5e/f110 x:0 0 0 2024-04-06T14:17:23.152 INFO:tasks.workunit.client.1.smithi086.stdout:4/737: symlink d0/d15/d20/d34/le0 0 2024-04-06T14:17:23.152 INFO:tasks.workunit.client.1.smithi086.stdout:3/928: write d5/d36/d5a/d6d/d72/d57/fd1 [1163423,63805] 0 2024-04-06T14:17:23.155 INFO:tasks.workunit.client.1.smithi086.stdout:1/916: truncate f2 405327 0 2024-04-06T14:17:23.159 INFO:tasks.workunit.client.1.smithi086.stdout:8/950: symlink d0/d71/d16/d39/d90/daf/df0/l148 0 2024-04-06T14:17:23.160 INFO:tasks.workunit.client.1.smithi086.stdout:0/905: truncate d0/d1/d2/d1a/f56 105430 0 2024-04-06T14:17:23.160 INFO:tasks.workunit.client.1.smithi086.stdout:2/950: dwrite d0/d20/d27/d2b/d35/d36/fa2 [0,4194304] 0 2024-04-06T14:17:23.161 INFO:tasks.workunit.client.1.smithi086.stdout:4/738: creat d0/d2c/d38/d54/d61/d7a/dc2/dd3/fe1 x:0 0 0 2024-04-06T14:17:23.164 INFO:tasks.workunit.client.1.smithi086.stdout:9/845: fdatasync d5/d13/d1a/d5f/d88/d14/d19/f4b 0 2024-04-06T14:17:23.166 INFO:tasks.workunit.client.1.smithi086.stdout:0/906: unlink d0/d1/d2/d1a/f63 0 2024-04-06T14:17:23.169 INFO:tasks.workunit.client.1.smithi086.stdout:4/739: dwrite d0/d2c/d30/d24/d5d/f8f [0,4194304] 0 2024-04-06T14:17:23.176 INFO:tasks.workunit.client.1.smithi086.stdout:8/951: rename d0/d3b/f51 to d0/d3b/d79/dc6/d123/f149 0 2024-04-06T14:17:23.177 INFO:tasks.workunit.client.1.smithi086.stdout:3/929: getdents d5/d36/d5a/d6d/d72/d64 0 2024-04-06T14:17:23.179 INFO:tasks.workunit.client.1.smithi086.stdout:0/907: creat d0/d1/d4/de/dae/f12b x:0 0 0 2024-04-06T14:17:23.179 INFO:tasks.workunit.client.1.smithi086.stdout:3/930: truncate d5/d36/d7d/f109 706871 0 2024-04-06T14:17:23.180 INFO:tasks.workunit.client.1.smithi086.stdout:4/740: unlink d0/d2c/d30/d24/d93/fdc 0 2024-04-06T14:17:23.181 INFO:tasks.workunit.client.1.smithi086.stdout:0/908: dread - d0/d15/d26/d33/d9f/f101 zero size 2024-04-06T14:17:23.182 INFO:tasks.workunit.client.1.smithi086.stdout:8/952: mkdir d0/d71/d22/d14a 0 2024-04-06T14:17:23.183 INFO:tasks.workunit.client.1.smithi086.stdout:4/741: write d0/f5 [4624919,118296] 0 2024-04-06T14:17:23.186 INFO:tasks.workunit.client.1.smithi086.stdout:8/953: chown d0/d5a/fcb 59 1 2024-04-06T14:17:23.186 INFO:tasks.workunit.client.1.smithi086.stdout:0/909: creat d0/d1/dfe/f12c x:0 0 0 2024-04-06T14:17:23.188 INFO:tasks.workunit.client.1.smithi086.stdout:4/742: unlink d0/d15/l49 0 2024-04-06T14:17:23.189 INFO:tasks.workunit.client.1.smithi086.stdout:9/846: dread d5/d13/d1a/fb6 [0,4194304] 0 2024-04-06T14:17:23.190 INFO:tasks.workunit.client.1.smithi086.stdout:8/954: read d0/d71/d82/f13f [261284,7687] 0 2024-04-06T14:17:23.192 INFO:tasks.workunit.client.1.smithi086.stdout:1/917: dwrite db/d11/d1a/d79/da5/f115 [0,4194304] 0 2024-04-06T14:17:23.192 INFO:tasks.workunit.client.1.smithi086.stdout:3/931: rename d5/dea/db4/da7/d89/ccb to d5/d36/d5a/d6d/d72/c12a 0 2024-04-06T14:17:23.193 INFO:tasks.workunit.client.1.smithi086.stdout:3/932: chown d5 5 1 2024-04-06T14:17:23.195 INFO:tasks.workunit.client.1.smithi086.stdout:3/933: chown d5/f44 159102935 1 2024-04-06T14:17:23.196 INFO:tasks.workunit.client.1.smithi086.stdout:2/951: dwrite d0/d20/d44/dbb/dc4/de3/d76/fb9 [0,4194304] 0 2024-04-06T14:17:23.196 INFO:tasks.workunit.client.1.smithi086.stdout:4/743: mkdir d0/d2c/d30/d24/de2 0 2024-04-06T14:17:23.199 INFO:tasks.workunit.client.1.smithi086.stdout:0/910: mkdir d0/d15/d4e/da3/d44/d5d/d74/d120/d12d 0 2024-04-06T14:17:23.205 INFO:tasks.workunit.client.1.smithi086.stdout:9/847: mknod d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/c111 0 2024-04-06T14:17:23.207 INFO:tasks.workunit.client.1.smithi086.stdout:2/952: creat d0/d20/d44/d120/f154 x:0 0 0 2024-04-06T14:17:23.209 INFO:tasks.workunit.client.1.smithi086.stdout:3/934: write d5/dea/dd2/ddb/de6/f85 [3804599,105540] 0 2024-04-06T14:17:23.210 INFO:tasks.workunit.client.1.smithi086.stdout:4/744: creat d0/d2c/d38/fe3 x:0 0 0 2024-04-06T14:17:23.210 INFO:tasks.workunit.client.1.smithi086.stdout:3/935: stat d5/dea/dd2/ddb/de6/d10b/fdd 0 2024-04-06T14:17:23.214 INFO:tasks.workunit.client.1.smithi086.stdout:9/848: creat d5/d13/d1a/d5f/d88/d14/d19/d1b/f112 x:0 0 0 2024-04-06T14:17:23.214 INFO:tasks.workunit.client.1.smithi086.stdout:9/849: chown d5/d18 15 1 2024-04-06T14:17:23.216 INFO:tasks.workunit.client.1.smithi086.stdout:3/936: unlink d5/d33/c4a 0 2024-04-06T14:17:23.216 INFO:tasks.workunit.client.1.smithi086.stdout:0/911: getdents d0/d15/d4e/da3/d46/d52/dd0/dbe/d107 0 2024-04-06T14:17:23.217 INFO:tasks.workunit.client.1.smithi086.stdout:4/745: rmdir d0/d2c/d38/d54/d61/d7a 39 2024-04-06T14:17:23.217 INFO:tasks.workunit.client.1.smithi086.stdout:3/937: write d5/dea/f88 [3340179,123231] 0 2024-04-06T14:17:23.219 INFO:tasks.workunit.client.1.smithi086.stdout:9/850: read d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f81 [438444,85263] 0 2024-04-06T14:17:23.220 INFO:tasks.workunit.client.1.smithi086.stdout:2/953: truncate d0/fd 3208541 0 2024-04-06T14:17:23.220 INFO:tasks.workunit.client.1.smithi086.stdout:9/851: readlink d5/d13/d1a/d5f/d88/l4a 0 2024-04-06T14:17:23.223 INFO:tasks.workunit.client.1.smithi086.stdout:1/918: write db/d11/d1f/f44 [4488825,129896] 0 2024-04-06T14:17:23.225 INFO:tasks.workunit.client.1.smithi086.stdout:2/954: readlink d0/d20/d82/lc5 0 2024-04-06T14:17:23.229 INFO:tasks.workunit.client.1.smithi086.stdout:4/746: mknod d0/d15/ce4 0 2024-04-06T14:17:23.229 INFO:tasks.workunit.client.1.smithi086.stdout:1/919: dwrite db/d13/d55/da3/fc4 [0,4194304] 0 2024-04-06T14:17:23.229 INFO:tasks.workunit.client.1.smithi086.stdout:8/955: truncate d0/d71/d22/d43/daa/f113 4146857 0 2024-04-06T14:17:23.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:22 smithi032 ceph-mon[39730]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:23.232 INFO:tasks.workunit.client.1.smithi086.stdout:3/938: symlink d5/d26/d7f/l12b 0 2024-04-06T14:17:23.232 INFO:tasks.workunit.client.1.smithi086.stdout:3/939: chown d5/dea/db4/f100 10006798 1 2024-04-06T14:17:23.234 INFO:tasks.workunit.client.1.smithi086.stdout:2/955: readlink d0/l7 0 2024-04-06T14:17:23.234 INFO:tasks.workunit.client.1.smithi086.stdout:3/940: write d5/d36/d56/f105 [670369,119079] 0 2024-04-06T14:17:23.235 INFO:tasks.workunit.client.1.smithi086.stdout:0/912: write d0/d1/d4/de/dae/f119 [1567842,51300] 0 2024-04-06T14:17:23.241 INFO:tasks.workunit.client.1.smithi086.stdout:3/941: rmdir d5/d36/d56 39 2024-04-06T14:17:23.242 INFO:tasks.workunit.client.1.smithi086.stdout:2/956: mknod d0/df/d5e/d9d/d102/d100/c155 0 2024-04-06T14:17:23.243 INFO:tasks.workunit.client.1.smithi086.stdout:4/747: creat d0/dc0/dbb/fe5 x:0 0 0 2024-04-06T14:17:23.245 INFO:tasks.workunit.client.1.smithi086.stdout:7/850: sync 2024-04-06T14:17:23.245 INFO:tasks.workunit.client.1.smithi086.stdout:0/913: creat d0/d15/d4e/da3/d44/d8e/d8f/f12e x:0 0 0 2024-04-06T14:17:23.246 INFO:tasks.workunit.client.1.smithi086.stdout:4/748: truncate d0/d2c/d38/d54/d61/fd4 284400 0 2024-04-06T14:17:23.247 INFO:tasks.workunit.client.1.smithi086.stdout:4/749: chown d0/de/c29 14 1 2024-04-06T14:17:23.248 INFO:tasks.workunit.client.1.smithi086.stdout:4/750: write d0/d15/d2b/fda [363540,108056] 0 2024-04-06T14:17:23.250 INFO:tasks.workunit.client.1.smithi086.stdout:8/956: write d0/d71/d82/f13f [184260,19335] 0 2024-04-06T14:17:23.251 INFO:tasks.workunit.client.1.smithi086.stdout:7/851: creat d7/d10/d102/d5c/d66/d7e/db9/dd2/f12e x:0 0 0 2024-04-06T14:17:23.252 INFO:tasks.workunit.client.1.smithi086.stdout:9/852: dwrite d5/d13/d1a/d5f/d88/d14/d19/f5d [0,4194304] 0 2024-04-06T14:17:23.253 INFO:tasks.workunit.client.1.smithi086.stdout:0/914: mkdir d0/d15/d4e/da3/d46/d52/dd0/d12f 0 2024-04-06T14:17:23.253 INFO:tasks.workunit.client.1.smithi086.stdout:1/920: link db/d11/d1a/c1c db/d11/d1f/d5c/ddb/de9/c121 0 2024-04-06T14:17:23.257 INFO:tasks.workunit.client.1.smithi086.stdout:3/942: rename d5/dea/dd2/ddb/de6/d9e/db5 to d5/d36/d12c 0 2024-04-06T14:17:23.258 INFO:tasks.workunit.client.1.smithi086.stdout:8/957: mknod d0/d3b/d79/dc6/c14b 0 2024-04-06T14:17:23.259 INFO:tasks.workunit.client.1.smithi086.stdout:7/852: mknod d7/c12f 0 2024-04-06T14:17:23.261 INFO:tasks.workunit.client.1.smithi086.stdout:2/957: creat d0/d20/d44/dbb/dc4/de3/d76/f156 x:0 0 0 2024-04-06T14:17:23.261 INFO:tasks.workunit.client.1.smithi086.stdout:4/751: mkdir d0/d2c/d38/dd7/de6 0 2024-04-06T14:17:23.261 INFO:tasks.workunit.client.1.smithi086.stdout:8/958: dread d0/d5a/d100/fcf [0,4194304] 0 2024-04-06T14:17:23.263 INFO:tasks.workunit.client.1.smithi086.stdout:1/921: symlink db/d13/d55/da3/l122 0 2024-04-06T14:17:23.263 INFO:tasks.workunit.client.1.smithi086.stdout:0/915: dwrite d0/d15/d4e/da3/d46/d52/dd0/fa7 [0,4194304] 0 2024-04-06T14:17:23.270 INFO:tasks.workunit.client.1.smithi086.stdout:1/922: creat db/d11/d1a/d2a/d52/d6b/f123 x:0 0 0 2024-04-06T14:17:23.271 INFO:tasks.workunit.client.1.smithi086.stdout:3/943: mknod d5/d26/d7f/c12d 0 2024-04-06T14:17:23.272 INFO:tasks.workunit.client.1.smithi086.stdout:7/853: symlink d7/d10/d102/d5c/l130 0 2024-04-06T14:17:23.272 INFO:tasks.workunit.client.1.smithi086.stdout:8/959: fsync d0/f5 0 2024-04-06T14:17:23.273 INFO:tasks.workunit.client.1.smithi086.stdout:8/960: readlink d0/db/leb 0 2024-04-06T14:17:23.273 INFO:tasks.workunit.client.1.smithi086.stdout:2/958: truncate d0/d20/d27/d2b/d35/d36/f13a 763173 0 2024-04-06T14:17:23.275 INFO:tasks.workunit.client.1.smithi086.stdout:1/923: creat db/d11/d1a/d79/f124 x:0 0 0 2024-04-06T14:17:23.275 INFO:tasks.workunit.client.1.smithi086.stdout:8/961: truncate d0/d71/d16/d39/d90/f122 532817 0 2024-04-06T14:17:23.276 INFO:tasks.workunit.client.1.smithi086.stdout:7/854: creat d7/f131 x:0 0 0 2024-04-06T14:17:23.280 INFO:tasks.workunit.client.1.smithi086.stdout:1/924: dread db/d11/d1a/d79/f9d [0,4194304] 0 2024-04-06T14:17:23.281 INFO:tasks.workunit.client.1.smithi086.stdout:3/944: mkdir d5/d36/d5a/d6d/d72/d108/d12e 0 2024-04-06T14:17:23.281 INFO:tasks.workunit.client.1.smithi086.stdout:1/925: write db/d11/fc1 [1505207,94093] 0 2024-04-06T14:17:23.283 INFO:tasks.workunit.client.1.smithi086.stdout:2/959: creat d0/d20/d27/d2b/d35/d36/d4d/d59/f157 x:0 0 0 2024-04-06T14:17:23.285 INFO:tasks.workunit.client.1.smithi086.stdout:8/962: rmdir d0/d13 39 2024-04-06T14:17:23.288 INFO:tasks.workunit.client.1.smithi086.stdout:7/855: mknod d7/d10/d102/daf/db5/d7a/d100/c132 0 2024-04-06T14:17:23.288 INFO:tasks.workunit.client.1.smithi086.stdout:9/853: truncate d5/d13/d1a/d5f/d88/d14/d19/ffc 3921342 0 2024-04-06T14:17:23.288 INFO:tasks.workunit.client.1.smithi086.stdout:4/752: link d0/d2c/d38/d54/d61/cc6 d0/de/ce7 0 2024-04-06T14:17:23.291 INFO:tasks.workunit.client.1.smithi086.stdout:3/945: symlink d5/d36/d65/l12f 0 2024-04-06T14:17:23.296 INFO:tasks.workunit.client.1.smithi086.stdout:7/856: read - d7/d10/d102/d5c/d66/d7e/db9/fd9 zero size 2024-04-06T14:17:23.297 INFO:tasks.workunit.client.1.smithi086.stdout:4/753: readlink d0/d2c/d30/d24/da3/l1f 0 2024-04-06T14:17:23.297 INFO:tasks.workunit.client.1.smithi086.stdout:3/946: dwrite d5/dea/dd2/ddb/de6/d10b/fdd [0,4194304] 0 2024-04-06T14:17:23.298 INFO:tasks.workunit.client.1.smithi086.stdout:4/754: chown d0/dc0/dbb 415340901 1 2024-04-06T14:17:23.299 INFO:tasks.workunit.client.1.smithi086.stdout:3/947: readlink d5/d36/d5a/d6d/d72/d57/lc6 0 2024-04-06T14:17:23.301 INFO:tasks.workunit.client.1.smithi086.stdout:2/960: mkdir d0/d20/d44/dbb/dc4/de3/d76/d158 0 2024-04-06T14:17:23.303 INFO:tasks.workunit.client.1.smithi086.stdout:8/963: stat d0/d71/d22/d43/d4d/da3/dc7/l105 0 2024-04-06T14:17:23.305 INFO:tasks.workunit.client.1.smithi086.stdout:1/926: link db/d13/f83 db/d13/d55/da3/f125 0 2024-04-06T14:17:23.308 INFO:tasks.workunit.client.1.smithi086.stdout:2/961: mknod d0/d20/d44/dbb/c159 0 2024-04-06T14:17:23.310 INFO:tasks.workunit.client.1.smithi086.stdout:7/857: dread d7/d10/d11/d123/f4a [0,4194304] 0 2024-04-06T14:17:23.313 INFO:tasks.workunit.client.1.smithi086.stdout:1/927: symlink db/d11/d1a/d2a/l126 0 2024-04-06T14:17:23.314 INFO:tasks.workunit.client.1.smithi086.stdout:3/948: mkdir d5/dea/dd2/ddb/de6/dee/d130 0 2024-04-06T14:17:23.314 INFO:tasks.workunit.client.1.smithi086.stdout:2/962: dwrite d0/df/f84 [0,4194304] 0 2024-04-06T14:17:23.315 INFO:tasks.workunit.client.1.smithi086.stdout:3/949: chown d5/dea/db4/f127 79667894 1 2024-04-06T14:17:23.316 INFO:tasks.workunit.client.1.smithi086.stdout:9/854: getdents d5/d13/d1a/d5f/d88/d14/d19/d94 0 2024-04-06T14:17:23.318 INFO:tasks.workunit.client.1.smithi086.stdout:8/964: mkdir d0/d71/d16/d39/d14c 0 2024-04-06T14:17:23.322 INFO:tasks.workunit.client.1.smithi086.stdout:1/928: creat db/d11/d1a/d79/da5/f127 x:0 0 0 2024-04-06T14:17:23.322 INFO:tasks.workunit.client.1.smithi086.stdout:2/963: mknod d0/d20/d27/d2b/d35/d36/d4d/d59/c15a 0 2024-04-06T14:17:23.323 INFO:tasks.workunit.client.1.smithi086.stdout:3/950: creat d5/d36/d5a/d6d/d72/d108/f131 x:0 0 0 2024-04-06T14:17:23.324 INFO:tasks.workunit.client.1.smithi086.stdout:8/965: creat d0/d71/d16/d39/d14c/f14d x:0 0 0 2024-04-06T14:17:23.325 INFO:tasks.workunit.client.1.smithi086.stdout:9/855: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dce/f113 x:0 0 0 2024-04-06T14:17:23.326 INFO:tasks.workunit.client.1.smithi086.stdout:4/755: write d0/d15/d20/d74/f7b [282563,27830] 0 2024-04-06T14:17:23.328 INFO:tasks.workunit.client.1.smithi086.stdout:7/858: rename d7/d10/d11/d1c/l96 to d7/d10/d102/l133 0 2024-04-06T14:17:23.329 INFO:tasks.workunit.client.1.smithi086.stdout:7/859: dread - d7/fd6 zero size 2024-04-06T14:17:23.330 INFO:tasks.workunit.client.1.smithi086.stdout:3/951: mknod d5/d36/d12c/c132 0 2024-04-06T14:17:23.330 INFO:tasks.workunit.client.1.smithi086.stdout:8/966: mkdir d0/d71/d16/d52/d14e 0 2024-04-06T14:17:23.333 INFO:tasks.workunit.client.1.smithi086.stdout:9/856: symlink d5/d13/d1a/d5f/d88/d14/d19/d94/l114 0 2024-04-06T14:17:23.336 INFO:tasks.workunit.client.1.smithi086.stdout:3/952: unlink d5/d26/d7f/dae/d11e/lfb 0 2024-04-06T14:17:23.337 INFO:tasks.workunit.client.1.smithi086.stdout:2/964: dread d0/d20/d27/d2b/d35/f12f [0,4194304] 0 2024-04-06T14:17:23.338 INFO:tasks.workunit.client.1.smithi086.stdout:8/967: getdents d0/d71/d22/d43/daa 0 2024-04-06T14:17:23.339 INFO:tasks.workunit.client.1.smithi086.stdout:7/860: creat d7/d10/d11/d1c/d24/d2c/f134 x:0 0 0 2024-04-06T14:17:23.341 INFO:tasks.workunit.client.1.smithi086.stdout:4/756: dwrite d0/de/d4c/f5a [0,4194304] 0 2024-04-06T14:17:23.343 INFO:tasks.workunit.client.1.smithi086.stdout:1/929: fsync db/d11/d1f/d5c/ff6 0 2024-04-06T14:17:23.344 INFO:tasks.workunit.client.1.smithi086.stdout:4/757: chown d0/d2c/d30/l9d 6 1 2024-04-06T14:17:23.344 INFO:tasks.workunit.client.1.smithi086.stdout:3/953: write d5/dea/dd2/ddb/de6/f45 [4704897,81535] 0 2024-04-06T14:17:23.344 INFO:tasks.workunit.client.1.smithi086.stdout:9/857: dwrite d5/d13/d1a/d5f/fc8 [0,4194304] 0 2024-04-06T14:17:23.345 INFO:tasks.workunit.client.1.smithi086.stdout:7/861: creat d7/d10/d102/d5c/d66/d7e/f135 x:0 0 0 2024-04-06T14:17:23.347 INFO:tasks.workunit.client.1.smithi086.stdout:9/858: dread - d5/d13/d1a/d5f/d88/d14/f101 zero size 2024-04-06T14:17:23.347 INFO:tasks.workunit.client.1.smithi086.stdout:3/954: write d5/d36/d5a/d6d/d72/d64/fdc [4332108,1732] 0 2024-04-06T14:17:23.349 INFO:tasks.workunit.client.1.smithi086.stdout:4/758: fsync d0/d2c/d30/d24/d50/f72 0 2024-04-06T14:17:23.350 INFO:tasks.workunit.client.1.smithi086.stdout:8/968: link d0/d71/d16/d52/d5f/l138 d0/d71/d93/l14f 0 2024-04-06T14:17:23.351 INFO:tasks.workunit.client.1.smithi086.stdout:4/759: truncate d0/fb4 566281 0 2024-04-06T14:17:23.351 INFO:tasks.workunit.client.1.smithi086.stdout:2/965: dread d0/f104 [0,4194304] 0 2024-04-06T14:17:23.353 INFO:tasks.workunit.client.1.smithi086.stdout:3/955: truncate d5/dea/dd2/ddb/de6/ff0 4228692 0 2024-04-06T14:17:23.353 INFO:tasks.workunit.client.1.smithi086.stdout:7/862: creat d7/d10/d11/d1c/d24/d2c/d69/d8d/f136 x:0 0 0 2024-04-06T14:17:23.356 INFO:tasks.workunit.client.1.smithi086.stdout:9/859: rmdir d5/d13/d1a/d5f/d88/d14 39 2024-04-06T14:17:23.357 INFO:tasks.workunit.client.1.smithi086.stdout:1/930: truncate db/d11/d1a/d2a/d9a/d68/d6e/d81/f99 1921822 0 2024-04-06T14:17:23.359 INFO:tasks.workunit.client.1.smithi086.stdout:5/778: sync 2024-04-06T14:17:23.360 INFO:tasks.workunit.client.1.smithi086.stdout:9/860: dread - d5/d18/dff/ff1 zero size 2024-04-06T14:17:23.360 INFO:tasks.workunit.client.1.smithi086.stdout:8/969: creat d0/d71/d16/d52/dd0/f150 x:0 0 0 2024-04-06T14:17:23.362 INFO:tasks.workunit.client.1.smithi086.stdout:8/970: write d0/d71/f9f [364794,28824] 0 2024-04-06T14:17:23.363 INFO:tasks.workunit.client.1.smithi086.stdout:3/956: rename d5/dea/dd2/ddb/de6/l1e to d5/dea/db4/l133 0 2024-04-06T14:17:23.370 INFO:tasks.workunit.client.1.smithi086.stdout:5/779: creat d0/d16/d18/d35/d9e/dd7/ffd x:0 0 0 2024-04-06T14:17:23.371 INFO:tasks.workunit.client.1.smithi086.stdout:9/861: mkdir d5/d13/d1a/d5f/d88/d14/d19/d1b/d115 0 2024-04-06T14:17:23.374 INFO:tasks.workunit.client.1.smithi086.stdout:4/760: dwrite d0/d2c/d38/f5b [0,4194304] 0 2024-04-06T14:17:23.374 INFO:tasks.workunit.client.1.smithi086.stdout:8/971: link d0/f72 d0/d71/d22/d43/d6d/d75/d7d/f151 0 2024-04-06T14:17:23.376 INFO:tasks.workunit.client.1.smithi086.stdout:4/761: readlink d0/d2c/d30/d24/l69 0 2024-04-06T14:17:23.376 INFO:tasks.workunit.client.1.smithi086.stdout:5/780: unlink d0/d16/d18/l34 0 2024-04-06T14:17:23.377 INFO:tasks.workunit.client.1.smithi086.stdout:7/863: write d7/d1d/fb8 [1223634,46866] 0 2024-04-06T14:17:23.379 INFO:tasks.workunit.client.1.smithi086.stdout:8/972: creat d0/d71/d22/d43/d7f/f152 x:0 0 0 2024-04-06T14:17:23.380 INFO:tasks.workunit.client.1.smithi086.stdout:4/762: dread d0/d2c/d30/d24/da3/f52 [0,4194304] 0 2024-04-06T14:17:23.381 INFO:tasks.workunit.client.1.smithi086.stdout:7/864: stat d7/d10/d11/d1c/d24/df0/d116/f12d 0 2024-04-06T14:17:23.382 INFO:tasks.workunit.client.1.smithi086.stdout:3/957: write d5/f8d [1739792,123177] 0 2024-04-06T14:17:23.385 INFO:tasks.workunit.client.1.smithi086.stdout:4/763: chown d0/d2c/d30/d24/d50/c92 13222 1 2024-04-06T14:17:23.385 INFO:tasks.workunit.client.1.smithi086.stdout:9/862: getdents d5/d13/d1a 0 2024-04-06T14:17:23.389 INFO:tasks.workunit.client.1.smithi086.stdout:3/958: write d5/d36/d65/f111 [283508,44950] 0 2024-04-06T14:17:23.390 INFO:tasks.workunit.client.1.smithi086.stdout:5/781: getdents d0/d16/d18/d21/d23/d28/d94/d80/d85 0 2024-04-06T14:17:23.391 INFO:tasks.workunit.client.1.smithi086.stdout:1/931: dwrite db/d11/d1a/d2a/d9a/d68/d6f/fb9 [0,4194304] 0 2024-04-06T14:17:23.393 INFO:tasks.workunit.client.1.smithi086.stdout:9/863: write d5/d13/d47/d5e/fcd [695399,7128] 0 2024-04-06T14:17:23.395 INFO:tasks.workunit.client.1.smithi086.stdout:5/782: rmdir d0/d16/d71/d89 39 2024-04-06T14:17:23.396 INFO:tasks.workunit.client.1.smithi086.stdout:3/959: creat d5/dea/db4/da7/f134 x:0 0 0 2024-04-06T14:17:23.397 INFO:tasks.workunit.client.1.smithi086.stdout:8/973: getdents d0/d3b/d79/dc6/d118 0 2024-04-06T14:17:23.402 INFO:tasks.workunit.client.1.smithi086.stdout:4/764: link d0/d2c/d30/d24/d50/d79/lc8 d0/d2c/d38/dd7/le8 0 2024-04-06T14:17:23.407 INFO:tasks.workunit.client.1.smithi086.stdout:7/865: dwrite d7/d10/d11/d1c/d36/dbc/fbd [0,4194304] 0 2024-04-06T14:17:23.408 INFO:tasks.workunit.client.1.smithi086.stdout:9/864: dread d5/d13/d47/fc4 [0,4194304] 0 2024-04-06T14:17:23.409 INFO:tasks.workunit.client.1.smithi086.stdout:4/765: symlink d0/d2c/d30/d24/d50/le9 0 2024-04-06T14:17:23.411 INFO:tasks.workunit.client.1.smithi086.stdout:1/932: dwrite db/d11/d1a/d2a/d9a/d68/fc0 [0,4194304] 0 2024-04-06T14:17:23.414 INFO:tasks.workunit.client.1.smithi086.stdout:9/865: fdatasync d5/d13/d1a/d5f/d88/d14/f51 0 2024-04-06T14:17:23.414 INFO:tasks.workunit.client.1.smithi086.stdout:7/866: mkdir d7/d10/d102/daf/db5/d7a/d100/d137 0 2024-04-06T14:17:23.417 INFO:tasks.workunit.client.1.smithi086.stdout:4/766: truncate d0/f59 2997542 0 2024-04-06T14:17:23.420 INFO:tasks.workunit.client.1.smithi086.stdout:1/933: creat db/d11/d1f/d5c/f128 x:0 0 0 2024-04-06T14:17:23.423 INFO:tasks.workunit.client.1.smithi086.stdout:7/867: mkdir d7/d10/d11/d1c/d36/dbc/d138 0 2024-04-06T14:17:23.425 INFO:tasks.workunit.client.1.smithi086.stdout:5/783: dwrite d0/d16/d18/d21/d23/d28/d94/f7a [0,4194304] 0 2024-04-06T14:17:23.426 INFO:tasks.workunit.client.1.smithi086.stdout:8/974: dwrite d0/d71/d16/d39/d90/f94 [0,4194304] 0 2024-04-06T14:17:23.433 INFO:tasks.workunit.client.1.smithi086.stdout:8/975: creat d0/d71/d82/f153 x:0 0 0 2024-04-06T14:17:23.434 INFO:tasks.workunit.client.1.smithi086.stdout:9/866: dwrite d5/d13/d47/ff9 [0,4194304] 0 2024-04-06T14:17:23.435 INFO:tasks.workunit.client.1.smithi086.stdout:5/784: mknod d0/d43/da6/cfe 0 2024-04-06T14:17:23.437 INFO:tasks.workunit.client.1.smithi086.stdout:3/960: dread d5/d36/d7d/fec [0,4194304] 0 2024-04-06T14:17:23.438 INFO:tasks.workunit.client.1.smithi086.stdout:1/934: link db/c30 db/c129 0 2024-04-06T14:17:23.438 INFO:tasks.workunit.client.1.smithi086.stdout:5/785: write d0/d16/d18/d35/d9e/de7/f99 [1455166,1098] 0 2024-04-06T14:17:23.441 INFO:tasks.workunit.client.1.smithi086.stdout:1/935: chown db/d13/d55/da3 166179733 1 2024-04-06T14:17:23.441 INFO:tasks.workunit.client.1.smithi086.stdout:4/767: link d0/d15/d20/f57 d0/d2c/d38/d54/d61/d7a/dc2/dd3/fea 0 2024-04-06T14:17:23.441 INFO:tasks.workunit.client.1.smithi086.stdout:1/936: chown f1 1173635 1 2024-04-06T14:17:23.442 INFO:tasks.workunit.client.1.smithi086.stdout:7/868: truncate d7/d10/d11/d1c/f58 8228854 0 2024-04-06T14:17:23.443 INFO:tasks.workunit.client.1.smithi086.stdout:3/961: mknod d5/dea/dd2/c135 0 2024-04-06T14:17:23.444 INFO:tasks.workunit.client.1.smithi086.stdout:1/937: chown db/d11/d1a/d2a/d9a/d68/d6e/d81/f90 20734484 1 2024-04-06T14:17:23.445 INFO:tasks.workunit.client.1.smithi086.stdout:4/768: chown d0/de/l14 337434 1 2024-04-06T14:17:23.445 INFO:tasks.workunit.client.1.smithi086.stdout:7/869: fsync d7/d10/d11/d1c/d24/fb2 0 2024-04-06T14:17:23.447 INFO:tasks.workunit.client.1.smithi086.stdout:5/786: unlink d0/d16/d18/c1d 0 2024-04-06T14:17:23.447 INFO:tasks.workunit.client.1.smithi086.stdout:4/769: chown d0/d2c/d30/d24/fa2 93185276 1 2024-04-06T14:17:23.448 INFO:tasks.workunit.client.1.smithi086.stdout:3/962: dread d5/dea/dd2/ddb/de6/d9e/da5/ff8 [0,4194304] 0 2024-04-06T14:17:23.450 INFO:tasks.workunit.client.1.smithi086.stdout:8/976: link d0/db/c2f d0/d3b/d79/dc6/d123/c154 0 2024-04-06T14:17:23.453 INFO:tasks.workunit.client.1.smithi086.stdout:0/916: sync 2024-04-06T14:17:23.466 INFO:tasks.workunit.client.1.smithi086.stdout:9/867: dwrite d5/d13/d1a/d5f/d88/d14/d19/f76 [4194304,4194304] 0 2024-04-06T14:17:23.466 INFO:tasks.workunit.client.1.smithi086.stdout:3/963: creat d5/d36/d5c/f136 x:0 0 0 2024-04-06T14:17:23.467 INFO:tasks.workunit.client.1.smithi086.stdout:1/938: dread db/d11/d1f/d5c/f112 [0,4194304] 0 2024-04-06T14:17:23.468 INFO:tasks.workunit.client.1.smithi086.stdout:3/964: fdatasync d5/d36/d7d/f126 0 2024-04-06T14:17:23.469 INFO:tasks.workunit.client.1.smithi086.stdout:8/977: write d0/d71/d22/d43/d7f/f152 [902735,17379] 0 2024-04-06T14:17:23.472 INFO:tasks.workunit.client.1.smithi086.stdout:5/787: dread d0/d16/f67 [0,4194304] 0 2024-04-06T14:17:23.476 INFO:tasks.workunit.client.1.smithi086.stdout:5/788: mkdir d0/d16/d18/d21/dff 0 2024-04-06T14:17:23.477 INFO:tasks.workunit.client.1.smithi086.stdout:0/917: creat d0/d15/d4e/da3/d46/d52/dd0/d12f/f130 x:0 0 0 2024-04-06T14:17:23.478 INFO:tasks.workunit.client.1.smithi086.stdout:9/868: dwrite d5/d13/f7d [0,4194304] 0 2024-04-06T14:17:23.479 INFO:tasks.workunit.client.1.smithi086.stdout:8/978: rename d0/d71/d22/d43/l1b to d0/d71/d22/d43/d6d/d75/db2/l155 0 2024-04-06T14:17:23.479 INFO:tasks.workunit.client.1.smithi086.stdout:3/965: dread d5/d36/d5a/d6d/d72/d57/fd1 [0,4194304] 0 2024-04-06T14:17:23.480 INFO:tasks.workunit.client.1.smithi086.stdout:1/939: mknod db/d11/d1a/d2a/d9a/d68/c12a 0 2024-04-06T14:17:23.480 INFO:tasks.workunit.client.1.smithi086.stdout:8/979: fsync d0/d71/d16/ff1 0 2024-04-06T14:17:23.481 INFO:tasks.workunit.client.1.smithi086.stdout:4/770: link d0/de/cad d0/d2c/d38/d54/ceb 0 2024-04-06T14:17:23.482 INFO:tasks.workunit.client.1.smithi086.stdout:0/918: creat d0/d1/d4/de/dae/f131 x:0 0 0 2024-04-06T14:17:23.483 INFO:tasks.workunit.client.1.smithi086.stdout:3/966: symlink d5/dea/dd2/l137 0 2024-04-06T14:17:23.487 INFO:tasks.workunit.client.1.smithi086.stdout:3/967: fsync d5/d33/fd0 0 2024-04-06T14:17:23.488 INFO:tasks.workunit.client.1.smithi086.stdout:1/940: creat db/d11/d1a/d2a/d9a/d68/daf/d104/f12b x:0 0 0 2024-04-06T14:17:23.490 INFO:tasks.workunit.client.1.smithi086.stdout:8/980: dread d0/f72 [0,4194304] 0 2024-04-06T14:17:23.490 INFO:tasks.workunit.client.1.smithi086.stdout:4/771: symlink d0/d2c/d38/dd7/de6/lec 0 2024-04-06T14:17:23.493 INFO:tasks.workunit.client.1.smithi086.stdout:0/919: rename d0/d1/d2/d1a/d51/dd7 to d0/d15/d4e/da3/d44/d5d/d74/d120/d12d/d132 0 2024-04-06T14:17:23.495 INFO:tasks.workunit.client.1.smithi086.stdout:1/941: creat db/d11/d24/f12c x:0 0 0 2024-04-06T14:17:23.496 INFO:tasks.workunit.client.1.smithi086.stdout:1/942: fsync db/d11/d1f/fbc 0 2024-04-06T14:17:23.497 INFO:tasks.workunit.client.1.smithi086.stdout:9/869: link d5/d13/d1a/d5f/d88/l4a d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dfa/l116 0 2024-04-06T14:17:23.499 INFO:tasks.workunit.client.1.smithi086.stdout:9/870: write d5/d13/d1a/d5f/d88/d14/f4e [5149736,6414] 0 2024-04-06T14:17:23.500 INFO:tasks.workunit.client.1.smithi086.stdout:8/981: unlink d0/d71/d16/cfd 0 2024-04-06T14:17:23.500 INFO:tasks.workunit.client.1.smithi086.stdout:5/789: dread d0/d16/f47 [0,4194304] 0 2024-04-06T14:17:23.501 INFO:tasks.workunit.client.1.smithi086.stdout:4/772: mknod d0/d2c/d38/dd7/de6/ced 0 2024-04-06T14:17:23.502 INFO:tasks.workunit.client.1.smithi086.stdout:8/982: fsync d0/d71/d22/d65/d6e/d98/ffe 0 2024-04-06T14:17:23.504 INFO:tasks.workunit.client.1.smithi086.stdout:0/920: unlink d0/d15/d4e/da3/d44/d5d/d74/d120/d12d/d132/de2/f100 0 2024-04-06T14:17:23.505 INFO:tasks.workunit.client.1.smithi086.stdout:8/983: chown d0/d71/d16/f13b 17 1 2024-04-06T14:17:23.506 INFO:tasks.workunit.client.1.smithi086.stdout:0/921: fdatasync d0/d1/d4/de/dae/f12b 0 2024-04-06T14:17:23.507 INFO:tasks.workunit.client.1.smithi086.stdout:9/871: dwrite d5/d13/d1a/d5f/d88/d14/d19/d25/fcb [0,4194304] 0 2024-04-06T14:17:23.508 INFO:tasks.workunit.client.1.smithi086.stdout:0/922: write d0/d1/d2/d1a/d79/ff6 [313531,29299] 0 2024-04-06T14:17:23.510 INFO:tasks.workunit.client.1.smithi086.stdout:3/968: getdents d5/d36/d5c/d8e 0 2024-04-06T14:17:23.510 INFO:tasks.workunit.client.1.smithi086.stdout:9/872: write d5/d13/d1a/d5f/d88/d14/d19/d25/fcb [2786413,39452] 0 2024-04-06T14:17:23.515 INFO:tasks.workunit.client.1.smithi086.stdout:4/773: rename d0/d2c/d30/d24/da3/d88/c9c to d0/d2c/d30/d24/d5d/cee 0 2024-04-06T14:17:23.516 INFO:tasks.workunit.client.1.smithi086.stdout:0/923: dwrite d0/d15/d4e/da3/d44/fbc [0,4194304] 0 2024-04-06T14:17:23.517 INFO:tasks.workunit.client.1.smithi086.stdout:8/984: creat d0/d71/d16/d135/f156 x:0 0 0 2024-04-06T14:17:23.520 INFO:tasks.workunit.client.1.smithi086.stdout:5/790: stat d0/d16/d18/d21/d23/lb2 0 2024-04-06T14:17:23.525 INFO:tasks.workunit.client.1.smithi086.stdout:1/943: link db/d11/d1a/d2a/d9a/d68/d6e/d81/c8b db/d11/d1a/d2a/d9a/d78/c12d 0 2024-04-06T14:17:23.526 INFO:tasks.workunit.client.1.smithi086.stdout:8/985: dwrite d0/d71/d16/d39/d90/f94 [0,4194304] 0 2024-04-06T14:17:23.527 INFO:tasks.workunit.client.1.smithi086.stdout:9/873: creat d5/d13/d47/d5e/dc2/d105/f117 x:0 0 0 2024-04-06T14:17:23.530 INFO:tasks.workunit.client.1.smithi086.stdout:3/969: link d5/d33/d5d/lfd d5/d36/d5a/d6d/d72/d57/l138 0 2024-04-06T14:17:23.531 INFO:tasks.workunit.client.1.smithi086.stdout:3/970: stat d5/dea/lb9 0 2024-04-06T14:17:23.531 INFO:tasks.workunit.client.1.smithi086.stdout:5/791: unlink d0/d16/d71/d89/cc4 0 2024-04-06T14:17:23.532 INFO:tasks.workunit.client.1.smithi086.stdout:3/971: fsync d5/dea/db4/da7/f63 0 2024-04-06T14:17:23.533 INFO:tasks.workunit.client.1.smithi086.stdout:9/874: creat d5/d13/d1a/d5f/d88/d14/d19/d1b/f118 x:0 0 0 2024-04-06T14:17:23.534 INFO:tasks.workunit.client.1.smithi086.stdout:2/966: sync 2024-04-06T14:17:23.536 INFO:tasks.workunit.client.1.smithi086.stdout:1/944: rename db/d11/d1a/d2a/d9a/d68/daf/dc2/fcd to db/d11/d1a/d2a/d9a/d68/d6e/f12e 0 2024-04-06T14:17:23.537 INFO:tasks.workunit.client.1.smithi086.stdout:3/972: creat d5/d36/d5a/d6d/d72/d64/f139 x:0 0 0 2024-04-06T14:17:23.537 INFO:tasks.workunit.client.1.smithi086.stdout:3/973: readlink d5/d33/l106 0 2024-04-06T14:17:23.538 INFO:tasks.workunit.client.1.smithi086.stdout:9/875: unlink d5/d13/d47/d5e/l7e 0 2024-04-06T14:17:23.539 INFO:tasks.workunit.client.1.smithi086.stdout:2/967: mkdir d0/d20/d27/d2b/d35/d36/dbe/d15b 0 2024-04-06T14:17:23.539 INFO:tasks.workunit.client.1.smithi086.stdout:1/945: rmdir db/d11/d42 39 2024-04-06T14:17:23.541 INFO:tasks.workunit.client.1.smithi086.stdout:2/968: write d0/d20/d27/d2b/d35/d36/d4d/d59/f157 [465131,83162] 0 2024-04-06T14:17:23.542 INFO:tasks.workunit.client.1.smithi086.stdout:1/946: chown db/d11/d1a/d2a/d9a/d68/c7e 0 1 2024-04-06T14:17:23.542 INFO:tasks.workunit.client.1.smithi086.stdout:9/876: dread d5/d18/f1d [0,4194304] 0 2024-04-06T14:17:23.544 INFO:tasks.workunit.client.1.smithi086.stdout:5/792: getdents d0/d5e 0 2024-04-06T14:17:23.546 INFO:tasks.workunit.client.1.smithi086.stdout:5/793: read d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc0 [768430,117926] 0 2024-04-06T14:17:23.547 INFO:tasks.workunit.client.1.smithi086.stdout:1/947: mknod db/d11/d1a/d2a/d9a/d68/d6e/d81/c12f 0 2024-04-06T14:17:23.548 INFO:tasks.workunit.client.1.smithi086.stdout:1/948: chown db/d11/d1a/d2a/d9a/d68/daf/dc2/fe2 128455 1 2024-04-06T14:17:23.549 INFO:tasks.workunit.client.1.smithi086.stdout:5/794: symlink d0/d16/d18/d21/l100 0 2024-04-06T14:17:23.550 INFO:tasks.workunit.client.1.smithi086.stdout:2/969: getdents d0/d20/d27/d2b/d35/d36/d4d 0 2024-04-06T14:17:23.554 INFO:tasks.workunit.client.1.smithi086.stdout:5/795: truncate d0/d43/da6/fc3 641521 0 2024-04-06T14:17:23.554 INFO:tasks.workunit.client.1.smithi086.stdout:0/924: dwrite d0/d15/d4e/da3/d44/d5d/fc8 [0,4194304] 0 2024-04-06T14:17:23.555 INFO:tasks.workunit.client.1.smithi086.stdout:1/949: link db/d13/d55/f59 db/d11/d1a/d2a/f130 0 2024-04-06T14:17:23.558 INFO:tasks.workunit.client.1.smithi086.stdout:3/974: write d5/d26/d7f/dae/fcd [3593674,74788] 0 2024-04-06T14:17:23.561 INFO:tasks.workunit.client.1.smithi086.stdout:5/796: rename d0/d5e/f60 to d0/d16/d18/d35/f101 0 2024-04-06T14:17:23.562 INFO:tasks.workunit.client.1.smithi086.stdout:8/986: dwrite d0/d13/f7c [4194304,4194304] 0 2024-04-06T14:17:23.565 INFO:tasks.workunit.client.1.smithi086.stdout:2/970: write d0/d20/d27/d2b/d35/d36/d4d/d6f/f77 [2426137,28034] 0 2024-04-06T14:17:23.565 INFO:tasks.workunit.client.1.smithi086.stdout:3/975: creat d5/dea/dd2/ddb/de6/d10b/f13a x:0 0 0 2024-04-06T14:17:23.566 INFO:tasks.workunit.client.1.smithi086.stdout:5/797: write d0/d16/d18/d21/d23/d28/df7/fcd [3774740,87711] 0 2024-04-06T14:17:23.567 INFO:tasks.workunit.client.1.smithi086.stdout:8/987: rename d0/db/la8 to d0/d71/d22/l157 0 2024-04-06T14:17:23.570 INFO:tasks.workunit.client.1.smithi086.stdout:8/988: readlink d0/d3b/l120 0 2024-04-06T14:17:23.572 INFO:tasks.workunit.client.1.smithi086.stdout:5/798: chown d0/d16/d65/c83 233943599 1 2024-04-06T14:17:23.575 INFO:tasks.workunit.client.1.smithi086.stdout:3/976: write d5/d36/d12c/f117 [1320773,44381] 0 2024-04-06T14:17:23.576 INFO:tasks.workunit.client.1.smithi086.stdout:8/989: mknod d0/d71/d22/d43/d6d/d75/db2/c158 0 2024-04-06T14:17:23.578 INFO:tasks.workunit.client.1.smithi086.stdout:5/799: chown d0/d16/d18/d21/d23/d28/d94/d52/d7c/c91 17269 1 2024-04-06T14:17:23.580 INFO:tasks.workunit.client.1.smithi086.stdout:0/925: dread d0/d15/d4e/da3/d46/d52/f7f [0,4194304] 0 2024-04-06T14:17:23.580 INFO:tasks.workunit.client.1.smithi086.stdout:1/950: dwrite db/d11/d42/d53/f64 [0,4194304] 0 2024-04-06T14:17:23.580 INFO:tasks.workunit.client.1.smithi086.stdout:8/990: fsync d0/d71/d16/d52/dd0/fb4 0 2024-04-06T14:17:23.582 INFO:tasks.workunit.client.1.smithi086.stdout:3/977: symlink d5/dea/dd2/ddb/de6/d10b/d70/l13b 0 2024-04-06T14:17:23.583 INFO:tasks.workunit.client.1.smithi086.stdout:3/978: readlink d5/d26/d7f/l12b 0 2024-04-06T14:17:23.584 INFO:tasks.workunit.client.1.smithi086.stdout:8/991: truncate d0/d71/d22/d43/d6d/d75/dee/f115 255886 0 2024-04-06T14:17:23.584 INFO:tasks.workunit.client.1.smithi086.stdout:5/800: fsync d0/d16/d18/d2a/d50/fc6 0 2024-04-06T14:17:23.586 INFO:tasks.workunit.client.1.smithi086.stdout:5/801: truncate d0/d16/d18/d2a/d50/f93 755714 0 2024-04-06T14:17:23.587 INFO:tasks.workunit.client.1.smithi086.stdout:1/951: mknod db/d11/d1a/d2a/d9a/d68/d6e/d81/c131 0 2024-04-06T14:17:23.588 INFO:tasks.workunit.client.1.smithi086.stdout:3/979: creat d5/d36/d5a/d6d/d72/d64/f13c x:0 0 0 2024-04-06T14:17:23.588 INFO:tasks.workunit.client.1.smithi086.stdout:1/952: stat db/d11/d1a/d79/dce/ce3 0 2024-04-06T14:17:23.589 INFO:tasks.workunit.client.1.smithi086.stdout:5/802: read d0/d16/d18/d21/d23/d28/df7/fcd [2572749,34591] 0 2024-04-06T14:17:23.590 INFO:tasks.workunit.client.1.smithi086.stdout:8/992: creat d0/d71/d16/d39/d90/daf/f159 x:0 0 0 2024-04-06T14:17:23.591 INFO:tasks.workunit.client.1.smithi086.stdout:8/993: fsync d0/d71/d22/d43/d7f/f88 0 2024-04-06T14:17:23.593 INFO:tasks.workunit.client.1.smithi086.stdout:8/994: chown d0/d71/f139 4581892 1 2024-04-06T14:17:23.593 INFO:tasks.workunit.client.1.smithi086.stdout:3/980: mkdir d5/d26/d7f/dae/d11e/d13d 0 2024-04-06T14:17:23.594 INFO:tasks.workunit.client.1.smithi086.stdout:3/981: chown d5/dea/dd2/ddb/de6/c115 19713 1 2024-04-06T14:17:23.594 INFO:tasks.workunit.client.1.smithi086.stdout:1/953: rmdir db/d11/d1a/d79/dce 39 2024-04-06T14:17:23.601 INFO:tasks.workunit.client.1.smithi086.stdout:3/982: dwrite d5/dea/dd2/ddb/de6/d9e/da5/fad [0,4194304] 0 2024-04-06T14:17:23.603 INFO:tasks.workunit.client.1.smithi086.stdout:3/983: write d5/d36/d5a/fd4 [684800,48675] 0 2024-04-06T14:17:23.604 INFO:tasks.workunit.client.1.smithi086.stdout:5/803: link d0/l2c d0/d16/d18/d21/d23/d28/d94/d52/d7c/l102 0 2024-04-06T14:17:23.606 INFO:tasks.workunit.client.1.smithi086.stdout:1/954: write db/d13/f3d [1666335,55610] 0 2024-04-06T14:17:23.608 INFO:tasks.workunit.client.1.smithi086.stdout:3/984: symlink d5/d36/d56/dd3/dfc/l13e 0 2024-04-06T14:17:23.611 INFO:tasks.workunit.client.1.smithi086.stdout:5/804: write d0/d16/d18/d21/d23/f6c [3189523,52207] 0 2024-04-06T14:17:23.614 INFO:tasks.workunit.client.1.smithi086.stdout:3/985: fsync d5/d36/d5a/d6d/d72/d9a/f124 0 2024-04-06T14:17:23.617 INFO:tasks.workunit.client.1.smithi086.stdout:0/926: dwrite d0/d1/d2/d1a/f56 [0,4194304] 0 2024-04-06T14:17:23.618 INFO:tasks.workunit.client.1.smithi086.stdout:5/805: mkdir d0/d16/d18/d21/d23/d28/d94/d52/d103 0 2024-04-06T14:17:23.619 INFO:tasks.workunit.client.1.smithi086.stdout:1/955: dwrite db/d13/f74 [0,4194304] 0 2024-04-06T14:17:23.622 INFO:tasks.workunit.client.1.smithi086.stdout:0/927: chown d0/d15/d4e/da3/d44/d5d/d74 5660353 1 2024-04-06T14:17:23.623 INFO:tasks.workunit.client.1.smithi086.stdout:5/806: creat d0/d43/de5/f104 x:0 0 0 2024-04-06T14:17:23.624 INFO:tasks.workunit.client.1.smithi086.stdout:3/986: read d5/d26/f3d [2155267,62298] 0 2024-04-06T14:17:23.626 INFO:tasks.workunit.client.1.smithi086.stdout:1/956: symlink db/d13/d55/dfa/d10d/l132 0 2024-04-06T14:17:23.629 INFO:tasks.workunit.client.1.smithi086.stdout:0/928: fdatasync d0/d15/fce 0 2024-04-06T14:17:23.633 INFO:tasks.workunit.client.1.smithi086.stdout:3/987: symlink d5/d36/d5c/d8e/dc1/l13f 0 2024-04-06T14:17:23.634 INFO:tasks.workunit.client.1.smithi086.stdout:5/807: dwrite d0/d16/d18/d21/d23/d28/df7/fca [0,4194304] 0 2024-04-06T14:17:23.636 INFO:tasks.workunit.client.1.smithi086.stdout:0/929: creat d0/d15/d4e/da3/d44/d5d/d74/d120/f133 x:0 0 0 2024-04-06T14:17:23.638 INFO:tasks.workunit.client.1.smithi086.stdout:3/988: mknod d5/d36/d5c/c140 0 2024-04-06T14:17:23.639 INFO:tasks.workunit.client.1.smithi086.stdout:1/957: link db/d11/d1a/d2a/d9a/d68/daf/lf3 db/d11/d105/l133 0 2024-04-06T14:17:23.639 INFO:tasks.workunit.client.1.smithi086.stdout:3/989: write d5/dea/dd2/ddb/de6/ff0 [3471956,88454] 0 2024-04-06T14:17:23.641 INFO:tasks.workunit.client.1.smithi086.stdout:5/808: creat d0/d16/d18/d21/f105 x:0 0 0 2024-04-06T14:17:23.643 INFO:tasks.workunit.client.1.smithi086.stdout:1/958: creat db/d11/d42/d53/f134 x:0 0 0 2024-04-06T14:17:23.644 INFO:tasks.workunit.client.1.smithi086.stdout:5/809: creat d0/d16/d18/d35/d9e/de7/f106 x:0 0 0 2024-04-06T14:17:23.645 INFO:tasks.workunit.client.1.smithi086.stdout:1/959: write db/d13/d71/f88 [89903,80486] 0 2024-04-06T14:17:23.646 INFO:tasks.workunit.client.1.smithi086.stdout:3/990: mkdir d5/d26/d141 0 2024-04-06T14:17:23.647 INFO:tasks.workunit.client.1.smithi086.stdout:5/810: write d0/d16/d18/d35/d9e/dad/fed [597590,84524] 0 2024-04-06T14:17:23.649 INFO:tasks.workunit.client.1.smithi086.stdout:1/960: creat db/d11/d1a/d2a/f135 x:0 0 0 2024-04-06T14:17:23.649 INFO:tasks.workunit.client.1.smithi086.stdout:3/991: rmdir d5/d36/d65 39 2024-04-06T14:17:23.650 INFO:tasks.workunit.client.1.smithi086.stdout:1/961: fdatasync db/d13/f1e 0 2024-04-06T14:17:23.655 INFO:tasks.workunit.client.0.smithi032.stdout:0/925: sync 2024-04-06T14:17:23.655 INFO:tasks.workunit.client.1.smithi086.stdout:5/811: rename d0/d43/de5/c1e to d0/d16/d18/d21/d23/d28/d94/d52/d7c/c107 0 2024-04-06T14:17:23.657 INFO:tasks.workunit.client.1.smithi086.stdout:1/962: creat db/d13/d55/dfa/f136 x:0 0 0 2024-04-06T14:17:23.657 INFO:tasks.workunit.client.1.smithi086.stdout:7/870: sync 2024-04-06T14:17:23.662 INFO:tasks.workunit.client.1.smithi086.stdout:0/930: write d0/d1/d2/d1a/d79/f8c [272583,36075] 0 2024-04-06T14:17:23.662 INFO:tasks.workunit.client.1.smithi086.stdout:7/871: truncate d7/d10/d102/daf/db5/da1/f10e 544453 0 2024-04-06T14:17:23.662 INFO:tasks.workunit.client.0.smithi032.stdout:0/926: dwrite d3/dc2/fda [0,4194304] 0 2024-04-06T14:17:23.663 INFO:tasks.workunit.client.0.smithi032.stdout:0/927: read d3/d83/ff6 [3070710,92519] 0 2024-04-06T14:17:23.663 INFO:tasks.workunit.client.1.smithi086.stdout:5/812: symlink d0/d16/d18/d21/d23/d28/df7/l108 0 2024-04-06T14:17:23.664 INFO:tasks.workunit.client.1.smithi086.stdout:1/963: rename fa to db/d11/d1f/d5c/f137 0 2024-04-06T14:17:23.666 INFO:tasks.workunit.client.1.smithi086.stdout:1/964: chown db/d11/d1a/d2a/d9a/d68/d6e/d81/l10f 3636 1 2024-04-06T14:17:23.667 INFO:tasks.workunit.client.1.smithi086.stdout:5/813: creat d0/d16/d71/d96/f109 x:0 0 0 2024-04-06T14:17:23.668 INFO:tasks.workunit.client.1.smithi086.stdout:1/965: write db/fe [1624014,32485] 0 2024-04-06T14:17:23.669 INFO:tasks.workunit.client.1.smithi086.stdout:0/931: dwrite d0/d1/dfe/f114 [0,4194304] 0 2024-04-06T14:17:23.671 INFO:tasks.workunit.client.1.smithi086.stdout:7/872: creat d7/d10/d102/daf/db5/d7a/d100/d137/f139 x:0 0 0 2024-04-06T14:17:23.671 INFO:tasks.workunit.client.0.smithi032.stdout:0/928: truncate d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/d9c/fc7 1548583 0 2024-04-06T14:17:23.672 INFO:tasks.workunit.client.1.smithi086.stdout:5/814: mkdir d0/d16/d65/d10a 0 2024-04-06T14:17:23.673 INFO:tasks.workunit.client.0.smithi032.stdout:0/929: chown d3/da/d25/d2b/d4a/d79/d97/de2 36074866 1 2024-04-06T14:17:23.673 INFO:tasks.workunit.client.1.smithi086.stdout:0/932: dread - d0/d15/d26/d33/d9f/f101 zero size 2024-04-06T14:17:23.679 INFO:tasks.workunit.client.1.smithi086.stdout:5/815: rmdir d0/d16/d18/d2a/d50 39 2024-04-06T14:17:23.679 INFO:tasks.workunit.client.1.smithi086.stdout:0/933: creat d0/d1/d4/de/db4/f134 x:0 0 0 2024-04-06T14:17:23.681 INFO:tasks.workunit.client.0.smithi032.stdout:0/930: dwrite d3/da/d25/f94 [0,4194304] 0 2024-04-06T14:17:23.685 INFO:tasks.workunit.client.1.smithi086.stdout:5/816: getdents d0/d16/d71/d96 0 2024-04-06T14:17:23.686 INFO:tasks.workunit.client.1.smithi086.stdout:3/992: dwrite d5/dea/dd2/ddb/de6/f6b [0,4194304] 0 2024-04-06T14:17:23.686 INFO:tasks.workunit.client.1.smithi086.stdout:0/934: dread - d0/d15/d26/d33/d9f/fab zero size 2024-04-06T14:17:23.688 INFO:tasks.workunit.client.0.smithi032.stdout:0/931: creat d3/da/d25/d2b/f135 x:0 0 0 2024-04-06T14:17:23.691 INFO:tasks.workunit.client.1.smithi086.stdout:1/966: dwrite db/d11/d1a/d79/f9d [0,4194304] 0 2024-04-06T14:17:23.694 INFO:tasks.workunit.client.1.smithi086.stdout:0/935: read d0/d15/d4e/da3/d44/d5d/d74/d120/ff7 [158757,85437] 0 2024-04-06T14:17:23.697 INFO:tasks.workunit.client.1.smithi086.stdout:1/967: write db/d11/d1a/d2a/d9a/d78/fed [667146,9034] 0 2024-04-06T14:17:23.698 INFO:tasks.workunit.client.0.smithi032.stdout:0/932: creat d3/d83/dfd/f136 x:0 0 0 2024-04-06T14:17:23.701 INFO:tasks.workunit.client.1.smithi086.stdout:3/993: dwrite d5/dea/dd2/ddb/de6/f2c [8388608,4194304] 0 2024-04-06T14:17:23.702 INFO:tasks.workunit.client.1.smithi086.stdout:5/817: dwrite d0/d16/d18/d35/fa2 [0,4194304] 0 2024-04-06T14:17:23.714 INFO:tasks.workunit.client.1.smithi086.stdout:5/818: read d0/f6 [355272,59734] 0 2024-04-06T14:17:23.720 INFO:tasks.workunit.client.1.smithi086.stdout:3/994: dwrite d5/d36/d5a/d6d/d72/f10a [0,4194304] 0 2024-04-06T14:17:23.720 INFO:tasks.workunit.client.1.smithi086.stdout:0/936: dwrite d0/d15/d4e/da3/d46/d52/fe8 [4194304,4194304] 0 2024-04-06T14:17:23.721 INFO:tasks.workunit.client.1.smithi086.stdout:1/968: dwrite db/d11/d24/fba [4194304,4194304] 0 2024-04-06T14:17:23.728 INFO:tasks.workunit.client.1.smithi086.stdout:5/819: truncate d0/d16/d18/d21/d23/d28/df7/fcd 413108 0 2024-04-06T14:17:23.729 INFO:tasks.workunit.client.1.smithi086.stdout:0/937: symlink d0/d15/d4e/da3/l135 0 2024-04-06T14:17:23.729 INFO:tasks.workunit.client.1.smithi086.stdout:1/969: symlink db/d11/d42/dee/l138 0 2024-04-06T14:17:23.733 INFO:tasks.workunit.client.1.smithi086.stdout:5/820: creat d0/d16/d18/d21/dff/f10b x:0 0 0 2024-04-06T14:17:23.734 INFO:tasks.workunit.client.1.smithi086.stdout:1/970: mknod db/d13/c139 0 2024-04-06T14:17:23.736 INFO:tasks.workunit.client.1.smithi086.stdout:1/971: write db/d11/d1f/d5c/ddb/de9/f107 [1403418,25176] 0 2024-04-06T14:17:23.737 INFO:tasks.workunit.client.1.smithi086.stdout:5/821: read d0/d16/d18/d2a/d50/fc6 [3020525,20458] 0 2024-04-06T14:17:23.743 INFO:tasks.workunit.client.1.smithi086.stdout:5/822: dwrite d0/d16/d18/d21/d23/d28/d94/d8a/fbd [0,4194304] 0 2024-04-06T14:17:23.747 INFO:tasks.workunit.client.1.smithi086.stdout:5/823: read d0/d43/de5/f4b [1078106,23858] 0 2024-04-06T14:17:23.752 INFO:tasks.workunit.client.1.smithi086.stdout:3/995: truncate d5/dea/db4/fd8 2815095 0 2024-04-06T14:17:23.754 INFO:tasks.workunit.client.1.smithi086.stdout:5/824: rename d0/d16/d71/d96/lc1 to d0/d16/d71/l10c 0 2024-04-06T14:17:23.756 INFO:tasks.workunit.client.1.smithi086.stdout:3/996: symlink d5/d36/d5a/d6d/d72/d108/l142 0 2024-04-06T14:17:23.757 INFO:tasks.workunit.client.0.smithi032.stdout:0/933: dwrite d3/f4 [0,4194304] 0 2024-04-06T14:17:23.758 INFO:tasks.workunit.client.1.smithi086.stdout:0/938: write d0/d15/d4e/da3/d44/d5d/d74/d120/d12d/d132/de2/f14 [2352334,14234] 0 2024-04-06T14:17:23.759 INFO:tasks.workunit.client.1.smithi086.stdout:5/825: symlink d0/d16/d18/d21/d23/l10d 0 2024-04-06T14:17:23.760 INFO:tasks.workunit.client.0.smithi032.stdout:0/934: mkdir d3/da/d25/d57/d137 0 2024-04-06T14:17:23.761 INFO:tasks.workunit.client.1.smithi086.stdout:1/972: write db/d11/d1a/d2a/f130 [4040667,58491] 0 2024-04-06T14:17:23.764 INFO:tasks.workunit.client.0.smithi032.stdout:0/935: symlink d3/da/d25/d2b/d4a/d79/d97/de2/l138 0 2024-04-06T14:17:23.766 INFO:tasks.workunit.client.0.smithi032.stdout:0/936: mknod d3/dc2/d123/c139 0 2024-04-06T14:17:23.767 INFO:tasks.workunit.client.1.smithi086.stdout:3/997: getdents d5/d36/d5a/d6d 0 2024-04-06T14:17:23.767 INFO:tasks.workunit.client.1.smithi086.stdout:0/939: link d0/d1/d2/d1a/d51/cb8 d0/d15/d26/d33/d9f/c136 0 2024-04-06T14:17:23.767 INFO:tasks.workunit.client.0.smithi032.stdout:0/937: dread - d3/da/d25/d2b/d4a/d86/dd6/f11d zero size 2024-04-06T14:17:23.769 INFO:tasks.workunit.client.1.smithi086.stdout:1/973: rename db/c49 to db/d11/d1a/c13a 0 2024-04-06T14:17:23.770 INFO:tasks.workunit.client.0.smithi032.stdout:0/938: getdents d3/da/d25/d2b/d80/d100 0 2024-04-06T14:17:23.771 INFO:tasks.workunit.client.1.smithi086.stdout:5/826: dwrite d0/f6 [4194304,4194304] 0 2024-04-06T14:17:23.771 INFO:tasks.workunit.client.1.smithi086.stdout:3/998: write d5/dea/dd2/ddb/de6/ff0 [761295,20736] 0 2024-04-06T14:17:23.771 INFO:tasks.workunit.client.1.smithi086.stdout:1/974: readlink db/d11/d42/d53/lae 0 2024-04-06T14:17:23.772 INFO:tasks.workunit.client.0.smithi032.stdout:0/939: write d3/da/d41/d107/d125/f12e [85603,110657] 0 2024-04-06T14:17:23.773 INFO:tasks.workunit.client.1.smithi086.stdout:0/940: mkdir d0/d15/d4e/da3/d73/d137 0 2024-04-06T14:17:23.773 INFO:tasks.workunit.client.1.smithi086.stdout:1/975: fsync db/d11/d1f/d5c/ddb/de9/ff9 0 2024-04-06T14:17:23.776 INFO:tasks.workunit.client.0.smithi032.stdout:0/940: creat d3/da/d25/d2b/d80/db3/db7/f13a x:0 0 0 2024-04-06T14:17:23.780 INFO:tasks.workunit.client.1.smithi086.stdout:3/999: rename d5/d36/d56/dd3 to d5/d36/d7d/df6/d143 0 2024-04-06T14:17:23.780 INFO:tasks.workunit.client.1.smithi086.stdout:0/941: rmdir d0/d1/d2/d1a/d79/dac/db7 39 2024-04-06T14:17:23.781 INFO:tasks.workunit.client.1.smithi086.stdout:4/774: sync 2024-04-06T14:17:23.782 INFO:tasks.workunit.client.1.smithi086.stdout:1/976: creat db/d11/d42/d53/f13b x:0 0 0 2024-04-06T14:17:23.783 INFO:tasks.workunit.client.1.smithi086.stdout:9/877: sync 2024-04-06T14:17:23.783 INFO:tasks.workunit.client.1.smithi086.stdout:1/977: chown db/fe 33448458 1 2024-04-06T14:17:23.786 INFO:tasks.workunit.client.1.smithi086.stdout:4/775: dread d0/d2c/d38/d54/f80 [0,4194304] 0 2024-04-06T14:17:23.787 INFO:tasks.workunit.client.0.smithi032.stdout:0/941: truncate d3/d83/fc5 1636155 0 2024-04-06T14:17:23.787 INFO:tasks.workunit.client.1.smithi086.stdout:0/942: symlink d0/d15/d4e/da3/d44/d5d/d74/d120/d110/l138 0 2024-04-06T14:17:23.788 INFO:tasks.workunit.client.1.smithi086.stdout:5/827: dread d0/d16/d18/d21/d23/d28/d94/d52/d7c/f8f [0,4194304] 0 2024-04-06T14:17:23.789 INFO:tasks.workunit.client.0.smithi032.stdout:0/942: mkdir d3/da/d25/d2b/d80/db3/dcd/d13b 0 2024-04-06T14:17:23.793 INFO:tasks.workunit.client.1.smithi086.stdout:4/776: creat d0/d15/d20/d74/fef x:0 0 0 2024-04-06T14:17:23.793 INFO:tasks.workunit.client.1.smithi086.stdout:1/978: dwrite db/d11/d1a/d79/da5/f127 [0,4194304] 0 2024-04-06T14:17:23.793 INFO:tasks.workunit.client.1.smithi086.stdout:9/878: mkdir d5/d13/d1a/d5f/d119 0 2024-04-06T14:17:23.797 INFO:tasks.workunit.client.0.smithi032.stdout:0/943: write d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/f90 [4336769,46554] 0 2024-04-06T14:17:23.799 INFO:tasks.workunit.client.1.smithi086.stdout:9/879: fdatasync d5/d13/d1a/d5f/d88/d14/d19/fa6 0 2024-04-06T14:17:23.799 INFO:tasks.workunit.client.1.smithi086.stdout:5/828: creat d0/d16/d18/d35/d9e/de7/f10e x:0 0 0 2024-04-06T14:17:23.801 INFO:tasks.workunit.client.1.smithi086.stdout:4/777: mknod d0/d15/cf0 0 2024-04-06T14:17:23.802 INFO:tasks.workunit.client.1.smithi086.stdout:5/829: unlink d0/d16/le2 0 2024-04-06T14:17:23.804 INFO:tasks.workunit.client.1.smithi086.stdout:1/979: dread db/d11/d42/d53/f64 [0,4194304] 0 2024-04-06T14:17:23.805 INFO:tasks.workunit.client.1.smithi086.stdout:0/943: rename d0/d15/d4e/da3/d46/d52/dd0/d12f/f130 to d0/d15/d26/f139 0 2024-04-06T14:17:23.806 INFO:tasks.workunit.client.1.smithi086.stdout:4/778: creat d0/d2c/d38/d54/d61/d7a/ff1 x:0 0 0 2024-04-06T14:17:23.808 INFO:tasks.workunit.client.1.smithi086.stdout:5/830: mkdir d0/d16/d65/d10f 0 2024-04-06T14:17:23.808 INFO:tasks.workunit.client.1.smithi086.stdout:4/779: dread - d0/d2c/d38/d54/d83/fd1 zero size 2024-04-06T14:17:23.810 INFO:tasks.workunit.client.1.smithi086.stdout:1/980: truncate db/d11/d42/d53/f57 2292723 0 2024-04-06T14:17:23.811 INFO:tasks.workunit.client.1.smithi086.stdout:0/944: dread d0/d1/d2/d1a/d79/fb3 [0,4194304] 0 2024-04-06T14:17:23.812 INFO:tasks.workunit.client.0.smithi032.stdout:0/944: dwrite d3/da/d25/d101/fb2 [0,4194304] 0 2024-04-06T14:17:23.812 INFO:tasks.workunit.client.1.smithi086.stdout:0/945: write d0/d1/dfe/f114 [3005753,93445] 0 2024-04-06T14:17:23.815 INFO:tasks.workunit.client.0.smithi032.stdout:0/945: creat d3/f13c x:0 0 0 2024-04-06T14:17:23.815 INFO:tasks.workunit.client.1.smithi086.stdout:1/981: dread db/d11/d1a/d2a/d9a/d68/d6e/f12e [0,4194304] 0 2024-04-06T14:17:23.816 INFO:tasks.workunit.client.0.smithi032.stdout:0/946: chown d3/da/f56 1 1 2024-04-06T14:17:23.816 INFO:tasks.workunit.client.1.smithi086.stdout:1/982: chown db/d11/d1a/d2a/f43 2794 1 2024-04-06T14:17:23.816 INFO:tasks.workunit.client.1.smithi086.stdout:4/780: rename d0/d2c/d38/d54/d61/d7a/c7e to d0/db0/ddd/cf2 0 2024-04-06T14:17:23.818 INFO:tasks.workunit.client.1.smithi086.stdout:1/983: read db/d11/d42/d53/f64 [1072501,17185] 0 2024-04-06T14:17:23.818 INFO:tasks.workunit.client.1.smithi086.stdout:4/781: write d0/de/d4c/fba [697295,110770] 0 2024-04-06T14:17:23.818 INFO:tasks.workunit.client.1.smithi086.stdout:9/880: link d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f81 d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dfa/f11a 0 2024-04-06T14:17:23.819 INFO:tasks.workunit.client.1.smithi086.stdout:4/782: dread - d0/d2c/d30/faf zero size 2024-04-06T14:17:23.821 INFO:tasks.workunit.client.1.smithi086.stdout:0/946: rename d0/d1/d2/d1a/ca2 to d0/d15/d4e/da3/d44/d10f/c13a 0 2024-04-06T14:17:23.822 INFO:tasks.workunit.client.1.smithi086.stdout:4/783: dread - d0/d2c/d38/d54/d61/d7a/ff1 zero size 2024-04-06T14:17:23.823 INFO:tasks.workunit.client.1.smithi086.stdout:0/947: read d0/d1/d4/f6b [788634,60545] 0 2024-04-06T14:17:23.825 INFO:tasks.workunit.client.1.smithi086.stdout:4/784: readlink d0/d2c/d38/dd7/de6/lec 0 2024-04-06T14:17:23.826 INFO:tasks.workunit.client.1.smithi086.stdout:9/881: dread d5/d18/dff/f9a [0,4194304] 0 2024-04-06T14:17:23.828 INFO:tasks.workunit.client.1.smithi086.stdout:1/984: dwrite db/d11/d1a/d79/da5/fe4 [0,4194304] 0 2024-04-06T14:17:23.830 INFO:tasks.workunit.client.1.smithi086.stdout:4/785: mknod d0/cf3 0 2024-04-06T14:17:23.831 INFO:tasks.workunit.client.1.smithi086.stdout:0/948: mkdir d0/d15/d4e/da3/d46/d52/dd0/d13b 0 2024-04-06T14:17:23.833 INFO:tasks.workunit.client.1.smithi086.stdout:9/882: chown d5/d13/d1a/d5f/d88/c9c 2359 1 2024-04-06T14:17:23.834 INFO:tasks.workunit.client.1.smithi086.stdout:9/883: dread - d5/d13/d47/d5e/f93 zero size 2024-04-06T14:17:23.835 INFO:tasks.workunit.client.1.smithi086.stdout:1/985: rename db/d11/d1f to db/d11/d1a/d2a/d9a/d68/daf/dc2/dd8/d13c 0 2024-04-06T14:17:23.838 INFO:tasks.workunit.client.1.smithi086.stdout:5/831: truncate d0/d16/d18/fec 865549 0 2024-04-06T14:17:23.840 INFO:tasks.workunit.client.0.smithi032.stdout:0/947: dread d3/d83/fc5 [0,4194304] 0 2024-04-06T14:17:23.840 INFO:tasks.workunit.client.1.smithi086.stdout:0/949: creat d0/d1/d4/de/df9/f13c x:0 0 0 2024-04-06T14:17:23.843 INFO:tasks.workunit.client.1.smithi086.stdout:9/884: mknod d5/d13/d47/d55/d87/ddc/c11b 0 2024-04-06T14:17:23.843 INFO:tasks.workunit.client.0.smithi032.stdout:0/948: dread d3/da/d25/d36/f51 [0,4194304] 0 2024-04-06T14:17:23.843 INFO:tasks.workunit.client.1.smithi086.stdout:9/885: readlink d5/d13/d1a/l3a 0 2024-04-06T14:17:23.845 INFO:tasks.workunit.client.1.smithi086.stdout:4/786: rename d0/d2c/d38/d54/d83/f98 to d0/d2c/d38/d54/d61/d7a/dc2/ff4 0 2024-04-06T14:17:23.845 INFO:tasks.workunit.client.0.smithi032.stdout:0/949: dread - d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/fea zero size 2024-04-06T14:17:23.845 INFO:tasks.workunit.client.1.smithi086.stdout:1/986: mknod db/d11/d24/c13d 0 2024-04-06T14:17:23.847 INFO:tasks.workunit.client.1.smithi086.stdout:0/950: rmdir d0/d15/d4e/da3/d46/d52 39 2024-04-06T14:17:23.848 INFO:tasks.workunit.client.1.smithi086.stdout:0/951: chown d0/d1/d2/f67 171837 1 2024-04-06T14:17:23.848 INFO:tasks.workunit.client.1.smithi086.stdout:9/886: mknod d5/d13/d47/d5e/dc2/c11c 0 2024-04-06T14:17:23.849 INFO:tasks.workunit.client.0.smithi032.stdout:0/950: rename d3/da/d25/d2b/d80/db3/dcd/cfa to d3/da/d25/d2b/d4a/d79/d97/de0/c13d 0 2024-04-06T14:17:23.849 INFO:tasks.workunit.client.1.smithi086.stdout:4/787: symlink d0/d2c/d38/lf5 0 2024-04-06T14:17:23.850 INFO:tasks.workunit.client.1.smithi086.stdout:9/887: write d5/d13/d1a/d5f/d88/f53 [2958421,54136] 0 2024-04-06T14:17:23.852 INFO:tasks.workunit.client.1.smithi086.stdout:9/888: dread - d5/d13/d47/d5e/f110 zero size 2024-04-06T14:17:23.857 INFO:tasks.workunit.client.0.smithi032.stdout:0/951: dwrite d3/f5d [4194304,4194304] 0 2024-04-06T14:17:23.858 INFO:tasks.workunit.client.0.smithi032.stdout:0/952: fdatasync d3/da/d25/d2b/f135 0 2024-04-06T14:17:23.862 INFO:tasks.workunit.client.1.smithi086.stdout:9/889: creat d5/d13/d1a/d5f/d88/d14/d19/f11d x:0 0 0 2024-04-06T14:17:23.865 INFO:tasks.workunit.client.1.smithi086.stdout:9/890: dread - d5/d13/d1a/d5f/d88/d14/ff8 zero size 2024-04-06T14:17:23.866 INFO:tasks.workunit.client.1.smithi086.stdout:0/952: truncate d0/d15/d4e/d48/f31 3656133 0 2024-04-06T14:17:23.866 INFO:tasks.workunit.client.1.smithi086.stdout:4/788: truncate d0/d2c/d30/d24/d50/d79/fa8 62836 0 2024-04-06T14:17:23.866 INFO:tasks.workunit.client.1.smithi086.stdout:0/953: read - d0/d1/d2/d5b/fed zero size 2024-04-06T14:17:23.868 INFO:tasks.workunit.client.0.smithi032.stdout:0/953: dwrite d3/da/d25/d57/fd7 [0,4194304] 0 2024-04-06T14:17:23.868 INFO:tasks.workunit.client.1.smithi086.stdout:9/891: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/f11e x:0 0 0 2024-04-06T14:17:23.868 INFO:tasks.workunit.client.1.smithi086.stdout:5/832: write d0/d16/d18/fec [937620,36009] 0 2024-04-06T14:17:23.870 INFO:tasks.workunit.client.1.smithi086.stdout:4/789: mkdir d0/d2c/d30/d24/df6 0 2024-04-06T14:17:23.871 INFO:tasks.workunit.client.1.smithi086.stdout:4/790: fdatasync d0/de/f26 0 2024-04-06T14:17:23.871 INFO:tasks.workunit.client.0.smithi032.stdout:0/954: dread - d3/da/d25/d2b/fdc zero size 2024-04-06T14:17:23.871 INFO:tasks.workunit.client.1.smithi086.stdout:0/954: symlink d0/d1/d4/l13d 0 2024-04-06T14:17:23.872 INFO:tasks.workunit.client.1.smithi086.stdout:4/791: chown d0/d2c/d38/d54/lbc 841 1 2024-04-06T14:17:23.873 INFO:tasks.workunit.client.0.smithi032.stdout:0/955: write d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/f90 [2987435,48785] 0 2024-04-06T14:17:23.876 INFO:tasks.workunit.client.1.smithi086.stdout:9/892: creat d5/d13/d1a/d5f/d88/d14/d19/d1b/d115/f11f x:0 0 0 2024-04-06T14:17:23.877 INFO:tasks.workunit.client.1.smithi086.stdout:4/792: rename d0/l4e to d0/d2c/d30/d24/df6/lf7 0 2024-04-06T14:17:23.878 INFO:tasks.workunit.client.1.smithi086.stdout:5/833: creat d0/d16/d18/d21/d23/d28/d94/d8a/dbf/f110 x:0 0 0 2024-04-06T14:17:23.879 INFO:tasks.workunit.client.1.smithi086.stdout:0/955: mkdir d0/d15/d26/d33/def/d13e 0 2024-04-06T14:17:23.879 INFO:tasks.workunit.client.1.smithi086.stdout:9/893: fsync d5/d13/d1a/d5f/d88/d14/ff8 0 2024-04-06T14:17:23.882 INFO:tasks.workunit.client.0.smithi032.stdout:0/956: dwrite d3/d83/dfd/f136 [0,4194304] 0 2024-04-06T14:17:23.887 INFO:tasks.workunit.client.0.smithi032.stdout:0/957: mkdir d3/da/d41/d13e 0 2024-04-06T14:17:23.889 INFO:tasks.workunit.client.1.smithi086.stdout:5/834: fdatasync d0/d16/d18/d21/d23/d28/d94/d52/fb1 0 2024-04-06T14:17:23.890 INFO:tasks.workunit.client.1.smithi086.stdout:5/835: fsync d0/d43/de5/ff6 0 2024-04-06T14:17:23.893 INFO:tasks.workunit.client.0.smithi032.stdout:0/958: link d3/da/d25/d2b/d4a/d79/d97/de2/l138 d3/da/d25/d2b/d4a/d130/l13f 0 2024-04-06T14:17:23.902 INFO:tasks.workunit.client.0.smithi032.stdout:0/959: write d3/da/d25/d101/fae [409702,108271] 0 2024-04-06T14:17:23.904 INFO:tasks.workunit.client.0.smithi032.stdout:0/960: chown d3/da/d41/d107/l11a 122026 1 2024-04-06T14:17:23.905 INFO:tasks.workunit.client.0.smithi032.stdout:0/961: chown d3/da/d25/d2b/d4a/d79/d97/de0/df5/l60 16 1 2024-04-06T14:17:23.906 INFO:tasks.workunit.client.0.smithi032.stdout:0/962: stat d3/da/d25/d2b/d4a/d79/da8/f9f 0 2024-04-06T14:17:23.908 INFO:tasks.workunit.client.1.smithi086.stdout:0/956: dread d0/d15/d4e/da3/f39 [0,4194304] 0 2024-04-06T14:17:23.909 INFO:tasks.workunit.client.1.smithi086.stdout:4/793: symlink d0/d2c/d30/d24/d71/lf8 0 2024-04-06T14:17:23.909 INFO:tasks.workunit.client.0.smithi032.stdout:0/963: symlink d3/da/d41/d107/d125/l140 0 2024-04-06T14:17:23.909 INFO:tasks.workunit.client.1.smithi086.stdout:9/894: mkdir d5/d13/d47/d55/d87/ddc/d120 0 2024-04-06T14:17:23.911 INFO:tasks.workunit.client.1.smithi086.stdout:5/836: chown d0/d16/d18/c6a 48473 1 2024-04-06T14:17:23.912 INFO:tasks.workunit.client.1.smithi086.stdout:4/794: unlink d0/d2c/d30/d24/d71/lf8 0 2024-04-06T14:17:23.913 INFO:tasks.workunit.client.0.smithi032.stdout:0/964: getdents d3/da/d25/d2b/d4a/d86 0 2024-04-06T14:17:23.916 INFO:tasks.workunit.client.1.smithi086.stdout:9/895: rename d5/d13/d1a/d5f/d88/d14/d19/d25/l106 to d5/d13/d47/d55/l121 0 2024-04-06T14:17:23.917 INFO:tasks.workunit.client.1.smithi086.stdout:0/957: dwrite d0/d15/d4e/da3/d44/df4/f116 [0,4194304] 0 2024-04-06T14:17:23.917 INFO:tasks.workunit.client.0.smithi032.stdout:0/965: write d3/dc2/dd8/fde [230004,85018] 0 2024-04-06T14:17:23.919 INFO:tasks.workunit.client.1.smithi086.stdout:0/958: dread - d0/d1/d4/de/db4/fb5 zero size 2024-04-06T14:17:23.920 INFO:tasks.workunit.client.1.smithi086.stdout:0/959: chown d0/d15/d4e/da3/d46 3619 1 2024-04-06T14:17:23.922 INFO:tasks.workunit.client.1.smithi086.stdout:4/795: rename d0/d2c/d38/l4f to d0/de/d4c/lf9 0 2024-04-06T14:17:23.922 INFO:tasks.workunit.client.0.smithi032.stdout:0/966: dread d3/da/f23 [0,4194304] 0 2024-04-06T14:17:23.923 INFO:tasks.workunit.client.1.smithi086.stdout:0/960: chown d0/d15/d4e/da3/d44/d5d/d74/d120/d12d/d132/de2/l59 53833 1 2024-04-06T14:17:23.924 INFO:tasks.workunit.client.1.smithi086.stdout:9/896: creat d5/d13/d1a/d5f/d88/d14/d90/f122 x:0 0 0 2024-04-06T14:17:23.925 INFO:tasks.workunit.client.1.smithi086.stdout:0/961: write d0/d15/f11c [228414,28205] 0 2024-04-06T14:17:23.926 INFO:tasks.workunit.client.1.smithi086.stdout:0/962: fsync d0/d1/d4/de/dae/f119 0 2024-04-06T14:17:23.927 INFO:tasks.workunit.client.0.smithi032.stdout:0/967: creat d3/da/d41/d107/f141 x:0 0 0 2024-04-06T14:17:23.931 INFO:tasks.workunit.client.1.smithi086.stdout:5/837: dread d0/d43/f5c [0,4194304] 0 2024-04-06T14:17:23.932 INFO:tasks.workunit.client.1.smithi086.stdout:0/963: symlink d0/d1/d2/d1a/d79/dac/db7/de6/d129/l13f 0 2024-04-06T14:17:23.945 INFO:tasks.workunit.client.0.smithi032.stdout:0/968: dread d3/da/d25/d2b/d4a/d79/d97/de0/df5/d8b/f90 [0,4194304] 0 2024-04-06T14:17:23.946 INFO:tasks.workunit.client.1.smithi086.stdout:2/971: sync 2024-04-06T14:17:23.948 INFO:tasks.workunit.client.0.smithi032.stdout:0/969: rename d3/da/f6a to d3/da/d25/d2b/d80/db3/df1/d120/f142 0 2024-04-06T14:17:23.948 INFO:tasks.workunit.client.0.smithi032.stdout:0/970: stat d3/da/d25/d2b/d4a/d86/f114 0 2024-04-06T14:17:23.950 INFO:tasks.workunit.client.0.smithi032.stdout:0/971: fdatasync d3/fd 0 2024-04-06T14:17:23.953 INFO:tasks.workunit.client.0.smithi032.stdout:0/972: creat d3/da/d25/d2b/d80/db3/f143 x:0 0 0 2024-04-06T14:17:23.960 INFO:tasks.workunit.client.1.smithi086.stdout:0/964: creat d0/d15/d4e/da3/f140 x:0 0 0 2024-04-06T14:17:23.967 INFO:tasks.workunit.client.1.smithi086.stdout:0/965: truncate d0/d15/d26/fa0 1172390 0 2024-04-06T14:17:23.967 INFO:tasks.workunit.client.1.smithi086.stdout:5/838: dwrite d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc0 [4194304,4194304] 0 2024-04-06T14:17:23.969 INFO:tasks.workunit.client.0.smithi032.stdout:0/973: dread d3/da/d25/d2b/d4a/d79/d97/de0/df5/fbc [0,4194304] 0 2024-04-06T14:17:23.971 INFO:tasks.workunit.client.1.smithi086.stdout:0/966: unlink d0/d1/d2/d1a/l1f 0 2024-04-06T14:17:23.972 INFO:tasks.workunit.client.1.smithi086.stdout:4/796: dread d0/d2c/d30/d24/d50/d79/d89/fa6 [0,4194304] 0 2024-04-06T14:17:23.972 INFO:tasks.workunit.client.1.smithi086.stdout:2/972: dwrite d0/d20/d44/dbb/dc4/f11f [0,4194304] 0 2024-04-06T14:17:23.973 INFO:tasks.workunit.client.1.smithi086.stdout:0/967: stat d0/d15/d4e/da3/d44/d5d/f5f 0 2024-04-06T14:17:23.974 INFO:tasks.workunit.client.1.smithi086.stdout:2/973: chown d0/d20/c29 11503262 1 2024-04-06T14:17:23.977 INFO:tasks.workunit.client.1.smithi086.stdout:5/839: truncate d0/d16/d18/d21/d23/d28/df7/fbe 11651 0 2024-04-06T14:17:23.978 INFO:tasks.workunit.client.1.smithi086.stdout:4/797: symlink d0/d15/d20/lfa 0 2024-04-06T14:17:23.981 INFO:tasks.workunit.client.1.smithi086.stdout:0/968: rename d0/d15/d4e/da3/d73/f123 to d0/d1/d4/de/dae/f141 0 2024-04-06T14:17:23.985 INFO:tasks.workunit.client.1.smithi086.stdout:5/840: symlink d0/d16/d65/d10f/l111 0 2024-04-06T14:17:23.986 INFO:tasks.workunit.client.1.smithi086.stdout:4/798: dwrite d0/d2c/d30/d24/d5d/f8f [0,4194304] 0 2024-04-06T14:17:23.987 INFO:tasks.workunit.client.0.smithi032.stdout:0/974: dread d3/da/d25/d2b/d4a/d79/d97/de2/ff4 [0,4194304] 0 2024-04-06T14:17:23.988 INFO:tasks.workunit.client.1.smithi086.stdout:5/841: readlink d0/d16/d18/d21/d23/d28/d94/l31 0 2024-04-06T14:17:23.988 INFO:tasks.workunit.client.1.smithi086.stdout:0/969: getdents d0/d15 0 2024-04-06T14:17:23.988 INFO:tasks.workunit.client.0.smithi032.stdout:0/975: chown d3/da/d25/d2b/d4a/l8d 849975742 1 2024-04-06T14:17:23.993 INFO:tasks.workunit.client.1.smithi086.stdout:4/799: chown d0/d2c/d38/d54/d83/f77 8126272 1 2024-04-06T14:17:23.994 INFO:tasks.workunit.client.1.smithi086.stdout:4/800: chown d0/d2c/d38/dd7/de6 4322 1 2024-04-06T14:17:23.994 INFO:tasks.workunit.client.1.smithi086.stdout:0/970: write d0/d1/d4/de/df9/f13c [598878,127095] 0 2024-04-06T14:17:23.994 INFO:tasks.workunit.client.1.smithi086.stdout:5/842: write d0/d16/d18/d21/d23/d28/d94/f56 [4223130,4269] 0 2024-04-06T14:17:23.995 INFO:tasks.workunit.client.1.smithi086.stdout:0/971: mknod d0/d15/d4e/da3/d44/d10f/c142 0 2024-04-06T14:17:23.999 INFO:tasks.workunit.client.1.smithi086.stdout:0/972: symlink d0/d15/d4e/da3/d44/d5d/d74/d120/l143 0 2024-04-06T14:17:24.000 INFO:tasks.workunit.client.1.smithi086.stdout:4/801: dwrite d0/d2c/d38/d54/fa1 [0,4194304] 0 2024-04-06T14:17:24.004 INFO:tasks.workunit.client.1.smithi086.stdout:4/802: symlink d0/d2c/d38/lfb 0 2024-04-06T14:17:24.004 INFO:tasks.workunit.client.1.smithi086.stdout:4/803: chown d0/d2c/d30/d24/d5d/d8a/f9a 629055905 1 2024-04-06T14:17:24.012 INFO:tasks.workunit.client.1.smithi086.stdout:4/804: dwrite d0/d2c/d30/d24/da3/d88/fcc [0,4194304] 0 2024-04-06T14:17:24.018 INFO:tasks.workunit.client.1.smithi086.stdout:4/805: creat d0/d2c/d30/d24/d50/dd0/ffc x:0 0 0 2024-04-06T14:17:24.018 INFO:tasks.workunit.client.1.smithi086.stdout:2/974: write d0/df/d5e/fc7 [974842,100575] 0 2024-04-06T14:17:24.020 INFO:tasks.workunit.client.1.smithi086.stdout:2/975: rmdir d0/d20/d44 39 2024-04-06T14:17:24.024 INFO:tasks.workunit.client.1.smithi086.stdout:4/806: getdents d0/d2c/d38/d54/d61 0 2024-04-06T14:17:24.027 INFO:tasks.workunit.client.1.smithi086.stdout:5/843: dwrite d0/d16/d18/d35/f3a [0,4194304] 0 2024-04-06T14:17:24.030 INFO:tasks.workunit.client.1.smithi086.stdout:0/973: dwrite d0/d15/d4e/da3/d44/d5d/f5f [0,4194304] 0 2024-04-06T14:17:24.032 INFO:tasks.workunit.client.1.smithi086.stdout:4/807: write d0/d2c/d30/d24/da3/f52 [1262918,31834] 0 2024-04-06T14:17:24.069 INFO:tasks.workunit.client.1.smithi086.stdout:8/995: sync 2024-04-06T14:17:24.069 INFO:tasks.workunit.client.1.smithi086.stdout:1/987: sync 2024-04-06T14:17:24.069 INFO:tasks.workunit.client.1.smithi086.stdout:7/873: sync 2024-04-06T14:17:24.070 INFO:tasks.workunit.client.1.smithi086.stdout:7/874: chown d7/d10/d11/d1c/d24/d2c/d69/f107 78 1 2024-04-06T14:17:24.101 INFO:tasks.workunit.client.1.smithi086.stdout:0/974: mknod d0/c144 0 2024-04-06T14:17:24.103 INFO:tasks.workunit.client.1.smithi086.stdout:1/988: rmdir db/d13/d55 39 2024-04-06T14:17:24.104 INFO:tasks.workunit.client.1.smithi086.stdout:8/996: symlink d0/d71/d22/l15a 0 2024-04-06T14:17:24.110 INFO:tasks.workunit.client.1.smithi086.stdout:0/975: rename d0/d1/d2 to d0/d15/d4e/da3/d44/d8e/d145 0 2024-04-06T14:17:24.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:23 smithi086 ceph-mon[47925]: pgmap v29: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 131 MiB/s rd, 294 MiB/s wr, 673 op/s 2024-04-06T14:17:24.141 INFO:tasks.workunit.client.1.smithi086.stdout:7/875: write d7/d1d/f13 [398622,26778] 0 2024-04-06T14:17:24.142 INFO:tasks.workunit.client.1.smithi086.stdout:1/989: link db/d11/d24/leb db/d11/d1a/d2a/d9a/d78/l13e 0 2024-04-06T14:17:24.142 INFO:tasks.workunit.client.1.smithi086.stdout:7/876: readlink d7/d1d/l3a 0 2024-04-06T14:17:24.144 INFO:tasks.workunit.client.1.smithi086.stdout:0/976: write d0/d15/d4e/da3/d44/d8e/d145/d1a/d79/dac/db7/f11d [239825,43145] 0 2024-04-06T14:17:24.182 INFO:tasks.workunit.client.1.smithi086.stdout:9/897: sync 2024-04-06T14:17:24.185 INFO:tasks.workunit.client.1.smithi086.stdout:4/808: dwrite d0/de/d4c/fc5 [0,4194304] 0 2024-04-06T14:17:24.185 INFO:tasks.workunit.client.1.smithi086.stdout:0/977: creat d0/d1/d4/de/df9/f146 x:0 0 0 2024-04-06T14:17:24.188 INFO:tasks.workunit.client.1.smithi086.stdout:1/990: rename db/d11/d1a/d2a/d9a/d68/f91 to db/f13f 0 2024-04-06T14:17:24.189 INFO:tasks.workunit.client.1.smithi086.stdout:7/877: truncate d7/d10/d11/d123/f4a 3488141 0 2024-04-06T14:17:24.194 INFO:tasks.workunit.client.1.smithi086.stdout:4/809: dwrite d0/d2c/d30/d24/d5d/d8a/f9a [0,4194304] 0 2024-04-06T14:17:24.198 INFO:tasks.workunit.client.1.smithi086.stdout:0/978: mkdir d0/d15/d4e/da3/d46/d147 0 2024-04-06T14:17:24.199 INFO:tasks.workunit.client.1.smithi086.stdout:7/878: dwrite d7/d1d/f13 [0,4194304] 0 2024-04-06T14:17:24.199 INFO:tasks.workunit.client.1.smithi086.stdout:9/898: creat d5/d13/d1a/d5f/d88/d14/d19/f123 x:0 0 0 2024-04-06T14:17:24.204 INFO:tasks.workunit.client.1.smithi086.stdout:9/899: creat d5/d58/d8d/f124 x:0 0 0 2024-04-06T14:17:24.205 INFO:tasks.workunit.client.1.smithi086.stdout:1/991: mknod db/d13/d55/df1/d110/c140 0 2024-04-06T14:17:24.207 INFO:tasks.workunit.client.1.smithi086.stdout:4/810: symlink d0/d2c/lfd 0 2024-04-06T14:17:24.209 INFO:tasks.workunit.client.1.smithi086.stdout:1/992: dread - db/d11/d1a/d2a/fcb zero size 2024-04-06T14:17:24.211 INFO:tasks.workunit.client.1.smithi086.stdout:4/811: mkdir d0/d2c/d30/d24/d5d/dfe 0 2024-04-06T14:17:24.211 INFO:tasks.workunit.client.1.smithi086.stdout:0/979: getdents d0/d15/d26/d33/d9f 0 2024-04-06T14:17:24.212 INFO:tasks.workunit.client.1.smithi086.stdout:1/993: symlink db/d13/d71/l141 0 2024-04-06T14:17:24.214 INFO:tasks.workunit.client.1.smithi086.stdout:4/812: dread d0/fb4 [0,4194304] 0 2024-04-06T14:17:24.216 INFO:tasks.workunit.client.1.smithi086.stdout:0/980: truncate d0/d15/d4e/da3/fe7 1695304 0 2024-04-06T14:17:24.219 INFO:tasks.workunit.client.1.smithi086.stdout:9/900: dwrite d5/d13/d1a/d5f/d88/d14/d19/fbf [0,4194304] 0 2024-04-06T14:17:24.220 INFO:tasks.workunit.client.1.smithi086.stdout:0/981: rmdir d0/d15/d26/d33/d7a 39 2024-04-06T14:17:24.222 INFO:tasks.workunit.client.1.smithi086.stdout:7/879: dread d7/d1d/f3e [0,4194304] 0 2024-04-06T14:17:24.222 INFO:tasks.workunit.client.1.smithi086.stdout:0/982: readlink d0/d15/l1e 0 2024-04-06T14:17:24.225 INFO:tasks.workunit.client.1.smithi086.stdout:1/994: dwrite db/d11/d1a/d2a/d9a/d68/d6e/f9e [4194304,4194304] 0 2024-04-06T14:17:24.227 INFO:tasks.workunit.client.1.smithi086.stdout:0/983: symlink d0/d15/d26/d33/def/d13e/l148 0 2024-04-06T14:17:24.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:23 smithi032 ceph-mon[39730]: pgmap v29: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 131 MiB/s rd, 294 MiB/s wr, 673 op/s 2024-04-06T14:17:24.230 INFO:tasks.workunit.client.1.smithi086.stdout:1/995: dread db/d11/d87/fec [0,4194304] 0 2024-04-06T14:17:24.237 INFO:tasks.workunit.client.1.smithi086.stdout:4/813: dread d0/d15/d20/d34/f55 [0,4194304] 0 2024-04-06T14:17:24.238 INFO:tasks.workunit.client.1.smithi086.stdout:4/814: stat d0/de/d4c/fc5 0 2024-04-06T14:17:24.262 INFO:tasks.workunit.client.1.smithi086.stdout:2/976: sync 2024-04-06T14:17:24.285 INFO:tasks.workunit.client.1.smithi086.stdout:9/901: rename d5/d13/d1a/d5f/d88/d14/d19/fbf to d5/d13/d1a/d5f/d88/d14/f125 0 2024-04-06T14:17:24.288 INFO:tasks.workunit.client.1.smithi086.stdout:7/880: write d7/d10/d11/d123/f4a [3131928,120451] 0 2024-04-06T14:17:24.289 INFO:tasks.workunit.client.1.smithi086.stdout:7/881: dread - d7/d10/d102/daf/db5/d7a/d100/d137/f139 zero size 2024-04-06T14:17:24.297 INFO:tasks.workunit.client.1.smithi086.stdout:1/996: mknod db/c142 0 2024-04-06T14:17:24.297 INFO:tasks.workunit.client.1.smithi086.stdout:7/882: rmdir d7/d10/d92 39 2024-04-06T14:17:24.297 INFO:tasks.workunit.client.1.smithi086.stdout:4/815: mknod d0/d2c/d30/d24/da3/cff 0 2024-04-06T14:17:24.297 INFO:tasks.workunit.client.1.smithi086.stdout:9/902: symlink d5/l126 0 2024-04-06T14:17:24.298 INFO:tasks.workunit.client.1.smithi086.stdout:7/883: fsync d7/d10/d11/d1c/d36/dbc/fbd 0 2024-04-06T14:17:24.299 INFO:tasks.workunit.client.1.smithi086.stdout:9/903: chown f2 3723 1 2024-04-06T14:17:24.300 INFO:tasks.workunit.client.1.smithi086.stdout:2/977: link d0/f33 d0/d20/d27/d2b/d35/f15c 0 2024-04-06T14:17:24.301 INFO:tasks.workunit.client.1.smithi086.stdout:0/984: dwrite d0/d1/d4/f6b [0,4194304] 0 2024-04-06T14:17:24.302 INFO:tasks.workunit.client.1.smithi086.stdout:7/884: symlink d7/d1d/l13a 0 2024-04-06T14:17:24.303 INFO:tasks.workunit.client.1.smithi086.stdout:1/997: write db/d11/d87/fec [658918,103542] 0 2024-04-06T14:17:24.306 INFO:tasks.workunit.client.1.smithi086.stdout:9/904: rename d5/d13/d1a/d5f/d119 to d5/d13/d1a/d5f/d88/db3/d127 0 2024-04-06T14:17:24.306 INFO:tasks.workunit.client.1.smithi086.stdout:1/998: write db/d11/d1a/d79/da5/f11d [547707,33193] 0 2024-04-06T14:17:24.307 INFO:tasks.workunit.client.1.smithi086.stdout:2/978: dread d0/d20/d27/f7f [0,4194304] 0 2024-04-06T14:17:24.308 INFO:tasks.workunit.client.1.smithi086.stdout:8/997: sync 2024-04-06T14:17:24.308 INFO:tasks.workunit.client.1.smithi086.stdout:5/844: sync 2024-04-06T14:17:24.309 INFO:tasks.workunit.client.1.smithi086.stdout:0/985: creat d0/d15/d4e/da3/f149 x:0 0 0 2024-04-06T14:17:24.310 INFO:tasks.workunit.client.1.smithi086.stdout:8/998: write d0/d71/d16/d39/d90/fa0 [4622153,42950] 0 2024-04-06T14:17:24.311 INFO:tasks.workunit.client.1.smithi086.stdout:5/845: mknod d0/d43/d74/c112 0 2024-04-06T14:17:24.313 INFO:tasks.workunit.client.1.smithi086.stdout:1/999: creat db/d11/d1a/d2a/d9a/d68/d6e/d81/f143 x:0 0 0 2024-04-06T14:17:24.315 INFO:tasks.workunit.client.1.smithi086.stdout:4/816: truncate d0/de/f62 2746549 0 2024-04-06T14:17:24.315 INFO:tasks.workunit.client.1.smithi086.stdout:4/817: read - d0/d15/d20/d74/fef zero size 2024-04-06T14:17:24.318 INFO:tasks.workunit.client.1.smithi086.stdout:0/986: rename d0/d15/d4e/da3/d44/d8e/d8f/ff0 to d0/d15/d4e/da3/d44/d8e/d145/d1a/d79/f14a 0 2024-04-06T14:17:24.318 INFO:tasks.workunit.client.1.smithi086.stdout:2/979: getdents d0/df/d110 0 2024-04-06T14:17:24.321 INFO:tasks.workunit.client.1.smithi086.stdout:4/818: write d0/de/f11 [8053185,91479] 0 2024-04-06T14:17:24.322 INFO:tasks.workunit.client.1.smithi086.stdout:4/819: write d0/d2c/d30/f3b [1712333,33139] 0 2024-04-06T14:17:24.323 INFO:tasks.workunit.client.1.smithi086.stdout:2/980: dread - d0/d20/d27/d37/f98 zero size 2024-04-06T14:17:24.323 INFO:tasks.workunit.client.1.smithi086.stdout:4/820: chown d0/d15/d2b/f5f 2017535351 1 2024-04-06T14:17:24.324 INFO:tasks.workunit.client.1.smithi086.stdout:8/999: creat d0/d71/d22/d43/d4d/f15b x:0 0 0 2024-04-06T14:17:24.325 INFO:tasks.workunit.client.1.smithi086.stdout:4/821: read d0/f10 [956950,36549] 0 2024-04-06T14:17:24.328 INFO:tasks.workunit.client.1.smithi086.stdout:4/822: chown d0/de/l6d 271639 1 2024-04-06T14:17:24.331 INFO:tasks.workunit.client.1.smithi086.stdout:9/905: truncate d5/d13/d1a/d5f/d88/d14/d19/d25/f67 2798562 0 2024-04-06T14:17:24.332 INFO:tasks.workunit.client.1.smithi086.stdout:5/846: truncate d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc5 3326549 0 2024-04-06T14:17:24.335 INFO:tasks.workunit.client.1.smithi086.stdout:4/823: getdents d0/d2c/d38 0 2024-04-06T14:17:24.342 INFO:tasks.workunit.client.1.smithi086.stdout:9/906: dwrite d5/d13/d1a/d5f/d88/d14/d19/d25/fcb [0,4194304] 0 2024-04-06T14:17:24.343 INFO:tasks.workunit.client.1.smithi086.stdout:5/847: dwrite d0/d16/fd3 [0,4194304] 0 2024-04-06T14:17:24.345 INFO:tasks.workunit.client.1.smithi086.stdout:4/824: mknod d0/d2c/d30/d24/de2/c100 0 2024-04-06T14:17:24.346 INFO:tasks.workunit.client.1.smithi086.stdout:2/981: dwrite d0/d20/d27/d2b/d35/d36/d4a/ff8 [0,4194304] 0 2024-04-06T14:17:24.349 INFO:tasks.workunit.client.1.smithi086.stdout:5/848: mkdir d0/d16/d71/d96/d113 0 2024-04-06T14:17:24.350 INFO:tasks.workunit.client.1.smithi086.stdout:5/849: write d0/d16/d18/d35/f3a [802337,50828] 0 2024-04-06T14:17:24.351 INFO:tasks.workunit.client.1.smithi086.stdout:4/825: mkdir d0/d2c/d30/d24/d5d/d101 0 2024-04-06T14:17:24.353 INFO:tasks.workunit.client.1.smithi086.stdout:2/982: unlink d0/df/d48/d7d/f87 0 2024-04-06T14:17:24.361 INFO:tasks.workunit.client.1.smithi086.stdout:4/826: creat d0/d2c/d30/d24/da3/f102 x:0 0 0 2024-04-06T14:17:24.361 INFO:tasks.workunit.client.1.smithi086.stdout:9/907: link d5/d13/d1a/d5f/d88/d14/d19/d94/lea d5/d13/d1a/d5f/l128 0 2024-04-06T14:17:24.364 INFO:tasks.workunit.client.1.smithi086.stdout:9/908: write d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dce/f113 [639164,57645] 0 2024-04-06T14:17:24.367 INFO:tasks.workunit.client.1.smithi086.stdout:9/909: creat d5/f129 x:0 0 0 2024-04-06T14:17:24.376 INFO:tasks.workunit.client.1.smithi086.stdout:2/983: dwrite d0/d20/d27/d2b/d35/f62 [4194304,4194304] 0 2024-04-06T14:17:24.377 INFO:tasks.workunit.client.1.smithi086.stdout:4/827: dwrite d0/d15/d2b/f82 [0,4194304] 0 2024-04-06T14:17:24.378 INFO:tasks.workunit.client.1.smithi086.stdout:4/828: write d0/d2c/d38/d54/d83/fd1 [1026356,55003] 0 2024-04-06T14:17:24.387 INFO:tasks.workunit.client.1.smithi086.stdout:2/984: write d0/d20/d44/dbb/fdb [3593574,74323] 0 2024-04-06T14:17:24.387 INFO:tasks.workunit.client.1.smithi086.stdout:4/829: read d0/d2c/d30/d24/d50/d79/fa8 [22939,106586] 0 2024-04-06T14:17:24.387 INFO:tasks.workunit.client.1.smithi086.stdout:9/910: dread d5/d13/d47/d55/f64 [0,4194304] 0 2024-04-06T14:17:24.389 INFO:tasks.workunit.client.1.smithi086.stdout:4/830: mknod d0/d2c/d30/d24/da3/d88/c103 0 2024-04-06T14:17:24.391 INFO:tasks.workunit.client.1.smithi086.stdout:2/985: creat d0/d20/d82/d121/f15d x:0 0 0 2024-04-06T14:17:24.392 INFO:tasks.workunit.client.1.smithi086.stdout:9/911: mknod d5/d13/d47/d55/d87/ddc/c12a 0 2024-04-06T14:17:24.392 INFO:tasks.workunit.client.1.smithi086.stdout:9/912: chown d5/d13/d47/fc4 1500 1 2024-04-06T14:17:24.394 INFO:tasks.workunit.client.1.smithi086.stdout:9/913: truncate d5/d18/dff/fb8 389301 0 2024-04-06T14:17:24.394 INFO:tasks.workunit.client.1.smithi086.stdout:4/831: rename d0/dc0/l5c to d0/d2c/d30/d24/d93/l104 0 2024-04-06T14:17:24.396 INFO:tasks.workunit.client.1.smithi086.stdout:2/986: mkdir d0/d20/d44/dbb/dc4/de3/d76/d78/d15e 0 2024-04-06T14:17:24.398 INFO:tasks.workunit.client.1.smithi086.stdout:4/832: mknod d0/d2c/d38/d54/d90/c105 0 2024-04-06T14:17:24.399 INFO:tasks.workunit.client.1.smithi086.stdout:2/987: fdatasync d0/d20/d44/dbb/dc4/de3/d76/d78/d125/fd9 0 2024-04-06T14:17:24.401 INFO:tasks.workunit.client.1.smithi086.stdout:9/914: creat d5/d13/d1a/d5f/d88/d14/d19/f12b x:0 0 0 2024-04-06T14:17:24.401 INFO:tasks.workunit.client.1.smithi086.stdout:9/915: stat d5 0 2024-04-06T14:17:24.401 INFO:tasks.workunit.client.1.smithi086.stdout:9/916: chown d5/f129 2 1 2024-04-06T14:17:24.402 INFO:tasks.workunit.client.1.smithi086.stdout:9/917: chown d5/d13/d1a 13645637 1 2024-04-06T14:17:24.407 INFO:tasks.workunit.client.1.smithi086.stdout:2/988: dwrite d0/d20/d27/d2b/d35/d36/d4d/d59/f119 [0,4194304] 0 2024-04-06T14:17:24.408 INFO:tasks.workunit.client.1.smithi086.stdout:9/918: creat d5/f12c x:0 0 0 2024-04-06T14:17:24.411 INFO:tasks.workunit.client.1.smithi086.stdout:9/919: creat d5/d13/d1a/f12d x:0 0 0 2024-04-06T14:17:24.412 INFO:tasks.workunit.client.1.smithi086.stdout:4/833: truncate d0/d2c/d30/d24/d5d/f8f 2500627 0 2024-04-06T14:17:24.416 INFO:tasks.workunit.client.1.smithi086.stdout:4/834: mknod d0/d15/d20/c106 0 2024-04-06T14:17:24.418 INFO:tasks.workunit.client.1.smithi086.stdout:2/989: dwrite d0/d20/f61 [0,4194304] 0 2024-04-06T14:17:24.420 INFO:tasks.workunit.client.1.smithi086.stdout:2/990: readlink d0/d20/d27/d2b/d35/d36/d4a/d11d/l138 0 2024-04-06T14:17:24.421 INFO:tasks.workunit.client.1.smithi086.stdout:4/835: link d0/dc0/la5 d0/d2c/d38/d54/d61/d7a/dc2/dd3/l107 0 2024-04-06T14:17:24.422 INFO:tasks.workunit.client.1.smithi086.stdout:2/991: write d0/f136 [215000,70948] 0 2024-04-06T14:17:24.424 INFO:tasks.workunit.client.1.smithi086.stdout:2/992: chown d0/df/d48/d7d/fe9 107 1 2024-04-06T14:17:24.426 INFO:tasks.workunit.client.1.smithi086.stdout:4/836: symlink d0/db0/l108 0 2024-04-06T14:17:24.428 INFO:tasks.workunit.client.1.smithi086.stdout:0/987: sync 2024-04-06T14:17:24.429 INFO:tasks.workunit.client.1.smithi086.stdout:7/885: sync 2024-04-06T14:17:24.429 INFO:tasks.workunit.client.1.smithi086.stdout:9/920: dread d5/d18/dff/fb1 [0,4194304] 0 2024-04-06T14:17:24.430 INFO:tasks.workunit.client.1.smithi086.stdout:2/993: symlink d0/l15f 0 2024-04-06T14:17:24.430 INFO:tasks.workunit.client.1.smithi086.stdout:7/886: write d7/d10/d11/d1c/d24/f7d [2153823,18274] 0 2024-04-06T14:17:24.432 INFO:tasks.workunit.client.1.smithi086.stdout:4/837: symlink d0/de/l109 0 2024-04-06T14:17:24.436 INFO:tasks.workunit.client.1.smithi086.stdout:0/988: getdents d0/d15/d4e/da3/d128 0 2024-04-06T14:17:24.437 INFO:tasks.workunit.client.1.smithi086.stdout:2/994: mkdir d0/d160 0 2024-04-06T14:17:24.439 INFO:tasks.workunit.client.1.smithi086.stdout:7/887: dwrite d7/d10/d11/d1c/d24/d2c/d69/f12a [0,4194304] 0 2024-04-06T14:17:24.441 INFO:tasks.workunit.client.1.smithi086.stdout:0/989: rmdir d0/d15/d4e/da3/d46 39 2024-04-06T14:17:24.443 INFO:tasks.workunit.client.1.smithi086.stdout:4/838: link d0/d2c/d30/d24/d93/fa4 d0/d2c/d30/d24/d5d/dfe/f10a 0 2024-04-06T14:17:24.444 INFO:tasks.workunit.client.1.smithi086.stdout:4/839: truncate d0/d15/d20/d74/fef 349331 0 2024-04-06T14:17:24.445 INFO:tasks.workunit.client.1.smithi086.stdout:2/995: rename d0/d160 to d0/df/d48/dce/d101/d161 0 2024-04-06T14:17:24.445 INFO:tasks.workunit.client.1.smithi086.stdout:4/840: chown d0/d15/d2b/f5f 3 1 2024-04-06T14:17:24.449 INFO:tasks.workunit.client.1.smithi086.stdout:0/990: creat d0/d1/d4/de/db4/f14b x:0 0 0 2024-04-06T14:17:24.450 INFO:tasks.workunit.client.1.smithi086.stdout:2/996: symlink d0/d20/d44/d120/d147/l162 0 2024-04-06T14:17:24.451 INFO:tasks.workunit.client.1.smithi086.stdout:0/991: read - d0/d1/dfe/f12c zero size 2024-04-06T14:17:24.451 INFO:tasks.workunit.client.1.smithi086.stdout:4/841: fdatasync d0/d2c/d30/d24/d93/fa4 0 2024-04-06T14:17:24.452 INFO:tasks.workunit.client.1.smithi086.stdout:0/992: stat d0/d15/d4e/da3/d44/d8e/d145/d1a/d79/dac/dc2/c10d 0 2024-04-06T14:17:24.453 INFO:tasks.workunit.client.1.smithi086.stdout:4/842: write d0/d2c/d30/d24/d50/d79/f7c [2982735,34656] 0 2024-04-06T14:17:24.456 INFO:tasks.workunit.client.1.smithi086.stdout:9/921: write d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f81 [1101179,66635] 0 2024-04-06T14:17:24.458 INFO:tasks.workunit.client.1.smithi086.stdout:4/843: creat d0/db0/f10b x:0 0 0 2024-04-06T14:17:24.459 INFO:tasks.workunit.client.1.smithi086.stdout:0/993: creat d0/d15/d4e/da3/d44/d8e/d145/d1a/d79/f14c x:0 0 0 2024-04-06T14:17:24.463 INFO:tasks.workunit.client.1.smithi086.stdout:7/888: dwrite d7/d10/d102/d5c/d66/f55 [0,4194304] 0 2024-04-06T14:17:24.464 INFO:tasks.workunit.client.1.smithi086.stdout:4/844: write d0/d2c/d30/d24/d50/d79/fa8 [62075,106270] 0 2024-04-06T14:17:24.465 INFO:tasks.workunit.client.1.smithi086.stdout:2/997: dread d0/df/d48/fb7 [0,4194304] 0 2024-04-06T14:17:24.468 INFO:tasks.workunit.client.1.smithi086.stdout:0/994: dread d0/d15/d4e/f42 [0,4194304] 0 2024-04-06T14:17:24.469 INFO:tasks.workunit.client.1.smithi086.stdout:7/889: creat d7/d10/d11/d1c/d24/d2c/d69/f13b x:0 0 0 2024-04-06T14:17:24.470 INFO:tasks.workunit.client.1.smithi086.stdout:4/845: creat d0/de/f10c x:0 0 0 2024-04-06T14:17:24.470 INFO:tasks.workunit.client.1.smithi086.stdout:2/998: mkdir d0/d20/d27/d2b/d35/d36/d163 0 2024-04-06T14:17:24.471 INFO:tasks.workunit.client.1.smithi086.stdout:0/995: rmdir d0/d15/d4e/da3/d44/d8e/d145 39 2024-04-06T14:17:24.473 INFO:tasks.workunit.client.1.smithi086.stdout:7/890: mkdir d7/d10/d11/d1c/d13c 0 2024-04-06T14:17:24.474 INFO:tasks.workunit.client.1.smithi086.stdout:0/996: mkdir d0/d15/d26/d14d 0 2024-04-06T14:17:24.474 INFO:tasks.workunit.client.1.smithi086.stdout:7/891: write d7/d10/d102/d5c/d66/d7e/db9/dd2/fe0 [1121784,119937] 0 2024-04-06T14:17:24.476 INFO:tasks.workunit.client.1.smithi086.stdout:0/997: truncate d0/d15/d26/d33/d9f/f106 954327 0 2024-04-06T14:17:24.476 INFO:tasks.workunit.client.1.smithi086.stdout:7/892: read d7/d10/d102/d5c/d66/d7e/db9/fc3 [819511,5963] 0 2024-04-06T14:17:24.478 INFO:tasks.workunit.client.1.smithi086.stdout:0/998: chown d0/d15/d4e/d48/d89/fa9 318 1 2024-04-06T14:17:24.478 INFO:tasks.workunit.client.1.smithi086.stdout:2/999: truncate d0/d20/d27/d2b/d35/d36/f3b 2270875 0 2024-04-06T14:17:24.481 INFO:tasks.workunit.client.1.smithi086.stdout:7/893: chown d7/d10/d11/d1c/d24/d2c/d69/c12b 3642435 1 2024-04-06T14:17:24.481 INFO:tasks.workunit.client.1.smithi086.stdout:7/894: readlink d7/d94/la5 0 2024-04-06T14:17:24.482 INFO:tasks.workunit.client.1.smithi086.stdout:0/999: rename d0/d15/d4e/f45 to d0/d1/d4/de/df9/f14e 0 2024-04-06T14:17:24.483 INFO:tasks.workunit.client.1.smithi086.stdout:7/895: symlink d7/d10/d11/d123/l13d 0 2024-04-06T14:17:24.486 INFO:tasks.workunit.client.1.smithi086.stdout:4/846: dread d0/d2c/d30/d24/d93/fa4 [0,4194304] 0 2024-04-06T14:17:24.487 INFO:tasks.workunit.client.1.smithi086.stdout:4/847: chown d0/d2c/d38/d54/d90/l9e 1980092776 1 2024-04-06T14:17:24.492 INFO:tasks.workunit.client.1.smithi086.stdout:4/848: rename d0/d15/d20/d34/l4b to d0/d2c/d30/d24/d50/dd0/l10d 0 2024-04-06T14:17:24.492 INFO:tasks.workunit.client.1.smithi086.stdout:7/896: write d7/d10/f64 [3770630,21880] 0 2024-04-06T14:17:24.494 INFO:tasks.workunit.client.1.smithi086.stdout:4/849: fdatasync d0/d2c/d30/f6c 0 2024-04-06T14:17:24.498 INFO:tasks.workunit.client.1.smithi086.stdout:7/897: dwrite d7/d10/d11/d21/f120 [0,4194304] 0 2024-04-06T14:17:24.500 INFO:tasks.workunit.client.1.smithi086.stdout:7/898: rmdir d7/d10/d11/d1c/d24/d2c/d69/d8d 39 2024-04-06T14:17:24.504 INFO:tasks.workunit.client.1.smithi086.stdout:7/899: mkdir d7/d10/d11/d1c/d24/df0/d116/d13e 0 2024-04-06T14:17:24.504 INFO:tasks.workunit.client.1.smithi086.stdout:7/900: fdatasync d7/d10/d102/d5c/d66/d7e/f135 0 2024-04-06T14:17:24.505 INFO:tasks.workunit.client.1.smithi086.stdout:7/901: write d7/d1d/f6b [378984,75058] 0 2024-04-06T14:17:24.506 INFO:tasks.workunit.client.1.smithi086.stdout:7/902: fsync d7/d10/d102/d5c/d66/d7e/f108 0 2024-04-06T14:17:24.508 INFO:tasks.workunit.client.1.smithi086.stdout:7/903: creat d7/f13f x:0 0 0 2024-04-06T14:17:24.509 INFO:tasks.workunit.client.1.smithi086.stdout:7/904: fdatasync d7/d10/d11/d1c/d24/d2c/fc0 0 2024-04-06T14:17:24.510 INFO:tasks.workunit.client.1.smithi086.stdout:7/905: write d7/d1d/fb8 [2781520,92193] 0 2024-04-06T14:17:24.513 INFO:tasks.workunit.client.1.smithi086.stdout:7/906: mkdir d7/d10/d11/d1c/d24/d140 0 2024-04-06T14:17:24.514 INFO:tasks.workunit.client.1.smithi086.stdout:5/850: sync 2024-04-06T14:17:24.514 INFO:tasks.workunit.client.1.smithi086.stdout:5/851: dread - d0/d16/d71/ff2 zero size 2024-04-06T14:17:24.515 INFO:tasks.workunit.client.1.smithi086.stdout:5/852: chown d0/l9a 3 1 2024-04-06T14:17:24.516 INFO:tasks.workunit.client.1.smithi086.stdout:7/907: symlink d7/d10/d11/d1c/d36/dbc/d138/l141 0 2024-04-06T14:17:24.517 INFO:tasks.workunit.client.1.smithi086.stdout:5/853: symlink d0/d16/d18/d35/d9e/dd7/l114 0 2024-04-06T14:17:24.518 INFO:tasks.workunit.client.1.smithi086.stdout:7/908: truncate d7/d10/d102/d5c/d66/d7e/f108 744517 0 2024-04-06T14:17:24.522 INFO:tasks.workunit.client.1.smithi086.stdout:7/909: creat d7/d10/d92/f142 x:0 0 0 2024-04-06T14:17:24.523 INFO:tasks.workunit.client.1.smithi086.stdout:7/910: write d7/d10/d102/daf/db5/fff [666290,81340] 0 2024-04-06T14:17:24.527 INFO:tasks.workunit.client.1.smithi086.stdout:7/911: mknod d7/d10/d11/d1c/d24/d2c/c143 0 2024-04-06T14:17:24.528 INFO:tasks.workunit.client.1.smithi086.stdout:5/854: dread d0/d43/de5/f26 [0,4194304] 0 2024-04-06T14:17:24.528 INFO:tasks.workunit.client.1.smithi086.stdout:7/912: mknod d7/d10/d11/d1c/d24/dc8/dcd/c144 0 2024-04-06T14:17:24.534 INFO:tasks.workunit.client.1.smithi086.stdout:5/855: dwrite d0/d16/d71/fb9 [0,4194304] 0 2024-04-06T14:17:24.534 INFO:tasks.workunit.client.1.smithi086.stdout:7/913: dwrite d7/d1d/f38 [4194304,4194304] 0 2024-04-06T14:17:24.538 INFO:tasks.workunit.client.1.smithi086.stdout:5/856: creat d0/d43/da6/f115 x:0 0 0 2024-04-06T14:17:24.543 INFO:tasks.workunit.client.1.smithi086.stdout:5/857: link d0/d43/de5/c57 d0/d16/d71/d96/d113/c116 0 2024-04-06T14:17:24.551 INFO:tasks.workunit.client.1.smithi086.stdout:5/858: dwrite d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc0 [8388608,4194304] 0 2024-04-06T14:17:24.555 INFO:tasks.workunit.client.1.smithi086.stdout:7/914: dwrite d7/d10/d11/fbe [0,4194304] 0 2024-04-06T14:17:24.555 INFO:tasks.workunit.client.1.smithi086.stdout:5/859: read d0/d16/d18/d35/f101 [1352040,65486] 0 2024-04-06T14:17:24.556 INFO:tasks.workunit.client.1.smithi086.stdout:7/915: chown d7/d10/d102/daf/db5/d7a/c11f 70882 1 2024-04-06T14:17:24.563 INFO:tasks.workunit.client.1.smithi086.stdout:5/860: creat d0/d16/d18/d21/d23/d28/d94/d52/d7c/f117 x:0 0 0 2024-04-06T14:17:24.564 INFO:tasks.workunit.client.1.smithi086.stdout:7/916: mknod d7/d10/d11/d123/c145 0 2024-04-06T14:17:24.565 INFO:tasks.workunit.client.1.smithi086.stdout:5/861: write d0/d16/d18/d21/d23/f6c [2681373,88453] 0 2024-04-06T14:17:24.565 INFO:tasks.workunit.client.1.smithi086.stdout:5/862: stat d0/d16/d18/d21/d23/lb2 0 2024-04-06T14:17:24.567 INFO:tasks.workunit.client.1.smithi086.stdout:7/917: creat d7/d10/d11/d1c/d24/dc8/f146 x:0 0 0 2024-04-06T14:17:24.570 INFO:tasks.workunit.client.1.smithi086.stdout:7/918: creat d7/d10/d11/d1c/d36/f147 x:0 0 0 2024-04-06T14:17:24.572 INFO:tasks.workunit.client.1.smithi086.stdout:5/863: dwrite d0/d16/d18/d2a/d50/f73 [0,4194304] 0 2024-04-06T14:17:24.573 INFO:tasks.workunit.client.1.smithi086.stdout:7/919: symlink d7/d10/d102/d5c/l148 0 2024-04-06T14:17:24.575 INFO:tasks.workunit.client.1.smithi086.stdout:7/920: symlink d7/d10/d11/d123/l149 0 2024-04-06T14:17:24.576 INFO:tasks.workunit.client.1.smithi086.stdout:9/922: sync 2024-04-06T14:17:24.576 INFO:tasks.workunit.client.1.smithi086.stdout:7/921: fdatasync d7/d10/d11/d1c/d24/d2c/fb6 0 2024-04-06T14:17:24.577 INFO:tasks.workunit.client.1.smithi086.stdout:5/864: dread d0/d16/d18/d35/d9e/de7/f79 [0,4194304] 0 2024-04-06T14:17:24.579 INFO:tasks.workunit.client.1.smithi086.stdout:7/922: read d7/d10/d11/d123/f4b [435061,46596] 0 2024-04-06T14:17:24.581 INFO:tasks.workunit.client.1.smithi086.stdout:9/923: unlink d5/d13/d1a/d5f/d88/d91/fbc 0 2024-04-06T14:17:24.582 INFO:tasks.workunit.client.1.smithi086.stdout:7/923: rmdir d7/d10/d11/d1c/d104 39 2024-04-06T14:17:24.584 INFO:tasks.workunit.client.1.smithi086.stdout:9/924: mknod d5/d13/d47/d55/d87/ddc/d120/c12e 0 2024-04-06T14:17:24.585 INFO:tasks.workunit.client.1.smithi086.stdout:7/924: unlink d7/d10/d11/d1c/d36/f147 0 2024-04-06T14:17:24.586 INFO:tasks.workunit.client.1.smithi086.stdout:7/925: fsync d7/d10/d102/daf/db5/f11e 0 2024-04-06T14:17:24.587 INFO:tasks.workunit.client.1.smithi086.stdout:9/925: rmdir d5/d13/d1a/d5f/d88/d14/d19/d25/d59 39 2024-04-06T14:17:24.589 INFO:tasks.workunit.client.1.smithi086.stdout:7/926: unlink d7/d10/d11/d1c/l37 0 2024-04-06T14:17:24.593 INFO:tasks.workunit.client.1.smithi086.stdout:9/926: dwrite d5/d13/d1a/d5f/d88/fe5 [0,4194304] 0 2024-04-06T14:17:24.594 INFO:tasks.workunit.client.1.smithi086.stdout:9/927: chown d5/d13/d1a/d5f/d88/d14/d19/d25/d74 28 1 2024-04-06T14:17:24.594 INFO:tasks.workunit.client.1.smithi086.stdout:7/927: unlink d7/d10/d11/d1c/d24/d2c/faa 0 2024-04-06T14:17:24.598 INFO:tasks.workunit.client.1.smithi086.stdout:9/928: unlink d5/l6 0 2024-04-06T14:17:24.602 INFO:tasks.workunit.client.1.smithi086.stdout:9/929: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/f12f x:0 0 0 2024-04-06T14:17:24.607 INFO:tasks.workunit.client.1.smithi086.stdout:7/928: write d7/d10/d11/d1c/d24/fa2 [3053102,46597] 0 2024-04-06T14:17:24.616 INFO:tasks.workunit.client.1.smithi086.stdout:9/930: dread d5/d13/d47/d55/d87/ddc/fde [0,4194304] 0 2024-04-06T14:17:24.620 INFO:tasks.workunit.client.1.smithi086.stdout:9/931: symlink d5/d13/d1a/d5f/d88/d14/d19/l130 0 2024-04-06T14:17:24.621 INFO:tasks.workunit.client.1.smithi086.stdout:9/932: write d5/d13/d1a/d5f/d88/d14/d19/d1b/f45 [490864,7287] 0 2024-04-06T14:17:24.625 INFO:tasks.workunit.client.1.smithi086.stdout:9/933: creat d5/d13/d1a/d5f/d88/d14/d19/d25/f131 x:0 0 0 2024-04-06T14:17:24.627 INFO:tasks.workunit.client.1.smithi086.stdout:9/934: mknod d5/d58/d100/c132 0 2024-04-06T14:17:24.628 INFO:tasks.workunit.client.1.smithi086.stdout:9/935: dread - d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/f12f zero size 2024-04-06T14:17:24.639 INFO:tasks.workunit.client.1.smithi086.stdout:9/936: dread f0 [4194304,4194304] 0 2024-04-06T14:17:24.641 INFO:tasks.workunit.client.1.smithi086.stdout:9/937: unlink d5/d13/d1a/d5f/d88/d91/fed 0 2024-04-06T14:17:24.642 INFO:tasks.workunit.client.1.smithi086.stdout:9/938: chown d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5 7 1 2024-04-06T14:17:24.643 INFO:tasks.workunit.client.1.smithi086.stdout:9/939: write d5/d13/d47/d5e/fcd [1189206,108487] 0 2024-04-06T14:17:24.645 INFO:tasks.workunit.client.1.smithi086.stdout:9/940: write d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dfa/f11a [791605,46545] 0 2024-04-06T14:17:24.648 INFO:tasks.workunit.client.1.smithi086.stdout:9/941: read d5/d13/d1a/d5f/d88/d14/d19/d25/f2f [396997,110509] 0 2024-04-06T14:17:24.653 INFO:tasks.workunit.client.1.smithi086.stdout:9/942: creat d5/d13/d1a/d5f/d88/d14/d19/d25/deb/f133 x:0 0 0 2024-04-06T14:17:24.653 INFO:tasks.workunit.client.1.smithi086.stdout:9/943: fdatasync d5/d13/d47/d5e/f93 0 2024-04-06T14:17:24.654 INFO:tasks.workunit.client.1.smithi086.stdout:9/944: fdatasync d5/d13/f4f 0 2024-04-06T14:17:24.654 INFO:tasks.workunit.client.1.smithi086.stdout:9/945: write d5/d13/d1a/d5f/d88/d14/f4e [2976749,97607] 0 2024-04-06T14:17:24.657 INFO:tasks.workunit.client.1.smithi086.stdout:9/946: getdents d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5 0 2024-04-06T14:17:24.659 INFO:tasks.workunit.client.1.smithi086.stdout:4/850: sync 2024-04-06T14:17:24.661 INFO:tasks.workunit.client.1.smithi086.stdout:4/851: symlink d0/d15/d2b/l10e 0 2024-04-06T14:17:24.662 INFO:tasks.workunit.client.1.smithi086.stdout:4/852: creat d0/dc0/f10f x:0 0 0 2024-04-06T14:17:24.663 INFO:tasks.workunit.client.1.smithi086.stdout:9/947: dwrite d5/d13/d47/d55/d87/f109 [0,4194304] 0 2024-04-06T14:17:24.667 INFO:tasks.workunit.client.1.smithi086.stdout:9/948: mknod d5/d13/d1a/c134 0 2024-04-06T14:17:24.675 INFO:tasks.workunit.client.0.smithi032.stdout:0/976: sync 2024-04-06T14:17:24.676 INFO:tasks.workunit.client.1.smithi086.stdout:9/949: rename d5/d13/d1a/d5f/d88/d14/d19/d25/deb to d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/d135 0 2024-04-06T14:17:24.676 INFO:tasks.workunit.client.0.smithi032.stdout:0/977: dread - d3/da/d25/d2b/d4a/d79/da8/f133 zero size 2024-04-06T14:17:24.680 INFO:tasks.workunit.client.1.smithi086.stdout:4/853: truncate d0/de/f11 6148177 0 2024-04-06T14:17:24.681 INFO:tasks.workunit.client.1.smithi086.stdout:4/854: dread - d0/d2c/d30/d24/d50/dd0/ffc zero size 2024-04-06T14:17:24.681 INFO:tasks.workunit.client.1.smithi086.stdout:4/855: readlink d0/d2c/d38/d54/lbc 0 2024-04-06T14:17:24.682 INFO:tasks.workunit.client.0.smithi032.stdout:0/978: link d3/da/d25/d2b/d80/db3/cff d3/da/d25/d2b/d4a/c144 0 2024-04-06T14:17:24.683 INFO:tasks.workunit.client.1.smithi086.stdout:4/856: symlink d0/d2c/d38/d54/d61/d7a/dc2/dd3/l110 0 2024-04-06T14:17:24.684 INFO:tasks.workunit.client.1.smithi086.stdout:4/857: fsync d0/d2c/d38/d54/d61/d7a/ff1 0 2024-04-06T14:17:24.685 INFO:tasks.workunit.client.1.smithi086.stdout:9/950: dread d5/d13/d1a/d5f/d88/d14/f22 [0,4194304] 0 2024-04-06T14:17:24.686 INFO:tasks.workunit.client.1.smithi086.stdout:4/858: creat d0/d2c/d30/d24/df6/f111 x:0 0 0 2024-04-06T14:17:24.689 INFO:tasks.workunit.client.1.smithi086.stdout:4/859: fdatasync d0/de/f11 0 2024-04-06T14:17:24.689 INFO:tasks.workunit.client.1.smithi086.stdout:4/860: readlink d0/d2c/d30/d24/da3/l58 0 2024-04-06T14:17:24.690 INFO:tasks.workunit.client.1.smithi086.stdout:9/951: rename d5/d13/d1a/c10e to d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/c136 0 2024-04-06T14:17:24.690 INFO:tasks.workunit.client.1.smithi086.stdout:9/952: stat d5/d13/d47/l104 0 2024-04-06T14:17:24.741 INFO:tasks.workunit.client.1.smithi086.stdout:5/865: sync 2024-04-06T14:17:24.742 INFO:tasks.workunit.client.1.smithi086.stdout:5/866: write d0/d43/da6/fc3 [910932,30840] 0 2024-04-06T14:17:24.782 INFO:tasks.workunit.client.1.smithi086.stdout:7/929: sync 2024-04-06T14:17:24.794 INFO:tasks.workunit.client.1.smithi086.stdout:9/953: creat d5/d13/d1a/d5f/d88/d14/d90/f137 x:0 0 0 2024-04-06T14:17:24.800 INFO:tasks.workunit.client.1.smithi086.stdout:9/954: dread d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/f10c [0,4194304] 0 2024-04-06T14:17:24.831 INFO:tasks.workunit.client.1.smithi086.stdout:4/861: unlink d0/dc0/la5 0 2024-04-06T14:17:24.834 INFO:tasks.workunit.client.1.smithi086.stdout:5/867: creat d0/d16/d18/d2a/f118 x:0 0 0 2024-04-06T14:17:24.836 INFO:tasks.workunit.client.1.smithi086.stdout:7/930: getdents d7/d10/d102/d5c 0 2024-04-06T14:17:24.837 INFO:tasks.workunit.client.1.smithi086.stdout:4/862: dwrite d0/d2c/d30/d24/d50/d79/d89/fae [0,4194304] 0 2024-04-06T14:17:24.837 INFO:tasks.workunit.client.1.smithi086.stdout:7/931: readlink d7/d10/d11/d21/l117 0 2024-04-06T14:17:24.839 INFO:tasks.workunit.client.1.smithi086.stdout:9/955: link d5/d13/f71 d5/d13/d1a/d5f/d88/f138 0 2024-04-06T14:17:24.840 INFO:tasks.workunit.client.1.smithi086.stdout:4/863: mkdir d0/d2c/d30/d24/d93/d112 0 2024-04-06T14:17:24.841 INFO:tasks.workunit.client.1.smithi086.stdout:7/932: symlink d7/d10/d11/d1c/d36/dbc/l14a 0 2024-04-06T14:17:24.845 INFO:tasks.workunit.client.1.smithi086.stdout:9/956: creat d5/d13/d47/d55/d87/ddc/d120/f139 x:0 0 0 2024-04-06T14:17:24.846 INFO:tasks.workunit.client.1.smithi086.stdout:5/868: dwrite d0/d16/d18/d35/f4d [0,4194304] 0 2024-04-06T14:17:24.847 INFO:tasks.workunit.client.1.smithi086.stdout:7/933: truncate d7/d10/d102/daf/db5/d7a/d100/d137/f139 327873 0 2024-04-06T14:17:24.849 INFO:tasks.workunit.client.1.smithi086.stdout:9/957: write f0 [1139118,107367] 0 2024-04-06T14:17:24.858 INFO:tasks.workunit.client.1.smithi086.stdout:4/864: dread d0/d2c/d38/f5b [0,4194304] 0 2024-04-06T14:17:24.860 INFO:tasks.workunit.client.1.smithi086.stdout:7/934: symlink d7/d10/d11/d1c/d13c/l14b 0 2024-04-06T14:17:24.861 INFO:tasks.workunit.client.1.smithi086.stdout:4/865: mkdir d0/d15/d20/d74/d8d/d113 0 2024-04-06T14:17:24.864 INFO:tasks.workunit.client.1.smithi086.stdout:9/958: dread d5/d13/d1a/d5f/d88/d14/d19/f1e [0,4194304] 0 2024-04-06T14:17:24.865 INFO:tasks.workunit.client.1.smithi086.stdout:5/869: write d0/d16/f4e [2699331,117414] 0 2024-04-06T14:17:24.867 INFO:tasks.workunit.client.1.smithi086.stdout:5/870: write d0/d16/f39 [3189011,1473] 0 2024-04-06T14:17:24.870 INFO:tasks.workunit.client.1.smithi086.stdout:7/935: link d7/d10/d102/daf/db5/d7a/c11f d7/d10/d102/d5c/d66/c14c 0 2024-04-06T14:17:24.871 INFO:tasks.workunit.client.1.smithi086.stdout:5/871: dread d0/d16/d18/d35/fa2 [4194304,4194304] 0 2024-04-06T14:17:24.872 INFO:tasks.workunit.client.1.smithi086.stdout:5/872: fdatasync d0/d43/de5/f104 0 2024-04-06T14:17:24.872 INFO:tasks.workunit.client.1.smithi086.stdout:4/866: getdents d0/d2c/d38/dd7 0 2024-04-06T14:17:24.874 INFO:tasks.workunit.client.1.smithi086.stdout:9/959: rename d5/d13/d1a/d5f/d88/d14/f86 to d5/d13/d1a/d5f/d88/d14/f13a 0 2024-04-06T14:17:24.879 INFO:tasks.workunit.client.1.smithi086.stdout:4/867: rename d0/d2c/d30/d24/d71/cb3 to d0/d2c/d30/d24/d50/dd0/c114 0 2024-04-06T14:17:24.880 INFO:tasks.workunit.client.1.smithi086.stdout:5/873: truncate d0/d16/f63 454573 0 2024-04-06T14:17:24.881 INFO:tasks.workunit.client.1.smithi086.stdout:5/874: chown d0/d16/d18/d21/d23/f6c 60593712 1 2024-04-06T14:17:24.882 INFO:tasks.workunit.client.1.smithi086.stdout:4/868: rename d0/de/d4c/f5a to d0/d2c/d30/d24/d50/d79/f115 0 2024-04-06T14:17:24.883 INFO:tasks.workunit.client.1.smithi086.stdout:7/936: dread d7/d10/d102/d5c/d66/f85 [0,4194304] 0 2024-04-06T14:17:24.883 INFO:tasks.workunit.client.1.smithi086.stdout:5/875: mknod d0/d16/d18/d21/dff/c119 0 2024-04-06T14:17:24.884 INFO:tasks.workunit.client.1.smithi086.stdout:7/937: chown d7/d10/d102/daf/db5/da1/fce 629034 1 2024-04-06T14:17:24.886 INFO:tasks.workunit.client.1.smithi086.stdout:5/876: creat d0/d16/d71/d96/deb/f11a x:0 0 0 2024-04-06T14:17:24.897 INFO:tasks.workunit.client.1.smithi086.stdout:4/869: dwrite d0/d2c/d30/d24/d50/dd0/ffc [0,4194304] 0 2024-04-06T14:17:24.909 INFO:tasks.workunit.client.0.smithi032.stdout:0/979: sync 2024-04-06T14:17:24.910 INFO:tasks.workunit.client.1.smithi086.stdout:4/870: dread d0/f3 [4194304,4194304] 0 2024-04-06T14:17:24.910 INFO:tasks.workunit.client.1.smithi086.stdout:7/938: dread d7/d10/d11/d1c/d24/d2c/d69/d8d/f105 [0,4194304] 0 2024-04-06T14:17:24.911 INFO:tasks.workunit.client.1.smithi086.stdout:4/871: write d0/d2c/d38/faa [2035745,48535] 0 2024-04-06T14:17:24.913 INFO:tasks.workunit.client.0.smithi032.stdout:0/980: dread d3/f63 [0,4194304] 0 2024-04-06T14:17:24.916 INFO:tasks.workunit.client.0.smithi032.stdout:0/981: truncate d3/f55 2025829 0 2024-04-06T14:17:24.918 INFO:tasks.workunit.client.0.smithi032.stdout:0/982: creat d3/d83/dfd/f145 x:0 0 0 2024-04-06T14:17:24.922 INFO:tasks.workunit.client.0.smithi032.stdout:0/983: getdents d3/da/d25/d57 0 2024-04-06T14:17:24.924 INFO:tasks.workunit.client.0.smithi032.stdout:0/984: rmdir d3/da/d25/d2b/d4a/d130 39 2024-04-06T14:17:24.926 INFO:tasks.workunit.client.0.smithi032.stdout:0/985: creat d3/dc2/d123/f146 x:0 0 0 2024-04-06T14:17:24.930 INFO:tasks.workunit.client.1.smithi086.stdout:5/877: rename d0/d43/de5 to d0/d16/d18/d21/dff/d11b 0 2024-04-06T14:17:24.932 INFO:tasks.workunit.client.1.smithi086.stdout:5/878: fdatasync d0/d16/d18/d21/d23/d28/d94/d52/fb1 0 2024-04-06T14:17:24.935 INFO:tasks.workunit.client.1.smithi086.stdout:4/872: rename d0/d15/d20/d74/d8d/fac to d0/d2c/d38/dd7/f116 0 2024-04-06T14:17:24.937 INFO:tasks.workunit.client.1.smithi086.stdout:4/873: write d0/d2c/d38/faa [1199177,93278] 0 2024-04-06T14:17:24.938 INFO:tasks.workunit.client.1.smithi086.stdout:5/879: rename d0/d16/d18/d21/dff/d11b/fb7 to d0/d16/d71/d96/f11c 0 2024-04-06T14:17:24.939 INFO:tasks.workunit.client.1.smithi086.stdout:7/939: dwrite d7/d1d/f3e [0,4194304] 0 2024-04-06T14:17:24.944 INFO:tasks.workunit.client.1.smithi086.stdout:5/880: rename d0/d16/d18/d21/dff/d11b/f3e to d0/d43/f11d 0 2024-04-06T14:17:24.945 INFO:tasks.workunit.client.1.smithi086.stdout:4/874: truncate d0/d2c/d30/d24/d71/f84 311424 0 2024-04-06T14:17:24.947 INFO:tasks.workunit.client.1.smithi086.stdout:5/881: mkdir d0/d43/da6/d11e 0 2024-04-06T14:17:24.949 INFO:tasks.workunit.client.1.smithi086.stdout:4/875: mkdir d0/d2c/d38/d54/d83/d117 0 2024-04-06T14:17:24.952 INFO:tasks.workunit.client.1.smithi086.stdout:4/876: mkdir d0/d2c/d38/d54/d83/d117/d118 0 2024-04-06T14:17:24.963 INFO:tasks.workunit.client.1.smithi086.stdout:7/940: truncate d7/d10/d92/f121 2213563 0 2024-04-06T14:17:24.965 INFO:tasks.workunit.client.1.smithi086.stdout:7/941: creat d7/d10/d11/d123/f14d x:0 0 0 2024-04-06T14:17:24.967 INFO:tasks.workunit.client.1.smithi086.stdout:4/877: dread d0/f5 [0,4194304] 0 2024-04-06T14:17:24.968 INFO:tasks.workunit.client.1.smithi086.stdout:7/942: mkdir d7/d10/d11/d1c/d14e 0 2024-04-06T14:17:24.973 INFO:tasks.workunit.client.1.smithi086.stdout:4/878: dwrite d0/dc0/dbb/fe5 [0,4194304] 0 2024-04-06T14:17:24.979 INFO:tasks.workunit.client.1.smithi086.stdout:4/879: creat d0/d2c/d30/d24/da3/f119 x:0 0 0 2024-04-06T14:17:24.982 INFO:tasks.workunit.client.1.smithi086.stdout:7/943: dwrite d7/d1d/f125 [0,4194304] 0 2024-04-06T14:17:24.983 INFO:tasks.workunit.client.1.smithi086.stdout:4/880: mknod d0/d2c/d38/d54/d61/d7a/c11a 0 2024-04-06T14:17:24.992 INFO:tasks.workunit.client.1.smithi086.stdout:4/881: dwrite d0/d2c/d38/d54/d61/d7a/dc2/fde [0,4194304] 0 2024-04-06T14:17:24.992 INFO:tasks.workunit.client.1.smithi086.stdout:7/944: dwrite d7/d10/d11/d1c/d24/dc8/fec [4194304,4194304] 0 2024-04-06T14:17:24.993 INFO:tasks.workunit.client.1.smithi086.stdout:7/945: stat d7/d10/d92 0 2024-04-06T14:17:24.999 INFO:tasks.workunit.client.1.smithi086.stdout:7/946: mknod d7/d10/d102/d5c/d66/d7e/db9/dd2/c14f 0 2024-04-06T14:17:25.001 INFO:tasks.workunit.client.1.smithi086.stdout:7/947: creat d7/d10/d11/d1c/d13c/f150 x:0 0 0 2024-04-06T14:17:25.002 INFO:tasks.workunit.client.1.smithi086.stdout:7/948: readlink d7/d10/d11/l17 0 2024-04-06T14:17:25.009 INFO:tasks.workunit.client.1.smithi086.stdout:7/949: dwrite d7/d10/d11/d1c/d24/df0/d116/f12d [0,4194304] 0 2024-04-06T14:17:25.009 INFO:tasks.workunit.client.1.smithi086.stdout:7/950: stat d7/d10/d11/c12c 0 2024-04-06T14:17:25.021 INFO:tasks.workunit.client.1.smithi086.stdout:9/960: sync 2024-04-06T14:17:25.023 INFO:tasks.workunit.client.1.smithi086.stdout:9/961: creat d5/d13/d1a/d5f/d88/d91/f13b x:0 0 0 2024-04-06T14:17:25.028 INFO:tasks.workunit.client.1.smithi086.stdout:7/951: fsync d7/d10/d11/d1c/d24/dc8/fec 0 2024-04-06T14:17:25.032 INFO:tasks.workunit.client.1.smithi086.stdout:7/952: dread d7/d1d/f125 [0,4194304] 0 2024-04-06T14:17:25.035 INFO:tasks.workunit.client.1.smithi086.stdout:7/953: creat d7/d10/d102/daf/db5/d7a/d100/d137/f151 x:0 0 0 2024-04-06T14:17:25.039 INFO:tasks.workunit.client.1.smithi086.stdout:7/954: rmdir d7/d10/d11/d1c/d24/d140 0 2024-04-06T14:17:25.040 INFO:tasks.workunit.client.1.smithi086.stdout:7/955: read - d7/d10/d102/daf/db5/ff3 zero size 2024-04-06T14:17:25.044 INFO:tasks.workunit.client.1.smithi086.stdout:7/956: creat d7/d10/d102/f152 x:0 0 0 2024-04-06T14:17:25.053 INFO:tasks.workunit.client.1.smithi086.stdout:7/957: dwrite d7/d10/d11/d1c/d24/fb2 [0,4194304] 0 2024-04-06T14:17:25.057 INFO:tasks.workunit.client.1.smithi086.stdout:7/958: symlink d7/d10/d11/d1c/d14e/l153 0 2024-04-06T14:17:25.060 INFO:tasks.workunit.client.1.smithi086.stdout:7/959: rename d7/d1d/l3b to d7/d10/d92/l154 0 2024-04-06T14:17:25.062 INFO:tasks.workunit.client.1.smithi086.stdout:7/960: truncate d7/fb 904813 0 2024-04-06T14:17:25.066 INFO:tasks.workunit.client.1.smithi086.stdout:7/961: rename d7/d10/d11/d1c/d24/ff2 to d7/d10/d102/d5c/d66/d7e/d103/f155 0 2024-04-06T14:17:25.068 INFO:tasks.workunit.client.1.smithi086.stdout:7/962: write d7/d10/f127 [576949,2686] 0 2024-04-06T14:17:25.073 INFO:tasks.workunit.client.1.smithi086.stdout:7/963: dwrite d7/d10/d11/d1c/d24/d2c/d69/f12a [0,4194304] 0 2024-04-06T14:17:25.075 INFO:tasks.workunit.client.1.smithi086.stdout:7/964: creat d7/f156 x:0 0 0 2024-04-06T14:17:25.079 INFO:tasks.workunit.client.1.smithi086.stdout:7/965: read d7/d1d/f6b [1403965,56034] 0 2024-04-06T14:17:25.082 INFO:tasks.workunit.client.1.smithi086.stdout:7/966: creat d7/f157 x:0 0 0 2024-04-06T14:17:25.085 INFO:tasks.workunit.client.1.smithi086.stdout:7/967: truncate d7/d10/d102/daf/db5/d7a/d100/d137/f139 1121119 0 2024-04-06T14:17:25.088 INFO:tasks.workunit.client.1.smithi086.stdout:7/968: symlink d7/d10/d11/d1c/d24/d2c/l158 0 2024-04-06T14:17:25.089 INFO:tasks.workunit.client.1.smithi086.stdout:7/969: chown d7/d10/d102/daf/db5/d7a/d100/d137/f151 11 1 2024-04-06T14:17:25.090 INFO:tasks.workunit.client.1.smithi086.stdout:7/970: write d7/d10/d11/d1c/d24/d2c/fb6 [2098491,70587] 0 2024-04-06T14:17:25.092 INFO:tasks.workunit.client.1.smithi086.stdout:7/971: readlink d7/d10/d92/lb4 0 2024-04-06T14:17:25.097 INFO:tasks.workunit.client.1.smithi086.stdout:7/972: truncate d7/d1d/f125 781181 0 2024-04-06T14:17:25.100 INFO:tasks.workunit.client.1.smithi086.stdout:7/973: rename d7/d1d/c5e to d7/d10/d11/d1c/d24/d2c/c159 0 2024-04-06T14:17:25.105 INFO:tasks.workunit.client.1.smithi086.stdout:7/974: dread d7/d10/d11/d1c/d24/d2c/fe3 [0,4194304] 0 2024-04-06T14:17:25.106 INFO:tasks.workunit.client.1.smithi086.stdout:7/975: fsync d7/d10/d11/d1c/d13c/f150 0 2024-04-06T14:17:25.108 INFO:tasks.workunit.client.1.smithi086.stdout:7/976: mkdir d7/d10/d11/d1c/d24/df0/d15a 0 2024-04-06T14:17:25.110 INFO:tasks.workunit.client.1.smithi086.stdout:7/977: mknod d7/d10/d11/d1c/d24/dc8/c15b 0 2024-04-06T14:17:25.110 INFO:tasks.workunit.client.0.smithi032.stdout:0/986: sync 2024-04-06T14:17:25.111 INFO:tasks.workunit.client.0.smithi032.stdout:0/987: write d3/dc2/dd8/fde [1206171,84724] 0 2024-04-06T14:17:25.111 INFO:tasks.workunit.client.1.smithi086.stdout:5/882: sync 2024-04-06T14:17:25.112 INFO:tasks.workunit.client.1.smithi086.stdout:7/978: rename d7/d10/d102/d5c/d66/cb1 to d7/d10/d11/d123/c15c 0 2024-04-06T14:17:25.113 INFO:tasks.workunit.client.1.smithi086.stdout:5/883: fdatasync d0/d16/d18/d21/dff/d11b/f26 0 2024-04-06T14:17:25.114 INFO:tasks.workunit.client.0.smithi032.stdout:0/988: truncate d3/f2e 1793566 0 2024-04-06T14:17:25.114 INFO:tasks.workunit.client.1.smithi086.stdout:7/979: creat d7/d10/d11/d1c/d24/dc8/dcd/f15d x:0 0 0 2024-04-06T14:17:25.116 INFO:tasks.workunit.client.0.smithi032.stdout:0/989: getdents d3/da/d41/d13e 0 2024-04-06T14:17:25.119 INFO:tasks.workunit.client.0.smithi032.stdout:0/990: mkdir d3/da/d41/d13e/d147 0 2024-04-06T14:17:25.119 INFO:tasks.workunit.client.1.smithi086.stdout:5/884: readlink d0/d16/d18/d35/ld0 0 2024-04-06T14:17:25.120 INFO:tasks.workunit.client.0.smithi032.stdout:0/991: fdatasync d3/dc2/dd8/fde 0 2024-04-06T14:17:25.121 INFO:tasks.workunit.client.0.smithi032.stdout:0/992: chown d3/da/d25/d2b/d4a/fb9 10693106 1 2024-04-06T14:17:25.121 INFO:tasks.workunit.client.1.smithi086.stdout:7/980: mkdir d7/d10/d11/d1c/d24/d2c/d69/d8d/d15e 0 2024-04-06T14:17:25.122 INFO:tasks.workunit.client.1.smithi086.stdout:7/981: dread - d7/d10/d102/f110 zero size 2024-04-06T14:17:25.125 INFO:tasks.workunit.client.1.smithi086.stdout:5/885: rename d0/d16/d71/d96/deb to d0/d16/d71/d96/d11f 0 2024-04-06T14:17:25.131 INFO:tasks.workunit.client.1.smithi086.stdout:5/886: dwrite d0/d16/d18/d21/d23/d28/d94/f56 [0,4194304] 0 2024-04-06T14:17:25.133 INFO:tasks.workunit.client.1.smithi086.stdout:7/982: write d7/d10/d11/d1c/d24/d2c/d69/f78 [356706,85854] 0 2024-04-06T14:17:25.133 INFO:tasks.workunit.client.1.smithi086.stdout:5/887: truncate d0/d16/d18/d2a/f118 644045 0 2024-04-06T14:17:25.136 INFO:tasks.workunit.client.1.smithi086.stdout:4/882: sync 2024-04-06T14:17:25.136 INFO:tasks.workunit.client.1.smithi086.stdout:9/962: sync 2024-04-06T14:17:25.139 INFO:tasks.workunit.client.1.smithi086.stdout:7/983: mkdir d7/d10/d11/d1c/d24/df0/d15a/d15f 0 2024-04-06T14:17:25.141 INFO:tasks.workunit.client.1.smithi086.stdout:4/883: fdatasync d0/d15/f91 0 2024-04-06T14:17:25.143 INFO:tasks.workunit.client.1.smithi086.stdout:9/963: readlink d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/ld5 0 2024-04-06T14:17:25.146 INFO:tasks.workunit.client.1.smithi086.stdout:5/888: dwrite d0/d16/d18/d21/dff/d11b/fe1 [0,4194304] 0 2024-04-06T14:17:25.147 INFO:tasks.workunit.client.1.smithi086.stdout:7/984: symlink d7/d10/d11/d1c/d24/df0/d116/l160 0 2024-04-06T14:17:25.148 INFO:tasks.workunit.client.1.smithi086.stdout:9/964: mkdir d5/d13/d1a/d5f/d88/d91/d13c 0 2024-04-06T14:17:25.149 INFO:tasks.workunit.client.1.smithi086.stdout:5/889: symlink d0/d16/d65/d10a/l120 0 2024-04-06T14:17:25.150 INFO:tasks.workunit.client.1.smithi086.stdout:7/985: write d7/d10/d11/d1c/d24/fb2 [2926503,12977] 0 2024-04-06T14:17:25.152 INFO:tasks.workunit.client.1.smithi086.stdout:5/890: fsync d0/d16/d18/d21/d23/d28/d94/d52/fb1 0 2024-04-06T14:17:25.152 INFO:tasks.workunit.client.1.smithi086.stdout:9/965: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/d13d 0 2024-04-06T14:17:25.157 INFO:tasks.workunit.client.0.smithi032.stdout:0/993: dwrite d3/da/d25/d2b/d4a/d79/d97/de0/df5/d12c/fbf [4194304,4194304] 0 2024-04-06T14:17:25.157 INFO:tasks.workunit.client.1.smithi086.stdout:9/966: mkdir d5/d13/d1a/d5f/d88/d13e 0 2024-04-06T14:17:25.158 INFO:tasks.workunit.client.0.smithi032.stdout:0/994: read d3/d83/dfd/f136 [1082858,25403] 0 2024-04-06T14:17:25.163 INFO:tasks.workunit.client.1.smithi086.stdout:5/891: dwrite d0/d16/d18/d21/d23/d28/d94/f4f [0,4194304] 0 2024-04-06T14:17:25.163 INFO:tasks.workunit.client.1.smithi086.stdout:7/986: dwrite d7/d10/d102/daf/db5/d7a/d100/f10a [0,4194304] 0 2024-04-06T14:17:25.171 INFO:tasks.workunit.client.1.smithi086.stdout:9/967: dread d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/f10c [0,4194304] 0 2024-04-06T14:17:25.172 INFO:tasks.workunit.client.1.smithi086.stdout:4/884: dwrite d0/f36 [0,4194304] 0 2024-04-06T14:17:25.175 INFO:tasks.workunit.client.1.smithi086.stdout:9/968: dread - d5/d13/d47/d5e/f110 zero size 2024-04-06T14:17:25.176 INFO:tasks.workunit.client.1.smithi086.stdout:5/892: creat d0/d16/d18/d35/d9e/dad/f121 x:0 0 0 2024-04-06T14:17:25.178 INFO:tasks.workunit.client.1.smithi086.stdout:4/885: chown d0/d2c/d30/d24/d50/d79/lc8 131799780 1 2024-04-06T14:17:25.180 INFO:tasks.workunit.client.1.smithi086.stdout:9/969: write d5/d13/d1a/d5f/d88/d14/f51 [4183410,110010] 0 2024-04-06T14:17:25.182 INFO:tasks.workunit.client.1.smithi086.stdout:5/893: dread - d0/d16/d18/d21/d23/d28/d94/d52/fb1 zero size 2024-04-06T14:17:25.183 INFO:tasks.workunit.client.1.smithi086.stdout:7/987: dwrite d7/d1d/f124 [0,4194304] 0 2024-04-06T14:17:25.183 INFO:tasks.workunit.client.1.smithi086.stdout:4/886: fsync d0/d2c/d30/d24/fa2 0 2024-04-06T14:17:25.184 INFO:tasks.workunit.client.1.smithi086.stdout:5/894: readlink d0/d43/d74/l8b 0 2024-04-06T14:17:25.186 INFO:tasks.workunit.client.1.smithi086.stdout:7/988: mknod d7/d1d/c161 0 2024-04-06T14:17:25.187 INFO:tasks.workunit.client.1.smithi086.stdout:7/989: stat d7/d10/d102/daf/db5/f6e 0 2024-04-06T14:17:25.188 INFO:tasks.workunit.client.1.smithi086.stdout:7/990: stat d7/d1d/fb8 0 2024-04-06T14:17:25.189 INFO:tasks.workunit.client.1.smithi086.stdout:9/970: truncate d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/d135/f133 420539 0 2024-04-06T14:17:25.192 INFO:tasks.workunit.client.1.smithi086.stdout:4/887: rename d0/d2c/d38/d54/d83/d117/d118 to d0/d2c/d38/d54/d61/d7a/dc2/d11b 0 2024-04-06T14:17:25.192 INFO:tasks.workunit.client.1.smithi086.stdout:9/971: write d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/dce/f113 [495618,38655] 0 2024-04-06T14:17:25.194 INFO:tasks.workunit.client.1.smithi086.stdout:5/895: getdents d0/d5e 0 2024-04-06T14:17:25.196 INFO:tasks.workunit.client.1.smithi086.stdout:4/888: fdatasync d0/d2c/d38/d54/d83/fd1 0 2024-04-06T14:17:25.196 INFO:tasks.workunit.client.1.smithi086.stdout:5/896: readlink d0/d16/d71/lfb 0 2024-04-06T14:17:25.199 INFO:tasks.workunit.client.1.smithi086.stdout:7/991: dwrite d7/d10/d11/d1c/d24/dc8/dcd/f15d [0,4194304] 0 2024-04-06T14:17:25.199 INFO:tasks.workunit.client.1.smithi086.stdout:4/889: write d0/d15/f37 [1853477,123802] 0 2024-04-06T14:17:25.201 INFO:tasks.workunit.client.1.smithi086.stdout:9/972: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/d13d/f13f x:0 0 0 2024-04-06T14:17:25.206 INFO:tasks.workunit.client.1.smithi086.stdout:5/897: creat d0/d16/d18/d21/d23/d28/df4/f122 x:0 0 0 2024-04-06T14:17:25.206 INFO:tasks.workunit.client.1.smithi086.stdout:4/890: mknod d0/d2c/d30/d24/d50/d79/d89/c11c 0 2024-04-06T14:17:25.206 INFO:tasks.workunit.client.1.smithi086.stdout:9/973: mkdir d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/ddb/d140 0 2024-04-06T14:17:25.206 INFO:tasks.workunit.client.1.smithi086.stdout:5/898: mknod d0/d16/d65/d10f/c123 0 2024-04-06T14:17:25.207 INFO:tasks.workunit.client.1.smithi086.stdout:9/974: read d5/d13/d47/d55/f64 [3616902,18720] 0 2024-04-06T14:17:25.207 INFO:tasks.workunit.client.1.smithi086.stdout:4/891: mkdir d0/d2c/d38/d54/d61/d7a/dc2/d11b/d11d 0 2024-04-06T14:17:25.207 INFO:tasks.workunit.client.1.smithi086.stdout:5/899: dread d0/d16/d18/d2a/f118 [0,4194304] 0 2024-04-06T14:17:25.210 INFO:tasks.workunit.client.1.smithi086.stdout:5/900: mknod d0/d16/d18/d21/d23/d28/d94/d80/d85/c124 0 2024-04-06T14:17:25.211 INFO:tasks.workunit.client.1.smithi086.stdout:4/892: getdents d0/db0 0 2024-04-06T14:17:25.214 INFO:tasks.workunit.client.1.smithi086.stdout:5/901: symlink d0/d16/d18/d21/d23/l125 0 2024-04-06T14:17:25.215 INFO:tasks.workunit.client.1.smithi086.stdout:4/893: dread d0/d2c/d38/d54/d83/fd1 [0,4194304] 0 2024-04-06T14:17:25.220 INFO:tasks.workunit.client.1.smithi086.stdout:5/902: write d0/d16/d71/f76 [1652542,24126] 0 2024-04-06T14:17:25.220 INFO:tasks.workunit.client.1.smithi086.stdout:4/894: write d0/d2c/d38/d54/d61/d7a/dc2/ff4 [1103349,86267] 0 2024-04-06T14:17:25.221 INFO:tasks.workunit.client.1.smithi086.stdout:7/992: dwrite d7/d10/d11/d1c/f2e [0,4194304] 0 2024-04-06T14:17:25.226 INFO:tasks.workunit.client.1.smithi086.stdout:9/975: dwrite d5/d13/d1a/d5f/d88/d14/d19/d25/f2f [0,4194304] 0 2024-04-06T14:17:25.226 INFO:tasks.workunit.client.1.smithi086.stdout:5/903: creat d0/d16/d65/d10a/f126 x:0 0 0 2024-04-06T14:17:25.227 INFO:tasks.workunit.client.1.smithi086.stdout:4/895: mknod d0/c11e 0 2024-04-06T14:17:25.229 INFO:tasks.workunit.client.1.smithi086.stdout:5/904: creat d0/d16/d65/f127 x:0 0 0 2024-04-06T14:17:25.231 INFO:tasks.workunit.client.1.smithi086.stdout:5/905: chown d0/d16/d18/d21/d23/d28/d94/d8a/le3 8350 1 2024-04-06T14:17:25.232 INFO:tasks.workunit.client.1.smithi086.stdout:5/906: stat d0/d16/d71/d96/ddc 0 2024-04-06T14:17:25.234 INFO:tasks.workunit.client.1.smithi086.stdout:4/896: link d0/de/d4c/l70 d0/d2c/d30/d24/d50/d79/d89/l11f 0 2024-04-06T14:17:25.234 INFO:tasks.workunit.client.1.smithi086.stdout:7/993: dread d7/d10/d11/d1c/d36/f74 [0,4194304] 0 2024-04-06T14:17:25.235 INFO:tasks.workunit.client.1.smithi086.stdout:5/907: rmdir d0/d16/d65/d10f 39 2024-04-06T14:17:25.236 INFO:tasks.workunit.client.1.smithi086.stdout:4/897: creat d0/d15/d20/d34/f120 x:0 0 0 2024-04-06T14:17:25.240 INFO:tasks.workunit.client.1.smithi086.stdout:5/908: rename d0/d16/d71/d89/lb6 to d0/d16/d18/d21/d23/d28/df7/l128 0 2024-04-06T14:17:25.245 INFO:tasks.workunit.client.1.smithi086.stdout:7/994: truncate d7/d10/d11/fbe 2050757 0 2024-04-06T14:17:25.245 INFO:tasks.workunit.client.1.smithi086.stdout:4/898: dwrite d0/dc0/f10f [0,4194304] 0 2024-04-06T14:17:25.246 INFO:tasks.workunit.client.1.smithi086.stdout:7/995: dread - d7/d10/d102/f110 zero size 2024-04-06T14:17:25.249 INFO:tasks.workunit.client.1.smithi086.stdout:7/996: creat d7/d10/d11/d1c/d24/dc8/dcd/f162 x:0 0 0 2024-04-06T14:17:25.250 INFO:tasks.workunit.client.1.smithi086.stdout:4/899: getdents d0/d2c/d30/d24/d5d/d8a 0 2024-04-06T14:17:25.251 INFO:tasks.workunit.client.1.smithi086.stdout:5/909: dread d0/d16/f67 [0,4194304] 0 2024-04-06T14:17:25.252 INFO:tasks.workunit.client.1.smithi086.stdout:9/976: dread d5/d13/f7d [0,4194304] 0 2024-04-06T14:17:25.253 INFO:tasks.workunit.client.1.smithi086.stdout:4/900: unlink d0/d15/d20/c106 0 2024-04-06T14:17:25.254 INFO:tasks.workunit.client.1.smithi086.stdout:7/997: write d7/d10/d11/d1c/d24/d2c/d69/d8d/f105 [3068437,95979] 0 2024-04-06T14:17:25.254 INFO:tasks.workunit.client.1.smithi086.stdout:4/901: chown d0/d15/c97 470 1 2024-04-06T14:17:25.254 INFO:tasks.workunit.client.1.smithi086.stdout:5/910: rmdir d0/d16/d18/d35/d9e/de7 39 2024-04-06T14:17:25.256 INFO:tasks.workunit.client.1.smithi086.stdout:7/998: mkdir d7/d1d/d163 0 2024-04-06T14:17:25.257 INFO:tasks.workunit.client.1.smithi086.stdout:9/977: unlink d5/d13/d1a/d5f/d88/d14/d19/d25/d74/f97 0 2024-04-06T14:17:25.258 INFO:tasks.workunit.client.1.smithi086.stdout:7/999: write d7/d10/d11/d1c/d24/dc8/fda [2189901,70757] 0 2024-04-06T14:17:25.259 INFO:tasks.workunit.client.1.smithi086.stdout:4/902: dread d0/f10 [0,4194304] 0 2024-04-06T14:17:25.259 INFO:tasks.workunit.client.1.smithi086.stdout:9/978: stat d5/d13/d47/d55/fcf 0 2024-04-06T14:17:25.262 INFO:tasks.workunit.client.1.smithi086.stdout:5/911: getdents d0/d16/d71/d96 0 2024-04-06T14:17:25.264 INFO:tasks.workunit.client.1.smithi086.stdout:4/903: dread d0/d2c/d38/f5b [0,4194304] 0 2024-04-06T14:17:25.265 INFO:tasks.workunit.client.1.smithi086.stdout:9/979: rename d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/dc5/lf4 to d5/d13/d1a/d5f/d88/d13e/l141 0 2024-04-06T14:17:25.266 INFO:tasks.workunit.client.1.smithi086.stdout:5/912: rmdir d0/d16/d18/d21/d23/d28/d94/d8a 39 2024-04-06T14:17:25.270 INFO:tasks.workunit.client.1.smithi086.stdout:5/913: creat d0/d16/d71/d96/ddc/f129 x:0 0 0 2024-04-06T14:17:25.271 INFO:tasks.workunit.client.1.smithi086.stdout:9/980: dwrite d5/d13/d1a/d5f/fc8 [0,4194304] 0 2024-04-06T14:17:25.273 INFO:tasks.workunit.client.1.smithi086.stdout:9/981: fsync d5/d13/d1a/d5f/d88/d14/d19/d25/d74/dab/db5/f11e 0 2024-04-06T14:17:25.273 INFO:tasks.workunit.client.1.smithi086.stdout:5/914: chown d0/d16/d18/de6/lf5 1 1 2024-04-06T14:17:25.273 INFO:tasks.workunit.client.1.smithi086.stdout:4/904: link d0/d2c/d30/d24/d71/f84 d0/d2c/d30/d24/da3/d88/f121 0 2024-04-06T14:17:25.275 INFO:tasks.workunit.client.1.smithi086.stdout:4/905: write d0/d2c/d38/d54/d83/fd1 [2097829,34701] 0 2024-04-06T14:17:25.276 INFO:tasks.workunit.client.1.smithi086.stdout:9/982: creat d5/d13/d1a/d5f/d88/d91/f142 x:0 0 0 2024-04-06T14:17:25.279 INFO:tasks.workunit.client.1.smithi086.stdout:5/915: mkdir d0/d43/da6/d11e/d12a 0 2024-04-06T14:17:25.280 INFO:tasks.workunit.client.1.smithi086.stdout:4/906: unlink d0/d2c/d30/d24/da3/f102 0 2024-04-06T14:17:25.281 INFO:tasks.workunit.client.1.smithi086.stdout:9/983: mknod d5/d58/c143 0 2024-04-06T14:17:25.282 INFO:tasks.workunit.client.1.smithi086.stdout:9/984: write d5/d13/d47/d55/fcf [552365,12942] 0 2024-04-06T14:17:25.284 INFO:tasks.workunit.client.1.smithi086.stdout:5/916: rmdir d0/d43/d74/dee 0 2024-04-06T14:17:25.286 INFO:tasks.workunit.client.1.smithi086.stdout:4/907: getdents d0/d2c/d38/d54/d61/d7a/dc2/dd3 0 2024-04-06T14:17:25.291 INFO:tasks.workunit.client.1.smithi086.stdout:4/908: dwrite d0/d15/d20/f44 [0,4194304] 0 2024-04-06T14:17:25.293 INFO:tasks.workunit.client.1.smithi086.stdout:4/909: creat d0/d2c/d30/d24/d93/f122 x:0 0 0 2024-04-06T14:17:25.296 INFO:tasks.workunit.client.1.smithi086.stdout:4/910: symlink d0/d2c/d30/d24/d5d/dfe/l123 0 2024-04-06T14:17:25.298 INFO:tasks.workunit.client.1.smithi086.stdout:4/911: write d0/d2c/d30/d24/d50/d79/f7c [4142653,110629] 0 2024-04-06T14:17:25.305 INFO:tasks.workunit.client.1.smithi086.stdout:4/912: dwrite d0/d2c/d30/d24/da3/d88/fcc [0,4194304] 0 2024-04-06T14:17:25.305 INFO:tasks.workunit.client.1.smithi086.stdout:5/917: dread d0/d16/d18/d21/d23/d28/d94/f4f [4194304,4194304] 0 2024-04-06T14:17:25.306 INFO:tasks.workunit.client.1.smithi086.stdout:5/918: chown d0/l64 255353177 1 2024-04-06T14:17:25.308 INFO:tasks.workunit.client.1.smithi086.stdout:4/913: mkdir d0/dc0/dbb/d124 0 2024-04-06T14:17:25.314 INFO:tasks.workunit.client.1.smithi086.stdout:4/914: truncate d0/d2c/d30/d24/fa2 221564 0 2024-04-06T14:17:25.317 INFO:tasks.workunit.client.1.smithi086.stdout:5/919: dwrite d0/f3 [0,4194304] 0 2024-04-06T14:17:25.319 INFO:tasks.workunit.client.1.smithi086.stdout:4/915: link d0/d2c/d30/d24/d5d/d8a/f9a d0/d2c/d38/d54/d83/d117/f125 0 2024-04-06T14:17:25.321 INFO:tasks.workunit.client.1.smithi086.stdout:4/916: write d0/d2c/d30/d24/da3/f52 [2017668,127183] 0 2024-04-06T14:17:25.333 INFO:tasks.workunit.client.1.smithi086.stdout:5/920: dread d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc5 [0,4194304] 0 2024-04-06T14:17:25.335 INFO:tasks.workunit.client.1.smithi086.stdout:5/921: chown d0/d16/d18/d2a/c6f 3381825 1 2024-04-06T14:17:25.337 INFO:tasks.workunit.client.1.smithi086.stdout:4/917: dwrite d0/d2c/d30/f6c [0,4194304] 0 2024-04-06T14:17:25.338 INFO:tasks.workunit.client.1.smithi086.stdout:5/922: mknod d0/d16/d71/d89/c12b 0 2024-04-06T14:17:25.345 INFO:tasks.workunit.client.1.smithi086.stdout:5/923: getdents d0/d16/d71/d96/d11f 0 2024-04-06T14:17:25.347 INFO:tasks.workunit.client.1.smithi086.stdout:4/918: dwrite d0/f59 [0,4194304] 0 2024-04-06T14:17:25.349 INFO:tasks.workunit.client.1.smithi086.stdout:4/919: chown d0/d2c/d30/d24/d50/d79/d89/c11c 365122828 1 2024-04-06T14:17:25.353 INFO:tasks.workunit.client.1.smithi086.stdout:4/920: write d0/d2c/d38/d54/d61/d7a/dc2/ff4 [2654805,50874] 0 2024-04-06T14:17:25.355 INFO:tasks.workunit.client.1.smithi086.stdout:5/924: dwrite d0/d16/d18/d21/dff/d11b/f104 [0,4194304] 0 2024-04-06T14:17:25.356 INFO:tasks.workunit.client.1.smithi086.stdout:4/921: write d0/d2c/d38/d54/d61/d7a/dc2/ff4 [3796961,43408] 0 2024-04-06T14:17:25.360 INFO:tasks.workunit.client.0.smithi032.stdout:0/995: sync 2024-04-06T14:17:25.362 INFO:tasks.workunit.client.1.smithi086.stdout:5/925: getdents d0/d16/d18/d35/d9e/dd7 0 2024-04-06T14:17:25.366 INFO:tasks.workunit.client.1.smithi086.stdout:5/926: creat d0/d16/d18/d35/d9e/de7/f12c x:0 0 0 2024-04-06T14:17:25.367 INFO:tasks.workunit.client.0.smithi032.stdout:0/996: getdents d3/da/d25/d57/db5/dc9 0 2024-04-06T14:17:25.369 INFO:tasks.workunit.client.1.smithi086.stdout:5/927: chown d0/d16/d18/d2a/d50/cd4 12 1 2024-04-06T14:17:25.370 INFO:tasks.workunit.client.1.smithi086.stdout:5/928: readlink d0/d16/d71/d96/lcc 0 2024-04-06T14:17:25.370 INFO:tasks.workunit.client.0.smithi032.stdout:0/997: dread d3/da/d25/d2b/d4a/f62 [0,4194304] 0 2024-04-06T14:17:25.372 INFO:tasks.workunit.client.1.smithi086.stdout:4/922: dwrite d0/d15/d2b/f5f [0,4194304] 0 2024-04-06T14:17:25.372 INFO:tasks.workunit.client.1.smithi086.stdout:5/929: dread - d0/d16/d18/d21/d23/d28/df4/f122 zero size 2024-04-06T14:17:25.373 INFO:tasks.workunit.client.1.smithi086.stdout:5/930: read - d0/d16/d71/d96/f11c zero size 2024-04-06T14:17:25.379 INFO:tasks.workunit.client.1.smithi086.stdout:4/923: dwrite d0/d15/d20/fc9 [0,4194304] 0 2024-04-06T14:17:25.381 INFO:tasks.workunit.client.1.smithi086.stdout:4/924: chown d0/d2c/d38/faa 7904445 1 2024-04-06T14:17:25.381 INFO:tasks.workunit.client.1.smithi086.stdout:5/931: dread d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc0 [8388608,4194304] 0 2024-04-06T14:17:25.382 INFO:tasks.workunit.client.1.smithi086.stdout:4/925: fsync d0/d2c/d30/d24/da3/d88/fcc 0 2024-04-06T14:17:25.385 INFO:tasks.workunit.client.1.smithi086.stdout:5/932: dread d0/d16/d18/d21/dff/d11b/f26 [0,4194304] 0 2024-04-06T14:17:25.388 INFO:tasks.workunit.client.1.smithi086.stdout:5/933: symlink d0/d43/d74/l12d 0 2024-04-06T14:17:25.389 INFO:tasks.workunit.client.1.smithi086.stdout:4/926: dread d0/d15/f91 [0,4194304] 0 2024-04-06T14:17:25.390 INFO:tasks.workunit.client.0.smithi032.stdout:0/998: dwrite d3/f14 [0,4194304] 0 2024-04-06T14:17:25.391 INFO:tasks.workunit.client.1.smithi086.stdout:5/934: creat d0/f12e x:0 0 0 2024-04-06T14:17:25.392 INFO:tasks.workunit.client.0.smithi032.stdout:0/999: chown d3/da/d25/d2b/d80/db3/dcd 21113868 1 2024-04-06T14:17:25.394 INFO:tasks.workunit.client.1.smithi086.stdout:5/935: fsync d0/d16/d71/d96/f11c 0 2024-04-06T14:17:25.398 INFO:tasks.workunit.client.1.smithi086.stdout:5/936: creat d0/d16/d18/d21/d23/d28/d94/d52/d103/f12f x:0 0 0 2024-04-06T14:17:25.401 INFO:tasks.workunit.client.1.smithi086.stdout:4/927: truncate d0/dc0/dbb/fe5 2409588 0 2024-04-06T14:17:25.402 INFO:tasks.workunit.client.1.smithi086.stdout:5/937: mkdir d0/d16/d18/d21/d23/d28/df7/d130 0 2024-04-06T14:17:25.406 INFO:tasks.workunit.client.0.smithi032.stderr:+ rm -rf -- ./tmp.mbqFNLcCvl 2024-04-06T14:17:25.407 INFO:tasks.workunit.client.1.smithi086.stdout:5/938: creat d0/d16/d18/d2a/f131 x:0 0 0 2024-04-06T14:17:25.409 INFO:tasks.workunit.client.1.smithi086.stdout:5/939: mknod d0/d16/d71/d96/d11f/c132 0 2024-04-06T14:17:25.412 INFO:tasks.workunit.client.1.smithi086.stdout:5/940: creat d0/d16/d65/d10a/f133 x:0 0 0 2024-04-06T14:17:25.413 INFO:tasks.workunit.client.1.smithi086.stdout:4/928: write d0/d2c/d30/d24/d5d/f8f [2697063,100135] 0 2024-04-06T14:17:25.413 INFO:tasks.workunit.client.1.smithi086.stdout:5/941: chown d0/d43/d74/l8b 11859 1 2024-04-06T14:17:25.418 INFO:tasks.workunit.client.1.smithi086.stdout:4/929: write d0/f2 [8706880,81095] 0 2024-04-06T14:17:25.418 INFO:tasks.workunit.client.1.smithi086.stdout:5/942: link d0/d16/c6d d0/d16/d18/d21/d23/d28/df4/c134 0 2024-04-06T14:17:25.421 INFO:tasks.workunit.client.1.smithi086.stdout:5/943: creat d0/d16/d18/d21/d23/d28/d94/d80/f135 x:0 0 0 2024-04-06T14:17:25.426 INFO:tasks.workunit.client.1.smithi086.stdout:4/930: truncate d0/de/d4c/fc5 2260060 0 2024-04-06T14:17:25.426 INFO:tasks.workunit.client.1.smithi086.stdout:5/944: write d0/d16/d18/d21/d23/d28/df7/fcd [1162030,126918] 0 2024-04-06T14:17:25.429 INFO:tasks.workunit.client.1.smithi086.stdout:9/985: sync 2024-04-06T14:17:25.429 INFO:tasks.workunit.client.1.smithi086.stdout:4/931: creat d0/d2c/d38/d54/d61/d7a/dc2/d11b/d11d/f126 x:0 0 0 2024-04-06T14:17:25.430 INFO:tasks.workunit.client.1.smithi086.stdout:5/945: creat d0/d16/d18/d21/d23/d28/d94/d80/f136 x:0 0 0 2024-04-06T14:17:25.432 INFO:tasks.workunit.client.1.smithi086.stdout:9/986: creat d5/d13/d47/d55/d87/ddc/d120/f144 x:0 0 0 2024-04-06T14:17:25.434 INFO:tasks.workunit.client.1.smithi086.stdout:5/946: creat d0/d16/d18/d21/d23/d28/f137 x:0 0 0 2024-04-06T14:17:25.437 INFO:tasks.workunit.client.1.smithi086.stdout:4/932: dwrite d0/d2c/d38/d54/fb1 [0,4194304] 0 2024-04-06T14:17:25.437 INFO:tasks.workunit.client.1.smithi086.stdout:5/947: dread - d0/d16/d18/d35/d9e/de7/f10e zero size 2024-04-06T14:17:25.441 INFO:tasks.workunit.client.1.smithi086.stdout:9/987: dwrite d5/d13/d1a/d5f/d88/d14/f101 [0,4194304] 0 2024-04-06T14:17:25.443 INFO:tasks.workunit.client.1.smithi086.stdout:4/933: mknod d0/d2c/d30/d24/d5d/d101/c127 0 2024-04-06T14:17:25.446 INFO:tasks.workunit.client.1.smithi086.stdout:4/934: write d0/f2 [8278536,38933] 0 2024-04-06T14:17:25.449 INFO:tasks.workunit.client.1.smithi086.stdout:4/935: write d0/d15/d20/d74/fef [443011,125267] 0 2024-04-06T14:17:25.450 INFO:tasks.workunit.client.1.smithi086.stdout:9/988: rename d5/d13/d1a/d5f/d88/d14/d19/f5d to d5/d13/d1a/d5f/d88/d14/d90/f145 0 2024-04-06T14:17:25.450 INFO:tasks.workunit.client.1.smithi086.stdout:5/948: dwrite d0/d16/d18/d35/f101 [0,4194304] 0 2024-04-06T14:17:25.455 INFO:tasks.workunit.client.1.smithi086.stdout:4/936: symlink d0/d15/d20/d74/l128 0 2024-04-06T14:17:25.457 INFO:tasks.workunit.client.1.smithi086.stdout:9/989: dwrite d5/d13/d1a/d5f/d88/d14/f101 [0,4194304] 0 2024-04-06T14:17:25.457 INFO:tasks.workunit.client.1.smithi086.stdout:5/949: symlink d0/d16/d18/d21/dff/d11b/l138 0 2024-04-06T14:17:25.461 INFO:tasks.workunit.client.1.smithi086.stdout:5/950: mknod d0/d16/d18/d35/d9e/c139 0 2024-04-06T14:17:25.471 INFO:tasks.workunit.client.1.smithi086.stdout:4/937: getdents d0/d2c/d30 0 2024-04-06T14:17:25.471 INFO:tasks.workunit.client.1.smithi086.stdout:5/951: mknod d0/d16/d18/d35/d9e/c13a 0 2024-04-06T14:17:25.473 INFO:tasks.workunit.client.1.smithi086.stdout:9/990: creat d5/d13/d1a/d5f/d88/d14/d19/d25/d59/f146 x:0 0 0 2024-04-06T14:17:25.475 INFO:tasks.workunit.client.1.smithi086.stdout:5/952: truncate d0/f1 991576 0 2024-04-06T14:17:25.476 INFO:tasks.workunit.client.1.smithi086.stdout:9/991: mkdir d5/d13/d1a/d5f/d88/d14/d19/d94/d147 0 2024-04-06T14:17:25.479 INFO:tasks.workunit.client.1.smithi086.stdout:5/953: rename d0/d16/d18/d21/dff/d11b/l53 to d0/d16/d18/d2a/d50/l13b 0 2024-04-06T14:17:25.480 INFO:tasks.workunit.client.1.smithi086.stdout:5/954: stat d0/d16/d18/d21/d23/l37 0 2024-04-06T14:17:25.480 INFO:tasks.workunit.client.1.smithi086.stdout:4/938: dread d0/d2c/d30/f3b [0,4194304] 0 2024-04-06T14:17:25.481 INFO:tasks.workunit.client.1.smithi086.stdout:4/939: write d0/f36 [2397113,18116] 0 2024-04-06T14:17:25.482 INFO:tasks.workunit.client.1.smithi086.stdout:9/992: link d5/d58/d100/c132 d5/d13/d1a/d5f/d88/d14/d19/d94/c148 0 2024-04-06T14:17:25.484 INFO:tasks.workunit.client.1.smithi086.stdout:9/993: symlink d5/d13/d47/d55/d87/ddc/l149 0 2024-04-06T14:17:25.486 INFO:tasks.workunit.client.1.smithi086.stdout:4/940: dread d0/dc0/dbb/fe5 [0,4194304] 0 2024-04-06T14:17:25.490 INFO:tasks.workunit.client.1.smithi086.stdout:4/941: creat d0/d15/d20/f129 x:0 0 0 2024-04-06T14:17:25.494 INFO:tasks.workunit.client.1.smithi086.stdout:4/942: dread d0/d15/d20/fc9 [0,4194304] 0 2024-04-06T14:17:25.495 INFO:tasks.workunit.client.1.smithi086.stdout:4/943: chown d0/d15/d20/c73 49 1 2024-04-06T14:17:25.496 INFO:tasks.workunit.client.1.smithi086.stdout:4/944: truncate d0/d2c/d30/d24/da3/f119 627470 0 2024-04-06T14:17:25.498 INFO:tasks.workunit.client.1.smithi086.stdout:9/994: truncate d5/d13/d1a/d5f/d88/d14/f4e 4269978 0 2024-04-06T14:17:25.498 INFO:tasks.workunit.client.1.smithi086.stdout:4/945: chown d0/d2c/d38/d54/d61/cc6 2003 1 2024-04-06T14:17:25.501 INFO:tasks.workunit.client.1.smithi086.stdout:4/946: creat d0/d2c/d38/d54/d61/d7a/dc2/f12a x:0 0 0 2024-04-06T14:17:25.502 INFO:tasks.workunit.client.1.smithi086.stdout:4/947: chown d0/d2c/d38/lfb 17312859 1 2024-04-06T14:17:25.510 INFO:tasks.workunit.client.1.smithi086.stdout:4/948: truncate d0/d2c/d30/d24/d50/dd0/ffc 32242 0 2024-04-06T14:17:25.511 INFO:tasks.workunit.client.1.smithi086.stdout:4/949: fdatasync d0/d15/f48 0 2024-04-06T14:17:25.511 INFO:tasks.workunit.client.1.smithi086.stdout:9/995: dread d5/d13/d47/d5e/f61 [0,4194304] 0 2024-04-06T14:17:25.516 INFO:tasks.workunit.client.1.smithi086.stdout:4/950: dread d0/dc0/f10f [0,4194304] 0 2024-04-06T14:17:25.519 INFO:tasks.workunit.client.1.smithi086.stdout:9/996: write d5/d13/f8c [499234,101286] 0 2024-04-06T14:17:25.519 INFO:tasks.workunit.client.1.smithi086.stdout:4/951: write d0/d2c/d38/f5b [520814,28353] 0 2024-04-06T14:17:25.522 INFO:tasks.workunit.client.1.smithi086.stdout:4/952: symlink d0/d2c/d30/d24/da3/d88/l12b 0 2024-04-06T14:17:25.526 INFO:tasks.workunit.client.1.smithi086.stdout:4/953: mkdir d0/d2c/d30/d24/d93/d112/d12c 0 2024-04-06T14:17:25.526 INFO:tasks.workunit.client.1.smithi086.stdout:9/997: truncate d5/d13/d1a/d5f/d88/d14/f125 2130175 0 2024-04-06T14:17:25.527 INFO:tasks.workunit.client.1.smithi086.stdout:9/998: chown d5/d13/d47/d55/d87/ddc 0 1 2024-04-06T14:17:25.531 INFO:tasks.workunit.client.1.smithi086.stdout:4/954: link d0/d2c/d38/dd7/f116 d0/d15/d2b/f12d 0 2024-04-06T14:17:25.532 INFO:tasks.workunit.client.1.smithi086.stdout:9/999: mknod d5/d13/d1a/d5f/d88/d14/c14a 0 2024-04-06T14:17:25.534 INFO:tasks.workunit.client.1.smithi086.stdout:4/955: mknod d0/d2c/d30/d24/d93/c12e 0 2024-04-06T14:17:25.534 INFO:tasks.workunit.client.1.smithi086.stdout:4/956: chown d0/d15/d20/c56 1919 1 2024-04-06T14:17:25.536 INFO:tasks.workunit.client.1.smithi086.stdout:4/957: dread d0/d2c/d30/d24/d71/f84 [0,4194304] 0 2024-04-06T14:17:25.550 INFO:tasks.workunit.client.1.smithi086.stdout:4/958: dread d0/d2c/d38/d54/fa1 [0,4194304] 0 2024-04-06T14:17:25.552 INFO:tasks.workunit.client.1.smithi086.stdout:4/959: write d0/d2c/d38/d54/d61/d7a/dc2/f12a [597737,112604] 0 2024-04-06T14:17:25.553 INFO:tasks.workunit.client.1.smithi086.stdout:4/960: fdatasync d0/d15/d20/d34/fcd 0 2024-04-06T14:17:25.553 INFO:tasks.workunit.client.1.smithi086.stdout:4/961: chown d0/f36 236068124 1 2024-04-06T14:17:25.556 INFO:tasks.workunit.client.1.smithi086.stdout:4/962: symlink d0/d2c/d38/d54/d61/d7a/dc2/d11b/l12f 0 2024-04-06T14:17:25.557 INFO:tasks.workunit.client.1.smithi086.stdout:4/963: mknod d0/d2c/d30/d24/d5d/d101/c130 0 2024-04-06T14:17:25.561 INFO:tasks.workunit.client.1.smithi086.stdout:4/964: rename d0/d2c/d30/d24/da3/l58 to d0/d15/l131 0 2024-04-06T14:17:25.570 INFO:tasks.workunit.client.1.smithi086.stdout:4/965: dread d0/d2c/d30/d24/d5d/f8f [0,4194304] 0 2024-04-06T14:17:25.571 INFO:tasks.workunit.client.1.smithi086.stdout:4/966: dread - d0/d15/d2b/fbe zero size 2024-04-06T14:17:25.585 INFO:tasks.workunit.client.1.smithi086.stdout:4/967: dread d0/d2c/d38/d54/d61/d7a/dc2/dd3/fea [0,4194304] 0 2024-04-06T14:17:25.586 INFO:tasks.workunit.client.1.smithi086.stdout:4/968: stat d0/d2c/d30/d24/d93/f122 0 2024-04-06T14:17:25.586 INFO:tasks.workunit.client.1.smithi086.stdout:4/969: dread - d0/d2c/d30/faf zero size 2024-04-06T14:17:25.587 INFO:tasks.workunit.client.1.smithi086.stdout:4/970: readlink d0/d2c/d30/l76 0 2024-04-06T14:17:25.588 INFO:tasks.workunit.client.1.smithi086.stdout:4/971: chown d0/d15/d2b/l10e 12288 1 2024-04-06T14:17:25.589 INFO:tasks.workunit.client.1.smithi086.stdout:4/972: rmdir d0/d2c/d30/d24/d5d/dfe 39 2024-04-06T14:17:25.590 INFO:tasks.workunit.client.1.smithi086.stdout:4/973: write d0/d15/d20/d34/fcf [327954,58116] 0 2024-04-06T14:17:25.594 INFO:tasks.workunit.client.1.smithi086.stdout:4/974: unlink d0/d15/d20/d34/le0 0 2024-04-06T14:17:25.594 INFO:tasks.workunit.client.1.smithi086.stdout:4/975: readlink d0/d2c/d38/d54/d61/l6b 0 2024-04-06T14:17:25.595 INFO:tasks.workunit.client.1.smithi086.stdout:4/976: chown d0/d2c/d38/d54/d61/c78 118170766 1 2024-04-06T14:17:25.597 INFO:tasks.workunit.client.1.smithi086.stdout:4/977: mkdir d0/d2c/d30/d24/da3/d132 0 2024-04-06T14:17:25.672 INFO:tasks.workunit.client.1.smithi086.stdout:5/955: sync 2024-04-06T14:17:25.675 INFO:tasks.workunit.client.1.smithi086.stdout:5/956: creat d0/d16/d18/d21/d23/d28/d94/d52/f13c x:0 0 0 2024-04-06T14:17:25.676 INFO:tasks.workunit.client.1.smithi086.stdout:5/957: chown d0/d16/d18/d21/d23/d28/d94/d52/d7c/fc0 251407 1 2024-04-06T14:17:25.678 INFO:tasks.workunit.client.1.smithi086.stdout:5/958: read d0/d16/d71/f76 [809670,17176] 0 2024-04-06T14:17:25.679 INFO:tasks.workunit.client.1.smithi086.stdout:5/959: symlink d0/d43/l13d 0 2024-04-06T14:17:25.685 INFO:tasks.workunit.client.1.smithi086.stdout:5/960: dwrite d0/d16/d18/d21/d23/d28/d94/f7a [0,4194304] 0 2024-04-06T14:17:25.689 INFO:tasks.workunit.client.1.smithi086.stdout:5/961: mknod d0/d16/d18/d35/d9e/c13e 0 2024-04-06T14:17:25.695 INFO:tasks.workunit.client.1.smithi086.stdout:5/962: rename d0/d16/d18/l27 to d0/d43/da6/d11e/d12a/l13f 0 2024-04-06T14:17:25.701 INFO:tasks.workunit.client.1.smithi086.stdout:5/963: dwrite d0/d16/d18/d35/fcf [0,4194304] 0 2024-04-06T14:17:25.704 INFO:tasks.workunit.client.1.smithi086.stdout:5/964: rmdir d0/d16/d71 39 2024-04-06T14:17:25.704 INFO:tasks.workunit.client.1.smithi086.stdout:5/965: chown d0/c42 486300819 1 2024-04-06T14:17:25.710 INFO:tasks.workunit.client.1.smithi086.stdout:5/966: dread d0/d16/d18/d21/d23/d28/d94/f7a [0,4194304] 0 2024-04-06T14:17:25.713 INFO:tasks.workunit.client.1.smithi086.stdout:5/967: creat d0/d16/d71/d96/ddc/f140 x:0 0 0 2024-04-06T14:17:25.720 INFO:tasks.workunit.client.1.smithi086.stdout:5/968: dwrite d0/d16/d18/d21/dff/f10b [0,4194304] 0 2024-04-06T14:17:25.734 INFO:tasks.workunit.client.1.smithi086.stdout:4/978: sync 2024-04-06T14:17:25.737 INFO:tasks.workunit.client.1.smithi086.stdout:4/979: creat d0/d2c/d30/d24/d50/f133 x:0 0 0 2024-04-06T14:17:25.742 INFO:tasks.workunit.client.1.smithi086.stdout:5/969: dread d0/d16/d18/d21/d23/d28/df7/fca [0,4194304] 0 2024-04-06T14:17:25.744 INFO:tasks.workunit.client.1.smithi086.stdout:5/970: unlink d0/d16/d18/d21/d23/d28/d94/d80/d85/l8c 0 2024-04-06T14:17:25.746 INFO:tasks.workunit.client.1.smithi086.stdout:5/971: symlink d0/d5e/l141 0 2024-04-06T14:17:25.747 INFO:tasks.workunit.client.1.smithi086.stdout:5/972: rmdir d0/d5e 39 2024-04-06T14:17:25.749 INFO:tasks.workunit.client.1.smithi086.stdout:5/973: truncate d0/d16/d18/d21/d23/d28/d94/d52/d7c/f8f 1370564 0 2024-04-06T14:17:25.752 INFO:tasks.workunit.client.1.smithi086.stdout:5/974: rename d0/d16/d18/d21/dff/d11b/f4b to d0/d16/d18/d2a/d50/f142 0 2024-04-06T14:17:25.755 INFO:tasks.workunit.client.1.smithi086.stdout:5/975: truncate d0/d16/d18/d21/d23/d28/faa 302467 0 2024-04-06T14:17:25.756 INFO:tasks.workunit.client.1.smithi086.stdout:5/976: fsync d0/d16/d18/d35/d9e/fb3 0 2024-04-06T14:17:25.760 INFO:tasks.workunit.client.1.smithi086.stdout:5/977: rename d0/d16/d18/d35/d9e/dd7/ffd to d0/d16/d71/d96/d11f/f143 0 2024-04-06T14:17:25.761 INFO:tasks.workunit.client.1.smithi086.stdout:5/978: fsync d0/d16/d18/d35/d9e/fb3 0 2024-04-06T14:17:25.774 INFO:tasks.workunit.client.1.smithi086.stdout:5/979: dwrite d0/d16/d71/d96/f11c [0,4194304] 0 2024-04-06T14:17:25.775 INFO:tasks.workunit.client.1.smithi086.stdout:5/980: chown d0/d16/d65/d10a/l120 465 1 2024-04-06T14:17:25.777 INFO:tasks.workunit.client.1.smithi086.stdout:5/981: chown d0/d16/d18/d35/f87 107758748 1 2024-04-06T14:17:25.784 INFO:tasks.workunit.client.1.smithi086.stdout:5/982: dwrite d0/d16/d18/d21/d23/d28/df7/fca [0,4194304] 0 2024-04-06T14:17:25.786 INFO:tasks.workunit.client.1.smithi086.stdout:5/983: write d0/d16/d71/ff2 [935320,84472] 0 2024-04-06T14:17:25.791 INFO:tasks.workunit.client.1.smithi086.stdout:5/984: rename d0/d16/d18/d35/d9e/de7/f79 to d0/d16/d18/d21/d23/d28/d94/d80/d85/f144 0 2024-04-06T14:17:25.792 INFO:tasks.workunit.client.1.smithi086.stdout:5/985: chown d0/d16/d18/d21/d23/d28/df4 7979 1 2024-04-06T14:17:25.798 INFO:tasks.workunit.client.1.smithi086.stdout:5/986: rename d0/d16/d18/d21/dff/d11b/f3c to d0/d16/d18/d21/d23/d28/d94/d52/d7c/f145 0 2024-04-06T14:17:25.798 INFO:tasks.workunit.client.1.smithi086.stdout:5/987: write d0/d16/d18/d2a/d50/fc6 [3509156,51140] 0 2024-04-06T14:17:25.802 INFO:tasks.workunit.client.1.smithi086.stdout:5/988: creat d0/d16/d18/d21/d23/d28/d94/d8a/f146 x:0 0 0 2024-04-06T14:17:25.808 INFO:tasks.workunit.client.1.smithi086.stdout:5/989: dwrite d0/d43/f11d [0,4194304] 0 2024-04-06T14:17:25.809 INFO:tasks.workunit.client.1.smithi086.stdout:5/990: dread - d0/d16/d71/d96/d11f/f11a zero size 2024-04-06T14:17:25.822 INFO:tasks.workunit.client.1.smithi086.stdout:5/991: dwrite d0/d16/d18/f7f [0,4194304] 0 2024-04-06T14:17:25.828 INFO:tasks.workunit.client.1.smithi086.stdout:5/992: link d0/d16/d18/f9d d0/d43/da6/d11e/d12a/f147 0 2024-04-06T14:17:25.831 INFO:tasks.workunit.client.1.smithi086.stdout:5/993: getdents d0/d16/d18/d35 0 2024-04-06T14:17:25.838 INFO:tasks.workunit.client.1.smithi086.stdout:5/994: dwrite d0/d16/d71/d96/d11f/f143 [0,4194304] 0 2024-04-06T14:17:25.841 INFO:tasks.workunit.client.1.smithi086.stdout:5/995: mkdir d0/d16/d18/d21/d23/d28/d94/d80/d148 0 2024-04-06T14:17:25.844 INFO:tasks.workunit.client.1.smithi086.stdout:5/996: mknod d0/d16/d18/d21/d23/d28/c149 0 2024-04-06T14:17:25.852 INFO:tasks.workunit.client.1.smithi086.stdout:5/997: write d0/f3f [387771,44227] 0 2024-04-06T14:17:25.857 INFO:tasks.workunit.client.1.smithi086.stdout:5/998: getdents d0/d16/d18/d35/d9e/de7 0 2024-04-06T14:17:25.859 INFO:tasks.workunit.client.1.smithi086.stdout:5/999: unlink d0/d43/da6/cfe 0 2024-04-06T14:17:25.877 INFO:tasks.workunit.client.1.smithi086.stdout:4/980: sync 2024-04-06T14:17:26.055 INFO:tasks.workunit.client.1.smithi086.stdout:4/981: sync 2024-04-06T14:17:26.055 INFO:tasks.workunit.client.1.smithi086.stdout:4/982: chown d0/d15/d20/d74/ld6 143760313 1 2024-04-06T14:17:26.059 INFO:tasks.workunit.client.1.smithi086.stdout:4/983: rename d0/d2c/d30/ldb to d0/d15/l134 0 2024-04-06T14:17:26.060 INFO:tasks.workunit.client.1.smithi086.stdout:4/984: write d0/d2c/d38/faa [3665983,62790] 0 2024-04-06T14:17:26.062 INFO:tasks.workunit.client.1.smithi086.stdout:4/985: write d0/d2c/d38/d54/d61/d7a/dc2/dd3/fe1 [463826,32112] 0 2024-04-06T14:17:26.063 INFO:tasks.workunit.client.1.smithi086.stdout:4/986: dread - d0/d15/d20/d34/f120 zero size 2024-04-06T14:17:26.081 INFO:tasks.workunit.client.1.smithi086.stdout:4/987: link d0/d2c/d30/d24/da3/d88/fcc d0/d2c/d30/d24/d50/d79/d89/f135 0 2024-04-06T14:17:26.082 INFO:tasks.workunit.client.1.smithi086.stdout:4/988: chown d0/d15/d2b/f82 1859 1 2024-04-06T14:17:26.088 INFO:tasks.workunit.client.1.smithi086.stdout:4/989: dwrite d0/db0/f10b [0,4194304] 0 2024-04-06T14:17:26.089 INFO:tasks.workunit.client.1.smithi086.stdout:4/990: chown d0/d15/d20/c73 1978735203 1 2024-04-06T14:17:26.090 INFO:tasks.workunit.client.1.smithi086.stdout:4/991: readlink d0/d15/d2b/l10e 0 2024-04-06T14:17:26.180 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:25 smithi032 ceph-mon[39730]: pgmap v30: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 95 MiB/s rd, 214 MiB/s wr, 491 op/s 2024-04-06T14:17:26.180 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:25 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:26.180 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:25 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:26.285 INFO:tasks.workunit.client.1.smithi086.stdout:4/992: sync 2024-04-06T14:17:26.298 INFO:tasks.workunit.client.1.smithi086.stdout:4/993: dread d0/d2c/d30/d24/f85 [0,4194304] 0 2024-04-06T14:17:26.298 INFO:tasks.workunit.client.1.smithi086.stdout:4/994: fsync d0/d2c/d38/f5b 0 2024-04-06T14:17:26.300 INFO:tasks.workunit.client.1.smithi086.stdout:4/995: mkdir d0/d2c/d30/d24/d50/dd0/d136 0 2024-04-06T14:17:26.305 INFO:tasks.workunit.client.1.smithi086.stdout:4/996: dwrite d0/d2c/d38/d54/f8c [0,4194304] 0 2024-04-06T14:17:26.320 INFO:tasks.workunit.client.1.smithi086.stdout:4/997: dread d0/d2c/d38/d54/f80 [0,4194304] 0 2024-04-06T14:17:26.322 INFO:tasks.workunit.client.1.smithi086.stdout:4/998: mknod d0/d2c/d30/d24/d50/dd0/d136/c137 0 2024-04-06T14:17:26.324 INFO:tasks.workunit.client.1.smithi086.stdout:4/999: unlink d0/d2c/lfd 0 2024-04-06T14:17:26.328 INFO:tasks.workunit.client.1.smithi086.stderr:+ rm -rf -- ./tmp.uBXlKvppeW 2024-04-06T14:17:26.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:25 smithi086 ceph-mon[47925]: pgmap v30: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 95 MiB/s rd, 214 MiB/s wr, 491 op/s 2024-04-06T14:17:26.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:25 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:26.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:25 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:27.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:26 smithi032 ceph-mon[39730]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-04-06T14:17:27.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:26 smithi032 ceph-mon[39730]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-04-06T14:17:27.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:26 smithi086 ceph-mon[47925]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-04-06T14:17:27.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:26 smithi086 ceph-mon[47925]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-04-06T14:17:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:27 smithi032 ceph-mon[39730]: pgmap v31: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 95 MiB/s rd, 214 MiB/s wr, 491 op/s 2024-04-06T14:17:28.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:27 smithi086 ceph-mon[47925]: pgmap v31: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 95 MiB/s rd, 214 MiB/s wr, 491 op/s 2024-04-06T14:17:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:28 smithi032 ceph-mon[39730]: pgmap v32: 65 pgs: 65 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 121 MiB/s rd, 282 MiB/s wr, 664 op/s 2024-04-06T14:17:29.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:28 smithi086 ceph-mon[47925]: pgmap v32: 65 pgs: 65 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 121 MiB/s rd, 282 MiB/s wr, 664 op/s 2024-04-06T14:17:32.067 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:31 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:32.067 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:31 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:32.067 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:31 smithi032 ceph-mon[39730]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-06T14:17:32.067 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:31 smithi032 ceph-mon[39730]: pgmap v33: 65 pgs: 65 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 79 MiB/s rd, 185 MiB/s wr, 429 op/s 2024-04-06T14:17:32.067 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:31 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:17:32.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:31 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:32.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:31 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:32.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:31 smithi086 ceph-mon[47925]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-06T14:17:32.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:31 smithi086 ceph-mon[47925]: pgmap v33: 65 pgs: 65 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 79 MiB/s rd, 185 MiB/s wr, 429 op/s 2024-04-06T14:17:32.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:31 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:17:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:32 smithi032 ceph-mon[39730]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-06T14:17:33.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:32 smithi086 ceph-mon[47925]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-06T14:17:33.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:33 smithi032 ceph-mon[39730]: pgmap v34: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 79 MiB/s rd, 186 MiB/s wr, 538 op/s 2024-04-06T14:17:34.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:33 smithi086 ceph-mon[47925]: pgmap v34: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 79 MiB/s rd, 186 MiB/s wr, 538 op/s 2024-04-06T14:17:36.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:35 smithi086 ceph-mon[47925]: pgmap v35: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 26 MiB/s rd, 69 MiB/s wr, 282 op/s 2024-04-06T14:17:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:35 smithi032 ceph-mon[39730]: pgmap v35: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 26 MiB/s rd, 69 MiB/s wr, 282 op/s 2024-04-06T14:17:37.718 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-06T14:17:37.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-06T14:17:38.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:37 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:38.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:37 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:38.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:37 smithi086 ceph-mon[47925]: pgmap v36: 65 pgs: 65 active+clean; 2.2 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 26 MiB/s rd, 70 MiB/s wr, 371 op/s 2024-04-06T14:17:38.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:37 smithi086 ceph-mon[47925]: Reconfiguring node-exporter.smithi086 (dependencies changed)... 2024-04-06T14:17:38.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:37 smithi086 ceph-mon[47925]: Reconfiguring daemon node-exporter.smithi086 on smithi086 2024-04-06T14:17:38.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:37 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:38.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:37 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:38.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:37 smithi032 ceph-mon[39730]: pgmap v36: 65 pgs: 65 active+clean; 2.2 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 26 MiB/s rd, 70 MiB/s wr, 371 op/s 2024-04-06T14:17:38.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:37 smithi032 ceph-mon[39730]: Reconfiguring node-exporter.smithi086 (dependencies changed)... 2024-04-06T14:17:38.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:37 smithi032 ceph-mon[39730]: Reconfiguring daemon node-exporter.smithi086 on smithi086 2024-04-06T14:17:39.833 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-06T14:17:39.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-06T14:17:40.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:39 smithi032 ceph-mon[39730]: pgmap v37: 65 pgs: 65 active+clean; 1.4 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 26 MiB/s rd, 70 MiB/s wr, 398 op/s 2024-04-06T14:17:40.114 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:39 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:40.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:39 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:40.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:39 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:17:40.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:39 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:17:40.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:39 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:17:40.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:39 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:17:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:39 smithi086 ceph-mon[47925]: pgmap v37: 65 pgs: 65 active+clean; 1.4 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 26 MiB/s rd, 70 MiB/s wr, 398 op/s 2024-04-06T14:17:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:39 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:39 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:39 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:17:40.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:39 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:17:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:39 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:17:40.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:39 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:17:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:40 smithi086 ceph-mon[47925]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:17:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:40 smithi086 ceph-mon[47925]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:17:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:40 smithi086 ceph-mon[47925]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:17:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:40 smithi086 ceph-mon[47925]: Upgrade: Need to upgrade myself (mgr.smithi086.fyyhxm) 2024-04-06T14:17:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:40 smithi086 ceph-mon[47925]: Upgrade: Need to upgrade myself (mgr.smithi086.fyyhxm) 2024-04-06T14:17:41.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:40 smithi032 ceph-mon[39730]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:17:41.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:40 smithi032 ceph-mon[39730]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:17:41.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:40 smithi032 ceph-mon[39730]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:17:41.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:40 smithi032 ceph-mon[39730]: Upgrade: Need to upgrade myself (mgr.smithi086.fyyhxm) 2024-04-06T14:17:41.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:40 smithi032 ceph-mon[39730]: Upgrade: Need to upgrade myself (mgr.smithi086.fyyhxm) 2024-04-06T14:17:42.124 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:41 smithi032 ceph-mon[39730]: pgmap v38: 65 pgs: 65 active+clean; 1.4 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 30 KiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-04-06T14:17:42.124 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:41 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:42.124 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:41 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.peohxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:17:42.124 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:41 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.peohxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:17:42.124 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:41 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:17:42.125 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:41 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:17:42.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:41 smithi086 ceph-mon[47925]: pgmap v38: 65 pgs: 65 active+clean; 1.4 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 30 KiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-04-06T14:17:42.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:41 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:41 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.peohxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:17:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:41 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.peohxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:17:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:41 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:17:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:41 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:17:43.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:42 smithi086 ceph-mon[47925]: Upgrade: Updating mgr.smithi032.peohxu 2024-04-06T14:17:43.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:42 smithi086 ceph-mon[47925]: Deploying daemon mgr.smithi032.peohxu on smithi032 2024-04-06T14:17:43.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:42 smithi032 ceph-mon[39730]: Upgrade: Updating mgr.smithi032.peohxu 2024-04-06T14:17:43.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:42 smithi032 ceph-mon[39730]: Deploying daemon mgr.smithi032.peohxu on smithi032 2024-04-06T14:17:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:43 smithi086 ceph-mon[47925]: pgmap v39: 65 pgs: 65 active+clean; 300 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 2.3 MiB/s wr, 303 op/s 2024-04-06T14:17:44.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:43 smithi032 ceph-mon[39730]: pgmap v39: 65 pgs: 65 active+clean; 300 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 2.3 MiB/s wr, 303 op/s 2024-04-06T14:17:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:45 smithi032 ceph-mon[39730]: pgmap v40: 65 pgs: 65 active+clean; 300 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 32 KiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-04-06T14:17:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:45 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:45 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:45 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:17:46.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:45 smithi086 ceph-mon[47925]: pgmap v40: 65 pgs: 65 active+clean; 300 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 32 KiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-04-06T14:17:46.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:45 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:46.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:45 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:46.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:45 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:17:46.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:46 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:17:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:46 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:46 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:47.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:46 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:17:47.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:46 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:47.199 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:46 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:47 smithi086 ceph-mon[47925]: pgmap v41: 65 pgs: 65 active+clean; 316 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-04-06T14:17:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:47 smithi032 ceph-mon[39730]: pgmap v41: 65 pgs: 65 active+clean; 316 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-04-06T14:17:49.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:49 smithi032 ceph-mon[39730]: pgmap v42: 65 pgs: 65 active+clean; 326 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 2.9 MiB/s wr, 132 op/s 2024-04-06T14:17:50.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:49 smithi086 ceph-mon[47925]: pgmap v42: 65 pgs: 65 active+clean; 326 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 2.9 MiB/s wr, 132 op/s 2024-04-06T14:17:51.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.146+0000 7f9520b0f700 1 -- 172.21.15.32:0/4209447456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c06ed80 msgr2=0x7f951c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:51.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.146+0000 7f9520b0f700 1 --2- 172.21.15.32:0/4209447456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c06ed80 0x7f951c06f1f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9510004880 tx=0x7f9510009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.147+0000 7f9520b0f700 1 -- 172.21.15.32:0/4209447456 shutdown_connections 2024-04-06T14:17:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.147+0000 7f9520b0f700 1 --2- 172.21.15.32:0/4209447456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c06ed80 0x7f951c06f1f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.147+0000 7f9520b0f700 1 --2- 172.21.15.32:0/4209447456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c105210 0x7f951c105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.147+0000 7f9520b0f700 1 -- 172.21.15.32:0/4209447456 >> 172.21.15.32:0/4209447456 conn(0x7f951c069de0 msgr2=0x7f951c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.147+0000 7f9520b0f700 1 -- 172.21.15.32:0/4209447456 shutdown_connections 2024-04-06T14:17:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.147+0000 7f9520b0f700 1 -- 172.21.15.32:0/4209447456 wait complete. 2024-04-06T14:17:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.147+0000 7f9520b0f700 1 Processor -- start 2024-04-06T14:17:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.147+0000 7f9520b0f700 1 -- start start 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9520b0f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c06ed80 0x7f951c116040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9520b0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c105210 0x7f951c116580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9520b0f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c116b80 con 0x7f951c105210 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9520b0f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c116cf0 con 0x7f951c06ed80 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9519d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c105210 0x7f951c116580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9519d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c105210 0x7f951c116580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48082/0 (socket says 172.21.15.32:48082) 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9519d9b700 1 -- 172.21.15.32:0/1975391024 learned_addr learned my addr 172.21.15.32:0/1975391024 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f951a59c700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c06ed80 0x7f951c116040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9519d9b700 1 -- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c06ed80 msgr2=0x7f951c116040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9519d9b700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c06ed80 0x7f951c116040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.148+0000 7f9519d9b700 1 -- 172.21.15.32:0/1975391024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c005040 con 0x7f951c105210 2024-04-06T14:17:51.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.149+0000 7f9519d9b700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c105210 0x7f951c116580 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f95100093b0 tx=0x7f9510009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:51.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.149+0000 7f950affd700 1 -- 172.21.15.32:0/1975391024 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951000e070 con 0x7f951c105210 2024-04-06T14:17:51.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.149+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9510005040 con 0x7f951c105210 2024-04-06T14:17:51.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.149+0000 7f950affd700 1 -- 172.21.15.32:0/1975391024 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9510013d00 con 0x7f951c105210 2024-04-06T14:17:51.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.149+0000 7f950affd700 1 -- 172.21.15.32:0/1975391024 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95100123c0 con 0x7f951c105210 2024-04-06T14:17:51.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.149+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951c106750 con 0x7f951c105210 2024-04-06T14:17:51.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.150+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951c04c710 con 0x7f951c105210 2024-04-06T14:17:51.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.152+0000 7f950affd700 1 -- 172.21.15.32:0/1975391024 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f951001b450 con 0x7f951c105210 2024-04-06T14:17:51.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.152+0000 7f950affd700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f950406afa0 0x7f950406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:51.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.152+0000 7f950affd700 1 -- 172.21.15.32:0/1975391024 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f951008a8e0 con 0x7f951c105210 2024-04-06T14:17:51.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.153+0000 7f951a59c700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f950406afa0 0x7f950406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.160+0000 7f951a59c700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f950406afa0 0x7f950406d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f951c06ed10 tx=0x7f950c00c040 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:51.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.163+0000 7f950affd700 1 -- 172.21.15.32:0/1975391024 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f9510058800 con 0x7f951c105210 2024-04-06T14:17:51.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.485+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f951c109620 con 0x7f950406afa0 2024-04-06T14:17:51.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.486+0000 7f950affd700 1 -- 172.21.15.32:0/1975391024 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f951c109620 con 0x7f950406afa0 2024-04-06T14:17:51.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f950406afa0 msgr2=0x7f950406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:51.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f950406afa0 0x7f950406d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f951c06ed10 tx=0x7f950c00c040 comp rx=0 tx=0).stop 2024-04-06T14:17:51.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c105210 msgr2=0x7f951c116580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:51.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c105210 0x7f951c116580 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f95100093b0 tx=0x7f9510009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 shutdown_connections 2024-04-06T14:17:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f950406afa0 0x7f950406d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c06ed80 0x7f951c116040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 --2- 172.21.15.32:0/1975391024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c105210 0x7f951c116580 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 >> 172.21.15.32:0/1975391024 conn(0x7f951c069de0 msgr2=0x7f951c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 shutdown_connections 2024-04-06T14:17:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.489+0000 7f9520b0f700 1 -- 172.21.15.32:0/1975391024 wait complete. 2024-04-06T14:17:51.505 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:17:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.605+0000 7fdd08907700 1 -- 172.21.15.32:0/2751981231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fce40 msgr2=0x7fdd040fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.605+0000 7fdd08907700 1 --2- 172.21.15.32:0/2751981231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fce40 0x7fdd040fd250 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4004940 tx=0x7fdcf4009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.605+0000 7fdd08907700 1 -- 172.21.15.32:0/2751981231 shutdown_connections 2024-04-06T14:17:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.605+0000 7fdd08907700 1 --2- 172.21.15.32:0/2751981231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040fe040 0x7fdd040fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.605+0000 7fdd08907700 1 --2- 172.21.15.32:0/2751981231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fce40 0x7fdd040fd250 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.605+0000 7fdd08907700 1 -- 172.21.15.32:0/2751981231 >> 172.21.15.32:0/2751981231 conn(0x7fdd040f8570 msgr2=0x7fdd040fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.605+0000 7fdd08907700 1 -- 172.21.15.32:0/2751981231 shutdown_connections 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.605+0000 7fdd08907700 1 -- 172.21.15.32:0/2751981231 wait complete. 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd08907700 1 Processor -- start 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd08907700 1 -- start start 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd08907700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040fce40 0x7fdd04071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd08907700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fe040 0x7fdd04070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd08907700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd040708c0 con 0x7fdd040fe040 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd08907700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04070a30 con 0x7fdd040fce40 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd01d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fe040 0x7fdd04070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd01d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fe040 0x7fdd04070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48106/0 (socket says 172.21.15.32:48106) 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd01d9b700 1 -- 172.21.15.32:0/1843619323 learned_addr learned my addr 172.21.15.32:0/1843619323 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd0259c700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040fce40 0x7fdd04071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd01d9b700 1 -- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040fce40 msgr2=0x7fdd04071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd01d9b700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040fce40 0x7fdd04071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd01d9b700 1 -- 172.21.15.32:0/1843619323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf4005040 con 0x7fdd040fe040 2024-04-06T14:17:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.606+0000 7fdd01d9b700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fe040 0x7fdd04070310 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8002a50 tx=0x7fdcf80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.607+0000 7fdcf2ffd700 1 -- 172.21.15.32:0/1843619323 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcf8007cc0 con 0x7fdd040fe040 2024-04-06T14:17:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.607+0000 7fdcf2ffd700 1 -- 172.21.15.32:0/1843619323 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcf800f040 con 0x7fdd040fe040 2024-04-06T14:17:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.607+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd04070d10 con 0x7fdd040fe040 2024-04-06T14:17:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.607+0000 7fdcf2ffd700 1 -- 172.21.15.32:0/1843619323 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcf800d9c0 con 0x7fdd040fe040 2024-04-06T14:17:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.607+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd041159a0 con 0x7fdd040fe040 2024-04-06T14:17:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.608+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0405f5f0 con 0x7fdd040fe040 2024-04-06T14:17:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.608+0000 7fdcf2ffd700 1 -- 172.21.15.32:0/1843619323 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7fdcf800db20 con 0x7fdd040fe040 2024-04-06T14:17:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.609+0000 7fdcf2ffd700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fdcec06aec0 0x7fdcec06d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.609+0000 7fdcf2ffd700 1 -- 172.21.15.32:0/1843619323 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fdcf8089560 con 0x7fdd040fe040 2024-04-06T14:17:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.609+0000 7fdd0259c700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fdcec06aec0 0x7fdcec06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.609+0000 7fdd0259c700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fdcec06aec0 0x7fdcec06d370 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdd04071bd0 tx=0x7fdcf4009e30 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:51.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.614+0000 7fdcf2ffd700 1 -- 172.21.15.32:0/1843619323 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fdcf8053970 con 0x7fdd040fe040 2024-04-06T14:17:51.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.892+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd04102470 con 0x7fdcec06aec0 2024-04-06T14:17:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.893+0000 7fdcf2ffd700 1 -- 172.21.15.32:0/1843619323 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fdd04102470 con 0x7fdcec06aec0 2024-04-06T14:17:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fdcec06aec0 msgr2=0x7fdcec06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fdcec06aec0 0x7fdcec06d370 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdd04071bd0 tx=0x7fdcf4009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fe040 msgr2=0x7fdd04070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fe040 0x7fdd04070310 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8002a50 tx=0x7fdcf80046b0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 shutdown_connections 2024-04-06T14:17:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fdcec06aec0 0x7fdcec06d370 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040fce40 0x7fdd04071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 --2- 172.21.15.32:0/1843619323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd040fe040 0x7fdd04070310 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 >> 172.21.15.32:0/1843619323 conn(0x7fdd040f8570 msgr2=0x7fdd040f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 shutdown_connections 2024-04-06T14:17:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:51.896+0000 7fdd08907700 1 -- 172.21.15.32:0/1843619323 wait complete. 2024-04-06T14:17:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.025+0000 7f03b8f76700 1 -- 172.21.15.32:0/3665354547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 msgr2=0x7f03b40fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.025+0000 7f03b8f76700 1 --2- 172.21.15.32:0/3665354547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 0x7f03b40fd400 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f03a4004940 tx=0x7f03a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.025+0000 7f03b8f76700 1 -- 172.21.15.32:0/3665354547 shutdown_connections 2024-04-06T14:17:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.025+0000 7f03b8f76700 1 --2- 172.21.15.32:0/3665354547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 0x7f03b40fd400 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.025+0000 7f03b8f76700 1 --2- 172.21.15.32:0/3665354547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03b40fbce0 0x7f03b40fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.025+0000 7f03b8f76700 1 -- 172.21.15.32:0/3665354547 >> 172.21.15.32:0/3665354547 conn(0x7f03b40f7440 msgr2=0x7f03b40f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.025+0000 7f03b8f76700 1 -- 172.21.15.32:0/3665354547 shutdown_connections 2024-04-06T14:17:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.025+0000 7f03b8f76700 1 -- 172.21.15.32:0/3665354547 wait complete. 2024-04-06T14:17:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.026+0000 7f03b8f76700 1 Processor -- start 2024-04-06T14:17:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.026+0000 7f03b8f76700 1 -- start start 2024-04-06T14:17:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.026+0000 7f03b8f76700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03b40fbce0 0x7f03b4113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.026+0000 7f03b8f76700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 0x7f03b4114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.026+0000 7f03b8f76700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b41151d0 con 0x7f03b40fcf90 2024-04-06T14:17:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.026+0000 7f03b8f76700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4115340 con 0x7f03b40fbce0 2024-04-06T14:17:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.026+0000 7f03b1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 0x7f03b4114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.026+0000 7f03b259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03b40fbce0 0x7f03b4113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.026+0000 7f03b1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 0x7f03b4114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48130/0 (socket says 172.21.15.32:48130) 2024-04-06T14:17:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03b1d9b700 1 -- 172.21.15.32:0/1908923642 learned_addr learned my addr 172.21.15.32:0/1908923642 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03b259c700 1 -- 172.21.15.32:0/1908923642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 msgr2=0x7f03b4114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03b259c700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 0x7f03b4114460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03b259c700 1 -- 172.21.15.32:0/1908923642 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a4005040 con 0x7f03b40fbce0 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03b1d9b700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 0x7f03b4114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03b259c700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03b40fbce0 0x7f03b4113f20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f03a8009fd0 tx=0x7f03a800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03a2ffd700 1 -- 172.21.15.32:0/1908923642 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a8004e90 con 0x7f03b40fbce0 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b4115560 con 0x7f03b40fbce0 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03a2ffd700 1 -- 172.21.15.32:0/1908923642 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03a800f040 con 0x7f03b40fbce0 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03a2ffd700 1 -- 172.21.15.32:0/1908923642 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a800c9b0 con 0x7f03b40fbce0 2024-04-06T14:17:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.027+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b4115b20 con 0x7f03b40fbce0 2024-04-06T14:17:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.029+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03940052e0 con 0x7f03b40fbce0 2024-04-06T14:17:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.029+0000 7f03a2ffd700 1 -- 172.21.15.32:0/1908923642 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f03a800cb10 con 0x7f03b40fbce0 2024-04-06T14:17:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.029+0000 7f03a2ffd700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f039c06aec0 0x7f039c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.030+0000 7f03a2ffd700 1 -- 172.21.15.32:0/1908923642 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f03a8089360 con 0x7f03b40fbce0 2024-04-06T14:17:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.030+0000 7f03b1d9b700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f039c06aec0 0x7f039c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.030+0000 7f03b1d9b700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f039c06aec0 0x7f039c06d370 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f03a4013680 tx=0x7f03a4009e30 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.035+0000 7f03a2ffd700 1 -- 172.21.15.32:0/1908923642 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f03a804f560 con 0x7f03b40fbce0 2024-04-06T14:17:52.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:51 smithi086 ceph-mon[47925]: pgmap v43: 65 pgs: 65 active+clean; 326 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 2.9 MiB/s wr, 105 op/s 2024-04-06T14:17:52.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:51 smithi032 ceph-mon[39730]: pgmap v43: 65 pgs: 65 active+clean; 326 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 2.9 MiB/s wr, 105 op/s 2024-04-06T14:17:52.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.359+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0394000bf0 con 0x7f039c06aec0 2024-04-06T14:17:52.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.379+0000 7f03a2ffd700 1 -- 172.21.15.32:0/1908923642 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0394000bf0 con 0x7f039c06aec0 2024-04-06T14:17:52.382 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:17:52.382 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 starting - - - - 2024-04-06T14:17:52.382 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 73s ago 9m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:17:52.382 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (8m) 75s ago 8m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:17:52.382 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 starting - - - - 2024-04-06T14:17:52.382 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (3m) 73s ago 3m 11.4M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:17:52.382 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (3m) 73s ago 3m 108M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:17:52.382 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (3m) 75s ago 3m 108M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:17:52.382 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (3m) 75s ago 3m 14.2M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 starting - - - - 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283 running (94s) 75s ago 8m 486M - 18.2.2-1086-g052adcad 87261d5ece5a f68af56edf8d 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 73s ago 11m 47.3M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (8m) 75s ago 8m 38.6M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 starting - - - - 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 starting - - - - 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 73s ago 7m 76.0M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 73s ago 7m 83.0M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 73s ago 6m 76.6M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (6m) 75s ago 6m 90.2M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (5m) 75s ago 5m 83.9M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (5m) 75s ago 5m 75.4M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:17:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 starting - - - - 2024-04-06T14:17:52.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.383+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f039c06aec0 msgr2=0x7f039c06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:52.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.383+0000 7f03b8f76700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f039c06aec0 0x7f039c06d370 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f03a4013680 tx=0x7f03a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:52.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.383+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03b40fbce0 msgr2=0x7f03b4113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:52.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.383+0000 7f03b8f76700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03b40fbce0 0x7f03b4113f20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f03a8009fd0 tx=0x7f03a800b040 comp rx=0 tx=0).stop 2024-04-06T14:17:52.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.385+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 shutdown_connections 2024-04-06T14:17:52.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.385+0000 7f03b8f76700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f039c06aec0 0x7f039c06d370 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.385+0000 7f03b8f76700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03b40fbce0 0x7f03b4113f20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.385+0000 7f03b8f76700 1 --2- 172.21.15.32:0/1908923642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b40fcf90 0x7f03b4114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.385+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 >> 172.21.15.32:0/1908923642 conn(0x7f03b40f7440 msgr2=0x7f03b40f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:52.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.385+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 shutdown_connections 2024-04-06T14:17:52.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.385+0000 7f03b8f76700 1 -- 172.21.15.32:0/1908923642 wait complete. 2024-04-06T14:17:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.519+0000 7f54d15c6700 1 -- 172.21.15.32:0/3158364803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 msgr2=0x7f54cc102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.519+0000 7f54d15c6700 1 --2- 172.21.15.32:0/3158364803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 0x7f54cc102320 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f54c0004940 tx=0x7f54c0009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.519+0000 7f54d15c6700 1 -- 172.21.15.32:0/3158364803 shutdown_connections 2024-04-06T14:17:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.519+0000 7f54d15c6700 1 --2- 172.21.15.32:0/3158364803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 0x7f54cc102320 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.519+0000 7f54d15c6700 1 --2- 172.21.15.32:0/3158364803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc066f50 0x7f54cc0ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.519+0000 7f54d15c6700 1 -- 172.21.15.32:0/3158364803 >> 172.21.15.32:0/3158364803 conn(0x7f54cc0f7450 msgr2=0x7f54cc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.519+0000 7f54d15c6700 1 -- 172.21.15.32:0/3158364803 shutdown_connections 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.519+0000 7f54d15c6700 1 -- 172.21.15.32:0/3158364803 wait complete. 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54d15c6700 1 Processor -- start 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54d15c6700 1 -- start start 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54d15c6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc066f50 0x7f54cc10d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54d15c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 0x7f54cc10dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54d15c6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc10e340 con 0x7f54cc0fff00 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54d15c6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc10e4b0 con 0x7f54cc066f50 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54ca7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 0x7f54cc10dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54ca7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 0x7f54cc10dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48150/0 (socket says 172.21.15.32:48150) 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54ca7fc700 1 -- 172.21.15.32:0/100149436 learned_addr learned my addr 172.21.15.32:0/100149436 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54caffd700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc066f50 0x7f54cc10d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54ca7fc700 1 -- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc066f50 msgr2=0x7f54cc10d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54ca7fc700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc066f50 0x7f54cc10d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.520+0000 7f54ca7fc700 1 -- 172.21.15.32:0/100149436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc005040 con 0x7f54cc0fff00 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.521+0000 7f54ca7fc700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 0x7f54cc10dd40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f54c0007d40 tx=0x7f54c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.521+0000 7f54b37fe700 1 -- 172.21.15.32:0/100149436 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54c000e030 con 0x7f54cc0fff00 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.521+0000 7f54b37fe700 1 -- 172.21.15.32:0/100149436 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54c001c400 con 0x7f54cc0fff00 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.521+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c0005040 con 0x7f54cc0fff00 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.521+0000 7f54b37fe700 1 -- 172.21.15.32:0/100149436 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54c000b650 con 0x7f54cc0fff00 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.521+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54cc1132e0 con 0x7f54cc0fff00 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.522+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54cc04c710 con 0x7f54cc0fff00 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.522+0000 7f54b37fe700 1 -- 172.21.15.32:0/100149436 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f54c001c570 con 0x7f54cc0fff00 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.523+0000 7f54b37fe700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f54b406aed0 0x7f54b406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.523+0000 7f54b37fe700 1 -- 172.21.15.32:0/100149436 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f54c008ad20 con 0x7f54cc0fff00 2024-04-06T14:17:52.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.523+0000 7f54caffd700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f54b406aed0 0x7f54b406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:52.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.523+0000 7f54caffd700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f54b406aed0 0x7f54b406d380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54cc066c90 tx=0x7f54bc00b040 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:52.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.528+0000 7f54b37fe700 1 -- 172.21.15.32:0/100149436 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f54c0058d20 con 0x7f54cc0fff00 2024-04-06T14:17:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.909+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f54cc105790 con 0x7f54cc0fff00 2024-04-06T14:17:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.909+0000 7f54b37fe700 1 -- 172.21.15.32:0/100149436 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7f54c0004490 con 0x7f54cc0fff00 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 1 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:17:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:17:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:17:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:17:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:17:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:17:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:17:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 13, 2024-04-06T14:17:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 1 2024-04-06T14:17:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:17:52.911 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:17:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.912+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f54b406aed0 msgr2=0x7f54b406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.912+0000 7f54d15c6700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f54b406aed0 0x7f54b406d380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54cc066c90 tx=0x7f54bc00b040 comp rx=0 tx=0).stop 2024-04-06T14:17:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.912+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 msgr2=0x7f54cc10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.912+0000 7f54d15c6700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 0x7f54cc10dd40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f54c0007d40 tx=0x7f54c0009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.913+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 shutdown_connections 2024-04-06T14:17:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.913+0000 7f54d15c6700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7f54b406aed0 0x7f54b406d380 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.913+0000 7f54d15c6700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc066f50 0x7f54cc10d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.913+0000 7f54d15c6700 1 --2- 172.21.15.32:0/100149436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0fff00 0x7f54cc10dd40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:52.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.913+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 >> 172.21.15.32:0/100149436 conn(0x7f54cc0f7450 msgr2=0x7f54cc0f7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:52.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.913+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 shutdown_connections 2024-04-06T14:17:52.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:52.913+0000 7f54d15c6700 1 -- 172.21.15.32:0/100149436 wait complete. 2024-04-06T14:17:52.937 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:52 smithi032 ceph-mon[39730]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.054+0000 7fe423091700 1 -- 172.21.15.32:0/2946574473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0fcf90 msgr2=0x7fe41c0fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.054+0000 7fe423091700 1 --2- 172.21.15.32:0/2946574473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0fcf90 0x7fe41c0fd400 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe408004940 tx=0x7fe408009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.054+0000 7fe423091700 1 -- 172.21.15.32:0/2946574473 shutdown_connections 2024-04-06T14:17:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.054+0000 7fe423091700 1 --2- 172.21.15.32:0/2946574473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0fcf90 0x7fe41c0fd400 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.054+0000 7fe423091700 1 --2- 172.21.15.32:0/2946574473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0fbce0 0x7fe41c0fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.054+0000 7fe423091700 1 -- 172.21.15.32:0/2946574473 >> 172.21.15.32:0/2946574473 conn(0x7fe41c0f7480 msgr2=0x7fe41c0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.054+0000 7fe423091700 1 -- 172.21.15.32:0/2946574473 shutdown_connections 2024-04-06T14:17:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.054+0000 7fe423091700 1 -- 172.21.15.32:0/2946574473 wait complete. 2024-04-06T14:17:53.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.055+0000 7fe423091700 1 Processor -- start 2024-04-06T14:17:53.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.055+0000 7fe423091700 1 -- start start 2024-04-06T14:17:53.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.055+0000 7fe423091700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0fbce0 0x7fe41c10d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.055+0000 7fe423091700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0fcf90 0x7fe41c10dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.055+0000 7fe423091700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c10e3e0 con 0x7fe41c0fcf90 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.055+0000 7fe423091700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c10ec30 con 0x7fe41c0fbce0 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe41bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0fcf90 0x7fe41c10dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe41bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0fcf90 0x7fe41c10dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48154/0 (socket says 172.21.15.32:48154) 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe41bfff700 1 -- 172.21.15.32:0/1666467228 learned_addr learned my addr 172.21.15.32:0/1666467228 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe420e2d700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0fbce0 0x7fe41c10d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe420e2d700 1 -- 172.21.15.32:0/1666467228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0fcf90 msgr2=0x7fe41c10dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe420e2d700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0fcf90 0x7fe41c10dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe420e2d700 1 -- 172.21.15.32:0/1666467228 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe408005040 con 0x7fe41c0fbce0 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe420e2d700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0fbce0 0x7fe41c10d8a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe410009fd0 tx=0x7fe41000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe4197fa700 1 -- 172.21.15.32:0/1666467228 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe410015070 con 0x7fe41c0fbce0 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c10eeb0 con 0x7fe41c0fbce0 2024-04-06T14:17:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.056+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c10f3f0 con 0x7fe41c0fbce0 2024-04-06T14:17:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.057+0000 7fe4197fa700 1 -- 172.21.15.32:0/1666467228 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe410010e50 con 0x7fe41c0fbce0 2024-04-06T14:17:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.057+0000 7fe4197fa700 1 -- 172.21.15.32:0/1666467228 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe410009970 con 0x7fe41c0fbce0 2024-04-06T14:17:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.058+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41c05f5f0 con 0x7fe41c0fbce0 2024-04-06T14:17:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.058+0000 7fe4197fa700 1 -- 172.21.15.32:0/1666467228 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) v1 ==== 88764+0+0 (secure 0 0 0) 0x7fe41001b470 con 0x7fe41c0fbce0 2024-04-06T14:17:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.058+0000 7fe4197fa700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fe40c06aec0 0x7fe40c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:17:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.059+0000 7fe4197fa700 1 -- 172.21.15.32:0/1666467228 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fe41008e9f0 con 0x7fe41c0fbce0 2024-04-06T14:17:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.059+0000 7fe41bfff700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fe40c06aec0 0x7fe40c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:17:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.059+0000 7fe41bfff700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fe40c06aec0 0x7fe40c06d370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4080043d0 tx=0x7fe408009e30 comp rx=0 tx=0).ready entity=mgr.24409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:17:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.065+0000 7fe4197fa700 1 -- 172.21.15.32:0/1666467228 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fe41005c8d0 con 0x7fe41c0fbce0 2024-04-06T14:17:53.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:52 smithi086 ceph-mon[47925]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.404+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 --> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe41c10f7d0 con 0x7fe40c06aec0 2024-04-06T14:17:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.405+0000 7fe4197fa700 1 -- 172.21.15.32:0/1666467228 <== mgr.24409 v2:172.21.15.86:6828/1279185277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fe41c10f7d0 con 0x7fe40c06aec0 2024-04-06T14:17:53.405 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:17:53.405 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:17:53.405 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:17:53.406 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T14:17:53.406 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-06T14:17:53.406 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "", 2024-04-06T14:17:53.406 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mgr daemons", 2024-04-06T14:17:53.406 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:17:53.406 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.407+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fe40c06aec0 msgr2=0x7fe40c06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.407+0000 7fe423091700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fe40c06aec0 0x7fe40c06d370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4080043d0 tx=0x7fe408009e30 comp rx=0 tx=0).stop 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.407+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0fbce0 msgr2=0x7fe41c10d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.407+0000 7fe423091700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0fbce0 0x7fe41c10d8a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe410009fd0 tx=0x7fe41000b040 comp rx=0 tx=0).stop 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.407+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 shutdown_connections 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.407+0000 7fe423091700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:6828/1279185277,v1:172.21.15.86:6829/1279185277] conn(0x7fe40c06aec0 0x7fe40c06d370 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.407+0000 7fe423091700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0fbce0 0x7fe41c10d8a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.407+0000 7fe423091700 1 --2- 172.21.15.32:0/1666467228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0fcf90 0x7fe41c10dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.407+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 >> 172.21.15.32:0/1666467228 conn(0x7fe41c0f7480 msgr2=0x7fe41c0f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.408+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 shutdown_connections 2024-04-06T14:17:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:17:53.408+0000 7fe423091700 1 -- 172.21.15.32:0/1666467228 wait complete. 2024-04-06T14:17:53.885 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:53 smithi032 ceph-mon[39730]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:53.885 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:53 smithi032 ceph-mon[39730]: from='client.24481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:53.885 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:53 smithi032 ceph-mon[39730]: pgmap v44: 65 pgs: 65 active+clean; 334 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 41 KiB/s rd, 3.8 MiB/s wr, 143 op/s 2024-04-06T14:17:53.885 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:53 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/100149436' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:17:54.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:53 smithi086 ceph-mon[47925]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:54.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:53 smithi086 ceph-mon[47925]: from='client.24481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:54.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:53 smithi086 ceph-mon[47925]: pgmap v44: 65 pgs: 65 active+clean; 334 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 41 KiB/s rd, 3.8 MiB/s wr, 143 op/s 2024-04-06T14:17:54.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:53 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/100149436' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:17:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:54 smithi032 ceph-mon[39730]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:54 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:54 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:54 smithi032 ceph-mon[39730]: Standby manager daemon smithi032.peohxu restarted 2024-04-06T14:17:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:54 smithi032 ceph-mon[39730]: Standby manager daemon smithi032.peohxu started 2024-04-06T14:17:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:54 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.peohxu/crt"}]: dispatch 2024-04-06T14:17:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:54 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:17:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:54 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.peohxu/key"}]: dispatch 2024-04-06T14:17:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:54 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:17:55.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:54 smithi086 ceph-mon[47925]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:17:55.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:54 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:55.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:54 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:55.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:54 smithi086 ceph-mon[47925]: Standby manager daemon smithi032.peohxu restarted 2024-04-06T14:17:55.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:54 smithi086 ceph-mon[47925]: Standby manager daemon smithi032.peohxu started 2024-04-06T14:17:55.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:54 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.peohxu/crt"}]: dispatch 2024-04-06T14:17:55.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:54 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:17:55.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:54 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.peohxu/key"}]: dispatch 2024-04-06T14:17:55.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:54 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:17:56.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:55 smithi032 ceph-mon[39730]: pgmap v45: 65 pgs: 65 active+clean; 334 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 26 KiB/s rd, 3.2 MiB/s wr, 64 op/s 2024-04-06T14:17:56.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:55 smithi032 ceph-mon[39730]: mgrmap e26: smithi086.fyyhxm(active, since 84s), standbys: smithi032.peohxu 2024-04-06T14:17:56.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:55 smithi086 ceph-mon[47925]: pgmap v45: 65 pgs: 65 active+clean; 334 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 26 KiB/s rd, 3.2 MiB/s wr, 64 op/s 2024-04-06T14:17:56.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:55 smithi086 ceph-mon[47925]: mgrmap e26: smithi086.fyyhxm(active, since 84s), standbys: smithi032.peohxu 2024-04-06T14:17:57.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:56 smithi032 ceph-mon[39730]: pgmap v46: 65 pgs: 65 active+clean; 342 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 3.9 MiB/s wr, 87 op/s 2024-04-06T14:17:57.324 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:56 smithi086 ceph-mon[47925]: pgmap v46: 65 pgs: 65 active+clean; 342 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 3.9 MiB/s wr, 87 op/s 2024-04-06T14:17:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:58 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:58 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:58 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:58.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:58 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:59.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:58 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:59.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:58 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:59.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:58 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:59.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:58 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:17:59.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:17:59 smithi032 ceph-mon[39730]: pgmap v47: 65 pgs: 65 active+clean; 346 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 2.8 MiB/s wr, 86 op/s 2024-04-06T14:18:00.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:17:59 smithi086 ceph-mon[47925]: pgmap v47: 65 pgs: 65 active+clean; 346 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 2.8 MiB/s wr, 86 op/s 2024-04-06T14:18:01.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:01 smithi032 ceph-mon[39730]: pgmap v48: 65 pgs: 65 active+clean; 346 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 1.9 MiB/s wr, 74 op/s 2024-04-06T14:18:01.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:01 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:02.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:01 smithi086 ceph-mon[47925]: pgmap v48: 65 pgs: 65 active+clean; 346 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 1.9 MiB/s wr, 74 op/s 2024-04-06T14:18:02.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:01 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:03.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:02 smithi086 ceph-mon[47925]: mgrmap e27: smithi086.fyyhxm(active, since 91s), standbys: smithi032.peohxu 2024-04-06T14:18:03.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:02 smithi032 ceph-mon[39730]: mgrmap e27: smithi086.fyyhxm(active, since 91s), standbys: smithi032.peohxu 2024-04-06T14:18:04.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:03 smithi086 ceph-mon[47925]: pgmap v49: 65 pgs: 65 active+clean; 352 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 59 KiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-04-06T14:18:04.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:03 smithi032 ceph-mon[39730]: pgmap v49: 65 pgs: 65 active+clean; 352 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 59 KiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-04-06T14:18:06.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: pgmap v50: 65 pgs: 65 active+clean; 352 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 41 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-04-06T14:18:06.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:18:06.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:18:06.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:18:06.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:18:06.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr fail", "who": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr fail", "who": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: osdmap e49: 6 total, 6 up, 6 in 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd='[{"prefix": "mgr fail", "who": "smithi086.fyyhxm"}]': finished 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: mgrmap e28: smithi032.peohxu(active, starting, since 0.00559991s) 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:18:06.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:18:06.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:18:06.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:18:06.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:18:06.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:18:06.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:18:06.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:18:06.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:18:06.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:18:06.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:05 smithi086 ceph-mon[47925]: Manager daemon smithi032.peohxu is now available 2024-04-06T14:18:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: pgmap v50: 65 pgs: 65 active+clean; 352 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 41 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-04-06T14:18:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24409 172.21.15.86:0/1365203396' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr fail", "who": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "mgr fail", "who": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: osdmap e49: 6 total, 6 up, 6 in 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24409 ' entity='mgr.smithi086.fyyhxm' cmd='[{"prefix": "mgr fail", "who": "smithi086.fyyhxm"}]': finished 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: mgrmap e28: smithi032.peohxu(active, starting, since 0.00559991s) 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:18:06.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:18:06.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:05 smithi032 ceph-mon[39730]: Manager daemon smithi032.peohxu is now available 2024-04-06T14:18:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:18:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/trash_purge_schedule"}]: dispatch 2024-04-06T14:18:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:06 smithi086 ceph-mon[47925]: [06/Apr/2024:14:18:06] ENGINE Bus STARTING 2024-04-06T14:18:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:06 smithi086 ceph-mon[47925]: [06/Apr/2024:14:18:06] ENGINE Serving on https://172.21.15.32:7150 2024-04-06T14:18:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:06 smithi086 ceph-mon[47925]: mgrmap e29: smithi032.peohxu(active, since 1.00905s) 2024-04-06T14:18:07.138 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:07.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:07.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:18:07.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/trash_purge_schedule"}]: dispatch 2024-04-06T14:18:07.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:06 smithi032 ceph-mon[39730]: [06/Apr/2024:14:18:06] ENGINE Bus STARTING 2024-04-06T14:18:07.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:06 smithi032 ceph-mon[39730]: [06/Apr/2024:14:18:06] ENGINE Serving on https://172.21.15.32:7150 2024-04-06T14:18:07.139 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:06 smithi032 ceph-mon[39730]: mgrmap e29: smithi032.peohxu(active, since 1.00905s) 2024-04-06T14:18:07.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:07 smithi086 ceph-mon[47925]: [06/Apr/2024:14:18:06] ENGINE Serving on http://172.21.15.32:8765 2024-04-06T14:18:07.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:07 smithi086 ceph-mon[47925]: [06/Apr/2024:14:18:06] ENGINE Bus STARTED 2024-04-06T14:18:07.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:07 smithi086 ceph-mon[47925]: pgmap v3: 65 pgs: 65 active+clean; 343 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:18:08.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:07 smithi032 ceph-mon[39730]: [06/Apr/2024:14:18:06] ENGINE Serving on http://172.21.15.32:8765 2024-04-06T14:18:08.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:07 smithi032 ceph-mon[39730]: [06/Apr/2024:14:18:06] ENGINE Bus STARTED 2024-04-06T14:18:08.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:07 smithi032 ceph-mon[39730]: pgmap v3: 65 pgs: 65 active+clean; 343 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:18:09.215 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:08 smithi086 ceph-mon[47925]: pgmap v4: 65 pgs: 65 active+clean; 343 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:18:09.229 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:08 smithi032 ceph-mon[39730]: pgmap v4: 65 pgs: 65 active+clean; 343 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:18:10.176 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:09 smithi086 ceph-mon[47925]: mgrmap e30: smithi032.peohxu(active, since 3s) 2024-04-06T14:18:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:09 smithi032 ceph-mon[39730]: mgrmap e30: smithi032.peohxu(active, since 3s) 2024-04-06T14:18:11.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:11 smithi086 ceph-mon[47925]: pgmap v5: 65 pgs: 65 active+clean; 343 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:18:11.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:11 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:11.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:11 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:11.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:11 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:11.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:11 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:11 smithi032 ceph-mon[39730]: pgmap v5: 65 pgs: 65 active+clean; 343 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:18:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:11 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:11 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:11 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:11 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:13.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:13 smithi086 ceph-mon[47925]: pgmap v6: 65 pgs: 65 active+clean; 349 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 103 KiB/s rd, 2.2 MiB/s wr, 102 op/s 2024-04-06T14:18:13.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:13.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:13.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:13.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:13.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:13 smithi032 ceph-mon[39730]: pgmap v6: 65 pgs: 65 active+clean; 349 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 103 KiB/s rd, 2.2 MiB/s wr, 102 op/s 2024-04-06T14:18:13.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:13.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:13.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:13.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:15.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:15 smithi086 ceph-mon[47925]: pgmap v7: 65 pgs: 65 active+clean; 349 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 73 KiB/s rd, 2.0 MiB/s wr, 96 op/s 2024-04-06T14:18:15.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:15 smithi086 ceph-mon[47925]: Standby manager daemon smithi086.fyyhxm started 2024-04-06T14:18:15.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:15 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1421513326' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:18:15.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:15 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1421513326' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:18:15.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:15 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1421513326' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:18:15.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:15 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1421513326' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:18:15.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:15 smithi032 ceph-mon[39730]: pgmap v7: 65 pgs: 65 active+clean; 349 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 73 KiB/s rd, 2.0 MiB/s wr, 96 op/s 2024-04-06T14:18:15.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:15 smithi032 ceph-mon[39730]: Standby manager daemon smithi086.fyyhxm started 2024-04-06T14:18:15.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:15 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/1421513326' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:18:15.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:15 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/1421513326' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:18:15.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:15 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/1421513326' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:18:15.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:15 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/1421513326' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:18:16.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:16 smithi086 ceph-mon[47925]: mgrmap e31: smithi032.peohxu(active, since 9s), standbys: smithi086.fyyhxm 2024-04-06T14:18:16.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:16 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi086.fyyhxm", "id": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:18:16.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:16 smithi032 ceph-mon[39730]: mgrmap e31: smithi032.peohxu(active, since 9s), standbys: smithi086.fyyhxm 2024-04-06T14:18:16.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:16 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi086.fyyhxm", "id": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:18:17.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:17 smithi086 ceph-mon[47925]: pgmap v8: 65 pgs: 65 active+clean; 349 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 57 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-04-06T14:18:17.819 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:17 smithi032 ceph-mon[39730]: pgmap v8: 65 pgs: 65 active+clean; 349 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 57 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-04-06T14:18:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:19 smithi086 ceph-mon[47925]: pgmap v9: 65 pgs: 65 active+clean; 356 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 66 KiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-04-06T14:18:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:19 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:19 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:19 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:18:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:19 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:18:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:19 smithi086 ceph-mon[47925]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:18:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:19 smithi086 ceph-mon[47925]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:18:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:19 smithi086 ceph-mon[47925]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:18:19.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:19 smithi032 ceph-mon[39730]: pgmap v9: 65 pgs: 65 active+clean; 356 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 66 KiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-04-06T14:18:19.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:19 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:19.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:19 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:19.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:19 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:18:19.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:19 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:18:19.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:19 smithi032 ceph-mon[39730]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:18:19.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:19 smithi032 ceph-mon[39730]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:18:19.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:19 smithi032 ceph-mon[39730]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:18:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:18:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:18:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:18:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:18:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:18:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:20.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:20.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:20.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:20.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:20.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:18:20.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:18:20.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:18:20.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:18:20.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:18:20.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:20.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:20.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:20.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:20.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:21 smithi032 ceph-mon[39730]: pgmap v10: 65 pgs: 65 active+clean; 356 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 60 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-04-06T14:18:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:21 smithi032 ceph-mon[39730]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-06T14:18:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:21 smithi032 ceph-mon[39730]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-06T14:18:21.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:21 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:21.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:21 smithi086 ceph-mon[47925]: pgmap v10: 65 pgs: 65 active+clean; 356 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 60 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-04-06T14:18:21.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:21 smithi086 ceph-mon[47925]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-06T14:18:21.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:21 smithi086 ceph-mon[47925]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-06T14:18:21.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:21 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:23.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.527+0000 7f4b60950700 1 -- 172.21.15.32:0/2471614155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0fff00 msgr2=0x7f4b5c102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:23.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.527+0000 7f4b60950700 1 --2- 172.21.15.32:0/2471614155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0fff00 0x7f4b5c102320 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4b50004940 tx=0x7f4b50009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:23.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.527+0000 7f4b60950700 1 -- 172.21.15.32:0/2471614155 shutdown_connections 2024-04-06T14:18:23.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.527+0000 7f4b60950700 1 --2- 172.21.15.32:0/2471614155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0fff00 0x7f4b5c102320 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:23.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.527+0000 7f4b60950700 1 --2- 172.21.15.32:0/2471614155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b5c0662e0 0x7f4b5c0ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:23.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.527+0000 7f4b60950700 1 -- 172.21.15.32:0/2471614155 >> 172.21.15.32:0/2471614155 conn(0x7f4b5c0f7430 msgr2=0x7f4b5c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:23.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.528+0000 7f4b60950700 1 -- 172.21.15.32:0/2471614155 shutdown_connections 2024-04-06T14:18:23.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.528+0000 7f4b60950700 1 -- 172.21.15.32:0/2471614155 wait complete. 2024-04-06T14:18:23.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.528+0000 7f4b60950700 1 Processor -- start 2024-04-06T14:18:23.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.528+0000 7f4b60950700 1 -- start start 2024-04-06T14:18:23.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b60950700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0662e0 0x7f4b5c10d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b60950700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b5c0fff00 0x7f4b5c10de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b60950700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c10e460 con 0x7f4b5c0662e0 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b60950700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c10ec60 con 0x7f4b5c0fff00 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b5a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0662e0 0x7f4b5c10d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b5a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0662e0 0x7f4b5c10d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45942/0 (socket says 172.21.15.32:45942) 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b5a59c700 1 -- 172.21.15.32:0/1280973915 learned_addr learned my addr 172.21.15.32:0/1280973915 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b59d9b700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b5c0fff00 0x7f4b5c10de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b5a59c700 1 -- 172.21.15.32:0/1280973915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b5c0fff00 msgr2=0x7f4b5c10de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b5a59c700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b5c0fff00 0x7f4b5c10de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b5a59c700 1 -- 172.21.15.32:0/1280973915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b50005040 con 0x7f4b5c0662e0 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b5a59c700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0662e0 0x7f4b5c10d920 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00eb40 tx=0x7f4b4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b4affd700 1 -- 172.21.15.32:0/1280973915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b4c007cc0 con 0x7f4b5c0662e0 2024-04-06T14:18:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b4affd700 1 -- 172.21.15.32:0/1280973915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b4c00f040 con 0x7f4b5c0662e0 2024-04-06T14:18:23.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c10eee0 con 0x7f4b5c0662e0 2024-04-06T14:18:23.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b4affd700 1 -- 172.21.15.32:0/1280973915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b4c00c8f0 con 0x7f4b5c0662e0 2024-04-06T14:18:23.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.529+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c10f420 con 0x7f4b5c0662e0 2024-04-06T14:18:23.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.530+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c05f5f0 con 0x7f4b5c0662e0 2024-04-06T14:18:23.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.531+0000 7f4b4affd700 1 -- 172.21.15.32:0/1280973915 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 31) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f4b4c0090d0 con 0x7f4b5c0662e0 2024-04-06T14:18:23.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.531+0000 7f4b4affd700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b4406c9c0 0x7f4b4406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:23.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.531+0000 7f4b4affd700 1 -- 172.21.15.32:0/1280973915 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f4b4c08b6b0 con 0x7f4b5c0662e0 2024-04-06T14:18:23.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.532+0000 7f4b59d9b700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b4406c9c0 0x7f4b4406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:23.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.532+0000 7f4b59d9b700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b4406c9c0 0x7f4b4406ee70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4b50004890 tx=0x7f4b50009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:23.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.537+0000 7f4b4affd700 1 -- 172.21.15.32:0/1280973915 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f4b4c059530 con 0x7f4b5c0662e0 2024-04-06T14:18:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:23 smithi032 ceph-mon[39730]: pgmap v11: 65 pgs: 65 active+clean; 356 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 76 KiB/s rd, 2.3 MiB/s wr, 161 op/s 2024-04-06T14:18:23.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.839+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b5c10f7f0 con 0x7f4b4406c9c0 2024-04-06T14:18:23.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.841+0000 7f4b4affd700 1 -- 172.21.15.32:0/1280973915 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f4b5c10f7f0 con 0x7f4b4406c9c0 2024-04-06T14:18:23.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.844+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b4406c9c0 msgr2=0x7f4b4406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:23.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.844+0000 7f4b60950700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b4406c9c0 0x7f4b4406ee70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4b50004890 tx=0x7f4b50009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:23.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.844+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0662e0 msgr2=0x7f4b5c10d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:23.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.844+0000 7f4b60950700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0662e0 0x7f4b5c10d920 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00eb40 tx=0x7f4b4c00b040 comp rx=0 tx=0).stop 2024-04-06T14:18:23.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.844+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 shutdown_connections 2024-04-06T14:18:23.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.844+0000 7f4b60950700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b4406c9c0 0x7f4b4406ee70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:23.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.844+0000 7f4b60950700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b5c0662e0 0x7f4b5c10d920 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00eb40 tx=0x7f4b4c00b040 comp rx=0 tx=0).stop 2024-04-06T14:18:23.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.844+0000 7f4b60950700 1 --2- 172.21.15.32:0/1280973915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b5c0fff00 0x7f4b5c10de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:23.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.844+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 >> 172.21.15.32:0/1280973915 conn(0x7f4b5c0f7430 msgr2=0x7f4b5c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:23.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.845+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 shutdown_connections 2024-04-06T14:18:23.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.845+0000 7f4b60950700 1 -- 172.21.15.32:0/1280973915 wait complete. 2024-04-06T14:18:23.858 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:18:23.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:23 smithi086 ceph-mon[47925]: pgmap v11: 65 pgs: 65 active+clean; 356 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 76 KiB/s rd, 2.3 MiB/s wr, 161 op/s 2024-04-06T14:18:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.961+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2685727161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 msgr2=0x7f5b400fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.961+0000 7f5b47e0f700 1 --2- 172.21.15.32:0/2685727161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 0x7f5b400fe4b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5b30004940 tx=0x7f5b30009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.961+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2685727161 shutdown_connections 2024-04-06T14:18:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.961+0000 7f5b47e0f700 1 --2- 172.21.15.32:0/2685727161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 0x7f5b400fe4b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.961+0000 7f5b47e0f700 1 --2- 172.21.15.32:0/2685727161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b400fce40 0x7f5b400fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.961+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2685727161 >> 172.21.15.32:0/2685727161 conn(0x7f5b400f8570 msgr2=0x7f5b400fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.961+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2685727161 shutdown_connections 2024-04-06T14:18:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.961+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2685727161 wait complete. 2024-04-06T14:18:23.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b47e0f700 1 Processor -- start 2024-04-06T14:18:23.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b47e0f700 1 -- start start 2024-04-06T14:18:23.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b47e0f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b400fce40 0x7f5b4010fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:23.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b47e0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 0x7f5b40110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:23.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b47e0f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b40110660 con 0x7f5b400fe040 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b47e0f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b40110eb0 con 0x7f5b400fce40 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b453aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 0x7f5b40110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b453aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 0x7f5b40110060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45958/0 (socket says 172.21.15.32:45958) 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b453aa700 1 -- 172.21.15.32:0/2656184662 learned_addr learned my addr 172.21.15.32:0/2656184662 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.962+0000 7f5b45bab700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b400fce40 0x7f5b4010fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b45bab700 1 -- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 msgr2=0x7f5b40110060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b45bab700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 0x7f5b40110060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b45bab700 1 -- 172.21.15.32:0/2656184662 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b30005040 con 0x7f5b400fce40 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b453aa700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 0x7f5b40110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b45bab700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b400fce40 0x7f5b4010fb20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5b400fdbb0 tx=0x7f5b3c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b367fc700 1 -- 172.21.15.32:0/2656184662 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b3c004e90 con 0x7f5b400fce40 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b40111130 con 0x7f5b400fce40 2024-04-06T14:18:23.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b367fc700 1 -- 172.21.15.32:0/2656184662 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b3c00f040 con 0x7f5b400fce40 2024-04-06T14:18:23.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b367fc700 1 -- 172.21.15.32:0/2656184662 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b3c00c890 con 0x7f5b400fce40 2024-04-06T14:18:23.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.963+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b40111670 con 0x7f5b400fce40 2024-04-06T14:18:23.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.964+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b401019d0 con 0x7f5b400fce40 2024-04-06T14:18:23.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.965+0000 7f5b367fc700 1 -- 172.21.15.32:0/2656184662 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 31) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f5b3c00cb00 con 0x7f5b400fce40 2024-04-06T14:18:23.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.965+0000 7f5b367fc700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5b2c06c9c0 0x7f5b2c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:23.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.965+0000 7f5b367fc700 1 -- 172.21.15.32:0/2656184662 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f5b3c08b750 con 0x7f5b400fce40 2024-04-06T14:18:23.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.965+0000 7f5b453aa700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5b2c06c9c0 0x7f5b2c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:23.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.966+0000 7f5b453aa700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5b2c06c9c0 0x7f5b2c06ee70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5b300135e0 tx=0x7f5b30009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:23.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:23.974+0000 7f5b367fc700 1 -- 172.21.15.32:0/2656184662 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f5b3c059550 con 0x7f5b400fce40 2024-04-06T14:18:24.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.213+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b40111890 con 0x7f5b2c06c9c0 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.214+0000 7f5b367fc700 1 -- 172.21.15.32:0/2656184662 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f5b40111890 con 0x7f5b2c06c9c0 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5b2c06c9c0 msgr2=0x7f5b2c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5b2c06c9c0 0x7f5b2c06ee70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5b300135e0 tx=0x7f5b30009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b400fce40 msgr2=0x7f5b4010fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b400fce40 0x7f5b4010fb20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5b400fdbb0 tx=0x7f5b3c00b040 comp rx=0 tx=0).stop 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 shutdown_connections 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5b2c06c9c0 0x7f5b2c06ee70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b400fce40 0x7f5b4010fb20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 --2- 172.21.15.32:0/2656184662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b400fe040 0x7f5b40110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 >> 172.21.15.32:0/2656184662 conn(0x7f5b400f8570 msgr2=0x7f5b400f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 shutdown_connections 2024-04-06T14:18:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.216+0000 7f5b47e0f700 1 -- 172.21.15.32:0/2656184662 wait complete. 2024-04-06T14:18:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.325+0000 7f0848160700 1 -- 172.21.15.32:0/2186415350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08400fff30 msgr2=0x7f0840102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.325+0000 7f0848160700 1 --2- 172.21.15.32:0/2186415350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08400fff30 0x7f0840102350 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0830004940 tx=0x7f0830009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 -- 172.21.15.32:0/2186415350 shutdown_connections 2024-04-06T14:18:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 --2- 172.21.15.32:0/2186415350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08400fff30 0x7f0840102350 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 --2- 172.21.15.32:0/2186415350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0840066ef0 0x7f08400ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 -- 172.21.15.32:0/2186415350 >> 172.21.15.32:0/2186415350 conn(0x7f08400f7440 msgr2=0x7f08400f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 -- 172.21.15.32:0/2186415350 shutdown_connections 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 -- 172.21.15.32:0/2186415350 wait complete. 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 Processor -- start 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 -- start start 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0840066ef0 0x7f084010d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08400fff30 0x7f084010dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084010e330 con 0x7f0840066ef0 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.326+0000 7f0848160700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084010e4a0 con 0x7f08400fff30 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f0845efc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0840066ef0 0x7f084010d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f0845efc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0840066ef0 0x7f084010d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45972/0 (socket says 172.21.15.32:45972) 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f0845efc700 1 -- 172.21.15.32:0/1454400050 learned_addr learned my addr 172.21.15.32:0/1454400050 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f08456fb700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08400fff30 0x7f084010dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f0845efc700 1 -- 172.21.15.32:0/1454400050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08400fff30 msgr2=0x7f084010dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f0845efc700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08400fff30 0x7f084010dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f0845efc700 1 -- 172.21.15.32:0/1454400050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0830005040 con 0x7f0840066ef0 2024-04-06T14:18:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f0845efc700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0840066ef0 0x7f084010d7f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f083c00eb40 tx=0x7f083c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f08367fc700 1 -- 172.21.15.32:0/1454400050 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f083c007cc0 con 0x7f0840066ef0 2024-04-06T14:18:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f08367fc700 1 -- 172.21.15.32:0/1454400050 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f083c00f040 con 0x7f0840066ef0 2024-04-06T14:18:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f08367fc700 1 -- 172.21.15.32:0/1454400050 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f083c00c8f0 con 0x7f0840066ef0 2024-04-06T14:18:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0840112ea0 con 0x7f0840066ef0 2024-04-06T14:18:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.327+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08401133d0 con 0x7f0840066ef0 2024-04-06T14:18:24.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.328+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084004c710 con 0x7f0840066ef0 2024-04-06T14:18:24.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.329+0000 7f08367fc700 1 -- 172.21.15.32:0/1454400050 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 31) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f083c0090d0 con 0x7f0840066ef0 2024-04-06T14:18:24.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.329+0000 7f08367fc700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f082c06c9d0 0x7f082c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:24.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.330+0000 7f08367fc700 1 -- 172.21.15.32:0/1454400050 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f083c08b690 con 0x7f0840066ef0 2024-04-06T14:18:24.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.330+0000 7f08456fb700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f082c06c9d0 0x7f082c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:24.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.330+0000 7f08456fb700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f082c06c9d0 0x7f082c06ee80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0830013680 tx=0x7f0830009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.335+0000 7f08367fc700 1 -- 172.21.15.32:0/1454400050 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f083c059510 con 0x7f0840066ef0 2024-04-06T14:18:24.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.562+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0840113610 con 0x7f082c06c9d0 2024-04-06T14:18:24.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.573+0000 7f08367fc700 1 -- 172.21.15.32:0/1454400050 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0840113610 con 0x7f082c06c9d0 2024-04-06T14:18:24.573 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:18:24.573 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (61s) 12s ago 10m 28.5M - 0.25.0 c8568f914cd2 73706679ea15 2024-04-06T14:18:24.573 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 12s ago 10m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:18:24.573 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (9m) 14s ago 9m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:18:24.573 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (54s) 12s ago 9m 75.5M - 9.4.7 954c08fa6188 88d7529c0ea5 2024-04-06T14:18:24.573 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (4m) 12s ago 4m 12.4M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:18:24.573 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (4m) 12s ago 4m 191M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (4m) 14s ago 4m 194M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (4m) 14s ago 4m 15.2M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (39s) 12s ago 11m 472M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283 running (2m) 14s ago 9m 61.7M - 18.2.2-1086-g052adcad 87261d5ece5a f68af56edf8d 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (11m) 12s ago 11m 53.7M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (9m) 14s ago 9m 39.6M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (87s) 12s ago 10m 16.3M - 1.5.0 0da6a335fe13 c3cb793eb173 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (45s) 14s ago 9m 14.4M - 1.5.0 0da6a335fe13 05e92e59b710 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 12s ago 8m 177M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 12s ago 7m 180M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 12s ago 7m 148M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (6m) 14s ago 6m 222M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (6m) 14s ago 6m 191M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:18:24.574 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (5m) 14s ago 5m 176M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:18:24.575 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (48s) 12s ago 9m 43.8M - 2.43.0 a07b618ecd1d 466cc29cf0f5 2024-04-06T14:18:24.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f082c06c9d0 msgr2=0x7f082c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:24.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f082c06c9d0 0x7f082c06ee80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0830013680 tx=0x7f0830009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:24.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0840066ef0 msgr2=0x7f084010d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:24.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0840066ef0 0x7f084010d7f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f083c00eb40 tx=0x7f083c00b040 comp rx=0 tx=0).stop 2024-04-06T14:18:24.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 shutdown_connections 2024-04-06T14:18:24.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f082c06c9d0 0x7f082c06ee80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0840066ef0 0x7f084010d7f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 --2- 172.21.15.32:0/1454400050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08400fff30 0x7f084010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 >> 172.21.15.32:0/1454400050 conn(0x7f08400f7440 msgr2=0x7f08400f7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:24.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 shutdown_connections 2024-04-06T14:18:24.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.575+0000 7f0848160700 1 -- 172.21.15.32:0/1454400050 wait complete. 2024-04-06T14:18:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 -- 172.21.15.32:0/2422020730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 msgr2=0x7f35bc0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 --2- 172.21.15.32:0/2422020730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 0x7f35bc0fd3f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f35b0004940 tx=0x7f35b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 -- 172.21.15.32:0/2422020730 shutdown_connections 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 --2- 172.21.15.32:0/2422020730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 0x7f35bc0fd3f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 --2- 172.21.15.32:0/2422020730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35bc0fbcd0 0x7f35bc0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 -- 172.21.15.32:0/2422020730 >> 172.21.15.32:0/2422020730 conn(0x7f35bc0f7430 msgr2=0x7f35bc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 -- 172.21.15.32:0/2422020730 shutdown_connections 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 -- 172.21.15.32:0/2422020730 wait complete. 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 Processor -- start 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 -- start start 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35bc0fbcd0 0x7f35bc110590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 0x7f35bc112ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc066370 con 0x7f35bc0fcf80 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.680+0000 7f35c3a50700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0664b0 con 0x7f35bc0fbcd0 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c0feb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 0x7f35bc112ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c0feb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 0x7f35bc112ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60086/0 (socket says 172.21.15.32:60086) 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c0feb700 1 -- 172.21.15.32:0/613314564 learned_addr learned my addr 172.21.15.32:0/613314564 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c17ec700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35bc0fbcd0 0x7f35bc110590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c0feb700 1 -- 172.21.15.32:0/613314564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35bc0fbcd0 msgr2=0x7f35bc110590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c0feb700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35bc0fbcd0 0x7f35bc110590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c0feb700 1 -- 172.21.15.32:0/613314564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b0005040 con 0x7f35bc0fcf80 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c17ec700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35bc0fbcd0 0x7f35bc110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c0feb700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 0x7f35bc112ae0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f35b00137b0 tx=0x7f35b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35adffb700 1 -- 172.21.15.32:0/613314564 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b000e070 con 0x7f35bc0fcf80 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35adffb700 1 -- 172.21.15.32:0/613314564 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35b001c400 con 0x7f35bc0fcf80 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc113080 con 0x7f35bc0fcf80 2024-04-06T14:18:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.681+0000 7f35adffb700 1 -- 172.21.15.32:0/613314564 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b000b650 con 0x7f35bc0fcf80 2024-04-06T14:18:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.682+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc1135b0 con 0x7f35bc0fcf80 2024-04-06T14:18:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.682+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc05f5f0 con 0x7f35bc0fcf80 2024-04-06T14:18:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.683+0000 7f35adffb700 1 -- 172.21.15.32:0/613314564 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 31) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f35b001c9e0 con 0x7f35bc0fcf80 2024-04-06T14:18:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.683+0000 7f35adffb700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f35a806c9c0 0x7f35a806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.683+0000 7f35adffb700 1 -- 172.21.15.32:0/613314564 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f35b008d230 con 0x7f35bc0fcf80 2024-04-06T14:18:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.684+0000 7f35c17ec700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f35a806c9c0 0x7f35a806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.684+0000 7f35c17ec700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f35a806c9c0 0x7f35a806ee70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f35bc065ff0 tx=0x7f35b800b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:24.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:24.688+0000 7f35adffb700 1 -- 172.21.15.32:0/613314564 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f35b0057610 con 0x7f35bc0fcf80 2024-04-06T14:18:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.048+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f35bc04bed0 con 0x7f35bc0fcf80 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.048+0000 7f35adffb700 1 -- 172.21.15.32:0/613314564 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f35b0004490 con 0x7f35bc0fcf80 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:18:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:18:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:18:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:18:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 12, 2024-04-06T14:18:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:18:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:18:25.050 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:18:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f35a806c9c0 msgr2=0x7f35a806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f35a806c9c0 0x7f35a806ee70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f35bc065ff0 tx=0x7f35b800b040 comp rx=0 tx=0).stop 2024-04-06T14:18:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 msgr2=0x7f35bc112ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 0x7f35bc112ae0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f35b00137b0 tx=0x7f35b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 shutdown_connections 2024-04-06T14:18:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f35a806c9c0 0x7f35a806ee70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35bc0fbcd0 0x7f35bc110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 --2- 172.21.15.32:0/613314564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35bc0fcf80 0x7f35bc112ae0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 >> 172.21.15.32:0/613314564 conn(0x7f35bc0f7430 msgr2=0x7f35bc0f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 shutdown_connections 2024-04-06T14:18:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.051+0000 7f35c3a50700 1 -- 172.21.15.32:0/613314564 wait complete. 2024-04-06T14:18:25.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 -- 172.21.15.32:0/3802942452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 msgr2=0x7f63dc1046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 --2- 172.21.15.32:0/3802942452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 0x7f63dc1046b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f63cc004970 tx=0x7f63cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 -- 172.21.15.32:0/3802942452 shutdown_connections 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 --2- 172.21.15.32:0/3802942452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 0x7f63dc1046b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 --2- 172.21.15.32:0/3802942452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0fd410 0x7f63dc0fd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 -- 172.21.15.32:0/3802942452 >> 172.21.15.32:0/3802942452 conn(0x7f63dc0f6ba0 msgr2=0x7f63dc0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 -- 172.21.15.32:0/3802942452 shutdown_connections 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 -- 172.21.15.32:0/3802942452 wait complete. 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 Processor -- start 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.166+0000 7f63e36d4700 1 -- start start 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e36d4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0fd410 0x7f63dc10fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e36d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 0x7f63dc110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e36d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc110620 con 0x7f63dc0fdd60 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e36d4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc110e20 con 0x7f63dc0fd410 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e0c6f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 0x7f63dc110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e0c6f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 0x7f63dc110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60104/0 (socket says 172.21.15.32:60104) 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e0c6f700 1 -- 172.21.15.32:0/3624495215 learned_addr learned my addr 172.21.15.32:0/3624495215 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e1470700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0fd410 0x7f63dc10fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e0c6f700 1 -- 172.21.15.32:0/3624495215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0fd410 msgr2=0x7f63dc10fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e0c6f700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0fd410 0x7f63dc10fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e0c6f700 1 -- 172.21.15.32:0/3624495215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc005040 con 0x7f63dc0fdd60 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e0c6f700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 0x7f63dc110020 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00b820 tx=0x7f63cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63d1ffb700 1 -- 172.21.15.32:0/3624495215 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63cc00e040 con 0x7f63dc0fdd60 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63dc1110a0 con 0x7f63dc0fdd60 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63d1ffb700 1 -- 172.21.15.32:0/3624495215 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63cc013420 con 0x7f63dc0fdd60 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.167+0000 7f63d1ffb700 1 -- 172.21.15.32:0/3624495215 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63cc01c630 con 0x7f63dc0fdd60 2024-04-06T14:18:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.168+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63dc1115e0 con 0x7f63dc0fdd60 2024-04-06T14:18:25.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.168+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63dc05f5f0 con 0x7f63dc0fdd60 2024-04-06T14:18:25.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.169+0000 7f63d1ffb700 1 -- 172.21.15.32:0/3624495215 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 31) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f63cc0139e0 con 0x7f63dc0fdd60 2024-04-06T14:18:25.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.169+0000 7f63d1ffb700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f63c806ca10 0x7f63c806eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:25.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.170+0000 7f63d1ffb700 1 -- 172.21.15.32:0/3624495215 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f63cc08d030 con 0x7f63dc0fdd60 2024-04-06T14:18:25.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.170+0000 7f63e1470700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f63c806ca10 0x7f63c806eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:25.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.170+0000 7f63e1470700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f63c806ca10 0x7f63c806eec0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009b80 tx=0x7f63d800b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:25.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.174+0000 7f63d1ffb700 1 -- 172.21.15.32:0/3624495215 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f63cc05ad60 con 0x7f63dc0fdd60 2024-04-06T14:18:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.415+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f63dc111910 con 0x7f63c806ca10 2024-04-06T14:18:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.416+0000 7f63d1ffb700 1 -- 172.21.15.32:0/3624495215 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f63dc111910 con 0x7f63c806ca10 2024-04-06T14:18:25.416 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:18:25.416 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:18:25.416 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:18:25.416 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T14:18:25.416 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:18:25.416 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-06T14:18:25.417 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:18:25.417 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/2 daemons upgraded", 2024-04-06T14:18:25.417 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-04-06T14:18:25.417 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:18:25.417 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:18:25.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f63c806ca10 msgr2=0x7f63c806eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f63c806ca10 0x7f63c806eec0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009b80 tx=0x7f63d800b040 comp rx=0 tx=0).stop 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 msgr2=0x7f63dc110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 0x7f63dc110020 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00b820 tx=0x7f63cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 shutdown_connections 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f63c806ca10 0x7f63c806eec0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0fd410 0x7f63dc10fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 --2- 172.21.15.32:0/3624495215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc0fdd60 0x7f63dc110020 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 >> 172.21.15.32:0/3624495215 conn(0x7f63dc0f6ba0 msgr2=0x7f63dc0f6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 shutdown_connections 2024-04-06T14:18:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:25.418+0000 7f63e36d4700 1 -- 172.21.15.32:0/3624495215 wait complete. 2024-04-06T14:18:25.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:25 smithi032 ceph-mon[39730]: pgmap v12: 65 pgs: 65 active+clean; 358 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-04-06T14:18:25.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:25 smithi032 ceph-mon[39730]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:25.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:25 smithi032 ceph-mon[39730]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:25.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:25 smithi032 ceph-mon[39730]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:25.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:25 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/613314564' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:25.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:25 smithi086 ceph-mon[47925]: pgmap v12: 65 pgs: 65 active+clean; 358 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-04-06T14:18:25.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:25 smithi086 ceph-mon[47925]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:25.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:25 smithi086 ceph-mon[47925]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:25.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:25 smithi086 ceph-mon[47925]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:25.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:25 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/613314564' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:26 smithi032 ceph-mon[39730]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:26 smithi086 ceph-mon[47925]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:27.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:27 smithi086 ceph-mon[47925]: pgmap v13: 65 pgs: 65 active+clean; 358 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-04-06T14:18:27.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:27 smithi032 ceph-mon[39730]: pgmap v13: 65 pgs: 65 active+clean; 358 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-04-06T14:18:29.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:29 smithi086 ceph-mon[47925]: pgmap v14: 65 pgs: 65 active+clean; 363 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 49 KiB/s rd, 2.2 MiB/s wr, 174 op/s 2024-04-06T14:18:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:29 smithi032 ceph-mon[39730]: pgmap v14: 65 pgs: 65 active+clean; 363 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 49 KiB/s rd, 2.2 MiB/s wr, 174 op/s 2024-04-06T14:18:31.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:31 smithi086 ceph-mon[47925]: pgmap v15: 65 pgs: 65 active+clean; 363 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 32 KiB/s rd, 1.4 MiB/s wr, 110 op/s 2024-04-06T14:18:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:31 smithi032 ceph-mon[39730]: pgmap v15: 65 pgs: 65 active+clean; 363 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 32 KiB/s rd, 1.4 MiB/s wr, 110 op/s 2024-04-06T14:18:33.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:33 smithi086 ceph-mon[47925]: pgmap v16: 65 pgs: 65 active+clean; 358 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-06T14:18:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:33 smithi032 ceph-mon[39730]: pgmap v16: 65 pgs: 65 active+clean; 358 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-06T14:18:35.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:35 smithi086 ceph-mon[47925]: pgmap v17: 65 pgs: 65 active+clean; 357 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-04-06T14:18:35.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:35 smithi032 ceph-mon[39730]: pgmap v17: 65 pgs: 65 active+clean; 357 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-04-06T14:18:36.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:36 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:36 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:37.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:37 smithi086 ceph-mon[47925]: pgmap v18: 65 pgs: 65 active+clean; 357 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-04-06T14:18:37.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:37 smithi032 ceph-mon[39730]: pgmap v18: 65 pgs: 65 active+clean; 357 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-04-06T14:18:38.504 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-06T14:18:38.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-06T14:18:38.862 DEBUG:teuthology.parallel:result is None 2024-04-06T14:18:39.417 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-06T14:18:39.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-06T14:18:39.710 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:39 smithi086 ceph-mon[47925]: pgmap v19: 65 pgs: 65 active+clean; 360 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 44 KiB/s rd, 2.0 MiB/s wr, 231 op/s 2024-04-06T14:18:39.711 DEBUG:teuthology.parallel:result is None 2024-04-06T14:18:39.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-06T14:18:39.750 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:39 smithi032 ceph-mon[39730]: pgmap v19: 65 pgs: 65 active+clean; 360 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 44 KiB/s rd, 2.0 MiB/s wr, 231 op/s 2024-04-06T14:18:39.792 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-06T14:18:39.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-06T14:18:39.848 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-06T14:18:39.848 DEBUG:teuthology.parallel:result is None 2024-04-06T14:18:40.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:40 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:40.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:40 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:40.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:40 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:18:40.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:40 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:40 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:40 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:40 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:18:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:40 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:41.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:41 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:18:41.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:41 smithi086 ceph-mon[47925]: pgmap v20: 65 pgs: 65 active+clean; 360 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 187 op/s 2024-04-06T14:18:41.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:41 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:18:41.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:41 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:18:41.911 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:41 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:18:41.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:41 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:18:41.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:41 smithi032 ceph-mon[39730]: pgmap v20: 65 pgs: 65 active+clean; 360 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 187 op/s 2024-04-06T14:18:41.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:41 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:18:41.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:41 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:18:41.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:41 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:18:42.811 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:42 smithi086 ceph-mon[47925]: Upgrade: Updating mgr.smithi086.fyyhxm 2024-04-06T14:18:42.811 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:42 smithi086 ceph-mon[47925]: Deploying daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:18:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:42 smithi032 ceph-mon[39730]: Upgrade: Updating mgr.smithi086.fyyhxm 2024-04-06T14:18:42.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:42 smithi032 ceph-mon[39730]: Deploying daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:18:43.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:43 smithi032 ceph-mon[39730]: pgmap v21: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 1.7 MiB/s wr, 270 op/s 2024-04-06T14:18:44.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:43 smithi086 ceph-mon[47925]: pgmap v21: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 1.7 MiB/s wr, 270 op/s 2024-04-06T14:18:45.838 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:45 smithi086 ceph-mon[47925]: pgmap v22: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.2 MiB/s wr, 188 op/s 2024-04-06T14:18:45.838 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:45 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:45.838 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:45 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:45.838 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:45 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:45 smithi032 ceph-mon[39730]: pgmap v22: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.2 MiB/s wr, 188 op/s 2024-04-06T14:18:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:45 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:45 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:45.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:45 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:47.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:47 smithi032 ceph-mon[39730]: pgmap v23: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-04-06T14:18:48.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:47 smithi086 ceph-mon[47925]: pgmap v23: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-04-06T14:18:49.349 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:49 smithi032 ceph-mon[39730]: pgmap v24: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.2 MiB/s wr, 183 op/s 2024-04-06T14:18:49.349 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:49.349 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:49.350 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:49.350 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:49.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:49 smithi086 ceph-mon[47925]: pgmap v24: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.2 MiB/s wr, 183 op/s 2024-04-06T14:18:49.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:51.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:51 smithi086 ceph-mon[47925]: pgmap v25: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 461 KiB/s wr, 84 op/s 2024-04-06T14:18:51.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:51 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:51.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:51 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:51.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:51 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:51 smithi032 ceph-mon[39730]: pgmap v25: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 461 KiB/s wr, 84 op/s 2024-04-06T14:18:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:51 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:51 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:51 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:18:52.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:52 smithi086 ceph-mon[47925]: Standby manager daemon smithi086.fyyhxm restarted 2024-04-06T14:18:52.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:52 smithi086 ceph-mon[47925]: Standby manager daemon smithi086.fyyhxm started 2024-04-06T14:18:52.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:52 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1064237544' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:18:52.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:52 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1064237544' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:18:52.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:52 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1064237544' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:18:52.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:52 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1064237544' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:18:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:52 smithi032 ceph-mon[39730]: Standby manager daemon smithi086.fyyhxm restarted 2024-04-06T14:18:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:52 smithi032 ceph-mon[39730]: Standby manager daemon smithi086.fyyhxm started 2024-04-06T14:18:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:52 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/1064237544' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:18:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:52 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/1064237544' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:18:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:52 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/1064237544' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:18:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:52 smithi032 ceph-mon[39730]: from='mgr.? 172.21.15.86:0/1064237544' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:18:53.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:53 smithi086 ceph-mon[47925]: pgmap v26: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 462 KiB/s wr, 84 op/s 2024-04-06T14:18:53.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:53 smithi086 ceph-mon[47925]: mgrmap e32: smithi032.peohxu(active, since 46s), standbys: smithi086.fyyhxm 2024-04-06T14:18:53.400 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:53 smithi032 ceph-mon[39730]: pgmap v26: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 462 KiB/s wr, 84 op/s 2024-04-06T14:18:53.400 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:53 smithi032 ceph-mon[39730]: mgrmap e32: smithi032.peohxu(active, since 46s), standbys: smithi086.fyyhxm 2024-04-06T14:18:55.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:18:55.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:18:55.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: pgmap v27: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 3.2 KiB/s wr, 1 op/s 2024-04-06T14:18:55.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: Upgrade: Setting container_image for all mgr 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.peohxu"}]: dispatch 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.peohxu"}]': finished 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi086.fyyhxm"}]: dispatch 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi086.fyyhxm"}]': finished 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: Upgrade: Setting container_image for all rgw 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: Upgrade: Setting container_image for all rbd-mirror 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: Upgrade: Setting container_image for all ceph-exporter 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: Upgrade: Setting container_image for all iscsi 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: Upgrade: Setting container_image for all nfs 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: Upgrade: Setting container_image for all nvmeof 2024-04-06T14:18:55.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:54 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.176 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.176 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.176 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:18:55.176 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: pgmap v27: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 3.2 KiB/s wr, 1 op/s 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: Upgrade: Setting container_image for all mgr 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.peohxu"}]: dispatch 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.peohxu"}]': finished 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi086.fyyhxm"}]: dispatch 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi086.fyyhxm"}]': finished 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: Upgrade: Setting container_image for all rgw 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: Upgrade: Setting container_image for all rbd-mirror 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: Upgrade: Setting container_image for all ceph-exporter 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: Upgrade: Setting container_image for all iscsi 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: Upgrade: Setting container_image for all nfs 2024-04-06T14:18:55.179 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.179 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:55.179 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: Upgrade: Setting container_image for all nvmeof 2024-04-06T14:18:55.179 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:54 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 -- 172.21.15.32:0/65898589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fcf80 msgr2=0x7fea100fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 --2- 172.21.15.32:0/65898589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fcf80 0x7fea100fd3f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fea00004940 tx=0x7fea00009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 -- 172.21.15.32:0/65898589 shutdown_connections 2024-04-06T14:18:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 --2- 172.21.15.32:0/65898589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fcf80 0x7fea100fd3f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 --2- 172.21.15.32:0/65898589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea100fbcd0 0x7fea100fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 -- 172.21.15.32:0/65898589 >> 172.21.15.32:0/65898589 conn(0x7fea100f7430 msgr2=0x7fea100f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 -- 172.21.15.32:0/65898589 shutdown_connections 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 -- 172.21.15.32:0/65898589 wait complete. 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 Processor -- start 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.528+0000 7fea14f71700 1 -- start start 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea14f71700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fbcd0 0x7fea10110670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea14f71700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea100fcf80 0x7fea10112bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea14f71700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10066350 con 0x7fea100fbcd0 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea14f71700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10066490 con 0x7fea100fcf80 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea0e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fbcd0 0x7fea10110670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea0e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fbcd0 0x7fea10110670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37310/0 (socket says 172.21.15.32:37310) 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea0e59c700 1 -- 172.21.15.32:0/2281341523 learned_addr learned my addr 172.21.15.32:0/2281341523 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea0dd9b700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea100fcf80 0x7fea10112bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea0e59c700 1 -- 172.21.15.32:0/2281341523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea100fcf80 msgr2=0x7fea10112bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea0e59c700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea100fcf80 0x7fea10112bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea0e59c700 1 -- 172.21.15.32:0/2281341523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea00005040 con 0x7fea100fbcd0 2024-04-06T14:18:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea0dd9b700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea100fcf80 0x7fea10112bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:18:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea0e59c700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fbcd0 0x7fea10110670 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8002a50 tx=0x7fe9f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea06ffd700 1 -- 172.21.15.32:0/2281341523 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9f8007cc0 con 0x7fea100fbcd0 2024-04-06T14:18:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea06ffd700 1 -- 172.21.15.32:0/2281341523 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9f800f040 con 0x7fea100fbcd0 2024-04-06T14:18:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea10113160 con 0x7fea100fbcd0 2024-04-06T14:18:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea06ffd700 1 -- 172.21.15.32:0/2281341523 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9f8009540 con 0x7fea100fbcd0 2024-04-06T14:18:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.529+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea10113720 con 0x7fea100fbcd0 2024-04-06T14:18:55.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.530+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1005f5f0 con 0x7fea100fbcd0 2024-04-06T14:18:55.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.531+0000 7fea06ffd700 1 -- 172.21.15.32:0/2281341523 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fe9f80096a0 con 0x7fea100fbcd0 2024-04-06T14:18:55.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.531+0000 7fea06ffd700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fe9fc06c9c0 0x7fe9fc06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:55.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.531+0000 7fea06ffd700 1 -- 172.21.15.32:0/2281341523 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fe9f808ba40 con 0x7fea100fbcd0 2024-04-06T14:18:55.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.532+0000 7fea0dd9b700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fe9fc06c9c0 0x7fe9fc06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:55.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.532+0000 7fea0dd9b700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fe9fc06c9c0 0x7fe9fc06ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fea00004f90 tx=0x7fea00009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.536+0000 7fea06ffd700 1 -- 172.21.15.32:0/2281341523 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fe9f8055d80 con 0x7fea100fbcd0 2024-04-06T14:18:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.760+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea10101380 con 0x7fe9fc06c9c0 2024-04-06T14:18:55.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.762+0000 7fea06ffd700 1 -- 172.21.15.32:0/2281341523 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7fea10101380 con 0x7fe9fc06c9c0 2024-04-06T14:18:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fe9fc06c9c0 msgr2=0x7fe9fc06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fe9fc06c9c0 0x7fe9fc06ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fea00004f90 tx=0x7fea00009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fbcd0 msgr2=0x7fea10110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fbcd0 0x7fea10110670 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8002a50 tx=0x7fe9f80046b0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 shutdown_connections 2024-04-06T14:18:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fe9fc06c9c0 0x7fe9fc06ee70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea100fbcd0 0x7fea10110670 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 --2- 172.21.15.32:0/2281341523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea100fcf80 0x7fea10112bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 >> 172.21.15.32:0/2281341523 conn(0x7fea100f7430 msgr2=0x7fea100f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 shutdown_connections 2024-04-06T14:18:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.764+0000 7fea14f71700 1 -- 172.21.15.32:0/2281341523 wait complete. 2024-04-06T14:18:55.774 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:18:55.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.873+0000 7fc833242700 1 -- 172.21.15.32:0/468041245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 msgr2=0x7fc82c102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:55.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.873+0000 7fc833242700 1 --2- 172.21.15.32:0/468041245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 0x7fc82c102320 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc818004940 tx=0x7fc818009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:55.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.873+0000 7fc833242700 1 -- 172.21.15.32:0/468041245 shutdown_connections 2024-04-06T14:18:55.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.873+0000 7fc833242700 1 --2- 172.21.15.32:0/468041245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 0x7fc82c102320 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.873+0000 7fc833242700 1 --2- 172.21.15.32:0/468041245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc82c0662e0 0x7fc82c0ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.873+0000 7fc833242700 1 -- 172.21.15.32:0/468041245 >> 172.21.15.32:0/468041245 conn(0x7fc82c0f7430 msgr2=0x7fc82c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:55.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.873+0000 7fc833242700 1 -- 172.21.15.32:0/468041245 shutdown_connections 2024-04-06T14:18:55.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.873+0000 7fc833242700 1 -- 172.21.15.32:0/468041245 wait complete. 2024-04-06T14:18:55.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc833242700 1 Processor -- start 2024-04-06T14:18:55.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc833242700 1 -- start start 2024-04-06T14:18:55.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc833242700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc82c0662e0 0x7fc82c10d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc833242700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 0x7fc82c10dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc833242700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c10e4c0 con 0x7fc82c0fff00 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc833242700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c10ed10 con 0x7fc82c0662e0 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc82bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 0x7fc82c10dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc82bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 0x7fc82c10dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37330/0 (socket says 172.21.15.32:37330) 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc82bfff700 1 -- 172.21.15.32:0/370948312 learned_addr learned my addr 172.21.15.32:0/370948312 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.874+0000 7fc830fde700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc82c0662e0 0x7fc82c10d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.875+0000 7fc82bfff700 1 -- 172.21.15.32:0/370948312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc82c0662e0 msgr2=0x7fc82c10d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.875+0000 7fc82bfff700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc82c0662e0 0x7fc82c10d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.875+0000 7fc82bfff700 1 -- 172.21.15.32:0/370948312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc818005040 con 0x7fc82c0fff00 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.875+0000 7fc82bfff700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 0x7fc82c10dec0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc818009380 tx=0x7fc818009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:55.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.875+0000 7fc8297fa700 1 -- 172.21.15.32:0/370948312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc81800e040 con 0x7fc82c0fff00 2024-04-06T14:18:55.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.875+0000 7fc8297fa700 1 -- 172.21.15.32:0/370948312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc81800bd20 con 0x7fc82c0fff00 2024-04-06T14:18:55.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.875+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82c10ef90 con 0x7fc82c0fff00 2024-04-06T14:18:55.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.875+0000 7fc8297fa700 1 -- 172.21.15.32:0/370948312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc81801c630 con 0x7fc82c0fff00 2024-04-06T14:18:55.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.875+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82c10f4d0 con 0x7fc82c0fff00 2024-04-06T14:18:55.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.876+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82c05f5f0 con 0x7fc82c0fff00 2024-04-06T14:18:55.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.877+0000 7fc8297fa700 1 -- 172.21.15.32:0/370948312 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fc818013420 con 0x7fc82c0fff00 2024-04-06T14:18:55.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.877+0000 7fc8297fa700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc81c06c9c0 0x7fc81c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:55.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.877+0000 7fc8297fa700 1 -- 172.21.15.32:0/370948312 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc81808ca00 con 0x7fc82c0fff00 2024-04-06T14:18:55.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.877+0000 7fc830fde700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc81c06c9c0 0x7fc81c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:55.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.878+0000 7fc830fde700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc81c06c9c0 0x7fc81c06ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc820009fd0 tx=0x7fc820009f20 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:55.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:55.882+0000 7fc8297fa700 1 -- 172.21.15.32:0/370948312 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc81805a7d0 con 0x7fc82c0fff00 2024-04-06T14:18:55.940 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:55 smithi032 ceph-mon[39730]: Upgrade: Updating node-exporter.smithi032 (1/2) 2024-04-06T14:18:55.940 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:55 smithi032 ceph-mon[39730]: Deploying daemon node-exporter.smithi032 on smithi032 2024-04-06T14:18:56.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.116+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc82c10f800 con 0x7fc81c06c9c0 2024-04-06T14:18:56.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.118+0000 7fc8297fa700 1 -- 172.21.15.32:0/370948312 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7fc82c10f800 con 0x7fc81c06c9c0 2024-04-06T14:18:56.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc81c06c9c0 msgr2=0x7fc81c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc81c06c9c0 0x7fc81c06ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc820009fd0 tx=0x7fc820009f20 comp rx=0 tx=0).stop 2024-04-06T14:18:56.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 msgr2=0x7fc82c10dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 0x7fc82c10dec0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc818009380 tx=0x7fc818009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:56.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 shutdown_connections 2024-04-06T14:18:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc81c06c9c0 0x7fc81c06ee70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc82c0662e0 0x7fc82c10d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 --2- 172.21.15.32:0/370948312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc82c0fff00 0x7fc82c10dec0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 >> 172.21.15.32:0/370948312 conn(0x7fc82c0f7430 msgr2=0x7fc82c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 shutdown_connections 2024-04-06T14:18:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.120+0000 7fc833242700 1 -- 172.21.15.32:0/370948312 wait complete. 2024-04-06T14:18:56.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:55 smithi086 ceph-mon[47925]: Upgrade: Updating node-exporter.smithi032 (1/2) 2024-04-06T14:18:56.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:55 smithi086 ceph-mon[47925]: Deploying daemon node-exporter.smithi032 on smithi032 2024-04-06T14:18:56.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.231+0000 7f89056ea700 1 -- 172.21.15.32:0/3787615238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fb490 msgr2=0x7f89000fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.231+0000 7f89056ea700 1 --2- 172.21.15.32:0/3787615238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fb490 0x7f89000fb8a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f88f0004a50 tx=0x7f88f000af40 comp rx=0 tx=0).stop 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.231+0000 7f89056ea700 1 -- 172.21.15.32:0/3787615238 shutdown_connections 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.231+0000 7f89056ea700 1 --2- 172.21.15.32:0/3787615238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89000fc740 0x7f89000fcbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.231+0000 7f89056ea700 1 --2- 172.21.15.32:0/3787615238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fb490 0x7f89000fb8a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.231+0000 7f89056ea700 1 -- 172.21.15.32:0/3787615238 >> 172.21.15.32:0/3787615238 conn(0x7f89000f6c10 msgr2=0x7f89000f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.231+0000 7f89056ea700 1 -- 172.21.15.32:0/3787615238 shutdown_connections 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.231+0000 7f89056ea700 1 -- 172.21.15.32:0/3787615238 wait complete. 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f89056ea700 1 Processor -- start 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f89056ea700 1 -- start start 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f89056ea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89000fb490 0x7f890010d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f89056ea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fc740 0x7f890010de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f89056ea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890010e400 con 0x7f89000fc740 2024-04-06T14:18:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f89056ea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890010e570 con 0x7f89000fb490 2024-04-06T14:18:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f88fe7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fc740 0x7f890010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f88feffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89000fb490 0x7f890010d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f88fe7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fc740 0x7f890010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37348/0 (socket says 172.21.15.32:37348) 2024-04-06T14:18:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.232+0000 7f88fe7fc700 1 -- 172.21.15.32:0/365334871 learned_addr learned my addr 172.21.15.32:0/365334871 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f88feffd700 1 -- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fc740 msgr2=0x7f890010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f88feffd700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fc740 0x7f890010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f88feffd700 1 -- 172.21.15.32:0/365334871 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f0005040 con 0x7f89000fb490 2024-04-06T14:18:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f88fe7fc700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fc740 0x7f890010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:18:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f88feffd700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89000fb490 0x7f890010d8c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f88f0000c00 tx=0x7f88f000af40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f88e77fe700 1 -- 172.21.15.32:0/365334871 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f000f050 con 0x7f89000fb490 2024-04-06T14:18:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8900112f70 con 0x7f89000fb490 2024-04-06T14:18:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8900113390 con 0x7f89000fb490 2024-04-06T14:18:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f88e77fe700 1 -- 172.21.15.32:0/365334871 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88f000c8c0 con 0x7f89000fb490 2024-04-06T14:18:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.233+0000 7f88e77fe700 1 -- 172.21.15.32:0/365334871 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f00276a0 con 0x7f89000fb490 2024-04-06T14:18:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.234+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890004c710 con 0x7f89000fb490 2024-04-06T14:18:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.235+0000 7f88e77fe700 1 -- 172.21.15.32:0/365334871 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f88f001d480 con 0x7f89000fb490 2024-04-06T14:18:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.235+0000 7f88e77fe700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f88e806c9d0 0x7f88e806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.236+0000 7f88e77fe700 1 -- 172.21.15.32:0/365334871 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f88f008d770 con 0x7f89000fb490 2024-04-06T14:18:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.236+0000 7f88fe7fc700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f88e806c9d0 0x7f88e806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.236+0000 7f88fe7fc700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f88e806c9d0 0x7f88e806ee80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88f40041d0 tx=0x7f88f400b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:56.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.241+0000 7f88e77fe700 1 -- 172.21.15.32:0/365334871 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f88f005b540 con 0x7f89000fb490 2024-04-06T14:18:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.454+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f89001135d0 con 0x7f88e806c9d0 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.464+0000 7f88e77fe700 1 -- 172.21.15.32:0/365334871 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f89001135d0 con 0x7f88e806c9d0 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (93s) 5s ago 10m 28.5M - 0.25.0 c8568f914cd2 73706679ea15 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 5s ago 10m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (9m) 8s ago 9m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (86s) 5s ago 10m 78.4M - 9.4.7 954c08fa6188 88d7529c0ea5 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (4m) 5s ago 4m 12.6M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (4m) 5s ago 4m 154M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (4m) 8s ago 4m 151M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (4m) 8s ago 4m 15.4M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:18:56.464 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (71s) 5s ago 11m 492M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (11s) 8s ago 9m 55.7M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 5s ago 12m 54.4M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (9m) 8s ago 9m 41.8M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (119s) 5s ago 10m 16.5M - 1.5.0 0da6a335fe13 c3cb793eb173 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (77s) 8s ago 9m 16.9M - 1.5.0 0da6a335fe13 05e92e59b710 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 5s ago 8m 194M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 5s ago 8m 190M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 5s ago 7m 161M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (7m) 8s ago 7m 224M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (6m) 8s ago 6m 202M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (6m) 8s ago 6m 184M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:18:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (17s) 5s ago 10m 38.4M - 2.43.0 a07b618ecd1d 880bde2897ef 2024-04-06T14:18:56.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f88e806c9d0 msgr2=0x7f88e806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f88e806c9d0 0x7f88e806ee80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88f40041d0 tx=0x7f88f400b040 comp rx=0 tx=0).stop 2024-04-06T14:18:56.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89000fb490 msgr2=0x7f890010d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89000fb490 0x7f890010d8c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f88f0000c00 tx=0x7f88f000af40 comp rx=0 tx=0).stop 2024-04-06T14:18:56.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 shutdown_connections 2024-04-06T14:18:56.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f88e806c9d0 0x7f88e806ee80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89000fb490 0x7f890010d8c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 --2- 172.21.15.32:0/365334871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89000fc740 0x7f890010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 >> 172.21.15.32:0/365334871 conn(0x7f89000f6c10 msgr2=0x7f89000f82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:56.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 shutdown_connections 2024-04-06T14:18:56.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.466+0000 7f89056ea700 1 -- 172.21.15.32:0/365334871 wait complete. 2024-04-06T14:18:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.568+0000 7f17c80f2700 1 -- 172.21.15.32:0/4122452065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 msgr2=0x7f17c00fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.568+0000 7f17c80f2700 1 --2- 172.21.15.32:0/4122452065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 0x7f17c00fd3f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f17b4004940 tx=0x7f17b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.568+0000 7f17c80f2700 1 -- 172.21.15.32:0/4122452065 shutdown_connections 2024-04-06T14:18:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.568+0000 7f17c80f2700 1 --2- 172.21.15.32:0/4122452065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 0x7f17c00fd3f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.568+0000 7f17c80f2700 1 --2- 172.21.15.32:0/4122452065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17c00fbcd0 0x7f17c00fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.568+0000 7f17c80f2700 1 -- 172.21.15.32:0/4122452065 >> 172.21.15.32:0/4122452065 conn(0x7f17c00f7430 msgr2=0x7f17c00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.568+0000 7f17c80f2700 1 -- 172.21.15.32:0/4122452065 shutdown_connections 2024-04-06T14:18:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.568+0000 7f17c80f2700 1 -- 172.21.15.32:0/4122452065 wait complete. 2024-04-06T14:18:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.569+0000 7f17c80f2700 1 Processor -- start 2024-04-06T14:18:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.569+0000 7f17c80f2700 1 -- start start 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.569+0000 7f17c80f2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17c00fbcd0 0x7f17c010c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.569+0000 7f17c80f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 0x7f17c010ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.569+0000 7f17c80f2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c010d2d0 con 0x7f17c00fcf80 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.569+0000 7f17c80f2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c010db20 con 0x7f17c00fbcd0 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c568d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 0x7f17c010ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c568d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 0x7f17c010ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37366/0 (socket says 172.21.15.32:37366) 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c568d700 1 -- 172.21.15.32:0/1065681279 learned_addr learned my addr 172.21.15.32:0/1065681279 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c5e8e700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17c00fbcd0 0x7f17c010c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c568d700 1 -- 172.21.15.32:0/1065681279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17c00fbcd0 msgr2=0x7f17c010c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c568d700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17c00fbcd0 0x7f17c010c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c568d700 1 -- 172.21.15.32:0/1065681279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b4005040 con 0x7f17c00fcf80 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c568d700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 0x7f17c010ccd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f17b40137b0 tx=0x7f17b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17b27fc700 1 -- 172.21.15.32:0/1065681279 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17b400e070 con 0x7f17c00fcf80 2024-04-06T14:18:56.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17b27fc700 1 -- 172.21.15.32:0/1065681279 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17b401c400 con 0x7f17c00fcf80 2024-04-06T14:18:56.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c010dda0 con 0x7f17c00fcf80 2024-04-06T14:18:56.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17b27fc700 1 -- 172.21.15.32:0/1065681279 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17b400b650 con 0x7f17c00fcf80 2024-04-06T14:18:56.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.570+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c010e280 con 0x7f17c00fcf80 2024-04-06T14:18:56.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.572+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c005f5f0 con 0x7f17c00fcf80 2024-04-06T14:18:56.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.572+0000 7f17b27fc700 1 -- 172.21.15.32:0/1065681279 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f17b401c9e0 con 0x7f17c00fcf80 2024-04-06T14:18:56.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.573+0000 7f17b27fc700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f17ac06c9c0 0x7f17ac06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:56.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.573+0000 7f17b27fc700 1 -- 172.21.15.32:0/1065681279 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f17b408d330 con 0x7f17c00fcf80 2024-04-06T14:18:56.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.573+0000 7f17c5e8e700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f17ac06c9c0 0x7f17ac06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:56.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.573+0000 7f17c5e8e700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f17ac06c9c0 0x7f17ac06ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009b80 tx=0x7f17bc00b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:56.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.578+0000 7f17b27fc700 1 -- 172.21.15.32:0/1065681279 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f17b405b100 con 0x7f17c00fcf80 2024-04-06T14:18:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.869+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f17c004bed0 con 0x7f17c00fcf80 2024-04-06T14:18:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.870+0000 7f17b27fc700 1 -- 172.21.15.32:0/1065681279 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f17b4004490 con 0x7f17c00fcf80 2024-04-06T14:18:56.870 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:18:56.870 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:18:56.870 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:18:56.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:18:56.870 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:18:56.870 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 12, 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:18:56.871 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:18:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f17ac06c9c0 msgr2=0x7f17ac06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f17ac06c9c0 0x7f17ac06ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009b80 tx=0x7f17bc00b040 comp rx=0 tx=0).stop 2024-04-06T14:18:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 msgr2=0x7f17c010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 0x7f17c010ccd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f17b40137b0 tx=0x7f17b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 shutdown_connections 2024-04-06T14:18:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f17ac06c9c0 0x7f17ac06ee70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17c00fbcd0 0x7f17c010c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 --2- 172.21.15.32:0/1065681279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c00fcf80 0x7f17c010ccd0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 >> 172.21.15.32:0/1065681279 conn(0x7f17c00f7430 msgr2=0x7f17c00f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:56.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 shutdown_connections 2024-04-06T14:18:56.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.872+0000 7f17c80f2700 1 -- 172.21.15.32:0/1065681279 wait complete. 2024-04-06T14:18:56.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.977+0000 7ff4b217d700 1 -- 172.21.15.32:0/1120113653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 msgr2=0x7ff4ac0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.977+0000 7ff4b217d700 1 --2- 172.21.15.32:0/1120113653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 0x7ff4ac0fc0e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff49c004940 tx=0x7ff49c009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.977+0000 7ff4b217d700 1 -- 172.21.15.32:0/1120113653 shutdown_connections 2024-04-06T14:18:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.977+0000 7ff4b217d700 1 --2- 172.21.15.32:0/1120113653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4ac0fcf80 0x7ff4ac0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.977+0000 7ff4b217d700 1 --2- 172.21.15.32:0/1120113653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 0x7ff4ac0fc0e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.977+0000 7ff4b217d700 1 -- 172.21.15.32:0/1120113653 >> 172.21.15.32:0/1120113653 conn(0x7ff4ac0f7430 msgr2=0x7ff4ac0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.977+0000 7ff4b217d700 1 -- 172.21.15.32:0/1120113653 shutdown_connections 2024-04-06T14:18:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.977+0000 7ff4b217d700 1 -- 172.21.15.32:0/1120113653 wait complete. 2024-04-06T14:18:56.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4b217d700 1 Processor -- start 2024-04-06T14:18:56.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4b217d700 1 -- start start 2024-04-06T14:18:56.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4b217d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 0x7ff4ac10d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:56.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4b217d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4ac0fcf80 0x7ff4ac10ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:56.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4b217d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac10e4d0 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4b217d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac10ecd0 con 0x7ff4ac0fcf80 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4ab7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 0x7ff4ac10d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4ab7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 0x7ff4ac10d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37384/0 (socket says 172.21.15.32:37384) 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4ab7fe700 1 -- 172.21.15.32:0/3622306710 learned_addr learned my addr 172.21.15.32:0/3622306710 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4aaffd700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4ac0fcf80 0x7ff4ac10ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4ab7fe700 1 -- 172.21.15.32:0/3622306710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4ac0fcf80 msgr2=0x7ff4ac10ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4ab7fe700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4ac0fcf80 0x7ff4ac10ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.978+0000 7ff4ab7fe700 1 -- 172.21.15.32:0/3622306710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c005040 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.979+0000 7ff4ab7fe700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 0x7ff4ac10d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009380 tx=0x7ff49c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.979+0000 7ff493fff700 1 -- 172.21.15.32:0/3622306710 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff49c00e040 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.979+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ac10ef50 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.979+0000 7ff493fff700 1 -- 172.21.15.32:0/3622306710 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff49c00bd20 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.979+0000 7ff493fff700 1 -- 172.21.15.32:0/3622306710 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff49c01c630 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.979+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac10f490 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:56 smithi032 ceph-mon[39730]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:56.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:56 smithi032 ceph-mon[39730]: pgmap v28: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 3.2 KiB/s wr, 1 op/s 2024-04-06T14:18:56.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:56 smithi032 ceph-mon[39730]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:56.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:56 smithi032 ceph-mon[39730]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.980+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac101b10 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.980+0000 7ff493fff700 1 -- 172.21.15.32:0/3622306710 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7ff49c013420 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.981+0000 7ff493fff700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff49406c9c0 0x7ff49406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:18:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.981+0000 7ff493fff700 1 -- 172.21.15.32:0/3622306710 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff49c08caa0 con 0x7ff4ac0fbcd0 2024-04-06T14:18:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.981+0000 7ff4aaffd700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff49406c9c0 0x7ff49406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:18:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.981+0000 7ff4aaffd700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff49406c9c0 0x7ff49406ee70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0007fd0 tx=0x7ff4a0009040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:18:56.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:56.985+0000 7ff493fff700 1 -- 172.21.15.32:0/3622306710 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff49c05a7d0 con 0x7ff4ac0fbcd0 2024-04-06T14:18:57.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:56 smithi086 ceph-mon[47925]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:57.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:56 smithi086 ceph-mon[47925]: pgmap v28: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 3.2 KiB/s wr, 1 op/s 2024-04-06T14:18:57.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:56 smithi086 ceph-mon[47925]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:57.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:56 smithi086 ceph-mon[47925]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:57.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.243+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4ac10f6b0 con 0x7ff49406c9c0 2024-04-06T14:18:57.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.245+0000 7ff493fff700 1 -- 172.21.15.32:0/3622306710 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7ff4ac10f6b0 con 0x7ff49406c9c0 2024-04-06T14:18:57.245 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:18:57.245 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:18:57.245 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:18:57.245 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T14:18:57.245 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:18:57.245 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-06T14:18:57.246 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:18:57.246 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/2 daemons upgraded", 2024-04-06T14:18:57.246 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-06T14:18:57.246 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:18:57.246 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:18:57.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff49406c9c0 msgr2=0x7ff49406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:57.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff49406c9c0 0x7ff49406ee70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0007fd0 tx=0x7ff4a0009040 comp rx=0 tx=0).stop 2024-04-06T14:18:57.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 msgr2=0x7ff4ac10d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:18:57.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 0x7ff4ac10d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009380 tx=0x7ff49c009e30 comp rx=0 tx=0).stop 2024-04-06T14:18:57.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 shutdown_connections 2024-04-06T14:18:57.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff49406c9c0 0x7ff49406ee70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4ac0fbcd0 0x7ff4ac10d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 --2- 172.21.15.32:0/3622306710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4ac0fcf80 0x7ff4ac10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:18:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 >> 172.21.15.32:0/3622306710 conn(0x7ff4ac0f7430 msgr2=0x7ff4ac065c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:18:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 shutdown_connections 2024-04-06T14:18:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:18:57.247+0000 7ff4b217d700 1 -- 172.21.15.32:0/3622306710 wait complete. 2024-04-06T14:18:58.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:57 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1065681279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:58.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:57 smithi086 ceph-mon[47925]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:58.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:57 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1065681279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:18:58.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:57 smithi032 ceph-mon[39730]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:18:59.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:58 smithi086 ceph-mon[47925]: pgmap v29: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 3.2 KiB/s wr, 1 op/s 2024-04-06T14:18:59.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:58 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:59.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:58 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:59.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:58 smithi032 ceph-mon[39730]: pgmap v29: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 3.2 KiB/s wr, 1 op/s 2024-04-06T14:18:59.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:58 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:18:59.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:58 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:00.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:59 smithi086 ceph-mon[47925]: Upgrade: Updating node-exporter.smithi086 (2/2) 2024-04-06T14:19:00.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:18:59 smithi086 ceph-mon[47925]: Deploying daemon node-exporter.smithi086 on smithi086 2024-04-06T14:19:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:59 smithi032 ceph-mon[39730]: Upgrade: Updating node-exporter.smithi086 (2/2) 2024-04-06T14:19:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:18:59 smithi032 ceph-mon[39730]: Deploying daemon node-exporter.smithi086 on smithi086 2024-04-06T14:19:01.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:00 smithi086 ceph-mon[47925]: pgmap v30: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-06T14:19:01.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:00 smithi032 ceph-mon[39730]: pgmap v30: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-06T14:19:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:02 smithi032 ceph-mon[39730]: pgmap v31: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-06T14:19:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:02 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:02 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:02 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:02.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:02 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:02.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:02 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:03.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:02 smithi086 ceph-mon[47925]: pgmap v31: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-06T14:19:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:02 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:02 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:02 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:02 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:03.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:02 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:05.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:04 smithi086 ceph-mon[47925]: pgmap v32: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:05.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:04 smithi032 ceph-mon[39730]: pgmap v32: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:19:06.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:06.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:06.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:06.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:06.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:19:07.358 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:07 smithi086 ceph-mon[47925]: pgmap v33: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:07.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:07 smithi032 ceph-mon[39730]: pgmap v33: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:08.666 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:08 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:08.666 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:08 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:08.666 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:08 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:08.666 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:08 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:08.759 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:08 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:08.759 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:08 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:08.759 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:08 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:08.759 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:08 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:09 smithi032 ceph-mon[39730]: pgmap v34: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:09.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:09 smithi086 ceph-mon[47925]: pgmap v34: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:10 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:10 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:10.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:10 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:10.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:10 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:11.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:11 smithi032 ceph-mon[39730]: pgmap v35: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:11.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:11 smithi086 ceph-mon[47925]: pgmap v35: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:13.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: pgmap v36: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:13.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:13.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:13.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:19:13.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:19:13.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:13.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:13.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.575 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.576 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:13 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: pgmap v36: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:13.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:13 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:15.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:15 smithi032 ceph-mon[39730]: pgmap v37: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:15.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:15 smithi032 ceph-mon[39730]: Upgrade: Updating prometheus.smithi032 2024-04-06T14:19:15.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:15 smithi032 ceph-mon[39730]: Deploying daemon prometheus.smithi032 on smithi032 2024-04-06T14:19:15.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:15 smithi086 ceph-mon[47925]: pgmap v37: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:15 smithi086 ceph-mon[47925]: Upgrade: Updating prometheus.smithi032 2024-04-06T14:19:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:15 smithi086 ceph-mon[47925]: Deploying daemon prometheus.smithi032 on smithi032 2024-04-06T14:19:17.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:17 smithi086 ceph-mon[47925]: pgmap v38: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:17.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:17 smithi032 ceph-mon[39730]: pgmap v38: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:19.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:19 smithi032 ceph-mon[39730]: pgmap v39: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:19.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:19 smithi086 ceph-mon[47925]: pgmap v39: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:20 smithi032 ceph-mon[39730]: pgmap v40: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:21.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:19:21.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:20 smithi086 ceph-mon[47925]: pgmap v40: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:19:23.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:22 smithi032 ceph-mon[39730]: pgmap v41: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:23.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:22 smithi086 ceph-mon[47925]: pgmap v41: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:24.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:24 smithi032 ceph-mon[39730]: pgmap v42: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:25.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:24 smithi086 ceph-mon[47925]: pgmap v42: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:26 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:26 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:26 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:26 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:26 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:26 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:26 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:26.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:26 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.360+0000 7fb83e72e700 1 -- 172.21.15.32:0/366888472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fbcd0 msgr2=0x7fb8380fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.360+0000 7fb83e72e700 1 --2- 172.21.15.32:0/366888472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fbcd0 0x7fb8380fc0e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb828004940 tx=0x7fb828009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.360+0000 7fb83e72e700 1 -- 172.21.15.32:0/366888472 shutdown_connections 2024-04-06T14:19:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.360+0000 7fb83e72e700 1 --2- 172.21.15.32:0/366888472 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8380fcf80 0x7fb8380fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.360+0000 7fb83e72e700 1 --2- 172.21.15.32:0/366888472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fbcd0 0x7fb8380fc0e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.360+0000 7fb83e72e700 1 -- 172.21.15.32:0/366888472 >> 172.21.15.32:0/366888472 conn(0x7fb8380f7430 msgr2=0x7fb8380f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.360+0000 7fb83e72e700 1 -- 172.21.15.32:0/366888472 shutdown_connections 2024-04-06T14:19:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.360+0000 7fb83e72e700 1 -- 172.21.15.32:0/366888472 wait complete. 2024-04-06T14:19:27.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.361+0000 7fb83e72e700 1 Processor -- start 2024-04-06T14:19:27.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.361+0000 7fb83e72e700 1 -- start start 2024-04-06T14:19:27.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.361+0000 7fb83e72e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8380fbcd0 0x7fb83810c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.361+0000 7fb83e72e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fcf80 0x7fb83810cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.361+0000 7fb83e72e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83810d370 con 0x7fb8380fcf80 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb83e72e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83810dbc0 con 0x7fb8380fbcd0 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb8377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fcf80 0x7fb83810cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb8377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fcf80 0x7fb83810cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39214/0 (socket says 172.21.15.32:39214) 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb8377fe700 1 -- 172.21.15.32:0/2592164543 learned_addr learned my addr 172.21.15.32:0/2592164543 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb837fff700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8380fbcd0 0x7fb83810c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb8377fe700 1 -- 172.21.15.32:0/2592164543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8380fbcd0 msgr2=0x7fb83810c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb8377fe700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8380fbcd0 0x7fb83810c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb8377fe700 1 -- 172.21.15.32:0/2592164543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb828005040 con 0x7fb8380fcf80 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb8377fe700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fcf80 0x7fb83810cd70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00fce0 tx=0x7fb82c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb834ff9700 1 -- 172.21.15.32:0/2592164543 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb82c007ea0 con 0x7fb8380fcf80 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83810de40 con 0x7fb8380fcf80 2024-04-06T14:19:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb834ff9700 1 -- 172.21.15.32:0/2592164543 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb82c00dd80 con 0x7fb8380fcf80 2024-04-06T14:19:27.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb834ff9700 1 -- 172.21.15.32:0/2592164543 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb82c012460 con 0x7fb8380fcf80 2024-04-06T14:19:27.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.362+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83810e400 con 0x7fb8380fcf80 2024-04-06T14:19:27.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.363+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83805f5f0 con 0x7fb8380fcf80 2024-04-06T14:19:27.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.364+0000 7fb834ff9700 1 -- 172.21.15.32:0/2592164543 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fb82c009360 con 0x7fb8380fcf80 2024-04-06T14:19:27.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.364+0000 7fb834ff9700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb82006c9c0 0x7fb82006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:27.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.364+0000 7fb834ff9700 1 -- 172.21.15.32:0/2592164543 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb82c08e410 con 0x7fb8380fcf80 2024-04-06T14:19:27.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.364+0000 7fb837fff700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb82006c9c0 0x7fb82006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:27.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.365+0000 7fb837fff700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb82006c9c0 0x7fb82006ee70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb82800b900 tx=0x7fb828009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:27.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.369+0000 7fb834ff9700 1 -- 172.21.15.32:0/2592164543 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fb82c05c160 con 0x7fb8380fcf80 2024-04-06T14:19:27.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.595+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb83810e760 con 0x7fb82006c9c0 2024-04-06T14:19:27.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.596+0000 7fb834ff9700 1 -- 172.21.15.32:0/2592164543 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fb83810e760 con 0x7fb82006c9c0 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.598+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb82006c9c0 msgr2=0x7fb82006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.598+0000 7fb83e72e700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb82006c9c0 0x7fb82006ee70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb82800b900 tx=0x7fb828009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.598+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fcf80 msgr2=0x7fb83810cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.599+0000 7fb83e72e700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fcf80 0x7fb83810cd70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00fce0 tx=0x7fb82c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.599+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 shutdown_connections 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.599+0000 7fb83e72e700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb82006c9c0 0x7fb82006ee70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.599+0000 7fb83e72e700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8380fbcd0 0x7fb83810c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.599+0000 7fb83e72e700 1 --2- 172.21.15.32:0/2592164543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8380fcf80 0x7fb83810cd70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.599+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 >> 172.21.15.32:0/2592164543 conn(0x7fb8380f7430 msgr2=0x7fb8380f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.599+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 shutdown_connections 2024-04-06T14:19:27.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.599+0000 7fb83e72e700 1 -- 172.21.15.32:0/2592164543 wait complete. 2024-04-06T14:19:27.609 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:19:27.690 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:27 smithi032 ceph-mon[39730]: pgmap v43: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.706+0000 7f136ec8a700 1 -- 172.21.15.32:0/162315105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 msgr2=0x7f13680ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.706+0000 7f136ec8a700 1 --2- 172.21.15.32:0/162315105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 0x7f13680ff9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f135c004940 tx=0x7f135c009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.706+0000 7f136ec8a700 1 -- 172.21.15.32:0/162315105 shutdown_connections 2024-04-06T14:19:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.706+0000 7f136ec8a700 1 --2- 172.21.15.32:0/162315105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13680fff00 0x7f1368102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.706+0000 7f136ec8a700 1 --2- 172.21.15.32:0/162315105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 0x7f13680ff9c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.706+0000 7f136ec8a700 1 -- 172.21.15.32:0/162315105 >> 172.21.15.32:0/162315105 conn(0x7f13680f7430 msgr2=0x7f13680f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.707+0000 7f136ec8a700 1 -- 172.21.15.32:0/162315105 shutdown_connections 2024-04-06T14:19:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.707+0000 7f136ec8a700 1 -- 172.21.15.32:0/162315105 wait complete. 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.707+0000 7f136ec8a700 1 Processor -- start 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.707+0000 7f136ec8a700 1 -- start start 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ec8a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 0x7f136810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ec8a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13680fff00 0x7f136810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ec8a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136810e4b0 con 0x7f13680662e0 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ec8a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136810ed00 con 0x7f13680fff00 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ca26700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 0x7f136810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ca26700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 0x7f136810d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39242/0 (socket says 172.21.15.32:39242) 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ca26700 1 -- 172.21.15.32:0/2980227269 learned_addr learned my addr 172.21.15.32:0/2980227269 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:19:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f1367fff700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13680fff00 0x7f136810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ca26700 1 -- 172.21.15.32:0/2980227269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13680fff00 msgr2=0x7f136810deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ca26700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13680fff00 0x7f136810deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ca26700 1 -- 172.21.15.32:0/2980227269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c005040 con 0x7f13680662e0 2024-04-06T14:19:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f136ca26700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 0x7f136810d970 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f135c00b3b0 tx=0x7f135c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.708+0000 7f13657fa700 1 -- 172.21.15.32:0/2980227269 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f135c00e040 con 0x7f13680662e0 2024-04-06T14:19:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.709+0000 7f13657fa700 1 -- 172.21.15.32:0/2980227269 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f135c00bc50 con 0x7f13680662e0 2024-04-06T14:19:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.709+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136810ef80 con 0x7f13680662e0 2024-04-06T14:19:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.709+0000 7f13657fa700 1 -- 172.21.15.32:0/2980227269 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f135c01c3f0 con 0x7f13680662e0 2024-04-06T14:19:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.709+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136810f4c0 con 0x7f13680662e0 2024-04-06T14:19:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.709+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136805f5f0 con 0x7f13680662e0 2024-04-06T14:19:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.710+0000 7f13657fa700 1 -- 172.21.15.32:0/2980227269 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f135c013420 con 0x7f13680662e0 2024-04-06T14:19:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.711+0000 7f13657fa700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f135806c9c0 0x7f135806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.711+0000 7f13657fa700 1 -- 172.21.15.32:0/2980227269 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f135c08c8f0 con 0x7f13680662e0 2024-04-06T14:19:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.711+0000 7f1367fff700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f135806c9c0 0x7f135806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:27.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.711+0000 7f1367fff700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f135806c9c0 0x7f135806ee70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1354004f40 tx=0x7f135400b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:27.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.715+0000 7f13657fa700 1 -- 172.21.15.32:0/2980227269 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f135c05a6c0 con 0x7f13680662e0 2024-04-06T14:19:27.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:27 smithi086 ceph-mon[47925]: pgmap v43: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:27.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.945+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f136810f840 con 0x7f135806c9c0 2024-04-06T14:19:27.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.947+0000 7f13657fa700 1 -- 172.21.15.32:0/2980227269 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f136810f840 con 0x7f135806c9c0 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f135806c9c0 msgr2=0x7f135806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f135806c9c0 0x7f135806ee70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1354004f40 tx=0x7f135400b040 comp rx=0 tx=0).stop 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 msgr2=0x7f136810d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 0x7f136810d970 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f135c00b3b0 tx=0x7f135c009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 shutdown_connections 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f135806c9c0 0x7f135806ee70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680662e0 0x7f136810d970 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 --2- 172.21.15.32:0/2980227269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13680fff00 0x7f136810deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 >> 172.21.15.32:0/2980227269 conn(0x7f13680f7430 msgr2=0x7f13680f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 shutdown_connections 2024-04-06T14:19:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:27.949+0000 7f136ec8a700 1 -- 172.21.15.32:0/2980227269 wait complete. 2024-04-06T14:19:28.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 -- 172.21.15.32:0/3563571206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 msgr2=0x7fbc84104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 --2- 172.21.15.32:0/3563571206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 0x7fbc84104760 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbc70004940 tx=0x7fbc70009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 -- 172.21.15.32:0/3563571206 shutdown_connections 2024-04-06T14:19:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 --2- 172.21.15.32:0/3563571206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 0x7fbc84104760 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 --2- 172.21.15.32:0/3563571206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84066260 0x7fbc84101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 -- 172.21.15.32:0/3563571206 >> 172.21.15.32:0/3563571206 conn(0x7fbc840f7430 msgr2=0x7fbc840f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 -- 172.21.15.32:0/3563571206 shutdown_connections 2024-04-06T14:19:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 -- 172.21.15.32:0/3563571206 wait complete. 2024-04-06T14:19:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 Processor -- start 2024-04-06T14:19:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.059+0000 7fbc8b231700 1 -- start start 2024-04-06T14:19:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc8b231700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84066260 0x7fbc8410fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc8b231700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 0x7fbc84110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc8b231700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc84110780 con 0x7fbc84102380 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc8b231700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc84110f80 con 0x7fbc84066260 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc83fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 0x7fbc84110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc83fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 0x7fbc84110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39264/0 (socket says 172.21.15.32:39264) 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc83fff700 1 -- 172.21.15.32:0/4215326974 learned_addr learned my addr 172.21.15.32:0/4215326974 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc88fcd700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84066260 0x7fbc8410fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc83fff700 1 -- 172.21.15.32:0/4215326974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84066260 msgr2=0x7fbc8410fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc83fff700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84066260 0x7fbc8410fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc83fff700 1 -- 172.21.15.32:0/4215326974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc70005040 con 0x7fbc84102380 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.060+0000 7fbc83fff700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 0x7fbc84110180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbc70000c00 tx=0x7fbc70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.061+0000 7fbc817fa700 1 -- 172.21.15.32:0/4215326974 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc7000e040 con 0x7fbc84102380 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.061+0000 7fbc817fa700 1 -- 172.21.15.32:0/4215326974 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc70004290 con 0x7fbc84102380 2024-04-06T14:19:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.061+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc84111200 con 0x7fbc84102380 2024-04-06T14:19:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.061+0000 7fbc817fa700 1 -- 172.21.15.32:0/4215326974 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc7001c3f0 con 0x7fbc84102380 2024-04-06T14:19:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.061+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc84111740 con 0x7fbc84102380 2024-04-06T14:19:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.062+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8405f5f0 con 0x7fbc84102380 2024-04-06T14:19:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.062+0000 7fbc817fa700 1 -- 172.21.15.32:0/4215326974 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fbc7000bbf0 con 0x7fbc84102380 2024-04-06T14:19:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.063+0000 7fbc817fa700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fbc7406c9c0 0x7fbc7406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.063+0000 7fbc817fa700 1 -- 172.21.15.32:0/4215326974 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fbc7008d0d0 con 0x7fbc84102380 2024-04-06T14:19:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.063+0000 7fbc88fcd700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fbc7406c9c0 0x7fbc7406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.063+0000 7fbc88fcd700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fbc7406c9c0 0x7fbc7406ee70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbc84065fa0 tx=0x7fbc78009040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:28.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.067+0000 7fbc817fa700 1 -- 172.21.15.32:0/4215326974 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fbc7005ae20 con 0x7fbc84102380 2024-04-06T14:19:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.281+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc84111ad0 con 0x7fbc7406c9c0 2024-04-06T14:19:28.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.291+0000 7fbc817fa700 1 -- 172.21.15.32:0/4215326974 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fbc84111ad0 con 0x7fbc7406c9c0 2024-04-06T14:19:28.291 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:19:28.291 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (2m) 2s ago 11m 28.5M - 0.25.0 c8568f914cd2 73706679ea15 2024-04-06T14:19:28.291 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 2s ago 11m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:19:28.292 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (10m) 23s ago 10m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:19:28.292 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (118s) 2s ago 10m 82.3M - 9.4.7 954c08fa6188 88d7529c0ea5 2024-04-06T14:19:28.292 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (5m) 2s ago 5m 12.7M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:19:28.292 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (5m) 2s ago 5m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:19:28.292 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (5m) 23s ago 5m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:19:28.292 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (5m) 23s ago 5m 15.5M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:19:28.293 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (103s) 2s ago 12m 497M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:19:28.293 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (43s) 23s ago 10m 449M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:19:28.293 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 2s ago 12m 53.9M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:19:28.293 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (10m) 23s ago 10m 43.8M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:19:28.293 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (30s) 2s ago 11m 8484k - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:19:28.293 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (27s) 23s ago 10m 3397k - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:19:28.293 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 2s ago 9m 194M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:19:28.294 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 2s ago 8m 191M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:19:28.294 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (8m) 2s ago 8m 162M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:19:28.294 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (7m) 23s ago 7m 224M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:19:28.294 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (7m) 23s ago 7m 202M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:19:28.294 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (6m) 23s ago 6m 183M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:19:28.294 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (8s) 2s ago 10m 37.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fbc7406c9c0 msgr2=0x7fbc7406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fbc7406c9c0 0x7fbc7406ee70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbc84065fa0 tx=0x7fbc78009040 comp rx=0 tx=0).stop 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 msgr2=0x7fbc84110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 0x7fbc84110180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbc70000c00 tx=0x7fbc70009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 shutdown_connections 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fbc7406c9c0 0x7fbc7406ee70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84066260 0x7fbc8410fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 --2- 172.21.15.32:0/4215326974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc84102380 0x7fbc84110180 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 >> 172.21.15.32:0/4215326974 conn(0x7fbc840f7430 msgr2=0x7fbc840f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 shutdown_connections 2024-04-06T14:19:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.293+0000 7fbc8b231700 1 -- 172.21.15.32:0/4215326974 wait complete. 2024-04-06T14:19:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.398+0000 7f7c24f30700 1 -- 172.21.15.32:0/60832403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 msgr2=0x7f7c200fd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.398+0000 7f7c24f30700 1 --2- 172.21.15.32:0/60832403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 0x7f7c200fd260 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7c10004940 tx=0x7f7c10009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.398+0000 7f7c24f30700 1 -- 172.21.15.32:0/60832403 shutdown_connections 2024-04-06T14:19:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.398+0000 7f7c24f30700 1 --2- 172.21.15.32:0/60832403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200fe050 0x7f7c200fe4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.398+0000 7f7c24f30700 1 --2- 172.21.15.32:0/60832403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 0x7f7c200fd260 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.398+0000 7f7c24f30700 1 -- 172.21.15.32:0/60832403 >> 172.21.15.32:0/60832403 conn(0x7f7c200f8580 msgr2=0x7f7c200fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.398+0000 7f7c24f30700 1 -- 172.21.15.32:0/60832403 shutdown_connections 2024-04-06T14:19:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.398+0000 7f7c24f30700 1 -- 172.21.15.32:0/60832403 wait complete. 2024-04-06T14:19:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.398+0000 7f7c24f30700 1 Processor -- start 2024-04-06T14:19:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c24f30700 1 -- start start 2024-04-06T14:19:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c24f30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 0x7f7c2010d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c24f30700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200fe050 0x7f7c2010ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c24f30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2010e3c0 con 0x7f7c200fce50 2024-04-06T14:19:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c24f30700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2010e530 con 0x7f7c200fe050 2024-04-06T14:19:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c1e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 0x7f7c2010d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c1e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 0x7f7c2010d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39290/0 (socket says 172.21.15.32:39290) 2024-04-06T14:19:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c1e59c700 1 -- 172.21.15.32:0/3630721603 learned_addr learned my addr 172.21.15.32:0/3630721603 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:19:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c1dd9b700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200fe050 0x7f7c2010ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c1e59c700 1 -- 172.21.15.32:0/3630721603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200fe050 msgr2=0x7f7c2010ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c1e59c700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200fe050 0x7f7c2010ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c1e59c700 1 -- 172.21.15.32:0/3630721603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c10005040 con 0x7f7c200fce50 2024-04-06T14:19:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c1dd9b700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200fe050 0x7f7c2010ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c1e59c700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 0x7f7c2010d880 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000b3b0 tx=0x7f7c10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.399+0000 7f7c0effd700 1 -- 172.21.15.32:0/3630721603 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c1000e040 con 0x7f7c200fce50 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.400+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c20112f30 con 0x7f7c200fce50 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.400+0000 7f7c0effd700 1 -- 172.21.15.32:0/3630721603 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c1000bd20 con 0x7f7c200fce50 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.400+0000 7f7c0effd700 1 -- 172.21.15.32:0/3630721603 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c1001c630 con 0x7f7c200fce50 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.400+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c20113460 con 0x7f7c200fce50 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.400+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2004c710 con 0x7f7c200fce50 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.401+0000 7f7c0effd700 1 -- 172.21.15.32:0/3630721603 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f7c10013960 con 0x7f7c200fce50 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.401+0000 7f7c0effd700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7c0806c9d0 0x7f7c0806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.402+0000 7f7c0effd700 1 -- 172.21.15.32:0/3630721603 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f7c1008ca30 con 0x7f7c200fce50 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.402+0000 7f7c1dd9b700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7c0806c9d0 0x7f7c0806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.402+0000 7f7c1dd9b700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7c0806c9d0 0x7f7c0806ee80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7c14009fd0 tx=0x7f7c14009ee0 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.406+0000 7f7c0effd700 1 -- 172.21.15.32:0/3630721603 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f7c1005a800 con 0x7f7c200fce50 2024-04-06T14:19:28.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.718+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7c20113680 con 0x7f7c200fce50 2024-04-06T14:19:28.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.718+0000 7f7c0effd700 1 -- 172.21.15.32:0/3630721603 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f7c1005a390 con 0x7f7c200fce50 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:19:28.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:19:28.720 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:19:28.720 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 12, 2024-04-06T14:19:28.720 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:19:28.720 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:19:28.720 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.720+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7c0806c9d0 msgr2=0x7f7c0806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.720+0000 7f7c24f30700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7c0806c9d0 0x7f7c0806ee80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7c14009fd0 tx=0x7f7c14009ee0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.720+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 msgr2=0x7f7c2010d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.720+0000 7f7c24f30700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 0x7f7c2010d880 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000b3b0 tx=0x7f7c10009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.720+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 shutdown_connections 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.720+0000 7f7c24f30700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7c0806c9d0 0x7f7c0806ee80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.720+0000 7f7c24f30700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200fce50 0x7f7c2010d880 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.720+0000 7f7c24f30700 1 --2- 172.21.15.32:0/3630721603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200fe050 0x7f7c2010ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.720+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 >> 172.21.15.32:0/3630721603 conn(0x7f7c200f8580 msgr2=0x7f7c200f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.721+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 shutdown_connections 2024-04-06T14:19:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.721+0000 7f7c24f30700 1 -- 172.21.15.32:0/3630721603 wait complete. 2024-04-06T14:19:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:28 smithi032 ceph-mon[39730]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:19:28.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.843+0000 7f2a214dc700 1 -- 172.21.15.32:0/3557822940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 msgr2=0x7f2a1c101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.843+0000 7f2a214dc700 1 --2- 172.21.15.32:0/3557822940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 0x7f2a1c101e40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c004940 tx=0x7f2a0c009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:28.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.843+0000 7f2a214dc700 1 -- 172.21.15.32:0/3557822940 shutdown_connections 2024-04-06T14:19:28.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.843+0000 7f2a214dc700 1 --2- 172.21.15.32:0/3557822940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a1c102380 0x7f2a1c104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.843+0000 7f2a214dc700 1 --2- 172.21.15.32:0/3557822940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 0x7f2a1c101e40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.843+0000 7f2a214dc700 1 -- 172.21.15.32:0/3557822940 >> 172.21.15.32:0/3557822940 conn(0x7f2a1c0f7430 msgr2=0x7f2a1c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:28.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.844+0000 7f2a214dc700 1 -- 172.21.15.32:0/3557822940 shutdown_connections 2024-04-06T14:19:28.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.844+0000 7f2a214dc700 1 -- 172.21.15.32:0/3557822940 wait complete. 2024-04-06T14:19:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.844+0000 7f2a214dc700 1 Processor -- start 2024-04-06T14:19:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.844+0000 7f2a214dc700 1 -- start start 2024-04-06T14:19:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a214dc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 0x7f2a1c10fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a214dc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a1c102380 0x7f2a1c110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a214dc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c110680 con 0x7f2a1c066260 2024-04-06T14:19:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a214dc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c110ed0 con 0x7f2a1c102380 2024-04-06T14:19:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a1affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 0x7f2a1c10fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a1affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 0x7f2a1c10fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39310/0 (socket says 172.21.15.32:39310) 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a1affd700 1 -- 172.21.15.32:0/1245372643 learned_addr learned my addr 172.21.15.32:0/1245372643 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a1a7fc700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a1c102380 0x7f2a1c110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a1affd700 1 -- 172.21.15.32:0/1245372643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a1c102380 msgr2=0x7f2a1c110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a1affd700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a1c102380 0x7f2a1c110080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a1affd700 1 -- 172.21.15.32:0/1245372643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c005040 con 0x7f2a1c066260 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a1affd700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 0x7f2a1c10fb40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009380 tx=0x7f2a0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.845+0000 7f2a037fe700 1 -- 172.21.15.32:0/1245372643 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a0c00e040 con 0x7f2a1c066260 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.846+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c111150 con 0x7f2a1c066260 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.846+0000 7f2a037fe700 1 -- 172.21.15.32:0/1245372643 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a0c013420 con 0x7f2a1c066260 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.846+0000 7f2a037fe700 1 -- 172.21.15.32:0/1245372643 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a0c01c630 con 0x7f2a1c066260 2024-04-06T14:19:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.846+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c111690 con 0x7f2a1c066260 2024-04-06T14:19:28.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.847+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a080052e0 con 0x7f2a1c066260 2024-04-06T14:19:28.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.847+0000 7f2a037fe700 1 -- 172.21.15.32:0/1245372643 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f2a0c013900 con 0x7f2a1c066260 2024-04-06T14:19:28.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.848+0000 7f2a037fe700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f2a0406c9c0 0x7f2a0406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:28.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.848+0000 7f2a037fe700 1 -- 172.21.15.32:0/1245372643 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f2a0c08ce70 con 0x7f2a1c066260 2024-04-06T14:19:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.848+0000 7f2a1a7fc700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f2a0406c9c0 0x7f2a0406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:28.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.848+0000 7f2a1a7fc700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f2a0406c9c0 0x7f2a0406ee70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009fd0 tx=0x7f2a1000b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:28.852+0000 7f2a037fe700 1 -- 172.21.15.32:0/1245372643 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f2a0c05acf0 con 0x7f2a1c066260 2024-04-06T14:19:28.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:28 smithi086 ceph-mon[47925]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:19:29.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.079+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a08000bf0 con 0x7f2a0406c9c0 2024-04-06T14:19:29.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.080+0000 7f2a037fe700 1 -- 172.21.15.32:0/1245372643 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f2a08000bf0 con 0x7f2a0406c9c0 2024-04-06T14:19:29.080 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/2 daemons upgraded", 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:19:29.081 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.082+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f2a0406c9c0 msgr2=0x7f2a0406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.082+0000 7f2a214dc700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f2a0406c9c0 0x7f2a0406ee70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009fd0 tx=0x7f2a1000b040 comp rx=0 tx=0).stop 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.082+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 msgr2=0x7f2a1c10fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.082+0000 7f2a214dc700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 0x7f2a1c10fb40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009380 tx=0x7f2a0c009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.082+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 shutdown_connections 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.082+0000 7f2a214dc700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f2a0406c9c0 0x7f2a0406ee70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.082+0000 7f2a214dc700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a1c066260 0x7f2a1c10fb40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.082+0000 7f2a214dc700 1 --2- 172.21.15.32:0/1245372643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a1c102380 0x7f2a1c110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.083+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 >> 172.21.15.32:0/1245372643 conn(0x7f2a1c0f7430 msgr2=0x7f2a1c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.083+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 shutdown_connections 2024-04-06T14:19:29.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:29.083+0000 7f2a214dc700 1 -- 172.21.15.32:0/1245372643 wait complete. 2024-04-06T14:19:29.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:29 smithi032 ceph-mon[39730]: pgmap v44: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:29.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:29 smithi032 ceph-mon[39730]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:19:29.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:29 smithi032 ceph-mon[39730]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:19:29.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:29 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3630721603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:29.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:29 smithi086 ceph-mon[47925]: pgmap v44: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:29.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:29 smithi086 ceph-mon[47925]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:19:29.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:29 smithi086 ceph-mon[47925]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:19:29.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:29 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3630721603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:19:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:30 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:19:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:19:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:19:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:19:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:30.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:30 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:31.682 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:31 smithi032 ceph-mon[39730]: pgmap v45: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:31.682 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:31 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:19:31.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:31 smithi086 ceph-mon[47925]: pgmap v45: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:31.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:31 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T14:19:33.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:33 smithi032 ceph-mon[39730]: Upgrade: Updating alertmanager.smithi032 2024-04-06T14:19:33.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:33 smithi032 ceph-mon[39730]: Deploying daemon alertmanager.smithi032 on smithi032 2024-04-06T14:19:33.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:33 smithi032 ceph-mon[39730]: pgmap v46: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:33.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:33 smithi086 ceph-mon[47925]: Upgrade: Updating alertmanager.smithi032 2024-04-06T14:19:33.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:33 smithi086 ceph-mon[47925]: Deploying daemon alertmanager.smithi032 on smithi032 2024-04-06T14:19:33.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:33 smithi086 ceph-mon[47925]: pgmap v46: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:35.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:35 smithi032 ceph-mon[39730]: pgmap v47: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:35.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:35 smithi086 ceph-mon[47925]: pgmap v47: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:36.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:36 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:19:36.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:36 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:19:37.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:37 smithi032 ceph-mon[39730]: pgmap v48: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:37.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:37 smithi086 ceph-mon[47925]: pgmap v48: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:38.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:38 smithi032 ceph-mon[39730]: mgrmap e33: smithi032.peohxu(active, since 91s), standbys: smithi086.fyyhxm 2024-04-06T14:19:38.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:38 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:38.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:38 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:38.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:38 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:38.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:38 smithi086 ceph-mon[47925]: mgrmap e33: smithi032.peohxu(active, since 91s), standbys: smithi086.fyyhxm 2024-04-06T14:19:38.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:38 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:38.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:38 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:38.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:38 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:39.669 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:39 smithi032 ceph-mon[39730]: pgmap v49: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:39.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:39 smithi086 ceph-mon[47925]: pgmap v49: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:41.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:41 smithi032 ceph-mon[39730]: pgmap v50: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:41.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:41 smithi086 ceph-mon[47925]: pgmap v50: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:43 smithi032 ceph-mon[39730]: pgmap v51: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:43 smithi086 ceph-mon[47925]: pgmap v51: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:44.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:44 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:44.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:44 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:44.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:44 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:44.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:44 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:44.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:44 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:44.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:44 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:44.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:44 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:44.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:44 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:45.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:45 smithi032 ceph-mon[39730]: pgmap v52: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:45.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:45 smithi086 ceph-mon[47925]: pgmap v52: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:47.672 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:47 smithi032 ceph-mon[39730]: pgmap v53: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:47.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:47 smithi086 ceph-mon[47925]: pgmap v53: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: pgmap v54: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:19:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:19:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:19:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:49 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: pgmap v54: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:19:49.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:49.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:49.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:19:49.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:19:49.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:49.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:19:49.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:49 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:19:51.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:51 smithi086 ceph-mon[47925]: pgmap v55: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:51.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:51 smithi086 ceph-mon[47925]: Upgrade: Updating grafana.smithi032 2024-04-06T14:19:51.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:51 smithi086 ceph-mon[47925]: Deploying daemon grafana.smithi032 on smithi032 2024-04-06T14:19:51.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:51 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:19:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:51 smithi032 ceph-mon[39730]: pgmap v55: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:51 smithi032 ceph-mon[39730]: Upgrade: Updating grafana.smithi032 2024-04-06T14:19:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:51 smithi032 ceph-mon[39730]: Deploying daemon grafana.smithi032 on smithi032 2024-04-06T14:19:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:51 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:19:53.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:53 smithi086 ceph-mon[47925]: pgmap v56: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:53.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:53 smithi032 ceph-mon[39730]: pgmap v56: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:55.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:55 smithi086 ceph-mon[47925]: pgmap v57: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:55.427 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:55 smithi032 ceph-mon[39730]: pgmap v57: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:56.551 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:56 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:56.552 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:56 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:56.552 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:56 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:56.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:56 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:56.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:56 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:19:56.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:56 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:19:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:57 smithi032 ceph-mon[39730]: pgmap v58: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:57.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:57 smithi086 ceph-mon[47925]: pgmap v58: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:59.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.199+0000 7f4b1de1e700 1 -- 172.21.15.32:0/551141326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 msgr2=0x7f4b180fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.199+0000 7f4b1de1e700 1 --2- 172.21.15.32:0/551141326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 0x7f4b180fd3f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4b08004940 tx=0x7f4b08009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.199+0000 7f4b1de1e700 1 -- 172.21.15.32:0/551141326 shutdown_connections 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.199+0000 7f4b1de1e700 1 --2- 172.21.15.32:0/551141326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 0x7f4b180fd3f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.199+0000 7f4b1de1e700 1 --2- 172.21.15.32:0/551141326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b180fbcd0 0x7f4b180fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.199+0000 7f4b1de1e700 1 -- 172.21.15.32:0/551141326 >> 172.21.15.32:0/551141326 conn(0x7f4b180f7430 msgr2=0x7f4b180f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.199+0000 7f4b1de1e700 1 -- 172.21.15.32:0/551141326 shutdown_connections 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.199+0000 7f4b1de1e700 1 -- 172.21.15.32:0/551141326 wait complete. 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b1de1e700 1 Processor -- start 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b1de1e700 1 -- start start 2024-04-06T14:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b1de1e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b180fbcd0 0x7f4b1810c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b1de1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 0x7f4b1810cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b1de1e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1810d390 con 0x7f4b180fcf80 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b1de1e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1810dbe0 con 0x7f4b180fbcd0 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b16ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 0x7f4b1810cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b16ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 0x7f4b1810cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58134/0 (socket says 172.21.15.32:58134) 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b16ffd700 1 -- 172.21.15.32:0/3900388959 learned_addr learned my addr 172.21.15.32:0/3900388959 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.200+0000 7f4b177fe700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b180fbcd0 0x7f4b1810c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.201+0000 7f4b16ffd700 1 -- 172.21.15.32:0/3900388959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b180fbcd0 msgr2=0x7f4b1810c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.201+0000 7f4b16ffd700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b180fbcd0 0x7f4b1810c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.201+0000 7f4b16ffd700 1 -- 172.21.15.32:0/3900388959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b08005040 con 0x7f4b180fcf80 2024-04-06T14:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.201+0000 7f4b16ffd700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 0x7f4b1810cd90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800b3b0 tx=0x7f4b08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:59.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.201+0000 7f4afffff700 1 -- 172.21.15.32:0/3900388959 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0800e040 con 0x7f4b180fcf80 2024-04-06T14:19:59.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.201+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1810de60 con 0x7f4b180fcf80 2024-04-06T14:19:59.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.201+0000 7f4afffff700 1 -- 172.21.15.32:0/3900388959 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b08004290 con 0x7f4b180fcf80 2024-04-06T14:19:59.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.201+0000 7f4afffff700 1 -- 172.21.15.32:0/3900388959 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b08013660 con 0x7f4b180fcf80 2024-04-06T14:19:59.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.201+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1810e340 con 0x7f4b180fcf80 2024-04-06T14:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.202+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1805f5f0 con 0x7f4b180fcf80 2024-04-06T14:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.202+0000 7f4afffff700 1 -- 172.21.15.32:0/3900388959 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f4b0801e420 con 0x7f4b180fcf80 2024-04-06T14:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.203+0000 7f4afffff700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b0006c8b0 0x7f4b0006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.203+0000 7f4afffff700 1 -- 172.21.15.32:0/3900388959 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f4b0808fa00 con 0x7f4b180fcf80 2024-04-06T14:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.203+0000 7f4b177fe700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b0006c8b0 0x7f4b0006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.203+0000 7f4b177fe700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b0006c8b0 0x7f4b0006ed60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c004f40 tx=0x7f4b0c00b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:59.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.208+0000 7f4afffff700 1 -- 172.21.15.32:0/3900388959 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f4b0805d970 con 0x7f4b180fcf80 2024-04-06T14:19:59.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.438+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b1805f5f0 con 0x7f4b0006c8b0 2024-04-06T14:19:59.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.439+0000 7f4afffff700 1 -- 172.21.15.32:0/3900388959 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f4b1805f5f0 con 0x7f4b0006c8b0 2024-04-06T14:19:59.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.441+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b0006c8b0 msgr2=0x7f4b0006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.441+0000 7f4b1de1e700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b0006c8b0 0x7f4b0006ed60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c004f40 tx=0x7f4b0c00b040 comp rx=0 tx=0).stop 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.441+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 msgr2=0x7f4b1810cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.441+0000 7f4b1de1e700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 0x7f4b1810cd90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800b3b0 tx=0x7f4b08009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.441+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 shutdown_connections 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.441+0000 7f4b1de1e700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f4b0006c8b0 0x7f4b0006ed60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.441+0000 7f4b1de1e700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b180fbcd0 0x7f4b1810c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.441+0000 7f4b1de1e700 1 --2- 172.21.15.32:0/3900388959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b180fcf80 0x7f4b1810cd90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.441+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 >> 172.21.15.32:0/3900388959 conn(0x7f4b180f7430 msgr2=0x7f4b180f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.442+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 shutdown_connections 2024-04-06T14:19:59.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.442+0000 7f4b1de1e700 1 -- 172.21.15.32:0/3900388959 wait complete. 2024-04-06T14:19:59.452 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:19:59.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.546+0000 7fefd6f71700 1 -- 172.21.15.32:0/3143635687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 msgr2=0x7fefd00fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.546+0000 7fefd6f71700 1 --2- 172.21.15.32:0/3143635687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 0x7fefd00fd3f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fefbc004940 tx=0x7fefbc009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:59.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 -- 172.21.15.32:0/3143635687 shutdown_connections 2024-04-06T14:19:59.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 --2- 172.21.15.32:0/3143635687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 0x7fefd00fd3f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 --2- 172.21.15.32:0/3143635687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefd00fbcd0 0x7fefd00fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 -- 172.21.15.32:0/3143635687 >> 172.21.15.32:0/3143635687 conn(0x7fefd00f7430 msgr2=0x7fefd00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:59.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 -- 172.21.15.32:0/3143635687 shutdown_connections 2024-04-06T14:19:59.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 -- 172.21.15.32:0/3143635687 wait complete. 2024-04-06T14:19:59.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 Processor -- start 2024-04-06T14:19:59.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 -- start start 2024-04-06T14:19:59.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefd00fbcd0 0x7fefd00ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 0x7fefd00fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd00fe850 con 0x7fefd00fcf80 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.547+0000 7fefd6f71700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd00fe990 con 0x7fefd00fbcd0 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 0x7fefd00fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 0x7fefd00fe2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58154/0 (socket says 172.21.15.32:58154) 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcffff700 1 -- 172.21.15.32:0/3925130002 learned_addr learned my addr 172.21.15.32:0/3925130002 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefd4d0d700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefd00fbcd0 0x7fefd00ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcffff700 1 -- 172.21.15.32:0/3925130002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefd00fbcd0 msgr2=0x7fefd00ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcffff700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefd00fbcd0 0x7fefd00ffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcffff700 1 -- 172.21.15.32:0/3925130002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc005040 con 0x7fefd00fcf80 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcffff700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 0x7fefd00fe2c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00b3b0 tx=0x7fefbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcd7fa700 1 -- 172.21.15.32:0/3925130002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefbc00e040 con 0x7fefd00fcf80 2024-04-06T14:19:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcd7fa700 1 -- 172.21.15.32:0/3925130002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefbc00bc50 con 0x7fefd00fcf80 2024-04-06T14:19:59.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd00fec10 con 0x7fefd00fcf80 2024-04-06T14:19:59.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefcd7fa700 1 -- 172.21.15.32:0/3925130002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefbc01c630 con 0x7fefd00fcf80 2024-04-06T14:19:59.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.548+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd01159d0 con 0x7fefd00fcf80 2024-04-06T14:19:59.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.549+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd0101b10 con 0x7fefd00fcf80 2024-04-06T14:19:59.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.550+0000 7fefcd7fa700 1 -- 172.21.15.32:0/3925130002 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fefbc013420 con 0x7fefd00fcf80 2024-04-06T14:19:59.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.550+0000 7fefcd7fa700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fefc006c8a0 0x7fefc006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:59.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.550+0000 7fefcd7fa700 1 -- 172.21.15.32:0/3925130002 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fefbc08c720 con 0x7fefd00fcf80 2024-04-06T14:19:59.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.550+0000 7fefd4d0d700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fefc006c8a0 0x7fefc006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:59.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.551+0000 7fefd4d0d700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fefc006c8a0 0x7fefc006ed50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fefd00ffb30 tx=0x7fefc400b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:59.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.555+0000 7fefcd7fa700 1 -- 172.21.15.32:0/3925130002 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fefbc056be0 con 0x7fefd00fcf80 2024-04-06T14:19:59.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.796+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefd01047f0 con 0x7fefc006c8a0 2024-04-06T14:19:59.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.797+0000 7fefcd7fa700 1 -- 172.21.15.32:0/3925130002 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fefd01047f0 con 0x7fefc006c8a0 2024-04-06T14:19:59.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.799+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fefc006c8a0 msgr2=0x7fefc006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fefc006c8a0 0x7fefc006ed50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fefd00ffb30 tx=0x7fefc400b040 comp rx=0 tx=0).stop 2024-04-06T14:19:59.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 msgr2=0x7fefd00fe2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 0x7fefd00fe2c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00b3b0 tx=0x7fefbc009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:59.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 shutdown_connections 2024-04-06T14:19:59.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fefc006c8a0 0x7fefc006ed50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefd00fbcd0 0x7fefd00ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 --2- 172.21.15.32:0/3925130002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefd00fcf80 0x7fefd00fe2c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 >> 172.21.15.32:0/3925130002 conn(0x7fefd00f7430 msgr2=0x7fefd0065c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:59.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 shutdown_connections 2024-04-06T14:19:59.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.800+0000 7fefd6f71700 1 -- 172.21.15.32:0/3925130002 wait complete. 2024-04-06T14:19:59.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:19:59 smithi032 ceph-mon[39730]: pgmap v59: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:59.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:19:59 smithi086 ceph-mon[47925]: pgmap v59: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:19:59.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.916+0000 7f0e3628a700 1 -- 172.21.15.32:0/2429471447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 msgr2=0x7f0e30102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.916+0000 7f0e3628a700 1 --2- 172.21.15.32:0/2429471447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 0x7f0e30102320 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0e24004940 tx=0x7f0e24009e30 comp rx=0 tx=0).stop 2024-04-06T14:19:59.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.916+0000 7f0e3628a700 1 -- 172.21.15.32:0/2429471447 shutdown_connections 2024-04-06T14:19:59.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.916+0000 7f0e3628a700 1 --2- 172.21.15.32:0/2429471447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 0x7f0e30102320 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.916+0000 7f0e3628a700 1 --2- 172.21.15.32:0/2429471447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e300662e0 0x7f0e300ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.916+0000 7f0e3628a700 1 -- 172.21.15.32:0/2429471447 >> 172.21.15.32:0/2429471447 conn(0x7f0e300f7430 msgr2=0x7f0e300f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.916+0000 7f0e3628a700 1 -- 172.21.15.32:0/2429471447 shutdown_connections 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.916+0000 7f0e3628a700 1 -- 172.21.15.32:0/2429471447 wait complete. 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e3628a700 1 Processor -- start 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e3628a700 1 -- start start 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e3628a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e300662e0 0x7f0e3010d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e3628a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 0x7f0e3010dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e3628a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3010e550 con 0x7f0e300fff00 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e3628a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3010ed40 con 0x7f0e300662e0 2024-04-06T14:19:59.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e2f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 0x7f0e3010dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e2f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 0x7f0e3010dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58162/0 (socket says 172.21.15.32:58162) 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e2f7fe700 1 -- 172.21.15.32:0/300623570 learned_addr learned my addr 172.21.15.32:0/300623570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.917+0000 7f0e2ffff700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e300662e0 0x7f0e3010d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e2f7fe700 1 -- 172.21.15.32:0/300623570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e300662e0 msgr2=0x7f0e3010d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e2f7fe700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e300662e0 0x7f0e3010d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e2f7fe700 1 -- 172.21.15.32:0/300623570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e24005040 con 0x7f0e300fff00 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e2ffff700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e300662e0 0x7f0e3010d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e2f7fe700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 0x7f0e3010dec0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400b3b0 tx=0x7f0e24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e2cff9700 1 -- 172.21.15.32:0/300623570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2400e040 con 0x7f0e300fff00 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e2cff9700 1 -- 172.21.15.32:0/300623570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e24004290 con 0x7f0e300fff00 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3010efc0 con 0x7f0e300fff00 2024-04-06T14:19:59.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e2cff9700 1 -- 172.21.15.32:0/300623570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2401c3f0 con 0x7f0e300fff00 2024-04-06T14:19:59.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.918+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3010f580 con 0x7f0e300fff00 2024-04-06T14:19:59.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.919+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3005f5f0 con 0x7f0e300fff00 2024-04-06T14:19:59.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.919+0000 7f0e2cff9700 1 -- 172.21.15.32:0/300623570 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f0e24013420 con 0x7f0e300fff00 2024-04-06T14:19:59.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.920+0000 7f0e2cff9700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f0e1806c8b0 0x7f0e1806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:19:59.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.920+0000 7f0e2cff9700 1 -- 172.21.15.32:0/300623570 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0e2408cd20 con 0x7f0e300fff00 2024-04-06T14:19:59.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.920+0000 7f0e2ffff700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f0e1806c8b0 0x7f0e1806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:19:59.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.920+0000 7f0e2ffff700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f0e1806c8b0 0x7f0e1806ed60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0e20006fd0 tx=0x7f0e20008040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:19:59.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:19:59.924+0000 7f0e2cff9700 1 -- 172.21.15.32:0/300623570 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0e2405adb0 con 0x7f0e300fff00 2024-04-06T14:20:00.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.144+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e3010f860 con 0x7f0e1806c8b0 2024-04-06T14:20:00.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.159+0000 7f0e2cff9700 1 -- 172.21.15.32:0/300623570 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0e3010f860 con 0x7f0e1806c8b0 2024-04-06T14:20:00.159 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:20:00.159 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (23s) 16s ago 11m 15.4M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 16s ago 11m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (10m) 55s ago 10m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 starting - - - - 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (5m) 16s ago 5m 12.9M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (5m) 16s ago 5m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (5m) 55s ago 5m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (5m) 55s ago 5m 15.5M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (2m) 16s ago 13m 498M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (75s) 55s ago 10m 449M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (13m) 16s ago 13m 54.2M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (10m) 55s ago 10m 43.8M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (62s) 16s ago 11m 8778k - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (58s) 55s ago 10m 3397k - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:20:00.160 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 16s ago 9m 195M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:20:00.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (9m) 16s ago 9m 191M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:20:00.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (8m) 16s ago 8m 161M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:20:00.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (8m) 55s ago 8m 224M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:20:00.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (7m) 55s ago 7m 202M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:20:00.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (7m) 55s ago 7m 183M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:20:00.161 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (40s) 16s ago 11m 42.1M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:20:00.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f0e1806c8b0 msgr2=0x7f0e1806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f0e1806c8b0 0x7f0e1806ed60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0e20006fd0 tx=0x7f0e20008040 comp rx=0 tx=0).stop 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 msgr2=0x7f0e3010dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 0x7f0e3010dec0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400b3b0 tx=0x7f0e24009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 shutdown_connections 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f0e1806c8b0 0x7f0e1806ed60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e300662e0 0x7f0e3010d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 --2- 172.21.15.32:0/300623570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e300fff00 0x7f0e3010dec0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 >> 172.21.15.32:0/300623570 conn(0x7f0e300f7430 msgr2=0x7f0e300f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 shutdown_connections 2024-04-06T14:20:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.161+0000 7f0e3628a700 1 -- 172.21.15.32:0/300623570 wait complete. 2024-04-06T14:20:00.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.268+0000 7ff10d9df700 1 -- 172.21.15.32:0/354065638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fed90 msgr2=0x7ff1080faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.268+0000 7ff10d9df700 1 --2- 172.21.15.32:0/354065638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fed90 0x7ff1080faed0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc004940 tx=0x7ff0fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.268+0000 7ff10d9df700 1 -- 172.21.15.32:0/354065638 shutdown_connections 2024-04-06T14:20:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.268+0000 7ff10d9df700 1 --2- 172.21.15.32:0/354065638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fed90 0x7ff1080faed0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.268+0000 7ff10d9df700 1 --2- 172.21.15.32:0/354065638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1080fe440 0x7ff1080fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.268+0000 7ff10d9df700 1 -- 172.21.15.32:0/354065638 >> 172.21.15.32:0/354065638 conn(0x7ff1080f6be0 msgr2=0x7ff1080f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.268+0000 7ff10d9df700 1 -- 172.21.15.32:0/354065638 shutdown_connections 2024-04-06T14:20:00.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.268+0000 7ff10d9df700 1 -- 172.21.15.32:0/354065638 wait complete. 2024-04-06T14:20:00.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff10d9df700 1 Processor -- start 2024-04-06T14:20:00.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff10d9df700 1 -- start start 2024-04-06T14:20:00.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff10d9df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fe440 0x7ff10810d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:00.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff10d9df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1080fed90 0x7ff10810de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:00.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff10d9df700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10810e400 con 0x7ff1080fe440 2024-04-06T14:20:00.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff10d9df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10810e570 con 0x7ff1080fed90 2024-04-06T14:20:00.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff106ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fe440 0x7ff10810d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:00.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff106ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fe440 0x7ff10810d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58180/0 (socket says 172.21.15.32:58180) 2024-04-06T14:20:00.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff106ffd700 1 -- 172.21.15.32:0/4178032670 learned_addr learned my addr 172.21.15.32:0/4178032670 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.269+0000 7ff1067fc700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1080fed90 0x7ff10810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.270+0000 7ff106ffd700 1 -- 172.21.15.32:0/4178032670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1080fed90 msgr2=0x7ff10810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.270+0000 7ff106ffd700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1080fed90 0x7ff10810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.270+0000 7ff106ffd700 1 -- 172.21.15.32:0/4178032670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc005040 con 0x7ff1080fe440 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.270+0000 7ff106ffd700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fe440 0x7ff10810d8c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8002a50 tx=0x7ff0f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.270+0000 7ff0ef7fe700 1 -- 172.21.15.32:0/4178032670 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f8007cc0 con 0x7ff1080fe440 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.270+0000 7ff0ef7fe700 1 -- 172.21.15.32:0/4178032670 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0f800f040 con 0x7ff1080fe440 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.270+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff108112f80 con 0x7ff1080fe440 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.270+0000 7ff0ef7fe700 1 -- 172.21.15.32:0/4178032670 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f800d9c0 con 0x7ff1080fe440 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.270+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1081134b0 con 0x7ff1080fe440 2024-04-06T14:20:00.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.271+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10804c710 con 0x7ff1080fe440 2024-04-06T14:20:00.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.272+0000 7ff0ef7fe700 1 -- 172.21.15.32:0/4178032670 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7ff0f800db60 con 0x7ff1080fe440 2024-04-06T14:20:00.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.272+0000 7ff0ef7fe700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff0f006c8a0 0x7ff0f006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:00.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.272+0000 7ff0ef7fe700 1 -- 172.21.15.32:0/4178032670 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff0f808b0e0 con 0x7ff1080fe440 2024-04-06T14:20:00.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.272+0000 7ff1067fc700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff0f006c8a0 0x7ff0f006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:00.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.273+0000 7ff1067fc700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff0f006c8a0 0x7ff0f006ed50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00b3b0 tx=0x7ff0fc009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:00.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.277+0000 7ff0ef7fe700 1 -- 172.21.15.32:0/4178032670 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff0f8059050 con 0x7ff1080fe440 2024-04-06T14:20:00.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.574+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff108113760 con 0x7ff1080fe440 2024-04-06T14:20:00.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.575+0000 7ff0ef7fe700 1 -- 172.21.15.32:0/4178032670 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7ff0f8058be0 con 0x7ff1080fe440 2024-04-06T14:20:00.575 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:20:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:20:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:20:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 12, 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:20:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:20:00.577 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:20:00.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff0f006c8a0 msgr2=0x7ff0f006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff0f006c8a0 0x7ff0f006ed50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00b3b0 tx=0x7ff0fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fe440 msgr2=0x7ff10810d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fe440 0x7ff10810d8c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8002a50 tx=0x7ff0f80046b0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 shutdown_connections 2024-04-06T14:20:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff0f006c8a0 0x7ff0f006ed50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080fe440 0x7ff10810d8c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 --2- 172.21.15.32:0/4178032670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1080fed90 0x7ff10810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 >> 172.21.15.32:0/4178032670 conn(0x7ff1080f6be0 msgr2=0x7ff1080f8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 shutdown_connections 2024-04-06T14:20:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.577+0000 7ff10d9df700 1 -- 172.21.15.32:0/4178032670 wait complete. 2024-04-06T14:20:00.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 -- 172.21.15.32:0/984987717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fe040 msgr2=0x7fae640fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 --2- 172.21.15.32:0/984987717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fe040 0x7fae640fe4b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fae58004940 tx=0x7fae58009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 -- 172.21.15.32:0/984987717 shutdown_connections 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 --2- 172.21.15.32:0/984987717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fe040 0x7fae640fe4b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 --2- 172.21.15.32:0/984987717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640fce40 0x7fae640fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 -- 172.21.15.32:0/984987717 >> 172.21.15.32:0/984987717 conn(0x7fae640f8570 msgr2=0x7fae640fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 -- 172.21.15.32:0/984987717 shutdown_connections 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 -- 172.21.15.32:0/984987717 wait complete. 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 Processor -- start 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.682+0000 7fae6a795700 1 -- start start 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae6a795700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fce40 0x7fae6410d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae6a795700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640fe040 0x7fae6410de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae6a795700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6410e520 con 0x7fae640fce40 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae6a795700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6410ed10 con 0x7fae640fe040 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae63fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fce40 0x7fae6410d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae63fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fce40 0x7fae6410d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58206/0 (socket says 172.21.15.32:58206) 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae63fff700 1 -- 172.21.15.32:0/942887108 learned_addr learned my addr 172.21.15.32:0/942887108 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae637fe700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640fe040 0x7fae6410de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae63fff700 1 -- 172.21.15.32:0/942887108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640fe040 msgr2=0x7fae6410de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae63fff700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640fe040 0x7fae6410de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae63fff700 1 -- 172.21.15.32:0/942887108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae58005040 con 0x7fae640fce40 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae63fff700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fce40 0x7fae6410d950 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fae5400b790 tx=0x7fae54009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae60ff9700 1 -- 172.21.15.32:0/942887108 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae5400f6f0 con 0x7fae640fce40 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae60ff9700 1 -- 172.21.15.32:0/942887108 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae5400fd00 con 0x7fae640fce40 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6410eff0 con 0x7fae640fce40 2024-04-06T14:20:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae60ff9700 1 -- 172.21.15.32:0/942887108 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae540103a0 con 0x7fae640fce40 2024-04-06T14:20:00.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.683+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6410f5b0 con 0x7fae640fce40 2024-04-06T14:20:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.684+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6405f5f0 con 0x7fae640fce40 2024-04-06T14:20:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.685+0000 7fae60ff9700 1 -- 172.21.15.32:0/942887108 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fae5400f850 con 0x7fae640fce40 2024-04-06T14:20:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.685+0000 7fae60ff9700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fae4c06c8a0 0x7fae4c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.685+0000 7fae60ff9700 1 -- 172.21.15.32:0/942887108 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fae5408b860 con 0x7fae640fce40 2024-04-06T14:20:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.685+0000 7fae637fe700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fae4c06c8a0 0x7fae4c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.686+0000 7fae637fe700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fae4c06c8a0 0x7fae4c06ed50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fae58000c00 tx=0x7fae58009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:00.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.690+0000 7fae60ff9700 1 -- 172.21.15.32:0/942887108 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fae54059850 con 0x7fae640fce40 2024-04-06T14:20:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:00 smithi032 ceph-mon[39730]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:00 smithi032 ceph-mon[39730]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:20:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:00 smithi032 ceph-mon[39730]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:20:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:00 smithi032 ceph-mon[39730]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:20:00.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:00 smithi086 ceph-mon[47925]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:00.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:00 smithi086 ceph-mon[47925]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:20:00.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:00 smithi086 ceph-mon[47925]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:20:00.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:00 smithi086 ceph-mon[47925]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:20:00.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.911+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae6410f920 con 0x7fae4c06c8a0 2024-04-06T14:20:00.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.913+0000 7fae60ff9700 1 -- 172.21.15.32:0/942887108 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fae6410f920 con 0x7fae4c06c8a0 2024-04-06T14:20:00.913 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:20:00.913 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:20:00.913 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:20:00.913 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T14:20:00.913 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:20:00.913 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-06T14:20:00.913 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:20:00.914 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/2 daemons upgraded", 2024-04-06T14:20:00.914 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading grafana daemons", 2024-04-06T14:20:00.914 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:20:00.914 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:20:00.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fae4c06c8a0 msgr2=0x7fae4c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fae4c06c8a0 0x7fae4c06ed50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fae58000c00 tx=0x7fae58009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:00.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fce40 msgr2=0x7fae6410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:00.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fce40 0x7fae6410d950 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fae5400b790 tx=0x7fae54009220 comp rx=0 tx=0).stop 2024-04-06T14:20:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 shutdown_connections 2024-04-06T14:20:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fae4c06c8a0 0x7fae4c06ed50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640fce40 0x7fae6410d950 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 --2- 172.21.15.32:0/942887108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640fe040 0x7fae6410de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 >> 172.21.15.32:0/942887108 conn(0x7fae640f8570 msgr2=0x7fae640f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 shutdown_connections 2024-04-06T14:20:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:00.915+0000 7fae6a795700 1 -- 172.21.15.32:0/942887108 wait complete. 2024-04-06T14:20:01.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:01 smithi086 ceph-mon[47925]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:01.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:01 smithi086 ceph-mon[47925]: pgmap v60: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:01.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:01 smithi086 ceph-mon[47925]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:01.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:01 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/4178032670' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:01.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:01 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:01.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:01 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:01.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:01 smithi032 ceph-mon[39730]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:01.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:01 smithi032 ceph-mon[39730]: pgmap v60: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:01.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:01 smithi032 ceph-mon[39730]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:01.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:01 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/4178032670' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:01.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:01 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:01.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:01 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:02.658 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:02 smithi032 ceph-mon[39730]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:02.658 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:02 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:02.658 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:02 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:02.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:02 smithi086 ceph-mon[47925]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:02.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:02 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:02.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:02 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:03.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:03 smithi032 ceph-mon[39730]: pgmap v61: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:03.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:03 smithi086 ceph-mon[47925]: pgmap v61: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:05.624 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:05 smithi032 ceph-mon[39730]: pgmap v62: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:05.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:05 smithi086 ceph-mon[47925]: pgmap v62: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:06.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: pgmap v63: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: Upgrade: Finalizing container_image settings 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-06T14:20:06.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-06T14:20:06.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: Upgrade: Complete! 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-06T14:20:06.984 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:20:06.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:20:06.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:06.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:20:06.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:20:06.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:06.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:20:06.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:20:06.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:20:06.985 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:07.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:07.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:07.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:20:07.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:20:07.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:07.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:20:07.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T14:20:07.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: pgmap v63: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:07.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: Upgrade: Finalizing container_image settings 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-06T14:20:07.130 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-06T14:20:07.131 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: Upgrade: Complete! 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:20:07.132 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:09.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:08 smithi032 ceph-mon[39730]: pgmap v64: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:09.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:08 smithi086 ceph-mon[47925]: pgmap v64: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:11.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:10 smithi032 ceph-mon[39730]: pgmap v65: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:11.271 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:10 smithi086 ceph-mon[47925]: pgmap v65: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:12 smithi032 ceph-mon[39730]: pgmap v66: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:13.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:12 smithi086 ceph-mon[47925]: pgmap v66: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:14 smithi032 ceph-mon[39730]: pgmap v67: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:15.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:14 smithi086 ceph-mon[47925]: pgmap v67: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:16 smithi032 ceph-mon[39730]: pgmap v68: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:17.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:16 smithi086 ceph-mon[47925]: pgmap v68: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:19.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:18 smithi032 ceph-mon[39730]: pgmap v69: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:19.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:18 smithi086 ceph-mon[47925]: pgmap v69: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:20 smithi032 ceph-mon[39730]: pgmap v70: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:21.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:20 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:20:21.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:20 smithi086 ceph-mon[47925]: pgmap v70: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:20 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:20:23.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:22 smithi032 ceph-mon[39730]: pgmap v71: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:23.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:22 smithi086 ceph-mon[47925]: pgmap v71: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:25.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:24 smithi032 ceph-mon[39730]: pgmap v72: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:25.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:24 smithi086 ceph-mon[47925]: pgmap v72: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:26 smithi032 ceph-mon[39730]: pgmap v73: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:27.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:26 smithi086 ceph-mon[47925]: pgmap v73: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:29.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:28 smithi032 ceph-mon[39730]: pgmap v74: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:29.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:28 smithi086 ceph-mon[47925]: pgmap v74: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.034+0000 7f5a38aed700 1 -- 172.21.15.32:0/1272160255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 msgr2=0x7f5a340ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.034+0000 7f5a38aed700 1 --2- 172.21.15.32:0/1272160255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 0x7f5a340ffa40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5a24004940 tx=0x7f5a24009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.034+0000 7f5a38aed700 1 -- 172.21.15.32:0/1272160255 shutdown_connections 2024-04-06T14:20:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.034+0000 7f5a38aed700 1 --2- 172.21.15.32:0/1272160255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a340fff80 0x7f5a34102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.034+0000 7f5a38aed700 1 --2- 172.21.15.32:0/1272160255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 0x7f5a340ffa40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.034+0000 7f5a38aed700 1 -- 172.21.15.32:0/1272160255 >> 172.21.15.32:0/1272160255 conn(0x7f5a340f7430 msgr2=0x7f5a340f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.034+0000 7f5a38aed700 1 -- 172.21.15.32:0/1272160255 shutdown_connections 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.034+0000 7f5a38aed700 1 -- 172.21.15.32:0/1272160255 wait complete. 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.034+0000 7f5a38aed700 1 Processor -- start 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a38aed700 1 -- start start 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a38aed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 0x7f5a3410d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a38aed700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a340fff80 0x7f5a3410df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a38aed700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3410e530 con 0x7f5a34066260 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a38aed700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3410ed30 con 0x7f5a340fff80 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a3259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 0x7f5a3410d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a3259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 0x7f5a3410d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48964/0 (socket says 172.21.15.32:48964) 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a3259c700 1 -- 172.21.15.32:0/3976667240 learned_addr learned my addr 172.21.15.32:0/3976667240 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a31d9b700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a340fff80 0x7f5a3410df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:31.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a3259c700 1 -- 172.21.15.32:0/3976667240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a340fff80 msgr2=0x7f5a3410df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a3259c700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a340fff80 0x7f5a3410df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a3259c700 1 -- 172.21.15.32:0/3976667240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a24005040 con 0x7f5a34066260 2024-04-06T14:20:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.035+0000 7f5a3259c700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 0x7f5a3410d9f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400b3b0 tx=0x7f5a24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.036+0000 7f5a22ffd700 1 -- 172.21.15.32:0/3976667240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a2400e040 con 0x7f5a34066260 2024-04-06T14:20:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.036+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3410efb0 con 0x7f5a34066260 2024-04-06T14:20:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.036+0000 7f5a22ffd700 1 -- 172.21.15.32:0/3976667240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a2400bd20 con 0x7f5a34066260 2024-04-06T14:20:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.036+0000 7f5a22ffd700 1 -- 172.21.15.32:0/3976667240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a2401c630 con 0x7f5a34066260 2024-04-06T14:20:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.036+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3410f4f0 con 0x7f5a34066260 2024-04-06T14:20:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.037+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3405f5f0 con 0x7f5a34066260 2024-04-06T14:20:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.037+0000 7f5a22ffd700 1 -- 172.21.15.32:0/3976667240 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f5a24013960 con 0x7f5a34066260 2024-04-06T14:20:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.038+0000 7f5a22ffd700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5a1c06c8a0 0x7f5a1c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.038+0000 7f5a22ffd700 1 -- 172.21.15.32:0/3976667240 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f5a2408c750 con 0x7f5a34066260 2024-04-06T14:20:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.038+0000 7f5a31d9b700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5a1c06c8a0 0x7f5a1c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.038+0000 7f5a31d9b700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5a1c06c8a0 0x7f5a1c06ed50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a28004d60 tx=0x7f5a2800b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.042+0000 7f5a22ffd700 1 -- 172.21.15.32:0/3976667240 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f5a2405a740 con 0x7f5a34066260 2024-04-06T14:20:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:30 smithi032 ceph-mon[39730]: pgmap v75: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:31.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.262+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a3410f820 con 0x7f5a1c06c8a0 2024-04-06T14:20:31.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.263+0000 7f5a22ffd700 1 -- 172.21.15.32:0/3976667240 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f5a3410f820 con 0x7f5a1c06c8a0 2024-04-06T14:20:31.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.265+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5a1c06c8a0 msgr2=0x7f5a1c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:31.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.265+0000 7f5a38aed700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5a1c06c8a0 0x7f5a1c06ed50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a28004d60 tx=0x7f5a2800b040 comp rx=0 tx=0).stop 2024-04-06T14:20:31.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.265+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 msgr2=0x7f5a3410d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:31.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.265+0000 7f5a38aed700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 0x7f5a3410d9f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400b3b0 tx=0x7f5a24009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:31.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.265+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 shutdown_connections 2024-04-06T14:20:31.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.266+0000 7f5a38aed700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f5a1c06c8a0 0x7f5a1c06ed50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:31.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.266+0000 7f5a38aed700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a34066260 0x7f5a3410d9f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:31.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.266+0000 7f5a38aed700 1 --2- 172.21.15.32:0/3976667240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a340fff80 0x7f5a3410df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:31.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.266+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 >> 172.21.15.32:0/3976667240 conn(0x7f5a340f7430 msgr2=0x7f5a340f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:31.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.266+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 shutdown_connections 2024-04-06T14:20:31.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:31.266+0000 7f5a38aed700 1 -- 172.21.15.32:0/3976667240 wait complete. 2024-04-06T14:20:31.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:30 smithi086 ceph-mon[47925]: pgmap v75: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:31.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-06T14:20:32.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:31 smithi032 ceph-mon[39730]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:32.257 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:20:32.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:31 smithi086 ceph-mon[47925]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:33.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:32 smithi032 ceph-mon[39730]: pgmap v76: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:33.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:32 smithi086 ceph-mon[47925]: pgmap v76: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:34.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.248+0000 7f389d0b0700 1 -- 172.21.15.32:0/1846187998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 msgr2=0x7f38980fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.248+0000 7f389d0b0700 1 --2- 172.21.15.32:0/1846187998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 0x7f38980fe260 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3888004940 tx=0x7f3888009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.248+0000 7f389d0b0700 1 -- 172.21.15.32:0/1846187998 shutdown_connections 2024-04-06T14:20:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.248+0000 7f389d0b0700 1 --2- 172.21.15.32:0/1846187998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38980fe830 0x7f3898106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.248+0000 7f389d0b0700 1 --2- 172.21.15.32:0/1846187998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 0x7f38980fe260 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.248+0000 7f389d0b0700 1 -- 172.21.15.32:0/1846187998 >> 172.21.15.32:0/1846187998 conn(0x7f3898072bc0 msgr2=0x7f3898072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.248+0000 7f389d0b0700 1 -- 172.21.15.32:0/1846187998 shutdown_connections 2024-04-06T14:20:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.248+0000 7f389d0b0700 1 -- 172.21.15.32:0/1846187998 wait complete. 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.249+0000 7f389d0b0700 1 Processor -- start 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.249+0000 7f389d0b0700 1 -- start start 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.249+0000 7f389d0b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 0x7f3898113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.249+0000 7f389d0b0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38980fe830 0x7f3898114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.249+0000 7f389d0b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3898114710 con 0x7f38980fde90 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.249+0000 7f389d0b0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389810da70 con 0x7f38980fe830 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.249+0000 7f3896d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 0x7f3898113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.249+0000 7f3896d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 0x7f3898113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48972/0 (socket says 172.21.15.32:48972) 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.249+0000 7f3896d9d700 1 -- 172.21.15.32:0/1658192619 learned_addr learned my addr 172.21.15.32:0/1658192619 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f389659c700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38980fe830 0x7f3898114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f3896d9d700 1 -- 172.21.15.32:0/1658192619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38980fe830 msgr2=0x7f3898114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f3896d9d700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38980fe830 0x7f3898114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f3896d9d700 1 -- 172.21.15.32:0/1658192619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3888005040 con 0x7f38980fde90 2024-04-06T14:20:34.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f389659c700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38980fe830 0x7f3898114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:20:34.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f3896d9d700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 0x7f3898113af0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3888009390 tx=0x7f3888009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:34.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f387f7fe700 1 -- 172.21.15.32:0/1658192619 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f388800e040 con 0x7f38980fde90 2024-04-06T14:20:34.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389810dcf0 con 0x7f38980fde90 2024-04-06T14:20:34.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f387f7fe700 1 -- 172.21.15.32:0/1658192619 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3888004290 con 0x7f38980fde90 2024-04-06T14:20:34.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f387f7fe700 1 -- 172.21.15.32:0/1658192619 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f388801c3f0 con 0x7f38980fde90 2024-04-06T14:20:34.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.250+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389810e230 con 0x7f38980fde90 2024-04-06T14:20:34.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.251+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3898065e30 con 0x7f38980fde90 2024-04-06T14:20:34.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.252+0000 7f387f7fe700 1 -- 172.21.15.32:0/1658192619 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f3888013420 con 0x7f38980fde90 2024-04-06T14:20:34.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.252+0000 7f387f7fe700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f388006c8a0 0x7f388006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:34.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.252+0000 7f387f7fe700 1 -- 172.21.15.32:0/1658192619 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f388808ccf0 con 0x7f38980fde90 2024-04-06T14:20:34.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.252+0000 7f389659c700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f388006c8a0 0x7f388006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:34.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.253+0000 7f389659c700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f388006c8a0 0x7f388006ed50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f388c006fd0 tx=0x7f388c008040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:34.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.257+0000 7f387f7fe700 1 -- 172.21.15.32:0/1658192619 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f388805ace0 con 0x7f38980fde90 2024-04-06T14:20:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.550+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3898065e30 con 0x7f38980fde90 2024-04-06T14:20:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.551+0000 7f387f7fe700 1 -- 172.21.15.32:0/1658192619 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f388805a870 con 0x7f38980fde90 2024-04-06T14:20:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f388006c8a0 msgr2=0x7f388006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f388006c8a0 0x7f388006ed50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f388c006fd0 tx=0x7f388c008040 comp rx=0 tx=0).stop 2024-04-06T14:20:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 msgr2=0x7f3898113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 0x7f3898113af0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3888009390 tx=0x7f3888009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 shutdown_connections 2024-04-06T14:20:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f388006c8a0 0x7f388006ed50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980fde90 0x7f3898113af0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 --2- 172.21.15.32:0/1658192619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38980fe830 0x7f3898114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 >> 172.21.15.32:0/1658192619 conn(0x7f3898072bc0 msgr2=0x7f38981037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 shutdown_connections 2024-04-06T14:20:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:34.553+0000 7f389d0b0700 1 -- 172.21.15.32:0/1658192619 wait complete. 2024-04-06T14:20:34.563 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:20:35.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:34 smithi032 ceph-mon[39730]: pgmap v77: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:35.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:34 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/1658192619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:35.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-06T14:20:35.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:34 smithi086 ceph-mon[47925]: pgmap v77: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:35.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:34 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/1658192619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:35.665 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:20:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:35 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:20:36.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:35 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:20:37.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:36 smithi032 ceph-mon[39730]: pgmap v78: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:37.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:36 smithi086 ceph-mon[47925]: pgmap v78: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:37.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.726+0000 7f9ff20ea700 1 -- 172.21.15.32:0/2935472921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec1035d0 msgr2=0x7f9fec0fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.726+0000 7f9ff20ea700 1 --2- 172.21.15.32:0/2935472921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec1035d0 0x7f9fec0fcc40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0004940 tx=0x7f9fe0009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.726+0000 7f9ff20ea700 1 -- 172.21.15.32:0/2935472921 shutdown_connections 2024-04-06T14:20:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.726+0000 7f9ff20ea700 1 --2- 172.21.15.32:0/2935472921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec1035d0 0x7f9fec0fcc40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.726+0000 7f9ff20ea700 1 --2- 172.21.15.32:0/2935472921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fec102c30 0x7f9fec103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.726+0000 7f9ff20ea700 1 -- 172.21.15.32:0/2935472921 >> 172.21.15.32:0/2935472921 conn(0x7f9fec0f8760 msgr2=0x7f9fec0fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.726+0000 7f9ff20ea700 1 -- 172.21.15.32:0/2935472921 shutdown_connections 2024-04-06T14:20:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.726+0000 7f9ff20ea700 1 -- 172.21.15.32:0/2935472921 wait complete. 2024-04-06T14:20:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff20ea700 1 Processor -- start 2024-04-06T14:20:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff20ea700 1 -- start start 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff20ea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec102c30 0x7f9fec115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff20ea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fec1035d0 0x7f9fec116020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff20ea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec116670 con 0x7f9fec102c30 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff20ea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec10fa60 con 0x7f9fec1035d0 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff10e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec102c30 0x7f9fec115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff10e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec102c30 0x7f9fec115ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42218/0 (socket says 172.21.15.32:42218) 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff10e8700 1 -- 172.21.15.32:0/3495872113 learned_addr learned my addr 172.21.15.32:0/3495872113 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.727+0000 7f9ff08e7700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fec1035d0 0x7f9fec116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.728+0000 7f9ff10e8700 1 -- 172.21.15.32:0/3495872113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fec1035d0 msgr2=0x7f9fec116020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.728+0000 7f9ff10e8700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fec1035d0 0x7f9fec116020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.728+0000 7f9ff10e8700 1 -- 172.21.15.32:0/3495872113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0005040 con 0x7f9fec102c30 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.728+0000 7f9ff10e8700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec102c30 0x7f9fec115ae0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8002a50 tx=0x7f9fe80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.728+0000 7f9fddffb700 1 -- 172.21.15.32:0/3495872113 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fe8007cc0 con 0x7f9fec102c30 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.728+0000 7f9fddffb700 1 -- 172.21.15.32:0/3495872113 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fe800f040 con 0x7f9fec102c30 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.728+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec10fd40 con 0x7f9fec102c30 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.728+0000 7f9fddffb700 1 -- 172.21.15.32:0/3495872113 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fe800d9c0 con 0x7f9fec102c30 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.728+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec1102d0 con 0x7f9fec102c30 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.729+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec0fe380 con 0x7f9fec102c30 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.730+0000 7f9fddffb700 1 -- 172.21.15.32:0/3495872113 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f9fe800db20 con 0x7f9fec102c30 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.730+0000 7f9fddffb700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f9fd806c8a0 0x7f9fd806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.730+0000 7f9fddffb700 1 -- 172.21.15.32:0/3495872113 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f9fe808b6f0 con 0x7f9fec102c30 2024-04-06T14:20:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.730+0000 7f9ff08e7700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f9fd806c8a0 0x7f9fd806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:37.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.731+0000 7f9ff08e7700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f9fd806c8a0 0x7f9fd806ed50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9fec110d40 tx=0x7f9fe0009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:37.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:37.735+0000 7f9fddffb700 1 -- 172.21.15.32:0/3495872113 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f9fe80596e0 con 0x7f9fec102c30 2024-04-06T14:20:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.023+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9fec0fc3c0 con 0x7f9fec102c30 2024-04-06T14:20:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.023+0000 7f9fddffb700 1 -- 172.21.15.32:0/3495872113 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f9fe8059270 con 0x7f9fec102c30 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.025+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f9fd806c8a0 msgr2=0x7f9fd806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.025+0000 7f9ff20ea700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f9fd806c8a0 0x7f9fd806ed50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9fec110d40 tx=0x7f9fe0009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.025+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec102c30 msgr2=0x7f9fec115ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.025+0000 7f9ff20ea700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec102c30 0x7f9fec115ae0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8002a50 tx=0x7f9fe80046b0 comp rx=0 tx=0).stop 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.025+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 shutdown_connections 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.025+0000 7f9ff20ea700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f9fd806c8a0 0x7f9fd806ed50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.026+0000 7f9ff20ea700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fec102c30 0x7f9fec115ae0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.026+0000 7f9ff20ea700 1 --2- 172.21.15.32:0/3495872113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fec1035d0 0x7f9fec116020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.026+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 >> 172.21.15.32:0/3495872113 conn(0x7f9fec0f8760 msgr2=0x7f9fec0f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.026+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 shutdown_connections 2024-04-06T14:20:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:38.026+0000 7f9ff20ea700 1 -- 172.21.15.32:0/3495872113 wait complete. 2024-04-06T14:20:38.036 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)" 2024-04-06T14:20:38.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-06T14:20:39.073 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:20:39.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:38 smithi032 ceph-mon[39730]: pgmap v79: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:39.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:38 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3495872113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:39.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:38 smithi086 ceph-mon[47925]: pgmap v79: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:39.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:38 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3495872113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.117+0000 7fdc42449700 1 -- 172.21.15.32:0/838319617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 msgr2=0x7fdc3c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.117+0000 7fdc42449700 1 --2- 172.21.15.32:0/838319617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 0x7fdc3c103240 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c004940 tx=0x7fdc2c009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.117+0000 7fdc42449700 1 -- 172.21.15.32:0/838319617 shutdown_connections 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.117+0000 7fdc42449700 1 --2- 172.21.15.32:0/838319617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc3c0fb4a0 0x7fdc3c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.117+0000 7fdc42449700 1 --2- 172.21.15.32:0/838319617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 0x7fdc3c103240 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.117+0000 7fdc42449700 1 -- 172.21.15.32:0/838319617 >> 172.21.15.32:0/838319617 conn(0x7fdc3c0714c0 msgr2=0x7fdc3c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.117+0000 7fdc42449700 1 -- 172.21.15.32:0/838319617 shutdown_connections 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.117+0000 7fdc42449700 1 -- 172.21.15.32:0/838319617 wait complete. 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc42449700 1 Processor -- start 2024-04-06T14:20:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc42449700 1 -- start start 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc42449700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc3c0fb4a0 0x7fdc3c113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc42449700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 0x7fdc3c114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc42449700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c1146e0 con 0x7fdc3c102e70 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc42449700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c10dad0 con 0x7fdc3c0fb4a0 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc3b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 0x7fdc3c114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc3b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 0x7fdc3c114090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42244/0 (socket says 172.21.15.32:42244) 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc3b7fe700 1 -- 172.21.15.32:0/2706421247 learned_addr learned my addr 172.21.15.32:0/2706421247 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc3bfff700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc3c0fb4a0 0x7fdc3c113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc3b7fe700 1 -- 172.21.15.32:0/2706421247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc3c0fb4a0 msgr2=0x7fdc3c113b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc3b7fe700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc3c0fb4a0 0x7fdc3c113b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.118+0000 7fdc3b7fe700 1 -- 172.21.15.32:0/2706421247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c005040 con 0x7fdc3c102e70 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.119+0000 7fdc3b7fe700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 0x7fdc3c114090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdc30002a50 tx=0x7fdc300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.119+0000 7fdc38ff9700 1 -- 172.21.15.32:0/2706421247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc30007cc0 con 0x7fdc3c102e70 2024-04-06T14:20:41.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.119+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c10ddb0 con 0x7fdc3c102e70 2024-04-06T14:20:41.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.119+0000 7fdc38ff9700 1 -- 172.21.15.32:0/2706421247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc3000f040 con 0x7fdc3c102e70 2024-04-06T14:20:41.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.119+0000 7fdc38ff9700 1 -- 172.21.15.32:0/2706421247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc3000d9c0 con 0x7fdc3c102e70 2024-04-06T14:20:41.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.119+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c10e2f0 con 0x7fdc3c102e70 2024-04-06T14:20:41.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.120+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c0653a0 con 0x7fdc3c102e70 2024-04-06T14:20:41.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.120+0000 7fdc38ff9700 1 -- 172.21.15.32:0/2706421247 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fdc3000db20 con 0x7fdc3c102e70 2024-04-06T14:20:41.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.121+0000 7fdc38ff9700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fdc2406c8a0 0x7fdc2406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:41.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.121+0000 7fdc38ff9700 1 -- 172.21.15.32:0/2706421247 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fdc3008b7b0 con 0x7fdc3c102e70 2024-04-06T14:20:41.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.121+0000 7fdc3bfff700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fdc2406c8a0 0x7fdc2406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:41.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.121+0000 7fdc3bfff700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fdc2406c8a0 0x7fdc2406ed50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c004f90 tx=0x7fdc2c009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:41.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.126+0000 7fdc38ff9700 1 -- 172.21.15.32:0/2706421247 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fdc30059720 con 0x7fdc3c102e70 2024-04-06T14:20:41.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:40 smithi032 ceph-mon[39730]: pgmap v80: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:41.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:40 smithi086 ceph-mon[47925]: pgmap v80: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:41.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.412+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdc3c105300 con 0x7fdc3c102e70 2024-04-06T14:20:41.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.412+0000 7fdc38ff9700 1 -- 172.21.15.32:0/2706421247 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7fdc300592b0 con 0x7fdc3c102e70 2024-04-06T14:20:41.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.414+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fdc2406c8a0 msgr2=0x7fdc2406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:41.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.414+0000 7fdc42449700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fdc2406c8a0 0x7fdc2406ed50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c004f90 tx=0x7fdc2c009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:41.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.414+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 msgr2=0x7fdc3c114090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:41.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.414+0000 7fdc42449700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 0x7fdc3c114090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdc30002a50 tx=0x7fdc300046b0 comp rx=0 tx=0).stop 2024-04-06T14:20:41.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.414+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 shutdown_connections 2024-04-06T14:20:41.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.414+0000 7fdc42449700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fdc2406c8a0 0x7fdc2406ed50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:41.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.414+0000 7fdc42449700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc3c0fb4a0 0x7fdc3c113b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:41.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.415+0000 7fdc42449700 1 --2- 172.21.15.32:0/2706421247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c102e70 0x7fdc3c114090 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:41.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.415+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 >> 172.21.15.32:0/2706421247 conn(0x7fdc3c0714c0 msgr2=0x7fdc3c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:41.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.415+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 shutdown_connections 2024-04-06T14:20:41.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:41.415+0000 7fdc42449700 1 -- 172.21.15.32:0/2706421247 wait complete. 2024-04-06T14:20:41.425 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:20:42.130 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:41 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2706421247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:42.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-06T14:20:42.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:41 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2706421247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:20:42.437 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:20:43.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:42 smithi032 ceph-mon[39730]: pgmap v81: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:43.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:42 smithi086 ceph-mon[47925]: pgmap v81: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:44.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.327+0000 7ff62e224700 1 -- 172.21.15.32:0/3341616209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6280fb4a0 msgr2=0x7ff6280fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.327+0000 7ff62e224700 1 --2- 172.21.15.32:0/3341616209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6280fb4a0 0x7ff6280fb910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff61c004940 tx=0x7ff61c009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.327+0000 7ff62e224700 1 -- 172.21.15.32:0/3341616209 shutdown_connections 2024-04-06T14:20:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.327+0000 7ff62e224700 1 --2- 172.21.15.32:0/3341616209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6280fb4a0 0x7ff6280fb910 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.327+0000 7ff62e224700 1 --2- 172.21.15.32:0/3341616209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff628102e70 0x7ff628103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.327+0000 7ff62e224700 1 -- 172.21.15.32:0/3341616209 >> 172.21.15.32:0/3341616209 conn(0x7ff6280714c0 msgr2=0x7ff6280718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.327+0000 7ff62e224700 1 -- 172.21.15.32:0/3341616209 shutdown_connections 2024-04-06T14:20:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.327+0000 7ff62e224700 1 -- 172.21.15.32:0/3341616209 wait complete. 2024-04-06T14:20:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.328+0000 7ff62e224700 1 Processor -- start 2024-04-06T14:20:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.328+0000 7ff62e224700 1 -- start start 2024-04-06T14:20:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.328+0000 7ff62e224700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6280fb4a0 0x7ff628113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.328+0000 7ff62e224700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff628102e70 0x7ff628113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.328+0000 7ff62e224700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff628114620 con 0x7ff628102e70 2024-04-06T14:20:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.328+0000 7ff62e224700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62810da10 con 0x7ff6280fb4a0 2024-04-06T14:20:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff626ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff628102e70 0x7ff628113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff626ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff628102e70 0x7ff628113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42254/0 (socket says 172.21.15.32:42254) 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff626ffd700 1 -- 172.21.15.32:0/2643915736 learned_addr learned my addr 172.21.15.32:0/2643915736 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff6277fe700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6280fb4a0 0x7ff628113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff626ffd700 1 -- 172.21.15.32:0/2643915736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6280fb4a0 msgr2=0x7ff628113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff626ffd700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6280fb4a0 0x7ff628113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff626ffd700 1 -- 172.21.15.32:0/2643915736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c005040 con 0x7ff628102e70 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff626ffd700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff628102e70 0x7ff628113fd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00b3b0 tx=0x7ff61c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff60ffff700 1 -- 172.21.15.32:0/2643915736 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff61c00e040 con 0x7ff628102e70 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff60ffff700 1 -- 172.21.15.32:0/2643915736 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff61c013420 con 0x7ff628102e70 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62810dc90 con 0x7ff628102e70 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff60ffff700 1 -- 172.21.15.32:0/2643915736 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff61c01c610 con 0x7ff628102e70 2024-04-06T14:20:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.329+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62810e250 con 0x7ff628102e70 2024-04-06T14:20:44.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.330+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff628100110 con 0x7ff628102e70 2024-04-06T14:20:44.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.331+0000 7ff60ffff700 1 -- 172.21.15.32:0/2643915736 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7ff61c013a00 con 0x7ff628102e70 2024-04-06T14:20:44.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.331+0000 7ff60ffff700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff61006c8a0 0x7ff61006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:44.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.331+0000 7ff60ffff700 1 -- 172.21.15.32:0/2643915736 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff61c08cc00 con 0x7ff628102e70 2024-04-06T14:20:44.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.331+0000 7ff6277fe700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff61006c8a0 0x7ff61006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:44.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.332+0000 7ff6277fe700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff61006c8a0 0x7ff61006ed50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff618004f40 tx=0x7ff61800b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:44.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.336+0000 7ff60ffff700 1 -- 172.21.15.32:0/2643915736 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff61c092020 con 0x7ff628102e70 2024-04-06T14:20:44.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:44.556+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46", "target": ["mon-mgr", ""]}) v1 -- 0x7ff62810f0e0 con 0x7ff61006c8a0 2024-04-06T14:20:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:44 smithi032 ceph-mon[39730]: pgmap v82: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:44 smithi032 ceph-mon[39730]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:45.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:44 smithi086 ceph-mon[47925]: pgmap v82: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:45.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:44 smithi086 ceph-mon[47925]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:46.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.025+0000 7ff60ffff700 1 -- 172.21.15.32:0/2643915736 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5094 (secure 0 0 0) 0x7ff62810f0e0 con 0x7ff61006c8a0 2024-04-06T14:20:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff61006c8a0 msgr2=0x7ff61006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff61006c8a0 0x7ff61006ed50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff618004f40 tx=0x7ff61800b040 comp rx=0 tx=0).stop 2024-04-06T14:20:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff628102e70 msgr2=0x7ff628113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff628102e70 0x7ff628113fd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00b3b0 tx=0x7ff61c009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 shutdown_connections 2024-04-06T14:20:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7ff61006c8a0 0x7ff61006ed50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6280fb4a0 0x7ff628113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 --2- 172.21.15.32:0/2643915736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff628102e70 0x7ff628113fd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 >> 172.21.15.32:0/2643915736 conn(0x7ff6280714c0 msgr2=0x7ff6280fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 shutdown_connections 2024-04-06T14:20:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:46.028+0000 7ff62e224700 1 -- 172.21.15.32:0/2643915736 wait complete. 2024-04-06T14:20:46.040 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:20:46.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph orch ps' 2024-04-06T14:20:47.095 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:20:47.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:46 smithi032 ceph-mon[39730]: pgmap v83: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:47.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:46 smithi086 ceph-mon[47925]: pgmap v83: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 -- 172.21.15.32:0/240812970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 msgr2=0x7f55f00fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 --2- 172.21.15.32:0/240812970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 0x7f55f00fbfe0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f55e0004940 tx=0x7f55e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 -- 172.21.15.32:0/240812970 shutdown_connections 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 --2- 172.21.15.32:0/240812970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 0x7f55f00fbfe0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 --2- 172.21.15.32:0/240812970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f0101ba0 0x7f55f0101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 -- 172.21.15.32:0/240812970 >> 172.21.15.32:0/240812970 conn(0x7f55f0071b70 msgr2=0x7f55f0071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 -- 172.21.15.32:0/240812970 shutdown_connections 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 -- 172.21.15.32:0/240812970 wait complete. 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 Processor -- start 2024-04-06T14:20:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.880+0000 7f55f739f700 1 -- start start 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f739f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 0x7f55f0113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f739f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f0101ba0 0x7f55f0113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f739f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0114520 con 0x7f55f00fbb70 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f739f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f010d960 con 0x7f55f0101ba0 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f639d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 0x7f55f0113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f639d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 0x7f55f0113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49276/0 (socket says 172.21.15.32:49276) 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f639d700 1 -- 172.21.15.32:0/646775395 learned_addr learned my addr 172.21.15.32:0/646775395 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f5b9c700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f0101ba0 0x7f55f0113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f639d700 1 -- 172.21.15.32:0/646775395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f0101ba0 msgr2=0x7f55f0113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f639d700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f0101ba0 0x7f55f0113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f639d700 1 -- 172.21.15.32:0/646775395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0005040 con 0x7f55f00fbb70 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55f639d700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 0x7f55f0113990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00c830 tx=0x7f55ec004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.881+0000 7f55e6ffd700 1 -- 172.21.15.32:0/646775395 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55ec00a760 con 0x7f55f00fbb70 2024-04-06T14:20:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.882+0000 7f55e6ffd700 1 -- 172.21.15.32:0/646775395 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55ec00ad70 con 0x7f55f00fbb70 2024-04-06T14:20:48.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.882+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f010dc40 con 0x7f55f00fbb70 2024-04-06T14:20:48.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.882+0000 7f55e6ffd700 1 -- 172.21.15.32:0/646775395 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55ec0113a0 con 0x7f55f00fbb70 2024-04-06T14:20:48.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.882+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f010e180 con 0x7f55f00fbb70 2024-04-06T14:20:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.883+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f00fd200 con 0x7f55f00fbb70 2024-04-06T14:20:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.883+0000 7f55e6ffd700 1 -- 172.21.15.32:0/646775395 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f55ec012400 con 0x7f55f00fbb70 2024-04-06T14:20:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.883+0000 7f55e6ffd700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f55dc06c8b0 0x7f55dc06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.884+0000 7f55e6ffd700 1 -- 172.21.15.32:0/646775395 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f55ec08ba50 con 0x7f55f00fbb70 2024-04-06T14:20:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.884+0000 7f55f5b9c700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f55dc06c8b0 0x7f55dc06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.884+0000 7f55f5b9c700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f55dc06c8b0 0x7f55dc06ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f55f010ead0 tx=0x7f55e0009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:48.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:48.888+0000 7f55e6ffd700 1 -- 172.21.15.32:0/646775395 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f55ec059ae0 con 0x7f55f00fbb70 2024-04-06T14:20:49.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.099+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f55f0108ae0 con 0x7f55dc06c8b0 2024-04-06T14:20:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.109+0000 7f55e6ffd700 1 -- 172.21.15.32:0/646775395 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f55f0108ae0 con 0x7f55dc06c8b0 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (72s) 47s ago 12m 15.4M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (12m) 47s ago 12m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (11m) 104s ago 11m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (54s) 47s ago 12m 50.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (6m) 47s ago 6m 13.0M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (6m) 47s ago 6m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (6m) 104s ago 6m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (6m) 104s ago 6m 15.5M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (3m) 47s ago 13m 499M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (2m) 104s ago 11m 449M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:20:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (13m) 47s ago 14m 54.2M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (11m) 104s ago 11m 43.8M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (111s) 47s ago 12m 9361k - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (107s) 104s ago 11m 3397k - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (10m) 47s ago 10m 193M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 47s ago 10m 191M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (9m) 47s ago 9m 161M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (9m) 104s ago 9m 224M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (8m) 104s ago 8m 202M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (8m) 104s ago 8m 183M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:20:49.111 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (89s) 47s ago 11m 47.6M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:20:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.111+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f55dc06c8b0 msgr2=0x7f55dc06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.111+0000 7f55f739f700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f55dc06c8b0 0x7f55dc06ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f55f010ead0 tx=0x7f55e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.111+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 msgr2=0x7f55f0113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.111+0000 7f55f739f700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 0x7f55f0113990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00c830 tx=0x7f55ec004470 comp rx=0 tx=0).stop 2024-04-06T14:20:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.111+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 shutdown_connections 2024-04-06T14:20:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.111+0000 7f55f739f700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f55dc06c8b0 0x7f55dc06ed60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.111+0000 7f55f739f700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55f00fbb70 0x7f55f0113990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.111+0000 7f55f739f700 1 --2- 172.21.15.32:0/646775395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f0101ba0 0x7f55f0113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.111+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 >> 172.21.15.32:0/646775395 conn(0x7f55f0071b70 msgr2=0x7f55f0065600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.112+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 shutdown_connections 2024-04-06T14:20:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:49.112+0000 7f55f739f700 1 -- 172.21.15.32:0/646775395 wait complete. 2024-04-06T14:20:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:48 smithi032 ceph-mon[39730]: pgmap v84: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:48 smithi086 ceph-mon[47925]: pgmap v84: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:49.733 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-06T14:20:49.734 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:20:49.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-06T14:20:50.014 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:20:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:49 smithi032 ceph-mon[39730]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:50.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:49 smithi086 ceph-mon[47925]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:20:51.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:50 smithi086 ceph-mon[47925]: pgmap v85: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:51.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:50 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:20:51.428 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:50 smithi032 ceph-mon[39730]: pgmap v85: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:51.428 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:50 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:20:51.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.957+0000 7f7e097ad700 1 -- 172.21.15.32:0/3208151107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e041037b0 msgr2=0x7f7e040fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.957+0000 7f7e097ad700 1 --2- 172.21.15.32:0/3208151107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e041037b0 0x7f7e040fce20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7df4004940 tx=0x7f7df4009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.957+0000 7f7e097ad700 1 -- 172.21.15.32:0/3208151107 shutdown_connections 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.957+0000 7f7e097ad700 1 --2- 172.21.15.32:0/3208151107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e041037b0 0x7f7e040fce20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.957+0000 7f7e097ad700 1 --2- 172.21.15.32:0/3208151107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e04102e10 0x7f7e041031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.957+0000 7f7e097ad700 1 -- 172.21.15.32:0/3208151107 >> 172.21.15.32:0/3208151107 conn(0x7f7e040f89a0 msgr2=0x7f7e040fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.957+0000 7f7e097ad700 1 -- 172.21.15.32:0/3208151107 shutdown_connections 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.957+0000 7f7e097ad700 1 -- 172.21.15.32:0/3208151107 wait complete. 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.958+0000 7f7e097ad700 1 Processor -- start 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.958+0000 7f7e097ad700 1 -- start start 2024-04-06T14:20:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.958+0000 7f7e097ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e04102e10 0x7f7e04113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.958+0000 7f7e097ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e041037b0 0x7f7e04113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.958+0000 7f7e097ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e041145c0 con 0x7f7e04102e10 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.958+0000 7f7e097ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0410d9b0 con 0x7f7e041037b0 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.958+0000 7f7e02ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e04102e10 0x7f7e04113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.958+0000 7f7e02ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e04102e10 0x7f7e04113a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49294/0 (socket says 172.21.15.32:49294) 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.958+0000 7f7e02ffd700 1 -- 172.21.15.32:0/2468554200 learned_addr learned my addr 172.21.15.32:0/2468554200 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7e027fc700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e041037b0 0x7f7e04113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7e02ffd700 1 -- 172.21.15.32:0/2468554200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e041037b0 msgr2=0x7f7e04113f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7e02ffd700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e041037b0 0x7f7e04113f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7e02ffd700 1 -- 172.21.15.32:0/2468554200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df4005040 con 0x7f7e04102e10 2024-04-06T14:20:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7e02ffd700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e04102e10 0x7f7e04113a30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7df800eb40 tx=0x7f7df800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7deb7fe700 1 -- 172.21.15.32:0/2468554200 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7df8007cc0 con 0x7f7e04102e10 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7deb7fe700 1 -- 172.21.15.32:0/2468554200 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7df800f040 con 0x7f7e04102e10 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7deb7fe700 1 -- 172.21.15.32:0/2468554200 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7df8009570 con 0x7f7e04102e10 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0410dc30 con 0x7f7e04102e10 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.959+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0410e170 con 0x7f7e04102e10 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.960+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e040fe560 con 0x7f7e04102e10 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.961+0000 7f7deb7fe700 1 -- 172.21.15.32:0/2468554200 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f7df80096d0 con 0x7f7e04102e10 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.961+0000 7f7deb7fe700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7dec06c8a0 0x7f7dec06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.961+0000 7f7deb7fe700 1 -- 172.21.15.32:0/2468554200 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f7df808b770 con 0x7f7e04102e10 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.961+0000 7f7e027fc700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7dec06c8a0 0x7f7dec06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.962+0000 7f7e027fc700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7dec06c8a0 0x7f7dec06ed50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e0410ec90 tx=0x7f7df4009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:51.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:51.966+0000 7f7deb7fe700 1 -- 172.21.15.32:0/2468554200 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f7df8059790 con 0x7f7e04102e10 2024-04-06T14:20:52.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.167+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f7e04109db0 con 0x7f7e04102e10 2024-04-06T14:20:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.171+0000 7f7deb7fe700 1 -- 172.21.15.32:0/2468554200 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v47)=0 v47) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7df8059320 con 0x7f7e04102e10 2024-04-06T14:20:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7dec06c8a0 msgr2=0x7f7dec06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7dec06c8a0 0x7f7dec06ed50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e0410ec90 tx=0x7f7df4009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e04102e10 msgr2=0x7f7e04113a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e04102e10 0x7f7e04113a30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7df800eb40 tx=0x7f7df800b040 comp rx=0 tx=0).stop 2024-04-06T14:20:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 shutdown_connections 2024-04-06T14:20:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f7dec06c8a0 0x7f7dec06ed50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e04102e10 0x7f7e04113a30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 --2- 172.21.15.32:0/2468554200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e041037b0 0x7f7e04113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:52.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 >> 172.21.15.32:0/2468554200 conn(0x7f7e040f89a0 msgr2=0x7f7e040f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:52.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 shutdown_connections 2024-04-06T14:20:52.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:52.174+0000 7f7e097ad700 1 -- 172.21.15.32:0/2468554200 wait complete. 2024-04-06T14:20:53.052 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-06T14:20:53.053 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:20:53.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-06T14:20:53.342 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:20:53.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:53 smithi032 ceph-mon[39730]: pgmap v86: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:53.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:53 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2468554200' entity='client.admin' 2024-04-06T14:20:53.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:53 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:20:53.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:53 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:20:53.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:53 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:20:53.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:53 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:53.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:53 smithi086 ceph-mon[47925]: pgmap v86: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:53.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:53 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2468554200' entity='client.admin' 2024-04-06T14:20:53.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:53 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:20:53.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:53 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:20:53.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:53 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:20:53.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:53 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:20:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.255+0000 7f8da9fa6700 1 -- 172.21.15.32:0/3065264229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 msgr2=0x7f8da40706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.255+0000 7f8da9fa6700 1 --2- 172.21.15.32:0/3065264229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 0x7f8da40706d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c004970 tx=0x7f8d8c009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.255+0000 7f8da9fa6700 1 -- 172.21.15.32:0/3065264229 shutdown_connections 2024-04-06T14:20:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.255+0000 7f8da9fa6700 1 --2- 172.21.15.32:0/3065264229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8da4070c10 0x7f8da4109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.255+0000 7f8da9fa6700 1 --2- 172.21.15.32:0/3065264229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 0x7f8da40706d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.255+0000 7f8da9fa6700 1 -- 172.21.15.32:0/3065264229 >> 172.21.15.32:0/3065264229 conn(0x7f8da40f89a0 msgr2=0x7f8da40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.255+0000 7f8da9fa6700 1 -- 172.21.15.32:0/3065264229 shutdown_connections 2024-04-06T14:20:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.255+0000 7f8da9fa6700 1 -- 172.21.15.32:0/3065264229 wait complete. 2024-04-06T14:20:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.256+0000 7f8da9fa6700 1 Processor -- start 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.256+0000 7f8da9fa6700 1 -- start start 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.256+0000 7f8da9fa6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 0x7f8da4115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.256+0000 7f8da9fa6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8da4070c10 0x7f8da4116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.256+0000 7f8da9fa6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da4116860 con 0x7f8da4070300 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.256+0000 7f8da9fa6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da410fc30 con 0x7f8da4070c10 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.256+0000 7f8da37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 0x7f8da4115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.256+0000 7f8da37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 0x7f8da4115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36752/0 (socket says 172.21.15.32:36752) 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.256+0000 7f8da37fe700 1 -- 172.21.15.32:0/213973443 learned_addr learned my addr 172.21.15.32:0/213973443 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8da2ffd700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8da4070c10 0x7f8da4116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8da37fe700 1 -- 172.21.15.32:0/213973443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8da4070c10 msgr2=0x7f8da4116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8da37fe700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8da4070c10 0x7f8da4116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8da37fe700 1 -- 172.21.15.32:0/213973443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c005040 con 0x7f8da4070300 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8da37fe700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 0x7f8da4115cd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0043d0 tx=0x7f8d8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8d9bfff700 1 -- 172.21.15.32:0/213973443 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d8c00e040 con 0x7f8da4070300 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8d9bfff700 1 -- 172.21.15.32:0/213973443 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d8c013420 con 0x7f8da4070300 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8d9bfff700 1 -- 172.21.15.32:0/213973443 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d8c01c630 con 0x7f8da4070300 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da410feb0 con 0x7f8da4070300 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.257+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da41103f0 con 0x7f8da4070300 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.258+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da4103ba0 con 0x7f8da4070300 2024-04-06T14:20:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.259+0000 7f8d9bfff700 1 -- 172.21.15.32:0/213973443 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f8d8c013970 con 0x7f8da4070300 2024-04-06T14:20:55.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.259+0000 7f8d9bfff700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f8d9006c8a0 0x7f8d9006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:55.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.259+0000 7f8d9bfff700 1 -- 172.21.15.32:0/213973443 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f8d8c08cc30 con 0x7f8da4070300 2024-04-06T14:20:55.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.259+0000 7f8da2ffd700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f8d9006c8a0 0x7f8d9006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:55.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.260+0000 7f8da2ffd700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f8d9006c8a0 0x7f8d9006ed50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8d94007fd0 tx=0x7f8d94009040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:55.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.264+0000 7f8d9bfff700 1 -- 172.21.15.32:0/213973443 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f8d8c05acc0 con 0x7f8da4070300 2024-04-06T14:20:55.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.466+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f8da410bfc0 con 0x7f8da4070300 2024-04-06T14:20:55.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.466+0000 7f8d9bfff700 1 -- 172.21.15.32:0/213973443 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v47)=0 v47) v1 ==== 155+0+0 (secure 0 0 0) 0x7f8d8c004de0 con 0x7f8da4070300 2024-04-06T14:20:55.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f8d9006c8a0 msgr2=0x7f8d9006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:55.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f8d9006c8a0 0x7f8d9006ed50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8d94007fd0 tx=0x7f8d94009040 comp rx=0 tx=0).stop 2024-04-06T14:20:55.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 msgr2=0x7f8da4115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:55.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 0x7f8da4115cd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0043d0 tx=0x7f8d8c009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:55.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 shutdown_connections 2024-04-06T14:20:55.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f8d9006c8a0 0x7f8d9006ed50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:55.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da4070300 0x7f8da4115cd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:55.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 --2- 172.21.15.32:0/213973443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8da4070c10 0x7f8da4116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:55.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 >> 172.21.15.32:0/213973443 conn(0x7f8da40f89a0 msgr2=0x7f8da4101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:55.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 shutdown_connections 2024-04-06T14:20:55.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:55.468+0000 7f8da9fa6700 1 -- 172.21.15.32:0/213973443 wait complete. 2024-04-06T14:20:55.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:55 smithi032 ceph-mon[39730]: pgmap v87: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:55.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:55 smithi086 ceph-mon[47925]: pgmap v87: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:56.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-06T14:20:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:20:57.392 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:57 smithi032 ceph-mon[39730]: pgmap v88: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:57.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:57 smithi086 ceph-mon[47925]: pgmap v88: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:58.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.256+0000 7f16c988c700 1 -- 172.21.15.32:0/3920584844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 msgr2=0x7f16c4106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:58.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.256+0000 7f16c988c700 1 --2- 172.21.15.32:0/3920584844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 0x7f16c4106e80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f16b4004940 tx=0x7f16b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:58.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.256+0000 7f16c988c700 1 -- 172.21.15.32:0/3920584844 shutdown_connections 2024-04-06T14:20:58.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.256+0000 7f16c988c700 1 --2- 172.21.15.32:0/3920584844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 0x7f16c4106e80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:58.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.256+0000 7f16c988c700 1 --2- 172.21.15.32:0/3920584844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c40fde10 0x7f16c40fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:58.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.256+0000 7f16c988c700 1 -- 172.21.15.32:0/3920584844 >> 172.21.15.32:0/3920584844 conn(0x7f16c40f7870 msgr2=0x7f16c40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:58.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.256+0000 7f16c988c700 1 -- 172.21.15.32:0/3920584844 shutdown_connections 2024-04-06T14:20:58.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.256+0000 7f16c988c700 1 -- 172.21.15.32:0/3920584844 wait complete. 2024-04-06T14:20:58.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c988c700 1 Processor -- start 2024-04-06T14:20:58.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c988c700 1 -- start start 2024-04-06T14:20:58.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c988c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c40fde10 0x7f16c4113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:58.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c988c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 0x7f16c4113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c988c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4114520 con 0x7f16c40fe7b0 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c988c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c410db20 con 0x7f16c40fde10 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 0x7f16c4113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 0x7f16c4113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36764/0 (socket says 172.21.15.32:36764) 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c27fc700 1 -- 172.21.15.32:0/2477979091 learned_addr learned my addr 172.21.15.32:0/2477979091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c2ffd700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c40fde10 0x7f16c4113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c27fc700 1 -- 172.21.15.32:0/2477979091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c40fde10 msgr2=0x7f16c4113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c27fc700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c40fde10 0x7f16c4113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c27fc700 1 -- 172.21.15.32:0/2477979091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b4005040 con 0x7f16c40fe7b0 2024-04-06T14:20:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c2ffd700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c40fde10 0x7f16c4113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.257+0000 7f16c27fc700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 0x7f16c4113ed0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f16b4004890 tx=0x7f16b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.258+0000 7f16bb7fe700 1 -- 172.21.15.32:0/2477979091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16b400e040 con 0x7f16c40fe7b0 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.258+0000 7f16bb7fe700 1 -- 172.21.15.32:0/2477979091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16b4009af0 con 0x7f16c40fe7b0 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.258+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c410dda0 con 0x7f16c40fe7b0 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.258+0000 7f16bb7fe700 1 -- 172.21.15.32:0/2477979091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16b401c3f0 con 0x7f16c40fe7b0 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.258+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c410e2e0 con 0x7f16c40fe7b0 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.259+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c4066ad0 con 0x7f16c40fe7b0 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.259+0000 7f16bb7fe700 1 -- 172.21.15.32:0/2477979091 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f16b4013420 con 0x7f16c40fe7b0 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.260+0000 7f16bb7fe700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f16b006c8a0 0x7f16b006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:20:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.260+0000 7f16bb7fe700 1 -- 172.21.15.32:0/2477979091 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f16b408cd50 con 0x7f16c40fe7b0 2024-04-06T14:20:58.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.260+0000 7f16c2ffd700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f16b006c8a0 0x7f16b006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:20:58.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.260+0000 7f16c2ffd700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f16b006c8a0 0x7f16b006ed50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f16ac007fd0 tx=0x7f16ac009040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:20:58.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.264+0000 7f16bb7fe700 1 -- 172.21.15.32:0/2477979091 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f16b405ade0 con 0x7f16c40fe7b0 2024-04-06T14:20:58.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.468+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f16c4109d60 con 0x7f16c40fe7b0 2024-04-06T14:20:58.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.468+0000 7f16bb7fe700 1 -- 172.21.15.32:0/2477979091 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v47)=0 v47) v1 ==== 163+0+0 (secure 0 0 0) 0x7f16b405a970 con 0x7f16c40fe7b0 2024-04-06T14:20:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.470+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f16b006c8a0 msgr2=0x7f16b006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.470+0000 7f16c988c700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f16b006c8a0 0x7f16b006ed50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f16ac007fd0 tx=0x7f16ac009040 comp rx=0 tx=0).stop 2024-04-06T14:20:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.470+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 msgr2=0x7f16c4113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:20:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.470+0000 7f16c988c700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 0x7f16c4113ed0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f16b4004890 tx=0x7f16b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:20:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.470+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 shutdown_connections 2024-04-06T14:20:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.470+0000 7f16c988c700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f16b006c8a0 0x7f16b006ed50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.470+0000 7f16c988c700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c40fde10 0x7f16c4113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.471+0000 7f16c988c700 1 --2- 172.21.15.32:0/2477979091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c40fe7b0 0x7f16c4113ed0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:20:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.471+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 >> 172.21.15.32:0/2477979091 conn(0x7f16c40f7870 msgr2=0x7f16c41063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:20:58.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.471+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 shutdown_connections 2024-04-06T14:20:58.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:20:58.471+0000 7f16c988c700 1 -- 172.21.15.32:0/2477979091 wait complete. 2024-04-06T14:20:59.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:20:59 smithi032 ceph-mon[39730]: pgmap v89: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:20:59.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-06T14:20:59.590 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:20:59.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:20:59 smithi086 ceph-mon[47925]: pgmap v89: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:01.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:01 smithi032 ceph-mon[39730]: pgmap v90: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:01.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.515+0000 7faedb029700 1 -- 172.21.15.32:0/577934118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 msgr2=0x7faed41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:01.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.515+0000 7faedb029700 1 --2- 172.21.15.32:0/577934118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 0x7faed41031e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7faec8004940 tx=0x7faec8009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:01.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.515+0000 7faedb029700 1 -- 172.21.15.32:0/577934118 shutdown_connections 2024-04-06T14:21:01.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.515+0000 7faedb029700 1 --2- 172.21.15.32:0/577934118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed41037b0 0x7faed40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:01.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.515+0000 7faedb029700 1 --2- 172.21.15.32:0/577934118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 0x7faed41031e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:01.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.515+0000 7faedb029700 1 -- 172.21.15.32:0/577934118 >> 172.21.15.32:0/577934118 conn(0x7faed40f8980 msgr2=0x7faed40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:01.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.515+0000 7faedb029700 1 -- 172.21.15.32:0/577934118 shutdown_connections 2024-04-06T14:21:01.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.515+0000 7faedb029700 1 -- 172.21.15.32:0/577934118 wait complete. 2024-04-06T14:21:01.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.515+0000 7faedb029700 1 Processor -- start 2024-04-06T14:21:01.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faedb029700 1 -- start start 2024-04-06T14:21:01.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faedb029700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 0x7faed4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:01.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faedb029700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed41037b0 0x7faed4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:01.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faedb029700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4114690 con 0x7faed4102e10 2024-04-06T14:21:01.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faedb029700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed410da80 con 0x7faed41037b0 2024-04-06T14:21:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faed8dc5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 0x7faed4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faed8dc5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 0x7faed4113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36776/0 (socket says 172.21.15.32:36776) 2024-04-06T14:21:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faed8dc5700 1 -- 172.21.15.32:0/1694928177 learned_addr learned my addr 172.21.15.32:0/1694928177 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faed3fff700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed41037b0 0x7faed4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faed8dc5700 1 -- 172.21.15.32:0/1694928177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed41037b0 msgr2=0x7faed4114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faed8dc5700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed41037b0 0x7faed4114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.516+0000 7faed8dc5700 1 -- 172.21.15.32:0/1694928177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec8005040 con 0x7faed4102e10 2024-04-06T14:21:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.517+0000 7faed3fff700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed41037b0 0x7faed4114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.517+0000 7faed8dc5700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 0x7faed4113b00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7faec800b3b0 tx=0x7faec8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.517+0000 7faed17fa700 1 -- 172.21.15.32:0/1694928177 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faec800e040 con 0x7faed4102e10 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.517+0000 7faed17fa700 1 -- 172.21.15.32:0/1694928177 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faec8013420 con 0x7faed4102e10 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.517+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed410dd00 con 0x7faed4102e10 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.517+0000 7faed17fa700 1 -- 172.21.15.32:0/1694928177 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faec801c630 con 0x7faed4102e10 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.517+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed410e2c0 con 0x7faed4102e10 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.518+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed40fe560 con 0x7faed4102e10 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.518+0000 7faed17fa700 1 -- 172.21.15.32:0/1694928177 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7faec801c8f0 con 0x7faed4102e10 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.519+0000 7faed17fa700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7faec406c8a0 0x7faec406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.519+0000 7faed17fa700 1 -- 172.21.15.32:0/1694928177 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7faec808ccc0 con 0x7faed4102e10 2024-04-06T14:21:01.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.519+0000 7faed3fff700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7faec406c8a0 0x7faec406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:01.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.519+0000 7faed3fff700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7faec406c8a0 0x7faec406ed50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faec0007fd0 tx=0x7faec0009040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:01.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.524+0000 7faed17fa700 1 -- 172.21.15.32:0/1694928177 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7faec805acb0 con 0x7faed4102e10 2024-04-06T14:21:01.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:01 smithi086 ceph-mon[47925]: pgmap v90: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:01.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.723+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7faed4109e40 con 0x7faed4102e10 2024-04-06T14:21:01.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.723+0000 7faed17fa700 1 -- 172.21.15.32:0/1694928177 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v47)=0 v47) v1 ==== 135+0+0 (secure 0 0 0) 0x7faec805a840 con 0x7faed4102e10 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7faec406c8a0 msgr2=0x7faec406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7faec406c8a0 0x7faec406ed50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faec0007fd0 tx=0x7faec0009040 comp rx=0 tx=0).stop 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 msgr2=0x7faed4113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 0x7faed4113b00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7faec800b3b0 tx=0x7faec8009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 shutdown_connections 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7faec406c8a0 0x7faec406ed50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faed4102e10 0x7faed4113b00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 --2- 172.21.15.32:0/1694928177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed41037b0 0x7faed4114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 >> 172.21.15.32:0/1694928177 conn(0x7faed40f8980 msgr2=0x7faed40f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.725+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 shutdown_connections 2024-04-06T14:21:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:01.726+0000 7faedb029700 1 -- 172.21.15.32:0/1694928177 wait complete. 2024-04-06T14:21:02.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-06T14:21:02.755 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:21:03.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:03 smithi032 ceph-mon[39730]: pgmap v91: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:03.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:03 smithi086 ceph-mon[47925]: pgmap v91: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.540+0000 7f789f86f700 1 -- 172.21.15.32:0/3396847550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898102710 msgr2=0x7f78980fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.540+0000 7f789f86f700 1 --2- 172.21.15.32:0/3396847550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898102710 0x7f78980fbd80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7894004940 tx=0x7f7894009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.541+0000 7f789f86f700 1 -- 172.21.15.32:0/3396847550 shutdown_connections 2024-04-06T14:21:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.541+0000 7f789f86f700 1 --2- 172.21.15.32:0/3396847550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898102710 0x7f78980fbd80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.541+0000 7f789f86f700 1 --2- 172.21.15.32:0/3396847550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7898101d70 0x7f7898102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.541+0000 7f789f86f700 1 -- 172.21.15.32:0/3396847550 >> 172.21.15.32:0/3396847550 conn(0x7f7898072bc0 msgr2=0x7f7898072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.541+0000 7f789f86f700 1 -- 172.21.15.32:0/3396847550 shutdown_connections 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.541+0000 7f789f86f700 1 -- 172.21.15.32:0/3396847550 wait complete. 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.541+0000 7f789f86f700 1 Processor -- start 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.541+0000 7f789f86f700 1 -- start start 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789f86f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898101d70 0x7f789810e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789f86f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7898102710 0x7f789810eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789f86f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789810f3e0 con 0x7f7898101d70 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789f86f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789810f520 con 0x7f7898102710 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789d60b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898101d70 0x7f789810e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789d60b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898101d70 0x7f789810e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53648/0 (socket says 172.21.15.32:53648) 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789d60b700 1 -- 172.21.15.32:0/950772511 learned_addr learned my addr 172.21.15.32:0/950772511 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789ce0a700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7898102710 0x7f789810eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789d60b700 1 -- 172.21.15.32:0/950772511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7898102710 msgr2=0x7f789810eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789d60b700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7898102710 0x7f789810eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789d60b700 1 -- 172.21.15.32:0/950772511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7894005040 con 0x7f7898101d70 2024-04-06T14:21:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789d60b700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898101d70 0x7f789810e960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7888002a50 tx=0x7f78880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f788dffb700 1 -- 172.21.15.32:0/950772511 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7888007cc0 con 0x7f7898101d70 2024-04-06T14:21:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f788dffb700 1 -- 172.21.15.32:0/950772511 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f788800f040 con 0x7f7898101d70 2024-04-06T14:21:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7898117810 con 0x7f7898101d70 2024-04-06T14:21:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.542+0000 7f788dffb700 1 -- 172.21.15.32:0/950772511 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f788800d9c0 con 0x7f7898101d70 2024-04-06T14:21:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.543+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7898117da0 con 0x7f7898101d70 2024-04-06T14:21:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.544+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78980ff080 con 0x7f7898101d70 2024-04-06T14:21:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.544+0000 7f788dffb700 1 -- 172.21.15.32:0/950772511 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f788800db80 con 0x7f7898101d70 2024-04-06T14:21:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.545+0000 7f788dffb700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f788406c8b0 0x7f788406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.545+0000 7f788dffb700 1 -- 172.21.15.32:0/950772511 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f788808b6e0 con 0x7f7898101d70 2024-04-06T14:21:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.545+0000 7f789ce0a700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f788406c8b0 0x7f788406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.545+0000 7f789ce0a700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f788406c8b0 0x7f788406ed60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7898066020 tx=0x7f7894009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.549+0000 7f788dffb700 1 -- 172.21.15.32:0/950772511 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f7888055c40 con 0x7f7898101d70 2024-04-06T14:21:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.776+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46", "target": ["mon-mgr", ""]}) v1 -- 0x7f789804bed0 con 0x7f788406c8b0 2024-04-06T14:21:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.781+0000 7f788dffb700 1 -- 172.21.15.32:0/950772511 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f789804bed0 con 0x7f788406c8b0 2024-04-06T14:21:04.782 INFO:teuthology.orchestra.run.smithi032.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:21:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f788406c8b0 msgr2=0x7f788406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f788406c8b0 0x7f788406ed60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7898066020 tx=0x7f7894009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898101d70 msgr2=0x7f789810e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898101d70 0x7f789810e960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7888002a50 tx=0x7f78880046b0 comp rx=0 tx=0).stop 2024-04-06T14:21:04.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 shutdown_connections 2024-04-06T14:21:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f788406c8b0 0x7f788406ed60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7898101d70 0x7f789810e960 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 --2- 172.21.15.32:0/950772511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7898102710 0x7f789810eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 >> 172.21.15.32:0/950772511 conn(0x7f7898072bc0 msgr2=0x7f78980fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 shutdown_connections 2024-04-06T14:21:04.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:04.784+0000 7f789f86f700 1 -- 172.21.15.32:0/950772511 wait complete. 2024-04-06T14:21:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:05 smithi032 ceph-mon[39730]: pgmap v92: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:05 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:05.585 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-06T14:21:05.585 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:21:05.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-06T14:21:05.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:05 smithi086 ceph-mon[47925]: pgmap v92: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:05.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:05 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:06.233 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:21:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:06 smithi032 ceph-mon[39730]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:06 smithi032 ceph-mon[39730]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:21:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:21:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:06.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:06 smithi032 ceph-mon[39730]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:21:06.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:06 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:21:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:06 smithi086 ceph-mon[47925]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:06 smithi086 ceph-mon[47925]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:21:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:21:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:06 smithi086 ceph-mon[47925]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T14:21:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:06 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:21:07.404 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:07 smithi032 ceph-mon[39730]: pgmap v93: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:07.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:07 smithi086 ceph-mon[47925]: pgmap v93: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.693+0000 7fd8192af700 1 -- 172.21.15.32:0/3506099211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 msgr2=0x7fd8140fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.693+0000 7fd8192af700 1 --2- 172.21.15.32:0/3506099211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 0x7fd8140fbd80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd804004940 tx=0x7fd804009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.693+0000 7fd8192af700 1 -- 172.21.15.32:0/3506099211 shutdown_connections 2024-04-06T14:21:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.693+0000 7fd8192af700 1 --2- 172.21.15.32:0/3506099211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 0x7fd8140fbd80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.693+0000 7fd8192af700 1 --2- 172.21.15.32:0/3506099211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd814101d70 0x7fd814102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.693+0000 7fd8192af700 1 -- 172.21.15.32:0/3506099211 >> 172.21.15.32:0/3506099211 conn(0x7fd814072bc0 msgr2=0x7fd814072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.693+0000 7fd8192af700 1 -- 172.21.15.32:0/3506099211 shutdown_connections 2024-04-06T14:21:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.693+0000 7fd8192af700 1 -- 172.21.15.32:0/3506099211 wait complete. 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8192af700 1 Processor -- start 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8192af700 1 -- start start 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8192af700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd814101d70 0x7fd814103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8192af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 0x7fd814103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8192af700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd814103e10 con 0x7fd814102710 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8192af700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd814103f50 con 0x7fd814101d70 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8127fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 0x7fd814103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8127fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 0x7fd814103880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53658/0 (socket says 172.21.15.32:53658) 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8127fc700 1 -- 172.21.15.32:0/2886753484 learned_addr learned my addr 172.21.15.32:0/2886753484 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd812ffd700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd814101d70 0x7fd814103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8127fc700 1 -- 172.21.15.32:0/2886753484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd814101d70 msgr2=0x7fd814103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8127fc700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd814101d70 0x7fd814103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd8127fc700 1 -- 172.21.15.32:0/2886753484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd804005040 con 0x7fd814102710 2024-04-06T14:21:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.694+0000 7fd812ffd700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd814101d70 0x7fd814103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:21:08.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.695+0000 7fd8127fc700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 0x7fd814103880 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd804009380 tx=0x7fd804009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:08.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.695+0000 7fd7fb7fe700 1 -- 172.21.15.32:0/2886753484 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80400e040 con 0x7fd814102710 2024-04-06T14:21:08.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.695+0000 7fd7fb7fe700 1 -- 172.21.15.32:0/2886753484 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd804013420 con 0x7fd814102710 2024-04-06T14:21:08.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.695+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8141041a0 con 0x7fd814102710 2024-04-06T14:21:08.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.695+0000 7fd7fb7fe700 1 -- 172.21.15.32:0/2886753484 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80401c630 con 0x7fd814102710 2024-04-06T14:21:08.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.695+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81411a1d0 con 0x7fd814102710 2024-04-06T14:21:08.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.696+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8140fd4c0 con 0x7fd814102710 2024-04-06T14:21:08.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.696+0000 7fd7fb7fe700 1 -- 172.21.15.32:0/2886753484 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fd8040139e0 con 0x7fd814102710 2024-04-06T14:21:08.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.697+0000 7fd7fb7fe700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fd7fc06c8a0 0x7fd7fc06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:08.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.697+0000 7fd7fb7fe700 1 -- 172.21.15.32:0/2886753484 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd80408cc50 con 0x7fd814102710 2024-04-06T14:21:08.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.697+0000 7fd812ffd700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fd7fc06c8a0 0x7fd7fc06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:08.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.697+0000 7fd812ffd700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fd7fc06c8a0 0x7fd7fc06ed50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd808009fd0 tx=0x7fd808009ee0 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:08.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.701+0000 7fd7fb7fe700 1 -- 172.21.15.32:0/2886753484 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fd8040571b0 con 0x7fd814102710 2024-04-06T14:21:08.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.924+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd814108d80 con 0x7fd7fc06c8a0 2024-04-06T14:21:08.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.925+0000 7fd7fb7fe700 1 -- 172.21.15.32:0/2886753484 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fd814108d80 con 0x7fd7fc06c8a0 2024-04-06T14:21:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fd7fc06c8a0 msgr2=0x7fd7fc06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fd7fc06c8a0 0x7fd7fc06ed50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd808009fd0 tx=0x7fd808009ee0 comp rx=0 tx=0).stop 2024-04-06T14:21:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 msgr2=0x7fd814103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 0x7fd814103880 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd804009380 tx=0x7fd804009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 shutdown_connections 2024-04-06T14:21:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fd7fc06c8a0 0x7fd7fc06ed50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd814101d70 0x7fd814103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 --2- 172.21.15.32:0/2886753484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102710 0x7fd814103880 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 >> 172.21.15.32:0/2886753484 conn(0x7fd814072bc0 msgr2=0x7fd8141058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 shutdown_connections 2024-04-06T14:21:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:08.928+0000 7fd8192af700 1 -- 172.21.15.32:0/2886753484 wait complete. 2024-04-06T14:21:08.940 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:21:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.035+0000 7fa0c1298700 1 -- 172.21.15.32:0/3951172695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc0fb4a0 msgr2=0x7fa0bc0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.035+0000 7fa0c1298700 1 --2- 172.21.15.32:0/3951172695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc0fb4a0 0x7fa0bc0fb910 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0004940 tx=0x7fa0b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.035+0000 7fa0c1298700 1 -- 172.21.15.32:0/3951172695 shutdown_connections 2024-04-06T14:21:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.035+0000 7fa0c1298700 1 --2- 172.21.15.32:0/3951172695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc0fb4a0 0x7fa0bc0fb910 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.035+0000 7fa0c1298700 1 --2- 172.21.15.32:0/3951172695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc102e70 0x7fa0bc103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.035+0000 7fa0c1298700 1 -- 172.21.15.32:0/3951172695 >> 172.21.15.32:0/3951172695 conn(0x7fa0bc0714c0 msgr2=0x7fa0bc0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.035+0000 7fa0c1298700 1 -- 172.21.15.32:0/3951172695 shutdown_connections 2024-04-06T14:21:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.035+0000 7fa0c1298700 1 -- 172.21.15.32:0/3951172695 wait complete. 2024-04-06T14:21:09.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.036+0000 7fa0c1298700 1 Processor -- start 2024-04-06T14:21:09.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.036+0000 7fa0c1298700 1 -- start start 2024-04-06T14:21:09.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0c1298700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc0fb4a0 0x7fa0bc113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:09.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0c1298700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc102e70 0x7fa0bc113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:09.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0c1298700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc114600 con 0x7fa0bc102e70 2024-04-06T14:21:09.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0c1298700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc10d9f0 con 0x7fa0bc0fb4a0 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0ba7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc102e70 0x7fa0bc113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0ba7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc102e70 0x7fa0bc113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53670/0 (socket says 172.21.15.32:53670) 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0ba7fc700 1 -- 172.21.15.32:0/4266785409 learned_addr learned my addr 172.21.15.32:0/4266785409 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0baffd700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc0fb4a0 0x7fa0bc113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0ba7fc700 1 -- 172.21.15.32:0/4266785409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc0fb4a0 msgr2=0x7fa0bc113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0ba7fc700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc0fb4a0 0x7fa0bc113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0ba7fc700 1 -- 172.21.15.32:0/4266785409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b0005040 con 0x7fa0bc102e70 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0ba7fc700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc102e70 0x7fa0bc113fb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000b3b0 tx=0x7fa0b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.037+0000 7fa0a37fe700 1 -- 172.21.15.32:0/4266785409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0b000e040 con 0x7fa0bc102e70 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.038+0000 7fa0a37fe700 1 -- 172.21.15.32:0/4266785409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0b0013420 con 0x7fa0bc102e70 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.038+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc10dc70 con 0x7fa0bc102e70 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.038+0000 7fa0a37fe700 1 -- 172.21.15.32:0/4266785409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0b001c610 con 0x7fa0bc102e70 2024-04-06T14:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.038+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc10e230 con 0x7fa0bc102e70 2024-04-06T14:21:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.039+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc100110 con 0x7fa0bc102e70 2024-04-06T14:21:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.039+0000 7fa0a37fe700 1 -- 172.21.15.32:0/4266785409 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fa0b000ba80 con 0x7fa0bc102e70 2024-04-06T14:21:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.039+0000 7fa0a37fe700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fa0a406c8a0 0x7fa0a406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.040+0000 7fa0a37fe700 1 -- 172.21.15.32:0/4266785409 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fa0b008cb20 con 0x7fa0bc102e70 2024-04-06T14:21:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.040+0000 7fa0baffd700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fa0a406c8a0 0x7fa0a406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:09.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.040+0000 7fa0baffd700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fa0a406c8a0 0x7fa0a406ed50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac004f40 tx=0x7fa0ac00b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:09.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.044+0000 7fa0a37fe700 1 -- 172.21.15.32:0/4266785409 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa0b005abb0 con 0x7fa0bc102e70 2024-04-06T14:21:09.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.269+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0bc113740 con 0x7fa0a406c8a0 2024-04-06T14:21:09.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.271+0000 7fa0a37fe700 1 -- 172.21.15.32:0/4266785409 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fa0bc113740 con 0x7fa0a406c8a0 2024-04-06T14:21:09.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fa0a406c8a0 msgr2=0x7fa0a406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fa0a406c8a0 0x7fa0a406ed50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac004f40 tx=0x7fa0ac00b040 comp rx=0 tx=0).stop 2024-04-06T14:21:09.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc102e70 msgr2=0x7fa0bc113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc102e70 0x7fa0bc113fb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000b3b0 tx=0x7fa0b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 shutdown_connections 2024-04-06T14:21:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fa0a406c8a0 0x7fa0a406ed50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc0fb4a0 0x7fa0bc113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 --2- 172.21.15.32:0/4266785409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc102e70 0x7fa0bc113fb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 >> 172.21.15.32:0/4266785409 conn(0x7fa0bc0714c0 msgr2=0x7fa0bc0fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 shutdown_connections 2024-04-06T14:21:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.274+0000 7fa0c1298700 1 -- 172.21.15.32:0/4266785409 wait complete. 2024-04-06T14:21:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.379+0000 7f266c112700 1 -- 172.21.15.32:0/3288964329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 msgr2=0x7f26640fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.379+0000 7f266c112700 1 --2- 172.21.15.32:0/3288964329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 0x7f26640fbd80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2654004940 tx=0x7f2654009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.379+0000 7f266c112700 1 -- 172.21.15.32:0/3288964329 shutdown_connections 2024-04-06T14:21:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.379+0000 7f266c112700 1 --2- 172.21.15.32:0/3288964329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 0x7f26640fbd80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.379+0000 7f266c112700 1 --2- 172.21.15.32:0/3288964329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2664101d70 0x7f2664102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.379+0000 7f266c112700 1 -- 172.21.15.32:0/3288964329 >> 172.21.15.32:0/3288964329 conn(0x7f2664072bc0 msgr2=0x7f2664072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.380+0000 7f266c112700 1 -- 172.21.15.32:0/3288964329 shutdown_connections 2024-04-06T14:21:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.380+0000 7f266c112700 1 -- 172.21.15.32:0/3288964329 wait complete. 2024-04-06T14:21:09.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.380+0000 7f266c112700 1 Processor -- start 2024-04-06T14:21:09.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.380+0000 7f266c112700 1 -- start start 2024-04-06T14:21:09.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.380+0000 7f266c112700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2664101d70 0x7f2664112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:09.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.380+0000 7f266c112700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 0x7f2664112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:09.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.380+0000 7f266c112700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2664113490 con 0x7f2664102710 2024-04-06T14:21:09.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.380+0000 7f266c112700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266410c880 con 0x7f2664101d70 2024-04-06T14:21:09.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.380+0000 7f26696ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 0x7f2664112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:09.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f26696ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 0x7f2664112e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53694/0 (socket says 172.21.15.32:53694) 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f26696ad700 1 -- 172.21.15.32:0/1110228178 learned_addr learned my addr 172.21.15.32:0/1110228178 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f2669eae700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2664101d70 0x7f2664112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f26696ad700 1 -- 172.21.15.32:0/1110228178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2664101d70 msgr2=0x7f2664112900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f26696ad700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2664101d70 0x7f2664112900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f26696ad700 1 -- 172.21.15.32:0/1110228178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654005040 con 0x7f2664102710 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f26696ad700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 0x7f2664112e40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2654009180 tx=0x7f2654009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f265a7fc700 1 -- 172.21.15.32:0/1110228178 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f265400e040 con 0x7f2664102710 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f265a7fc700 1 -- 172.21.15.32:0/1110228178 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2654004290 con 0x7f2664102710 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266410cb00 con 0x7f2664102710 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f265a7fc700 1 -- 172.21.15.32:0/1110228178 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2654013420 con 0x7f2664102710 2024-04-06T14:21:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.381+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266410d060 con 0x7f2664102710 2024-04-06T14:21:09.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.382+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26640fd4c0 con 0x7f2664102710 2024-04-06T14:21:09.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.383+0000 7f265a7fc700 1 -- 172.21.15.32:0/1110228178 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f265400bc90 con 0x7f2664102710 2024-04-06T14:21:09.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.383+0000 7f265a7fc700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f265006c8a0 0x7f265006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:09.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.383+0000 7f265a7fc700 1 -- 172.21.15.32:0/1110228178 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f265408f7c0 con 0x7f2664102710 2024-04-06T14:21:09.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.383+0000 7f2669eae700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f265006c8a0 0x7f265006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:09.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.384+0000 7f2669eae700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f265006c8a0 0x7f265006ed50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2660009fd0 tx=0x7f266000b040 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:09.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.388+0000 7f265a7fc700 1 -- 172.21.15.32:0/1110228178 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f265405d7b0 con 0x7f2664102710 2024-04-06T14:21:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: pgmap v94: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: Upgrade: Target is version 18.2.2-1086-g052adcad (reef) 2024-04-06T14:21:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e'] 2024-04-06T14:21:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:21:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: Upgrade: Setting container_image for all mgr 2024-04-06T14:21:09.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:09.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:09 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T14:21:09.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.601+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f26641125d0 con 0x7f265006c8a0 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.611+0000 7f265a7fc700 1 -- 172.21.15.32:0/1110228178 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f26641125d0 con 0x7f265006c8a0 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (92s) 68s ago 12m 15.4M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (13m) 68s ago 13m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (12m) 2m ago 12m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (74s) 68s ago 12m 50.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (7m) 68s ago 7m 13.0M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (6m) 68s ago 6m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (7m) 2m ago 7m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:21:09.611 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (6m) 2m ago 6m 15.5M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (3m) 68s ago 14m 499M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (2m) 2m ago 11m 449M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (14m) 68s ago 14m 54.2M 2048M 17.2.7-819-g6699495c ec300eaa768b 71e43d3091dd 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (11m) 2m ago 11m 43.8M 2048M 17.2.7-819-g6699495c ec300eaa768b 8dd9df00b367 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (2m) 68s ago 12m 9361k - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (2m) 2m ago 11m 3397k - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (11m) 68s ago 11m 193M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 68s ago 10m 191M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (9m) 68s ago 9m 161M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (9m) 2m ago 9m 224M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (9m) 2m ago 9m 202M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (8m) 2m ago 8m 183M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:21:09.612 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (110s) 68s ago 12m 47.6M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:21:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f265006c8a0 msgr2=0x7f265006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f265006c8a0 0x7f265006ed50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2660009fd0 tx=0x7f266000b040 comp rx=0 tx=0).stop 2024-04-06T14:21:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 msgr2=0x7f2664112e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 0x7f2664112e40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2654009180 tx=0x7f2654009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 shutdown_connections 2024-04-06T14:21:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f265006c8a0 0x7f265006ed50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2664101d70 0x7f2664112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 --2- 172.21.15.32:0/1110228178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2664102710 0x7f2664112e40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 >> 172.21.15.32:0/1110228178 conn(0x7f2664072bc0 msgr2=0x7f26640faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 shutdown_connections 2024-04-06T14:21:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.613+0000 7f266c112700 1 -- 172.21.15.32:0/1110228178 wait complete. 2024-04-06T14:21:09.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: pgmap v94: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:09.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: Upgrade: Target is version 18.2.2-1086-g052adcad (reef) 2024-04-06T14:21:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e'] 2024-04-06T14:21:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:21:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: Upgrade: Setting container_image for all mgr 2024-04-06T14:21:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:09.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:09 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T14:21:09.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.716+0000 7fb95b2d3700 1 -- 172.21.15.32:0/3123924927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fdde0 msgr2=0x7fb9540fe1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.716+0000 7fb95b2d3700 1 --2- 172.21.15.32:0/3123924927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fdde0 0x7fb9540fe1b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb944009ce0 tx=0x7fb944004d10 comp rx=0 tx=0).stop 2024-04-06T14:21:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.716+0000 7fb95b2d3700 1 -- 172.21.15.32:0/3123924927 shutdown_connections 2024-04-06T14:21:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.716+0000 7fb95b2d3700 1 --2- 172.21.15.32:0/3123924927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9540fe780 0x7fb954106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.716+0000 7fb95b2d3700 1 --2- 172.21.15.32:0/3123924927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fdde0 0x7fb9540fe1b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.716+0000 7fb95b2d3700 1 -- 172.21.15.32:0/3123924927 >> 172.21.15.32:0/3123924927 conn(0x7fb9540f7890 msgr2=0x7fb9540f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.716+0000 7fb95b2d3700 1 -- 172.21.15.32:0/3123924927 shutdown_connections 2024-04-06T14:21:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.716+0000 7fb95b2d3700 1 -- 172.21.15.32:0/3123924927 wait complete. 2024-04-06T14:21:09.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95b2d3700 1 Processor -- start 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95b2d3700 1 -- start start 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95b2d3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9540fdde0 0x7fb9540fbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95b2d3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fe780 0x7fb9540fc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95b2d3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540fdb60 con 0x7fb9540fe780 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95b2d3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540fc690 con 0x7fb9540fdde0 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95886e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fe780 0x7fb9540fc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95886e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fe780 0x7fb9540fc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53712/0 (socket says 172.21.15.32:53712) 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95886e700 1 -- 172.21.15.32:0/2054088223 learned_addr learned my addr 172.21.15.32:0/2054088223 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95906f700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9540fdde0 0x7fb9540fbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95886e700 1 -- 172.21.15.32:0/2054088223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9540fdde0 msgr2=0x7fb9540fbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.717+0000 7fb95886e700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9540fdde0 0x7fb9540fbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.718+0000 7fb95886e700 1 -- 172.21.15.32:0/2054088223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944009940 con 0x7fb9540fe780 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.718+0000 7fb95886e700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fe780 0x7fb9540fc150 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb94800c810 tx=0x7fb948004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:09.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.718+0000 7fb951ffb700 1 -- 172.21.15.32:0/2054088223 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb948012810 con 0x7fb9540fe780 2024-04-06T14:21:09.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.718+0000 7fb951ffb700 1 -- 172.21.15.32:0/2054088223 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb948012e20 con 0x7fb9540fe780 2024-04-06T14:21:09.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.718+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9540fc970 con 0x7fb9540fe780 2024-04-06T14:21:09.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.718+0000 7fb951ffb700 1 -- 172.21.15.32:0/2054088223 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb94800a370 con 0x7fb9540fe780 2024-04-06T14:21:09.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.718+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb954119ff0 con 0x7fb9540fe780 2024-04-06T14:21:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.719+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb954066bd0 con 0x7fb9540fe780 2024-04-06T14:21:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.719+0000 7fb951ffb700 1 -- 172.21.15.32:0/2054088223 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fb948011780 con 0x7fb9540fe780 2024-04-06T14:21:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.720+0000 7fb951ffb700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb93c06c8a0 0x7fb93c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.720+0000 7fb951ffb700 1 -- 172.21.15.32:0/2054088223 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb94808bf10 con 0x7fb9540fe780 2024-04-06T14:21:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.720+0000 7fb95906f700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb93c06c8a0 0x7fb93c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.720+0000 7fb95906f700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb93c06c8a0 0x7fb93c06ed50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb944009f90 tx=0x7fb944004d10 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:09.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:09.724+0000 7fb951ffb700 1 -- 172.21.15.32:0/2054088223 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fb9480563f0 con 0x7fb9540fe780 2024-04-06T14:21:10.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.015+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb95411a240 con 0x7fb9540fe780 2024-04-06T14:21:10.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.015+0000 7fb951ffb700 1 -- 172.21.15.32:0/2054088223 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7fb948059a10 con 0x7fb9540fe780 2024-04-06T14:21:10.019 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 12, 2024-04-06T14:21:10.021 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:21:10.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:21:10.021 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:21:10.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.017+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb93c06c8a0 msgr2=0x7fb93c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.017+0000 7fb95b2d3700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb93c06c8a0 0x7fb93c06ed50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb944009f90 tx=0x7fb944004d10 comp rx=0 tx=0).stop 2024-04-06T14:21:10.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.017+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fe780 msgr2=0x7fb9540fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.017+0000 7fb95b2d3700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fe780 0x7fb9540fc150 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb94800c810 tx=0x7fb948004470 comp rx=0 tx=0).stop 2024-04-06T14:21:10.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.018+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 shutdown_connections 2024-04-06T14:21:10.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.018+0000 7fb95b2d3700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fb93c06c8a0 0x7fb93c06ed50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.018+0000 7fb95b2d3700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9540fdde0 0x7fb9540fbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.018+0000 7fb95b2d3700 1 --2- 172.21.15.32:0/2054088223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9540fe780 0x7fb9540fc150 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.018+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 >> 172.21.15.32:0/2054088223 conn(0x7fb9540f7890 msgr2=0x7fb9540f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:10.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.018+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 shutdown_connections 2024-04-06T14:21:10.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.018+0000 7fb95b2d3700 1 -- 172.21.15.32:0/2054088223 wait complete. 2024-04-06T14:21:10.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.126+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/102345303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8101d70 msgr2=0x7fc3d8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.126+0000 7fc3dcb7a700 1 --2- 172.21.15.32:0/102345303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8101d70 0x7fc3d8102140 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8004940 tx=0x7fc3c8009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.126+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/102345303 shutdown_connections 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.126+0000 7fc3dcb7a700 1 --2- 172.21.15.32:0/102345303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d8102710 0x7fc3d80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.126+0000 7fc3dcb7a700 1 --2- 172.21.15.32:0/102345303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8101d70 0x7fc3d8102140 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.126+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/102345303 >> 172.21.15.32:0/102345303 conn(0x7fc3d8072bc0 msgr2=0x7fc3d8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.126+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/102345303 shutdown_connections 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.126+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/102345303 wait complete. 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3dcb7a700 1 Processor -- start 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3dcb7a700 1 -- start start 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3dcb7a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d8101d70 0x7fc3d8102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3dcb7a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8102710 0x7fc3d8103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3dcb7a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8103650 con 0x7fc3d8102710 2024-04-06T14:21:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3dcb7a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8103790 con 0x7fc3d8101d70 2024-04-06T14:21:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3d5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8102710 0x7fc3d8103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3d5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8102710 0x7fc3d8103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53726/0 (socket says 172.21.15.32:53726) 2024-04-06T14:21:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3d5d9b700 1 -- 172.21.15.32:0/3968582869 learned_addr learned my addr 172.21.15.32:0/3968582869 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3d659c700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d8101d70 0x7fc3d8102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3d5d9b700 1 -- 172.21.15.32:0/3968582869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d8101d70 msgr2=0x7fc3d8102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3d5d9b700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d8101d70 0x7fc3d8102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.127+0000 7fc3d5d9b700 1 -- 172.21.15.32:0/3968582869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8005040 con 0x7fc3d8102710 2024-04-06T14:21:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.128+0000 7fc3d5d9b700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8102710 0x7fc3d8103110 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00c890 tx=0x7fc3cc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:10.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.128+0000 7fc3c6ffd700 1 -- 172.21.15.32:0/3968582869 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3cc00a7d0 con 0x7fc3d8102710 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.128+0000 7fc3c6ffd700 1 -- 172.21.15.32:0/3968582869 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3cc00ade0 con 0x7fc3d8102710 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.128+0000 7fc3c6ffd700 1 -- 172.21.15.32:0/3968582869 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3cc0113a0 con 0x7fc3d8102710 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.128+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d8103a40 con 0x7fc3d8102710 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.128+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d811a210 con 0x7fc3d8102710 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.129+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d80fd4c0 con 0x7fc3d8102710 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.129+0000 7fc3c6ffd700 1 -- 172.21.15.32:0/3968582869 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fc3cc012400 con 0x7fc3d8102710 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.130+0000 7fc3c6ffd700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc3c006c8a0 0x7fc3c006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.130+0000 7fc3c6ffd700 1 -- 172.21.15.32:0/3968582869 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc3cc08b9c0 con 0x7fc3d8102710 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.130+0000 7fc3d659c700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc3c006c8a0 0x7fc3c006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.130+0000 7fc3d659c700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc3c006c8a0 0x7fc3c006ed50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8004f90 tx=0x7fc3c8009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:10.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.135+0000 7fc3c6ffd700 1 -- 172.21.15.32:0/3968582869 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc3cc055f20 con 0x7fc3d8102710 2024-04-06T14:21:10.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.390+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc3d8108d60 con 0x7fc3d8102710 2024-04-06T14:21:10.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.390+0000 7fc3c6ffd700 1 -- 172.21.15.32:0/3968582869 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1847 (secure 0 0 0) 0x7fc3cc059540 con 0x7fc3d8102710 2024-04-06T14:21:10.390 INFO:teuthology.orchestra.run.smithi032.stdout:e20 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 20 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:17:08.851867+0000 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:21:10.391 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:21:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:21:10.393 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:10.393 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:10.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:21:10.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:21:10.393 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:21:10.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:21:10.393 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:10.393 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.392+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc3c006c8a0 msgr2=0x7fc3c006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.392+0000 7fc3dcb7a700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc3c006c8a0 0x7fc3c006ed50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8004f90 tx=0x7fc3c8009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.392+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8102710 msgr2=0x7fc3d8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.392+0000 7fc3dcb7a700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8102710 0x7fc3d8103110 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00c890 tx=0x7fc3cc0044c0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.392+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 shutdown_connections 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.392+0000 7fc3dcb7a700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc3c006c8a0 0x7fc3c006ed50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.392+0000 7fc3dcb7a700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d8101d70 0x7fc3d8102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.392+0000 7fc3dcb7a700 1 --2- 172.21.15.32:0/3968582869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d8102710 0x7fc3d8103110 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.392+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 >> 172.21.15.32:0/3968582869 conn(0x7fc3d8072bc0 msgr2=0x7fc3d8065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.393+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 shutdown_connections 2024-04-06T14:21:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.393+0000 7fc3dcb7a700 1 -- 172.21.15.32:0/3968582869 wait complete. 2024-04-06T14:21:10.398 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 20 2024-04-06T14:21:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:10 smithi032 ceph-mon[39730]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:10 smithi032 ceph-mon[39730]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:10 smithi032 ceph-mon[39730]: from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:10 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/2054088223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:21:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:10 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:10 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:21:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:10 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:21:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:10 smithi032 ceph-mon[39730]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:10.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.497+0000 7fc025088700 1 -- 172.21.15.32:0/3339144126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 msgr2=0x7fc0200fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.497+0000 7fc025088700 1 --2- 172.21.15.32:0/3339144126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 0x7fc0200fb910 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc014004940 tx=0x7fc014009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:10.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.497+0000 7fc025088700 1 -- 172.21.15.32:0/3339144126 shutdown_connections 2024-04-06T14:21:10.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.497+0000 7fc025088700 1 --2- 172.21.15.32:0/3339144126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 0x7fc0200fb910 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.497+0000 7fc025088700 1 --2- 172.21.15.32:0/3339144126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc020102e70 0x7fc020103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.497+0000 7fc025088700 1 -- 172.21.15.32:0/3339144126 >> 172.21.15.32:0/3339144126 conn(0x7fc0200714c0 msgr2=0x7fc0200718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:10.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.497+0000 7fc025088700 1 -- 172.21.15.32:0/3339144126 shutdown_connections 2024-04-06T14:21:10.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.497+0000 7fc025088700 1 -- 172.21.15.32:0/3339144126 wait complete. 2024-04-06T14:21:10.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.497+0000 7fc025088700 1 Processor -- start 2024-04-06T14:21:10.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc025088700 1 -- start start 2024-04-06T14:21:10.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc025088700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 0x7fc0200fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc025088700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc020102e70 0x7fc0200fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc025088700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200fd150 con 0x7fc0200fb4a0 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc025088700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200fd290 con 0x7fc020102e70 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc01ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 0x7fc0200fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc01ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 0x7fc0200fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53736/0 (socket says 172.21.15.32:53736) 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc01ed9d700 1 -- 172.21.15.32:0/3539591413 learned_addr learned my addr 172.21.15.32:0/3539591413 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc01e59c700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc020102e70 0x7fc0200fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc01ed9d700 1 -- 172.21.15.32:0/3539591413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc020102e70 msgr2=0x7fc0200fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc01ed9d700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc020102e70 0x7fc0200fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc01ed9d700 1 -- 172.21.15.32:0/3539591413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014005040 con 0x7fc0200fb4a0 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc01ed9d700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 0x7fc0200fc680 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc01000c830 tx=0x7fc010004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.498+0000 7fc0077fe700 1 -- 172.21.15.32:0/3539591413 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0100106f0 con 0x7fc0200fb4a0 2024-04-06T14:21:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.499+0000 7fc0077fe700 1 -- 172.21.15.32:0/3539591413 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc010010d00 con 0x7fc0200fb4a0 2024-04-06T14:21:10.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.499+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc020119e50 con 0x7fc0200fb4a0 2024-04-06T14:21:10.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.499+0000 7fc0077fe700 1 -- 172.21.15.32:0/3539591413 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc010012e90 con 0x7fc0200fb4a0 2024-04-06T14:21:10.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.499+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02011a250 con 0x7fc0200fb4a0 2024-04-06T14:21:10.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.499+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc020100170 con 0x7fc0200fb4a0 2024-04-06T14:21:10.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.500+0000 7fc0077fe700 1 -- 172.21.15.32:0/3539591413 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fc010010850 con 0x7fc0200fb4a0 2024-04-06T14:21:10.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.500+0000 7fc0077fe700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc00806c8a0 0x7fc00806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:10.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.501+0000 7fc0077fe700 1 -- 172.21.15.32:0/3539591413 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc01008b9a0 con 0x7fc0200fb4a0 2024-04-06T14:21:10.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.501+0000 7fc01e59c700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc00806c8a0 0x7fc00806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:10.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.501+0000 7fc01e59c700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc00806c8a0 0x7fc00806ed50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc0200fdfc0 tx=0x7fc014009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:10.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.505+0000 7fc0077fe700 1 -- 172.21.15.32:0/3539591413 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc010055fa0 con 0x7fc0200fb4a0 2024-04-06T14:21:10.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:10 smithi086 ceph-mon[47925]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:10.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:10 smithi086 ceph-mon[47925]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:10.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:10 smithi086 ceph-mon[47925]: from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:10.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:10 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/2054088223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:21:10.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:10 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:10.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:10 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:21:10.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:10 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:21:10.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:10 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:10.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.735+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 --> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc02004bed0 con 0x7fc00806c8a0 2024-04-06T14:21:10.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.736+0000 7fc0077fe700 1 -- 172.21.15.32:0/3539591413 <== mgr.24469 v2:172.21.15.32:6800/2780016783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fc02004bed0 con 0x7fc00806c8a0 2024-04-06T14:21:10.736 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:21:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:21:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:21:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:21:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:21:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-06T14:21:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:21:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/21 daemons upgraded", 2024-04-06T14:21:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mon daemons", 2024-04-06T14:21:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:21:10.737 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:21:10.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc00806c8a0 msgr2=0x7fc00806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc00806c8a0 0x7fc00806ed50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc0200fdfc0 tx=0x7fc014009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 msgr2=0x7fc0200fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 0x7fc0200fc680 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc01000c830 tx=0x7fc010004470 comp rx=0 tx=0).stop 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 shutdown_connections 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7fc00806c8a0 0x7fc00806ed50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0200fb4a0 0x7fc0200fc680 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 --2- 172.21.15.32:0/3539591413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc020102e70 0x7fc0200fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 >> 172.21.15.32:0/3539591413 conn(0x7fc0200714c0 msgr2=0x7fc0200fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 shutdown_connections 2024-04-06T14:21:10.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.738+0000 7fc025088700 1 -- 172.21.15.32:0/3539591413 wait complete. 2024-04-06T14:21:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.842+0000 7f37973bf700 1 -- 172.21.15.32:0/2058801903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fc4e0 msgr2=0x7f3790109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.842+0000 7f37973bf700 1 --2- 172.21.15.32:0/2058801903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fc4e0 0x7f3790109240 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3780004970 tx=0x7f3780009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 -- 172.21.15.32:0/2058801903 shutdown_connections 2024-04-06T14:21:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 --2- 172.21.15.32:0/2058801903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fc4e0 0x7f3790109240 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 --2- 172.21.15.32:0/2058801903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37900fbbd0 0x7f37900fbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 -- 172.21.15.32:0/2058801903 >> 172.21.15.32:0/2058801903 conn(0x7f37900f7860 msgr2=0x7f37900f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 -- 172.21.15.32:0/2058801903 shutdown_connections 2024-04-06T14:21:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 -- 172.21.15.32:0/2058801903 wait complete. 2024-04-06T14:21:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 Processor -- start 2024-04-06T14:21:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 -- start start 2024-04-06T14:21:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fbbd0 0x7f3790115c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37900fc4e0 0x7f3790116170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37901167c0 con 0x7f37900fbbd0 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.843+0000 7f37973bf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379010fbe0 con 0x7f37900fc4e0 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f379515b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fbbd0 0x7f3790115c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f379515b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fbbd0 0x7f3790115c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53756/0 (socket says 172.21.15.32:53756) 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f379515b700 1 -- 172.21.15.32:0/4118836517 learned_addr learned my addr 172.21.15.32:0/4118836517 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f379495a700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37900fc4e0 0x7f3790116170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f379515b700 1 -- 172.21.15.32:0/4118836517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37900fc4e0 msgr2=0x7f3790116170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f379515b700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37900fc4e0 0x7f3790116170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f379515b700 1 -- 172.21.15.32:0/4118836517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780005040 con 0x7f37900fbbd0 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f379495a700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37900fc4e0 0x7f3790116170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f379515b700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fbbd0 0x7f3790115c30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f378c002a50 tx=0x7f378c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f3785ffb700 1 -- 172.21.15.32:0/4118836517 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f378c007cc0 con 0x7f37900fbbd0 2024-04-06T14:21:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f3785ffb700 1 -- 172.21.15.32:0/4118836517 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f378c00f040 con 0x7f37900fbbd0 2024-04-06T14:21:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379010fec0 con 0x7f37900fbbd0 2024-04-06T14:21:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f3785ffb700 1 -- 172.21.15.32:0/4118836517 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f378c009540 con 0x7f37900fbbd0 2024-04-06T14:21:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.844+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3790110450 con 0x7f37900fbbd0 2024-04-06T14:21:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.845+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3790107080 con 0x7f37900fbbd0 2024-04-06T14:21:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.846+0000 7f3785ffb700 1 -- 172.21.15.32:0/4118836517 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f378c0096a0 con 0x7f37900fbbd0 2024-04-06T14:21:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.846+0000 7f3785ffb700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f377c06c8a0 0x7f377c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.846+0000 7f3785ffb700 1 -- 172.21.15.32:0/4118836517 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f378c08b8b0 con 0x7f37900fbbd0 2024-04-06T14:21:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.846+0000 7f379495a700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f377c06c8a0 0x7f377c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.847+0000 7f379495a700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f377c06c8a0 0x7f377c06ed50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3780004f50 tx=0x7f3780009e30 comp rx=0 tx=0).ready entity=mgr.24469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:10.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:10.851+0000 7f3785ffb700 1 -- 172.21.15.32:0/4118836517 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f378c059820 con 0x7f37900fbbd0 2024-04-06T14:21:11.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.138+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f379004bed0 con 0x7f37900fbbd0 2024-04-06T14:21:11.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.139+0000 7f3785ffb700 1 -- 172.21.15.32:0/4118836517 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f378c0593b0 con 0x7f37900fbbd0 2024-04-06T14:21:11.139 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:11.139 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:11.139 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:21:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f377c06c8a0 msgr2=0x7f377c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f377c06c8a0 0x7f377c06ed50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3780004f50 tx=0x7f3780009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fbbd0 msgr2=0x7f3790115c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fbbd0 0x7f3790115c30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f378c002a50 tx=0x7f378c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:21:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 shutdown_connections 2024-04-06T14:21:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:6800/2780016783,v1:172.21.15.32:6801/2780016783] conn(0x7f377c06c8a0 0x7f377c06ed50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37900fbbd0 0x7f3790115c30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 --2- 172.21.15.32:0/4118836517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37900fc4e0 0x7f3790116170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 >> 172.21.15.32:0/4118836517 conn(0x7f37900f7860 msgr2=0x7f3790104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 shutdown_connections 2024-04-06T14:21:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:11.141+0000 7f37973bf700 1 -- 172.21.15.32:0/4118836517 wait complete. 2024-04-06T14:21:11.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:11 smithi032 ceph-mon[39730]: pgmap v95: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:11.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:11 smithi032 ceph-mon[39730]: Upgrade: Updating mon.smithi032 2024-04-06T14:21:11.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:11 smithi032 ceph-mon[39730]: Deploying daemon mon.smithi032 on smithi032 2024-04-06T14:21:11.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:11 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/3968582869' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:21:11.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:11 smithi032 ceph-mon[39730]: from='client.? 172.21.15.32:0/4118836517' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:21:11.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:11 smithi086 ceph-mon[47925]: pgmap v95: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:11.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:11 smithi086 ceph-mon[47925]: Upgrade: Updating mon.smithi032 2024-04-06T14:21:11.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:11 smithi086 ceph-mon[47925]: Deploying daemon mon.smithi032 on smithi032 2024-04-06T14:21:11.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:11 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/3968582869' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:21:11.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:11 smithi086 ceph-mon[47925]: from='client.? 172.21.15.32:0/4118836517' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:21:12.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:12 smithi032 systemd[1]: Stopping Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:21:12.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:12 smithi032 ceph-mon[39730]: from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:12.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:12 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[39707]: 2024-04-06T14:21:12.299+0000 7fb103a5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi032 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T14:21:12.308 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:12 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[39707]: 2024-04-06T14:21:12.299+0000 7fb103a5e700 -1 mon.smithi032@0(leader) e2 *** Got Signal Terminated *** 2024-04-06T14:21:12.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:12 smithi086 ceph-mon[47925]: from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:12.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:12 smithi032 bash[147210]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032 2024-04-06T14:21:13.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi032.service: Succeeded. 2024-04-06T14:21:13.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 systemd[1]: Stopped Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:21:13.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 systemd[1]: Starting Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:21:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T14:21:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable), process ceph-mon, pid 7 2024-04-06T14:21:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: pidfile_write: ignore empty --pid-file 2024-04-06T14:21:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: load: jerasure load: lrc 2024-04-06T14:21:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: RocksDB version: 7.9.2 2024-04-06T14:21:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Git sha 0 2024-04-06T14:21:14.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Compile date 2024-04-03 02:24:10 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: DB SUMMARY 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: DB Session ID: IHPJVIXV600838ZCODXK 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: CURRENT file: CURRENT 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1166 Bytes 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi032/store.db dir, Total Num: 1, files: 000033.sst 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi032/store.db: 000031.log size: 2276976 ; 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.error_if_exists: 0 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.create_if_missing: 0 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.paranoid_checks: 1 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.env: 0x5625985907a0 2024-04-06T14:21:14.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.fs: PosixFileSystem 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.info_log: 0x56259aa15380 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.statistics: (nil) 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.use_fsync: 0 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_log_file_size: 0 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.allow_fallocate: 1 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.use_direct_reads: 0 2024-04-06T14:21:14.233 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.db_log_dir: 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.wal_dir: 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.write_buffer_manager: 0x562599c8c140 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T14:21:14.234 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.unordered_write: 0 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.row_cache: None 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.wal_filter: None 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T14:21:14.235 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.two_write_queues: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.wal_compression: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.atomic_flush: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.log_readahead_size: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T14:21:14.236 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_background_jobs: 2 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_background_compactions: -1 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_subcompactions: 1 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_open_files: -1 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T14:21:14.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_background_flushes: -1 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Compression algorithms supported: 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: kZSTD supported: 0 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: kXpressCompression supported: 0 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: kZlibCompression supported: 1 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: kSnappyCompression supported: 1 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: kLZ4Compression supported: 1 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: kBZip2Compression supported: 0 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T14:21:14.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000015 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.merge_operator: 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_filter: None 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56259aa154a0) 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks: 1 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_top_level_index_and_filter: 1 2024-04-06T14:21:14.239 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_type: 0 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_index_type: 0 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_shortening: 1 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: checksum: 4 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: no_block_cache: 0 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache: 0x562599d2d1f0 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_name: BinnedLRUCache 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_options: 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: capacity : 536870912 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: num_shard_bits : 4 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: strict_capacity_limit : 0 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: high_pri_pool_ratio: 0.000 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_compressed: (nil) 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: persistent_cache: (nil) 2024-04-06T14:21:14.240 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size: 4096 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size_deviation: 10 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_restart_interval: 16 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_block_restart_interval: 1 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: metadata_block_size: 4096 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: partition_filters: 0 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: use_delta_encoding: 1 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: filter_policy: bloomfilter 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: whole_key_filtering: 1 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: verify_compression: 0 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: read_amp_bytes_per_bit: 0 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: format_version: 5 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_index_compression: 1 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_align: 0 2024-04-06T14:21:14.241 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_auto_readahead_size: 262144 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: prepopulate_block_cache: 0 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: initial_auto_readahead_size: 8192 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: num_file_reads_for_auto_readahead: 2 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression: NoCompression 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.num_levels: 7 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T14:21:14.242 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T14:21:14.243 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T14:21:14.244 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.arena_block_size: 1048576 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T14:21:14.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T14:21:14.246 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T14:21:14.246 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T14:21:14.246 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.table_properties_collectors: 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.inplace_update_support: 0 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.bloom_locality: 0 2024-04-06T14:21:14.247 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.max_successive_merges: 0 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.ttl: 2592000 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.enable_blob_files: false 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.min_blob_size: 0 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T14:21:14.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.blob_file_starting_level: 0 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 11979, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 64bbb27e-29c9-44ab-a860-51947a052ba2 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712413273913342, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712413273921091, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 2002932, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11980, "largest_seqno": 13769, "table_properties": {"data_size": 1994807, "index_size": 5125, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1989, "raw_key_size": 18430, "raw_average_key_size": 23, "raw_value_size": 1978055, "raw_average_value_size": 2513, "num_data_blocks": 235, "num_entries": 787, "num_filter_entries": 787, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712413273, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "64bbb27e-29c9-44ab-a860-51947a052ba2", "db_session_id": "IHPJVIXV600838ZCODXK", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712413273921163, "job": 1, "event": "recovery_finished"} 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T14:21:14.249 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi032/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T14:21:14.250 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562599dca000 2024-04-06T14:21:14.250 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: DB pointer 0x562599db8000 2024-04-06T14:21:14.250 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T14:21:14.250 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T14:21:14.250 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** DB Stats ** 2024-04-06T14:21:14.250 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:21:14.250 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T14:21:14.250 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L0 1/0 1.91 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 522.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L6 1/0 6.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Sum 2/0 8.59 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 522.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 522.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:21:14.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 522.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative compaction: 0.00 GB write, 188.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:21:14.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval compaction: 0.00 GB write, 188.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Block cache BinnedLRUCache@0x562599d2d1f0#7 capacity: 512.00 MB usage: 8.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: starting mon.smithi032 rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi032 fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: mon.smithi032@-1(???) e2 preinit fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: mon.smithi032@-1(???).mds e20 new map 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: mon.smithi032@-1(???).mds e20 print_map 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: e20 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: legacy client fscid: 1 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Filesystem 'cephfs' (1) 2024-04-06T14:21:14.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: fs_name cephfs 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: epoch 20 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: modified 2024-04-06T14:17:08.851867+0000 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: tableserver 0 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: root 0 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: session_timeout 60 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: session_autoclose 300 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_file_size 1099511627776 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_xattr_size 65536 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: required_client_features {} 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: last_failure 0 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: last_failure_osd_epoch 46 2024-04-06T14:21:14.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_mds 2 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: in 0,1 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: up {0=14460,1=24301} 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: failed 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: damaged 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: stopped 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_pools [3] 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: metadata_pool 2 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: inline_data enabled 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: balancer 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: bal_rank_mask -1 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: standby_count_wanted 1 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:14.255 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Standby daemons: 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: mon.smithi032@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: mon.smithi032@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: mon.smithi032@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: mon.smithi032@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 ceph-mon[147414]: mon.smithi032@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 bash[147377]: 2c109e84844d03f78c1656ff65f493795e35f3ad932e5ad0c95ad65f5e846416 2024-04-06T14:21:14.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:13 smithi032 systemd[1]: Started Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:21:15.706 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: pgmap v97: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: mon.smithi032 calling monitor election 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: mon.smithi032 is new leader, mons smithi032,smithi086 in quorum (ranks 0,1) 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: osdmap e49: 6 total, 6 up, 6 in 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: mgrmap e33: smithi032.peohxu(active, since 3m), standbys: smithi086.fyyhxm 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: mgrmap e34: smithi032.peohxu(active, since 3m), standbys: smithi086.fyyhxm 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:21:15.707 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:15 smithi086 ceph-mon[47925]: from='mgr.24469 ' entity='' 2024-04-06T14:21:15.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:21:15.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: from='mgr.24469 172.21.15.32:0/1653852631' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:21:15.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: pgmap v97: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:15.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: mon.smithi032 calling monitor election 2024-04-06T14:21:15.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: mon.smithi032 is new leader, mons smithi032,smithi086 in quorum (ranks 0,1) 2024-04-06T14:21:15.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T14:21:15.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:21:15.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: osdmap e49: 6 total, 6 up, 6 in 2024-04-06T14:21:15.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: mgrmap e33: smithi032.peohxu(active, since 3m), standbys: smithi086.fyyhxm 2024-04-06T14:21:15.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: mgrmap e34: smithi032.peohxu(active, since 3m), standbys: smithi086.fyyhxm 2024-04-06T14:21:15.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:15.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:15.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:21:15.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:15 smithi032 ceph-mon[147414]: from='mgr.24469 ' entity='' 2024-04-06T14:21:18.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:18 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:21:18.659 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:18 smithi032 ceph-mon[147414]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:21:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:22 smithi032 ceph-mon[147414]: Standby manager daemon smithi086.fyyhxm restarted 2024-04-06T14:21:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:22 smithi032 ceph-mon[147414]: Standby manager daemon smithi086.fyyhxm started 2024-04-06T14:21:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:22 smithi032 ceph-mon[147414]: from='mgr.? 172.21.15.86:0/1707672796' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:21:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:22 smithi032 ceph-mon[147414]: from='mgr.? 172.21.15.86:0/1707672796' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:21:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:22 smithi032 ceph-mon[147414]: from='mgr.? 172.21.15.86:0/1707672796' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:21:22.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:22 smithi032 ceph-mon[147414]: from='mgr.? 172.21.15.86:0/1707672796' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:21:22.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:22 smithi086 ceph-mon[47925]: Standby manager daemon smithi086.fyyhxm restarted 2024-04-06T14:21:22.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:22 smithi086 ceph-mon[47925]: Standby manager daemon smithi086.fyyhxm started 2024-04-06T14:21:22.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:22 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1707672796' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/crt"}]: dispatch 2024-04-06T14:21:22.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:22 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1707672796' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T14:21:22.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:22 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1707672796' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.fyyhxm/key"}]: dispatch 2024-04-06T14:21:22.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:22 smithi086 ceph-mon[47925]: from='mgr.? 172.21.15.86:0/1707672796' entity='mgr.smithi086.fyyhxm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T14:21:23.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: mgrmap e35: smithi032.peohxu(active, since 3m), standbys: smithi086.fyyhxm 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: Active manager daemon smithi032.peohxu restarted 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: Activating manager daemon smithi032.peohxu 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: osdmap e50: 6 total, 6 up, 6 in 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: mgrmap e36: smithi032.peohxu(active, starting, since 0.00550818s), standbys: smithi086.fyyhxm 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:21:23.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi086.fyyhxm", "id": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: Manager daemon smithi032.peohxu is now available 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:21:23.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/trash_purge_schedule"}]: dispatch 2024-04-06T14:21:23.601 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: mgrmap e35: smithi032.peohxu(active, since 3m), standbys: smithi086.fyyhxm 2024-04-06T14:21:23.601 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: Active manager daemon smithi032.peohxu restarted 2024-04-06T14:21:23.601 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: Activating manager daemon smithi032.peohxu 2024-04-06T14:21:23.601 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: osdmap e50: 6 total, 6 up, 6 in 2024-04-06T14:21:23.601 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: mgrmap e36: smithi032.peohxu(active, starting, since 0.00550818s), standbys: smithi086.fyyhxm 2024-04-06T14:21:23.601 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:21:23.601 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi032.peohxu", "id": "smithi032.peohxu"}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr metadata", "who": "smithi086.fyyhxm", "id": "smithi086.fyyhxm"}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T14:21:23.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T14:21:23.603 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T14:21:23.603 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: Manager daemon smithi032.peohxu is now available 2024-04-06T14:21:23.603 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:23.603 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:21:23.603 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/mirror_snapshot_schedule"}]: dispatch 2024-04-06T14:21:23.603 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:23 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.peohxu/trash_purge_schedule"}]: dispatch 2024-04-06T14:21:24.434 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:24 smithi032 ceph-mon[147414]: [06/Apr/2024:14:21:23] ENGINE Bus STARTING 2024-04-06T14:21:24.434 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:24 smithi032 ceph-mon[147414]: [06/Apr/2024:14:21:23] ENGINE Serving on https://172.21.15.32:7150 2024-04-06T14:21:24.434 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:24 smithi032 ceph-mon[147414]: [06/Apr/2024:14:21:23] ENGINE Serving on http://172.21.15.32:8765 2024-04-06T14:21:24.434 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:24 smithi032 ceph-mon[147414]: [06/Apr/2024:14:21:23] ENGINE Bus STARTED 2024-04-06T14:21:24.434 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:24 smithi032 ceph-mon[147414]: mgrmap e37: smithi032.peohxu(active, since 1.0103s), standbys: smithi086.fyyhxm 2024-04-06T14:21:24.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:24 smithi086 ceph-mon[47925]: [06/Apr/2024:14:21:23] ENGINE Bus STARTING 2024-04-06T14:21:24.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:24 smithi086 ceph-mon[47925]: [06/Apr/2024:14:21:23] ENGINE Serving on https://172.21.15.32:7150 2024-04-06T14:21:24.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:24 smithi086 ceph-mon[47925]: [06/Apr/2024:14:21:23] ENGINE Serving on http://172.21.15.32:8765 2024-04-06T14:21:24.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:24 smithi086 ceph-mon[47925]: [06/Apr/2024:14:21:23] ENGINE Bus STARTED 2024-04-06T14:21:24.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:24 smithi086 ceph-mon[47925]: mgrmap e37: smithi032.peohxu(active, since 1.0103s), standbys: smithi086.fyyhxm 2024-04-06T14:21:25.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:25 smithi032 ceph-mon[147414]: pgmap v3: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:25.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:25 smithi086 ceph-mon[47925]: pgmap v3: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:26 smithi032 ceph-mon[147414]: pgmap v4: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:26 smithi032 ceph-mon[147414]: mgrmap e38: smithi032.peohxu(active, since 2s), standbys: smithi086.fyyhxm 2024-04-06T14:21:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:26 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:26 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:26.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:26 smithi086 ceph-mon[47925]: pgmap v4: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:26.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:26 smithi086 ceph-mon[47925]: mgrmap e38: smithi032.peohxu(active, since 2s), standbys: smithi086.fyyhxm 2024-04-06T14:21:26.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:26 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:26.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:26 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:27.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:27 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:27.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:27 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:27.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:27 smithi032 ceph-mon[147414]: pgmap v5: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:27.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:27 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:27.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:27 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:27.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:27 smithi086 ceph-mon[47925]: pgmap v5: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:28.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:28 smithi086 ceph-mon[47925]: mgrmap e39: smithi032.peohxu(active, since 4s), standbys: smithi086.fyyhxm 2024-04-06T14:21:28.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:28 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:28.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:28 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:28 smithi032 ceph-mon[147414]: mgrmap e39: smithi032.peohxu(active, since 4s), standbys: smithi086.fyyhxm 2024-04-06T14:21:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:28 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:28 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:29.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:29 smithi086 ceph-mon[47925]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:21:29.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:29 smithi086 ceph-mon[47925]: pgmap v6: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:29.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:29 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:29.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:29 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:29.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:29 smithi032 ceph-mon[147414]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:21:29.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:29 smithi032 ceph-mon[147414]: pgmap v6: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:29.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:29 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:29.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:29 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:31 smithi032 ceph-mon[147414]: Detected new or changed devices on smithi086 2024-04-06T14:21:31.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:31.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:31.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:31 smithi086 ceph-mon[47925]: Detected new or changed devices on smithi086 2024-04-06T14:21:31.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:31 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:31.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:31 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:32.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:32 smithi032 ceph-mon[147414]: pgmap v7: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-06T14:21:32.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:32 smithi086 ceph-mon[47925]: pgmap v7: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-06T14:21:34.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: pgmap v8: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: Detected new or changed devices on smithi032 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:21:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:33 smithi086 ceph-mon[47925]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:21:34.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: pgmap v8: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T14:21:34.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: Detected new or changed devices on smithi032 2024-04-06T14:21:34.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:34.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:34.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:34.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:21:34.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-06T14:21:34.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: Updating smithi086:/etc/ceph/ceph.conf 2024-04-06T14:21:34.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:21:34.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T14:21:34.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:21:34.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:33 smithi032 ceph-mon[147414]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-04-06T14:21:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:21:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:21:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:35.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:21:35.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:34 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T14:21:35.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: Updating smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:21:35.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: Updating smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T14:21:35.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:35.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:35.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:21:35.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T14:21:36.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:35 smithi032 ceph-mon[147414]: pgmap v9: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T14:21:36.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:35 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:36.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:35 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:21:36.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:35 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:21:36.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:35 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:36.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:35 smithi086 ceph-mon[47925]: pgmap v9: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T14:21:36.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:35 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:36.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:35 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:21:36.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:35 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:21:36.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:35 smithi086 ceph-mon[47925]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:37.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:36 smithi086 ceph-mon[47925]: Upgrade: Updating mon.smithi086 2024-04-06T14:21:37.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:36 smithi086 ceph-mon[47925]: Deploying daemon mon.smithi086 on smithi086 2024-04-06T14:21:37.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:36 smithi032 ceph-mon[147414]: Upgrade: Updating mon.smithi086 2024-04-06T14:21:37.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:36 smithi032 ceph-mon[147414]: Deploying daemon mon.smithi086 on smithi086 2024-04-06T14:21:37.601 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:37 smithi086 systemd[1]: Stopping Ceph mon.smithi086 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:21:37.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:37 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi086[47886]: 2024-04-06T14:21:37.599+0000 7f417df48700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi086 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T14:21:37.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:37 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi086[47886]: 2024-04-06T14:21:37.599+0000 7f417df48700 -1 mon.smithi086@1(peon) e2 *** Got Signal Terminated *** 2024-04-06T14:21:38.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:38 smithi086 bash[110785]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi086 2024-04-06T14:21:38.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:38 smithi086 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi086.service: Succeeded. 2024-04-06T14:21:38.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:38 smithi086 systemd[1]: Stopped Ceph mon.smithi086 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:21:38.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:38 smithi086 systemd[1]: Starting Ceph mon.smithi086 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:21:39.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T14:21:39.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable), process ceph-mon, pid 7 2024-04-06T14:21:39.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: pidfile_write: ignore empty --pid-file 2024-04-06T14:21:39.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: load: jerasure load: lrc 2024-04-06T14:21:39.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: RocksDB version: 7.9.2 2024-04-06T14:21:39.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Git sha 0 2024-04-06T14:21:39.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Compile date 2024-04-03 02:24:10 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: DB SUMMARY 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: DB Session ID: 0JAFGPG6EVFW8NDSS7ZT 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: CURRENT file: CURRENT 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1091 Bytes 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi086/store.db dir, Total Num: 1, files: 000027.sst 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi086/store.db: 000025.log size: 6361137 ; 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.error_if_exists: 0 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.create_if_missing: 0 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.paranoid_checks: 1 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T14:21:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.env: 0x555a891fb7a0 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.fs: PosixFileSystem 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.info_log: 0x555a8c295380 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.statistics: (nil) 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.use_fsync: 0 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_log_file_size: 0 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.allow_fallocate: 1 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T14:21:39.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.use_direct_reads: 0 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.db_log_dir: 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.wal_dir: 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.write_buffer_manager: 0x555a8b50c140 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T14:21:39.365 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.unordered_write: 0 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.row_cache: None 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.wal_filter: None 2024-04-06T14:21:39.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.two_write_queues: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.wal_compression: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.atomic_flush: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.log_readahead_size: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T14:21:39.367 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_background_jobs: 2 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_background_compactions: -1 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_subcompactions: 1 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_open_files: -1 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T14:21:39.368 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_background_flushes: -1 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Compression algorithms supported: 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: kZSTD supported: 0 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: kXpressCompression supported: 0 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: kZlibCompression supported: 1 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: kSnappyCompression supported: 1 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: kLZ4Compression supported: 1 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: kBZip2Compression supported: 0 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T14:21:39.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000009 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.merge_operator: 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_filter: None 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555a8c2954a0) 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T14:21:39.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_type: 0 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_block_index_type: 0 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_shortening: 1 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: checksum: 4 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: no_block_cache: 0 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache: 0x555a8b5ad1f0 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_options: 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: capacity : 536870912 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: num_shard_bits : 4 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: strict_capacity_limit : 0 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_compressed: (nil) 2024-04-06T14:21:39.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: persistent_cache: (nil) 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_size: 4096 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_size_deviation: 10 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_restart_interval: 16 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_block_restart_interval: 1 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: metadata_block_size: 4096 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: partition_filters: 0 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: use_delta_encoding: 1 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: filter_policy: bloomfilter 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: whole_key_filtering: 1 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: verify_compression: 0 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: format_version: 5 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: enable_index_compression: 1 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_align: 0 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: max_auto_readahead_size: 262144 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: prepopulate_block_cache: 0 2024-04-06T14:21:39.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: initial_auto_readahead_size: 8192 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: num_file_reads_for_auto_readahead: 2 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression: NoCompression 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.num_levels: 7 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T14:21:39.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T14:21:39.374 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.arena_block_size: 1048576 2024-04-06T14:21:39.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.table_properties_collectors: 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.inplace_update_support: 0 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T14:21:39.376 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.bloom_locality: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.max_successive_merges: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.ttl: 2592000 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.enable_blob_files: false 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.min_blob_size: 0 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T14:21:39.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T14:21:39.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T14:21:39.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T14:21:39.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.blob_file_starting_level: 0 2024-04-06T14:21:39.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T14:21:39.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-06T14:21:39.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12245, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 086c9831-528e-4fb2-9455-8c93a6754a25 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712413299216886, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712413299232729, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3853710, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12250, "largest_seqno": 15096, "table_properties": {"data_size": 3842554, "index_size": 6871, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3269, "raw_key_size": 28287, "raw_average_key_size": 22, "raw_value_size": 3817317, "raw_average_value_size": 2979, "num_data_blocks": 312, "num_entries": 1281, "num_filter_entries": 1281, "num_deletions": 258, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712413299, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "086c9831-528e-4fb2-9455-8c93a6754a25", "db_session_id": "0JAFGPG6EVFW8NDSS7ZT", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712413299232819, "job": 1, "event": "recovery_finished"} 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi086/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555a8b64a000 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: DB pointer 0x555a8b638000 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T14:21:39.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** DB Stats ** 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: L0 1/0 3.68 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 520.9 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: L6 1/0 6.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Sum 2/0 10.36 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 520.9 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 520.9 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:21:39.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 520.9 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative compaction: 0.00 GB write, 180.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval compaction: 0.00 GB write, 180.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T14:21:39.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Block cache BinnedLRUCache@0x555a8b5ad1f0#7 capacity: 512.00 MB usage: 11.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,3.28 KB,0.000625849%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: starting mon.smithi086 rank 1 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi086 fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: mon.smithi086@-1(???) e2 preinit fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: mon.smithi086@-1(???).mds e22 new map 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: mon.smithi086@-1(???).mds e22 print_map 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: e22 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: legacy client fscid: 1 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Filesystem 'cephfs' (1) 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: fs_name cephfs 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: epoch 22 2024-04-06T14:21:39.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: modified 2024-04-06T14:21:28.458361+0000 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: tableserver 0 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: root 0 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: session_timeout 60 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: session_autoclose 300 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: max_file_size 1099511627776 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: max_xattr_size 65536 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: required_client_features {} 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: last_failure 0 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: last_failure_osd_epoch 46 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: max_mds 2 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: in 0,1 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: up {0=14460,1=24301} 2024-04-06T14:21:39.383 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: failed 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: damaged 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: stopped 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_pools [3] 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: metadata_pool 2 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: inline_data enabled 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: balancer 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: bal_rank_mask -1 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: standby_count_wanted 1 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: [mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: [mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Standby daemons: 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: [mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:39.384 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: [mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:39.385 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: mon.smithi086@-1(???).osd e50 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-06T14:21:39.385 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: mon.smithi086@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-04-06T14:21:39.385 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: mon.smithi086@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-04-06T14:21:39.385 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: mon.smithi086@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-04-06T14:21:39.385 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: mon.smithi086@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-04-06T14:21:39.385 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 bash[110938]: 7224a981bc2919448331655660916ef565cb2b7193fab002334624ed334b4060 2024-04-06T14:21:39.385 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 systemd[1]: Started Ceph mon.smithi086 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:21:39.385 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: pgmap v10: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-06T14:21:39.385 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:39 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:21:39.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:39 smithi032 ceph-mon[147414]: pgmap v10: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-06T14:21:39.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:39 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:21:40.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: pgmap v11: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-06T14:21:40.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:40.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: mon.smithi032 calling monitor election 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: mon.smithi086 calling monitor election 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: mon.smithi032 is new leader, mons smithi032,smithi086 in quorum (ranks 0,1) 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: monmap e3: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: osdmap e50: 6 total, 6 up, 6 in 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: mgrmap e39: smithi032.peohxu(active, since 16s), standbys: smithi086.fyyhxm 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:40.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:40.732 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:40 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:40.822 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: pgmap v11: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-06T14:21:40.822 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:40.822 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:40.822 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:21:40.822 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-06T14:21:40.822 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: mon.smithi032 calling monitor election 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: mon.smithi086 calling monitor election 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: mon.smithi032 is new leader, mons smithi032,smithi086 in quorum (ranks 0,1) 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: monmap e3: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: fsmap cephfs:2 {0=cephfs.smithi032.mqoaof=up:active,1=cephfs.smithi086.nikfjj=up:active} 2 up:standby 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: osdmap e50: 6 total, 6 up, 6 in 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: mgrmap e39: smithi032.peohxu(active, since 16s), standbys: smithi086.fyyhxm 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:40.823 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:40 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:21:41.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.260+0000 7f89e7c6f700 1 -- 172.21.15.32:0/3884912280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 msgr2=0x7f89e01033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:41.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.260+0000 7f89e7c6f700 1 --2- 172.21.15.32:0/3884912280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 0x7f89e01033a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f89dc004940 tx=0x7f89dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:41.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.260+0000 7f89e7c6f700 1 -- 172.21.15.32:0/3884912280 shutdown_connections 2024-04-06T14:21:41.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.260+0000 7f89e7c6f700 1 --2- 172.21.15.32:0/3884912280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 0x7f89e00fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.260+0000 7f89e7c6f700 1 --2- 172.21.15.32:0/3884912280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 0x7f89e01033a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.260+0000 7f89e7c6f700 1 -- 172.21.15.32:0/3884912280 >> 172.21.15.32:0/3884912280 conn(0x7f89e0071530 msgr2=0x7f89e0071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:41.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.260+0000 7f89e7c6f700 1 -- 172.21.15.32:0/3884912280 shutdown_connections 2024-04-06T14:21:41.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.260+0000 7f89e7c6f700 1 -- 172.21.15.32:0/3884912280 wait complete. 2024-04-06T14:21:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e7c6f700 1 Processor -- start 2024-04-06T14:21:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e7c6f700 1 -- start start 2024-04-06T14:21:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e7c6f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 0x7f89e0113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e7c6f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 0x7f89e0113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e7c6f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e0114420 con 0x7f89e0102fd0 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e7c6f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e010da20 con 0x7f89e00fb540 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e520a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 0x7f89e0113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e520a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 0x7f89e0113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59078/0 (socket says 172.21.15.32:59078) 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e520a700 1 -- 172.21.15.32:0/1030654347 learned_addr learned my addr 172.21.15.32:0/1030654347 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e5a0b700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 0x7f89e0113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e5a0b700 1 -- 172.21.15.32:0/1030654347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 msgr2=0x7f89e0113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e5a0b700 1 -- 172.21.15.32:0/1030654347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 msgr2=0x7f89e0113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e5a0b700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 0x7f89e0113890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.261+0000 7f89e5a0b700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 0x7f89e0113890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.262+0000 7f89e520a700 1 -- 172.21.15.32:0/1030654347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 msgr2=0x7f89e0113890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.262+0000 7f89e520a700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 0x7f89e0113890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.262+0000 7f89e520a700 1 -- 172.21.15.32:0/1030654347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc005040 con 0x7f89e0102fd0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.262+0000 7f89e520a700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 0x7f89e0113dd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89d00041d0 tx=0x7f89d00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.262+0000 7f89d67fc700 1 -- 172.21.15.32:0/1030654347 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89d0007cc0 con 0x7f89e0102fd0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.262+0000 7f89d67fc700 1 -- 172.21.15.32:0/1030654347 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89d000f040 con 0x7f89e0102fd0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.262+0000 7f89d67fc700 1 -- 172.21.15.32:0/1030654347 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89d000dae0 con 0x7f89e0102fd0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.262+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e010dd00 con 0x7f89e0102fd0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.262+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e010e2c0 con 0x7f89e0102fd0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.263+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e0100270 con 0x7f89e0102fd0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.263+0000 7f89d67fc700 1 -- 172.21.15.32:0/1030654347 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f89d000dc40 con 0x7f89e0102fd0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.264+0000 7f89d67fc700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f89cc06cd50 0x7f89cc06f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.264+0000 7f89d67fc700 1 -- 172.21.15.32:0/1030654347 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f89d008ba10 con 0x7f89e0102fd0 2024-04-06T14:21:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.264+0000 7f89e5a0b700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f89cc06cd50 0x7f89cc06f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.264+0000 7f89e5a0b700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f89cc06cd50 0x7f89cc06f200 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00b3b0 tx=0x7f89dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:41.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.269+0000 7f89d67fc700 1 -- 172.21.15.32:0/1030654347 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f89d00592e0 con 0x7f89e0102fd0 2024-04-06T14:21:41.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.491+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f89e0113560 con 0x7f89cc06cd50 2024-04-06T14:21:41.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.492+0000 7f89d67fc700 1 -- 172.21.15.32:0/1030654347 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f89e0113560 con 0x7f89cc06cd50 2024-04-06T14:21:41.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f89cc06cd50 msgr2=0x7f89cc06f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:41.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f89cc06cd50 0x7f89cc06f200 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00b3b0 tx=0x7f89dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 msgr2=0x7f89e0113dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 0x7f89e0113dd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89d00041d0 tx=0x7f89d00046b0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 shutdown_connections 2024-04-06T14:21:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f89cc06cd50 0x7f89cc06f200 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89e00fb540 0x7f89e0113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 --2- 172.21.15.32:0/1030654347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e0102fd0 0x7f89e0113dd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 >> 172.21.15.32:0/1030654347 conn(0x7f89e0071530 msgr2=0x7f89e00fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 shutdown_connections 2024-04-06T14:21:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.494+0000 7f89e7c6f700 1 -- 172.21.15.32:0/1030654347 wait complete. 2024-04-06T14:21:41.505 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:21:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.598+0000 7ff0c3c37700 1 -- 172.21.15.32:0/1660000614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fde90 msgr2=0x7ff0bc0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.598+0000 7ff0c3c37700 1 --2- 172.21.15.32:0/1660000614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fde90 0x7ff0bc0fe260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac004940 tx=0x7ff0ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.598+0000 7ff0c3c37700 1 -- 172.21.15.32:0/1660000614 shutdown_connections 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.598+0000 7ff0c3c37700 1 --2- 172.21.15.32:0/1660000614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fe830 0x7ff0bc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.598+0000 7ff0c3c37700 1 --2- 172.21.15.32:0/1660000614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fde90 0x7ff0bc0fe260 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.598+0000 7ff0c3c37700 1 -- 172.21.15.32:0/1660000614 >> 172.21.15.32:0/1660000614 conn(0x7ff0bc072bc0 msgr2=0x7ff0bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.598+0000 7ff0c3c37700 1 -- 172.21.15.32:0/1660000614 shutdown_connections 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.598+0000 7ff0c3c37700 1 -- 172.21.15.32:0/1660000614 wait complete. 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c3c37700 1 Processor -- start 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c3c37700 1 -- start start 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c3c37700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fde90 0x7ff0bc113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c3c37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fe830 0x7ff0bc114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c3c37700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc114680 con 0x7ff0bc0fe830 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c3c37700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc10da70 con 0x7ff0bc0fde90 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c11d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fe830 0x7ff0bc114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c11d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fe830 0x7ff0bc114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59086/0 (socket says 172.21.15.32:59086) 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c11d2700 1 -- 172.21.15.32:0/444418478 learned_addr learned my addr 172.21.15.32:0/444418478 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c19d3700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fde90 0x7ff0bc113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c19d3700 1 -- 172.21.15.32:0/444418478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fde90 msgr2=0x7ff0bc113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c19d3700 1 -- 172.21.15.32:0/444418478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fde90 msgr2=0x7ff0bc113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c19d3700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fde90 0x7ff0bc113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c19d3700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fde90 0x7ff0bc113af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c11d2700 1 -- 172.21.15.32:0/444418478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fde90 msgr2=0x7ff0bc113af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c11d2700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fde90 0x7ff0bc113af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.599+0000 7ff0c11d2700 1 -- 172.21.15.32:0/444418478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac005040 con 0x7ff0bc0fe830 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.600+0000 7ff0c11d2700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fe830 0x7ff0bc114030 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8002a50 tx=0x7ff0b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.600+0000 7ff0b27fc700 1 -- 172.21.15.32:0/444418478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0b8007cc0 con 0x7ff0bc0fe830 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.600+0000 7ff0b27fc700 1 -- 172.21.15.32:0/444418478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0b800f040 con 0x7ff0bc0fe830 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.600+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0bc10dd50 con 0x7ff0bc0fe830 2024-04-06T14:21:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.600+0000 7ff0b27fc700 1 -- 172.21.15.32:0/444418478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0b8009540 con 0x7ff0bc0fe830 2024-04-06T14:21:41.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.600+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0bc10e290 con 0x7ff0bc0fe830 2024-04-06T14:21:41.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.601+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0bc1019b0 con 0x7ff0bc0fe830 2024-04-06T14:21:41.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.601+0000 7ff0b27fc700 1 -- 172.21.15.32:0/444418478 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7ff0b80096a0 con 0x7ff0bc0fe830 2024-04-06T14:21:41.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.602+0000 7ff0b27fc700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0a806cd40 0x7ff0a806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:41.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.602+0000 7ff0b27fc700 1 -- 172.21.15.32:0/444418478 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff0b808bf50 con 0x7ff0bc0fe830 2024-04-06T14:21:41.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.602+0000 7ff0c19d3700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0a806cd40 0x7ff0a806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:41.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.602+0000 7ff0c19d3700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0a806cd40 0x7ff0a806f1f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00b3b0 tx=0x7ff0ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:41.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.606+0000 7ff0b27fc700 1 -- 172.21.15.32:0/444418478 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff0b80bfab0 con 0x7ff0bc0fe830 2024-04-06T14:21:41.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.833+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0bc1137c0 con 0x7ff0a806cd40 2024-04-06T14:21:41.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.834+0000 7ff0b27fc700 1 -- 172.21.15.32:0/444418478 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7ff0bc1137c0 con 0x7ff0a806cd40 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0a806cd40 msgr2=0x7ff0a806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0a806cd40 0x7ff0a806f1f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00b3b0 tx=0x7ff0ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fe830 msgr2=0x7ff0bc114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fe830 0x7ff0bc114030 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8002a50 tx=0x7ff0b80046b0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 shutdown_connections 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0a806cd40 0x7ff0a806f1f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc0fde90 0x7ff0bc113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 --2- 172.21.15.32:0/444418478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0bc0fe830 0x7ff0bc114030 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 >> 172.21.15.32:0/444418478 conn(0x7ff0bc072bc0 msgr2=0x7ff0bc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.836+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 shutdown_connections 2024-04-06T14:21:41.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.837+0000 7ff0c3c37700 1 -- 172.21.15.32:0/444418478 wait complete. 2024-04-06T14:21:41.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.939+0000 7f2e1d358700 1 -- 172.21.15.32:0/4276442721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 msgr2=0x7f2e18103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.939+0000 7f2e1d358700 1 --2- 172.21.15.32:0/4276442721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 0x7f2e18103340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2e08004940 tx=0x7f2e08009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.939+0000 7f2e1d358700 1 -- 172.21.15.32:0/4276442721 shutdown_connections 2024-04-06T14:21:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.939+0000 7f2e1d358700 1 --2- 172.21.15.32:0/4276442721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 0x7f2e180fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.939+0000 7f2e1d358700 1 --2- 172.21.15.32:0/4276442721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 0x7f2e18103340 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.939+0000 7f2e1d358700 1 -- 172.21.15.32:0/4276442721 >> 172.21.15.32:0/4276442721 conn(0x7f2e18072310 msgr2=0x7f2e18072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.939+0000 7f2e1d358700 1 -- 172.21.15.32:0/4276442721 shutdown_connections 2024-04-06T14:21:41.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.939+0000 7f2e1d358700 1 -- 172.21.15.32:0/4276442721 wait complete. 2024-04-06T14:21:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e1d358700 1 Processor -- start 2024-04-06T14:21:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e1d358700 1 -- start start 2024-04-06T14:21:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e1d358700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 0x7f2e18113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e1d358700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 0x7f2e18113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e1d358700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e181143b0 con 0x7f2e18102f70 2024-04-06T14:21:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e1d358700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1810d9b0 con 0x7f2e180fb3b0 2024-04-06T14:21:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e167fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 0x7f2e18113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e167fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 0x7f2e18113d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59110/0 (socket says 172.21.15.32:59110) 2024-04-06T14:21:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e167fc700 1 -- 172.21.15.32:0/2483375962 learned_addr learned my addr 172.21.15.32:0/2483375962 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.940+0000 7f2e16ffd700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 0x7f2e18113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e16ffd700 1 -- 172.21.15.32:0/2483375962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 msgr2=0x7f2e18113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e16ffd700 1 -- 172.21.15.32:0/2483375962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 msgr2=0x7f2e18113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e16ffd700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 0x7f2e18113820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e16ffd700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 0x7f2e18113820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e167fc700 1 -- 172.21.15.32:0/2483375962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 msgr2=0x7f2e18113820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e167fc700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 0x7f2e18113820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e167fc700 1 -- 172.21.15.32:0/2483375962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08005040 con 0x7f2e18102f70 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e167fc700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 0x7f2e18113d60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c002a50 tx=0x7f2e0c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2dff7fe700 1 -- 172.21.15.32:0/2483375962 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0c007cc0 con 0x7f2e18102f70 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2dff7fe700 1 -- 172.21.15.32:0/2483375962 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e0c00f040 con 0x7f2e18102f70 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1810dc90 con 0x7f2e18102f70 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2dff7fe700 1 -- 172.21.15.32:0/2483375962 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0c009540 con 0x7f2e18102f70 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.941+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1810e250 con 0x7f2e18102f70 2024-04-06T14:21:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.942+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e180fe6d0 con 0x7f2e18102f70 2024-04-06T14:21:41.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.943+0000 7f2dff7fe700 1 -- 172.21.15.32:0/2483375962 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f2e0c0096a0 con 0x7f2e18102f70 2024-04-06T14:21:41.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.943+0000 7f2dff7fe700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e0006cd50 0x7f2e0006f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:41.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.943+0000 7f2dff7fe700 1 -- 172.21.15.32:0/2483375962 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2e0c08c670 con 0x7f2e18102f70 2024-04-06T14:21:41.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.943+0000 7f2e16ffd700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e0006cd50 0x7f2e0006f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:41.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.944+0000 7f2e16ffd700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e0006cd50 0x7f2e0006f200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2e08005010 tx=0x7f2e08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:41.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:41.947+0000 7f2dff7fe700 1 -- 172.21.15.32:0/2483375962 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2e0c059fc0 con 0x7f2e18102f70 2024-04-06T14:21:42.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.159+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e18105400 con 0x7f2e0006cd50 2024-04-06T14:21:42.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.169+0000 7f2dff7fe700 1 -- 172.21.15.32:0/2483375962 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f2e18105400 con 0x7f2e0006cd50 2024-04-06T14:21:42.169 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:21:42.169 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (2m) 13s ago 13m 19.1M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (13m) 13s ago 13m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (12m) 16s ago 12m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (107s) 13s ago 13m 80.0M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (7m) 13s ago 7m 13.4M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (7m) 13s ago 7m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (7m) 16s ago 7m 128M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (7m) 16s ago 7m 16.2M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (3m) 13s ago 14m 470M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (2m) 16s ago 12m 445M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (28s) 13s ago 14m 26.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 starting - - - 2048M 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (2m) 13s ago 13m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (2m) 16s ago 12m 17.3M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (11m) 13s ago 11m 195M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:21:42.170 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 13s ago 11m 192M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:21:42.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (10m) 13s ago 10m 163M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:21:42.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (10m) 16s ago 10m 226M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:21:42.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (9m) 16s ago 9m 204M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:21:42.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (9m) 16s ago 9m 186M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:21:42.171 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (2m) 13s ago 12m 56.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:21:42.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e0006cd50 msgr2=0x7f2e0006f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:42.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e0006cd50 0x7f2e0006f200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2e08005010 tx=0x7f2e08009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 msgr2=0x7f2e18113d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 0x7f2e18113d60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c002a50 tx=0x7f2e0c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 shutdown_connections 2024-04-06T14:21:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e0006cd50 0x7f2e0006f200 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e180fb3b0 0x7f2e18113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 --2- 172.21.15.32:0/2483375962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e18102f70 0x7f2e18113d60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 >> 172.21.15.32:0/2483375962 conn(0x7f2e18072310 msgr2=0x7f2e180f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 shutdown_connections 2024-04-06T14:21:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.171+0000 7f2e1d358700 1 -- 172.21.15.32:0/2483375962 wait complete. 2024-04-06T14:21:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.275+0000 7f11a999e700 1 -- 172.21.15.32:0/953016744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 msgr2=0x7f11a4103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.275+0000 7f11a999e700 1 --2- 172.21.15.32:0/953016744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 0x7f11a4103070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1198004940 tx=0x7f1198009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.275+0000 7f11a999e700 1 -- 172.21.15.32:0/953016744 shutdown_connections 2024-04-06T14:21:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.275+0000 7f11a999e700 1 --2- 172.21.15.32:0/953016744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 0x7f11a4065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.275+0000 7f11a999e700 1 --2- 172.21.15.32:0/953016744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 0x7f11a4103070 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.275+0000 7f11a999e700 1 -- 172.21.15.32:0/953016744 >> 172.21.15.32:0/953016744 conn(0x7f11a4071b70 msgr2=0x7f11a4071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.275+0000 7f11a999e700 1 -- 172.21.15.32:0/953016744 shutdown_connections 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.275+0000 7f11a999e700 1 -- 172.21.15.32:0/953016744 wait complete. 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a999e700 1 Processor -- start 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a999e700 1 -- start start 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a999e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 0x7f11a41138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a999e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 0x7f11a4113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a999e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a4114430 con 0x7f11a4102ca0 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a999e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a410d820 con 0x7f11a4065630 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 0x7f11a4113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 0x7f11a4113de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59132/0 (socket says 172.21.15.32:59132) 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a3fff700 1 -- 172.21.15.32:0/1375604017 learned_addr learned my addr 172.21.15.32:0/1375604017 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.276+0000 7f11a899c700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 0x7f11a41138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a899c700 1 -- 172.21.15.32:0/1375604017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 msgr2=0x7f11a41138a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a899c700 1 -- 172.21.15.32:0/1375604017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 msgr2=0x7f11a41138a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a899c700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 0x7f11a41138a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a899c700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 0x7f11a41138a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a3fff700 1 -- 172.21.15.32:0/1375604017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 msgr2=0x7f11a41138a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a3fff700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 0x7f11a41138a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a3fff700 1 -- 172.21.15.32:0/1375604017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1198005040 con 0x7f11a4102ca0 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a3fff700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 0x7f11a4113de0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1190002a50 tx=0x7f11900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a17fa700 1 -- 172.21.15.32:0/1375604017 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1190007cc0 con 0x7f11a4102ca0 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a17fa700 1 -- 172.21.15.32:0/1375604017 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f119000f040 con 0x7f11a4102ca0 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a410db00 con 0x7f11a4102ca0 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a17fa700 1 -- 172.21.15.32:0/1375604017 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1190009540 con 0x7f11a4102ca0 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.277+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a410e0c0 con 0x7f11a4102ca0 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.278+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a40fe380 con 0x7f11a4102ca0 2024-04-06T14:21:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.279+0000 7f11a17fa700 1 -- 172.21.15.32:0/1375604017 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f11900096a0 con 0x7f11a4102ca0 2024-04-06T14:21:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.279+0000 7f11a17fa700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f119406cd40 0x7f119406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.279+0000 7f11a17fa700 1 -- 172.21.15.32:0/1375604017 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f119008c050 con 0x7f11a4102ca0 2024-04-06T14:21:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.279+0000 7f11a899c700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f119406cd40 0x7f119406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.279+0000 7f11a899c700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f119406cd40 0x7f119406f1f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f119800b3b0 tx=0x7f1198009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.283+0000 7f11a17fa700 1 -- 172.21.15.32:0/1375604017 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1190059920 con 0x7f11a4102ca0 2024-04-06T14:21:42.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.576+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f11a4109ba0 con 0x7f11a4102ca0 2024-04-06T14:21:42.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.576+0000 7f11a17fa700 1 -- 172.21.15.32:0/1375604017 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+742 (secure 0 0 0) 0x7f11900594b0 con 0x7f11a4102ca0 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:21:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:21:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:21:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:21:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:21:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 10, 2024-04-06T14:21:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:21:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:21:42.578 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.578+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f119406cd40 msgr2=0x7f119406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.578+0000 7f11a999e700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f119406cd40 0x7f119406f1f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f119800b3b0 tx=0x7f1198009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.578+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 msgr2=0x7f11a4113de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.578+0000 7f11a999e700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 0x7f11a4113de0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1190002a50 tx=0x7f11900046b0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.579+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 shutdown_connections 2024-04-06T14:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.579+0000 7f11a999e700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f119406cd40 0x7f119406f1f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.579+0000 7f11a999e700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11a4065630 0x7f11a41138a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.579+0000 7f11a999e700 1 --2- 172.21.15.32:0/1375604017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11a4102ca0 0x7f11a4113de0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.579+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 >> 172.21.15.32:0/1375604017 conn(0x7f11a4071b70 msgr2=0x7f11a41059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.579+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 shutdown_connections 2024-04-06T14:21:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.579+0000 7f11a999e700 1 -- 172.21.15.32:0/1375604017 wait complete. 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.686+0000 7f69b1892700 1 -- 172.21.15.32:0/574696645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 msgr2=0x7f69ac102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.686+0000 7f69b1892700 1 --2- 172.21.15.32:0/574696645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 0x7f69ac102140 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f699c009ce0 tx=0x7f699c004d10 comp rx=0 tx=0).stop 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.686+0000 7f69b1892700 1 -- 172.21.15.32:0/574696645 shutdown_connections 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.686+0000 7f69b1892700 1 --2- 172.21.15.32:0/574696645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 0x7f69ac0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.686+0000 7f69b1892700 1 --2- 172.21.15.32:0/574696645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 0x7f69ac102140 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.686+0000 7f69b1892700 1 -- 172.21.15.32:0/574696645 >> 172.21.15.32:0/574696645 conn(0x7f69ac072bc0 msgr2=0x7f69ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.686+0000 7f69b1892700 1 -- 172.21.15.32:0/574696645 shutdown_connections 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.686+0000 7f69b1892700 1 -- 172.21.15.32:0/574696645 wait complete. 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69b1892700 1 Processor -- start 2024-04-06T14:21:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69b1892700 1 -- start start 2024-04-06T14:21:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69b1892700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 0x7f69ac115e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69b1892700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 0x7f69ac116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69b1892700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac1169b0 con 0x7f69ac101d70 2024-04-06T14:21:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69b1892700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac10fd80 con 0x7f69ac102710 2024-04-06T14:21:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 0x7f69ac115e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 0x7f69ac115e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59158/0 (socket says 172.21.15.32:59158) 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aaffd700 1 -- 172.21.15.32:0/2262538117 learned_addr learned my addr 172.21.15.32:0/2262538117 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aa7fc700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 0x7f69ac116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aa7fc700 1 -- 172.21.15.32:0/2262538117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 msgr2=0x7f69ac116360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aa7fc700 1 -- 172.21.15.32:0/2262538117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 msgr2=0x7f69ac116360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aa7fc700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 0x7f69ac116360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aa7fc700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 0x7f69ac116360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aaffd700 1 -- 172.21.15.32:0/2262538117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 msgr2=0x7f69ac116360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aaffd700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 0x7f69ac116360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aaffd700 1 -- 172.21.15.32:0/2262538117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c009940 con 0x7f69ac101d70 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.687+0000 7f69aaffd700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 0x7f69ac115e20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f699c003400 tx=0x7f699c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.688+0000 7f69a37fe700 1 -- 172.21.15.32:0/2262538117 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f699c00e040 con 0x7f69ac101d70 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.688+0000 7f69a37fe700 1 -- 172.21.15.32:0/2262538117 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f699c004610 con 0x7f69ac101d70 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.688+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac110000 con 0x7f69ac101d70 2024-04-06T14:21:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.688+0000 7f69a37fe700 1 -- 172.21.15.32:0/2262538117 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f699c01c640 con 0x7f69ac101d70 2024-04-06T14:21:42.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.688+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac1105c0 con 0x7f69ac101d70 2024-04-06T14:21:42.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.689+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac0fd4c0 con 0x7f69ac101d70 2024-04-06T14:21:42.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.689+0000 7f69a37fe700 1 -- 172.21.15.32:0/2262538117 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f699c012410 con 0x7f69ac101d70 2024-04-06T14:21:42.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.689+0000 7f69a37fe700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f699806cd40 0x7f699806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:42.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.690+0000 7f69a37fe700 1 -- 172.21.15.32:0/2262538117 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f699c08dcb0 con 0x7f69ac101d70 2024-04-06T14:21:42.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.690+0000 7f69aa7fc700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f699806cd40 0x7f699806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:42.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.690+0000 7f69aa7fc700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f699806cd40 0x7f699806f1f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f69ac1116c0 tx=0x7f6994009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.694+0000 7f69a37fe700 1 -- 172.21.15.32:0/2262538117 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f699c05b600 con 0x7f69ac101d70 2024-04-06T14:21:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:42 smithi032 ceph-mon[147414]: pgmap v12: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-06T14:21:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:42 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:42 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:42.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:42 smithi086 ceph-mon[110975]: pgmap v12: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-06T14:21:42.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:42 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:42.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:42 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.946+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f69ac108d50 con 0x7f69ac101d70 2024-04-06T14:21:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.946+0000 7f69a37fe700 1 -- 172.21.15.32:0/2262538117 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1851 (secure 0 0 0) 0x7f699c020750 con 0x7f69ac101d70 2024-04-06T14:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:21:28.458361+0000 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:21:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f699806cd40 msgr2=0x7f699806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f699806cd40 0x7f699806f1f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f69ac1116c0 tx=0x7f6994009f70 comp rx=0 tx=0).stop 2024-04-06T14:21:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 msgr2=0x7f69ac115e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 0x7f69ac115e20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f699c003400 tx=0x7f699c004d10 comp rx=0 tx=0).stop 2024-04-06T14:21:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 shutdown_connections 2024-04-06T14:21:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f699806cd40 0x7f699806f1f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac101d70 0x7f69ac115e20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 --2- 172.21.15.32:0/2262538117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69ac102710 0x7f69ac116360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 >> 172.21.15.32:0/2262538117 conn(0x7f69ac072bc0 msgr2=0x7f69ac0659d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 shutdown_connections 2024-04-06T14:21:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:42.948+0000 7f69b1892700 1 -- 172.21.15.32:0/2262538117 wait complete. 2024-04-06T14:21:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-06T14:21:43.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.053+0000 7fa8c9c35700 1 -- 172.21.15.32:0/2541357219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c4101d20 msgr2=0x7fa8c41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:43.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.053+0000 7fa8c9c35700 1 --2- 172.21.15.32:0/2541357219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c4101d20 0x7fa8c41020f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac004940 tx=0x7fa8ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:43.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.053+0000 7fa8c9c35700 1 -- 172.21.15.32:0/2541357219 shutdown_connections 2024-04-06T14:21:43.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.053+0000 7fa8c9c35700 1 --2- 172.21.15.32:0/2541357219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c41026c0 0x7fa8c40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.053+0000 7fa8c9c35700 1 --2- 172.21.15.32:0/2541357219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c4101d20 0x7fa8c41020f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.053+0000 7fa8c9c35700 1 -- 172.21.15.32:0/2541357219 >> 172.21.15.32:0/2541357219 conn(0x7fa8c40f78d0 msgr2=0x7fa8c40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.053+0000 7fa8c9c35700 1 -- 172.21.15.32:0/2541357219 shutdown_connections 2024-04-06T14:21:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.053+0000 7fa8c9c35700 1 -- 172.21.15.32:0/2541357219 wait complete. 2024-04-06T14:21:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.053+0000 7fa8c9c35700 1 Processor -- start 2024-04-06T14:21:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c9c35700 1 -- start start 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c9c35700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c4101d20 0x7fa8c410eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c9c35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c41026c0 0x7fa8c410efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c9c35700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c410f630 con 0x7fa8c41026c0 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c9c35700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c410f770 con 0x7fa8c4101d20 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c41026c0 0x7fa8c410efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c41026c0 0x7fa8c410efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59182/0 (socket says 172.21.15.32:59182) 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c2ffd700 1 -- 172.21.15.32:0/3691772210 learned_addr learned my addr 172.21.15.32:0/3691772210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c37fe700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c4101d20 0x7fa8c410eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c37fe700 1 -- 172.21.15.32:0/3691772210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c4101d20 msgr2=0x7fa8c410eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c37fe700 1 -- 172.21.15.32:0/3691772210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c4101d20 msgr2=0x7fa8c410eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c37fe700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c4101d20 0x7fa8c410eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c37fe700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c4101d20 0x7fa8c410eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c2ffd700 1 -- 172.21.15.32:0/3691772210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c4101d20 msgr2=0x7fa8c410eaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:21:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c2ffd700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c4101d20 0x7fa8c410eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c2ffd700 1 -- 172.21.15.32:0/3691772210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac005040 con 0x7fa8c41026c0 2024-04-06T14:21:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.054+0000 7fa8c2ffd700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c41026c0 0x7fa8c410efe0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4002a80 tx=0x7fa8b4004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.055+0000 7fa8bbfff700 1 -- 172.21.15.32:0/3691772210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8b4007cc0 con 0x7fa8c41026c0 2024-04-06T14:21:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.055+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c4117b10 con 0x7fa8c41026c0 2024-04-06T14:21:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.055+0000 7fa8bbfff700 1 -- 172.21.15.32:0/3691772210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8b400f040 con 0x7fa8c41026c0 2024-04-06T14:21:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.055+0000 7fa8bbfff700 1 -- 172.21.15.32:0/3691772210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8b400da30 con 0x7fa8c41026c0 2024-04-06T14:21:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.055+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c4118070 con 0x7fa8c41026c0 2024-04-06T14:21:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.056+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c40fd470 con 0x7fa8c41026c0 2024-04-06T14:21:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.056+0000 7fa8bbfff700 1 -- 172.21.15.32:0/3691772210 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fa8b400dbd0 con 0x7fa8c41026c0 2024-04-06T14:21:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.057+0000 7fa8bbfff700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa8b006cd40 0x7fa8b006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.057+0000 7fa8bbfff700 1 -- 172.21.15.32:0/3691772210 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa8b408b940 con 0x7fa8c41026c0 2024-04-06T14:21:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.057+0000 7fa8c37fe700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa8b006cd40 0x7fa8b006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.057+0000 7fa8c37fe700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa8b006cd40 0x7fa8b006f1f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac004f90 tx=0x7fa8ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.061+0000 7fa8bbfff700 1 -- 172.21.15.32:0/3691772210 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa8b4055780 con 0x7fa8c41026c0 2024-04-06T14:21:43.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.284+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8c404bed0 con 0x7fa8b006cd40 2024-04-06T14:21:43.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.285+0000 7fa8bbfff700 1 -- 172.21.15.32:0/3691772210 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fa8c404bed0 con 0x7fa8b006cd40 2024-04-06T14:21:43.285 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:21:43.285 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:21:43.285 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:21:43.285 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:21:43.285 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:21:43.285 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:21:43.285 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-06T14:21:43.285 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:21:43.286 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "4/21 daemons upgraded", 2024-04-06T14:21:43.286 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mon daemons", 2024-04-06T14:21:43.286 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:21:43.286 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:21:43.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa8b006cd40 msgr2=0x7fa8b006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa8b006cd40 0x7fa8b006f1f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac004f90 tx=0x7fa8ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c41026c0 msgr2=0x7fa8c410efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c41026c0 0x7fa8c410efe0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4002a80 tx=0x7fa8b4004630 comp rx=0 tx=0).stop 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 shutdown_connections 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa8b006cd40 0x7fa8b006f1f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8c4101d20 0x7fa8c410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 --2- 172.21.15.32:0/3691772210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8c41026c0 0x7fa8c410efe0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 >> 172.21.15.32:0/3691772210 conn(0x7fa8c40f78d0 msgr2=0x7fa8c40f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 shutdown_connections 2024-04-06T14:21:43.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.287+0000 7fa8c9c35700 1 -- 172.21.15.32:0/3691772210 wait complete. 2024-04-06T14:21:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.390+0000 7faaf8887700 1 -- 172.21.15.32:0/3749082322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf4101d70 msgr2=0x7faaf4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.390+0000 7faaf8887700 1 --2- 172.21.15.32:0/3749082322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf4101d70 0x7faaf4102140 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faae4004940 tx=0x7faae4009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.390+0000 7faaf8887700 1 -- 172.21.15.32:0/3749082322 shutdown_connections 2024-04-06T14:21:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.390+0000 7faaf8887700 1 --2- 172.21.15.32:0/3749082322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf40fc5b0 0x7faaf40fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.390+0000 7faaf8887700 1 --2- 172.21.15.32:0/3749082322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf4101d70 0x7faaf4102140 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.390+0000 7faaf8887700 1 -- 172.21.15.32:0/3749082322 >> 172.21.15.32:0/3749082322 conn(0x7faaf40714c0 msgr2=0x7faaf40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf8887700 1 -- 172.21.15.32:0/3749082322 shutdown_connections 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf8887700 1 -- 172.21.15.32:0/3749082322 wait complete. 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf8887700 1 Processor -- start 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf8887700 1 -- start start 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf8887700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf40fc5b0 0x7faaf41129d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf8887700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf4101d70 0x7faaf4112f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf8887700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf4113560 con 0x7faaf40fc5b0 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf8887700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf410c950 con 0x7faaf4101d70 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf40fc5b0 0x7faaf41129d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf40fc5b0 0x7faaf41129d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59200/0 (socket says 172.21.15.32:59200) 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf259c700 1 -- 172.21.15.32:0/3877648369 learned_addr learned my addr 172.21.15.32:0/3877648369 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.391+0000 7faaf1d9b700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf4101d70 0x7faaf4112f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf1d9b700 1 -- 172.21.15.32:0/3877648369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf4101d70 msgr2=0x7faaf4112f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf1d9b700 1 -- 172.21.15.32:0/3877648369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf4101d70 msgr2=0x7faaf4112f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf1d9b700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf4101d70 0x7faaf4112f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf1d9b700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf4101d70 0x7faaf4112f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf259c700 1 -- 172.21.15.32:0/3877648369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf4101d70 msgr2=0x7faaf4112f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf259c700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf4101d70 0x7faaf4112f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf259c700 1 -- 172.21.15.32:0/3877648369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae4005040 con 0x7faaf40fc5b0 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf259c700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf40fc5b0 0x7faaf41129d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faae4009380 tx=0x7faae4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faae2ffd700 1 -- 172.21.15.32:0/3877648369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faae400e040 con 0x7faaf40fc5b0 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf410cbd0 con 0x7faaf40fc5b0 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faae2ffd700 1 -- 172.21.15.32:0/3877648369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faae4013420 con 0x7faaf40fc5b0 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faae2ffd700 1 -- 172.21.15.32:0/3877648369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faae401c630 con 0x7faaf40fc5b0 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.392+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf410d190 con 0x7faaf40fc5b0 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.393+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf40fd5c0 con 0x7faaf40fc5b0 2024-04-06T14:21:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.394+0000 7faae2ffd700 1 -- 172.21.15.32:0/3877648369 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7faae40139e0 con 0x7faaf40fc5b0 2024-04-06T14:21:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.394+0000 7faae2ffd700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faadc06cd40 0x7faadc06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:21:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.394+0000 7faaf1d9b700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faadc06cd40 0x7faadc06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:21:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.394+0000 7faae2ffd700 1 -- 172.21.15.32:0/3877648369 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7faae408d310 con 0x7faaf40fc5b0 2024-04-06T14:21:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.395+0000 7faaf1d9b700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faadc06cd40 0x7faadc06f1f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faaf410db10 tx=0x7faae8009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:21:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.399+0000 7faae2ffd700 1 -- 172.21.15.32:0/3877648369 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faae405ad00 con 0x7faaf40fc5b0 2024-04-06T14:21:43.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.685+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faaf405f5f0 con 0x7faaf40fc5b0 2024-04-06T14:21:43.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.686+0000 7faae2ffd700 1 -- 172.21.15.32:0/3877648369 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7faae405a890 con 0x7faaf40fc5b0 2024-04-06T14:21:43.687 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:43.687 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:21:43.687 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faadc06cd40 msgr2=0x7faadc06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faadc06cd40 0x7faadc06f1f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faaf410db10 tx=0x7faae8009f70 comp rx=0 tx=0).stop 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf40fc5b0 msgr2=0x7faaf41129d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf40fc5b0 0x7faaf41129d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faae4009380 tx=0x7faae4009e30 comp rx=0 tx=0).stop 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 shutdown_connections 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faadc06cd40 0x7faadc06f1f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf40fc5b0 0x7faaf41129d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 --2- 172.21.15.32:0/3877648369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf4101d70 0x7faaf4112f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 >> 172.21.15.32:0/3877648369 conn(0x7faaf40714c0 msgr2=0x7faaf4104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 shutdown_connections 2024-04-06T14:21:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:21:43.688+0000 7faaf8887700 1 -- 172.21.15.32:0/3877648369 wait complete. 2024-04-06T14:21:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:43 smithi032 ceph-mon[147414]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:43 smithi032 ceph-mon[147414]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:43 smithi032 ceph-mon[147414]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:43.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:43 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1375604017' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:21:43.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:43 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:43.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:43 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:43.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:43 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2262538117' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:21:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:43 smithi086 ceph-mon[110975]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:43 smithi086 ceph-mon[110975]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:43 smithi086 ceph-mon[110975]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:43 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1375604017' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:21:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:43 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:43 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:43 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2262538117' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:21:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:44 smithi032 ceph-mon[147414]: pgmap v13: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:44 smithi032 ceph-mon[147414]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:44 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3877648369' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:21:44.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:44 smithi086 ceph-mon[110975]: pgmap v13: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:44.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:44 smithi086 ceph-mon[110975]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:21:44.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:44 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3877648369' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:21:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:46 smithi032 ceph-mon[147414]: pgmap v14: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:46 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:46 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:46 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:46 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:21:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:46 smithi086 ceph-mon[110975]: pgmap v14: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:46 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:46 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:46 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:46.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:46 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:21:47.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:47 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:47.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:47 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:21:47.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:47 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:21:47.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:47 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:47.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:47 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:47.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:47 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:21:47.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:47 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:21:47.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:47 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:48.660 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:48 smithi032 ceph-mon[147414]: Reconfiguring mon.smithi032 (monmap changed)... 2024-04-06T14:21:48.660 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:48 smithi032 ceph-mon[147414]: Reconfiguring daemon mon.smithi032 on smithi032 2024-04-06T14:21:48.660 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:48 smithi032 ceph-mon[147414]: pgmap v15: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:48.660 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:48.660 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:48.660 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.peohxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:21:48.660 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:21:48.661 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:48.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:48 smithi086 ceph-mon[110975]: Reconfiguring mon.smithi032 (monmap changed)... 2024-04-06T14:21:48.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:48 smithi086 ceph-mon[110975]: Reconfiguring daemon mon.smithi032 on smithi032 2024-04-06T14:21:48.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:48 smithi086 ceph-mon[110975]: pgmap v15: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:48.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:48.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:48.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.peohxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:21:48.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:21:48.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:49.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:49 smithi032 ceph-mon[147414]: Reconfiguring mgr.smithi032.peohxu (monmap changed)... 2024-04-06T14:21:49.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:49 smithi032 ceph-mon[147414]: Reconfiguring daemon mgr.smithi032.peohxu on smithi032 2024-04-06T14:21:49.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:49 smithi086 ceph-mon[110975]: Reconfiguring mgr.smithi032.peohxu (monmap changed)... 2024-04-06T14:21:49.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:49 smithi086 ceph-mon[110975]: Reconfiguring daemon mgr.smithi032.peohxu on smithi032 2024-04-06T14:21:50.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:50 smithi032 ceph-mon[147414]: pgmap v16: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:50.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:50.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:50.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:21:50.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:50.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:50 smithi086 ceph-mon[110975]: pgmap v16: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:50.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:50.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:50.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:21:50.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:51.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:51 smithi032 ceph-mon[147414]: Reconfiguring crash.smithi032 (monmap changed)... 2024-04-06T14:21:51.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:51 smithi032 ceph-mon[147414]: Reconfiguring daemon crash.smithi032 on smithi032 2024-04-06T14:21:51.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:51 smithi086 ceph-mon[110975]: Reconfiguring crash.smithi032 (monmap changed)... 2024-04-06T14:21:51.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:51 smithi086 ceph-mon[110975]: Reconfiguring daemon crash.smithi032 on smithi032 2024-04-06T14:21:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:52 smithi032 ceph-mon[147414]: pgmap v17: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T14:21:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:52.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:52 smithi086 ceph-mon[110975]: pgmap v17: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:52.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:52.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:52.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T14:21:52.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:53.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:53 smithi032 ceph-mon[147414]: Reconfiguring osd.0 (monmap changed)... 2024-04-06T14:21:53.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:53 smithi032 ceph-mon[147414]: Reconfiguring daemon osd.0 on smithi032 2024-04-06T14:21:53.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:21:53.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:53.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:53.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T14:21:53.667 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:53 smithi086 ceph-mon[110975]: Reconfiguring osd.0 (monmap changed)... 2024-04-06T14:21:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:53 smithi086 ceph-mon[110975]: Reconfiguring daemon osd.0 on smithi032 2024-04-06T14:21:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:21:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T14:21:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:54.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:54 smithi032 ceph-mon[147414]: pgmap v18: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:54.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:54 smithi032 ceph-mon[147414]: Reconfiguring osd.1 (monmap changed)... 2024-04-06T14:21:54.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:54 smithi032 ceph-mon[147414]: Reconfiguring daemon osd.1 on smithi032 2024-04-06T14:21:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:54 smithi086 ceph-mon[110975]: pgmap v18: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:54 smithi086 ceph-mon[110975]: Reconfiguring osd.1 (monmap changed)... 2024-04-06T14:21:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:54 smithi086 ceph-mon[110975]: Reconfiguring daemon osd.1 on smithi032 2024-04-06T14:21:55.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:55 smithi032 ceph-mon[147414]: pgmap v19: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:55.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:55.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:55.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:55 smithi032 ceph-mon[147414]: Reconfiguring osd.2 (monmap changed)... 2024-04-06T14:21:55.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T14:21:55.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:55.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:55 smithi032 ceph-mon[147414]: Reconfiguring daemon osd.2 on smithi032 2024-04-06T14:21:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:55 smithi086 ceph-mon[110975]: pgmap v19: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:55 smithi086 ceph-mon[110975]: Reconfiguring osd.2 (monmap changed)... 2024-04-06T14:21:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T14:21:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:55 smithi086 ceph-mon[110975]: Reconfiguring daemon osd.2 on smithi032 2024-04-06T14:21:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:57 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:57 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:57 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.gcjdjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:21:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:57 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:57.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:57 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:57.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:57 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:57.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:57 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.gcjdjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:21:57.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:57 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:58 smithi032 ceph-mon[147414]: Reconfiguring mds.cephfs.smithi032.gcjdjo (monmap changed)... 2024-04-06T14:21:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:58 smithi032 ceph-mon[147414]: Reconfiguring daemon mds.cephfs.smithi032.gcjdjo on smithi032 2024-04-06T14:21:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:58 smithi032 ceph-mon[147414]: pgmap v20: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:58 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:58 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:58 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.mqoaof", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:21:58.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:58 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:58.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:58 smithi086 ceph-mon[110975]: Reconfiguring mds.cephfs.smithi032.gcjdjo (monmap changed)... 2024-04-06T14:21:58.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:58 smithi086 ceph-mon[110975]: Reconfiguring daemon mds.cephfs.smithi032.gcjdjo on smithi032 2024-04-06T14:21:58.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:58 smithi086 ceph-mon[110975]: pgmap v20: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:58.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:58 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:58.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:58 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:21:58.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:58 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.mqoaof", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:21:58.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:58 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:21:59.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:59 smithi032 ceph-mon[147414]: Reconfiguring mds.cephfs.smithi032.mqoaof (monmap changed)... 2024-04-06T14:21:59.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:59 smithi032 ceph-mon[147414]: Reconfiguring daemon mds.cephfs.smithi032.mqoaof on smithi032 2024-04-06T14:21:59.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:21:59 smithi032 ceph-mon[147414]: pgmap v21: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:21:59.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:59 smithi086 ceph-mon[110975]: Reconfiguring mds.cephfs.smithi032.mqoaof (monmap changed)... 2024-04-06T14:21:59.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:59 smithi086 ceph-mon[110975]: Reconfiguring daemon mds.cephfs.smithi032.mqoaof on smithi032 2024-04-06T14:21:59.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:21:59 smithi086 ceph-mon[110975]: pgmap v21: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:00.642 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:00 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:00.643 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:00 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:00.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:00 smithi086 ceph-mon[110975]: Reconfiguring crash.smithi086 (monmap changed)... 2024-04-06T14:22:00.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:00 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:22:00.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:00 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:00.896 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:00 smithi086 ceph-mon[110975]: Reconfiguring daemon crash.smithi086 on smithi086 2024-04-06T14:22:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:00 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:00 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:00 smithi032 ceph-mon[147414]: Reconfiguring crash.smithi086 (monmap changed)... 2024-04-06T14:22:00.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:00 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:22:00.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:00 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:00.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:00 smithi032 ceph-mon[147414]: Reconfiguring daemon crash.smithi086 on smithi086 2024-04-06T14:22:02.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:02 smithi086 ceph-mon[110975]: pgmap v22: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:02 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:02 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:02 smithi086 ceph-mon[110975]: Reconfiguring mgr.smithi086.fyyhxm (monmap changed)... 2024-04-06T14:22:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:02 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:22:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:02 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:22:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:02 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:02 smithi086 ceph-mon[110975]: Reconfiguring daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:22:02.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:02 smithi032 ceph-mon[147414]: pgmap v22: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:02.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:02 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:02.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:02 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:02.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:02 smithi032 ceph-mon[147414]: Reconfiguring mgr.smithi086.fyyhxm (monmap changed)... 2024-04-06T14:22:02.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:02 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.fyyhxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T14:22:02.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:02 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T14:22:02.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:02 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:02.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:02 smithi032 ceph-mon[147414]: Reconfiguring daemon mgr.smithi086.fyyhxm on smithi086 2024-04-06T14:22:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:03 smithi086 ceph-mon[110975]: pgmap v23: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:03 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:03 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:03 smithi086 ceph-mon[110975]: Reconfiguring mon.smithi086 (monmap changed)... 2024-04-06T14:22:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:03 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:22:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:03 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:22:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:03 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:03 smithi086 ceph-mon[110975]: Reconfiguring daemon mon.smithi086 on smithi086 2024-04-06T14:22:04.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:03 smithi032 ceph-mon[147414]: pgmap v23: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:04.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:03 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:04.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:03 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:04.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:03 smithi032 ceph-mon[147414]: Reconfiguring mon.smithi086 (monmap changed)... 2024-04-06T14:22:04.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:03 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T14:22:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:03 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T14:22:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:03 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:03 smithi032 ceph-mon[147414]: Reconfiguring daemon mon.smithi086 on smithi086 2024-04-06T14:22:05.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:05 smithi086 ceph-mon[110975]: Reconfiguring osd.3 (monmap changed)... 2024-04-06T14:22:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T14:22:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:05 smithi086 ceph-mon[110975]: Reconfiguring daemon osd.3 on smithi086 2024-04-06T14:22:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:05 smithi032 ceph-mon[147414]: Reconfiguring osd.3 (monmap changed)... 2024-04-06T14:22:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T14:22:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:05.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:05 smithi032 ceph-mon[147414]: Reconfiguring daemon osd.3 on smithi086 2024-04-06T14:22:06.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:06 smithi086 ceph-mon[110975]: pgmap v24: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:06 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:06 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:06 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T14:22:06.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:06 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:06 smithi032 ceph-mon[147414]: pgmap v24: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:06 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:06 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:06 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T14:22:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:06 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:07.510 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:07 smithi086 ceph-mon[110975]: Reconfiguring osd.4 (monmap changed)... 2024-04-06T14:22:07.510 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:07 smithi086 ceph-mon[110975]: Reconfiguring daemon osd.4 on smithi086 2024-04-06T14:22:07.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:07 smithi032 ceph-mon[147414]: Reconfiguring osd.4 (monmap changed)... 2024-04-06T14:22:07.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:07 smithi032 ceph-mon[147414]: Reconfiguring daemon osd.4 on smithi086 2024-04-06T14:22:08.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:08 smithi086 ceph-mon[110975]: pgmap v25: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:22:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T14:22:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:08.659 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:08 smithi032 ceph-mon[147414]: pgmap v25: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:08.659 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:22:08.659 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:08.659 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:08.659 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T14:22:08.659 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:09.612 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:09 smithi086 ceph-mon[110975]: Reconfiguring osd.5 (monmap changed)... 2024-04-06T14:22:09.612 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:09 smithi086 ceph-mon[110975]: Reconfiguring daemon osd.5 on smithi086 2024-04-06T14:22:09.612 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:09 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:09.612 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:09 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:09.612 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:09 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.nikfjj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:22:09.612 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:09 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:09 smithi032 ceph-mon[147414]: Reconfiguring osd.5 (monmap changed)... 2024-04-06T14:22:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:09 smithi032 ceph-mon[147414]: Reconfiguring daemon osd.5 on smithi086 2024-04-06T14:22:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:09 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:09 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:09.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:09 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.nikfjj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:22:09.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:09 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:10.600 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:10 smithi086 ceph-mon[110975]: pgmap v26: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:10.600 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:10 smithi086 ceph-mon[110975]: Reconfiguring mds.cephfs.smithi086.nikfjj (monmap changed)... 2024-04-06T14:22:10.600 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:10 smithi086 ceph-mon[110975]: Reconfiguring daemon mds.cephfs.smithi086.nikfjj on smithi086 2024-04-06T14:22:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:10 smithi032 ceph-mon[147414]: pgmap v26: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:10 smithi032 ceph-mon[147414]: Reconfiguring mds.cephfs.smithi086.nikfjj (monmap changed)... 2024-04-06T14:22:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:10 smithi032 ceph-mon[147414]: Reconfiguring daemon mds.cephfs.smithi086.nikfjj on smithi086 2024-04-06T14:22:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:11 smithi032 ceph-mon[147414]: pgmap v27: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:11 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:11 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:11 smithi032 ceph-mon[147414]: Reconfiguring mds.cephfs.smithi086.ojmchr (monmap changed)... 2024-04-06T14:22:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:11 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.ojmchr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:22:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:11 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:11.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:11 smithi032 ceph-mon[147414]: Reconfiguring daemon mds.cephfs.smithi086.ojmchr on smithi086 2024-04-06T14:22:12.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:11 smithi086 ceph-mon[110975]: pgmap v27: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:12.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:11 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:12.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:11 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:12.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:11 smithi086 ceph-mon[110975]: Reconfiguring mds.cephfs.smithi086.ojmchr (monmap changed)... 2024-04-06T14:22:12.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:11 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.ojmchr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:22:12.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:11 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:12.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:11 smithi086 ceph-mon[110975]: Reconfiguring daemon mds.cephfs.smithi086.ojmchr on smithi086 2024-04-06T14:22:13.332 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:13.332 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:13.332 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:13.332 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:13.332 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:13.332 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all mon 2024-04-06T14:22:13.332 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:13.333 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]: dispatch 2024-04-06T14:22:13.333 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]': finished 2024-04-06T14:22:13.333 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi086"}]: dispatch 2024-04-06T14:22:13.333 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:13 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi086"}]': finished 2024-04-06T14:22:13.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all mon 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]: dispatch 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]': finished 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi086"}]: dispatch 2024-04-06T14:22:13.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:13 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi086"}]': finished 2024-04-06T14:22:13.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.800+0000 7f320f4e1700 1 -- 172.21.15.32:0/4112769814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 msgr2=0x7f32080fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.800+0000 7f320f4e1700 1 --2- 172.21.15.32:0/4112769814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 0x7f32080fe260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3204004940 tx=0x7f3204009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.800+0000 7f320f4e1700 1 -- 172.21.15.32:0/4112769814 shutdown_connections 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.800+0000 7f320f4e1700 1 --2- 172.21.15.32:0/4112769814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32080fe830 0x7f3208106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.800+0000 7f320f4e1700 1 --2- 172.21.15.32:0/4112769814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 0x7f32080fe260 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.800+0000 7f320f4e1700 1 -- 172.21.15.32:0/4112769814 >> 172.21.15.32:0/4112769814 conn(0x7f3208072bc0 msgr2=0x7f3208072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.800+0000 7f320f4e1700 1 -- 172.21.15.32:0/4112769814 shutdown_connections 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.800+0000 7f320f4e1700 1 -- 172.21.15.32:0/4112769814 wait complete. 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320f4e1700 1 Processor -- start 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320f4e1700 1 -- start start 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320f4e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 0x7f3208113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320f4e1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32080fe830 0x7f3208114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320f4e1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3208114680 con 0x7f32080fde90 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320f4e1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320810da70 con 0x7f32080fe830 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320d27d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 0x7f3208113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320d27d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 0x7f3208113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40826/0 (socket says 172.21.15.32:40826) 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320d27d700 1 -- 172.21.15.32:0/2479615431 learned_addr learned my addr 172.21.15.32:0/2479615431 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.801+0000 7f320ca7c700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32080fe830 0x7f3208114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.802+0000 7f320d27d700 1 -- 172.21.15.32:0/2479615431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32080fe830 msgr2=0x7f3208114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.802+0000 7f320d27d700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32080fe830 0x7f3208114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.802+0000 7f320d27d700 1 -- 172.21.15.32:0/2479615431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3204005040 con 0x7f32080fde90 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.802+0000 7f320d27d700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 0x7f3208113af0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32040093b0 tx=0x7f3204009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.802+0000 7f31fdffb700 1 -- 172.21.15.32:0/2479615431 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f320400e040 con 0x7f32080fde90 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.802+0000 7f31fdffb700 1 -- 172.21.15.32:0/2479615431 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f320400bed0 con 0x7f32080fde90 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.802+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320810dcf0 con 0x7f32080fde90 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.802+0000 7f31fdffb700 1 -- 172.21.15.32:0/2479615431 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f320401c3f0 con 0x7f32080fde90 2024-04-06T14:22:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.802+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320810e230 con 0x7f32080fde90 2024-04-06T14:22:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.804+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32081019b0 con 0x7f32080fde90 2024-04-06T14:22:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.804+0000 7f31fdffb700 1 -- 172.21.15.32:0/2479615431 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f320400ba20 con 0x7f32080fde90 2024-04-06T14:22:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.804+0000 7f31fdffb700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31f406cd40 0x7f31f406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.804+0000 7f31fdffb700 1 -- 172.21.15.32:0/2479615431 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f320408d470 con 0x7f32080fde90 2024-04-06T14:22:13.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.804+0000 7f320ca7c700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31f406cd40 0x7f31f406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:13.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.805+0000 7f320ca7c700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31f406cd40 0x7f31f406f1f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f320810ec30 tx=0x7f31f800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:13.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:13.809+0000 7f31fdffb700 1 -- 172.21.15.32:0/2479615431 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f320405ae60 con 0x7f32080fde90 2024-04-06T14:22:14.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.037+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f320810f140 con 0x7f31f406cd40 2024-04-06T14:22:14.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.039+0000 7f31fdffb700 1 -- 172.21.15.32:0/2479615431 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f320810f140 con 0x7f31f406cd40 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.041+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31f406cd40 msgr2=0x7f31f406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.041+0000 7f320f4e1700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31f406cd40 0x7f31f406f1f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f320810ec30 tx=0x7f31f800b040 comp rx=0 tx=0).stop 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.041+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 msgr2=0x7f3208113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.041+0000 7f320f4e1700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 0x7f3208113af0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32040093b0 tx=0x7f3204009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.041+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 shutdown_connections 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.041+0000 7f320f4e1700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31f406cd40 0x7f31f406f1f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.041+0000 7f320f4e1700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32080fde90 0x7f3208113af0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.041+0000 7f320f4e1700 1 --2- 172.21.15.32:0/2479615431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32080fe830 0x7f3208114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.041+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 >> 172.21.15.32:0/2479615431 conn(0x7f3208072bc0 msgr2=0x7f32080ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.042+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 shutdown_connections 2024-04-06T14:22:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.042+0000 7f320f4e1700 1 -- 172.21.15.32:0/2479615431 wait complete. 2024-04-06T14:22:14.054 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:22:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.151+0000 7f1a018f4700 1 -- 172.21.15.32:0/3630451889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 msgr2=0x7f19fc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.151+0000 7f1a018f4700 1 --2- 172.21.15.32:0/3630451889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 0x7f19fc106f00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f19f0004940 tx=0x7f19f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.151+0000 7f1a018f4700 1 -- 172.21.15.32:0/3630451889 shutdown_connections 2024-04-06T14:22:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.151+0000 7f1a018f4700 1 --2- 172.21.15.32:0/3630451889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 0x7f19fc106f00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.151+0000 7f1a018f4700 1 --2- 172.21.15.32:0/3630451889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc0fde90 0x7f19fc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.151+0000 7f1a018f4700 1 -- 172.21.15.32:0/3630451889 >> 172.21.15.32:0/3630451889 conn(0x7f19fc072bc0 msgr2=0x7f19fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.151+0000 7f1a018f4700 1 -- 172.21.15.32:0/3630451889 shutdown_connections 2024-04-06T14:22:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.152+0000 7f1a018f4700 1 -- 172.21.15.32:0/3630451889 wait complete. 2024-04-06T14:22:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.152+0000 7f1a018f4700 1 Processor -- start 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.152+0000 7f1a018f4700 1 -- start start 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.152+0000 7f1a018f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc0fde90 0x7f19fc113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.152+0000 7f1a018f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 0x7f19fc114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.152+0000 7f1a018f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc114670 con 0x7f19fc0fe830 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.152+0000 7f1a018f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc10da60 con 0x7f19fc0fde90 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19fa7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 0x7f19fc114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19fa7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 0x7f19fc114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40836/0 (socket says 172.21.15.32:40836) 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19fa7fc700 1 -- 172.21.15.32:0/2173628736 learned_addr learned my addr 172.21.15.32:0/2173628736 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19faffd700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc0fde90 0x7f19fc113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19fa7fc700 1 -- 172.21.15.32:0/2173628736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc0fde90 msgr2=0x7f19fc113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19fa7fc700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc0fde90 0x7f19fc113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19fa7fc700 1 -- 172.21.15.32:0/2173628736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f0005040 con 0x7f19fc0fe830 2024-04-06T14:22:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19fa7fc700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 0x7f19fc114020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19f000b3b0 tx=0x7f19f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19e37fe700 1 -- 172.21.15.32:0/2173628736 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19f000e040 con 0x7f19fc0fe830 2024-04-06T14:22:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19e37fe700 1 -- 172.21.15.32:0/2173628736 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19f0013420 con 0x7f19fc0fe830 2024-04-06T14:22:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19fc10dce0 con 0x7f19fc0fe830 2024-04-06T14:22:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f19e37fe700 1 -- 172.21.15.32:0/2173628736 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19f001c5d0 con 0x7f19fc0fe830 2024-04-06T14:22:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.153+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19fc10e220 con 0x7f19fc0fe830 2024-04-06T14:22:14.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.154+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc065e30 con 0x7f19fc0fe830 2024-04-06T14:22:14.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.155+0000 7f19e37fe700 1 -- 172.21.15.32:0/2173628736 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f19f0013590 con 0x7f19fc0fe830 2024-04-06T14:22:14.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.155+0000 7f19e37fe700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19e406cd40 0x7f19e406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:14.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.155+0000 7f19e37fe700 1 -- 172.21.15.32:0/2173628736 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f19f008d4a0 con 0x7f19fc0fe830 2024-04-06T14:22:14.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.155+0000 7f19faffd700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19e406cd40 0x7f19e406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:14.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.156+0000 7f19faffd700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19e406cd40 0x7f19e406f1f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19ec004990 tx=0x7f19ec009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:14.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.160+0000 7f19e37fe700 1 -- 172.21.15.32:0/2173628736 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f19f005ad70 con 0x7f19fc0fe830 2024-04-06T14:22:14.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.388+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19fc1137b0 con 0x7f19e406cd40 2024-04-06T14:22:14.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.389+0000 7f19e37fe700 1 -- 172.21.15.32:0/2173628736 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f19fc1137b0 con 0x7f19e406cd40 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.391+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19e406cd40 msgr2=0x7f19e406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.391+0000 7f1a018f4700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19e406cd40 0x7f19e406f1f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19ec004990 tx=0x7f19ec009ee0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.391+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 msgr2=0x7f19fc114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.391+0000 7f1a018f4700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 0x7f19fc114020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19f000b3b0 tx=0x7f19f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.391+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 shutdown_connections 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.391+0000 7f1a018f4700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19e406cd40 0x7f19e406f1f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.391+0000 7f1a018f4700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc0fde90 0x7f19fc113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.391+0000 7f1a018f4700 1 --2- 172.21.15.32:0/2173628736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0fe830 0x7f19fc114020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.391+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 >> 172.21.15.32:0/2173628736 conn(0x7f19fc072bc0 msgr2=0x7f19fc103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.392+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 shutdown_connections 2024-04-06T14:22:14.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.392+0000 7f1a018f4700 1 -- 172.21.15.32:0/2173628736 wait complete. 2024-04-06T14:22:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:14 smithi032 ceph-mon[147414]: pgmap v28: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:22:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:14.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.496+0000 7f61a03fb700 1 -- 172.21.15.32:0/70936314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61980fde90 msgr2=0x7f61980fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.496+0000 7f61a03fb700 1 --2- 172.21.15.32:0/70936314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61980fde90 0x7f61980fe260 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6194004940 tx=0x7f6194009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:14.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.498+0000 7f61a03fb700 1 -- 172.21.15.32:0/70936314 shutdown_connections 2024-04-06T14:22:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.498+0000 7f61a03fb700 1 --2- 172.21.15.32:0/70936314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61980fe830 0x7f6198106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.498+0000 7f61a03fb700 1 --2- 172.21.15.32:0/70936314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61980fde90 0x7f61980fe260 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.498+0000 7f61a03fb700 1 -- 172.21.15.32:0/70936314 >> 172.21.15.32:0/70936314 conn(0x7f6198072bc0 msgr2=0x7f6198072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.498+0000 7f61a03fb700 1 -- 172.21.15.32:0/70936314 shutdown_connections 2024-04-06T14:22:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.498+0000 7f61a03fb700 1 -- 172.21.15.32:0/70936314 wait complete. 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.499+0000 7f61a03fb700 1 Processor -- start 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.499+0000 7f61a03fb700 1 -- start start 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.499+0000 7f61a03fb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61980fde90 0x7f61980fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.499+0000 7f61a03fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61980fe830 0x7f61980fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.499+0000 7f61a03fb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980fc820 con 0x7f61980fe830 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.499+0000 7f61a03fb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980fc960 con 0x7f61980fde90 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.499+0000 7f619d996700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61980fe830 0x7f61980fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.499+0000 7f619d996700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61980fe830 0x7f61980fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58228/0 (socket says 172.21.15.32:58228) 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f619d996700 1 -- 172.21.15.32:0/1754115191 learned_addr learned my addr 172.21.15.32:0/1754115191 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f619e197700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61980fde90 0x7f61980fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f619d996700 1 -- 172.21.15.32:0/1754115191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61980fde90 msgr2=0x7f61980fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f619d996700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61980fde90 0x7f61980fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f619d996700 1 -- 172.21.15.32:0/1754115191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6194005040 con 0x7f61980fe830 2024-04-06T14:22:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f619d996700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61980fe830 0x7f61980fc290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f618800eb40 tx=0x7f618800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f618effd700 1 -- 172.21.15.32:0/1754115191 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6188007cc0 con 0x7f61980fe830 2024-04-06T14:22:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f618effd700 1 -- 172.21.15.32:0/1754115191 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f618800f040 con 0x7f61980fe830 2024-04-06T14:22:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61980fcbb0 con 0x7f61980fe830 2024-04-06T14:22:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f618effd700 1 -- 172.21.15.32:0/1754115191 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f618800c990 con 0x7f61980fe830 2024-04-06T14:22:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.500+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619811a0a0 con 0x7f61980fe830 2024-04-06T14:22:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.502+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6198065e30 con 0x7f61980fe830 2024-04-06T14:22:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.502+0000 7f618effd700 1 -- 172.21.15.32:0/1754115191 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f61880090d0 con 0x7f61980fe830 2024-04-06T14:22:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.503+0000 7f618effd700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f618406cd40 0x7f618406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.503+0000 7f618effd700 1 -- 172.21.15.32:0/1754115191 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f618808bd60 con 0x7f61980fe830 2024-04-06T14:22:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.503+0000 7f619e197700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f618406cd40 0x7f618406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.503+0000 7f619e197700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f618406cd40 0x7f618406f1f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6194009fd0 tx=0x7f6194009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:14.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.508+0000 7f618effd700 1 -- 172.21.15.32:0/1754115191 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6188055cd0 con 0x7f61980fe830 2024-04-06T14:22:14.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:14 smithi086 ceph-mon[110975]: pgmap v28: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:14.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:14.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:22:14.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:14.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.725+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f619804bed0 con 0x7f618406cd40 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.735+0000 7f618effd700 1 -- 172.21.15.32:0/1754115191 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f619804bed0 con 0x7f618406cd40 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (2m) 46s ago 13m 19.1M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (14m) 46s ago 14m 6651k - 17.2.7-819-g6699495c ec300eaa768b 0dd0bdf837c1 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (13m) 32s ago 13m 6660k - 17.2.7-819-g6699495c ec300eaa768b bae54552a931 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (2m) 46s ago 13m 80.0M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (8m) 46s ago 8m 13.4M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (8m) 46s ago 8m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (8m) 32s ago 8m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:22:14.735 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (8m) 32s ago 7m 16.4M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (4m) 46s ago 15m 470M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (3m) 32s ago 12m 446M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (61s) 46s ago 15m 26.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (35s) 32s ago 12m 22.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (3m) 46s ago 14m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (3m) 32s ago 13m 17.1M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (12m) 46s ago 12m 195M 1570M 17.2.7-819-g6699495c ec300eaa768b bf09582cddc4 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 46s ago 11m 192M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 46s ago 11m 163M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (10m) 32s ago 10m 225M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (10m) 32s ago 10m 202M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (9m) 32s ago 9m 187M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:22:14.736 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (2m) 46s ago 13m 56.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:22:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.736+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f618406cd40 msgr2=0x7f618406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.736+0000 7f61a03fb700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f618406cd40 0x7f618406f1f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6194009fd0 tx=0x7f6194009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.736+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61980fe830 msgr2=0x7f61980fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.736+0000 7f61a03fb700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61980fe830 0x7f61980fc290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f618800eb40 tx=0x7f618800b040 comp rx=0 tx=0).stop 2024-04-06T14:22:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.737+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 shutdown_connections 2024-04-06T14:22:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.737+0000 7f61a03fb700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f618406cd40 0x7f618406f1f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.737+0000 7f61a03fb700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61980fde90 0x7f61980fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.737+0000 7f61a03fb700 1 --2- 172.21.15.32:0/1754115191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61980fe830 0x7f61980fc290 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.737+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 >> 172.21.15.32:0/1754115191 conn(0x7f6198072bc0 msgr2=0x7f61980ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:14.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.737+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 shutdown_connections 2024-04-06T14:22:14.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.737+0000 7f61a03fb700 1 -- 172.21.15.32:0/1754115191 wait complete. 2024-04-06T14:22:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.840+0000 7fe10dd0b700 1 -- 172.21.15.32:0/3939927978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1081037b0 msgr2=0x7fe1080fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.840+0000 7fe10dd0b700 1 --2- 172.21.15.32:0/3939927978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1081037b0 0x7fe1080fce20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0004940 tx=0x7fe0f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.840+0000 7fe10dd0b700 1 -- 172.21.15.32:0/3939927978 shutdown_connections 2024-04-06T14:22:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.840+0000 7fe10dd0b700 1 --2- 172.21.15.32:0/3939927978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1081037b0 0x7fe1080fce20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.840+0000 7fe10dd0b700 1 --2- 172.21.15.32:0/3939927978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe108102e10 0x7fe1081031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.840+0000 7fe10dd0b700 1 -- 172.21.15.32:0/3939927978 >> 172.21.15.32:0/3939927978 conn(0x7fe1080f89a0 msgr2=0x7fe1080fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.840+0000 7fe10dd0b700 1 -- 172.21.15.32:0/3939927978 shutdown_connections 2024-04-06T14:22:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.840+0000 7fe10dd0b700 1 -- 172.21.15.32:0/3939927978 wait complete. 2024-04-06T14:22:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe10dd0b700 1 Processor -- start 2024-04-06T14:22:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe10dd0b700 1 -- start start 2024-04-06T14:22:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe10dd0b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe108102e10 0x7fe108113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe10dd0b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1081037b0 0x7fe108114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe10dd0b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe108114660 con 0x7fe108102e10 2024-04-06T14:22:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe10dd0b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10810da50 con 0x7fe1081037b0 2024-04-06T14:22:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe1077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe108102e10 0x7fe108113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe1077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe108102e10 0x7fe108113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58254/0 (socket says 172.21.15.32:58254) 2024-04-06T14:22:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe1077fe700 1 -- 172.21.15.32:0/1949562871 learned_addr learned my addr 172.21.15.32:0/1949562871 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.841+0000 7fe106ffd700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1081037b0 0x7fe108114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.842+0000 7fe1077fe700 1 -- 172.21.15.32:0/1949562871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1081037b0 msgr2=0x7fe108114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.842+0000 7fe1077fe700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1081037b0 0x7fe108114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.842+0000 7fe1077fe700 1 -- 172.21.15.32:0/1949562871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f0005040 con 0x7fe108102e10 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.842+0000 7fe1077fe700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe108102e10 0x7fe108113ad0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800c830 tx=0x7fe0f8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.842+0000 7fe0fffff700 1 -- 172.21.15.32:0/1949562871 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f80106f0 con 0x7fe108102e10 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.842+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10810dd30 con 0x7fe108102e10 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.842+0000 7fe0fffff700 1 -- 172.21.15.32:0/1949562871 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0f8010d00 con 0x7fe108102e10 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.842+0000 7fe0fffff700 1 -- 172.21.15.32:0/1949562871 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f8012e90 con 0x7fe108102e10 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.842+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10810e2c0 con 0x7fe108102e10 2024-04-06T14:22:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.843+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1080fe560 con 0x7fe108102e10 2024-04-06T14:22:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.843+0000 7fe0fffff700 1 -- 172.21.15.32:0/1949562871 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fe0f8010850 con 0x7fe108102e10 2024-04-06T14:22:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.844+0000 7fe0fffff700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f406cd40 0x7fe0f406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.844+0000 7fe0fffff700 1 -- 172.21.15.32:0/1949562871 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fe0f808c0e0 con 0x7fe108102e10 2024-04-06T14:22:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.844+0000 7fe106ffd700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f406cd40 0x7fe0f406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.844+0000 7fe106ffd700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f406cd40 0x7fe0f406f1f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe10810ed30 tx=0x7fe0f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:14.848+0000 7fe0fffff700 1 -- 172.21.15.32:0/1949562871 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe0f8059b40 con 0x7fe108102e10 2024-04-06T14:22:15.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.142+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe10805f5f0 con 0x7fe108102e10 2024-04-06T14:22:15.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.143+0000 7fe0fffff700 1 -- 172.21.15.32:0/1949562871 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+742 (secure 0 0 0) 0x7fe0f80596d0 con 0x7fe108102e10 2024-04-06T14:22:15.143 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:22:15.143 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:22:15.143 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:22:15.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:22:15.143 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:22:15.143 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 10, 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:22:15.144 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:22:15.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f406cd40 msgr2=0x7fe0f406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f406cd40 0x7fe0f406f1f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe10810ed30 tx=0x7fe0f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:15.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe108102e10 msgr2=0x7fe108113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe108102e10 0x7fe108113ad0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800c830 tx=0x7fe0f8004470 comp rx=0 tx=0).stop 2024-04-06T14:22:15.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 shutdown_connections 2024-04-06T14:22:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f406cd40 0x7fe0f406f1f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe108102e10 0x7fe108113ad0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 --2- 172.21.15.32:0/1949562871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1081037b0 0x7fe108114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 >> 172.21.15.32:0/1949562871 conn(0x7fe1080f89a0 msgr2=0x7fe1080f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 shutdown_connections 2024-04-06T14:22:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.145+0000 7fe10dd0b700 1 -- 172.21.15.32:0/1949562871 wait complete. 2024-04-06T14:22:15.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:15 smithi032 ceph-mon[147414]: Upgrade: Updating crash.smithi032 (1/2) 2024-04-06T14:22:15.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:15 smithi032 ceph-mon[147414]: Deploying daemon crash.smithi032 on smithi032 2024-04-06T14:22:15.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:15 smithi032 ceph-mon[147414]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:15.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:15 smithi032 ceph-mon[147414]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:15.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:15 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1949562871' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:15.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.250+0000 7fac3f9ef700 1 -- 172.21.15.32:0/601993267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fbc80 msgr2=0x7fac380fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.250+0000 7fac3f9ef700 1 --2- 172.21.15.32:0/601993267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fbc80 0x7fac380fc050 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fac34004940 tx=0x7fac34009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.250+0000 7fac3f9ef700 1 -- 172.21.15.32:0/601993267 shutdown_connections 2024-04-06T14:22:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.250+0000 7fac3f9ef700 1 --2- 172.21.15.32:0/601993267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac380fc590 0x7fac381092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.250+0000 7fac3f9ef700 1 --2- 172.21.15.32:0/601993267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fbc80 0x7fac380fc050 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.250+0000 7fac3f9ef700 1 -- 172.21.15.32:0/601993267 >> 172.21.15.32:0/601993267 conn(0x7fac38072bc0 msgr2=0x7fac38072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.250+0000 7fac3f9ef700 1 -- 172.21.15.32:0/601993267 shutdown_connections 2024-04-06T14:22:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.250+0000 7fac3f9ef700 1 -- 172.21.15.32:0/601993267 wait complete. 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3f9ef700 1 Processor -- start 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3f9ef700 1 -- start start 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3f9ef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac380fbc80 0x7fac38115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3f9ef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fc590 0x7fac38116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3f9ef700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac381168c0 con 0x7fac380fc590 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3f9ef700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3810fc90 con 0x7fac380fbc80 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3cf8a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fc590 0x7fac38116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3cf8a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fc590 0x7fac38116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58258/0 (socket says 172.21.15.32:58258) 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3cf8a700 1 -- 172.21.15.32:0/733526837 learned_addr learned my addr 172.21.15.32:0/733526837 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.251+0000 7fac3d78b700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac380fbc80 0x7fac38115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.252+0000 7fac3cf8a700 1 -- 172.21.15.32:0/733526837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac380fbc80 msgr2=0x7fac38115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.252+0000 7fac3cf8a700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac380fbc80 0x7fac38115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.252+0000 7fac3cf8a700 1 -- 172.21.15.32:0/733526837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac34005040 con 0x7fac380fc590 2024-04-06T14:22:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.252+0000 7fac3cf8a700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fc590 0x7fac38116270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac28002a50 tx=0x7fac280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.252+0000 7fac2dffb700 1 -- 172.21.15.32:0/733526837 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac28007cc0 con 0x7fac380fc590 2024-04-06T14:22:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.252+0000 7fac2dffb700 1 -- 172.21.15.32:0/733526837 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac2800f040 con 0x7fac380fc590 2024-04-06T14:22:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.252+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3810ff70 con 0x7fac380fc590 2024-04-06T14:22:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.252+0000 7fac2dffb700 1 -- 172.21.15.32:0/733526837 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac2800d9c0 con 0x7fac380fc590 2024-04-06T14:22:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.252+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac381104b0 con 0x7fac380fc590 2024-04-06T14:22:15.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.253+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac38065e30 con 0x7fac380fc590 2024-04-06T14:22:15.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.254+0000 7fac2dffb700 1 -- 172.21.15.32:0/733526837 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fac2800db20 con 0x7fac380fc590 2024-04-06T14:22:15.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.255+0000 7fac2dffb700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac2406cd40 0x7fac2406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:15.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.255+0000 7fac2dffb700 1 -- 172.21.15.32:0/733526837 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fac2808bcb0 con 0x7fac380fc590 2024-04-06T14:22:15.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.255+0000 7fac3d78b700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac2406cd40 0x7fac2406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:15.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.255+0000 7fac3d78b700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac2406cd40 0x7fac2406f1f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fac34009fd0 tx=0x7fac34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.259+0000 7fac2dffb700 1 -- 172.21.15.32:0/733526837 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fac280596a0 con 0x7fac380fc590 2024-04-06T14:22:15.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.518+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fac3810c020 con 0x7fac380fc590 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:21:28.458361+0000 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:22:15.520 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:22:15.521 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:22:15.522 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:22:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.519+0000 7fac2dffb700 1 -- 172.21.15.32:0/733526837 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1851 (secure 0 0 0) 0x7fac28022750 con 0x7fac380fc590 2024-04-06T14:22:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac2406cd40 msgr2=0x7fac2406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac2406cd40 0x7fac2406f1f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fac34009fd0 tx=0x7fac34009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fc590 msgr2=0x7fac38116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fc590 0x7fac38116270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac28002a50 tx=0x7fac280046b0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 shutdown_connections 2024-04-06T14:22:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac2406cd40 0x7fac2406f1f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac380fbc80 0x7fac38115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 --2- 172.21.15.32:0/733526837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac380fc590 0x7fac38116270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 >> 172.21.15.32:0/733526837 conn(0x7fac38072bc0 msgr2=0x7fac38105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 shutdown_connections 2024-04-06T14:22:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.521+0000 7fac3f9ef700 1 -- 172.21.15.32:0/733526837 wait complete. 2024-04-06T14:22:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-06T14:22:15.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:15 smithi086 ceph-mon[110975]: Upgrade: Updating crash.smithi032 (1/2) 2024-04-06T14:22:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:15 smithi086 ceph-mon[110975]: Deploying daemon crash.smithi032 on smithi032 2024-04-06T14:22:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:15 smithi086 ceph-mon[110975]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:15 smithi086 ceph-mon[110975]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:15 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1949562871' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.638+0000 7fab6f022700 1 -- 172.21.15.32:0/4242563722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68102710 msgr2=0x7fab680fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.638+0000 7fab6f022700 1 --2- 172.21.15.32:0/4242563722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68102710 0x7fab680fbd80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fab50004940 tx=0x7fab50009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.638+0000 7fab6f022700 1 -- 172.21.15.32:0/4242563722 shutdown_connections 2024-04-06T14:22:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.638+0000 7fab6f022700 1 --2- 172.21.15.32:0/4242563722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68102710 0x7fab680fbd80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.638+0000 7fab6f022700 1 --2- 172.21.15.32:0/4242563722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab68101d70 0x7fab68102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.638+0000 7fab6f022700 1 -- 172.21.15.32:0/4242563722 >> 172.21.15.32:0/4242563722 conn(0x7fab68072bc0 msgr2=0x7fab68072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.638+0000 7fab6f022700 1 -- 172.21.15.32:0/4242563722 shutdown_connections 2024-04-06T14:22:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.638+0000 7fab6f022700 1 -- 172.21.15.32:0/4242563722 wait complete. 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.639+0000 7fab6f022700 1 Processor -- start 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.639+0000 7fab6f022700 1 -- start start 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6f022700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68101d70 0x7fab6810ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6f022700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab68102710 0x7fab6810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6f022700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6810f5f0 con 0x7fab68101d70 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6f022700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6810f730 con 0x7fab68102710 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6cdbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68101d70 0x7fab6810ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6cdbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68101d70 0x7fab6810ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58264/0 (socket says 172.21.15.32:58264) 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6cdbe700 1 -- 172.21.15.32:0/250672740 learned_addr learned my addr 172.21.15.32:0/250672740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6cdbe700 1 -- 172.21.15.32:0/250672740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab68102710 msgr2=0x7fab6810efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6cdbe700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab68102710 0x7fab6810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6cdbe700 1 -- 172.21.15.32:0/250672740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab50005040 con 0x7fab68101d70 2024-04-06T14:22:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6cdbe700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68101d70 0x7fab6810ea60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab6000cb10 tx=0x7fab60007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab657fa700 1 -- 172.21.15.32:0/250672740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab60004200 con 0x7fab68101d70 2024-04-06T14:22:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab657fa700 1 -- 172.21.15.32:0/250672740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab6000aca0 con 0x7fab68101d70 2024-04-06T14:22:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab68117ad0 con 0x7fab68101d70 2024-04-06T14:22:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.640+0000 7fab657fa700 1 -- 172.21.15.32:0/250672740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab60003ce0 con 0x7fab68101d70 2024-04-06T14:22:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.641+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab68118030 con 0x7fab68101d70 2024-04-06T14:22:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.642+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab680ff080 con 0x7fab68101d70 2024-04-06T14:22:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.642+0000 7fab657fa700 1 -- 172.21.15.32:0/250672740 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fab6000a7f0 con 0x7fab68101d70 2024-04-06T14:22:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.642+0000 7fab657fa700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab5406cd40 0x7fab5406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.643+0000 7fab657fa700 1 -- 172.21.15.32:0/250672740 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fab6008c6b0 con 0x7fab68101d70 2024-04-06T14:22:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.643+0000 7fab67fff700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab5406cd40 0x7fab5406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.643+0000 7fab67fff700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab5406cd40 0x7fab5406f1f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab50004f90 tx=0x7fab50009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:15.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.647+0000 7fab657fa700 1 -- 172.21.15.32:0/250672740 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fab60056600 con 0x7fab68101d70 2024-04-06T14:22:15.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.875+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fab68108d10 con 0x7fab5406cd40 2024-04-06T14:22:15.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.876+0000 7fab657fa700 1 -- 172.21.15.32:0/250672740 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7fab68108d10 con 0x7fab5406cd40 2024-04-06T14:22:15.876 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "4/21 daemons upgraded", 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading crash daemons", 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:22:15.877 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab5406cd40 msgr2=0x7fab5406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab5406cd40 0x7fab5406f1f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab50004f90 tx=0x7fab50009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68101d70 msgr2=0x7fab6810ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68101d70 0x7fab6810ea60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab6000cb10 tx=0x7fab60007f90 comp rx=0 tx=0).stop 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 shutdown_connections 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab5406cd40 0x7fab5406f1f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab68101d70 0x7fab6810ea60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 --2- 172.21.15.32:0/250672740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab68102710 0x7fab6810efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 >> 172.21.15.32:0/250672740 conn(0x7fab68072bc0 msgr2=0x7fab680fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.878+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 shutdown_connections 2024-04-06T14:22:15.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.879+0000 7fab6f022700 1 -- 172.21.15.32:0/250672740 wait complete. 2024-04-06T14:22:15.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.983+0000 7f7aaa911700 1 -- 172.21.15.32:0/2182533025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4102710 msgr2=0x7f7aa40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.983+0000 7f7aaa911700 1 --2- 172.21.15.32:0/2182533025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4102710 0x7f7aa40fbd80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a98004940 tx=0x7f7a98009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.983+0000 7f7aaa911700 1 -- 172.21.15.32:0/2182533025 shutdown_connections 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.983+0000 7f7aaa911700 1 --2- 172.21.15.32:0/2182533025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4102710 0x7f7aa40fbd80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.983+0000 7f7aaa911700 1 --2- 172.21.15.32:0/2182533025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aa4101d70 0x7f7aa4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.983+0000 7f7aaa911700 1 -- 172.21.15.32:0/2182533025 >> 172.21.15.32:0/2182533025 conn(0x7f7aa4072bc0 msgr2=0x7f7aa4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.984+0000 7f7aaa911700 1 -- 172.21.15.32:0/2182533025 shutdown_connections 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.984+0000 7f7aaa911700 1 -- 172.21.15.32:0/2182533025 wait complete. 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.984+0000 7f7aaa911700 1 Processor -- start 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.984+0000 7f7aaa911700 1 -- start start 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.984+0000 7f7aaa911700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4101d70 0x7f7aa4113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.984+0000 7f7aaa911700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aa4102710 0x7f7aa4114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.984+0000 7f7aaa911700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa4114650 con 0x7f7aa4101d70 2024-04-06T14:22:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.984+0000 7f7aaa911700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa410da40 con 0x7f7aa4102710 2024-04-06T14:22:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4101d70 0x7f7aa4113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4101d70 0x7f7aa4113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58290/0 (socket says 172.21.15.32:58290) 2024-04-06T14:22:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa3fff700 1 -- 172.21.15.32:0/482196808 learned_addr learned my addr 172.21.15.32:0/482196808 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa37fe700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aa4102710 0x7f7aa4114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa3fff700 1 -- 172.21.15.32:0/482196808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aa4102710 msgr2=0x7f7aa4114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa3fff700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aa4102710 0x7f7aa4114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa3fff700 1 -- 172.21.15.32:0/482196808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a98005040 con 0x7f7aa4101d70 2024-04-06T14:22:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa3fff700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4101d70 0x7f7aa4113ac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7a94002a50 tx=0x7f7a940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa0ff9700 1 -- 172.21.15.32:0/482196808 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a94007cc0 con 0x7f7aa4101d70 2024-04-06T14:22:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa0ff9700 1 -- 172.21.15.32:0/482196808 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a9400f040 con 0x7f7aa4101d70 2024-04-06T14:22:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa410dd20 con 0x7f7aa4101d70 2024-04-06T14:22:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aa0ff9700 1 -- 172.21.15.32:0/482196808 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a9400d9c0 con 0x7f7aa4101d70 2024-04-06T14:22:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.985+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa410e2b0 con 0x7f7aa4101d70 2024-04-06T14:22:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.986+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa40fd4c0 con 0x7f7aa4101d70 2024-04-06T14:22:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.987+0000 7f7aa0ff9700 1 -- 172.21.15.32:0/482196808 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f7a9400db20 con 0x7f7aa4101d70 2024-04-06T14:22:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.987+0000 7f7aa0ff9700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a8c06cd40 0x7f7a8c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.987+0000 7f7aa0ff9700 1 -- 172.21.15.32:0/482196808 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7a9408be90 con 0x7f7aa4101d70 2024-04-06T14:22:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.987+0000 7f7aa37fe700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a8c06cd40 0x7f7a8c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.988+0000 7f7aa37fe700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a8c06cd40 0x7f7a8c06f1f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7aa410ed20 tx=0x7f7a98009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:15.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:15.992+0000 7f7aa0ff9700 1 -- 172.21.15.32:0/482196808 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7a940597e0 con 0x7f7aa4101d70 2024-04-06T14:22:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.284+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7aa405f5f0 con 0x7f7aa4101d70 2024-04-06T14:22:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.285+0000 7f7aa0ff9700 1 -- 172.21.15.32:0/482196808 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f7a94059370 con 0x7f7aa4101d70 2024-04-06T14:22:16.285 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:22:16.285 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:22:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:22:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a8c06cd40 msgr2=0x7f7a8c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a8c06cd40 0x7f7a8c06f1f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7aa410ed20 tx=0x7f7a98009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4101d70 msgr2=0x7f7aa4113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4101d70 0x7f7aa4113ac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7a94002a50 tx=0x7f7a940046b0 comp rx=0 tx=0).stop 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 shutdown_connections 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a8c06cd40 0x7f7a8c06f1f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa4101d70 0x7f7aa4113ac0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 --2- 172.21.15.32:0/482196808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aa4102710 0x7f7aa4114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 >> 172.21.15.32:0/482196808 conn(0x7f7aa4072bc0 msgr2=0x7f7aa40659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 shutdown_connections 2024-04-06T14:22:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:16.287+0000 7f7aaa911700 1 -- 172.21.15.32:0/482196808 wait complete. 2024-04-06T14:22:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:16 smithi032 ceph-mon[147414]: pgmap v29: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:16 smithi032 ceph-mon[147414]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:16 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/733526837' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:22:16.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:16 smithi086 ceph-mon[110975]: pgmap v29: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:16.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:16 smithi086 ceph-mon[110975]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:16.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:16 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/733526837' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:22:17.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:17 smithi032 ceph-mon[147414]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:17.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:17 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/482196808' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:22:17.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:17 smithi086 ceph-mon[110975]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:17.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:17 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/482196808' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:22:18.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:18 smithi086 ceph-mon[110975]: pgmap v30: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:18.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:18 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:18.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:18 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:18 smithi032 ceph-mon[147414]: pgmap v30: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:18 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:18 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:20.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:20 smithi086 ceph-mon[110975]: pgmap v31: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:20 smithi086 ceph-mon[110975]: Upgrade: Updating crash.smithi086 (2/2) 2024-04-06T14:22:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:20 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:20 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:22:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:20 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:20 smithi086 ceph-mon[110975]: Deploying daemon crash.smithi086 on smithi086 2024-04-06T14:22:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:20 smithi032 ceph-mon[147414]: pgmap v31: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:20 smithi032 ceph-mon[147414]: Upgrade: Updating crash.smithi086 (2/2) 2024-04-06T14:22:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:20 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:20 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T14:22:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:20 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:20 smithi032 ceph-mon[147414]: Deploying daemon crash.smithi086 on smithi086 2024-04-06T14:22:22.366 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:22 smithi086 ceph-mon[110975]: pgmap v32: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:22.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:22 smithi032 ceph-mon[147414]: pgmap v32: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:23 smithi032 ceph-mon[147414]: pgmap v33: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:22:23.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:23.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:23.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:23.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:23 smithi086 ceph-mon[110975]: pgmap v33: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:23.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:22:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:26 smithi032 ceph-mon[147414]: pgmap v34: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:26 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:26 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:26 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:26.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:26 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:26.572 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:26 smithi086 ceph-mon[110975]: pgmap v34: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:26.572 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:26 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:26.572 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:26 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:26.572 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:26 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:26.572 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:26 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:28.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:28 smithi032 ceph-mon[147414]: pgmap v35: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:28.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:28 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:28.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:28 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:28.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:28 smithi086 ceph-mon[110975]: pgmap v35: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:28.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:28 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:28.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:28 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:29 smithi032 ceph-mon[147414]: pgmap v36: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:29 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:29.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:29 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:29.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:29 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:29.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:29 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:29 smithi086 ceph-mon[110975]: pgmap v36: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:29 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:29 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:29 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:30.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:29 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:31.943 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:31 smithi032 ceph-mon[147414]: pgmap v37: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:31 smithi086 ceph-mon[110975]: pgmap v37: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:33.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:33.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:33.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:33.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:22:33.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:33.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:33.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:33.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:33.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:33.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:33.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]: dispatch 2024-04-06T14:22:33.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]': finished 2024-04-06T14:22:33.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi086"}]: dispatch 2024-04-06T14:22:33.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi086"}]': finished 2024-04-06T14:22:33.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T14:22:33.629 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:33.629 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]: dispatch 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]': finished 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi086"}]: dispatch 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi086"}]': finished 2024-04-06T14:22:33.630 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T14:22:34.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:34 smithi086 ceph-mon[110975]: pgmap v38: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:34.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:34 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all crash 2024-04-06T14:22:34.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:34 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T14:22:34.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:34 smithi086 ceph-mon[110975]: Upgrade: osd.0 is safe to restart 2024-04-06T14:22:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:34 smithi032 ceph-mon[147414]: pgmap v38: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:34 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all crash 2024-04-06T14:22:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:34 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T14:22:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:34 smithi032 ceph-mon[147414]: Upgrade: osd.0 is safe to restart 2024-04-06T14:22:35.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:35 smithi086 ceph-mon[110975]: Upgrade: Updating osd.0 2024-04-06T14:22:35.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:35 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:35.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:35 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T14:22:35.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:35 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:35.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:35 smithi086 ceph-mon[110975]: Deploying daemon osd.0 on smithi032 2024-04-06T14:22:35.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:35 smithi086 ceph-mon[110975]: pgmap v39: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:35.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:35 smithi032 ceph-mon[147414]: Upgrade: Updating osd.0 2024-04-06T14:22:35.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:35 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:35.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:35 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T14:22:35.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:35 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:35.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:35 smithi032 ceph-mon[147414]: Deploying daemon osd.0 on smithi032 2024-04-06T14:22:35.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:35 smithi032 ceph-mon[147414]: pgmap v39: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:37.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:37 smithi086 ceph-mon[110975]: pgmap v40: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:37.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:22:37.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:37 smithi032 ceph-mon[147414]: pgmap v40: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:37.922 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:22:38.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:38 smithi032 systemd[1]: Stopping Ceph osd.0 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:22:38.602 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:38 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[63509]: 2024-04-06T14:22:38.341+0000 7f8dd2fa1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T14:22:38.602 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:38 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[63509]: 2024-04-06T14:22:38.341+0000 7f8dd2fa1700 -1 osd.0 50 *** Got signal Terminated *** 2024-04-06T14:22:38.603 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:38 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[63509]: 2024-04-06T14:22:38.341+0000 7f8dd2fa1700 -1 osd.0 50 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T14:22:38.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:38 smithi086 ceph-mon[110975]: osd.0 marked itself down and dead 2024-04-06T14:22:38.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:38 smithi032 ceph-mon[147414]: osd.0 marked itself down and dead 2024-04-06T14:22:39.610 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:39 smithi032 bash[160776]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0 2024-04-06T14:22:39.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:39 smithi086 ceph-mon[110975]: pgmap v41: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:39.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:39 smithi086 ceph-mon[110975]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:22:39.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:39 smithi086 ceph-mon[110975]: osdmap e51: 6 total, 5 up, 6 in 2024-04-06T14:22:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:39 smithi032 ceph-mon[147414]: pgmap v41: 65 pgs: 65 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:39 smithi032 ceph-mon[147414]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:22:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:39 smithi032 ceph-mon[147414]: osdmap e51: 6 total, 5 up, 6 in 2024-04-06T14:22:40.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:40 smithi086 ceph-mon[110975]: osdmap e52: 6 total, 5 up, 6 in 2024-04-06T14:22:40.926 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:40 smithi032 ceph-mon[147414]: osdmap e52: 6 total, 5 up, 6 in 2024-04-06T14:22:41.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:41 smithi032 ceph-mon[147414]: pgmap v44: 65 pgs: 9 stale+active+clean, 56 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:41.731 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:41 smithi032 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.0.service: Succeeded. 2024-04-06T14:22:41.731 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:41 smithi032 systemd[1]: Stopped Ceph osd.0 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:22:41.731 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:41 smithi032 systemd[1]: Starting Ceph osd.0 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:22:41.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:41 smithi086 ceph-mon[110975]: pgmap v44: 65 pgs: 9 stale+active+clean, 56 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:42.909 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:42 smithi032 bash[161052]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T14:22:42.909 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:42 smithi032 bash[161052]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T14:22:42.909 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:42 smithi032 bash[161052]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T14:22:42.910 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:42 smithi032 bash[161052]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T14:22:42.910 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:42 smithi032 bash[161052]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-06T14:22:42.910 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:42 smithi032 bash[161052]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T14:22:42.910 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:42 smithi032 bash[161052]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-06T14:22:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:43 smithi032 ceph-mon[147414]: pgmap v45: 65 pgs: 9 stale+active+clean, 56 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:43 smithi086 ceph-mon[110975]: pgmap v45: 65 pgs: 9 stale+active+clean, 56 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:22:44.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:43 smithi032 ceph-osd[161349]: -- 172.21.15.32:0/3019035950 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56490036f1e0 con 0x56490033ec00 2024-04-06T14:22:44.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:43 smithi032 bash[161311]: 5c8657a4a93cbaeae7e9831cdfc1c0e601d0a115f73e6028dad1d0ecfa36fb2c 2024-04-06T14:22:44.230 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:43 smithi032 systemd[1]: Started Ceph osd.0 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:22:44.613 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:44 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[161326]: 2024-04-06T14:22:44.286+0000 7fce2c711640 -1 Falling back to public interface 2024-04-06T14:22:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:44 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:44 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:45.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:44 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:45.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:44 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:45.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:44 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:45.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:44 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:46.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:45 smithi032 ceph-mon[147414]: pgmap v46: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:46.198 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:45 smithi032 ceph-mon[147414]: Health check failed: Degraded data redundancy: 80/393 objects degraded (20.356%), 20 pgs degraded (PG_DEGRADED) 2024-04-06T14:22:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:45 smithi086 ceph-mon[110975]: pgmap v46: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:45 smithi086 ceph-mon[110975]: Health check failed: Degraded data redundancy: 80/393 objects degraded (20.356%), 20 pgs degraded (PG_DEGRADED) 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 -- 172.21.15.32:0/2955772615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070c10 msgr2=0x7f76b0109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 --2- 172.21.15.32:0/2955772615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070c10 0x7f76b0109180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f76a4004970 tx=0x7f76a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 -- 172.21.15.32:0/2955772615 shutdown_connections 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 --2- 172.21.15.32:0/2955772615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070c10 0x7f76b0109180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 --2- 172.21.15.32:0/2955772615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76b0070300 0x7f76b00706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 -- 172.21.15.32:0/2955772615 >> 172.21.15.32:0/2955772615 conn(0x7f76b00f89a0 msgr2=0x7f76b00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 -- 172.21.15.32:0/2955772615 shutdown_connections 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 -- 172.21.15.32:0/2955772615 wait complete. 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 Processor -- start 2024-04-06T14:22:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.394+0000 7f76b48bc700 1 -- start start 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76b48bc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070300 0x7f76b0115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76b48bc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76b0070c10 0x7f76b01161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76b48bc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b0116800 con 0x7f76b0070300 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76b48bc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b010fbd0 con 0x7f76b0070c10 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76ae59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070300 0x7f76b0115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76ae59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070300 0x7f76b0115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42060/0 (socket says 172.21.15.32:42060) 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76ae59c700 1 -- 172.21.15.32:0/3872114136 learned_addr learned my addr 172.21.15.32:0/3872114136 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76add9b700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76b0070c10 0x7f76b01161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76ae59c700 1 -- 172.21.15.32:0/3872114136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76b0070c10 msgr2=0x7f76b01161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76ae59c700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76b0070c10 0x7f76b01161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76ae59c700 1 -- 172.21.15.32:0/3872114136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a4005040 con 0x7f76b0070300 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76ae59c700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070300 0x7f76b0115c70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76a0002a50 tx=0x7f76a00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f769effd700 1 -- 172.21.15.32:0/3872114136 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76a0007cc0 con 0x7f76b0070300 2024-04-06T14:22:46.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f769effd700 1 -- 172.21.15.32:0/3872114136 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76a000f040 con 0x7f76b0070300 2024-04-06T14:22:46.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b010feb0 con 0x7f76b0070300 2024-04-06T14:22:46.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f769effd700 1 -- 172.21.15.32:0/3872114136 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76a000d9c0 con 0x7f76b0070300 2024-04-06T14:22:46.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.395+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b01103c0 con 0x7f76b0070300 2024-04-06T14:22:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.396+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b0071a40 con 0x7f76b0070300 2024-04-06T14:22:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.397+0000 7f769effd700 1 -- 172.21.15.32:0/3872114136 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f76a000db20 con 0x7f76b0070300 2024-04-06T14:22:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.398+0000 7f769effd700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f769806cd40 0x7f769806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.398+0000 7f769effd700 1 -- 172.21.15.32:0/3872114136 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f76a008c520 con 0x7f76b0070300 2024-04-06T14:22:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.398+0000 7f76add9b700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f769806cd40 0x7f769806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:46.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.398+0000 7f76add9b700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f769806cd40 0x7f769806f1f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f76a4004f50 tx=0x7f76a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:46.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.402+0000 7f769effd700 1 -- 172.21.15.32:0/3872114136 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f76a0059fc0 con 0x7f76b0070300 2024-04-06T14:22:46.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.626+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f76b004bf70 con 0x7f769806cd40 2024-04-06T14:22:46.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.627+0000 7f769effd700 1 -- 172.21.15.32:0/3872114136 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f76a4007e00 con 0x7f769806cd40 2024-04-06T14:22:46.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f769806cd40 msgr2=0x7f769806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f769806cd40 0x7f769806f1f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f76a4004f50 tx=0x7f76a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070300 msgr2=0x7f76b0115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070300 0x7f76b0115c70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76a0002a50 tx=0x7f76a00046b0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 shutdown_connections 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f769806cd40 0x7f769806f1f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b0070300 0x7f76b0115c70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 --2- 172.21.15.32:0/3872114136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76b0070c10 0x7f76b01161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 >> 172.21.15.32:0/3872114136 conn(0x7f76b00f89a0 msgr2=0x7f76b00f9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 shutdown_connections 2024-04-06T14:22:46.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.629+0000 7f76b48bc700 1 -- 172.21.15.32:0/3872114136 wait complete. 2024-04-06T14:22:46.641 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.747+0000 7f390bddd700 1 -- 172.21.15.32:0/2986510231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904102710 msgr2=0x7f39040fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.747+0000 7f390bddd700 1 --2- 172.21.15.32:0/2986510231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904102710 0x7f39040fbd80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f38f4004940 tx=0x7f38f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.747+0000 7f390bddd700 1 -- 172.21.15.32:0/2986510231 shutdown_connections 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.747+0000 7f390bddd700 1 --2- 172.21.15.32:0/2986510231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904102710 0x7f39040fbd80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.747+0000 7f390bddd700 1 --2- 172.21.15.32:0/2986510231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3904101d70 0x7f3904102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.747+0000 7f390bddd700 1 -- 172.21.15.32:0/2986510231 >> 172.21.15.32:0/2986510231 conn(0x7f3904072bc0 msgr2=0x7f3904072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.747+0000 7f390bddd700 1 -- 172.21.15.32:0/2986510231 shutdown_connections 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.747+0000 7f390bddd700 1 -- 172.21.15.32:0/2986510231 wait complete. 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f390bddd700 1 Processor -- start 2024-04-06T14:22:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f390bddd700 1 -- start start 2024-04-06T14:22:46.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f390bddd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904101d70 0x7f39041117e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f390bddd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3904102710 0x7f3904111d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f390bddd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904112370 con 0x7f3904101d70 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f390bddd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390410b760 con 0x7f3904102710 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f3909b79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904101d70 0x7f39041117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f3909b79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904101d70 0x7f39041117e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42088/0 (socket says 172.21.15.32:42088) 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f3909b79700 1 -- 172.21.15.32:0/46943066 learned_addr learned my addr 172.21.15.32:0/46943066 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f3909378700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3904102710 0x7f3904111d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f3909b79700 1 -- 172.21.15.32:0/46943066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3904102710 msgr2=0x7f3904111d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f3909b79700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3904102710 0x7f3904111d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.748+0000 7f3909b79700 1 -- 172.21.15.32:0/46943066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4005040 con 0x7f3904101d70 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.749+0000 7f3909b79700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904101d70 0x7f39041117e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3900002a80 tx=0x7f39000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.749+0000 7f38fa7fc700 1 -- 172.21.15.32:0/46943066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3900007cc0 con 0x7f3904101d70 2024-04-06T14:22:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.749+0000 7f38fa7fc700 1 -- 172.21.15.32:0/46943066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f390000f040 con 0x7f3904101d70 2024-04-06T14:22:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.749+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390410ba40 con 0x7f3904101d70 2024-04-06T14:22:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.749+0000 7f38fa7fc700 1 -- 172.21.15.32:0/46943066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f390000da60 con 0x7f3904101d70 2024-04-06T14:22:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.749+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390410bfd0 con 0x7f3904101d70 2024-04-06T14:22:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.750+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39040ff080 con 0x7f3904101d70 2024-04-06T14:22:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.751+0000 7f38fa7fc700 1 -- 172.21.15.32:0/46943066 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f390000dc60 con 0x7f3904101d70 2024-04-06T14:22:46.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.751+0000 7f38fa7fc700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38f0071150 0x7f38f0073600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:46.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.751+0000 7f38fa7fc700 1 -- 172.21.15.32:0/46943066 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f390008bde0 con 0x7f3904101d70 2024-04-06T14:22:46.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.751+0000 7f3909378700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38f0071150 0x7f38f0073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:46.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.752+0000 7f3909378700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38f0071150 0x7f38f0073600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f390410ca40 tx=0x7f38f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:46.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.756+0000 7f38fa7fc700 1 -- 172.21.15.32:0/46943066 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3900059730 con 0x7f3904101d70 2024-04-06T14:22:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.985+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3904107bf0 con 0x7f38f0071150 2024-04-06T14:22:46.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.986+0000 7f38fa7fc700 1 -- 172.21.15.32:0/46943066 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f3904107bf0 con 0x7f38f0071150 2024-04-06T14:22:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38f0071150 msgr2=0x7f38f0073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38f0071150 0x7f38f0073600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f390410ca40 tx=0x7f38f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904101d70 msgr2=0x7f39041117e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904101d70 0x7f39041117e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3900002a80 tx=0x7f39000046b0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 shutdown_connections 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38f0071150 0x7f38f0073600 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3904101d70 0x7f39041117e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 --2- 172.21.15.32:0/46943066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3904102710 0x7f3904111d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 >> 172.21.15.32:0/46943066 conn(0x7f3904072bc0 msgr2=0x7f39040faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 shutdown_connections 2024-04-06T14:22:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:46.988+0000 7f390bddd700 1 -- 172.21.15.32:0/46943066 wait complete. 2024-04-06T14:22:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.097+0000 7f7793f91700 1 -- 172.21.15.32:0/363203465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 msgr2=0x7f778c1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.097+0000 7f7793f91700 1 --2- 172.21.15.32:0/363203465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 0x7f778c1092f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f777c004940 tx=0x7f777c009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.097+0000 7f7793f91700 1 -- 172.21.15.32:0/363203465 shutdown_connections 2024-04-06T14:22:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.097+0000 7f7793f91700 1 --2- 172.21.15.32:0/363203465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 0x7f778c1092f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.097+0000 7f7793f91700 1 --2- 172.21.15.32:0/363203465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f778c0fbc80 0x7f778c0fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.097+0000 7f7793f91700 1 -- 172.21.15.32:0/363203465 >> 172.21.15.32:0/363203465 conn(0x7f778c072bc0 msgr2=0x7f778c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.098+0000 7f7793f91700 1 -- 172.21.15.32:0/363203465 shutdown_connections 2024-04-06T14:22:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.098+0000 7f7793f91700 1 -- 172.21.15.32:0/363203465 wait complete. 2024-04-06T14:22:47.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.098+0000 7f7793f91700 1 Processor -- start 2024-04-06T14:22:47.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.098+0000 7f7793f91700 1 -- start start 2024-04-06T14:22:47.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f7793f91700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f778c0fbc80 0x7f778c115ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:47.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f7793f91700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 0x7f778c1160e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:47.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f7793f91700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c116730 con 0x7f778c0fc590 2024-04-06T14:22:47.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f7793f91700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c10fb90 con 0x7f778c0fbc80 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f779152c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 0x7f778c1160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f779152c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 0x7f778c1160e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42100/0 (socket says 172.21.15.32:42100) 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f779152c700 1 -- 172.21.15.32:0/3040087080 learned_addr learned my addr 172.21.15.32:0/3040087080 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f7791d2d700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f778c0fbc80 0x7f778c115ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f779152c700 1 -- 172.21.15.32:0/3040087080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f778c0fbc80 msgr2=0x7f778c115ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f779152c700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f778c0fbc80 0x7f778c115ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f779152c700 1 -- 172.21.15.32:0/3040087080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c005040 con 0x7f778c0fc590 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f779152c700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 0x7f778c1160e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f777c009180 tx=0x7f777c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.099+0000 7f77827fc700 1 -- 172.21.15.32:0/3040087080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f777c00e040 con 0x7f778c0fc590 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.100+0000 7f77827fc700 1 -- 172.21.15.32:0/3040087080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f777c004290 con 0x7f778c0fc590 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.100+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c10fe10 con 0x7f778c0fc590 2024-04-06T14:22:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.100+0000 7f77827fc700 1 -- 172.21.15.32:0/3040087080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f777c013e60 con 0x7f778c0fc590 2024-04-06T14:22:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.100+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c1103d0 con 0x7f778c0fc590 2024-04-06T14:22:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.101+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778c065e30 con 0x7f778c0fc590 2024-04-06T14:22:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.101+0000 7f77827fc700 1 -- 172.21.15.32:0/3040087080 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f777c0136b0 con 0x7f778c0fc590 2024-04-06T14:22:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.102+0000 7f77827fc700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f777806cd40 0x7f777806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.102+0000 7f77827fc700 1 -- 172.21.15.32:0/3040087080 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f777c08d3a0 con 0x7f778c0fc590 2024-04-06T14:22:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.102+0000 7f7791d2d700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f777806cd40 0x7f777806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.102+0000 7f7791d2d700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f777806cd40 0x7f777806f1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7788009fd0 tx=0x7f778800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:47.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.106+0000 7f77827fc700 1 -- 172.21.15.32:0/3040087080 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f777c057300 con 0x7f778c0fc590 2024-04-06T14:22:47.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.330+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f778c10c020 con 0x7f777806cd40 2024-04-06T14:22:47.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.340+0000 7f77827fc700 1 -- 172.21.15.32:0/3040087080 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f778c10c020 con 0x7f777806cd40 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (3m) 19s ago 14m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (30s) 19s ago 14m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (25s) 22s ago 13m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (2m) 19s ago 14m 80.2M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (8m) 19s ago 8m 14.1M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (8m) 19s ago 8m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (8m) 22s ago 8m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (8m) 22s ago 8m 16.5M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (5m) 19s ago 15m 494M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (4m) 22s ago 13m 448M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:22:47.341 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (93s) 19s ago 15m 41.6M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (68s) 22s ago 13m 32.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (3m) 19s ago 14m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (3m) 22s ago 13m 17.0M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 starting - - - 1570M 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (12m) 19s ago 12m 193M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 19s ago 11m 164M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (11m) 22s ago 11m 226M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (10m) 22s ago 10m 203M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (10m) 22s ago 10m 187M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:22:47.342 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (3m) 19s ago 13m 58.6M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:22:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f777806cd40 msgr2=0x7f777806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f777806cd40 0x7f777806f1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7788009fd0 tx=0x7f778800b040 comp rx=0 tx=0).stop 2024-04-06T14:22:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 msgr2=0x7f778c1160e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 0x7f778c1160e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f777c009180 tx=0x7f777c009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 shutdown_connections 2024-04-06T14:22:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f777806cd40 0x7f777806f1f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f778c0fbc80 0x7f778c115ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 --2- 172.21.15.32:0/3040087080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f778c0fc590 0x7f778c1160e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 >> 172.21.15.32:0/3040087080 conn(0x7f778c072bc0 msgr2=0x7f778c105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:47.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 shutdown_connections 2024-04-06T14:22:47.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.343+0000 7f7793f91700 1 -- 172.21.15.32:0/3040087080 wait complete. 2024-04-06T14:22:47.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.457+0000 7f43a7284700 1 -- 172.21.15.32:0/1158821015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a01026b0 msgr2=0x7f43a00fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.457+0000 7f43a7284700 1 --2- 172.21.15.32:0/1158821015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a01026b0 0x7f43a00fbd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4390004940 tx=0x7f4390009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:47.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.457+0000 7f43a7284700 1 -- 172.21.15.32:0/1158821015 shutdown_connections 2024-04-06T14:22:47.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.457+0000 7f43a7284700 1 --2- 172.21.15.32:0/1158821015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a01026b0 0x7f43a00fbd20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.457+0000 7f43a7284700 1 --2- 172.21.15.32:0/1158821015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43a0101d10 0x7f43a01020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.457+0000 7f43a7284700 1 -- 172.21.15.32:0/1158821015 >> 172.21.15.32:0/1158821015 conn(0x7f43a00f7880 msgr2=0x7f43a00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:47.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.457+0000 7f43a7284700 1 -- 172.21.15.32:0/1158821015 shutdown_connections 2024-04-06T14:22:47.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.457+0000 7f43a7284700 1 -- 172.21.15.32:0/1158821015 wait complete. 2024-04-06T14:22:47.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a7284700 1 Processor -- start 2024-04-06T14:22:47.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a7284700 1 -- start start 2024-04-06T14:22:47.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a7284700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a0101d10 0x7f43a0112c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:47.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a7284700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43a01026b0 0x7f43a01131b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a7284700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a0113800 con 0x7f43a0101d10 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a7284700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a010cbd0 con 0x7f43a01026b0 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a5020700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a0101d10 0x7f43a0112c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a5020700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a0101d10 0x7f43a0112c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42106/0 (socket says 172.21.15.32:42106) 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a5020700 1 -- 172.21.15.32:0/2916228223 learned_addr learned my addr 172.21.15.32:0/2916228223 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.458+0000 7f43a481f700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43a01026b0 0x7f43a01131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.459+0000 7f43a5020700 1 -- 172.21.15.32:0/2916228223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43a01026b0 msgr2=0x7f43a01131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.459+0000 7f43a5020700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43a01026b0 0x7f43a01131b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.459+0000 7f43a5020700 1 -- 172.21.15.32:0/2916228223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4390005040 con 0x7f43a0101d10 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.459+0000 7f43a5020700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a0101d10 0x7f43a0112c70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f439c00eb40 tx=0x7f439c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.459+0000 7f4395ffb700 1 -- 172.21.15.32:0/2916228223 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f439c007cc0 con 0x7f43a0101d10 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.459+0000 7f4395ffb700 1 -- 172.21.15.32:0/2916228223 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f439c00f040 con 0x7f43a0101d10 2024-04-06T14:22:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.459+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a010ce50 con 0x7f43a0101d10 2024-04-06T14:22:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.459+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a010d410 con 0x7f43a0101d10 2024-04-06T14:22:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.460+0000 7f4395ffb700 1 -- 172.21.15.32:0/2916228223 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f439c00c990 con 0x7f43a0101d10 2024-04-06T14:22:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.461+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a00fd460 con 0x7f43a0101d10 2024-04-06T14:22:47.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.461+0000 7f4395ffb700 1 -- 172.21.15.32:0/2916228223 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f439c00cbb0 con 0x7f43a0101d10 2024-04-06T14:22:47.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.462+0000 7f4395ffb700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f438c06cd40 0x7f438c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:47.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.462+0000 7f4395ffb700 1 -- 172.21.15.32:0/2916228223 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f439c08bdb0 con 0x7f43a0101d10 2024-04-06T14:22:47.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.462+0000 7f43a481f700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f438c06cd40 0x7f438c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.462+0000 7f43a481f700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f438c06cd40 0x7f438c06f1f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f43a010e510 tx=0x7f4390009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.468+0000 7f4395ffb700 1 -- 172.21.15.32:0/2916228223 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f439c05a660 con 0x7f43a0101d10 2024-04-06T14:22:47.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.770+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f43a004bed0 con 0x7f43a0101d10 2024-04-06T14:22:47.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.771+0000 7f4395ffb700 1 -- 172.21.15.32:0/2916228223 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f439c00ce60 con 0x7f43a0101d10 2024-04-06T14:22:47.771 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:22:47.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 5 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 9, 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:22:47.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:22:47.773 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f438c06cd40 msgr2=0x7f438c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f438c06cd40 0x7f438c06f1f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f43a010e510 tx=0x7f4390009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a0101d10 msgr2=0x7f43a0112c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a0101d10 0x7f43a0112c70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f439c00eb40 tx=0x7f439c00b040 comp rx=0 tx=0).stop 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 shutdown_connections 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f438c06cd40 0x7f438c06f1f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a0101d10 0x7f43a0112c70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 --2- 172.21.15.32:0/2916228223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43a01026b0 0x7f43a01131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 >> 172.21.15.32:0/2916228223 conn(0x7f43a00f7880 msgr2=0x7f43a00f88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.773+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 shutdown_connections 2024-04-06T14:22:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.774+0000 7f43a7284700 1 -- 172.21.15.32:0/2916228223 wait complete. 2024-04-06T14:22:47.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.884+0000 7fdf73613700 1 -- 172.21.15.32:0/3640810195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 msgr2=0x7fdf6c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.884+0000 7fdf73613700 1 --2- 172.21.15.32:0/3640810195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 0x7fdf6c1031e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf68004940 tx=0x7fdf68009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:47.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.884+0000 7fdf73613700 1 -- 172.21.15.32:0/3640810195 shutdown_connections 2024-04-06T14:22:47.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.884+0000 7fdf73613700 1 --2- 172.21.15.32:0/3640810195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf6c1037b0 0x7fdf6c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.884+0000 7fdf73613700 1 --2- 172.21.15.32:0/3640810195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 0x7fdf6c1031e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.884+0000 7fdf73613700 1 -- 172.21.15.32:0/3640810195 >> 172.21.15.32:0/3640810195 conn(0x7fdf6c0f89a0 msgr2=0x7fdf6c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:47.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.884+0000 7fdf73613700 1 -- 172.21.15.32:0/3640810195 shutdown_connections 2024-04-06T14:22:47.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.884+0000 7fdf73613700 1 -- 172.21.15.32:0/3640810195 wait complete. 2024-04-06T14:22:47.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.885+0000 7fdf73613700 1 Processor -- start 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.885+0000 7fdf73613700 1 -- start start 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.885+0000 7fdf73613700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 0x7fdf6c113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.885+0000 7fdf73613700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf6c1037b0 0x7fdf6c114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.885+0000 7fdf73613700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c1146a0 con 0x7fdf6c102e10 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.885+0000 7fdf73613700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c10da90 con 0x7fdf6c1037b0 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.885+0000 7fdf713af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 0x7fdf6c113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.885+0000 7fdf713af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 0x7fdf6c113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42124/0 (socket says 172.21.15.32:42124) 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.885+0000 7fdf713af700 1 -- 172.21.15.32:0/235915950 learned_addr learned my addr 172.21.15.32:0/235915950 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf70bae700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf6c1037b0 0x7fdf6c114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf713af700 1 -- 172.21.15.32:0/235915950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf6c1037b0 msgr2=0x7fdf6c114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf713af700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf6c1037b0 0x7fdf6c114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf713af700 1 -- 172.21.15.32:0/235915950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf68005040 con 0x7fdf6c102e10 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf713af700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 0x7fdf6c113b10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800b3b0 tx=0x7fdf68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf5dffb700 1 -- 172.21.15.32:0/235915950 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf6800e040 con 0x7fdf6c102e10 2024-04-06T14:22:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf5dffb700 1 -- 172.21.15.32:0/235915950 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf68013420 con 0x7fdf6c102e10 2024-04-06T14:22:47.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf5dffb700 1 -- 172.21.15.32:0/235915950 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf6801c630 con 0x7fdf6c102e10 2024-04-06T14:22:47.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c10dd10 con 0x7fdf6c102e10 2024-04-06T14:22:47.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.886+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c10e2d0 con 0x7fdf6c102e10 2024-04-06T14:22:47.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.887+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c0fe560 con 0x7fdf6c102e10 2024-04-06T14:22:47.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.888+0000 7fdf5dffb700 1 -- 172.21.15.32:0/235915950 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fdf68013970 con 0x7fdf6c102e10 2024-04-06T14:22:47.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.888+0000 7fdf5dffb700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf5806cd40 0x7fdf5806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:47.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.888+0000 7fdf5dffb700 1 -- 172.21.15.32:0/235915950 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fdf6808d380 con 0x7fdf6c102e10 2024-04-06T14:22:47.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.888+0000 7fdf70bae700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf5806cd40 0x7fdf5806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:47.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.889+0000 7fdf70bae700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf5806cd40 0x7fdf5806f1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009fd0 tx=0x7fdf60009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:47.893+0000 7fdf5dffb700 1 -- 172.21.15.32:0/235915950 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdf6805ad70 con 0x7fdf6c102e10 2024-04-06T14:22:48.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.146+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdf6c05f5f0 con 0x7fdf6c102e10 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.146+0000 7fdf5dffb700 1 -- 172.21.15.32:0/235915950 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1851 (secure 0 0 0) 0x7fdf6805a900 con 0x7fdf6c102e10 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-06T14:22:48.147 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:21:28.458361+0000 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:22:48.148 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:22:48.150 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:22:48.150 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:22:48.150 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:22:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf5806cd40 msgr2=0x7fdf5806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf5806cd40 0x7fdf5806f1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009fd0 tx=0x7fdf60009f20 comp rx=0 tx=0).stop 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 msgr2=0x7fdf6c113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 0x7fdf6c113b10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800b3b0 tx=0x7fdf68009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 shutdown_connections 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf5806cd40 0x7fdf5806f1f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf6c102e10 0x7fdf6c113b10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 --2- 172.21.15.32:0/235915950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf6c1037b0 0x7fdf6c114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 >> 172.21.15.32:0/235915950 conn(0x7fdf6c0f89a0 msgr2=0x7fdf6c0f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 shutdown_connections 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.149+0000 7fdf73613700 1 -- 172.21.15.32:0/235915950 wait complete. 2024-04-06T14:22:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-06T14:22:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:47 smithi032 ceph-mon[147414]: pgmap v47: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:47 smithi032 ceph-mon[147414]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:47 smithi032 ceph-mon[147414]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:47 smithi032 ceph-mon[147414]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:47 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2916228223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:48.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.262+0000 7f7b93b00700 1 -- 172.21.15.32:0/2345058248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c070300 msgr2=0x7f7b8c0706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.262+0000 7f7b93b00700 1 --2- 172.21.15.32:0/2345058248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c070300 0x7f7b8c0706d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7b88004970 tx=0x7f7b88009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:48.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 -- 172.21.15.32:0/2345058248 shutdown_connections 2024-04-06T14:22:48.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 --2- 172.21.15.32:0/2345058248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b8c070c10 0x7f7b8c109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 --2- 172.21.15.32:0/2345058248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c070300 0x7f7b8c0706d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 -- 172.21.15.32:0/2345058248 >> 172.21.15.32:0/2345058248 conn(0x7f7b8c0f89a0 msgr2=0x7f7b8c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 -- 172.21.15.32:0/2345058248 shutdown_connections 2024-04-06T14:22:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 -- 172.21.15.32:0/2345058248 wait complete. 2024-04-06T14:22:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 Processor -- start 2024-04-06T14:22:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 -- start start 2024-04-06T14:22:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b8c070300 0x7f7b8c115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c070c10 0x7f7b8c1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c116840 con 0x7f7b8c070c10 2024-04-06T14:22:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.263+0000 7f7b93b00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c10fc10 con 0x7f7b8c070300 2024-04-06T14:22:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.264+0000 7f7b9109b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c070c10 0x7f7b8c1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.264+0000 7f7b9109b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c070c10 0x7f7b8c1161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42138/0 (socket says 172.21.15.32:42138) 2024-04-06T14:22:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.264+0000 7f7b9109b700 1 -- 172.21.15.32:0/1004347229 learned_addr learned my addr 172.21.15.32:0/1004347229 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.264+0000 7f7b9189c700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b8c070300 0x7f7b8c115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.264+0000 7f7b9189c700 1 -- 172.21.15.32:0/1004347229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c070c10 msgr2=0x7f7b8c1161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.264+0000 7f7b9189c700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c070c10 0x7f7b8c1161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.264+0000 7f7b9189c700 1 -- 172.21.15.32:0/1004347229 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b88005040 con 0x7f7b8c070300 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.264+0000 7f7b9189c700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b8c070300 0x7f7b8c115cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7b88005010 tx=0x7f7b88009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.264+0000 7f7b827fc700 1 -- 172.21.15.32:0/1004347229 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b8800e040 con 0x7f7b8c070300 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.265+0000 7f7b827fc700 1 -- 172.21.15.32:0/1004347229 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b88009830 con 0x7f7b8c070300 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.265+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c10fe90 con 0x7f7b8c070300 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.265+0000 7f7b827fc700 1 -- 172.21.15.32:0/1004347229 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b88013d50 con 0x7f7b8c070300 2024-04-06T14:22:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.265+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c1103d0 con 0x7f7b8c070300 2024-04-06T14:22:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.266+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8c103b90 con 0x7f7b8c070300 2024-04-06T14:22:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.266+0000 7f7b827fc700 1 -- 172.21.15.32:0/1004347229 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f7b88013520 con 0x7f7b8c070300 2024-04-06T14:22:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.267+0000 7f7b827fc700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b7806cd40 0x7f7b7806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.267+0000 7f7b827fc700 1 -- 172.21.15.32:0/1004347229 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7b88062990 con 0x7f7b8c070300 2024-04-06T14:22:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.267+0000 7f7b9109b700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b7806cd40 0x7f7b7806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.268+0000 7f7b9109b700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b7806cd40 0x7f7b7806f1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c008fd0 tx=0x7f7b7c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.272+0000 7f7b827fc700 1 -- 172.21.15.32:0/1004347229 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7b8805a370 con 0x7f7b8c070300 2024-04-06T14:22:48.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:47 smithi086 ceph-mon[110975]: pgmap v47: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:47 smithi086 ceph-mon[110975]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:47 smithi086 ceph-mon[110975]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:47 smithi086 ceph-mon[110975]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:47 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2916228223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:48.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.540+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b8c10c030 con 0x7f7b7806cd40 2024-04-06T14:22:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.542+0000 7f7b827fc700 1 -- 172.21.15.32:0/1004347229 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7b8c10c030 con 0x7f7b7806cd40 2024-04-06T14:22:48.542 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:22:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:22:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:22:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:22:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:22:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:22:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:22:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:22:48.543 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:22:48.543 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "6/21 daemons upgraded", 2024-04-06T14:22:48.543 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T14:22:48.543 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:22:48.543 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.545+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b7806cd40 msgr2=0x7f7b7806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.545+0000 7f7b93b00700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b7806cd40 0x7f7b7806f1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c008fd0 tx=0x7f7b7c00b040 comp rx=0 tx=0).stop 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.545+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b8c070300 msgr2=0x7f7b8c115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.545+0000 7f7b93b00700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b8c070300 0x7f7b8c115cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7b88005010 tx=0x7f7b88009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.545+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 shutdown_connections 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.545+0000 7f7b93b00700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b7806cd40 0x7f7b7806f1f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.545+0000 7f7b93b00700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b8c070300 0x7f7b8c115cb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.545+0000 7f7b93b00700 1 --2- 172.21.15.32:0/1004347229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c070c10 0x7f7b8c1161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.545+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 >> 172.21.15.32:0/1004347229 conn(0x7f7b8c0f89a0 msgr2=0x7f7b8c101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.546+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 shutdown_connections 2024-04-06T14:22:48.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.546+0000 7f7b93b00700 1 -- 172.21.15.32:0/1004347229 wait complete. 2024-04-06T14:22:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.656+0000 7f0443163700 1 -- 172.21.15.32:0/4261572851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 msgr2=0x7f043c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.656+0000 7f0443163700 1 --2- 172.21.15.32:0/4261572851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 0x7f043c106f00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0428004940 tx=0x7f0428009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.656+0000 7f0443163700 1 -- 172.21.15.32:0/4261572851 shutdown_connections 2024-04-06T14:22:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.656+0000 7f0443163700 1 --2- 172.21.15.32:0/4261572851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 0x7f043c106f00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.656+0000 7f0443163700 1 --2- 172.21.15.32:0/4261572851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f043c0fde90 0x7f043c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.656+0000 7f0443163700 1 -- 172.21.15.32:0/4261572851 >> 172.21.15.32:0/4261572851 conn(0x7f043c072bc0 msgr2=0x7f043c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.656+0000 7f0443163700 1 -- 172.21.15.32:0/4261572851 shutdown_connections 2024-04-06T14:22:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.656+0000 7f0443163700 1 -- 172.21.15.32:0/4261572851 wait complete. 2024-04-06T14:22:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.657+0000 7f0443163700 1 Processor -- start 2024-04-06T14:22:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.657+0000 7f0443163700 1 -- start start 2024-04-06T14:22:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f0443163700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f043c0fde90 0x7f043c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f0443163700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 0x7f043c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f0443163700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0fdb40 con 0x7f043c0fe830 2024-04-06T14:22:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f0443163700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0fdcb0 con 0x7f043c0fde90 2024-04-06T14:22:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f043bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 0x7f043c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f043bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 0x7f043c0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42160/0 (socket says 172.21.15.32:42160) 2024-04-06T14:22:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f043bfff700 1 -- 172.21.15.32:0/102598560 learned_addr learned my addr 172.21.15.32:0/102598560 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:22:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f0440eff700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f043c0fde90 0x7f043c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:48.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f043bfff700 1 -- 172.21.15.32:0/102598560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f043c0fde90 msgr2=0x7f043c0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f043bfff700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f043c0fde90 0x7f043c0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.658+0000 7f043bfff700 1 -- 172.21.15.32:0/102598560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0428005040 con 0x7f043c0fe830 2024-04-06T14:22:48.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.659+0000 7f043bfff700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 0x7f043c0fc290 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0428000c00 tx=0x7f0428009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:48.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.659+0000 7f04397fa700 1 -- 172.21.15.32:0/102598560 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0428021070 con 0x7f043c0fe830 2024-04-06T14:22:48.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.659+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c0fc890 con 0x7f043c0fe830 2024-04-06T14:22:48.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.659+0000 7f04397fa700 1 -- 172.21.15.32:0/102598560 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0428013e00 con 0x7f043c0fe830 2024-04-06T14:22:48.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.659+0000 7f04397fa700 1 -- 172.21.15.32:0/102598560 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f042800e070 con 0x7f043c0fe830 2024-04-06T14:22:48.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.659+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c11e710 con 0x7f043c0fe830 2024-04-06T14:22:48.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.660+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043c101980 con 0x7f043c0fe830 2024-04-06T14:22:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.661+0000 7f04397fa700 1 -- 172.21.15.32:0/102598560 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f042801c590 con 0x7f043c0fe830 2024-04-06T14:22:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.661+0000 7f04397fa700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f042c06cd40 0x7f042c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:22:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.661+0000 7f04397fa700 1 -- 172.21.15.32:0/102598560 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f042808d190 con 0x7f043c0fe830 2024-04-06T14:22:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.662+0000 7f0440eff700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f042c06cd40 0x7f042c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:22:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.662+0000 7f0440eff700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f042c06cd40 0x7f042c06f1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0430009b80 tx=0x7f043000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:22:48.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.670+0000 7f04397fa700 1 -- 172.21.15.32:0/102598560 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f042805ab80 con 0x7f043c0fe830 2024-04-06T14:22:48.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.975+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f043c065e30 con 0x7f043c0fe830 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.975+0000 7f04397fa700 1 -- 172.21.15.32:0/102598560 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1573 (secure 0 0 0) 0x7f042805a710 con 0x7f043c0fe830 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; Degraded data redundancy: 80/393 objects degraded (20.356%), 20 pgs degraded 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout: osd.0 (root=default,host=smithi032) is down 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 80/393 objects degraded (20.356%), 20 pgs degraded 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout: pg 1.0 is active+undersized+degraded, acting [3,1] 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.0 is active+undersized+degraded, acting [3,1] 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1 is active+undersized+degraded, acting [2,1] 2024-04-06T14:22:48.976 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.4 is active+undersized+degraded, acting [1,4] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.5 is active+undersized+degraded, acting [3,4] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.8 is active+undersized+degraded, acting [3,5] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.9 is active+undersized+degraded, acting [1,4] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.c is active+undersized+degraded, acting [2,3] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.f is active+undersized+degraded, acting [4,5] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.12 is active+undersized+degraded, acting [3,1] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.13 is active+undersized+degraded, acting [4,2] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.19 is active+undersized+degraded, acting [4,2] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1b is active+undersized+degraded, acting [1,5] 2024-04-06T14:22:48.977 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1d is active+undersized+degraded, acting [3,5] 2024-04-06T14:22:48.978 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1e is active+undersized+degraded, acting [2,5] 2024-04-06T14:22:48.978 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1f is active+undersized+degraded, acting [3,4] 2024-04-06T14:22:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.977+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f042c06cd40 msgr2=0x7f042c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.977+0000 7f0443163700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f042c06cd40 0x7f042c06f1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0430009b80 tx=0x7f043000b040 comp rx=0 tx=0).stop 2024-04-06T14:22:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.977+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 msgr2=0x7f043c0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:22:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.977+0000 7f0443163700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 0x7f043c0fc290 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0428000c00 tx=0x7f0428009e30 comp rx=0 tx=0).stop 2024-04-06T14:22:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.978+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 shutdown_connections 2024-04-06T14:22:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.978+0000 7f0443163700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f042c06cd40 0x7f042c06f1f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.978+0000 7f0443163700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f043c0fde90 0x7f043c0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.978+0000 7f0443163700 1 --2- 172.21.15.32:0/102598560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f043c0fe830 0x7f043c0fc290 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:22:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.978+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 >> 172.21.15.32:0/102598560 conn(0x7f043c072bc0 msgr2=0x7f043c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:22:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.978+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 shutdown_connections 2024-04-06T14:22:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:22:48.978+0000 7f0443163700 1 -- 172.21.15.32:0/102598560 wait complete. 2024-04-06T14:22:48.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:48 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/235915950' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:22:48.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:48 smithi032 ceph-mon[147414]: from='osd.0 [v2:172.21.15.32:6802/969207178,v1:172.21.15.32:6803/969207178]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T14:22:48.980 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:48 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[161326]: 2024-04-06T14:22:48.826+0000 7fce2c711640 -1 osd.0 50 log_to_monitors true 2024-04-06T14:22:48.980 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 14:22:48 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[161326]: 2024-04-06T14:22:48.912+0000 7fce1db6a700 -1 osd.0 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:22:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:48 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/235915950' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:22:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:48 smithi086 ceph-mon[110975]: from='osd.0 [v2:172.21.15.32:6802/969207178,v1:172.21.15.32:6803/969207178]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T14:22:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: pgmap v48: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: from='osd.0 [v2:172.21.15.32:6802/969207178,v1:172.21.15.32:6803/969207178]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T14:22:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: osdmap e53: 6 total, 5 up, 6 in 2024-04-06T14:22:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: from='osd.0 [v2:172.21.15.32:6802/969207178,v1:172.21.15.32:6803/969207178]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:22:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/102598560' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:22:50.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:50.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:50.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:50.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:49 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: pgmap v48: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: from='osd.0 [v2:172.21.15.32:6802/969207178,v1:172.21.15.32:6803/969207178]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: osdmap e53: 6 total, 5 up, 6 in 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: from='osd.0 [v2:172.21.15.32:6802/969207178,v1:172.21.15.32:6803/969207178]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/102598560' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:49 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:50 smithi032 ceph-mon[147414]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:22:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:50 smithi032 ceph-mon[147414]: osd.0 [v2:172.21.15.32:6802/969207178,v1:172.21.15.32:6803/969207178] boot 2024-04-06T14:22:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:50 smithi032 ceph-mon[147414]: osdmap e54: 6 total, 6 up, 6 in 2024-04-06T14:22:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:22:51.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:50 smithi032 ceph-mon[147414]: osdmap e55: 6 total, 6 up, 6 in 2024-04-06T14:22:51.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:50 smithi086 ceph-mon[110975]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:22:51.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:50 smithi086 ceph-mon[110975]: osd.0 [v2:172.21.15.32:6802/969207178,v1:172.21.15.32:6803/969207178] boot 2024-04-06T14:22:51.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:50 smithi086 ceph-mon[110975]: osdmap e54: 6 total, 6 up, 6 in 2024-04-06T14:22:51.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T14:22:51.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:50 smithi086 ceph-mon[110975]: osdmap e55: 6 total, 6 up, 6 in 2024-04-06T14:22:52.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:51 smithi032 ceph-mon[147414]: pgmap v51: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:52.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:51 smithi086 ceph-mon[110975]: pgmap v51: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:53 smithi032 ceph-mon[147414]: pgmap v53: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:22:53.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:53 smithi086 ceph-mon[110975]: pgmap v53: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 80/393 objects degraded (20.356%) 2024-04-06T14:22:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:22:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: mgrmap e40: smithi032.peohxu(active, since 91s), standbys: smithi086.fyyhxm 2024-04-06T14:22:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:22:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:54.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:54.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:54.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:54.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:54.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T14:22:54.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T14:22:54.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:54 smithi086 ceph-mon[110975]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-06T14:22:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: mgrmap e40: smithi032.peohxu(active, since 91s), standbys: smithi086.fyyhxm 2024-04-06T14:22:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:22:54.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:22:54.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:22:54.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:22:54.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:54.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:54.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:22:54.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T14:22:54.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T14:22:54.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:54 smithi032 ceph-mon[147414]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-06T14:22:55.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:55 smithi086 ceph-mon[110975]: pgmap v54: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-04-06T14:22:55.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:55 smithi086 ceph-mon[110975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 80/393 objects degraded (20.356%), 20 pgs degraded) 2024-04-06T14:22:55.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:55 smithi032 ceph-mon[147414]: pgmap v54: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-04-06T14:22:55.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:55 smithi032 ceph-mon[147414]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 80/393 objects degraded (20.356%), 20 pgs degraded) 2024-04-06T14:22:57.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:57 smithi086 ceph-mon[110975]: pgmap v55: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 802 B/s wr, 0 op/s 2024-04-06T14:22:57.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:57 smithi032 ceph-mon[147414]: pgmap v55: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 802 B/s wr, 0 op/s 2024-04-06T14:23:00.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:22:59 smithi032 ceph-mon[147414]: pgmap v56: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 710 B/s wr, 0 op/s 2024-04-06T14:23:00.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:22:59 smithi086 ceph-mon[110975]: pgmap v56: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 710 B/s wr, 0 op/s 2024-04-06T14:23:02.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:01 smithi032 ceph-mon[147414]: pgmap v57: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 614 B/s wr, 0 op/s 2024-04-06T14:23:02.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:01 smithi086 ceph-mon[110975]: pgmap v57: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 614 B/s wr, 0 op/s 2024-04-06T14:23:03.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:03 smithi032 ceph-mon[147414]: pgmap v58: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 527 B/s wr, 0 op/s 2024-04-06T14:23:03.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:03 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:03 smithi086 ceph-mon[110975]: pgmap v58: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 527 B/s wr, 0 op/s 2024-04-06T14:23:04.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:03 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:05 smithi032 ceph-mon[147414]: pgmap v59: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:23:06.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:05 smithi086 ceph-mon[110975]: pgmap v59: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T14:23:08.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:07 smithi032 ceph-mon[147414]: pgmap v60: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:08.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:08.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:23:08.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:07 smithi086 ceph-mon[110975]: pgmap v60: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:23:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:09 smithi032 ceph-mon[147414]: pgmap v61: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:09 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T14:23:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:09 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T14:23:10.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:09 smithi032 ceph-mon[147414]: Upgrade: osd.1 is safe to restart 2024-04-06T14:23:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:09 smithi086 ceph-mon[110975]: pgmap v61: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:09 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T14:23:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:09 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T14:23:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:09 smithi086 ceph-mon[110975]: Upgrade: osd.1 is safe to restart 2024-04-06T14:23:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:11 smithi032 ceph-mon[147414]: pgmap v62: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:11 smithi032 ceph-mon[147414]: Upgrade: Updating osd.1 2024-04-06T14:23:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:11 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:11 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T14:23:11.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:11 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:23:11.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:11 smithi032 ceph-mon[147414]: Deploying daemon osd.1 on smithi032 2024-04-06T14:23:12.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:11 smithi086 ceph-mon[110975]: pgmap v62: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:12.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:11 smithi086 ceph-mon[110975]: Upgrade: Updating osd.1 2024-04-06T14:23:12.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:11 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:12.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:11 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T14:23:12.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:11 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:23:12.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:11 smithi086 ceph-mon[110975]: Deploying daemon osd.1 on smithi032 2024-04-06T14:23:13.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:13 smithi032 ceph-mon[147414]: pgmap v63: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:14.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:13 smithi086 ceph-mon[110975]: pgmap v63: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:14.382 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:14 smithi032 systemd[1]: Stopping Ceph osd.1 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:23:14.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:14 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[72327]: 2024-04-06T14:23:14.381+0000 7f0b7a912700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T14:23:14.731 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:14 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[72327]: 2024-04-06T14:23:14.381+0000 7f0b7a912700 -1 osd.1 55 *** Got signal Terminated *** 2024-04-06T14:23:14.731 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:14 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[72327]: 2024-04-06T14:23:14.381+0000 7f0b7a912700 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T14:23:15.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:14 smithi032 ceph-mon[147414]: osd.1 marked itself down and dead 2024-04-06T14:23:15.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:14 smithi086 ceph-mon[110975]: osd.1 marked itself down and dead 2024-04-06T14:23:16.230 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:15 smithi032 bash[168055]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1 2024-04-06T14:23:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:15 smithi032 ceph-mon[147414]: pgmap v64: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:16.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:15 smithi032 ceph-mon[147414]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:23:16.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:15 smithi032 ceph-mon[147414]: osdmap e56: 6 total, 5 up, 6 in 2024-04-06T14:23:16.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:15 smithi086 ceph-mon[110975]: pgmap v64: 65 pgs: 65 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:16.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:15 smithi086 ceph-mon[110975]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:23:16.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:15 smithi086 ceph-mon[110975]: osdmap e56: 6 total, 5 up, 6 in 2024-04-06T14:23:17.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:16 smithi032 ceph-mon[147414]: osdmap e57: 6 total, 5 up, 6 in 2024-04-06T14:23:17.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:16 smithi086 ceph-mon[110975]: osdmap e57: 6 total, 5 up, 6 in 2024-04-06T14:23:18.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:17 smithi032 ceph-mon[147414]: pgmap v67: 65 pgs: 12 stale+active+clean, 53 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:18.231 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:17 smithi032 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.1.service: Succeeded. 2024-04-06T14:23:18.231 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:17 smithi032 systemd[1]: Stopped Ceph osd.1 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:23:18.231 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:18 smithi032 systemd[1]: Starting Ceph osd.1 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:23:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:17 smithi086 ceph-mon[110975]: pgmap v67: 65 pgs: 12 stale+active+clean, 53 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:19.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.090+0000 7fda36ba8700 1 -- 172.21.15.32:0/2854819043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 msgr2=0x7fda30106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.090+0000 7fda36ba8700 1 --2- 172.21.15.32:0/2854819043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 0x7fda30106e70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fda1c004940 tx=0x7fda1c009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.090+0000 7fda36ba8700 1 -- 172.21.15.32:0/2854819043 shutdown_connections 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.090+0000 7fda36ba8700 1 --2- 172.21.15.32:0/2854819043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 0x7fda30106e70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.090+0000 7fda36ba8700 1 --2- 172.21.15.32:0/2854819043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda300fde00 0x7fda300fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.090+0000 7fda36ba8700 1 -- 172.21.15.32:0/2854819043 >> 172.21.15.32:0/2854819043 conn(0x7fda300f7860 msgr2=0x7fda300f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.090+0000 7fda36ba8700 1 -- 172.21.15.32:0/2854819043 shutdown_connections 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.090+0000 7fda36ba8700 1 -- 172.21.15.32:0/2854819043 wait complete. 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.091+0000 7fda36ba8700 1 Processor -- start 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.091+0000 7fda36ba8700 1 -- start start 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.091+0000 7fda36ba8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda300fde00 0x7fda300fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda36ba8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 0x7fda300fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda36ba8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300fc790 con 0x7fda300fe7a0 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda36ba8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300fc8d0 con 0x7fda300fde00 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda34944700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda300fde00 0x7fda300fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda2ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 0x7fda300fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda34944700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda300fde00 0x7fda300fbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:49136/0 (socket says 172.21.15.32:49136) 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda2ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 0x7fda300fc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32790/0 (socket says 172.21.15.32:32790) 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda34944700 1 -- 172.21.15.32:0/3992717940 learned_addr learned my addr 172.21.15.32:0/3992717940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda2ffff700 1 -- 172.21.15.32:0/3992717940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda300fde00 msgr2=0x7fda300fbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda2ffff700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda300fde00 0x7fda300fbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda2ffff700 1 -- 172.21.15.32:0/3992717940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c005040 con 0x7fda300fe7a0 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda2ffff700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 0x7fda300fc200 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00b3b0 tx=0x7fda1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.092+0000 7fda2d7fa700 1 -- 172.21.15.32:0/3992717940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda1c00e040 con 0x7fda300fe7a0 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.093+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda300fcb20 con 0x7fda300fe7a0 2024-04-06T14:23:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.093+0000 7fda2d7fa700 1 -- 172.21.15.32:0/3992717940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda1c01c430 con 0x7fda300fe7a0 2024-04-06T14:23:19.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.093+0000 7fda2d7fa700 1 -- 172.21.15.32:0/3992717940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda1c013660 con 0x7fda300fe7a0 2024-04-06T14:23:19.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.093+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3011a110 con 0x7fda300fe7a0 2024-04-06T14:23:19.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.094+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda300666f0 con 0x7fda300fe7a0 2024-04-06T14:23:19.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.094+0000 7fda2d7fa700 1 -- 172.21.15.32:0/3992717940 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fda1c0138d0 con 0x7fda300fe7a0 2024-04-06T14:23:19.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.094+0000 7fda2d7fa700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fda2006cc20 0x7fda2006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:19.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.095+0000 7fda2d7fa700 1 -- 172.21.15.32:0/3992717940 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fda1c08caf0 con 0x7fda300fe7a0 2024-04-06T14:23:19.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.095+0000 7fda34944700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fda2006cc20 0x7fda2006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:19.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.095+0000 7fda34944700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fda2006cc20 0x7fda2006f0d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda240102e0 tx=0x7fda2400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:19.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.099+0000 7fda2d7fa700 1 -- 172.21.15.32:0/3992717940 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fda1c056c40 con 0x7fda300fe7a0 2024-04-06T14:23:19.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.324+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda3004bed0 con 0x7fda2006cc20 2024-04-06T14:23:19.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.325+0000 7fda2d7fa700 1 -- 172.21.15.32:0/3992717940 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fda3004bed0 con 0x7fda2006cc20 2024-04-06T14:23:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.327+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fda2006cc20 msgr2=0x7fda2006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.327+0000 7fda36ba8700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fda2006cc20 0x7fda2006f0d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda240102e0 tx=0x7fda2400b040 comp rx=0 tx=0).stop 2024-04-06T14:23:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.328+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 msgr2=0x7fda300fc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.328+0000 7fda36ba8700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 0x7fda300fc200 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00b3b0 tx=0x7fda1c009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.328+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 shutdown_connections 2024-04-06T14:23:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.328+0000 7fda36ba8700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fda2006cc20 0x7fda2006f0d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.328+0000 7fda36ba8700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda300fde00 0x7fda300fbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.328+0000 7fda36ba8700 1 --2- 172.21.15.32:0/3992717940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda300fe7a0 0x7fda300fc200 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.328+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 >> 172.21.15.32:0/3992717940 conn(0x7fda300f7860 msgr2=0x7fda300f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.328+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 shutdown_connections 2024-04-06T14:23:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.328+0000 7fda36ba8700 1 -- 172.21.15.32:0/3992717940 wait complete. 2024-04-06T14:23:19.338 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:23:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.434+0000 7f63acfcc700 1 -- 172.21.15.32:0/1465157525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81037b0 msgr2=0x7f63a80fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.434+0000 7f63acfcc700 1 --2- 172.21.15.32:0/1465157525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81037b0 0x7f63a80fce20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f639c004940 tx=0x7f639c009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 -- 172.21.15.32:0/1465157525 shutdown_connections 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 --2- 172.21.15.32:0/1465157525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81037b0 0x7f63a80fce20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 --2- 172.21.15.32:0/1465157525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a8102e10 0x7f63a81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 -- 172.21.15.32:0/1465157525 >> 172.21.15.32:0/1465157525 conn(0x7f63a80f89a0 msgr2=0x7f63a80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 -- 172.21.15.32:0/1465157525 shutdown_connections 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 -- 172.21.15.32:0/1465157525 wait complete. 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 Processor -- start 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 -- start start 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a8102e10 0x7f63a8113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a81037b0 0x7f63a8114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a8114660 con 0x7f63a8102e10 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63acfcc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a810da50 con 0x7f63a81037b0 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63a659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a8102e10 0x7f63a8113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63a659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a8102e10 0x7f63a8113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32812/0 (socket says 172.21.15.32:32812) 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.435+0000 7f63a659c700 1 -- 172.21.15.32:0/2026482727 learned_addr learned my addr 172.21.15.32:0/2026482727 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f63a5d9b700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a81037b0 0x7f63a8114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f63a659c700 1 -- 172.21.15.32:0/2026482727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a81037b0 msgr2=0x7f63a8114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f63a659c700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a81037b0 0x7f63a8114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f63a659c700 1 -- 172.21.15.32:0/2026482727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c005040 con 0x7f63a8102e10 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f63a5d9b700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a81037b0 0x7f63a8114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f63a659c700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a8102e10 0x7f63a8113ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6398002a50 tx=0x7f63980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f6396ffd700 1 -- 172.21.15.32:0/2026482727 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6398007cc0 con 0x7f63a8102e10 2024-04-06T14:23:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f6396ffd700 1 -- 172.21.15.32:0/2026482727 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f639800f040 con 0x7f63a8102e10 2024-04-06T14:23:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a810dd30 con 0x7f63a8102e10 2024-04-06T14:23:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f6396ffd700 1 -- 172.21.15.32:0/2026482727 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6398009540 con 0x7f63a8102e10 2024-04-06T14:23:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.436+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a810e2f0 con 0x7f63a8102e10 2024-04-06T14:23:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.437+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a80fe560 con 0x7f63a8102e10 2024-04-06T14:23:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.438+0000 7f6396ffd700 1 -- 172.21.15.32:0/2026482727 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f63980096a0 con 0x7f63a8102e10 2024-04-06T14:23:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.438+0000 7f6396ffd700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f639006cc20 0x7f639006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.438+0000 7f6396ffd700 1 -- 172.21.15.32:0/2026482727 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f639808bcd0 con 0x7f63a8102e10 2024-04-06T14:23:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.438+0000 7f63a5d9b700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f639006cc20 0x7f639006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.438+0000 7f63a5d9b700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f639006cc20 0x7f639006f0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63a810ec10 tx=0x7f639c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:19.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.443+0000 7f6396ffd700 1 -- 172.21.15.32:0/2026482727 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6398059840 con 0x7f63a8102e10 2024-04-06T14:23:19.480 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:19 smithi032 bash[168330]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T14:23:19.480 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:19 smithi032 bash[168330]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T14:23:19.480 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:19 smithi032 bash[168330]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T14:23:19.480 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:19 smithi032 bash[168330]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T14:23:19.480 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:19 smithi032 bash[168330]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-06T14:23:19.480 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:19 smithi032 bash[168330]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T14:23:19.480 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:19 smithi032 bash[168330]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-06T14:23:19.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.667+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f63a8109db0 con 0x7f639006cc20 2024-04-06T14:23:19.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.668+0000 7f6396ffd700 1 -- 172.21.15.32:0/2026482727 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f63a8109db0 con 0x7f639006cc20 2024-04-06T14:23:19.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f639006cc20 msgr2=0x7f639006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f639006cc20 0x7f639006f0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63a810ec10 tx=0x7f639c009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:19.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a8102e10 msgr2=0x7f63a8113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a8102e10 0x7f63a8113ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6398002a50 tx=0x7f63980046b0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 shutdown_connections 2024-04-06T14:23:19.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f639006cc20 0x7f639006f0d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a8102e10 0x7f63a8113ad0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 --2- 172.21.15.32:0/2026482727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a81037b0 0x7f63a8114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 >> 172.21.15.32:0/2026482727 conn(0x7f63a80f89a0 msgr2=0x7f63a80f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:19.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 shutdown_connections 2024-04-06T14:23:19.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.670+0000 7f63acfcc700 1 -- 172.21.15.32:0/2026482727 wait complete. 2024-04-06T14:23:19.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.774+0000 7ff03d6ec700 1 -- 172.21.15.32:0/931072944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff038101d20 msgr2=0x7ff0381020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.774+0000 7ff03d6ec700 1 --2- 172.21.15.32:0/931072944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff038101d20 0x7ff0381020f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff028004940 tx=0x7ff028009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:19.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.774+0000 7ff03d6ec700 1 -- 172.21.15.32:0/931072944 shutdown_connections 2024-04-06T14:23:19.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.774+0000 7ff03d6ec700 1 --2- 172.21.15.32:0/931072944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0381026c0 0x7ff0380fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.774+0000 7ff03d6ec700 1 --2- 172.21.15.32:0/931072944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff038101d20 0x7ff0381020f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.774+0000 7ff03d6ec700 1 -- 172.21.15.32:0/931072944 >> 172.21.15.32:0/931072944 conn(0x7ff0380f7870 msgr2=0x7ff0380f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:19.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.774+0000 7ff03d6ec700 1 -- 172.21.15.32:0/931072944 shutdown_connections 2024-04-06T14:23:19.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.774+0000 7ff03d6ec700 1 -- 172.21.15.32:0/931072944 wait complete. 2024-04-06T14:23:19.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.775+0000 7ff03d6ec700 1 Processor -- start 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.775+0000 7ff03d6ec700 1 -- start start 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.775+0000 7ff03d6ec700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff038101d20 0x7ff0381127b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.775+0000 7ff03d6ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0381026c0 0x7ff038112cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.775+0000 7ff03d6ec700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0381133d0 con 0x7ff0381026c0 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.775+0000 7ff03d6ec700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03810c940 con 0x7ff038101d20 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.775+0000 7ff0367fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0381026c0 0x7ff038112cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.775+0000 7ff0367fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0381026c0 0x7ff038112cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32818/0 (socket says 172.21.15.32:32818) 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.775+0000 7ff0367fc700 1 -- 172.21.15.32:0/1178884509 learned_addr learned my addr 172.21.15.32:0/1178884509 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff036ffd700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff038101d20 0x7ff0381127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff0367fc700 1 -- 172.21.15.32:0/1178884509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff038101d20 msgr2=0x7ff0381127b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff0367fc700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff038101d20 0x7ff0381127b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff0367fc700 1 -- 172.21.15.32:0/1178884509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff028005040 con 0x7ff0381026c0 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff0367fc700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0381026c0 0x7ff038112cf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff02000daf0 tx=0x7ff02000b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff02f7fe700 1 -- 172.21.15.32:0/1178884509 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff02000ed70 con 0x7ff0381026c0 2024-04-06T14:23:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff02f7fe700 1 -- 172.21.15.32:0/1178884509 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff020017920 con 0x7ff0381026c0 2024-04-06T14:23:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03810cbc0 con 0x7ff0381026c0 2024-04-06T14:23:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff02f7fe700 1 -- 172.21.15.32:0/1178884509 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff020016970 con 0x7ff0381026c0 2024-04-06T14:23:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.776+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03810d180 con 0x7ff0381026c0 2024-04-06T14:23:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.777+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0380fd470 con 0x7ff0381026c0 2024-04-06T14:23:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.778+0000 7ff02f7fe700 1 -- 172.21.15.32:0/1178884509 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff020014070 con 0x7ff0381026c0 2024-04-06T14:23:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.778+0000 7ff02f7fe700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff02406cc20 0x7ff02406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.778+0000 7ff02f7fe700 1 -- 172.21.15.32:0/1178884509 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff02008e090 con 0x7ff0381026c0 2024-04-06T14:23:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.778+0000 7ff036ffd700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff02406cc20 0x7ff02406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.779+0000 7ff036ffd700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff02406cc20 0x7ff02406f0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff028004f90 tx=0x7ff028009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:19.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:19.783+0000 7ff02f7fe700 1 -- 172.21.15.32:0/1178884509 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff02005bca0 con 0x7ff0381026c0 2024-04-06T14:23:20.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.006+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff038108d50 con 0x7ff02406cc20 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.016+0000 7ff02f7fe700 1 -- 172.21.15.32:0/1178884509 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7ff038108d50 con 0x7ff02406cc20 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (3m) 30s ago 14m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (63s) 30s ago 15m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (58s) 54s ago 14m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (3m) 30s ago 14m 81.0M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (9m) 30s ago 9m 14.2M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (9m) 30s ago 9m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (9m) 54s ago 9m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (9m) 54s ago 9m 16.5M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:23:20.017 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (5m) 30s ago 16m 501M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (4m) 54s ago 14m 448M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (2m) 30s ago 16m 42.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (101s) 54s ago 13m 32.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (4m) 30s ago 15m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (4m) 54s ago 14m 17.0M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (36s) 30s ago 13m 11.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (12m) 30s ago 12m 193M 1570M 17.2.7-819-g6699495c ec300eaa768b 48da31acad03 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (12m) 30s ago 12m 163M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (11m) 54s ago 11m 226M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (11m) 54s ago 11m 203M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (10m) 54s ago 10m 187M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:23:20.018 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (4m) 30s ago 14m 58.6M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:23:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.018+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff02406cc20 msgr2=0x7ff02406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.018+0000 7ff03d6ec700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff02406cc20 0x7ff02406f0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff028004f90 tx=0x7ff028009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.018+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0381026c0 msgr2=0x7ff038112cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.018+0000 7ff03d6ec700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0381026c0 0x7ff038112cf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff02000daf0 tx=0x7ff02000b4c0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.019+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 shutdown_connections 2024-04-06T14:23:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.019+0000 7ff03d6ec700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff02406cc20 0x7ff02406f0d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.019+0000 7ff03d6ec700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff038101d20 0x7ff0381127b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.019+0000 7ff03d6ec700 1 --2- 172.21.15.32:0/1178884509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0381026c0 0x7ff038112cf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.019+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 >> 172.21.15.32:0/1178884509 conn(0x7ff0380f7870 msgr2=0x7ff0380f88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:20.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.019+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 shutdown_connections 2024-04-06T14:23:20.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.019+0000 7ff03d6ec700 1 -- 172.21.15.32:0/1178884509 wait complete. 2024-04-06T14:23:20.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.122+0000 7fc1ba835700 1 -- 172.21.15.32:0/466378189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 msgr2=0x7fc1b40fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.122+0000 7fc1ba835700 1 --2- 172.21.15.32:0/466378189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 0x7fc1b40fe2a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4004940 tx=0x7fc1a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:20.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.123+0000 7fc1ba835700 1 -- 172.21.15.32:0/466378189 shutdown_connections 2024-04-06T14:23:20.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.123+0000 7fc1ba835700 1 --2- 172.21.15.32:0/466378189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b40fe870 0x7fc1b4106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.123+0000 7fc1ba835700 1 --2- 172.21.15.32:0/466378189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 0x7fc1b40fe2a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.123+0000 7fc1ba835700 1 -- 172.21.15.32:0/466378189 >> 172.21.15.32:0/466378189 conn(0x7fc1b4072bc0 msgr2=0x7fc1b4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:20.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.123+0000 7fc1ba835700 1 -- 172.21.15.32:0/466378189 shutdown_connections 2024-04-06T14:23:20.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.123+0000 7fc1ba835700 1 -- 172.21.15.32:0/466378189 wait complete. 2024-04-06T14:23:20.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.123+0000 7fc1ba835700 1 Processor -- start 2024-04-06T14:23:20.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.123+0000 7fc1ba835700 1 -- start start 2024-04-06T14:23:20.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1ba835700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 0x7fc1b4113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:20.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1ba835700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b40fe870 0x7fc1b4113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:20.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1ba835700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4114520 con 0x7fc1b40fded0 2024-04-06T14:23:20.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1ba835700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b410d960 con 0x7fc1b40fe870 2024-04-06T14:23:20.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 0x7fc1b4113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:20.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 0x7fc1b4113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32844/0 (socket says 172.21.15.32:32844) 2024-04-06T14:23:20.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b3fff700 1 -- 172.21.15.32:0/543035180 learned_addr learned my addr 172.21.15.32:0/543035180 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b37fe700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b40fe870 0x7fc1b4113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b3fff700 1 -- 172.21.15.32:0/543035180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b40fe870 msgr2=0x7fc1b4113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b3fff700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b40fe870 0x7fc1b4113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b3fff700 1 -- 172.21.15.32:0/543035180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a4005040 con 0x7fc1b40fded0 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b3fff700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 0x7fc1b4113990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4009380 tx=0x7fc1a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b0ff9700 1 -- 172.21.15.32:0/543035180 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a400e040 con 0x7fc1b40fded0 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b0ff9700 1 -- 172.21.15.32:0/543035180 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1a4004290 con 0x7fc1b40fded0 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b410dc40 con 0x7fc1b40fded0 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1b0ff9700 1 -- 172.21.15.32:0/543035180 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a401c3f0 con 0x7fc1b40fded0 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.124+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b410e180 con 0x7fc1b40fded0 2024-04-06T14:23:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.126+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b4065df0 con 0x7fc1b40fded0 2024-04-06T14:23:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.126+0000 7fc1b0ff9700 1 -- 172.21.15.32:0/543035180 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc1a400bb50 con 0x7fc1b40fded0 2024-04-06T14:23:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.126+0000 7fc1b0ff9700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1a006cc20 0x7fc1a006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.126+0000 7fc1b0ff9700 1 -- 172.21.15.32:0/543035180 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc1a408d2d0 con 0x7fc1b40fded0 2024-04-06T14:23:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.126+0000 7fc1b37fe700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1a006cc20 0x7fc1a006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.127+0000 7fc1b37fe700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1a006cc20 0x7fc1a006f0d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009b80 tx=0x7fc19c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:20.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.131+0000 7fc1b0ff9700 1 -- 172.21.15.32:0/543035180 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc1a405ae40 con 0x7fc1b40fded0 2024-04-06T14:23:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:19 smithi032 ceph-mon[147414]: pgmap v68: 65 pgs: 14 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 33 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 51/393 objects degraded (12.977%) 2024-04-06T14:23:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:19 smithi032 ceph-mon[147414]: Health check failed: Degraded data redundancy: 51/393 objects degraded (12.977%), 14 pgs degraded (PG_DEGRADED) 2024-04-06T14:23:20.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:19 smithi032 ceph-mon[147414]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:20.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:19 smithi086 ceph-mon[110975]: pgmap v68: 65 pgs: 14 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 33 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 51/393 objects degraded (12.977%) 2024-04-06T14:23:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:19 smithi086 ceph-mon[110975]: Health check failed: Degraded data redundancy: 51/393 objects degraded (12.977%), 14 pgs degraded (PG_DEGRADED) 2024-04-06T14:23:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:19 smithi086 ceph-mon[110975]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:20.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.422+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc1b4109d50 con 0x7fc1b40fded0 2024-04-06T14:23:20.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.423+0000 7fc1b0ff9700 1 -- 172.21.15.32:0/543035180 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fc1a405a9d0 con 0x7fc1b40fded0 2024-04-06T14:23:20.423 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:23:20.423 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4, 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 1 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 8, 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 5 2024-04-06T14:23:20.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:23:20.425 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:23:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1a006cc20 msgr2=0x7fc1a006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1a006cc20 0x7fc1a006f0d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009b80 tx=0x7fc19c00b040 comp rx=0 tx=0).stop 2024-04-06T14:23:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 msgr2=0x7fc1b4113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 0x7fc1b4113990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4009380 tx=0x7fc1a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 shutdown_connections 2024-04-06T14:23:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1a006cc20 0x7fc1a006f0d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b40fded0 0x7fc1b4113990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 --2- 172.21.15.32:0/543035180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b40fe870 0x7fc1b4113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 >> 172.21.15.32:0/543035180 conn(0x7fc1b4072bc0 msgr2=0x7fc1b4103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 shutdown_connections 2024-04-06T14:23:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.425+0000 7fc1ba835700 1 -- 172.21.15.32:0/543035180 wait complete. 2024-04-06T14:23:20.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 -- 172.21.15.32:0/120478138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 msgr2=0x7fe2d8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 --2- 172.21.15.32:0/120478138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 0x7fe2d8106f00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc004940 tx=0x7fe2cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 -- 172.21.15.32:0/120478138 shutdown_connections 2024-04-06T14:23:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 --2- 172.21.15.32:0/120478138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 0x7fe2d8106f00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 --2- 172.21.15.32:0/120478138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2d80fde90 0x7fe2d80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 -- 172.21.15.32:0/120478138 >> 172.21.15.32:0/120478138 conn(0x7fe2d8072bc0 msgr2=0x7fe2d8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 -- 172.21.15.32:0/120478138 shutdown_connections 2024-04-06T14:23:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 -- 172.21.15.32:0/120478138 wait complete. 2024-04-06T14:23:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 Processor -- start 2024-04-06T14:23:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.540+0000 7fe2de283700 1 -- start start 2024-04-06T14:23:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2de283700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2d80fde90 0x7fe2d8113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2de283700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 0x7fe2d8114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2de283700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d8114670 con 0x7fe2d80fe830 2024-04-06T14:23:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2de283700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d810da60 con 0x7fe2d80fde90 2024-04-06T14:23:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 0x7fe2d8114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 0x7fe2d8114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32884/0 (socket says 172.21.15.32:32884) 2024-04-06T14:23:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d77fe700 1 -- 172.21.15.32:0/1652997701 learned_addr learned my addr 172.21.15.32:0/1652997701 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d7fff700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2d80fde90 0x7fe2d8113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d77fe700 1 -- 172.21.15.32:0/1652997701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2d80fde90 msgr2=0x7fe2d8113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d77fe700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2d80fde90 0x7fe2d8113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d77fe700 1 -- 172.21.15.32:0/1652997701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc005040 con 0x7fe2d80fe830 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d7fff700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2d80fde90 0x7fe2d8113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d77fe700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 0x7fe2d8114020 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00b3b0 tx=0x7fe2cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d4ff9700 1 -- 172.21.15.32:0/1652997701 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2cc00e040 con 0x7fe2d80fe830 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d810dce0 con 0x7fe2d80fe830 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d4ff9700 1 -- 172.21.15.32:0/1652997701 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2cc013420 con 0x7fe2d80fe830 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.541+0000 7fe2d4ff9700 1 -- 172.21.15.32:0/1652997701 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2cc01c5d0 con 0x7fe2d80fe830 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.542+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d810e220 con 0x7fe2d80fe830 2024-04-06T14:23:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.543+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d8065e30 con 0x7fe2d80fe830 2024-04-06T14:23:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.544+0000 7fe2d4ff9700 1 -- 172.21.15.32:0/1652997701 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe2cc013590 con 0x7fe2d80fe830 2024-04-06T14:23:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.544+0000 7fe2d4ff9700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe2c006cc20 0x7fe2c006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.544+0000 7fe2d4ff9700 1 -- 172.21.15.32:0/1652997701 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fe2cc08d240 con 0x7fe2d80fe830 2024-04-06T14:23:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.544+0000 7fe2d7fff700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe2c006cc20 0x7fe2c006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.545+0000 7fe2d7fff700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe2c006cc20 0x7fe2c006f0d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8007fd0 tx=0x7fe2c8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:20.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.549+0000 7fe2d4ff9700 1 -- 172.21.15.32:0/1652997701 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe2cc05ad30 con 0x7fe2d80fe830 2024-04-06T14:23:20.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:20 smithi032 bash[168662]: afc7d0d625a8ce361fc872e49d022bc6e9bfb749d1ccffe5d41ae2abfca01ac7 2024-04-06T14:23:20.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:20 smithi032 systemd[1]: Started Ceph osd.1 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:23:20.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.810+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe2d810f130 con 0x7fe2d80fe830 2024-04-06T14:23:20.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.811+0000 7fe2d4ff9700 1 -- 172.21.15.32:0/1652997701 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1851 (secure 0 0 0) 0x7fe2cc05a8c0 con 0x7fe2d80fe830 2024-04-06T14:23:20.811 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-06T14:23:20.811 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:23:20.811 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:21:28.458361+0000 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:23:20.812 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:23:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:23:20.814 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:23:20.814 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:23:20.814 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:23:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:23:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:23:20.814 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:23:20.814 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:23:20.814 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:23:20.814 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe2c006cc20 msgr2=0x7fe2c006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe2c006cc20 0x7fe2c006f0d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8007fd0 tx=0x7fe2c8009040 comp rx=0 tx=0).stop 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 msgr2=0x7fe2d8114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 0x7fe2d8114020 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00b3b0 tx=0x7fe2cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 shutdown_connections 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe2c006cc20 0x7fe2c006f0d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2d80fde90 0x7fe2d8113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 --2- 172.21.15.32:0/1652997701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2d80fe830 0x7fe2d8114020 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 >> 172.21.15.32:0/1652997701 conn(0x7fe2d8072bc0 msgr2=0x7fe2d8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:20.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 shutdown_connections 2024-04-06T14:23:20.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.813+0000 7fe2de283700 1 -- 172.21.15.32:0/1652997701 wait complete. 2024-04-06T14:23:20.816 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-06T14:23:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.923+0000 7f43259d1700 1 -- 172.21.15.32:0/4245264199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 msgr2=0x7f43200fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.923+0000 7f43259d1700 1 --2- 172.21.15.32:0/4245264199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 0x7f43200fce20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4308004940 tx=0x7f4308009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.923+0000 7f43259d1700 1 -- 172.21.15.32:0/4245264199 shutdown_connections 2024-04-06T14:23:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.923+0000 7f43259d1700 1 --2- 172.21.15.32:0/4245264199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 0x7f43200fce20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.923+0000 7f43259d1700 1 --2- 172.21.15.32:0/4245264199 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4320102e10 0x7f43201031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.923+0000 7f43259d1700 1 -- 172.21.15.32:0/4245264199 >> 172.21.15.32:0/4245264199 conn(0x7f43200f89a0 msgr2=0x7f43200fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.923+0000 7f43259d1700 1 -- 172.21.15.32:0/4245264199 shutdown_connections 2024-04-06T14:23:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.923+0000 7f43259d1700 1 -- 172.21.15.32:0/4245264199 wait complete. 2024-04-06T14:23:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f43259d1700 1 Processor -- start 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f43259d1700 1 -- start start 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f43259d1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4320102e10 0x7f4320113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f43259d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 0x7f4320113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f43259d1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43201145d0 con 0x7f43201037b0 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f43259d1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432010d9c0 con 0x7f4320102e10 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f431e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 0x7f4320113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f431e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 0x7f4320113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32902/0 (socket says 172.21.15.32:32902) 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f431e7fc700 1 -- 172.21.15.32:0/1286766626 learned_addr learned my addr 172.21.15.32:0/1286766626 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f431effd700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4320102e10 0x7f4320113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f431e7fc700 1 -- 172.21.15.32:0/1286766626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4320102e10 msgr2=0x7f4320113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f431e7fc700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4320102e10 0x7f4320113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f431e7fc700 1 -- 172.21.15.32:0/1286766626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4308005040 con 0x7f43201037b0 2024-04-06T14:23:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.924+0000 7f431e7fc700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 0x7f4320113f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4308008d50 tx=0x7f4308009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.925+0000 7f43177fe700 1 -- 172.21.15.32:0/1286766626 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f430800e030 con 0x7f43201037b0 2024-04-06T14:23:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.925+0000 7f43177fe700 1 -- 172.21.15.32:0/1286766626 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4308013de0 con 0x7f43201037b0 2024-04-06T14:23:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.925+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432010dc40 con 0x7f43201037b0 2024-04-06T14:23:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.925+0000 7f43177fe700 1 -- 172.21.15.32:0/1286766626 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f430801cd90 con 0x7f43201037b0 2024-04-06T14:23:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.925+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432010e200 con 0x7f43201037b0 2024-04-06T14:23:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.926+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43200fe560 con 0x7f43201037b0 2024-04-06T14:23:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.926+0000 7f43177fe700 1 -- 172.21.15.32:0/1286766626 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4308013930 con 0x7f43201037b0 2024-04-06T14:23:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.927+0000 7f43177fe700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f430c06cc30 0x7f430c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.927+0000 7f43177fe700 1 -- 172.21.15.32:0/1286766626 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f430808d790 con 0x7f43201037b0 2024-04-06T14:23:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.927+0000 7f431effd700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f430c06cc30 0x7f430c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.927+0000 7f431effd700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f430c06cc30 0x7f430c06f0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4318009f10 tx=0x7f431800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:20.931+0000 7f43177fe700 1 -- 172.21.15.32:0/1286766626 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f430805b3a0 con 0x7f43201037b0 2024-04-06T14:23:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.153+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4320109db0 con 0x7f430c06cc30 2024-04-06T14:23:21.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.155+0000 7f43177fe700 1 -- 172.21.15.32:0/1286766626 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4320109db0 con 0x7f430c06cc30 2024-04-06T14:23:21.155 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:23:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:23:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "7/21 daemons upgraded", 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:23:21.156 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:23:21.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.158+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f430c06cc30 msgr2=0x7f430c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:21.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f430c06cc30 0x7f430c06f0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4318009f10 tx=0x7f431800b040 comp rx=0 tx=0).stop 2024-04-06T14:23:21.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 msgr2=0x7f4320113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:21.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 0x7f4320113f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4308008d50 tx=0x7f4308009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:21.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 shutdown_connections 2024-04-06T14:23:21.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f430c06cc30 0x7f430c06f0e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:21.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4320102e10 0x7f4320113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:21.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 --2- 172.21.15.32:0/1286766626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43201037b0 0x7f4320113f80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:21.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 >> 172.21.15.32:0/1286766626 conn(0x7f43200f89a0 msgr2=0x7f43200f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:21.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 shutdown_connections 2024-04-06T14:23:21.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.159+0000 7f43259d1700 1 -- 172.21.15.32:0/1286766626 wait complete. 2024-04-06T14:23:21.184 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:20 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[168700]: 2024-04-06T14:23:20.871+0000 7fa44278f640 -1 Falling back to public interface 2024-04-06T14:23:21.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:20 smithi032 ceph-mon[147414]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:21.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:20 smithi032 ceph-mon[147414]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:21.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:20 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/543035180' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:21.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:20 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:21.184 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:20 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:21.185 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:20 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:23:21.185 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:20 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1652997701' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:23:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.270+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2899986165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 msgr2=0x7f319c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.270+0000 7f31a3fbd700 1 --2- 172.21.15.32:0/2899986165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 0x7f319c106f00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f318c004940 tx=0x7f318c009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.270+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2899986165 shutdown_connections 2024-04-06T14:23:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.270+0000 7f31a3fbd700 1 --2- 172.21.15.32:0/2899986165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 0x7f319c106f00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.270+0000 7f31a3fbd700 1 --2- 172.21.15.32:0/2899986165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f319c0fde90 0x7f319c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.270+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2899986165 >> 172.21.15.32:0/2899986165 conn(0x7f319c072bc0 msgr2=0x7f319c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.270+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2899986165 shutdown_connections 2024-04-06T14:23:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.270+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2899986165 wait complete. 2024-04-06T14:23:21.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a3fbd700 1 Processor -- start 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a3fbd700 1 -- start start 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a3fbd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f319c0fde90 0x7f319c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a3fbd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 0x7f319c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a3fbd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c0fc820 con 0x7f319c0fe830 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a3fbd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c0fc960 con 0x7f319c0fde90 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a1558700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 0x7f319c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a1558700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 0x7f319c0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32920/0 (socket says 172.21.15.32:32920) 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a1558700 1 -- 172.21.15.32:0/2216915784 learned_addr learned my addr 172.21.15.32:0/2216915784 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a1d59700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f319c0fde90 0x7f319c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a1558700 1 -- 172.21.15.32:0/2216915784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f319c0fde90 msgr2=0x7f319c0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a1558700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f319c0fde90 0x7f319c0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.271+0000 7f31a1558700 1 -- 172.21.15.32:0/2216915784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318c005040 con 0x7f319c0fe830 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.272+0000 7f31a1558700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 0x7f319c0fc290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f318c00b3b0 tx=0x7f318c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:21.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.272+0000 7f31927fc700 1 -- 172.21.15.32:0/2216915784 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f318c00e040 con 0x7f319c0fe830 2024-04-06T14:23:21.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.272+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319c0fcbb0 con 0x7f319c0fe830 2024-04-06T14:23:21.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.272+0000 7f31927fc700 1 -- 172.21.15.32:0/2216915784 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f318c013420 con 0x7f319c0fe830 2024-04-06T14:23:21.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.272+0000 7f31927fc700 1 -- 172.21.15.32:0/2216915784 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f318c01c5f0 con 0x7f319c0fe830 2024-04-06T14:23:21.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.272+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c11a190 con 0x7f319c0fe830 2024-04-06T14:23:21.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.273+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c065e30 con 0x7f319c0fe830 2024-04-06T14:23:21.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.273+0000 7f31927fc700 1 -- 172.21.15.32:0/2216915784 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f318c013590 con 0x7f319c0fe830 2024-04-06T14:23:21.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.274+0000 7f31927fc700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f318806cc20 0x7f318806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:21.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.274+0000 7f31927fc700 1 -- 172.21.15.32:0/2216915784 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f318c08d340 con 0x7f319c0fe830 2024-04-06T14:23:21.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.274+0000 7f31a1d59700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f318806cc20 0x7f318806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:21.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.274+0000 7f31a1d59700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f318806cc20 0x7f318806f0d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3198004f40 tx=0x7f319800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:21.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.278+0000 7f31927fc700 1 -- 172.21.15.32:0/2216915784 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f318c057380 con 0x7f319c0fe830 2024-04-06T14:23:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:20 smithi086 ceph-mon[110975]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:20 smithi086 ceph-mon[110975]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:20 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/543035180' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:20 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:20 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:20 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:23:21.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:20 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1652997701' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:23:21.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.576+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f319c109e00 con 0x7f319c0fe830 2024-04-06T14:23:21.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.577+0000 7f31927fc700 1 -- 172.21.15.32:0/2216915784 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1350 (secure 0 0 0) 0x7f318c05a9a0 con 0x7f319c0fe830 2024-04-06T14:23:21.577 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; Degraded data redundancy: 60/393 objects degraded (15.267%), 16 pgs degraded 2024-04-06T14:23:21.577 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: osd.1 (root=default,host=smithi032) is down 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 60/393 objects degraded (15.267%), 16 pgs degraded 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.0 is active+undersized+degraded, acting [3,0] 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.2 is active+undersized+degraded, acting [5,0] 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.4 is active+undersized+degraded, acting [0,4] 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-04-06T14:23:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-04-06T14:23:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.a is active+undersized+degraded, acting [4,3] 2024-04-06T14:23:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-04-06T14:23:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-04-06T14:23:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.11 is active+undersized+degraded, acting [4,3] 2024-04-06T14:23:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.12 is active+undersized+degraded, acting [3,0] 2024-04-06T14:23:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-04-06T14:23:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-06T14:23:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1b is active+undersized+degraded, acting [0,5] 2024-04-06T14:23:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f318806cc20 msgr2=0x7f318806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f318806cc20 0x7f318806f0d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3198004f40 tx=0x7f319800b040 comp rx=0 tx=0).stop 2024-04-06T14:23:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 msgr2=0x7f319c0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 0x7f319c0fc290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f318c00b3b0 tx=0x7f318c009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 shutdown_connections 2024-04-06T14:23:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f318806cc20 0x7f318806f0d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f319c0fde90 0x7f319c0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 --2- 172.21.15.32:0/2216915784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f319c0fe830 0x7f319c0fc290 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 >> 172.21.15.32:0/2216915784 conn(0x7f319c072bc0 msgr2=0x7f319c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 shutdown_connections 2024-04-06T14:23:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:21.580+0000 7f31a3fbd700 1 -- 172.21.15.32:0/2216915784 wait complete. 2024-04-06T14:23:22.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:22 smithi032 ceph-mon[147414]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 60/393 objects degraded (15.267%) 2024-04-06T14:23:22.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:22 smithi032 ceph-mon[147414]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:22.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:22 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2216915784' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:23:22.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:22 smithi086 ceph-mon[110975]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 60/393 objects degraded (15.267%) 2024-04-06T14:23:22.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:22 smithi086 ceph-mon[110975]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:22.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:22 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2216915784' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:23:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:23 smithi032 ceph-mon[147414]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 60/393 objects degraded (15.267%) 2024-04-06T14:23:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:23:23.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:23 smithi086 ceph-mon[110975]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 60/393 objects degraded (15.267%) 2024-04-06T14:23:23.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:23.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:23:25.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:25 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[168700]: 2024-04-06T14:23:25.467+0000 7fa44278f640 -1 osd.1 55 log_to_monitors true 2024-04-06T14:23:26.230 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 14:23:25 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[168700]: 2024-04-06T14:23:25.980+0000 7fa433be8700 -1 osd.1 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:23:26.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:25 smithi032 ceph-mon[147414]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 60/393 objects degraded (15.267%) 2024-04-06T14:23:26.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:25 smithi032 ceph-mon[147414]: from='osd.1 [v2:172.21.15.32:6810/197940852,v1:172.21.15.32:6811/197940852]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T14:23:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:25 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:26.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:25 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:26.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:25 smithi086 ceph-mon[110975]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 60/393 objects degraded (15.267%) 2024-04-06T14:23:26.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:25 smithi086 ceph-mon[110975]: from='osd.1 [v2:172.21.15.32:6810/197940852,v1:172.21.15.32:6811/197940852]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T14:23:26.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:25 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:26.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:25 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:26 smithi032 ceph-mon[147414]: from='osd.1 [v2:172.21.15.32:6810/197940852,v1:172.21.15.32:6811/197940852]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T14:23:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:26 smithi032 ceph-mon[147414]: osdmap e58: 6 total, 5 up, 6 in 2024-04-06T14:23:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:26 smithi032 ceph-mon[147414]: from='osd.1 [v2:172.21.15.32:6810/197940852,v1:172.21.15.32:6811/197940852]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:23:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:26 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:27.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:26 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:27.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:26 smithi086 ceph-mon[110975]: from='osd.1 [v2:172.21.15.32:6810/197940852,v1:172.21.15.32:6811/197940852]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T14:23:27.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:26 smithi086 ceph-mon[110975]: osdmap e58: 6 total, 5 up, 6 in 2024-04-06T14:23:27.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:26 smithi086 ceph-mon[110975]: from='osd.1 [v2:172.21.15.32:6810/197940852,v1:172.21.15.32:6811/197940852]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:23:27.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:26 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:27.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:26 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:28.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:27 smithi032 ceph-mon[147414]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 60/393 objects degraded (15.267%) 2024-04-06T14:23:28.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:27 smithi032 ceph-mon[147414]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:23:28.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:27 smithi032 ceph-mon[147414]: osd.1 [v2:172.21.15.32:6810/197940852,v1:172.21.15.32:6811/197940852] boot 2024-04-06T14:23:28.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:27 smithi032 ceph-mon[147414]: osdmap e59: 6 total, 6 up, 6 in 2024-04-06T14:23:28.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:27 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:23:28.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:27 smithi032 ceph-mon[147414]: osdmap e60: 6 total, 6 up, 6 in 2024-04-06T14:23:28.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:27 smithi086 ceph-mon[110975]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 60/393 objects degraded (15.267%) 2024-04-06T14:23:28.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:27 smithi086 ceph-mon[110975]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:23:28.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:27 smithi086 ceph-mon[110975]: osd.1 [v2:172.21.15.32:6810/197940852,v1:172.21.15.32:6811/197940852] boot 2024-04-06T14:23:28.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:27 smithi086 ceph-mon[110975]: osdmap e59: 6 total, 6 up, 6 in 2024-04-06T14:23:28.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:27 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T14:23:28.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:27 smithi086 ceph-mon[110975]: osdmap e60: 6 total, 6 up, 6 in 2024-04-06T14:23:29.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:28 smithi086 ceph-mon[110975]: Health check update: Degraded data redundancy: 60/393 objects degraded (15.267%), 16 pgs degraded (PG_DEGRADED) 2024-04-06T14:23:29.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:28 smithi032 ceph-mon[147414]: Health check update: Degraded data redundancy: 60/393 objects degraded (15.267%), 16 pgs degraded (PG_DEGRADED) 2024-04-06T14:23:30.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:29 smithi032 ceph-mon[147414]: pgmap v76: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 25/393 objects degraded (6.361%) 2024-04-06T14:23:30.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:29 smithi086 ceph-mon[110975]: pgmap v76: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 25/393 objects degraded (6.361%) 2024-04-06T14:23:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: pgmap v77: 65 pgs: 7 active+undersized+degraded, 5 active+undersized, 53 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 25/393 objects degraded (6.361%) 2024-04-06T14:23:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:23:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:23:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:23:31.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:31.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:31.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:31.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T14:23:31.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T14:23:31.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:31 smithi032 ceph-mon[147414]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: pgmap v77: 65 pgs: 7 active+undersized+degraded, 5 active+undersized, 53 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 25/393 objects degraded (6.361%) 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T14:23:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T14:23:32.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:31 smithi086 ceph-mon[110975]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-06T14:23:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:32 smithi032 ceph-mon[147414]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/393 objects degraded (6.361%), 7 pgs degraded) 2024-04-06T14:23:33.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:32 smithi086 ceph-mon[110975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/393 objects degraded (6.361%), 7 pgs degraded) 2024-04-06T14:23:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:33 smithi032 ceph-mon[147414]: pgmap v78: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:34.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:33 smithi086 ceph-mon[110975]: pgmap v78: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:36.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:35 smithi032 ceph-mon[147414]: pgmap v79: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:36.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:35 smithi086 ceph-mon[110975]: pgmap v79: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:38.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:37 smithi032 ceph-mon[147414]: pgmap v80: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:38.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:38.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:23:38.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:38.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:37 smithi086 ceph-mon[110975]: pgmap v80: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:23:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:40.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:39 smithi032 ceph-mon[147414]: pgmap v81: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:40.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:39 smithi086 ceph-mon[110975]: pgmap v81: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:42.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:41 smithi032 ceph-mon[147414]: pgmap v82: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:42.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:41 smithi086 ceph-mon[110975]: pgmap v82: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:44.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:43 smithi032 ceph-mon[147414]: pgmap v83: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:44.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:43 smithi086 ceph-mon[110975]: pgmap v83: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:46.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:45 smithi032 ceph-mon[147414]: pgmap v84: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:46.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:45 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T14:23:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:45 smithi086 ceph-mon[110975]: pgmap v84: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:45 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T14:23:46.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:46 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T14:23:46.982 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:46 smithi032 ceph-mon[147414]: Upgrade: osd.2 is safe to restart 2024-04-06T14:23:47.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:46 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T14:23:47.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:46 smithi086 ceph-mon[110975]: Upgrade: osd.2 is safe to restart 2024-04-06T14:23:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:47 smithi032 ceph-mon[147414]: pgmap v85: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:47 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:47 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T14:23:48.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:47 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:23:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:47 smithi086 ceph-mon[110975]: pgmap v85: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:47 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:47 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T14:23:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:47 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:23:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:48 smithi032 ceph-mon[147414]: Upgrade: Updating osd.2 2024-04-06T14:23:49.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:48 smithi032 ceph-mon[147414]: Deploying daemon osd.2 on smithi032 2024-04-06T14:23:49.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:48 smithi086 ceph-mon[110975]: Upgrade: Updating osd.2 2024-04-06T14:23:49.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:48 smithi086 ceph-mon[110975]: Deploying daemon osd.2 on smithi032 2024-04-06T14:23:50.221 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:49 smithi032 ceph-mon[147414]: pgmap v86: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:50.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:49 smithi086 ceph-mon[110975]: pgmap v86: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:51.230 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:50 smithi032 systemd[1]: Stopping Ceph osd.2 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:23:51.230 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:51 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[82087]: 2024-04-06T14:23:51.147+0000 7fbd2c806700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T14:23:51.230 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:51 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[82087]: 2024-04-06T14:23:51.147+0000 7fbd2c806700 -1 osd.2 60 *** Got signal Terminated *** 2024-04-06T14:23:51.230 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:51 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[82087]: 2024-04-06T14:23:51.147+0000 7fbd2c806700 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T14:23:51.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 -- 172.21.15.32:0/2272763964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 msgr2=0x7f35c00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:51.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 --2- 172.21.15.32:0/2272763964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 0x7f35c00fbd80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f35b4004940 tx=0x7f35b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:51.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 -- 172.21.15.32:0/2272763964 shutdown_connections 2024-04-06T14:23:51.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 --2- 172.21.15.32:0/2272763964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 0x7f35c00fbd80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 --2- 172.21.15.32:0/2272763964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c0101d70 0x7f35c0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 -- 172.21.15.32:0/2272763964 >> 172.21.15.32:0/2272763964 conn(0x7f35c0072bc0 msgr2=0x7f35c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 -- 172.21.15.32:0/2272763964 shutdown_connections 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 -- 172.21.15.32:0/2272763964 wait complete. 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 Processor -- start 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.701+0000 7f35c690a700 1 -- start start 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35c690a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c0101d70 0x7f35c0102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35c690a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 0x7f35c0103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35c690a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c01036a0 con 0x7f35c0102710 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35c690a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c01037e0 con 0x7f35c0101d70 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 0x7f35c0103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 0x7f35c0103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51426/0 (socket says 172.21.15.32:51426) 2024-04-06T14:23:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35bf7fe700 1 -- 172.21.15.32:0/2200451192 learned_addr learned my addr 172.21.15.32:0/2200451192 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35bffff700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c0101d70 0x7f35c0102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35bf7fe700 1 -- 172.21.15.32:0/2200451192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c0101d70 msgr2=0x7f35c0102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35bf7fe700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c0101d70 0x7f35c0102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35bf7fe700 1 -- 172.21.15.32:0/2200451192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b4005040 con 0x7f35c0102710 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.702+0000 7f35bf7fe700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 0x7f35c0103110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009390 tx=0x7f35b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.703+0000 7f35bcff9700 1 -- 172.21.15.32:0/2200451192 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b400e040 con 0x7f35c0102710 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.703+0000 7f35bcff9700 1 -- 172.21.15.32:0/2200451192 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35b4013420 con 0x7f35c0102710 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.703+0000 7f35bcff9700 1 -- 172.21.15.32:0/2200451192 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b401c630 con 0x7f35c0102710 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.703+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c0103a30 con 0x7f35c0102710 2024-04-06T14:23:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.703+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c011a1a0 con 0x7f35c0102710 2024-04-06T14:23:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.704+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c00fd4c0 con 0x7f35c0102710 2024-04-06T14:23:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.704+0000 7f35bcff9700 1 -- 172.21.15.32:0/2200451192 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f35b4013a00 con 0x7f35c0102710 2024-04-06T14:23:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.705+0000 7f35bcff9700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f35a806cc30 0x7f35a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.705+0000 7f35bcff9700 1 -- 172.21.15.32:0/2200451192 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f35b408dcd0 con 0x7f35c0102710 2024-04-06T14:23:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.705+0000 7f35bffff700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f35a806cc30 0x7f35a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.705+0000 7f35bffff700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f35a806cc30 0x7f35a806f0e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f35b0004d60 tx=0x7f35b000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.709+0000 7f35bcff9700 1 -- 172.21.15.32:0/2200451192 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f35b4057e50 con 0x7f35c0102710 2024-04-06T14:23:51.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.932+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f35c004bed0 con 0x7f35a806cc30 2024-04-06T14:23:51.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.933+0000 7f35bcff9700 1 -- 172.21.15.32:0/2200451192 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f35c004bed0 con 0x7f35a806cc30 2024-04-06T14:23:51.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f35a806cc30 msgr2=0x7f35a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:51.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f35a806cc30 0x7f35a806f0e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f35b0004d60 tx=0x7f35b000b040 comp rx=0 tx=0).stop 2024-04-06T14:23:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 msgr2=0x7f35c0103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 0x7f35c0103110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009390 tx=0x7f35b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 shutdown_connections 2024-04-06T14:23:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f35a806cc30 0x7f35a806f0e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c0101d70 0x7f35c0102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 --2- 172.21.15.32:0/2200451192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c0102710 0x7f35c0103110 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 >> 172.21.15.32:0/2200451192 conn(0x7f35c0072bc0 msgr2=0x7f35c0065a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 shutdown_connections 2024-04-06T14:23:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:51.935+0000 7f35c690a700 1 -- 172.21.15.32:0/2200451192 wait complete. 2024-04-06T14:23:51.946 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:23:52.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.050+0000 7f4fbe520700 1 -- 172.21.15.32:0/2806336726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fdde0 msgr2=0x7f4fb80fe1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.050+0000 7f4fbe520700 1 --2- 172.21.15.32:0/2806336726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fdde0 0x7f4fb80fe1b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8004940 tx=0x7f4fa8009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:52.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.050+0000 7f4fbe520700 1 -- 172.21.15.32:0/2806336726 shutdown_connections 2024-04-06T14:23:52.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.050+0000 7f4fbe520700 1 --2- 172.21.15.32:0/2806336726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb80fe780 0x7f4fb8106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.050+0000 7f4fbe520700 1 --2- 172.21.15.32:0/2806336726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fdde0 0x7f4fb80fe1b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.050+0000 7f4fbe520700 1 -- 172.21.15.32:0/2806336726 >> 172.21.15.32:0/2806336726 conn(0x7f4fb80f7860 msgr2=0x7f4fb80f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:52.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.050+0000 7f4fbe520700 1 -- 172.21.15.32:0/2806336726 shutdown_connections 2024-04-06T14:23:52.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.050+0000 7f4fbe520700 1 -- 172.21.15.32:0/2806336726 wait complete. 2024-04-06T14:23:52.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.050+0000 7f4fbe520700 1 Processor -- start 2024-04-06T14:23:52.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fbe520700 1 -- start start 2024-04-06T14:23:52.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fbe520700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb80fdde0 0x7f4fb8115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:52.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fbe520700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fe780 0x7f4fb8116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fbe520700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb81167e0 con 0x7f4fb80fe780 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fbe520700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb810fbb0 con 0x7f4fb80fdde0 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fe780 0x7f4fb8116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fe780 0x7f4fb8116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51448/0 (socket says 172.21.15.32:51448) 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb77fe700 1 -- 172.21.15.32:0/2034310249 learned_addr learned my addr 172.21.15.32:0/2034310249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb77fe700 1 -- 172.21.15.32:0/2034310249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb80fdde0 msgr2=0x7f4fb8115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb7fff700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb80fdde0 0x7f4fb8115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb77fe700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb80fdde0 0x7f4fb8115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb77fe700 1 -- 172.21.15.32:0/2034310249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa8005040 con 0x7f4fb80fe780 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb7fff700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb80fdde0 0x7f4fb8115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb77fe700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fe780 0x7f4fb8116190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4fac002a50 tx=0x7f4fac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb4ff9700 1 -- 172.21.15.32:0/2034310249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fac007cc0 con 0x7f4fb80fe780 2024-04-06T14:23:52.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.051+0000 7f4fb4ff9700 1 -- 172.21.15.32:0/2034310249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4fac00f040 con 0x7f4fb80fe780 2024-04-06T14:23:52.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.052+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb810fe90 con 0x7f4fb80fe780 2024-04-06T14:23:52.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.052+0000 7f4fb4ff9700 1 -- 172.21.15.32:0/2034310249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fac009540 con 0x7f4fb80fe780 2024-04-06T14:23:52.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.052+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb81103d0 con 0x7f4fb80fe780 2024-04-06T14:23:52.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.052+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb80666f0 con 0x7f4fb80fe780 2024-04-06T14:23:52.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.053+0000 7f4fb4ff9700 1 -- 172.21.15.32:0/2034310249 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4fac0096a0 con 0x7f4fb80fe780 2024-04-06T14:23:52.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.054+0000 7f4fb4ff9700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4fa006cc20 0x7f4fa006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:52.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.054+0000 7f4fb4ff9700 1 -- 172.21.15.32:0/2034310249 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f4fac08bbc0 con 0x7f4fb80fe780 2024-04-06T14:23:52.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.054+0000 7f4fb7fff700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4fa006cc20 0x7f4fa006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:52.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.054+0000 7f4fb7fff700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4fa006cc20 0x7f4fa006f0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8000c00 tx=0x7f4fa8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:52.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.058+0000 7f4fb4ff9700 1 -- 172.21.15.32:0/2034310249 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4fac059730 con 0x7f4fb80fe780 2024-04-06T14:23:52.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.278+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fb8109da0 con 0x7f4fa006cc20 2024-04-06T14:23:52.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.280+0000 7f4fb4ff9700 1 -- 172.21.15.32:0/2034310249 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4fb8109da0 con 0x7f4fa006cc20 2024-04-06T14:23:52.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4fa006cc20 msgr2=0x7f4fa006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4fa006cc20 0x7f4fa006f0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8000c00 tx=0x7f4fa8009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:52.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fe780 msgr2=0x7f4fb8116190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fe780 0x7f4fb8116190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4fac002a50 tx=0x7f4fac0046b0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 shutdown_connections 2024-04-06T14:23:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4fa006cc20 0x7f4fa006f0d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb80fdde0 0x7f4fb8115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 --2- 172.21.15.32:0/2034310249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fb80fe780 0x7f4fb8116190 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 >> 172.21.15.32:0/2034310249 conn(0x7f4fb80f7860 msgr2=0x7f4fb80f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 shutdown_connections 2024-04-06T14:23:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.282+0000 7f4fbe520700 1 -- 172.21.15.32:0/2034310249 wait complete. 2024-04-06T14:23:52.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:51 smithi086 ceph-mon[110975]: pgmap v87: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:52.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:51 smithi086 ceph-mon[110975]: osd.2 marked itself down and dead 2024-04-06T14:23:52.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.386+0000 7f441c66c700 1 -- 172.21.15.32:0/2926721079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 msgr2=0x7f44140fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.386+0000 7f441c66c700 1 --2- 172.21.15.32:0/2926721079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 0x7f44140fce20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4408004940 tx=0x7f4408009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:52.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.386+0000 7f441c66c700 1 -- 172.21.15.32:0/2926721079 shutdown_connections 2024-04-06T14:23:52.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.386+0000 7f441c66c700 1 --2- 172.21.15.32:0/2926721079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 0x7f44140fce20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.386+0000 7f441c66c700 1 --2- 172.21.15.32:0/2926721079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4414102e10 0x7f44141031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.386+0000 7f441c66c700 1 -- 172.21.15.32:0/2926721079 >> 172.21.15.32:0/2926721079 conn(0x7f44140f89a0 msgr2=0x7f44140fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.386+0000 7f441c66c700 1 -- 172.21.15.32:0/2926721079 shutdown_connections 2024-04-06T14:23:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.386+0000 7f441c66c700 1 -- 172.21.15.32:0/2926721079 wait complete. 2024-04-06T14:23:52.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f441c66c700 1 Processor -- start 2024-04-06T14:23:52.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f441c66c700 1 -- start start 2024-04-06T14:23:52.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f441c66c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4414102e10 0x7f4414113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:52.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f441c66c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 0x7f4414113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:52.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f441c66c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44141145c0 con 0x7f44141037b0 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f441c66c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441410d9b0 con 0x7f4414102e10 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f4419c07700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 0x7f4414113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f4419c07700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 0x7f4414113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51478/0 (socket says 172.21.15.32:51478) 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f4419c07700 1 -- 172.21.15.32:0/881579138 learned_addr learned my addr 172.21.15.32:0/881579138 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f441a408700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4414102e10 0x7f4414113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f4419c07700 1 -- 172.21.15.32:0/881579138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4414102e10 msgr2=0x7f4414113a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f4419c07700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4414102e10 0x7f4414113a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f4419c07700 1 -- 172.21.15.32:0/881579138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4408005040 con 0x7f44141037b0 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.387+0000 7f441a408700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4414102e10 0x7f4414113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.388+0000 7f4419c07700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 0x7f4414113f70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4408000c00 tx=0x7f4408009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.388+0000 7f4406ffd700 1 -- 172.21.15.32:0/881579138 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f440800e070 con 0x7f44141037b0 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.388+0000 7f4406ffd700 1 -- 172.21.15.32:0/881579138 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4408013de0 con 0x7f44141037b0 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.388+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441410dc30 con 0x7f44141037b0 2024-04-06T14:23:52.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.388+0000 7f4406ffd700 1 -- 172.21.15.32:0/881579138 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f440801cde0 con 0x7f44141037b0 2024-04-06T14:23:52.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.388+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441410e190 con 0x7f44141037b0 2024-04-06T14:23:52.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.389+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44140fe560 con 0x7f44141037b0 2024-04-06T14:23:52.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.389+0000 7f4406ffd700 1 -- 172.21.15.32:0/881579138 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f440801c600 con 0x7f44141037b0 2024-04-06T14:23:52.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.390+0000 7f4406ffd700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f440006cc20 0x7f440006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:52.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.390+0000 7f4406ffd700 1 -- 172.21.15.32:0/881579138 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f440808d400 con 0x7f44141037b0 2024-04-06T14:23:52.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.390+0000 7f441a408700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f440006cc20 0x7f440006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:52.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.390+0000 7f441a408700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f440006cc20 0x7f440006f0d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4410004600 tx=0x7f441000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:52.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.394+0000 7f4406ffd700 1 -- 172.21.15.32:0/881579138 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f440805b010 con 0x7f44141037b0 2024-04-06T14:23:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:51 smithi032 ceph-mon[147414]: pgmap v87: 65 pgs: 65 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:51 smithi032 ceph-mon[147414]: osd.2 marked itself down and dead 2024-04-06T14:23:52.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.615+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4414109db0 con 0x7f440006cc20 2024-04-06T14:23:52.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.625+0000 7f4406ffd700 1 -- 172.21.15.32:0/881579138 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4414109db0 con 0x7f440006cc20 2024-04-06T14:23:52.625 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:23:52.625 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (4m) 26s ago 15m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (95s) 26s ago 15m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (90s) 87s ago 14m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (3m) 26s ago 15m 81.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (9m) 26s ago 9m 14.3M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (9m) 26s ago 9m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (9m) 87s ago 9m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (9m) 87s ago 9m 16.5M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (6m) 26s ago 16m 501M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (5m) 87s ago 14m 448M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (2m) 26s ago 17m 46.2M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (2m) 87s ago 14m 32.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (4m) 26s ago 15m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (4m) 87s ago 14m 17.0M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:23:52.626 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (69s) 26s ago 13m 78.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:23:52.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (32s) 26s ago 13m 10.7M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:23:52.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (12m) 26s ago 12m 163M 1570M 17.2.7-819-g6699495c ec300eaa768b 1bf391939c7e 2024-04-06T14:23:52.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (12m) 87s ago 12m 226M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:23:52.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (11m) 87s ago 11m 203M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:23:52.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (11m) 87s ago 11m 187M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:23:52.627 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (4m) 26s ago 15m 59.8M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.627+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f440006cc20 msgr2=0x7f440006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.627+0000 7f441c66c700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f440006cc20 0x7f440006f0d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4410004600 tx=0x7f441000b040 comp rx=0 tx=0).stop 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.627+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 msgr2=0x7f4414113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.627+0000 7f441c66c700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 0x7f4414113f70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4408000c00 tx=0x7f4408009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.627+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 shutdown_connections 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.628+0000 7f441c66c700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f440006cc20 0x7f440006f0d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.628+0000 7f441c66c700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4414102e10 0x7f4414113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.628+0000 7f441c66c700 1 --2- 172.21.15.32:0/881579138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44141037b0 0x7f4414113f70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.628+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 >> 172.21.15.32:0/881579138 conn(0x7f44140f89a0 msgr2=0x7f44140f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.628+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 shutdown_connections 2024-04-06T14:23:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.628+0000 7f441c66c700 1 -- 172.21.15.32:0/881579138 wait complete. 2024-04-06T14:23:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.730+0000 7f5d00015700 1 -- 172.21.15.32:0/693348679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 msgr2=0x7f5cf8103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.730+0000 7f5d00015700 1 --2- 172.21.15.32:0/693348679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 0x7f5cf8103240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4004940 tx=0x7f5cf4009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.730+0000 7f5d00015700 1 -- 172.21.15.32:0/693348679 shutdown_connections 2024-04-06T14:23:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.730+0000 7f5d00015700 1 --2- 172.21.15.32:0/693348679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf80fb4a0 0x7f5cf80fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.730+0000 7f5d00015700 1 --2- 172.21.15.32:0/693348679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 0x7f5cf8103240 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.730+0000 7f5d00015700 1 -- 172.21.15.32:0/693348679 >> 172.21.15.32:0/693348679 conn(0x7f5cf80714c0 msgr2=0x7f5cf80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.730+0000 7f5d00015700 1 -- 172.21.15.32:0/693348679 shutdown_connections 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.730+0000 7f5d00015700 1 -- 172.21.15.32:0/693348679 wait complete. 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.731+0000 7f5d00015700 1 Processor -- start 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.731+0000 7f5d00015700 1 -- start start 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.731+0000 7f5d00015700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf80fb4a0 0x7f5cf8113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.731+0000 7f5d00015700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 0x7f5cf8114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.731+0000 7f5d00015700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf8114730 con 0x7f5cf8102e70 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.731+0000 7f5d00015700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf810da90 con 0x7f5cf80fb4a0 2024-04-06T14:23:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.731+0000 7f5cfd5b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 0x7f5cf8114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.731+0000 7f5cfd5b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 0x7f5cf8114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51498/0 (socket says 172.21.15.32:51498) 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.731+0000 7f5cfd5b0700 1 -- 172.21.15.32:0/1026679786 learned_addr learned my addr 172.21.15.32:0/1026679786 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5cfddb1700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf80fb4a0 0x7f5cf8113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5cfd5b0700 1 -- 172.21.15.32:0/1026679786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf80fb4a0 msgr2=0x7f5cf8113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5cfd5b0700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf80fb4a0 0x7f5cf8113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5cfd5b0700 1 -- 172.21.15.32:0/1026679786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf4005040 con 0x7f5cf8102e70 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5cfddb1700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf80fb4a0 0x7f5cf8113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5cfd5b0700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 0x7f5cf8114050 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00b790 tx=0x7f5cec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5cea7fc700 1 -- 172.21.15.32:0/1026679786 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cec00f770 con 0x7f5cf8102e70 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5cea7fc700 1 -- 172.21.15.32:0/1026679786 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cec00fd80 con 0x7f5cf8102e70 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5cea7fc700 1 -- 172.21.15.32:0/1026679786 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cec00d310 con 0x7f5cf8102e70 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf810dd70 con 0x7f5cf8102e70 2024-04-06T14:23:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.732+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf810e2b0 con 0x7f5cf8102e70 2024-04-06T14:23:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.733+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf8100170 con 0x7f5cf8102e70 2024-04-06T14:23:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.734+0000 7f5cea7fc700 1 -- 172.21.15.32:0/1026679786 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5cec010420 con 0x7f5cf8102e70 2024-04-06T14:23:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.734+0000 7f5cea7fc700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5ce406cc30 0x7f5ce406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.734+0000 7f5cea7fc700 1 -- 172.21.15.32:0/1026679786 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f5cec08bfb0 con 0x7f5cf8102e70 2024-04-06T14:23:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.734+0000 7f5cfddb1700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5ce406cc30 0x7f5ce406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.734+0000 7f5cfddb1700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5ce406cc30 0x7f5ce406f0e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400b840 tx=0x7f5cf4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:52.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:52.738+0000 7f5cea7fc700 1 -- 172.21.15.32:0/1026679786 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5cec059b20 con 0x7f5cf8102e70 2024-04-06T14:23:53.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.040+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5cf810f160 con 0x7f5cf8102e70 2024-04-06T14:23:53.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.041+0000 7f5cea7fc700 1 -- 172.21.15.32:0/1026679786 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f5cec0596b0 con 0x7f5cf8102e70 2024-04-06T14:23:53.041 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:23:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:23:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:23:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:23:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 3, 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 7, 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:23:53.042 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:23:53.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5ce406cc30 msgr2=0x7f5ce406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5ce406cc30 0x7f5ce406f0e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400b840 tx=0x7f5cf4009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:53.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 msgr2=0x7f5cf8114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 0x7f5cf8114050 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00b790 tx=0x7f5cec009250 comp rx=0 tx=0).stop 2024-04-06T14:23:53.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 shutdown_connections 2024-04-06T14:23:53.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5ce406cc30 0x7f5ce406f0e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf80fb4a0 0x7f5cf8113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 --2- 172.21.15.32:0/1026679786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cf8102e70 0x7f5cf8114050 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 >> 172.21.15.32:0/1026679786 conn(0x7f5cf80714c0 msgr2=0x7f5cf80fa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:53.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 shutdown_connections 2024-04-06T14:23:53.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.043+0000 7f5d00015700 1 -- 172.21.15.32:0/1026679786 wait complete. 2024-04-06T14:23:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.153+0000 7fbb93018700 1 -- 172.21.15.32:0/425794131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c0fb4d0 msgr2=0x7fbb8c0fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.153+0000 7fbb93018700 1 --2- 172.21.15.32:0/425794131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c0fb4d0 0x7fbb8c0fb940 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbb78004940 tx=0x7fbb78009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.153+0000 7fbb93018700 1 -- 172.21.15.32:0/425794131 shutdown_connections 2024-04-06T14:23:53.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.153+0000 7fbb93018700 1 --2- 172.21.15.32:0/425794131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c0fb4d0 0x7fbb8c0fb940 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.153+0000 7fbb93018700 1 --2- 172.21.15.32:0/425794131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb8c102f50 0x7fbb8c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.153+0000 7fbb93018700 1 -- 172.21.15.32:0/425794131 >> 172.21.15.32:0/425794131 conn(0x7fbb8c0714c0 msgr2=0x7fbb8c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:53.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.153+0000 7fbb93018700 1 -- 172.21.15.32:0/425794131 shutdown_connections 2024-04-06T14:23:53.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.153+0000 7fbb93018700 1 -- 172.21.15.32:0/425794131 wait complete. 2024-04-06T14:23:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.153+0000 7fbb93018700 1 Processor -- start 2024-04-06T14:23:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb93018700 1 -- start start 2024-04-06T14:23:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb93018700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb8c0fb4d0 0x7fbb8c113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb93018700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c102f50 0x7fbb8c113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb93018700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8c1144b0 con 0x7fbb8c102f50 2024-04-06T14:23:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb93018700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8c10dab0 con 0x7fbb8c0fb4d0 2024-04-06T14:23:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb90db4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb8c0fb4d0 0x7fbb8c113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb8bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c102f50 0x7fbb8c113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb90db4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb8c0fb4d0 0x7fbb8c113920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:39780/0 (socket says 172.21.15.32:39780) 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb90db4700 1 -- 172.21.15.32:0/3742121872 learned_addr learned my addr 172.21.15.32:0/3742121872 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb8bfff700 1 -- 172.21.15.32:0/3742121872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb8c0fb4d0 msgr2=0x7fbb8c113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb8bfff700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb8c0fb4d0 0x7fbb8c113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb8bfff700 1 -- 172.21.15.32:0/3742121872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb78005040 con 0x7fbb8c102f50 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.154+0000 7fbb8bfff700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c102f50 0x7fbb8c113e60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800b3b0 tx=0x7fbb78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.155+0000 7fbb897fa700 1 -- 172.21.15.32:0/3742121872 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb7800e040 con 0x7fbb8c102f50 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.155+0000 7fbb897fa700 1 -- 172.21.15.32:0/3742121872 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb78009b10 con 0x7fbb8c102f50 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.155+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8c10dc50 con 0x7fbb8c102f50 2024-04-06T14:23:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.155+0000 7fbb897fa700 1 -- 172.21.15.32:0/3742121872 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb7801ce70 con 0x7fbb8c102f50 2024-04-06T14:23:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.155+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8c10e190 con 0x7fbb8c102f50 2024-04-06T14:23:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.156+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8c105330 con 0x7fbb8c102f50 2024-04-06T14:23:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.156+0000 7fbb897fa700 1 -- 172.21.15.32:0/3742121872 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbb7801c6c0 con 0x7fbb8c102f50 2024-04-06T14:23:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.157+0000 7fbb897fa700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb7c06cc20 0x7fbb7c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.157+0000 7fbb897fa700 1 -- 172.21.15.32:0/3742121872 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fbb7808c9d0 con 0x7fbb8c102f50 2024-04-06T14:23:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.157+0000 7fbb90db4700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb7c06cc20 0x7fbb7c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.157+0000 7fbb90db4700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb7c06cc20 0x7fbb7c06f0d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb80010210 tx=0x7fbb8000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.161+0000 7fbb897fa700 1 -- 172.21.15.32:0/3742121872 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbb7805a5f0 con 0x7fbb8c102f50 2024-04-06T14:23:53.230 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:53 smithi032 bash[175316]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2 2024-04-06T14:23:53.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:52 smithi032 ceph-mon[147414]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:53.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:52 smithi032 ceph-mon[147414]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:23:53.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:52 smithi032 ceph-mon[147414]: osdmap e61: 6 total, 5 up, 6 in 2024-04-06T14:23:53.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:52 smithi032 ceph-mon[147414]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:53.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:53.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:23:53.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:52 smithi086 ceph-mon[110975]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:53.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:52 smithi086 ceph-mon[110975]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:23:53.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:52 smithi086 ceph-mon[110975]: osdmap e61: 6 total, 5 up, 6 in 2024-04-06T14:23:53.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:52 smithi086 ceph-mon[110975]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:53.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:53.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:23:53.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.413+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbb8c0fe7f0 con 0x7fbb8c102f50 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.414+0000 7fbb897fa700 1 -- 172.21.15.32:0/3742121872 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1851 (secure 0 0 0) 0x7fbb7805a180 con 0x7fbb8c102f50 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:23:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:21:28.458361+0000 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:23:53.415 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:23:53.416 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:23:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb7c06cc20 msgr2=0x7fbb7c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb7c06cc20 0x7fbb7c06f0d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb80010210 tx=0x7fbb8000b040 comp rx=0 tx=0).stop 2024-04-06T14:23:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c102f50 msgr2=0x7fbb8c113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c102f50 0x7fbb8c113e60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800b3b0 tx=0x7fbb78009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 shutdown_connections 2024-04-06T14:23:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb7c06cc20 0x7fbb7c06f0d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb8c0fb4d0 0x7fbb8c113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 --2- 172.21.15.32:0/3742121872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c102f50 0x7fbb8c113e60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 >> 172.21.15.32:0/3742121872 conn(0x7fbb8c0714c0 msgr2=0x7fbb8c0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 shutdown_connections 2024-04-06T14:23:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.416+0000 7fbb93018700 1 -- 172.21.15.32:0/3742121872 wait complete. 2024-04-06T14:23:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-06T14:23:53.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.529+0000 7f7f814c5700 1 -- 172.21.15.32:0/2225835134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070300 msgr2=0x7f7f7c0706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.529+0000 7f7f814c5700 1 --2- 172.21.15.32:0/2225835134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070300 0x7f7f7c0706d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c004970 tx=0x7f7f6c009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:53.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.529+0000 7f7f814c5700 1 -- 172.21.15.32:0/2225835134 shutdown_connections 2024-04-06T14:23:53.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.529+0000 7f7f814c5700 1 --2- 172.21.15.32:0/2225835134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f7c070c10 0x7f7f7c109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.529+0000 7f7f814c5700 1 --2- 172.21.15.32:0/2225835134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070300 0x7f7f7c0706d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.529+0000 7f7f814c5700 1 -- 172.21.15.32:0/2225835134 >> 172.21.15.32:0/2225835134 conn(0x7f7f7c0f8980 msgr2=0x7f7f7c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:53.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.529+0000 7f7f814c5700 1 -- 172.21.15.32:0/2225835134 shutdown_connections 2024-04-06T14:23:53.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.529+0000 7f7f814c5700 1 -- 172.21.15.32:0/2225835134 wait complete. 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.530+0000 7f7f814c5700 1 Processor -- start 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.530+0000 7f7f814c5700 1 -- start start 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.530+0000 7f7f814c5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f7c070300 0x7f7f7c115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.530+0000 7f7f814c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070c10 0x7f7f7c116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.530+0000 7f7f814c5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c116880 con 0x7f7f7c070c10 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.530+0000 7f7f814c5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c10fc50 con 0x7f7f7c070300 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.530+0000 7f7f7a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070c10 0x7f7f7c116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f7a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070c10 0x7f7f7c116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51540/0 (socket says 172.21.15.32:51540) 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f7a7fc700 1 -- 172.21.15.32:0/97631115 learned_addr learned my addr 172.21.15.32:0/97631115 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f7affd700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f7c070300 0x7f7f7c115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f7a7fc700 1 -- 172.21.15.32:0/97631115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f7c070300 msgr2=0x7f7f7c115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f7a7fc700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f7c070300 0x7f7f7c115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f7a7fc700 1 -- 172.21.15.32:0/97631115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c005040 con 0x7f7f7c070c10 2024-04-06T14:23:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f7a7fc700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070c10 0x7f7f7c116230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c8c0 tx=0x7f7f700044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f637fe700 1 -- 172.21.15.32:0/97631115 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f7000a760 con 0x7f7f7c070c10 2024-04-06T14:23:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7c10ff30 con 0x7f7f7c070c10 2024-04-06T14:23:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f637fe700 1 -- 172.21.15.32:0/97631115 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f7000ad70 con 0x7f7f7c070c10 2024-04-06T14:23:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f637fe700 1 -- 172.21.15.32:0/97631115 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f700113a0 con 0x7f7f7c070c10 2024-04-06T14:23:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.531+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7c110470 con 0x7f7f7c070c10 2024-04-06T14:23:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.533+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7c103b90 con 0x7f7f7c070c10 2024-04-06T14:23:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.533+0000 7f7f637fe700 1 -- 172.21.15.32:0/97631115 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7f70012400 con 0x7f7f7c070c10 2024-04-06T14:23:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.533+0000 7f7f637fe700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7f6406cc20 0x7f7f6406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.533+0000 7f7f637fe700 1 -- 172.21.15.32:0/97631115 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7f7008bef0 con 0x7f7f7c070c10 2024-04-06T14:23:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.533+0000 7f7f7affd700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7f6406cc20 0x7f7f6406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:53.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.534+0000 7f7f7affd700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7f6406cc20 0x7f7f6406f0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c004f50 tx=0x7f7f6c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:53.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.538+0000 7f7f637fe700 1 -- 172.21.15.32:0/97631115 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7f70059b00 con 0x7f7f7c070c10 2024-04-06T14:23:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.765+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f7c111340 con 0x7f7f6406cc20 2024-04-06T14:23:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.767+0000 7f7f637fe700 1 -- 172.21.15.32:0/97631115 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7f7c111340 con 0x7f7f6406cc20 2024-04-06T14:23:53.767 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:23:53.767 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:23:53.767 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:23:53.767 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:23:53.767 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:23:53.767 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:23:53.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:23:53.768 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:23:53.768 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:23:53.768 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "8/21 daemons upgraded", 2024-04-06T14:23:53.768 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T14:23:53.768 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:23:53.768 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:23:53.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7f6406cc20 msgr2=0x7f7f6406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7f6406cc20 0x7f7f6406f0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c004f50 tx=0x7f7f6c009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070c10 msgr2=0x7f7f7c116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070c10 0x7f7f7c116230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c8c0 tx=0x7f7f700044c0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 shutdown_connections 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7f6406cc20 0x7f7f6406f0d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f7c070300 0x7f7f7c115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 --2- 172.21.15.32:0/97631115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f7c070c10 0x7f7f7c116230 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 >> 172.21.15.32:0/97631115 conn(0x7f7f7c0f8980 msgr2=0x7f7f7c101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 shutdown_connections 2024-04-06T14:23:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.769+0000 7f7f814c5700 1 -- 172.21.15.32:0/97631115 wait complete. 2024-04-06T14:23:53.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.878+0000 7fe97837a700 1 -- 172.21.15.32:0/2539754074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 msgr2=0x7fe970106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.878+0000 7fe97837a700 1 --2- 172.21.15.32:0/2539754074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 0x7fe970106f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe960004940 tx=0x7fe960009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 -- 172.21.15.32:0/2539754074 shutdown_connections 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 --2- 172.21.15.32:0/2539754074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 0x7fe970106f00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 --2- 172.21.15.32:0/2539754074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9700fde90 0x7fe9700fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 -- 172.21.15.32:0/2539754074 >> 172.21.15.32:0/2539754074 conn(0x7fe970072bc0 msgr2=0x7fe970072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 -- 172.21.15.32:0/2539754074 shutdown_connections 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 -- 172.21.15.32:0/2539754074 wait complete. 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 Processor -- start 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 -- start start 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9700fde90 0x7fe970113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 0x7fe970113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe970114610 con 0x7fe9700fe830 2024-04-06T14:23:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.879+0000 7fe97837a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97010da00 con 0x7fe9700fde90 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe975915700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 0x7fe970113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe975915700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 0x7fe970113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51562/0 (socket says 172.21.15.32:51562) 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe975915700 1 -- 172.21.15.32:0/2344367111 learned_addr learned my addr 172.21.15.32:0/2344367111 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe976116700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9700fde90 0x7fe970113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe975915700 1 -- 172.21.15.32:0/2344367111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9700fde90 msgr2=0x7fe970113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe975915700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9700fde90 0x7fe970113a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe975915700 1 -- 172.21.15.32:0/2344367111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe960005040 con 0x7fe9700fe830 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe975915700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 0x7fe970113fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe960000c00 tx=0x7fe960009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe966ffd700 1 -- 172.21.15.32:0/2344367111 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe960021070 con 0x7fe9700fe830 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe966ffd700 1 -- 172.21.15.32:0/2344367111 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe960013e00 con 0x7fe9700fe830 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe97010dc80 con 0x7fe9700fe830 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe966ffd700 1 -- 172.21.15.32:0/2344367111 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe96000e070 con 0x7fe9700fe830 2024-04-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.880+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe97010e1c0 con 0x7fe9700fe830 2024-04-06T14:23:53.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.881+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe970101980 con 0x7fe9700fe830 2024-04-06T14:23:53.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.882+0000 7fe966ffd700 1 -- 172.21.15.32:0/2344367111 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe96001c590 con 0x7fe9700fe830 2024-04-06T14:23:53.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.882+0000 7fe966ffd700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe95c06cc20 0x7fe95c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:23:53.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.882+0000 7fe966ffd700 1 -- 172.21.15.32:0/2344367111 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fe96008d030 con 0x7fe9700fe830 2024-04-06T14:23:53.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.882+0000 7fe976116700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe95c06cc20 0x7fe95c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:23:53.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.883+0000 7fe976116700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe95c06cc20 0x7fe95c06f0d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009b80 tx=0x7fe96c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:23:53.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:53.887+0000 7fe966ffd700 1 -- 172.21.15.32:0/2344367111 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe96005aba0 con 0x7fe9700fe830 2024-04-06T14:23:54.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.179+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe97010f0d0 con 0x7fe9700fe830 2024-04-06T14:23:54.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.179+0000 7fe966ffd700 1 -- 172.21.15.32:0/2344367111 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+290 (secure 0 0 0) 0x7fe96005a730 con 0x7fe9700fe830 2024-04-06T14:23:54.180 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down 2024-04-06T14:23:54.180 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:23:54.180 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:23:54.180 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T14:23:54.180 INFO:teuthology.orchestra.run.smithi032.stdout: osd.2 (root=default,host=smithi032) is down 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe95c06cc20 msgr2=0x7fe95c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe95c06cc20 0x7fe95c06f0d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009b80 tx=0x7fe96c00b040 comp rx=0 tx=0).stop 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 msgr2=0x7fe970113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 0x7fe970113fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe960000c00 tx=0x7fe960009e30 comp rx=0 tx=0).stop 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 shutdown_connections 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe95c06cc20 0x7fe95c06f0d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9700fde90 0x7fe970113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 --2- 172.21.15.32:0/2344367111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9700fe830 0x7fe970113fc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 >> 172.21.15.32:0/2344367111 conn(0x7fe970072bc0 msgr2=0x7fe9700ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 shutdown_connections 2024-04-06T14:23:54.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:23:54.181+0000 7fe97837a700 1 -- 172.21.15.32:0/2344367111 wait complete. 2024-04-06T14:23:54.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:53 smithi032 ceph-mon[147414]: pgmap v89: 65 pgs: 9 stale+active+clean, 56 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:54.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:53 smithi032 ceph-mon[147414]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:54.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:53 smithi032 ceph-mon[147414]: osdmap e62: 6 total, 5 up, 6 in 2024-04-06T14:23:54.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:53 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1026679786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:54.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:53 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3742121872' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:23:54.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:54 smithi086 ceph-mon[110975]: pgmap v89: 65 pgs: 9 stale+active+clean, 56 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:23:54.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:54 smithi086 ceph-mon[110975]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:54.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:54 smithi086 ceph-mon[110975]: osdmap e62: 6 total, 5 up, 6 in 2024-04-06T14:23:54.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:54 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1026679786' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:23:54.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:54 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3742121872' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:23:55.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:55 smithi086 ceph-mon[110975]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:55.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:55 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2344367111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:23:55.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:55 smithi032 ceph-mon[147414]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:23:55.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:55 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2344367111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:23:55.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:55 smithi032 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.2.service: Succeeded. 2024-04-06T14:23:55.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:55 smithi032 systemd[1]: Stopped Ceph osd.2 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:23:55.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:55 smithi032 systemd[1]: Starting Ceph osd.2 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:23:56.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:56 smithi086 ceph-mon[110975]: pgmap v91: 65 pgs: 10 active+undersized+degraded, 4 stale+active+clean, 10 active+undersized, 41 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 34/393 objects degraded (8.651%) 2024-04-06T14:23:56.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:56 smithi086 ceph-mon[110975]: Health check failed: Degraded data redundancy: 34/393 objects degraded (8.651%), 10 pgs degraded (PG_DEGRADED) 2024-04-06T14:23:56.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:56 smithi032 ceph-mon[147414]: pgmap v91: 65 pgs: 10 active+undersized+degraded, 4 stale+active+clean, 10 active+undersized, 41 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 34/393 objects degraded (8.651%) 2024-04-06T14:23:56.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:56 smithi032 ceph-mon[147414]: Health check failed: Degraded data redundancy: 34/393 objects degraded (8.651%), 10 pgs degraded (PG_DEGRADED) 2024-04-06T14:23:56.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:56 smithi032 bash[175752]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T14:23:56.481 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:56 smithi032 bash[175752]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T14:23:56.481 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:56 smithi032 bash[175752]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T14:23:56.481 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:56 smithi032 bash[175752]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T14:23:56.481 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:56 smithi032 bash[175752]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-06T14:23:56.481 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:56 smithi032 bash[175752]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T14:23:56.481 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:56 smithi032 bash[175752]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-06T14:23:57.730 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:57 smithi032 ceph-osd[176078]: -- 172.21.15.32:0/3825341248 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ca8076b1e0 con 0x55ca8073b000 2024-04-06T14:23:57.730 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:57 smithi032 bash[176039]: beb745b5f249531cd73146460177db282e58a9b8a55862075bfde6442f61fd9a 2024-04-06T14:23:57.730 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:57 smithi032 systemd[1]: Started Ceph osd.2 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:23:58.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:58 smithi086 ceph-mon[110975]: pgmap v92: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:23:58.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:58 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:58.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:58 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:58.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:23:58 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:23:58.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:23:57 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[176055]: 2024-04-06T14:23:57.984+0000 7fab7291e640 -1 Falling back to public interface 2024-04-06T14:23:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:58 smithi032 ceph-mon[147414]: pgmap v92: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:23:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:58 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:58 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:23:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:23:58 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:24:00.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:00 smithi086 ceph-mon[110975]: pgmap v93: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:24:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:00 smithi032 ceph-mon[147414]: pgmap v93: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:24:02.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:02 smithi086 ceph-mon[110975]: pgmap v94: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:24:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:02 smithi032 ceph-mon[147414]: pgmap v94: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:24:02.480 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:24:02 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[176055]: 2024-04-06T14:24:02.120+0000 7fab7291e640 -1 osd.2 60 log_to_monitors true 2024-04-06T14:24:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:03 smithi086 ceph-mon[110975]: from='osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T14:24:03.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:03 smithi032 ceph-mon[147414]: from='osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T14:24:04.230 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 14:24:03 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[176055]: 2024-04-06T14:24:03.849+0000 7fab5fd6f700 -1 osd.2 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:24:04.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: pgmap v95: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:24:04.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: from='osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T14:24:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: osdmap e63: 6 total, 5 up, 6 in 2024-04-06T14:24:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: from='osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:24:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: from='osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204]' entity='osd.2' 2024-04-06T14:24:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: Health check update: Degraded data redundancy: 48/393 objects degraded (12.214%), 13 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:04.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:04 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:04.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: pgmap v95: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:24:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: from='osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T14:24:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: osdmap e63: 6 total, 5 up, 6 in 2024-04-06T14:24:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: from='osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-06T14:24:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: from='osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204]' entity='osd.2' 2024-04-06T14:24:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: Health check update: Degraded data redundancy: 48/393 objects degraded (12.214%), 13 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:04 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:05.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:05 smithi086 ceph-mon[110975]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:24:05.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:05 smithi086 ceph-mon[110975]: osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204] boot 2024-04-06T14:24:05.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:05 smithi086 ceph-mon[110975]: osdmap e64: 6 total, 6 up, 6 in 2024-04-06T14:24:05.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:24:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:05 smithi032 ceph-mon[147414]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:24:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:05 smithi032 ceph-mon[147414]: osd.2 [v2:172.21.15.32:6818/1957975204,v1:172.21.15.32:6819/1957975204] boot 2024-04-06T14:24:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:05 smithi032 ceph-mon[147414]: osdmap e64: 6 total, 6 up, 6 in 2024-04-06T14:24:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T14:24:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:06 smithi032 ceph-mon[147414]: pgmap v97: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:24:06.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:06 smithi032 ceph-mon[147414]: osdmap e65: 6 total, 6 up, 6 in 2024-04-06T14:24:06.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:06 smithi086 ceph-mon[110975]: pgmap v97: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 48/393 objects degraded (12.214%) 2024-04-06T14:24:06.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:06 smithi086 ceph-mon[110975]: osdmap e65: 6 total, 6 up, 6 in 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: pgmap v100: 65 pgs: 12 active+undersized+degraded, 12 active+undersized, 41 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 45/393 objects degraded (11.450%) 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:08.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T14:24:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: pgmap v100: 65 pgs: 12 active+undersized+degraded, 12 active+undersized, 41 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 45/393 objects degraded (11.450%) 2024-04-06T14:24:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:24:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:24:08.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:24:08.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:08.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:24:08.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:08.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:08.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:08.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T14:24:09.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:09 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T14:24:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:09 smithi086 ceph-mon[110975]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-04-06T14:24:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:09 smithi086 ceph-mon[110975]: Health check update: Degraded data redundancy: 45/393 objects degraded (11.450%), 12 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:09 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T14:24:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:09 smithi032 ceph-mon[147414]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-04-06T14:24:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:09 smithi032 ceph-mon[147414]: Health check update: Degraded data redundancy: 45/393 objects degraded (11.450%), 12 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:10.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:10 smithi086 ceph-mon[110975]: pgmap v101: 65 pgs: 2 active+undersized+degraded, 3 active+undersized, 60 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 10/393 objects degraded (2.545%) 2024-04-06T14:24:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:10 smithi032 ceph-mon[147414]: pgmap v101: 65 pgs: 2 active+undersized+degraded, 3 active+undersized, 60 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 10/393 objects degraded (2.545%) 2024-04-06T14:24:11.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:11 smithi086 ceph-mon[110975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/393 objects degraded (2.545%), 2 pgs degraded) 2024-04-06T14:24:11.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:11 smithi032 ceph-mon[147414]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/393 objects degraded (2.545%), 2 pgs degraded) 2024-04-06T14:24:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:12 smithi086 ceph-mon[110975]: pgmap v102: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:12.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:12 smithi032 ceph-mon[147414]: pgmap v102: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:14.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:14 smithi086 ceph-mon[110975]: pgmap v103: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:14 smithi032 ceph-mon[147414]: pgmap v103: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:16.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:16 smithi086 ceph-mon[110975]: pgmap v104: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:16 smithi032 ceph-mon[147414]: pgmap v104: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:18.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:18 smithi086 ceph-mon[110975]: pgmap v105: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:18 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:18.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:18 smithi032 ceph-mon[147414]: pgmap v105: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:18.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:18 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:20.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:20 smithi086 ceph-mon[110975]: pgmap v106: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:20 smithi032 ceph-mon[147414]: pgmap v106: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:22.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:22 smithi086 ceph-mon[110975]: pgmap v107: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:22.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:22 smithi032 ceph-mon[147414]: pgmap v107: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:23.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:23 smithi086 ceph-mon[110975]: pgmap v108: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:23.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:23.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:24:23.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T14:24:23.722 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:23 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T14:24:23.722 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:23 smithi086 ceph-mon[110975]: Upgrade: osd.3 is safe to restart 2024-04-06T14:24:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:23 smithi032 ceph-mon[147414]: pgmap v108: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:24:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T14:24:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:23 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T14:24:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:23 smithi032 ceph-mon[147414]: Upgrade: osd.3 is safe to restart 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.301+0000 7fb9b2f22700 1 -- 172.21.15.32:0/1870592049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 msgr2=0x7fb9ac0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.301+0000 7fb9b2f22700 1 --2- 172.21.15.32:0/1870592049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 0x7fb9ac0fbd80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb998004940 tx=0x7fb998009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.301+0000 7fb9b2f22700 1 -- 172.21.15.32:0/1870592049 shutdown_connections 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.301+0000 7fb9b2f22700 1 --2- 172.21.15.32:0/1870592049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 0x7fb9ac0fbd80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.301+0000 7fb9b2f22700 1 --2- 172.21.15.32:0/1870592049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ac101d70 0x7fb9ac102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.301+0000 7fb9b2f22700 1 -- 172.21.15.32:0/1870592049 >> 172.21.15.32:0/1870592049 conn(0x7fb9ac072bc0 msgr2=0x7fb9ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.301+0000 7fb9b2f22700 1 -- 172.21.15.32:0/1870592049 shutdown_connections 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.301+0000 7fb9b2f22700 1 -- 172.21.15.32:0/1870592049 wait complete. 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.302+0000 7fb9b2f22700 1 Processor -- start 2024-04-06T14:24:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.302+0000 7fb9b2f22700 1 -- start start 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.302+0000 7fb9b2f22700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ac101d70 0x7fb9ac112a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.302+0000 7fb9b2f22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 0x7fb9ac112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.302+0000 7fb9b2f22700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac1135b0 con 0x7fb9ac102710 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.302+0000 7fb9b2f22700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac10c9a0 con 0x7fb9ac101d70 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9abfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 0x7fb9ac112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9abfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 0x7fb9ac112f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52222/0 (socket says 172.21.15.32:52222) 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9abfff700 1 -- 172.21.15.32:0/647762399 learned_addr learned my addr 172.21.15.32:0/647762399 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9b0cbe700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ac101d70 0x7fb9ac112a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9abfff700 1 -- 172.21.15.32:0/647762399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ac101d70 msgr2=0x7fb9ac112a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9abfff700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ac101d70 0x7fb9ac112a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9abfff700 1 -- 172.21.15.32:0/647762399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb998005040 con 0x7fb9ac102710 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9abfff700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 0x7fb9ac112f60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb99800b3b0 tx=0x7fb998009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9a97fa700 1 -- 172.21.15.32:0/647762399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb99800e040 con 0x7fb9ac102710 2024-04-06T14:24:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9a97fa700 1 -- 172.21.15.32:0/647762399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb99800bc90 con 0x7fb9ac102710 2024-04-06T14:24:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9a97fa700 1 -- 172.21.15.32:0/647762399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb998013660 con 0x7fb9ac102710 2024-04-06T14:24:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ac10cc20 con 0x7fb9ac102710 2024-04-06T14:24:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.303+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ac10d180 con 0x7fb9ac102710 2024-04-06T14:24:24.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.304+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ac0fd4c0 con 0x7fb9ac102710 2024-04-06T14:24:24.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.305+0000 7fb9a97fa700 1 -- 172.21.15.32:0/647762399 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb9980138d0 con 0x7fb9ac102710 2024-04-06T14:24:24.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.305+0000 7fb9a97fa700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:24.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.305+0000 7fb9a97fa700 1 -- 172.21.15.32:0/647762399 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb99808fdc0 con 0x7fb9ac102710 2024-04-06T14:24:24.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.305+0000 7fb9b0cbe700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:24.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.306+0000 7fb9b0cbe700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0004f60 tx=0x7fb9a000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:24.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.310+0000 7fb9a97fa700 1 -- 172.21.15.32:0/647762399 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb99805d930 con 0x7fb9ac102710 2024-04-06T14:24:24.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.534+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9ac1126f0 con 0x7fb99c06cc20 2024-04-06T14:24:24.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.535+0000 7fb9a97fa700 1 -- 172.21.15.32:0/647762399 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fb9ac1126f0 con 0x7fb99c06cc20 2024-04-06T14:24:24.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 msgr2=0x7fb99c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0004f60 tx=0x7fb9a000b040 comp rx=0 tx=0).stop 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 msgr2=0x7fb9ac112f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 0x7fb9ac112f60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb99800b3b0 tx=0x7fb998009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 shutdown_connections 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ac101d70 0x7fb9ac112a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 --2- 172.21.15.32:0/647762399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ac102710 0x7fb9ac112f60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 >> 172.21.15.32:0/647762399 conn(0x7fb9ac072bc0 msgr2=0x7fb9ac0fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 shutdown_connections 2024-04-06T14:24:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.537+0000 7fb9b2f22700 1 -- 172.21.15.32:0/647762399 wait complete. 2024-04-06T14:24:24.548 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:24:24.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.642+0000 7fc26a3c8700 1 -- 172.21.15.32:0/1982936391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264102710 msgr2=0x7fc2640fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.642+0000 7fc26a3c8700 1 --2- 172.21.15.32:0/1982936391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264102710 0x7fc2640fbd80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc258004940 tx=0x7fc258009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:24.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.642+0000 7fc26a3c8700 1 -- 172.21.15.32:0/1982936391 shutdown_connections 2024-04-06T14:24:24.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.642+0000 7fc26a3c8700 1 --2- 172.21.15.32:0/1982936391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264102710 0x7fc2640fbd80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.642+0000 7fc26a3c8700 1 --2- 172.21.15.32:0/1982936391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc264101d70 0x7fc264102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.642+0000 7fc26a3c8700 1 -- 172.21.15.32:0/1982936391 >> 172.21.15.32:0/1982936391 conn(0x7fc264072bc0 msgr2=0x7fc264072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:24.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.642+0000 7fc26a3c8700 1 -- 172.21.15.32:0/1982936391 shutdown_connections 2024-04-06T14:24:24.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.642+0000 7fc26a3c8700 1 -- 172.21.15.32:0/1982936391 wait complete. 2024-04-06T14:24:24.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.643+0000 7fc26a3c8700 1 Processor -- start 2024-04-06T14:24:24.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.643+0000 7fc26a3c8700 1 -- start start 2024-04-06T14:24:24.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.643+0000 7fc26a3c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264101d70 0x7fc264112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:24.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.643+0000 7fc26a3c8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc264102710 0x7fc264112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:24.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.643+0000 7fc26a3c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc264113490 con 0x7fc264101d70 2024-04-06T14:24:24.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.643+0000 7fc26a3c8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26410c880 con 0x7fc264102710 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.643+0000 7fc263fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264101d70 0x7fc264112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.643+0000 7fc263fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264101d70 0x7fc264112900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58924/0 (socket says 172.21.15.32:58924) 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.643+0000 7fc263fff700 1 -- 172.21.15.32:0/2367817077 learned_addr learned my addr 172.21.15.32:0/2367817077 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc2637fe700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc264102710 0x7fc264112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc263fff700 1 -- 172.21.15.32:0/2367817077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc264102710 msgr2=0x7fc264112e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc263fff700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc264102710 0x7fc264112e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc263fff700 1 -- 172.21.15.32:0/2367817077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc254005040 con 0x7fc264101d70 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc263fff700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264101d70 0x7fc264112900 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2540096d0 tx=0x7fc254009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc260ff9700 1 -- 172.21.15.32:0/2367817077 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25400e840 con 0x7fc264101d70 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc260ff9700 1 -- 172.21.15.32:0/2367817077 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc25400f040 con 0x7fc264101d70 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc258005040 con 0x7fc264101d70 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc260ff9700 1 -- 172.21.15.32:0/2367817077 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc254013d30 con 0x7fc264101d70 2024-04-06T14:24:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.644+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26410cf00 con 0x7fc264101d70 2024-04-06T14:24:24.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.645+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2640ff080 con 0x7fc264101d70 2024-04-06T14:24:24.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.646+0000 7fc260ff9700 1 -- 172.21.15.32:0/2367817077 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc25400e9a0 con 0x7fc264101d70 2024-04-06T14:24:24.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.646+0000 7fc260ff9700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc24c06cc20 0x7fc24c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:24.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.646+0000 7fc260ff9700 1 -- 172.21.15.32:0/2367817077 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc25408bcc0 con 0x7fc264101d70 2024-04-06T14:24:24.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.646+0000 7fc2637fe700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc24c06cc20 0x7fc24c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:24.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.646+0000 7fc2637fe700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc24c06cc20 0x7fc24c06f0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc26410db60 tx=0x7fc258009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:24.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.650+0000 7fc260ff9700 1 -- 172.21.15.32:0/2367817077 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc254059830 con 0x7fc264101d70 2024-04-06T14:24:24.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.870+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc26410dfc0 con 0x7fc24c06cc20 2024-04-06T14:24:24.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.872+0000 7fc260ff9700 1 -- 172.21.15.32:0/2367817077 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fc26410dfc0 con 0x7fc24c06cc20 2024-04-06T14:24:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc24c06cc20 msgr2=0x7fc24c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc24c06cc20 0x7fc24c06f0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc26410db60 tx=0x7fc258009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264101d70 msgr2=0x7fc264112900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264101d70 0x7fc264112900 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2540096d0 tx=0x7fc254009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 shutdown_connections 2024-04-06T14:24:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc24c06cc20 0x7fc24c06f0d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc264101d70 0x7fc264112900 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 --2- 172.21.15.32:0/2367817077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc264102710 0x7fc264112e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 >> 172.21.15.32:0/2367817077 conn(0x7fc264072bc0 msgr2=0x7fc2640faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:24.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 shutdown_connections 2024-04-06T14:24:24.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.874+0000 7fc26a3c8700 1 -- 172.21.15.32:0/2367817077 wait complete. 2024-04-06T14:24:24.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.977+0000 7f81fcfa9700 1 -- 172.21.15.32:0/3542404460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f81026b0 msgr2=0x7f81f80fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.977+0000 7f81fcfa9700 1 --2- 172.21.15.32:0/3542404460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f81026b0 0x7f81f80fbd20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f81ec004940 tx=0x7f81ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:24.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.977+0000 7f81fcfa9700 1 -- 172.21.15.32:0/3542404460 shutdown_connections 2024-04-06T14:24:24.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.977+0000 7f81fcfa9700 1 --2- 172.21.15.32:0/3542404460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f81026b0 0x7f81f80fbd20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.977+0000 7f81fcfa9700 1 --2- 172.21.15.32:0/3542404460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81f8101d10 0x7f81f81020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.977+0000 7f81fcfa9700 1 -- 172.21.15.32:0/3542404460 >> 172.21.15.32:0/3542404460 conn(0x7f81f80f7880 msgr2=0x7f81f80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:24.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.977+0000 7f81fcfa9700 1 -- 172.21.15.32:0/3542404460 shutdown_connections 2024-04-06T14:24:24.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.977+0000 7f81fcfa9700 1 -- 172.21.15.32:0/3542404460 wait complete. 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.978+0000 7f81fcfa9700 1 Processor -- start 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.978+0000 7f81fcfa9700 1 -- start start 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.978+0000 7f81fcfa9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f8101d10 0x7f81f8112d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.978+0000 7f81fcfa9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81f81026b0 0x7f81f8113240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.978+0000 7f81fcfa9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f8113890 con 0x7f81f8101d10 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.978+0000 7f81fcfa9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f810cc60 con 0x7f81f81026b0 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.978+0000 7f81f659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f8101d10 0x7f81f8112d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81f659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f8101d10 0x7f81f8112d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58938/0 (socket says 172.21.15.32:58938) 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81f659c700 1 -- 172.21.15.32:0/756870728 learned_addr learned my addr 172.21.15.32:0/756870728 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:24.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81f5d9b700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81f81026b0 0x7f81f8113240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:24.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81f659c700 1 -- 172.21.15.32:0/756870728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81f81026b0 msgr2=0x7f81f8113240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:24.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81f659c700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81f81026b0 0x7f81f8113240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:24.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81f659c700 1 -- 172.21.15.32:0/756870728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec005040 con 0x7f81f8101d10 2024-04-06T14:24:24.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81f659c700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f8101d10 0x7f81f8112d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f81e8002a50 tx=0x7f81e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:24.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81e6ffd700 1 -- 172.21.15.32:0/756870728 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e8007cc0 con 0x7f81f8101d10 2024-04-06T14:24:24.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81e6ffd700 1 -- 172.21.15.32:0/756870728 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81e800f040 con 0x7f81f8101d10 2024-04-06T14:24:24.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f810cf40 con 0x7f81f8101d10 2024-04-06T14:24:24.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81e6ffd700 1 -- 172.21.15.32:0/756870728 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e800d9c0 con 0x7f81f8101d10 2024-04-06T14:24:24.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.979+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f810d500 con 0x7f81f8101d10 2024-04-06T14:24:24.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.980+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f80fd460 con 0x7f81f8101d10 2024-04-06T14:24:24.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.981+0000 7f81e6ffd700 1 -- 172.21.15.32:0/756870728 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f81e800db20 con 0x7f81f8101d10 2024-04-06T14:24:24.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.981+0000 7f81e6ffd700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81e006cc20 0x7f81e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:24.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.981+0000 7f81e6ffd700 1 -- 172.21.15.32:0/756870728 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f81e808bcd0 con 0x7f81f8101d10 2024-04-06T14:24:24.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.981+0000 7f81f5d9b700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81e006cc20 0x7f81e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:24.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.981+0000 7f81f5d9b700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81e006cc20 0x7f81e006f0d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f81f810e5a0 tx=0x7f81ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:24.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:24.986+0000 7f81e6ffd700 1 -- 172.21.15.32:0/756870728 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f81e80597c0 con 0x7f81f8101d10 2024-04-06T14:24:25.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.198+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f81f804bf40 con 0x7f81e006cc20 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.208+0000 7f81e6ffd700 1 -- 172.21.15.32:0/756870728 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f81f804bf40 con 0x7f81e006cc20 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (4m) 22s ago 16m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (2m) 22s ago 16m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (2m) 119s ago 15m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (4m) 22s ago 15m 87.2M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (10m) 22s ago 10m 14.5M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (10m) 22s ago 10m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (10m) 119s ago 10m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:24:25.208 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (10m) 119s ago 10m 16.5M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (6m) 22s ago 17m 503M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (5m) 119s ago 15m 448M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (3m) 22s ago 17m 47.1M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (2m) 119s ago 15m 32.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (5m) 22s ago 16m 16.6M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (5m) 119s ago 15m 17.0M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (101s) 22s ago 14m 78.7M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (65s) 22s ago 13m 80.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (28s) 22s ago 13m 11.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (12m) 119s ago 12m 226M 2592M 17.2.7-819-g6699495c ec300eaa768b d22f860aec04 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (12m) 119s ago 12m 203M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (11m) 119s ago 11m 187M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:24:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (5m) 22s ago 15m 59.8M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:24:25.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81e006cc20 msgr2=0x7f81e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81e006cc20 0x7f81e006f0d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f81f810e5a0 tx=0x7f81ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:25.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f8101d10 msgr2=0x7f81f8112d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f8101d10 0x7f81f8112d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f81e8002a50 tx=0x7f81e80046b0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 shutdown_connections 2024-04-06T14:24:25.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81e006cc20 0x7f81e006f0d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81f8101d10 0x7f81f8112d00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 --2- 172.21.15.32:0/756870728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81f81026b0 0x7f81f8113240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 >> 172.21.15.32:0/756870728 conn(0x7f81f80f7880 msgr2=0x7f81f80f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:25.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 shutdown_connections 2024-04-06T14:24:25.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.210+0000 7f81fcfa9700 1 -- 172.21.15.32:0/756870728 wait complete. 2024-04-06T14:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.312+0000 7f283b34b700 1 -- 172.21.15.32:0/1446403451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 msgr2=0x7f2834102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.312+0000 7f283b34b700 1 --2- 172.21.15.32:0/1446403451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 0x7f2834102140 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2824004940 tx=0x7f2824009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.312+0000 7f283b34b700 1 -- 172.21.15.32:0/1446403451 shutdown_connections 2024-04-06T14:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.312+0000 7f283b34b700 1 --2- 172.21.15.32:0/1446403451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2834102710 0x7f28340fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.312+0000 7f283b34b700 1 --2- 172.21.15.32:0/1446403451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 0x7f2834102140 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.312+0000 7f283b34b700 1 -- 172.21.15.32:0/1446403451 >> 172.21.15.32:0/1446403451 conn(0x7f2834072bc0 msgr2=0x7f2834072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.312+0000 7f283b34b700 1 -- 172.21.15.32:0/1446403451 shutdown_connections 2024-04-06T14:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.312+0000 7f283b34b700 1 -- 172.21.15.32:0/1446403451 wait complete. 2024-04-06T14:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.313+0000 7f283b34b700 1 Processor -- start 2024-04-06T14:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.313+0000 7f283b34b700 1 -- start start 2024-04-06T14:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.313+0000 7f283b34b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 0x7f2834112920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f283b34b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2834102710 0x7f2834112e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f283b34b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28341134b0 con 0x7f2834101d70 2024-04-06T14:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f283b34b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283410c8a0 con 0x7f2834102710 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f28390e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 0x7f2834112920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f28390e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 0x7f2834112920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58962/0 (socket says 172.21.15.32:58962) 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f28390e7700 1 -- 172.21.15.32:0/2374370259 learned_addr learned my addr 172.21.15.32:0/2374370259 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f28388e6700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2834102710 0x7f2834112e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f28390e7700 1 -- 172.21.15.32:0/2374370259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2834102710 msgr2=0x7f2834112e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f28390e7700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2834102710 0x7f2834112e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f28390e7700 1 -- 172.21.15.32:0/2374370259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2824005040 con 0x7f2834101d70 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f28388e6700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2834102710 0x7f2834112e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f28390e7700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 0x7f2834112920 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2824000c00 tx=0x7f2824009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f2829ffb700 1 -- 172.21.15.32:0/2374370259 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2824021070 con 0x7f2834101d70 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f2829ffb700 1 -- 172.21.15.32:0/2374370259 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28240093b0 con 0x7f2834101d70 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283410cb20 con 0x7f2834101d70 2024-04-06T14:24:25.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.314+0000 7f2829ffb700 1 -- 172.21.15.32:0/2374370259 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f282400e070 con 0x7f2834101d70 2024-04-06T14:24:25.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.315+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283410d0e0 con 0x7f2834101d70 2024-04-06T14:24:25.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.316+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28340ff080 con 0x7f2834101d70 2024-04-06T14:24:25.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.316+0000 7f2829ffb700 1 -- 172.21.15.32:0/2374370259 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f282401c490 con 0x7f2834101d70 2024-04-06T14:24:25.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.316+0000 7f2829ffb700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282006cc20 0x7f282006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:25.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.317+0000 7f2829ffb700 1 -- 172.21.15.32:0/2374370259 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f282408d110 con 0x7f2834101d70 2024-04-06T14:24:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.317+0000 7f28388e6700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282006cc20 0x7f282006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.317+0000 7f28388e6700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282006cc20 0x7f282006f0d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f283410da60 tx=0x7f283000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:25.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.321+0000 7f2829ffb700 1 -- 172.21.15.32:0/2374370259 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f282405abe0 con 0x7f2834101d70 2024-04-06T14:24:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.610+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f283410df70 con 0x7f2834101d70 2024-04-06T14:24:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.610+0000 7f2829ffb700 1 -- 172.21.15.32:0/2374370259 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f282405a770 con 0x7f2834101d70 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 3, 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:24:25.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:24:25.612 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:24:25.612 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:24:25.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:24:25.612 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:24:25.612 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 7, 2024-04-06T14:24:25.612 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 7 2024-04-06T14:24:25.612 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:24:25.612 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282006cc20 msgr2=0x7f282006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282006cc20 0x7f282006f0d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f283410da60 tx=0x7f283000b040 comp rx=0 tx=0).stop 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 msgr2=0x7f2834112920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 0x7f2834112920 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2824000c00 tx=0x7f2824009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 shutdown_connections 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282006cc20 0x7f282006f0d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2834101d70 0x7f2834112920 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 --2- 172.21.15.32:0/2374370259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2834102710 0x7f2834112e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 >> 172.21.15.32:0/2374370259 conn(0x7f2834072bc0 msgr2=0x7f28340faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 shutdown_connections 2024-04-06T14:24:25.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.612+0000 7f283b34b700 1 -- 172.21.15.32:0/2374370259 wait complete. 2024-04-06T14:24:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.716+0000 7f27e1660700 1 -- 172.21.15.32:0/604435619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc1037a0 msgr2=0x7f27dc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.716+0000 7f27e1660700 1 --2- 172.21.15.32:0/604435619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc1037a0 0x7f27dc0fce10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f27d0004940 tx=0x7f27d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.716+0000 7f27e1660700 1 -- 172.21.15.32:0/604435619 shutdown_connections 2024-04-06T14:24:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.717+0000 7f27e1660700 1 --2- 172.21.15.32:0/604435619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc1037a0 0x7f27dc0fce10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.717+0000 7f27e1660700 1 --2- 172.21.15.32:0/604435619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27dc102e00 0x7f27dc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.717+0000 7f27e1660700 1 -- 172.21.15.32:0/604435619 >> 172.21.15.32:0/604435619 conn(0x7f27dc0f89b0 msgr2=0x7f27dc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.717+0000 7f27e1660700 1 -- 172.21.15.32:0/604435619 shutdown_connections 2024-04-06T14:24:25.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.717+0000 7f27e1660700 1 -- 172.21.15.32:0/604435619 wait complete. 2024-04-06T14:24:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.717+0000 7f27e1660700 1 Processor -- start 2024-04-06T14:24:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27e1660700 1 -- start start 2024-04-06T14:24:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27e1660700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc102e00 0x7f27dc1138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27e1660700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27dc1037a0 0x7f27dc113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27e1660700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc114430 con 0x7f27dc102e00 2024-04-06T14:24:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27e1660700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc10da30 con 0x7f27dc1037a0 2024-04-06T14:24:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27daffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc102e00 0x7f27dc1138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27daffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc102e00 0x7f27dc1138a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58988/0 (socket says 172.21.15.32:58988) 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27daffd700 1 -- 172.21.15.32:0/4048101226 learned_addr learned my addr 172.21.15.32:0/4048101226 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27da7fc700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27dc1037a0 0x7f27dc113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27daffd700 1 -- 172.21.15.32:0/4048101226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27dc1037a0 msgr2=0x7f27dc113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27daffd700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27dc1037a0 0x7f27dc113de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27daffd700 1 -- 172.21.15.32:0/4048101226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d0005040 con 0x7f27dc102e00 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27da7fc700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27dc1037a0 0x7f27dc113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.718+0000 7f27daffd700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc102e00 0x7f27dc1138a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00c830 tx=0x7f27cc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.719+0000 7f27c37fe700 1 -- 172.21.15.32:0/4048101226 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27cc00a760 con 0x7f27dc102e00 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.719+0000 7f27c37fe700 1 -- 172.21.15.32:0/4048101226 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27cc00ad70 con 0x7f27dc102e00 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.719+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27dc10dd10 con 0x7f27dc102e00 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.719+0000 7f27c37fe700 1 -- 172.21.15.32:0/4048101226 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27cc0113a0 con 0x7f27dc102e00 2024-04-06T14:24:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.719+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27dc10e2d0 con 0x7f27dc102e00 2024-04-06T14:24:25.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.720+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27dc0fe550 con 0x7f27dc102e00 2024-04-06T14:24:25.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.720+0000 7f27c37fe700 1 -- 172.21.15.32:0/4048101226 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f27cc012400 con 0x7f27dc102e00 2024-04-06T14:24:25.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.721+0000 7f27c37fe700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27c406cc20 0x7f27c406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:25.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.721+0000 7f27da7fc700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27c406cc20 0x7f27c406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:25.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.721+0000 7f27c37fe700 1 -- 172.21.15.32:0/4048101226 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f27cc08bf70 con 0x7f27dc102e00 2024-04-06T14:24:25.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.721+0000 7f27da7fc700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27c406cc20 0x7f27c406f0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f27dc10ebf0 tx=0x7f27d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:25.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.725+0000 7f27c37fe700 1 -- 172.21.15.32:0/4048101226 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f27cc0bfab0 con 0x7f27dc102e00 2024-04-06T14:24:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:25 smithi086 ceph-mon[110975]: Upgrade: Updating osd.3 2024-04-06T14:24:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:25 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:25 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T14:24:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:25 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:24:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:25 smithi086 ceph-mon[110975]: Deploying daemon osd.3 on smithi086 2024-04-06T14:24:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:25 smithi086 ceph-mon[110975]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:25 smithi086 ceph-mon[110975]: pgmap v109: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:25.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:25 smithi086 ceph-mon[110975]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:25.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:25 smithi086 ceph-mon[110975]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.972+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f27dc109da0 con 0x7f27dc102e00 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.972+0000 7f27c37fe700 1 -- 172.21.15.32:0/4048101226 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1851 (secure 0 0 0) 0x7f27cc059b80 con 0x7f27dc102e00 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:24:25.973 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:21:28.458361+0000 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:24:25.974 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:24:25.975 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:24:25.976 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:24:25.976 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:24:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27c406cc20 msgr2=0x7f27c406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27c406cc20 0x7f27c406f0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f27dc10ebf0 tx=0x7f27d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc102e00 msgr2=0x7f27dc1138a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc102e00 0x7f27dc1138a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00c830 tx=0x7f27cc004470 comp rx=0 tx=0).stop 2024-04-06T14:24:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 shutdown_connections 2024-04-06T14:24:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27c406cc20 0x7f27c406f0d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27dc102e00 0x7f27dc1138a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 --2- 172.21.15.32:0/4048101226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27dc1037a0 0x7f27dc113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 >> 172.21.15.32:0/4048101226 conn(0x7f27dc0f89b0 msgr2=0x7f27dc0f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 shutdown_connections 2024-04-06T14:24:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:25.975+0000 7f27e1660700 1 -- 172.21.15.32:0/4048101226 wait complete. 2024-04-06T14:24:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-06T14:24:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:25 smithi032 ceph-mon[147414]: Upgrade: Updating osd.3 2024-04-06T14:24:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:25 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:25 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T14:24:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:25 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:24:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:25 smithi032 ceph-mon[147414]: Deploying daemon osd.3 on smithi086 2024-04-06T14:24:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:25 smithi032 ceph-mon[147414]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:25 smithi032 ceph-mon[147414]: pgmap v109: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:25 smithi032 ceph-mon[147414]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:25 smithi032 ceph-mon[147414]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.079+0000 7fe4b4d22700 1 -- 172.21.15.32:0/1803391340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0101d70 msgr2=0x7fe4b0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.079+0000 7fe4b4d22700 1 --2- 172.21.15.32:0/1803391340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0101d70 0x7fe4b0102140 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0004940 tx=0x7fe4a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 -- 172.21.15.32:0/1803391340 shutdown_connections 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 --2- 172.21.15.32:0/1803391340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4b0102710 0x7fe4b00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 --2- 172.21.15.32:0/1803391340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0101d70 0x7fe4b0102140 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 -- 172.21.15.32:0/1803391340 >> 172.21.15.32:0/1803391340 conn(0x7fe4b0072bc0 msgr2=0x7fe4b0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 -- 172.21.15.32:0/1803391340 shutdown_connections 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 -- 172.21.15.32:0/1803391340 wait complete. 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 Processor -- start 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 -- start start 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4b0101d70 0x7fe4b0102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0102710 0x7fe4b0103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b0103650 con 0x7fe4b0102710 2024-04-06T14:24:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.080+0000 7fe4b4d22700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b0103790 con 0x7fe4b0101d70 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4add9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0102710 0x7fe4b0103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4add9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0102710 0x7fe4b0103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59008/0 (socket says 172.21.15.32:59008) 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4add9b700 1 -- 172.21.15.32:0/2132928156 learned_addr learned my addr 172.21.15.32:0/2132928156 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4ae59c700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4b0101d70 0x7fe4b0102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4add9b700 1 -- 172.21.15.32:0/2132928156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4b0101d70 msgr2=0x7fe4b0102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4add9b700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4b0101d70 0x7fe4b0102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4add9b700 1 -- 172.21.15.32:0/2132928156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a0005040 con 0x7fe4b0102710 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4add9b700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0102710 0x7fe4b0103110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4002a50 tx=0x7fe4a4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe49effd700 1 -- 172.21.15.32:0/2132928156 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4a4007cc0 con 0x7fe4b0102710 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b0103a40 con 0x7fe4b0102710 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe49effd700 1 -- 172.21.15.32:0/2132928156 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4a400f040 con 0x7fe4b0102710 2024-04-06T14:24:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe49effd700 1 -- 172.21.15.32:0/2132928156 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4a400d920 con 0x7fe4b0102710 2024-04-06T14:24:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.081+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b011a210 con 0x7fe4b0102710 2024-04-06T14:24:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.082+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b00fd4c0 con 0x7fe4b0102710 2024-04-06T14:24:26.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.083+0000 7fe49effd700 1 -- 172.21.15.32:0/2132928156 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe4a4009680 con 0x7fe4b0102710 2024-04-06T14:24:26.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.083+0000 7fe49effd700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe49806cc30 0x7fe49806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:26.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.084+0000 7fe4ae59c700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe49806cc30 0x7fe49806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:26.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.084+0000 7fe49effd700 1 -- 172.21.15.32:0/2132928156 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fe4a408bc50 con 0x7fe4b0102710 2024-04-06T14:24:26.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.084+0000 7fe4ae59c700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe49806cc30 0x7fe49806f0e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009fd0 tx=0x7fe4a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:26.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.088+0000 7fe49effd700 1 -- 172.21.15.32:0/2132928156 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe4a4055c10 con 0x7fe4b0102710 2024-04-06T14:24:26.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.311+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4b0103a40 con 0x7fe49806cc30 2024-04-06T14:24:26.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.312+0000 7fe49effd700 1 -- 172.21.15.32:0/2132928156 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe4b0103a40 con 0x7fe49806cc30 2024-04-06T14:24:26.312 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "9/21 daemons upgraded", 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:24:26.313 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:24:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe49806cc30 msgr2=0x7fe49806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe49806cc30 0x7fe49806f0e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009fd0 tx=0x7fe4a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0102710 msgr2=0x7fe4b0103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0102710 0x7fe4b0103110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4002a50 tx=0x7fe4a4004600 comp rx=0 tx=0).stop 2024-04-06T14:24:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 shutdown_connections 2024-04-06T14:24:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe49806cc30 0x7fe49806f0e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4b0101d70 0x7fe4b0102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 --2- 172.21.15.32:0/2132928156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4b0102710 0x7fe4b0103110 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 >> 172.21.15.32:0/2132928156 conn(0x7fe4b0072bc0 msgr2=0x7fe4b0065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:26.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 shutdown_connections 2024-04-06T14:24:26.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.315+0000 7fe4b4d22700 1 -- 172.21.15.32:0/2132928156 wait complete. 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.418+0000 7f3f1293d700 1 -- 172.21.15.32:0/3016280597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c101d70 msgr2=0x7f3f0c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.418+0000 7f3f1293d700 1 --2- 172.21.15.32:0/3016280597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c101d70 0x7f3f0c102140 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3efc004940 tx=0x7f3efc009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.418+0000 7f3f1293d700 1 -- 172.21.15.32:0/3016280597 shutdown_connections 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.418+0000 7f3f1293d700 1 --2- 172.21.15.32:0/3016280597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c102710 0x7f3f0c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.418+0000 7f3f1293d700 1 --2- 172.21.15.32:0/3016280597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c101d70 0x7f3f0c102140 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.418+0000 7f3f1293d700 1 -- 172.21.15.32:0/3016280597 >> 172.21.15.32:0/3016280597 conn(0x7f3f0c072bc0 msgr2=0x7f3f0c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.418+0000 7f3f1293d700 1 -- 172.21.15.32:0/3016280597 shutdown_connections 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.418+0000 7f3f1293d700 1 -- 172.21.15.32:0/3016280597 wait complete. 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.418+0000 7f3f1293d700 1 Processor -- start 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f1293d700 1 -- start start 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f1293d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c101d70 0x7f3f0c102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f1293d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c102710 0x7f3f0c103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f1293d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c1036a0 con 0x7f3f0c102710 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f1293d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c1037e0 con 0x7f3f0c101d70 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f0b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c102710 0x7f3f0c103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f0b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c102710 0x7f3f0c103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59028/0 (socket says 172.21.15.32:59028) 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f0b7fe700 1 -- 172.21.15.32:0/128840882 learned_addr learned my addr 172.21.15.32:0/128840882 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f0bfff700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c101d70 0x7f3f0c102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f0b7fe700 1 -- 172.21.15.32:0/128840882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c101d70 msgr2=0x7f3f0c102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f0b7fe700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c101d70 0x7f3f0c102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f0b7fe700 1 -- 172.21.15.32:0/128840882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc005040 con 0x7f3f0c102710 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.419+0000 7f3f0b7fe700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c102710 0x7f3f0c103110 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000c800 tx=0x7f3f00004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.420+0000 7f3f08ff9700 1 -- 172.21.15.32:0/128840882 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0000a760 con 0x7f3f0c102710 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.420+0000 7f3f08ff9700 1 -- 172.21.15.32:0/128840882 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f0000ad70 con 0x7f3f0c102710 2024-04-06T14:24:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.420+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c119d60 con 0x7f3f0c102710 2024-04-06T14:24:26.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.420+0000 7f3f08ff9700 1 -- 172.21.15.32:0/128840882 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f00012e80 con 0x7f3f0c102710 2024-04-06T14:24:26.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.420+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c11a1e0 con 0x7f3f0c102710 2024-04-06T14:24:26.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.421+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c0fd4c0 con 0x7f3f0c102710 2024-04-06T14:24:26.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.421+0000 7f3f08ff9700 1 -- 172.21.15.32:0/128840882 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3f0000a8c0 con 0x7f3f0c102710 2024-04-06T14:24:26.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.421+0000 7f3f08ff9700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ef406cc30 0x7f3ef406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:26.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.422+0000 7f3f08ff9700 1 -- 172.21.15.32:0/128840882 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3f0008c020 con 0x7f3f0c102710 2024-04-06T14:24:26.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.422+0000 7f3f0bfff700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ef406cc30 0x7f3ef406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:26.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.422+0000 7f3f0bfff700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ef406cc30 0x7f3ef406f0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00b020 tx=0x7f3efc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:26.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.426+0000 7f3f08ff9700 1 -- 172.21.15.32:0/128840882 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3f00056100 con 0x7f3f0c102710 2024-04-06T14:24:26.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:26 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2374370259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:26.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:26 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4048101226' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:24:26.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:26 smithi086 ceph-mon[110975]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:26.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.718+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3f0c05f5f0 con 0x7f3f0c102710 2024-04-06T14:24:26.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.719+0000 7f3f08ff9700 1 -- 172.21.15.32:0/128840882 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f3f00059720 con 0x7f3f0c102710 2024-04-06T14:24:26.719 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:24:26.719 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:24:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:24:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ef406cc30 msgr2=0x7f3ef406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ef406cc30 0x7f3ef406f0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00b020 tx=0x7f3efc009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c102710 msgr2=0x7f3f0c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c102710 0x7f3f0c103110 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000c800 tx=0x7f3f00004470 comp rx=0 tx=0).stop 2024-04-06T14:24:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 shutdown_connections 2024-04-06T14:24:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ef406cc30 0x7f3ef406f0e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c101d70 0x7f3f0c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 --2- 172.21.15.32:0/128840882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f0c102710 0x7f3f0c103110 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 >> 172.21.15.32:0/128840882 conn(0x7f3f0c072bc0 msgr2=0x7f3f0c0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 shutdown_connections 2024-04-06T14:24:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:26.721+0000 7f3f1293d700 1 -- 172.21.15.32:0/128840882 wait complete. 2024-04-06T14:24:26.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:26 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2374370259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:26.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:26 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4048101226' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:24:26.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:26 smithi032 ceph-mon[147414]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:27.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:27 smithi086 ceph-mon[110975]: pgmap v110: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:27.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:27 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/128840882' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:24:27.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:27 smithi032 ceph-mon[147414]: pgmap v110: 65 pgs: 65 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:27.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:27 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/128840882' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:24:28.128 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:27 smithi086 systemd[1]: Stopping Ceph osd.3 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:24:28.128 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:27 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[54402]: 2024-04-06T14:24:27.895+0000 7f4697e88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T14:24:28.128 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:27 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[54402]: 2024-04-06T14:24:27.895+0000 7f4697e88700 -1 osd.3 65 *** Got signal Terminated *** 2024-04-06T14:24:28.128 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:27 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[54402]: 2024-04-06T14:24:27.895+0000 7f4697e88700 -1 osd.3 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T14:24:28.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:28 smithi086 ceph-mon[110975]: osd.3 marked itself down and dead 2024-04-06T14:24:28.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:28 smithi032 ceph-mon[147414]: osd.3 marked itself down and dead 2024-04-06T14:24:29.628 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:29 smithi086 bash[122628]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3 2024-04-06T14:24:29.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:29 smithi086 ceph-mon[110975]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:24:29.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:29 smithi086 ceph-mon[110975]: osdmap e66: 6 total, 5 up, 6 in 2024-04-06T14:24:29.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:29 smithi086 ceph-mon[110975]: pgmap v112: 65 pgs: 18 peering, 6 stale+active+clean, 41 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:29 smithi032 ceph-mon[147414]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:24:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:29 smithi032 ceph-mon[147414]: osdmap e66: 6 total, 5 up, 6 in 2024-04-06T14:24:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:29 smithi032 ceph-mon[147414]: pgmap v112: 65 pgs: 18 peering, 6 stale+active+clean, 41 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:30.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:30 smithi086 ceph-mon[110975]: Health check failed: Reduced data availability: 10 pgs peering (PG_AVAILABILITY) 2024-04-06T14:24:30.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:30 smithi086 ceph-mon[110975]: osdmap e67: 6 total, 5 up, 6 in 2024-04-06T14:24:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:30 smithi032 ceph-mon[147414]: Health check failed: Reduced data availability: 10 pgs peering (PG_AVAILABILITY) 2024-04-06T14:24:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:30 smithi032 ceph-mon[147414]: osdmap e67: 6 total, 5 up, 6 in 2024-04-06T14:24:31.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:31 smithi086 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.3.service: Succeeded. 2024-04-06T14:24:31.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:31 smithi086 systemd[1]: Stopped Ceph osd.3 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:24:31.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:31 smithi086 systemd[1]: Starting Ceph osd.3 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:24:31.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:31 smithi086 ceph-mon[110975]: pgmap v114: 65 pgs: 25 peering, 3 stale+active+clean, 37 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:31 smithi032 ceph-mon[147414]: pgmap v114: 65 pgs: 25 peering, 3 stale+active+clean, 37 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-06T14:24:32.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:32 smithi086 bash[122902]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T14:24:32.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:32 smithi086 bash[122902]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T14:24:32.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:32 smithi086 bash[122902]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T14:24:32.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:32 smithi086 bash[122902]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T14:24:32.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:32 smithi086 bash[122902]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-06T14:24:32.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:32 smithi086 bash[122902]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T14:24:32.378 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:32 smithi086 bash[122902]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-06T14:24:32.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:32 smithi086 ceph-mon[110975]: Health check failed: Degraded data redundancy: 21/393 objects degraded (5.344%), 6 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:32 smithi032 ceph-mon[147414]: Health check failed: Degraded data redundancy: 21/393 objects degraded (5.344%), 6 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:33.627 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:33 smithi086 ceph-osd[123198]: -- 172.21.15.86:0/540406698 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5610069071e0 con 0x5610068d6c00 2024-04-06T14:24:33.628 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:33 smithi086 bash[123160]: bcdfa7f8b79b24fcb31e32cbdc43f82ba4717ae2cbe79c698c38c0458e727e93 2024-04-06T14:24:33.628 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:33 smithi086 systemd[1]: Started Ceph osd.3 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:24:34.127 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:33 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[123175]: 2024-04-06T14:24:33.786+0000 7fc130ce4640 -1 Falling back to public interface 2024-04-06T14:24:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:34 smithi032 ceph-mon[147414]: pgmap v115: 65 pgs: 6 active+undersized+degraded, 25 peering, 1 stale+active+clean, 3 active+undersized, 30 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 21/393 objects degraded (5.344%) 2024-04-06T14:24:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:34 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:24:34.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:34 smithi086 ceph-mon[110975]: pgmap v115: 65 pgs: 6 active+undersized+degraded, 25 peering, 1 stale+active+clean, 3 active+undersized, 30 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 21/393 objects degraded (5.344%) 2024-04-06T14:24:34.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:34 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:34.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:34 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:34.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:34 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:24:35.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:35 smithi032 ceph-mon[147414]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs peering) 2024-04-06T14:24:35.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:35 smithi086 ceph-mon[110975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs peering) 2024-04-06T14:24:36.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:36 smithi032 ceph-mon[147414]: pgmap v116: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:36.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:36 smithi086 ceph-mon[110975]: pgmap v116: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:37.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:37.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:37.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:37 smithi032 ceph-mon[147414]: pgmap v117: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:37.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:37.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:37.732 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:37.733 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:37.733 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:37 smithi086 ceph-mon[110975]: pgmap v117: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:37.733 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:37.733 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:38.603 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:38.603 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:24:38.878 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:38 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[123175]: 2024-04-06T14:24:38.620+0000 7fc130ce4640 -1 osd.3 65 log_to_monitors true 2024-04-06T14:24:38.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:38.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:24:39.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:39 smithi086 ceph-mon[110975]: pgmap v118: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:39.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:39 smithi086 ceph-mon[110975]: from='osd.3 [v2:172.21.15.86:6800/3817339654,v1:172.21.15.86:6801/3817339654]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T14:24:39.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:39 smithi086 ceph-mon[110975]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T14:24:39.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:39 smithi086 ceph-mon[110975]: Health check update: Degraded data redundancy: 79/393 objects degraded (20.102%), 20 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:39 smithi032 ceph-mon[147414]: pgmap v118: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:39 smithi032 ceph-mon[147414]: from='osd.3 [v2:172.21.15.86:6800/3817339654,v1:172.21.15.86:6801/3817339654]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T14:24:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:39 smithi032 ceph-mon[147414]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T14:24:39.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:39 smithi032 ceph-mon[147414]: Health check update: Degraded data redundancy: 79/393 objects degraded (20.102%), 20 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:40.702 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 14:24:40 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[123175]: 2024-04-06T14:24:40.478+0000 7fc11e135700 -1 osd.3 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:24:40.703 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:40 smithi086 ceph-mon[110975]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T14:24:40.703 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:40 smithi086 ceph-mon[110975]: osdmap e68: 6 total, 5 up, 6 in 2024-04-06T14:24:40.703 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:40 smithi086 ceph-mon[110975]: from='osd.3 [v2:172.21.15.86:6800/3817339654,v1:172.21.15.86:6801/3817339654]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:24:40.703 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:40 smithi086 ceph-mon[110975]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:24:40.703 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:40 smithi086 ceph-mon[110975]: from='osd.3 ' entity='osd.3' 2024-04-06T14:24:40.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:40 smithi032 ceph-mon[147414]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T14:24:40.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:40 smithi032 ceph-mon[147414]: osdmap e68: 6 total, 5 up, 6 in 2024-04-06T14:24:40.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:40 smithi032 ceph-mon[147414]: from='osd.3 [v2:172.21.15.86:6800/3817339654,v1:172.21.15.86:6801/3817339654]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:24:40.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:40 smithi032 ceph-mon[147414]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:24:40.903 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:40 smithi032 ceph-mon[147414]: from='osd.3 ' entity='osd.3' 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: pgmap v120: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T14:24:42.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T14:24:42.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-06T14:24:42.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:24:42.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: osd.3 [v2:172.21.15.86:6800/3817339654,v1:172.21.15.86:6801/3817339654] boot 2024-04-06T14:24:42.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: osdmap e69: 6 total, 6 up, 6 in 2024-04-06T14:24:42.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:24:42.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: pgmap v120: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:42.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:42.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:42.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:24:42.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: osd.3 [v2:172.21.15.86:6800/3817339654,v1:172.21.15.86:6801/3817339654] boot 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: osdmap e69: 6 total, 6 up, 6 in 2024-04-06T14:24:42.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T14:24:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:43 smithi032 ceph-mon[147414]: osdmap e70: 6 total, 6 up, 6 in 2024-04-06T14:24:43.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:43 smithi032 ceph-mon[147414]: pgmap v123: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:43 smithi086 ceph-mon[110975]: osdmap e70: 6 total, 6 up, 6 in 2024-04-06T14:24:43.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:43 smithi086 ceph-mon[110975]: pgmap v123: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 351 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 79/393 objects degraded (20.102%) 2024-04-06T14:24:45.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:45 smithi086 ceph-mon[110975]: Health check update: Degraded data redundancy: 59/393 objects degraded (15.013%), 14 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:45.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:45 smithi032 ceph-mon[147414]: Health check update: Degraded data redundancy: 59/393 objects degraded (15.013%), 14 pgs degraded (PG_DEGRADED) 2024-04-06T14:24:46.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:46 smithi086 ceph-mon[110975]: pgmap v124: 65 pgs: 14 active+undersized+degraded, 9 active+undersized, 42 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 59/393 objects degraded (15.013%) 2024-04-06T14:24:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:46 smithi032 ceph-mon[147414]: pgmap v124: 65 pgs: 14 active+undersized+degraded, 9 active+undersized, 42 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 59/393 objects degraded (15.013%) 2024-04-06T14:24:47.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:47 smithi086 ceph-mon[110975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/393 objects degraded (15.013%), 14 pgs degraded) 2024-04-06T14:24:47.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:47 smithi032 ceph-mon[147414]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/393 objects degraded (15.013%), 14 pgs degraded) 2024-04-06T14:24:48.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:48 smithi086 ceph-mon[110975]: pgmap v125: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:48.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:48 smithi032 ceph-mon[147414]: pgmap v125: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:50 smithi086 ceph-mon[110975]: pgmap v126: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:50 smithi032 ceph-mon[147414]: pgmap v126: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:52.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:52 smithi086 ceph-mon[110975]: pgmap v127: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:52 smithi032 ceph-mon[147414]: pgmap v127: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:53 smithi032 ceph-mon[147414]: pgmap v128: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:53.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:24:53.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:53 smithi086 ceph-mon[110975]: pgmap v128: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:24:53.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:56.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:55 smithi032 ceph-mon[147414]: pgmap v129: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:56.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T14:24:56.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:55 smithi086 ceph-mon[110975]: pgmap v129: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:56.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T14:24:56.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.836+0000 7ff7cd85b700 1 -- 172.21.15.32:0/3560275298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c81037a0 msgr2=0x7ff7c80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.836+0000 7ff7cd85b700 1 --2- 172.21.15.32:0/3560275298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c81037a0 0x7ff7c80fce10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8004940 tx=0x7ff7b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.836+0000 7ff7cd85b700 1 -- 172.21.15.32:0/3560275298 shutdown_connections 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.836+0000 7ff7cd85b700 1 --2- 172.21.15.32:0/3560275298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c81037a0 0x7ff7c80fce10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.836+0000 7ff7cd85b700 1 --2- 172.21.15.32:0/3560275298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7c8102e00 0x7ff7c81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.836+0000 7ff7cd85b700 1 -- 172.21.15.32:0/3560275298 >> 172.21.15.32:0/3560275298 conn(0x7ff7c80f89b0 msgr2=0x7ff7c80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.836+0000 7ff7cd85b700 1 -- 172.21.15.32:0/3560275298 shutdown_connections 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.836+0000 7ff7cd85b700 1 -- 172.21.15.32:0/3560275298 wait complete. 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7cd85b700 1 Processor -- start 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7cd85b700 1 -- start start 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7cd85b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7cd85b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7cd85b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8070d90 con 0x7ff7c8102e00 2024-04-06T14:24:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7cd85b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8070ed0 con 0x7ff7c81037a0 2024-04-06T14:24:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41368/0 (socket says 172.21.15.32:41368) 2024-04-06T14:24:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7c6ffd700 1 -- 172.21.15.32:0/4189180023 learned_addr learned my addr 172.21.15.32:0/4189180023 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.837+0000 7ff7c67fc700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.838+0000 7ff7c6ffd700 1 -- 172.21.15.32:0/4189180023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7c81037a0 msgr2=0x7ff7c8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.838+0000 7ff7c6ffd700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.838+0000 7ff7c6ffd700 1 -- 172.21.15.32:0/4189180023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b8005040 con 0x7ff7c8102e00 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.838+0000 7ff7c6ffd700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0002a50 tx=0x7ff7b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.838+0000 7ff7bf7fe700 1 -- 172.21.15.32:0/4189180023 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7b0007cc0 con 0x7ff7c8102e00 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.838+0000 7ff7bf7fe700 1 -- 172.21.15.32:0/4189180023 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7b000f040 con 0x7ff7c8102e00 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.838+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c8119dc0 con 0x7ff7c8102e00 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.838+0000 7ff7bf7fe700 1 -- 172.21.15.32:0/4189180023 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7b000d9c0 con 0x7ff7c8102e00 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.838+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c811a240 con 0x7ff7c8102e00 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.839+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c80fe550 con 0x7ff7c8102e00 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.840+0000 7ff7bf7fe700 1 -- 172.21.15.32:0/4189180023 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff7b000db20 con 0x7ff7c8102e00 2024-04-06T14:24:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.840+0000 7ff7bf7fe700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7b406cc20 0x7ff7b406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:56.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.840+0000 7ff7bf7fe700 1 -- 172.21.15.32:0/4189180023 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff7b008bd10 con 0x7ff7c8102e00 2024-04-06T14:24:56.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.840+0000 7ff7c67fc700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7b406cc20 0x7ff7b406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:56.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.841+0000 7ff7c67fc700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7b406cc20 0x7ff7b406f0d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8004890 tx=0x7ff7b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:56.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:56.845+0000 7ff7bf7fe700 1 -- 172.21.15.32:0/4189180023 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff7b0055d70 con 0x7ff7c8102e00 2024-04-06T14:24:57.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.069+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7c80719b0 con 0x7ff7b406cc20 2024-04-06T14:24:57.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.070+0000 7ff7bf7fe700 1 -- 172.21.15.32:0/4189180023 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7ff7c80719b0 con 0x7ff7b406cc20 2024-04-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7b406cc20 msgr2=0x7ff7b406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7b406cc20 0x7ff7b406f0d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8004890 tx=0x7ff7b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c8102e00 msgr2=0x7ff7c80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0002a50 tx=0x7ff7b00046b0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 shutdown_connections 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7b406cc20 0x7ff7b406f0d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 --2- 172.21.15.32:0/4189180023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 >> 172.21.15.32:0/4189180023 conn(0x7ff7c80f89b0 msgr2=0x7ff7c80f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 shutdown_connections 2024-04-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.072+0000 7ff7cd85b700 1 -- 172.21.15.32:0/4189180023 wait complete. 2024-04-06T14:24:57.083 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:24:57.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:56 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T14:24:57.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:56 smithi086 ceph-mon[110975]: Upgrade: osd.4 is safe to restart 2024-04-06T14:24:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.176+0000 7fe1c1120700 1 -- 172.21.15.32:0/4128782856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc102e10 msgr2=0x7fe1bc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.176+0000 7fe1c1120700 1 --2- 172.21.15.32:0/4128782856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc102e10 0x7fe1bc1031e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0004940 tx=0x7fe1b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.176+0000 7fe1c1120700 1 -- 172.21.15.32:0/4128782856 shutdown_connections 2024-04-06T14:24:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.176+0000 7fe1c1120700 1 --2- 172.21.15.32:0/4128782856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1bc1037b0 0x7fe1bc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.176+0000 7fe1c1120700 1 --2- 172.21.15.32:0/4128782856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc102e10 0x7fe1bc1031e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.176+0000 7fe1c1120700 1 -- 172.21.15.32:0/4128782856 >> 172.21.15.32:0/4128782856 conn(0x7fe1bc0f89a0 msgr2=0x7fe1bc0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.176+0000 7fe1c1120700 1 -- 172.21.15.32:0/4128782856 shutdown_connections 2024-04-06T14:24:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.177+0000 7fe1c1120700 1 -- 172.21.15.32:0/4128782856 wait complete. 2024-04-06T14:24:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.177+0000 7fe1c1120700 1 Processor -- start 2024-04-06T14:24:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.177+0000 7fe1c1120700 1 -- start start 2024-04-06T14:24:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.177+0000 7fe1c1120700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1bc102e10 0x7fe1bc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.177+0000 7fe1c1120700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc1037b0 0x7fe1bc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.177+0000 7fe1c1120700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc070e20 con 0x7fe1bc1037b0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.177+0000 7fe1c1120700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc070f60 con 0x7fe1bc102e10 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1ba59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc1037b0 0x7fe1bc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1ba59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc1037b0 0x7fe1bc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41376/0 (socket says 172.21.15.32:41376) 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1ba59c700 1 -- 172.21.15.32:0/1004043898 learned_addr learned my addr 172.21.15.32:0/1004043898 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1bad9d700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1bc102e10 0x7fe1bc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1ba59c700 1 -- 172.21.15.32:0/1004043898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1bc102e10 msgr2=0x7fe1bc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1ba59c700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1bc102e10 0x7fe1bc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1ba59c700 1 -- 172.21.15.32:0/1004043898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b0005040 con 0x7fe1bc1037b0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1ba59c700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc1037b0 0x7fe1bc070800 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00b790 tx=0x7fe1ac009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1a37fe700 1 -- 172.21.15.32:0/1004043898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1ac00f6f0 con 0x7fe1bc1037b0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1bc119db0 con 0x7fe1bc1037b0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1a37fe700 1 -- 172.21.15.32:0/1004043898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1ac00fd00 con 0x7fe1bc1037b0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1a37fe700 1 -- 172.21.15.32:0/1004043898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1ac0103a0 con 0x7fe1bc1037b0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.178+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1bc11a2c0 con 0x7fe1bc1037b0 2024-04-06T14:24:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.179+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1bc0fe560 con 0x7fe1bc1037b0 2024-04-06T14:24:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.180+0000 7fe1a37fe700 1 -- 172.21.15.32:0/1004043898 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe1ac00f850 con 0x7fe1bc1037b0 2024-04-06T14:24:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.180+0000 7fe1a37fe700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1a406cc20 0x7fe1a406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.180+0000 7fe1a37fe700 1 -- 172.21.15.32:0/1004043898 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fe1ac08bda0 con 0x7fe1bc1037b0 2024-04-06T14:24:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.180+0000 7fe1bad9d700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1a406cc20 0x7fe1a406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.181+0000 7fe1bad9d700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1a406cc20 0x7fe1a406f0d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0004f90 tx=0x7fe1b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:57.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.185+0000 7fe1a37fe700 1 -- 172.21.15.32:0/1004043898 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe1ac055e80 con 0x7fe1bc1037b0 2024-04-06T14:24:57.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.411+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1bc04bed0 con 0x7fe1a406cc20 2024-04-06T14:24:57.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.412+0000 7fe1a37fe700 1 -- 172.21.15.32:0/1004043898 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe1bc04bed0 con 0x7fe1a406cc20 2024-04-06T14:24:57.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1a406cc20 msgr2=0x7fe1a406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1a406cc20 0x7fe1a406f0d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0004f90 tx=0x7fe1b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:57.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc1037b0 msgr2=0x7fe1bc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc1037b0 0x7fe1bc070800 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00b790 tx=0x7fe1ac009220 comp rx=0 tx=0).stop 2024-04-06T14:24:57.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 shutdown_connections 2024-04-06T14:24:57.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1a406cc20 0x7fe1a406f0d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1bc102e10 0x7fe1bc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 --2- 172.21.15.32:0/1004043898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1bc1037b0 0x7fe1bc070800 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 >> 172.21.15.32:0/1004043898 conn(0x7fe1bc0f89a0 msgr2=0x7fe1bc0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:57.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 shutdown_connections 2024-04-06T14:24:57.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.414+0000 7fe1c1120700 1 -- 172.21.15.32:0/1004043898 wait complete. 2024-04-06T14:24:57.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:56 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T14:24:57.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:56 smithi032 ceph-mon[147414]: Upgrade: osd.4 is safe to restart 2024-04-06T14:24:57.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.520+0000 7f42a176c700 1 -- 172.21.15.32:0/4031528188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c101d20 msgr2=0x7f429c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.520+0000 7f42a176c700 1 --2- 172.21.15.32:0/4031528188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c101d20 0x7f429c1020f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f428c004940 tx=0x7f428c009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:57.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.520+0000 7f42a176c700 1 -- 172.21.15.32:0/4031528188 shutdown_connections 2024-04-06T14:24:57.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.520+0000 7f42a176c700 1 --2- 172.21.15.32:0/4031528188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f429c1026c0 0x7f429c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.520+0000 7f42a176c700 1 --2- 172.21.15.32:0/4031528188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c101d20 0x7f429c1020f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.520+0000 7f42a176c700 1 -- 172.21.15.32:0/4031528188 >> 172.21.15.32:0/4031528188 conn(0x7f429c0f7870 msgr2=0x7f429c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:57.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.520+0000 7f42a176c700 1 -- 172.21.15.32:0/4031528188 shutdown_connections 2024-04-06T14:24:57.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.520+0000 7f42a176c700 1 -- 172.21.15.32:0/4031528188 wait complete. 2024-04-06T14:24:57.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f42a176c700 1 Processor -- start 2024-04-06T14:24:57.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f42a176c700 1 -- start start 2024-04-06T14:24:57.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f42a176c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f429c101d20 0x7f429c1138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f42a176c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c1026c0 0x7f429c113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f42a176c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c114440 con 0x7f429c1026c0 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f42a176c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c10da40 con 0x7f429c101d20 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f429a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c1026c0 0x7f429c113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f429a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c1026c0 0x7f429c113df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41394/0 (socket says 172.21.15.32:41394) 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f429affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f429c101d20 0x7f429c1138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.521+0000 7f429a7fc700 1 -- 172.21.15.32:0/3027111802 learned_addr learned my addr 172.21.15.32:0/3027111802 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.522+0000 7f429a7fc700 1 -- 172.21.15.32:0/3027111802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f429c101d20 msgr2=0x7f429c1138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.522+0000 7f429a7fc700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f429c101d20 0x7f429c1138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.522+0000 7f429a7fc700 1 -- 172.21.15.32:0/3027111802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c005040 con 0x7f429c1026c0 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.522+0000 7f429a7fc700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c1026c0 0x7f429c113df0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4290002a50 tx=0x7f4290004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.522+0000 7f42837fe700 1 -- 172.21.15.32:0/3027111802 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4290007cc0 con 0x7f429c1026c0 2024-04-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.522+0000 7f42837fe700 1 -- 172.21.15.32:0/3027111802 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f429000f040 con 0x7f429c1026c0 2024-04-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.522+0000 7f42837fe700 1 -- 172.21.15.32:0/3027111802 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f429000d920 con 0x7f429c1026c0 2024-04-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.522+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429c10dd20 con 0x7f429c1026c0 2024-04-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.522+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429c10e2e0 con 0x7f429c1026c0 2024-04-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.523+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429c0fd470 con 0x7f429c1026c0 2024-04-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.524+0000 7f42837fe700 1 -- 172.21.15.32:0/3027111802 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4290009680 con 0x7f429c1026c0 2024-04-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.525+0000 7f42837fe700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f428406cc20 0x7f428406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.525+0000 7f42837fe700 1 -- 172.21.15.32:0/3027111802 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f429008c3a0 con 0x7f429c1026c0 2024-04-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.525+0000 7f429affd700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f428406cc20 0x7f428406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.525+0000 7f429affd700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f428406cc20 0x7f428406f0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f428c00b840 tx=0x7f428c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.528+0000 7f42837fe700 1 -- 172.21.15.32:0/3027111802 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4290059fa0 con 0x7f429c1026c0 2024-04-06T14:24:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.741+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f429c108d80 con 0x7f428406cc20 2024-04-06T14:24:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.750+0000 7f42837fe700 1 -- 172.21.15.32:0/3027111802 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f428c008ee0 con 0x7f428406cc20 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (5m) 54s ago 16m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (2m) 54s ago 16m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (2m) 21s ago 15m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (5m) 54s ago 16m 87.2M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (10m) 54s ago 10m 14.5M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (10m) 54s ago 10m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (10m) 21s ago 10m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (10m) 21s ago 10m 17.1M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (7m) 54s ago 18m 503M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:24:57.751 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (6m) 21s ago 15m 450M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (3m) 54s ago 18m 47.1M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (3m) 21s ago 15m 37.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (6m) 54s ago 16m 16.6M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (5m) 21s ago 15m 17.5M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (2m) 54s ago 14m 78.7M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (97s) 54s ago 14m 80.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (60s) 54s ago 13m 11.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (24s) 21s ago 13m 10.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (12m) 21s ago 12m 205M 2592M 17.2.7-819-g6699495c ec300eaa768b 9bbac5cec109 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (12m) 21s ago 12m 187M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:24:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (5m) 54s ago 16m 59.8M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:24:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.752+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f428406cc20 msgr2=0x7f428406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.752+0000 7f42a176c700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f428406cc20 0x7f428406f0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f428c00b840 tx=0x7f428c009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.752+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c1026c0 msgr2=0x7f429c113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.752+0000 7f42a176c700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c1026c0 0x7f429c113df0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4290002a50 tx=0x7f4290004600 comp rx=0 tx=0).stop 2024-04-06T14:24:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.752+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 shutdown_connections 2024-04-06T14:24:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.752+0000 7f42a176c700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f428406cc20 0x7f428406f0d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.752+0000 7f42a176c700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f429c101d20 0x7f429c1138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.752+0000 7f42a176c700 1 --2- 172.21.15.32:0/3027111802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c1026c0 0x7f429c113df0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.752+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 >> 172.21.15.32:0/3027111802 conn(0x7f429c0f7870 msgr2=0x7f429c0662c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.753+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 shutdown_connections 2024-04-06T14:24:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.753+0000 7f42a176c700 1 -- 172.21.15.32:0/3027111802 wait complete. 2024-04-06T14:24:57.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.854+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/2885006331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a241037b0 msgr2=0x7f8a240fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.854+0000 7f8a2ae6f700 1 --2- 172.21.15.32:0/2885006331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a241037b0 0x7f8a240fce20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8a10004940 tx=0x7f8a10009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.854+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/2885006331 shutdown_connections 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.854+0000 7f8a2ae6f700 1 --2- 172.21.15.32:0/2885006331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a241037b0 0x7f8a240fce20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.854+0000 7f8a2ae6f700 1 --2- 172.21.15.32:0/2885006331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a24102e10 0x7f8a241031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.854+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/2885006331 >> 172.21.15.32:0/2885006331 conn(0x7f8a240f89a0 msgr2=0x7f8a240fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.854+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/2885006331 shutdown_connections 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.854+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/2885006331 wait complete. 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a2ae6f700 1 Processor -- start 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a2ae6f700 1 -- start start 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a2ae6f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a24102e10 0x7f8a24113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a2ae6f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a241037b0 0x7f8a24114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a2ae6f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a24114660 con 0x7f8a24102e10 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a2ae6f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2410da50 con 0x7f8a241037b0 2024-04-06T14:24:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a28c0b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a24102e10 0x7f8a24113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a28c0b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a24102e10 0x7f8a24113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41418/0 (socket says 172.21.15.32:41418) 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a28c0b700 1 -- 172.21.15.32:0/1484965983 learned_addr learned my addr 172.21.15.32:0/1484965983 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.855+0000 7f8a23fff700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a241037b0 0x7f8a24114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.856+0000 7f8a28c0b700 1 -- 172.21.15.32:0/1484965983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a241037b0 msgr2=0x7f8a24114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.856+0000 7f8a28c0b700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a241037b0 0x7f8a24114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.856+0000 7f8a28c0b700 1 -- 172.21.15.32:0/1484965983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a10005040 con 0x7f8a24102e10 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.856+0000 7f8a28c0b700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a24102e10 0x7f8a24113ad0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800c830 tx=0x7f8a18004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.856+0000 7f8a217fa700 1 -- 172.21.15.32:0/1484965983 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a180106f0 con 0x7f8a24102e10 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.856+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2410dd30 con 0x7f8a24102e10 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.856+0000 7f8a217fa700 1 -- 172.21.15.32:0/1484965983 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a18010d00 con 0x7f8a24102e10 2024-04-06T14:24:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.856+0000 7f8a217fa700 1 -- 172.21.15.32:0/1484965983 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a18012e90 con 0x7f8a24102e10 2024-04-06T14:24:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.856+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2410e2c0 con 0x7f8a24102e10 2024-04-06T14:24:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.857+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a240fe560 con 0x7f8a24102e10 2024-04-06T14:24:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.858+0000 7f8a217fa700 1 -- 172.21.15.32:0/1484965983 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8a18010850 con 0x7f8a24102e10 2024-04-06T14:24:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.858+0000 7f8a217fa700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a1406cc30 0x7f8a1406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.858+0000 7f8a217fa700 1 -- 172.21.15.32:0/1484965983 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f8a1808bf60 con 0x7f8a24102e10 2024-04-06T14:24:57.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.858+0000 7f8a23fff700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a1406cc30 0x7f8a1406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:57.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.858+0000 7f8a23fff700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a1406cc30 0x7f8a1406f0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a2410ed30 tx=0x7f8a10009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:57.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:57.863+0000 7f8a217fa700 1 -- 172.21.15.32:0/1484965983 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8a180bfab0 con 0x7f8a24102e10 2024-04-06T14:24:58.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.147+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8a24109db0 con 0x7f8a24102e10 2024-04-06T14:24:58.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.148+0000 7f8a217fa700 1 -- 172.21.15.32:0/1484965983 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f8a18059ad0 con 0x7f8a24102e10 2024-04-06T14:24:58.148 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:24:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:24:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:24:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:24:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:24:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 2, 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 6, 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 8 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:24:58.149 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:24:58.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.149+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a1406cc30 msgr2=0x7f8a1406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a1406cc30 0x7f8a1406f0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a2410ed30 tx=0x7f8a10009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:58.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a24102e10 msgr2=0x7f8a24113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a24102e10 0x7f8a24113ad0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800c830 tx=0x7f8a18004470 comp rx=0 tx=0).stop 2024-04-06T14:24:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 shutdown_connections 2024-04-06T14:24:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a1406cc30 0x7f8a1406f0e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a24102e10 0x7f8a24113ad0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 --2- 172.21.15.32:0/1484965983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a241037b0 0x7f8a24114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 >> 172.21.15.32:0/1484965983 conn(0x7f8a240f89a0 msgr2=0x7f8a240f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 shutdown_connections 2024-04-06T14:24:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.150+0000 7f8a2ae6f700 1 -- 172.21.15.32:0/1484965983 wait complete. 2024-04-06T14:24:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.253+0000 7f84452b8700 1 -- 172.21.15.32:0/1694798407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8440102e00 msgr2=0x7f84401031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.253+0000 7f84452b8700 1 --2- 172.21.15.32:0/1694798407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8440102e00 0x7f84401031d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8428004940 tx=0x7f8428009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.253+0000 7f84452b8700 1 -- 172.21.15.32:0/1694798407 shutdown_connections 2024-04-06T14:24:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.253+0000 7f84452b8700 1 --2- 172.21.15.32:0/1694798407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84401037a0 0x7f84400fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.253+0000 7f84452b8700 1 --2- 172.21.15.32:0/1694798407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8440102e00 0x7f84401031d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.253+0000 7f84452b8700 1 -- 172.21.15.32:0/1694798407 >> 172.21.15.32:0/1694798407 conn(0x7f84400f89f0 msgr2=0x7f84400fae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.253+0000 7f84452b8700 1 -- 172.21.15.32:0/1694798407 shutdown_connections 2024-04-06T14:24:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.253+0000 7f84452b8700 1 -- 172.21.15.32:0/1694798407 wait complete. 2024-04-06T14:24:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f84452b8700 1 Processor -- start 2024-04-06T14:24:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f84452b8700 1 -- start start 2024-04-06T14:24:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f84452b8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8440102e00 0x7f8440113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f84452b8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84401037a0 0x7f8440113d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f84452b8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84401143c0 con 0x7f84401037a0 2024-04-06T14:24:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f84452b8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844010d9c0 con 0x7f8440102e00 2024-04-06T14:24:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f843e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84401037a0 0x7f8440113d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f843e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84401037a0 0x7f8440113d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41438/0 (socket says 172.21.15.32:41438) 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f843e7fc700 1 -- 172.21.15.32:0/188177128 learned_addr learned my addr 172.21.15.32:0/188177128 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f843effd700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8440102e00 0x7f8440113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f843e7fc700 1 -- 172.21.15.32:0/188177128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8440102e00 msgr2=0x7f8440113830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f843e7fc700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8440102e00 0x7f8440113830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.254+0000 7f843e7fc700 1 -- 172.21.15.32:0/188177128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8428005040 con 0x7f84401037a0 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.255+0000 7f843e7fc700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84401037a0 0x7f8440113d70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f843000eb40 tx=0x7f843000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.255+0000 7f84377fe700 1 -- 172.21.15.32:0/188177128 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8430007cc0 con 0x7f84401037a0 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.255+0000 7f84377fe700 1 -- 172.21.15.32:0/188177128 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f843000f040 con 0x7f84401037a0 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.255+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844010dc40 con 0x7f84401037a0 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.255+0000 7f84377fe700 1 -- 172.21.15.32:0/188177128 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8430009570 con 0x7f84401037a0 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.255+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844010e200 con 0x7f84401037a0 2024-04-06T14:24:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.256+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84400fe550 con 0x7f84401037a0 2024-04-06T14:24:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.256+0000 7f84377fe700 1 -- 172.21.15.32:0/188177128 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f84300096d0 con 0x7f84401037a0 2024-04-06T14:24:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.257+0000 7f84377fe700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f842c06cc20 0x7f842c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.257+0000 7f84377fe700 1 -- 172.21.15.32:0/188177128 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f843008b9f0 con 0x7f84401037a0 2024-04-06T14:24:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.257+0000 7f843effd700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f842c06cc20 0x7f842c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.257+0000 7f843effd700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f842c06cc20 0x7f842c06f0d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f842800b0d0 tx=0x7f8428009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:58.260 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:57 smithi086 ceph-mon[110975]: pgmap v130: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:58.260 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:57 smithi086 ceph-mon[110975]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:58.260 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:57 smithi086 ceph-mon[110975]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:58.260 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:57 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:58.260 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:57 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T14:24:58.260 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:57 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:24:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.261+0000 7f84377fe700 1 -- 172.21.15.32:0/188177128 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f843008de60 con 0x7f84401037a0 2024-04-06T14:24:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:57 smithi032 ceph-mon[147414]: pgmap v130: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:24:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:57 smithi032 ceph-mon[147414]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:57 smithi032 ceph-mon[147414]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:57 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:24:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:57 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T14:24:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:57 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:24:58.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.512+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8440109dc0 con 0x7f84401037a0 2024-04-06T14:24:58.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.512+0000 7f84377fe700 1 -- 172.21.15.32:0/188177128 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1851 (secure 0 0 0) 0x7f8430019750 con 0x7f84401037a0 2024-04-06T14:24:58.512 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:21:28.458361+0000 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:24:58.513 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:24:58.514 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:24:58.515 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:24:58.516 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:24:58.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f842c06cc20 msgr2=0x7f842c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f842c06cc20 0x7f842c06f0d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f842800b0d0 tx=0x7f8428009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:58.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84401037a0 msgr2=0x7f8440113d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84401037a0 0x7f8440113d70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f843000eb40 tx=0x7f843000b040 comp rx=0 tx=0).stop 2024-04-06T14:24:58.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 shutdown_connections 2024-04-06T14:24:58.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f842c06cc20 0x7f842c06f0d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8440102e00 0x7f8440113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 --2- 172.21.15.32:0/188177128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84401037a0 0x7f8440113d70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 >> 172.21.15.32:0/188177128 conn(0x7f84400f89f0 msgr2=0x7f84400f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:58.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 shutdown_connections 2024-04-06T14:24:58.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.514+0000 7f84452b8700 1 -- 172.21.15.32:0/188177128 wait complete. 2024-04-06T14:24:58.517 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-06T14:24:58.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.619+0000 7f025de1f700 1 -- 172.21.15.32:0/1006677140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 msgr2=0x7f0258106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.619+0000 7f025de1f700 1 --2- 172.21.15.32:0/1006677140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 0x7f0258106ec0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f024c004940 tx=0x7f024c009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:58.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.619+0000 7f025de1f700 1 -- 172.21.15.32:0/1006677140 shutdown_connections 2024-04-06T14:24:58.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.619+0000 7f025de1f700 1 --2- 172.21.15.32:0/1006677140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 0x7f0258106ec0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.619+0000 7f025de1f700 1 --2- 172.21.15.32:0/1006677140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02580fdee0 0x7f02580fe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.619+0000 7f025de1f700 1 -- 172.21.15.32:0/1006677140 >> 172.21.15.32:0/1006677140 conn(0x7f02580f78f0 msgr2=0x7f02580f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.620+0000 7f025de1f700 1 -- 172.21.15.32:0/1006677140 shutdown_connections 2024-04-06T14:24:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.620+0000 7f025de1f700 1 -- 172.21.15.32:0/1006677140 wait complete. 2024-04-06T14:24:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.620+0000 7f025de1f700 1 Processor -- start 2024-04-06T14:24:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.620+0000 7f025de1f700 1 -- start start 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.620+0000 7f025de1f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02580fdee0 0x7f02580fbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.620+0000 7f025de1f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 0x7f02580fc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.620+0000 7f025de1f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580fdc60 con 0x7f02580fe880 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.620+0000 7f025de1f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580fc790 con 0x7f02580fdee0 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f0256ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 0x7f02580fc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f0256ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 0x7f02580fc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41464/0 (socket says 172.21.15.32:41464) 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f0256ffd700 1 -- 172.21.15.32:0/3537292306 learned_addr learned my addr 172.21.15.32:0/3537292306 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f02577fe700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02580fdee0 0x7f02580fbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f0256ffd700 1 -- 172.21.15.32:0/3537292306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02580fdee0 msgr2=0x7f02580fbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f0256ffd700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02580fdee0 0x7f02580fbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f0256ffd700 1 -- 172.21.15.32:0/3537292306 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c005040 con 0x7f02580fe880 2024-04-06T14:24:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f0256ffd700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 0x7f02580fc250 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f024c004890 tx=0x7f024c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f023ffff700 1 -- 172.21.15.32:0/3537292306 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f024c00e040 con 0x7f02580fe880 2024-04-06T14:24:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02580fca70 con 0x7f02580fe880 2024-04-06T14:24:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f023ffff700 1 -- 172.21.15.32:0/3537292306 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f024c009af0 con 0x7f02580fe880 2024-04-06T14:24:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f023ffff700 1 -- 172.21.15.32:0/3537292306 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f024c01c3f0 con 0x7f02580fe880 2024-04-06T14:24:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.621+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0258119eb0 con 0x7f02580fe880 2024-04-06T14:24:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.622+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0258066a90 con 0x7f02580fe880 2024-04-06T14:24:58.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.623+0000 7f023ffff700 1 -- 172.21.15.32:0/3537292306 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f024c013420 con 0x7f02580fe880 2024-04-06T14:24:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.623+0000 7f023ffff700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f024006cc20 0x7f024006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.623+0000 7f023ffff700 1 -- 172.21.15.32:0/3537292306 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f024c08d210 con 0x7f02580fe880 2024-04-06T14:24:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.623+0000 7f02577fe700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f024006cc20 0x7f024006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:58.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.624+0000 7f02577fe700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f024006cc20 0x7f024006f0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0248009fd0 tx=0x7f024800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:58.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.628+0000 7f023ffff700 1 -- 172.21.15.32:0/3537292306 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f024c0573a0 con 0x7f02580fe880 2024-04-06T14:24:58.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.847+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f025805e400 con 0x7f024006cc20 2024-04-06T14:24:58.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.848+0000 7f023ffff700 1 -- 172.21.15.32:0/3537292306 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f025805e400 con 0x7f024006cc20 2024-04-06T14:24:58.848 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:24:58.848 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:24:58.848 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "10/21 daemons upgraded", 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:24:58.849 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:24:58.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f024006cc20 msgr2=0x7f024006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f024006cc20 0x7f024006f0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0248009fd0 tx=0x7f024800b040 comp rx=0 tx=0).stop 2024-04-06T14:24:58.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 msgr2=0x7f02580fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 0x7f02580fc250 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f024c004890 tx=0x7f024c009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 shutdown_connections 2024-04-06T14:24:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f024006cc20 0x7f024006f0d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02580fdee0 0x7f02580fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 --2- 172.21.15.32:0/3537292306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02580fe880 0x7f02580fc250 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 >> 172.21.15.32:0/3537292306 conn(0x7f02580f78f0 msgr2=0x7f02580ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 shutdown_connections 2024-04-06T14:24:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.850+0000 7f025de1f700 1 -- 172.21.15.32:0/3537292306 wait complete. 2024-04-06T14:24:58.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.953+0000 7fd504192700 1 -- 172.21.15.32:0/2860490622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 msgr2=0x7fd4fc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.953+0000 7fd504192700 1 --2- 172.21.15.32:0/2860490622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 0x7fd4fc102140 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8004940 tx=0x7fd4f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:58.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.953+0000 7fd504192700 1 -- 172.21.15.32:0/2860490622 shutdown_connections 2024-04-06T14:24:58.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.953+0000 7fd504192700 1 --2- 172.21.15.32:0/2860490622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4fc102710 0x7fd4fc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.953+0000 7fd504192700 1 --2- 172.21.15.32:0/2860490622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 0x7fd4fc102140 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.953+0000 7fd504192700 1 -- 172.21.15.32:0/2860490622 >> 172.21.15.32:0/2860490622 conn(0x7fd4fc072bc0 msgr2=0x7fd4fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:58.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.953+0000 7fd504192700 1 -- 172.21.15.32:0/2860490622 shutdown_connections 2024-04-06T14:24:58.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd504192700 1 -- 172.21.15.32:0/2860490622 wait complete. 2024-04-06T14:24:58.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd504192700 1 Processor -- start 2024-04-06T14:24:58.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd504192700 1 -- start start 2024-04-06T14:24:58.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd504192700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 0x7fd4fc102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:58.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd504192700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4fc102710 0x7fd4fc103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:58.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd504192700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc103750 con 0x7fd4fc101d70 2024-04-06T14:24:58.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd504192700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc1038c0 con 0x7fd4fc102710 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd501f2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 0x7fd4fc102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd501f2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 0x7fd4fc102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41486/0 (socket says 172.21.15.32:41486) 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.954+0000 7fd501f2e700 1 -- 172.21.15.32:0/1585060204 learned_addr learned my addr 172.21.15.32:0/1585060204 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd50172d700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4fc102710 0x7fd4fc103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd501f2e700 1 -- 172.21.15.32:0/1585060204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4fc102710 msgr2=0x7fd4fc103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd501f2e700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4fc102710 0x7fd4fc103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd501f2e700 1 -- 172.21.15.32:0/1585060204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f8005040 con 0x7fd4fc101d70 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd501f2e700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 0x7fd4fc102bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009390 tx=0x7fd4f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd4f27fc700 1 -- 172.21.15.32:0/1585060204 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4f800e040 con 0x7fd4fc101d70 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd4f27fc700 1 -- 172.21.15.32:0/1585060204 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4f8013420 con 0x7fd4fc101d70 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd4f27fc700 1 -- 172.21.15.32:0/1585060204 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4f801c630 con 0x7fd4fc101d70 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc119e00 con 0x7fd4fc101d70 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.955+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc11a2b0 con 0x7fd4fc101d70 2024-04-06T14:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.956+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc0fd4c0 con 0x7fd4fc101d70 2024-04-06T14:24:58.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.957+0000 7fd4f27fc700 1 -- 172.21.15.32:0/1585060204 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd4f80139e0 con 0x7fd4fc101d70 2024-04-06T14:24:58.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.957+0000 7fd4f27fc700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd4e806cc30 0x7fd4e806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:24:58.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.957+0000 7fd4f27fc700 1 -- 172.21.15.32:0/1585060204 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd4f808d2a0 con 0x7fd4fc101d70 2024-04-06T14:24:58.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.957+0000 7fd50172d700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd4e806cc30 0x7fd4e806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:24:58.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.958+0000 7fd50172d700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd4e806cc30 0x7fd4e806f0e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc104510 tx=0x7fd4ec0091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:24:58.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:58.962+0000 7fd4f27fc700 1 -- 172.21.15.32:0/1585060204 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd4f8057380 con 0x7fd4fc101d70 2024-04-06T14:24:59.118 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:58 smithi086 ceph-mon[110975]: Upgrade: Updating osd.4 2024-04-06T14:24:59.118 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:58 smithi086 ceph-mon[110975]: Deploying daemon osd.4 on smithi086 2024-04-06T14:24:59.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:58 smithi086 ceph-mon[110975]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:59.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:58 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1484965983' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:59.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:58 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/188177128' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:24:59.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.251+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd4fc05f5f0 con 0x7fd4fc101d70 2024-04-06T14:24:59.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.251+0000 7fd4f27fc700 1 -- 172.21.15.32:0/1585060204 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fd4f805a9a0 con 0x7fd4fc101d70 2024-04-06T14:24:59.251 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:24:59.251 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:24:59.251 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:24:59.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd4e806cc30 msgr2=0x7fd4e806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:59.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd4e806cc30 0x7fd4e806f0e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc104510 tx=0x7fd4ec0091a0 comp rx=0 tx=0).stop 2024-04-06T14:24:59.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 msgr2=0x7fd4fc102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:24:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 0x7fd4fc102bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009390 tx=0x7fd4f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:24:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 shutdown_connections 2024-04-06T14:24:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd4e806cc30 0x7fd4e806f0e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4fc101d70 0x7fd4fc102bd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 --2- 172.21.15.32:0/1585060204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4fc102710 0x7fd4fc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:24:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 >> 172.21.15.32:0/1585060204 conn(0x7fd4fc072bc0 msgr2=0x7fd4fc0658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:24:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 shutdown_connections 2024-04-06T14:24:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:24:59.253+0000 7fd504192700 1 -- 172.21.15.32:0/1585060204 wait complete. 2024-04-06T14:24:59.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:58 smithi032 ceph-mon[147414]: Upgrade: Updating osd.4 2024-04-06T14:24:59.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:58 smithi032 ceph-mon[147414]: Deploying daemon osd.4 on smithi086 2024-04-06T14:24:59.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:58 smithi032 ceph-mon[147414]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:24:59.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:58 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1484965983' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:24:59.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:58 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/188177128' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:25:00.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:59 smithi086 ceph-mon[110975]: pgmap v131: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:00.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:59 smithi086 ceph-mon[110975]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:00.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:24:59 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1585060204' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:25:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:59 smithi032 ceph-mon[147414]: pgmap v131: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:59 smithi032 ceph-mon[147414]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:24:59 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1585060204' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:25:01.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:00 smithi086 systemd[1]: Stopping Ceph osd.4 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:25:01.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:01 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[63137]: 2024-04-06T14:25:01.052+0000 7fdfd51a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T14:25:01.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:01 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[63137]: 2024-04-06T14:25:01.052+0000 7fdfd51a5700 -1 osd.4 70 *** Got signal Terminated *** 2024-04-06T14:25:01.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:01 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[63137]: 2024-04-06T14:25:01.052+0000 7fdfd51a5700 -1 osd.4 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T14:25:02.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:01 smithi086 ceph-mon[110975]: pgmap v132: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:01 smithi086 ceph-mon[110975]: osd.4 marked itself down and dead 2024-04-06T14:25:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:01 smithi032 ceph-mon[147414]: pgmap v132: 65 pgs: 65 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:01 smithi032 ceph-mon[147414]: osd.4 marked itself down and dead 2024-04-06T14:25:03.378 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:02 smithi086 bash[129523]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4 2024-04-06T14:25:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:02 smithi086 ceph-mon[110975]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:25:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:02 smithi086 ceph-mon[110975]: osdmap e71: 6 total, 5 up, 6 in 2024-04-06T14:25:03.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:02 smithi032 ceph-mon[147414]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:25:03.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:02 smithi032 ceph-mon[147414]: osdmap e71: 6 total, 5 up, 6 in 2024-04-06T14:25:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:04 smithi086 ceph-mon[110975]: pgmap v134: 65 pgs: 6 stale+active+clean, 59 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:04 smithi086 ceph-mon[110975]: osdmap e72: 6 total, 5 up, 6 in 2024-04-06T14:25:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:04 smithi032 ceph-mon[147414]: pgmap v134: 65 pgs: 6 stale+active+clean, 59 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:04 smithi032 ceph-mon[147414]: osdmap e72: 6 total, 5 up, 6 in 2024-04-06T14:25:05.058 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:04 smithi086 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.4.service: Succeeded. 2024-04-06T14:25:05.058 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:04 smithi086 systemd[1]: Stopped Ceph osd.4 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:25:05.058 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:04 smithi086 systemd[1]: Starting Ceph osd.4 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:25:05.318 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:05 smithi086 ceph-mon[110975]: Health check failed: Degraded data redundancy: 24/393 objects degraded (6.107%), 6 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:05 smithi032 ceph-mon[147414]: Health check failed: Degraded data redundancy: 24/393 objects degraded (6.107%), 6 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:06.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:06 smithi086 ceph-mon[110975]: pgmap v136: 65 pgs: 6 active+undersized+degraded, 2 stale+active+clean, 9 active+undersized, 48 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 24/393 objects degraded (6.107%) 2024-04-06T14:25:06.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:05 smithi086 bash[129791]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T14:25:06.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:05 smithi086 bash[129791]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T14:25:06.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:05 smithi086 bash[129791]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T14:25:06.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:05 smithi086 bash[129791]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T14:25:06.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:05 smithi086 bash[129791]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-06T14:25:06.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:05 smithi086 bash[129791]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T14:25:06.128 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:05 smithi086 bash[129791]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-06T14:25:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:06 smithi032 ceph-mon[147414]: pgmap v136: 65 pgs: 6 active+undersized+degraded, 2 stale+active+clean, 9 active+undersized, 48 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 24/393 objects degraded (6.107%) 2024-04-06T14:25:07.247 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:07 smithi086 ceph-osd[130086]: -- 172.21.15.86:0/654758261 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556c8382d1e0 con 0x556c837fcc00 2024-04-06T14:25:07.627 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:07 smithi086 bash[130048]: d0008d3fcf95b4104a21bff65f04b39ede8500723f2cca1d9177bb3aa567f467 2024-04-06T14:25:07.627 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:07 smithi086 systemd[1]: Started Ceph osd.4 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:25:08.022 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:07 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T14:25:07.685+0000 7f2accc11640 -1 Falling back to public interface 2024-04-06T14:25:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:08 smithi086 ceph-mon[110975]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:25:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:08 smithi032 ceph-mon[147414]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:25:10.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:10 smithi086 ceph-mon[110975]: pgmap v138: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:10.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:10 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:10.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:10 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:10 smithi032 ceph-mon[147414]: pgmap v138: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:10 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:10 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:12.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:11 smithi086 ceph-mon[110975]: pgmap v139: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:12.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:11 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:12.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:11 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:12.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:11 smithi032 ceph-mon[147414]: pgmap v139: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:12.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:11 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:12.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:11 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:12.801 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:12 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T14:25:12.519+0000 7f2accc11640 -1 osd.4 70 log_to_monitors true 2024-04-06T14:25:13.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:12 smithi086 ceph-mon[110975]: from='osd.4 [v2:172.21.15.86:6808/1198497064,v1:172.21.15.86:6809/1198497064]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T14:25:13.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:12 smithi086 ceph-mon[110975]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T14:25:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:12 smithi032 ceph-mon[147414]: from='osd.4 [v2:172.21.15.86:6808/1198497064,v1:172.21.15.86:6809/1198497064]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T14:25:13.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:12 smithi032 ceph-mon[147414]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T14:25:14.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 14:25:13 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T14:25:13.712+0000 7f2aba062700 -1 osd.4 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:25:14.089 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:13 smithi086 ceph-mon[110975]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:14.089 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:13 smithi086 ceph-mon[110975]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T14:25:14.089 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:13 smithi086 ceph-mon[110975]: osdmap e73: 6 total, 5 up, 6 in 2024-04-06T14:25:14.089 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:13 smithi086 ceph-mon[110975]: from='osd.4 [v2:172.21.15.86:6808/1198497064,v1:172.21.15.86:6809/1198497064]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:25:14.089 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:13 smithi086 ceph-mon[110975]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:25:14.089 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:13 smithi086 ceph-mon[110975]: from='osd.4 ' entity='osd.4' 2024-04-06T14:25:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:13 smithi032 ceph-mon[147414]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:13 smithi032 ceph-mon[147414]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T14:25:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:13 smithi032 ceph-mon[147414]: osdmap e73: 6 total, 5 up, 6 in 2024-04-06T14:25:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:13 smithi032 ceph-mon[147414]: from='osd.4 [v2:172.21.15.86:6808/1198497064,v1:172.21.15.86:6809/1198497064]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:25:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:13 smithi032 ceph-mon[147414]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:25:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:13 smithi032 ceph-mon[147414]: from='osd.4 ' entity='osd.4' 2024-04-06T14:25:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: Health check update: Degraded data redundancy: 68/393 objects degraded (17.303%), 16 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:25:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: osd.4 [v2:172.21.15.86:6808/1198497064,v1:172.21.15.86:6809/1198497064] boot 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: osdmap e74: 6 total, 6 up, 6 in 2024-04-06T14:25:14.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: Health check update: Degraded data redundancy: 68/393 objects degraded (17.303%), 16 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: osd.4 [v2:172.21.15.86:6808/1198497064,v1:172.21.15.86:6809/1198497064] boot 2024-04-06T14:25:15.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: osdmap e74: 6 total, 6 up, 6 in 2024-04-06T14:25:15.129 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T14:25:16.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:16 smithi032 ceph-mon[147414]: pgmap v142: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:16.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:16 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T14:25:16.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:16 smithi032 ceph-mon[147414]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-06T14:25:16.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:16 smithi032 ceph-mon[147414]: osdmap e75: 6 total, 6 up, 6 in 2024-04-06T14:25:16.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:16 smithi086 ceph-mon[110975]: pgmap v142: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 68/393 objects degraded (17.303%) 2024-04-06T14:25:16.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:16 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T14:25:16.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:16 smithi086 ceph-mon[110975]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-06T14:25:16.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:16 smithi086 ceph-mon[110975]: osdmap e75: 6 total, 6 up, 6 in 2024-04-06T14:25:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:18 smithi032 ceph-mon[147414]: pgmap v145: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/393 objects degraded (13.740%) 2024-04-06T14:25:18.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:18 smithi086 ceph-mon[110975]: pgmap v145: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/393 objects degraded (13.740%) 2024-04-06T14:25:19.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:19 smithi032 ceph-mon[147414]: Health check update: Degraded data redundancy: 54/393 objects degraded (13.740%), 12 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:19.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:19 smithi086 ceph-mon[110975]: Health check update: Degraded data redundancy: 54/393 objects degraded (13.740%), 12 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:20 smithi032 ceph-mon[147414]: pgmap v146: 65 pgs: 8 active+undersized+degraded, 8 active+undersized, 49 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 42/393 objects degraded (10.687%) 2024-04-06T14:25:20.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:20 smithi086 ceph-mon[110975]: pgmap v146: 65 pgs: 8 active+undersized+degraded, 8 active+undersized, 49 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 42/393 objects degraded (10.687%) 2024-04-06T14:25:21.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:21 smithi086 ceph-mon[110975]: pgmap v147: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:21.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:21 smithi086 ceph-mon[110975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/393 objects degraded (10.687%), 8 pgs degraded) 2024-04-06T14:25:21.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:21 smithi032 ceph-mon[147414]: pgmap v147: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:21.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:21 smithi032 ceph-mon[147414]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/393 objects degraded (10.687%), 8 pgs degraded) 2024-04-06T14:25:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:23 smithi032 ceph-mon[147414]: pgmap v148: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:25:23.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:23 smithi086 ceph-mon[110975]: pgmap v148: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:23.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:23.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:25:26.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:25 smithi086 ceph-mon[110975]: pgmap v149: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:25 smithi032 ceph-mon[147414]: pgmap v149: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:28.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:27 smithi086 ceph-mon[110975]: pgmap v150: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:28.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:27 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:28.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:27 smithi032 ceph-mon[147414]: pgmap v150: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:28.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:27 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.371+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/2487114255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 msgr2=0x7fdfd80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.371+0000 7fdfdcb8b700 1 --2- 172.21.15.32:0/2487114255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 0x7fdfd80fbd80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0004940 tx=0x7fdfc0009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.371+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/2487114255 shutdown_connections 2024-04-06T14:25:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.371+0000 7fdfdcb8b700 1 --2- 172.21.15.32:0/2487114255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 0x7fdfd80fbd80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.371+0000 7fdfdcb8b700 1 --2- 172.21.15.32:0/2487114255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd8101d70 0x7fdfd8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.371+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/2487114255 >> 172.21.15.32:0/2487114255 conn(0x7fdfd8072bc0 msgr2=0x7fdfd8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.371+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/2487114255 shutdown_connections 2024-04-06T14:25:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.371+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/2487114255 wait complete. 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.372+0000 7fdfdcb8b700 1 Processor -- start 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.372+0000 7fdfdcb8b700 1 -- start start 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.372+0000 7fdfdcb8b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd8101d70 0x7fdfd8103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.372+0000 7fdfdcb8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 0x7fdfd8103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.372+0000 7fdfdcb8b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd8103e10 con 0x7fdfd8102710 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.372+0000 7fdfdcb8b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd8103f50 con 0x7fdfd8101d70 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.372+0000 7fdfd5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 0x7fdfd8103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.372+0000 7fdfd5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 0x7fdfd8103880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37388/0 (socket says 172.21.15.32:37388) 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.372+0000 7fdfd5d9b700 1 -- 172.21.15.32:0/3545614050 learned_addr learned my addr 172.21.15.32:0/3545614050 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfd659c700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd8101d70 0x7fdfd8103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfd5d9b700 1 -- 172.21.15.32:0/3545614050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd8101d70 msgr2=0x7fdfd8103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfd5d9b700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd8101d70 0x7fdfd8103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfd5d9b700 1 -- 172.21.15.32:0/3545614050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc0005040 con 0x7fdfd8102710 2024-04-06T14:25:29.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfd659c700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd8101d70 0x7fdfd8103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:25:29.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfd5d9b700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 0x7fdfd8103880 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009380 tx=0x7fdfc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfceffd700 1 -- 172.21.15.32:0/3545614050 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc000e040 con 0x7fdfd8102710 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfceffd700 1 -- 172.21.15.32:0/3545614050 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfc0013420 con 0x7fdfd8102710 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd81041a0 con 0x7fdfd8102710 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfceffd700 1 -- 172.21.15.32:0/3545614050 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc001c630 con 0x7fdfd8102710 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.373+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd811a1d0 con 0x7fdfd8102710 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.374+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd80fd4c0 con 0x7fdfd8102710 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.375+0000 7fdfceffd700 1 -- 172.21.15.32:0/3545614050 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdfc00139e0 con 0x7fdfd8102710 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.375+0000 7fdfceffd700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdfc406cc20 0x7fdfc406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.375+0000 7fdfceffd700 1 -- 172.21.15.32:0/3545614050 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fdfc008d170 con 0x7fdfd8102710 2024-04-06T14:25:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.375+0000 7fdfd659c700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdfc406cc20 0x7fdfc406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.376+0000 7fdfd659c700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdfc406cc20 0x7fdfc406f0d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009fd0 tx=0x7fdfc8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:29.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.380+0000 7fdfceffd700 1 -- 172.21.15.32:0/3545614050 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdfc0057250 con 0x7fdfd8102710 2024-04-06T14:25:29.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.605+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfd8108d80 con 0x7fdfc406cc20 2024-04-06T14:25:29.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.610+0000 7fdfceffd700 1 -- 172.21.15.32:0/3545614050 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fdfd8108d80 con 0x7fdfc406cc20 2024-04-06T14:25:29.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdfc406cc20 msgr2=0x7fdfc406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:29.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdfc406cc20 0x7fdfc406f0d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009fd0 tx=0x7fdfc8009ee0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 msgr2=0x7fdfd8103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:29.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 0x7fdfd8103880 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009380 tx=0x7fdfc0009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:29.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 shutdown_connections 2024-04-06T14:25:29.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdfc406cc20 0x7fdfc406f0d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd8101d70 0x7fdfd8103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 --2- 172.21.15.32:0/3545614050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfd8102710 0x7fdfd8103880 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 >> 172.21.15.32:0/3545614050 conn(0x7fdfd8072bc0 msgr2=0x7fdfd81058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 shutdown_connections 2024-04-06T14:25:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.613+0000 7fdfdcb8b700 1 -- 172.21.15.32:0/3545614050 wait complete. 2024-04-06T14:25:29.625 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:25:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 -- 172.21.15.32:0/4283459232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff01026c0 msgr2=0x7f2ff00fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 --2- 172.21.15.32:0/4283459232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff01026c0 0x7f2ff00fbd30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0004940 tx=0x7f2fe0009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 -- 172.21.15.32:0/4283459232 shutdown_connections 2024-04-06T14:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 --2- 172.21.15.32:0/4283459232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff01026c0 0x7f2ff00fbd30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 --2- 172.21.15.32:0/4283459232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff0101d20 0x7f2ff01020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 -- 172.21.15.32:0/4283459232 >> 172.21.15.32:0/4283459232 conn(0x7f2ff00f78d0 msgr2=0x7f2ff00f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 -- 172.21.15.32:0/4283459232 shutdown_connections 2024-04-06T14:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 -- 172.21.15.32:0/4283459232 wait complete. 2024-04-06T14:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 Processor -- start 2024-04-06T14:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.719+0000 7f2ff7f37700 1 -- start start 2024-04-06T14:25:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff7f37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0101d20 0x7f2ff010eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff7f37700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff01026c0 0x7f2ff010f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff7f37700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff010f5f0 con 0x7f2ff0101d20 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff7f37700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff010f730 con 0x7f2ff01026c0 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff5cd3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0101d20 0x7f2ff010eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff5cd3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0101d20 0x7f2ff010eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37412/0 (socket says 172.21.15.32:37412) 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff5cd3700 1 -- 172.21.15.32:0/3802269589 learned_addr learned my addr 172.21.15.32:0/3802269589 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff54d2700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff01026c0 0x7f2ff010f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff5cd3700 1 -- 172.21.15.32:0/3802269589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff01026c0 msgr2=0x7f2ff010f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff5cd3700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff01026c0 0x7f2ff010f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff5cd3700 1 -- 172.21.15.32:0/3802269589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0005040 con 0x7f2ff0101d20 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff5cd3700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0101d20 0x7f2ff010eb70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00eb40 tx=0x7f2fec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2fe67fc700 1 -- 172.21.15.32:0/3802269589 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fec007cc0 con 0x7f2ff0101d20 2024-04-06T14:25:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2fe67fc700 1 -- 172.21.15.32:0/3802269589 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fec00f040 con 0x7f2ff0101d20 2024-04-06T14:25:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff0117a20 con 0x7f2ff0101d20 2024-04-06T14:25:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2fe67fc700 1 -- 172.21.15.32:0/3802269589 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fec00c990 con 0x7f2ff0101d20 2024-04-06T14:25:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.720+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff0117f80 con 0x7f2ff0101d20 2024-04-06T14:25:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.721+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff00fd470 con 0x7f2ff0101d20 2024-04-06T14:25:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.722+0000 7f2fe67fc700 1 -- 172.21.15.32:0/3802269589 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2fec00caf0 con 0x7f2ff0101d20 2024-04-06T14:25:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.722+0000 7f2fe67fc700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2fdc06cc30 0x7f2fdc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.723+0000 7f2ff54d2700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2fdc06cc30 0x7f2fdc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.723+0000 7f2fe67fc700 1 -- 172.21.15.32:0/3802269589 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2fec08baf0 con 0x7f2ff0101d20 2024-04-06T14:25:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.723+0000 7f2ff54d2700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2fdc06cc30 0x7f2fdc06f0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00668f0 tx=0x7f2fe0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.727+0000 7f2fe67fc700 1 -- 172.21.15.32:0/3802269589 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2fec055c80 con 0x7f2ff0101d20 2024-04-06T14:25:29.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.949+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ff004bf40 con 0x7f2fdc06cc30 2024-04-06T14:25:29.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.950+0000 7f2fe67fc700 1 -- 172.21.15.32:0/3802269589 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f2ff004bf40 con 0x7f2fdc06cc30 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2fdc06cc30 msgr2=0x7f2fdc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2fdc06cc30 0x7f2fdc06f0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00668f0 tx=0x7f2fe0009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0101d20 msgr2=0x7f2ff010eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0101d20 0x7f2ff010eb70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00eb40 tx=0x7f2fec00b040 comp rx=0 tx=0).stop 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 shutdown_connections 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2fdc06cc30 0x7f2fdc06f0e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0101d20 0x7f2ff010eb70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 --2- 172.21.15.32:0/3802269589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff01026c0 0x7f2ff010f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 >> 172.21.15.32:0/3802269589 conn(0x7f2ff00f78d0 msgr2=0x7f2ff00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 shutdown_connections 2024-04-06T14:25:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:29.952+0000 7f2ff7f37700 1 -- 172.21.15.32:0/3802269589 wait complete. 2024-04-06T14:25:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.056+0000 7fb911e88700 1 -- 172.21.15.32:0/2457169285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fde10 msgr2=0x7fb90c0fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.056+0000 7fb911e88700 1 --2- 172.21.15.32:0/2457169285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fde10 0x7fb90c0fe1e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc004940 tx=0x7fb8fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.056+0000 7fb911e88700 1 -- 172.21.15.32:0/2457169285 shutdown_connections 2024-04-06T14:25:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.056+0000 7fb911e88700 1 --2- 172.21.15.32:0/2457169285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.056+0000 7fb911e88700 1 --2- 172.21.15.32:0/2457169285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fde10 0x7fb90c0fe1e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.056+0000 7fb911e88700 1 -- 172.21.15.32:0/2457169285 >> 172.21.15.32:0/2457169285 conn(0x7fb90c0f7880 msgr2=0x7fb90c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.056+0000 7fb911e88700 1 -- 172.21.15.32:0/2457169285 shutdown_connections 2024-04-06T14:25:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.056+0000 7fb911e88700 1 -- 172.21.15.32:0/2457169285 wait complete. 2024-04-06T14:25:30.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.057+0000 7fb911e88700 1 Processor -- start 2024-04-06T14:25:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.057+0000 7fb911e88700 1 -- start start 2024-04-06T14:25:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb911e88700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb90c0fde10 0x7fb90c113d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb911e88700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c114290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb911e88700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c1148e0 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb911e88700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c10dcb0 con 0x7fb90c0fde10 2024-04-06T14:25:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb90affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c114290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb90affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c114290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37442/0 (socket says 172.21.15.32:37442) 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb90affd700 1 -- 172.21.15.32:0/3654351266 learned_addr learned my addr 172.21.15.32:0/3654351266 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb90b7fe700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb90c0fde10 0x7fb90c113d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb90affd700 1 -- 172.21.15.32:0/3654351266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb90c0fde10 msgr2=0x7fb90c113d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb90affd700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb90c0fde10 0x7fb90c113d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb90affd700 1 -- 172.21.15.32:0/3654351266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc005040 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb90affd700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c114290 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb90000eb40 tx=0x7fb90000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb8f3fff700 1 -- 172.21.15.32:0/3654351266 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb900007cc0 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c10df30 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb8f3fff700 1 -- 172.21.15.32:0/3654351266 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb90000f040 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb8f3fff700 1 -- 172.21.15.32:0/3654351266 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb90000c990 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.058+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c10e470 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.059+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c066770 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.060+0000 7fb8f3fff700 1 -- 172.21.15.32:0/3654351266 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb90000cc00 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.060+0000 7fb8f3fff700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb8f406cc30 0x7fb8f406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:30.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.060+0000 7fb8f3fff700 1 -- 172.21.15.32:0/3654351266 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb90008bc60 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.060+0000 7fb90b7fe700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb8f406cc30 0x7fb8f406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:30.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.061+0000 7fb90b7fe700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb8f406cc30 0x7fb8f406f0e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009fd0 tx=0x7fb8fc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:30.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.065+0000 7fb8f3fff700 1 -- 172.21.15.32:0/3654351266 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb900059760 con 0x7fb90c0fe7b0 2024-04-06T14:25:30.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.278+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb90c109d60 con 0x7fb8f406cc30 2024-04-06T14:25:30.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.288+0000 7fb8f3fff700 1 -- 172.21.15.32:0/3654351266 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb90c109d60 con 0x7fb8f406cc30 2024-04-06T14:25:30.289 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:25:30.289 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (5m) 87s ago 17m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:25:30.289 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (3m) 87s ago 17m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:25:30.289 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (3m) 20s ago 16m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:25:30.289 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (5m) 87s ago 16m 87.2M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:25:30.289 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (11m) 87s ago 11m 14.5M - 17.2.7-819-g6699495c ec300eaa768b 6a54464908b3 2024-04-06T14:25:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (11m) 87s ago 11m 129M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:25:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (11m) 20s ago 11m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:25:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (11m) 20s ago 11m 17.3M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:25:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (7m) 87s ago 18m 503M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:25:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (6m) 20s ago 16m 450M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:25:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (4m) 87s ago 18m 47.1M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:25:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (3m) 20s ago 16m 37.1M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:25:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (6m) 87s ago 17m 16.6M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:25:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (6m) 20s ago 16m 17.2M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:25:30.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (2m) 87s ago 15m 78.7M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:25:30.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (2m) 87s ago 14m 80.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:25:30.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (93s) 87s ago 14m 11.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:25:30.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (57s) 20s ago 13m 90.7M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:25:30.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (23s) 20s ago 13m 13.9M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:25:30.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (13m) 20s ago 12m 188M 2592M 17.2.7-819-g6699495c ec300eaa768b 0902d50f1d41 2024-04-06T14:25:30.291 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 87s ago 16m 59.8M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:25:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb8f406cc30 msgr2=0x7fb8f406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb8f406cc30 0x7fb8f406f0e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009fd0 tx=0x7fb8fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fe7b0 msgr2=0x7fb90c114290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c114290 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb90000eb40 tx=0x7fb90000b040 comp rx=0 tx=0).stop 2024-04-06T14:25:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 shutdown_connections 2024-04-06T14:25:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb8f406cc30 0x7fb8f406f0e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb90c0fde10 0x7fb90c113d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 --2- 172.21.15.32:0/3654351266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c114290 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 >> 172.21.15.32:0/3654351266 conn(0x7fb90c0f7880 msgr2=0x7fb90c0f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 shutdown_connections 2024-04-06T14:25:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.290+0000 7fb911e88700 1 -- 172.21.15.32:0/3654351266 wait complete. 2024-04-06T14:25:30.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:29 smithi086 ceph-mon[110975]: pgmap v151: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:30.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:29 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T14:25:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 -- 172.21.15.32:0/3696609001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0102710 msgr2=0x7f4fa00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 --2- 172.21.15.32:0/3696609001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0102710 0x7f4fa00fbd80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4f94004940 tx=0x7f4f94009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 -- 172.21.15.32:0/3696609001 shutdown_connections 2024-04-06T14:25:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 --2- 172.21.15.32:0/3696609001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0102710 0x7f4fa00fbd80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 --2- 172.21.15.32:0/3696609001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fa0101d70 0x7f4fa0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 -- 172.21.15.32:0/3696609001 >> 172.21.15.32:0/3696609001 conn(0x7f4fa0072bc0 msgr2=0x7f4fa0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 -- 172.21.15.32:0/3696609001 shutdown_connections 2024-04-06T14:25:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 -- 172.21.15.32:0/3696609001 wait complete. 2024-04-06T14:25:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 Processor -- start 2024-04-06T14:25:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.393+0000 7f4fa6653700 1 -- start start 2024-04-06T14:25:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4fa6653700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0101d70 0x7f4fa0113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4fa6653700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fa0102710 0x7f4fa0113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4fa6653700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa0114630 con 0x7f4fa0101d70 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4fa6653700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa010da20 con 0x7f4fa0102710 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0101d70 0x7f4fa0113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0101d70 0x7f4fa0113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37466/0 (socket says 172.21.15.32:37466) 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9ffff700 1 -- 172.21.15.32:0/2592535936 learned_addr learned my addr 172.21.15.32:0/2592535936 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9f7fe700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fa0102710 0x7f4fa0113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9ffff700 1 -- 172.21.15.32:0/2592535936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fa0102710 msgr2=0x7f4fa0113fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9ffff700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fa0102710 0x7f4fa0113fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9ffff700 1 -- 172.21.15.32:0/2592535936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f94005040 con 0x7f4fa0101d70 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9ffff700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0101d70 0x7f4fa0113aa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000eb70 tx=0x7f4f9000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9cff9700 1 -- 172.21.15.32:0/2592535936 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f90007cc0 con 0x7f4fa0101d70 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa010dca0 con 0x7f4fa0101d70 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9cff9700 1 -- 172.21.15.32:0/2592535936 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f9000f040 con 0x7f4fa0101d70 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.394+0000 7f4f9cff9700 1 -- 172.21.15.32:0/2592535936 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f9000c9f0 con 0x7f4fa0101d70 2024-04-06T14:25:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.395+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa010e230 con 0x7f4fa0101d70 2024-04-06T14:25:30.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.396+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa00fd4c0 con 0x7f4fa0101d70 2024-04-06T14:25:30.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.396+0000 7f4f9cff9700 1 -- 172.21.15.32:0/2592535936 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4f9000cb50 con 0x7f4fa0101d70 2024-04-06T14:25:30.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.396+0000 7f4f9cff9700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f8806cc20 0x7f4f8806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:30.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.396+0000 7f4f9cff9700 1 -- 172.21.15.32:0/2592535936 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f4f9008bb90 con 0x7f4fa0101d70 2024-04-06T14:25:30.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.396+0000 7f4f9f7fe700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f8806cc20 0x7f4f8806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:30.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.397+0000 7f4f9f7fe700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f8806cc20 0x7f4f8806f0d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400b0d0 tx=0x7f4f94009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:30.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.401+0000 7f4f9cff9700 1 -- 172.21.15.32:0/2592535936 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4f90059680 con 0x7f4fa0101d70 2024-04-06T14:25:30.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:29 smithi032 ceph-mon[147414]: pgmap v151: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:30.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:29 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T14:25:30.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.694+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4fa0108d30 con 0x7f4fa0101d70 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.694+0000 7f4f9cff9700 1 -- 172.21.15.32:0/2592535936 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f4f90059210 con 0x7f4fa0101d70 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:25:30.695 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 5 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 4 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 5, 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 9 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:25:30.696 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:25:30.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.696+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f8806cc20 msgr2=0x7f4f8806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.696+0000 7f4fa6653700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f8806cc20 0x7f4f8806f0d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400b0d0 tx=0x7f4f94009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:30.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.697+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0101d70 msgr2=0x7f4fa0113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.697+0000 7f4fa6653700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0101d70 0x7f4fa0113aa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000eb70 tx=0x7f4f9000b040 comp rx=0 tx=0).stop 2024-04-06T14:25:30.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.697+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 shutdown_connections 2024-04-06T14:25:30.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.697+0000 7f4fa6653700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f8806cc20 0x7f4f8806f0d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.697+0000 7f4fa6653700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa0101d70 0x7f4fa0113aa0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.697+0000 7f4fa6653700 1 --2- 172.21.15.32:0/2592535936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fa0102710 0x7f4fa0113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.697+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 >> 172.21.15.32:0/2592535936 conn(0x7f4fa0072bc0 msgr2=0x7f4fa0065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.697+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 shutdown_connections 2024-04-06T14:25:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.697+0000 7f4fa6653700 1 -- 172.21.15.32:0/2592535936 wait complete. 2024-04-06T14:25:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.799+0000 7f9a6fc35700 1 -- 172.21.15.32:0/4117884567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 msgr2=0x7f9a680fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.799+0000 7f9a6fc35700 1 --2- 172.21.15.32:0/4117884567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fb910 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9a58004940 tx=0x7f9a58009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.799+0000 7f9a6fc35700 1 -- 172.21.15.32:0/4117884567 shutdown_connections 2024-04-06T14:25:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.799+0000 7f9a6fc35700 1 --2- 172.21.15.32:0/4117884567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fb910 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.799+0000 7f9a6fc35700 1 --2- 172.21.15.32:0/4117884567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a68103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.800+0000 7f9a6fc35700 1 -- 172.21.15.32:0/4117884567 >> 172.21.15.32:0/4117884567 conn(0x7f9a680714c0 msgr2=0x7f9a680718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.800+0000 7f9a6fc35700 1 -- 172.21.15.32:0/4117884567 shutdown_connections 2024-04-06T14:25:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.800+0000 7f9a6fc35700 1 -- 172.21.15.32:0/4117884567 wait complete. 2024-04-06T14:25:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.800+0000 7f9a6fc35700 1 Processor -- start 2024-04-06T14:25:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.800+0000 7f9a6fc35700 1 -- start start 2024-04-06T14:25:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6fc35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a68113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6fc35700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a68113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6fc35700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a681145c0 con 0x7f9a680fb4a0 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6fc35700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6810d9b0 con 0x7f9a68102e70 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6d9d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a68113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6d9d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a68113a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37480/0 (socket says 172.21.15.32:37480) 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6d9d1700 1 -- 172.21.15.32:0/3647138808 learned_addr learned my addr 172.21.15.32:0/3647138808 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6d1d0700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a68113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6d1d0700 1 -- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 msgr2=0x7f9a68113a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6d1d0700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a68113a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6d1d0700 1 -- 172.21.15.32:0/3647138808 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a58005040 con 0x7f9a68102e70 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6d9d1700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a68113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a6d1d0700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a68113f70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800b0d0 tx=0x7f9a58009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.801+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/3647138808 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a5800e070 con 0x7f9a68102e70 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.802+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/3647138808 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a58013e00 con 0x7f9a68102e70 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.802+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6810dc30 con 0x7f9a68102e70 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.802+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/3647138808 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a5800bc30 con 0x7f9a68102e70 2024-04-06T14:25:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.802+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6810e1f0 con 0x7f9a68102e70 2024-04-06T14:25:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.803+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a68100110 con 0x7f9a68102e70 2024-04-06T14:25:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.803+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/3647138808 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9a58013950 con 0x7f9a68102e70 2024-04-06T14:25:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.803+0000 7f9a5e7fc700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:30.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.803+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/3647138808 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f9a5808c790 con 0x7f9a68102e70 2024-04-06T14:25:30.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.803+0000 7f9a6d9d1700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:30.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.804+0000 7f9a6d9d1700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9a640041d0 tx=0x7f9a6400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:30.808+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/3647138808 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9a5805a300 con 0x7f9a68102e70 2024-04-06T14:25:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.057+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9a6810f080 con 0x7f9a68102e70 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.057+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/3647138808 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1851 (secure 0 0 0) 0x7f9a58022070 con 0x7f9a68102e70 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-06T14:25:31.058 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:21:28.458361+0000 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:25:31.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14460,1=24301} 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{0:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/1704327767,v1:172.21.15.32:6829/1704327767] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{1:24301} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.86:6824/3246813202,v1:172.21.15.86:6825/3246813202] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:25:31.060 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:25:31.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:25:31.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:14472} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/179354627,v1:172.21.15.32:6827/179354627] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:25:31.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:25:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.059+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 msgr2=0x7f9a5406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.059+0000 7f9a6fc35700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9a640041d0 tx=0x7f9a6400b040 comp rx=0 tx=0).stop 2024-04-06T14:25:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.060+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 msgr2=0x7f9a68113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.060+0000 7f9a6fc35700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a68113f70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800b0d0 tx=0x7f9a58009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.060+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 shutdown_connections 2024-04-06T14:25:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.060+0000 7f9a6fc35700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.060+0000 7f9a6fc35700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a68113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.060+0000 7f9a6fc35700 1 --2- 172.21.15.32:0/3647138808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a68113f70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.060+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 >> 172.21.15.32:0/3647138808 conn(0x7f9a680714c0 msgr2=0x7f9a680fa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.060+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 shutdown_connections 2024-04-06T14:25:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.060+0000 7f9a6fc35700 1 -- 172.21.15.32:0/3647138808 wait complete. 2024-04-06T14:25:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-06T14:25:31.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:31 smithi086 ceph-mon[110975]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T14:25:31.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:31 smithi086 ceph-mon[110975]: Upgrade: osd.5 is safe to restart 2024-04-06T14:25:31.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:31 smithi086 ceph-mon[110975]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:31.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:31 smithi086 ceph-mon[110975]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:31.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:31 smithi086 ceph-mon[110975]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:31.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:31 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2592535936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.165+0000 7fc098062700 1 -- 172.21.15.32:0/1379439268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 msgr2=0x7fc0900fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.165+0000 7fc098062700 1 --2- 172.21.15.32:0/1379439268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 0x7fc0900fe260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc08c004940 tx=0x7fc08c009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.165+0000 7fc098062700 1 -- 172.21.15.32:0/1379439268 shutdown_connections 2024-04-06T14:25:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.165+0000 7fc098062700 1 --2- 172.21.15.32:0/1379439268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0900fe830 0x7fc090106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.165+0000 7fc098062700 1 --2- 172.21.15.32:0/1379439268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 0x7fc0900fe260 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.165+0000 7fc098062700 1 -- 172.21.15.32:0/1379439268 >> 172.21.15.32:0/1379439268 conn(0x7fc090072bc0 msgr2=0x7fc090072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.165+0000 7fc098062700 1 -- 172.21.15.32:0/1379439268 shutdown_connections 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.165+0000 7fc098062700 1 -- 172.21.15.32:0/1379439268 wait complete. 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc098062700 1 Processor -- start 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc098062700 1 -- start start 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc098062700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 0x7fc0900fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc098062700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0900fe830 0x7fc0900fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc098062700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0900fc820 con 0x7fc0900fde90 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc098062700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0900fc960 con 0x7fc0900fe830 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc095dfe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 0x7fc0900fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc095dfe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 0x7fc0900fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37490/0 (socket says 172.21.15.32:37490) 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc095dfe700 1 -- 172.21.15.32:0/550579446 learned_addr learned my addr 172.21.15.32:0/550579446 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.166+0000 7fc0955fd700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0900fe830 0x7fc0900fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.167+0000 7fc095dfe700 1 -- 172.21.15.32:0/550579446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0900fe830 msgr2=0x7fc0900fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.167+0000 7fc095dfe700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0900fe830 0x7fc0900fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.167+0000 7fc095dfe700 1 -- 172.21.15.32:0/550579446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c005040 con 0x7fc0900fde90 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.167+0000 7fc095dfe700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 0x7fc0900fbd50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00b3b0 tx=0x7fc08c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.167+0000 7fc0867fc700 1 -- 172.21.15.32:0/550579446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc08c00e040 con 0x7fc0900fde90 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.167+0000 7fc0867fc700 1 -- 172.21.15.32:0/550579446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc08c013420 con 0x7fc0900fde90 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.167+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0900fcbb0 con 0x7fc0900fde90 2024-04-06T14:25:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.167+0000 7fc0867fc700 1 -- 172.21.15.32:0/550579446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc08c01c5f0 con 0x7fc0900fde90 2024-04-06T14:25:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.167+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09011a1b0 con 0x7fc0900fde90 2024-04-06T14:25:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.168+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc090065e30 con 0x7fc0900fde90 2024-04-06T14:25:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.169+0000 7fc0867fc700 1 -- 172.21.15.32:0/550579446 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc08c013590 con 0x7fc0900fde90 2024-04-06T14:25:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.169+0000 7fc0867fc700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07c06cc20 0x7fc07c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.169+0000 7fc0867fc700 1 -- 172.21.15.32:0/550579446 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc08c08d1b0 con 0x7fc0900fde90 2024-04-06T14:25:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.169+0000 7fc0955fd700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07c06cc20 0x7fc07c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.170+0000 7fc0955fd700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07c06cc20 0x7fc07c06f0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc0900fd690 tx=0x7fc08000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:31.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.174+0000 7fc0867fc700 1 -- 172.21.15.32:0/550579446 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc08c057330 con 0x7fc0900fde90 2024-04-06T14:25:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.393+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc09004bf70 con 0x7fc07c06cc20 2024-04-06T14:25:31.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.394+0000 7fc0867fc700 1 -- 172.21.15.32:0/550579446 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fc08000f040 con 0x7fc07c06cc20 2024-04-06T14:25:31.395 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:25:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:25:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:25:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:25:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:25:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:25:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:25:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:25:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:25:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "11/21 daemons upgraded", 2024-04-06T14:25:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T14:25:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:25:31.397 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:25:31.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07c06cc20 msgr2=0x7fc07c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07c06cc20 0x7fc07c06f0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc0900fd690 tx=0x7fc08000b040 comp rx=0 tx=0).stop 2024-04-06T14:25:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 msgr2=0x7fc0900fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 0x7fc0900fbd50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00b3b0 tx=0x7fc08c009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 shutdown_connections 2024-04-06T14:25:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07c06cc20 0x7fc07c06f0d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0900fde90 0x7fc0900fbd50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 --2- 172.21.15.32:0/550579446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0900fe830 0x7fc0900fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 >> 172.21.15.32:0/550579446 conn(0x7fc090072bc0 msgr2=0x7fc090103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:31.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 shutdown_connections 2024-04-06T14:25:31.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.396+0000 7fc098062700 1 -- 172.21.15.32:0/550579446 wait complete. 2024-04-06T14:25:31.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:31 smithi032 ceph-mon[147414]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T14:25:31.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:31 smithi032 ceph-mon[147414]: Upgrade: osd.5 is safe to restart 2024-04-06T14:25:31.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:31 smithi032 ceph-mon[147414]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:31.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:31 smithi032 ceph-mon[147414]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:31.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:31 smithi032 ceph-mon[147414]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:31.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:31 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2592535936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:31.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.500+0000 7f3849286700 1 -- 172.21.15.32:0/2274118593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 msgr2=0x7f3844102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.500+0000 7f3849286700 1 --2- 172.21.15.32:0/2274118593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 0x7f3844102140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3834004940 tx=0x7f3834009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.500+0000 7f3849286700 1 -- 172.21.15.32:0/2274118593 shutdown_connections 2024-04-06T14:25:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.500+0000 7f3849286700 1 --2- 172.21.15.32:0/2274118593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3844102710 0x7f38440fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.500+0000 7f3849286700 1 --2- 172.21.15.32:0/2274118593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 0x7f3844102140 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.500+0000 7f3849286700 1 -- 172.21.15.32:0/2274118593 >> 172.21.15.32:0/2274118593 conn(0x7f3844072bc0 msgr2=0x7f3844072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.500+0000 7f3849286700 1 -- 172.21.15.32:0/2274118593 shutdown_connections 2024-04-06T14:25:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.500+0000 7f3849286700 1 -- 172.21.15.32:0/2274118593 wait complete. 2024-04-06T14:25:31.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3849286700 1 Processor -- start 2024-04-06T14:25:31.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3849286700 1 -- start start 2024-04-06T14:25:31.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3849286700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 0x7f38441139b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:31.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3849286700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3844102710 0x7f3844113ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3849286700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3844114540 con 0x7f3844101d70 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3849286700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384410d970 con 0x7f3844102710 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3842ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 0x7f38441139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3842ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 0x7f38441139b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37500/0 (socket says 172.21.15.32:37500) 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3842ffd700 1 -- 172.21.15.32:0/3998232500 learned_addr learned my addr 172.21.15.32:0/3998232500 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f383a7fc700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3844102710 0x7f3844113ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3842ffd700 1 -- 172.21.15.32:0/3998232500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3844102710 msgr2=0x7f3844113ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3842ffd700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3844102710 0x7f3844113ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f3842ffd700 1 -- 172.21.15.32:0/3998232500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3834005040 con 0x7f3844101d70 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.501+0000 7f383a7fc700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3844102710 0x7f3844113ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.502+0000 7f3842ffd700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 0x7f38441139b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3834008d80 tx=0x7f3834009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:31.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.502+0000 7f383bfff700 1 -- 172.21.15.32:0/3998232500 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f383400e030 con 0x7f3844101d70 2024-04-06T14:25:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.502+0000 7f383bfff700 1 -- 172.21.15.32:0/3998232500 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3834013de0 con 0x7f3844101d70 2024-04-06T14:25:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.502+0000 7f383bfff700 1 -- 172.21.15.32:0/3998232500 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f383401cd60 con 0x7f3844101d70 2024-04-06T14:25:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.502+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384410dbf0 con 0x7f3844101d70 2024-04-06T14:25:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.502+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384410e1b0 con 0x7f3844101d70 2024-04-06T14:25:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.503+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38440fd4c0 con 0x7f3844101d70 2024-04-06T14:25:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.503+0000 7f383bfff700 1 -- 172.21.15.32:0/3998232500 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f383401c590 con 0x7f3844101d70 2024-04-06T14:25:31.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.504+0000 7f383bfff700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f383006cc30 0x7f383006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:25:31.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.504+0000 7f383bfff700 1 -- 172.21.15.32:0/3998232500 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f383408d3b0 con 0x7f3844101d70 2024-04-06T14:25:31.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.504+0000 7f383a7fc700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f383006cc30 0x7f383006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:25:31.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.504+0000 7f383a7fc700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f383006cc30 0x7f383006f0e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f382c004600 tx=0x7f382c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:25:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.508+0000 7f383bfff700 1 -- 172.21.15.32:0/3998232500 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f383405ae80 con 0x7f3844101d70 2024-04-06T14:25:31.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.798+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f384404bed0 con 0x7f3844101d70 2024-04-06T14:25:31.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.799+0000 7f383bfff700 1 -- 172.21.15.32:0/3998232500 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f383405aa10 con 0x7f3844101d70 2024-04-06T14:25:31.799 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:25:31.799 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:25:31.799 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:25:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f383006cc30 msgr2=0x7f383006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f383006cc30 0x7f383006f0e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f382c004600 tx=0x7f382c00b040 comp rx=0 tx=0).stop 2024-04-06T14:25:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 msgr2=0x7f38441139b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:25:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 0x7f38441139b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3834008d80 tx=0x7f3834009e30 comp rx=0 tx=0).stop 2024-04-06T14:25:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 shutdown_connections 2024-04-06T14:25:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f383006cc30 0x7f383006f0e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3844101d70 0x7f38441139b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 --2- 172.21.15.32:0/3998232500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3844102710 0x7f3844113ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:25:31.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 >> 172.21.15.32:0/3998232500 conn(0x7f3844072bc0 msgr2=0x7f3844065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:25:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 shutdown_connections 2024-04-06T14:25:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:25:31.801+0000 7f3849286700 1 -- 172.21.15.32:0/3998232500 wait complete. 2024-04-06T14:25:32.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:32 smithi086 ceph-mon[110975]: pgmap v152: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:32 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3647138808' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:25:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:32 smithi086 ceph-mon[110975]: Upgrade: Updating osd.5 2024-04-06T14:25:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:32 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:32 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T14:25:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:32 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:25:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:32 smithi086 ceph-mon[110975]: Deploying daemon osd.5 on smithi086 2024-04-06T14:25:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:32 smithi086 ceph-mon[110975]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:32.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:32 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3998232500' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:25:32.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:32 smithi032 ceph-mon[147414]: pgmap v152: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:32.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:32 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3647138808' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:25:32.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:32 smithi032 ceph-mon[147414]: Upgrade: Updating osd.5 2024-04-06T14:25:32.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:32 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:32.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:32 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T14:25:32.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:32 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:25:32.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:32 smithi032 ceph-mon[147414]: Deploying daemon osd.5 on smithi086 2024-04-06T14:25:32.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:32 smithi032 ceph-mon[147414]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:25:32.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:32 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3998232500' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:25:34.290 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:34 smithi086 ceph-mon[110975]: pgmap v153: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:34.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:34 smithi032 ceph-mon[147414]: pgmap v153: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:34.545 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:34 smithi086 systemd[1]: Stopping Ceph osd.5 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:25:34.877 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:34 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[71241]: 2024-04-06T14:25:34.543+0000 7f6d4c6d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T14:25:34.877 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:34 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[71241]: 2024-04-06T14:25:34.543+0000 7f6d4c6d6700 -1 osd.5 75 *** Got signal Terminated *** 2024-04-06T14:25:34.878 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:34 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[71241]: 2024-04-06T14:25:34.543+0000 7f6d4c6d6700 -1 osd.5 75 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T14:25:35.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:35 smithi086 ceph-mon[110975]: osd.5 marked itself down and dead 2024-04-06T14:25:35.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:35 smithi032 ceph-mon[147414]: osd.5 marked itself down and dead 2024-04-06T14:25:36.012 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:35 smithi086 bash[135617]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5 2024-04-06T14:25:36.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:36 smithi086 ceph-mon[110975]: pgmap v154: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:36.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:36 smithi086 ceph-mon[110975]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:25:36.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:36 smithi086 ceph-mon[110975]: osdmap e76: 6 total, 5 up, 6 in 2024-04-06T14:25:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:36 smithi032 ceph-mon[147414]: pgmap v154: 65 pgs: 65 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:36 smithi032 ceph-mon[147414]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T14:25:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:36 smithi032 ceph-mon[147414]: osdmap e76: 6 total, 5 up, 6 in 2024-04-06T14:25:37.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:37 smithi086 ceph-mon[110975]: osdmap e77: 6 total, 5 up, 6 in 2024-04-06T14:25:37.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:37 smithi032 ceph-mon[147414]: osdmap e77: 6 total, 5 up, 6 in 2024-04-06T14:25:37.878 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:37 smithi086 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.5.service: Succeeded. 2024-04-06T14:25:37.878 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:37 smithi086 systemd[1]: Stopped Ceph osd.5 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:25:37.878 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:37 smithi086 systemd[1]: Starting Ceph osd.5 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T14:25:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:38 smithi086 ceph-mon[110975]: pgmap v157: 65 pgs: 3 peering, 12 stale+active+clean, 50 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:38 smithi086 ceph-mon[110975]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-06T14:25:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:25:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:38 smithi032 ceph-mon[147414]: pgmap v157: 65 pgs: 3 peering, 12 stale+active+clean, 50 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:38 smithi032 ceph-mon[147414]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-06T14:25:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:38.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:25:39.051 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:38 smithi086 bash[135899]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T14:25:39.051 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:38 smithi086 bash[135899]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T14:25:39.051 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:38 smithi086 bash[135899]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T14:25:39.051 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:38 smithi086 bash[135899]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T14:25:39.051 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:38 smithi086 bash[135899]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-06T14:25:39.051 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:38 smithi086 bash[135899]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T14:25:39.051 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:38 smithi086 bash[135899]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-06T14:25:40.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:39 smithi086 ceph-mon[110975]: pgmap v158: 65 pgs: 6 active+undersized+degraded, 3 peering, 5 stale+active+clean, 6 active+undersized, 45 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 19/393 objects degraded (4.835%) 2024-04-06T14:25:40.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:39 smithi086 ceph-mon[110975]: Health check failed: Degraded data redundancy: 19/393 objects degraded (4.835%), 6 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:40.378 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:39 smithi086 ceph-osd[136221]: -- 172.21.15.86:0/3045399179 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564189edb1e0 con 0x564189eab000 2024-04-06T14:25:40.378 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:40 smithi086 bash[136184]: c6fe4d60741d63439b5f0d7599847d9f3f8a1ab9d5f939b959b0e8479c6629df 2024-04-06T14:25:40.378 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:40 smithi086 systemd[1]: Started Ceph osd.5 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T14:25:40.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:39 smithi032 ceph-mon[147414]: pgmap v158: 65 pgs: 6 active+undersized+degraded, 3 peering, 5 stale+active+clean, 6 active+undersized, 45 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 19/393 objects degraded (4.835%) 2024-04-06T14:25:40.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:39 smithi032 ceph-mon[147414]: Health check failed: Degraded data redundancy: 19/393 objects degraded (4.835%), 6 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:40.809 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:40 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T14:25:40.417+0000 7f52b60ee640 -1 Falling back to public interface 2024-04-06T14:25:41.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:41.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:42.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:42 smithi086 ceph-mon[110975]: pgmap v159: 65 pgs: 12 active+undersized+degraded, 3 peering, 13 active+undersized, 37 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 51/393 objects degraded (12.977%) 2024-04-06T14:25:42.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:42 smithi032 ceph-mon[147414]: pgmap v159: 65 pgs: 12 active+undersized+degraded, 3 peering, 13 active+undersized, 37 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 51/393 objects degraded (12.977%) 2024-04-06T14:25:43.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:43 smithi086 ceph-mon[110975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-06T14:25:43.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:43 smithi032 ceph-mon[147414]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-06T14:25:44.434 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:44 smithi086 ceph-mon[110975]: pgmap v160: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 58/393 objects degraded (14.758%) 2024-04-06T14:25:44.434 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:44 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:44.434 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:44 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:44.434 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:44 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:44.434 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:44 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:44.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:44 smithi032 ceph-mon[147414]: pgmap v160: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 58/393 objects degraded (14.758%) 2024-04-06T14:25:44.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:44 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:44.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:44 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:44.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:44 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:44.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:44 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:44.877 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:44 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T14:25:44.592+0000 7f52b60ee640 -1 osd.5 75 log_to_monitors true 2024-04-06T14:25:45.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:45 smithi086 ceph-mon[110975]: from='osd.5 [v2:172.21.15.86:6816/3374872376,v1:172.21.15.86:6817/3374872376]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T14:25:45.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:45 smithi086 ceph-mon[110975]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T14:25:45.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:45 smithi032 ceph-mon[147414]: from='osd.5 [v2:172.21.15.86:6816/3374872376,v1:172.21.15.86:6817/3374872376]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T14:25:45.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:45 smithi032 ceph-mon[147414]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T14:25:46.378 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 14:25:46 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T14:25:46.042+0000 7f52a353f700 -1 osd.5 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T14:25:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:46 smithi086 ceph-mon[110975]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 58/393 objects degraded (14.758%) 2024-04-06T14:25:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:46 smithi086 ceph-mon[110975]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T14:25:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:46 smithi086 ceph-mon[110975]: osdmap e78: 6 total, 5 up, 6 in 2024-04-06T14:25:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:46 smithi086 ceph-mon[110975]: from='osd.5 [v2:172.21.15.86:6816/3374872376,v1:172.21.15.86:6817/3374872376]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:25:46.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:46 smithi086 ceph-mon[110975]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:25:46.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:46 smithi086 ceph-mon[110975]: from='osd.5 ' entity='osd.5' 2024-04-06T14:25:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:46 smithi032 ceph-mon[147414]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 351 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 58/393 objects degraded (14.758%) 2024-04-06T14:25:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:46 smithi032 ceph-mon[147414]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T14:25:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:46 smithi032 ceph-mon[147414]: osdmap e78: 6 total, 5 up, 6 in 2024-04-06T14:25:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:46 smithi032 ceph-mon[147414]: from='osd.5 [v2:172.21.15.86:6816/3374872376,v1:172.21.15.86:6817/3374872376]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:25:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:46 smithi032 ceph-mon[147414]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-04-06T14:25:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:46 smithi032 ceph-mon[147414]: from='osd.5 ' entity='osd.5' 2024-04-06T14:25:47.440 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:47 smithi086 ceph-mon[110975]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:25:47.440 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:47 smithi086 ceph-mon[110975]: osd.5 [v2:172.21.15.86:6816/3374872376,v1:172.21.15.86:6817/3374872376] boot 2024-04-06T14:25:47.440 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:47 smithi086 ceph-mon[110975]: osdmap e79: 6 total, 6 up, 6 in 2024-04-06T14:25:47.440 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:47 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:25:47.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:47 smithi032 ceph-mon[147414]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T14:25:47.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:47 smithi032 ceph-mon[147414]: osd.5 [v2:172.21.15.86:6816/3374872376,v1:172.21.15.86:6817/3374872376] boot 2024-04-06T14:25:47.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:47 smithi032 ceph-mon[147414]: osdmap e79: 6 total, 6 up, 6 in 2024-04-06T14:25:47.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:47 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T14:25:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: pgmap v163: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 58/393 objects degraded (14.758%) 2024-04-06T14:25:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:25:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:25:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-06T14:25:48.379 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: osdmap e80: 6 total, 6 up, 6 in 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: osdmap e81: 6 total, 6 up, 6 in 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-06T14:25:48.380 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:48 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-06T14:25:48.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:25:48.052+0000 7fa221483700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T14:25:48.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: pgmap v163: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 58/393 objects degraded (14.758%) 2024-04-06T14:25:48.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-06T14:25:48.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: osdmap e80: 6 total, 6 up, 6 in 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T14:25:48.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: osdmap e81: 6 total, 6 up, 6 in 2024-04-06T14:25:48.483 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-06T14:25:48.483 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:48 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-06T14:25:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:49 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all osd 2024-04-06T14:25:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:49 smithi032 ceph-mon[147414]: Upgrade: Setting require_osd_release to 18 reef 2024-04-06T14:25:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:49 smithi032 ceph-mon[147414]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-06T14:25:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:49 smithi032 ceph-mon[147414]: Health check update: Degraded data redundancy: 58/393 objects degraded (14.758%), 14 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:49 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.86:6824/4145187217,v1:172.21.15.86:6825/4145187217] up:boot 2024-04-06T14:25:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:49 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6828/629297924,v1:172.21.15.32:6829/629297924] up:boot 2024-04-06T14:25:49.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:49 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:25:49.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:49 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:25:49.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:49 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:25:49.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:49 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all osd 2024-04-06T14:25:49.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:49 smithi086 ceph-mon[110975]: Upgrade: Setting require_osd_release to 18 reef 2024-04-06T14:25:49.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:49 smithi086 ceph-mon[110975]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-06T14:25:49.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:49 smithi086 ceph-mon[110975]: Health check update: Degraded data redundancy: 58/393 objects degraded (14.758%), 14 pgs degraded (PG_DEGRADED) 2024-04-06T14:25:49.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:49 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.86:6824/4145187217,v1:172.21.15.86:6825/4145187217] up:boot 2024-04-06T14:25:49.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:49 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6828/629297924,v1:172.21.15.32:6829/629297924] up:boot 2024-04-06T14:25:49.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:49 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:25:49.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:49 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:25:49.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:49 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:25:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:50 smithi032 ceph-mon[147414]: pgmap v167: 65 pgs: 5 peering, 11 active+undersized+degraded, 12 active+undersized, 37 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 50/393 objects degraded (12.723%) 2024-04-06T14:25:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.gcjdjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:25:50.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:25:50.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:50 smithi086 ceph-mon[110975]: pgmap v167: 65 pgs: 5 peering, 11 active+undersized+degraded, 12 active+undersized, 37 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 50/393 objects degraded (12.723%) 2024-04-06T14:25:50.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:50.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.gcjdjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:25:50.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:25:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:51 smithi032 ceph-mon[147414]: Upgrade: Updating mds.cephfs.smithi032.gcjdjo 2024-04-06T14:25:51.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:51 smithi032 ceph-mon[147414]: Deploying daemon mds.cephfs.smithi032.gcjdjo on smithi032 2024-04-06T14:25:51.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:51 smithi086 ceph-mon[110975]: Upgrade: Updating mds.cephfs.smithi032.gcjdjo 2024-04-06T14:25:51.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:51 smithi086 ceph-mon[110975]: Deploying daemon mds.cephfs.smithi032.gcjdjo on smithi032 2024-04-06T14:25:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:52 smithi032 ceph-mon[147414]: pgmap v168: 65 pgs: 5 peering, 1 active+undersized+degraded, 1 active+undersized, 58 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 4/393 objects degraded (1.018%) 2024-04-06T14:25:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:52 smithi032 ceph-mon[147414]: osdmap e82: 6 total, 6 up, 6 in 2024-04-06T14:25:52.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:52 smithi086 ceph-mon[110975]: pgmap v168: 65 pgs: 5 peering, 1 active+undersized+degraded, 1 active+undersized, 58 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 4/393 objects degraded (1.018%) 2024-04-06T14:25:52.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:52 smithi086 ceph-mon[110975]: osdmap e82: 6 total, 6 up, 6 in 2024-04-06T14:25:53.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:53 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-06T14:25:53.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:53.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:25:53.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:53 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-06T14:25:53.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:53.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:25:54.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:54 smithi032 ceph-mon[147414]: pgmap v170: 65 pgs: 5 peering, 60 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:54.495 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:54 smithi032 ceph-mon[147414]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/393 objects degraded (1.018%), 1 pg degraded) 2024-04-06T14:25:54.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:54 smithi086 ceph-mon[110975]: pgmap v170: 65 pgs: 5 peering, 60 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:54.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:54 smithi086 ceph-mon[110975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/393 objects degraded (1.018%), 1 pg degraded) 2024-04-06T14:25:55.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:55 smithi032 ceph-mon[147414]: pgmap v171: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:55.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:55.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:55.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:55.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:55 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:boot 2024-04-06T14:25:55.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:55 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:25:55.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:25:56.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:55 smithi086 ceph-mon[110975]: pgmap v171: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:25:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:55 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:boot 2024-04-06T14:25:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:55 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:25:56.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:25:58.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:57 smithi086 ceph-mon[110975]: pgmap v172: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:58.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:25:57 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:25:58.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:57 smithi032 ceph-mon[147414]: pgmap v172: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:25:58.134 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:25:57 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:00.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:00 smithi032 ceph-mon[147414]: pgmap v173: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:00.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:00 smithi086 ceph-mon[110975]: pgmap v173: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:01.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:01 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:01.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:01 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:01.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:01 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:01.571 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:01 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:01.578 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:01 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:01.578 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:01 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:01.578 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:01 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:01.578 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:01 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:01.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.914+0000 7fb3ba722700 1 -- 172.21.15.32:0/142506368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4102710 msgr2=0x7fb3b40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.914+0000 7fb3ba722700 1 --2- 172.21.15.32:0/142506368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4102710 0x7fb3b40fbd80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8004940 tx=0x7fb3a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.914+0000 7fb3ba722700 1 -- 172.21.15.32:0/142506368 shutdown_connections 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.914+0000 7fb3ba722700 1 --2- 172.21.15.32:0/142506368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4102710 0x7fb3b40fbd80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.914+0000 7fb3ba722700 1 --2- 172.21.15.32:0/142506368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3b4101d70 0x7fb3b4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.914+0000 7fb3ba722700 1 -- 172.21.15.32:0/142506368 >> 172.21.15.32:0/142506368 conn(0x7fb3b4072bc0 msgr2=0x7fb3b4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.914+0000 7fb3ba722700 1 -- 172.21.15.32:0/142506368 shutdown_connections 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.914+0000 7fb3ba722700 1 -- 172.21.15.32:0/142506368 wait complete. 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.914+0000 7fb3ba722700 1 Processor -- start 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3ba722700 1 -- start start 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3ba722700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4101d70 0x7fb3b4113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3ba722700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3b4102710 0x7fb3b4114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3ba722700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4114670 con 0x7fb3b4101d70 2024-04-06T14:26:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3ba722700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b410da60 con 0x7fb3b4102710 2024-04-06T14:26:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4101d70 0x7fb3b4113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4101d70 0x7fb3b4113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32964/0 (socket says 172.21.15.32:32964) 2024-04-06T14:26:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3b3fff700 1 -- 172.21.15.32:0/2854987955 learned_addr learned my addr 172.21.15.32:0/2854987955 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3b37fe700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3b4102710 0x7fb3b4114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3b3fff700 1 -- 172.21.15.32:0/2854987955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3b4102710 msgr2=0x7fb3b4114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3b3fff700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3b4102710 0x7fb3b4114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.915+0000 7fb3b3fff700 1 -- 172.21.15.32:0/2854987955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8005040 con 0x7fb3b4101d70 2024-04-06T14:26:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.916+0000 7fb3b3fff700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4101d70 0x7fb3b4113ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4002a50 tx=0x7fb3a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.916+0000 7fb3b0ff9700 1 -- 172.21.15.32:0/2854987955 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a4007cc0 con 0x7fb3b4101d70 2024-04-06T14:26:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.916+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b410dd40 con 0x7fb3b4101d70 2024-04-06T14:26:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.916+0000 7fb3b0ff9700 1 -- 172.21.15.32:0/2854987955 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3a400f040 con 0x7fb3b4101d70 2024-04-06T14:26:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.916+0000 7fb3b0ff9700 1 -- 172.21.15.32:0/2854987955 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a400d9c0 con 0x7fb3b4101d70 2024-04-06T14:26:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.916+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b410e2d0 con 0x7fb3b4101d70 2024-04-06T14:26:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.917+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b40fd4c0 con 0x7fb3b4101d70 2024-04-06T14:26:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.917+0000 7fb3b0ff9700 1 -- 172.21.15.32:0/2854987955 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb3a400db60 con 0x7fb3b4101d70 2024-04-06T14:26:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.918+0000 7fb3b0ff9700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb39c06cc30 0x7fb39c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.918+0000 7fb3b0ff9700 1 -- 172.21.15.32:0/2854987955 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb3a408b6f0 con 0x7fb3b4101d70 2024-04-06T14:26:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.918+0000 7fb3b37fe700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb39c06cc30 0x7fb39c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.918+0000 7fb3b37fe700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb39c06cc30 0x7fb39c06f0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb3b410ed40 tx=0x7fb3a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:01.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:01.923+0000 7fb3b0ff9700 1 -- 172.21.15.32:0/2854987955 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb3a4059140 con 0x7fb3b4101d70 2024-04-06T14:26:02.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.150+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3b4108da0 con 0x7fb39c06cc30 2024-04-06T14:26:02.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.152+0000 7fb3b0ff9700 1 -- 172.21.15.32:0/2854987955 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb3b4108da0 con 0x7fb39c06cc30 2024-04-06T14:26:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.153+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb39c06cc30 msgr2=0x7fb39c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.153+0000 7fb3ba722700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb39c06cc30 0x7fb39c06f0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb3b410ed40 tx=0x7fb3a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.154+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4101d70 msgr2=0x7fb3b4113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.154+0000 7fb3ba722700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4101d70 0x7fb3b4113ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4002a50 tx=0x7fb3a40046b0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.154+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 shutdown_connections 2024-04-06T14:26:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.154+0000 7fb3ba722700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb39c06cc30 0x7fb39c06f0e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.154+0000 7fb3ba722700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b4101d70 0x7fb3b4113ae0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.154+0000 7fb3ba722700 1 --2- 172.21.15.32:0/2854987955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3b4102710 0x7fb3b4114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.154+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 >> 172.21.15.32:0/2854987955 conn(0x7fb3b4072bc0 msgr2=0x7fb3b4065a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.154+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 shutdown_connections 2024-04-06T14:26:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.154+0000 7fb3ba722700 1 -- 172.21.15.32:0/2854987955 wait complete. 2024-04-06T14:26:02.164 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:26:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.260+0000 7f993a2d4700 1 -- 172.21.15.32:0/3632433446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 msgr2=0x7f9934102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.260+0000 7f993a2d4700 1 --2- 172.21.15.32:0/3632433446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 0x7f9934102140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f991c004940 tx=0x7f991c009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.260+0000 7f993a2d4700 1 -- 172.21.15.32:0/3632433446 shutdown_connections 2024-04-06T14:26:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.260+0000 7f993a2d4700 1 --2- 172.21.15.32:0/3632433446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9934102710 0x7f99340fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.260+0000 7f993a2d4700 1 --2- 172.21.15.32:0/3632433446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 0x7f9934102140 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.260+0000 7f993a2d4700 1 -- 172.21.15.32:0/3632433446 >> 172.21.15.32:0/3632433446 conn(0x7f9934072bc0 msgr2=0x7f9934072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.260+0000 7f993a2d4700 1 -- 172.21.15.32:0/3632433446 shutdown_connections 2024-04-06T14:26:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.260+0000 7f993a2d4700 1 -- 172.21.15.32:0/3632433446 wait complete. 2024-04-06T14:26:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.260+0000 7f993a2d4700 1 Processor -- start 2024-04-06T14:26:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f993a2d4700 1 -- start start 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f993a2d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 0x7f993410e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f993a2d4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9934102710 0x7f993410ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f993a2d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993410f550 con 0x7f9934101d70 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f993a2d4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993410f690 con 0x7f9934102710 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f9933fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 0x7f993410e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f9933fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 0x7f993410e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32984/0 (socket says 172.21.15.32:32984) 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f9933fff700 1 -- 172.21.15.32:0/584120394 learned_addr learned my addr 172.21.15.32:0/584120394 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f99337fe700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9934102710 0x7f993410ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f9933fff700 1 -- 172.21.15.32:0/584120394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9934102710 msgr2=0x7f993410ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f9933fff700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9934102710 0x7f993410ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f9933fff700 1 -- 172.21.15.32:0/584120394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c005040 con 0x7f9934101d70 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f9933fff700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 0x7f993410e9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f991c000c00 tx=0x7f991c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.261+0000 7f9930ff9700 1 -- 172.21.15.32:0/584120394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c021070 con 0x7f9934101d70 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.262+0000 7f9930ff9700 1 -- 172.21.15.32:0/584120394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f991c0093b0 con 0x7f9934101d70 2024-04-06T14:26:02.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.262+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9934117a30 con 0x7f9934101d70 2024-04-06T14:26:02.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.262+0000 7f9930ff9700 1 -- 172.21.15.32:0/584120394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c00e070 con 0x7f9934101d70 2024-04-06T14:26:02.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.262+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9934117f40 con 0x7f9934101d70 2024-04-06T14:26:02.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.262+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99340ff080 con 0x7f9934101d70 2024-04-06T14:26:02.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.263+0000 7f9930ff9700 1 -- 172.21.15.32:0/584120394 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f991c01c490 con 0x7f9934101d70 2024-04-06T14:26:02.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.263+0000 7f9930ff9700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f992006cc20 0x7f992006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:02.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.264+0000 7f9930ff9700 1 -- 172.21.15.32:0/584120394 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f991c08d100 con 0x7f9934101d70 2024-04-06T14:26:02.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.264+0000 7f99337fe700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f992006cc20 0x7f992006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:02.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.264+0000 7f99337fe700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f992006cc20 0x7f992006f0d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9934066000 tx=0x7f992400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:02.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.268+0000 7f9930ff9700 1 -- 172.21.15.32:0/584120394 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f991c057140 con 0x7f9934101d70 2024-04-06T14:26:02.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.492+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f993404bed0 con 0x7f992006cc20 2024-04-06T14:26:02.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.494+0000 7f9930ff9700 1 -- 172.21.15.32:0/584120394 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f993404bed0 con 0x7f992006cc20 2024-04-06T14:26:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f992006cc20 msgr2=0x7f992006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f992006cc20 0x7f992006f0d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9934066000 tx=0x7f992400b040 comp rx=0 tx=0).stop 2024-04-06T14:26:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 msgr2=0x7f993410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 0x7f993410e9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f991c000c00 tx=0x7f991c009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 shutdown_connections 2024-04-06T14:26:02.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f992006cc20 0x7f992006f0d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9934101d70 0x7f993410e9c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 --2- 172.21.15.32:0/584120394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9934102710 0x7f993410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 >> 172.21.15.32:0/584120394 conn(0x7f9934072bc0 msgr2=0x7f99340faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:02.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 shutdown_connections 2024-04-06T14:26:02.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.496+0000 7f993a2d4700 1 -- 172.21.15.32:0/584120394 wait complete. 2024-04-06T14:26:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.600+0000 7fc183087700 1 -- 172.21.15.32:0/4113895492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fde10 msgr2=0x7fc17c0fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.600+0000 7fc183087700 1 --2- 172.21.15.32:0/4113895492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fde10 0x7fc17c0fe1e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc170004940 tx=0x7fc170009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.600+0000 7fc183087700 1 -- 172.21.15.32:0/4113895492 shutdown_connections 2024-04-06T14:26:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.600+0000 7fc183087700 1 --2- 172.21.15.32:0/4113895492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc17c0fe7b0 0x7fc17c106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.600+0000 7fc183087700 1 --2- 172.21.15.32:0/4113895492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fde10 0x7fc17c0fe1e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.601+0000 7fc183087700 1 -- 172.21.15.32:0/4113895492 >> 172.21.15.32:0/4113895492 conn(0x7fc17c0f78d0 msgr2=0x7fc17c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.601+0000 7fc183087700 1 -- 172.21.15.32:0/4113895492 shutdown_connections 2024-04-06T14:26:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.601+0000 7fc183087700 1 -- 172.21.15.32:0/4113895492 wait complete. 2024-04-06T14:26:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.601+0000 7fc183087700 1 Processor -- start 2024-04-06T14:26:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.601+0000 7fc183087700 1 -- start start 2024-04-06T14:26:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.601+0000 7fc183087700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc17c0fde10 0x7fc17c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc183087700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fe7b0 0x7fc17c1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc183087700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c1167e0 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc183087700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c116950 con 0x7fc17c0fde10 2024-04-06T14:26:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc17bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fe7b0 0x7fc17c1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc17bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fe7b0 0x7fc17c1161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33002/0 (socket says 172.21.15.32:33002) 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc17bfff700 1 -- 172.21.15.32:0/281063161 learned_addr learned my addr 172.21.15.32:0/281063161 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc180e23700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc17c0fde10 0x7fc17c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc17bfff700 1 -- 172.21.15.32:0/281063161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc17c0fde10 msgr2=0x7fc17c115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc17bfff700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc17c0fde10 0x7fc17c115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc17bfff700 1 -- 172.21.15.32:0/281063161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc170005040 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc17bfff700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fe7b0 0x7fc17c1161e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c8c0 tx=0x7fc1680044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc1797fa700 1 -- 172.21.15.32:0/281063161 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16800a980 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c10fe60 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc1797fa700 1 -- 172.21.15.32:0/281063161 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc16800ce60 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc1797fa700 1 -- 172.21.15.32:0/281063161 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1680113a0 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.602+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c1103a0 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.604+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c0666f0 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.604+0000 7fc1797fa700 1 -- 172.21.15.32:0/281063161 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc168012400 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.604+0000 7fc1797fa700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc16c06cc20 0x7fc16c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.604+0000 7fc1797fa700 1 -- 172.21.15.32:0/281063161 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc16808c160 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.604+0000 7fc180e23700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc16c06cc20 0x7fc16c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.605+0000 7fc180e23700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc16c06cc20 0x7fc16c06f0d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc170004f90 tx=0x7fc170009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:02.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.611+0000 7fc1797fa700 1 -- 172.21.15.32:0/281063161 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc1680561a0 con 0x7fc17c0fe7b0 2024-04-06T14:26:02.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:02 smithi086 ceph-mon[110975]: pgmap v174: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:02.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:02 smithi032 ceph-mon[147414]: pgmap v174: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:02.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.867+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc17c109dd0 con 0x7fc16c06cc20 2024-04-06T14:26:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.877+0000 7fc1797fa700 1 -- 172.21.15.32:0/281063161 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc17c109dd0 con 0x7fc16c06cc20 2024-04-06T14:26:02.877 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:26:02.877 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 2s ago 17m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:26:02.877 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (3m) 2s ago 17m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:26:02.877 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (3m) 19s ago 16m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:26:02.877 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 2s ago 17m 87.2M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:26:02.877 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (8s) 2s ago 11m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (11m) 2s ago 11m 10.4M - 17.2.7-819-g6699495c ec300eaa768b 06c72f127f41 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (11m) 19s ago 11m 127M - 17.2.7-819-g6699495c ec300eaa768b 412bafe45f57 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (11m) 19s ago 11m 17.4M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (8m) 2s ago 19m 507M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (7m) 19s ago 16m 450M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (4m) 2s ago 19m 53.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (4m) 19s ago 16m 40.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (7m) 2s ago 17m 16.6M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (7m) 19s ago 16m 17.2M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (3m) 2s ago 15m 83.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (2m) 2s ago 15m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:26:02.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (2m) 2s ago 14m 78.9M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:26:02.879 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (89s) 19s ago 14m 91.3M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:26:02.879 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (56s) 19s ago 13m 90.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:26:02.879 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (23s) 19s ago 13m 10.7M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:26:02.879 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 2s ago 17m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:26:02.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc16c06cc20 msgr2=0x7fc16c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc16c06cc20 0x7fc16c06f0d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc170004f90 tx=0x7fc170009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fe7b0 msgr2=0x7fc17c1161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fe7b0 0x7fc17c1161e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c8c0 tx=0x7fc1680044c0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 shutdown_connections 2024-04-06T14:26:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc16c06cc20 0x7fc16c06f0d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc17c0fde10 0x7fc17c115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 --2- 172.21.15.32:0/281063161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc17c0fe7b0 0x7fc17c1161e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 >> 172.21.15.32:0/281063161 conn(0x7fc17c0f78d0 msgr2=0x7fc17c0f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 shutdown_connections 2024-04-06T14:26:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.879+0000 7fc183087700 1 -- 172.21.15.32:0/281063161 wait complete. 2024-04-06T14:26:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 -- 172.21.15.32:0/511795813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fde10 msgr2=0x7fa0540fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 --2- 172.21.15.32:0/511795813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fde10 0x7fa0540fe1e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa044004940 tx=0x7fa044009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 -- 172.21.15.32:0/511795813 shutdown_connections 2024-04-06T14:26:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 --2- 172.21.15.32:0/511795813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0540fe7b0 0x7fa054106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 --2- 172.21.15.32:0/511795813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fde10 0x7fa0540fe1e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 -- 172.21.15.32:0/511795813 >> 172.21.15.32:0/511795813 conn(0x7fa0540f7880 msgr2=0x7fa0540f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 -- 172.21.15.32:0/511795813 shutdown_connections 2024-04-06T14:26:02.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 -- 172.21.15.32:0/511795813 wait complete. 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 Processor -- start 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.983+0000 7fa05c636700 1 -- start start 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa05c636700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0540fde10 0x7fa054113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa05c636700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fe7b0 0x7fa054114310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa05c636700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0541149f0 con 0x7fa0540fe7b0 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa05c636700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05410dd30 con 0x7fa0540fde10 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa059bd1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fe7b0 0x7fa054114310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa059bd1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fe7b0 0x7fa054114310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33020/0 (socket says 172.21.15.32:33020) 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa059bd1700 1 -- 172.21.15.32:0/911238844 learned_addr learned my addr 172.21.15.32:0/911238844 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa05a3d2700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0540fde10 0x7fa054113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa059bd1700 1 -- 172.21.15.32:0/911238844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0540fde10 msgr2=0x7fa054113dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa059bd1700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0540fde10 0x7fa054113dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa059bd1700 1 -- 172.21.15.32:0/911238844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa044005040 con 0x7fa0540fe7b0 2024-04-06T14:26:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa059bd1700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fe7b0 0x7fa054114310 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa05000b790 tx=0x7fa050009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa04affd700 1 -- 172.21.15.32:0/911238844 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa05000f770 con 0x7fa0540fe7b0 2024-04-06T14:26:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05410dfb0 con 0x7fa0540fe7b0 2024-04-06T14:26:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa04affd700 1 -- 172.21.15.32:0/911238844 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa05000fd80 con 0x7fa0540fe7b0 2024-04-06T14:26:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.984+0000 7fa04affd700 1 -- 172.21.15.32:0/911238844 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa05000d310 con 0x7fa0540fe7b0 2024-04-06T14:26:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.985+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05410e4f0 con 0x7fa0540fe7b0 2024-04-06T14:26:02.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.986+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa054066770 con 0x7fa0540fe7b0 2024-04-06T14:26:02.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.986+0000 7fa04affd700 1 -- 172.21.15.32:0/911238844 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa050010420 con 0x7fa0540fe7b0 2024-04-06T14:26:02.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.986+0000 7fa04affd700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa04006cc20 0x7fa04006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.986+0000 7fa04affd700 1 -- 172.21.15.32:0/911238844 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa05008c050 con 0x7fa0540fe7b0 2024-04-06T14:26:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.986+0000 7fa05a3d2700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa04006cc20 0x7fa04006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.987+0000 7fa05a3d2700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa04006cc20 0x7fa04006f0d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa044004f90 tx=0x7fa044009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:02.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:02.991+0000 7fa04affd700 1 -- 172.21.15.32:0/911238844 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa050059b20 con 0x7fa0540fe7b0 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.283+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa05404bed0 con 0x7fa0540fe7b0 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.283+0000 7fa04affd700 1 -- 172.21.15.32:0/911238844 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fa0500596b0 con 0x7fa0540fe7b0 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:26:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 3, 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 1 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 3, 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 11 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:26:03.285 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:26:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa04006cc20 msgr2=0x7fa04006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa04006cc20 0x7fa04006f0d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa044004f90 tx=0x7fa044009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fe7b0 msgr2=0x7fa054114310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fe7b0 0x7fa054114310 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa05000b790 tx=0x7fa050009250 comp rx=0 tx=0).stop 2024-04-06T14:26:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 shutdown_connections 2024-04-06T14:26:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa04006cc20 0x7fa04006f0d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0540fde10 0x7fa054113dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 --2- 172.21.15.32:0/911238844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0540fe7b0 0x7fa054114310 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 >> 172.21.15.32:0/911238844 conn(0x7fa0540f7880 msgr2=0x7fa0540f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:03.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 shutdown_connections 2024-04-06T14:26:03.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.285+0000 7fa05c636700 1 -- 172.21.15.32:0/911238844 wait complete. 2024-04-06T14:26:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 -- 172.21.15.32:0/1792840752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 msgr2=0x7f0164106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 --2- 172.21.15.32:0/1792840752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 0x7f0164106f20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0150004970 tx=0x7f0150009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 -- 172.21.15.32:0/1792840752 shutdown_connections 2024-04-06T14:26:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 --2- 172.21.15.32:0/1792840752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 0x7f0164106f20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 --2- 172.21.15.32:0/1792840752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01640fdeb0 0x7f01640fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 -- 172.21.15.32:0/1792840752 >> 172.21.15.32:0/1792840752 conn(0x7f0164072bc0 msgr2=0x7f0164072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 -- 172.21.15.32:0/1792840752 shutdown_connections 2024-04-06T14:26:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 -- 172.21.15.32:0/1792840752 wait complete. 2024-04-06T14:26:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 Processor -- start 2024-04-06T14:26:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.393+0000 7f016aef2700 1 -- start start 2024-04-06T14:26:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f016aef2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01640fdeb0 0x7f01640fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f016aef2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 0x7f01640fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f016aef2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01640fc840 con 0x7f01640fe850 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f016aef2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01640fc980 con 0x7f01640fdeb0 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f0163fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 0x7f01640fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f0163fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 0x7f01640fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33044/0 (socket says 172.21.15.32:33044) 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f0163fff700 1 -- 172.21.15.32:0/1701944566 learned_addr learned my addr 172.21.15.32:0/1701944566 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f0168c8e700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01640fdeb0 0x7f01640fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f0163fff700 1 -- 172.21.15.32:0/1701944566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01640fdeb0 msgr2=0x7f01640fbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f0163fff700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01640fdeb0 0x7f01640fbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f0163fff700 1 -- 172.21.15.32:0/1701944566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0150005040 con 0x7f01640fe850 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.394+0000 7f0163fff700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 0x7f01640fc2b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0150009380 tx=0x7f0150009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.395+0000 7f01617fa700 1 -- 172.21.15.32:0/1701944566 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f015000e040 con 0x7f01640fe850 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.395+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01640fcbd0 con 0x7f01640fe850 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.395+0000 7f01617fa700 1 -- 172.21.15.32:0/1701944566 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0150013420 con 0x7f01640fe850 2024-04-06T14:26:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.395+0000 7f01617fa700 1 -- 172.21.15.32:0/1701944566 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f015001c630 con 0x7f01640fe850 2024-04-06T14:26:03.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.395+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016411a0c0 con 0x7f01640fe850 2024-04-06T14:26:03.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.396+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0164065e10 con 0x7f01640fe850 2024-04-06T14:26:03.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.396+0000 7f01617fa700 1 -- 172.21.15.32:0/1701944566 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f01500139e0 con 0x7f01640fe850 2024-04-06T14:26:03.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.397+0000 7f01617fa700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f015406cc20 0x7f015406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:03.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.397+0000 7f01617fa700 1 -- 172.21.15.32:0/1701944566 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f015008d2c0 con 0x7f01640fe850 2024-04-06T14:26:03.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.397+0000 7f0168c8e700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f015406cc20 0x7f015406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:03.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.397+0000 7f0168c8e700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f015406cc20 0x7f015406f0d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0158004c40 tx=0x7f0158009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:03.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:03 smithi032 ceph-mon[147414]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:03.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:03 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/911238844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:03.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.402+0000 7f01617fa700 1 -- 172.21.15.32:0/1701944566 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0150057300 con 0x7f01640fe850 2024-04-06T14:26:03.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:03 smithi086 ceph-mon[110975]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:03.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:03 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/911238844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:03.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.651+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f016404bed0 con 0x7f01640fe850 2024-04-06T14:26:03.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.651+0000 7f01617fa700 1 -- 172.21.15.32:0/1701944566 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1834 (secure 0 0 0) 0x7f015005a920 con 0x7f01640fe850 2024-04-06T14:26:03.652 INFO:teuthology.orchestra.run.smithi032.stdout:e26 2024-04-06T14:26:03.652 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:26:03.652 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:26:03.652 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:26:03.652 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:26:03.652 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:26:03.652 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:26:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:26:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:26:03.655 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34378} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/629297924,v1:172.21.15.32:6829/629297924] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:26:03.655 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:26:03.655 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44283} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/4145187217,v1:172.21.15.86:6825/4145187217] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:26:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.655+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f015406cc20 msgr2=0x7f015406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.655+0000 7f016aef2700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f015406cc20 0x7f015406f0d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0158004c40 tx=0x7f0158009040 comp rx=0 tx=0).stop 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.655+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 msgr2=0x7f01640fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.655+0000 7f016aef2700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 0x7f01640fc2b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0150009380 tx=0x7f0150009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.655+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 shutdown_connections 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.655+0000 7f016aef2700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f015406cc20 0x7f015406f0d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.655+0000 7f016aef2700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01640fdeb0 0x7f01640fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.655+0000 7f016aef2700 1 --2- 172.21.15.32:0/1701944566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01640fe850 0x7f01640fc2b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.655+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 >> 172.21.15.32:0/1701944566 conn(0x7f0164072bc0 msgr2=0x7f0164103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.656+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 shutdown_connections 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.656+0000 7f016aef2700 1 -- 172.21.15.32:0/1701944566 wait complete. 2024-04-06T14:26:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-04-06T14:26:03.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.764+0000 7fe70c947700 1 -- 172.21.15.32:0/164611904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 msgr2=0x7fe7080fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 --2- 172.21.15.32:0/164611904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 0x7fe7080fb910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc004940 tx=0x7fe6fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 -- 172.21.15.32:0/164611904 shutdown_connections 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 --2- 172.21.15.32:0/164611904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 0x7fe7080fb910 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 --2- 172.21.15.32:0/164611904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe708102e70 0x7fe708103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 -- 172.21.15.32:0/164611904 >> 172.21.15.32:0/164611904 conn(0x7fe7080714c0 msgr2=0x7fe7080718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 -- 172.21.15.32:0/164611904 shutdown_connections 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 -- 172.21.15.32:0/164611904 wait complete. 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 Processor -- start 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 -- start start 2024-04-06T14:26:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.765+0000 7fe70c947700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 0x7fe708113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70c947700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe708102e70 0x7fe708114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70c947700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7081146e0 con 0x7fe7080fb4a0 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70c947700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70810dad0 con 0x7fe708102e70 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 0x7fe708113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 0x7fe708113b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33062/0 (socket says 172.21.15.32:33062) 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70659c700 1 -- 172.21.15.32:0/2631582193 learned_addr learned my addr 172.21.15.32:0/2631582193 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70659c700 1 -- 172.21.15.32:0/2631582193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe708102e70 msgr2=0x7fe708114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70659c700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe708102e70 0x7fe708114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70659c700 1 -- 172.21.15.32:0/2631582193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc005040 con 0x7fe7080fb4a0 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70659c700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 0x7fe708113b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8002a50 tx=0x7fe6f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe6f6ffd700 1 -- 172.21.15.32:0/2631582193 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6f8007cc0 con 0x7fe7080fb4a0 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70810ddb0 con 0x7fe7080fb4a0 2024-04-06T14:26:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe6f6ffd700 1 -- 172.21.15.32:0/2631582193 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6f800f040 con 0x7fe7080fb4a0 2024-04-06T14:26:03.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe6f6ffd700 1 -- 172.21.15.32:0/2631582193 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6f8009540 con 0x7fe7080fb4a0 2024-04-06T14:26:03.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.766+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70810e2f0 con 0x7fe7080fb4a0 2024-04-06T14:26:03.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.767+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe708100170 con 0x7fe7080fb4a0 2024-04-06T14:26:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.768+0000 7fe6f6ffd700 1 -- 172.21.15.32:0/2631582193 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe6f80096a0 con 0x7fe7080fb4a0 2024-04-06T14:26:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.768+0000 7fe6f6ffd700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe6f006cc30 0x7fe6f006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.769+0000 7fe6f6ffd700 1 -- 172.21.15.32:0/2631582193 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe6f808be70 con 0x7fe7080fb4a0 2024-04-06T14:26:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.769+0000 7fe705d9b700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe6f006cc30 0x7fe6f006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.769+0000 7fe705d9b700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe6f006cc30 0x7fe6f006f0e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc004f90 tx=0x7fe6fc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:03.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:03.773+0000 7fe6f6ffd700 1 -- 172.21.15.32:0/2631582193 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe6f80598c0 con 0x7fe7080fb4a0 2024-04-06T14:26:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.012+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe70810f240 con 0x7fe6f006cc30 2024-04-06T14:26:04.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.013+0000 7fe6f6ffd700 1 -- 172.21.15.32:0/2631582193 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe70810f240 con 0x7fe6f006cc30 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "13/21 daemons upgraded", 2024-04-06T14:26:04.014 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:26:04.015 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:26:04.015 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.015+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe6f006cc30 msgr2=0x7fe6f006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.015+0000 7fe70c947700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe6f006cc30 0x7fe6f006f0e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc004f90 tx=0x7fe6fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.015+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 msgr2=0x7fe708113b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.015+0000 7fe70c947700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 0x7fe708113b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8002a50 tx=0x7fe6f80046b0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.015+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 shutdown_connections 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.015+0000 7fe70c947700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe6f006cc30 0x7fe6f006f0e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.015+0000 7fe70c947700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7080fb4a0 0x7fe708113b50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.016+0000 7fe70c947700 1 --2- 172.21.15.32:0/2631582193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe708102e70 0x7fe708114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.016+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 >> 172.21.15.32:0/2631582193 conn(0x7fe7080714c0 msgr2=0x7fe7080fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.016+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 shutdown_connections 2024-04-06T14:26:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.016+0000 7fe70c947700 1 -- 172.21.15.32:0/2631582193 wait complete. 2024-04-06T14:26:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 -- 172.21.15.32:0/3644231238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 msgr2=0x7f24c8103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 --2- 172.21.15.32:0/3644231238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 0x7f24c8103240 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24c4004940 tx=0x7f24c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 -- 172.21.15.32:0/3644231238 shutdown_connections 2024-04-06T14:26:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 --2- 172.21.15.32:0/3644231238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24c80fb4a0 0x7f24c80fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 --2- 172.21.15.32:0/3644231238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 0x7f24c8103240 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 -- 172.21.15.32:0/3644231238 >> 172.21.15.32:0/3644231238 conn(0x7f24c80714c0 msgr2=0x7f24c80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 -- 172.21.15.32:0/3644231238 shutdown_connections 2024-04-06T14:26:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 -- 172.21.15.32:0/3644231238 wait complete. 2024-04-06T14:26:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 Processor -- start 2024-04-06T14:26:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.125+0000 7f24d0235700 1 -- start start 2024-04-06T14:26:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24d0235700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24c80fb4a0 0x7f24c8113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24d0235700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 0x7f24c8113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24d0235700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c8114600 con 0x7f24c8102e70 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24d0235700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c810d9f0 con 0x7f24c80fb4a0 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24cd7d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 0x7f24c8113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24cd7d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 0x7f24c8113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33078/0 (socket says 172.21.15.32:33078) 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24cd7d0700 1 -- 172.21.15.32:0/1401253826 learned_addr learned my addr 172.21.15.32:0/1401253826 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24cdfd1700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24c80fb4a0 0x7f24c8113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24cd7d0700 1 -- 172.21.15.32:0/1401253826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24c80fb4a0 msgr2=0x7f24c8113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24cd7d0700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24c80fb4a0 0x7f24c8113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24cd7d0700 1 -- 172.21.15.32:0/1401253826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c4005040 con 0x7f24c8102e70 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24cdfd1700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24c80fb4a0 0x7f24c8113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.126+0000 7f24cd7d0700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 0x7f24c8113fb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f24b8002a50 tx=0x7f24b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.127+0000 7f24be7fc700 1 -- 172.21.15.32:0/1401253826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24b8007cc0 con 0x7f24c8102e70 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.127+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c810dcd0 con 0x7f24c8102e70 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.127+0000 7f24be7fc700 1 -- 172.21.15.32:0/1401253826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24b800f040 con 0x7f24c8102e70 2024-04-06T14:26:04.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.127+0000 7f24be7fc700 1 -- 172.21.15.32:0/1401253826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24b8009540 con 0x7f24c8102e70 2024-04-06T14:26:04.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.127+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c810e290 con 0x7f24c8102e70 2024-04-06T14:26:04.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.128+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c8100110 con 0x7f24c8102e70 2024-04-06T14:26:04.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.128+0000 7f24be7fc700 1 -- 172.21.15.32:0/1401253826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f24b80096a0 con 0x7f24c8102e70 2024-04-06T14:26:04.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.129+0000 7f24be7fc700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f24b406cc30 0x7f24b406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:04.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.129+0000 7f24be7fc700 1 -- 172.21.15.32:0/1401253826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f24b808be70 con 0x7f24c8102e70 2024-04-06T14:26:04.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.129+0000 7f24cdfd1700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f24b406cc30 0x7f24b406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:04.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.130+0000 7f24cdfd1700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f24b406cc30 0x7f24b406f0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24c4000c00 tx=0x7f24c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:04.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.134+0000 7f24be7fc700 1 -- 172.21.15.32:0/1401253826 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f24b80598c0 con 0x7f24c8102e70 2024-04-06T14:26:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.427+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f24c810f0c0 con 0x7f24c8102e70 2024-04-06T14:26:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.428+0000 7f24be7fc700 1 -- 172.21.15.32:0/1401253826 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f24b8059450 con 0x7f24c8102e70 2024-04-06T14:26:04.428 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:26:04.428 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:26:04.428 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:26:04.429 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:26:04.429 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:26:04.429 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:26:04.429 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:26:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f24b406cc30 msgr2=0x7f24b406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f24b406cc30 0x7f24b406f0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24c4000c00 tx=0x7f24c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 msgr2=0x7f24c8113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 0x7f24c8113fb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f24b8002a50 tx=0x7f24b80046b0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 shutdown_connections 2024-04-06T14:26:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f24b406cc30 0x7f24b406f0e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24c80fb4a0 0x7f24c8113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 --2- 172.21.15.32:0/1401253826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8102e70 0x7f24c8113fb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 >> 172.21.15.32:0/1401253826 conn(0x7f24c80714c0 msgr2=0x7f24c80fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 shutdown_connections 2024-04-06T14:26:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:04.430+0000 7f24d0235700 1 -- 172.21.15.32:0/1401253826 wait complete. 2024-04-06T14:26:04.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:04 smithi086 ceph-mon[110975]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:04.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:04 smithi086 ceph-mon[110975]: pgmap v175: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:04.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:04 smithi086 ceph-mon[110975]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:04.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:04 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1701944566' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:26:04.709 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:04 smithi032 ceph-mon[147414]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:04.709 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:04 smithi032 ceph-mon[147414]: pgmap v175: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:04.709 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:04 smithi032 ceph-mon[147414]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:04.710 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:04 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1701944566' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:26:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1401253826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:26:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:05.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:26:05.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:26:05.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:05.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:26:05.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:05.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:05.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:05.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:05 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1401253826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:26:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:26:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:26:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:05.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:26:05.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:05.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:05.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:05.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:05 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:06 smithi032 ceph-mon[147414]: pgmap v176: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:06.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:06 smithi086 ceph-mon[110975]: pgmap v176: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:08.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:07 smithi032 ceph-mon[147414]: pgmap v177: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:08.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:07 smithi032 ceph-mon[147414]: Upgrade: Updating mds.cephfs.smithi032.mqoaof 2024-04-06T14:26:08.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:08.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.mqoaof", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:26:08.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:26:08.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:07 smithi032 ceph-mon[147414]: Deploying daemon mds.cephfs.smithi032.mqoaof on smithi032 2024-04-06T14:26:08.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:08.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:26:08.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:07 smithi086 ceph-mon[110975]: pgmap v177: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:07 smithi086 ceph-mon[110975]: Upgrade: Updating mds.cephfs.smithi032.mqoaof 2024-04-06T14:26:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.mqoaof", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:26:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:26:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:07 smithi086 ceph-mon[110975]: Deploying daemon mds.cephfs.smithi032.mqoaof on smithi032 2024-04-06T14:26:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:26:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:10 smithi086 ceph-mon[110975]: pgmap v178: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:10 smithi086 ceph-mon[110975]: osdmap e83: 6 total, 6 up, 6 in 2024-04-06T14:26:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:10 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-06T14:26:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:10 smithi032 ceph-mon[147414]: pgmap v178: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:10 smithi032 ceph-mon[147414]: osdmap e83: 6 total, 6 up, 6 in 2024-04-06T14:26:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:10 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-06T14:26:12.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:12 smithi086 ceph-mon[110975]: pgmap v180: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:12.439 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:12 smithi032 ceph-mon[147414]: pgmap v180: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:14 smithi032 ceph-mon[147414]: pgmap v181: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:14.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:14 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:26:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:14 smithi086 ceph-mon[110975]: pgmap v181: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:14.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:14 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:26:15.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:15 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:boot 2024-04-06T14:26:15.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:15 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:26:15.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:15 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:26:15.417 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:15 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:boot 2024-04-06T14:26:15.417 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:15 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:26:15.417 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:15 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:26:16.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:16 smithi086 ceph-mon[110975]: pgmap v182: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:16 smithi032 ceph-mon[147414]: pgmap v182: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:18.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:18 smithi086 ceph-mon[110975]: pgmap v183: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:18.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:18 smithi032 ceph-mon[147414]: pgmap v183: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:19.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:19 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:19.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:19 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:19.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:19 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:19.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:19 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:19.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:19 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:19.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:19 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:19.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:19 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:19.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:19 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:20.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:20 smithi032 ceph-mon[147414]: pgmap v184: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:20.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:20 smithi086 ceph-mon[110975]: pgmap v184: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:21.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:21 smithi086 ceph-mon[110975]: pgmap v185: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:21.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:21 smithi032 ceph-mon[147414]: pgmap v185: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:22.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:26:23.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: pgmap v186: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:24.128 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:24.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: pgmap v186: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:24.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:24.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:24.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:26:24.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:26:24.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:24.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:26:24.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:24.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:24.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:24.231 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:25 smithi086 ceph-mon[110975]: Upgrade: Updating mds.cephfs.smithi086.nikfjj 2024-04-06T14:26:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:25 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:25 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.nikfjj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:26:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:25 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:26:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:25 smithi086 ceph-mon[110975]: Deploying daemon mds.cephfs.smithi086.nikfjj on smithi086 2024-04-06T14:26:25.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:25 smithi086 ceph-mon[110975]: pgmap v187: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:25 smithi032 ceph-mon[147414]: Upgrade: Updating mds.cephfs.smithi086.nikfjj 2024-04-06T14:26:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:25 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:25 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.nikfjj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:26:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:25 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:26:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:25 smithi032 ceph-mon[147414]: Deploying daemon mds.cephfs.smithi086.nikfjj on smithi086 2024-04-06T14:26:25.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:25 smithi032 ceph-mon[147414]: pgmap v187: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:27.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:27 smithi032 ceph-mon[147414]: osdmap e84: 6 total, 6 up, 6 in 2024-04-06T14:26:27.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:27 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-06T14:26:27.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:27 smithi086 ceph-mon[110975]: osdmap e84: 6 total, 6 up, 6 in 2024-04-06T14:26:27.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:27 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-06T14:26:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:28 smithi032 ceph-mon[147414]: pgmap v189: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:28.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:28 smithi086 ceph-mon[110975]: pgmap v189: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:30 smithi032 ceph-mon[147414]: pgmap v190: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:30 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:30 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:30.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:30 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:26:30.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:30 smithi086 ceph-mon[110975]: pgmap v190: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:30 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:30 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:30 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:26:31.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:31 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] up:boot 2024-04-06T14:26:31.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:31 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:26:31.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:26:31.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:31 smithi086 ceph-mon[110975]: pgmap v191: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:31 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] up:boot 2024-04-06T14:26:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:31 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:26:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:26:31.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:31 smithi032 ceph-mon[147414]: pgmap v191: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:33 smithi032 ceph-mon[147414]: pgmap v192: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:33.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:33 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:34.077 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:33 smithi086 ceph-mon[110975]: pgmap v192: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:34.077 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:34.077 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:34.077 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:34.077 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:33 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.551+0000 7fd821dfd700 1 -- 172.21.15.32:0/3498824508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c102710 msgr2=0x7fd81c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.551+0000 7fd821dfd700 1 --2- 172.21.15.32:0/3498824508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c102710 0x7fd81c0fbd80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd810004940 tx=0x7fd810009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.551+0000 7fd821dfd700 1 -- 172.21.15.32:0/3498824508 shutdown_connections 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.551+0000 7fd821dfd700 1 --2- 172.21.15.32:0/3498824508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c102710 0x7fd81c0fbd80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.551+0000 7fd821dfd700 1 --2- 172.21.15.32:0/3498824508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd81c101d70 0x7fd81c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.551+0000 7fd821dfd700 1 -- 172.21.15.32:0/3498824508 >> 172.21.15.32:0/3498824508 conn(0x7fd81c072bc0 msgr2=0x7fd81c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.551+0000 7fd821dfd700 1 -- 172.21.15.32:0/3498824508 shutdown_connections 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.551+0000 7fd821dfd700 1 -- 172.21.15.32:0/3498824508 wait complete. 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.551+0000 7fd821dfd700 1 Processor -- start 2024-04-06T14:26:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd821dfd700 1 -- start start 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd821dfd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c101d70 0x7fd81c113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd821dfd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd81c102710 0x7fd81c113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd821dfd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c114630 con 0x7fd81c101d70 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd821dfd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c10da20 con 0x7fd81c102710 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd81b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c101d70 0x7fd81c113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd81b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c101d70 0x7fd81c113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54282/0 (socket says 172.21.15.32:54282) 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd81b7fe700 1 -- 172.21.15.32:0/2491098150 learned_addr learned my addr 172.21.15.32:0/2491098150 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd81affd700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd81c102710 0x7fd81c113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd81b7fe700 1 -- 172.21.15.32:0/2491098150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd81c102710 msgr2=0x7fd81c113fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd81b7fe700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd81c102710 0x7fd81c113fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.552+0000 7fd81b7fe700 1 -- 172.21.15.32:0/2491098150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd810005040 con 0x7fd81c101d70 2024-04-06T14:26:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.553+0000 7fd81b7fe700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c101d70 0x7fd81c113aa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd80c002a50 tx=0x7fd80c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.553+0000 7fd803fff700 1 -- 172.21.15.32:0/2491098150 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80c007cc0 con 0x7fd81c101d70 2024-04-06T14:26:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.553+0000 7fd803fff700 1 -- 172.21.15.32:0/2491098150 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd80c00f040 con 0x7fd81c101d70 2024-04-06T14:26:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.553+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c10dd00 con 0x7fd81c101d70 2024-04-06T14:26:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.553+0000 7fd803fff700 1 -- 172.21.15.32:0/2491098150 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80c00d9c0 con 0x7fd81c101d70 2024-04-06T14:26:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.553+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c10e290 con 0x7fd81c101d70 2024-04-06T14:26:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.554+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c0fd4c0 con 0x7fd81c101d70 2024-04-06T14:26:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.555+0000 7fd803fff700 1 -- 172.21.15.32:0/2491098150 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd80c00db20 con 0x7fd81c101d70 2024-04-06T14:26:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.555+0000 7fd803fff700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd80406cc30 0x7fd80406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.555+0000 7fd803fff700 1 -- 172.21.15.32:0/2491098150 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd80c08bd70 con 0x7fd81c101d70 2024-04-06T14:26:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.555+0000 7fd81affd700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd80406cc30 0x7fd80406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.556+0000 7fd81affd700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd80406cc30 0x7fd80406f0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd81c10ed00 tx=0x7fd810009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.560+0000 7fd803fff700 1 -- 172.21.15.32:0/2491098150 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd80c0597c0 con 0x7fd81c101d70 2024-04-06T14:26:34.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.781+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd81c0666a0 con 0x7fd80406cc30 2024-04-06T14:26:34.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.782+0000 7fd803fff700 1 -- 172.21.15.32:0/2491098150 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd81c0666a0 con 0x7fd80406cc30 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.784+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd80406cc30 msgr2=0x7fd80406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.784+0000 7fd821dfd700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd80406cc30 0x7fd80406f0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd81c10ed00 tx=0x7fd810009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.784+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c101d70 msgr2=0x7fd81c113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.784+0000 7fd821dfd700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c101d70 0x7fd81c113aa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd80c002a50 tx=0x7fd80c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.784+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 shutdown_connections 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.784+0000 7fd821dfd700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd80406cc30 0x7fd80406f0e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.784+0000 7fd821dfd700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd81c101d70 0x7fd81c113aa0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.784+0000 7fd821dfd700 1 --2- 172.21.15.32:0/2491098150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd81c102710 0x7fd81c113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.785+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 >> 172.21.15.32:0/2491098150 conn(0x7fd81c072bc0 msgr2=0x7fd81c1058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.785+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 shutdown_connections 2024-04-06T14:26:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.785+0000 7fd821dfd700 1 -- 172.21.15.32:0/2491098150 wait complete. 2024-04-06T14:26:34.795 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:26:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.889+0000 7fa45c8f1700 1 -- 172.21.15.32:0/731973070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 msgr2=0x7fa458106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.889+0000 7fa45c8f1700 1 --2- 172.21.15.32:0/731973070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 0x7fa458106f00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa440004940 tx=0x7fa440009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.889+0000 7fa45c8f1700 1 -- 172.21.15.32:0/731973070 shutdown_connections 2024-04-06T14:26:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.889+0000 7fa45c8f1700 1 --2- 172.21.15.32:0/731973070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 0x7fa458106f00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.889+0000 7fa45c8f1700 1 --2- 172.21.15.32:0/731973070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4580fde90 0x7fa4580fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.889+0000 7fa45c8f1700 1 -- 172.21.15.32:0/731973070 >> 172.21.15.32:0/731973070 conn(0x7fa458072bc0 msgr2=0x7fa458072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.889+0000 7fa45c8f1700 1 -- 172.21.15.32:0/731973070 shutdown_connections 2024-04-06T14:26:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.889+0000 7fa45c8f1700 1 -- 172.21.15.32:0/731973070 wait complete. 2024-04-06T14:26:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa45c8f1700 1 Processor -- start 2024-04-06T14:26:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa45c8f1700 1 -- start start 2024-04-06T14:26:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa45c8f1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4580fde90 0x7fa4580fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa45c8f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 0x7fa4580fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa45c8f1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580fc820 con 0x7fa4580fe830 2024-04-06T14:26:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa45c8f1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580fc960 con 0x7fa4580fde90 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa455d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 0x7fa4580fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa455d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 0x7fa4580fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54294/0 (socket says 172.21.15.32:54294) 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa455d9b700 1 -- 172.21.15.32:0/3377064984 learned_addr learned my addr 172.21.15.32:0/3377064984 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa45659c700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4580fde90 0x7fa4580fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa455d9b700 1 -- 172.21.15.32:0/3377064984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4580fde90 msgr2=0x7fa4580fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa455d9b700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4580fde90 0x7fa4580fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.890+0000 7fa455d9b700 1 -- 172.21.15.32:0/3377064984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa440005040 con 0x7fa4580fe830 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.891+0000 7fa455d9b700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 0x7fa4580fc290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa44000b3b0 tx=0x7fa440009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.891+0000 7fa44effd700 1 -- 172.21.15.32:0/3377064984 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa44000e040 con 0x7fa4580fe830 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.891+0000 7fa44effd700 1 -- 172.21.15.32:0/3377064984 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa440013420 con 0x7fa4580fe830 2024-04-06T14:26:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.891+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4580fcbb0 con 0x7fa4580fe830 2024-04-06T14:26:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.891+0000 7fa44effd700 1 -- 172.21.15.32:0/3377064984 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa44001c5f0 con 0x7fa4580fe830 2024-04-06T14:26:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.891+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45811a190 con 0x7fa4580fe830 2024-04-06T14:26:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.892+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa458065e30 con 0x7fa4580fe830 2024-04-06T14:26:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.892+0000 7fa44effd700 1 -- 172.21.15.32:0/3377064984 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa440013590 con 0x7fa4580fe830 2024-04-06T14:26:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.893+0000 7fa44effd700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa44406cc20 0x7fa44406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.893+0000 7fa44effd700 1 -- 172.21.15.32:0/3377064984 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa44008d230 con 0x7fa4580fe830 2024-04-06T14:26:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.893+0000 7fa45659c700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa44406cc20 0x7fa44406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.893+0000 7fa45659c700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa44406cc20 0x7fa44406f0d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa448004f40 tx=0x7fa44800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:34.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:34.897+0000 7fa44effd700 1 -- 172.21.15.32:0/3377064984 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa440057330 con 0x7fa4580fe830 2024-04-06T14:26:35.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.119+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa458109e70 con 0x7fa44406cc20 2024-04-06T14:26:35.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.120+0000 7fa44effd700 1 -- 172.21.15.32:0/3377064984 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa44800f040 con 0x7fa44406cc20 2024-04-06T14:26:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.122+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa44406cc20 msgr2=0x7fa44406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.122+0000 7fa45c8f1700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa44406cc20 0x7fa44406f0d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa448004f40 tx=0x7fa44800b040 comp rx=0 tx=0).stop 2024-04-06T14:26:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.122+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 msgr2=0x7fa4580fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.122+0000 7fa45c8f1700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 0x7fa4580fc290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa44000b3b0 tx=0x7fa440009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.123+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 shutdown_connections 2024-04-06T14:26:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.123+0000 7fa45c8f1700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa44406cc20 0x7fa44406f0d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.123+0000 7fa45c8f1700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4580fde90 0x7fa4580fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.123+0000 7fa45c8f1700 1 --2- 172.21.15.32:0/3377064984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4580fe830 0x7fa4580fc290 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.123+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 >> 172.21.15.32:0/3377064984 conn(0x7fa458072bc0 msgr2=0x7fa458103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:35.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.123+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 shutdown_connections 2024-04-06T14:26:35.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.123+0000 7fa45c8f1700 1 -- 172.21.15.32:0/3377064984 wait complete. 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.226+0000 7f3b92d64700 1 -- 172.21.15.32:0/2444271227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c1037c0 msgr2=0x7f3b8c0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.226+0000 7f3b92d64700 1 --2- 172.21.15.32:0/2444271227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c1037c0 0x7f3b8c0fce30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3b78004940 tx=0x7f3b78009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.226+0000 7f3b92d64700 1 -- 172.21.15.32:0/2444271227 shutdown_connections 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.226+0000 7f3b92d64700 1 --2- 172.21.15.32:0/2444271227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c1037c0 0x7f3b8c0fce30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.226+0000 7f3b92d64700 1 --2- 172.21.15.32:0/2444271227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c102e20 0x7f3b8c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.226+0000 7f3b92d64700 1 -- 172.21.15.32:0/2444271227 >> 172.21.15.32:0/2444271227 conn(0x7f3b8c0f89d0 msgr2=0x7f3b8c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.226+0000 7f3b92d64700 1 -- 172.21.15.32:0/2444271227 shutdown_connections 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.226+0000 7f3b92d64700 1 -- 172.21.15.32:0/2444271227 wait complete. 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b92d64700 1 Processor -- start 2024-04-06T14:26:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b92d64700 1 -- start start 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b92d64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c102e20 0x7f3b8c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b92d64700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c1037c0 0x7f3b8c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b92d64700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c070e20 con 0x7f3b8c102e20 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b92d64700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c070f60 con 0x7f3b8c1037c0 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b90b00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c102e20 0x7f3b8c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b90b00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c102e20 0x7f3b8c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54300/0 (socket says 172.21.15.32:54300) 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b90b00700 1 -- 172.21.15.32:0/3169153866 learned_addr learned my addr 172.21.15.32:0/3169153866 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b8bfff700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c1037c0 0x7f3b8c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b90b00700 1 -- 172.21.15.32:0/3169153866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c1037c0 msgr2=0x7f3b8c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b90b00700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c1037c0 0x7f3b8c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.227+0000 7f3b90b00700 1 -- 172.21.15.32:0/3169153866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b78005040 con 0x7f3b8c102e20 2024-04-06T14:26:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.228+0000 7f3b90b00700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c102e20 0x7f3b8c0702c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000b790 tx=0x7f3b80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.228+0000 7f3b897fa700 1 -- 172.21.15.32:0/3169153866 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b8000f6f0 con 0x7f3b8c102e20 2024-04-06T14:26:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.228+0000 7f3b897fa700 1 -- 172.21.15.32:0/3169153866 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b8000fd00 con 0x7f3b8c102e20 2024-04-06T14:26:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.228+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8c119da0 con 0x7f3b8c102e20 2024-04-06T14:26:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.228+0000 7f3b897fa700 1 -- 172.21.15.32:0/3169153866 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b800103a0 con 0x7f3b8c102e20 2024-04-06T14:26:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.228+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8c11a2b0 con 0x7f3b8c102e20 2024-04-06T14:26:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.229+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8c0fe570 con 0x7f3b8c102e20 2024-04-06T14:26:35.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.229+0000 7f3b897fa700 1 -- 172.21.15.32:0/3169153866 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3b8000f850 con 0x7f3b8c102e20 2024-04-06T14:26:35.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.230+0000 7f3b897fa700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b7c06cc30 0x7f3b7c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:35.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.230+0000 7f3b897fa700 1 -- 172.21.15.32:0/3169153866 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3b8008be80 con 0x7f3b8c102e20 2024-04-06T14:26:35.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.230+0000 7f3b8bfff700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b7c06cc30 0x7f3b7c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:35.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.230+0000 7f3b8bfff700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b7c06cc30 0x7f3b7c06f0e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c071c00 tx=0x7f3b78009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:35.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.234+0000 7f3b897fa700 1 -- 172.21.15.32:0/3169153866 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3b80055ec0 con 0x7f3b8c102e20 2024-04-06T14:26:35.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.447+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b8c0719b0 con 0x7f3b7c06cc30 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.458+0000 7f3b897fa700 1 -- 172.21.15.32:0/3169153866 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3b8c0719b0 con 0x7f3b7c06cc30 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 16s ago 18m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (4m) 16s ago 18m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (4m) 2s ago 17m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 16s ago 17m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (41s) 16s ago 12m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (22s) 16s ago 12m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (6s) 2s ago 12m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (12m) 2s ago 12m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (8m) 16s ago 19m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:26:35.458 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (7m) 2s ago 17m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (5m) 16s ago 19m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (4m) 2s ago 17m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (7m) 16s ago 18m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (7m) 2s ago 17m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (3m) 16s ago 16m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (3m) 16s ago 15m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (2m) 16s ago 15m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (2m) 2s ago 14m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (88s) 2s ago 14m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:26:35.459 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (55s) 2s ago 14m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:26:35.460 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 16s ago 17m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:26:35.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b7c06cc30 msgr2=0x7f3b7c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b7c06cc30 0x7f3b7c06f0e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c071c00 tx=0x7f3b78009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:35.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c102e20 msgr2=0x7f3b8c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c102e20 0x7f3b8c0702c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000b790 tx=0x7f3b80009250 comp rx=0 tx=0).stop 2024-04-06T14:26:35.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 shutdown_connections 2024-04-06T14:26:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b7c06cc30 0x7f3b7c06f0e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c102e20 0x7f3b8c0702c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 --2- 172.21.15.32:0/3169153866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c1037c0 0x7f3b8c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 >> 172.21.15.32:0/3169153866 conn(0x7f3b8c0f89d0 msgr2=0x7f3b8c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 shutdown_connections 2024-04-06T14:26:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.460+0000 7f3b92d64700 1 -- 172.21.15.32:0/3169153866 wait complete. 2024-04-06T14:26:35.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.562+0000 7f80c67b8700 1 -- 172.21.15.32:0/3306126150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 msgr2=0x7f80c00fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.562+0000 7f80c67b8700 1 --2- 172.21.15.32:0/3306126150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 0x7f80c00fce20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f80b4004940 tx=0x7f80b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:35.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.562+0000 7f80c67b8700 1 -- 172.21.15.32:0/3306126150 shutdown_connections 2024-04-06T14:26:35.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.562+0000 7f80c67b8700 1 --2- 172.21.15.32:0/3306126150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 0x7f80c00fce20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.562+0000 7f80c67b8700 1 --2- 172.21.15.32:0/3306126150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80c0102e10 0x7f80c01031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.562+0000 7f80c67b8700 1 -- 172.21.15.32:0/3306126150 >> 172.21.15.32:0/3306126150 conn(0x7f80c00f8980 msgr2=0x7f80c00fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:35.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.562+0000 7f80c67b8700 1 -- 172.21.15.32:0/3306126150 shutdown_connections 2024-04-06T14:26:35.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.562+0000 7f80c67b8700 1 -- 172.21.15.32:0/3306126150 wait complete. 2024-04-06T14:26:35.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80c67b8700 1 Processor -- start 2024-04-06T14:26:35.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80c67b8700 1 -- start start 2024-04-06T14:26:35.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80c67b8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80c0102e10 0x7f80c00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80c67b8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 0x7f80c0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80c67b8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c0070d90 con 0x7f80c01037b0 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80c67b8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c0070ed0 con 0x7f80c0102e10 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 0x7f80c0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 0x7f80c0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54318/0 (socket says 172.21.15.32:54318) 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80bf7fe700 1 -- 172.21.15.32:0/975161416 learned_addr learned my addr 172.21.15.32:0/975161416 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.563+0000 7f80bffff700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80c0102e10 0x7f80c00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.564+0000 7f80bf7fe700 1 -- 172.21.15.32:0/975161416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80c0102e10 msgr2=0x7f80c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.564+0000 7f80bf7fe700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80c0102e10 0x7f80c00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.564+0000 7f80bf7fe700 1 -- 172.21.15.32:0/975161416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b4005040 con 0x7f80c01037b0 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.564+0000 7f80bf7fe700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 0x7f80c0070800 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80b4008d50 tx=0x7f80b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.564+0000 7f80bcff9700 1 -- 172.21.15.32:0/975161416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80b400e030 con 0x7f80c01037b0 2024-04-06T14:26:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.564+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c0071120 con 0x7f80c01037b0 2024-04-06T14:26:35.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.564+0000 7f80bcff9700 1 -- 172.21.15.32:0/975161416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80b40093b0 con 0x7f80c01037b0 2024-04-06T14:26:35.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.564+0000 7f80bcff9700 1 -- 172.21.15.32:0/975161416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80b401cbb0 con 0x7f80c01037b0 2024-04-06T14:26:35.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.564+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c011a190 con 0x7f80c01037b0 2024-04-06T14:26:35.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.565+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c00fe560 con 0x7f80c01037b0 2024-04-06T14:26:35.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.565+0000 7f80bcff9700 1 -- 172.21.15.32:0/975161416 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f80b4013930 con 0x7f80c01037b0 2024-04-06T14:26:35.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.566+0000 7f80bcff9700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80a806cc30 0x7f80a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:35.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.566+0000 7f80bcff9700 1 -- 172.21.15.32:0/975161416 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f80b408d810 con 0x7f80c01037b0 2024-04-06T14:26:35.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.566+0000 7f80bffff700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80a806cc30 0x7f80a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:35.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.566+0000 7f80bffff700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80a806cc30 0x7f80a806f0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009b80 tx=0x7f80b000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:35.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.570+0000 7f80bcff9700 1 -- 172.21.15.32:0/975161416 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f80b40577d0 con 0x7f80c01037b0 2024-04-06T14:26:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.865+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f80c0109de0 con 0x7f80c01037b0 2024-04-06T14:26:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.865+0000 7f80bcff9700 1 -- 172.21.15.32:0/975161416 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f80b405adf0 con 0x7f80c01037b0 2024-04-06T14:26:35.866 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:26:35.866 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:26:35.866 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:26:35.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:26:35.866 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:26:35.867 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:26:35.868 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:26:35.868 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:26:35.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.867+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80a806cc30 msgr2=0x7f80a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.867+0000 7f80c67b8700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80a806cc30 0x7f80a806f0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009b80 tx=0x7f80b000b040 comp rx=0 tx=0).stop 2024-04-06T14:26:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.867+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 msgr2=0x7f80c0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.867+0000 7f80c67b8700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 0x7f80c0070800 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80b4008d50 tx=0x7f80b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.867+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 shutdown_connections 2024-04-06T14:26:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.867+0000 7f80c67b8700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80a806cc30 0x7f80a806f0e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.867+0000 7f80c67b8700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80c0102e10 0x7f80c00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.867+0000 7f80c67b8700 1 --2- 172.21.15.32:0/975161416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80c01037b0 0x7f80c0070800 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.867+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 >> 172.21.15.32:0/975161416 conn(0x7f80c00f8980 msgr2=0x7f80c00f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.868+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 shutdown_connections 2024-04-06T14:26:35.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.868+0000 7f80c67b8700 1 -- 172.21.15.32:0/975161416 wait complete. 2024-04-06T14:26:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.971+0000 7f5785dd7700 1 -- 172.21.15.32:0/2096134162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5780101d70 msgr2=0x7f5780102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.971+0000 7f5785dd7700 1 --2- 172.21.15.32:0/2096134162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5780101d70 0x7f5780102140 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5770004940 tx=0x7f5770009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.971+0000 7f5785dd7700 1 -- 172.21.15.32:0/2096134162 shutdown_connections 2024-04-06T14:26:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.971+0000 7f5785dd7700 1 --2- 172.21.15.32:0/2096134162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57800fc5b0 0x7f57800fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.971+0000 7f5785dd7700 1 --2- 172.21.15.32:0/2096134162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5780101d70 0x7f5780102140 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.971+0000 7f5785dd7700 1 -- 172.21.15.32:0/2096134162 >> 172.21.15.32:0/2096134162 conn(0x7f57800714c0 msgr2=0x7f57800718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:35.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.971+0000 7f5785dd7700 1 -- 172.21.15.32:0/2096134162 shutdown_connections 2024-04-06T14:26:35.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.971+0000 7f5785dd7700 1 -- 172.21.15.32:0/2096134162 wait complete. 2024-04-06T14:26:35.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.972+0000 7f5785dd7700 1 Processor -- start 2024-04-06T14:26:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.972+0000 7f5785dd7700 1 -- start start 2024-04-06T14:26:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f5785dd7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800fc5b0 0x7f578010eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f5785dd7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5780101d70 0x7f578010f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f5785dd7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578010f610 con 0x7f57800fc5b0 2024-04-06T14:26:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f5785dd7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578010f750 con 0x7f5780101d70 2024-04-06T14:26:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f577f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800fc5b0 0x7f578010eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f577f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800fc5b0 0x7f578010eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54326/0 (socket says 172.21.15.32:54326) 2024-04-06T14:26:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f577f7fe700 1 -- 172.21.15.32:0/1154076953 learned_addr learned my addr 172.21.15.32:0/1154076953 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f577effd700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5780101d70 0x7f578010f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f577f7fe700 1 -- 172.21.15.32:0/1154076953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5780101d70 msgr2=0x7f578010f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f577f7fe700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5780101d70 0x7f578010f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f577f7fe700 1 -- 172.21.15.32:0/1154076953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5770005040 con 0x7f57800fc5b0 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f577f7fe700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800fc5b0 0x7f578010eb90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5770000c00 tx=0x7f5770009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f5767fff700 1 -- 172.21.15.32:0/1154076953 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5770021070 con 0x7f57800fc5b0 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5780117a40 con 0x7f57800fc5b0 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f5767fff700 1 -- 172.21.15.32:0/1154076953 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5770013de0 con 0x7f57800fc5b0 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f5767fff700 1 -- 172.21.15.32:0/1154076953 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f577000e070 con 0x7f57800fc5b0 2024-04-06T14:26:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.973+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5780117f70 con 0x7f57800fc5b0 2024-04-06T14:26:35.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.974+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57800fd5c0 con 0x7f57800fc5b0 2024-04-06T14:26:35.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.975+0000 7f5767fff700 1 -- 172.21.15.32:0/1154076953 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f577001c600 con 0x7f57800fc5b0 2024-04-06T14:26:35.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.975+0000 7f5767fff700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f576806cc20 0x7f576806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:35.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.975+0000 7f5767fff700 1 -- 172.21.15.32:0/1154076953 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f577008d0b0 con 0x7f57800fc5b0 2024-04-06T14:26:35.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.975+0000 7f577effd700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f576806cc20 0x7f576806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:35.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.976+0000 7f577effd700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f576806cc20 0x7f576806f0d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f57800fbec0 tx=0x7f577400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:35.980+0000 7f5767fff700 1 -- 172.21.15.32:0/1154076953 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f57700570f0 con 0x7f57800fc5b0 2024-04-06T14:26:36.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.231+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f578004bf00 con 0x7f57800fc5b0 2024-04-06T14:26:36.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.232+0000 7f5767fff700 1 -- 172.21.15.32:0/1154076953 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f5770004490 con 0x7f57800fc5b0 2024-04-06T14:26:36.232 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:26:36.232 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:26:36.232 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:26:36.232 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:26:36.232 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:26:36.233 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:26:36.234 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:26:36.235 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:26:36.235 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:26:36.235 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:26:36.235 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:26:36.235 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:26:36.235 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:26:36.235 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:26:36.235 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:26:36.235 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f576806cc20 msgr2=0x7f576806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f576806cc20 0x7f576806f0d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f57800fbec0 tx=0x7f577400b040 comp rx=0 tx=0).stop 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800fc5b0 msgr2=0x7f578010eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800fc5b0 0x7f578010eb90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5770000c00 tx=0x7f5770009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 shutdown_connections 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f576806cc20 0x7f576806f0d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800fc5b0 0x7f578010eb90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 --2- 172.21.15.32:0/1154076953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5780101d70 0x7f578010f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 >> 172.21.15.32:0/1154076953 conn(0x7f57800714c0 msgr2=0x7f5780104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 shutdown_connections 2024-04-06T14:26:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.234+0000 7f5785dd7700 1 -- 172.21.15.32:0/1154076953 wait complete. 2024-04-06T14:26:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 -- 172.21.15.32:0/70558679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 msgr2=0x7f81d80fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 --2- 172.21.15.32:0/70558679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 0x7f81d80fb910 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f81d4004940 tx=0x7f81d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 -- 172.21.15.32:0/70558679 shutdown_connections 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 --2- 172.21.15.32:0/70558679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 0x7f81d80fb910 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 --2- 172.21.15.32:0/70558679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d8102e70 0x7f81d8103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 -- 172.21.15.32:0/70558679 >> 172.21.15.32:0/70558679 conn(0x7f81d80714c0 msgr2=0x7f81d80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 -- 172.21.15.32:0/70558679 shutdown_connections 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 -- 172.21.15.32:0/70558679 wait complete. 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 Processor -- start 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.339+0000 7f81df768700 1 -- start start 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81df768700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 0x7f81d8113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81df768700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d8102e70 0x7f81d8113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81df768700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d81145c0 con 0x7f81d80fb4a0 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81df768700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d810d9b0 con 0x7f81d8102e70 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81dd504700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 0x7f81d8113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81dd504700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 0x7f81d8113a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54338/0 (socket says 172.21.15.32:54338) 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81dd504700 1 -- 172.21.15.32:0/3342733828 learned_addr learned my addr 172.21.15.32:0/3342733828 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81dcd03700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d8102e70 0x7f81d8113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81dd504700 1 -- 172.21.15.32:0/3342733828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d8102e70 msgr2=0x7f81d8113f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81dd504700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d8102e70 0x7f81d8113f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81dd504700 1 -- 172.21.15.32:0/3342733828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d4005040 con 0x7f81d80fb4a0 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81dd504700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 0x7f81d8113a30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f81c800eb40 tx=0x7f81c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81cdffb700 1 -- 172.21.15.32:0/3342733828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81c8007cc0 con 0x7f81d80fb4a0 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d810dc30 con 0x7f81d80fb4a0 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81cdffb700 1 -- 172.21.15.32:0/3342733828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81c800f040 con 0x7f81d80fb4a0 2024-04-06T14:26:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.340+0000 7f81cdffb700 1 -- 172.21.15.32:0/3342733828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81c800c990 con 0x7f81d80fb4a0 2024-04-06T14:26:36.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.341+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d810e1c0 con 0x7f81d80fb4a0 2024-04-06T14:26:36.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.341+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d8100110 con 0x7f81d80fb4a0 2024-04-06T14:26:36.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.342+0000 7f81cdffb700 1 -- 172.21.15.32:0/3342733828 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f81c800cc00 con 0x7f81d80fb4a0 2024-04-06T14:26:36.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.342+0000 7f81cdffb700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81c406cc20 0x7f81c406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:36.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.342+0000 7f81cdffb700 1 -- 172.21.15.32:0/3342733828 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f81c808bc60 con 0x7f81d80fb4a0 2024-04-06T14:26:36.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.342+0000 7f81dcd03700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81c406cc20 0x7f81c406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:36.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.343+0000 7f81dcd03700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81c406cc20 0x7f81c406f0d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81d810ec90 tx=0x7f81d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:36.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.347+0000 7f81cdffb700 1 -- 172.21.15.32:0/3342733828 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f81c8059760 con 0x7f81d80fb4a0 2024-04-06T14:26:36.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:36 smithi086 ceph-mon[110975]: pgmap v193: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:36.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:36 smithi086 ceph-mon[110975]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:36.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:36 smithi086 ceph-mon[110975]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:36.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:36 smithi086 ceph-mon[110975]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:36.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:36 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/975161416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:36 smithi032 ceph-mon[147414]: pgmap v193: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:36 smithi032 ceph-mon[147414]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:36 smithi032 ceph-mon[147414]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:36 smithi032 ceph-mon[147414]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:36 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/975161416' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:26:36.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.567+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f81d810f0f0 con 0x7f81c406cc20 2024-04-06T14:26:36.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.568+0000 7f81cdffb700 1 -- 172.21.15.32:0/3342733828 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f81d810f0f0 con 0x7f81c406cc20 2024-04-06T14:26:36.568 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:26:36.569 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81c406cc20 msgr2=0x7f81c406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81c406cc20 0x7f81c406f0d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81d810ec90 tx=0x7f81d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 msgr2=0x7f81d8113a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 0x7f81d8113a30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f81c800eb40 tx=0x7f81c800b040 comp rx=0 tx=0).stop 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 shutdown_connections 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81c406cc20 0x7f81c406f0d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d80fb4a0 0x7f81d8113a30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 --2- 172.21.15.32:0/3342733828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d8102e70 0x7f81d8113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 >> 172.21.15.32:0/3342733828 conn(0x7f81d80714c0 msgr2=0x7f81d80fa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 shutdown_connections 2024-04-06T14:26:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.570+0000 7f81df768700 1 -- 172.21.15.32:0/3342733828 wait complete. 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.674+0000 7fd5bde40700 1 -- 172.21.15.32:0/674112865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fe830 msgr2=0x7fd5b8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.674+0000 7fd5bde40700 1 --2- 172.21.15.32:0/674112865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fe830 0x7fd5b8106f00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8004940 tx=0x7fd5a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.674+0000 7fd5bde40700 1 -- 172.21.15.32:0/674112865 shutdown_connections 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.674+0000 7fd5bde40700 1 --2- 172.21.15.32:0/674112865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fe830 0x7fd5b8106f00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.674+0000 7fd5bde40700 1 --2- 172.21.15.32:0/674112865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b80fde90 0x7fd5b80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.674+0000 7fd5bde40700 1 -- 172.21.15.32:0/674112865 >> 172.21.15.32:0/674112865 conn(0x7fd5b8072bc0 msgr2=0x7fd5b8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.674+0000 7fd5bde40700 1 -- 172.21.15.32:0/674112865 shutdown_connections 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.674+0000 7fd5bde40700 1 -- 172.21.15.32:0/674112865 wait complete. 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5bde40700 1 Processor -- start 2024-04-06T14:26:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5bde40700 1 -- start start 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5bde40700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fde90 0x7fd5b8113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5bde40700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b80fe830 0x7fd5b8114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5bde40700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b8114670 con 0x7fd5b80fde90 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5bde40700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b810da60 con 0x7fd5b80fe830 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5b77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fde90 0x7fd5b8113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5b77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fde90 0x7fd5b8113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54366/0 (socket says 172.21.15.32:54366) 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5b77fe700 1 -- 172.21.15.32:0/2620063428 learned_addr learned my addr 172.21.15.32:0/2620063428 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5b6ffd700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b80fe830 0x7fd5b8114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5b77fe700 1 -- 172.21.15.32:0/2620063428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b80fe830 msgr2=0x7fd5b8114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5b77fe700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b80fe830 0x7fd5b8114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.675+0000 7fd5b77fe700 1 -- 172.21.15.32:0/2620063428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8005040 con 0x7fd5b80fde90 2024-04-06T14:26:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.676+0000 7fd5b77fe700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fde90 0x7fd5b8113ae0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac002a50 tx=0x7fd5ac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.676+0000 7fd59ffff700 1 -- 172.21.15.32:0/2620063428 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5ac007cc0 con 0x7fd5b80fde90 2024-04-06T14:26:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.676+0000 7fd59ffff700 1 -- 172.21.15.32:0/2620063428 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5ac00f040 con 0x7fd5b80fde90 2024-04-06T14:26:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.676+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b810dd40 con 0x7fd5b80fde90 2024-04-06T14:26:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.676+0000 7fd59ffff700 1 -- 172.21.15.32:0/2620063428 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5ac00d9c0 con 0x7fd5b80fde90 2024-04-06T14:26:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.676+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b810e280 con 0x7fd5b80fde90 2024-04-06T14:26:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.677+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b8065e30 con 0x7fd5b80fde90 2024-04-06T14:26:36.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.677+0000 7fd59ffff700 1 -- 172.21.15.32:0/2620063428 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd5ac00db20 con 0x7fd5b80fde90 2024-04-06T14:26:36.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.678+0000 7fd59ffff700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5a006cc20 0x7fd5a006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:26:36.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.678+0000 7fd59ffff700 1 -- 172.21.15.32:0/2620063428 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd5ac08bcb0 con 0x7fd5b80fde90 2024-04-06T14:26:36.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.678+0000 7fd5b6ffd700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5a006cc20 0x7fd5a006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:26:36.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.678+0000 7fd5b6ffd700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5a006cc20 0x7fd5a006f0d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd5b810ec20 tx=0x7fd5a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:26:36.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.682+0000 7fd59ffff700 1 -- 172.21.15.32:0/2620063428 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd5ac059780 con 0x7fd5b80fde90 2024-04-06T14:26:36.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.966+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd5b8109e30 con 0x7fd5b80fde90 2024-04-06T14:26:36.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.966+0000 7fd59ffff700 1 -- 172.21.15.32:0/2620063428 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fd5ac059310 con 0x7fd5b80fde90 2024-04-06T14:26:36.967 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:26:36.967 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:26:36.967 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:26:36.967 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:26:36.967 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:26:36.967 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:26:36.967 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5a006cc20 msgr2=0x7fd5a006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5a006cc20 0x7fd5a006f0d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd5b810ec20 tx=0x7fd5a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fde90 msgr2=0x7fd5b8113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fde90 0x7fd5b8113ae0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac002a50 tx=0x7fd5ac0046b0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 shutdown_connections 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5a006cc20 0x7fd5a006f0d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5b80fde90 0x7fd5b8113ae0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 --2- 172.21.15.32:0/2620063428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b80fe830 0x7fd5b8114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 >> 172.21.15.32:0/2620063428 conn(0x7fd5b8072bc0 msgr2=0x7fd5b8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 shutdown_connections 2024-04-06T14:26:36.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:26:36.969+0000 7fd5bde40700 1 -- 172.21.15.32:0/2620063428 wait complete. 2024-04-06T14:26:37.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:37 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1154076953' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:26:37.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:37 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2620063428' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:26:37.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:37 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1154076953' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:26:37.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:37 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2620063428' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:26:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:38 smithi086 ceph-mon[110975]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:38 smithi086 ceph-mon[110975]: pgmap v194: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:26:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:38 smithi032 ceph-mon[147414]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:26:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:38 smithi032 ceph-mon[147414]: pgmap v194: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:26:40.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:40 smithi086 ceph-mon[110975]: pgmap v195: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:40.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:40 smithi032 ceph-mon[147414]: pgmap v195: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:42.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:42 smithi086 ceph-mon[110975]: pgmap v196: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:42.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:42 smithi032 ceph-mon[147414]: pgmap v196: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:44.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:44 smithi086 ceph-mon[110975]: pgmap v197: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:44.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:44 smithi032 ceph-mon[147414]: pgmap v197: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:46.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:46 smithi086 ceph-mon[110975]: pgmap v198: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:46 smithi032 ceph-mon[147414]: pgmap v198: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:48.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:48 smithi086 ceph-mon[110975]: pgmap v199: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:48.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:48 smithi032 ceph-mon[147414]: pgmap v199: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:50 smithi086 ceph-mon[110975]: pgmap v200: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:50 smithi032 ceph-mon[147414]: pgmap v200: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:52.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:52 smithi086 ceph-mon[110975]: pgmap v201: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:52 smithi032 ceph-mon[147414]: pgmap v201: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:53.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:26:53.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:26:54.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:54 smithi086 ceph-mon[110975]: pgmap v202: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:54.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:54 smithi032 ceph-mon[147414]: pgmap v202: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:56.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:56 smithi086 ceph-mon[110975]: pgmap v203: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:56.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:56 smithi032 ceph-mon[147414]: pgmap v203: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:58.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:26:58 smithi086 ceph-mon[110975]: pgmap v204: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:26:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:26:58 smithi032 ceph-mon[147414]: pgmap v204: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:00.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:00 smithi086 ceph-mon[110975]: pgmap v205: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:00 smithi032 ceph-mon[147414]: pgmap v205: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:02 smithi086 ceph-mon[110975]: pgmap v206: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:02 smithi032 ceph-mon[147414]: pgmap v206: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:04.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:04 smithi086 ceph-mon[110975]: pgmap v207: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:04 smithi032 ceph-mon[147414]: pgmap v207: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:06.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:06 smithi086 ceph-mon[110975]: pgmap v208: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:06.481 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:06 smithi032 ceph-mon[147414]: pgmap v208: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.087+0000 7f3d36c60700 1 -- 172.21.15.32:0/2108587889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 msgr2=0x7f3d301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.087+0000 7f3d36c60700 1 --2- 172.21.15.32:0/2108587889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 0x7f3d301031d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3d24004940 tx=0x7f3d24009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.088+0000 7f3d36c60700 1 -- 172.21.15.32:0/2108587889 shutdown_connections 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.088+0000 7f3d36c60700 1 --2- 172.21.15.32:0/2108587889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d301037a0 0x7f3d300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.088+0000 7f3d36c60700 1 --2- 172.21.15.32:0/2108587889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 0x7f3d301031d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.088+0000 7f3d36c60700 1 -- 172.21.15.32:0/2108587889 >> 172.21.15.32:0/2108587889 conn(0x7f3d300f89b0 msgr2=0x7f3d300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.088+0000 7f3d36c60700 1 -- 172.21.15.32:0/2108587889 shutdown_connections 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.088+0000 7f3d36c60700 1 -- 172.21.15.32:0/2108587889 wait complete. 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.088+0000 7f3d36c60700 1 Processor -- start 2024-04-06T14:27:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.088+0000 7f3d36c60700 1 -- start start 2024-04-06T14:27:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d36c60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 0x7f3d30113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d36c60700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d301037a0 0x7f3d30114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d36c60700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30114650 con 0x7f3d30102e00 2024-04-06T14:27:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d36c60700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010da40 con 0x7f3d301037a0 2024-04-06T14:27:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d349fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 0x7f3d30113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d349fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 0x7f3d30113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44500/0 (socket says 172.21.15.32:44500) 2024-04-06T14:27:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d349fc700 1 -- 172.21.15.32:0/4102359759 learned_addr learned my addr 172.21.15.32:0/4102359759 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d2ffff700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d301037a0 0x7f3d30114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d349fc700 1 -- 172.21.15.32:0/4102359759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d301037a0 msgr2=0x7f3d30114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d349fc700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d301037a0 0x7f3d30114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.089+0000 7f3d349fc700 1 -- 172.21.15.32:0/4102359759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d24005040 con 0x7f3d30102e00 2024-04-06T14:27:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.090+0000 7f3d349fc700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 0x7f3d30113ac0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009360 tx=0x7f3d24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.090+0000 7f3d2d7fa700 1 -- 172.21.15.32:0/4102359759 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d2400e040 con 0x7f3d30102e00 2024-04-06T14:27:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.090+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3010dcc0 con 0x7f3d30102e00 2024-04-06T14:27:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.090+0000 7f3d2d7fa700 1 -- 172.21.15.32:0/4102359759 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d24013420 con 0x7f3d30102e00 2024-04-06T14:27:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.090+0000 7f3d2d7fa700 1 -- 172.21.15.32:0/4102359759 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d2401c5d0 con 0x7f3d30102e00 2024-04-06T14:27:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.090+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3010e280 con 0x7f3d30102e00 2024-04-06T14:27:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.091+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d300fe550 con 0x7f3d30102e00 2024-04-06T14:27:07.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.092+0000 7f3d2d7fa700 1 -- 172.21.15.32:0/4102359759 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3d24013590 con 0x7f3d30102e00 2024-04-06T14:27:07.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.092+0000 7f3d2d7fa700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3d2006cc30 0x7f3d2006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:07.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.092+0000 7f3d2d7fa700 1 -- 172.21.15.32:0/4102359759 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3d2408dce0 con 0x7f3d30102e00 2024-04-06T14:27:07.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.092+0000 7f3d2ffff700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3d2006cc30 0x7f3d2006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:07.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.093+0000 7f3d2ffff700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3d2006cc30 0x7f3d2006f0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3d3010ec00 tx=0x7f3d1c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:07.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.096+0000 7f3d2d7fa700 1 -- 172.21.15.32:0/4102359759 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3d2405b8d0 con 0x7f3d30102e00 2024-04-06T14:27:07.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.316+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d3004bed0 con 0x7f3d2006cc30 2024-04-06T14:27:07.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.318+0000 7f3d2d7fa700 1 -- 172.21.15.32:0/4102359759 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3d3004bed0 con 0x7f3d2006cc30 2024-04-06T14:27:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3d2006cc30 msgr2=0x7f3d2006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3d2006cc30 0x7f3d2006f0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3d3010ec00 tx=0x7f3d1c00b040 comp rx=0 tx=0).stop 2024-04-06T14:27:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 msgr2=0x7f3d30113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 0x7f3d30113ac0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009360 tx=0x7f3d24009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 shutdown_connections 2024-04-06T14:27:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3d2006cc30 0x7f3d2006f0e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d30102e00 0x7f3d30113ac0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 --2- 172.21.15.32:0/4102359759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d301037a0 0x7f3d30114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 >> 172.21.15.32:0/4102359759 conn(0x7f3d300f89b0 msgr2=0x7f3d300f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 shutdown_connections 2024-04-06T14:27:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.320+0000 7f3d36c60700 1 -- 172.21.15.32:0/4102359759 wait complete. 2024-04-06T14:27:07.331 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.425+0000 7f50b2a8d700 1 -- 172.21.15.32:0/2104421083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 msgr2=0x7f50ac102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.425+0000 7f50b2a8d700 1 --2- 172.21.15.32:0/2104421083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 0x7f50ac102140 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f50a0004940 tx=0x7f50a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.425+0000 7f50b2a8d700 1 -- 172.21.15.32:0/2104421083 shutdown_connections 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.425+0000 7f50b2a8d700 1 --2- 172.21.15.32:0/2104421083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50ac102710 0x7f50ac0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.425+0000 7f50b2a8d700 1 --2- 172.21.15.32:0/2104421083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 0x7f50ac102140 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.425+0000 7f50b2a8d700 1 -- 172.21.15.32:0/2104421083 >> 172.21.15.32:0/2104421083 conn(0x7f50ac072bc0 msgr2=0x7f50ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.425+0000 7f50b2a8d700 1 -- 172.21.15.32:0/2104421083 shutdown_connections 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.425+0000 7f50b2a8d700 1 -- 172.21.15.32:0/2104421083 wait complete. 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b2a8d700 1 Processor -- start 2024-04-06T14:27:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b2a8d700 1 -- start start 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b2a8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 0x7f50ac10eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b2a8d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50ac102710 0x7f50ac10f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b2a8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac10f5e0 con 0x7f50ac101d70 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b2a8d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac10f720 con 0x7f50ac102710 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b0829700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 0x7f50ac10eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b0829700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 0x7f50ac10eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44518/0 (socket says 172.21.15.32:44518) 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b0829700 1 -- 172.21.15.32:0/1472996574 learned_addr learned my addr 172.21.15.32:0/1472996574 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50abfff700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50ac102710 0x7f50ac10f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b0829700 1 -- 172.21.15.32:0/1472996574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50ac102710 msgr2=0x7f50ac10f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b0829700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50ac102710 0x7f50ac10f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.426+0000 7f50b0829700 1 -- 172.21.15.32:0/1472996574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a0005040 con 0x7f50ac101d70 2024-04-06T14:27:07.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.427+0000 7f50b0829700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 0x7f50ac10eb60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f50a0008d80 tx=0x7f50a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:07.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.427+0000 7f50a97fa700 1 -- 172.21.15.32:0/1472996574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50a000e030 con 0x7f50ac101d70 2024-04-06T14:27:07.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.427+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ac117a10 con 0x7f50ac101d70 2024-04-06T14:27:07.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.427+0000 7f50a97fa700 1 -- 172.21.15.32:0/1472996574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50a0013de0 con 0x7f50ac101d70 2024-04-06T14:27:07.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.427+0000 7f50a97fa700 1 -- 172.21.15.32:0/1472996574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50a001cd60 con 0x7f50ac101d70 2024-04-06T14:27:07.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.427+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ac117fa0 con 0x7f50ac101d70 2024-04-06T14:27:07.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.428+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ac0ff080 con 0x7f50ac101d70 2024-04-06T14:27:07.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.428+0000 7f50a97fa700 1 -- 172.21.15.32:0/1472996574 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f50a001c590 con 0x7f50ac101d70 2024-04-06T14:27:07.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.429+0000 7f50a97fa700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f509c06cc30 0x7f509c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:07.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.429+0000 7f50a97fa700 1 -- 172.21.15.32:0/1472996574 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f50a008d950 con 0x7f50ac101d70 2024-04-06T14:27:07.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.429+0000 7f50abfff700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f509c06cc30 0x7f509c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:07.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.429+0000 7f50abfff700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f509c06cc30 0x7f509c06f0e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5098004600 tx=0x7f509800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:07.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.433+0000 7f50a97fa700 1 -- 172.21.15.32:0/1472996574 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f50a0057a40 con 0x7f50ac101d70 2024-04-06T14:27:07.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.654+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50ac04bed0 con 0x7f509c06cc30 2024-04-06T14:27:07.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.655+0000 7f50a97fa700 1 -- 172.21.15.32:0/1472996574 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f50ac04bed0 con 0x7f509c06cc30 2024-04-06T14:27:07.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f509c06cc30 msgr2=0x7f509c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f509c06cc30 0x7f509c06f0e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5098004600 tx=0x7f509800b040 comp rx=0 tx=0).stop 2024-04-06T14:27:07.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 msgr2=0x7f50ac10eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 0x7f50ac10eb60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f50a0008d80 tx=0x7f50a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:07.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 shutdown_connections 2024-04-06T14:27:07.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f509c06cc30 0x7f509c06f0e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ac101d70 0x7f50ac10eb60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 --2- 172.21.15.32:0/1472996574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50ac102710 0x7f50ac10f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 >> 172.21.15.32:0/1472996574 conn(0x7f50ac072bc0 msgr2=0x7f50ac0faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:07.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 shutdown_connections 2024-04-06T14:27:07.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.658+0000 7f50b2a8d700 1 -- 172.21.15.32:0/1472996574 wait complete. 2024-04-06T14:27:07.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.763+0000 7fed19d1d700 1 -- 172.21.15.32:0/1488601903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed141037a0 msgr2=0x7fed140fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.763+0000 7fed19d1d700 1 --2- 172.21.15.32:0/1488601903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed141037a0 0x7fed140fce10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fed08004940 tx=0x7fed08009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:07.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.763+0000 7fed19d1d700 1 -- 172.21.15.32:0/1488601903 shutdown_connections 2024-04-06T14:27:07.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.763+0000 7fed19d1d700 1 --2- 172.21.15.32:0/1488601903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed141037a0 0x7fed140fce10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.763+0000 7fed19d1d700 1 --2- 172.21.15.32:0/1488601903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14102e00 0x7fed141031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.763+0000 7fed19d1d700 1 -- 172.21.15.32:0/1488601903 >> 172.21.15.32:0/1488601903 conn(0x7fed140f89b0 msgr2=0x7fed140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:07.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.763+0000 7fed19d1d700 1 -- 172.21.15.32:0/1488601903 shutdown_connections 2024-04-06T14:27:07.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.764+0000 7fed19d1d700 1 -- 172.21.15.32:0/1488601903 wait complete. 2024-04-06T14:27:07.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.764+0000 7fed19d1d700 1 Processor -- start 2024-04-06T14:27:07.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.764+0000 7fed19d1d700 1 -- start start 2024-04-06T14:27:07.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.764+0000 7fed19d1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed14102e00 0x7fed14113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:07.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.764+0000 7fed19d1d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed141037a0 0x7fed14114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:07.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.764+0000 7fed19d1d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed14114690 con 0x7fed14102e00 2024-04-06T14:27:07.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.764+0000 7fed19d1d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1410da80 con 0x7fed141037a0 2024-04-06T14:27:07.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed14102e00 0x7fed14113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:07.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed14102e00 0x7fed14113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44538/0 (socket says 172.21.15.32:44538) 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed137fe700 1 -- 172.21.15.32:0/300938749 learned_addr learned my addr 172.21.15.32:0/300938749 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed12ffd700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed141037a0 0x7fed14114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed137fe700 1 -- 172.21.15.32:0/300938749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed141037a0 msgr2=0x7fed14114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed137fe700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed141037a0 0x7fed14114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed137fe700 1 -- 172.21.15.32:0/300938749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08005040 con 0x7fed14102e00 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed12ffd700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed141037a0 0x7fed14114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed137fe700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed14102e00 0x7fed14113b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fed04002a50 tx=0x7fed040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fecfbfff700 1 -- 172.21.15.32:0/300938749 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed04007cc0 con 0x7fed14102e00 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fecfbfff700 1 -- 172.21.15.32:0/300938749 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed0400f040 con 0x7fed14102e00 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fecfbfff700 1 -- 172.21.15.32:0/300938749 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed04009540 con 0x7fed14102e00 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1410dd60 con 0x7fed14102e00 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.765+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1410e320 con 0x7fed14102e00 2024-04-06T14:27:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.766+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed140fe550 con 0x7fed14102e00 2024-04-06T14:27:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.767+0000 7fecfbfff700 1 -- 172.21.15.32:0/300938749 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fed040096a0 con 0x7fed14102e00 2024-04-06T14:27:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.767+0000 7fecfbfff700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecfc06cc20 0x7fecfc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.767+0000 7fecfbfff700 1 -- 172.21.15.32:0/300938749 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fed0408be70 con 0x7fed14102e00 2024-04-06T14:27:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.767+0000 7fed12ffd700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecfc06cc20 0x7fecfc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:07.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.768+0000 7fed12ffd700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecfc06cc20 0x7fecfc06f0d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fed1410ec40 tx=0x7fed08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:07.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.772+0000 7fecfbfff700 1 -- 172.21.15.32:0/300938749 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fed040598c0 con 0x7fed14102e00 2024-04-06T14:27:07.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.983+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fed14109da0 con 0x7fecfc06cc20 2024-04-06T14:27:07.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.993+0000 7fecfbfff700 1 -- 172.21.15.32:0/300938749 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fed14109da0 con 0x7fecfc06cc20 2024-04-06T14:27:07.994 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:27:07.994 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (7m) 49s ago 18m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:27:07.994 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (4m) 49s ago 18m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:27:07.994 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (4m) 35s ago 17m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:27:07.994 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 49s ago 18m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:27:07.994 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (73s) 49s ago 13m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:27:07.994 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (54s) 49s ago 12m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:27:07.994 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (38s) 35s ago 13m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:27:07.994 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (12m) 35s ago 12m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:27:07.995 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (9m) 49s ago 20m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:27:07.995 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (8m) 35s ago 17m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:27:07.995 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (5m) 49s ago 20m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:27:07.995 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (5m) 35s ago 17m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:27:07.995 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 49s ago 18m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:27:07.995 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (8m) 35s ago 17m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:27:07.995 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (4m) 49s ago 16m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:27:07.995 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (3m) 49s ago 16m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:27:07.995 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (3m) 49s ago 15m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:27:07.996 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (2m) 35s ago 15m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:27:07.996 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (2m) 35s ago 15m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:27:07.996 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (88s) 35s ago 14m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:27:07.996 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 49s ago 18m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:27:07.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.995+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecfc06cc20 msgr2=0x7fecfc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.995+0000 7fed19d1d700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecfc06cc20 0x7fecfc06f0d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fed1410ec40 tx=0x7fed08009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:07.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.995+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed14102e00 msgr2=0x7fed14113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:07.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.995+0000 7fed19d1d700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed14102e00 0x7fed14113b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fed04002a50 tx=0x7fed040046b0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.995+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 shutdown_connections 2024-04-06T14:27:07.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.995+0000 7fed19d1d700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecfc06cc20 0x7fecfc06f0d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.995+0000 7fed19d1d700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed14102e00 0x7fed14113b00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.995+0000 7fed19d1d700 1 --2- 172.21.15.32:0/300938749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed141037a0 0x7fed14114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:07.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.996+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 >> 172.21.15.32:0/300938749 conn(0x7fed140f89b0 msgr2=0x7fed140f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:07.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.996+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 shutdown_connections 2024-04-06T14:27:07.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:07.996+0000 7fed19d1d700 1 -- 172.21.15.32:0/300938749 wait complete. 2024-04-06T14:27:08.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 -- 172.21.15.32:0/393549319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fde90 msgr2=0x7fe5040fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 --2- 172.21.15.32:0/393549319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fde90 0x7fe5040fe260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8004940 tx=0x7fe4f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 -- 172.21.15.32:0/393549319 shutdown_connections 2024-04-06T14:27:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 --2- 172.21.15.32:0/393549319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5040fe830 0x7fe504106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 --2- 172.21.15.32:0/393549319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fde90 0x7fe5040fe260 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 -- 172.21.15.32:0/393549319 >> 172.21.15.32:0/393549319 conn(0x7fe504072bc0 msgr2=0x7fe504072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 -- 172.21.15.32:0/393549319 shutdown_connections 2024-04-06T14:27:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 -- 172.21.15.32:0/393549319 wait complete. 2024-04-06T14:27:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 Processor -- start 2024-04-06T14:27:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.100+0000 7fe50919d700 1 -- start start 2024-04-06T14:27:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50919d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5040fde90 0x7fe5040fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50919d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fe830 0x7fe5040fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50919d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040fc820 con 0x7fe5040fe830 2024-04-06T14:27:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50919d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040fc960 con 0x7fe5040fde90 2024-04-06T14:27:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fe830 0x7fe5040fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fe830 0x7fe5040fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44566/0 (socket says 172.21.15.32:44566) 2024-04-06T14:27:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50259c700 1 -- 172.21.15.32:0/3014070705 learned_addr learned my addr 172.21.15.32:0/3014070705 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe502d9d700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5040fde90 0x7fe5040fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50259c700 1 -- 172.21.15.32:0/3014070705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5040fde90 msgr2=0x7fe5040fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50259c700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5040fde90 0x7fe5040fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50259c700 1 -- 172.21.15.32:0/3014070705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f8005040 con 0x7fe5040fe830 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50259c700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fe830 0x7fe5040fc290 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4002a50 tx=0x7fe4f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe4eb7fe700 1 -- 172.21.15.32:0/3014070705 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4f4007cc0 con 0x7fe5040fe830 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe504119df0 con 0x7fe5040fe830 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.101+0000 7fe4eb7fe700 1 -- 172.21.15.32:0/3014070705 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4f400f040 con 0x7fe5040fe830 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.102+0000 7fe4eb7fe700 1 -- 172.21.15.32:0/3014070705 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4f400d9c0 con 0x7fe5040fe830 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.102+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50411a1f0 con 0x7fe5040fe830 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.102+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe504065e30 con 0x7fe5040fe830 2024-04-06T14:27:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.103+0000 7fe4eb7fe700 1 -- 172.21.15.32:0/3014070705 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe4f400db20 con 0x7fe5040fe830 2024-04-06T14:27:08.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.103+0000 7fe4eb7fe700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4ec06cc30 0x7fe4ec06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:08.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.104+0000 7fe4eb7fe700 1 -- 172.21.15.32:0/3014070705 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe4f408bdb0 con 0x7fe5040fe830 2024-04-06T14:27:08.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.104+0000 7fe502d9d700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4ec06cc30 0x7fe4ec06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:08.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.104+0000 7fe502d9d700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4ec06cc30 0x7fe4ec06f0e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe4f800b3b0 tx=0x7fe4f8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:08.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.108+0000 7fe4eb7fe700 1 -- 172.21.15.32:0/3014070705 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe4f4055d70 con 0x7fe5040fe830 2024-04-06T14:27:08.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:08 smithi086 ceph-mon[110975]: pgmap v209: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:08 smithi086 ceph-mon[110975]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:27:08.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.395+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe50404bed0 con 0x7fe5040fe830 2024-04-06T14:27:08.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.395+0000 7fe4eb7fe700 1 -- 172.21.15.32:0/3014070705 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fe4f4059390 con 0x7fe5040fe830 2024-04-06T14:27:08.395 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:27:08.395 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:27:08.395 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:27:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:27:08.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:27:08.397 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:27:08.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4ec06cc30 msgr2=0x7fe4ec06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4ec06cc30 0x7fe4ec06f0e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe4f800b3b0 tx=0x7fe4f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:08.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fe830 msgr2=0x7fe5040fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fe830 0x7fe5040fc290 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4002a50 tx=0x7fe4f40046b0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 shutdown_connections 2024-04-06T14:27:08.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4ec06cc30 0x7fe4ec06f0e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5040fde90 0x7fe5040fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 --2- 172.21.15.32:0/3014070705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5040fe830 0x7fe5040fc290 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 >> 172.21.15.32:0/3014070705 conn(0x7fe504072bc0 msgr2=0x7fe5040ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:08.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 shutdown_connections 2024-04-06T14:27:08.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.397+0000 7fe50919d700 1 -- 172.21.15.32:0/3014070705 wait complete. 2024-04-06T14:27:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:08 smithi032 ceph-mon[147414]: pgmap v209: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:08 smithi032 ceph-mon[147414]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:27:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.501+0000 7fbd73753700 1 -- 172.21.15.32:0/823955311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 msgr2=0x7fbd6c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.501+0000 7fbd73753700 1 --2- 172.21.15.32:0/823955311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 0x7fbd6c0fce20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbd68004940 tx=0x7fbd68009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.501+0000 7fbd73753700 1 -- 172.21.15.32:0/823955311 shutdown_connections 2024-04-06T14:27:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.501+0000 7fbd73753700 1 --2- 172.21.15.32:0/823955311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 0x7fbd6c0fce20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.501+0000 7fbd73753700 1 --2- 172.21.15.32:0/823955311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd6c102e10 0x7fbd6c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.501+0000 7fbd73753700 1 -- 172.21.15.32:0/823955311 >> 172.21.15.32:0/823955311 conn(0x7fbd6c0f8980 msgr2=0x7fbd6c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.501+0000 7fbd73753700 1 -- 172.21.15.32:0/823955311 shutdown_connections 2024-04-06T14:27:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.501+0000 7fbd73753700 1 -- 172.21.15.32:0/823955311 wait complete. 2024-04-06T14:27:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.502+0000 7fbd73753700 1 Processor -- start 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.502+0000 7fbd73753700 1 -- start start 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.502+0000 7fbd73753700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd6c102e10 0x7fbd6c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.502+0000 7fbd73753700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 0x7fbd6c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.502+0000 7fbd73753700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c070e20 con 0x7fbd6c1037b0 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.502+0000 7fbd73753700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c070f60 con 0x7fbd6c102e10 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.502+0000 7fbd70cee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 0x7fbd6c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.502+0000 7fbd70cee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 0x7fbd6c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44576/0 (socket says 172.21.15.32:44576) 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.502+0000 7fbd70cee700 1 -- 172.21.15.32:0/949221938 learned_addr learned my addr 172.21.15.32:0/949221938 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd714ef700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd6c102e10 0x7fbd6c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd70cee700 1 -- 172.21.15.32:0/949221938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd6c102e10 msgr2=0x7fbd6c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd70cee700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd6c102e10 0x7fbd6c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd70cee700 1 -- 172.21.15.32:0/949221938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd68005040 con 0x7fbd6c1037b0 2024-04-06T14:27:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd714ef700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd6c102e10 0x7fbd6c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:27:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd70cee700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 0x7fbd6c070800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbd680093b0 tx=0x7fbd68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd61ffb700 1 -- 172.21.15.32:0/949221938 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd6800e040 con 0x7fbd6c1037b0 2024-04-06T14:27:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd61ffb700 1 -- 172.21.15.32:0/949221938 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd68013420 con 0x7fbd6c1037b0 2024-04-06T14:27:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c119dd0 con 0x7fbd6c1037b0 2024-04-06T14:27:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd61ffb700 1 -- 172.21.15.32:0/949221938 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd6801c630 con 0x7fbd6c1037b0 2024-04-06T14:27:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.503+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c11a2b0 con 0x7fbd6c1037b0 2024-04-06T14:27:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.504+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6c0fe560 con 0x7fbd6c1037b0 2024-04-06T14:27:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.505+0000 7fbd61ffb700 1 -- 172.21.15.32:0/949221938 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbd680139e0 con 0x7fbd6c1037b0 2024-04-06T14:27:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.505+0000 7fbd61ffb700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd5806cc20 0x7fbd5806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.505+0000 7fbd61ffb700 1 -- 172.21.15.32:0/949221938 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbd6808d2b0 con 0x7fbd6c1037b0 2024-04-06T14:27:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.505+0000 7fbd714ef700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd5806cc20 0x7fbd5806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.506+0000 7fbd714ef700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd5806cc20 0x7fbd5806f0d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009b10 tx=0x7fbd5c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.510+0000 7fbd61ffb700 1 -- 172.21.15.32:0/949221938 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbd680573a0 con 0x7fbd6c1037b0 2024-04-06T14:27:08.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.759+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbd6c04bed0 con 0x7fbd6c1037b0 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.759+0000 7fbd61ffb700 1 -- 172.21.15.32:0/949221938 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fbd6805a9c0 con 0x7fbd6c1037b0 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:27:08.760 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:27:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:27:08.762 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:27:08.763 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:27:08.763 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:27:08.763 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:27:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd5806cc20 msgr2=0x7fbd5806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd5806cc20 0x7fbd5806f0d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009b10 tx=0x7fbd5c008040 comp rx=0 tx=0).stop 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 msgr2=0x7fbd6c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 0x7fbd6c070800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbd680093b0 tx=0x7fbd68009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 shutdown_connections 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd5806cc20 0x7fbd5806f0d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd6c102e10 0x7fbd6c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 --2- 172.21.15.32:0/949221938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd6c1037b0 0x7fbd6c070800 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 >> 172.21.15.32:0/949221938 conn(0x7fbd6c0f8980 msgr2=0x7fbd6c0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 shutdown_connections 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.762+0000 7fbd73753700 1 -- 172.21.15.32:0/949221938 wait complete. 2024-04-06T14:27:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:27:08.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 -- 172.21.15.32:0/3043281101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 msgr2=0x7fd400106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 --2- 172.21.15.32:0/3043281101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 0x7fd400106e80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4004940 tx=0x7fd3f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:08.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 -- 172.21.15.32:0/3043281101 shutdown_connections 2024-04-06T14:27:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 --2- 172.21.15.32:0/3043281101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 0x7fd400106e80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 --2- 172.21.15.32:0/3043281101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4000fde10 0x7fd4000fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 -- 172.21.15.32:0/3043281101 >> 172.21.15.32:0/3043281101 conn(0x7fd4000f78d0 msgr2=0x7fd4000f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 -- 172.21.15.32:0/3043281101 shutdown_connections 2024-04-06T14:27:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 -- 172.21.15.32:0/3043281101 wait complete. 2024-04-06T14:27:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 Processor -- start 2024-04-06T14:27:08.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.866+0000 7fd404bd4700 1 -- start start 2024-04-06T14:27:08.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd404bd4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4000fde10 0x7fd4000fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:08.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd404bd4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 0x7fd4000fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:08.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd404bd4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4000fc7a0 con 0x7fd4000fe7b0 2024-04-06T14:27:08.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd404bd4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4000fc8e0 con 0x7fd4000fde10 2024-04-06T14:27:08.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3fdd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 0x7fd4000fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:08.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3fdd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 0x7fd4000fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44596/0 (socket says 172.21.15.32:44596) 2024-04-06T14:27:08.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3fdd9b700 1 -- 172.21.15.32:0/3257905844 learned_addr learned my addr 172.21.15.32:0/3257905844 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:08.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3fdd9b700 1 -- 172.21.15.32:0/3257905844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4000fde10 msgr2=0x7fd4000fbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:08.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3fe59c700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4000fde10 0x7fd4000fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:08.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3fdd9b700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4000fde10 0x7fd4000fbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:08.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3fdd9b700 1 -- 172.21.15.32:0/3257905844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f4005040 con 0x7fd4000fe7b0 2024-04-06T14:27:08.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3fdd9b700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 0x7fd4000fc210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009390 tx=0x7fd3f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:08.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3eeffd700 1 -- 172.21.15.32:0/3257905844 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3f400e040 con 0x7fd4000fe7b0 2024-04-06T14:27:08.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3eeffd700 1 -- 172.21.15.32:0/3257905844 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3f4013420 con 0x7fd4000fe7b0 2024-04-06T14:27:08.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4000fcb30 con 0x7fd4000fe7b0 2024-04-06T14:27:08.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd3eeffd700 1 -- 172.21.15.32:0/3257905844 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3f401c630 con 0x7fd4000fe7b0 2024-04-06T14:27:08.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.867+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40011a190 con 0x7fd4000fe7b0 2024-04-06T14:27:08.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.868+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4000666f0 con 0x7fd4000fe7b0 2024-04-06T14:27:08.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.869+0000 7fd3eeffd700 1 -- 172.21.15.32:0/3257905844 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd3f4013a20 con 0x7fd4000fe7b0 2024-04-06T14:27:08.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.869+0000 7fd3eeffd700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd3e806cc30 0x7fd3e806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:08.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.869+0000 7fd3eeffd700 1 -- 172.21.15.32:0/3257905844 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd3f408d320 con 0x7fd4000fe7b0 2024-04-06T14:27:08.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.870+0000 7fd3fe59c700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd3e806cc30 0x7fd3e806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:08.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.870+0000 7fd3fe59c700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd3e806cc30 0x7fd3e806f0e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0004d60 tx=0x7fd3f000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:08.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:08.874+0000 7fd3eeffd700 1 -- 172.21.15.32:0/3257905844 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd3f40572e0 con 0x7fd4000fe7b0 2024-04-06T14:27:09.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.097+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd40004bed0 con 0x7fd3e806cc30 2024-04-06T14:27:09.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.098+0000 7fd3eeffd700 1 -- 172.21.15.32:0/3257905844 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd40004bed0 con 0x7fd3e806cc30 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:27:09.099 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.100+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd3e806cc30 msgr2=0x7fd3e806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.100+0000 7fd404bd4700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd3e806cc30 0x7fd3e806f0e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0004d60 tx=0x7fd3f000b040 comp rx=0 tx=0).stop 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.100+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 msgr2=0x7fd4000fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.100+0000 7fd404bd4700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 0x7fd4000fc210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009390 tx=0x7fd3f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.100+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 shutdown_connections 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.100+0000 7fd404bd4700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd3e806cc30 0x7fd3e806f0e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.100+0000 7fd404bd4700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4000fde10 0x7fd4000fbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.100+0000 7fd404bd4700 1 --2- 172.21.15.32:0/3257905844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4000fe7b0 0x7fd4000fc210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.100+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 >> 172.21.15.32:0/3257905844 conn(0x7fd4000f78d0 msgr2=0x7fd4000f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.101+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 shutdown_connections 2024-04-06T14:27:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.101+0000 7fd404bd4700 1 -- 172.21.15.32:0/3257905844 wait complete. 2024-04-06T14:27:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 -- 172.21.15.32:0/2266353735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 msgr2=0x7f5c680fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 --2- 172.21.15.32:0/2266353735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 0x7f5c680fbd80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c004940 tx=0x7f5c5c009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 -- 172.21.15.32:0/2266353735 shutdown_connections 2024-04-06T14:27:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 --2- 172.21.15.32:0/2266353735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 0x7f5c680fbd80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 --2- 172.21.15.32:0/2266353735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c68101d70 0x7f5c68102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 -- 172.21.15.32:0/2266353735 >> 172.21.15.32:0/2266353735 conn(0x7f5c68072bc0 msgr2=0x7f5c68072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 -- 172.21.15.32:0/2266353735 shutdown_connections 2024-04-06T14:27:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 -- 172.21.15.32:0/2266353735 wait complete. 2024-04-06T14:27:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 Processor -- start 2024-04-06T14:27:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.204+0000 7f5c6d03e700 1 -- start start 2024-04-06T14:27:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6d03e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c68101d70 0x7f5c6810ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6d03e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 0x7f5c6810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6d03e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6810f5f0 con 0x7f5c68102710 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6d03e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6810f730 con 0x7f5c68101d70 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 0x7f5c6810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 0x7f5c6810efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44604/0 (socket says 172.21.15.32:44604) 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6659c700 1 -- 172.21.15.32:0/1211432997 learned_addr learned my addr 172.21.15.32:0/1211432997 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c66d9d700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c68101d70 0x7f5c6810ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6659c700 1 -- 172.21.15.32:0/1211432997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c68101d70 msgr2=0x7f5c6810ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6659c700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c68101d70 0x7f5c6810ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6659c700 1 -- 172.21.15.32:0/1211432997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c005040 con 0x7f5c68102710 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6659c700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 0x7f5c6810efa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c00b3b0 tx=0x7f5c5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c4f7fe700 1 -- 172.21.15.32:0/1211432997 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c5c00e040 con 0x7f5c68102710 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c68117ad0 con 0x7f5c68102710 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c4f7fe700 1 -- 172.21.15.32:0/1211432997 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c5c009b10 con 0x7f5c68102710 2024-04-06T14:27:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.205+0000 7f5c4f7fe700 1 -- 172.21.15.32:0/1211432997 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c5c013420 con 0x7f5c68102710 2024-04-06T14:27:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.206+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c68117fe0 con 0x7f5c68102710 2024-04-06T14:27:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.206+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c540052b0 con 0x7f5c68102710 2024-04-06T14:27:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.207+0000 7f5c4f7fe700 1 -- 172.21.15.32:0/1211432997 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5c5c01c430 con 0x7f5c68102710 2024-04-06T14:27:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.207+0000 7f5c4f7fe700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5006cc20 0x7f5c5006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.208+0000 7f5c66d9d700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5006cc20 0x7f5c5006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.208+0000 7f5c4f7fe700 1 -- 172.21.15.32:0/1211432997 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5c5c08cc90 con 0x7f5c68102710 2024-04-06T14:27:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.208+0000 7f5c66d9d700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5006cc20 0x7f5c5006f0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5c58004d60 tx=0x7f5c5800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.212+0000 7f5c4f7fe700 1 -- 172.21.15.32:0/1211432997 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5c5c056d00 con 0x7f5c68102710 2024-04-06T14:27:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:09 smithi086 ceph-mon[110975]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:09 smithi086 ceph-mon[110975]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:09 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3014070705' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:27:09.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:09 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/949221938' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:27:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:09 smithi032 ceph-mon[147414]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:09 smithi032 ceph-mon[147414]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:09 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3014070705' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:27:09.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:09 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/949221938' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:27:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.498+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5c54005560 con 0x7f5c68102710 2024-04-06T14:27:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.498+0000 7f5c4f7fe700 1 -- 172.21.15.32:0/1211432997 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f5c5c05a320 con 0x7f5c68102710 2024-04-06T14:27:09.499 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:27:09.499 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:27:09.499 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:27:09.499 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:27:09.499 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:27:09.499 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:27:09.499 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.500+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5006cc20 msgr2=0x7f5c5006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.500+0000 7f5c6d03e700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5006cc20 0x7f5c5006f0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5c58004d60 tx=0x7f5c5800b040 comp rx=0 tx=0).stop 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.500+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 msgr2=0x7f5c6810efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.500+0000 7f5c6d03e700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 0x7f5c6810efa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c00b3b0 tx=0x7f5c5c009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.501+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 shutdown_connections 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.501+0000 7f5c6d03e700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5006cc20 0x7f5c5006f0d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.501+0000 7f5c6d03e700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c68101d70 0x7f5c6810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.501+0000 7f5c6d03e700 1 --2- 172.21.15.32:0/1211432997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c68102710 0x7f5c6810efa0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.501+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 >> 172.21.15.32:0/1211432997 conn(0x7f5c68072bc0 msgr2=0x7f5c680fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.501+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 shutdown_connections 2024-04-06T14:27:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:09.501+0000 7f5c6d03e700 1 -- 172.21.15.32:0/1211432997 wait complete. 2024-04-06T14:27:10.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:10 smithi086 ceph-mon[110975]: pgmap v210: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:10 smithi086 ceph-mon[110975]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:10.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:10 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1211432997' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:27:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:10 smithi032 ceph-mon[147414]: pgmap v210: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:10 smithi032 ceph-mon[147414]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:10 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1211432997' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:27:12.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:12 smithi086 ceph-mon[110975]: pgmap v211: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:12.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:12 smithi032 ceph-mon[147414]: pgmap v211: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:14.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:14 smithi086 ceph-mon[110975]: pgmap v212: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:14 smithi032 ceph-mon[147414]: pgmap v212: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:16.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:16 smithi086 ceph-mon[110975]: pgmap v213: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:16 smithi032 ceph-mon[147414]: pgmap v213: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:18.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:18 smithi086 ceph-mon[110975]: pgmap v214: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:18.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:18 smithi032 ceph-mon[147414]: pgmap v214: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:20.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:20 smithi086 ceph-mon[110975]: pgmap v215: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:20 smithi032 ceph-mon[147414]: pgmap v215: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:22.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:22 smithi086 ceph-mon[110975]: pgmap v216: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:22.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:22 smithi032 ceph-mon[147414]: pgmap v216: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:23.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:27:23.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:27:24.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:24 smithi086 ceph-mon[110975]: pgmap v217: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:24.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:24 smithi032 ceph-mon[147414]: pgmap v217: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:26.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:26 smithi086 ceph-mon[110975]: pgmap v218: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:26 smithi032 ceph-mon[147414]: pgmap v218: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:28.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:28 smithi086 ceph-mon[110975]: pgmap v219: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:28.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:28 smithi032 ceph-mon[147414]: pgmap v219: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:30.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:30 smithi086 ceph-mon[110975]: pgmap v220: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:30.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:30 smithi032 ceph-mon[147414]: pgmap v220: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:32.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:32 smithi086 ceph-mon[110975]: pgmap v221: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:32.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:32 smithi032 ceph-mon[147414]: pgmap v221: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:34.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:34 smithi086 ceph-mon[110975]: pgmap v222: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:34.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:34 smithi032 ceph-mon[147414]: pgmap v222: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:36.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:36 smithi086 ceph-mon[110975]: pgmap v223: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:36 smithi032 ceph-mon[147414]: pgmap v223: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:38.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:38 smithi086 ceph-mon[110975]: pgmap v224: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:38.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:27:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:38 smithi032 ceph-mon[147414]: pgmap v224: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:27:39.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.616+0000 7f9726917700 1 -- 172.21.15.32:0/2877917105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 msgr2=0x7f97200fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:39.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.616+0000 7f9726917700 1 --2- 172.21.15.32:0/2877917105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 0x7f97200fbd20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9708004940 tx=0x7f9708009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:39.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.616+0000 7f9726917700 1 -- 172.21.15.32:0/2877917105 shutdown_connections 2024-04-06T14:27:39.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.616+0000 7f9726917700 1 --2- 172.21.15.32:0/2877917105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 0x7f97200fbd20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.616+0000 7f9726917700 1 --2- 172.21.15.32:0/2877917105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9720101d10 0x7f97201020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.616+0000 7f9726917700 1 -- 172.21.15.32:0/2877917105 >> 172.21.15.32:0/2877917105 conn(0x7f97200f7880 msgr2=0x7f97200f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:39.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.616+0000 7f9726917700 1 -- 172.21.15.32:0/2877917105 shutdown_connections 2024-04-06T14:27:39.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.616+0000 7f9726917700 1 -- 172.21.15.32:0/2877917105 wait complete. 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f9726917700 1 Processor -- start 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f9726917700 1 -- start start 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f9726917700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9720101d10 0x7f9720102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f9726917700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 0x7f97201030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f9726917700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9720103640 con 0x7f97201026b0 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f9726917700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9720103780 con 0x7f9720101d10 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f9717fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 0x7f97201030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f9717fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 0x7f97201030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54514/0 (socket says 172.21.15.32:54514) 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f9717fff700 1 -- 172.21.15.32:0/2854606411 learned_addr learned my addr 172.21.15.32:0/2854606411 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.617+0000 7f971ffff700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9720101d10 0x7f9720102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.618+0000 7f9717fff700 1 -- 172.21.15.32:0/2854606411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9720101d10 msgr2=0x7f9720102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:39.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.618+0000 7f9717fff700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9720101d10 0x7f9720102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.618+0000 7f9717fff700 1 -- 172.21.15.32:0/2854606411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9708005040 con 0x7f97201026b0 2024-04-06T14:27:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.618+0000 7f9717fff700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 0x7f97201030b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f97080093b0 tx=0x7f9708009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.618+0000 7f971d7fa700 1 -- 172.21.15.32:0/2854606411 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f970800e040 con 0x7f97201026b0 2024-04-06T14:27:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.618+0000 7f971d7fa700 1 -- 172.21.15.32:0/2854606411 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9708013420 con 0x7f97201026b0 2024-04-06T14:27:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.618+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97201039d0 con 0x7f97201026b0 2024-04-06T14:27:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.618+0000 7f971d7fa700 1 -- 172.21.15.32:0/2854606411 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f970801c630 con 0x7f97201026b0 2024-04-06T14:27:39.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.618+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972011a530 con 0x7f97201026b0 2024-04-06T14:27:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.619+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97200fd460 con 0x7f97201026b0 2024-04-06T14:27:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.620+0000 7f971d7fa700 1 -- 172.21.15.32:0/2854606411 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f97080139e0 con 0x7f97201026b0 2024-04-06T14:27:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.620+0000 7f971d7fa700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f970c06cc20 0x7f970c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.620+0000 7f971d7fa700 1 -- 172.21.15.32:0/2854606411 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f970808d290 con 0x7f97201026b0 2024-04-06T14:27:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.620+0000 7f971ffff700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f970c06cc20 0x7f970c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.621+0000 7f971ffff700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f970c06cc20 0x7f970c06f0d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9710004990 tx=0x7f9710009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:39.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.625+0000 7f971d7fa700 1 -- 172.21.15.32:0/2854606411 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f97080572d0 con 0x7f97201026b0 2024-04-06T14:27:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.846+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9720108d20 con 0x7f970c06cc20 2024-04-06T14:27:39.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.847+0000 7f971d7fa700 1 -- 172.21.15.32:0/2854606411 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9720108d20 con 0x7f970c06cc20 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f970c06cc20 msgr2=0x7f970c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f970c06cc20 0x7f970c06f0d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9710004990 tx=0x7f9710009ee0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 msgr2=0x7f97201030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 0x7f97201030b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f97080093b0 tx=0x7f9708009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 shutdown_connections 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f970c06cc20 0x7f970c06f0d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9720101d10 0x7f9720102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 --2- 172.21.15.32:0/2854606411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97201026b0 0x7f97201030b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 >> 172.21.15.32:0/2854606411 conn(0x7f97200f7880 msgr2=0x7f97200f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 shutdown_connections 2024-04-06T14:27:39.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.849+0000 7f9726917700 1 -- 172.21.15.32:0/2854606411 wait complete. 2024-04-06T14:27:39.859 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:27:39.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 -- 172.21.15.32:0/3494935777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 msgr2=0x7f19a81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:39.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 --2- 172.21.15.32:0/3494935777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 0x7f19a81031f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f19a4004940 tx=0x7f19a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 -- 172.21.15.32:0/3494935777 shutdown_connections 2024-04-06T14:27:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 --2- 172.21.15.32:0/3494935777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a81037c0 0x7f19a80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 --2- 172.21.15.32:0/3494935777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 0x7f19a81031f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 -- 172.21.15.32:0/3494935777 >> 172.21.15.32:0/3494935777 conn(0x7f19a80f89d0 msgr2=0x7f19a80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 -- 172.21.15.32:0/3494935777 shutdown_connections 2024-04-06T14:27:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 -- 172.21.15.32:0/3494935777 wait complete. 2024-04-06T14:27:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 Processor -- start 2024-04-06T14:27:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.955+0000 7f19af488700 1 -- start start 2024-04-06T14:27:39.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19af488700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 0x7f19a80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:39.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19af488700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a81037c0 0x7f19a8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:39.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19af488700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8070d90 con 0x7f19a8102e20 2024-04-06T14:27:39.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19af488700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8070ed0 con 0x7f19a81037c0 2024-04-06T14:27:39.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19ad224700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 0x7f19a80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:39.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19ad224700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 0x7f19a80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54534/0 (socket says 172.21.15.32:54534) 2024-04-06T14:27:39.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19ad224700 1 -- 172.21.15.32:0/3359132218 learned_addr learned my addr 172.21.15.32:0/3359132218 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:39.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19ad224700 1 -- 172.21.15.32:0/3359132218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a81037c0 msgr2=0x7f19a8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:39.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19aca23700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a81037c0 0x7f19a8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19ad224700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a81037c0 0x7f19a8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19ad224700 1 -- 172.21.15.32:0/3359132218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a4005040 con 0x7f19a8102e20 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19aca23700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a81037c0 0x7f19a8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19ad224700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 0x7f19a80702c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f19a4009360 tx=0x7f19a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f199dffb700 1 -- 172.21.15.32:0/3359132218 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19a400e040 con 0x7f19a8102e20 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f199dffb700 1 -- 172.21.15.32:0/3359132218 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19a4013420 con 0x7f19a8102e20 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a8071120 con 0x7f19a8102e20 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.956+0000 7f199dffb700 1 -- 172.21.15.32:0/3359132218 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19a401c5f0 con 0x7f19a8102e20 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.957+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a811a210 con 0x7f19a8102e20 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.958+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a80fe570 con 0x7f19a8102e20 2024-04-06T14:27:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.958+0000 7f199dffb700 1 -- 172.21.15.32:0/3359132218 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f19a4013590 con 0x7f19a8102e20 2024-04-06T14:27:39.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.958+0000 7f199dffb700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f199406cc30 0x7f199406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:39.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.959+0000 7f199dffb700 1 -- 172.21.15.32:0/3359132218 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f19a408d2d0 con 0x7f19a8102e20 2024-04-06T14:27:39.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.959+0000 7f19aca23700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f199406cc30 0x7f199406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:39.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.959+0000 7f19aca23700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f199406cc30 0x7f199406f0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1998009fd0 tx=0x7f199800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:39.963+0000 7f199dffb700 1 -- 172.21.15.32:0/3359132218 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f19a4057310 con 0x7f19a8102e20 2024-04-06T14:27:40.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.190+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19a80719b0 con 0x7f199406cc30 2024-04-06T14:27:40.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.191+0000 7f199dffb700 1 -- 172.21.15.32:0/3359132218 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f19a80719b0 con 0x7f199406cc30 2024-04-06T14:27:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f199406cc30 msgr2=0x7f199406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f199406cc30 0x7f199406f0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1998009fd0 tx=0x7f199800b040 comp rx=0 tx=0).stop 2024-04-06T14:27:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 msgr2=0x7f19a80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 0x7f19a80702c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f19a4009360 tx=0x7f19a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:40.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 shutdown_connections 2024-04-06T14:27:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f199406cc30 0x7f199406f0e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a8102e20 0x7f19a80702c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 --2- 172.21.15.32:0/3359132218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a81037c0 0x7f19a8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 >> 172.21.15.32:0/3359132218 conn(0x7f19a80f89d0 msgr2=0x7f19a80f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 shutdown_connections 2024-04-06T14:27:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.193+0000 7f19af488700 1 -- 172.21.15.32:0/3359132218 wait complete. 2024-04-06T14:27:40.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.296+0000 7fe5144b0700 1 -- 172.21.15.32:0/3175169225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c101d70 msgr2=0x7fe50c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.296+0000 7fe5144b0700 1 --2- 172.21.15.32:0/3175169225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c101d70 0x7fe50c102140 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe508004940 tx=0x7fe508009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:40.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.296+0000 7fe5144b0700 1 -- 172.21.15.32:0/3175169225 shutdown_connections 2024-04-06T14:27:40.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.296+0000 7fe5144b0700 1 --2- 172.21.15.32:0/3175169225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c102710 0x7fe50c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.296+0000 7fe5144b0700 1 --2- 172.21.15.32:0/3175169225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c101d70 0x7fe50c102140 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.296+0000 7fe5144b0700 1 -- 172.21.15.32:0/3175169225 >> 172.21.15.32:0/3175169225 conn(0x7fe50c072bc0 msgr2=0x7fe50c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:40.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.297+0000 7fe5144b0700 1 -- 172.21.15.32:0/3175169225 shutdown_connections 2024-04-06T14:27:40.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.297+0000 7fe5144b0700 1 -- 172.21.15.32:0/3175169225 wait complete. 2024-04-06T14:27:40.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.297+0000 7fe5144b0700 1 Processor -- start 2024-04-06T14:27:40.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.297+0000 7fe5144b0700 1 -- start start 2024-04-06T14:27:40.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe5144b0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c101d70 0x7fe50c1129b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:40.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe5144b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c102710 0x7fe50c112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:40.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe5144b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c1135d0 con 0x7fe50c102710 2024-04-06T14:27:40.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe5144b0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c10c930 con 0x7fe50c101d70 2024-04-06T14:27:40.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe511a4b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c102710 0x7fe50c112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe511a4b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c102710 0x7fe50c112ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54558/0 (socket says 172.21.15.32:54558) 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe511a4b700 1 -- 172.21.15.32:0/2175536207 learned_addr learned my addr 172.21.15.32:0/2175536207 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe51224c700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c101d70 0x7fe50c1129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe511a4b700 1 -- 172.21.15.32:0/2175536207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c101d70 msgr2=0x7fe50c1129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe511a4b700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c101d70 0x7fe50c1129b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe511a4b700 1 -- 172.21.15.32:0/2175536207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe508005040 con 0x7fe50c102710 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe511a4b700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c102710 0x7fe50c112ef0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00b790 tx=0x7fe4fc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe502ffd700 1 -- 172.21.15.32:0/2175536207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4fc00f6f0 con 0x7fe50c102710 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe502ffd700 1 -- 172.21.15.32:0/2175536207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4fc00fd00 con 0x7fe50c102710 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50c10cc10 con 0x7fe50c102710 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.298+0000 7fe502ffd700 1 -- 172.21.15.32:0/2175536207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4fc0103a0 con 0x7fe50c102710 2024-04-06T14:27:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.299+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50c10d1d0 con 0x7fe50c102710 2024-04-06T14:27:40.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.299+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe50c0ff080 con 0x7fe50c102710 2024-04-06T14:27:40.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.300+0000 7fe502ffd700 1 -- 172.21.15.32:0/2175536207 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe4fc00f850 con 0x7fe50c102710 2024-04-06T14:27:40.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.300+0000 7fe502ffd700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4f806cc20 0x7fe4f806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:40.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.300+0000 7fe502ffd700 1 -- 172.21.15.32:0/2175536207 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe4fc08be50 con 0x7fe50c102710 2024-04-06T14:27:40.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.300+0000 7fe51224c700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4f806cc20 0x7fe4f806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:40.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.301+0000 7fe51224c700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4f806cc20 0x7fe4f806f0d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe508004f90 tx=0x7fe508009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.305+0000 7fe502ffd700 1 -- 172.21.15.32:0/2175536207 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe4fc059920 con 0x7fe50c102710 2024-04-06T14:27:40.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:40 smithi086 ceph-mon[110975]: pgmap v225: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:40.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:40 smithi032 ceph-mon[147414]: pgmap v225: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:40.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.518+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe50c10e000 con 0x7fe4f806cc20 2024-04-06T14:27:40.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.528+0000 7fe502ffd700 1 -- 172.21.15.32:0/2175536207 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fe50c10e000 con 0x7fe4f806cc20 2024-04-06T14:27:40.527 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:27:40.527 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (8m) 82s ago 19m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:27:40.527 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (5m) 82s ago 19m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:27:40.527 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (5m) 67s ago 18m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 82s ago 19m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (106s) 82s ago 13m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (87s) 82s ago 13m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (71s) 67s ago 13m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (13m) 67s ago 13m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (9m) 82s ago 20m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (8m) 67s ago 18m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (6m) 82s ago 20m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (6m) 67s ago 18m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 82s ago 19m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (8m) 67s ago 18m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (4m) 82s ago 17m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (4m) 82s ago 17m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (3m) 82s ago 16m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:27:40.528 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (3m) 67s ago 16m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:27:40.529 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (2m) 67s ago 15m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:27:40.529 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (2m) 67s ago 15m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:27:40.529 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (8m) 82s ago 18m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:27:40.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4f806cc20 msgr2=0x7fe4f806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4f806cc20 0x7fe4f806f0d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe508004f90 tx=0x7fe508009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:40.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c102710 msgr2=0x7fe50c112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c102710 0x7fe50c112ef0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00b790 tx=0x7fe4fc009220 comp rx=0 tx=0).stop 2024-04-06T14:27:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 shutdown_connections 2024-04-06T14:27:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe4f806cc20 0x7fe4f806f0d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c101d70 0x7fe50c1129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 --2- 172.21.15.32:0/2175536207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c102710 0x7fe50c112ef0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 >> 172.21.15.32:0/2175536207 conn(0x7fe50c072bc0 msgr2=0x7fe50c0faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 shutdown_connections 2024-04-06T14:27:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.530+0000 7fe5144b0700 1 -- 172.21.15.32:0/2175536207 wait complete. 2024-04-06T14:27:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.632+0000 7f0a85a4a700 1 -- 172.21.15.32:0/3458964090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a801026c0 msgr2=0x7f0a800fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.632+0000 7f0a85a4a700 1 --2- 172.21.15.32:0/3458964090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a801026c0 0x7f0a800fbd30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0a68004940 tx=0x7f0a68009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.632+0000 7f0a85a4a700 1 -- 172.21.15.32:0/3458964090 shutdown_connections 2024-04-06T14:27:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.632+0000 7f0a85a4a700 1 --2- 172.21.15.32:0/3458964090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a801026c0 0x7f0a800fbd30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.632+0000 7f0a85a4a700 1 --2- 172.21.15.32:0/3458964090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a80101d20 0x7f0a801020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.632+0000 7f0a85a4a700 1 -- 172.21.15.32:0/3458964090 >> 172.21.15.32:0/3458964090 conn(0x7f0a800f7850 msgr2=0x7f0a800f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.632+0000 7f0a85a4a700 1 -- 172.21.15.32:0/3458964090 shutdown_connections 2024-04-06T14:27:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.632+0000 7f0a85a4a700 1 -- 172.21.15.32:0/3458964090 wait complete. 2024-04-06T14:27:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a85a4a700 1 Processor -- start 2024-04-06T14:27:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a85a4a700 1 -- start start 2024-04-06T14:27:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a85a4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a80101d20 0x7f0a801127b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a85a4a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a801026c0 0x7f0a80112cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a85a4a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a80113340 con 0x7f0a80101d20 2024-04-06T14:27:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a85a4a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8010c940 con 0x7f0a801026c0 2024-04-06T14:27:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a7effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a80101d20 0x7f0a801127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a7effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a80101d20 0x7f0a801127b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54578/0 (socket says 172.21.15.32:54578) 2024-04-06T14:27:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a7effd700 1 -- 172.21.15.32:0/2425468302 learned_addr learned my addr 172.21.15.32:0/2425468302 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.633+0000 7f0a7e7fc700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a801026c0 0x7f0a80112cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.634+0000 7f0a7effd700 1 -- 172.21.15.32:0/2425468302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a801026c0 msgr2=0x7f0a80112cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.634+0000 7f0a7effd700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a801026c0 0x7f0a80112cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.634+0000 7f0a7effd700 1 -- 172.21.15.32:0/2425468302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a68005040 con 0x7f0a80101d20 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.634+0000 7f0a7effd700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a80101d20 0x7f0a801127b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0a7000c8c0 tx=0x7f0a700044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.634+0000 7f0a777fe700 1 -- 172.21.15.32:0/2425468302 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a7000a760 con 0x7f0a80101d20 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.634+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8010cc20 con 0x7f0a80101d20 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.634+0000 7f0a777fe700 1 -- 172.21.15.32:0/2425468302 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a7000ad70 con 0x7f0a80101d20 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.634+0000 7f0a777fe700 1 -- 172.21.15.32:0/2425468302 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a700113a0 con 0x7f0a80101d20 2024-04-06T14:27:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.634+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8010d0b0 con 0x7f0a80101d20 2024-04-06T14:27:40.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.636+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a800fd470 con 0x7f0a80101d20 2024-04-06T14:27:40.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.636+0000 7f0a777fe700 1 -- 172.21.15.32:0/2425468302 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0a70012400 con 0x7f0a80101d20 2024-04-06T14:27:40.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.636+0000 7f0a777fe700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a6c071030 0x7f0a6c0734e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:40.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.636+0000 7f0a7e7fc700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a6c071030 0x7f0a6c0734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:40.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.636+0000 7f0a777fe700 1 -- 172.21.15.32:0/2425468302 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0a7008c090 con 0x7f0a80101d20 2024-04-06T14:27:40.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.637+0000 7f0a7e7fc700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a6c071030 0x7f0a6c0734e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0a8010dc20 tx=0x7f0a68009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:40.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.641+0000 7f0a777fe700 1 -- 172.21.15.32:0/2425468302 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0a70059b60 con 0x7f0a80101d20 2024-04-06T14:27:40.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.929+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0a80108d50 con 0x7f0a80101d20 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.929+0000 7f0a777fe700 1 -- 172.21.15.32:0/2425468302 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f0a700596f0 con 0x7f0a80101d20 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:27:40.929 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:27:40.930 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:27:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a6c071030 msgr2=0x7f0a6c0734e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a6c071030 0x7f0a6c0734e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0a8010dc20 tx=0x7f0a68009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a80101d20 msgr2=0x7f0a801127b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a80101d20 0x7f0a801127b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0a7000c8c0 tx=0x7f0a700044c0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 shutdown_connections 2024-04-06T14:27:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a6c071030 0x7f0a6c0734e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a80101d20 0x7f0a801127b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 --2- 172.21.15.32:0/2425468302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a801026c0 0x7f0a80112cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 >> 172.21.15.32:0/2425468302 conn(0x7f0a800f7850 msgr2=0x7f0a800f8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:40.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 shutdown_connections 2024-04-06T14:27:40.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:40.932+0000 7f0a85a4a700 1 -- 172.21.15.32:0/2425468302 wait complete. 2024-04-06T14:27:41.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.035+0000 7f20915b2700 1 -- 172.21.15.32:0/335288102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c1026b0 msgr2=0x7f208c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.035+0000 7f20915b2700 1 --2- 172.21.15.32:0/335288102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c1026b0 0x7f208c0fbd20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2080004940 tx=0x7f2080009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:41.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.036+0000 7f20915b2700 1 -- 172.21.15.32:0/335288102 shutdown_connections 2024-04-06T14:27:41.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.036+0000 7f20915b2700 1 --2- 172.21.15.32:0/335288102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c1026b0 0x7f208c0fbd20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.036+0000 7f20915b2700 1 --2- 172.21.15.32:0/335288102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f208c101d10 0x7f208c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.036+0000 7f20915b2700 1 -- 172.21.15.32:0/335288102 >> 172.21.15.32:0/335288102 conn(0x7f208c0f7880 msgr2=0x7f208c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:41.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.036+0000 7f20915b2700 1 -- 172.21.15.32:0/335288102 shutdown_connections 2024-04-06T14:27:41.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.036+0000 7f20915b2700 1 -- 172.21.15.32:0/335288102 wait complete. 2024-04-06T14:27:41.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.036+0000 7f20915b2700 1 Processor -- start 2024-04-06T14:27:41.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.036+0000 7f20915b2700 1 -- start start 2024-04-06T14:27:41.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f20915b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c101d10 0x7f208c115ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:41.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f20915b2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f208c1026b0 0x7f208c116010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:41.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f20915b2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c116660 con 0x7f208c101d10 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f20915b2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c10fc40 con 0x7f208c1026b0 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f208affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c101d10 0x7f208c115ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f208affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c101d10 0x7f208c115ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54604/0 (socket says 172.21.15.32:54604) 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f208affd700 1 -- 172.21.15.32:0/1635086876 learned_addr learned my addr 172.21.15.32:0/1635086876 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f208a7fc700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f208c1026b0 0x7f208c116010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f208affd700 1 -- 172.21.15.32:0/1635086876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f208c1026b0 msgr2=0x7f208c116010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f208affd700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f208c1026b0 0x7f208c116010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f208affd700 1 -- 172.21.15.32:0/1635086876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2080005040 con 0x7f208c101d10 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f208affd700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c101d10 0x7f208c115ad0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f207c002a50 tx=0x7f207c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f20737fe700 1 -- 172.21.15.32:0/1635086876 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f207c007cc0 con 0x7f208c101d10 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c10ff20 con 0x7f208c101d10 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f20737fe700 1 -- 172.21.15.32:0/1635086876 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f207c00f040 con 0x7f208c101d10 2024-04-06T14:27:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f20737fe700 1 -- 172.21.15.32:0/1635086876 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f207c009540 con 0x7f208c101d10 2024-04-06T14:27:41.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.037+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c1104e0 con 0x7f208c101d10 2024-04-06T14:27:41.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.038+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c0fd460 con 0x7f208c101d10 2024-04-06T14:27:41.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.039+0000 7f20737fe700 1 -- 172.21.15.32:0/1635086876 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f207c0096a0 con 0x7f208c101d10 2024-04-06T14:27:41.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.039+0000 7f20737fe700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f207406cc20 0x7f207406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:41.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.039+0000 7f20737fe700 1 -- 172.21.15.32:0/1635086876 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f207c08be70 con 0x7f208c101d10 2024-04-06T14:27:41.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.039+0000 7f208a7fc700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f207406cc20 0x7f207406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:41.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.040+0000 7f208a7fc700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f207406cc20 0x7f207406f0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f208c111370 tx=0x7f2080009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.044+0000 7f20737fe700 1 -- 172.21.15.32:0/1635086876 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f207c0598c0 con 0x7f208c101d10 2024-04-06T14:27:41.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.290+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f208c111120 con 0x7f208c101d10 2024-04-06T14:27:41.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.290+0000 7f20737fe700 1 -- 172.21.15.32:0/1635086876 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f207c059450 con 0x7f208c101d10 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:27:41.290 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:27:41.291 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:27:41.292 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:27:41.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f207406cc20 msgr2=0x7f207406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f207406cc20 0x7f207406f0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f208c111370 tx=0x7f2080009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:41.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c101d10 msgr2=0x7f208c115ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c101d10 0x7f208c115ad0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f207c002a50 tx=0x7f207c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 shutdown_connections 2024-04-06T14:27:41.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f207406cc20 0x7f207406f0d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f208c101d10 0x7f208c115ad0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 --2- 172.21.15.32:0/1635086876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f208c1026b0 0x7f208c116010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 >> 172.21.15.32:0/1635086876 conn(0x7f208c0f7880 msgr2=0x7f208c0f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:41.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 shutdown_connections 2024-04-06T14:27:41.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.292+0000 7f20915b2700 1 -- 172.21.15.32:0/1635086876 wait complete. 2024-04-06T14:27:41.294 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:27:41.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:41 smithi086 ceph-mon[110975]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:41.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:41 smithi086 ceph-mon[110975]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:41.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:41 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2425468302' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:27:41.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 -- 172.21.15.32:0/314308588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68741037b0 msgr2=0x7f68740fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 --2- 172.21.15.32:0/314308588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68741037b0 0x7f68740fce20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6864004940 tx=0x7f6864009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:41.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 -- 172.21.15.32:0/314308588 shutdown_connections 2024-04-06T14:27:41.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 --2- 172.21.15.32:0/314308588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68741037b0 0x7f68740fce20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 --2- 172.21.15.32:0/314308588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6874102e10 0x7f68741031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 -- 172.21.15.32:0/314308588 >> 172.21.15.32:0/314308588 conn(0x7f68740f8980 msgr2=0x7f68740fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 -- 172.21.15.32:0/314308588 shutdown_connections 2024-04-06T14:27:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 -- 172.21.15.32:0/314308588 wait complete. 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 Processor -- start 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.397+0000 7f687c415700 1 -- start start 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687c415700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6874102e10 0x7f6874113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687c415700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68741037b0 0x7f6874114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687c415700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6874114670 con 0x7f6874102e10 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687c415700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687410da60 con 0x7f68741037b0 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687a1b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6874102e10 0x7f6874113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687a1b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6874102e10 0x7f6874113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54618/0 (socket says 172.21.15.32:54618) 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687a1b1700 1 -- 172.21.15.32:0/699932450 learned_addr learned my addr 172.21.15.32:0/699932450 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687a1b1700 1 -- 172.21.15.32:0/699932450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68741037b0 msgr2=0x7f6874114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687a1b1700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68741037b0 0x7f6874114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687a1b1700 1 -- 172.21.15.32:0/699932450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6864005040 con 0x7f6874102e10 2024-04-06T14:27:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687a1b1700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6874102e10 0x7f6874113ae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6870002a50 tx=0x7f68700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f686affd700 1 -- 172.21.15.32:0/699932450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6870007cc0 con 0x7f6874102e10 2024-04-06T14:27:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f686affd700 1 -- 172.21.15.32:0/699932450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f687000f040 con 0x7f6874102e10 2024-04-06T14:27:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687410dd40 con 0x7f6874102e10 2024-04-06T14:27:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f686affd700 1 -- 172.21.15.32:0/699932450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6870009540 con 0x7f6874102e10 2024-04-06T14:27:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.398+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687410e300 con 0x7f6874102e10 2024-04-06T14:27:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.399+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68740fe560 con 0x7f6874102e10 2024-04-06T14:27:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.400+0000 7f686affd700 1 -- 172.21.15.32:0/699932450 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f68700096a0 con 0x7f6874102e10 2024-04-06T14:27:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.400+0000 7f686affd700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f686006cc20 0x7f686006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.400+0000 7f686affd700 1 -- 172.21.15.32:0/699932450 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f687008bd70 con 0x7f6874102e10 2024-04-06T14:27:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.400+0000 7f68799b0700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f686006cc20 0x7f686006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.401+0000 7f68799b0700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f686006cc20 0x7f686006f0d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6864004f90 tx=0x7f6864009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:41.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.405+0000 7f686affd700 1 -- 172.21.15.32:0/699932450 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6870059840 con 0x7f6874102e10 2024-04-06T14:27:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:41 smithi032 ceph-mon[147414]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:41 smithi032 ceph-mon[147414]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:41.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:41 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2425468302' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:27:41.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.627+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6874109e20 con 0x7f686006cc20 2024-04-06T14:27:41.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.628+0000 7f686affd700 1 -- 172.21.15.32:0/699932450 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6874109e20 con 0x7f686006cc20 2024-04-06T14:27:41.627 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:27:41.627 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:27:41.627 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:27:41.627 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:27:41.628 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:27:41.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f686006cc20 msgr2=0x7f686006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f686006cc20 0x7f686006f0d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6864004f90 tx=0x7f6864009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6874102e10 msgr2=0x7f6874113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6874102e10 0x7f6874113ae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6870002a50 tx=0x7f68700046b0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 shutdown_connections 2024-04-06T14:27:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f686006cc20 0x7f686006f0d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6874102e10 0x7f6874113ae0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 --2- 172.21.15.32:0/699932450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68741037b0 0x7f6874114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 >> 172.21.15.32:0/699932450 conn(0x7f68740f8980 msgr2=0x7f68740f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 shutdown_connections 2024-04-06T14:27:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.630+0000 7f687c415700 1 -- 172.21.15.32:0/699932450 wait complete. 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.732+0000 7f43035c6700 1 -- 172.21.15.32:0/2167117845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 msgr2=0x7f42fc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.732+0000 7f43035c6700 1 --2- 172.21.15.32:0/2167117845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 0x7f42fc0fbd80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f42f0004940 tx=0x7f42f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.732+0000 7f43035c6700 1 -- 172.21.15.32:0/2167117845 shutdown_connections 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.732+0000 7f43035c6700 1 --2- 172.21.15.32:0/2167117845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 0x7f42fc0fbd80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.732+0000 7f43035c6700 1 --2- 172.21.15.32:0/2167117845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42fc101d70 0x7f42fc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.732+0000 7f43035c6700 1 -- 172.21.15.32:0/2167117845 >> 172.21.15.32:0/2167117845 conn(0x7f42fc072bc0 msgr2=0x7f42fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.732+0000 7f43035c6700 1 -- 172.21.15.32:0/2167117845 shutdown_connections 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.732+0000 7f43035c6700 1 -- 172.21.15.32:0/2167117845 wait complete. 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f43035c6700 1 Processor -- start 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f43035c6700 1 -- start start 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f43035c6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42fc101d70 0x7f42fc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f43035c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 0x7f42fc109a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:41.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f43035c6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc10d670 con 0x7f42fc102710 2024-04-06T14:27:41.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f43035c6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc109f60 con 0x7f42fc101d70 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f4300b61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 0x7f42fc109a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f4300b61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 0x7f42fc109a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54636/0 (socket says 172.21.15.32:54636) 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f4300b61700 1 -- 172.21.15.32:0/3444391949 learned_addr learned my addr 172.21.15.32:0/3444391949 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f4300b61700 1 -- 172.21.15.32:0/3444391949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42fc101d70 msgr2=0x7f42fc1094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f4301362700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42fc101d70 0x7f42fc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f4300b61700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42fc101d70 0x7f42fc1094e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f4300b61700 1 -- 172.21.15.32:0/3444391949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f0005040 con 0x7f42fc102710 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f4301362700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42fc101d70 0x7f42fc1094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.733+0000 7f4300b61700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 0x7f42fc109a20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009380 tx=0x7f42f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.734+0000 7f42edffb700 1 -- 172.21.15.32:0/3444391949 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42f000e040 con 0x7f42fc102710 2024-04-06T14:27:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.734+0000 7f42edffb700 1 -- 172.21.15.32:0/3444391949 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42f0013420 con 0x7f42fc102710 2024-04-06T14:27:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.734+0000 7f42edffb700 1 -- 172.21.15.32:0/3444391949 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42f001c630 con 0x7f42fc102710 2024-04-06T14:27:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.734+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42fc10a1e0 con 0x7f42fc102710 2024-04-06T14:27:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.734+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42fc11e610 con 0x7f42fc102710 2024-04-06T14:27:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.735+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42fc0fd4c0 con 0x7f42fc102710 2024-04-06T14:27:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.735+0000 7f42edffb700 1 -- 172.21.15.32:0/3444391949 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f42f00139e0 con 0x7f42fc102710 2024-04-06T14:27:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.736+0000 7f42edffb700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f42e806cc20 0x7f42e806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:27:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.736+0000 7f42edffb700 1 -- 172.21.15.32:0/3444391949 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f42f008d1b0 con 0x7f42fc102710 2024-04-06T14:27:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.736+0000 7f4301362700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f42e806cc20 0x7f42e806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:27:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.736+0000 7f4301362700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f42e806cc20 0x7f42e806f0d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f42f8009fd0 tx=0x7f42f800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:27:41.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:41.740+0000 7f42edffb700 1 -- 172.21.15.32:0/3444391949 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f42f00571f0 con 0x7f42fc102710 2024-04-06T14:27:42.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.032+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f42fc04bed0 con 0x7f42fc102710 2024-04-06T14:27:42.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.032+0000 7f42edffb700 1 -- 172.21.15.32:0/3444391949 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f42f005a810 con 0x7f42fc102710 2024-04-06T14:27:42.033 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:27:42.033 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:27:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:27:42.033 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:27:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:27:42.033 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:27:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f42e806cc20 msgr2=0x7f42e806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f42e806cc20 0x7f42e806f0d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f42f8009fd0 tx=0x7f42f800b040 comp rx=0 tx=0).stop 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 msgr2=0x7f42fc109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 0x7f42fc109a20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009380 tx=0x7f42f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 shutdown_connections 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f42e806cc20 0x7f42e806f0d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42fc101d70 0x7f42fc1094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 --2- 172.21.15.32:0/3444391949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42fc102710 0x7f42fc109a20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 >> 172.21.15.32:0/3444391949 conn(0x7f42fc072bc0 msgr2=0x7f42fc0659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.034+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 shutdown_connections 2024-04-06T14:27:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:27:42.035+0000 7f43035c6700 1 -- 172.21.15.32:0/3444391949 wait complete. 2024-04-06T14:27:42.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:42 smithi086 ceph-mon[110975]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:42.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:42 smithi086 ceph-mon[110975]: pgmap v226: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:42.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:42 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1635086876' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:27:42.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:42 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3444391949' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:27:42.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:42 smithi032 ceph-mon[147414]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:42.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:42 smithi032 ceph-mon[147414]: pgmap v226: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:42.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:42 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1635086876' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:27:42.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:42 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3444391949' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:27:43.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:43 smithi032 ceph-mon[147414]: from='client.34496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:43.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:43 smithi086 ceph-mon[110975]: from='client.34496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:27:44.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:44 smithi032 ceph-mon[147414]: pgmap v227: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:44.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:44 smithi086 ceph-mon[110975]: pgmap v227: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:46 smithi032 ceph-mon[147414]: pgmap v228: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:46.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:46 smithi086 ceph-mon[110975]: pgmap v228: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:48.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:48 smithi032 ceph-mon[147414]: pgmap v229: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:48.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:48 smithi086 ceph-mon[110975]: pgmap v229: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:50 smithi032 ceph-mon[147414]: pgmap v230: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:50.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:50 smithi086 ceph-mon[110975]: pgmap v230: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:52 smithi032 ceph-mon[147414]: pgmap v231: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:52.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:52 smithi086 ceph-mon[110975]: pgmap v231: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:53.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:27:53.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:27:54.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:54 smithi032 ceph-mon[147414]: pgmap v232: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:54.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:54 smithi086 ceph-mon[110975]: pgmap v232: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:56.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:56 smithi032 ceph-mon[147414]: pgmap v233: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:56.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:56 smithi086 ceph-mon[110975]: pgmap v233: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:27:58 smithi032 ceph-mon[147414]: pgmap v234: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:27:58.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:27:58 smithi086 ceph-mon[110975]: pgmap v234: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:00 smithi032 ceph-mon[147414]: pgmap v235: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:00.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:00 smithi086 ceph-mon[110975]: pgmap v235: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:02 smithi032 ceph-mon[147414]: pgmap v236: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:02.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:02 smithi086 ceph-mon[110975]: pgmap v236: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:04 smithi032 ceph-mon[147414]: pgmap v237: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:04.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:04 smithi086 ceph-mon[110975]: pgmap v237: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:06 smithi032 ceph-mon[147414]: pgmap v238: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:06.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:06 smithi086 ceph-mon[110975]: pgmap v238: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:08 smithi032 ceph-mon[147414]: pgmap v239: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:28:08.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:08 smithi086 ceph-mon[110975]: pgmap v239: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:28:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:10 smithi032 ceph-mon[147414]: pgmap v240: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:10.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:10 smithi086 ceph-mon[110975]: pgmap v240: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:12.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.154+0000 7f60d9847700 1 -- 172.21.15.32:0/1589878865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d40fb4a0 msgr2=0x7f60d40fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.154+0000 7f60d9847700 1 --2- 172.21.15.32:0/1589878865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d40fb4a0 0x7f60d40fb910 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f60c4004940 tx=0x7f60c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.154+0000 7f60d9847700 1 -- 172.21.15.32:0/1589878865 shutdown_connections 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.154+0000 7f60d9847700 1 --2- 172.21.15.32:0/1589878865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d40fb4a0 0x7f60d40fb910 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.154+0000 7f60d9847700 1 --2- 172.21.15.32:0/1589878865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60d4102e70 0x7f60d4103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.154+0000 7f60d9847700 1 -- 172.21.15.32:0/1589878865 >> 172.21.15.32:0/1589878865 conn(0x7f60d40714c0 msgr2=0x7f60d40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.154+0000 7f60d9847700 1 -- 172.21.15.32:0/1589878865 shutdown_connections 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.154+0000 7f60d9847700 1 -- 172.21.15.32:0/1589878865 wait complete. 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.155+0000 7f60d9847700 1 Processor -- start 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.155+0000 7f60d9847700 1 -- start start 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.155+0000 7f60d9847700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60d40fb4a0 0x7f60d40fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.155+0000 7f60d9847700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4102e70 0x7f60d40fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.155+0000 7f60d9847700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d40fd1e0 con 0x7f60d4102e70 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.155+0000 7f60d9847700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d40fd320 con 0x7f60d40fb4a0 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4102e70 0x7f60d40fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4102e70 0x7f60d40fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38882/0 (socket says 172.21.15.32:38882) 2024-04-06T14:28:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d27fc700 1 -- 172.21.15.32:0/1136967426 learned_addr learned my addr 172.21.15.32:0/1136967426 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d2ffd700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60d40fb4a0 0x7f60d40fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d27fc700 1 -- 172.21.15.32:0/1136967426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60d40fb4a0 msgr2=0x7f60d40fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d27fc700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60d40fb4a0 0x7f60d40fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d27fc700 1 -- 172.21.15.32:0/1136967426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c4005040 con 0x7f60d4102e70 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d2ffd700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60d40fb4a0 0x7f60d40fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d27fc700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4102e70 0x7f60d40fcbc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f60c4009390 tx=0x7f60c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60cb7fe700 1 -- 172.21.15.32:0/1136967426 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60c400e040 con 0x7f60d4102e70 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60cb7fe700 1 -- 172.21.15.32:0/1136967426 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60c4013420 con 0x7f60d4102e70 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d4119d90 con 0x7f60d4102e70 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60cb7fe700 1 -- 172.21.15.32:0/1136967426 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60c401c630 con 0x7f60d4102e70 2024-04-06T14:28:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.156+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d411a270 con 0x7f60d4102e70 2024-04-06T14:28:12.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.157+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d4100110 con 0x7f60d4102e70 2024-04-06T14:28:12.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.158+0000 7f60cb7fe700 1 -- 172.21.15.32:0/1136967426 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f60c40139e0 con 0x7f60d4102e70 2024-04-06T14:28:12.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.158+0000 7f60cb7fe700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f60c006cc30 0x7f60c006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:12.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.159+0000 7f60cb7fe700 1 -- 172.21.15.32:0/1136967426 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f60c408d2a0 con 0x7f60d4102e70 2024-04-06T14:28:12.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.159+0000 7f60d2ffd700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f60c006cc30 0x7f60c006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:12.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.159+0000 7f60d2ffd700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f60c006cc30 0x7f60c006f0e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009b10 tx=0x7f60bc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:12.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.163+0000 7f60cb7fe700 1 -- 172.21.15.32:0/1136967426 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f60c40572e0 con 0x7f60d4102e70 2024-04-06T14:28:12.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.384+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60d40fde10 con 0x7f60c006cc30 2024-04-06T14:28:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.385+0000 7f60cb7fe700 1 -- 172.21.15.32:0/1136967426 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f60d40fde10 con 0x7f60c006cc30 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.387+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f60c006cc30 msgr2=0x7f60c006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.387+0000 7f60d9847700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f60c006cc30 0x7f60c006f0e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009b10 tx=0x7f60bc008040 comp rx=0 tx=0).stop 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.387+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4102e70 msgr2=0x7f60d40fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.387+0000 7f60d9847700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4102e70 0x7f60d40fcbc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f60c4009390 tx=0x7f60c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.387+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 shutdown_connections 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.387+0000 7f60d9847700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f60c006cc30 0x7f60c006f0e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.387+0000 7f60d9847700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60d40fb4a0 0x7f60d40fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.387+0000 7f60d9847700 1 --2- 172.21.15.32:0/1136967426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60d4102e70 0x7f60d40fcbc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.387+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 >> 172.21.15.32:0/1136967426 conn(0x7f60d40714c0 msgr2=0x7f60d40fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.388+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 shutdown_connections 2024-04-06T14:28:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.388+0000 7f60d9847700 1 -- 172.21.15.32:0/1136967426 wait complete. 2024-04-06T14:28:12.398 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:28:12.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:12 smithi032 ceph-mon[147414]: pgmap v241: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.492+0000 7faaec33f700 1 -- 172.21.15.32:0/992108679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 msgr2=0x7faae40fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.492+0000 7faaec33f700 1 --2- 172.21.15.32:0/992108679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 0x7faae40fb910 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7faad8004940 tx=0x7faad8009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.492+0000 7faaec33f700 1 -- 172.21.15.32:0/992108679 shutdown_connections 2024-04-06T14:28:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.492+0000 7faaec33f700 1 --2- 172.21.15.32:0/992108679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 0x7faae40fb910 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.492+0000 7faaec33f700 1 --2- 172.21.15.32:0/992108679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faae4102e70 0x7faae4103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.492+0000 7faaec33f700 1 -- 172.21.15.32:0/992108679 >> 172.21.15.32:0/992108679 conn(0x7faae40714c0 msgr2=0x7faae40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.492+0000 7faaec33f700 1 -- 172.21.15.32:0/992108679 shutdown_connections 2024-04-06T14:28:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.492+0000 7faaec33f700 1 -- 172.21.15.32:0/992108679 wait complete. 2024-04-06T14:28:12.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.493+0000 7faaec33f700 1 Processor -- start 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.493+0000 7faaec33f700 1 -- start start 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.493+0000 7faaec33f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 0x7faae4113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.493+0000 7faaec33f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faae4102e70 0x7faae4113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.493+0000 7faaec33f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae4114600 con 0x7faae40fb4a0 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.493+0000 7faaec33f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae410d9f0 con 0x7faae4102e70 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faaea0db700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 0x7faae4113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faaea0db700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 0x7faae4113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38906/0 (socket says 172.21.15.32:38906) 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faaea0db700 1 -- 172.21.15.32:0/1153355815 learned_addr learned my addr 172.21.15.32:0/1153355815 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faae98da700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faae4102e70 0x7faae4113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faaea0db700 1 -- 172.21.15.32:0/1153355815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faae4102e70 msgr2=0x7faae4113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faaea0db700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faae4102e70 0x7faae4113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faaea0db700 1 -- 172.21.15.32:0/1153355815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad8005040 con 0x7faae40fb4a0 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faae98da700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faae4102e70 0x7faae4113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faaea0db700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 0x7faae4113a70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faae0002a50 tx=0x7faae00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faad6ffd700 1 -- 172.21.15.32:0/1153355815 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faae0007cc0 con 0x7faae40fb4a0 2024-04-06T14:28:12.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae410dcd0 con 0x7faae40fb4a0 2024-04-06T14:28:12.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faad6ffd700 1 -- 172.21.15.32:0/1153355815 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faae000f040 con 0x7faae40fb4a0 2024-04-06T14:28:12.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faad6ffd700 1 -- 172.21.15.32:0/1153355815 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faae0009540 con 0x7faae40fb4a0 2024-04-06T14:28:12.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.494+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae410e290 con 0x7faae40fb4a0 2024-04-06T14:28:12.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.495+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae4100110 con 0x7faae40fb4a0 2024-04-06T14:28:12.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.496+0000 7faad6ffd700 1 -- 172.21.15.32:0/1153355815 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faae00096a0 con 0x7faae40fb4a0 2024-04-06T14:28:12.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.496+0000 7faad6ffd700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faad006cc30 0x7faad006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:12.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.496+0000 7faad6ffd700 1 -- 172.21.15.32:0/1153355815 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faae008be70 con 0x7faae40fb4a0 2024-04-06T14:28:12.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.496+0000 7faae98da700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faad006cc30 0x7faad006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:12.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.497+0000 7faae98da700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faad006cc30 0x7faad006f0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faae410ebb0 tx=0x7faad8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.501+0000 7faad6ffd700 1 -- 172.21.15.32:0/1153355815 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faae00598c0 con 0x7faae40fb4a0 2024-04-06T14:28:12.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:12 smithi086 ceph-mon[110975]: pgmap v241: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.723+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faae4109e00 con 0x7faad006cc30 2024-04-06T14:28:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.724+0000 7faad6ffd700 1 -- 172.21.15.32:0/1153355815 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7faae4109e00 con 0x7faad006cc30 2024-04-06T14:28:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faad006cc30 msgr2=0x7faad006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faad006cc30 0x7faad006f0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faae410ebb0 tx=0x7faad8009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 msgr2=0x7faae4113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 0x7faae4113a70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faae0002a50 tx=0x7faae00046b0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 shutdown_connections 2024-04-06T14:28:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faad006cc30 0x7faad006f0e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faae40fb4a0 0x7faae4113a70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 --2- 172.21.15.32:0/1153355815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faae4102e70 0x7faae4113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 >> 172.21.15.32:0/1153355815 conn(0x7faae40714c0 msgr2=0x7faae40fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 shutdown_connections 2024-04-06T14:28:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.726+0000 7faaec33f700 1 -- 172.21.15.32:0/1153355815 wait complete. 2024-04-06T14:28:12.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.829+0000 7fae7975b700 1 -- 172.21.15.32:0/3415944812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 msgr2=0x7fae74102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.829+0000 7fae7975b700 1 --2- 172.21.15.32:0/3415944812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 0x7fae74102140 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fae64004940 tx=0x7fae64009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:12.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.829+0000 7fae7975b700 1 -- 172.21.15.32:0/3415944812 shutdown_connections 2024-04-06T14:28:12.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.829+0000 7fae7975b700 1 --2- 172.21.15.32:0/3415944812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae74102710 0x7fae740fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.829+0000 7fae7975b700 1 --2- 172.21.15.32:0/3415944812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 0x7fae74102140 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.829+0000 7fae7975b700 1 -- 172.21.15.32:0/3415944812 >> 172.21.15.32:0/3415944812 conn(0x7fae74072bc0 msgr2=0x7fae74072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.829+0000 7fae7975b700 1 -- 172.21.15.32:0/3415944812 shutdown_connections 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.829+0000 7fae7975b700 1 -- 172.21.15.32:0/3415944812 wait complete. 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae7975b700 1 Processor -- start 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae7975b700 1 -- start start 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae7975b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 0x7fae7410ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae7975b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae74102710 0x7fae7410f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae7975b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7410f660 con 0x7fae74101d70 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae7975b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7410f7a0 con 0x7fae74102710 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae72ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 0x7fae7410ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae72ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 0x7fae7410ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38916/0 (socket says 172.21.15.32:38916) 2024-04-06T14:28:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae72ffd700 1 -- 172.21.15.32:0/2634682473 learned_addr learned my addr 172.21.15.32:0/2634682473 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae727fc700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae74102710 0x7fae7410f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae72ffd700 1 -- 172.21.15.32:0/2634682473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae74102710 msgr2=0x7fae7410f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae72ffd700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae74102710 0x7fae7410f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae72ffd700 1 -- 172.21.15.32:0/2634682473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae64005040 con 0x7fae74101d70 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.830+0000 7fae72ffd700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 0x7fae7410ebe0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fae6400b3b0 tx=0x7fae64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.831+0000 7fae5b7fe700 1 -- 172.21.15.32:0/2634682473 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae6400e040 con 0x7fae74101d70 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.831+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae74117a90 con 0x7fae74101d70 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.831+0000 7fae5b7fe700 1 -- 172.21.15.32:0/2634682473 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae64013420 con 0x7fae74101d70 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.831+0000 7fae5b7fe700 1 -- 172.21.15.32:0/2634682473 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae6401c630 con 0x7fae74101d70 2024-04-06T14:28:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.831+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae74118020 con 0x7fae74101d70 2024-04-06T14:28:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.832+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae740ff080 con 0x7fae74101d70 2024-04-06T14:28:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.832+0000 7fae5b7fe700 1 -- 172.21.15.32:0/2634682473 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fae64013970 con 0x7fae74101d70 2024-04-06T14:28:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.832+0000 7fae5b7fe700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae5c06cc20 0x7fae5c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.833+0000 7fae5b7fe700 1 -- 172.21.15.32:0/2634682473 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fae6408d2b0 con 0x7fae74101d70 2024-04-06T14:28:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.833+0000 7fae727fc700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae5c06cc20 0x7fae5c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:12.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.833+0000 7fae727fc700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae5c06cc20 0x7fae5c06f0d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fae74066020 tx=0x7fae68009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:12.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:12.837+0000 7fae5b7fe700 1 -- 172.21.15.32:0/2634682473 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fae640572f0 con 0x7fae74101d70 2024-04-06T14:28:13.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.047+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fae7404bed0 con 0x7fae5c06cc20 2024-04-06T14:28:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.057+0000 7fae5b7fe700 1 -- 172.21.15.32:0/2634682473 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fae7404bed0 con 0x7fae5c06cc20 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (8m) 114s ago 19m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (5m) 114s ago 20m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (5m) 100s ago 19m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (8m) 114s ago 19m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (2m) 114s ago 14m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (119s) 114s ago 14m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (103s) 100s ago 14m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (13m) 100s ago 13m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (10m) 114s ago 21m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:28:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (9m) 100s ago 18m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (6m) 114s ago 21m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (6m) 100s ago 18m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (9m) 114s ago 19m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (9m) 100s ago 19m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (5m) 114s ago 18m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (4m) 114s ago 17m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (4m) 114s ago 17m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (3m) 100s ago 16m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (3m) 100s ago 16m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (2m) 100s ago 15m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:28:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (8m) 114s ago 19m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:28:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae5c06cc20 msgr2=0x7fae5c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae5c06cc20 0x7fae5c06f0d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fae74066020 tx=0x7fae68009ee0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 msgr2=0x7fae7410ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 0x7fae7410ebe0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fae6400b3b0 tx=0x7fae64009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 shutdown_connections 2024-04-06T14:28:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae5c06cc20 0x7fae5c06f0d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae74101d70 0x7fae7410ebe0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 --2- 172.21.15.32:0/2634682473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae74102710 0x7fae7410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 >> 172.21.15.32:0/2634682473 conn(0x7fae74072bc0 msgr2=0x7fae740fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:13.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 shutdown_connections 2024-04-06T14:28:13.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.060+0000 7fae7975b700 1 -- 172.21.15.32:0/2634682473 wait complete. 2024-04-06T14:28:13.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.162+0000 7fe90bf0b700 1 -- 172.21.15.32:0/3272534073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 msgr2=0x7fe9040fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.162+0000 7fe90bf0b700 1 --2- 172.21.15.32:0/3272534073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 0x7fe9040fbd80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe900004940 tx=0x7fe900009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:13.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.162+0000 7fe90bf0b700 1 -- 172.21.15.32:0/3272534073 shutdown_connections 2024-04-06T14:28:13.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.162+0000 7fe90bf0b700 1 --2- 172.21.15.32:0/3272534073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 0x7fe9040fbd80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.162+0000 7fe90bf0b700 1 --2- 172.21.15.32:0/3272534073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe904101d70 0x7fe904102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.162+0000 7fe90bf0b700 1 -- 172.21.15.32:0/3272534073 >> 172.21.15.32:0/3272534073 conn(0x7fe904072bc0 msgr2=0x7fe904072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:13.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.162+0000 7fe90bf0b700 1 -- 172.21.15.32:0/3272534073 shutdown_connections 2024-04-06T14:28:13.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.162+0000 7fe90bf0b700 1 -- 172.21.15.32:0/3272534073 wait complete. 2024-04-06T14:28:13.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe90bf0b700 1 Processor -- start 2024-04-06T14:28:13.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe90bf0b700 1 -- start start 2024-04-06T14:28:13.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe90bf0b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe904101d70 0x7fe904113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:13.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe90bf0b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 0x7fe904113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:13.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe90bf0b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe904114590 con 0x7fe904102710 2024-04-06T14:28:13.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe90bf0b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90410d980 con 0x7fe904101d70 2024-04-06T14:28:13.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe9094a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 0x7fe904113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:13.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe9094a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 0x7fe904113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38930/0 (socket says 172.21.15.32:38930) 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe9094a6700 1 -- 172.21.15.32:0/2816920550 learned_addr learned my addr 172.21.15.32:0/2816920550 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.163+0000 7fe909ca7700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe904101d70 0x7fe904113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.164+0000 7fe9094a6700 1 -- 172.21.15.32:0/2816920550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe904101d70 msgr2=0x7fe904113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.164+0000 7fe9094a6700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe904101d70 0x7fe904113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.164+0000 7fe9094a6700 1 -- 172.21.15.32:0/2816920550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe900005040 con 0x7fe904102710 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.164+0000 7fe9094a6700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 0x7fe904113f40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe900000c00 tx=0x7fe900009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.164+0000 7fe8fa7fc700 1 -- 172.21.15.32:0/2816920550 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe90000e070 con 0x7fe904102710 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.164+0000 7fe8fa7fc700 1 -- 172.21.15.32:0/2816920550 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9000093b0 con 0x7fe904102710 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.164+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90410dc00 con 0x7fe904102710 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.164+0000 7fe8fa7fc700 1 -- 172.21.15.32:0/2816920550 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe90001ccb0 con 0x7fe904102710 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.164+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90410e160 con 0x7fe904102710 2024-04-06T14:28:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.165+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9040fd4c0 con 0x7fe904102710 2024-04-06T14:28:13.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.165+0000 7fe8fa7fc700 1 -- 172.21.15.32:0/2816920550 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe90001c490 con 0x7fe904102710 2024-04-06T14:28:13.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.166+0000 7fe8fa7fc700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe8f006cc20 0x7fe8f006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:13.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.166+0000 7fe8fa7fc700 1 -- 172.21.15.32:0/2816920550 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe90008d4a0 con 0x7fe904102710 2024-04-06T14:28:13.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.166+0000 7fe909ca7700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe8f006cc20 0x7fe8f006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:13.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.166+0000 7fe909ca7700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe8f006cc20 0x7fe8f006f0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009b80 tx=0x7fe8f400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:13.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.170+0000 7fe8fa7fc700 1 -- 172.21.15.32:0/2816920550 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe90005af70 con 0x7fe904102710 2024-04-06T14:28:13.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.462+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe904108d10 con 0x7fe904102710 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.462+0000 7fe8fa7fc700 1 -- 172.21.15.32:0/2816920550 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fe90005ab00 con 0x7fe904102710 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:28:13.463 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:28:13.464 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:28:13.464 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:28:13.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:28:13.464 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:28:13.464 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:28:13.464 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:28:13.464 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:28:13.464 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe8f006cc20 msgr2=0x7fe8f006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe8f006cc20 0x7fe8f006f0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009b80 tx=0x7fe8f400b040 comp rx=0 tx=0).stop 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 msgr2=0x7fe904113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 0x7fe904113f40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe900000c00 tx=0x7fe900009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 shutdown_connections 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe8f006cc20 0x7fe8f006f0d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe904101d70 0x7fe904113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 --2- 172.21.15.32:0/2816920550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe904102710 0x7fe904113f40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 >> 172.21.15.32:0/2816920550 conn(0x7fe904072bc0 msgr2=0x7fe904065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 shutdown_connections 2024-04-06T14:28:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.465+0000 7fe90bf0b700 1 -- 172.21.15.32:0/2816920550 wait complete. 2024-04-06T14:28:13.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:13 smithi032 ceph-mon[147414]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.569+0000 7f7c36557700 1 -- 172.21.15.32:0/3199889639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 msgr2=0x7f7c30103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.569+0000 7f7c36557700 1 --2- 172.21.15.32:0/3199889639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 0x7f7c30103240 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7c20004940 tx=0x7f7c20009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.569+0000 7f7c36557700 1 -- 172.21.15.32:0/3199889639 shutdown_connections 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.569+0000 7f7c36557700 1 --2- 172.21.15.32:0/3199889639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c300fb4a0 0x7f7c300fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.569+0000 7f7c36557700 1 --2- 172.21.15.32:0/3199889639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 0x7f7c30103240 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.569+0000 7f7c36557700 1 -- 172.21.15.32:0/3199889639 >> 172.21.15.32:0/3199889639 conn(0x7f7c300714c0 msgr2=0x7f7c300718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.569+0000 7f7c36557700 1 -- 172.21.15.32:0/3199889639 shutdown_connections 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.569+0000 7f7c36557700 1 -- 172.21.15.32:0/3199889639 wait complete. 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.570+0000 7f7c36557700 1 Processor -- start 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.570+0000 7f7c36557700 1 -- start start 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.570+0000 7f7c36557700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c300fb4a0 0x7f7c30113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.570+0000 7f7c36557700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 0x7f7c30114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.570+0000 7f7c36557700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c301146e0 con 0x7f7c30102e70 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.570+0000 7f7c36557700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3010dad0 con 0x7f7c300fb4a0 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.570+0000 7f7c2f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 0x7f7c30114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.570+0000 7f7c2f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 0x7f7c30114090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38938/0 (socket says 172.21.15.32:38938) 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.570+0000 7f7c2f7fe700 1 -- 172.21.15.32:0/1283936868 learned_addr learned my addr 172.21.15.32:0/1283936868 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c2ffff700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c300fb4a0 0x7f7c30113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c2f7fe700 1 -- 172.21.15.32:0/1283936868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c300fb4a0 msgr2=0x7f7c30113b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c2f7fe700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c300fb4a0 0x7f7c30113b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c2f7fe700 1 -- 172.21.15.32:0/1283936868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c20005040 con 0x7f7c30102e70 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c2f7fe700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 0x7f7c30114090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c8c0 tx=0x7f7c240044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c2cff9700 1 -- 172.21.15.32:0/1283936868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2400a760 con 0x7f7c30102e70 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3010ddb0 con 0x7f7c30102e70 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c2cff9700 1 -- 172.21.15.32:0/1283936868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c2400ad70 con 0x7f7c30102e70 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c2cff9700 1 -- 172.21.15.32:0/1283936868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c240113a0 con 0x7f7c30102e70 2024-04-06T14:28:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.571+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3010e2f0 con 0x7f7c30102e70 2024-04-06T14:28:13.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.572+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c300653a0 con 0x7f7c30102e70 2024-04-06T14:28:13.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.572+0000 7f7c2cff9700 1 -- 172.21.15.32:0/1283936868 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7c2400a8c0 con 0x7f7c30102e70 2024-04-06T14:28:13.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.573+0000 7f7c2cff9700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c1806cc20 0x7f7c1806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:13.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.573+0000 7f7c2cff9700 1 -- 172.21.15.32:0/1283936868 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7c2408c090 con 0x7f7c30102e70 2024-04-06T14:28:13.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.573+0000 7f7c2ffff700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c1806cc20 0x7f7c1806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:13.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.573+0000 7f7c2ffff700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c1806cc20 0x7f7c1806f0d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7c20004f90 tx=0x7f7c20009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:13.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.577+0000 7f7c2cff9700 1 -- 172.21.15.32:0/1283936868 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7c24059b60 con 0x7f7c30102e70 2024-04-06T14:28:13.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:13 smithi086 ceph-mon[110975]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.824+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7c30105300 con 0x7f7c30102e70 2024-04-06T14:28:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.825+0000 7f7c2cff9700 1 -- 172.21.15.32:0/1283936868 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f7c240596f0 con 0x7f7c30102e70 2024-04-06T14:28:13.825 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:28:13.825 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:28:13.825 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:28:13.825 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:28:13.825 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:28:13.826 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:28:13.827 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:28:13.828 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:28:13.828 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:28:13.828 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:28:13.828 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:28:13.828 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:28:13.828 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:28:13.828 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:28:13.828 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c1806cc20 msgr2=0x7f7c1806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c1806cc20 0x7f7c1806f0d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7c20004f90 tx=0x7f7c20009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 msgr2=0x7f7c30114090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 0x7f7c30114090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c8c0 tx=0x7f7c240044c0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 shutdown_connections 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c1806cc20 0x7f7c1806f0d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c300fb4a0 0x7f7c30113b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 --2- 172.21.15.32:0/1283936868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c30102e70 0x7f7c30114090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 >> 172.21.15.32:0/1283936868 conn(0x7f7c300714c0 msgr2=0x7f7c300fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 shutdown_connections 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.827+0000 7f7c36557700 1 -- 172.21.15.32:0/1283936868 wait complete. 2024-04-06T14:28:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.932+0000 7f5c73bcb700 1 -- 172.21.15.32:0/891752061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 msgr2=0x7f5c6c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.932+0000 7f5c73bcb700 1 --2- 172.21.15.32:0/891752061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 0x7f5c6c102140 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5c68004940 tx=0x7f5c68009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.932+0000 7f5c73bcb700 1 -- 172.21.15.32:0/891752061 shutdown_connections 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.932+0000 7f5c73bcb700 1 --2- 172.21.15.32:0/891752061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6c102710 0x7f5c6c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.932+0000 7f5c73bcb700 1 --2- 172.21.15.32:0/891752061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 0x7f5c6c102140 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.932+0000 7f5c73bcb700 1 -- 172.21.15.32:0/891752061 >> 172.21.15.32:0/891752061 conn(0x7f5c6c072bc0 msgr2=0x7f5c6c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.932+0000 7f5c73bcb700 1 -- 172.21.15.32:0/891752061 shutdown_connections 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.932+0000 7f5c73bcb700 1 -- 172.21.15.32:0/891752061 wait complete. 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c73bcb700 1 Processor -- start 2024-04-06T14:28:13.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c73bcb700 1 -- start start 2024-04-06T14:28:13.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c73bcb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 0x7f5c6c113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:13.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c73bcb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6c102710 0x7f5c6c113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:13.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c73bcb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c114620 con 0x7f5c6c101d70 2024-04-06T14:28:13.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c73bcb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c10da60 con 0x7f5c6c102710 2024-04-06T14:28:13.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c71967700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 0x7f5c6c113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:13.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c71967700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 0x7f5c6c113a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38956/0 (socket says 172.21.15.32:38956) 2024-04-06T14:28:13.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c71967700 1 -- 172.21.15.32:0/3418715185 learned_addr learned my addr 172.21.15.32:0/3418715185 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:13.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c71166700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6c102710 0x7f5c6c113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c71967700 1 -- 172.21.15.32:0/3418715185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6c102710 msgr2=0x7f5c6c113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c71967700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6c102710 0x7f5c6c113fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.933+0000 7f5c71967700 1 -- 172.21.15.32:0/3418715185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c68005040 con 0x7f5c6c101d70 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.934+0000 7f5c71967700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 0x7f5c6c113a90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5c68009360 tx=0x7f5c68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.934+0000 7f5c5e7fc700 1 -- 172.21.15.32:0/3418715185 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c6800e040 con 0x7f5c6c101d70 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.934+0000 7f5c5e7fc700 1 -- 172.21.15.32:0/3418715185 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c68013420 con 0x7f5c6c101d70 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.934+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6c10dce0 con 0x7f5c6c101d70 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.934+0000 7f5c5e7fc700 1 -- 172.21.15.32:0/3418715185 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c6801c5f0 con 0x7f5c6c101d70 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.934+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6c10e2a0 con 0x7f5c6c101d70 2024-04-06T14:28:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.935+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6c0fd4c0 con 0x7f5c6c101d70 2024-04-06T14:28:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.935+0000 7f5c5e7fc700 1 -- 172.21.15.32:0/3418715185 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5c68013590 con 0x7f5c6c101d70 2024-04-06T14:28:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.936+0000 7f5c5e7fc700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5806cc30 0x7f5c5806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.936+0000 7f5c5e7fc700 1 -- 172.21.15.32:0/3418715185 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5c6808d2d0 con 0x7f5c6c101d70 2024-04-06T14:28:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.936+0000 7f5c71166700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5806cc30 0x7f5c5806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.936+0000 7f5c71166700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5806cc30 0x7f5c5806f0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c10ebd0 tx=0x7f5c60009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:13.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:13.940+0000 7f5c5e7fc700 1 -- 172.21.15.32:0/3418715185 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5c6805ada0 con 0x7f5c6c101d70 2024-04-06T14:28:14.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.162+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c6c108da0 con 0x7f5c5806cc30 2024-04-06T14:28:14.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.163+0000 7f5c5e7fc700 1 -- 172.21.15.32:0/3418715185 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5c6c108da0 con 0x7f5c5806cc30 2024-04-06T14:28:14.163 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:28:14.163 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:28:14.163 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:28:14.163 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:28:14.163 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:28:14.163 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:28:14.163 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:28:14.163 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:28:14.164 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:28:14.164 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:28:14.164 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:28:14.164 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:28:14.164 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:28:14.164 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:28:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5806cc30 msgr2=0x7f5c5806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5806cc30 0x7f5c5806f0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c10ebd0 tx=0x7f5c60009ee0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 msgr2=0x7f5c6c113a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 0x7f5c6c113a90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5c68009360 tx=0x7f5c68009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 shutdown_connections 2024-04-06T14:28:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5c5806cc30 0x7f5c5806f0e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c6c101d70 0x7f5c6c113a90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 --2- 172.21.15.32:0/3418715185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6c102710 0x7f5c6c113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 >> 172.21.15.32:0/3418715185 conn(0x7f5c6c072bc0 msgr2=0x7f5c6c065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:14.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 shutdown_connections 2024-04-06T14:28:14.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.165+0000 7f5c73bcb700 1 -- 172.21.15.32:0/3418715185 wait complete. 2024-04-06T14:28:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.269+0000 7fccc52c3700 1 -- 172.21.15.32:0/2520039106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc0102f70 msgr2=0x7fccc0103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.269+0000 7fccc52c3700 1 --2- 172.21.15.32:0/2520039106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc0102f70 0x7fccc0103340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fccb0004940 tx=0x7fccb0009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.269+0000 7fccc52c3700 1 -- 172.21.15.32:0/2520039106 shutdown_connections 2024-04-06T14:28:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.269+0000 7fccc52c3700 1 --2- 172.21.15.32:0/2520039106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc00fb3b0 0x7fccc00fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.269+0000 7fccc52c3700 1 --2- 172.21.15.32:0/2520039106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc0102f70 0x7fccc0103340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.269+0000 7fccc52c3700 1 -- 172.21.15.32:0/2520039106 >> 172.21.15.32:0/2520039106 conn(0x7fccc0072310 msgr2=0x7fccc0072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.269+0000 7fccc52c3700 1 -- 172.21.15.32:0/2520039106 shutdown_connections 2024-04-06T14:28:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.269+0000 7fccc52c3700 1 -- 172.21.15.32:0/2520039106 wait complete. 2024-04-06T14:28:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.269+0000 7fccc52c3700 1 Processor -- start 2024-04-06T14:28:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccc52c3700 1 -- start start 2024-04-06T14:28:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccc52c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc00fb3b0 0x7fccc00fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccc52c3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0102f70 0x7fccc00fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccc52c3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00fe4e0 con 0x7fccc00fb3b0 2024-04-06T14:28:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccc52c3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00fd010 con 0x7fccc0102f70 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccbeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc00fb3b0 0x7fccc00fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccbeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc00fb3b0 0x7fccc00fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38980/0 (socket says 172.21.15.32:38980) 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccbeffd700 1 -- 172.21.15.32:0/2838017654 learned_addr learned my addr 172.21.15.32:0/2838017654 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccbe7fc700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0102f70 0x7fccc00fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccbeffd700 1 -- 172.21.15.32:0/2838017654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0102f70 msgr2=0x7fccc00fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccbeffd700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0102f70 0x7fccc00fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccbeffd700 1 -- 172.21.15.32:0/2838017654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0005040 con 0x7fccc00fb3b0 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccbeffd700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc00fb3b0 0x7fccc00fc590 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009150 tx=0x7fccb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fcca77fe700 1 -- 172.21.15.32:0/2838017654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccb000e040 con 0x7fccc00fb3b0 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fcca77fe700 1 -- 172.21.15.32:0/2838017654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccb0013420 con 0x7fccc00fb3b0 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.270+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc00fd290 con 0x7fccc00fb3b0 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.271+0000 7fcca77fe700 1 -- 172.21.15.32:0/2838017654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccb001c630 con 0x7fccc00fb3b0 2024-04-06T14:28:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.271+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc0119f90 con 0x7fccc00fb3b0 2024-04-06T14:28:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.272+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc01053b0 con 0x7fccc00fb3b0 2024-04-06T14:28:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.272+0000 7fcca77fe700 1 -- 172.21.15.32:0/2838017654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fccb0013990 con 0x7fccc00fb3b0 2024-04-06T14:28:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.272+0000 7fcca77fe700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcca806cc20 0x7fcca806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.273+0000 7fcca77fe700 1 -- 172.21.15.32:0/2838017654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fccb008d270 con 0x7fccc00fb3b0 2024-04-06T14:28:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.273+0000 7fccbe7fc700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcca806cc20 0x7fcca806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.273+0000 7fccbe7fc700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcca806cc20 0x7fcca806f0d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009fd0 tx=0x7fccb4009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.277+0000 7fcca77fe700 1 -- 172.21.15.32:0/2838017654 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fccb00572b0 con 0x7fccc00fb3b0 2024-04-06T14:28:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:14 smithi032 ceph-mon[147414]: pgmap v242: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:14 smithi032 ceph-mon[147414]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:14 smithi032 ceph-mon[147414]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:14 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2816920550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:28:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:14 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1283936868' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:28:14.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.564+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fccc00fe6d0 con 0x7fccc00fb3b0 2024-04-06T14:28:14.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.564+0000 7fcca77fe700 1 -- 172.21.15.32:0/2838017654 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fccb005a8d0 con 0x7fccc00fb3b0 2024-04-06T14:28:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:28:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:28:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:28:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:28:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:28:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:28:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.566+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcca806cc20 msgr2=0x7fcca806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.566+0000 7fccc52c3700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcca806cc20 0x7fcca806f0d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009fd0 tx=0x7fccb4009f20 comp rx=0 tx=0).stop 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.566+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc00fb3b0 msgr2=0x7fccc00fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.566+0000 7fccc52c3700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc00fb3b0 0x7fccc00fc590 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009150 tx=0x7fccb0009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.566+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 shutdown_connections 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.566+0000 7fccc52c3700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcca806cc20 0x7fcca806f0d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.567+0000 7fccc52c3700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccc00fb3b0 0x7fccc00fc590 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.567+0000 7fccc52c3700 1 --2- 172.21.15.32:0/2838017654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0102f70 0x7fccc00fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.567+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 >> 172.21.15.32:0/2838017654 conn(0x7fccc0072310 msgr2=0x7fccc00f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.567+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 shutdown_connections 2024-04-06T14:28:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:14.567+0000 7fccc52c3700 1 -- 172.21.15.32:0/2838017654 wait complete. 2024-04-06T14:28:14.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:14 smithi086 ceph-mon[110975]: pgmap v242: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:14.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:14 smithi086 ceph-mon[110975]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:14.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:14 smithi086 ceph-mon[110975]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:14.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:14 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2816920550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:28:14.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:14 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1283936868' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:28:15.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:15 smithi032 ceph-mon[147414]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:15.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:15 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2838017654' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:28:15.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:15 smithi086 ceph-mon[110975]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:15.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:15 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2838017654' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:28:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:16 smithi032 ceph-mon[147414]: pgmap v243: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:16.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:16 smithi086 ceph-mon[110975]: pgmap v243: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:18.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:18 smithi032 ceph-mon[147414]: pgmap v244: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:18.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:18 smithi086 ceph-mon[110975]: pgmap v244: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:20 smithi032 ceph-mon[147414]: pgmap v245: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:20.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:20 smithi086 ceph-mon[110975]: pgmap v245: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:22.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:22 smithi032 ceph-mon[147414]: pgmap v246: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:22.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:22 smithi086 ceph-mon[110975]: pgmap v246: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:23.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:28:23.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:28:24.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:24 smithi032 ceph-mon[147414]: pgmap v247: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:24.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:24 smithi086 ceph-mon[110975]: pgmap v247: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:26 smithi032 ceph-mon[147414]: pgmap v248: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:26.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:26 smithi086 ceph-mon[110975]: pgmap v248: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:28.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:28 smithi032 ceph-mon[147414]: pgmap v249: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:28.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:28 smithi086 ceph-mon[110975]: pgmap v249: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:30.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:30 smithi032 ceph-mon[147414]: pgmap v250: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:30.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:30 smithi086 ceph-mon[110975]: pgmap v250: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:32.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:32 smithi032 ceph-mon[147414]: pgmap v251: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:32.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:32 smithi086 ceph-mon[110975]: pgmap v251: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:34.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:34 smithi032 ceph-mon[147414]: pgmap v252: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:34.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:34 smithi086 ceph-mon[110975]: pgmap v252: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:36 smithi032 ceph-mon[147414]: pgmap v253: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:36.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:36 smithi086 ceph-mon[110975]: pgmap v253: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:38 smithi032 ceph-mon[147414]: pgmap v254: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:28:38.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:38 smithi086 ceph-mon[110975]: pgmap v254: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:38.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:28:40.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:40 smithi032 ceph-mon[147414]: pgmap v255: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:40.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:40 smithi086 ceph-mon[110975]: pgmap v255: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:42.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:42 smithi086 ceph-mon[110975]: pgmap v256: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:42 smithi032 ceph-mon[147414]: pgmap v256: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:44.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:44 smithi086 ceph-mon[110975]: pgmap v257: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:44.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.685+0000 7f488f691700 1 -- 172.21.15.32:0/2021654716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fc590 msgr2=0x7f48881092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:44.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.685+0000 7f488f691700 1 --2- 172.21.15.32:0/2021654716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fc590 0x7f48881092f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4884004940 tx=0x7f4884009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:44.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.685+0000 7f488f691700 1 -- 172.21.15.32:0/2021654716 shutdown_connections 2024-04-06T14:28:44.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.685+0000 7f488f691700 1 --2- 172.21.15.32:0/2021654716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fc590 0x7f48881092f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:44.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.685+0000 7f488f691700 1 --2- 172.21.15.32:0/2021654716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48880fbc80 0x7f48880fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:44.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.685+0000 7f488f691700 1 -- 172.21.15.32:0/2021654716 >> 172.21.15.32:0/2021654716 conn(0x7f4888072bc0 msgr2=0x7f4888072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:44.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.685+0000 7f488f691700 1 -- 172.21.15.32:0/2021654716 shutdown_connections 2024-04-06T14:28:44.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.685+0000 7f488f691700 1 -- 172.21.15.32:0/2021654716 wait complete. 2024-04-06T14:28:44.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488f691700 1 Processor -- start 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488f691700 1 -- start start 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488f691700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fbc80 0x7f4888115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488f691700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48880fc590 0x7f48881161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488f691700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888116890 con 0x7f48880fbc80 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488f691700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488810fbd0 con 0x7f48880fc590 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488d42d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fbc80 0x7f4888115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488d42d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fbc80 0x7f4888115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58612/0 (socket says 172.21.15.32:58612) 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488d42d700 1 -- 172.21.15.32:0/3208845611 learned_addr learned my addr 172.21.15.32:0/3208845611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488cc2c700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48880fc590 0x7f48881161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488d42d700 1 -- 172.21.15.32:0/3208845611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48880fc590 msgr2=0x7f48881161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488d42d700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48880fc590 0x7f48881161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.686+0000 7f488d42d700 1 -- 172.21.15.32:0/3208845611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4884005040 con 0x7f48880fbc80 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.687+0000 7f488d42d700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fbc80 0x7f4888115c70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f487800b790 tx=0x7f4878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.687+0000 7f487dffb700 1 -- 172.21.15.32:0/3208845611 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487800f770 con 0x7f48880fbc80 2024-04-06T14:28:44.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.687+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488810fe50 con 0x7f48880fbc80 2024-04-06T14:28:44.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.687+0000 7f487dffb700 1 -- 172.21.15.32:0/3208845611 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f487800fd80 con 0x7f48880fbc80 2024-04-06T14:28:44.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.687+0000 7f487dffb700 1 -- 172.21.15.32:0/3208845611 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487800d310 con 0x7f48880fbc80 2024-04-06T14:28:44.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.687+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4888110410 con 0x7f48880fbc80 2024-04-06T14:28:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.688+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4888065e30 con 0x7f48880fbc80 2024-04-06T14:28:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.688+0000 7f487dffb700 1 -- 172.21.15.32:0/3208845611 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4878010420 con 0x7f48880fbc80 2024-04-06T14:28:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.689+0000 7f487dffb700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f487406cc20 0x7f487406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.689+0000 7f487dffb700 1 -- 172.21.15.32:0/3208845611 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f487808c0d0 con 0x7f48880fbc80 2024-04-06T14:28:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.689+0000 7f488cc2c700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f487406cc20 0x7f487406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.689+0000 7f488cc2c700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f487406cc20 0x7f487406f0d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4884004f90 tx=0x7f4884009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.693+0000 7f487dffb700 1 -- 172.21.15.32:0/3208845611 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4878059b20 con 0x7f48880fbc80 2024-04-06T14:28:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:44 smithi032 ceph-mon[147414]: pgmap v257: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:44.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.913+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f488804bf70 con 0x7f487406cc20 2024-04-06T14:28:44.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.915+0000 7f487dffb700 1 -- 172.21.15.32:0/3208845611 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4884007e00 con 0x7f487406cc20 2024-04-06T14:28:44.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f487406cc20 msgr2=0x7f487406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:44.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f487406cc20 0x7f487406f0d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4884004f90 tx=0x7f4884009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:44.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fbc80 msgr2=0x7f4888115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:44.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fbc80 0x7f4888115c70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f487800b790 tx=0x7f4878009250 comp rx=0 tx=0).stop 2024-04-06T14:28:44.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 shutdown_connections 2024-04-06T14:28:44.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f487406cc20 0x7f487406f0d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:44.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48880fbc80 0x7f4888115c70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:44.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 --2- 172.21.15.32:0/3208845611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48880fc590 0x7f48881161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:44.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 >> 172.21.15.32:0/3208845611 conn(0x7f4888072bc0 msgr2=0x7f4888105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:44.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 shutdown_connections 2024-04-06T14:28:44.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:44.917+0000 7f488f691700 1 -- 172.21.15.32:0/3208845611 wait complete. 2024-04-06T14:28:44.927 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 -- 172.21.15.32:0/1405153303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638102710 msgr2=0x7f86380fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 --2- 172.21.15.32:0/1405153303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638102710 0x7f86380fbd80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8628004940 tx=0x7f8628009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 -- 172.21.15.32:0/1405153303 shutdown_connections 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 --2- 172.21.15.32:0/1405153303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638102710 0x7f86380fbd80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 --2- 172.21.15.32:0/1405153303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8638101d70 0x7f8638102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 -- 172.21.15.32:0/1405153303 >> 172.21.15.32:0/1405153303 conn(0x7f8638072bc0 msgr2=0x7f8638072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 -- 172.21.15.32:0/1405153303 shutdown_connections 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 -- 172.21.15.32:0/1405153303 wait complete. 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 Processor -- start 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.021+0000 7f86402d1700 1 -- start start 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f86402d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638101d70 0x7f8638113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f86402d1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8638102710 0x7f8638114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f86402d1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638114650 con 0x7f8638101d70 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f86402d1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863810da40 con 0x7f8638102710 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f863e06d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638101d70 0x7f8638113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f863e06d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638101d70 0x7f8638113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58632/0 (socket says 172.21.15.32:58632) 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f863e06d700 1 -- 172.21.15.32:0/1531366764 learned_addr learned my addr 172.21.15.32:0/1531366764 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f863d86c700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8638102710 0x7f8638114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f863e06d700 1 -- 172.21.15.32:0/1531366764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8638102710 msgr2=0x7f8638114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f863e06d700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8638102710 0x7f8638114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f863e06d700 1 -- 172.21.15.32:0/1531366764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8628005040 con 0x7f8638101d70 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f863d86c700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8638102710 0x7f8638114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f863e06d700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638101d70 0x7f8638113ac0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f863400c830 tx=0x7f8634004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f862effd700 1 -- 172.21.15.32:0/1531366764 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f863400a760 con 0x7f8638101d70 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f862effd700 1 -- 172.21.15.32:0/1531366764 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f863400ad70 con 0x7f8638101d70 2024-04-06T14:28:45.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863810dd20 con 0x7f8638101d70 2024-04-06T14:28:45.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f862effd700 1 -- 172.21.15.32:0/1531366764 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86340113a0 con 0x7f8638101d70 2024-04-06T14:28:45.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.022+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863810e2e0 con 0x7f8638101d70 2024-04-06T14:28:45.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.023+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86380fd4c0 con 0x7f8638101d70 2024-04-06T14:28:45.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.024+0000 7f862effd700 1 -- 172.21.15.32:0/1531366764 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8634012400 con 0x7f8638101d70 2024-04-06T14:28:45.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.024+0000 7f862effd700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f862406cc20 0x7f862406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:45.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.024+0000 7f862effd700 1 -- 172.21.15.32:0/1531366764 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f863408bff0 con 0x7f8638101d70 2024-04-06T14:28:45.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.024+0000 7f863d86c700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f862406cc20 0x7f862406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:45.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.025+0000 7f863d86c700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f862406cc20 0x7f862406f0d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f862800b3b0 tx=0x7f8628009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.029+0000 7f862effd700 1 -- 172.21.15.32:0/1531366764 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8634059b80 con 0x7f8638101d70 2024-04-06T14:28:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.249+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8638108d10 con 0x7f862406cc20 2024-04-06T14:28:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.251+0000 7f862effd700 1 -- 172.21.15.32:0/1531366764 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8638108d10 con 0x7f862406cc20 2024-04-06T14:28:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f862406cc20 msgr2=0x7f862406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f862406cc20 0x7f862406f0d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f862800b3b0 tx=0x7f8628009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638101d70 msgr2=0x7f8638113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638101d70 0x7f8638113ac0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f863400c830 tx=0x7f8634004470 comp rx=0 tx=0).stop 2024-04-06T14:28:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 shutdown_connections 2024-04-06T14:28:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f862406cc20 0x7f862406f0d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8638101d70 0x7f8638113ac0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 --2- 172.21.15.32:0/1531366764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8638102710 0x7f8638114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 >> 172.21.15.32:0/1531366764 conn(0x7f8638072bc0 msgr2=0x7f8638065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 shutdown_connections 2024-04-06T14:28:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.253+0000 7f86402d1700 1 -- 172.21.15.32:0/1531366764 wait complete. 2024-04-06T14:28:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 -- 172.21.15.32:0/7207254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fde10 msgr2=0x7f9da40fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 --2- 172.21.15.32:0/7207254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fde10 0x7f9da40fe1e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9d98004940 tx=0x7f9d98009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 -- 172.21.15.32:0/7207254 shutdown_connections 2024-04-06T14:28:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 --2- 172.21.15.32:0/7207254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da40fe7b0 0x7f9da4106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 --2- 172.21.15.32:0/7207254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fde10 0x7f9da40fe1e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 -- 172.21.15.32:0/7207254 >> 172.21.15.32:0/7207254 conn(0x7f9da40f7880 msgr2=0x7f9da40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 -- 172.21.15.32:0/7207254 shutdown_connections 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 -- 172.21.15.32:0/7207254 wait complete. 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 Processor -- start 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.356+0000 7f9da8fcf700 1 -- start start 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da8fcf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da40fde10 0x7f9da4113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da8fcf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fe7b0 0x7f9da4114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da8fcf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da4114970 con 0x7f9da40fe7b0 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da8fcf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da410dd40 con 0x7f9da40fde10 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fe7b0 0x7f9da4114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fe7b0 0x7f9da4114320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58636/0 (socket says 172.21.15.32:58636) 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da1d9b700 1 -- 172.21.15.32:0/1557368290 learned_addr learned my addr 172.21.15.32:0/1557368290 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da259c700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da40fde10 0x7f9da4113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:45.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da1d9b700 1 -- 172.21.15.32:0/1557368290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da40fde10 msgr2=0x7f9da4113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da1d9b700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da40fde10 0x7f9da4113de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da1d9b700 1 -- 172.21.15.32:0/1557368290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d98005040 con 0x7f9da40fe7b0 2024-04-06T14:28:45.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da1d9b700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fe7b0 0x7f9da4114320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400c8c0 tx=0x7f9d940044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:45.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9d92ffd700 1 -- 172.21.15.32:0/1557368290 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d9400a760 con 0x7f9da40fe7b0 2024-04-06T14:28:45.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9d92ffd700 1 -- 172.21.15.32:0/1557368290 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d9400ad70 con 0x7f9da40fe7b0 2024-04-06T14:28:45.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da410e020 con 0x7f9da40fe7b0 2024-04-06T14:28:45.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.357+0000 7f9d92ffd700 1 -- 172.21.15.32:0/1557368290 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d940113a0 con 0x7f9da40fe7b0 2024-04-06T14:28:45.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.358+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da410e560 con 0x7f9da40fe7b0 2024-04-06T14:28:45.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.358+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da4066770 con 0x7f9da40fe7b0 2024-04-06T14:28:45.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.359+0000 7f9d92ffd700 1 -- 172.21.15.32:0/1557368290 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9d94012400 con 0x7f9da40fe7b0 2024-04-06T14:28:45.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.359+0000 7f9d92ffd700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d8c06cc20 0x7f9d8c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:45.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.359+0000 7f9d92ffd700 1 -- 172.21.15.32:0/1557368290 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9d9408c090 con 0x7f9da40fe7b0 2024-04-06T14:28:45.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.359+0000 7f9da259c700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d8c06cc20 0x7f9d8c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:45.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.360+0000 7f9da259c700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d8c06cc20 0x7f9d8c06f0d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9d98004f90 tx=0x7f9d98009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:45.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.364+0000 7f9d92ffd700 1 -- 172.21.15.32:0/1557368290 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9d94059b60 con 0x7f9da40fe7b0 2024-04-06T14:28:45.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.575+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9da4109d60 con 0x7f9d8c06cc20 2024-04-06T14:28:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.585+0000 7f9d92ffd700 1 -- 172.21.15.32:0/1557368290 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9da4109d60 con 0x7f9d8c06cc20 2024-04-06T14:28:45.585 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:28:45.585 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (9m) 2m ago 20m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:28:45.585 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (6m) 2m ago 20m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (6m) 2m ago 19m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (8m) 2m ago 20m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (2m) 2m ago 14m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (2m) 2m ago 14m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (2m) 2m ago 14m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (14m) 2m ago 14m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (11m) 2m ago 21m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (10m) 2m ago 19m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (7m) 2m ago 21m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (7m) 2m ago 19m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (9m) 2m ago 20m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (9m) 2m ago 19m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:28:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 2m ago 18m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:28:45.587 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 2m ago 18m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:28:45.587 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (4m) 2m ago 17m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:28:45.587 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (4m) 2m ago 17m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:28:45.587 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (3m) 2m ago 16m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:28:45.587 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (3m) 2m ago 16m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:28:45.587 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (9m) 2m ago 19m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d8c06cc20 msgr2=0x7f9d8c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d8c06cc20 0x7f9d8c06f0d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9d98004f90 tx=0x7f9d98009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fe7b0 msgr2=0x7f9da4114320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fe7b0 0x7f9da4114320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400c8c0 tx=0x7f9d940044c0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 shutdown_connections 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d8c06cc20 0x7f9d8c06f0d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da40fde10 0x7f9da4113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 --2- 172.21.15.32:0/1557368290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da40fe7b0 0x7f9da4114320 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 >> 172.21.15.32:0/1557368290 conn(0x7f9da40f7880 msgr2=0x7f9da41030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 shutdown_connections 2024-04-06T14:28:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.587+0000 7f9da8fcf700 1 -- 172.21.15.32:0/1557368290 wait complete. 2024-04-06T14:28:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.689+0000 7fcc92456700 1 -- 172.21.15.32:0/1024963140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 msgr2=0x7fcc8c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.689+0000 7fcc92456700 1 --2- 172.21.15.32:0/1024963140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 0x7fcc8c1031e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c004940 tx=0x7fcc7c009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.690+0000 7fcc92456700 1 -- 172.21.15.32:0/1024963140 shutdown_connections 2024-04-06T14:28:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.690+0000 7fcc92456700 1 --2- 172.21.15.32:0/1024963140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc8c1037b0 0x7fcc8c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.690+0000 7fcc92456700 1 --2- 172.21.15.32:0/1024963140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 0x7fcc8c1031e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.690+0000 7fcc92456700 1 -- 172.21.15.32:0/1024963140 >> 172.21.15.32:0/1024963140 conn(0x7fcc8c0f8980 msgr2=0x7fcc8c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.690+0000 7fcc92456700 1 -- 172.21.15.32:0/1024963140 shutdown_connections 2024-04-06T14:28:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.690+0000 7fcc92456700 1 -- 172.21.15.32:0/1024963140 wait complete. 2024-04-06T14:28:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.690+0000 7fcc92456700 1 Processor -- start 2024-04-06T14:28:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.690+0000 7fcc92456700 1 -- start start 2024-04-06T14:28:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc92456700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 0x7fcc8c113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc92456700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc8c1037b0 0x7fcc8c113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc92456700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c1145d0 con 0x7fcc8c102e10 2024-04-06T14:28:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc92456700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c10d9c0 con 0x7fcc8c1037b0 2024-04-06T14:28:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc8bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 0x7fcc8c113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc8bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 0x7fcc8c113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58648/0 (socket says 172.21.15.32:58648) 2024-04-06T14:28:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc8bfff700 1 -- 172.21.15.32:0/4041838303 learned_addr learned my addr 172.21.15.32:0/4041838303 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc8b7fe700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc8c1037b0 0x7fcc8c113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc8bfff700 1 -- 172.21.15.32:0/4041838303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc8c1037b0 msgr2=0x7fcc8c113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc8bfff700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc8c1037b0 0x7fcc8c113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc8bfff700 1 -- 172.21.15.32:0/4041838303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c005040 con 0x7fcc8c102e10 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc8b7fe700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc8c1037b0 0x7fcc8c113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc8bfff700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 0x7fcc8c113a40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c000c00 tx=0x7fcc7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc88ff9700 1 -- 172.21.15.32:0/4041838303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc7c021070 con 0x7fcc8c102e10 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc88ff9700 1 -- 172.21.15.32:0/4041838303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc7c013de0 con 0x7fcc8c102e10 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8c10dc40 con 0x7fcc8c102e10 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc88ff9700 1 -- 172.21.15.32:0/4041838303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc7c00e070 con 0x7fcc8c102e10 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.691+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8c10e1a0 con 0x7fcc8c102e10 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.692+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8c0fe560 con 0x7fcc8c102e10 2024-04-06T14:28:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.693+0000 7fcc88ff9700 1 -- 172.21.15.32:0/4041838303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcc7c01c600 con 0x7fcc8c102e10 2024-04-06T14:28:45.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.693+0000 7fcc88ff9700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7406cc20 0x7fcc7406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:45.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.693+0000 7fcc88ff9700 1 -- 172.21.15.32:0/4041838303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcc7c08d130 con 0x7fcc8c102e10 2024-04-06T14:28:45.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.693+0000 7fcc8b7fe700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7406cc20 0x7fcc7406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:45.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.694+0000 7fcc8b7fe700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7406cc20 0x7fcc7406f0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcc80004600 tx=0x7fcc8000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:45.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.698+0000 7fcc88ff9700 1 -- 172.21.15.32:0/4041838303 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcc7c05ac00 con 0x7fcc8c102e10 2024-04-06T14:28:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.988+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcc8c04bed0 con 0x7fcc8c102e10 2024-04-06T14:28:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.989+0000 7fcc88ff9700 1 -- 172.21.15.32:0/4041838303 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fcc7c05a790 con 0x7fcc8c102e10 2024-04-06T14:28:45.989 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:28:45.989 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:28:45.989 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:28:45.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:28:45.990 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:28:45.991 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:28:45.991 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:28:45.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:28:45.991 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:28:45.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7406cc20 msgr2=0x7fcc7406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7406cc20 0x7fcc7406f0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcc80004600 tx=0x7fcc8000b040 comp rx=0 tx=0).stop 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 msgr2=0x7fcc8c113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 0x7fcc8c113a40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c000c00 tx=0x7fcc7c009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 shutdown_connections 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7406cc20 0x7fcc7406f0d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc8c102e10 0x7fcc8c113a40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 --2- 172.21.15.32:0/4041838303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc8c1037b0 0x7fcc8c113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 >> 172.21.15.32:0/4041838303 conn(0x7fcc8c0f8980 msgr2=0x7fcc8c0f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 shutdown_connections 2024-04-06T14:28:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:45.991+0000 7fcc92456700 1 -- 172.21.15.32:0/4041838303 wait complete. 2024-04-06T14:28:46.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.094+0000 7fe215599700 1 -- 172.21.15.32:0/4231806588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 msgr2=0x7fe2100706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.095+0000 7fe215599700 1 --2- 172.21.15.32:0/4231806588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 0x7fe2100706d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe204004970 tx=0x7fe204009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:46.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.095+0000 7fe215599700 1 -- 172.21.15.32:0/4231806588 shutdown_connections 2024-04-06T14:28:46.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.095+0000 7fe215599700 1 --2- 172.21.15.32:0/4231806588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe210070c10 0x7fe210109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.095+0000 7fe215599700 1 --2- 172.21.15.32:0/4231806588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 0x7fe2100706d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.095+0000 7fe215599700 1 -- 172.21.15.32:0/4231806588 >> 172.21.15.32:0/4231806588 conn(0x7fe2100f89b0 msgr2=0x7fe2100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:46.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.095+0000 7fe215599700 1 -- 172.21.15.32:0/4231806588 shutdown_connections 2024-04-06T14:28:46.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.095+0000 7fe215599700 1 -- 172.21.15.32:0/4231806588 wait complete. 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe215599700 1 Processor -- start 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe215599700 1 -- start start 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe215599700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 0x7fe210115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe215599700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe210070c10 0x7fe210115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe215599700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2101165e0 con 0x7fe210070300 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe215599700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21010fbc0 con 0x7fe210070c10 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe20effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 0x7fe210115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe20effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 0x7fe210115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58662/0 (socket says 172.21.15.32:58662) 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe20effd700 1 -- 172.21.15.32:0/3494174906 learned_addr learned my addr 172.21.15.32:0/3494174906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe20e7fc700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe210070c10 0x7fe210115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:46.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe20effd700 1 -- 172.21.15.32:0/3494174906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe210070c10 msgr2=0x7fe210115f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe20effd700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe210070c10 0x7fe210115f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe20effd700 1 -- 172.21.15.32:0/3494174906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204005040 con 0x7fe210070300 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe20e7fc700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe210070c10 0x7fe210115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.096+0000 7fe20effd700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 0x7fe210115a50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe2040043d0 tx=0x7fe204009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.097+0000 7fe1f77fe700 1 -- 172.21.15.32:0/3494174906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20400e040 con 0x7fe210070300 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.097+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21010fe40 con 0x7fe210070300 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.097+0000 7fe1f77fe700 1 -- 172.21.15.32:0/3494174906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe204013420 con 0x7fe210070300 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.097+0000 7fe1f77fe700 1 -- 172.21.15.32:0/3494174906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20401c630 con 0x7fe210070300 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.097+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe210110380 con 0x7fe210070300 2024-04-06T14:28:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.098+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe210071a40 con 0x7fe210070300 2024-04-06T14:28:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.098+0000 7fe1f77fe700 1 -- 172.21.15.32:0/3494174906 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe204013970 con 0x7fe210070300 2024-04-06T14:28:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.099+0000 7fe1f77fe700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1f806cc20 0x7fe1f806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.099+0000 7fe1f77fe700 1 -- 172.21.15.32:0/3494174906 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe20408d2d0 con 0x7fe210070300 2024-04-06T14:28:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.099+0000 7fe20e7fc700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1f806cc20 0x7fe1f806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.099+0000 7fe20e7fc700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1f806cc20 0x7fe1f806f0d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe200007fd0 tx=0x7fe200009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.103+0000 7fe1f77fe700 1 -- 172.21.15.32:0/3494174906 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe20405ada0 con 0x7fe210070300 2024-04-06T14:28:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.354+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe2101110a0 con 0x7fe210070300 2024-04-06T14:28:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.354+0000 7fe1f77fe700 1 -- 172.21.15.32:0/3494174906 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fe204004de0 con 0x7fe210070300 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:28:46.355 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:28:46.356 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:28:46.357 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:28:46.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1f806cc20 msgr2=0x7fe1f806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1f806cc20 0x7fe1f806f0d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe200007fd0 tx=0x7fe200009040 comp rx=0 tx=0).stop 2024-04-06T14:28:46.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 msgr2=0x7fe210115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 0x7fe210115a50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe2040043d0 tx=0x7fe204009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:46.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 shutdown_connections 2024-04-06T14:28:46.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe1f806cc20 0x7fe1f806f0d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe210070300 0x7fe210115a50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 --2- 172.21.15.32:0/3494174906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe210070c10 0x7fe210115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 >> 172.21.15.32:0/3494174906 conn(0x7fe2100f89b0 msgr2=0x7fe210105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:46.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 shutdown_connections 2024-04-06T14:28:46.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.356+0000 7fe215599700 1 -- 172.21.15.32:0/3494174906 wait complete. 2024-04-06T14:28:46.359 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:28:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.461+0000 7f3f5457f700 1 -- 172.21.15.32:0/2250972444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 msgr2=0x7f3f4c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.461+0000 7f3f5457f700 1 --2- 172.21.15.32:0/2250972444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 0x7f3f4c0fb910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3f48004940 tx=0x7f3f48009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.461+0000 7f3f5457f700 1 -- 172.21.15.32:0/2250972444 shutdown_connections 2024-04-06T14:28:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.461+0000 7f3f5457f700 1 --2- 172.21.15.32:0/2250972444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 0x7f3f4c0fb910 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.461+0000 7f3f5457f700 1 --2- 172.21.15.32:0/2250972444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f4c102e70 0x7f3f4c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.461+0000 7f3f5457f700 1 -- 172.21.15.32:0/2250972444 >> 172.21.15.32:0/2250972444 conn(0x7f3f4c0714c0 msgr2=0x7f3f4c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.461+0000 7f3f5457f700 1 -- 172.21.15.32:0/2250972444 shutdown_connections 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.461+0000 7f3f5457f700 1 -- 172.21.15.32:0/2250972444 wait complete. 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f5457f700 1 Processor -- start 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f5457f700 1 -- start start 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f5457f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 0x7f3f4c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f5457f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f4c102e70 0x7f3f4c1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f5457f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c116830 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f5457f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c10fc00 con 0x7f3f4c102e70 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f5231b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 0x7f3f4c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f5231b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 0x7f3f4c115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58678/0 (socket says 172.21.15.32:58678) 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f5231b700 1 -- 172.21.15.32:0/573656139 learned_addr learned my addr 172.21.15.32:0/573656139 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.462+0000 7f3f51b1a700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f4c102e70 0x7f3f4c1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.463+0000 7f3f5231b700 1 -- 172.21.15.32:0/573656139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f4c102e70 msgr2=0x7f3f4c1161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.463+0000 7f3f5231b700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f4c102e70 0x7f3f4c1161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.463+0000 7f3f5231b700 1 -- 172.21.15.32:0/573656139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f48005040 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.463+0000 7f3f5231b700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 0x7f3f4c115ca0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00c8c0 tx=0x7f3f3c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.463+0000 7f3f42ffd700 1 -- 172.21.15.32:0/573656139 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3c00a760 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.463+0000 7f3f42ffd700 1 -- 172.21.15.32:0/573656139 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f3c00ad70 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.463+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c10fee0 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.463+0000 7f3f42ffd700 1 -- 172.21.15.32:0/573656139 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3c0113a0 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.463+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c1104a0 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.464+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c107290 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.465+0000 7f3f42ffd700 1 -- 172.21.15.32:0/573656139 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3f3c012400 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.465+0000 7f3f42ffd700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f3806cc20 0x7f3f3806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.465+0000 7f3f51b1a700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f3806cc20 0x7f3f3806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:46.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.465+0000 7f3f42ffd700 1 -- 172.21.15.32:0/573656139 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3f3c08c090 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.466+0000 7f3f51b1a700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f3806cc20 0x7f3f3806f0d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c111540 tx=0x7f3f48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:46.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.470+0000 7f3f42ffd700 1 -- 172.21.15.32:0/573656139 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3f3c059b60 con 0x7f3f4c0fb4a0 2024-04-06T14:28:46.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:46 smithi086 ceph-mon[110975]: pgmap v258: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:46.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:46 smithi086 ceph-mon[110975]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:46.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:46 smithi086 ceph-mon[110975]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:46.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:46 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4041838303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:28:46.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.692+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f4c109e00 con 0x7f3f3806cc20 2024-04-06T14:28:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.693+0000 7f3f42ffd700 1 -- 172.21.15.32:0/573656139 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3f4c109e00 con 0x7f3f3806cc20 2024-04-06T14:28:46.693 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:28:46.693 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:28:46.693 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:28:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:28:46.695 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:28:46.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f3806cc20 msgr2=0x7f3f3806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f3806cc20 0x7f3f3806f0d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c111540 tx=0x7f3f48009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:46.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 msgr2=0x7f3f4c115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 0x7f3f4c115ca0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00c8c0 tx=0x7f3f3c0044c0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 shutdown_connections 2024-04-06T14:28:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f3806cc20 0x7f3f3806f0d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f4c0fb4a0 0x7f3f4c115ca0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 --2- 172.21.15.32:0/573656139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f4c102e70 0x7f3f4c1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 >> 172.21.15.32:0/573656139 conn(0x7f3f4c0714c0 msgr2=0x7f3f4c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 shutdown_connections 2024-04-06T14:28:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.695+0000 7f3f5457f700 1 -- 172.21.15.32:0/573656139 wait complete. 2024-04-06T14:28:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:46 smithi032 ceph-mon[147414]: pgmap v258: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:46 smithi032 ceph-mon[147414]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:46 smithi032 ceph-mon[147414]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:46 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4041838303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:28:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.799+0000 7f5937c5c700 1 -- 172.21.15.32:0/2735927105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5930071f90 msgr2=0x7f5930070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.799+0000 7f5937c5c700 1 --2- 172.21.15.32:0/2735927105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5930071f90 0x7f5930070360 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f592c004970 tx=0x7f592c009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.799+0000 7f5937c5c700 1 -- 172.21.15.32:0/2735927105 shutdown_connections 2024-04-06T14:28:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.799+0000 7f5937c5c700 1 --2- 172.21.15.32:0/2735927105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59300708a0 0x7f5930070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.799+0000 7f5937c5c700 1 --2- 172.21.15.32:0/2735927105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5930071f90 0x7f5930070360 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.799+0000 7f5937c5c700 1 -- 172.21.15.32:0/2735927105 >> 172.21.15.32:0/2735927105 conn(0x7f59300f89b0 msgr2=0x7f59300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.799+0000 7f5937c5c700 1 -- 172.21.15.32:0/2735927105 shutdown_connections 2024-04-06T14:28:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.799+0000 7f5937c5c700 1 -- 172.21.15.32:0/2735927105 wait complete. 2024-04-06T14:28:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.800+0000 7f5937c5c700 1 Processor -- start 2024-04-06T14:28:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.800+0000 7f5937c5c700 1 -- start start 2024-04-06T14:28:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.800+0000 7f5937c5c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59300708a0 0x7f593011a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.800+0000 7f5937c5c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5930071f90 0x7f593011a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.800+0000 7f5937c5c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593011ac40 con 0x7f59300708a0 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.800+0000 7f5937c5c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5930114010 con 0x7f5930071f90 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.800+0000 7f59359f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59300708a0 0x7f593011a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.800+0000 7f59359f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59300708a0 0x7f593011a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58692/0 (socket says 172.21.15.32:58692) 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.800+0000 7f59359f8700 1 -- 172.21.15.32:0/2500613417 learned_addr learned my addr 172.21.15.32:0/2500613417 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f59351f7700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5930071f90 0x7f593011a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f59359f8700 1 -- 172.21.15.32:0/2500613417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5930071f90 msgr2=0x7f593011a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f59359f8700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5930071f90 0x7f593011a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f59359f8700 1 -- 172.21.15.32:0/2500613417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c005040 con 0x7f59300708a0 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f59351f7700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5930071f90 0x7f593011a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:28:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f59359f8700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59300708a0 0x7f593011a0b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f592c005010 tx=0x7f592c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f59267fc700 1 -- 172.21.15.32:0/2500613417 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f592c00e040 con 0x7f59300708a0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f59267fc700 1 -- 172.21.15.32:0/2500613417 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f592c009930 con 0x7f59300708a0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5930114290 con 0x7f59300708a0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f59267fc700 1 -- 172.21.15.32:0/2500613417 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f592c01c3f0 con 0x7f59300708a0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.801+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59301147d0 con 0x7f59300708a0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.802+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5930071aa0 con 0x7f59300708a0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.803+0000 7f59267fc700 1 -- 172.21.15.32:0/2500613417 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f592c013420 con 0x7f59300708a0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.803+0000 7f59267fc700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591c06cc30 0x7f591c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.803+0000 7f59267fc700 1 -- 172.21.15.32:0/2500613417 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f592c08d310 con 0x7f59300708a0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.803+0000 7f59351f7700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591c06cc30 0x7f591c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:28:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.803+0000 7f59351f7700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591c06cc30 0x7f591c06f0e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5920004600 tx=0x7f592000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:28:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:46.808+0000 7f59267fc700 1 -- 172.21.15.32:0/2500613417 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f592c05ade0 con 0x7f59300708a0 2024-04-06T14:28:47.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.093+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f593004bed0 con 0x7f59300708a0 2024-04-06T14:28:47.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.093+0000 7f59267fc700 1 -- 172.21.15.32:0/2500613417 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f592c05a970 con 0x7f59300708a0 2024-04-06T14:28:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:28:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:28:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:28:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:28:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:28:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:28:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:28:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.095+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591c06cc30 msgr2=0x7f591c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.095+0000 7f5937c5c700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591c06cc30 0x7f591c06f0e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5920004600 tx=0x7f592000b040 comp rx=0 tx=0).stop 2024-04-06T14:28:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.095+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59300708a0 msgr2=0x7f593011a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:28:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.095+0000 7f5937c5c700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59300708a0 0x7f593011a0b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f592c005010 tx=0x7f592c009e30 comp rx=0 tx=0).stop 2024-04-06T14:28:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.095+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 shutdown_connections 2024-04-06T14:28:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.095+0000 7f5937c5c700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591c06cc30 0x7f591c06f0e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.096+0000 7f5937c5c700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59300708a0 0x7f593011a0b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.096+0000 7f5937c5c700 1 --2- 172.21.15.32:0/2500613417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5930071f90 0x7f593011a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:28:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.096+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 >> 172.21.15.32:0/2500613417 conn(0x7f59300f89b0 msgr2=0x7f59300f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:28:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.096+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 shutdown_connections 2024-04-06T14:28:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:28:47.096+0000 7f5937c5c700 1 -- 172.21.15.32:0/2500613417 wait complete. 2024-04-06T14:28:47.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:47 smithi086 ceph-mon[110975]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:47.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:47 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3494174906' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:28:47.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:47 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2500613417' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:28:47.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:47 smithi032 ceph-mon[147414]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:47.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:47 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3494174906' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:28:47.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:47 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2500613417' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:28:48.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:48 smithi086 ceph-mon[110975]: pgmap v259: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:48.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:48 smithi086 ceph-mon[110975]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:48.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:48 smithi032 ceph-mon[147414]: pgmap v259: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:48.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:48 smithi032 ceph-mon[147414]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:28:50.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:50 smithi086 ceph-mon[110975]: pgmap v260: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:50.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:50 smithi032 ceph-mon[147414]: pgmap v260: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:52.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:52 smithi086 ceph-mon[110975]: pgmap v261: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:52 smithi032 ceph-mon[147414]: pgmap v261: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:53.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:28:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:28:54.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:54 smithi086 ceph-mon[110975]: pgmap v262: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:54.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:54 smithi032 ceph-mon[147414]: pgmap v262: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:56.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:56 smithi086 ceph-mon[110975]: pgmap v263: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:56.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:56 smithi032 ceph-mon[147414]: pgmap v263: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:58.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:28:58 smithi086 ceph-mon[110975]: pgmap v264: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:28:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:28:58 smithi032 ceph-mon[147414]: pgmap v264: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:00 smithi086 ceph-mon[110975]: pgmap v265: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:00 smithi032 ceph-mon[147414]: pgmap v265: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:02.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:02 smithi086 ceph-mon[110975]: pgmap v266: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:02.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:02 smithi032 ceph-mon[147414]: pgmap v266: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:04.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:04 smithi086 ceph-mon[110975]: pgmap v267: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:04.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:04 smithi032 ceph-mon[147414]: pgmap v267: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:06.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:06 smithi086 ceph-mon[110975]: pgmap v268: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:06 smithi032 ceph-mon[147414]: pgmap v268: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:08.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:08 smithi086 ceph-mon[110975]: pgmap v269: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:29:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:08 smithi032 ceph-mon[147414]: pgmap v269: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:29:10.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:10 smithi086 ceph-mon[110975]: pgmap v270: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:10 smithi032 ceph-mon[147414]: pgmap v270: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:12.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:12 smithi086 ceph-mon[110975]: pgmap v271: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:12.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:12 smithi032 ceph-mon[147414]: pgmap v271: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:14.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:14 smithi086 ceph-mon[110975]: pgmap v272: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:14 smithi032 ceph-mon[147414]: pgmap v272: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:16.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:16 smithi086 ceph-mon[110975]: pgmap v273: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:16.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:16 smithi032 ceph-mon[147414]: pgmap v273: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:17.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.214+0000 7fcfd1e26700 1 -- 172.21.15.32:0/4217793107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc102e70 msgr2=0x7fcfcc103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.214+0000 7fcfd1e26700 1 --2- 172.21.15.32:0/4217793107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc102e70 0x7fcfcc103240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc004940 tx=0x7fcfbc009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.214+0000 7fcfd1e26700 1 -- 172.21.15.32:0/4217793107 shutdown_connections 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.214+0000 7fcfd1e26700 1 --2- 172.21.15.32:0/4217793107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0fb4a0 0x7fcfcc0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.214+0000 7fcfd1e26700 1 --2- 172.21.15.32:0/4217793107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc102e70 0x7fcfcc103240 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.214+0000 7fcfd1e26700 1 -- 172.21.15.32:0/4217793107 >> 172.21.15.32:0/4217793107 conn(0x7fcfcc0714c0 msgr2=0x7fcfcc0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.214+0000 7fcfd1e26700 1 -- 172.21.15.32:0/4217793107 shutdown_connections 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.214+0000 7fcfd1e26700 1 -- 172.21.15.32:0/4217793107 wait complete. 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.215+0000 7fcfd1e26700 1 Processor -- start 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.215+0000 7fcfd1e26700 1 -- start start 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.215+0000 7fcfd1e26700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc0fb4a0 0x7fcfcc113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.215+0000 7fcfd1e26700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc102e70 0x7fcfcc113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.215+0000 7fcfd1e26700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc1146b0 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.215+0000 7fcfd1e26700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc10da10 con 0x7fcfcc102e70 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfcb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc0fb4a0 0x7fcfcc113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfcb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc0fb4a0 0x7fcfcc113a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54802/0 (socket says 172.21.15.32:54802) 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfcb7fe700 1 -- 172.21.15.32:0/1319764187 learned_addr learned my addr 172.21.15.32:0/1319764187 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfcaffd700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc102e70 0x7fcfcc113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfcb7fe700 1 -- 172.21.15.32:0/1319764187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc102e70 msgr2=0x7fcfcc113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfcb7fe700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc102e70 0x7fcfcc113fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfcb7fe700 1 -- 172.21.15.32:0/1319764187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfbc005040 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfcaffd700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc102e70 0x7fcfcc113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfcb7fe700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc0fb4a0 0x7fcfcc113a90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00b3b0 tx=0x7fcfbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfb3fff700 1 -- 172.21.15.32:0/1319764187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfbc00e040 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc10dc90 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfb3fff700 1 -- 172.21.15.32:0/1319764187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfbc013420 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfb3fff700 1 -- 172.21.15.32:0/1319764187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfbc01c630 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.216+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc10e250 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.217+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc100110 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.218+0000 7fcfb3fff700 1 -- 172.21.15.32:0/1319764187 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcfbc013970 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.218+0000 7fcfb3fff700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfb406cc20 0x7fcfb406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:17.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.218+0000 7fcfb3fff700 1 -- 172.21.15.32:0/1319764187 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcfbc08d2b0 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.219+0000 7fcfcaffd700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfb406cc20 0x7fcfb406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:17.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.219+0000 7fcfcaffd700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfb406cc20 0x7fcfb406f0d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc10ebd0 tx=0x7fcfc0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:17.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.223+0000 7fcfb3fff700 1 -- 172.21.15.32:0/1319764187 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcfbc05ad80 con 0x7fcfcc0fb4a0 2024-04-06T14:29:17.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.448+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfcc10f150 con 0x7fcfb406cc20 2024-04-06T14:29:17.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.449+0000 7fcfb3fff700 1 -- 172.21.15.32:0/1319764187 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcfcc10f150 con 0x7fcfb406cc20 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.451+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfb406cc20 msgr2=0x7fcfb406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.451+0000 7fcfd1e26700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfb406cc20 0x7fcfb406f0d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc10ebd0 tx=0x7fcfc0008040 comp rx=0 tx=0).stop 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.451+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc0fb4a0 msgr2=0x7fcfcc113a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.451+0000 7fcfd1e26700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc0fb4a0 0x7fcfcc113a90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00b3b0 tx=0x7fcfbc009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.451+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 shutdown_connections 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.451+0000 7fcfd1e26700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfb406cc20 0x7fcfb406f0d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.451+0000 7fcfd1e26700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfcc0fb4a0 0x7fcfcc113a90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.451+0000 7fcfd1e26700 1 --2- 172.21.15.32:0/1319764187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc102e70 0x7fcfcc113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.451+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 >> 172.21.15.32:0/1319764187 conn(0x7fcfcc0714c0 msgr2=0x7fcfcc0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.452+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 shutdown_connections 2024-04-06T14:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.452+0000 7fcfd1e26700 1 -- 172.21.15.32:0/1319764187 wait complete. 2024-04-06T14:29:17.461 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:29:17.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.556+0000 7f1f8be56700 1 -- 172.21.15.32:0/1402548071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 msgr2=0x7f1f840fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.556+0000 7f1f8be56700 1 --2- 172.21.15.32:0/1402548071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 0x7f1f840fe260 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1f80004940 tx=0x7f1f80009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:17.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.556+0000 7f1f8be56700 1 -- 172.21.15.32:0/1402548071 shutdown_connections 2024-04-06T14:29:17.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.556+0000 7f1f8be56700 1 --2- 172.21.15.32:0/1402548071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f840fe830 0x7f1f84106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.556+0000 7f1f8be56700 1 --2- 172.21.15.32:0/1402548071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 0x7f1f840fe260 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.556+0000 7f1f8be56700 1 -- 172.21.15.32:0/1402548071 >> 172.21.15.32:0/1402548071 conn(0x7f1f84072bc0 msgr2=0x7f1f84072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:17.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.556+0000 7f1f8be56700 1 -- 172.21.15.32:0/1402548071 shutdown_connections 2024-04-06T14:29:17.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.556+0000 7f1f8be56700 1 -- 172.21.15.32:0/1402548071 wait complete. 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f8be56700 1 Processor -- start 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f8be56700 1 -- start start 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f8be56700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 0x7f1f840fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f8be56700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f840fe830 0x7f1f840fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f8be56700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840fc820 con 0x7f1f840fde90 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f8be56700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840fc960 con 0x7f1f840fe830 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f89bf2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 0x7f1f840fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f89bf2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 0x7f1f840fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54820/0 (socket says 172.21.15.32:54820) 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f89bf2700 1 -- 172.21.15.32:0/1081264098 learned_addr learned my addr 172.21.15.32:0/1081264098 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.557+0000 7f1f893f1700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f840fe830 0x7f1f840fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.558+0000 7f1f89bf2700 1 -- 172.21.15.32:0/1081264098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f840fe830 msgr2=0x7f1f840fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.558+0000 7f1f89bf2700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f840fe830 0x7f1f840fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.558+0000 7f1f89bf2700 1 -- 172.21.15.32:0/1081264098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f80005040 con 0x7f1f840fde90 2024-04-06T14:29:17.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.558+0000 7f1f89bf2700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 0x7f1f840fbd50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1f800093b0 tx=0x7f1f80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:17.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.558+0000 7f1f767fc700 1 -- 172.21.15.32:0/1081264098 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f8000e040 con 0x7f1f840fde90 2024-04-06T14:29:17.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.558+0000 7f1f767fc700 1 -- 172.21.15.32:0/1081264098 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f8000bed0 con 0x7f1f840fde90 2024-04-06T14:29:17.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.558+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f840fcbb0 con 0x7f1f840fde90 2024-04-06T14:29:17.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.558+0000 7f1f767fc700 1 -- 172.21.15.32:0/1081264098 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f8001c3f0 con 0x7f1f840fde90 2024-04-06T14:29:17.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.558+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8411a1a0 con 0x7f1f840fde90 2024-04-06T14:29:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.559+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f84065e30 con 0x7f1f840fde90 2024-04-06T14:29:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.559+0000 7f1f767fc700 1 -- 172.21.15.32:0/1081264098 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1f80013790 con 0x7f1f840fde90 2024-04-06T14:29:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.560+0000 7f1f767fc700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1f7006cc30 0x7f1f7006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.560+0000 7f1f893f1700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1f7006cc30 0x7f1f7006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.560+0000 7f1f767fc700 1 -- 172.21.15.32:0/1081264098 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1f8008d3a0 con 0x7f1f840fde90 2024-04-06T14:29:17.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.561+0000 7f1f893f1700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1f7006cc30 0x7f1f7006f0e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f840fd690 tx=0x7f1f7800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:17.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.565+0000 7f1f767fc700 1 -- 172.21.15.32:0/1081264098 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1f800573e0 con 0x7f1f840fde90 2024-04-06T14:29:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.785+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f840fd490 con 0x7f1f7006cc30 2024-04-06T14:29:17.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.786+0000 7f1f767fc700 1 -- 172.21.15.32:0/1081264098 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1f840fd490 con 0x7f1f7006cc30 2024-04-06T14:29:17.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1f7006cc30 msgr2=0x7f1f7006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1f7006cc30 0x7f1f7006f0e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f840fd690 tx=0x7f1f7800b040 comp rx=0 tx=0).stop 2024-04-06T14:29:17.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 msgr2=0x7f1f840fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 0x7f1f840fbd50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1f800093b0 tx=0x7f1f80009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:17.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 shutdown_connections 2024-04-06T14:29:17.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1f7006cc30 0x7f1f7006f0e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840fde90 0x7f1f840fbd50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 --2- 172.21.15.32:0/1081264098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f840fe830 0x7f1f840fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 >> 172.21.15.32:0/1081264098 conn(0x7f1f84072bc0 msgr2=0x7f1f840ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:17.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 shutdown_connections 2024-04-06T14:29:17.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.788+0000 7f1f8be56700 1 -- 172.21.15.32:0/1081264098 wait complete. 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.891+0000 7f36f33c1700 1 -- 172.21.15.32:0/3734594509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 msgr2=0x7f36ec0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.891+0000 7f36f33c1700 1 --2- 172.21.15.32:0/3734594509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 0x7f36ec0fbd80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f36dc004940 tx=0x7f36dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.891+0000 7f36f33c1700 1 -- 172.21.15.32:0/3734594509 shutdown_connections 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.891+0000 7f36f33c1700 1 --2- 172.21.15.32:0/3734594509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 0x7f36ec0fbd80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.891+0000 7f36f33c1700 1 --2- 172.21.15.32:0/3734594509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec101d70 0x7f36ec102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.891+0000 7f36f33c1700 1 -- 172.21.15.32:0/3734594509 >> 172.21.15.32:0/3734594509 conn(0x7f36ec072bc0 msgr2=0x7f36ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.891+0000 7f36f33c1700 1 -- 172.21.15.32:0/3734594509 shutdown_connections 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.891+0000 7f36f33c1700 1 -- 172.21.15.32:0/3734594509 wait complete. 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.891+0000 7f36f33c1700 1 Processor -- start 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f33c1700 1 -- start start 2024-04-06T14:29:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f33c1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec101d70 0x7f36ec102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f33c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 0x7f36ec103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f33c1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec1036a0 con 0x7f36ec102710 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f33c1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec1037e0 con 0x7f36ec101d70 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f095c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 0x7f36ec103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f095c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 0x7f36ec103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54838/0 (socket says 172.21.15.32:54838) 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f095c700 1 -- 172.21.15.32:0/2520035676 learned_addr learned my addr 172.21.15.32:0/2520035676 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f115d700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec101d70 0x7f36ec102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f095c700 1 -- 172.21.15.32:0/2520035676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec101d70 msgr2=0x7f36ec102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f095c700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec101d70 0x7f36ec102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f095c700 1 -- 172.21.15.32:0/2520035676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc005040 con 0x7f36ec102710 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f095c700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 0x7f36ec103110 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009380 tx=0x7f36dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36e1ffb700 1 -- 172.21.15.32:0/2520035676 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36dc00e040 con 0x7f36ec102710 2024-04-06T14:29:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec103a30 con 0x7f36ec102710 2024-04-06T14:29:17.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36e1ffb700 1 -- 172.21.15.32:0/2520035676 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36dc013420 con 0x7f36ec102710 2024-04-06T14:29:17.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.892+0000 7f36e1ffb700 1 -- 172.21.15.32:0/2520035676 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36dc01c630 con 0x7f36ec102710 2024-04-06T14:29:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.893+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec11a1f0 con 0x7f36ec102710 2024-04-06T14:29:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.894+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ec0fd4c0 con 0x7f36ec102710 2024-04-06T14:29:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.894+0000 7f36e1ffb700 1 -- 172.21.15.32:0/2520035676 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f36dc0139e0 con 0x7f36ec102710 2024-04-06T14:29:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.895+0000 7f36e1ffb700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f36d806cc20 0x7f36d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.895+0000 7f36e1ffb700 1 -- 172.21.15.32:0/2520035676 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f36dc08d210 con 0x7f36ec102710 2024-04-06T14:29:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.895+0000 7f36f115d700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f36d806cc20 0x7f36d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.895+0000 7f36f115d700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f36d806cc20 0x7f36d806f0d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36e8007fd0 tx=0x7f36e8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:17.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:17.899+0000 7f36e1ffb700 1 -- 172.21.15.32:0/2520035676 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f36dc057250 con 0x7f36ec102710 2024-04-06T14:29:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.111+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f36ec108db0 con 0x7f36d806cc20 2024-04-06T14:29:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.121+0000 7f36e1ffb700 1 -- 172.21.15.32:0/2520035676 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f36e8004db0 con 0x7f36d806cc20 2024-04-06T14:29:18.120 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:29:18.120 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (9m) 2m ago 20m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:29:18.120 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 2m ago 21m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:29:18.120 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (6m) 2m ago 20m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:29:18.120 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (9m) 2m ago 20m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:29:18.120 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (3m) 2m ago 15m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (3m) 2m ago 15m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (2m) 2m ago 15m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (15m) 2m ago 15m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (11m) 2m ago 22m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (10m) 2m ago 20m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 2m ago 22m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (7m) 2m ago 19m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (10m) 2m ago 21m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (10m) 2m ago 20m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 2m ago 19m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 2m ago 18m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 2m ago 18m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:29:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (4m) 2m ago 17m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:29:18.122 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (4m) 2m ago 17m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:29:18.122 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (3m) 2m ago 16m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:29:18.122 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (9m) 2m ago 20m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f36d806cc20 msgr2=0x7f36d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f36d806cc20 0x7f36d806f0d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36e8007fd0 tx=0x7f36e8009040 comp rx=0 tx=0).stop 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 msgr2=0x7f36ec103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 0x7f36ec103110 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009380 tx=0x7f36dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 shutdown_connections 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f36d806cc20 0x7f36d806f0d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec101d70 0x7f36ec102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 --2- 172.21.15.32:0/2520035676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36ec102710 0x7f36ec103110 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 >> 172.21.15.32:0/2520035676 conn(0x7f36ec072bc0 msgr2=0x7f36ec065a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 shutdown_connections 2024-04-06T14:29:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.123+0000 7f36f33c1700 1 -- 172.21.15.32:0/2520035676 wait complete. 2024-04-06T14:29:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/2485260837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 msgr2=0x7f8e9c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 --2- 172.21.15.32:0/2485260837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 0x7f8e9c1031d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8e98004940 tx=0x7f8e98009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/2485260837 shutdown_connections 2024-04-06T14:29:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 --2- 172.21.15.32:0/2485260837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9c1037a0 0x7f8e9c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 --2- 172.21.15.32:0/2485260837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 0x7f8e9c1031d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/2485260837 >> 172.21.15.32:0/2485260837 conn(0x7f8e9c0f89b0 msgr2=0x7f8e9c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/2485260837 shutdown_connections 2024-04-06T14:29:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/2485260837 wait complete. 2024-04-06T14:29:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 Processor -- start 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.225+0000 7f8ea3ca3700 1 -- start start 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea3ca3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 0x7f8e9c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea3ca3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9c1037a0 0x7f8e9c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea3ca3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c070d90 con 0x7f8e9c102e00 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea3ca3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c070ed0 con 0x7f8e9c1037a0 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea1a3f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 0x7f8e9c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea1a3f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 0x7f8e9c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54852/0 (socket says 172.21.15.32:54852) 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea1a3f700 1 -- 172.21.15.32:0/1672554071 learned_addr learned my addr 172.21.15.32:0/1672554071 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea1a3f700 1 -- 172.21.15.32:0/1672554071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9c1037a0 msgr2=0x7f8e9c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea1a3f700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9c1037a0 0x7f8e9c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea1a3f700 1 -- 172.21.15.32:0/1672554071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e98005040 con 0x7f8e9c102e00 2024-04-06T14:29:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea1a3f700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 0x7f8e9c0702c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8e98000c00 tx=0x7f8e98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8e927fc700 1 -- 172.21.15.32:0/1672554071 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9800e040 con 0x7f8e9c102e00 2024-04-06T14:29:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8e927fc700 1 -- 172.21.15.32:0/1672554071 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e98013420 con 0x7f8e9c102e00 2024-04-06T14:29:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8e927fc700 1 -- 172.21.15.32:0/1672554071 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9801c630 con 0x7f8e9c102e00 2024-04-06T14:29:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9c071120 con 0x7f8e9c102e00 2024-04-06T14:29:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.226+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e9c11a270 con 0x7f8e9c102e00 2024-04-06T14:29:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.227+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9c0fe550 con 0x7f8e9c102e00 2024-04-06T14:29:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.228+0000 7f8e927fc700 1 -- 172.21.15.32:0/1672554071 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8e98013590 con 0x7f8e9c102e00 2024-04-06T14:29:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.228+0000 7f8e927fc700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e8806cc70 0x7f8e8806f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.228+0000 7f8e927fc700 1 -- 172.21.15.32:0/1672554071 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8e9808d300 con 0x7f8e9c102e00 2024-04-06T14:29:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.228+0000 7f8ea123e700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e8806cc70 0x7f8e8806f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.229+0000 7f8ea123e700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e8806cc70 0x7f8e8806f120 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c071c00 tx=0x7f8e8c009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:18.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.233+0000 7f8e927fc700 1 -- 172.21.15.32:0/1672554071 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8e980572c0 con 0x7f8e9c102e00 2024-04-06T14:29:18.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.519+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8e9c04bed0 con 0x7f8e9c102e00 2024-04-06T14:29:18.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.519+0000 7f8e927fc700 1 -- 172.21.15.32:0/1672554071 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f8e9805a8e0 con 0x7f8e9c102e00 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:29:18.519 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:29:18.520 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:29:18.520 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:29:18.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:29:18.520 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:29:18.520 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:29:18.520 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:29:18.520 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:29:18.520 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e8806cc70 msgr2=0x7f8e8806f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e8806cc70 0x7f8e8806f120 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c071c00 tx=0x7f8e8c009ce0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 msgr2=0x7f8e9c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 0x7f8e9c0702c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8e98000c00 tx=0x7f8e98009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 shutdown_connections 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e8806cc70 0x7f8e8806f120 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e9c102e00 0x7f8e9c0702c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 --2- 172.21.15.32:0/1672554071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9c1037a0 0x7f8e9c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 >> 172.21.15.32:0/1672554071 conn(0x7f8e9c0f89b0 msgr2=0x7f8e9c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 shutdown_connections 2024-04-06T14:29:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.521+0000 7f8ea3ca3700 1 -- 172.21.15.32:0/1672554071 wait complete. 2024-04-06T14:29:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/2437490788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b38102e00 msgr2=0x7f8b381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 --2- 172.21.15.32:0/2437490788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b38102e00 0x7f8b381031d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8b28004940 tx=0x7f8b28009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/2437490788 shutdown_connections 2024-04-06T14:29:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 --2- 172.21.15.32:0/2437490788 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b381037a0 0x7f8b380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 --2- 172.21.15.32:0/2437490788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b38102e00 0x7f8b381031d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/2437490788 >> 172.21.15.32:0/2437490788 conn(0x7f8b380f89b0 msgr2=0x7f8b380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/2437490788 shutdown_connections 2024-04-06T14:29:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/2437490788 wait complete. 2024-04-06T14:29:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 Processor -- start 2024-04-06T14:29:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.625+0000 7f8b3cdfd700 1 -- start start 2024-04-06T14:29:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b3cdfd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b38102e00 0x7f8b380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b3cdfd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b381037a0 0x7f8b38070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b3cdfd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b38070d90 con 0x7f8b381037a0 2024-04-06T14:29:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b3cdfd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b38070ed0 con 0x7f8b38102e00 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b35d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b381037a0 0x7f8b38070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b35d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b381037a0 0x7f8b38070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54880/0 (socket says 172.21.15.32:54880) 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b35d9b700 1 -- 172.21.15.32:0/3657067566 learned_addr learned my addr 172.21.15.32:0/3657067566 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b3659c700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b38102e00 0x7f8b380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b35d9b700 1 -- 172.21.15.32:0/3657067566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b38102e00 msgr2=0x7f8b380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b35d9b700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b38102e00 0x7f8b380702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b35d9b700 1 -- 172.21.15.32:0/3657067566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b28005040 con 0x7f8b381037a0 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b35d9b700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b381037a0 0x7f8b38070800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c002a50 tx=0x7f8b2c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b26ffd700 1 -- 172.21.15.32:0/3657067566 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b2c007cc0 con 0x7f8b381037a0 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b38119e20 con 0x7f8b381037a0 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b26ffd700 1 -- 172.21.15.32:0/3657067566 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b2c00f040 con 0x7f8b381037a0 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b26ffd700 1 -- 172.21.15.32:0/3657067566 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b2c00d920 con 0x7f8b381037a0 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.626+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3811a2a0 con 0x7f8b381037a0 2024-04-06T14:29:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.627+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b380fe550 con 0x7f8b381037a0 2024-04-06T14:29:18.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:18 smithi086 ceph-mon[110975]: pgmap v274: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:18.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:18 smithi086 ceph-mon[110975]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.628+0000 7f8b26ffd700 1 -- 172.21.15.32:0/3657067566 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8b2c009680 con 0x7f8b381037a0 2024-04-06T14:29:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.628+0000 7f8b26ffd700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b2006cc20 0x7f8b2006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.628+0000 7f8b26ffd700 1 -- 172.21.15.32:0/3657067566 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8b2c08be00 con 0x7f8b381037a0 2024-04-06T14:29:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.628+0000 7f8b3659c700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b2006cc20 0x7f8b2006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.629+0000 7f8b3659c700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b2006cc20 0x7f8b2006f0d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8b28004f90 tx=0x7f8b28009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:18.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.633+0000 7f8b26ffd700 1 -- 172.21.15.32:0/3657067566 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8b2c055dc0 con 0x7f8b381037a0 2024-04-06T14:29:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:18 smithi032 ceph-mon[147414]: pgmap v274: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:18 smithi032 ceph-mon[147414]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:18.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.881+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8b3804bed0 con 0x7f8b381037a0 2024-04-06T14:29:18.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.881+0000 7f8b26ffd700 1 -- 172.21.15.32:0/3657067566 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f8b2c0593e0 con 0x7f8b381037a0 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:29:18.881 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:29:18.882 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:29:18.883 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:29:18.884 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:29:18.884 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:29:18.884 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:29:18.884 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:29:18.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b2006cc20 msgr2=0x7f8b2006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b2006cc20 0x7f8b2006f0d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8b28004f90 tx=0x7f8b28009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:18.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b381037a0 msgr2=0x7f8b38070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b381037a0 0x7f8b38070800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c002a50 tx=0x7f8b2c004600 comp rx=0 tx=0).stop 2024-04-06T14:29:18.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 shutdown_connections 2024-04-06T14:29:18.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b2006cc20 0x7f8b2006f0d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b38102e00 0x7f8b380702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 --2- 172.21.15.32:0/3657067566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b381037a0 0x7f8b38070800 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 >> 172.21.15.32:0/3657067566 conn(0x7f8b380f89b0 msgr2=0x7f8b380f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:18.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.883+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 shutdown_connections 2024-04-06T14:29:18.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.884+0000 7f8b3cdfd700 1 -- 172.21.15.32:0/3657067566 wait complete. 2024-04-06T14:29:18.886 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:29:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.988+0000 7fc380716700 1 -- 172.21.15.32:0/4270786332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc378102e70 msgr2=0x7fc378103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.988+0000 7fc380716700 1 --2- 172.21.15.32:0/4270786332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc378102e70 0x7fc378103240 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc374004940 tx=0x7fc374009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.988+0000 7fc380716700 1 -- 172.21.15.32:0/4270786332 shutdown_connections 2024-04-06T14:29:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.988+0000 7fc380716700 1 --2- 172.21.15.32:0/4270786332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3780fb4a0 0x7fc3780fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.988+0000 7fc380716700 1 --2- 172.21.15.32:0/4270786332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc378102e70 0x7fc378103240 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.988+0000 7fc380716700 1 -- 172.21.15.32:0/4270786332 >> 172.21.15.32:0/4270786332 conn(0x7fc3780714c0 msgr2=0x7fc3780718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.988+0000 7fc380716700 1 -- 172.21.15.32:0/4270786332 shutdown_connections 2024-04-06T14:29:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.988+0000 7fc380716700 1 -- 172.21.15.32:0/4270786332 wait complete. 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.989+0000 7fc380716700 1 Processor -- start 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.989+0000 7fc380716700 1 -- start start 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.989+0000 7fc380716700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780fb4a0 0x7fc378113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.989+0000 7fc380716700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc378102e70 0x7fc378113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.989+0000 7fc380716700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378114600 con 0x7fc3780fb4a0 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.989+0000 7fc380716700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37810d9f0 con 0x7fc378102e70 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc37e4b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780fb4a0 0x7fc378113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc37e4b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780fb4a0 0x7fc378113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54890/0 (socket says 172.21.15.32:54890) 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc37e4b2700 1 -- 172.21.15.32:0/2994843881 learned_addr learned my addr 172.21.15.32:0/2994843881 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc37e4b2700 1 -- 172.21.15.32:0/2994843881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc378102e70 msgr2=0x7fc378113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc37e4b2700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc378102e70 0x7fc378113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc37e4b2700 1 -- 172.21.15.32:0/2994843881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374005040 con 0x7fc3780fb4a0 2024-04-06T14:29:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc37e4b2700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780fb4a0 0x7fc378113a70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc374009180 tx=0x7fc374009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc36affd700 1 -- 172.21.15.32:0/2994843881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc37400e040 con 0x7fc3780fb4a0 2024-04-06T14:29:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc36affd700 1 -- 172.21.15.32:0/2994843881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc374004290 con 0x7fc3780fb4a0 2024-04-06T14:29:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37810dc70 con 0x7fc3780fb4a0 2024-04-06T14:29:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc36affd700 1 -- 172.21.15.32:0/2994843881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc37401c3f0 con 0x7fc3780fb4a0 2024-04-06T14:29:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.990+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37810e1b0 con 0x7fc3780fb4a0 2024-04-06T14:29:18.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.991+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc378100110 con 0x7fc3780fb4a0 2024-04-06T14:29:18.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.992+0000 7fc36affd700 1 -- 172.21.15.32:0/2994843881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc374013790 con 0x7fc3780fb4a0 2024-04-06T14:29:18.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.992+0000 7fc36affd700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc364071030 0x7fc3640734e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:18.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.992+0000 7fc36affd700 1 -- 172.21.15.32:0/2994843881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc37408d300 con 0x7fc3780fb4a0 2024-04-06T14:29:18.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.992+0000 7fc37dcb1700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc364071030 0x7fc3640734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:18.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.993+0000 7fc37dcb1700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc364071030 0x7fc3640734e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc37810ebb0 tx=0x7fc36c007cc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:18.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:18.997+0000 7fc36affd700 1 -- 172.21.15.32:0/2994843881 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc37405add0 con 0x7fc3780fb4a0 2024-04-06T14:29:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.222+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc37810f130 con 0x7fc364071030 2024-04-06T14:29:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.223+0000 7fc36affd700 1 -- 172.21.15.32:0/2994843881 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc37810f130 con 0x7fc364071030 2024-04-06T14:29:19.222 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:29:19.222 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:29:19.222 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:29:19.222 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:29:19.222 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:29:19.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:29:19.223 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:29:19.223 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:29:19.223 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:29:19.223 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:29:19.223 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:29:19.223 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:29:19.223 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:29:19.223 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:29:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc364071030 msgr2=0x7fc3640734e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc364071030 0x7fc3640734e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc37810ebb0 tx=0x7fc36c007cc0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780fb4a0 msgr2=0x7fc378113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780fb4a0 0x7fc378113a70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc374009180 tx=0x7fc374009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 shutdown_connections 2024-04-06T14:29:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc364071030 0x7fc3640734e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780fb4a0 0x7fc378113a70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 --2- 172.21.15.32:0/2994843881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc378102e70 0x7fc378113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 >> 172.21.15.32:0/2994843881 conn(0x7fc3780714c0 msgr2=0x7fc3780fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 shutdown_connections 2024-04-06T14:29:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.225+0000 7fc380716700 1 -- 172.21.15.32:0/2994843881 wait complete. 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 -- 172.21.15.32:0/1466952107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2134102e70 msgr2=0x7f2134103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 --2- 172.21.15.32:0/1466952107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2134102e70 0x7f2134103240 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2124004940 tx=0x7f2124009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 -- 172.21.15.32:0/1466952107 shutdown_connections 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 --2- 172.21.15.32:0/1466952107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21340fb4a0 0x7f21340fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 --2- 172.21.15.32:0/1466952107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2134102e70 0x7f2134103240 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 -- 172.21.15.32:0/1466952107 >> 172.21.15.32:0/1466952107 conn(0x7f21340714c0 msgr2=0x7f21340718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 -- 172.21.15.32:0/1466952107 shutdown_connections 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 -- 172.21.15.32:0/1466952107 wait complete. 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 Processor -- start 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.329+0000 7f2138ef9700 1 -- start start 2024-04-06T14:29:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2138ef9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21340fb4a0 0x7f2134115d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2138ef9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2134102e70 0x7f21341162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2138ef9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2134116910 con 0x7f21340fb4a0 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2138ef9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213410fce0 con 0x7f2134102e70 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f213259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21340fb4a0 0x7f2134115d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f213259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21340fb4a0 0x7f2134115d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54898/0 (socket says 172.21.15.32:54898) 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f213259c700 1 -- 172.21.15.32:0/2436703519 learned_addr learned my addr 172.21.15.32:0/2436703519 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2131d9b700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2134102e70 0x7f21341162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f213259c700 1 -- 172.21.15.32:0/2436703519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2134102e70 msgr2=0x7f21341162c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f213259c700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2134102e70 0x7f21341162c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f213259c700 1 -- 172.21.15.32:0/2436703519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2124005040 con 0x7f21340fb4a0 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f213259c700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21340fb4a0 0x7f2134115d80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f212400b3b0 tx=0x7f2124009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2122ffd700 1 -- 172.21.15.32:0/2436703519 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f212400e040 con 0x7f21340fb4a0 2024-04-06T14:29:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213410ff60 con 0x7f21340fb4a0 2024-04-06T14:29:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2122ffd700 1 -- 172.21.15.32:0/2436703519 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2124013420 con 0x7f21340fb4a0 2024-04-06T14:29:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2122ffd700 1 -- 172.21.15.32:0/2436703519 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f212401c610 con 0x7f21340fb4a0 2024-04-06T14:29:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.330+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21341104a0 con 0x7f21340fb4a0 2024-04-06T14:29:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.331+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2134107330 con 0x7f21340fb4a0 2024-04-06T14:29:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.332+0000 7f2122ffd700 1 -- 172.21.15.32:0/2436703519 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2124013a00 con 0x7f21340fb4a0 2024-04-06T14:29:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.332+0000 7f2122ffd700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f211c06cc20 0x7f211c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.332+0000 7f2122ffd700 1 -- 172.21.15.32:0/2436703519 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f212408d240 con 0x7f21340fb4a0 2024-04-06T14:29:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.332+0000 7f2131d9b700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f211c06cc20 0x7f211c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.333+0000 7f2131d9b700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f211c06cc20 0x7f211c06f0d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2134111620 tx=0x7f212800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:19.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.337+0000 7f2122ffd700 1 -- 172.21.15.32:0/2436703519 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f212405ac90 con 0x7f21340fb4a0 2024-04-06T14:29:19.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.626+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f213404bed0 con 0x7f21340fb4a0 2024-04-06T14:29:19.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.626+0000 7f2122ffd700 1 -- 172.21.15.32:0/2436703519 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f212405a820 con 0x7f21340fb4a0 2024-04-06T14:29:19.626 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:29:19.626 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:29:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:29:19.626 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:29:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:29:19.627 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:29:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:29:19.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f211c06cc20 msgr2=0x7f211c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f211c06cc20 0x7f211c06f0d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2134111620 tx=0x7f212800b040 comp rx=0 tx=0).stop 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21340fb4a0 msgr2=0x7f2134115d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21340fb4a0 0x7f2134115d80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f212400b3b0 tx=0x7f2124009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 shutdown_connections 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f211c06cc20 0x7f211c06f0d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21340fb4a0 0x7f2134115d80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 --2- 172.21.15.32:0/2436703519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2134102e70 0x7f21341162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 >> 172.21.15.32:0/2436703519 conn(0x7f21340714c0 msgr2=0x7f21340fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 shutdown_connections 2024-04-06T14:29:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:19.628+0000 7f2138ef9700 1 -- 172.21.15.32:0/2436703519 wait complete. 2024-04-06T14:29:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:19 smithi086 ceph-mon[110975]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:19 smithi086 ceph-mon[110975]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:19 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1672554071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:29:19.629 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:19 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3657067566' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:29:19.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:19 smithi032 ceph-mon[147414]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:19.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:19 smithi032 ceph-mon[147414]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:19.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:19 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1672554071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:29:19.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:19 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3657067566' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:29:20.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:20 smithi086 ceph-mon[110975]: pgmap v275: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:20.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:20 smithi086 ceph-mon[110975]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:20.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:20 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2436703519' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:29:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:20 smithi032 ceph-mon[147414]: pgmap v275: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:20 smithi032 ceph-mon[147414]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:20 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2436703519' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:29:22.630 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:22 smithi086 ceph-mon[110975]: pgmap v276: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:22.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:22 smithi032 ceph-mon[147414]: pgmap v276: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:23.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:29:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:29:24.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:24 smithi086 ceph-mon[110975]: pgmap v277: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:24.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:24 smithi032 ceph-mon[147414]: pgmap v277: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:26.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:26 smithi086 ceph-mon[110975]: pgmap v278: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:26 smithi032 ceph-mon[147414]: pgmap v278: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:28.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:28 smithi086 ceph-mon[110975]: pgmap v279: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:28 smithi032 ceph-mon[147414]: pgmap v279: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:30.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:30 smithi086 ceph-mon[110975]: pgmap v280: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:30 smithi032 ceph-mon[147414]: pgmap v280: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:32.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:32 smithi086 ceph-mon[110975]: pgmap v281: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:32.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:32 smithi032 ceph-mon[147414]: pgmap v281: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:34.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:34 smithi086 ceph-mon[110975]: pgmap v282: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:34 smithi032 ceph-mon[147414]: pgmap v282: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:36.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:36 smithi086 ceph-mon[110975]: pgmap v283: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:36.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:36 smithi032 ceph-mon[147414]: pgmap v283: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:38.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:38 smithi086 ceph-mon[110975]: pgmap v284: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:38.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:29:38.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:38 smithi032 ceph-mon[147414]: pgmap v284: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:38.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:29:40.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:40 smithi086 ceph-mon[110975]: pgmap v285: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:40.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:40 smithi032 ceph-mon[147414]: pgmap v285: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:42.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:42 smithi086 ceph-mon[110975]: pgmap v286: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:42 smithi032 ceph-mon[147414]: pgmap v286: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:44.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:44 smithi086 ceph-mon[110975]: pgmap v287: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:44 smithi032 ceph-mon[147414]: pgmap v287: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:46.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:46 smithi086 ceph-mon[110975]: pgmap v288: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:46 smithi032 ceph-mon[147414]: pgmap v288: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:48.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:48 smithi086 ceph-mon[110975]: pgmap v289: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:48.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:48 smithi032 ceph-mon[147414]: pgmap v289: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.746+0000 7f83b4c91700 1 -- 172.21.15.32:0/3509157132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b00fb4d0 msgr2=0x7f83b00fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.746+0000 7f83b4c91700 1 --2- 172.21.15.32:0/3509157132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b00fb4d0 0x7f83b00fb940 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f83a4004940 tx=0x7f83a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.746+0000 7f83b4c91700 1 -- 172.21.15.32:0/3509157132 shutdown_connections 2024-04-06T14:29:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.746+0000 7f83b4c91700 1 --2- 172.21.15.32:0/3509157132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b00fb4d0 0x7f83b00fb940 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.746+0000 7f83b4c91700 1 --2- 172.21.15.32:0/3509157132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b0102f50 0x7f83b0103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.746+0000 7f83b4c91700 1 -- 172.21.15.32:0/3509157132 >> 172.21.15.32:0/3509157132 conn(0x7f83b00714c0 msgr2=0x7f83b00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.746+0000 7f83b4c91700 1 -- 172.21.15.32:0/3509157132 shutdown_connections 2024-04-06T14:29:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.746+0000 7f83b4c91700 1 -- 172.21.15.32:0/3509157132 wait complete. 2024-04-06T14:29:49.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83b4c91700 1 Processor -- start 2024-04-06T14:29:49.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83b4c91700 1 -- start start 2024-04-06T14:29:49.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83b4c91700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00fb4d0 0x7f83b0113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:49.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83b4c91700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b0102f50 0x7f83b0113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83b4c91700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b01144b0 con 0x7f83b0102f50 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83b4c91700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b010dab0 con 0x7f83b00fb4d0 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83add9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b0102f50 0x7f83b0113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83add9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b0102f50 0x7f83b0113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52582/0 (socket says 172.21.15.32:52582) 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83add9b700 1 -- 172.21.15.32:0/1818162963 learned_addr learned my addr 172.21.15.32:0/1818162963 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83ae59c700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00fb4d0 0x7f83b0113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83add9b700 1 -- 172.21.15.32:0/1818162963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00fb4d0 msgr2=0x7f83b0113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83add9b700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00fb4d0 0x7f83b0113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.747+0000 7f83add9b700 1 -- 172.21.15.32:0/1818162963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4005040 con 0x7f83b0102f50 2024-04-06T14:29:49.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.748+0000 7f83add9b700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b0102f50 0x7f83b0113e60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009390 tx=0x7f83a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:49.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.748+0000 7f839effd700 1 -- 172.21.15.32:0/1818162963 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a400e040 con 0x7f83b0102f50 2024-04-06T14:29:49.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.748+0000 7f839effd700 1 -- 172.21.15.32:0/1818162963 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83a4013420 con 0x7f83b0102f50 2024-04-06T14:29:49.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.748+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b010dd30 con 0x7f83b0102f50 2024-04-06T14:29:49.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.748+0000 7f839effd700 1 -- 172.21.15.32:0/1818162963 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a401c630 con 0x7f83b0102f50 2024-04-06T14:29:49.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.748+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b010e270 con 0x7f83b0102f50 2024-04-06T14:29:49.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.749+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b0105330 con 0x7f83b0102f50 2024-04-06T14:29:49.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.749+0000 7f839effd700 1 -- 172.21.15.32:0/1818162963 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f83a40139e0 con 0x7f83b0102f50 2024-04-06T14:29:49.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.750+0000 7f839effd700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f839806cc20 0x7f839806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:49.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.750+0000 7f83ae59c700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f839806cc20 0x7f839806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:49.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.750+0000 7f839effd700 1 -- 172.21.15.32:0/1818162963 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f83a408d360 con 0x7f83b0102f50 2024-04-06T14:29:49.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.750+0000 7f83ae59c700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f839806cc20 0x7f839806f0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f83b0065bf0 tx=0x7f83a0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:49.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.755+0000 7f839effd700 1 -- 172.21.15.32:0/1818162963 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f83a405ae30 con 0x7f83b0102f50 2024-04-06T14:29:49.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.976+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f83b010ef70 con 0x7f839806cc20 2024-04-06T14:29:49.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.978+0000 7f839effd700 1 -- 172.21.15.32:0/1818162963 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f83b010ef70 con 0x7f839806cc20 2024-04-06T14:29:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f839806cc20 msgr2=0x7f839806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f839806cc20 0x7f839806f0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f83b0065bf0 tx=0x7f83a0009040 comp rx=0 tx=0).stop 2024-04-06T14:29:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b0102f50 msgr2=0x7f83b0113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b0102f50 0x7f83b0113e60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009390 tx=0x7f83a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 shutdown_connections 2024-04-06T14:29:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f839806cc20 0x7f839806f0d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00fb4d0 0x7f83b0113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 --2- 172.21.15.32:0/1818162963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b0102f50 0x7f83b0113e60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 >> 172.21.15.32:0/1818162963 conn(0x7f83b00714c0 msgr2=0x7f83b00fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 shutdown_connections 2024-04-06T14:29:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:49.980+0000 7f83b4c91700 1 -- 172.21.15.32:0/1818162963 wait complete. 2024-04-06T14:29:49.990 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:29:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 -- 172.21.15.32:0/741117371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64102710 msgr2=0x7fde640fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 --2- 172.21.15.32:0/741117371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64102710 0x7fde640fbd80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fde54004940 tx=0x7fde54009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 -- 172.21.15.32:0/741117371 shutdown_connections 2024-04-06T14:29:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 --2- 172.21.15.32:0/741117371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64102710 0x7fde640fbd80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 --2- 172.21.15.32:0/741117371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde64101d70 0x7fde64102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 -- 172.21.15.32:0/741117371 >> 172.21.15.32:0/741117371 conn(0x7fde64072bc0 msgr2=0x7fde64072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 -- 172.21.15.32:0/741117371 shutdown_connections 2024-04-06T14:29:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 -- 172.21.15.32:0/741117371 wait complete. 2024-04-06T14:29:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 Processor -- start 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.085+0000 7fde6b34a700 1 -- start start 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde6b34a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64101d70 0x7fde6410e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde6b34a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde64102710 0x7fde6410eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde6b34a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6410f530 con 0x7fde64101d70 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde6b34a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6410f670 con 0x7fde64102710 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde690e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64101d70 0x7fde6410e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde690e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64101d70 0x7fde6410e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52590/0 (socket says 172.21.15.32:52590) 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde690e6700 1 -- 172.21.15.32:0/3612075042 learned_addr learned my addr 172.21.15.32:0/3612075042 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde688e5700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde64102710 0x7fde6410eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde690e6700 1 -- 172.21.15.32:0/3612075042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde64102710 msgr2=0x7fde6410eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde690e6700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde64102710 0x7fde6410eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde690e6700 1 -- 172.21.15.32:0/3612075042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde54005040 con 0x7fde64101d70 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde690e6700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64101d70 0x7fde6410e9a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fde6000eb40 tx=0x7fde6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde59ffb700 1 -- 172.21.15.32:0/3612075042 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde60007cc0 con 0x7fde64101d70 2024-04-06T14:29:50.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.086+0000 7fde59ffb700 1 -- 172.21.15.32:0/3612075042 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde6000f040 con 0x7fde64101d70 2024-04-06T14:29:50.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.087+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde64117a10 con 0x7fde64101d70 2024-04-06T14:29:50.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.087+0000 7fde59ffb700 1 -- 172.21.15.32:0/3612075042 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6000c990 con 0x7fde64101d70 2024-04-06T14:29:50.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.087+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde64117ef0 con 0x7fde64101d70 2024-04-06T14:29:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.088+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde640ff080 con 0x7fde64101d70 2024-04-06T14:29:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.088+0000 7fde59ffb700 1 -- 172.21.15.32:0/3612075042 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fde6000cc00 con 0x7fde64101d70 2024-04-06T14:29:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.088+0000 7fde59ffb700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde5006cc20 0x7fde5006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.089+0000 7fde59ffb700 1 -- 172.21.15.32:0/3612075042 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fde6008bc70 con 0x7fde64101d70 2024-04-06T14:29:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.089+0000 7fde688e5700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde5006cc20 0x7fde5006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.089+0000 7fde688e5700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde5006cc20 0x7fde5006f0d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fde5400b0d0 tx=0x7fde54009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:50.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.093+0000 7fde59ffb700 1 -- 172.21.15.32:0/3612075042 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fde60055ce0 con 0x7fde64101d70 2024-04-06T14:29:50.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.315+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde6404bed0 con 0x7fde5006cc20 2024-04-06T14:29:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.316+0000 7fde59ffb700 1 -- 172.21.15.32:0/3612075042 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fde6404bed0 con 0x7fde5006cc20 2024-04-06T14:29:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde5006cc20 msgr2=0x7fde5006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde5006cc20 0x7fde5006f0d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fde5400b0d0 tx=0x7fde54009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64101d70 msgr2=0x7fde6410e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64101d70 0x7fde6410e9a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fde6000eb40 tx=0x7fde6000b040 comp rx=0 tx=0).stop 2024-04-06T14:29:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 shutdown_connections 2024-04-06T14:29:50.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde5006cc20 0x7fde5006f0d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde64101d70 0x7fde6410e9a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 --2- 172.21.15.32:0/3612075042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde64102710 0x7fde6410eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 >> 172.21.15.32:0/3612075042 conn(0x7fde64072bc0 msgr2=0x7fde640faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:50.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 shutdown_connections 2024-04-06T14:29:50.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.318+0000 7fde6b34a700 1 -- 172.21.15.32:0/3612075042 wait complete. 2024-04-06T14:29:50.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.421+0000 7f14c12c2700 1 -- 172.21.15.32:0/226284896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 msgr2=0x7f14bc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.421+0000 7f14c12c2700 1 --2- 172.21.15.32:0/226284896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 0x7f14bc0fbd80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f14ac004940 tx=0x7f14ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:50.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.421+0000 7f14c12c2700 1 -- 172.21.15.32:0/226284896 shutdown_connections 2024-04-06T14:29:50.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.421+0000 7f14c12c2700 1 --2- 172.21.15.32:0/226284896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 0x7f14bc0fbd80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.421+0000 7f14c12c2700 1 --2- 172.21.15.32:0/226284896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14bc101d70 0x7f14bc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.421+0000 7f14c12c2700 1 -- 172.21.15.32:0/226284896 >> 172.21.15.32:0/226284896 conn(0x7f14bc072bc0 msgr2=0x7f14bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:50.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.421+0000 7f14c12c2700 1 -- 172.21.15.32:0/226284896 shutdown_connections 2024-04-06T14:29:50.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.421+0000 7f14c12c2700 1 -- 172.21.15.32:0/226284896 wait complete. 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14c12c2700 1 Processor -- start 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14c12c2700 1 -- start start 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14c12c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14bc101d70 0x7f14bc102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14c12c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 0x7f14bc103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14c12c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc1036a0 con 0x7f14bc102710 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14c12c2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc1037e0 con 0x7f14bc101d70 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14ba7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 0x7f14bc103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14ba7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 0x7f14bc103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52610/0 (socket says 172.21.15.32:52610) 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14ba7fc700 1 -- 172.21.15.32:0/941222846 learned_addr learned my addr 172.21.15.32:0/941222846 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14baffd700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14bc101d70 0x7f14bc102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14ba7fc700 1 -- 172.21.15.32:0/941222846 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14bc101d70 msgr2=0x7f14bc102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14ba7fc700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14bc101d70 0x7f14bc102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.422+0000 7f14ba7fc700 1 -- 172.21.15.32:0/941222846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac005040 con 0x7f14bc102710 2024-04-06T14:29:50.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.423+0000 7f14ba7fc700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 0x7f14bc103110 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009390 tx=0x7f14ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:50.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.423+0000 7f14a37fe700 1 -- 172.21.15.32:0/941222846 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14ac00e040 con 0x7f14bc102710 2024-04-06T14:29:50.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.423+0000 7f14a37fe700 1 -- 172.21.15.32:0/941222846 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14ac013420 con 0x7f14bc102710 2024-04-06T14:29:50.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.423+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14bc103a30 con 0x7f14bc102710 2024-04-06T14:29:50.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.423+0000 7f14a37fe700 1 -- 172.21.15.32:0/941222846 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14ac01c630 con 0x7f14bc102710 2024-04-06T14:29:50.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.423+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc11a130 con 0x7f14bc102710 2024-04-06T14:29:50.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.424+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc0fd4c0 con 0x7f14bc102710 2024-04-06T14:29:50.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.424+0000 7f14a37fe700 1 -- 172.21.15.32:0/941222846 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f14ac013a00 con 0x7f14bc102710 2024-04-06T14:29:50.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.425+0000 7f14a37fe700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14a406cc30 0x7f14a406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:50.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.425+0000 7f14a37fe700 1 -- 172.21.15.32:0/941222846 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f14ac08d140 con 0x7f14bc102710 2024-04-06T14:29:50.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.425+0000 7f14baffd700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14a406cc30 0x7f14a406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:50.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.425+0000 7f14baffd700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14a406cc30 0x7f14a406f0e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009fd0 tx=0x7f14b000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:50.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.429+0000 7f14a37fe700 1 -- 172.21.15.32:0/941222846 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f14ac057180 con 0x7f14bc102710 2024-04-06T14:29:50.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:50 smithi086 ceph-mon[110975]: pgmap v290: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:50.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.641+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f14bc108db0 con 0x7f14a406cc30 2024-04-06T14:29:50.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.651+0000 7f14a37fe700 1 -- 172.21.15.32:0/941222846 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f14b000f040 con 0x7f14a406cc30 2024-04-06T14:29:50.651 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (10m) 3m ago 21m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 3m ago 21m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (7m) 3m ago 20m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (9m) 3m ago 21m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (3m) 3m ago 15m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (3m) 3m ago 15m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (3m) 3m ago 15m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (15m) 3m ago 15m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (12m) 3m ago 22m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (11m) 3m ago 20m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 3m ago 23m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (8m) 3m ago 20m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (10m) 3m ago 21m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:29:50.652 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (10m) 3m ago 20m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:29:50.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 3m ago 19m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:29:50.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 3m ago 19m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:29:50.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 3m ago 18m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:29:50.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (5m) 3m ago 18m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:29:50.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (4m) 3m ago 17m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:29:50.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (4m) 3m ago 17m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:29:50.653 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (10m) 3m ago 21m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14a406cc30 msgr2=0x7f14a406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14a406cc30 0x7f14a406f0e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009fd0 tx=0x7f14b000b040 comp rx=0 tx=0).stop 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 msgr2=0x7f14bc103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 0x7f14bc103110 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009390 tx=0x7f14ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 shutdown_connections 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14a406cc30 0x7f14a406f0e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14bc101d70 0x7f14bc102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 --2- 172.21.15.32:0/941222846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14bc102710 0x7f14bc103110 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 >> 172.21.15.32:0/941222846 conn(0x7f14bc072bc0 msgr2=0x7f14bc065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 shutdown_connections 2024-04-06T14:29:50.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.653+0000 7f14c12c2700 1 -- 172.21.15.32:0/941222846 wait complete. 2024-04-06T14:29:50.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:50 smithi032 ceph-mon[147414]: pgmap v290: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:50.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.755+0000 7f185f5aa700 1 -- 172.21.15.32:0/3948942913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 msgr2=0x7f18580fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.755+0000 7f185f5aa700 1 --2- 172.21.15.32:0/3948942913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 0x7f18580fe260 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1854004940 tx=0x7f1854009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.755+0000 7f185f5aa700 1 -- 172.21.15.32:0/3948942913 shutdown_connections 2024-04-06T14:29:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.755+0000 7f185f5aa700 1 --2- 172.21.15.32:0/3948942913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18580fe830 0x7f1858106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.755+0000 7f185f5aa700 1 --2- 172.21.15.32:0/3948942913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 0x7f18580fe260 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.755+0000 7f185f5aa700 1 -- 172.21.15.32:0/3948942913 >> 172.21.15.32:0/3948942913 conn(0x7f1858072bc0 msgr2=0x7f1858072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.755+0000 7f185f5aa700 1 -- 172.21.15.32:0/3948942913 shutdown_connections 2024-04-06T14:29:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.755+0000 7f185f5aa700 1 -- 172.21.15.32:0/3948942913 wait complete. 2024-04-06T14:29:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185f5aa700 1 Processor -- start 2024-04-06T14:29:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185f5aa700 1 -- start start 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185f5aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 0x7f1858113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185f5aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18580fe830 0x7f1858114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185f5aa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1858114680 con 0x7f18580fde90 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185f5aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185810da70 con 0x7f18580fe830 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185d346700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 0x7f1858113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185d346700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 0x7f1858113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52626/0 (socket says 172.21.15.32:52626) 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185d346700 1 -- 172.21.15.32:0/3855805866 learned_addr learned my addr 172.21.15.32:0/3855805866 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.756+0000 7f185cb45700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18580fe830 0x7f1858114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.757+0000 7f185d346700 1 -- 172.21.15.32:0/3855805866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18580fe830 msgr2=0x7f1858114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.757+0000 7f185d346700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18580fe830 0x7f1858114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.757+0000 7f185d346700 1 -- 172.21.15.32:0/3855805866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1854005040 con 0x7f18580fde90 2024-04-06T14:29:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.757+0000 7f185d346700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 0x7f1858113af0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1854009360 tx=0x7f1854009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.757+0000 7f184dffb700 1 -- 172.21.15.32:0/3855805866 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f185400e040 con 0x7f18580fde90 2024-04-06T14:29:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.757+0000 7f184dffb700 1 -- 172.21.15.32:0/3855805866 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f185400bed0 con 0x7f18580fde90 2024-04-06T14:29:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.757+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185810dcf0 con 0x7f18580fde90 2024-04-06T14:29:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.757+0000 7f184dffb700 1 -- 172.21.15.32:0/3855805866 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f185401c3f0 con 0x7f18580fde90 2024-04-06T14:29:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.757+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185810e230 con 0x7f18580fde90 2024-04-06T14:29:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.758+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18581019b0 con 0x7f18580fde90 2024-04-06T14:29:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.759+0000 7f184dffb700 1 -- 172.21.15.32:0/3855805866 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1854013790 con 0x7f18580fde90 2024-04-06T14:29:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.759+0000 7f184dffb700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f184406cc20 0x7f184406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.759+0000 7f184dffb700 1 -- 172.21.15.32:0/3855805866 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f185408d320 con 0x7f18580fde90 2024-04-06T14:29:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.759+0000 7f185cb45700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f184406cc20 0x7f184406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.760+0000 7f185cb45700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f184406cc20 0x7f184406f0d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f185810ec30 tx=0x7f184800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:50.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:50.763+0000 7f184dffb700 1 -- 172.21.15.32:0/3855805866 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f185405adf0 con 0x7f18580fde90 2024-04-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.052+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f185810f140 con 0x7f18580fde90 2024-04-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.052+0000 7f184dffb700 1 -- 172.21.15.32:0/3855805866 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f1854013420 con 0x7f18580fde90 2024-04-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:29:51.054 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:29:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.054+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f184406cc20 msgr2=0x7f184406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.054+0000 7f185f5aa700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f184406cc20 0x7f184406f0d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f185810ec30 tx=0x7f184800b040 comp rx=0 tx=0).stop 2024-04-06T14:29:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.054+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 msgr2=0x7f1858113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.054+0000 7f185f5aa700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 0x7f1858113af0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1854009360 tx=0x7f1854009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.054+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 shutdown_connections 2024-04-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.054+0000 7f185f5aa700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f184406cc20 0x7f184406f0d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.054+0000 7f185f5aa700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580fde90 0x7f1858113af0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.054+0000 7f185f5aa700 1 --2- 172.21.15.32:0/3855805866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18580fe830 0x7f1858114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.054+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 >> 172.21.15.32:0/3855805866 conn(0x7f1858072bc0 msgr2=0x7f18580ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.055+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 shutdown_connections 2024-04-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.055+0000 7f185f5aa700 1 -- 172.21.15.32:0/3855805866 wait complete. 2024-04-06T14:29:51.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 -- 172.21.15.32:0/906740061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fe7a0 msgr2=0x7f86f4106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 --2- 172.21.15.32:0/906740061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fe7a0 0x7f86f4106e70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f86e4004940 tx=0x7f86e4009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:51.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 -- 172.21.15.32:0/906740061 shutdown_connections 2024-04-06T14:29:51.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 --2- 172.21.15.32:0/906740061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fe7a0 0x7f86f4106e70 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 --2- 172.21.15.32:0/906740061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86f40fde00 0x7f86f40fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 -- 172.21.15.32:0/906740061 >> 172.21.15.32:0/906740061 conn(0x7f86f40f7880 msgr2=0x7f86f40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:51.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 -- 172.21.15.32:0/906740061 shutdown_connections 2024-04-06T14:29:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 -- 172.21.15.32:0/906740061 wait complete. 2024-04-06T14:29:51.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 Processor -- start 2024-04-06T14:29:51.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.158+0000 7f86fc5c7700 1 -- start start 2024-04-06T14:29:51.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fc5c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fde00 0x7f86f40fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:51.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fc5c7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86f40fe7a0 0x7f86f40fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:51.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fc5c7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40fc790 con 0x7f86f40fde00 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fc5c7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40fc8d0 con 0x7f86f40fe7a0 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fa363700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fde00 0x7f86f40fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fa363700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fde00 0x7f86f40fbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52654/0 (socket says 172.21.15.32:52654) 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fa363700 1 -- 172.21.15.32:0/1913774036 learned_addr learned my addr 172.21.15.32:0/1913774036 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fa363700 1 -- 172.21.15.32:0/1913774036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86f40fe7a0 msgr2=0x7f86f40fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86f9b62700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86f40fe7a0 0x7f86f40fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fa363700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86f40fe7a0 0x7f86f40fc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fa363700 1 -- 172.21.15.32:0/1913774036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e4005040 con 0x7f86f40fde00 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86f9b62700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86f40fe7a0 0x7f86f40fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fa363700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fde00 0x7f86f40fbcc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c830 tx=0x7f86f0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86eaffd700 1 -- 172.21.15.32:0/1913774036 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86f000a760 con 0x7f86f40fde00 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86eaffd700 1 -- 172.21.15.32:0/1913774036 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86f000ad70 con 0x7f86f40fde00 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f411a090 con 0x7f86f40fde00 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86eaffd700 1 -- 172.21.15.32:0/1913774036 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86f00113a0 con 0x7f86f40fde00 2024-04-06T14:29:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.159+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f411a490 con 0x7f86f40fde00 2024-04-06T14:29:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.160+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f40666f0 con 0x7f86f40fde00 2024-04-06T14:29:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.161+0000 7f86eaffd700 1 -- 172.21.15.32:0/1913774036 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f86f0012400 con 0x7f86f40fde00 2024-04-06T14:29:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.161+0000 7f86eaffd700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86e006cc30 0x7f86e006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.161+0000 7f86eaffd700 1 -- 172.21.15.32:0/1913774036 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f86f008bff0 con 0x7f86f40fde00 2024-04-06T14:29:51.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.161+0000 7f86f9b62700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86e006cc30 0x7f86e006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:51.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.162+0000 7f86f9b62700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86e006cc30 0x7f86e006f0e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f86e400b3b0 tx=0x7f86e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:51.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.166+0000 7f86eaffd700 1 -- 172.21.15.32:0/1913774036 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f86f00560f0 con 0x7f86f40fde00 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.409+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f86f405f5f0 con 0x7f86f40fde00 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.410+0000 7f86eaffd700 1 -- 172.21.15.32:0/1913774036 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f86f0059710 con 0x7f86f40fde00 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:29:51.410 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:29:51.411 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:29:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:29:51.413 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:29:51.413 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:29:51.413 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:29:51.413 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:29:51.413 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86e006cc30 msgr2=0x7f86e006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86e006cc30 0x7f86e006f0e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f86e400b3b0 tx=0x7f86e4009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fde00 msgr2=0x7f86f40fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fde00 0x7f86f40fbcc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c830 tx=0x7f86f0004470 comp rx=0 tx=0).stop 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 shutdown_connections 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86e006cc30 0x7f86e006f0e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86f40fde00 0x7f86f40fbcc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 --2- 172.21.15.32:0/1913774036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86f40fe7a0 0x7f86f40fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 >> 172.21.15.32:0/1913774036 conn(0x7f86f40f7880 msgr2=0x7f86f40f8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 shutdown_connections 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.412+0000 7f86fc5c7700 1 -- 172.21.15.32:0/1913774036 wait complete. 2024-04-06T14:29:51.414 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:29:51.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.516+0000 7fb419bab700 1 -- 172.21.15.32:0/760522431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414102710 msgr2=0x7fb4140fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.516+0000 7fb419bab700 1 --2- 172.21.15.32:0/760522431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414102710 0x7fb4140fbd80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb404004940 tx=0x7fb404009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:51.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.516+0000 7fb419bab700 1 -- 172.21.15.32:0/760522431 shutdown_connections 2024-04-06T14:29:51.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.516+0000 7fb419bab700 1 --2- 172.21.15.32:0/760522431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414102710 0x7fb4140fbd80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.516+0000 7fb419bab700 1 --2- 172.21.15.32:0/760522431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb414101d70 0x7fb414102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.516+0000 7fb419bab700 1 -- 172.21.15.32:0/760522431 >> 172.21.15.32:0/760522431 conn(0x7fb414072bc0 msgr2=0x7fb414072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:51.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.516+0000 7fb419bab700 1 -- 172.21.15.32:0/760522431 shutdown_connections 2024-04-06T14:29:51.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.516+0000 7fb419bab700 1 -- 172.21.15.32:0/760522431 wait complete. 2024-04-06T14:29:51.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.516+0000 7fb419bab700 1 Processor -- start 2024-04-06T14:29:51.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb419bab700 1 -- start start 2024-04-06T14:29:51.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb419bab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414101d70 0x7fb41410ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:51.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb419bab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb414102710 0x7fb41410f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:51.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb419bab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41410f680 con 0x7fb414101d70 2024-04-06T14:29:51.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb419bab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41410f7c0 con 0x7fb414102710 2024-04-06T14:29:51.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb4137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414101d70 0x7fb41410ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:51.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb4137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414101d70 0x7fb41410ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52674/0 (socket says 172.21.15.32:52674) 2024-04-06T14:29:51.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb4137fe700 1 -- 172.21.15.32:0/3147140424 learned_addr learned my addr 172.21.15.32:0/3147140424 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb412ffd700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb414102710 0x7fb41410f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb4137fe700 1 -- 172.21.15.32:0/3147140424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb414102710 msgr2=0x7fb41410f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb4137fe700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb414102710 0x7fb41410f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb4137fe700 1 -- 172.21.15.32:0/3147140424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb404005040 con 0x7fb414101d70 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb4137fe700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414101d70 0x7fb41410ec00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb408002a50 tx=0x7fb4080046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.517+0000 7fb3fbfff700 1 -- 172.21.15.32:0/3147140424 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb408007cc0 con 0x7fb414101d70 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.518+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb414117ab0 con 0x7fb414101d70 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.518+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb414118040 con 0x7fb414101d70 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.518+0000 7fb3fbfff700 1 -- 172.21.15.32:0/3147140424 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb40800f040 con 0x7fb414101d70 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.518+0000 7fb3fbfff700 1 -- 172.21.15.32:0/3147140424 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb408009510 con 0x7fb414101d70 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.519+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4000052b0 con 0x7fb414101d70 2024-04-06T14:29:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.519+0000 7fb3fbfff700 1 -- 172.21.15.32:0/3147140424 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb408009670 con 0x7fb414101d70 2024-04-06T14:29:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.519+0000 7fb3fbfff700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3fc06cc20 0x7fb3fc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.519+0000 7fb3fbfff700 1 -- 172.21.15.32:0/3147140424 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb40808bdf0 con 0x7fb414101d70 2024-04-06T14:29:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.519+0000 7fb412ffd700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3fc06cc20 0x7fb3fc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.520+0000 7fb412ffd700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3fc06cc20 0x7fb3fc06f0d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb40400b3b0 tx=0x7fb404009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:51.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.524+0000 7fb3fbfff700 1 -- 172.21.15.32:0/3147140424 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb408055db0 con 0x7fb414101d70 2024-04-06T14:29:51.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:51 smithi086 ceph-mon[110975]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:51.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:51 smithi086 ceph-mon[110975]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:51.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:51 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3855805866' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:29:51.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:51 smithi032 ceph-mon[147414]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:51.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:51 smithi032 ceph-mon[147414]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:51.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:51 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3855805866' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:29:51.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.745+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb400000bc0 con 0x7fb3fc06cc20 2024-04-06T14:29:51.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.746+0000 7fb3fbfff700 1 -- 172.21.15.32:0/3147140424 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb400000bc0 con 0x7fb3fc06cc20 2024-04-06T14:29:51.746 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:29:51.746 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:29:51.746 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:29:51.747 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:29:51.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3fc06cc20 msgr2=0x7fb3fc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3fc06cc20 0x7fb3fc06f0d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb40400b3b0 tx=0x7fb404009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:51.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414101d70 msgr2=0x7fb41410ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414101d70 0x7fb41410ec00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb408002a50 tx=0x7fb4080046b0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 shutdown_connections 2024-04-06T14:29:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3fc06cc20 0x7fb3fc06f0d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb414101d70 0x7fb41410ec00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 --2- 172.21.15.32:0/3147140424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb414102710 0x7fb41410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 >> 172.21.15.32:0/3147140424 conn(0x7fb414072bc0 msgr2=0x7fb4140fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 shutdown_connections 2024-04-06T14:29:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.748+0000 7fb419bab700 1 -- 172.21.15.32:0/3147140424 wait complete. 2024-04-06T14:29:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.851+0000 7f6eb2b00700 1 -- 172.21.15.32:0/3844663733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac102e10 msgr2=0x7f6eac1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.851+0000 7f6eb2b00700 1 --2- 172.21.15.32:0/3844663733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac102e10 0x7f6eac1031e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0004940 tx=0x7f6ea0009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.851+0000 7f6eb2b00700 1 -- 172.21.15.32:0/3844663733 shutdown_connections 2024-04-06T14:29:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.851+0000 7f6eb2b00700 1 --2- 172.21.15.32:0/3844663733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6eac1037b0 0x7f6eac0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.851+0000 7f6eb2b00700 1 --2- 172.21.15.32:0/3844663733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac102e10 0x7f6eac1031e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.851+0000 7f6eb2b00700 1 -- 172.21.15.32:0/3844663733 >> 172.21.15.32:0/3844663733 conn(0x7f6eac0f89a0 msgr2=0x7f6eac0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.851+0000 7f6eb2b00700 1 -- 172.21.15.32:0/3844663733 shutdown_connections 2024-04-06T14:29:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.851+0000 7f6eb2b00700 1 -- 172.21.15.32:0/3844663733 wait complete. 2024-04-06T14:29:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eb2b00700 1 Processor -- start 2024-04-06T14:29:51.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eb2b00700 1 -- start start 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eb2b00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6eac102e10 0x7f6eac113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eb2b00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac1037b0 0x7f6eac113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eb2b00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac1146d0 con 0x7f6eac1037b0 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eb2b00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac10da30 con 0x7f6eac102e10 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eabfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac1037b0 0x7f6eac113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eabfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac1037b0 0x7f6eac113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52696/0 (socket says 172.21.15.32:52696) 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eabfff700 1 -- 172.21.15.32:0/2944080628 learned_addr learned my addr 172.21.15.32:0/2944080628 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eb089c700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6eac102e10 0x7f6eac113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.852+0000 7f6eabfff700 1 -- 172.21.15.32:0/2944080628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6eac102e10 msgr2=0x7f6eac113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:51.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.853+0000 7f6eabfff700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6eac102e10 0x7f6eac113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.853+0000 7f6eabfff700 1 -- 172.21.15.32:0/2944080628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea0005040 con 0x7f6eac1037b0 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.853+0000 7f6eb089c700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6eac102e10 0x7f6eac113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.853+0000 7f6eabfff700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac1037b0 0x7f6eac113ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800b790 tx=0x7f6e98009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.853+0000 7f6ea97fa700 1 -- 172.21.15.32:0/2944080628 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9800f6f0 con 0x7f6eac1037b0 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.853+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac10dd10 con 0x7f6eac1037b0 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.853+0000 7f6ea97fa700 1 -- 172.21.15.32:0/2944080628 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e9800fd00 con 0x7f6eac1037b0 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.853+0000 7f6ea97fa700 1 -- 172.21.15.32:0/2944080628 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e980103a0 con 0x7f6eac1037b0 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.853+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac10e2d0 con 0x7f6eac1037b0 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.854+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac0fe560 con 0x7f6eac1037b0 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.854+0000 7f6ea97fa700 1 -- 172.21.15.32:0/2944080628 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6e9800f850 con 0x7f6eac1037b0 2024-04-06T14:29:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.855+0000 7f6ea97fa700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6e9c06cc20 0x7f6e9c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:29:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.855+0000 7f6ea97fa700 1 -- 172.21.15.32:0/2944080628 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6e9808be40 con 0x7f6eac1037b0 2024-04-06T14:29:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.855+0000 7f6eb089c700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6e9c06cc20 0x7f6e9c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:29:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.855+0000 7f6eb089c700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6e9c06cc20 0x7f6e9c06f0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0004560 tx=0x7f6ea0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:29:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:51.859+0000 7f6ea97fa700 1 -- 172.21.15.32:0/2944080628 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6e98059910 con 0x7f6eac1037b0 2024-04-06T14:29:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.151+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6eac05f5f0 con 0x7f6eac1037b0 2024-04-06T14:29:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.151+0000 7f6ea97fa700 1 -- 172.21.15.32:0/2944080628 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f6e980594a0 con 0x7f6eac1037b0 2024-04-06T14:29:52.152 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:29:52.152 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:29:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:29:52.152 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:29:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:29:52.152 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:29:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:29:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6e9c06cc20 msgr2=0x7f6e9c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6e9c06cc20 0x7f6e9c06f0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0004560 tx=0x7f6ea0009e30 comp rx=0 tx=0).stop 2024-04-06T14:29:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac1037b0 msgr2=0x7f6eac113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:29:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac1037b0 0x7f6eac113ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800b790 tx=0x7f6e98009220 comp rx=0 tx=0).stop 2024-04-06T14:29:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 shutdown_connections 2024-04-06T14:29:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6e9c06cc20 0x7f6e9c06f0d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6eac102e10 0x7f6eac113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 --2- 172.21.15.32:0/2944080628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eac1037b0 0x7f6eac113ff0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:29:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 >> 172.21.15.32:0/2944080628 conn(0x7f6eac0f89a0 msgr2=0x7f6eac0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:29:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 shutdown_connections 2024-04-06T14:29:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:29:52.154+0000 7f6eb2b00700 1 -- 172.21.15.32:0/2944080628 wait complete. 2024-04-06T14:29:52.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:52 smithi086 ceph-mon[110975]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:52.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:52 smithi086 ceph-mon[110975]: pgmap v291: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:52.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:52 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1913774036' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:29:52.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:52 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2944080628' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:29:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:52 smithi032 ceph-mon[147414]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:52 smithi032 ceph-mon[147414]: pgmap v291: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:52 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1913774036' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:29:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:52 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2944080628' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:29:53.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:53 smithi086 ceph-mon[110975]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:53.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:29:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:53 smithi032 ceph-mon[147414]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:29:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:29:54.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:54 smithi086 ceph-mon[110975]: pgmap v292: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:54.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:54 smithi032 ceph-mon[147414]: pgmap v292: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:56.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:56 smithi086 ceph-mon[110975]: pgmap v293: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:56.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:56 smithi032 ceph-mon[147414]: pgmap v293: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:58.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:29:58 smithi086 ceph-mon[110975]: pgmap v294: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:29:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:29:58 smithi032 ceph-mon[147414]: pgmap v294: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:00.363 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:29:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:30:00.363 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:29:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:30:00.364 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:29:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-06T14:30:00.364 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:29:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:30:00.364 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:29:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:30:00.364 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:29:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:30:00.364 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:29:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-06T14:30:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:00 smithi086 ceph-mon[110975]: pgmap v295: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:00 smithi086 ceph-mon[110975]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:30:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:00 smithi086 ceph-mon[110975]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:30:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:00 smithi086 ceph-mon[110975]: fs cephfs is degraded 2024-04-06T14:30:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:00 smithi086 ceph-mon[110975]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:30:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:00 smithi086 ceph-mon[110975]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:30:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:00 smithi086 ceph-mon[110975]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:30:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:00 smithi086 ceph-mon[110975]: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:30:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-mon[147414]: pgmap v295: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-mon[147414]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:30:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-mon[147414]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:30:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-mon[147414]: fs cephfs is degraded 2024-04-06T14:30:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-mon[147414]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:30:00.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-mon[147414]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:30:00.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-mon[147414]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:30:00.731 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:00 smithi032 ceph-mon[147414]: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:30:02.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:02 smithi086 ceph-mon[110975]: pgmap v296: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:02.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:02 smithi032 ceph-mon[147414]: pgmap v296: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:04.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:04 smithi086 ceph-mon[110975]: pgmap v297: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:04.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:04 smithi032 ceph-mon[147414]: pgmap v297: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:06.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:06 smithi086 ceph-mon[110975]: pgmap v298: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:06 smithi032 ceph-mon[147414]: pgmap v298: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:08 smithi086 ceph-mon[110975]: pgmap v299: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:30:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:08 smithi032 ceph-mon[147414]: pgmap v299: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:08.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:30:10.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:10 smithi086 ceph-mon[110975]: pgmap v300: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:10.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:10 smithi032 ceph-mon[147414]: pgmap v300: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:12.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:12 smithi032 ceph-mon[147414]: pgmap v301: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:12.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:12 smithi086 ceph-mon[110975]: pgmap v301: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:14 smithi032 ceph-mon[147414]: pgmap v302: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:14.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:14 smithi086 ceph-mon[110975]: pgmap v302: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:16.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:16 smithi032 ceph-mon[147414]: pgmap v303: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:16.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:16 smithi086 ceph-mon[110975]: pgmap v303: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:18 smithi032 ceph-mon[147414]: pgmap v304: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:18.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:18 smithi086 ceph-mon[110975]: pgmap v304: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:20 smithi032 ceph-mon[147414]: pgmap v305: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:20.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:20 smithi086 ceph-mon[110975]: pgmap v305: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:22.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.271+0000 7f2267e08700 1 -- 172.21.15.32:0/966183312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260102e70 msgr2=0x7f2260103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.271+0000 7f2267e08700 1 --2- 172.21.15.32:0/966183312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260102e70 0x7f2260103240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f225c004940 tx=0x7f225c009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.271+0000 7f2267e08700 1 -- 172.21.15.32:0/966183312 shutdown_connections 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.271+0000 7f2267e08700 1 --2- 172.21.15.32:0/966183312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22600fb4a0 0x7f22600fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.271+0000 7f2267e08700 1 --2- 172.21.15.32:0/966183312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260102e70 0x7f2260103240 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.271+0000 7f2267e08700 1 -- 172.21.15.32:0/966183312 >> 172.21.15.32:0/966183312 conn(0x7f22600714c0 msgr2=0x7f22600718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.272+0000 7f2267e08700 1 -- 172.21.15.32:0/966183312 shutdown_connections 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.272+0000 7f2267e08700 1 -- 172.21.15.32:0/966183312 wait complete. 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.272+0000 7f2267e08700 1 Processor -- start 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.272+0000 7f2267e08700 1 -- start start 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.272+0000 7f2267e08700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600fb4a0 0x7f2260113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.272+0000 7f2267e08700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2260102e70 0x7f2260114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.272+0000 7f2267e08700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22601146e0 con 0x7f22600fb4a0 2024-04-06T14:30:22.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.272+0000 7f2267e08700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226010dad0 con 0x7f2260102e70 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.272+0000 7f2265ba4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600fb4a0 0x7f2260113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f2265ba4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600fb4a0 0x7f2260113b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50350/0 (socket says 172.21.15.32:50350) 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f2265ba4700 1 -- 172.21.15.32:0/396747186 learned_addr learned my addr 172.21.15.32:0/396747186 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f22653a3700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2260102e70 0x7f2260114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f2265ba4700 1 -- 172.21.15.32:0/396747186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2260102e70 msgr2=0x7f2260114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f2265ba4700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2260102e70 0x7f2260114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f2265ba4700 1 -- 172.21.15.32:0/396747186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c005040 con 0x7f22600fb4a0 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f22653a3700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2260102e70 0x7f2260114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f2265ba4700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600fb4a0 0x7f2260113b50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f225c00b3b0 tx=0x7f225c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f22567fc700 1 -- 172.21.15.32:0/396747186 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f225c00e040 con 0x7f22600fb4a0 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f22567fc700 1 -- 172.21.15.32:0/396747186 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f225c013420 con 0x7f22600fb4a0 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f22567fc700 1 -- 172.21.15.32:0/396747186 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f225c01c630 con 0x7f22600fb4a0 2024-04-06T14:30:22.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226010dd50 con 0x7f22600fb4a0 2024-04-06T14:30:22.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.273+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226010e290 con 0x7f22600fb4a0 2024-04-06T14:30:22.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.274+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2260105320 con 0x7f22600fb4a0 2024-04-06T14:30:22.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.275+0000 7f22567fc700 1 -- 172.21.15.32:0/396747186 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f225c013970 con 0x7f22600fb4a0 2024-04-06T14:30:22.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.276+0000 7f22567fc700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f224c06cc20 0x7f224c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:22.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.276+0000 7f22567fc700 1 -- 172.21.15.32:0/396747186 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f225c08dc80 con 0x7f22600fb4a0 2024-04-06T14:30:22.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.276+0000 7f22653a3700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f224c06cc20 0x7f224c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:22.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.276+0000 7f22653a3700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f224c06cc20 0x7f224c06f0d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f226010ec90 tx=0x7f2250009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:22.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.280+0000 7f22567fc700 1 -- 172.21.15.32:0/396747186 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f225c05b810 con 0x7f22600fb4a0 2024-04-06T14:30:22.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.502+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f226010f240 con 0x7f224c06cc20 2024-04-06T14:30:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.503+0000 7f22567fc700 1 -- 172.21.15.32:0/396747186 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f226010f240 con 0x7f224c06cc20 2024-04-06T14:30:22.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.505+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f224c06cc20 msgr2=0x7f224c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.505+0000 7f2267e08700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f224c06cc20 0x7f224c06f0d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f226010ec90 tx=0x7f2250009040 comp rx=0 tx=0).stop 2024-04-06T14:30:22.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.505+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600fb4a0 msgr2=0x7f2260113b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.505+0000 7f2267e08700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600fb4a0 0x7f2260113b50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f225c00b3b0 tx=0x7f225c009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:22.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.505+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 shutdown_connections 2024-04-06T14:30:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.506+0000 7f2267e08700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f224c06cc20 0x7f224c06f0d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.506+0000 7f2267e08700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600fb4a0 0x7f2260113b50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.506+0000 7f2267e08700 1 --2- 172.21.15.32:0/396747186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2260102e70 0x7f2260114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.506+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 >> 172.21.15.32:0/396747186 conn(0x7f22600714c0 msgr2=0x7f22600fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.506+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 shutdown_connections 2024-04-06T14:30:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.506+0000 7f2267e08700 1 -- 172.21.15.32:0/396747186 wait complete. 2024-04-06T14:30:22.516 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.611+0000 7fc8c9b39700 1 -- 172.21.15.32:0/3491009979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fe820 msgr2=0x7fc8c4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.611+0000 7fc8c9b39700 1 --2- 172.21.15.32:0/3491009979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fe820 0x7fc8c4106e60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8004940 tx=0x7fc8b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.611+0000 7fc8c9b39700 1 -- 172.21.15.32:0/3491009979 shutdown_connections 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.611+0000 7fc8c9b39700 1 --2- 172.21.15.32:0/3491009979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fe820 0x7fc8c4106e60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.611+0000 7fc8c9b39700 1 --2- 172.21.15.32:0/3491009979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c40fde80 0x7fc8c40fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.611+0000 7fc8c9b39700 1 -- 172.21.15.32:0/3491009979 >> 172.21.15.32:0/3491009979 conn(0x7fc8c40f78d0 msgr2=0x7fc8c40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.611+0000 7fc8c9b39700 1 -- 172.21.15.32:0/3491009979 shutdown_connections 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.611+0000 7fc8c9b39700 1 -- 172.21.15.32:0/3491009979 wait complete. 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c9b39700 1 Processor -- start 2024-04-06T14:30:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c9b39700 1 -- start start 2024-04-06T14:30:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c9b39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fde80 0x7fc8c40fbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c9b39700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c40fe820 0x7fc8c40fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c9b39700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40fc780 con 0x7fc8c40fde80 2024-04-06T14:30:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c9b39700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40fc8c0 con 0x7fc8c40fe820 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fde80 0x7fc8c40fbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fde80 0x7fc8c40fbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50366/0 (socket says 172.21.15.32:50366) 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c37fe700 1 -- 172.21.15.32:0/1886963872 learned_addr learned my addr 172.21.15.32:0/1886963872 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c2ffd700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c40fe820 0x7fc8c40fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c37fe700 1 -- 172.21.15.32:0/1886963872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c40fe820 msgr2=0x7fc8c40fc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c37fe700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c40fe820 0x7fc8c40fc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c37fe700 1 -- 172.21.15.32:0/1886963872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b8005040 con 0x7fc8c40fde80 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.612+0000 7fc8c2ffd700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c40fe820 0x7fc8c40fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.613+0000 7fc8c37fe700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fde80 0x7fc8c40fbcb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400c8c0 tx=0x7fc8b40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.613+0000 7fc8abfff700 1 -- 172.21.15.32:0/1886963872 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8b400a980 con 0x7fc8c40fde80 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.613+0000 7fc8abfff700 1 -- 172.21.15.32:0/1886963872 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8b400ce60 con 0x7fc8c40fde80 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.613+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c4119d60 con 0x7fc8c40fde80 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.613+0000 7fc8abfff700 1 -- 172.21.15.32:0/1886963872 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8b40113a0 con 0x7fc8c40fde80 2024-04-06T14:30:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.613+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c411a160 con 0x7fc8c40fde80 2024-04-06T14:30:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.614+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c4066710 con 0x7fc8c40fde80 2024-04-06T14:30:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.615+0000 7fc8abfff700 1 -- 172.21.15.32:0/1886963872 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc8b4012400 con 0x7fc8c40fde80 2024-04-06T14:30:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.615+0000 7fc8abfff700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc8ac06cc20 0x7fc8ac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.615+0000 7fc8abfff700 1 -- 172.21.15.32:0/1886963872 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc8b408c1f0 con 0x7fc8c40fde80 2024-04-06T14:30:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.615+0000 7fc8c2ffd700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc8ac06cc20 0x7fc8ac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.615+0000 7fc8c2ffd700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc8ac06cc20 0x7fc8ac06f0d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800b3b0 tx=0x7fc8b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.620+0000 7fc8abfff700 1 -- 172.21.15.32:0/1886963872 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc8b40561b0 con 0x7fc8c40fde80 2024-04-06T14:30:22.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:22 smithi032 ceph-mon[147414]: pgmap v306: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:22.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.843+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8c40fd3a0 con 0x7fc8ac06cc20 2024-04-06T14:30:22.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.844+0000 7fc8abfff700 1 -- 172.21.15.32:0/1886963872 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc8c40fd3a0 con 0x7fc8ac06cc20 2024-04-06T14:30:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.846+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc8ac06cc20 msgr2=0x7fc8ac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.846+0000 7fc8c9b39700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc8ac06cc20 0x7fc8ac06f0d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800b3b0 tx=0x7fc8b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.846+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fde80 msgr2=0x7fc8c40fbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.846+0000 7fc8c9b39700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fde80 0x7fc8c40fbcb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400c8c0 tx=0x7fc8b40044c0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.847+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 shutdown_connections 2024-04-06T14:30:22.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.847+0000 7fc8c9b39700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc8ac06cc20 0x7fc8ac06f0d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.847+0000 7fc8c9b39700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c40fde80 0x7fc8c40fbcb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.847+0000 7fc8c9b39700 1 --2- 172.21.15.32:0/1886963872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c40fe820 0x7fc8c40fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.847+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 >> 172.21.15.32:0/1886963872 conn(0x7fc8c40f78d0 msgr2=0x7fc8c40f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:22.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.847+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 shutdown_connections 2024-04-06T14:30:22.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.847+0000 7fc8c9b39700 1 -- 172.21.15.32:0/1886963872 wait complete. 2024-04-06T14:30:22.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:22 smithi086 ceph-mon[110975]: pgmap v306: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 -- 172.21.15.32:0/1909653244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fe830 msgr2=0x7f8890106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 --2- 172.21.15.32:0/1909653244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fe830 0x7f8890106f00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f887c004940 tx=0x7f887c009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 -- 172.21.15.32:0/1909653244 shutdown_connections 2024-04-06T14:30:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 --2- 172.21.15.32:0/1909653244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fe830 0x7f8890106f00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 --2- 172.21.15.32:0/1909653244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88900fde90 0x7f88900fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 -- 172.21.15.32:0/1909653244 >> 172.21.15.32:0/1909653244 conn(0x7f8890072bc0 msgr2=0x7f8890072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 -- 172.21.15.32:0/1909653244 shutdown_connections 2024-04-06T14:30:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 -- 172.21.15.32:0/1909653244 wait complete. 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 Processor -- start 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.950+0000 7f8896b9a700 1 -- start start 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8896b9a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fde90 0x7f8890113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8896b9a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88900fe830 0x7f8890114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8896b9a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8890114670 con 0x7f88900fde90 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8896b9a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889010da60 con 0x7f88900fe830 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8894936700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fde90 0x7f8890113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8894936700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fde90 0x7f8890113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50392/0 (socket says 172.21.15.32:50392) 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8894936700 1 -- 172.21.15.32:0/289772628 learned_addr learned my addr 172.21.15.32:0/289772628 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f888ffff700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88900fe830 0x7f8890114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8894936700 1 -- 172.21.15.32:0/289772628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88900fe830 msgr2=0x7f8890114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8894936700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88900fe830 0x7f8890114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8894936700 1 -- 172.21.15.32:0/289772628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c005040 con 0x7f88900fde90 2024-04-06T14:30:22.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f8894936700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fde90 0x7f8890113ae0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8884002a50 tx=0x7f88840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:22.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.951+0000 7f888d7fa700 1 -- 172.21.15.32:0/289772628 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8884007cc0 con 0x7f88900fde90 2024-04-06T14:30:22.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.952+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889010dd40 con 0x7f88900fde90 2024-04-06T14:30:22.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.952+0000 7f888d7fa700 1 -- 172.21.15.32:0/289772628 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f888400f040 con 0x7f88900fde90 2024-04-06T14:30:22.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.952+0000 7f888d7fa700 1 -- 172.21.15.32:0/289772628 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8884009540 con 0x7f88900fde90 2024-04-06T14:30:22.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.952+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889010e280 con 0x7f88900fde90 2024-04-06T14:30:22.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.952+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890065e30 con 0x7f88900fde90 2024-04-06T14:30:22.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.953+0000 7f888d7fa700 1 -- 172.21.15.32:0/289772628 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f88840096a0 con 0x7f88900fde90 2024-04-06T14:30:22.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.953+0000 7f888d7fa700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f888006cc20 0x7f888006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:22.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.953+0000 7f888d7fa700 1 -- 172.21.15.32:0/289772628 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f888408bd70 con 0x7f88900fde90 2024-04-06T14:30:22.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.953+0000 7f888ffff700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f888006cc20 0x7f888006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:22.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.954+0000 7f888ffff700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f888006cc20 0x7f888006f0d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f889010ec20 tx=0x7f887c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:22.958+0000 7f888d7fa700 1 -- 172.21.15.32:0/289772628 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8884059840 con 0x7f88900fde90 2024-04-06T14:30:23.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.171+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8890109dd0 con 0x7f888006cc20 2024-04-06T14:30:23.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.181+0000 7f888d7fa700 1 -- 172.21.15.32:0/289772628 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8890109dd0 con 0x7f888006cc20 2024-04-06T14:30:23.181 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (10m) 4m ago 21m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 4m ago 22m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (8m) 3m ago 21m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (10m) 4m ago 21m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (4m) 4m ago 16m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (4m) 4m ago 16m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (3m) 3m ago 16m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (16m) 3m ago 16m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (12m) 4m ago 23m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (11m) 3m ago 21m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 4m ago 23m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (8m) 3m ago 21m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (11m) 4m ago 22m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:30:23.182 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (11m) 3m ago 21m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:30:23.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 4m ago 20m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:30:23.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 4m ago 19m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:30:23.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 4m ago 19m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:30:23.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (5m) 3m ago 18m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:30:23.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (5m) 3m ago 18m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:30:23.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (4m) 3m ago 17m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:30:23.183 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (11m) 4m ago 21m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f888006cc20 msgr2=0x7f888006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f888006cc20 0x7f888006f0d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f889010ec20 tx=0x7f887c009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fde90 msgr2=0x7f8890113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fde90 0x7f8890113ae0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8884002a50 tx=0x7f88840046b0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 shutdown_connections 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f888006cc20 0x7f888006f0d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88900fde90 0x7f8890113ae0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 --2- 172.21.15.32:0/289772628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88900fe830 0x7f8890114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 >> 172.21.15.32:0/289772628 conn(0x7f8890072bc0 msgr2=0x7f8890103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 shutdown_connections 2024-04-06T14:30:23.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.183+0000 7f8896b9a700 1 -- 172.21.15.32:0/289772628 wait complete. 2024-04-06T14:30:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 -- 172.21.15.32:0/3738867877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75100fb4a0 msgr2=0x7f75100fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 --2- 172.21.15.32:0/3738867877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75100fb4a0 0x7f75100fb910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f74fc004940 tx=0x7f74fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 -- 172.21.15.32:0/3738867877 shutdown_connections 2024-04-06T14:30:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 --2- 172.21.15.32:0/3738867877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75100fb4a0 0x7f75100fb910 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 --2- 172.21.15.32:0/3738867877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7510102e70 0x7f7510103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 -- 172.21.15.32:0/3738867877 >> 172.21.15.32:0/3738867877 conn(0x7f75100714c0 msgr2=0x7f75100718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 -- 172.21.15.32:0/3738867877 shutdown_connections 2024-04-06T14:30:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 -- 172.21.15.32:0/3738867877 wait complete. 2024-04-06T14:30:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 Processor -- start 2024-04-06T14:30:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.286+0000 7f7516a77700 1 -- start start 2024-04-06T14:30:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f7516a77700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75100fb4a0 0x7f75100fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f7516a77700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7510102e70 0x7f75100fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f7516a77700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75100fd170 con 0x7f7510102e70 2024-04-06T14:30:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f7516a77700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75100fd2e0 con 0x7f75100fb4a0 2024-04-06T14:30:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f750ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7510102e70 0x7f75100fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f750ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7510102e70 0x7f75100fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50408/0 (socket says 172.21.15.32:50408) 2024-04-06T14:30:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f750ffff700 1 -- 172.21.15.32:0/1718850828 learned_addr learned my addr 172.21.15.32:0/1718850828 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f7514813700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75100fb4a0 0x7f75100fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f750ffff700 1 -- 172.21.15.32:0/1718850828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75100fb4a0 msgr2=0x7f75100fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f750ffff700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75100fb4a0 0x7f75100fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f750ffff700 1 -- 172.21.15.32:0/1718850828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc005040 con 0x7f7510102e70 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f750ffff700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7510102e70 0x7f75100fcbc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f74fc00b3b0 tx=0x7f74fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f750d7fa700 1 -- 172.21.15.32:0/1718850828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74fc00e040 con 0x7f7510102e70 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f750d7fa700 1 -- 172.21.15.32:0/1718850828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74fc013420 con 0x7f7510102e70 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.287+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7510119d70 con 0x7f7510102e70 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.288+0000 7f750d7fa700 1 -- 172.21.15.32:0/1718850828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74fc01c610 con 0x7f7510102e70 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.288+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751011a180 con 0x7f7510102e70 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.289+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7510100110 con 0x7f7510102e70 2024-04-06T14:30:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.289+0000 7f750d7fa700 1 -- 172.21.15.32:0/1718850828 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f74fc013a00 con 0x7f7510102e70 2024-04-06T14:30:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.289+0000 7f750d7fa700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f750006cc20 0x7f750006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.289+0000 7f750d7fa700 1 -- 172.21.15.32:0/1718850828 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f74fc08d0d0 con 0x7f7510102e70 2024-04-06T14:30:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.289+0000 7f7514813700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f750006cc20 0x7f750006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.290+0000 7f7514813700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f750006cc20 0x7f750006f0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7504004f40 tx=0x7f750400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:23.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.294+0000 7f750d7fa700 1 -- 172.21.15.32:0/1718850828 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f74fc0571d0 con 0x7f7510102e70 2024-04-06T14:30:23.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.583+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7510109e00 con 0x7f7510102e70 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.583+0000 7f750d7fa700 1 -- 172.21.15.32:0/1718850828 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f74fc05a7f0 con 0x7f7510102e70 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:30:23.584 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:30:23.585 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.585+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f750006cc20 msgr2=0x7f750006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.585+0000 7f7516a77700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f750006cc20 0x7f750006f0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7504004f40 tx=0x7f750400b040 comp rx=0 tx=0).stop 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.585+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7510102e70 msgr2=0x7f75100fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.585+0000 7f7516a77700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7510102e70 0x7f75100fcbc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f74fc00b3b0 tx=0x7f74fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.586+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 shutdown_connections 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.586+0000 7f7516a77700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f750006cc20 0x7f750006f0d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.586+0000 7f7516a77700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75100fb4a0 0x7f75100fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.586+0000 7f7516a77700 1 --2- 172.21.15.32:0/1718850828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7510102e70 0x7f75100fcbc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.586+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 >> 172.21.15.32:0/1718850828 conn(0x7f75100714c0 msgr2=0x7f75100fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.586+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 shutdown_connections 2024-04-06T14:30:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.586+0000 7f7516a77700 1 -- 172.21.15.32:0/1718850828 wait complete. 2024-04-06T14:30:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.689+0000 7fc9a4279700 1 -- 172.21.15.32:0/2927215150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c0fb4a0 msgr2=0x7fc99c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.689+0000 7fc9a4279700 1 --2- 172.21.15.32:0/2927215150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c0fb4a0 0x7fc99c0fb910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc990004940 tx=0x7fc990009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.689+0000 7fc9a4279700 1 -- 172.21.15.32:0/2927215150 shutdown_connections 2024-04-06T14:30:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.689+0000 7fc9a4279700 1 --2- 172.21.15.32:0/2927215150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c0fb4a0 0x7fc99c0fb910 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.689+0000 7fc9a4279700 1 --2- 172.21.15.32:0/2927215150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc99c102e70 0x7fc99c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.689+0000 7fc9a4279700 1 -- 172.21.15.32:0/2927215150 >> 172.21.15.32:0/2927215150 conn(0x7fc99c0714c0 msgr2=0x7fc99c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.689+0000 7fc9a4279700 1 -- 172.21.15.32:0/2927215150 shutdown_connections 2024-04-06T14:30:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.689+0000 7fc9a4279700 1 -- 172.21.15.32:0/2927215150 wait complete. 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.690+0000 7fc9a4279700 1 Processor -- start 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.690+0000 7fc9a4279700 1 -- start start 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.690+0000 7fc9a4279700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc99c0fb4a0 0x7fc99c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.690+0000 7fc9a4279700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c102e70 0x7fc99c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.690+0000 7fc9a4279700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c114600 con 0x7fc99c102e70 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.690+0000 7fc9a4279700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c10d9f0 con 0x7fc99c0fb4a0 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.690+0000 7fc9a1814700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c102e70 0x7fc99c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.690+0000 7fc9a1814700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c102e70 0x7fc99c113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50430/0 (socket says 172.21.15.32:50430) 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc9a1814700 1 -- 172.21.15.32:0/2134339545 learned_addr learned my addr 172.21.15.32:0/2134339545 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc9a2015700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc99c0fb4a0 0x7fc99c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc9a1814700 1 -- 172.21.15.32:0/2134339545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc99c0fb4a0 msgr2=0x7fc99c113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc9a1814700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc99c0fb4a0 0x7fc99c113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc9a1814700 1 -- 172.21.15.32:0/2134339545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc990005040 con 0x7fc99c102e70 2024-04-06T14:30:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc9a1814700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c102e70 0x7fc99c113fb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc990009360 tx=0x7fc990009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc98effd700 1 -- 172.21.15.32:0/2134339545 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc99000e040 con 0x7fc99c102e70 2024-04-06T14:30:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc98effd700 1 -- 172.21.15.32:0/2134339545 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc990013420 con 0x7fc99c102e70 2024-04-06T14:30:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c10dc70 con 0x7fc99c102e70 2024-04-06T14:30:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc98effd700 1 -- 172.21.15.32:0/2134339545 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc99001c5d0 con 0x7fc99c102e70 2024-04-06T14:30:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.691+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c10e230 con 0x7fc99c102e70 2024-04-06T14:30:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.692+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c100110 con 0x7fc99c102e70 2024-04-06T14:30:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.693+0000 7fc98effd700 1 -- 172.21.15.32:0/2134339545 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc990013590 con 0x7fc99c102e70 2024-04-06T14:30:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.693+0000 7fc98effd700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc98806cc30 0x7fc98806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.693+0000 7fc98effd700 1 -- 172.21.15.32:0/2134339545 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc99008d1e0 con 0x7fc99c102e70 2024-04-06T14:30:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.693+0000 7fc9a2015700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc98806cc30 0x7fc98806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.694+0000 7fc9a2015700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc98806cc30 0x7fc98806f0e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc998004f40 tx=0x7fc99800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.698+0000 7fc98effd700 1 -- 172.21.15.32:0/2134339545 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc99005ac30 con 0x7fc99c102e70 2024-04-06T14:30:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:23 smithi032 ceph-mon[147414]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:23.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:30:23.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:23 smithi086 ceph-mon[110975]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:23.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:30:23.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.955+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc99c10f0c0 con 0x7fc99c102e70 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.956+0000 7fc98effd700 1 -- 172.21.15.32:0/2134339545 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fc99005a7c0 con 0x7fc99c102e70 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:30:23.956 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:30:23.957 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:30:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:30:23.959 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:30:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc98806cc30 msgr2=0x7fc98806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc98806cc30 0x7fc98806f0e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc998004f40 tx=0x7fc99800b040 comp rx=0 tx=0).stop 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c102e70 msgr2=0x7fc99c113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c102e70 0x7fc99c113fb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc990009360 tx=0x7fc990009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 shutdown_connections 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc98806cc30 0x7fc98806f0e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc99c0fb4a0 0x7fc99c113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 --2- 172.21.15.32:0/2134339545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc99c102e70 0x7fc99c113fb0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 >> 172.21.15.32:0/2134339545 conn(0x7fc99c0714c0 msgr2=0x7fc99c0fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 shutdown_connections 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:23.958+0000 7fc9a4279700 1 -- 172.21.15.32:0/2134339545 wait complete. 2024-04-06T14:30:23.960 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:30:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.063+0000 7f3cf92bb700 1 -- 172.21.15.32:0/1573237577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4102710 msgr2=0x7f3cf40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.063+0000 7f3cf92bb700 1 --2- 172.21.15.32:0/1573237577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4102710 0x7f3cf40fbd80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8004940 tx=0x7f3ce8009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.063+0000 7f3cf92bb700 1 -- 172.21.15.32:0/1573237577 shutdown_connections 2024-04-06T14:30:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.063+0000 7f3cf92bb700 1 --2- 172.21.15.32:0/1573237577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4102710 0x7f3cf40fbd80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.063+0000 7f3cf92bb700 1 --2- 172.21.15.32:0/1573237577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cf4101d70 0x7f3cf4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.063+0000 7f3cf92bb700 1 -- 172.21.15.32:0/1573237577 >> 172.21.15.32:0/1573237577 conn(0x7f3cf4072bc0 msgr2=0x7f3cf4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.063+0000 7f3cf92bb700 1 -- 172.21.15.32:0/1573237577 shutdown_connections 2024-04-06T14:30:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.063+0000 7f3cf92bb700 1 -- 172.21.15.32:0/1573237577 wait complete. 2024-04-06T14:30:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.063+0000 7f3cf92bb700 1 Processor -- start 2024-04-06T14:30:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf92bb700 1 -- start start 2024-04-06T14:30:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf92bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4101d70 0x7f3cf410eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf92bb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cf4102710 0x7f3cf410f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf92bb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf410f580 con 0x7f3cf4101d70 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf92bb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf410f6c0 con 0x7f3cf4102710 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4101d70 0x7f3cf410eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4101d70 0x7f3cf410eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50458/0 (socket says 172.21.15.32:50458) 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf2ffd700 1 -- 172.21.15.32:0/499426393 learned_addr learned my addr 172.21.15.32:0/499426393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf2ffd700 1 -- 172.21.15.32:0/499426393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cf4102710 msgr2=0x7f3cf410f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf2ffd700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cf4102710 0x7f3cf410f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf2ffd700 1 -- 172.21.15.32:0/499426393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce8005040 con 0x7f3cf4101d70 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf2ffd700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4101d70 0x7f3cf410eb00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400eb40 tx=0x7f3ce400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cdb7fe700 1 -- 172.21.15.32:0/499426393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ce4007cc0 con 0x7f3cf4101d70 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cdb7fe700 1 -- 172.21.15.32:0/499426393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ce400f040 con 0x7f3cf4101d70 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf41179b0 con 0x7f3cf4101d70 2024-04-06T14:30:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.064+0000 7f3cdb7fe700 1 -- 172.21.15.32:0/499426393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ce4009570 con 0x7f3cf4101d70 2024-04-06T14:30:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.065+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf4117f10 con 0x7f3cf4101d70 2024-04-06T14:30:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.065+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf40ff080 con 0x7f3cf4101d70 2024-04-06T14:30:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.066+0000 7f3cdb7fe700 1 -- 172.21.15.32:0/499426393 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3ce40096d0 con 0x7f3cf4101d70 2024-04-06T14:30:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.066+0000 7f3cdb7fe700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cdc06cc30 0x7f3cdc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.066+0000 7f3cdb7fe700 1 -- 172.21.15.32:0/499426393 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3ce408bc10 con 0x7f3cf4101d70 2024-04-06T14:30:24.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.066+0000 7f3cf27fc700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cdc06cc30 0x7f3cdc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:24.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.067+0000 7f3cf27fc700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cdc06cc30 0x7f3cdc06f0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8000c00 tx=0x7f3ce8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.071+0000 7f3cdb7fe700 1 -- 172.21.15.32:0/499426393 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ce4055d00 con 0x7f3cf4101d70 2024-04-06T14:30:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.293+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cf4065e80 con 0x7f3cdc06cc30 2024-04-06T14:30:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.294+0000 7f3cdb7fe700 1 -- 172.21.15.32:0/499426393 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3cf4065e80 con 0x7f3cdc06cc30 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:30:24.295 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:30:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cdc06cc30 msgr2=0x7f3cdc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cdc06cc30 0x7f3cdc06f0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8000c00 tx=0x7f3ce8009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4101d70 msgr2=0x7f3cf410eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4101d70 0x7f3cf410eb00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400eb40 tx=0x7f3ce400b040 comp rx=0 tx=0).stop 2024-04-06T14:30:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 shutdown_connections 2024-04-06T14:30:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cdc06cc30 0x7f3cdc06f0e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cf4101d70 0x7f3cf410eb00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 --2- 172.21.15.32:0/499426393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cf4102710 0x7f3cf410f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 >> 172.21.15.32:0/499426393 conn(0x7f3cf4072bc0 msgr2=0x7f3cf40faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 shutdown_connections 2024-04-06T14:30:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.296+0000 7f3cf92bb700 1 -- 172.21.15.32:0/499426393 wait complete. 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.402+0000 7f824e21a700 1 -- 172.21.15.32:0/2398785646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 msgr2=0x7f82481020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.402+0000 7f824e21a700 1 --2- 172.21.15.32:0/2398785646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 0x7f82481020e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8238004940 tx=0x7f8238009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.402+0000 7f824e21a700 1 -- 172.21.15.32:0/2398785646 shutdown_connections 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.402+0000 7f824e21a700 1 --2- 172.21.15.32:0/2398785646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82481026b0 0x7f82480fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.402+0000 7f824e21a700 1 --2- 172.21.15.32:0/2398785646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 0x7f82481020e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.402+0000 7f824e21a700 1 -- 172.21.15.32:0/2398785646 >> 172.21.15.32:0/2398785646 conn(0x7f82480f7880 msgr2=0x7f82480f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.402+0000 7f824e21a700 1 -- 172.21.15.32:0/2398785646 shutdown_connections 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.402+0000 7f824e21a700 1 -- 172.21.15.32:0/2398785646 wait complete. 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f824e21a700 1 Processor -- start 2024-04-06T14:30:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f824e21a700 1 -- start start 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f824e21a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 0x7f8248102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f824e21a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82481026b0 0x7f82481030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f824e21a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8248103640 con 0x7f8248101d10 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f824e21a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8248103780 con 0x7f82481026b0 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f82477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 0x7f8248102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f82477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 0x7f8248102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56094/0 (socket says 172.21.15.32:56094) 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f82477fe700 1 -- 172.21.15.32:0/2886596868 learned_addr learned my addr 172.21.15.32:0/2886596868 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f823ffff700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82481026b0 0x7f82481030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f82477fe700 1 -- 172.21.15.32:0/2886596868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82481026b0 msgr2=0x7f82481030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f82477fe700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82481026b0 0x7f82481030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f82477fe700 1 -- 172.21.15.32:0/2886596868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8238005040 con 0x7f8248101d10 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.403+0000 7f82477fe700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 0x7f8248102b70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8238000c00 tx=0x7f8238009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.404+0000 7f8244ff9700 1 -- 172.21.15.32:0/2886596868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8238021070 con 0x7f8248101d10 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.404+0000 7f8244ff9700 1 -- 172.21.15.32:0/2886596868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8238013de0 con 0x7f8248101d10 2024-04-06T14:30:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.404+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82481039d0 con 0x7f8248101d10 2024-04-06T14:30:24.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.404+0000 7f8244ff9700 1 -- 172.21.15.32:0/2886596868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f823800e070 con 0x7f8248101d10 2024-04-06T14:30:24.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.404+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824811a440 con 0x7f8248101d10 2024-04-06T14:30:24.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.405+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82480fd460 con 0x7f8248101d10 2024-04-06T14:30:24.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.405+0000 7f8244ff9700 1 -- 172.21.15.32:0/2886596868 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f823801c600 con 0x7f8248101d10 2024-04-06T14:30:24.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.405+0000 7f8244ff9700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f823406cc20 0x7f823406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:24.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.405+0000 7f8244ff9700 1 -- 172.21.15.32:0/2886596868 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f823808d010 con 0x7f8248101d10 2024-04-06T14:30:24.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.406+0000 7f823ffff700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f823406cc20 0x7f823406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:24.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.406+0000 7f823ffff700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f823406cc20 0x7f823406f0d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8230004600 tx=0x7f823000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:24.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.410+0000 7f8244ff9700 1 -- 172.21.15.32:0/2886596868 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8238057110 con 0x7f8248101d10 2024-04-06T14:30:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.699+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f824804bed0 con 0x7f8248101d10 2024-04-06T14:30:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.699+0000 7f8244ff9700 1 -- 172.21.15.32:0/2886596868 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f823805a730 con 0x7f8248101d10 2024-04-06T14:30:24.699 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:30:24.700 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:30:24.700 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:30:24.700 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:30:24.700 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:30:24.700 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:30:24.700 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.701+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f823406cc20 msgr2=0x7f823406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.701+0000 7f824e21a700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f823406cc20 0x7f823406f0d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8230004600 tx=0x7f823000b040 comp rx=0 tx=0).stop 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.701+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 msgr2=0x7f8248102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.701+0000 7f824e21a700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 0x7f8248102b70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8238000c00 tx=0x7f8238009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.702+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 shutdown_connections 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.702+0000 7f824e21a700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f823406cc20 0x7f823406f0d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.702+0000 7f824e21a700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8248101d10 0x7f8248102b70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.702+0000 7f824e21a700 1 --2- 172.21.15.32:0/2886596868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82481026b0 0x7f82481030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.702+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 >> 172.21.15.32:0/2886596868 conn(0x7f82480f7880 msgr2=0x7f82480f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.702+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 shutdown_connections 2024-04-06T14:30:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:24.702+0000 7f824e21a700 1 -- 172.21.15.32:0/2886596868 wait complete. 2024-04-06T14:30:24.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:24 smithi032 ceph-mon[147414]: pgmap v307: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:24.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:24 smithi032 ceph-mon[147414]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:24.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:24 smithi032 ceph-mon[147414]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:24.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:24 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1718850828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:30:24.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:24 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2134339545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:30:24.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:24 smithi086 ceph-mon[110975]: pgmap v307: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:24.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:24 smithi086 ceph-mon[110975]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:24 smithi086 ceph-mon[110975]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:24 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1718850828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:30:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:24 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2134339545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:30:25.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:25 smithi032 ceph-mon[147414]: from='client.34636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:25.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:25 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2886596868' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:30:25.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:25 smithi086 ceph-mon[110975]: from='client.34636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:25.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:25 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2886596868' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:30:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:26 smithi032 ceph-mon[147414]: pgmap v308: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:26.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:26 smithi086 ceph-mon[110975]: pgmap v308: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:28 smithi032 ceph-mon[147414]: pgmap v309: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:28.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:28 smithi086 ceph-mon[110975]: pgmap v309: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:30.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:30 smithi032 ceph-mon[147414]: pgmap v310: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:30.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:30 smithi086 ceph-mon[110975]: pgmap v310: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:32.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:32 smithi032 ceph-mon[147414]: pgmap v311: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:32.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:32 smithi086 ceph-mon[110975]: pgmap v311: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:34 smithi032 ceph-mon[147414]: pgmap v312: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:34.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:34 smithi086 ceph-mon[110975]: pgmap v312: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:36.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:36 smithi032 ceph-mon[147414]: pgmap v313: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:36.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:36 smithi086 ceph-mon[110975]: pgmap v313: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:38.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:38 smithi032 ceph-mon[147414]: pgmap v314: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:38.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:30:38.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:38 smithi086 ceph-mon[110975]: pgmap v314: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:38.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:30:40.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:40 smithi032 ceph-mon[147414]: pgmap v315: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:40.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:40 smithi086 ceph-mon[110975]: pgmap v315: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:42.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:42 smithi032 ceph-mon[147414]: pgmap v316: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:42.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:42 smithi086 ceph-mon[110975]: pgmap v316: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:44.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:44 smithi032 ceph-mon[147414]: pgmap v317: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:44.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:44 smithi086 ceph-mon[110975]: pgmap v317: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:46.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:46 smithi032 ceph-mon[147414]: pgmap v318: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:46.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:46 smithi086 ceph-mon[110975]: pgmap v318: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:48.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:48 smithi032 ceph-mon[147414]: pgmap v319: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:48.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:48 smithi086 ceph-mon[110975]: pgmap v319: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:50.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:50 smithi032 ceph-mon[147414]: pgmap v320: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:50.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:50 smithi086 ceph-mon[110975]: pgmap v320: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:52.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:52 smithi032 ceph-mon[147414]: pgmap v321: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:52.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:52 smithi086 ceph-mon[110975]: pgmap v321: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:53.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:30:53.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:30:54.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:54 smithi032 ceph-mon[147414]: pgmap v322: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:54.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.819+0000 7f7b512d7700 1 -- 172.21.15.32:0/2368884047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 msgr2=0x7f7b4c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:54.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.819+0000 7f7b512d7700 1 --2- 172.21.15.32:0/2368884047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 0x7f7b4c1031e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c004940 tx=0x7f7b3c009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.819+0000 7f7b512d7700 1 -- 172.21.15.32:0/2368884047 shutdown_connections 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.819+0000 7f7b512d7700 1 --2- 172.21.15.32:0/2368884047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.819+0000 7f7b512d7700 1 --2- 172.21.15.32:0/2368884047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 0x7f7b4c1031e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.819+0000 7f7b512d7700 1 -- 172.21.15.32:0/2368884047 >> 172.21.15.32:0/2368884047 conn(0x7f7b4c0f8980 msgr2=0x7f7b4c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.819+0000 7f7b512d7700 1 -- 172.21.15.32:0/2368884047 shutdown_connections 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.819+0000 7f7b512d7700 1 -- 172.21.15.32:0/2368884047 wait complete. 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.820+0000 7f7b512d7700 1 Processor -- start 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.820+0000 7f7b512d7700 1 -- start start 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.820+0000 7f7b512d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.820+0000 7f7b512d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.820+0000 7f7b512d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c1146d0 con 0x7f7b4c102e10 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.820+0000 7f7b512d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c10da30 con 0x7f7b4c1037b0 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.820+0000 7f7b4affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:54.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.820+0000 7f7b4affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56278/0 (socket says 172.21.15.32:56278) 2024-04-06T14:30:54.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.820+0000 7f7b4affd700 1 -- 172.21.15.32:0/616052885 learned_addr learned my addr 172.21.15.32:0/616052885 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:54.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b4a7fc700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:54.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b4affd700 1 -- 172.21.15.32:0/616052885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b4c1037b0 msgr2=0x7f7b4c113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:54.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b4affd700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:54.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b4affd700 1 -- 172.21.15.32:0/616052885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3c005040 con 0x7f7b4c102e10 2024-04-06T14:30:54.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b4a7fc700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:30:54.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b4affd700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ab0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009390 tx=0x7f7b3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b337fe700 1 -- 172.21.15.32:0/616052885 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b3c00e040 con 0x7f7b4c102e10 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c10dcb0 con 0x7f7b4c102e10 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b337fe700 1 -- 172.21.15.32:0/616052885 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b3c013420 con 0x7f7b4c102e10 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b337fe700 1 -- 172.21.15.32:0/616052885 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b3c01c630 con 0x7f7b4c102e10 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.821+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c10e270 con 0x7f7b4c102e10 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.822+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c0fe560 con 0x7f7b4c102e10 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.823+0000 7f7b337fe700 1 -- 172.21.15.32:0/616052885 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7b3c0139e0 con 0x7f7b4c102e10 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.823+0000 7f7b337fe700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b3406cc30 0x7f7b3406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.823+0000 7f7b337fe700 1 -- 172.21.15.32:0/616052885 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7b3c08d2b0 con 0x7f7b4c102e10 2024-04-06T14:30:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.823+0000 7f7b4a7fc700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b3406cc30 0x7f7b3406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:54.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.824+0000 7f7b4a7fc700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b3406cc30 0x7f7b3406f0e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b40006fd0 tx=0x7f7b40008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:54.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:54.828+0000 7f7b337fe700 1 -- 172.21.15.32:0/616052885 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7b3c05ae30 con 0x7f7b4c102e10 2024-04-06T14:30:54.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:54 smithi086 ceph-mon[110975]: pgmap v322: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:55.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.054+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b4c109ee0 con 0x7f7b3406cc30 2024-04-06T14:30:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.055+0000 7f7b337fe700 1 -- 172.21.15.32:0/616052885 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7b4000c660 con 0x7f7b3406cc30 2024-04-06T14:30:55.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.057+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b3406cc30 msgr2=0x7f7b3406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.057+0000 7f7b512d7700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b3406cc30 0x7f7b3406f0e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b40006fd0 tx=0x7f7b40008040 comp rx=0 tx=0).stop 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.057+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 msgr2=0x7f7b4c113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.057+0000 7f7b512d7700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ab0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009390 tx=0x7f7b3c009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.057+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 shutdown_connections 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.057+0000 7f7b512d7700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7b3406cc30 0x7f7b3406f0e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.057+0000 7f7b512d7700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ab0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.057+0000 7f7b512d7700 1 --2- 172.21.15.32:0/616052885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.057+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 >> 172.21.15.32:0/616052885 conn(0x7f7b4c0f8980 msgr2=0x7f7b4c0f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.058+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 shutdown_connections 2024-04-06T14:30:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.058+0000 7f7b512d7700 1 -- 172.21.15.32:0/616052885 wait complete. 2024-04-06T14:30:55.068 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:30:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 -- 172.21.15.32:0/3797554085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fde90 msgr2=0x7f676c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 --2- 172.21.15.32:0/3797554085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fde90 0x7f676c0fe260 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f675c004940 tx=0x7f675c009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 -- 172.21.15.32:0/3797554085 shutdown_connections 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 --2- 172.21.15.32:0/3797554085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f676c0fe830 0x7f676c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 --2- 172.21.15.32:0/3797554085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fde90 0x7f676c0fe260 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 -- 172.21.15.32:0/3797554085 >> 172.21.15.32:0/3797554085 conn(0x7f676c072bc0 msgr2=0x7f676c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 -- 172.21.15.32:0/3797554085 shutdown_connections 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 -- 172.21.15.32:0/3797554085 wait complete. 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 Processor -- start 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 -- start start 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f676c0fde90 0x7f676c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fe830 0x7f676c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0fc820 con 0x7f676c0fe830 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.162+0000 7f6770aba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0fc960 con 0x7f676c0fde90 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f6769d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fe830 0x7f676c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f6769d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fe830 0x7f676c0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56300/0 (socket says 172.21.15.32:56300) 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f6769d9b700 1 -- 172.21.15.32:0/3700572985 learned_addr learned my addr 172.21.15.32:0/3700572985 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f676a59c700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f676c0fde90 0x7f676c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f6769d9b700 1 -- 172.21.15.32:0/3700572985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f676c0fde90 msgr2=0x7f676c0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f6769d9b700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f676c0fde90 0x7f676c0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f6769d9b700 1 -- 172.21.15.32:0/3700572985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c005040 con 0x7f676c0fe830 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f6769d9b700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fe830 0x7f676c0fc290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6760002a50 tx=0x7f67600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f675affd700 1 -- 172.21.15.32:0/3700572985 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6760007cc0 con 0x7f676c0fe830 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f675affd700 1 -- 172.21.15.32:0/3700572985 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f676000f040 con 0x7f676c0fe830 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c119df0 con 0x7f676c0fe830 2024-04-06T14:30:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f675affd700 1 -- 172.21.15.32:0/3700572985 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f676000d9c0 con 0x7f676c0fe830 2024-04-06T14:30:55.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.163+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c11a1f0 con 0x7f676c0fe830 2024-04-06T14:30:55.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.164+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c065e30 con 0x7f676c0fe830 2024-04-06T14:30:55.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.165+0000 7f675affd700 1 -- 172.21.15.32:0/3700572985 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f676000db20 con 0x7f676c0fe830 2024-04-06T14:30:55.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.165+0000 7f675affd700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f675406cc30 0x7f675406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:55.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.165+0000 7f675affd700 1 -- 172.21.15.32:0/3700572985 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f676008bc90 con 0x7f676c0fe830 2024-04-06T14:30:55.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.165+0000 7f676a59c700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f675406cc30 0x7f675406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:55.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.166+0000 7f676a59c700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f675406cc30 0x7f675406f0e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f675c00b3b0 tx=0x7f675c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.170+0000 7f675affd700 1 -- 172.21.15.32:0/3700572985 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6760055cd0 con 0x7f676c0fe830 2024-04-06T14:30:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.389+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f676c04bed0 con 0x7f675406cc30 2024-04-06T14:30:55.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.390+0000 7f675affd700 1 -- 172.21.15.32:0/3700572985 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f676c04bed0 con 0x7f675406cc30 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f675406cc30 msgr2=0x7f675406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f675406cc30 0x7f675406f0e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f675c00b3b0 tx=0x7f675c009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fe830 msgr2=0x7f676c0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fe830 0x7f676c0fc290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6760002a50 tx=0x7f67600046b0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 shutdown_connections 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f675406cc30 0x7f675406f0e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f676c0fde90 0x7f676c0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 --2- 172.21.15.32:0/3700572985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0fe830 0x7f676c0fc290 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 >> 172.21.15.32:0/3700572985 conn(0x7f676c072bc0 msgr2=0x7f676c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.392+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 shutdown_connections 2024-04-06T14:30:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.393+0000 7f6770aba700 1 -- 172.21.15.32:0/3700572985 wait complete. 2024-04-06T14:30:55.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.496+0000 7fceacb62700 1 -- 172.21.15.32:0/2508744356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 msgr2=0x7fcea80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.496+0000 7fceacb62700 1 --2- 172.21.15.32:0/2508744356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 0x7fcea80fbd80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fce90004940 tx=0x7fce90009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:55.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.496+0000 7fceacb62700 1 -- 172.21.15.32:0/2508744356 shutdown_connections 2024-04-06T14:30:55.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.496+0000 7fceacb62700 1 --2- 172.21.15.32:0/2508744356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 0x7fcea80fbd80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.496+0000 7fceacb62700 1 --2- 172.21.15.32:0/2508744356 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8101d70 0x7fcea8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.496+0000 7fceacb62700 1 -- 172.21.15.32:0/2508744356 >> 172.21.15.32:0/2508744356 conn(0x7fcea8072bc0 msgr2=0x7fcea8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:55.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.496+0000 7fceacb62700 1 -- 172.21.15.32:0/2508744356 shutdown_connections 2024-04-06T14:30:55.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.496+0000 7fceacb62700 1 -- 172.21.15.32:0/2508744356 wait complete. 2024-04-06T14:30:55.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fceacb62700 1 Processor -- start 2024-04-06T14:30:55.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fceacb62700 1 -- start start 2024-04-06T14:30:55.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fceacb62700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8101d70 0x7fcea81129f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:55.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fceacb62700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 0x7fcea8112f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:55.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fceacb62700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea8113610 con 0x7fcea8102710 2024-04-06T14:30:55.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fceacb62700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea810c970 con 0x7fcea8101d70 2024-04-06T14:30:55.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fce9dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 0x7fcea8112f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:55.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fce9dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 0x7fcea8112f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56306/0 (socket says 172.21.15.32:56306) 2024-04-06T14:30:55.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fce9dd9b700 1 -- 172.21.15.32:0/233415586 learned_addr learned my addr 172.21.15.32:0/233415586 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.497+0000 7fcea659c700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8101d70 0x7fcea81129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fce9dd9b700 1 -- 172.21.15.32:0/233415586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8101d70 msgr2=0x7fcea81129f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fce9dd9b700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8101d70 0x7fcea81129f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fce9dd9b700 1 -- 172.21.15.32:0/233415586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce90005040 con 0x7fcea8102710 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fcea659c700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8101d70 0x7fcea81129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fce9dd9b700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 0x7fcea8112f30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce9000b3b0 tx=0x7fce90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fce9f7fe700 1 -- 172.21.15.32:0/233415586 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce9000e040 con 0x7fcea8102710 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fce9f7fe700 1 -- 172.21.15.32:0/233415586 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce9000bd60 con 0x7fcea8102710 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea810cbf0 con 0x7fcea8102710 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fce9f7fe700 1 -- 172.21.15.32:0/233415586 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce90013660 con 0x7fcea8102710 2024-04-06T14:30:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.498+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea810d150 con 0x7fcea8102710 2024-04-06T14:30:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.499+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea80fd4c0 con 0x7fcea8102710 2024-04-06T14:30:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.500+0000 7fce9f7fe700 1 -- 172.21.15.32:0/233415586 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fce90013920 con 0x7fcea8102710 2024-04-06T14:30:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.500+0000 7fce9f7fe700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9406cc20 0x7fce9406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.500+0000 7fce9f7fe700 1 -- 172.21.15.32:0/233415586 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fce90090040 con 0x7fcea8102710 2024-04-06T14:30:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.500+0000 7fcea659c700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9406cc20 0x7fce9406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.500+0000 7fcea659c700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9406cc20 0x7fce9406f0d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fce98001370 tx=0x7fce980096a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:55.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.505+0000 7fce9f7fe700 1 -- 172.21.15.32:0/233415586 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fce9005da90 con 0x7fcea8102710 2024-04-06T14:30:55.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.718+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcea8108df0 con 0x7fce9406cc20 2024-04-06T14:30:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.727+0000 7fce9f7fe700 1 -- 172.21.15.32:0/233415586 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fcea8108df0 con 0x7fce9406cc20 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (11m) 4m ago 22m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 4m ago 22m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (8m) 4m ago 21m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (11m) 4m ago 22m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (5m) 4m ago 16m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (4m) 4m ago 16m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (4m) 4m ago 16m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (16m) 4m ago 16m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (13m) 4m ago 23m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:30:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (12m) 4m ago 21m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 4m ago 24m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (9m) 4m ago 21m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (11m) 4m ago 22m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (11m) 4m ago 21m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 4m ago 20m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 4m ago 20m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 4m ago 19m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (6m) 4m ago 19m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (5m) 4m ago 18m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (5m) 4m ago 18m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:30:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (11m) 4m ago 22m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:30:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.729+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9406cc20 msgr2=0x7fce9406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.729+0000 7fceacb62700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9406cc20 0x7fce9406f0d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fce98001370 tx=0x7fce980096a0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.729+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 msgr2=0x7fcea8112f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.729+0000 7fceacb62700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 0x7fcea8112f30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce9000b3b0 tx=0x7fce90009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.729+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 shutdown_connections 2024-04-06T14:30:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.729+0000 7fceacb62700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9406cc20 0x7fce9406f0d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.729+0000 7fceacb62700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8101d70 0x7fcea81129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.729+0000 7fceacb62700 1 --2- 172.21.15.32:0/233415586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8102710 0x7fcea8112f30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.729+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 >> 172.21.15.32:0/233415586 conn(0x7fcea8072bc0 msgr2=0x7fcea80fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.730+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 shutdown_connections 2024-04-06T14:30:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.730+0000 7fceacb62700 1 -- 172.21.15.32:0/233415586 wait complete. 2024-04-06T14:30:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.832+0000 7fb86b302700 1 -- 172.21.15.32:0/2812132972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 msgr2=0x7fb864102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.832+0000 7fb86b302700 1 --2- 172.21.15.32:0/2812132972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 0x7fb864102140 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb860004940 tx=0x7fb860009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.832+0000 7fb86b302700 1 -- 172.21.15.32:0/2812132972 shutdown_connections 2024-04-06T14:30:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.832+0000 7fb86b302700 1 --2- 172.21.15.32:0/2812132972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb864102710 0x7fb8640fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.832+0000 7fb86b302700 1 --2- 172.21.15.32:0/2812132972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 0x7fb864102140 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.832+0000 7fb86b302700 1 -- 172.21.15.32:0/2812132972 >> 172.21.15.32:0/2812132972 conn(0x7fb864072bc0 msgr2=0x7fb864072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:55.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.832+0000 7fb86b302700 1 -- 172.21.15.32:0/2812132972 shutdown_connections 2024-04-06T14:30:55.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.832+0000 7fb86b302700 1 -- 172.21.15.32:0/2812132972 wait complete. 2024-04-06T14:30:55.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86b302700 1 Processor -- start 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86b302700 1 -- start start 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86b302700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 0x7fb86410eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86b302700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb864102710 0x7fb86410f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86b302700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86410f600 con 0x7fb864101d70 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86b302700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86410f740 con 0x7fb864102710 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86909e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 0x7fb86410eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86909e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 0x7fb86410eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56322/0 (socket says 172.21.15.32:56322) 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86909e700 1 -- 172.21.15.32:0/1829926035 learned_addr learned my addr 172.21.15.32:0/1829926035 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86889d700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb864102710 0x7fb86410f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86909e700 1 -- 172.21.15.32:0/1829926035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb864102710 msgr2=0x7fb86410f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86909e700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb864102710 0x7fb86410f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86909e700 1 -- 172.21.15.32:0/1829926035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb860005040 con 0x7fb864101d70 2024-04-06T14:30:55.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.833+0000 7fb86889d700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb864102710 0x7fb86410f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:30:55.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.834+0000 7fb86909e700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 0x7fb86410eb80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb860000c00 tx=0x7fb860009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:55.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.834+0000 7fb859ffb700 1 -- 172.21.15.32:0/1829926035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb860021070 con 0x7fb864101d70 2024-04-06T14:30:55.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.834+0000 7fb859ffb700 1 -- 172.21.15.32:0/1829926035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8600093b0 con 0x7fb864101d70 2024-04-06T14:30:55.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.834+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb864117a30 con 0x7fb864101d70 2024-04-06T14:30:55.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.834+0000 7fb859ffb700 1 -- 172.21.15.32:0/1829926035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86000e070 con 0x7fb864101d70 2024-04-06T14:30:55.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.834+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb864117fc0 con 0x7fb864101d70 2024-04-06T14:30:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.835+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8640ff080 con 0x7fb864101d70 2024-04-06T14:30:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.836+0000 7fb859ffb700 1 -- 172.21.15.32:0/1829926035 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb86001c490 con 0x7fb864101d70 2024-04-06T14:30:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.836+0000 7fb859ffb700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb85006cc20 0x7fb85006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.836+0000 7fb859ffb700 1 -- 172.21.15.32:0/1829926035 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb86008d110 con 0x7fb864101d70 2024-04-06T14:30:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.836+0000 7fb86889d700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb85006cc20 0x7fb85006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.836+0000 7fb86889d700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb85006cc20 0x7fb85006f0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb854004fd0 tx=0x7fb85400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:55.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:55.841+0000 7fb859ffb700 1 -- 172.21.15.32:0/1829926035 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb860057150 con 0x7fb864101d70 2024-04-06T14:30:56.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.128+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb864108de0 con 0x7fb864101d70 2024-04-06T14:30:56.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.128+0000 7fb859ffb700 1 -- 172.21.15.32:0/1829926035 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fb86005a770 con 0x7fb864101d70 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:30:56.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:30:56.130 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:30:56.130 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:30:56.130 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:30:56.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:30:56.130 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb85006cc20 msgr2=0x7fb85006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb85006cc20 0x7fb85006f0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb854004fd0 tx=0x7fb85400b040 comp rx=0 tx=0).stop 2024-04-06T14:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 msgr2=0x7fb86410eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 0x7fb86410eb80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb860000c00 tx=0x7fb860009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 shutdown_connections 2024-04-06T14:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb85006cc20 0x7fb85006f0d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb864101d70 0x7fb86410eb80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 --2- 172.21.15.32:0/1829926035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb864102710 0x7fb86410f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 >> 172.21.15.32:0/1829926035 conn(0x7fb864072bc0 msgr2=0x7fb8640faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 shutdown_connections 2024-04-06T14:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.130+0000 7fb86b302700 1 -- 172.21.15.32:0/1829926035 wait complete. 2024-04-06T14:30:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.233+0000 7f723ffef700 1 -- 172.21.15.32:0/1653993041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7238102e70 msgr2=0x7f7238103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.233+0000 7f723ffef700 1 --2- 172.21.15.32:0/1653993041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7238102e70 0x7f7238103240 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7234004940 tx=0x7f7234009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.233+0000 7f723ffef700 1 -- 172.21.15.32:0/1653993041 shutdown_connections 2024-04-06T14:30:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.233+0000 7f723ffef700 1 --2- 172.21.15.32:0/1653993041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72380fb4a0 0x7f72380fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.233+0000 7f723ffef700 1 --2- 172.21.15.32:0/1653993041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7238102e70 0x7f7238103240 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.233+0000 7f723ffef700 1 -- 172.21.15.32:0/1653993041 >> 172.21.15.32:0/1653993041 conn(0x7f72380714c0 msgr2=0x7f72380718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.233+0000 7f723ffef700 1 -- 172.21.15.32:0/1653993041 shutdown_connections 2024-04-06T14:30:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.233+0000 7f723ffef700 1 -- 172.21.15.32:0/1653993041 wait complete. 2024-04-06T14:30:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.234+0000 7f723ffef700 1 Processor -- start 2024-04-06T14:30:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.234+0000 7f723ffef700 1 -- start start 2024-04-06T14:30:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.234+0000 7f723ffef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72380fb4a0 0x7f7238113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723ffef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7238102e70 0x7f7238113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723ffef700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7238114600 con 0x7f72380fb4a0 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723ffef700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723810d9f0 con 0x7f7238102e70 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723dd8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72380fb4a0 0x7f7238113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723dd8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72380fb4a0 0x7f7238113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56340/0 (socket says 172.21.15.32:56340) 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723dd8b700 1 -- 172.21.15.32:0/160359674 learned_addr learned my addr 172.21.15.32:0/160359674 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723dd8b700 1 -- 172.21.15.32:0/160359674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7238102e70 msgr2=0x7f7238113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723dd8b700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7238102e70 0x7f7238113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723dd8b700 1 -- 172.21.15.32:0/160359674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7234005040 con 0x7f72380fb4a0 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723dd8b700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72380fb4a0 0x7f7238113a70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f723400b3b0 tx=0x7f7234009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f722a7fc700 1 -- 172.21.15.32:0/160359674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f723400e040 con 0x7f72380fb4a0 2024-04-06T14:30:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f722a7fc700 1 -- 172.21.15.32:0/160359674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7234013420 con 0x7f72380fb4a0 2024-04-06T14:30:56.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723810dc70 con 0x7f72380fb4a0 2024-04-06T14:30:56.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f722a7fc700 1 -- 172.21.15.32:0/160359674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f723401c630 con 0x7f72380fb4a0 2024-04-06T14:30:56.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.235+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723810e230 con 0x7f72380fb4a0 2024-04-06T14:30:56.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.237+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7238100110 con 0x7f72380fb4a0 2024-04-06T14:30:56.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.237+0000 7f722a7fc700 1 -- 172.21.15.32:0/160359674 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7234013970 con 0x7f72380fb4a0 2024-04-06T14:30:56.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.237+0000 7f722a7fc700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f722406cc20 0x7f722406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:56.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.237+0000 7f722a7fc700 1 -- 172.21.15.32:0/160359674 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f723408d290 con 0x7f72380fb4a0 2024-04-06T14:30:56.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.237+0000 7f723d58a700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f722406cc20 0x7f722406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:56.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.238+0000 7f723d58a700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f722406cc20 0x7f722406f0d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f723810ebb0 tx=0x7f722c009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.243+0000 7f722a7fc700 1 -- 172.21.15.32:0/160359674 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f723405ad60 con 0x7f72380fb4a0 2024-04-06T14:30:56.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.488+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f723810f0c0 con 0x7f72380fb4a0 2024-04-06T14:30:56.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.489+0000 7f722a7fc700 1 -- 172.21.15.32:0/160359674 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f723405a8f0 con 0x7f72380fb4a0 2024-04-06T14:30:56.489 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:30:56.489 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:30:56.489 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:30:56.489 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:30:56.489 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:30:56.490 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:30:56.491 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:30:56.492 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:30:56.492 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:30:56.492 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:30:56.492 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:30:56.492 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:30:56.492 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:30:56.492 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:30:56.492 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:30:56.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f722406cc20 msgr2=0x7f722406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f722406cc20 0x7f722406f0d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f723810ebb0 tx=0x7f722c009ce0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72380fb4a0 msgr2=0x7f7238113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72380fb4a0 0x7f7238113a70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f723400b3b0 tx=0x7f7234009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:56.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 shutdown_connections 2024-04-06T14:30:56.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f722406cc20 0x7f722406f0d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72380fb4a0 0x7f7238113a70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 --2- 172.21.15.32:0/160359674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7238102e70 0x7f7238113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 >> 172.21.15.32:0/160359674 conn(0x7f72380714c0 msgr2=0x7f72380fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:56.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 shutdown_connections 2024-04-06T14:30:56.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.491+0000 7f723ffef700 1 -- 172.21.15.32:0/160359674 wait complete. 2024-04-06T14:30:56.494 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:30:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.596+0000 7f2076b93700 1 -- 172.21.15.32:0/2871130150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 msgr2=0x7f2070102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.596+0000 7f2076b93700 1 --2- 172.21.15.32:0/2871130150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 0x7f2070102140 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2064004940 tx=0x7f2064009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.596+0000 7f2076b93700 1 -- 172.21.15.32:0/2871130150 shutdown_connections 2024-04-06T14:30:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.596+0000 7f2076b93700 1 --2- 172.21.15.32:0/2871130150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2070102710 0x7f20700fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.596+0000 7f2076b93700 1 --2- 172.21.15.32:0/2871130150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 0x7f2070102140 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.596+0000 7f2076b93700 1 -- 172.21.15.32:0/2871130150 >> 172.21.15.32:0/2871130150 conn(0x7f2070072bc0 msgr2=0x7f2070072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.596+0000 7f2076b93700 1 -- 172.21.15.32:0/2871130150 shutdown_connections 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.596+0000 7f2076b93700 1 -- 172.21.15.32:0/2871130150 wait complete. 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.597+0000 7f2076b93700 1 Processor -- start 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.597+0000 7f2076b93700 1 -- start start 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.597+0000 7f2076b93700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 0x7f207010e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.597+0000 7f2076b93700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2070102710 0x7f207010ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.597+0000 7f2076b93700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207010f550 con 0x7f2070101d70 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.597+0000 7f2076b93700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207010f690 con 0x7f2070102710 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.597+0000 7f207492f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 0x7f207010e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.597+0000 7f207492f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 0x7f207010e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56350/0 (socket says 172.21.15.32:56350) 2024-04-06T14:30:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.597+0000 7f207492f700 1 -- 172.21.15.32:0/3881674849 learned_addr learned my addr 172.21.15.32:0/3881674849 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f206ffff700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2070102710 0x7f207010ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f207492f700 1 -- 172.21.15.32:0/3881674849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2070102710 msgr2=0x7f207010ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f207492f700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2070102710 0x7f207010ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f207492f700 1 -- 172.21.15.32:0/3881674849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2064005040 con 0x7f2070101d70 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f207492f700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 0x7f207010e9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2064000c00 tx=0x7f2064009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f206d7fa700 1 -- 172.21.15.32:0/3881674849 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2064021070 con 0x7f2070101d70 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f206d7fa700 1 -- 172.21.15.32:0/3881674849 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20640093b0 con 0x7f2070101d70 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2070117a30 con 0x7f2070101d70 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f206d7fa700 1 -- 172.21.15.32:0/3881674849 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f206400e070 con 0x7f2070101d70 2024-04-06T14:30:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.598+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2070117f40 con 0x7f2070101d70 2024-04-06T14:30:56.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.599+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20700ff080 con 0x7f2070101d70 2024-04-06T14:30:56.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.600+0000 7f206d7fa700 1 -- 172.21.15.32:0/3881674849 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f206401c490 con 0x7f2070101d70 2024-04-06T14:30:56.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.600+0000 7f206d7fa700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f206006cc20 0x7f206006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:56.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.600+0000 7f206d7fa700 1 -- 172.21.15.32:0/3881674849 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f206408d100 con 0x7f2070101d70 2024-04-06T14:30:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.600+0000 7f206ffff700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f206006cc20 0x7f206006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.600+0000 7f206ffff700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f206006cc20 0x7f206006f0d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2070066000 tx=0x7f205c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.605+0000 7f206d7fa700 1 -- 172.21.15.32:0/3881674849 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2064057140 con 0x7f2070101d70 2024-04-06T14:30:56.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:56 smithi032 ceph-mon[147414]: pgmap v323: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:56.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:56 smithi032 ceph-mon[147414]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:56.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:56 smithi032 ceph-mon[147414]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:56.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:56 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1829926035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:30:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.825+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f207004bed0 con 0x7f206006cc20 2024-04-06T14:30:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.826+0000 7f206d7fa700 1 -- 172.21.15.32:0/3881674849 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f207004bed0 con 0x7f206006cc20 2024-04-06T14:30:56.825 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:30:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:30:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:30:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:30:56.826 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:30:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f206006cc20 msgr2=0x7f206006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f206006cc20 0x7f206006f0d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2070066000 tx=0x7f205c00b040 comp rx=0 tx=0).stop 2024-04-06T14:30:56.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 msgr2=0x7f207010e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 0x7f207010e9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2064000c00 tx=0x7f2064009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:56.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 shutdown_connections 2024-04-06T14:30:56.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f206006cc20 0x7f206006f0d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2070101d70 0x7f207010e9c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 --2- 172.21.15.32:0/3881674849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2070102710 0x7f207010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 >> 172.21.15.32:0/3881674849 conn(0x7f2070072bc0 msgr2=0x7f20700faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 shutdown_connections 2024-04-06T14:30:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.828+0000 7f2076b93700 1 -- 172.21.15.32:0/3881674849 wait complete. 2024-04-06T14:30:56.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:56 smithi086 ceph-mon[110975]: pgmap v323: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:56.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:56 smithi086 ceph-mon[110975]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:56.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:56 smithi086 ceph-mon[110975]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:56.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:56 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1829926035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 -- 172.21.15.32:0/6691072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 msgr2=0x7fa5340fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 --2- 172.21.15.32:0/6691072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa5340fbd80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa520004940 tx=0x7fa520009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 -- 172.21.15.32:0/6691072 shutdown_connections 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 --2- 172.21.15.32:0/6691072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa5340fbd80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 --2- 172.21.15.32:0/6691072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 0x7fa534102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 -- 172.21.15.32:0/6691072 >> 172.21.15.32:0/6691072 conn(0x7fa534072bc0 msgr2=0x7fa534072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 -- 172.21.15.32:0/6691072 shutdown_connections 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 -- 172.21.15.32:0/6691072 wait complete. 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 Processor -- start 2024-04-06T14:30:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.932+0000 7fa53ac82700 1 -- start start 2024-04-06T14:30:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa53ac82700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 0x7fa5341129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa53ac82700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa53ac82700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa534113550 con 0x7fa534102710 2024-04-06T14:30:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa53ac82700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53410c940 con 0x7fa534101d70 2024-04-06T14:30:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa533fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa533fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56356/0 (socket says 172.21.15.32:56356) 2024-04-06T14:30:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa533fff700 1 -- 172.21.15.32:0/549066191 learned_addr learned my addr 172.21.15.32:0/549066191 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:30:56.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa533fff700 1 -- 172.21.15.32:0/549066191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 msgr2=0x7fa5341129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:56.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa533fff700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 0x7fa5341129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:56.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa533fff700 1 -- 172.21.15.32:0/549066191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa520005040 con 0x7fa534102710 2024-04-06T14:30:56.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa533fff700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534112f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa520009180 tx=0x7fa520009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa5317fa700 1 -- 172.21.15.32:0/549066191 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa52000e040 con 0x7fa534102710 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa5317fa700 1 -- 172.21.15.32:0/549066191 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa520004290 con 0x7fa534102710 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53410cbc0 con 0x7fa534102710 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.933+0000 7fa5317fa700 1 -- 172.21.15.32:0/549066191 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa52001eea0 con 0x7fa534102710 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.934+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53410d120 con 0x7fa534102710 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.935+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5340fd4c0 con 0x7fa534102710 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.935+0000 7fa5317fa700 1 -- 172.21.15.32:0/549066191 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa52001e420 con 0x7fa534102710 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.936+0000 7fa5317fa700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52406cc20 0x7fa52406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.936+0000 7fa5317fa700 1 -- 172.21.15.32:0/549066191 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa52008feb0 con 0x7fa534102710 2024-04-06T14:30:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.936+0000 7fa538a1e700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52406cc20 0x7fa52406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:30:56.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.936+0000 7fa538a1e700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52406cc20 0x7fa52406f0d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa528004ad0 tx=0x7fa528007d40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:30:56.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:56.940+0000 7fa5317fa700 1 -- 172.21.15.32:0/549066191 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa52005d900 con 0x7fa534102710 2024-04-06T14:30:57.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.227+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa53405f5f0 con 0x7fa534102710 2024-04-06T14:30:57.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.227+0000 7fa5317fa700 1 -- 172.21.15.32:0/549066191 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fa52005d490 con 0x7fa534102710 2024-04-06T14:30:57.226 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:30:57.226 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:30:57.226 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:30:57.226 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:30:57.227 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:30:57.227 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:30:57.227 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:30:57.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52406cc20 msgr2=0x7fa52406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:57.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52406cc20 0x7fa52406f0d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa528004ad0 tx=0x7fa528007d40 comp rx=0 tx=0).stop 2024-04-06T14:30:57.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 msgr2=0x7fa534112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:30:57.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534112f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa520009180 tx=0x7fa520009e30 comp rx=0 tx=0).stop 2024-04-06T14:30:57.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 shutdown_connections 2024-04-06T14:30:57.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52406cc20 0x7fa52406f0d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:57.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 0x7fa5341129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:57.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 --2- 172.21.15.32:0/549066191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534112f00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:30:57.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 >> 172.21.15.32:0/549066191 conn(0x7fa534072bc0 msgr2=0x7fa5340fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:30:57.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 shutdown_connections 2024-04-06T14:30:57.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:30:57.229+0000 7fa53ac82700 1 -- 172.21.15.32:0/549066191 wait complete. 2024-04-06T14:30:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:57 smithi032 ceph-mon[147414]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:57 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/160359674' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:30:57.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:57 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/549066191' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:30:57.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:57 smithi086 ceph-mon[110975]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:57.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:57 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/160359674' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:30:57.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:57 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/549066191' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:30:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:58 smithi032 ceph-mon[147414]: pgmap v324: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:58.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:30:58 smithi032 ceph-mon[147414]: from='client.34664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:30:58.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:58 smithi086 ceph-mon[110975]: pgmap v324: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:30:58.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:30:58 smithi086 ceph-mon[110975]: from='client.34664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:31:00.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:00 smithi032 ceph-mon[147414]: pgmap v325: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:00.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:00 smithi086 ceph-mon[110975]: pgmap v325: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:02.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:02 smithi032 ceph-mon[147414]: pgmap v326: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:02.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:02 smithi086 ceph-mon[110975]: pgmap v326: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:04.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:04 smithi032 ceph-mon[147414]: pgmap v327: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:05.979 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:04 smithi086 ceph-mon[110975]: pgmap v327: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:06.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:06 smithi032 ceph-mon[147414]: pgmap v328: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:06.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:06 smithi086 ceph-mon[110975]: pgmap v328: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:08.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:08 smithi086 ceph-mon[110975]: pgmap v329: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:08.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:31:08.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:08 smithi032 ceph-mon[147414]: pgmap v329: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:08.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:31:10.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:10 smithi086 ceph-mon[110975]: pgmap v330: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:10.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:10 smithi032 ceph-mon[147414]: pgmap v330: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:12.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:12 smithi086 ceph-mon[110975]: pgmap v331: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:12.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:12 smithi032 ceph-mon[147414]: pgmap v331: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:14.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:14 smithi086 ceph-mon[110975]: pgmap v332: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:14.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:14 smithi032 ceph-mon[147414]: pgmap v332: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:16.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:16 smithi086 ceph-mon[110975]: pgmap v333: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:16.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:16 smithi032 ceph-mon[147414]: pgmap v333: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:18.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:18 smithi086 ceph-mon[110975]: pgmap v334: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:18.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:18 smithi032 ceph-mon[147414]: pgmap v334: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:20.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:20 smithi086 ceph-mon[110975]: pgmap v335: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:20.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:20 smithi032 ceph-mon[147414]: pgmap v335: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:22.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:22 smithi086 ceph-mon[110975]: pgmap v336: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:22.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:22 smithi032 ceph-mon[147414]: pgmap v336: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:23.733 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:31:23.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:31:24.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:24 smithi086 ceph-mon[110975]: pgmap v337: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:24.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:24 smithi032 ceph-mon[147414]: pgmap v337: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:26.880 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:26 smithi086 ceph-mon[110975]: pgmap v338: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:26.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:26 smithi032 ceph-mon[147414]: pgmap v338: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:27.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.347+0000 7fb9d40a5700 1 -- 172.21.15.32:0/2004805193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc101d70 msgr2=0x7fb9cc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:27.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.347+0000 7fb9d40a5700 1 --2- 172.21.15.32:0/2004805193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc101d70 0x7fb9cc102140 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8004940 tx=0x7fb9c8009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:27.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.347+0000 7fb9d40a5700 1 -- 172.21.15.32:0/2004805193 shutdown_connections 2024-04-06T14:31:27.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.347+0000 7fb9d40a5700 1 --2- 172.21.15.32:0/2004805193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc102710 0x7fb9cc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.347+0000 7fb9d40a5700 1 --2- 172.21.15.32:0/2004805193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc101d70 0x7fb9cc102140 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.347+0000 7fb9d40a5700 1 -- 172.21.15.32:0/2004805193 >> 172.21.15.32:0/2004805193 conn(0x7fb9cc072bc0 msgr2=0x7fb9cc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:27.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.347+0000 7fb9d40a5700 1 -- 172.21.15.32:0/2004805193 shutdown_connections 2024-04-06T14:31:27.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.347+0000 7fb9d40a5700 1 -- 172.21.15.32:0/2004805193 wait complete. 2024-04-06T14:31:27.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d40a5700 1 Processor -- start 2024-04-06T14:31:27.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d40a5700 1 -- start start 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d40a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc101d70 0x7fb9cc10c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d40a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc102710 0x7fb9cc10cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d40a5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc10d2d0 con 0x7fb9cc102710 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d40a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc10d440 con 0x7fb9cc101d70 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d1640700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc102710 0x7fb9cc10cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d1640700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc102710 0x7fb9cc10cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54844/0 (socket says 172.21.15.32:54844) 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d1640700 1 -- 172.21.15.32:0/3185539081 learned_addr learned my addr 172.21.15.32:0/3185539081 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d1e41700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc101d70 0x7fb9cc10c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d1640700 1 -- 172.21.15.32:0/3185539081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc101d70 msgr2=0x7fb9cc10c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d1640700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc101d70 0x7fb9cc10c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.348+0000 7fb9d1640700 1 -- 172.21.15.32:0/3185539081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c8005040 con 0x7fb9cc102710 2024-04-06T14:31:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.349+0000 7fb9d1640700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc102710 0x7fb9cc10cd90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc002a80 tx=0x7fb9bc004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.349+0000 7fb9c27fc700 1 -- 172.21.15.32:0/3185539081 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9bc007cc0 con 0x7fb9cc102710 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.349+0000 7fb9c27fc700 1 -- 172.21.15.32:0/3185539081 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9bc00f040 con 0x7fb9cc102710 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.349+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc119ec0 con 0x7fb9cc102710 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.349+0000 7fb9c27fc700 1 -- 172.21.15.32:0/3185539081 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9bc00da40 con 0x7fb9cc102710 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.349+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc11a290 con 0x7fb9cc102710 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.350+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9cc0ff080 con 0x7fb9cc102710 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.350+0000 7fb9c27fc700 1 -- 172.21.15.32:0/3185539081 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb9bc00dbe0 con 0x7fb9cc102710 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.351+0000 7fb9c27fc700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9b806cc20 0x7fb9b806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.351+0000 7fb9c27fc700 1 -- 172.21.15.32:0/3185539081 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb9bc08b730 con 0x7fb9cc102710 2024-04-06T14:31:27.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.351+0000 7fb9d1e41700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9b806cc20 0x7fb9b806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:27.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.351+0000 7fb9d1e41700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9b806cc20 0x7fb9b806f0d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8004f90 tx=0x7fb9c8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:27.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.356+0000 7fb9c27fc700 1 -- 172.21.15.32:0/3185539081 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb9bc059180 con 0x7fb9cc102710 2024-04-06T14:31:27.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.578+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9cc108d50 con 0x7fb9b806cc20 2024-04-06T14:31:27.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.579+0000 7fb9c27fc700 1 -- 172.21.15.32:0/3185539081 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb9cc108d50 con 0x7fb9b806cc20 2024-04-06T14:31:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9b806cc20 msgr2=0x7fb9b806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9b806cc20 0x7fb9b806f0d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8004f90 tx=0x7fb9c8009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc102710 msgr2=0x7fb9cc10cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:27.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc102710 0x7fb9cc10cd90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc002a80 tx=0x7fb9bc004630 comp rx=0 tx=0).stop 2024-04-06T14:31:27.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 shutdown_connections 2024-04-06T14:31:27.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9b806cc20 0x7fb9b806f0d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc101d70 0x7fb9cc10c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 --2- 172.21.15.32:0/3185539081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9cc102710 0x7fb9cc10cd90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 >> 172.21.15.32:0/3185539081 conn(0x7fb9cc072bc0 msgr2=0x7fb9cc0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:27.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 shutdown_connections 2024-04-06T14:31:27.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.581+0000 7fb9d40a5700 1 -- 172.21.15.32:0/3185539081 wait complete. 2024-04-06T14:31:27.596 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:31:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 -- 172.21.15.32:0/3277195501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 msgr2=0x7f10d81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 --2- 172.21.15.32:0/3277195501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 0x7f10d81031e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f10c8004940 tx=0x7f10c8009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 -- 172.21.15.32:0/3277195501 shutdown_connections 2024-04-06T14:31:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 --2- 172.21.15.32:0/3277195501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d81037b0 0x7f10d80fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 --2- 172.21.15.32:0/3277195501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 0x7f10d81031e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 -- 172.21.15.32:0/3277195501 >> 172.21.15.32:0/3277195501 conn(0x7f10d80f89a0 msgr2=0x7f10d80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 -- 172.21.15.32:0/3277195501 shutdown_connections 2024-04-06T14:31:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 -- 172.21.15.32:0/3277195501 wait complete. 2024-04-06T14:31:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 Processor -- start 2024-04-06T14:31:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.686+0000 7f10dcafd700 1 -- start start 2024-04-06T14:31:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10dcafd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 0x7f10d80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10dcafd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d81037b0 0x7f10d8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10dcafd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8070d90 con 0x7f10d8102e10 2024-04-06T14:31:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10dcafd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8070ed0 con 0x7f10d81037b0 2024-04-06T14:31:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10d659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 0x7f10d80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10d659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 0x7f10d80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54864/0 (socket says 172.21.15.32:54864) 2024-04-06T14:31:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10d659c700 1 -- 172.21.15.32:0/4118890165 learned_addr learned my addr 172.21.15.32:0/4118890165 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:31:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10d5d9b700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d81037b0 0x7f10d8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10d659c700 1 -- 172.21.15.32:0/4118890165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d81037b0 msgr2=0x7f10d8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10d659c700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d81037b0 0x7f10d8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10d659c700 1 -- 172.21.15.32:0/4118890165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c8005040 con 0x7f10d8102e10 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10d5d9b700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d81037b0 0x7f10d8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10d659c700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 0x7f10d80702c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f10c800b3b0 tx=0x7f10c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10c6ffd700 1 -- 172.21.15.32:0/4118890165 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10c800e040 con 0x7f10d8102e10 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10c6ffd700 1 -- 172.21.15.32:0/4118890165 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10c8013420 con 0x7f10d8102e10 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10c6ffd700 1 -- 172.21.15.32:0/4118890165 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10c801c630 con 0x7f10d8102e10 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d8071120 con 0x7f10d8102e10 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.687+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d811a220 con 0x7f10d8102e10 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.688+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d80fe560 con 0x7f10d8102e10 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.689+0000 7f10c6ffd700 1 -- 172.21.15.32:0/4118890165 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f10c8013970 con 0x7f10d8102e10 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.689+0000 7f10c6ffd700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f10c006cc20 0x7f10c006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.689+0000 7f10c6ffd700 1 -- 172.21.15.32:0/4118890165 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f10c808d1b0 con 0x7f10d8102e10 2024-04-06T14:31:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.689+0000 7f10d5d9b700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f10c006cc20 0x7f10c006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.690+0000 7f10d5d9b700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f10c006cc20 0x7f10c006f0d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009fd0 tx=0x7f10cc009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.694+0000 7f10c6ffd700 1 -- 172.21.15.32:0/4118890165 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f10c80572b0 con 0x7f10d8102e10 2024-04-06T14:31:27.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.916+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f10d80719b0 con 0x7f10c006cc20 2024-04-06T14:31:27.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.917+0000 7f10c6ffd700 1 -- 172.21.15.32:0/4118890165 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f10d80719b0 con 0x7f10c006cc20 2024-04-06T14:31:27.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f10c006cc20 msgr2=0x7f10c006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f10c006cc20 0x7f10c006f0d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009fd0 tx=0x7f10cc009ee0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 msgr2=0x7f10d80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 0x7f10d80702c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f10c800b3b0 tx=0x7f10c8009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 shutdown_connections 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f10c006cc20 0x7f10c006f0d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10d8102e10 0x7f10d80702c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 --2- 172.21.15.32:0/4118890165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d81037b0 0x7f10d8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 >> 172.21.15.32:0/4118890165 conn(0x7f10d80f89a0 msgr2=0x7f10d80f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 shutdown_connections 2024-04-06T14:31:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:27.919+0000 7f10dcafd700 1 -- 172.21.15.32:0/4118890165 wait complete. 2024-04-06T14:31:28.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.022+0000 7fe92a174700 1 -- 172.21.15.32:0/3755262502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 msgr2=0x7fe9240fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.022+0000 7fe92a174700 1 --2- 172.21.15.32:0/3755262502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 0x7fe9240fb910 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe918004940 tx=0x7fe918009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:28.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.022+0000 7fe92a174700 1 -- 172.21.15.32:0/3755262502 shutdown_connections 2024-04-06T14:31:28.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.022+0000 7fe92a174700 1 --2- 172.21.15.32:0/3755262502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 0x7fe9240fb910 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.022+0000 7fe92a174700 1 --2- 172.21.15.32:0/3755262502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe924102e70 0x7fe924103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.022+0000 7fe92a174700 1 -- 172.21.15.32:0/3755262502 >> 172.21.15.32:0/3755262502 conn(0x7fe9240714c0 msgr2=0x7fe9240718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:28.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.022+0000 7fe92a174700 1 -- 172.21.15.32:0/3755262502 shutdown_connections 2024-04-06T14:31:28.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.023+0000 7fe92a174700 1 -- 172.21.15.32:0/3755262502 wait complete. 2024-04-06T14:31:28.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.023+0000 7fe92a174700 1 Processor -- start 2024-04-06T14:31:28.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.023+0000 7fe92a174700 1 -- start start 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe92a174700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 0x7fe924113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe92a174700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe924102e70 0x7fe924113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe92a174700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9241145c0 con 0x7fe9240fb4a0 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe92a174700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92410d9b0 con 0x7fe924102e70 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe9237fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 0x7fe924113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe9237fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 0x7fe924113a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54868/0 (socket says 172.21.15.32:54868) 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe9237fe700 1 -- 172.21.15.32:0/942698229 learned_addr learned my addr 172.21.15.32:0/942698229 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe922ffd700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe924102e70 0x7fe924113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe9237fe700 1 -- 172.21.15.32:0/942698229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe924102e70 msgr2=0x7fe924113f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe9237fe700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe924102e70 0x7fe924113f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe9237fe700 1 -- 172.21.15.32:0/942698229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918005040 con 0x7fe9240fb4a0 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe922ffd700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe924102e70 0x7fe924113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:31:28.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe9237fe700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 0x7fe924113a30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe91400eb40 tx=0x7fe91400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:28.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe90bfff700 1 -- 172.21.15.32:0/942698229 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe914007cc0 con 0x7fe9240fb4a0 2024-04-06T14:31:28.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.024+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92410dc30 con 0x7fe9240fb4a0 2024-04-06T14:31:28.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.025+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92410e1f0 con 0x7fe9240fb4a0 2024-04-06T14:31:28.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.025+0000 7fe90bfff700 1 -- 172.21.15.32:0/942698229 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe91400f040 con 0x7fe9240fb4a0 2024-04-06T14:31:28.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.025+0000 7fe90bfff700 1 -- 172.21.15.32:0/942698229 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe914009570 con 0x7fe9240fb4a0 2024-04-06T14:31:28.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.025+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9100052b0 con 0x7fe9240fb4a0 2024-04-06T14:31:28.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.026+0000 7fe90bfff700 1 -- 172.21.15.32:0/942698229 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe91400cc30 con 0x7fe9240fb4a0 2024-04-06T14:31:28.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.026+0000 7fe90bfff700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe90c06cc20 0x7fe90c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:28.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.027+0000 7fe90bfff700 1 -- 172.21.15.32:0/942698229 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe91408bc90 con 0x7fe9240fb4a0 2024-04-06T14:31:28.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.027+0000 7fe922ffd700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe90c06cc20 0x7fe90c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:28.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.027+0000 7fe922ffd700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe90c06cc20 0x7fe90c06f0d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe91800b0d0 tx=0x7fe918009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.031+0000 7fe90bfff700 1 -- 172.21.15.32:0/942698229 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe914059790 con 0x7fe9240fb4a0 2024-04-06T14:31:28.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.242+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe910000bc0 con 0x7fe90c06cc20 2024-04-06T14:31:28.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.252+0000 7fe90bfff700 1 -- 172.21.15.32:0/942698229 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe910000bc0 con 0x7fe90c06cc20 2024-04-06T14:31:28.252 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:31:28.252 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (11m) 5m ago 23m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:31:28.252 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 5m ago 23m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:31:28.252 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (9m) 4m ago 22m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (11m) 5m ago 22m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (5m) 5m ago 17m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (5m) 5m ago 17m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (4m) 4m ago 17m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (17m) 4m ago 17m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (13m) 5m ago 24m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (12m) 4m ago 22m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 5m ago 24m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (9m) 4m ago 22m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (12m) 5m ago 23m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (12m) 4m ago 22m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:31:28.253 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 5m ago 21m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:31:28.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 5m ago 20m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:31:28.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 5m ago 20m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:31:28.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (6m) 4m ago 19m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:31:28.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (6m) 4m ago 19m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:31:28.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (5m) 4m ago 18m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:31:28.254 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (12m) 5m ago 22m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:31:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe90c06cc20 msgr2=0x7fe90c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe90c06cc20 0x7fe90c06f0d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe91800b0d0 tx=0x7fe918009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 msgr2=0x7fe924113a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 0x7fe924113a30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe91400eb40 tx=0x7fe91400b040 comp rx=0 tx=0).stop 2024-04-06T14:31:28.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 shutdown_connections 2024-04-06T14:31:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe90c06cc20 0x7fe90c06f0d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9240fb4a0 0x7fe924113a30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 --2- 172.21.15.32:0/942698229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe924102e70 0x7fe924113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 >> 172.21.15.32:0/942698229 conn(0x7fe9240714c0 msgr2=0x7fe9240fa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 shutdown_connections 2024-04-06T14:31:28.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.254+0000 7fe92a174700 1 -- 172.21.15.32:0/942698229 wait complete. 2024-04-06T14:31:28.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.356+0000 7f5a37b8d700 1 -- 172.21.15.32:0/1501348256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fde90 msgr2=0x7f5a300fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.356+0000 7f5a37b8d700 1 --2- 172.21.15.32:0/1501348256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fde90 0x7f5a300fe260 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c004940 tx=0x7f5a2c009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:28.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.356+0000 7f5a37b8d700 1 -- 172.21.15.32:0/1501348256 shutdown_connections 2024-04-06T14:31:28.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.356+0000 7f5a37b8d700 1 --2- 172.21.15.32:0/1501348256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a300fe830 0x7f5a30106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.356+0000 7f5a37b8d700 1 --2- 172.21.15.32:0/1501348256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fde90 0x7f5a300fe260 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.356+0000 7f5a37b8d700 1 -- 172.21.15.32:0/1501348256 >> 172.21.15.32:0/1501348256 conn(0x7f5a30072bc0 msgr2=0x7f5a30072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.356+0000 7f5a37b8d700 1 -- 172.21.15.32:0/1501348256 shutdown_connections 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.356+0000 7f5a37b8d700 1 -- 172.21.15.32:0/1501348256 wait complete. 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a37b8d700 1 Processor -- start 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a37b8d700 1 -- start start 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a37b8d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a300fde90 0x7f5a300fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a37b8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fe830 0x7f5a300fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a37b8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300fc820 con 0x7f5a300fe830 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a37b8d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300fc960 con 0x7f5a300fde90 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a35128700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fe830 0x7f5a300fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a35128700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fe830 0x7f5a300fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54890/0 (socket says 172.21.15.32:54890) 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a35128700 1 -- 172.21.15.32:0/2556838236 learned_addr learned my addr 172.21.15.32:0/2556838236 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a35929700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a300fde90 0x7f5a300fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a35128700 1 -- 172.21.15.32:0/2556838236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a300fde90 msgr2=0x7f5a300fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a35128700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a300fde90 0x7f5a300fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a35128700 1 -- 172.21.15.32:0/2556838236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c005040 con 0x7f5a300fe830 2024-04-06T14:31:28.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.357+0000 7f5a35929700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a300fde90 0x7f5a300fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:31:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.358+0000 7f5a35128700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fe830 0x7f5a300fc290 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000c830 tx=0x7f5a20004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.358+0000 7f5a267fc700 1 -- 172.21.15.32:0/2556838236 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a2000a760 con 0x7f5a300fe830 2024-04-06T14:31:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.358+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a30119df0 con 0x7f5a300fe830 2024-04-06T14:31:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.358+0000 7f5a267fc700 1 -- 172.21.15.32:0/2556838236 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a2000ad70 con 0x7f5a300fe830 2024-04-06T14:31:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.358+0000 7f5a267fc700 1 -- 172.21.15.32:0/2556838236 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a200113a0 con 0x7f5a300fe830 2024-04-06T14:31:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.358+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3011a1f0 con 0x7f5a300fe830 2024-04-06T14:31:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.359+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a30065e30 con 0x7f5a300fe830 2024-04-06T14:31:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.359+0000 7f5a267fc700 1 -- 172.21.15.32:0/2556838236 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5a20012400 con 0x7f5a300fe830 2024-04-06T14:31:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.360+0000 7f5a267fc700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a1c06cc20 0x7f5a1c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.360+0000 7f5a267fc700 1 -- 172.21.15.32:0/2556838236 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5a2008c0b0 con 0x7f5a300fe830 2024-04-06T14:31:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.360+0000 7f5a35929700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a1c06cc20 0x7f5a1c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.360+0000 7f5a35929700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a1c06cc20 0x7f5a1c06f0d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00b3b0 tx=0x7f5a2c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:28.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.364+0000 7f5a267fc700 1 -- 172.21.15.32:0/2556838236 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5a200560f0 con 0x7f5a300fe830 2024-04-06T14:31:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.649+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5a3004bed0 con 0x7f5a300fe830 2024-04-06T14:31:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.650+0000 7f5a267fc700 1 -- 172.21.15.32:0/2556838236 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f5a20059710 con 0x7f5a300fe830 2024-04-06T14:31:28.650 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:31:28.650 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:31:28.650 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:31:28.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:31:28.650 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:31:28.651 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:31:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a1c06cc20 msgr2=0x7f5a1c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a1c06cc20 0x7f5a1c06f0d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00b3b0 tx=0x7f5a2c009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fe830 msgr2=0x7f5a300fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fe830 0x7f5a300fc290 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000c830 tx=0x7f5a20004470 comp rx=0 tx=0).stop 2024-04-06T14:31:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 shutdown_connections 2024-04-06T14:31:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a1c06cc20 0x7f5a1c06f0d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a300fde90 0x7f5a300fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 --2- 172.21.15.32:0/2556838236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a300fe830 0x7f5a300fc290 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 >> 172.21.15.32:0/2556838236 conn(0x7f5a30072bc0 msgr2=0x7f5a300ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 shutdown_connections 2024-04-06T14:31:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.652+0000 7f5a37b8d700 1 -- 172.21.15.32:0/2556838236 wait complete. 2024-04-06T14:31:28.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.755+0000 7f4f3e422700 1 -- 172.21.15.32:0/512024952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 msgr2=0x7f4f38102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.755+0000 7f4f3e422700 1 --2- 172.21.15.32:0/512024952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 0x7f4f38102140 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4f28004940 tx=0x7f4f28009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.755+0000 7f4f3e422700 1 -- 172.21.15.32:0/512024952 shutdown_connections 2024-04-06T14:31:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.755+0000 7f4f3e422700 1 --2- 172.21.15.32:0/512024952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f38102710 0x7f4f380fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.755+0000 7f4f3e422700 1 --2- 172.21.15.32:0/512024952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 0x7f4f38102140 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.755+0000 7f4f3e422700 1 -- 172.21.15.32:0/512024952 >> 172.21.15.32:0/512024952 conn(0x7f4f38072bc0 msgr2=0x7f4f38072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.755+0000 7f4f3e422700 1 -- 172.21.15.32:0/512024952 shutdown_connections 2024-04-06T14:31:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.755+0000 7f4f3e422700 1 -- 172.21.15.32:0/512024952 wait complete. 2024-04-06T14:31:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.755+0000 7f4f3e422700 1 Processor -- start 2024-04-06T14:31:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f3e422700 1 -- start start 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f3e422700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 0x7f4f38113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f3e422700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f38102710 0x7f4f38114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f3e422700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38114650 con 0x7f4f38101d70 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f3e422700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3810da40 con 0x7f4f38102710 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 0x7f4f38113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 0x7f4f38113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54906/0 (socket says 172.21.15.32:54906) 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f37fff700 1 -- 172.21.15.32:0/535543591 learned_addr learned my addr 172.21.15.32:0/535543591 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f377fe700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f38102710 0x7f4f38114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f37fff700 1 -- 172.21.15.32:0/535543591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f38102710 msgr2=0x7f4f38114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f37fff700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f38102710 0x7f4f38114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f37fff700 1 -- 172.21.15.32:0/535543591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f28005040 con 0x7f4f38101d70 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f377fe700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f38102710 0x7f4f38114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.756+0000 7f4f37fff700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 0x7f4f38113ac0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009390 tx=0x7f4f28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.757+0000 7f4f34ff9700 1 -- 172.21.15.32:0/535543591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f2800e040 con 0x7f4f38101d70 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.757+0000 7f4f34ff9700 1 -- 172.21.15.32:0/535543591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f28013420 con 0x7f4f38101d70 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.757+0000 7f4f34ff9700 1 -- 172.21.15.32:0/535543591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f2801c630 con 0x7f4f38101d70 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.757+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3810dcc0 con 0x7f4f38101d70 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.757+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3810e280 con 0x7f4f38101d70 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.758+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f380fd4c0 con 0x7f4f38101d70 2024-04-06T14:31:28.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.758+0000 7f4f34ff9700 1 -- 172.21.15.32:0/535543591 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4f280139e0 con 0x7f4f38101d70 2024-04-06T14:31:28.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.759+0000 7f4f34ff9700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f2006cc30 0x7f4f2006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:28.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.759+0000 7f4f377fe700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f2006cc30 0x7f4f2006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:28.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.759+0000 7f4f34ff9700 1 -- 172.21.15.32:0/535543591 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4f2808d260 con 0x7f4f38101d70 2024-04-06T14:31:28.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.759+0000 7f4f377fe700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f2006cc30 0x7f4f2006f0e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009fd0 tx=0x7f4f2c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:28.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:28.763+0000 7f4f34ff9700 1 -- 172.21.15.32:0/535543591 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4f2805ade0 con 0x7f4f38101d70 2024-04-06T14:31:28.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:28 smithi086 ceph-mon[110975]: pgmap v339: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:28.979 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:28 smithi032 ceph-mon[147414]: pgmap v339: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.011+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4f3804bed0 con 0x7f4f38101d70 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.011+0000 7f4f34ff9700 1 -- 172.21.15.32:0/535543591 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f4f2805a970 con 0x7f4f38101d70 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:31:29.012 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:31:29.013 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:31:29.014 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:31:29.015 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:31:29.015 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:31:29.015 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:31:29.015 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:31:29.015 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:31:29.015 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:31:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f2006cc30 msgr2=0x7f4f2006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f2006cc30 0x7f4f2006f0e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009fd0 tx=0x7f4f2c009ee0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 msgr2=0x7f4f38113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 0x7f4f38113ac0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009390 tx=0x7f4f28009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 shutdown_connections 2024-04-06T14:31:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4f2006cc30 0x7f4f2006f0e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f38101d70 0x7f4f38113ac0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 --2- 172.21.15.32:0/535543591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f38102710 0x7f4f38114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 >> 172.21.15.32:0/535543591 conn(0x7f4f38072bc0 msgr2=0x7f4f380659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 shutdown_connections 2024-04-06T14:31:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.013+0000 7f4f3e422700 1 -- 172.21.15.32:0/535543591 wait complete. 2024-04-06T14:31:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:31:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 -- 172.21.15.32:0/118547736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 msgr2=0x7f9e40103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 --2- 172.21.15.32:0/118547736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 0x7f9e40103240 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9e34004940 tx=0x7f9e34009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 -- 172.21.15.32:0/118547736 shutdown_connections 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 --2- 172.21.15.32:0/118547736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e400fb4a0 0x7f9e400fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 --2- 172.21.15.32:0/118547736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 0x7f9e40103240 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 -- 172.21.15.32:0/118547736 >> 172.21.15.32:0/118547736 conn(0x7f9e400714c0 msgr2=0x7f9e400718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 -- 172.21.15.32:0/118547736 shutdown_connections 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 -- 172.21.15.32:0/118547736 wait complete. 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 Processor -- start 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.118+0000 7f9e47228700 1 -- start start 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e47228700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e400fb4a0 0x7f9e400fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e47228700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 0x7f9e400fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e47228700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e400fd100 con 0x7f9e40102e70 2024-04-06T14:31:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e47228700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e400fd240 con 0x7f9e400fb4a0 2024-04-06T14:31:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e3ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 0x7f9e400fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e3ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 0x7f9e400fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54920/0 (socket says 172.21.15.32:54920) 2024-04-06T14:31:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e3ffff700 1 -- 172.21.15.32:0/314286330 learned_addr learned my addr 172.21.15.32:0/314286330 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e44fc4700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e400fb4a0 0x7f9e400fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e3ffff700 1 -- 172.21.15.32:0/314286330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e400fb4a0 msgr2=0x7f9e400fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e3ffff700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e400fb4a0 0x7f9e400fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e3ffff700 1 -- 172.21.15.32:0/314286330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e34005040 con 0x7f9e40102e70 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e3ffff700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 0x7f9e400fcbc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c002a80 tx=0x7f9e2c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.119+0000 7f9e3d7fa700 1 -- 172.21.15.32:0/314286330 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e2c007cc0 con 0x7f9e40102e70 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.120+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e400fd4f0 con 0x7f9e40102e70 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.120+0000 7f9e3d7fa700 1 -- 172.21.15.32:0/314286330 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e2c00f040 con 0x7f9e40102e70 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.120+0000 7f9e3d7fa700 1 -- 172.21.15.32:0/314286330 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e2c00da60 con 0x7f9e40102e70 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.120+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4011a140 con 0x7f9e40102e70 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.121+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e40100110 con 0x7f9e40102e70 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.121+0000 7f9e3d7fa700 1 -- 172.21.15.32:0/314286330 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9e2c00dbc0 con 0x7f9e40102e70 2024-04-06T14:31:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.121+0000 7f9e3d7fa700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9e3006cc30 0x7f9e3006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:29.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.122+0000 7f9e3d7fa700 1 -- 172.21.15.32:0/314286330 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9e2c08bc70 con 0x7f9e40102e70 2024-04-06T14:31:29.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.122+0000 7f9e44fc4700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9e3006cc30 0x7f9e3006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:29.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.122+0000 7f9e44fc4700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9e3006cc30 0x7f9e3006f0e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009fd0 tx=0x7f9e34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:29.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.126+0000 7f9e3d7fa700 1 -- 172.21.15.32:0/314286330 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9e2c055c30 con 0x7f9e40102e70 2024-04-06T14:31:29.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.348+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e400fd4f0 con 0x7f9e3006cc30 2024-04-06T14:31:29.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.349+0000 7f9e3d7fa700 1 -- 172.21.15.32:0/314286330 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9e400fd4f0 con 0x7f9e3006cc30 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:31:29.350 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:31:29.351 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:31:29.351 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:31:29.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.351+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9e3006cc30 msgr2=0x7f9e3006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.351+0000 7f9e47228700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9e3006cc30 0x7f9e3006f0e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009fd0 tx=0x7f9e34009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.351+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 msgr2=0x7f9e400fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.351+0000 7f9e47228700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 0x7f9e400fcbc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c002a80 tx=0x7f9e2c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.351+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 shutdown_connections 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.351+0000 7f9e47228700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9e3006cc30 0x7f9e3006f0e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.351+0000 7f9e47228700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e400fb4a0 0x7f9e400fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.351+0000 7f9e47228700 1 --2- 172.21.15.32:0/314286330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e40102e70 0x7f9e400fcbc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.351+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 >> 172.21.15.32:0/314286330 conn(0x7f9e400714c0 msgr2=0x7f9e400fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.352+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 shutdown_connections 2024-04-06T14:31:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.352+0000 7f9e47228700 1 -- 172.21.15.32:0/314286330 wait complete. 2024-04-06T14:31:29.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 -- 172.21.15.32:0/2385776832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc1014f0 msgr2=0x7ff4bc1018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 --2- 172.21.15.32:0/2385776832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc1014f0 0x7ff4bc1018c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac004940 tx=0x7ff4ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:29.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 -- 172.21.15.32:0/2385776832 shutdown_connections 2024-04-06T14:31:29.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 --2- 172.21.15.32:0/2385776832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4bc0fb4a0 0x7ff4bc0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 --2- 172.21.15.32:0/2385776832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc1014f0 0x7ff4bc1018c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 -- 172.21.15.32:0/2385776832 >> 172.21.15.32:0/2385776832 conn(0x7ff4bc0714c0 msgr2=0x7ff4bc0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:29.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 -- 172.21.15.32:0/2385776832 shutdown_connections 2024-04-06T14:31:29.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 -- 172.21.15.32:0/2385776832 wait complete. 2024-04-06T14:31:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 Processor -- start 2024-04-06T14:31:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.455+0000 7ff4c2924700 1 -- start start 2024-04-06T14:31:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4c2924700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc0fb4a0 0x7ff4bc10ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4c2924700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4bc1014f0 0x7ff4bc10ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4c2924700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc10f590 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4c2924700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc10f6d0 con 0x7ff4bc1014f0 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4bbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc0fb4a0 0x7ff4bc10ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4bbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc0fb4a0 0x7ff4bc10ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54946/0 (socket says 172.21.15.32:54946) 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4bbfff700 1 -- 172.21.15.32:0/537582057 learned_addr learned my addr 172.21.15.32:0/537582057 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4bb7fe700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4bc1014f0 0x7ff4bc10ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4bbfff700 1 -- 172.21.15.32:0/537582057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4bc1014f0 msgr2=0x7ff4bc10ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4bbfff700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4bc1014f0 0x7ff4bc10ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4bbfff700 1 -- 172.21.15.32:0/537582057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac005040 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4bbfff700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc0fb4a0 0x7ff4bc10ea00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0040b0 tx=0x7ff4ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4b8ff9700 1 -- 172.21.15.32:0/537582057 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4ac021070 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4b8ff9700 1 -- 172.21.15.32:0/537582057 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4ac013de0 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc117a70 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.456+0000 7ff4b8ff9700 1 -- 172.21.15.32:0/537582057 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4ac00e070 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.457+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc117f50 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.457+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc1038d0 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.458+0000 7ff4b8ff9700 1 -- 172.21.15.32:0/537582057 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff4ac013930 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.459+0000 7ff4b8ff9700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff4a406cc20 0x7ff4a406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.459+0000 7ff4b8ff9700 1 -- 172.21.15.32:0/537582057 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff4ac08da40 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.459+0000 7ff4bb7fe700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff4a406cc20 0x7ff4a406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.459+0000 7ff4bb7fe700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff4a406cc20 0x7ff4a406f0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0f9e90 tx=0x7ff4b000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:29.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.463+0000 7ff4b8ff9700 1 -- 172.21.15.32:0/537582057 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff4ac057a80 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.751+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff4bc0fcbd0 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.751+0000 7ff4b8ff9700 1 -- 172.21.15.32:0/537582057 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7ff4ac05b0a0 con 0x7ff4bc0fb4a0 2024-04-06T14:31:29.752 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:31:29.752 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:31:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:31:29.752 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:31:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:31:29.752 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:31:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:31:29.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.753+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff4a406cc20 msgr2=0x7ff4a406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.753+0000 7ff4c2924700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff4a406cc20 0x7ff4a406f0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0f9e90 tx=0x7ff4b000b040 comp rx=0 tx=0).stop 2024-04-06T14:31:29.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.753+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc0fb4a0 msgr2=0x7ff4bc10ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:29.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.753+0000 7ff4c2924700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc0fb4a0 0x7ff4bc10ea00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0040b0 tx=0x7ff4ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:29.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.753+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 shutdown_connections 2024-04-06T14:31:29.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.753+0000 7ff4c2924700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff4a406cc20 0x7ff4a406f0d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.753+0000 7ff4c2924700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4bc0fb4a0 0x7ff4bc10ea00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.753+0000 7ff4c2924700 1 --2- 172.21.15.32:0/537582057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4bc1014f0 0x7ff4bc10ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.753+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 >> 172.21.15.32:0/537582057 conn(0x7ff4bc0714c0 msgr2=0x7ff4bc0fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.754+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 shutdown_connections 2024-04-06T14:31:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:29.754+0000 7ff4c2924700 1 -- 172.21.15.32:0/537582057 wait complete. 2024-04-06T14:31:29.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:29 smithi086 ceph-mon[110975]: from='client.34672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:31:29.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:29 smithi086 ceph-mon[110975]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:31:29.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:29 smithi086 ceph-mon[110975]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:31:29.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:29 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2556838236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:31:29.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:29 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/535543591' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:31:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:29 smithi032 ceph-mon[147414]: from='client.34672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:31:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:29 smithi032 ceph-mon[147414]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:31:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:29 smithi032 ceph-mon[147414]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:31:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:29 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2556838236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:31:29.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:29 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/535543591' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:31:30.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:30 smithi086 ceph-mon[110975]: pgmap v340: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:30.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:30 smithi086 ceph-mon[110975]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:31:30.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:30 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/537582057' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:31:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:30 smithi032 ceph-mon[147414]: pgmap v340: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:30 smithi032 ceph-mon[147414]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:31:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:30 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/537582057' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:31:32.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:32 smithi086 ceph-mon[110975]: pgmap v341: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:32.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:32 smithi032 ceph-mon[147414]: pgmap v341: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:34.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:34 smithi086 ceph-mon[110975]: pgmap v342: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:34.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:34 smithi032 ceph-mon[147414]: pgmap v342: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:36.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:36 smithi086 ceph-mon[110975]: pgmap v343: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:36.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:36 smithi032 ceph-mon[147414]: pgmap v343: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:38.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:38 smithi086 ceph-mon[110975]: pgmap v344: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:38.878 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:31:38.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:38 smithi032 ceph-mon[147414]: pgmap v344: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:38.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:31:40.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:40 smithi086 ceph-mon[110975]: pgmap v345: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:40.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:40 smithi032 ceph-mon[147414]: pgmap v345: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:41.877 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:41 smithi086 ceph-mon[110975]: pgmap v346: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:41.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:41 smithi032 ceph-mon[147414]: pgmap v346: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:44.127 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:43 smithi086 ceph-mon[110975]: pgmap v347: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:44.230 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:43 smithi032 ceph-mon[147414]: pgmap v347: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:46.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:46 smithi086 ceph-mon[110975]: pgmap v348: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:46 smithi032 ceph-mon[147414]: pgmap v348: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:48.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:48 smithi086 ceph-mon[110975]: pgmap v349: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:48.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:48 smithi032 ceph-mon[147414]: pgmap v349: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:50.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:50 smithi086 ceph-mon[110975]: pgmap v350: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:50 smithi032 ceph-mon[147414]: pgmap v350: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:52.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:52 smithi086 ceph-mon[110975]: pgmap v351: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:52 smithi032 ceph-mon[147414]: pgmap v351: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:53.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:31:53.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:31:54.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:54 smithi086 ceph-mon[110975]: pgmap v352: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:54.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:54 smithi032 ceph-mon[147414]: pgmap v352: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:56.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:56 smithi086 ceph-mon[110975]: pgmap v353: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:56.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:56 smithi032 ceph-mon[147414]: pgmap v353: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:58.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:31:58 smithi086 ceph-mon[110975]: pgmap v354: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:31:58 smithi032 ceph-mon[147414]: pgmap v354: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:31:59.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.871+0000 7f8386125700 1 -- 172.21.15.32:0/1531702040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8380102e00 msgr2=0x7f83801031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.871+0000 7f8386125700 1 --2- 172.21.15.32:0/1531702040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8380102e00 0x7f83801031d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8370004940 tx=0x7f8370009e30 comp rx=0 tx=0).stop 2024-04-06T14:31:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.871+0000 7f8386125700 1 -- 172.21.15.32:0/1531702040 shutdown_connections 2024-04-06T14:31:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.871+0000 7f8386125700 1 --2- 172.21.15.32:0/1531702040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83801037a0 0x7f83800fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.871+0000 7f8386125700 1 --2- 172.21.15.32:0/1531702040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8380102e00 0x7f83801031d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.871+0000 7f8386125700 1 -- 172.21.15.32:0/1531702040 >> 172.21.15.32:0/1531702040 conn(0x7f83800f89b0 msgr2=0x7f83800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:31:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.871+0000 7f8386125700 1 -- 172.21.15.32:0/1531702040 shutdown_connections 2024-04-06T14:31:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.871+0000 7f8386125700 1 -- 172.21.15.32:0/1531702040 wait complete. 2024-04-06T14:31:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.871+0000 7f8386125700 1 Processor -- start 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.872+0000 7f8386125700 1 -- start start 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.872+0000 7f8386125700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8380102e00 0x7f8380113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.872+0000 7f8386125700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83801037a0 0x7f8380114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.872+0000 7f8386125700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83801146b0 con 0x7f83801037a0 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.872+0000 7f8386125700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838010daa0 con 0x7f8380102e00 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.872+0000 7f837effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83801037a0 0x7f8380114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.872+0000 7f837effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83801037a0 0x7f8380114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47584/0 (socket says 172.21.15.32:47584) 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.872+0000 7f837effd700 1 -- 172.21.15.32:0/2842945028 learned_addr learned my addr 172.21.15.32:0/2842945028 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.872+0000 7f837f7fe700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8380102e00 0x7f8380113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:59.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f837effd700 1 -- 172.21.15.32:0/2842945028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8380102e00 msgr2=0x7f8380113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f837effd700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8380102e00 0x7f8380113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f837effd700 1 -- 172.21.15.32:0/2842945028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8370005040 con 0x7f83801037a0 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f837f7fe700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8380102e00 0x7f8380113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f837effd700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83801037a0 0x7f8380114060 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f837400c800 tx=0x7f8374004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f8367fff700 1 -- 172.21.15.32:0/2842945028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f837400a760 con 0x7f83801037a0 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f8367fff700 1 -- 172.21.15.32:0/2842945028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f837400ad70 con 0x7f83801037a0 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838010dd80 con 0x7f83801037a0 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f8367fff700 1 -- 172.21.15.32:0/2842945028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83740113a0 con 0x7f83801037a0 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.873+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838010e340 con 0x7f83801037a0 2024-04-06T14:31:59.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.874+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83800fe550 con 0x7f83801037a0 2024-04-06T14:31:59.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.875+0000 7f8367fff700 1 -- 172.21.15.32:0/2842945028 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8374012400 con 0x7f83801037a0 2024-04-06T14:31:59.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.875+0000 7f8367fff700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f836806cc30 0x7f836806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:31:59.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.875+0000 7f8367fff700 1 -- 172.21.15.32:0/2842945028 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f837408c060 con 0x7f83801037a0 2024-04-06T14:31:59.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.875+0000 7f837f7fe700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f836806cc30 0x7f836806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:31:59.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.876+0000 7f837f7fe700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f836806cc30 0x7f836806f0e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f837000b3b0 tx=0x7f8370009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:31:59.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:31:59.880+0000 7f8367fff700 1 -- 172.21.15.32:0/2842945028 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8374059b30 con 0x7f83801037a0 2024-04-06T14:32:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.102+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8380109e60 con 0x7f836806cc30 2024-04-06T14:32:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.103+0000 7f8367fff700 1 -- 172.21.15.32:0/2842945028 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f83700096f0 con 0x7f836806cc30 2024-04-06T14:32:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.105+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f836806cc30 msgr2=0x7f836806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.105+0000 7f8386125700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f836806cc30 0x7f836806f0e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f837000b3b0 tx=0x7f8370009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.105+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83801037a0 msgr2=0x7f8380114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.105+0000 7f8386125700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83801037a0 0x7f8380114060 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f837400c800 tx=0x7f8374004470 comp rx=0 tx=0).stop 2024-04-06T14:32:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.105+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 shutdown_connections 2024-04-06T14:32:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.105+0000 7f8386125700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f836806cc30 0x7f836806f0e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.105+0000 7f8386125700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8380102e00 0x7f8380113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.105+0000 7f8386125700 1 --2- 172.21.15.32:0/2842945028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83801037a0 0x7f8380114060 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.105+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 >> 172.21.15.32:0/2842945028 conn(0x7f83800f89b0 msgr2=0x7f83800f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.106+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 shutdown_connections 2024-04-06T14:32:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.106+0000 7f8386125700 1 -- 172.21.15.32:0/2842945028 wait complete. 2024-04-06T14:32:00.116 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:32:00.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.210+0000 7f1481516700 1 -- 172.21.15.32:0/3342480421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 msgr2=0x7f147c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.210+0000 7f1481516700 1 --2- 172.21.15.32:0/3342480421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 0x7f147c103240 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1470004940 tx=0x7f1470009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:00.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.210+0000 7f1481516700 1 -- 172.21.15.32:0/3342480421 shutdown_connections 2024-04-06T14:32:00.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.210+0000 7f1481516700 1 --2- 172.21.15.32:0/3342480421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f147c0fb4a0 0x7f147c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.210+0000 7f1481516700 1 --2- 172.21.15.32:0/3342480421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 0x7f147c103240 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.210+0000 7f1481516700 1 -- 172.21.15.32:0/3342480421 >> 172.21.15.32:0/3342480421 conn(0x7f147c0714c0 msgr2=0x7f147c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:00.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.210+0000 7f1481516700 1 -- 172.21.15.32:0/3342480421 shutdown_connections 2024-04-06T14:32:00.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.210+0000 7f1481516700 1 -- 172.21.15.32:0/3342480421 wait complete. 2024-04-06T14:32:00.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.210+0000 7f1481516700 1 Processor -- start 2024-04-06T14:32:00.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f1481516700 1 -- start start 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f1481516700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f147c0fb4a0 0x7f147c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f1481516700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 0x7f147c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f1481516700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c114600 con 0x7f147c102e70 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f1481516700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c10d9f0 con 0x7f147c0fb4a0 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f147a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 0x7f147c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f147a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 0x7f147c113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47610/0 (socket says 172.21.15.32:47610) 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f147a7fc700 1 -- 172.21.15.32:0/2520027700 learned_addr learned my addr 172.21.15.32:0/2520027700 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f147affd700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f147c0fb4a0 0x7f147c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f147a7fc700 1 -- 172.21.15.32:0/2520027700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f147c0fb4a0 msgr2=0x7f147c113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f147a7fc700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f147c0fb4a0 0x7f147c113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f147a7fc700 1 -- 172.21.15.32:0/2520027700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470005040 con 0x7f147c102e70 2024-04-06T14:32:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f147a7fc700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 0x7f147c113fb0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f146c002a50 tx=0x7f146c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:00.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.211+0000 7f14637fe700 1 -- 172.21.15.32:0/2520027700 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146c007cc0 con 0x7f147c102e70 2024-04-06T14:32:00.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.212+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c10dcd0 con 0x7f147c102e70 2024-04-06T14:32:00.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.212+0000 7f14637fe700 1 -- 172.21.15.32:0/2520027700 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f146c00f040 con 0x7f147c102e70 2024-04-06T14:32:00.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.212+0000 7f14637fe700 1 -- 172.21.15.32:0/2520027700 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146c00d9c0 con 0x7f147c102e70 2024-04-06T14:32:00.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.212+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c10e290 con 0x7f147c102e70 2024-04-06T14:32:00.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.212+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147c100110 con 0x7f147c102e70 2024-04-06T14:32:00.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.213+0000 7f14637fe700 1 -- 172.21.15.32:0/2520027700 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f146c00db20 con 0x7f147c102e70 2024-04-06T14:32:00.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.214+0000 7f14637fe700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146406cc30 0x7f146406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:00.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.214+0000 7f14637fe700 1 -- 172.21.15.32:0/2520027700 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f146c08bd70 con 0x7f147c102e70 2024-04-06T14:32:00.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.214+0000 7f147affd700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146406cc30 0x7f146406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:00.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.214+0000 7f147affd700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146406cc30 0x7f146406f0e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1470000c00 tx=0x7f1470009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:00.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.218+0000 7f14637fe700 1 -- 172.21.15.32:0/2520027700 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f146c0597c0 con 0x7f147c102e70 2024-04-06T14:32:00.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:00 smithi086 ceph-mon[110975]: pgmap v355: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:00.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.439+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f147c10f130 con 0x7f146406cc30 2024-04-06T14:32:00.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.440+0000 7f14637fe700 1 -- 172.21.15.32:0/2520027700 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f147c10f130 con 0x7f146406cc30 2024-04-06T14:32:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146406cc30 msgr2=0x7f146406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146406cc30 0x7f146406f0e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1470000c00 tx=0x7f1470009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 msgr2=0x7f147c113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 0x7f147c113fb0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f146c002a50 tx=0x7f146c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 shutdown_connections 2024-04-06T14:32:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146406cc30 0x7f146406f0e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f147c0fb4a0 0x7f147c113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 --2- 172.21.15.32:0/2520027700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c102e70 0x7f147c113fb0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 >> 172.21.15.32:0/2520027700 conn(0x7f147c0714c0 msgr2=0x7f147c0fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 shutdown_connections 2024-04-06T14:32:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.442+0000 7f1481516700 1 -- 172.21.15.32:0/2520027700 wait complete. 2024-04-06T14:32:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:00 smithi032 ceph-mon[147414]: pgmap v355: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:00.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.545+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2977707738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 msgr2=0x7fbb28102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.545+0000 7fbb2e72d700 1 --2- 172.21.15.32:0/2977707738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 0x7fbb28102150 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c004940 tx=0x7fbb1c009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:00.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.545+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2977707738 shutdown_connections 2024-04-06T14:32:00.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.545+0000 7fbb2e72d700 1 --2- 172.21.15.32:0/2977707738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb28102720 0x7fbb280fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.545+0000 7fbb2e72d700 1 --2- 172.21.15.32:0/2977707738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 0x7fbb28102150 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.545+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2977707738 >> 172.21.15.32:0/2977707738 conn(0x7fbb280f78f0 msgr2=0x7fbb280f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:00.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.545+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2977707738 shutdown_connections 2024-04-06T14:32:00.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.545+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2977707738 wait complete. 2024-04-06T14:32:00.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.546+0000 7fbb2e72d700 1 Processor -- start 2024-04-06T14:32:00.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.546+0000 7fbb2e72d700 1 -- start start 2024-04-06T14:32:00.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.546+0000 7fbb2e72d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 0x7fbb28113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:00.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.546+0000 7fbb2e72d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb28102720 0x7fbb28113da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:00.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.546+0000 7fbb2e72d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb281143f0 con 0x7fbb28101d80 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.546+0000 7fbb2e72d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2810d9f0 con 0x7fbb28102720 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.546+0000 7fbb27fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 0x7fbb28113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.546+0000 7fbb27fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 0x7fbb28113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47632/0 (socket says 172.21.15.32:47632) 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.546+0000 7fbb27fff700 1 -- 172.21.15.32:0/2010813636 learned_addr learned my addr 172.21.15.32:0/2010813636 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb277fe700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb28102720 0x7fbb28113da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb27fff700 1 -- 172.21.15.32:0/2010813636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb28102720 msgr2=0x7fbb28113da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb27fff700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb28102720 0x7fbb28113da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb27fff700 1 -- 172.21.15.32:0/2010813636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c005040 con 0x7fbb28101d80 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb27fff700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 0x7fbb28113860 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00b3b0 tx=0x7fbb1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb24ff9700 1 -- 172.21.15.32:0/2010813636 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb1c00e040 con 0x7fbb28101d80 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2810dc70 con 0x7fbb28101d80 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb24ff9700 1 -- 172.21.15.32:0/2010813636 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb1c013420 con 0x7fbb28101d80 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb24ff9700 1 -- 172.21.15.32:0/2010813636 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb1c01c610 con 0x7fbb28101d80 2024-04-06T14:32:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.547+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2810e230 con 0x7fbb28101d80 2024-04-06T14:32:00.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.548+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb280fd4d0 con 0x7fbb28101d80 2024-04-06T14:32:00.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.549+0000 7fbb24ff9700 1 -- 172.21.15.32:0/2010813636 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbb1c013a00 con 0x7fbb28101d80 2024-04-06T14:32:00.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.549+0000 7fbb24ff9700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb1006cc20 0x7fbb1006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:00.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.550+0000 7fbb24ff9700 1 -- 172.21.15.32:0/2010813636 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbb1c08dcd0 con 0x7fbb28101d80 2024-04-06T14:32:00.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.550+0000 7fbb277fe700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb1006cc20 0x7fbb1006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:00.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.550+0000 7fbb277fe700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb1006cc20 0x7fbb1006f0d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbb2810ebb0 tx=0x7fbb18009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:00.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.554+0000 7fbb24ff9700 1 -- 172.21.15.32:0/2010813636 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbb1c05b7a0 con 0x7fbb28101d80 2024-04-06T14:32:00.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.765+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb28108d40 con 0x7fbb1006cc20 2024-04-06T14:32:00.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.774+0000 7fbb24ff9700 1 -- 172.21.15.32:0/2010813636 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbb28108d40 con 0x7fbb1006cc20 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (12m) 5m ago 23m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 5m ago 23m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (9m) 5m ago 22m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (12m) 5m ago 23m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (6m) 5m ago 17m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (5m) 5m ago 17m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (5m) 5m ago 17m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (17m) 5m ago 17m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (14m) 5m ago 25m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:32:00.775 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (13m) 5m ago 22m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 5m ago 25m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (10m) 5m ago 22m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (13m) 5m ago 23m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (12m) 5m ago 22m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 5m ago 21m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 5m ago 21m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (8m) 5m ago 20m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (7m) 5m ago 20m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (6m) 5m ago 19m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (6m) 5m ago 19m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:32:00.776 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (12m) 5m ago 23m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:32:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.776+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb1006cc20 msgr2=0x7fbb1006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.776+0000 7fbb2e72d700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb1006cc20 0x7fbb1006f0d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbb2810ebb0 tx=0x7fbb18009040 comp rx=0 tx=0).stop 2024-04-06T14:32:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.776+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 msgr2=0x7fbb28113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.776+0000 7fbb2e72d700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 0x7fbb28113860 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00b3b0 tx=0x7fbb1c009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.777+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 shutdown_connections 2024-04-06T14:32:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.777+0000 7fbb2e72d700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbb1006cc20 0x7fbb1006f0d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.777+0000 7fbb2e72d700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb28101d80 0x7fbb28113860 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.777+0000 7fbb2e72d700 1 --2- 172.21.15.32:0/2010813636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb28102720 0x7fbb28113da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.777+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 >> 172.21.15.32:0/2010813636 conn(0x7fbb280f78f0 msgr2=0x7fbb280665b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.777+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 shutdown_connections 2024-04-06T14:32:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.777+0000 7fbb2e72d700 1 -- 172.21.15.32:0/2010813636 wait complete. 2024-04-06T14:32:00.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.879+0000 7f153af87700 1 -- 172.21.15.32:0/1272570089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 msgr2=0x7f15341020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.879+0000 7f153af87700 1 --2- 172.21.15.32:0/1272570089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 0x7f15341020e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1528004940 tx=0x7f1528009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:00.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.879+0000 7f153af87700 1 -- 172.21.15.32:0/1272570089 shutdown_connections 2024-04-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.879+0000 7f153af87700 1 --2- 172.21.15.32:0/1272570089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15341026b0 0x7f15340fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.879+0000 7f153af87700 1 --2- 172.21.15.32:0/1272570089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 0x7f15341020e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.879+0000 7f153af87700 1 -- 172.21.15.32:0/1272570089 >> 172.21.15.32:0/1272570089 conn(0x7f15340f7880 msgr2=0x7f15340f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.879+0000 7f153af87700 1 -- 172.21.15.32:0/1272570089 shutdown_connections 2024-04-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.879+0000 7f153af87700 1 -- 172.21.15.32:0/1272570089 wait complete. 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f153af87700 1 Processor -- start 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f153af87700 1 -- start start 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f153af87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 0x7f153410eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f153af87700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15341026b0 0x7f153410f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f153af87700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153410f940 con 0x7f1534101d10 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f153af87700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153410fa80 con 0x7f15341026b0 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f1538d23700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 0x7f153410eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f1538d23700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 0x7f153410eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47654/0 (socket says 172.21.15.32:47654) 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f1538d23700 1 -- 172.21.15.32:0/3871802576 learned_addr learned my addr 172.21.15.32:0/3871802576 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.880+0000 7f1533fff700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15341026b0 0x7f153410f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.881+0000 7f1538d23700 1 -- 172.21.15.32:0/3871802576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15341026b0 msgr2=0x7f153410f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.881+0000 7f1538d23700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15341026b0 0x7f153410f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.881+0000 7f1538d23700 1 -- 172.21.15.32:0/3871802576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1528005040 con 0x7f1534101d10 2024-04-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.881+0000 7f1538d23700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 0x7f153410eec0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f152800b3b0 tx=0x7f1528009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.881+0000 7f15317fa700 1 -- 172.21.15.32:0/3871802576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f152800e040 con 0x7f1534101d10 2024-04-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.881+0000 7f15317fa700 1 -- 172.21.15.32:0/3871802576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1528013420 con 0x7f1534101d10 2024-04-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.881+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1534117d70 con 0x7f1534101d10 2024-04-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.881+0000 7f15317fa700 1 -- 172.21.15.32:0/3871802576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f152801c610 con 0x7f1534101d10 2024-04-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.881+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15341182a0 con 0x7f1534101d10 2024-04-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.882+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15340fd460 con 0x7f1534101d10 2024-04-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.882+0000 7f15317fa700 1 -- 172.21.15.32:0/3871802576 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1528013a00 con 0x7f1534101d10 2024-04-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.883+0000 7f15317fa700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f152406cc20 0x7f152406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.883+0000 7f15317fa700 1 -- 172.21.15.32:0/3871802576 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f152808d190 con 0x7f1534101d10 2024-04-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.883+0000 7f1533fff700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f152406cc20 0x7f152406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.883+0000 7f1533fff700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f152406cc20 0x7f152406f0d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f15340668b0 tx=0x7f152000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:00.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:00.887+0000 7f15317fa700 1 -- 172.21.15.32:0/3871802576 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f15280571d0 con 0x7f1534101d10 2024-04-06T14:32:01.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.173+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1534066660 con 0x7f1534101d10 2024-04-06T14:32:01.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.174+0000 7f15317fa700 1 -- 172.21.15.32:0/3871802576 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f152805a7f0 con 0x7f1534101d10 2024-04-06T14:32:01.174 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:32:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:32:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:32:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:32:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:32:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:32:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:32:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:32:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:32:01.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f152406cc20 msgr2=0x7f152406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f152406cc20 0x7f152406f0d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f15340668b0 tx=0x7f152000b040 comp rx=0 tx=0).stop 2024-04-06T14:32:01.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 msgr2=0x7f153410eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 0x7f153410eec0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f152800b3b0 tx=0x7f1528009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:01.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 shutdown_connections 2024-04-06T14:32:01.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f152406cc20 0x7f152406f0d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1534101d10 0x7f153410eec0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 --2- 172.21.15.32:0/3871802576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15341026b0 0x7f153410f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 >> 172.21.15.32:0/3871802576 conn(0x7f15340f7880 msgr2=0x7f15340f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:01.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 shutdown_connections 2024-04-06T14:32:01.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.176+0000 7f153af87700 1 -- 172.21.15.32:0/3871802576 wait complete. 2024-04-06T14:32:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 -- 172.21.15.32:0/1387003504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 msgr2=0x7f49a41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 --2- 172.21.15.32:0/1387003504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 0x7f49a41020f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4994004940 tx=0x7f4994009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 -- 172.21.15.32:0/1387003504 shutdown_connections 2024-04-06T14:32:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 --2- 172.21.15.32:0/1387003504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49a41026c0 0x7f49a40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 --2- 172.21.15.32:0/1387003504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 0x7f49a41020f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 -- 172.21.15.32:0/1387003504 >> 172.21.15.32:0/1387003504 conn(0x7f49a40f7870 msgr2=0x7f49a40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 -- 172.21.15.32:0/1387003504 shutdown_connections 2024-04-06T14:32:01.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 -- 172.21.15.32:0/1387003504 wait complete. 2024-04-06T14:32:01.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 Processor -- start 2024-04-06T14:32:01.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.279+0000 7f49a9457700 1 -- start start 2024-04-06T14:32:01.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a9457700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 0x7f49a41138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:01.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a9457700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49a41026c0 0x7f49a4113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:01.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a9457700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4114440 con 0x7f49a4101d20 2024-04-06T14:32:01.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a9457700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a410da40 con 0x7f49a41026c0 2024-04-06T14:32:01.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 0x7f49a41138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 0x7f49a41138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47666/0 (socket says 172.21.15.32:47666) 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a2ffd700 1 -- 172.21.15.32:0/1673762130 learned_addr learned my addr 172.21.15.32:0/1673762130 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a2ffd700 1 -- 172.21.15.32:0/1673762130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49a41026c0 msgr2=0x7f49a4113df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a2ffd700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49a41026c0 0x7f49a4113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a2ffd700 1 -- 172.21.15.32:0/1673762130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4994005040 con 0x7f49a4101d20 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a2ffd700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 0x7f49a41138b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4994009390 tx=0x7f4994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f498b7fe700 1 -- 172.21.15.32:0/1673762130 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f499400e040 con 0x7f49a4101d20 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a410dcc0 con 0x7f49a4101d20 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f498b7fe700 1 -- 172.21.15.32:0/1673762130 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4994013420 con 0x7f49a4101d20 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f498b7fe700 1 -- 172.21.15.32:0/1673762130 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f499401c630 con 0x7f49a4101d20 2024-04-06T14:32:01.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.280+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a410e280 con 0x7f49a4101d20 2024-04-06T14:32:01.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.281+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a40fd470 con 0x7f49a4101d20 2024-04-06T14:32:01.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.282+0000 7f498b7fe700 1 -- 172.21.15.32:0/1673762130 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f49940139e0 con 0x7f49a4101d20 2024-04-06T14:32:01.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.282+0000 7f498b7fe700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f498c06cc20 0x7f498c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:01.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.282+0000 7f498b7fe700 1 -- 172.21.15.32:0/1673762130 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f499408d2b0 con 0x7f49a4101d20 2024-04-06T14:32:01.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.282+0000 7f49a27fc700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f498c06cc20 0x7f498c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:01.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.283+0000 7f49a27fc700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f498c06cc20 0x7f498c06f0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f49a410ec00 tx=0x7f4998007cc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:01.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.287+0000 7f498b7fe700 1 -- 172.21.15.32:0/1673762130 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f499405ae30 con 0x7f49a4101d20 2024-04-06T14:32:01.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:01 smithi086 ceph-mon[110975]: from='client.34700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:01.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:01 smithi086 ceph-mon[110975]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:01.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:01 smithi032 ceph-mon[147414]: from='client.34700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:01.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:01 smithi032 ceph-mon[147414]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.531+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f49a4108ce0 con 0x7f49a4101d20 2024-04-06T14:32:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.531+0000 7f498b7fe700 1 -- 172.21.15.32:0/1673762130 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f499405a9c0 con 0x7f49a4101d20 2024-04-06T14:32:01.531 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:32:01.531 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:32:01.532 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:32:01.533 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:32:01.534 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:32:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f498c06cc20 msgr2=0x7f498c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f498c06cc20 0x7f498c06f0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f49a410ec00 tx=0x7f4998007cc0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 msgr2=0x7f49a41138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 0x7f49a41138b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4994009390 tx=0x7f4994009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 shutdown_connections 2024-04-06T14:32:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f498c06cc20 0x7f498c06f0d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a4101d20 0x7f49a41138b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 --2- 172.21.15.32:0/1673762130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49a41026c0 0x7f49a4113df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 >> 172.21.15.32:0/1673762130 conn(0x7f49a40f7870 msgr2=0x7f49a40f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:01.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 shutdown_connections 2024-04-06T14:32:01.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.533+0000 7f49a9457700 1 -- 172.21.15.32:0/1673762130 wait complete. 2024-04-06T14:32:01.536 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:32:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.638+0000 7f02b6a35700 1 -- 172.21.15.32:0/2122049669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 msgr2=0x7f02b00fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.638+0000 7f02b6a35700 1 --2- 172.21.15.32:0/2122049669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 0x7f02b00fe1d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f02a0004940 tx=0x7f02a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.638+0000 7f02b6a35700 1 -- 172.21.15.32:0/2122049669 shutdown_connections 2024-04-06T14:32:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.638+0000 7f02b6a35700 1 --2- 172.21.15.32:0/2122049669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b00fe7a0 0x7f02b0106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.638+0000 7f02b6a35700 1 --2- 172.21.15.32:0/2122049669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 0x7f02b00fe1d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.638+0000 7f02b6a35700 1 -- 172.21.15.32:0/2122049669 >> 172.21.15.32:0/2122049669 conn(0x7f02b00f7880 msgr2=0x7f02b00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:01.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.638+0000 7f02b6a35700 1 -- 172.21.15.32:0/2122049669 shutdown_connections 2024-04-06T14:32:01.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.638+0000 7f02b6a35700 1 -- 172.21.15.32:0/2122049669 wait complete. 2024-04-06T14:32:01.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.638+0000 7f02b6a35700 1 Processor -- start 2024-04-06T14:32:01.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02b6a35700 1 -- start start 2024-04-06T14:32:01.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02b6a35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 0x7f02b0113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:01.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02b6a35700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b00fe7a0 0x7f02b01143b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:01.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02b6a35700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0114a00 con 0x7f02b00fde00 2024-04-06T14:32:01.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02b6a35700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b010ddd0 con 0x7f02b00fe7a0 2024-04-06T14:32:01.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 0x7f02b0113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 0x7f02b0113e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47688/0 (socket says 172.21.15.32:47688) 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02affff700 1 -- 172.21.15.32:0/601583415 learned_addr learned my addr 172.21.15.32:0/601583415 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02af7fe700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b00fe7a0 0x7f02b01143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02affff700 1 -- 172.21.15.32:0/601583415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b00fe7a0 msgr2=0x7f02b01143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02affff700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b00fe7a0 0x7f02b01143b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02affff700 1 -- 172.21.15.32:0/601583415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a0005040 con 0x7f02b00fde00 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02affff700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 0x7f02b0113e70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009380 tx=0x7f02a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.639+0000 7f02acff9700 1 -- 172.21.15.32:0/601583415 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02a000e040 con 0x7f02b00fde00 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.640+0000 7f02acff9700 1 -- 172.21.15.32:0/601583415 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02a0004290 con 0x7f02b00fde00 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.640+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b010e050 con 0x7f02b00fde00 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.640+0000 7f02acff9700 1 -- 172.21.15.32:0/601583415 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02a0013e90 con 0x7f02b00fde00 2024-04-06T14:32:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.640+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b010e590 con 0x7f02b00fde00 2024-04-06T14:32:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.641+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b00666f0 con 0x7f02b00fde00 2024-04-06T14:32:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.641+0000 7f02acff9700 1 -- 172.21.15.32:0/601583415 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f02a00136b0 con 0x7f02b00fde00 2024-04-06T14:32:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.641+0000 7f02acff9700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f029c06cc20 0x7f029c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.642+0000 7f02acff9700 1 -- 172.21.15.32:0/601583415 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f02a008d370 con 0x7f02b00fde00 2024-04-06T14:32:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.642+0000 7f02af7fe700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f029c06cc20 0x7f029c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.642+0000 7f02af7fe700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f029c06cc20 0x7f029c06f0d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f02b010f710 tx=0x7f029800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:01.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.646+0000 7f02acff9700 1 -- 172.21.15.32:0/601583415 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f02a005ae40 con 0x7f02b00fde00 2024-04-06T14:32:01.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.868+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f02b004bed0 con 0x7f029c06cc20 2024-04-06T14:32:01.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.869+0000 7f02acff9700 1 -- 172.21.15.32:0/601583415 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f02b004bed0 con 0x7f029c06cc20 2024-04-06T14:32:01.869 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:32:01.869 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:32:01.869 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:32:01.869 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:32:01.869 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:32:01.869 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:32:01.869 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:32:01.870 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:32:01.870 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:32:01.870 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:32:01.870 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:32:01.870 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:32:01.870 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:32:01.870 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:32:01.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f029c06cc20 msgr2=0x7f029c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f029c06cc20 0x7f029c06f0d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f02b010f710 tx=0x7f029800b040 comp rx=0 tx=0).stop 2024-04-06T14:32:01.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 msgr2=0x7f02b0113e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 0x7f02b0113e70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009380 tx=0x7f02a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:01.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 shutdown_connections 2024-04-06T14:32:01.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f029c06cc20 0x7f029c06f0d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b00fde00 0x7f02b0113e70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 --2- 172.21.15.32:0/601583415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b00fe7a0 0x7f02b01143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 >> 172.21.15.32:0/601583415 conn(0x7f02b00f7880 msgr2=0x7f02b00f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:01.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 shutdown_connections 2024-04-06T14:32:01.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.871+0000 7f02b6a35700 1 -- 172.21.15.32:0/601583415 wait complete. 2024-04-06T14:32:01.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.975+0000 7f7032175700 1 -- 172.21.15.32:0/722469732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 msgr2=0x7f702c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.975+0000 7f7032175700 1 --2- 172.21.15.32:0/722469732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 0x7f702c0fce20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f701c004940 tx=0x7f701c009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:01.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.975+0000 7f7032175700 1 -- 172.21.15.32:0/722469732 shutdown_connections 2024-04-06T14:32:01.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.975+0000 7f7032175700 1 --2- 172.21.15.32:0/722469732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 0x7f702c0fce20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.975+0000 7f7032175700 1 --2- 172.21.15.32:0/722469732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f702c102e10 0x7f702c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.975+0000 7f7032175700 1 -- 172.21.15.32:0/722469732 >> 172.21.15.32:0/722469732 conn(0x7f702c0f8980 msgr2=0x7f702c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:01.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.975+0000 7f7032175700 1 -- 172.21.15.32:0/722469732 shutdown_connections 2024-04-06T14:32:01.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.975+0000 7f7032175700 1 -- 172.21.15.32:0/722469732 wait complete. 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f7032175700 1 Processor -- start 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f7032175700 1 -- start start 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f7032175700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f702c102e10 0x7f702c113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f7032175700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 0x7f702c113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f7032175700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c1145f0 con 0x7f702c1037b0 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f7032175700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c10d9e0 con 0x7f702c102e10 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f702affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 0x7f702c113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f702affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 0x7f702c113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47720/0 (socket says 172.21.15.32:47720) 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f702affd700 1 -- 172.21.15.32:0/1864904839 learned_addr learned my addr 172.21.15.32:0/1864904839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f702b7fe700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f702c102e10 0x7f702c113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f702affd700 1 -- 172.21.15.32:0/1864904839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f702c102e10 msgr2=0x7f702c113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f702affd700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f702c102e10 0x7f702c113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f702affd700 1 -- 172.21.15.32:0/1864904839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c005040 con 0x7f702c1037b0 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.976+0000 7f702b7fe700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f702c102e10 0x7f702c113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.977+0000 7f702affd700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 0x7f702c113fa0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f701c000c00 tx=0x7f701c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.977+0000 7f7013fff700 1 -- 172.21.15.32:0/1864904839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f701c00e070 con 0x7f702c1037b0 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.977+0000 7f7013fff700 1 -- 172.21.15.32:0/1864904839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f701c013de0 con 0x7f702c1037b0 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.977+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702c10dc60 con 0x7f702c1037b0 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.977+0000 7f7013fff700 1 -- 172.21.15.32:0/1864904839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f701c01cde0 con 0x7f702c1037b0 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.977+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702c10e1c0 con 0x7f702c1037b0 2024-04-06T14:32:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.978+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702c0fe560 con 0x7f702c1037b0 2024-04-06T14:32:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.978+0000 7f7013fff700 1 -- 172.21.15.32:0/1864904839 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f701c01c600 con 0x7f702c1037b0 2024-04-06T14:32:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.979+0000 7f7013fff700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701406cc20 0x7f701406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.979+0000 7f7013fff700 1 -- 172.21.15.32:0/1864904839 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f701c08d470 con 0x7f702c1037b0 2024-04-06T14:32:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.979+0000 7f702b7fe700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701406cc20 0x7f701406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.979+0000 7f702b7fe700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701406cc20 0x7f701406f0d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7020004600 tx=0x7f702000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:01.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:01.983+0000 7f7013fff700 1 -- 172.21.15.32:0/1864904839 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f701c05b000 con 0x7f702c1037b0 2024-04-06T14:32:02.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.270+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f702c05f5f0 con 0x7f702c1037b0 2024-04-06T14:32:02.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.271+0000 7f7013fff700 1 -- 172.21.15.32:0/1864904839 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f701c05ab90 con 0x7f702c1037b0 2024-04-06T14:32:02.271 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:32:02.271 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:32:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:32:02.271 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:32:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:32:02.272 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:32:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:32:02.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701406cc20 msgr2=0x7f701406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:02.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701406cc20 0x7f701406f0d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7020004600 tx=0x7f702000b040 comp rx=0 tx=0).stop 2024-04-06T14:32:02.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 msgr2=0x7f702c113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:02.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 0x7f702c113fa0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f701c000c00 tx=0x7f701c009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:02.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 shutdown_connections 2024-04-06T14:32:02.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701406cc20 0x7f701406f0d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:02.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f702c102e10 0x7f702c113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:02.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 --2- 172.21.15.32:0/1864904839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c1037b0 0x7f702c113fa0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:02.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 >> 172.21.15.32:0/1864904839 conn(0x7f702c0f8980 msgr2=0x7f702c0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:02.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 shutdown_connections 2024-04-06T14:32:02.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:02.273+0000 7f7032175700 1 -- 172.21.15.32:0/1864904839 wait complete. 2024-04-06T14:32:02.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:02 smithi086 ceph-mon[110975]: pgmap v356: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:02 smithi086 ceph-mon[110975]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:02 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3871802576' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:32:02.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:02 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1673762130' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:32:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:02 smithi032 ceph-mon[147414]: pgmap v356: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:02 smithi032 ceph-mon[147414]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:02 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3871802576' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:32:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:02 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1673762130' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:32:03.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:03 smithi086 ceph-mon[110975]: from='client.34720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:03.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:03 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1864904839' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:32:03.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:03 smithi032 ceph-mon[147414]: from='client.34720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:03.399 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:03 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1864904839' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:32:04.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:04 smithi086 ceph-mon[110975]: pgmap v357: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:04 smithi032 ceph-mon[147414]: pgmap v357: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:06.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:06 smithi086 ceph-mon[110975]: pgmap v358: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:06 smithi032 ceph-mon[147414]: pgmap v358: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:08.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:08 smithi086 ceph-mon[110975]: pgmap v359: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:08.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:32:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:08 smithi032 ceph-mon[147414]: pgmap v359: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:32:10.377 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:10 smithi086 ceph-mon[110975]: pgmap v360: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:10 smithi032 ceph-mon[147414]: pgmap v360: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:12.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:12 smithi032 ceph-mon[147414]: pgmap v361: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:12.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:12 smithi086 ceph-mon[110975]: pgmap v361: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:14.451 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:14 smithi086 ceph-mon[110975]: pgmap v362: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:14.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:14 smithi032 ceph-mon[147414]: pgmap v362: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:16.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:16 smithi032 ceph-mon[147414]: pgmap v363: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:16.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:16 smithi086 ceph-mon[110975]: pgmap v363: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:18.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:18 smithi032 ceph-mon[147414]: pgmap v364: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:18.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:18 smithi086 ceph-mon[110975]: pgmap v364: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:20.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:20 smithi032 ceph-mon[147414]: pgmap v365: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:20.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:20 smithi086 ceph-mon[110975]: pgmap v365: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:22.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:22 smithi032 ceph-mon[147414]: pgmap v366: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:22.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:22 smithi086 ceph-mon[110975]: pgmap v366: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:23.398 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:32:23.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:32:24.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:24 smithi032 ceph-mon[147414]: pgmap v367: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:24.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:24 smithi086 ceph-mon[110975]: pgmap v367: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:26.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:26 smithi032 ceph-mon[147414]: pgmap v368: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:26.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:26 smithi086 ceph-mon[110975]: pgmap v368: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:28.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:28 smithi032 ceph-mon[147414]: pgmap v369: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:28.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:28 smithi086 ceph-mon[110975]: pgmap v369: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:30.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:30 smithi032 ceph-mon[147414]: pgmap v370: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:30.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:30 smithi086 ceph-mon[110975]: pgmap v370: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.391+0000 7f9466648700 1 -- 172.21.15.32:0/176113220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460102e20 msgr2=0x7f94601031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.391+0000 7f9466648700 1 --2- 172.21.15.32:0/176113220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460102e20 0x7f94601031f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9450004940 tx=0x7f9450009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.391+0000 7f9466648700 1 -- 172.21.15.32:0/176113220 shutdown_connections 2024-04-06T14:32:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.392+0000 7f9466648700 1 --2- 172.21.15.32:0/176113220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94601037c0 0x7f94600fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.392+0000 7f9466648700 1 --2- 172.21.15.32:0/176113220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9460102e20 0x7f94601031f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.392+0000 7f9466648700 1 -- 172.21.15.32:0/176113220 >> 172.21.15.32:0/176113220 conn(0x7f94600f89d0 msgr2=0x7f94600fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.392+0000 7f9466648700 1 -- 172.21.15.32:0/176113220 shutdown_connections 2024-04-06T14:32:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.392+0000 7f9466648700 1 -- 172.21.15.32:0/176113220 wait complete. 2024-04-06T14:32:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.392+0000 7f9466648700 1 Processor -- start 2024-04-06T14:32:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.392+0000 7f9466648700 1 -- start start 2024-04-06T14:32:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f9466648700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9460102e20 0x7f9460113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f9466648700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94601037c0 0x7f9460113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f9466648700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9460114610 con 0x7f94601037c0 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f9466648700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946010da00 con 0x7f9460102e20 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f945f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94601037c0 0x7f9460113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f945f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94601037c0 0x7f9460113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49268/0 (socket says 172.21.15.32:49268) 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f945f7fe700 1 -- 172.21.15.32:0/2210289445 learned_addr learned my addr 172.21.15.32:0/2210289445 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f945ffff700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9460102e20 0x7f9460113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f945f7fe700 1 -- 172.21.15.32:0/2210289445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9460102e20 msgr2=0x7f9460113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f945f7fe700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9460102e20 0x7f9460113a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f945f7fe700 1 -- 172.21.15.32:0/2210289445 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9450005040 con 0x7f94601037c0 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.393+0000 7f945f7fe700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94601037c0 0x7f9460113fc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f945400eb40 tx=0x7f945400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.394+0000 7f945cff9700 1 -- 172.21.15.32:0/2210289445 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9454007cc0 con 0x7f94601037c0 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.394+0000 7f945cff9700 1 -- 172.21.15.32:0/2210289445 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f945400f040 con 0x7f94601037c0 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.394+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946010dc80 con 0x7f94601037c0 2024-04-06T14:32:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.394+0000 7f945cff9700 1 -- 172.21.15.32:0/2210289445 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f945400c8f0 con 0x7f94601037c0 2024-04-06T14:32:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.394+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946010e240 con 0x7f94601037c0 2024-04-06T14:32:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.395+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94600fe570 con 0x7f94601037c0 2024-04-06T14:32:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.395+0000 7f945cff9700 1 -- 172.21.15.32:0/2210289445 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f94540090d0 con 0x7f94601037c0 2024-04-06T14:32:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.396+0000 7f945cff9700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f944806cc20 0x7f944806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:32.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.396+0000 7f945cff9700 1 -- 172.21.15.32:0/2210289445 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f945408ba60 con 0x7f94601037c0 2024-04-06T14:32:32.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.396+0000 7f945ffff700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f944806cc20 0x7f944806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:32.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.396+0000 7f945ffff700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f944806cc20 0x7f944806f0d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f945000b0d0 tx=0x7f9450009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:32.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.401+0000 7f945cff9700 1 -- 172.21.15.32:0/2210289445 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9454059560 con 0x7f94601037c0 2024-04-06T14:32:32.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:32 smithi032 ceph-mon[147414]: pgmap v371: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:32.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:32 smithi086 ceph-mon[110975]: pgmap v371: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:32.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.632+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9460109e80 con 0x7f944806cc20 2024-04-06T14:32:32.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.633+0000 7f945cff9700 1 -- 172.21.15.32:0/2210289445 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9450007de0 con 0x7f944806cc20 2024-04-06T14:32:32.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f944806cc20 msgr2=0x7f944806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:32.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f944806cc20 0x7f944806f0d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f945000b0d0 tx=0x7f9450009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:32.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94601037c0 msgr2=0x7f9460113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:32.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94601037c0 0x7f9460113fc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f945400eb40 tx=0x7f945400b040 comp rx=0 tx=0).stop 2024-04-06T14:32:32.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 shutdown_connections 2024-04-06T14:32:32.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f944806cc20 0x7f944806f0d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9460102e20 0x7f9460113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 --2- 172.21.15.32:0/2210289445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94601037c0 0x7f9460113fc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 >> 172.21.15.32:0/2210289445 conn(0x7f94600f89d0 msgr2=0x7f94600f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:32.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 shutdown_connections 2024-04-06T14:32:32.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.635+0000 7f9466648700 1 -- 172.21.15.32:0/2210289445 wait complete. 2024-04-06T14:32:32.645 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:32:32.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 -- 172.21.15.32:0/1267708630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f140fb4a0 msgr2=0x7f5f140fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:32.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 --2- 172.21.15.32:0/1267708630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f140fb4a0 0x7f5f140fb910 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5f08004940 tx=0x7f5f08009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 -- 172.21.15.32:0/1267708630 shutdown_connections 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 --2- 172.21.15.32:0/1267708630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f140fb4a0 0x7f5f140fb910 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 --2- 172.21.15.32:0/1267708630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f14102e70 0x7f5f14103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 -- 172.21.15.32:0/1267708630 >> 172.21.15.32:0/1267708630 conn(0x7f5f140714c0 msgr2=0x7f5f140718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 -- 172.21.15.32:0/1267708630 shutdown_connections 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 -- 172.21.15.32:0/1267708630 wait complete. 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 Processor -- start 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 -- start start 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f140fb4a0 0x7f5f14113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f14102e70 0x7f5f14113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f14114600 con 0x7f5f14102e70 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.740+0000 7f5f18946700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1410d9f0 con 0x7f5f140fb4a0 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f11d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f14102e70 0x7f5f14113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f11d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f14102e70 0x7f5f14113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49290/0 (socket says 172.21.15.32:49290) 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f11d9b700 1 -- 172.21.15.32:0/668978460 learned_addr learned my addr 172.21.15.32:0/668978460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f1259c700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f140fb4a0 0x7f5f14113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f11d9b700 1 -- 172.21.15.32:0/668978460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f140fb4a0 msgr2=0x7f5f14113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f11d9b700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f140fb4a0 0x7f5f14113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f11d9b700 1 -- 172.21.15.32:0/668978460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f08005040 con 0x7f5f14102e70 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f11d9b700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f14102e70 0x7f5f14113fb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5f0800b3b0 tx=0x7f5f08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f02ffd700 1 -- 172.21.15.32:0/668978460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0800e040 con 0x7f5f14102e70 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f02ffd700 1 -- 172.21.15.32:0/668978460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f08013420 con 0x7f5f14102e70 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1410dc70 con 0x7f5f14102e70 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f02ffd700 1 -- 172.21.15.32:0/668978460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0801c610 con 0x7f5f14102e70 2024-04-06T14:32:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.741+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1410e230 con 0x7f5f14102e70 2024-04-06T14:32:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.742+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f14100110 con 0x7f5f14102e70 2024-04-06T14:32:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.743+0000 7f5f02ffd700 1 -- 172.21.15.32:0/668978460 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5f08013a00 con 0x7f5f14102e70 2024-04-06T14:32:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.743+0000 7f5f02ffd700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5efc06cc20 0x7f5efc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.743+0000 7f5f02ffd700 1 -- 172.21.15.32:0/668978460 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5f0808d090 con 0x7f5f14102e70 2024-04-06T14:32:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.743+0000 7f5f1259c700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5efc06cc20 0x7f5efc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.744+0000 7f5f1259c700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5efc06cc20 0x7f5efc06f0d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5f04004f40 tx=0x7f5f0400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.748+0000 7f5f02ffd700 1 -- 172.21.15.32:0/668978460 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5f0805ac20 con 0x7f5f14102e70 2024-04-06T14:32:32.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.970+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f14113740 con 0x7f5efc06cc20 2024-04-06T14:32:32.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.971+0000 7f5f02ffd700 1 -- 172.21.15.32:0/668978460 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5f14113740 con 0x7f5efc06cc20 2024-04-06T14:32:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5efc06cc20 msgr2=0x7f5efc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5efc06cc20 0x7f5efc06f0d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5f04004f40 tx=0x7f5f0400b040 comp rx=0 tx=0).stop 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f14102e70 msgr2=0x7f5f14113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f14102e70 0x7f5f14113fb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5f0800b3b0 tx=0x7f5f08009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 shutdown_connections 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5efc06cc20 0x7f5efc06f0d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f140fb4a0 0x7f5f14113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 --2- 172.21.15.32:0/668978460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f14102e70 0x7f5f14113fb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 >> 172.21.15.32:0/668978460 conn(0x7f5f140714c0 msgr2=0x7f5f140fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 shutdown_connections 2024-04-06T14:32:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:32.973+0000 7f5f18946700 1 -- 172.21.15.32:0/668978460 wait complete. 2024-04-06T14:32:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.076+0000 7f075eeb2700 1 -- 172.21.15.32:0/2075555084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758101d70 msgr2=0x7f0758102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.076+0000 7f075eeb2700 1 --2- 172.21.15.32:0/2075555084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758101d70 0x7f0758102140 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f074c004940 tx=0x7f074c009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.076+0000 7f075eeb2700 1 -- 172.21.15.32:0/2075555084 shutdown_connections 2024-04-06T14:32:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.076+0000 7f075eeb2700 1 --2- 172.21.15.32:0/2075555084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0758102710 0x7f07580fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.076+0000 7f075eeb2700 1 --2- 172.21.15.32:0/2075555084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758101d70 0x7f0758102140 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.076+0000 7f075eeb2700 1 -- 172.21.15.32:0/2075555084 >> 172.21.15.32:0/2075555084 conn(0x7f0758072bc0 msgr2=0x7f0758072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.076+0000 7f075eeb2700 1 -- 172.21.15.32:0/2075555084 shutdown_connections 2024-04-06T14:32:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.077+0000 7f075eeb2700 1 -- 172.21.15.32:0/2075555084 wait complete. 2024-04-06T14:32:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.077+0000 7f075eeb2700 1 Processor -- start 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.077+0000 7f075eeb2700 1 -- start start 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.077+0000 7f075eeb2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0758101d70 0x7f0758115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.077+0000 7f075eeb2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758102710 0x7f07581161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.077+0000 7f075eeb2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0758116820 con 0x7f0758102710 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.077+0000 7f075eeb2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075810fc40 con 0x7f0758101d70 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f0757fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758102710 0x7f07581161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f0757fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758102710 0x7f07581161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49314/0 (socket says 172.21.15.32:49314) 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f0757fff700 1 -- 172.21.15.32:0/1009515251 learned_addr learned my addr 172.21.15.32:0/1009515251 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f075cc4e700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0758101d70 0x7f0758115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f0757fff700 1 -- 172.21.15.32:0/1009515251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0758101d70 msgr2=0x7f0758115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f0757fff700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0758101d70 0x7f0758115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f0757fff700 1 -- 172.21.15.32:0/1009515251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c005040 con 0x7f0758102710 2024-04-06T14:32:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f0757fff700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758102710 0x7f07581161d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f074400c890 tx=0x7f07440044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f07557fa700 1 -- 172.21.15.32:0/1009515251 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f074400a700 con 0x7f0758102710 2024-04-06T14:32:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075810ff20 con 0x7f0758102710 2024-04-06T14:32:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f07557fa700 1 -- 172.21.15.32:0/1009515251 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f074400ad10 con 0x7f0758102710 2024-04-06T14:32:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f07557fa700 1 -- 172.21.15.32:0/1009515251 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0744012e70 con 0x7f0758102710 2024-04-06T14:32:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.078+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07581104e0 con 0x7f0758102710 2024-04-06T14:32:33.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.079+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07580fd4c0 con 0x7f0758102710 2024-04-06T14:32:33.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.080+0000 7f07557fa700 1 -- 172.21.15.32:0/1009515251 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f074400a860 con 0x7f0758102710 2024-04-06T14:32:33.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.080+0000 7f07557fa700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f074806cc20 0x7f074806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:33.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.080+0000 7f07557fa700 1 -- 172.21.15.32:0/1009515251 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f074408bfa0 con 0x7f0758102710 2024-04-06T14:32:33.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.080+0000 7f075cc4e700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f074806cc20 0x7f074806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:33.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.081+0000 7f075cc4e700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f074806cc20 0x7f074806f0d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f074c004f90 tx=0x7f074c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:33.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.085+0000 7f07557fa700 1 -- 172.21.15.32:0/1009515251 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0744059b30 con 0x7f0758102710 2024-04-06T14:32:33.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.295+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0758108da0 con 0x7f074806cc20 2024-04-06T14:32:33.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.305+0000 7f07557fa700 1 -- 172.21.15.32:0/1009515251 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0758108da0 con 0x7f074806cc20 2024-04-06T14:32:33.305 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (12m) 6m ago 24m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 6m ago 24m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (10m) 6m ago 23m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (12m) 6m ago 23m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (6m) 6m ago 18m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (6m) 6m ago 18m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (6m) 6m ago 18m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (18m) 6m ago 18m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (14m) 6m ago 25m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (13m) 6m ago 23m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (11m) 6m ago 25m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (10m) 6m ago 23m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (13m) 6m ago 24m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:32:33.306 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (13m) 6m ago 23m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:32:33.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 6m ago 22m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:32:33.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (9m) 6m ago 21m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:32:33.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (8m) 6m ago 21m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:32:33.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (8m) 6m ago 20m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:32:33.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (7m) 6m ago 20m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:32:33.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (6m) 6m ago 20m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:32:33.307 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (13m) 6m ago 23m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.307+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f074806cc20 msgr2=0x7f074806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.307+0000 7f075eeb2700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f074806cc20 0x7f074806f0d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f074c004f90 tx=0x7f074c009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.307+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758102710 msgr2=0x7f07581161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.307+0000 7f075eeb2700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758102710 0x7f07581161d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f074400c890 tx=0x7f07440044c0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.307+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 shutdown_connections 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.307+0000 7f075eeb2700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f074806cc20 0x7f074806f0d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.307+0000 7f075eeb2700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0758101d70 0x7f0758115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.307+0000 7f075eeb2700 1 --2- 172.21.15.32:0/1009515251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0758102710 0x7f07581161d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.307+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 >> 172.21.15.32:0/1009515251 conn(0x7f0758072bc0 msgr2=0x7f0758065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.308+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 shutdown_connections 2024-04-06T14:32:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.308+0000 7f075eeb2700 1 -- 172.21.15.32:0/1009515251 wait complete. 2024-04-06T14:32:33.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.410+0000 7ff606415700 1 -- 172.21.15.32:0/3683027159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff600102e10 msgr2=0x7ff6001031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.410+0000 7ff606415700 1 --2- 172.21.15.32:0/3683027159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff600102e10 0x7ff6001031e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0004940 tx=0x7ff5f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:33.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.410+0000 7ff606415700 1 -- 172.21.15.32:0/3683027159 shutdown_connections 2024-04-06T14:32:33.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.410+0000 7ff606415700 1 --2- 172.21.15.32:0/3683027159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6001037b0 0x7ff6000fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.410+0000 7ff606415700 1 --2- 172.21.15.32:0/3683027159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff600102e10 0x7ff6001031e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.410+0000 7ff606415700 1 -- 172.21.15.32:0/3683027159 >> 172.21.15.32:0/3683027159 conn(0x7ff6000f8980 msgr2=0x7ff6000fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:33.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.410+0000 7ff606415700 1 -- 172.21.15.32:0/3683027159 shutdown_connections 2024-04-06T14:32:33.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.410+0000 7ff606415700 1 -- 172.21.15.32:0/3683027159 wait complete. 2024-04-06T14:32:33.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.411+0000 7ff606415700 1 Processor -- start 2024-04-06T14:32:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.411+0000 7ff606415700 1 -- start start 2024-04-06T14:32:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.411+0000 7ff606415700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff600102e10 0x7ff600113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.411+0000 7ff606415700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6001037b0 0x7ff600114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.411+0000 7ff606415700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6001146a0 con 0x7ff6001037b0 2024-04-06T14:32:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.411+0000 7ff606415700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60010da90 con 0x7ff600102e10 2024-04-06T14:32:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.411+0000 7ff5ff7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6001037b0 0x7ff600114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.411+0000 7ff5ff7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6001037b0 0x7ff600114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49334/0 (socket says 172.21.15.32:49334) 2024-04-06T14:32:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.411+0000 7ff5ff7fe700 1 -- 172.21.15.32:0/3999552810 learned_addr learned my addr 172.21.15.32:0/3999552810 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff5fffff700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff600102e10 0x7ff600113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff5ff7fe700 1 -- 172.21.15.32:0/3999552810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff600102e10 msgr2=0x7ff600113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff5ff7fe700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff600102e10 0x7ff600113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff5ff7fe700 1 -- 172.21.15.32:0/3999552810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f0005040 con 0x7ff6001037b0 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff5ff7fe700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6001037b0 0x7ff600114050 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400c800 tx=0x7ff5f4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff5fcff9700 1 -- 172.21.15.32:0/3999552810 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5f400a760 con 0x7ff6001037b0 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff5fcff9700 1 -- 172.21.15.32:0/3999552810 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5f400ad70 con 0x7ff6001037b0 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60010dd70 con 0x7ff6001037b0 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff5fcff9700 1 -- 172.21.15.32:0/3999552810 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5f40113a0 con 0x7ff6001037b0 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.412+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60010e330 con 0x7ff6001037b0 2024-04-06T14:32:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.413+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6000fe560 con 0x7ff6001037b0 2024-04-06T14:32:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.414+0000 7ff5fcff9700 1 -- 172.21.15.32:0/3999552810 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff5f4012400 con 0x7ff6001037b0 2024-04-06T14:32:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.414+0000 7ff5fcff9700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff5e806cc20 0x7ff5e806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.414+0000 7ff5fcff9700 1 -- 172.21.15.32:0/3999552810 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff5f408bf60 con 0x7ff6001037b0 2024-04-06T14:32:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.414+0000 7ff5fffff700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff5e806cc20 0x7ff5e806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.414+0000 7ff5fffff700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff5e806cc20 0x7ff5e806f0d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0004f90 tx=0x7ff5f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.418+0000 7ff5fcff9700 1 -- 172.21.15.32:0/3999552810 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff5f40bfab0 con 0x7ff6001037b0 2024-04-06T14:32:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.705+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff600109e00 con 0x7ff6001037b0 2024-04-06T14:32:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.706+0000 7ff5fcff9700 1 -- 172.21.15.32:0/3999552810 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7ff5f4059af0 con 0x7ff6001037b0 2024-04-06T14:32:33.706 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:32:33.706 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:32:33.707 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:32:33.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:32:33.708 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:32:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff5e806cc20 msgr2=0x7ff5e806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff5e806cc20 0x7ff5e806f0d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0004f90 tx=0x7ff5f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6001037b0 msgr2=0x7ff600114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6001037b0 0x7ff600114050 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400c800 tx=0x7ff5f4004470 comp rx=0 tx=0).stop 2024-04-06T14:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 shutdown_connections 2024-04-06T14:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff5e806cc20 0x7ff5e806f0d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff600102e10 0x7ff600113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 --2- 172.21.15.32:0/3999552810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6001037b0 0x7ff600114050 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 >> 172.21.15.32:0/3999552810 conn(0x7ff6000f8980 msgr2=0x7ff6000f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 shutdown_connections 2024-04-06T14:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.708+0000 7ff606415700 1 -- 172.21.15.32:0/3999552810 wait complete. 2024-04-06T14:32:33.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.812+0000 7fb684e80700 1 -- 172.21.15.32:0/3545671001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680102710 msgr2=0x7fb6800fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.812+0000 7fb684e80700 1 --2- 172.21.15.32:0/3545671001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680102710 0x7fb6800fbd80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb674004940 tx=0x7fb674009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.812+0000 7fb684e80700 1 -- 172.21.15.32:0/3545671001 shutdown_connections 2024-04-06T14:32:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.812+0000 7fb684e80700 1 --2- 172.21.15.32:0/3545671001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680102710 0x7fb6800fbd80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.812+0000 7fb684e80700 1 --2- 172.21.15.32:0/3545671001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb680101d70 0x7fb680102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.812+0000 7fb684e80700 1 -- 172.21.15.32:0/3545671001 >> 172.21.15.32:0/3545671001 conn(0x7fb680072bc0 msgr2=0x7fb680072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.812+0000 7fb684e80700 1 -- 172.21.15.32:0/3545671001 shutdown_connections 2024-04-06T14:32:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.812+0000 7fb684e80700 1 -- 172.21.15.32:0/3545671001 wait complete. 2024-04-06T14:32:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.813+0000 7fb684e80700 1 Processor -- start 2024-04-06T14:32:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.813+0000 7fb684e80700 1 -- start start 2024-04-06T14:32:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.813+0000 7fb684e80700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680101d70 0x7fb6801129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.813+0000 7fb684e80700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb680102710 0x7fb680112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.813+0000 7fb684e80700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb680113550 con 0x7fb680101d70 2024-04-06T14:32:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.813+0000 7fb684e80700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68010c940 con 0x7fb680102710 2024-04-06T14:32:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb67e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680101d70 0x7fb6801129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb67e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680101d70 0x7fb6801129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49344/0 (socket says 172.21.15.32:49344) 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb67e59c700 1 -- 172.21.15.32:0/184189022 learned_addr learned my addr 172.21.15.32:0/184189022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb67dd9b700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb680102710 0x7fb680112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb67e59c700 1 -- 172.21.15.32:0/184189022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb680102710 msgr2=0x7fb680112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb67e59c700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb680102710 0x7fb680112f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb67e59c700 1 -- 172.21.15.32:0/184189022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb674005040 con 0x7fb680101d70 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb67dd9b700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb680102710 0x7fb680112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb67e59c700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680101d70 0x7fb6801129c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb67000c8c0 tx=0x7fb6700044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb66effd700 1 -- 172.21.15.32:0/184189022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb67000a980 con 0x7fb680101d70 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb66effd700 1 -- 172.21.15.32:0/184189022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb67000ce60 con 0x7fb680101d70 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb66effd700 1 -- 172.21.15.32:0/184189022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6700113a0 con 0x7fb680101d70 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68010cc20 con 0x7fb680101d70 2024-04-06T14:32:33.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.814+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68010d1e0 con 0x7fb680101d70 2024-04-06T14:32:33.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.816+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6800ff080 con 0x7fb680101d70 2024-04-06T14:32:33.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.816+0000 7fb66effd700 1 -- 172.21.15.32:0/184189022 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb670012400 con 0x7fb680101d70 2024-04-06T14:32:33.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.816+0000 7fb66effd700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb66806cc20 0x7fb66806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.816+0000 7fb66effd700 1 -- 172.21.15.32:0/184189022 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb67008c040 con 0x7fb680101d70 2024-04-06T14:32:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.817+0000 7fb67dd9b700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb66806cc20 0x7fb66806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.817+0000 7fb67dd9b700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb66806cc20 0x7fb66806f0d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb68010db00 tx=0x7fb674009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:33.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:33.821+0000 7fb66effd700 1 -- 172.21.15.32:0/184189022 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb670059bd0 con 0x7fb680101d70 2024-04-06T14:32:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.074+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb68010e010 con 0x7fb680101d70 2024-04-06T14:32:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.074+0000 7fb66effd700 1 -- 172.21.15.32:0/184189022 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fb670059760 con 0x7fb680101d70 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:32:34.075 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:32:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:32:34.077 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:32:34.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.076+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb66806cc20 msgr2=0x7fb66806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.076+0000 7fb684e80700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb66806cc20 0x7fb66806f0d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb68010db00 tx=0x7fb674009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:34.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.076+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680101d70 msgr2=0x7fb6801129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.076+0000 7fb684e80700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680101d70 0x7fb6801129c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb67000c8c0 tx=0x7fb6700044c0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.076+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 shutdown_connections 2024-04-06T14:32:34.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.076+0000 7fb684e80700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb66806cc20 0x7fb66806f0d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.076+0000 7fb684e80700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb680101d70 0x7fb6801129c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.076+0000 7fb684e80700 1 --2- 172.21.15.32:0/184189022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb680102710 0x7fb680112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.077+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 >> 172.21.15.32:0/184189022 conn(0x7fb680072bc0 msgr2=0x7fb6800fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:34.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.077+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 shutdown_connections 2024-04-06T14:32:34.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.077+0000 7fb684e80700 1 -- 172.21.15.32:0/184189022 wait complete. 2024-04-06T14:32:34.079 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:32:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 -- 172.21.15.32:0/4087426506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fde90 msgr2=0x7f11dc0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 --2- 172.21.15.32:0/4087426506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fde90 0x7f11dc0fe260 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f11c4004940 tx=0x7f11c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 -- 172.21.15.32:0/4087426506 shutdown_connections 2024-04-06T14:32:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 --2- 172.21.15.32:0/4087426506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11dc0fe830 0x7f11dc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 --2- 172.21.15.32:0/4087426506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fde90 0x7f11dc0fe260 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 -- 172.21.15.32:0/4087426506 >> 172.21.15.32:0/4087426506 conn(0x7f11dc072bc0 msgr2=0x7f11dc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 -- 172.21.15.32:0/4087426506 shutdown_connections 2024-04-06T14:32:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 -- 172.21.15.32:0/4087426506 wait complete. 2024-04-06T14:32:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 Processor -- start 2024-04-06T14:32:34.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.183+0000 7f11e1f6b700 1 -- start start 2024-04-06T14:32:34.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11e1f6b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11dc0fde90 0x7f11dc113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11e1f6b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fe830 0x7f11dc114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11e1f6b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc114680 con 0x7f11dc0fe830 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11e1f6b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc10da70 con 0x7f11dc0fde90 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11daffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fe830 0x7f11dc114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11daffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fe830 0x7f11dc114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49352/0 (socket says 172.21.15.32:49352) 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11daffd700 1 -- 172.21.15.32:0/984826765 learned_addr learned my addr 172.21.15.32:0/984826765 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11db7fe700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11dc0fde90 0x7f11dc113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11daffd700 1 -- 172.21.15.32:0/984826765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11dc0fde90 msgr2=0x7f11dc113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11daffd700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11dc0fde90 0x7f11dc113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11daffd700 1 -- 172.21.15.32:0/984826765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c4005040 con 0x7f11dc0fe830 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11db7fe700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11dc0fde90 0x7f11dc113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11daffd700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fe830 0x7f11dc114030 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f11cc002a50 tx=0x7f11cc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11d3fff700 1 -- 172.21.15.32:0/984826765 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11cc007cc0 con 0x7f11dc0fe830 2024-04-06T14:32:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11d3fff700 1 -- 172.21.15.32:0/984826765 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11cc00f040 con 0x7f11dc0fe830 2024-04-06T14:32:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc10dd50 con 0x7f11dc0fe830 2024-04-06T14:32:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.184+0000 7f11d3fff700 1 -- 172.21.15.32:0/984826765 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11cc009540 con 0x7f11dc0fe830 2024-04-06T14:32:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.185+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc10e290 con 0x7f11dc0fe830 2024-04-06T14:32:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.185+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc1019b0 con 0x7f11dc0fe830 2024-04-06T14:32:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.186+0000 7f11d3fff700 1 -- 172.21.15.32:0/984826765 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f11cc0096a0 con 0x7f11dc0fe830 2024-04-06T14:32:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.186+0000 7f11d3fff700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c806cc20 0x7f11c806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.186+0000 7f11d3fff700 1 -- 172.21.15.32:0/984826765 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f11cc08bd70 con 0x7f11dc0fe830 2024-04-06T14:32:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.186+0000 7f11db7fe700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c806cc20 0x7f11c806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.187+0000 7f11db7fe700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c806cc20 0x7f11c806f0d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f11c400b3b0 tx=0x7f11c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:34.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.191+0000 7f11d3fff700 1 -- 172.21.15.32:0/984826765 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f11cc059840 con 0x7f11dc0fe830 2024-04-06T14:32:34.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.412+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f11dc1137c0 con 0x7f11c806cc20 2024-04-06T14:32:34.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.413+0000 7f11d3fff700 1 -- 172.21.15.32:0/984826765 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f11dc1137c0 con 0x7f11c806cc20 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:32:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:32:34.415 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.415+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c806cc20 msgr2=0x7f11c806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.415+0000 7f11e1f6b700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c806cc20 0x7f11c806f0d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f11c400b3b0 tx=0x7f11c4009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.415+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fe830 msgr2=0x7f11dc114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.415+0000 7f11e1f6b700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fe830 0x7f11dc114030 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f11cc002a50 tx=0x7f11cc0046b0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.415+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 shutdown_connections 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.415+0000 7f11e1f6b700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c806cc20 0x7f11c806f0d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.415+0000 7f11e1f6b700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11dc0fde90 0x7f11dc113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.415+0000 7f11e1f6b700 1 --2- 172.21.15.32:0/984826765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11dc0fe830 0x7f11dc114030 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.415+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 >> 172.21.15.32:0/984826765 conn(0x7f11dc072bc0 msgr2=0x7f11dc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.416+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 shutdown_connections 2024-04-06T14:32:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.416+0000 7f11e1f6b700 1 -- 172.21.15.32:0/984826765 wait complete. 2024-04-06T14:32:34.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:34 smithi032 ceph-mon[147414]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:34.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:34 smithi032 ceph-mon[147414]: pgmap v372: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:34.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:34 smithi032 ceph-mon[147414]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:34.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:34 smithi032 ceph-mon[147414]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:34.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:34 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3999552810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:32:34.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:34 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/184189022' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:32:34.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.519+0000 7f27cd4ec700 1 -- 172.21.15.32:0/3319918743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 msgr2=0x7f27c80fb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.519+0000 7f27cd4ec700 1 --2- 172.21.15.32:0/3319918743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 0x7f27c80fb4c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f27b8004940 tx=0x7f27b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:34.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.519+0000 7f27cd4ec700 1 -- 172.21.15.32:0/3319918743 shutdown_connections 2024-04-06T14:32:34.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.519+0000 7f27cd4ec700 1 --2- 172.21.15.32:0/3319918743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c80fba90 0x7f27c8101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.519+0000 7f27cd4ec700 1 --2- 172.21.15.32:0/3319918743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 0x7f27c80fb4c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.519+0000 7f27cd4ec700 1 -- 172.21.15.32:0/3319918743 >> 172.21.15.32:0/3319918743 conn(0x7f27c80f6bd0 msgr2=0x7f27c80f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:34.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.519+0000 7f27cd4ec700 1 -- 172.21.15.32:0/3319918743 shutdown_connections 2024-04-06T14:32:34.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.519+0000 7f27cd4ec700 1 -- 172.21.15.32:0/3319918743 wait complete. 2024-04-06T14:32:34.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.520+0000 7f27cd4ec700 1 Processor -- start 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.520+0000 7f27cd4ec700 1 -- start start 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.520+0000 7f27cd4ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 0x7f27c8113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.520+0000 7f27cd4ec700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c80fba90 0x7f27c8113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.520+0000 7f27cd4ec700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c8114510 con 0x7f27c80fb0f0 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.520+0000 7f27cd4ec700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c810db30 con 0x7f27c80fba90 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.520+0000 7f27c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 0x7f27c8113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 0x7f27c8113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49142/0 (socket says 172.21.15.32:49142) 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27c6ffd700 1 -- 172.21.15.32:0/282962933 learned_addr learned my addr 172.21.15.32:0/282962933 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27c67fc700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c80fba90 0x7f27c8113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27c6ffd700 1 -- 172.21.15.32:0/282962933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c80fba90 msgr2=0x7f27c8113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27c6ffd700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c80fba90 0x7f27c8113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27c6ffd700 1 -- 172.21.15.32:0/282962933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b8005040 con 0x7f27c80fb0f0 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27c6ffd700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 0x7f27c8113980 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f27b800b3b0 tx=0x7f27b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:34.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27af7fe700 1 -- 172.21.15.32:0/282962933 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27b800e040 con 0x7f27c80fb0f0 2024-04-06T14:32:34.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27af7fe700 1 -- 172.21.15.32:0/282962933 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27b8013420 con 0x7f27c80fb0f0 2024-04-06T14:32:34.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c810ddb0 con 0x7f27c80fb0f0 2024-04-06T14:32:34.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27af7fe700 1 -- 172.21.15.32:0/282962933 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27b801c610 con 0x7f27c80fb0f0 2024-04-06T14:32:34.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.521+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c810e2f0 con 0x7f27c80fb0f0 2024-04-06T14:32:34.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.522+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c80fc320 con 0x7f27c80fb0f0 2024-04-06T14:32:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.523+0000 7f27af7fe700 1 -- 172.21.15.32:0/282962933 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f27b8013a00 con 0x7f27c80fb0f0 2024-04-06T14:32:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.523+0000 7f27af7fe700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27b006cc20 0x7f27b006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:32:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.523+0000 7f27af7fe700 1 -- 172.21.15.32:0/282962933 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f27b808d080 con 0x7f27c80fb0f0 2024-04-06T14:32:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.523+0000 7f27c67fc700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27b006cc20 0x7f27b006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:32:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.524+0000 7f27c67fc700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27b006cc20 0x7f27b006f0d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f27c810f220 tx=0x7f27bc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:32:34.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.528+0000 7f27af7fe700 1 -- 172.21.15.32:0/282962933 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f27b8057180 con 0x7f27c80fb0f0 2024-04-06T14:32:34.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:34 smithi086 ceph-mon[110975]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:34.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:34 smithi086 ceph-mon[110975]: pgmap v372: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:34.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:34 smithi086 ceph-mon[110975]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:34.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:34 smithi086 ceph-mon[110975]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:34.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:34 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3999552810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:32:34.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:34 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/184189022' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:32:34.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.815+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f27c8109df0 con 0x7f27c80fb0f0 2024-04-06T14:32:34.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.816+0000 7f27af7fe700 1 -- 172.21.15.32:0/282962933 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f27b805a7a0 con 0x7f27c80fb0f0 2024-04-06T14:32:34.816 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:32:34.816 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:32:34.816 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:32:34.816 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:32:34.816 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:32:34.816 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:32:34.816 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.817+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27b006cc20 msgr2=0x7f27b006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.817+0000 7f27cd4ec700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27b006cc20 0x7f27b006f0d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f27c810f220 tx=0x7f27bc00b040 comp rx=0 tx=0).stop 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.817+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 msgr2=0x7f27c8113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.817+0000 7f27cd4ec700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 0x7f27c8113980 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f27b800b3b0 tx=0x7f27b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.818+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 shutdown_connections 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.818+0000 7f27cd4ec700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27b006cc20 0x7f27b006f0d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.818+0000 7f27cd4ec700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c80fb0f0 0x7f27c8113980 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.818+0000 7f27cd4ec700 1 --2- 172.21.15.32:0/282962933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c80fba90 0x7f27c8113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.818+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 >> 172.21.15.32:0/282962933 conn(0x7f27c80f6bd0 msgr2=0x7f27c80f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.818+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 shutdown_connections 2024-04-06T14:32:34.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:32:34.818+0000 7f27cd4ec700 1 -- 172.21.15.32:0/282962933 wait complete. 2024-04-06T14:32:35.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:35 smithi032 ceph-mon[147414]: from='client.34748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:35.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:35 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/282962933' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:32:35.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:35 smithi086 ceph-mon[110975]: from='client.34748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:32:35.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:35 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/282962933' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:32:36.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:36 smithi032 ceph-mon[147414]: pgmap v373: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:36.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:36 smithi086 ceph-mon[110975]: pgmap v373: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:38 smithi032 ceph-mon[147414]: pgmap v374: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:38.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:32:38.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:38 smithi086 ceph-mon[110975]: pgmap v374: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:38.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:32:40.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:40 smithi032 ceph-mon[147414]: pgmap v375: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:40.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:40 smithi086 ceph-mon[110975]: pgmap v375: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:42.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:42 smithi032 ceph-mon[147414]: pgmap v376: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:42.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:42 smithi086 ceph-mon[110975]: pgmap v376: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:44.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:44 smithi032 ceph-mon[147414]: pgmap v377: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:44.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:44 smithi086 ceph-mon[110975]: pgmap v377: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:46.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:46 smithi032 ceph-mon[147414]: pgmap v378: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:46.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:46 smithi086 ceph-mon[110975]: pgmap v378: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:48.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:48 smithi032 ceph-mon[147414]: pgmap v379: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:48.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:48 smithi086 ceph-mon[110975]: pgmap v379: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:50.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:50 smithi032 ceph-mon[147414]: pgmap v380: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:50.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:50 smithi086 ceph-mon[110975]: pgmap v380: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:52.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:52 smithi032 ceph-mon[147414]: pgmap v381: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:52.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:52 smithi086 ceph-mon[110975]: pgmap v381: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:53.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:32:53.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:32:54.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:54 smithi032 ceph-mon[147414]: pgmap v382: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:54.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:54 smithi086 ceph-mon[110975]: pgmap v382: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:56.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:56 smithi032 ceph-mon[147414]: pgmap v383: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:56.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:56 smithi086 ceph-mon[110975]: pgmap v383: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:58.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:32:58 smithi032 ceph-mon[147414]: pgmap v384: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:32:58.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:32:58 smithi086 ceph-mon[110975]: pgmap v384: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:00.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:00 smithi032 ceph-mon[147414]: pgmap v385: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:00.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:00 smithi086 ceph-mon[110975]: pgmap v385: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:02.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:02 smithi032 ceph-mon[147414]: pgmap v386: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:02.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:02 smithi086 ceph-mon[110975]: pgmap v386: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:04.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:04 smithi032 ceph-mon[147414]: pgmap v387: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:04.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:04 smithi086 ceph-mon[110975]: pgmap v387: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.933+0000 7f4766749700 1 -- 172.21.15.32:0/3503255723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 msgr2=0x7f47600fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.933+0000 7f4766749700 1 --2- 172.21.15.32:0/3503255723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 0x7f47600fce30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4754004940 tx=0x7f4754009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.933+0000 7f4766749700 1 -- 172.21.15.32:0/3503255723 shutdown_connections 2024-04-06T14:33:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.933+0000 7f4766749700 1 --2- 172.21.15.32:0/3503255723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 0x7f47600fce30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.933+0000 7f4766749700 1 --2- 172.21.15.32:0/3503255723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4760102e20 0x7f47601031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.933+0000 7f4766749700 1 -- 172.21.15.32:0/3503255723 >> 172.21.15.32:0/3503255723 conn(0x7f47600f89d0 msgr2=0x7f47600fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.933+0000 7f4766749700 1 -- 172.21.15.32:0/3503255723 shutdown_connections 2024-04-06T14:33:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.933+0000 7f4766749700 1 -- 172.21.15.32:0/3503255723 wait complete. 2024-04-06T14:33:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.933+0000 7f4766749700 1 Processor -- start 2024-04-06T14:33:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f4766749700 1 -- start start 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f4766749700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4760102e20 0x7f4760113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f4766749700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 0x7f4760114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f4766749700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760114660 con 0x7f47601037c0 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f4766749700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476010da50 con 0x7f4760102e20 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f475f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 0x7f4760114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f475f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 0x7f4760114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39280/0 (socket says 172.21.15.32:39280) 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f475f7fe700 1 -- 172.21.15.32:0/90609016 learned_addr learned my addr 172.21.15.32:0/90609016 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f475ffff700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4760102e20 0x7f4760113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f475f7fe700 1 -- 172.21.15.32:0/90609016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4760102e20 msgr2=0x7f4760113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.934+0000 7f475f7fe700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4760102e20 0x7f4760113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.935+0000 7f475f7fe700 1 -- 172.21.15.32:0/90609016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4754005040 con 0x7f47601037c0 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.935+0000 7f475f7fe700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 0x7f4760114010 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4754009380 tx=0x7f4754009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.935+0000 7f475cff9700 1 -- 172.21.15.32:0/90609016 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f475400e040 con 0x7f47601037c0 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.935+0000 7f475cff9700 1 -- 172.21.15.32:0/90609016 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4754013420 con 0x7f47601037c0 2024-04-06T14:33:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.935+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476010dcd0 con 0x7f47601037c0 2024-04-06T14:33:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.935+0000 7f475cff9700 1 -- 172.21.15.32:0/90609016 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f475401c630 con 0x7f47601037c0 2024-04-06T14:33:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.935+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476010e290 con 0x7f47601037c0 2024-04-06T14:33:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.936+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47600fe570 con 0x7f47601037c0 2024-04-06T14:33:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.936+0000 7f475cff9700 1 -- 172.21.15.32:0/90609016 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f47540139e0 con 0x7f47601037c0 2024-04-06T14:33:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.937+0000 7f475cff9700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f474806cc20 0x7f474806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.937+0000 7f475cff9700 1 -- 172.21.15.32:0/90609016 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f475408d140 con 0x7f47601037c0 2024-04-06T14:33:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.937+0000 7f475ffff700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f474806cc20 0x7f474806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.937+0000 7f475ffff700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f474806cc20 0x7f474806f0d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4750004990 tx=0x7f4750009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:04.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:04.941+0000 7f475cff9700 1 -- 172.21.15.32:0/90609016 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f475405acd0 con 0x7f47601037c0 2024-04-06T14:33:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.161+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4760109dc0 con 0x7f474806cc20 2024-04-06T14:33:05.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.162+0000 7f475cff9700 1 -- 172.21.15.32:0/90609016 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4760109dc0 con 0x7f474806cc20 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.164+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f474806cc20 msgr2=0x7f474806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.164+0000 7f4766749700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f474806cc20 0x7f474806f0d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4750004990 tx=0x7f4750009ee0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.164+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 msgr2=0x7f4760114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.164+0000 7f4766749700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 0x7f4760114010 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4754009380 tx=0x7f4754009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.165+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 shutdown_connections 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.165+0000 7f4766749700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f474806cc20 0x7f474806f0d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.165+0000 7f4766749700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4760102e20 0x7f4760113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.165+0000 7f4766749700 1 --2- 172.21.15.32:0/90609016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47601037c0 0x7f4760114010 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.165+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 >> 172.21.15.32:0/90609016 conn(0x7f47600f89d0 msgr2=0x7f47600f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.165+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 shutdown_connections 2024-04-06T14:33:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.165+0000 7f4766749700 1 -- 172.21.15.32:0/90609016 wait complete. 2024-04-06T14:33:05.175 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.270+0000 7f960c6e9700 1 -- 172.21.15.32:0/389705558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 msgr2=0x7f96040fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.270+0000 7f960c6e9700 1 --2- 172.21.15.32:0/389705558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 0x7f96040fce20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9600004940 tx=0x7f9600009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.270+0000 7f960c6e9700 1 -- 172.21.15.32:0/389705558 shutdown_connections 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.270+0000 7f960c6e9700 1 --2- 172.21.15.32:0/389705558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 0x7f96040fce20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.270+0000 7f960c6e9700 1 --2- 172.21.15.32:0/389705558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9604102e10 0x7f96041031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.270+0000 7f960c6e9700 1 -- 172.21.15.32:0/389705558 >> 172.21.15.32:0/389705558 conn(0x7f96040f8980 msgr2=0x7f96040fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.270+0000 7f960c6e9700 1 -- 172.21.15.32:0/389705558 shutdown_connections 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.270+0000 7f960c6e9700 1 -- 172.21.15.32:0/389705558 wait complete. 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f960c6e9700 1 Processor -- start 2024-04-06T14:33:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f960c6e9700 1 -- start start 2024-04-06T14:33:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f960c6e9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9604102e10 0x7f9604113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f960c6e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 0x7f9604114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f960c6e9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9604114670 con 0x7f96041037b0 2024-04-06T14:33:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f960c6e9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960410da60 con 0x7f9604102e10 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f9609c84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 0x7f9604114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f9609c84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 0x7f9604114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39294/0 (socket says 172.21.15.32:39294) 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f9609c84700 1 -- 172.21.15.32:0/1881177843 learned_addr learned my addr 172.21.15.32:0/1881177843 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f960a485700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9604102e10 0x7f9604113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f9609c84700 1 -- 172.21.15.32:0/1881177843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9604102e10 msgr2=0x7f9604113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f9609c84700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9604102e10 0x7f9604113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.271+0000 7f9609c84700 1 -- 172.21.15.32:0/1881177843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9600005040 con 0x7f96041037b0 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.272+0000 7f9609c84700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 0x7f9604114020 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9600009380 tx=0x7f9600009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.272+0000 7f95faffd700 1 -- 172.21.15.32:0/1881177843 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f960000e040 con 0x7f96041037b0 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.272+0000 7f95faffd700 1 -- 172.21.15.32:0/1881177843 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9600013420 con 0x7f96041037b0 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.272+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960410dce0 con 0x7f96041037b0 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.272+0000 7f95faffd700 1 -- 172.21.15.32:0/1881177843 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f960001c630 con 0x7f96041037b0 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.272+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960410e2a0 con 0x7f96041037b0 2024-04-06T14:33:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.273+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96040fe560 con 0x7f96041037b0 2024-04-06T14:33:05.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.273+0000 7f95faffd700 1 -- 172.21.15.32:0/1881177843 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f96000139e0 con 0x7f96041037b0 2024-04-06T14:33:05.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.274+0000 7f95faffd700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f95f006cc20 0x7f95f006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:05.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.274+0000 7f95faffd700 1 -- 172.21.15.32:0/1881177843 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f960008d020 con 0x7f96041037b0 2024-04-06T14:33:05.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.274+0000 7f960a485700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f95f006cc20 0x7f95f006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:05.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.274+0000 7f960a485700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f95f006cc20 0x7f95f006f0d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f95f4004d60 tx=0x7f95f400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:05.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.279+0000 7f95faffd700 1 -- 172.21.15.32:0/1881177843 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f960005aaf0 con 0x7f96041037b0 2024-04-06T14:33:05.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.502+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9604109db0 con 0x7f95f006cc20 2024-04-06T14:33:05.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.503+0000 7f95faffd700 1 -- 172.21.15.32:0/1881177843 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9604109db0 con 0x7f95f006cc20 2024-04-06T14:33:05.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.505+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f95f006cc20 msgr2=0x7f95f006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.505+0000 7f960c6e9700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f95f006cc20 0x7f95f006f0d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f95f4004d60 tx=0x7f95f400b040 comp rx=0 tx=0).stop 2024-04-06T14:33:05.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.505+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 msgr2=0x7f9604114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.505+0000 7f960c6e9700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 0x7f9604114020 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9600009380 tx=0x7f9600009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:05.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.506+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 shutdown_connections 2024-04-06T14:33:05.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.506+0000 7f960c6e9700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f95f006cc20 0x7f95f006f0d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.506+0000 7f960c6e9700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9604102e10 0x7f9604113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.506+0000 7f960c6e9700 1 --2- 172.21.15.32:0/1881177843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96041037b0 0x7f9604114020 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.506+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 >> 172.21.15.32:0/1881177843 conn(0x7f96040f8980 msgr2=0x7f96040f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:05.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.506+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 shutdown_connections 2024-04-06T14:33:05.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.506+0000 7f960c6e9700 1 -- 172.21.15.32:0/1881177843 wait complete. 2024-04-06T14:33:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.608+0000 7fa96ac2b700 1 -- 172.21.15.32:0/1771791526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fde90 msgr2=0x7fa9640fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.608+0000 7fa96ac2b700 1 --2- 172.21.15.32:0/1771791526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fde90 0x7fa9640fe260 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa958004940 tx=0x7fa958009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.609+0000 7fa96ac2b700 1 -- 172.21.15.32:0/1771791526 shutdown_connections 2024-04-06T14:33:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.609+0000 7fa96ac2b700 1 --2- 172.21.15.32:0/1771791526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9640fe830 0x7fa964106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.609+0000 7fa96ac2b700 1 --2- 172.21.15.32:0/1771791526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fde90 0x7fa9640fe260 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.609+0000 7fa96ac2b700 1 -- 172.21.15.32:0/1771791526 >> 172.21.15.32:0/1771791526 conn(0x7fa964072bc0 msgr2=0x7fa964072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:05.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.609+0000 7fa96ac2b700 1 -- 172.21.15.32:0/1771791526 shutdown_connections 2024-04-06T14:33:05.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.609+0000 7fa96ac2b700 1 -- 172.21.15.32:0/1771791526 wait complete. 2024-04-06T14:33:05.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.609+0000 7fa96ac2b700 1 Processor -- start 2024-04-06T14:33:05.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.609+0000 7fa96ac2b700 1 -- start start 2024-04-06T14:33:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa96ac2b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9640fde90 0x7fa9640fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa96ac2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fe830 0x7fa9640fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa96ac2b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9640fc820 con 0x7fa9640fe830 2024-04-06T14:33:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa96ac2b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9640fc960 con 0x7fa9640fde90 2024-04-06T14:33:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa963fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fe830 0x7fa9640fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa963fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fe830 0x7fa9640fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39320/0 (socket says 172.21.15.32:39320) 2024-04-06T14:33:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa963fff700 1 -- 172.21.15.32:0/3433490548 learned_addr learned my addr 172.21.15.32:0/3433490548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa9689c7700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9640fde90 0x7fa9640fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa963fff700 1 -- 172.21.15.32:0/3433490548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9640fde90 msgr2=0x7fa9640fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa963fff700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9640fde90 0x7fa9640fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa963fff700 1 -- 172.21.15.32:0/3433490548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa958005040 con 0x7fa9640fe830 2024-04-06T14:33:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa963fff700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fe830 0x7fa9640fc290 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa950002a50 tx=0x7fa9500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa9617fa700 1 -- 172.21.15.32:0/3433490548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa950007cc0 con 0x7fa9640fe830 2024-04-06T14:33:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa964119df0 con 0x7fa9640fe830 2024-04-06T14:33:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa9617fa700 1 -- 172.21.15.32:0/3433490548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa95000f040 con 0x7fa9640fe830 2024-04-06T14:33:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.610+0000 7fa9617fa700 1 -- 172.21.15.32:0/3433490548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa950009540 con 0x7fa9640fe830 2024-04-06T14:33:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.611+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96411a1f0 con 0x7fa9640fe830 2024-04-06T14:33:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.612+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa964065e30 con 0x7fa9640fe830 2024-04-06T14:33:05.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.613+0000 7fa9617fa700 1 -- 172.21.15.32:0/3433490548 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa9500096a0 con 0x7fa9640fe830 2024-04-06T14:33:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.613+0000 7fa9617fa700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa95406cc20 0x7fa95406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.613+0000 7fa9617fa700 1 -- 172.21.15.32:0/3433490548 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa95008be60 con 0x7fa9640fe830 2024-04-06T14:33:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.613+0000 7fa9689c7700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa95406cc20 0x7fa95406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.613+0000 7fa9689c7700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa95406cc20 0x7fa95406f0d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa958000c00 tx=0x7fa958009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:05.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.618+0000 7fa9617fa700 1 -- 172.21.15.32:0/3433490548 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa950055e20 con 0x7fa9640fe830 2024-04-06T14:33:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.828+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa96404bed0 con 0x7fa95406cc20 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.837+0000 7fa9617fa700 1 -- 172.21.15.32:0/3433490548 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa96404bed0 con 0x7fa95406cc20 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (13m) 6m ago 24m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 6m ago 24m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (10m) 6m ago 23m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (13m) 6m ago 24m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (7m) 6m ago 19m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (6m) 6m ago 18m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (6m) 6m ago 18m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:33:05.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (18m) 6m ago 18m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (15m) 6m ago 26m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (14m) 6m ago 23m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (11m) 6m ago 26m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (11m) 6m ago 23m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (14m) 6m ago 24m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (14m) 6m ago 23m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (10m) 6m ago 22m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (9m) 6m ago 22m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (9m) 6m ago 21m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (8m) 6m ago 21m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (7m) 6m ago 21m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (7m) 6m ago 20m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:33:05.839 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (13m) 6m ago 24m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:33:05.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.839+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa95406cc20 msgr2=0x7fa95406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.839+0000 7fa96ac2b700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa95406cc20 0x7fa95406f0d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa958000c00 tx=0x7fa958009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:05.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.839+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fe830 msgr2=0x7fa9640fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.839+0000 7fa96ac2b700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fe830 0x7fa9640fc290 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa950002a50 tx=0x7fa9500046b0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.840+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 shutdown_connections 2024-04-06T14:33:05.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.840+0000 7fa96ac2b700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa95406cc20 0x7fa95406f0d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.840+0000 7fa96ac2b700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9640fde90 0x7fa9640fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.840+0000 7fa96ac2b700 1 --2- 172.21.15.32:0/3433490548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9640fe830 0x7fa9640fc290 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.840+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 >> 172.21.15.32:0/3433490548 conn(0x7fa964072bc0 msgr2=0x7fa9640ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:05.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.840+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 shutdown_connections 2024-04-06T14:33:05.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.840+0000 7fa96ac2b700 1 -- 172.21.15.32:0/3433490548 wait complete. 2024-04-06T14:33:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 -- 172.21.15.32:0/2013165526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 msgr2=0x7efd940fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 --2- 172.21.15.32:0/2013165526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 0x7efd940fb910 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efd88004940 tx=0x7efd88009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 -- 172.21.15.32:0/2013165526 shutdown_connections 2024-04-06T14:33:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 --2- 172.21.15.32:0/2013165526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 0x7efd940fb910 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 --2- 172.21.15.32:0/2013165526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd94102e70 0x7efd94103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 -- 172.21.15.32:0/2013165526 >> 172.21.15.32:0/2013165526 conn(0x7efd940714c0 msgr2=0x7efd940718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 -- 172.21.15.32:0/2013165526 shutdown_connections 2024-04-06T14:33:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 -- 172.21.15.32:0/2013165526 wait complete. 2024-04-06T14:33:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 Processor -- start 2024-04-06T14:33:05.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.942+0000 7efd9c335700 1 -- start start 2024-04-06T14:33:05.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9c335700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 0x7efd940fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:05.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9c335700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd94102e70 0x7efd940fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:05.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9c335700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940fd150 con 0x7efd940fb4a0 2024-04-06T14:33:05.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9c335700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940fd290 con 0x7efd94102e70 2024-04-06T14:33:05.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9a0d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 0x7efd940fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:05.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9a0d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 0x7efd940fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39346/0 (socket says 172.21.15.32:39346) 2024-04-06T14:33:05.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9a0d1700 1 -- 172.21.15.32:0/2897192352 learned_addr learned my addr 172.21.15.32:0/2897192352 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd998d0700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd94102e70 0x7efd940fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9a0d1700 1 -- 172.21.15.32:0/2897192352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd94102e70 msgr2=0x7efd940fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9a0d1700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd94102e70 0x7efd940fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9a0d1700 1 -- 172.21.15.32:0/2897192352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd88005040 con 0x7efd940fb4a0 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9a0d1700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 0x7efd940fc680 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efd90002a50 tx=0x7efd900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd86ffd700 1 -- 172.21.15.32:0/2897192352 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd90007cc0 con 0x7efd940fb4a0 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd94119e50 con 0x7efd940fb4a0 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd86ffd700 1 -- 172.21.15.32:0/2897192352 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd9000f040 con 0x7efd940fb4a0 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd86ffd700 1 -- 172.21.15.32:0/2897192352 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd9000d9c0 con 0x7efd940fb4a0 2024-04-06T14:33:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.943+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9411a250 con 0x7efd940fb4a0 2024-04-06T14:33:05.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.945+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd94100170 con 0x7efd940fb4a0 2024-04-06T14:33:05.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.945+0000 7efd86ffd700 1 -- 172.21.15.32:0/2897192352 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efd9000db20 con 0x7efd940fb4a0 2024-04-06T14:33:05.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.946+0000 7efd86ffd700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efd8006cc20 0x7efd8006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:05.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.946+0000 7efd86ffd700 1 -- 172.21.15.32:0/2897192352 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efd9008bdb0 con 0x7efd940fb4a0 2024-04-06T14:33:05.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.946+0000 7efd998d0700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efd8006cc20 0x7efd8006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:05.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.946+0000 7efd998d0700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efd8006cc20 0x7efd8006f0d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efd940fdfc0 tx=0x7efd88009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:05.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:05.950+0000 7efd86ffd700 1 -- 172.21.15.32:0/2897192352 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efd90055d70 con 0x7efd940fb4a0 2024-04-06T14:33:06.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.240+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efd9404bed0 con 0x7efd940fb4a0 2024-04-06T14:33:06.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.240+0000 7efd86ffd700 1 -- 172.21.15.32:0/2897192352 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7efd90059390 con 0x7efd940fb4a0 2024-04-06T14:33:06.242 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:33:06.242 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:33:06.242 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:33:06.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:33:06.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:33:06.244 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:33:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efd8006cc20 msgr2=0x7efd8006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efd8006cc20 0x7efd8006f0d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efd940fdfc0 tx=0x7efd88009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 msgr2=0x7efd940fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 0x7efd940fc680 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efd90002a50 tx=0x7efd900046b0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 shutdown_connections 2024-04-06T14:33:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efd8006cc20 0x7efd8006f0d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd940fb4a0 0x7efd940fc680 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 --2- 172.21.15.32:0/2897192352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd94102e70 0x7efd940fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 >> 172.21.15.32:0/2897192352 conn(0x7efd940714c0 msgr2=0x7efd940fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:06.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 shutdown_connections 2024-04-06T14:33:06.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.242+0000 7efd9c335700 1 -- 172.21.15.32:0/2897192352 wait complete. 2024-04-06T14:33:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.345+0000 7f0bc9da1700 1 -- 172.21.15.32:0/1884097749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 msgr2=0x7f0bc40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.345+0000 7f0bc9da1700 1 --2- 172.21.15.32:0/1884097749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 0x7f0bc40fbd80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8004940 tx=0x7f0bb8009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.345+0000 7f0bc9da1700 1 -- 172.21.15.32:0/1884097749 shutdown_connections 2024-04-06T14:33:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.345+0000 7f0bc9da1700 1 --2- 172.21.15.32:0/1884097749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 0x7f0bc40fbd80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.345+0000 7f0bc9da1700 1 --2- 172.21.15.32:0/1884097749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4101d70 0x7f0bc4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.345+0000 7f0bc9da1700 1 -- 172.21.15.32:0/1884097749 >> 172.21.15.32:0/1884097749 conn(0x7f0bc4072bc0 msgr2=0x7f0bc4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.345+0000 7f0bc9da1700 1 -- 172.21.15.32:0/1884097749 shutdown_connections 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.345+0000 7f0bc9da1700 1 -- 172.21.15.32:0/1884097749 wait complete. 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc9da1700 1 Processor -- start 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc9da1700 1 -- start start 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc9da1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4101d70 0x7f0bc410c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc9da1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 0x7f0bc410cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc9da1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc410d2d0 con 0x7f0bc4102710 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc9da1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc410d440 con 0x7f0bc4101d70 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 0x7f0bc410cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 0x7f0bc410cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39362/0 (socket says 172.21.15.32:39362) 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc2ffd700 1 -- 172.21.15.32:0/3034965365 learned_addr learned my addr 172.21.15.32:0/3034965365 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.346+0000 7f0bc37fe700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4101d70 0x7f0bc410c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.347+0000 7f0bc2ffd700 1 -- 172.21.15.32:0/3034965365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4101d70 msgr2=0x7f0bc410c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.347+0000 7f0bc2ffd700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4101d70 0x7f0bc410c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.347+0000 7f0bc2ffd700 1 -- 172.21.15.32:0/3034965365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8005040 con 0x7f0bc4102710 2024-04-06T14:33:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.347+0000 7f0bc2ffd700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 0x7f0bc410cd90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009380 tx=0x7f0bb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.347+0000 7f0babfff700 1 -- 172.21.15.32:0/3034965365 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bb800e040 con 0x7f0bc4102710 2024-04-06T14:33:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.347+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc410d6c0 con 0x7f0bc4102710 2024-04-06T14:33:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.347+0000 7f0babfff700 1 -- 172.21.15.32:0/3034965365 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bb8013420 con 0x7f0bc4102710 2024-04-06T14:33:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.347+0000 7f0babfff700 1 -- 172.21.15.32:0/3034965365 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bb801c630 con 0x7f0bc4102710 2024-04-06T14:33:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.347+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc411a240 con 0x7f0bc4102710 2024-04-06T14:33:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.348+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc40ff080 con 0x7f0bc4102710 2024-04-06T14:33:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.349+0000 7f0babfff700 1 -- 172.21.15.32:0/3034965365 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0bb80139b0 con 0x7f0bc4102710 2024-04-06T14:33:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.349+0000 7f0babfff700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bac06cc20 0x7f0bac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.349+0000 7f0babfff700 1 -- 172.21.15.32:0/3034965365 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0bb808cf80 con 0x7f0bc4102710 2024-04-06T14:33:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.349+0000 7f0bc37fe700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bac06cc20 0x7f0bac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.350+0000 7f0bc37fe700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bac06cc20 0x7f0bac06f0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4004f40 tx=0x7f0bb400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:06.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.354+0000 7f0babfff700 1 -- 172.21.15.32:0/3034965365 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0bb80c0ab0 con 0x7f0bc4102710 2024-04-06T14:33:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:06 smithi032 ceph-mon[147414]: pgmap v388: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:06 smithi032 ceph-mon[147414]: from='client.34756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:06.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:06 smithi032 ceph-mon[147414]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.604+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0bc410d660 con 0x7f0bc4102710 2024-04-06T14:33:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.604+0000 7f0babfff700 1 -- 172.21.15.32:0/3034965365 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f0bb805aa50 con 0x7f0bc4102710 2024-04-06T14:33:06.604 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:33:06.604 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:33:06.605 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:33:06.606 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:33:06.607 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:33:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bac06cc20 msgr2=0x7f0bac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bac06cc20 0x7f0bac06f0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4004f40 tx=0x7f0bb400b040 comp rx=0 tx=0).stop 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 msgr2=0x7f0bc410cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 0x7f0bc410cd90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009380 tx=0x7f0bb8009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 shutdown_connections 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bac06cc20 0x7f0bac06f0d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4101d70 0x7f0bc410c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 --2- 172.21.15.32:0/3034965365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc4102710 0x7f0bc410cd90 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 >> 172.21.15.32:0/3034965365 conn(0x7f0bc4072bc0 msgr2=0x7f0bc40fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 shutdown_connections 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.606+0000 7f0bc9da1700 1 -- 172.21.15.32:0/3034965365 wait complete. 2024-04-06T14:33:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:33:06.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:06 smithi086 ceph-mon[110975]: pgmap v388: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:06.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:06 smithi086 ceph-mon[110975]: from='client.34756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:06.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:06 smithi086 ceph-mon[110975]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.711+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/3894101283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db4102de0 msgr2=0x7f4db41031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.711+0000 7f4dbbf1c700 1 --2- 172.21.15.32:0/3894101283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db4102de0 0x7f4db41031b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4da4004940 tx=0x7f4da4009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.711+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/3894101283 shutdown_connections 2024-04-06T14:33:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.711+0000 7f4dbbf1c700 1 --2- 172.21.15.32:0/3894101283 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db40656a0 0x7f4db4065af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.711+0000 7f4dbbf1c700 1 --2- 172.21.15.32:0/3894101283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db4102de0 0x7f4db41031b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.711+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/3894101283 >> 172.21.15.32:0/3894101283 conn(0x7f4db4072bc0 msgr2=0x7f4db4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.711+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/3894101283 shutdown_connections 2024-04-06T14:33:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.711+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/3894101283 wait complete. 2024-04-06T14:33:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4dbbf1c700 1 Processor -- start 2024-04-06T14:33:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4dbbf1c700 1 -- start start 2024-04-06T14:33:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4dbbf1c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40656a0 0x7f4db40fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4dbbf1c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db4102de0 0x7f4db40fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4dbbf1c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40fc7a0 con 0x7f4db40656a0 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4dbbf1c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40fc8e0 con 0x7f4db4102de0 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4db9cb8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40656a0 0x7f4db40fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4db9cb8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40656a0 0x7f4db40fbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39384/0 (socket says 172.21.15.32:39384) 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4db9cb8700 1 -- 172.21.15.32:0/2817648903 learned_addr learned my addr 172.21.15.32:0/2817648903 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4db94b7700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db4102de0 0x7f4db40fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4db9cb8700 1 -- 172.21.15.32:0/2817648903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db4102de0 msgr2=0x7f4db40fc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4db9cb8700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db4102de0 0x7f4db40fc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4db9cb8700 1 -- 172.21.15.32:0/2817648903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da4005040 con 0x7f4db40656a0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4db94b7700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db4102de0 0x7f4db40fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.712+0000 7f4db9cb8700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40656a0 0x7f4db40fbcd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4da400b3b0 tx=0x7f4da4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.713+0000 7f4daa7fc700 1 -- 172.21.15.32:0/2817648903 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da400e040 con 0x7f4db40656a0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.713+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db40fcb30 con 0x7f4db40656a0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.713+0000 7f4daa7fc700 1 -- 172.21.15.32:0/2817648903 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4da4013420 con 0x7f4db40656a0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.713+0000 7f4daa7fc700 1 -- 172.21.15.32:0/2817648903 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da401c630 con 0x7f4db40656a0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.713+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db411a130 con 0x7f4db40656a0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.714+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db40fe4c0 con 0x7f4db40656a0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.714+0000 7f4daa7fc700 1 -- 172.21.15.32:0/2817648903 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4da4013970 con 0x7f4db40656a0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.715+0000 7f4daa7fc700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4da006cc20 0x7f4da006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.715+0000 7f4daa7fc700 1 -- 172.21.15.32:0/2817648903 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4da408d2c0 con 0x7f4db40656a0 2024-04-06T14:33:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.715+0000 7f4db94b7700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4da006cc20 0x7f4da006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.715+0000 7f4db94b7700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4da006cc20 0x7f4da006f0d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4db0007fd0 tx=0x7f4db0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:06.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.719+0000 7f4daa7fc700 1 -- 172.21.15.32:0/2817648903 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4da4057300 con 0x7f4db40656a0 2024-04-06T14:33:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.943+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4db404bf70 con 0x7f4da006cc20 2024-04-06T14:33:06.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.944+0000 7f4daa7fc700 1 -- 172.21.15.32:0/2817648903 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4db0004dd0 con 0x7f4da006cc20 2024-04-06T14:33:06.944 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:33:06.945 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:33:06.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.946+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4da006cc20 msgr2=0x7f4da006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.946+0000 7f4dbbf1c700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4da006cc20 0x7f4da006f0d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4db0007fd0 tx=0x7f4db0009040 comp rx=0 tx=0).stop 2024-04-06T14:33:06.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.946+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40656a0 msgr2=0x7f4db40fbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:06.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.946+0000 7f4dbbf1c700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40656a0 0x7f4db40fbcd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4da400b3b0 tx=0x7f4da4009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:06.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.947+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 shutdown_connections 2024-04-06T14:33:06.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.947+0000 7f4dbbf1c700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4da006cc20 0x7f4da006f0d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.947+0000 7f4dbbf1c700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40656a0 0x7f4db40fbcd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.947+0000 7f4dbbf1c700 1 --2- 172.21.15.32:0/2817648903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db4102de0 0x7f4db40fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:06.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.947+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 >> 172.21.15.32:0/2817648903 conn(0x7f4db4072bc0 msgr2=0x7f4db4105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.947+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 shutdown_connections 2024-04-06T14:33:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:06.947+0000 7f4dbbf1c700 1 -- 172.21.15.32:0/2817648903 wait complete. 2024-04-06T14:33:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 -- 172.21.15.32:0/992579775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2844102e10 msgr2=0x7f28441031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 --2- 172.21.15.32:0/992579775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2844102e10 0x7f28441031e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2834004940 tx=0x7f2834009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 -- 172.21.15.32:0/992579775 shutdown_connections 2024-04-06T14:33:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 --2- 172.21.15.32:0/992579775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28441037b0 0x7f28440fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 --2- 172.21.15.32:0/992579775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2844102e10 0x7f28441031e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 -- 172.21.15.32:0/992579775 >> 172.21.15.32:0/992579775 conn(0x7f28440f89a0 msgr2=0x7f28440fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 -- 172.21.15.32:0/992579775 shutdown_connections 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 -- 172.21.15.32:0/992579775 wait complete. 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 Processor -- start 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.050+0000 7f2848a60700 1 -- start start 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2848a60700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2844102e10 0x7f2844113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2848a60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28441037b0 0x7f2844114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2848a60700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2844114680 con 0x7f28441037b0 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2848a60700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284410da70 con 0x7f2844102e10 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2841d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28441037b0 0x7f2844114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2841d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28441037b0 0x7f2844114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39402/0 (socket says 172.21.15.32:39402) 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2841d9b700 1 -- 172.21.15.32:0/225862181 learned_addr learned my addr 172.21.15.32:0/225862181 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2841d9b700 1 -- 172.21.15.32:0/225862181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2844102e10 msgr2=0x7f2844113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2841d9b700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2844102e10 0x7f2844113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2841d9b700 1 -- 172.21.15.32:0/225862181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2834005040 con 0x7f28441037b0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2841d9b700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28441037b0 0x7f2844114030 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f283800c890 tx=0x7f28380044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2832ffd700 1 -- 172.21.15.32:0/225862181 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f283800a7d0 con 0x7f28441037b0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2832ffd700 1 -- 172.21.15.32:0/225862181 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f283800ade0 con 0x7f28441037b0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284410dd50 con 0x7f28441037b0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2832ffd700 1 -- 172.21.15.32:0/225862181 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28380113a0 con 0x7f28441037b0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.051+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284410e310 con 0x7f28441037b0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.052+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28440fe560 con 0x7f28441037b0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.053+0000 7f2832ffd700 1 -- 172.21.15.32:0/225862181 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2838012400 con 0x7f28441037b0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.053+0000 7f2832ffd700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282c06cc30 0x7f282c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.053+0000 7f2832ffd700 1 -- 172.21.15.32:0/225862181 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f283808c050 con 0x7f28441037b0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.053+0000 7f284259c700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282c06cc30 0x7f282c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.054+0000 7f284259c700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282c06cc30 0x7f282c06f0e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2834004f90 tx=0x7f2834009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:07.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.058+0000 7f2832ffd700 1 -- 172.21.15.32:0/225862181 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2838059be0 con 0x7f28441037b0 2024-04-06T14:33:07.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.346+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2844109dd0 con 0x7f28441037b0 2024-04-06T14:33:07.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.346+0000 7f2832ffd700 1 -- 172.21.15.32:0/225862181 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f2838059770 con 0x7f28441037b0 2024-04-06T14:33:07.347 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:33:07.347 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:33:07.347 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:33:07.347 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:33:07.347 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:33:07.347 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:33:07.347 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:33:07.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282c06cc30 msgr2=0x7f282c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:07.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282c06cc30 0x7f282c06f0e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2834004f90 tx=0x7f2834009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:07.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28441037b0 msgr2=0x7f2844114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:07.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28441037b0 0x7f2844114030 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f283800c890 tx=0x7f28380044c0 comp rx=0 tx=0).stop 2024-04-06T14:33:07.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 shutdown_connections 2024-04-06T14:33:07.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f282c06cc30 0x7f282c06f0e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:07.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2844102e10 0x7f2844113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:07.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 --2- 172.21.15.32:0/225862181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28441037b0 0x7f2844114030 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:07.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 >> 172.21.15.32:0/225862181 conn(0x7f28440f89a0 msgr2=0x7f28440f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:07.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 shutdown_connections 2024-04-06T14:33:07.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:07.348+0000 7f2848a60700 1 -- 172.21.15.32:0/225862181 wait complete. 2024-04-06T14:33:07.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:07 smithi032 ceph-mon[147414]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:07.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:07 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2897192352' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:33:07.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:07 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3034965365' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:33:07.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:07 smithi086 ceph-mon[110975]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:07.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:07 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2897192352' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:33:07.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:07 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3034965365' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:33:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:08 smithi032 ceph-mon[147414]: pgmap v389: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:08 smithi032 ceph-mon[147414]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:08 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/225862181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:33:08.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:33:08.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:08 smithi086 ceph-mon[110975]: pgmap v389: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:08 smithi086 ceph-mon[110975]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:08 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/225862181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:33:08.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:33:10.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:10 smithi032 ceph-mon[147414]: pgmap v390: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:10.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:10 smithi086 ceph-mon[110975]: pgmap v390: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:12.480 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:12 smithi032 ceph-mon[147414]: pgmap v391: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:12.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:12 smithi086 ceph-mon[110975]: pgmap v391: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:14.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:14 smithi086 ceph-mon[110975]: pgmap v392: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:14.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:14 smithi032 ceph-mon[147414]: pgmap v392: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:16.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:16 smithi086 ceph-mon[110975]: pgmap v393: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:16.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:16 smithi032 ceph-mon[147414]: pgmap v393: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:18.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:18 smithi086 ceph-mon[110975]: pgmap v394: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:18.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:18 smithi032 ceph-mon[147414]: pgmap v394: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:20.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:20 smithi086 ceph-mon[110975]: pgmap v395: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:20.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:20 smithi032 ceph-mon[147414]: pgmap v395: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:22.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:22 smithi086 ceph-mon[110975]: pgmap v396: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:22.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:22 smithi032 ceph-mon[147414]: pgmap v396: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:23.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:33:23.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:33:24.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:24 smithi086 ceph-mon[110975]: pgmap v397: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:24.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:24 smithi032 ceph-mon[147414]: pgmap v397: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:26.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:26 smithi086 ceph-mon[110975]: pgmap v398: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:26.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:26 smithi032 ceph-mon[147414]: pgmap v398: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:28.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:28 smithi086 ceph-mon[110975]: pgmap v399: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:28.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:28 smithi032 ceph-mon[147414]: pgmap v399: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:30.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:30 smithi086 ceph-mon[110975]: pgmap v400: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:30.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:30 smithi032 ceph-mon[147414]: pgmap v400: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:32.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:32 smithi086 ceph-mon[110975]: pgmap v401: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:32.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:32 smithi032 ceph-mon[147414]: pgmap v401: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:34.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:34 smithi086 ceph-mon[110975]: pgmap v402: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:34.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:34 smithi032 ceph-mon[147414]: pgmap v402: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:36.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:36 smithi086 ceph-mon[110975]: pgmap v403: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:36.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:36 smithi032 ceph-mon[147414]: pgmap v403: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.466+0000 7f8b63477700 1 -- 172.21.15.32:0/1654714984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 msgr2=0x7f8b5c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.466+0000 7f8b63477700 1 --2- 172.21.15.32:0/1654714984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 0x7f8b5c102140 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8b58004940 tx=0x7f8b58009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.466+0000 7f8b63477700 1 -- 172.21.15.32:0/1654714984 shutdown_connections 2024-04-06T14:33:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.466+0000 7f8b63477700 1 --2- 172.21.15.32:0/1654714984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b5c102710 0x7f8b5c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.466+0000 7f8b63477700 1 --2- 172.21.15.32:0/1654714984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 0x7f8b5c102140 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.466+0000 7f8b63477700 1 -- 172.21.15.32:0/1654714984 >> 172.21.15.32:0/1654714984 conn(0x7f8b5c072bc0 msgr2=0x7f8b5c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.466+0000 7f8b63477700 1 -- 172.21.15.32:0/1654714984 shutdown_connections 2024-04-06T14:33:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.466+0000 7f8b63477700 1 -- 172.21.15.32:0/1654714984 wait complete. 2024-04-06T14:33:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b63477700 1 Processor -- start 2024-04-06T14:33:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b63477700 1 -- start start 2024-04-06T14:33:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b63477700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 0x7f8b5c1129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b63477700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b5c102710 0x7f8b5c112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b63477700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c113530 con 0x7f8b5c101d70 2024-04-06T14:33:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b63477700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c10c920 con 0x7f8b5c102710 2024-04-06T14:33:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b61213700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 0x7f8b5c1129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b61213700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 0x7f8b5c1129a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57502/0 (socket says 172.21.15.32:57502) 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b61213700 1 -- 172.21.15.32:0/102638438 learned_addr learned my addr 172.21.15.32:0/102638438 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.467+0000 7f8b60a12700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b5c102710 0x7f8b5c112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.468+0000 7f8b61213700 1 -- 172.21.15.32:0/102638438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b5c102710 msgr2=0x7f8b5c112ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.468+0000 7f8b61213700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b5c102710 0x7f8b5c112ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.468+0000 7f8b61213700 1 -- 172.21.15.32:0/102638438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b58005040 con 0x7f8b5c101d70 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.468+0000 7f8b61213700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 0x7f8b5c1129a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800b3b0 tx=0x7f8b58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.468+0000 7f8b51ffb700 1 -- 172.21.15.32:0/102638438 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b5800e040 con 0x7f8b5c101d70 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.468+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5c10cba0 con 0x7f8b5c101d70 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.468+0000 7f8b51ffb700 1 -- 172.21.15.32:0/102638438 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b58013420 con 0x7f8b5c101d70 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.468+0000 7f8b51ffb700 1 -- 172.21.15.32:0/102638438 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b5801c610 con 0x7f8b5c101d70 2024-04-06T14:33:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.468+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5c10d160 con 0x7f8b5c101d70 2024-04-06T14:33:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.469+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5c0ff080 con 0x7f8b5c101d70 2024-04-06T14:33:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.470+0000 7f8b51ffb700 1 -- 172.21.15.32:0/102638438 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8b58013a00 con 0x7f8b5c101d70 2024-04-06T14:33:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.470+0000 7f8b51ffb700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b4806cc20 0x7f8b4806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.470+0000 7f8b51ffb700 1 -- 172.21.15.32:0/102638438 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8b5808d1c0 con 0x7f8b5c101d70 2024-04-06T14:33:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.470+0000 7f8b60a12700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b4806cc20 0x7f8b4806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.471+0000 7f8b60a12700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b4806cc20 0x7f8b4806f0d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c10dae0 tx=0x7f8b4c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:37.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.475+0000 7f8b51ffb700 1 -- 172.21.15.32:0/102638438 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8b5805ac90 con 0x7f8b5c101d70 2024-04-06T14:33:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.698+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b5c10e060 con 0x7f8b4806cc20 2024-04-06T14:33:37.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.699+0000 7f8b51ffb700 1 -- 172.21.15.32:0/102638438 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8b5c10e060 con 0x7f8b4806cc20 2024-04-06T14:33:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b4806cc20 msgr2=0x7f8b4806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b4806cc20 0x7f8b4806f0d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c10dae0 tx=0x7f8b4c00b040 comp rx=0 tx=0).stop 2024-04-06T14:33:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 msgr2=0x7f8b5c1129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 0x7f8b5c1129a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800b3b0 tx=0x7f8b58009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 shutdown_connections 2024-04-06T14:33:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8b4806cc20 0x7f8b4806f0d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b5c101d70 0x7f8b5c1129a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 --2- 172.21.15.32:0/102638438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b5c102710 0x7f8b5c112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:37.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 >> 172.21.15.32:0/102638438 conn(0x7f8b5c072bc0 msgr2=0x7f8b5c0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:37.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 shutdown_connections 2024-04-06T14:33:37.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.701+0000 7f8b63477700 1 -- 172.21.15.32:0/102638438 wait complete. 2024-04-06T14:33:37.711 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:33:37.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.806+0000 7f739c069700 1 -- 172.21.15.32:0/3192299929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7394101d10 msgr2=0x7f73941020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:37.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.806+0000 7f739c069700 1 --2- 172.21.15.32:0/3192299929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7394101d10 0x7f73941020e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7384004940 tx=0x7f7384009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:37.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.806+0000 7f739c069700 1 -- 172.21.15.32:0/3192299929 shutdown_connections 2024-04-06T14:33:37.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.806+0000 7f739c069700 1 --2- 172.21.15.32:0/3192299929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73941026b0 0x7f73940fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:37.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.806+0000 7f739c069700 1 --2- 172.21.15.32:0/3192299929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7394101d10 0x7f73941020e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:37.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.806+0000 7f739c069700 1 -- 172.21.15.32:0/3192299929 >> 172.21.15.32:0/3192299929 conn(0x7f73940f7880 msgr2=0x7f73940f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:37.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.806+0000 7f739c069700 1 -- 172.21.15.32:0/3192299929 shutdown_connections 2024-04-06T14:33:37.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f739c069700 1 -- 172.21.15.32:0/3192299929 wait complete. 2024-04-06T14:33:37.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f739c069700 1 Processor -- start 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f739c069700 1 -- start start 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f739c069700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7394101d10 0x7f7394112d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f739c069700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73941026b0 0x7f7394113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f739c069700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73941138c0 con 0x7f73941026b0 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f739c069700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739410cc90 con 0x7f7394101d10 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f7399604700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73941026b0 0x7f7394113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f7399604700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73941026b0 0x7f7394113270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57518/0 (socket says 172.21.15.32:57518) 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.807+0000 7f7399604700 1 -- 172.21.15.32:0/2326944900 learned_addr learned my addr 172.21.15.32:0/2326944900 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f7399e05700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7394101d10 0x7f7394112d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f7399604700 1 -- 172.21.15.32:0/2326944900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7394101d10 msgr2=0x7f7394112d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f7399604700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7394101d10 0x7f7394112d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f7399604700 1 -- 172.21.15.32:0/2326944900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7384005040 con 0x7f73941026b0 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f7399604700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73941026b0 0x7f7394113270 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f739000eaa0 tx=0x7f7390004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f738a7fc700 1 -- 172.21.15.32:0/2326944900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f739000cb90 con 0x7f73941026b0 2024-04-06T14:33:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f738a7fc700 1 -- 172.21.15.32:0/2326944900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f739000ccf0 con 0x7f73941026b0 2024-04-06T14:33:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f738a7fc700 1 -- 172.21.15.32:0/2326944900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7390010770 con 0x7f73941026b0 2024-04-06T14:33:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739410cf10 con 0x7f73941026b0 2024-04-06T14:33:37.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.808+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739410d4d0 con 0x7f73941026b0 2024-04-06T14:33:37.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.809+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73940fd460 con 0x7f73941026b0 2024-04-06T14:33:37.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.810+0000 7f738a7fc700 1 -- 172.21.15.32:0/2326944900 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7390011470 con 0x7f73941026b0 2024-04-06T14:33:37.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.810+0000 7f738a7fc700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f738006cc20 0x7f738006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:37.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.810+0000 7f738a7fc700 1 -- 172.21.15.32:0/2326944900 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f739008e780 con 0x7f73941026b0 2024-04-06T14:33:37.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.810+0000 7f7399e05700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f738006cc20 0x7f738006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:37.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.811+0000 7f7399e05700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f738006cc20 0x7f738006f0d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7384004f90 tx=0x7f7384009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:37.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:37.815+0000 7f738a7fc700 1 -- 172.21.15.32:0/2326944900 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f739005c310 con 0x7f73941026b0 2024-04-06T14:33:38.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.034+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7394108d40 con 0x7f738006cc20 2024-04-06T14:33:38.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.035+0000 7f738a7fc700 1 -- 172.21.15.32:0/2326944900 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7394108d40 con 0x7f738006cc20 2024-04-06T14:33:38.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.037+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f738006cc20 msgr2=0x7f738006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.037+0000 7f739c069700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f738006cc20 0x7f738006f0d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7384004f90 tx=0x7f7384009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:38.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.037+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73941026b0 msgr2=0x7f7394113270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.037+0000 7f739c069700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73941026b0 0x7f7394113270 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f739000eaa0 tx=0x7f7390004500 comp rx=0 tx=0).stop 2024-04-06T14:33:38.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.037+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 shutdown_connections 2024-04-06T14:33:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.037+0000 7f739c069700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f738006cc20 0x7f738006f0d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.037+0000 7f739c069700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7394101d10 0x7f7394112d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.037+0000 7f739c069700 1 --2- 172.21.15.32:0/2326944900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73941026b0 0x7f7394113270 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.037+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 >> 172.21.15.32:0/2326944900 conn(0x7f73940f7880 msgr2=0x7f73940f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.038+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 shutdown_connections 2024-04-06T14:33:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.038+0000 7f739c069700 1 -- 172.21.15.32:0/2326944900 wait complete. 2024-04-06T14:33:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.141+0000 7fbc8c598700 1 -- 172.21.15.32:0/3016110295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 msgr2=0x7fbc840fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.141+0000 7fbc8c598700 1 --2- 172.21.15.32:0/3016110295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 0x7fbc840fce20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbc74004940 tx=0x7fbc74009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.141+0000 7fbc8c598700 1 -- 172.21.15.32:0/3016110295 shutdown_connections 2024-04-06T14:33:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.141+0000 7fbc8c598700 1 --2- 172.21.15.32:0/3016110295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 0x7fbc840fce20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.141+0000 7fbc8c598700 1 --2- 172.21.15.32:0/3016110295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84102e10 0x7fbc841031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.141+0000 7fbc8c598700 1 -- 172.21.15.32:0/3016110295 >> 172.21.15.32:0/3016110295 conn(0x7fbc840f8980 msgr2=0x7fbc840fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc8c598700 1 -- 172.21.15.32:0/3016110295 shutdown_connections 2024-04-06T14:33:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc8c598700 1 -- 172.21.15.32:0/3016110295 wait complete. 2024-04-06T14:33:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc8c598700 1 Processor -- start 2024-04-06T14:33:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc8c598700 1 -- start start 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc8c598700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84102e10 0x7fbc84113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc8c598700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 0x7fbc84114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc8c598700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc84114680 con 0x7fbc841037b0 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc8c598700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8410da70 con 0x7fbc84102e10 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc89b33700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 0x7fbc84114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc89b33700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 0x7fbc84114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57532/0 (socket says 172.21.15.32:57532) 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc89b33700 1 -- 172.21.15.32:0/2304844697 learned_addr learned my addr 172.21.15.32:0/2304844697 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.142+0000 7fbc8a334700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84102e10 0x7fbc84113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.143+0000 7fbc89b33700 1 -- 172.21.15.32:0/2304844697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84102e10 msgr2=0x7fbc84113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.143+0000 7fbc89b33700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84102e10 0x7fbc84113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.143+0000 7fbc89b33700 1 -- 172.21.15.32:0/2304844697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc74005040 con 0x7fbc841037b0 2024-04-06T14:33:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.143+0000 7fbc89b33700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 0x7fbc84114030 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009380 tx=0x7fbc74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.143+0000 7fbc7affd700 1 -- 172.21.15.32:0/2304844697 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc7400e040 con 0x7fbc841037b0 2024-04-06T14:33:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.143+0000 7fbc7affd700 1 -- 172.21.15.32:0/2304844697 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc74013420 con 0x7fbc841037b0 2024-04-06T14:33:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.143+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8410dcf0 con 0x7fbc841037b0 2024-04-06T14:33:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.143+0000 7fbc7affd700 1 -- 172.21.15.32:0/2304844697 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc7401c630 con 0x7fbc841037b0 2024-04-06T14:33:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.143+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8410e2b0 con 0x7fbc841037b0 2024-04-06T14:33:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.144+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc840fe560 con 0x7fbc841037b0 2024-04-06T14:33:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.145+0000 7fbc7affd700 1 -- 172.21.15.32:0/2304844697 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbc740139b0 con 0x7fbc841037b0 2024-04-06T14:33:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.145+0000 7fbc7affd700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc7006cc20 0x7fbc7006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.145+0000 7fbc7affd700 1 -- 172.21.15.32:0/2304844697 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbc7408d1b0 con 0x7fbc841037b0 2024-04-06T14:33:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.145+0000 7fbc8a334700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc7006cc20 0x7fbc7006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.145+0000 7fbc8a334700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc7006cc20 0x7fbc7006f0d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbc80004f40 tx=0x7fbc8000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:38.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.149+0000 7fbc7affd700 1 -- 172.21.15.32:0/2304844697 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbc7405ac80 con 0x7fbc841037b0 2024-04-06T14:33:38.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.362+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc84109db0 con 0x7fbc7006cc20 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.372+0000 7fbc7affd700 1 -- 172.21.15.32:0/2304844697 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbc84109db0 con 0x7fbc7006cc20 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (14m) 7m ago 25m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 7m ago 25m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (11m) 7m ago 24m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (13m) 7m ago 24m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (7m) 7m ago 19m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (7m) 7m ago 19m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (7m) 7m ago 19m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:33:38.372 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (19m) 7m ago 19m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (15m) 7m ago 26m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (14m) 7m ago 24m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 7m ago 26m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (11m) 7m ago 24m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (14m) 7m ago 25m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (14m) 7m ago 24m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (10m) 7m ago 23m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 7m ago 22m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (9m) 7m ago 22m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (9m) 7m ago 22m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (8m) 7m ago 21m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (7m) 7m ago 21m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:33:38.373 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (14m) 7m ago 24m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:33:38.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc7006cc20 msgr2=0x7fbc7006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc7006cc20 0x7fbc7006f0d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbc80004f40 tx=0x7fbc8000b040 comp rx=0 tx=0).stop 2024-04-06T14:33:38.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 msgr2=0x7fbc84114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 0x7fbc84114030 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009380 tx=0x7fbc74009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:38.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 shutdown_connections 2024-04-06T14:33:38.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc7006cc20 0x7fbc7006f0d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc84102e10 0x7fbc84113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 --2- 172.21.15.32:0/2304844697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc841037b0 0x7fbc84114030 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 >> 172.21.15.32:0/2304844697 conn(0x7fbc840f8980 msgr2=0x7fbc840f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:38.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 shutdown_connections 2024-04-06T14:33:38.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.374+0000 7fbc8c598700 1 -- 172.21.15.32:0/2304844697 wait complete. 2024-04-06T14:33:38.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.477+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3270844146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc1026b0 msgr2=0x7f31dc0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.477+0000 7f31e0fbe700 1 --2- 172.21.15.32:0/3270844146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc1026b0 0x7f31dc0fbd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f31d0004940 tx=0x7f31d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.477+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3270844146 shutdown_connections 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.477+0000 7f31e0fbe700 1 --2- 172.21.15.32:0/3270844146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc1026b0 0x7f31dc0fbd20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.477+0000 7f31e0fbe700 1 --2- 172.21.15.32:0/3270844146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31dc101d10 0x7f31dc1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.477+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3270844146 >> 172.21.15.32:0/3270844146 conn(0x7f31dc0f7880 msgr2=0x7f31dc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.477+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3270844146 shutdown_connections 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.477+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3270844146 wait complete. 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31e0fbe700 1 Processor -- start 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31e0fbe700 1 -- start start 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31e0fbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc101d10 0x7f31dc113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31e0fbe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31dc1026b0 0x7f31dc114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31e0fbe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc114a00 con 0x7f31dc101d10 2024-04-06T14:33:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31e0fbe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc10dd40 con 0x7f31dc1026b0 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31da59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc101d10 0x7f31dc113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31da59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc101d10 0x7f31dc113de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57552/0 (socket says 172.21.15.32:57552) 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31da59c700 1 -- 172.21.15.32:0/3170565825 learned_addr learned my addr 172.21.15.32:0/3170565825 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.478+0000 7f31d9d9b700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31dc1026b0 0x7f31dc114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.479+0000 7f31da59c700 1 -- 172.21.15.32:0/3170565825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31dc1026b0 msgr2=0x7f31dc114320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.479+0000 7f31da59c700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31dc1026b0 0x7f31dc114320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.479+0000 7f31da59c700 1 -- 172.21.15.32:0/3170565825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d0005040 con 0x7f31dc101d10 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.479+0000 7f31da59c700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc101d10 0x7f31dc113de0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b790 tx=0x7f31cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.479+0000 7f31caffd700 1 -- 172.21.15.32:0/3170565825 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31cc00f770 con 0x7f31dc101d10 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.479+0000 7f31caffd700 1 -- 172.21.15.32:0/3170565825 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31cc00fd80 con 0x7f31dc101d10 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.479+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc10dfc0 con 0x7f31dc101d10 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.479+0000 7f31caffd700 1 -- 172.21.15.32:0/3170565825 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31cc00d310 con 0x7f31dc101d10 2024-04-06T14:33:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.479+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc10e580 con 0x7f31dc101d10 2024-04-06T14:33:38.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.480+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc0fd460 con 0x7f31dc101d10 2024-04-06T14:33:38.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.480+0000 7f31caffd700 1 -- 172.21.15.32:0/3170565825 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f31cc010420 con 0x7f31dc101d10 2024-04-06T14:33:38.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.481+0000 7f31caffd700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31c406cc20 0x7f31c406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:38.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.481+0000 7f31caffd700 1 -- 172.21.15.32:0/3170565825 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f31cc08c050 con 0x7f31dc101d10 2024-04-06T14:33:38.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.481+0000 7f31d9d9b700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31c406cc20 0x7f31c406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:38.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.481+0000 7f31d9d9b700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31c406cc20 0x7f31c406f0d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f31d000b3b0 tx=0x7f31d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:38.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.485+0000 7f31caffd700 1 -- 172.21.15.32:0/3170565825 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f31cc059b20 con 0x7f31dc101d10 2024-04-06T14:33:38.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:38 smithi086 ceph-mon[110975]: pgmap v404: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:38.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:33:38.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:38 smithi032 ceph-mon[147414]: pgmap v404: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:38.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:33:38.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.779+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f31dc04bf00 con 0x7f31dc101d10 2024-04-06T14:33:38.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.780+0000 7f31caffd700 1 -- 172.21.15.32:0/3170565825 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f31cc0596b0 con 0x7f31dc101d10 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:33:38.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:33:38.781 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:33:38.781 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:33:38.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:33:38.781 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:33:38.781 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:33:38.781 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:33:38.781 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:33:38.781 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31c406cc20 msgr2=0x7f31c406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31c406cc20 0x7f31c406f0d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f31d000b3b0 tx=0x7f31d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc101d10 msgr2=0x7f31dc113de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc101d10 0x7f31dc113de0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b790 tx=0x7f31cc009250 comp rx=0 tx=0).stop 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 shutdown_connections 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31c406cc20 0x7f31c406f0d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31dc101d10 0x7f31dc113de0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 --2- 172.21.15.32:0/3170565825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31dc1026b0 0x7f31dc114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 >> 172.21.15.32:0/3170565825 conn(0x7f31dc0f7880 msgr2=0x7f31dc0f88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 shutdown_connections 2024-04-06T14:33:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.782+0000 7f31e0fbe700 1 -- 172.21.15.32:0/3170565825 wait complete. 2024-04-06T14:33:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.885+0000 7f6835873700 1 -- 172.21.15.32:0/3773660103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 msgr2=0x7f68300fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.885+0000 7f6835873700 1 --2- 172.21.15.32:0/3773660103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 0x7f68300fce10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6824004940 tx=0x7f6824009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.885+0000 7f6835873700 1 -- 172.21.15.32:0/3773660103 shutdown_connections 2024-04-06T14:33:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.885+0000 7f6835873700 1 --2- 172.21.15.32:0/3773660103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 0x7f68300fce10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.885+0000 7f6835873700 1 --2- 172.21.15.32:0/3773660103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6830102e00 0x7f68301031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.885+0000 7f6835873700 1 -- 172.21.15.32:0/3773660103 >> 172.21.15.32:0/3773660103 conn(0x7f68300f89b0 msgr2=0x7f68300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.885+0000 7f6835873700 1 -- 172.21.15.32:0/3773660103 shutdown_connections 2024-04-06T14:33:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.885+0000 7f6835873700 1 -- 172.21.15.32:0/3773660103 wait complete. 2024-04-06T14:33:38.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f6835873700 1 Processor -- start 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f6835873700 1 -- start start 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f6835873700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6830102e00 0x7f68301138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f6835873700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 0x7f6830113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f6835873700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68301144e0 con 0x7f68301037a0 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f6835873700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683010da50 con 0x7f6830102e00 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f682e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 0x7f6830113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f682e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 0x7f6830113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57568/0 (socket says 172.21.15.32:57568) 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f682e7fc700 1 -- 172.21.15.32:0/597991073 learned_addr learned my addr 172.21.15.32:0/597991073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.886+0000 7f682effd700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6830102e00 0x7f68301138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.887+0000 7f682e7fc700 1 -- 172.21.15.32:0/597991073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6830102e00 msgr2=0x7f68301138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.887+0000 7f682e7fc700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6830102e00 0x7f68301138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.887+0000 7f682e7fc700 1 -- 172.21.15.32:0/597991073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6824005040 con 0x7f68301037a0 2024-04-06T14:33:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.887+0000 7f682e7fc700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 0x7f6830113e00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f68240093b0 tx=0x7f6824009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.887+0000 7f68177fe700 1 -- 172.21.15.32:0/597991073 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f682400e040 con 0x7f68301037a0 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.887+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683010dcd0 con 0x7f68301037a0 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.887+0000 7f68177fe700 1 -- 172.21.15.32:0/597991073 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6824013420 con 0x7f68301037a0 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.887+0000 7f68177fe700 1 -- 172.21.15.32:0/597991073 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f682401c630 con 0x7f68301037a0 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.887+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683010e290 con 0x7f68301037a0 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.888+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68300fe550 con 0x7f68301037a0 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.888+0000 7f68177fe700 1 -- 172.21.15.32:0/597991073 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f682400bb80 con 0x7f68301037a0 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.889+0000 7f68177fe700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f681806cc20 0x7f681806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.889+0000 7f68177fe700 1 -- 172.21.15.32:0/597991073 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f682408d220 con 0x7f68301037a0 2024-04-06T14:33:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.889+0000 7f682effd700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f681806cc20 0x7f681806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.889+0000 7f682effd700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f681806cc20 0x7f681806f0d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6820006fd0 tx=0x7f6820008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:38.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:38.893+0000 7f68177fe700 1 -- 172.21.15.32:0/597991073 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f682405acf0 con 0x7f68301037a0 2024-04-06T14:33:39.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.139+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f683004c710 con 0x7f68301037a0 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.139+0000 7f68177fe700 1 -- 172.21.15.32:0/597991073 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f682405a880 con 0x7f68301037a0 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:33:39.139 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:33:39.140 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:33:39.141 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:33:39.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f681806cc20 msgr2=0x7f681806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f681806cc20 0x7f681806f0d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6820006fd0 tx=0x7f6820008040 comp rx=0 tx=0).stop 2024-04-06T14:33:39.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 msgr2=0x7f6830113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 0x7f6830113e00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f68240093b0 tx=0x7f6824009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:39.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 shutdown_connections 2024-04-06T14:33:39.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f681806cc20 0x7f681806f0d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6830102e00 0x7f68301138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 --2- 172.21.15.32:0/597991073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68301037a0 0x7f6830113e00 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 >> 172.21.15.32:0/597991073 conn(0x7f68300f89b0 msgr2=0x7f68300f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:39.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 shutdown_connections 2024-04-06T14:33:39.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.141+0000 7f6835873700 1 -- 172.21.15.32:0/597991073 wait complete. 2024-04-06T14:33:39.143 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:33:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 -- 172.21.15.32:0/1591804233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec101d70 msgr2=0x7f45ec102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 --2- 172.21.15.32:0/1591804233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec101d70 0x7f45ec102140 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f45e8004940 tx=0x7f45e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 -- 172.21.15.32:0/1591804233 shutdown_connections 2024-04-06T14:33:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 --2- 172.21.15.32:0/1591804233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45ec102710 0x7f45ec0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 --2- 172.21.15.32:0/1591804233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec101d70 0x7f45ec102140 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 -- 172.21.15.32:0/1591804233 >> 172.21.15.32:0/1591804233 conn(0x7f45ec072bc0 msgr2=0x7f45ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 -- 172.21.15.32:0/1591804233 shutdown_connections 2024-04-06T14:33:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 -- 172.21.15.32:0/1591804233 wait complete. 2024-04-06T14:33:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 Processor -- start 2024-04-06T14:33:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 -- start start 2024-04-06T14:33:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45ec101d70 0x7f45ec10eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec102710 0x7f45ec10f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec10f620 con 0x7f45ec102710 2024-04-06T14:33:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.246+0000 7f45f3e4e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec10f760 con 0x7f45ec101d70 2024-04-06T14:33:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f13e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec102710 0x7f45ec10f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f13e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec102710 0x7f45ec10f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57598/0 (socket says 172.21.15.32:57598) 2024-04-06T14:33:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f13e9700 1 -- 172.21.15.32:0/76717137 learned_addr learned my addr 172.21.15.32:0/76717137 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f1bea700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45ec101d70 0x7f45ec10eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f13e9700 1 -- 172.21.15.32:0/76717137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45ec101d70 msgr2=0x7f45ec10eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f13e9700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45ec101d70 0x7f45ec10eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f13e9700 1 -- 172.21.15.32:0/76717137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e8005040 con 0x7f45ec102710 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f13e9700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec102710 0x7f45ec10f0e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00c800 tx=0x7f45dc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45e27fc700 1 -- 172.21.15.32:0/76717137 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45dc0106f0 con 0x7f45ec102710 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45e27fc700 1 -- 172.21.15.32:0/76717137 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45dc010d00 con 0x7f45ec102710 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ec117a50 con 0x7f45ec102710 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45e27fc700 1 -- 172.21.15.32:0/76717137 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45dc0113a0 con 0x7f45ec102710 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.247+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ec118010 con 0x7f45ec102710 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.248+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ec0ff080 con 0x7f45ec102710 2024-04-06T14:33:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.249+0000 7f45e27fc700 1 -- 172.21.15.32:0/76717137 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f45dc012400 con 0x7f45ec102710 2024-04-06T14:33:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.249+0000 7f45e27fc700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45d806cc20 0x7f45d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.249+0000 7f45f1bea700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45d806cc20 0x7f45d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.249+0000 7f45e27fc700 1 -- 172.21.15.32:0/76717137 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f45dc08bed0 con 0x7f45ec102710 2024-04-06T14:33:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.250+0000 7f45f1bea700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45d806cc20 0x7f45d806f0d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f45e8004f90 tx=0x7f45e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:39.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.254+0000 7f45e27fc700 1 -- 172.21.15.32:0/76717137 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f45dc055fd0 con 0x7f45ec102710 2024-04-06T14:33:39.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.476+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45ec108d30 con 0x7f45d806cc20 2024-04-06T14:33:39.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.478+0000 7f45e27fc700 1 -- 172.21.15.32:0/76717137 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f45ec108d30 con 0x7f45d806cc20 2024-04-06T14:33:39.477 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:33:39.477 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:33:39.477 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:33:39.477 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:33:39.478 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:33:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45d806cc20 msgr2=0x7f45d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45d806cc20 0x7f45d806f0d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f45e8004f90 tx=0x7f45e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec102710 msgr2=0x7f45ec10f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec102710 0x7f45ec10f0e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00c800 tx=0x7f45dc004470 comp rx=0 tx=0).stop 2024-04-06T14:33:39.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 shutdown_connections 2024-04-06T14:33:39.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45d806cc20 0x7f45d806f0d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45ec101d70 0x7f45ec10eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 --2- 172.21.15.32:0/76717137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ec102710 0x7f45ec10f0e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 >> 172.21.15.32:0/76717137 conn(0x7f45ec072bc0 msgr2=0x7f45ec0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:39.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 shutdown_connections 2024-04-06T14:33:39.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.480+0000 7f45f3e4e700 1 -- 172.21.15.32:0/76717137 wait complete. 2024-04-06T14:33:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.583+0000 7f1efc19e700 1 -- 172.21.15.32:0/3087004848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4102710 msgr2=0x7f1ef40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.583+0000 7f1efc19e700 1 --2- 172.21.15.32:0/3087004848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4102710 0x7f1ef40fbd80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4004940 tx=0x7f1ee4009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.583+0000 7f1efc19e700 1 -- 172.21.15.32:0/3087004848 shutdown_connections 2024-04-06T14:33:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.583+0000 7f1efc19e700 1 --2- 172.21.15.32:0/3087004848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4102710 0x7f1ef40fbd80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.583+0000 7f1efc19e700 1 --2- 172.21.15.32:0/3087004848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ef4101d70 0x7f1ef4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.583+0000 7f1efc19e700 1 -- 172.21.15.32:0/3087004848 >> 172.21.15.32:0/3087004848 conn(0x7f1ef4072bc0 msgr2=0x7f1ef4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.583+0000 7f1efc19e700 1 -- 172.21.15.32:0/3087004848 shutdown_connections 2024-04-06T14:33:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.583+0000 7f1efc19e700 1 -- 172.21.15.32:0/3087004848 wait complete. 2024-04-06T14:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1efc19e700 1 Processor -- start 2024-04-06T14:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1efc19e700 1 -- start start 2024-04-06T14:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1efc19e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4101d70 0x7f1ef4103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1efc19e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ef4102710 0x7f1ef4103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1efc19e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef4103e10 con 0x7f1ef4101d70 2024-04-06T14:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1efc19e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef4103f50 con 0x7f1ef4102710 2024-04-06T14:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1ef9f3a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4101d70 0x7f1ef4103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1ef9f3a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4101d70 0x7f1ef4103340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57614/0 (socket says 172.21.15.32:57614) 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1ef9f3a700 1 -- 172.21.15.32:0/3652836235 learned_addr learned my addr 172.21.15.32:0/3652836235 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1ef9739700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ef4102710 0x7f1ef4103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1ef9f3a700 1 -- 172.21.15.32:0/3652836235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ef4102710 msgr2=0x7f1ef4103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1ef9f3a700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ef4102710 0x7f1ef4103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1ef9f3a700 1 -- 172.21.15.32:0/3652836235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee4005040 con 0x7f1ef4101d70 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.584+0000 7f1ef9f3a700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4101d70 0x7f1ef4103340 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000c830 tx=0x7f1ef0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.585+0000 7f1eea7fc700 1 -- 172.21.15.32:0/3652836235 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ef00106f0 con 0x7f1ef4101d70 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.585+0000 7f1eea7fc700 1 -- 172.21.15.32:0/3652836235 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ef0010d00 con 0x7f1ef4101d70 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.585+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef4119da0 con 0x7f1ef4101d70 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.585+0000 7f1eea7fc700 1 -- 172.21.15.32:0/3652836235 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ef0012e90 con 0x7f1ef4101d70 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.585+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef411a220 con 0x7f1ef4101d70 2024-04-06T14:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.585+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef40fd4c0 con 0x7f1ef4101d70 2024-04-06T14:33:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.586+0000 7f1eea7fc700 1 -- 172.21.15.32:0/3652836235 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1ef0010850 con 0x7f1ef4101d70 2024-04-06T14:33:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.586+0000 7f1eea7fc700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ee006cc20 0x7f1ee006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:33:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.587+0000 7f1eea7fc700 1 -- 172.21.15.32:0/3652836235 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1ef008bf90 con 0x7f1ef4101d70 2024-04-06T14:33:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.587+0000 7f1ef9739700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ee006cc20 0x7f1ee006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:33:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.587+0000 7f1ef9739700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ee006cc20 0x7f1ee006f0d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4104c80 tx=0x7f1ee4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:33:39.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.591+0000 7f1eea7fc700 1 -- 172.21.15.32:0/3652836235 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1ef0056090 con 0x7f1ef4101d70 2024-04-06T14:33:39.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:39 smithi086 ceph-mon[110975]: from='client.34784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:39.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:39 smithi086 ceph-mon[110975]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:39.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:39 smithi086 ceph-mon[110975]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:39.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:39 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3170565825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:33:39.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:39 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/597991073' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:33:39.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:39 smithi032 ceph-mon[147414]: from='client.34784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:39.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:39 smithi032 ceph-mon[147414]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:39.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:39 smithi032 ceph-mon[147414]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:39.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:39 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3170565825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:33:39.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:39 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/597991073' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:33:39.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.877+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1ef405f5f0 con 0x7f1ef4101d70 2024-04-06T14:33:39.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.878+0000 7f1eea7fc700 1 -- 172.21.15.32:0/3652836235 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f1ef00596b0 con 0x7f1ef4101d70 2024-04-06T14:33:39.878 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:33:39.878 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:33:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:33:39.878 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:33:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:33:39.878 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:33:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:33:39.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ee006cc20 msgr2=0x7f1ee006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ee006cc20 0x7f1ee006f0d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4104c80 tx=0x7f1ee4009e30 comp rx=0 tx=0).stop 2024-04-06T14:33:39.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4101d70 msgr2=0x7f1ef4103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:33:39.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4101d70 0x7f1ef4103340 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000c830 tx=0x7f1ef0004470 comp rx=0 tx=0).stop 2024-04-06T14:33:39.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 shutdown_connections 2024-04-06T14:33:39.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ee006cc20 0x7f1ee006f0d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef4101d70 0x7f1ef4103340 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 --2- 172.21.15.32:0/3652836235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ef4102710 0x7f1ef4103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:33:39.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 >> 172.21.15.32:0/3652836235 conn(0x7f1ef4072bc0 msgr2=0x7f1ef41058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:33:39.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 shutdown_connections 2024-04-06T14:33:39.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:33:39.880+0000 7f1efc19e700 1 -- 172.21.15.32:0/3652836235 wait complete. 2024-04-06T14:33:40.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:40 smithi086 ceph-mon[110975]: pgmap v405: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:40.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:40 smithi086 ceph-mon[110975]: from='client.34804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:40.628 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:40 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3652836235' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:33:40.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:40 smithi032 ceph-mon[147414]: pgmap v405: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:40.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:40 smithi032 ceph-mon[147414]: from='client.34804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:33:40.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:40 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3652836235' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:33:42.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:42 smithi086 ceph-mon[110975]: pgmap v406: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:42.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:42 smithi032 ceph-mon[147414]: pgmap v406: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:44.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:44 smithi086 ceph-mon[110975]: pgmap v407: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:44.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:44 smithi032 ceph-mon[147414]: pgmap v407: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:46.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:46 smithi086 ceph-mon[110975]: pgmap v408: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:46.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:46 smithi032 ceph-mon[147414]: pgmap v408: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:48.630 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:48 smithi086 ceph-mon[110975]: pgmap v409: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:48.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:48 smithi032 ceph-mon[147414]: pgmap v409: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:50.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:50 smithi086 ceph-mon[110975]: pgmap v410: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:50.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:50 smithi032 ceph-mon[147414]: pgmap v410: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:52.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:52 smithi086 ceph-mon[110975]: pgmap v411: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:52.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:52 smithi032 ceph-mon[147414]: pgmap v411: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:53.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:33:53.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:33:54.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:54 smithi086 ceph-mon[110975]: pgmap v412: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:54.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:54 smithi032 ceph-mon[147414]: pgmap v412: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:56.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:56 smithi086 ceph-mon[110975]: pgmap v413: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:56.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:56 smithi032 ceph-mon[147414]: pgmap v413: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:58.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:33:58 smithi086 ceph-mon[110975]: pgmap v414: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:33:58.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:33:58 smithi032 ceph-mon[147414]: pgmap v414: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:00.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:00 smithi086 ceph-mon[110975]: pgmap v415: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:00.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:00 smithi032 ceph-mon[147414]: pgmap v415: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:02.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:02 smithi086 ceph-mon[110975]: pgmap v416: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:02.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:02 smithi032 ceph-mon[147414]: pgmap v416: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:04.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:04 smithi086 ceph-mon[110975]: pgmap v417: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:04.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:04 smithi032 ceph-mon[147414]: pgmap v417: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:06.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:06 smithi086 ceph-mon[110975]: pgmap v418: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:06.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:06 smithi032 ceph-mon[147414]: pgmap v418: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:08.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:08 smithi086 ceph-mon[110975]: pgmap v419: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:08.776 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:34:08.777 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:08 smithi032 ceph-mon[147414]: pgmap v419: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:08.777 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:34:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.998+0000 7f909f957700 1 -- 172.21.15.32:0/280377633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 msgr2=0x7f90981020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.998+0000 7f909f957700 1 --2- 172.21.15.32:0/280377633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 0x7f90981020f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9088004940 tx=0x7f9088009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.998+0000 7f909f957700 1 -- 172.21.15.32:0/280377633 shutdown_connections 2024-04-06T14:34:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.998+0000 7f909f957700 1 --2- 172.21.15.32:0/280377633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90981026c0 0x7f90980fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.998+0000 7f909f957700 1 --2- 172.21.15.32:0/280377633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 0x7f90981020f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.998+0000 7f909f957700 1 -- 172.21.15.32:0/280377633 >> 172.21.15.32:0/280377633 conn(0x7f90980f78d0 msgr2=0x7f90980f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.998+0000 7f909f957700 1 -- 172.21.15.32:0/280377633 shutdown_connections 2024-04-06T14:34:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.998+0000 7f909f957700 1 -- 172.21.15.32:0/280377633 wait complete. 2024-04-06T14:34:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.999+0000 7f909f957700 1 Processor -- start 2024-04-06T14:34:09.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.999+0000 7f909f957700 1 -- start start 2024-04-06T14:34:10.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.999+0000 7f909f957700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 0x7f9098102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:10.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.999+0000 7f909f957700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90981026c0 0x7f90981030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:10.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.999+0000 7f909f957700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098103650 con 0x7f9098101d20 2024-04-06T14:34:10.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.999+0000 7f909f957700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098103790 con 0x7f90981026c0 2024-04-06T14:34:10.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.999+0000 7f909d6f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 0x7f9098102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:10.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.999+0000 7f909d6f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 0x7f9098102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60438/0 (socket says 172.21.15.32:60438) 2024-04-06T14:34:10.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:09.999+0000 7f909d6f3700 1 -- 172.21.15.32:0/1651349709 learned_addr learned my addr 172.21.15.32:0/1651349709 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:10.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f909cef2700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90981026c0 0x7f90981030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f909d6f3700 1 -- 172.21.15.32:0/1651349709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90981026c0 msgr2=0x7f90981030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f909d6f3700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90981026c0 0x7f90981030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f909d6f3700 1 -- 172.21.15.32:0/1651349709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9088005040 con 0x7f9098101d20 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f909d6f3700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 0x7f9098102b80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9088009390 tx=0x7f9088009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f908dffb700 1 -- 172.21.15.32:0/1651349709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f908800e040 con 0x7f9098101d20 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f908dffb700 1 -- 172.21.15.32:0/1651349709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9088013420 con 0x7f9098101d20 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90981039e0 con 0x7f9098101d20 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f908dffb700 1 -- 172.21.15.32:0/1651349709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f908801c630 con 0x7f9098101d20 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.000+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909811a1a0 con 0x7f9098101d20 2024-04-06T14:34:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.001+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90980fd470 con 0x7f9098101d20 2024-04-06T14:34:10.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.002+0000 7f908dffb700 1 -- 172.21.15.32:0/1651349709 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f90880139e0 con 0x7f9098101d20 2024-04-06T14:34:10.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.002+0000 7f908dffb700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f908406cc30 0x7f908406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:10.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.002+0000 7f908dffb700 1 -- 172.21.15.32:0/1651349709 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f908808d320 con 0x7f9098101d20 2024-04-06T14:34:10.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.002+0000 7f909cef2700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f908406cc30 0x7f908406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:10.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.003+0000 7f909cef2700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f908406cc30 0x7f908406f0e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f90981044c0 tx=0x7f909400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:10.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.007+0000 7f908dffb700 1 -- 172.21.15.32:0/1651349709 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9088057360 con 0x7f9098101d20 2024-04-06T14:34:10.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.231+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9098104270 con 0x7f908406cc30 2024-04-06T14:34:10.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.233+0000 7f908dffb700 1 -- 172.21.15.32:0/1651349709 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9098104270 con 0x7f908406cc30 2024-04-06T14:34:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f908406cc30 msgr2=0x7f908406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f908406cc30 0x7f908406f0e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f90981044c0 tx=0x7f909400b040 comp rx=0 tx=0).stop 2024-04-06T14:34:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 msgr2=0x7f9098102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 0x7f9098102b80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9088009390 tx=0x7f9088009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 shutdown_connections 2024-04-06T14:34:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f908406cc30 0x7f908406f0e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098101d20 0x7f9098102b80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 --2- 172.21.15.32:0/1651349709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90981026c0 0x7f90981030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 >> 172.21.15.32:0/1651349709 conn(0x7f90980f78d0 msgr2=0x7f90980f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:10.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 shutdown_connections 2024-04-06T14:34:10.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.235+0000 7f909f957700 1 -- 172.21.15.32:0/1651349709 wait complete. 2024-04-06T14:34:10.244 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:34:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.340+0000 7f87b6c9a700 1 -- 172.21.15.32:0/1133780601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 msgr2=0x7f87b0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.340+0000 7f87b6c9a700 1 --2- 172.21.15.32:0/1133780601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 0x7f87b0106f00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f879c004940 tx=0x7f879c009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.340+0000 7f87b6c9a700 1 -- 172.21.15.32:0/1133780601 shutdown_connections 2024-04-06T14:34:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.340+0000 7f87b6c9a700 1 --2- 172.21.15.32:0/1133780601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 0x7f87b0106f00 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.340+0000 7f87b6c9a700 1 --2- 172.21.15.32:0/1133780601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87b00fde90 0x7f87b00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.340+0000 7f87b6c9a700 1 -- 172.21.15.32:0/1133780601 >> 172.21.15.32:0/1133780601 conn(0x7f87b0072bc0 msgr2=0x7f87b0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.340+0000 7f87b6c9a700 1 -- 172.21.15.32:0/1133780601 shutdown_connections 2024-04-06T14:34:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.340+0000 7f87b6c9a700 1 -- 172.21.15.32:0/1133780601 wait complete. 2024-04-06T14:34:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87b6c9a700 1 Processor -- start 2024-04-06T14:34:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87b6c9a700 1 -- start start 2024-04-06T14:34:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87b6c9a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87b00fde90 0x7f87b00fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87b6c9a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 0x7f87b00fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87b6c9a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00fc820 con 0x7f87b00fe830 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87b6c9a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00fc960 con 0x7f87b00fde90 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 0x7f87b00fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 0x7f87b00fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60460/0 (socket says 172.21.15.32:60460) 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87affff700 1 -- 172.21.15.32:0/59799175 learned_addr learned my addr 172.21.15.32:0/59799175 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87b4a36700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87b00fde90 0x7f87b00fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87affff700 1 -- 172.21.15.32:0/59799175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87b00fde90 msgr2=0x7f87b00fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87affff700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87b00fde90 0x7f87b00fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.341+0000 7f87affff700 1 -- 172.21.15.32:0/59799175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c005040 con 0x7f87b00fe830 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.342+0000 7f87affff700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 0x7f87b00fc290 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f879c0093b0 tx=0x7f879c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.342+0000 7f87ad7fa700 1 -- 172.21.15.32:0/59799175 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c00e040 con 0x7f87b00fe830 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.342+0000 7f87ad7fa700 1 -- 172.21.15.32:0/59799175 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f879c004290 con 0x7f87b00fe830 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.342+0000 7f87ad7fa700 1 -- 172.21.15.32:0/59799175 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c01c3f0 con 0x7f87b00fe830 2024-04-06T14:34:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.342+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b00fcbb0 con 0x7f87b00fe830 2024-04-06T14:34:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.342+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b011a190 con 0x7f87b00fe830 2024-04-06T14:34:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.343+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b0065e30 con 0x7f87b00fe830 2024-04-06T14:34:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.343+0000 7f87ad7fa700 1 -- 172.21.15.32:0/59799175 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f879c013790 con 0x7f87b00fe830 2024-04-06T14:34:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.344+0000 7f87ad7fa700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f87a006cc30 0x7f87a006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.344+0000 7f87ad7fa700 1 -- 172.21.15.32:0/59799175 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f879c08d240 con 0x7f87b00fe830 2024-04-06T14:34:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.344+0000 7f87b4a36700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f87a006cc30 0x7f87a006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.344+0000 7f87b4a36700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f87a006cc30 0x7f87a006f0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f87a4007fd0 tx=0x7f87a4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:10.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.348+0000 7f87ad7fa700 1 -- 172.21.15.32:0/59799175 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f879c0572c0 con 0x7f87b00fe830 2024-04-06T14:34:10.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.569+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87b0109e40 con 0x7f87a006cc30 2024-04-06T14:34:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.570+0000 7f87ad7fa700 1 -- 172.21.15.32:0/59799175 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f87b0109e40 con 0x7f87a006cc30 2024-04-06T14:34:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f87a006cc30 msgr2=0x7f87a006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f87a006cc30 0x7f87a006f0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f87a4007fd0 tx=0x7f87a4009040 comp rx=0 tx=0).stop 2024-04-06T14:34:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 msgr2=0x7f87b00fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 0x7f87b00fc290 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f879c0093b0 tx=0x7f879c009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 shutdown_connections 2024-04-06T14:34:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f87a006cc30 0x7f87a006f0e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87b00fde90 0x7f87b00fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 --2- 172.21.15.32:0/59799175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87b00fe830 0x7f87b00fc290 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 >> 172.21.15.32:0/59799175 conn(0x7f87b0072bc0 msgr2=0x7f87b0103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 shutdown_connections 2024-04-06T14:34:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.572+0000 7f87b6c9a700 1 -- 172.21.15.32:0/59799175 wait complete. 2024-04-06T14:34:10.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:10 smithi086 ceph-mon[110975]: pgmap v420: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:10.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.676+0000 7f394cbb8700 1 -- 172.21.15.32:0/3606984700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3948101d70 msgr2=0x7f3948102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.676+0000 7f394cbb8700 1 --2- 172.21.15.32:0/3606984700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3948101d70 0x7f3948102140 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3938004940 tx=0x7f3938009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:10.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.676+0000 7f394cbb8700 1 -- 172.21.15.32:0/3606984700 shutdown_connections 2024-04-06T14:34:10.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.676+0000 7f394cbb8700 1 --2- 172.21.15.32:0/3606984700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39480fc5b0 0x7f39480fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.676+0000 7f394cbb8700 1 --2- 172.21.15.32:0/3606984700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3948101d70 0x7f3948102140 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.676+0000 7f394cbb8700 1 -- 172.21.15.32:0/3606984700 >> 172.21.15.32:0/3606984700 conn(0x7f39480714c0 msgr2=0x7f39480718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.676+0000 7f394cbb8700 1 -- 172.21.15.32:0/3606984700 shutdown_connections 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.676+0000 7f394cbb8700 1 -- 172.21.15.32:0/3606984700 wait complete. 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394cbb8700 1 Processor -- start 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394cbb8700 1 -- start start 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394cbb8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39480fc5b0 0x7f394810ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394cbb8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3948101d70 0x7f394810efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394cbb8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394810f6b0 con 0x7f39480fc5b0 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394cbb8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394810f7f0 con 0x7f3948101d70 2024-04-06T14:34:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39480fc5b0 0x7f394810ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39480fc5b0 0x7f394810ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60480/0 (socket says 172.21.15.32:60480) 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394659c700 1 -- 172.21.15.32:0/2425022932 learned_addr learned my addr 172.21.15.32:0/2425022932 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f3945d9b700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3948101d70 0x7f394810efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394659c700 1 -- 172.21.15.32:0/2425022932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3948101d70 msgr2=0x7f394810efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394659c700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3948101d70 0x7f394810efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394659c700 1 -- 172.21.15.32:0/2425022932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3938005040 con 0x7f39480fc5b0 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.677+0000 7f394659c700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39480fc5b0 0x7f394810ea90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3938009390 tx=0x7f3938009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.678+0000 7f393effd700 1 -- 172.21.15.32:0/2425022932 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f393800e040 con 0x7f39480fc5b0 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.678+0000 7f393effd700 1 -- 172.21.15.32:0/2425022932 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3938013420 con 0x7f39480fc5b0 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.678+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3948117b00 con 0x7f39480fc5b0 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.678+0000 7f393effd700 1 -- 172.21.15.32:0/2425022932 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f393801c630 con 0x7f39480fc5b0 2024-04-06T14:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.678+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3948118090 con 0x7f39480fc5b0 2024-04-06T14:34:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.679+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39480fd5c0 con 0x7f39480fc5b0 2024-04-06T14:34:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.679+0000 7f393effd700 1 -- 172.21.15.32:0/2425022932 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3938013a00 con 0x7f39480fc5b0 2024-04-06T14:34:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.680+0000 7f393effd700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f393406cc30 0x7f393406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.680+0000 7f393effd700 1 -- 172.21.15.32:0/2425022932 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f393808d1f0 con 0x7f39480fc5b0 2024-04-06T14:34:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.680+0000 7f3945d9b700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f393406cc30 0x7f393406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.680+0000 7f3945d9b700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f393406cc30 0x7f393406f0e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3930006fd0 tx=0x7f3930008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:10.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.684+0000 7f393effd700 1 -- 172.21.15.32:0/2425022932 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f39380572f0 con 0x7f39480fc5b0 2024-04-06T14:34:10.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:10 smithi032 ceph-mon[147414]: pgmap v420: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:10.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.899+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f39480fbc50 con 0x7f393406cc30 2024-04-06T14:34:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.909+0000 7f393effd700 1 -- 172.21.15.32:0/2425022932 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f39480fbc50 con 0x7f393406cc30 2024-04-06T14:34:10.910 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (14m) 7m ago 25m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 7m ago 26m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (11m) 7m ago 25m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (14m) 7m ago 25m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (8m) 7m ago 20m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (7m) 7m ago 19m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (7m) 7m ago 20m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (19m) 7m ago 19m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (16m) 7m ago 27m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (15m) 7m ago 24m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 7m ago 27m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (12m) 7m ago 24m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (15m) 7m ago 25m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:34:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (15m) 7m ago 24m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:34:10.912 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (11m) 7m ago 24m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:34:10.912 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 7m ago 23m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:34:10.912 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (10m) 7m ago 22m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:34:10.913 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (9m) 7m ago 22m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:34:10.913 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (9m) 7m ago 22m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:34:10.913 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (8m) 7m ago 21m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:34:10.913 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (14m) 7m ago 25m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:34:10.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f393406cc30 msgr2=0x7f393406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f393406cc30 0x7f393406f0e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3930006fd0 tx=0x7f3930008040 comp rx=0 tx=0).stop 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39480fc5b0 msgr2=0x7f394810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39480fc5b0 0x7f394810ea90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3938009390 tx=0x7f3938009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 shutdown_connections 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f393406cc30 0x7f393406f0e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39480fc5b0 0x7f394810ea90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 --2- 172.21.15.32:0/2425022932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3948101d70 0x7f394810efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 >> 172.21.15.32:0/2425022932 conn(0x7f39480714c0 msgr2=0x7f3948104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 shutdown_connections 2024-04-06T14:34:10.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:10.911+0000 7f394cbb8700 1 -- 172.21.15.32:0/2425022932 wait complete. 2024-04-06T14:34:11.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.013+0000 7febac04e700 1 -- 172.21.15.32:0/2510892558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba4102e00 msgr2=0x7feba41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.013+0000 7febac04e700 1 --2- 172.21.15.32:0/2510892558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba4102e00 0x7feba41031d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feba0004940 tx=0x7feba0009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:11.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.013+0000 7febac04e700 1 -- 172.21.15.32:0/2510892558 shutdown_connections 2024-04-06T14:34:11.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.013+0000 7febac04e700 1 --2- 172.21.15.32:0/2510892558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba41037a0 0x7feba40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.013+0000 7febac04e700 1 --2- 172.21.15.32:0/2510892558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba4102e00 0x7feba41031d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.013+0000 7febac04e700 1 -- 172.21.15.32:0/2510892558 >> 172.21.15.32:0/2510892558 conn(0x7feba40f89b0 msgr2=0x7feba40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:11.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.013+0000 7febac04e700 1 -- 172.21.15.32:0/2510892558 shutdown_connections 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.013+0000 7febac04e700 1 -- 172.21.15.32:0/2510892558 wait complete. 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7febac04e700 1 Processor -- start 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7febac04e700 1 -- start start 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7febac04e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4102e00 0x7feba40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7febac04e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba41037a0 0x7feba4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7febac04e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4070d90 con 0x7feba41037a0 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7febac04e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4070ed0 con 0x7feba4102e00 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7feba95e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba41037a0 0x7feba4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7feba95e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba41037a0 0x7feba4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60484/0 (socket says 172.21.15.32:60484) 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7feba95e9700 1 -- 172.21.15.32:0/1516610182 learned_addr learned my addr 172.21.15.32:0/1516610182 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7feba9dea700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4102e00 0x7feba40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7feba95e9700 1 -- 172.21.15.32:0/1516610182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4102e00 msgr2=0x7feba40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7feba95e9700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4102e00 0x7feba40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.014+0000 7feba95e9700 1 -- 172.21.15.32:0/1516610182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba0005040 con 0x7feba41037a0 2024-04-06T14:34:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.015+0000 7feba95e9700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba41037a0 0x7feba4070800 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feb9800eb40 tx=0x7feb9800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.015+0000 7feb967fc700 1 -- 172.21.15.32:0/1516610182 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb98007cc0 con 0x7feba41037a0 2024-04-06T14:34:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.015+0000 7feb967fc700 1 -- 172.21.15.32:0/1516610182 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb9800f040 con 0x7feba41037a0 2024-04-06T14:34:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.015+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba4071120 con 0x7feba41037a0 2024-04-06T14:34:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.015+0000 7feb967fc700 1 -- 172.21.15.32:0/1516610182 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb98009570 con 0x7feba41037a0 2024-04-06T14:34:11.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.015+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba411a170 con 0x7feba41037a0 2024-04-06T14:34:11.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.016+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba40fe550 con 0x7feba41037a0 2024-04-06T14:34:11.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.016+0000 7feb967fc700 1 -- 172.21.15.32:0/1516610182 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7feb980096d0 con 0x7feba41037a0 2024-04-06T14:34:11.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.017+0000 7feb967fc700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7feb9006cc20 0x7feb9006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:11.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.017+0000 7feb967fc700 1 -- 172.21.15.32:0/1516610182 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feb9808bae0 con 0x7feba41037a0 2024-04-06T14:34:11.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.017+0000 7feba9dea700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7feb9006cc20 0x7feb9006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:11.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.017+0000 7feba9dea700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7feb9006cc20 0x7feb9006f0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feba0004540 tx=0x7feba0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:11.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.021+0000 7feb967fc700 1 -- 172.21.15.32:0/1516610182 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7feb98055b90 con 0x7feba41037a0 2024-04-06T14:34:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.313+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feba405f5f0 con 0x7feba41037a0 2024-04-06T14:34:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.314+0000 7feb967fc700 1 -- 172.21.15.32:0/1516610182 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7feb980591b0 con 0x7feba41037a0 2024-04-06T14:34:11.313 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:34:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:34:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:34:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:34:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:34:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:34:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:34:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:34:11.315 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:34:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7feb9006cc20 msgr2=0x7feb9006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7feb9006cc20 0x7feb9006f0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feba0004540 tx=0x7feba0009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:11.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba41037a0 msgr2=0x7feba4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba41037a0 0x7feba4070800 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feb9800eb40 tx=0x7feb9800b040 comp rx=0 tx=0).stop 2024-04-06T14:34:11.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 shutdown_connections 2024-04-06T14:34:11.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7feb9006cc20 0x7feb9006f0d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4102e00 0x7feba40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 --2- 172.21.15.32:0/1516610182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feba41037a0 0x7feba4070800 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 >> 172.21.15.32:0/1516610182 conn(0x7feba40f89b0 msgr2=0x7feba40f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:11.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 shutdown_connections 2024-04-06T14:34:11.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.316+0000 7febac04e700 1 -- 172.21.15.32:0/1516610182 wait complete. 2024-04-06T14:34:11.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.419+0000 7f2630c0d700 1 -- 172.21.15.32:0/3731817254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 msgr2=0x7f262c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.419+0000 7f2630c0d700 1 --2- 172.21.15.32:0/3731817254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 0x7f262c102140 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f261c004940 tx=0x7f261c009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:11.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.419+0000 7f2630c0d700 1 -- 172.21.15.32:0/3731817254 shutdown_connections 2024-04-06T14:34:11.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.419+0000 7f2630c0d700 1 --2- 172.21.15.32:0/3731817254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262c102710 0x7f262c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.419+0000 7f2630c0d700 1 --2- 172.21.15.32:0/3731817254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 0x7f262c102140 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.419+0000 7f2630c0d700 1 -- 172.21.15.32:0/3731817254 >> 172.21.15.32:0/3731817254 conn(0x7f262c072bc0 msgr2=0x7f262c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:11.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.419+0000 7f2630c0d700 1 -- 172.21.15.32:0/3731817254 shutdown_connections 2024-04-06T14:34:11.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.420+0000 7f2630c0d700 1 -- 172.21.15.32:0/3731817254 wait complete. 2024-04-06T14:34:11.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.420+0000 7f2630c0d700 1 Processor -- start 2024-04-06T14:34:11.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.420+0000 7f2630c0d700 1 -- start start 2024-04-06T14:34:11.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.420+0000 7f2630c0d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 0x7f262c10ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:11.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.420+0000 7f2630c0d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262c102710 0x7f262c10ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:11.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.420+0000 7f2630c0d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c10f670 con 0x7f262c101d70 2024-04-06T14:34:11.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.420+0000 7f2630c0d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c10f7b0 con 0x7f262c102710 2024-04-06T14:34:11.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f262a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 0x7f262c10ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f262a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 0x7f262c10ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60496/0 (socket says 172.21.15.32:60496) 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f262a59c700 1 -- 172.21.15.32:0/2038260164 learned_addr learned my addr 172.21.15.32:0/2038260164 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f2629d9b700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262c102710 0x7f262c10ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f262a59c700 1 -- 172.21.15.32:0/2038260164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262c102710 msgr2=0x7f262c10ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f262a59c700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262c102710 0x7f262c10ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f262a59c700 1 -- 172.21.15.32:0/2038260164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c005040 con 0x7f262c101d70 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f2629d9b700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262c102710 0x7f262c10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f262a59c700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 0x7f262c10ea50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f261c000c00 tx=0x7f261c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f261affd700 1 -- 172.21.15.32:0/2038260164 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f261c00e040 con 0x7f262c101d70 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f261affd700 1 -- 172.21.15.32:0/2038260164 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f261c013420 con 0x7f262c101d70 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262c117ac0 con 0x7f262c101d70 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f261affd700 1 -- 172.21.15.32:0/2038260164 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f261c01c630 con 0x7f262c101d70 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.421+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262c118050 con 0x7f262c101d70 2024-04-06T14:34:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.422+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262c0ff080 con 0x7f262c101d70 2024-04-06T14:34:11.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.423+0000 7f261affd700 1 -- 172.21.15.32:0/2038260164 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f261c013590 con 0x7f262c101d70 2024-04-06T14:34:11.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.423+0000 7f261affd700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f261406cc20 0x7f261406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:11.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.423+0000 7f261affd700 1 -- 172.21.15.32:0/2038260164 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f261c08dda0 con 0x7f262c101d70 2024-04-06T14:34:11.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.423+0000 7f2629d9b700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f261406cc20 0x7f261406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:11.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.424+0000 7f2629d9b700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f261406cc20 0x7f261406f0d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2620006fd0 tx=0x7f2620008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.427+0000 7f261affd700 1 -- 172.21.15.32:0/2038260164 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f261c057e20 con 0x7f262c101d70 2024-04-06T14:34:11.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:11 smithi086 ceph-mon[110975]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:11.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:11 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1516610182' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:34:11.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.674+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f262c0fd540 con 0x7f262c101d70 2024-04-06T14:34:11.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.675+0000 7f261affd700 1 -- 172.21.15.32:0/2038260164 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f261c05b440 con 0x7f262c101d70 2024-04-06T14:34:11.674 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:34:11.674 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:34:11.674 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:34:11.674 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:34:11.674 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:34:11.674 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:34:11.674 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:34:11.675 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:34:11.676 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:34:11.677 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:34:11.677 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:34:11.677 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:34:11.677 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:34:11.677 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:34:11.677 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:34:11.677 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:34:11.677 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:34:11.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f261406cc20 msgr2=0x7f261406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f261406cc20 0x7f261406f0d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2620006fd0 tx=0x7f2620008040 comp rx=0 tx=0).stop 2024-04-06T14:34:11.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 msgr2=0x7f262c10ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 0x7f262c10ea50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f261c000c00 tx=0x7f261c009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:11.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 shutdown_connections 2024-04-06T14:34:11.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f261406cc20 0x7f261406f0d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f262c101d70 0x7f262c10ea50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 --2- 172.21.15.32:0/2038260164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262c102710 0x7f262c10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 >> 172.21.15.32:0/2038260164 conn(0x7f262c072bc0 msgr2=0x7f262c0faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:11.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 shutdown_connections 2024-04-06T14:34:11.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.677+0000 7f2630c0d700 1 -- 172.21.15.32:0/2038260164 wait complete. 2024-04-06T14:34:11.679 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:34:11.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:11 smithi032 ceph-mon[147414]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:11.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:11 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1516610182' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:34:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.782+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2238575755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 msgr2=0x7fa89c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.782+0000 7fa8a37c9700 1 --2- 172.21.15.32:0/2238575755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 0x7fa89c0fb910 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa88c004940 tx=0x7fa88c009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.782+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2238575755 shutdown_connections 2024-04-06T14:34:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.782+0000 7fa8a37c9700 1 --2- 172.21.15.32:0/2238575755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 0x7fa89c0fb910 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.782+0000 7fa8a37c9700 1 --2- 172.21.15.32:0/2238575755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa89c102e70 0x7fa89c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.782+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2238575755 >> 172.21.15.32:0/2238575755 conn(0x7fa89c0714c0 msgr2=0x7fa89c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2238575755 shutdown_connections 2024-04-06T14:34:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2238575755 wait complete. 2024-04-06T14:34:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a37c9700 1 Processor -- start 2024-04-06T14:34:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a37c9700 1 -- start start 2024-04-06T14:34:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a37c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 0x7fa89c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a37c9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa89c102e70 0x7fa89c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a37c9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c114600 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a37c9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c10d9f0 con 0x7fa89c102e70 2024-04-06T14:34:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a1565700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 0x7fa89c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a1565700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 0x7fa89c113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60512/0 (socket says 172.21.15.32:60512) 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.783+0000 7fa8a1565700 1 -- 172.21.15.32:0/2428609116 learned_addr learned my addr 172.21.15.32:0/2428609116 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa8a0d64700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa89c102e70 0x7fa89c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa8a1565700 1 -- 172.21.15.32:0/2428609116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa89c102e70 msgr2=0x7fa89c113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa8a1565700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa89c102e70 0x7fa89c113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa8a1565700 1 -- 172.21.15.32:0/2428609116 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88c005040 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa8a1565700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 0x7fa89c113a70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa89800c830 tx=0x7fa898004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa891ffb700 1 -- 172.21.15.32:0/2428609116 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8980106f0 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa891ffb700 1 -- 172.21.15.32:0/2428609116 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa898010d00 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c10dcd0 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa891ffb700 1 -- 172.21.15.32:0/2428609116 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa898012e90 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.784+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c10e260 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.785+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c100110 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.786+0000 7fa891ffb700 1 -- 172.21.15.32:0/2428609116 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa898010850 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.786+0000 7fa891ffb700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa88806cc20 0x7fa88806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.786+0000 7fa891ffb700 1 -- 172.21.15.32:0/2428609116 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa89808bf60 con 0x7fa89c0fb4a0 2024-04-06T14:34:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.786+0000 7fa8a0d64700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa88806cc20 0x7fa88806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.787+0000 7fa8a0d64700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa88806cc20 0x7fa88806f0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa89c10f3a0 tx=0x7fa88c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:11.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:11.791+0000 7fa891ffb700 1 -- 172.21.15.32:0/2428609116 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa8980bfab0 con 0x7fa89c0fb4a0 2024-04-06T14:34:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.014+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa89c10f130 con 0x7fa88806cc20 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.015+0000 7fa891ffb700 1 -- 172.21.15.32:0/2428609116 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa89c10f130 con 0x7fa88806cc20 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:34:12.014 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:34:12.015 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:34:12.015 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:34:12.015 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:34:12.015 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:34:12.015 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:34:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa88806cc20 msgr2=0x7fa88806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa88806cc20 0x7fa88806f0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa89c10f3a0 tx=0x7fa88c009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 msgr2=0x7fa89c113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 0x7fa89c113a70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa89800c830 tx=0x7fa898004470 comp rx=0 tx=0).stop 2024-04-06T14:34:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 shutdown_connections 2024-04-06T14:34:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa88806cc20 0x7fa88806f0d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa89c0fb4a0 0x7fa89c113a70 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 --2- 172.21.15.32:0/2428609116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa89c102e70 0x7fa89c113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 >> 172.21.15.32:0/2428609116 conn(0x7fa89c0714c0 msgr2=0x7fa89c0fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 shutdown_connections 2024-04-06T14:34:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.017+0000 7fa8a37c9700 1 -- 172.21.15.32:0/2428609116 wait complete. 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 -- 172.21.15.32:0/3739174317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 msgr2=0x7f70240fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 --2- 172.21.15.32:0/3739174317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 0x7f70240fe260 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7018004940 tx=0x7f7018009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 -- 172.21.15.32:0/3739174317 shutdown_connections 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 --2- 172.21.15.32:0/3739174317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70240fe830 0x7f7024106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 --2- 172.21.15.32:0/3739174317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 0x7f70240fe260 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 -- 172.21.15.32:0/3739174317 >> 172.21.15.32:0/3739174317 conn(0x7f7024072bc0 msgr2=0x7f7024072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 -- 172.21.15.32:0/3739174317 shutdown_connections 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 -- 172.21.15.32:0/3739174317 wait complete. 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 Processor -- start 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.122+0000 7f702be90700 1 -- start start 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f702be90700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 0x7f7024113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f702be90700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70240fe830 0x7f7024113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f702be90700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7024114600 con 0x7f70240fde90 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f702be90700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702410d9f0 con 0x7f70240fe830 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f7029c2c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 0x7f7024113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f7029c2c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 0x7f7024113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60528/0 (socket says 172.21.15.32:60528) 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f7029c2c700 1 -- 172.21.15.32:0/2686375462 learned_addr learned my addr 172.21.15.32:0/2686375462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f702942b700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70240fe830 0x7f7024113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f7029c2c700 1 -- 172.21.15.32:0/2686375462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70240fe830 msgr2=0x7f7024113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f7029c2c700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70240fe830 0x7f7024113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f7029c2c700 1 -- 172.21.15.32:0/2686375462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7018005040 con 0x7f70240fde90 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f7029c2c700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 0x7f7024113a70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7018000c00 tx=0x7f7018009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f70167fc700 1 -- 172.21.15.32:0/2686375462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7018021070 con 0x7f70240fde90 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f70167fc700 1 -- 172.21.15.32:0/2686375462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7018013e00 con 0x7f70240fde90 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.123+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702410dc70 con 0x7f70240fde90 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.124+0000 7f70167fc700 1 -- 172.21.15.32:0/2686375462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f701800e070 con 0x7f70240fde90 2024-04-06T14:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.124+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702410e1b0 con 0x7f70240fde90 2024-04-06T14:34:12.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.125+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7024065e30 con 0x7f70240fde90 2024-04-06T14:34:12.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.125+0000 7f70167fc700 1 -- 172.21.15.32:0/2686375462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f701801c690 con 0x7f70240fde90 2024-04-06T14:34:12.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.125+0000 7f70167fc700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701006cc20 0x7f701006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:12.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.126+0000 7f70167fc700 1 -- 172.21.15.32:0/2686375462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f701808d160 con 0x7f70240fde90 2024-04-06T14:34:12.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.126+0000 7f702942b700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701006cc20 0x7f701006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:12.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.126+0000 7f702942b700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701006cc20 0x7f701006f0d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7020004600 tx=0x7f702000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:12.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.130+0000 7f70167fc700 1 -- 172.21.15.32:0/2686375462 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f701805ac30 con 0x7f70240fde90 2024-04-06T14:34:12.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.418+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7024109e10 con 0x7f70240fde90 2024-04-06T14:34:12.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.418+0000 7f70167fc700 1 -- 172.21.15.32:0/2686375462 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f701805a7c0 con 0x7f70240fde90 2024-04-06T14:34:12.417 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:34:12.417 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:34:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:34:12.417 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:34:12.417 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:34:12.418 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:34:12.418 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:34:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701006cc20 msgr2=0x7f701006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701006cc20 0x7f701006f0d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7020004600 tx=0x7f702000b040 comp rx=0 tx=0).stop 2024-04-06T14:34:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 msgr2=0x7f7024113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 0x7f7024113a70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7018000c00 tx=0x7f7018009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 shutdown_connections 2024-04-06T14:34:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701006cc20 0x7f701006f0d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70240fde90 0x7f7024113a70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:12.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 --2- 172.21.15.32:0/2686375462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70240fe830 0x7f7024113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:12.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 >> 172.21.15.32:0/2686375462 conn(0x7f7024072bc0 msgr2=0x7f70241037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:12.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 shutdown_connections 2024-04-06T14:34:12.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:12.420+0000 7f702be90700 1 -- 172.21.15.32:0/2686375462 wait complete. 2024-04-06T14:34:12.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:12 smithi086 ceph-mon[110975]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:12.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:12 smithi086 ceph-mon[110975]: pgmap v421: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:12.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:12 smithi086 ceph-mon[110975]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:12.627 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:12 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2038260164' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:34:12.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:12 smithi032 ceph-mon[147414]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:12.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:12 smithi032 ceph-mon[147414]: pgmap v421: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:12.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:12 smithi032 ceph-mon[147414]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:12.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:12 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2038260164' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:34:13.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:13 smithi086 ceph-mon[110975]: from='client.34832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:13.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:13 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2686375462' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:34:13.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:13 smithi032 ceph-mon[147414]: from='client.34832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:13.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:13 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2686375462' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:34:14.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:14 smithi086 ceph-mon[110975]: pgmap v422: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:14.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:14 smithi032 ceph-mon[147414]: pgmap v422: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:16.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:16 smithi086 ceph-mon[110975]: pgmap v423: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:16.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:16 smithi032 ceph-mon[147414]: pgmap v423: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:18.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:18 smithi086 ceph-mon[110975]: pgmap v424: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:18.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:18 smithi032 ceph-mon[147414]: pgmap v424: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:20.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:20 smithi086 ceph-mon[110975]: pgmap v425: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:20.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:20 smithi032 ceph-mon[147414]: pgmap v425: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:22.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:22 smithi086 ceph-mon[110975]: pgmap v426: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:22.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:22 smithi032 ceph-mon[147414]: pgmap v426: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:23.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:34:23.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:34:24.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:24 smithi086 ceph-mon[110975]: pgmap v427: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:24.729 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:24 smithi032 ceph-mon[147414]: pgmap v427: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:26.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:26 smithi086 ceph-mon[110975]: pgmap v428: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:26.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:26 smithi032 ceph-mon[147414]: pgmap v428: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:28.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:28 smithi086 ceph-mon[110975]: pgmap v429: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:28.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:28 smithi032 ceph-mon[147414]: pgmap v429: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:30.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:30 smithi086 ceph-mon[110975]: pgmap v430: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:30.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:30 smithi032 ceph-mon[147414]: pgmap v430: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:32.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:32 smithi086 ceph-mon[110975]: pgmap v431: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:32.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:32 smithi032 ceph-mon[147414]: pgmap v431: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:34.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:34 smithi086 ceph-mon[110975]: pgmap v432: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:34.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:34 smithi032 ceph-mon[147414]: pgmap v432: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:36.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:36 smithi086 ceph-mon[110975]: pgmap v433: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:36.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:36 smithi032 ceph-mon[147414]: pgmap v433: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:38.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:38 smithi086 ceph-mon[110975]: pgmap v434: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:38.626 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:34:38.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:38 smithi032 ceph-mon[147414]: pgmap v434: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:38.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:34:40.730 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:40 smithi032 ceph-mon[147414]: pgmap v435: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:41.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:40 smithi086 ceph-mon[110975]: pgmap v435: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:42.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.539+0000 7fcbf7404700 1 -- 172.21.15.32:0/3872705928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 msgr2=0x7fcbf00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:42.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.539+0000 7fcbf7404700 1 --2- 172.21.15.32:0/3872705928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 0x7fcbf00fce10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0004940 tx=0x7fcbe0009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:42.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.539+0000 7fcbf7404700 1 -- 172.21.15.32:0/3872705928 shutdown_connections 2024-04-06T14:34:42.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.539+0000 7fcbf7404700 1 --2- 172.21.15.32:0/3872705928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 0x7fcbf00fce10 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:42.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.539+0000 7fcbf7404700 1 --2- 172.21.15.32:0/3872705928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbf0102e00 0x7fcbf01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:42.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.539+0000 7fcbf7404700 1 -- 172.21.15.32:0/3872705928 >> 172.21.15.32:0/3872705928 conn(0x7fcbf00f89b0 msgr2=0x7fcbf00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:42.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.539+0000 7fcbf7404700 1 -- 172.21.15.32:0/3872705928 shutdown_connections 2024-04-06T14:34:42.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.539+0000 7fcbf7404700 1 -- 172.21.15.32:0/3872705928 wait complete. 2024-04-06T14:34:42.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf7404700 1 Processor -- start 2024-04-06T14:34:42.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf7404700 1 -- start start 2024-04-06T14:34:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf7404700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbf0102e00 0x7fcbf0113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf7404700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 0x7fcbf0114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf7404700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0114690 con 0x7fcbf01037a0 2024-04-06T14:34:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf7404700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf010da80 con 0x7fcbf0102e00 2024-04-06T14:34:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf499f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 0x7fcbf0114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf499f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 0x7fcbf0114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36344/0 (socket says 172.21.15.32:36344) 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf499f700 1 -- 172.21.15.32:0/389546257 learned_addr learned my addr 172.21.15.32:0/389546257 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.540+0000 7fcbf51a0700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbf0102e00 0x7fcbf0113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.541+0000 7fcbf499f700 1 -- 172.21.15.32:0/389546257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbf0102e00 msgr2=0x7fcbf0113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.541+0000 7fcbf499f700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbf0102e00 0x7fcbf0113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.541+0000 7fcbf499f700 1 -- 172.21.15.32:0/389546257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe0005040 con 0x7fcbf01037a0 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.541+0000 7fcbf499f700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 0x7fcbf0114040 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00093b0 tx=0x7fcbe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.541+0000 7fcbe5ffb700 1 -- 172.21.15.32:0/389546257 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbe000e040 con 0x7fcbf01037a0 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.541+0000 7fcbe5ffb700 1 -- 172.21.15.32:0/389546257 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbe0013420 con 0x7fcbf01037a0 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.541+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf010dd00 con 0x7fcbf01037a0 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.541+0000 7fcbe5ffb700 1 -- 172.21.15.32:0/389546257 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbe001c630 con 0x7fcbf01037a0 2024-04-06T14:34:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.541+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf010e2c0 con 0x7fcbf01037a0 2024-04-06T14:34:42.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.542+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf00fe550 con 0x7fcbf01037a0 2024-04-06T14:34:42.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.543+0000 7fcbe5ffb700 1 -- 172.21.15.32:0/389546257 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcbe00139e0 con 0x7fcbf01037a0 2024-04-06T14:34:42.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.543+0000 7fcbe5ffb700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcbdc06cc20 0x7fcbdc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:42.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.543+0000 7fcbe5ffb700 1 -- 172.21.15.32:0/389546257 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcbe008d290 con 0x7fcbf01037a0 2024-04-06T14:34:42.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.543+0000 7fcbf51a0700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcbdc06cc20 0x7fcbdc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:42.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.544+0000 7fcbf51a0700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcbdc06cc20 0x7fcbdc06f0d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcbec007fd0 tx=0x7fcbec009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:42.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.548+0000 7fcbe5ffb700 1 -- 172.21.15.32:0/389546257 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcbe005ae10 con 0x7fcbf01037a0 2024-04-06T14:34:42.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:42 smithi032 ceph-mon[147414]: pgmap v436: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.770+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbf0109da0 con 0x7fcbdc06cc20 2024-04-06T14:34:42.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.771+0000 7fcbe5ffb700 1 -- 172.21.15.32:0/389546257 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcbf0109da0 con 0x7fcbdc06cc20 2024-04-06T14:34:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcbdc06cc20 msgr2=0x7fcbdc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcbdc06cc20 0x7fcbdc06f0d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcbec007fd0 tx=0x7fcbec009040 comp rx=0 tx=0).stop 2024-04-06T14:34:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 msgr2=0x7fcbf0114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 0x7fcbf0114040 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00093b0 tx=0x7fcbe0009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 shutdown_connections 2024-04-06T14:34:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcbdc06cc20 0x7fcbdc06f0d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbf0102e00 0x7fcbf0113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 --2- 172.21.15.32:0/389546257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbf01037a0 0x7fcbf0114040 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 >> 172.21.15.32:0/389546257 conn(0x7fcbf00f89b0 msgr2=0x7fcbf00f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 shutdown_connections 2024-04-06T14:34:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.774+0000 7fcbf7404700 1 -- 172.21.15.32:0/389546257 wait complete. 2024-04-06T14:34:42.782 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:34:42.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:42 smithi086 ceph-mon[110975]: pgmap v436: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:42.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.878+0000 7fc6f0860700 1 -- 172.21.15.32:0/1800158545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec1037b0 msgr2=0x7fc6ec0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.878+0000 7fc6f0860700 1 --2- 172.21.15.32:0/1800158545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec1037b0 0x7fc6ec0fce20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0004940 tx=0x7fc6e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.878+0000 7fc6f0860700 1 -- 172.21.15.32:0/1800158545 shutdown_connections 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.878+0000 7fc6f0860700 1 --2- 172.21.15.32:0/1800158545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec1037b0 0x7fc6ec0fce20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.878+0000 7fc6f0860700 1 --2- 172.21.15.32:0/1800158545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6ec102e10 0x7fc6ec1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.878+0000 7fc6f0860700 1 -- 172.21.15.32:0/1800158545 >> 172.21.15.32:0/1800158545 conn(0x7fc6ec0f89a0 msgr2=0x7fc6ec0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6f0860700 1 -- 172.21.15.32:0/1800158545 shutdown_connections 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6f0860700 1 -- 172.21.15.32:0/1800158545 wait complete. 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6f0860700 1 Processor -- start 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6f0860700 1 -- start start 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6f0860700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec102e10 0x7fc6ec115dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6f0860700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6ec1037b0 0x7fc6ec116300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:42.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6f0860700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec1169e0 con 0x7fc6ec102e10 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6f0860700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec10fd20 con 0x7fc6ec1037b0 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6ea59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec102e10 0x7fc6ec115dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6ea59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec102e10 0x7fc6ec115dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36362/0 (socket says 172.21.15.32:36362) 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.879+0000 7fc6ea59c700 1 -- 172.21.15.32:0/3953117969 learned_addr learned my addr 172.21.15.32:0/3953117969 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6e9d9b700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6ec1037b0 0x7fc6ec116300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6ea59c700 1 -- 172.21.15.32:0/3953117969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6ec1037b0 msgr2=0x7fc6ec116300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6ea59c700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6ec1037b0 0x7fc6ec116300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6ea59c700 1 -- 172.21.15.32:0/3953117969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e0005040 con 0x7fc6ec102e10 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6e9d9b700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6ec1037b0 0x7fc6ec116300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6ea59c700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec102e10 0x7fc6ec115dc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00b790 tx=0x7fc6dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6daffd700 1 -- 172.21.15.32:0/3953117969 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6dc00f770 con 0x7fc6ec102e10 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6daffd700 1 -- 172.21.15.32:0/3953117969 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6dc00fd80 con 0x7fc6ec102e10 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6daffd700 1 -- 172.21.15.32:0/3953117969 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6dc00d310 con 0x7fc6ec102e10 2024-04-06T14:34:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ec10ffa0 con 0x7fc6ec102e10 2024-04-06T14:34:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.880+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ec1104e0 con 0x7fc6ec102e10 2024-04-06T14:34:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.881+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ec0fe560 con 0x7fc6ec102e10 2024-04-06T14:34:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.882+0000 7fc6daffd700 1 -- 172.21.15.32:0/3953117969 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc6dc010420 con 0x7fc6ec102e10 2024-04-06T14:34:42.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.882+0000 7fc6daffd700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc6d406cc20 0x7fc6d406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:42.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.882+0000 7fc6daffd700 1 -- 172.21.15.32:0/3953117969 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc6dc08c050 con 0x7fc6ec102e10 2024-04-06T14:34:42.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.882+0000 7fc6e9d9b700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc6d406cc20 0x7fc6d406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:42.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.883+0000 7fc6e9d9b700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc6d406cc20 0x7fc6d406f0d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000b3b0 tx=0x7fc6e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:42.887+0000 7fc6daffd700 1 -- 172.21.15.32:0/3953117969 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc6dc059b20 con 0x7fc6ec102e10 2024-04-06T14:34:43.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.107+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6ec04bf70 con 0x7fc6d406cc20 2024-04-06T14:34:43.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.108+0000 7fc6daffd700 1 -- 172.21.15.32:0/3953117969 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc6e0007d50 con 0x7fc6d406cc20 2024-04-06T14:34:43.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc6d406cc20 msgr2=0x7fc6d406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc6d406cc20 0x7fc6d406f0d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000b3b0 tx=0x7fc6e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:43.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec102e10 msgr2=0x7fc6ec115dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec102e10 0x7fc6ec115dc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00b790 tx=0x7fc6dc009250 comp rx=0 tx=0).stop 2024-04-06T14:34:43.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 shutdown_connections 2024-04-06T14:34:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc6d406cc20 0x7fc6d406f0d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6ec102e10 0x7fc6ec115dc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 --2- 172.21.15.32:0/3953117969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6ec1037b0 0x7fc6ec116300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 >> 172.21.15.32:0/3953117969 conn(0x7fc6ec0f89a0 msgr2=0x7fc6ec0f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 shutdown_connections 2024-04-06T14:34:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.111+0000 7fc6f0860700 1 -- 172.21.15.32:0/3953117969 wait complete. 2024-04-06T14:34:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 -- 172.21.15.32:0/3155249399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f4102e70 msgr2=0x7f20f4103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 --2- 172.21.15.32:0/3155249399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f4102e70 0x7f20f4103240 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f20e8004940 tx=0x7f20e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 -- 172.21.15.32:0/3155249399 shutdown_connections 2024-04-06T14:34:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 --2- 172.21.15.32:0/3155249399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f40fb4a0 0x7f20f40fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 --2- 172.21.15.32:0/3155249399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f4102e70 0x7f20f4103240 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 -- 172.21.15.32:0/3155249399 >> 172.21.15.32:0/3155249399 conn(0x7f20f40714c0 msgr2=0x7f20f40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 -- 172.21.15.32:0/3155249399 shutdown_connections 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 -- 172.21.15.32:0/3155249399 wait complete. 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 Processor -- start 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.214+0000 7f20f8885700 1 -- start start 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f8885700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f40fb4a0 0x7f20f4113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f8885700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f4102e70 0x7f20f4113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f8885700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f4114600 con 0x7f20f40fb4a0 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f8885700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f410d9f0 con 0x7f20f4102e70 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f40fb4a0 0x7f20f4113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f40fb4a0 0x7f20f4113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36378/0 (socket says 172.21.15.32:36378) 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f259c700 1 -- 172.21.15.32:0/2488086238 learned_addr learned my addr 172.21.15.32:0/2488086238 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f1d9b700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f4102e70 0x7f20f4113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f259c700 1 -- 172.21.15.32:0/2488086238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f4102e70 msgr2=0x7f20f4113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f259c700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f4102e70 0x7f20f4113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f259c700 1 -- 172.21.15.32:0/2488086238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e8005040 con 0x7f20f40fb4a0 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f259c700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f40fb4a0 0x7f20f4113a70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f20e800b3b0 tx=0x7f20e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20e2ffd700 1 -- 172.21.15.32:0/2488086238 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20e800e040 con 0x7f20f40fb4a0 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20e2ffd700 1 -- 172.21.15.32:0/2488086238 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20e8013420 con 0x7f20f40fb4a0 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20e2ffd700 1 -- 172.21.15.32:0/2488086238 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20e801c630 con 0x7f20f40fb4a0 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f410dc70 con 0x7f20f40fb4a0 2024-04-06T14:34:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.215+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f410e230 con 0x7f20f40fb4a0 2024-04-06T14:34:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.216+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f4100110 con 0x7f20f40fb4a0 2024-04-06T14:34:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.217+0000 7f20e2ffd700 1 -- 172.21.15.32:0/2488086238 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f20e8013970 con 0x7f20f40fb4a0 2024-04-06T14:34:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.217+0000 7f20e2ffd700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f20dc06cc20 0x7f20dc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.217+0000 7f20e2ffd700 1 -- 172.21.15.32:0/2488086238 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f20e808d180 con 0x7f20f40fb4a0 2024-04-06T14:34:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.217+0000 7f20f1d9b700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f20dc06cc20 0x7f20dc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.218+0000 7f20f1d9b700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f20dc06cc20 0x7f20dc06f0d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f20e4007fd0 tx=0x7f20e4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.222+0000 7f20e2ffd700 1 -- 172.21.15.32:0/2488086238 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f20e805ad10 con 0x7f20f40fb4a0 2024-04-06T14:34:43.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.434+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f20f410f0c0 con 0x7f20dc06cc20 2024-04-06T14:34:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.444+0000 7f20e2ffd700 1 -- 172.21.15.32:0/2488086238 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f20f410f0c0 con 0x7f20dc06cc20 2024-04-06T14:34:43.442 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:34:43.442 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (15m) 8m ago 26m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:34:43.442 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (12m) 8m ago 26m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:34:43.442 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (12m) 8m ago 25m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:34:43.442 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (14m) 8m ago 26m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:34:43.442 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (8m) 8m ago 20m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (8m) 8m ago 20m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (8m) 8m ago 20m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (20m) 8m ago 20m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (16m) 8m ago 27m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (15m) 8m ago 25m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (13m) 8m ago 27m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (13m) 8m ago 25m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (15m) 8m ago 26m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (15m) 8m ago 25m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (12m) 8m ago 24m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 8m ago 24m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (10m) 8m ago 23m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (10m) 8m ago 23m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:34:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (9m) 8m ago 22m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:34:43.444 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (9m) 8m ago 22m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:34:43.444 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (15m) 8m ago 25m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:34:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f20dc06cc20 msgr2=0x7f20dc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f20dc06cc20 0x7f20dc06f0d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f20e4007fd0 tx=0x7f20e4009040 comp rx=0 tx=0).stop 2024-04-06T14:34:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f40fb4a0 msgr2=0x7f20f4113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f40fb4a0 0x7f20f4113a70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f20e800b3b0 tx=0x7f20e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 shutdown_connections 2024-04-06T14:34:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f20dc06cc20 0x7f20dc06f0d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20f40fb4a0 0x7f20f4113a70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 --2- 172.21.15.32:0/2488086238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f4102e70 0x7f20f4113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 >> 172.21.15.32:0/2488086238 conn(0x7f20f40714c0 msgr2=0x7f20f40fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 shutdown_connections 2024-04-06T14:34:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.446+0000 7f20f8885700 1 -- 172.21.15.32:0/2488086238 wait complete. 2024-04-06T14:34:43.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.548+0000 7f59cdc66700 1 -- 172.21.15.32:0/1612548562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c81026b0 msgr2=0x7f59c80fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.548+0000 7f59cdc66700 1 --2- 172.21.15.32:0/1612548562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c81026b0 0x7f59c80fbd20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f59b8004940 tx=0x7f59b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.548+0000 7f59cdc66700 1 -- 172.21.15.32:0/1612548562 shutdown_connections 2024-04-06T14:34:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.548+0000 7f59cdc66700 1 --2- 172.21.15.32:0/1612548562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c81026b0 0x7f59c80fbd20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.548+0000 7f59cdc66700 1 --2- 172.21.15.32:0/1612548562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59c8101d10 0x7f59c81020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.548+0000 7f59cdc66700 1 -- 172.21.15.32:0/1612548562 >> 172.21.15.32:0/1612548562 conn(0x7f59c80f7880 msgr2=0x7f59c80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.548+0000 7f59cdc66700 1 -- 172.21.15.32:0/1612548562 shutdown_connections 2024-04-06T14:34:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.548+0000 7f59cdc66700 1 -- 172.21.15.32:0/1612548562 wait complete. 2024-04-06T14:34:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.548+0000 7f59cdc66700 1 Processor -- start 2024-04-06T14:34:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59cdc66700 1 -- start start 2024-04-06T14:34:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59cdc66700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c8101d10 0x7f59c810eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59cdc66700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59c81026b0 0x7f59c810f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59cdc66700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c810f930 con 0x7f59c8101d10 2024-04-06T14:34:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59cdc66700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c810fa70 con 0x7f59c81026b0 2024-04-06T14:34:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59c77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c8101d10 0x7f59c810eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59c77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c8101d10 0x7f59c810eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36400/0 (socket says 172.21.15.32:36400) 2024-04-06T14:34:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59c77fe700 1 -- 172.21.15.32:0/3174692839 learned_addr learned my addr 172.21.15.32:0/3174692839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59c6ffd700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59c81026b0 0x7f59c810f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59c77fe700 1 -- 172.21.15.32:0/3174692839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59c81026b0 msgr2=0x7f59c810f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59c77fe700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59c81026b0 0x7f59c810f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59c77fe700 1 -- 172.21.15.32:0/3174692839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8005040 con 0x7f59c8101d10 2024-04-06T14:34:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59c77fe700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c8101d10 0x7f59c810eda0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00c830 tx=0x7f59bc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59affff700 1 -- 172.21.15.32:0/3174692839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59bc00a760 con 0x7f59c8101d10 2024-04-06T14:34:43.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.549+0000 7f59affff700 1 -- 172.21.15.32:0/3174692839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59bc00ad70 con 0x7f59c8101d10 2024-04-06T14:34:43.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.550+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c8117e10 con 0x7f59c8101d10 2024-04-06T14:34:43.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.550+0000 7f59affff700 1 -- 172.21.15.32:0/3174692839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59bc0113a0 con 0x7f59c8101d10 2024-04-06T14:34:43.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.550+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c8118340 con 0x7f59c8101d10 2024-04-06T14:34:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.551+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c80fd460 con 0x7f59c8101d10 2024-04-06T14:34:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.551+0000 7f59affff700 1 -- 172.21.15.32:0/3174692839 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f59bc012400 con 0x7f59c8101d10 2024-04-06T14:34:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.551+0000 7f59affff700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f59b006cc20 0x7f59b006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.551+0000 7f59affff700 1 -- 172.21.15.32:0/3174692839 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f59bc08c0b0 con 0x7f59c8101d10 2024-04-06T14:34:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.552+0000 7f59c6ffd700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f59b006cc20 0x7f59b006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.552+0000 7f59c6ffd700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f59b006cc20 0x7f59b006f0d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f59b800b3b0 tx=0x7f59b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:43.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.556+0000 7f59affff700 1 -- 172.21.15.32:0/3174692839 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f59bc0560f0 con 0x7f59c8101d10 2024-04-06T14:34:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.844+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f59c804bed0 con 0x7f59c8101d10 2024-04-06T14:34:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.844+0000 7f59affff700 1 -- 172.21.15.32:0/3174692839 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f59bc059710 con 0x7f59c8101d10 2024-04-06T14:34:43.843 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:34:43.844 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:34:43.844 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:34:43.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:34:43.844 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:34:43.844 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:34:43.845 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:34:43.846 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:34:43.846 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:34:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.846+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f59b006cc20 msgr2=0x7f59b006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.846+0000 7f59cdc66700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f59b006cc20 0x7f59b006f0d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f59b800b3b0 tx=0x7f59b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.846+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c8101d10 msgr2=0x7f59c810eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.846+0000 7f59cdc66700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c8101d10 0x7f59c810eda0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00c830 tx=0x7f59bc004470 comp rx=0 tx=0).stop 2024-04-06T14:34:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.846+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 shutdown_connections 2024-04-06T14:34:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.846+0000 7f59cdc66700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f59b006cc20 0x7f59b006f0d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.846+0000 7f59cdc66700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59c8101d10 0x7f59c810eda0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.846+0000 7f59cdc66700 1 --2- 172.21.15.32:0/3174692839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59c81026b0 0x7f59c810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.846+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 >> 172.21.15.32:0/3174692839 conn(0x7f59c80f7880 msgr2=0x7f59c80f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.847+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 shutdown_connections 2024-04-06T14:34:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.847+0000 7f59cdc66700 1 -- 172.21.15.32:0/3174692839 wait complete. 2024-04-06T14:34:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.950+0000 7fe93c15d700 1 -- 172.21.15.32:0/588121043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 msgr2=0x7fe9341031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.950+0000 7fe93c15d700 1 --2- 172.21.15.32:0/588121043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 0x7fe9341031e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe930004940 tx=0x7fe930009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.950+0000 7fe93c15d700 1 -- 172.21.15.32:0/588121043 shutdown_connections 2024-04-06T14:34:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.950+0000 7fe93c15d700 1 --2- 172.21.15.32:0/588121043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9341037b0 0x7fe9340fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.950+0000 7fe93c15d700 1 --2- 172.21.15.32:0/588121043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 0x7fe9341031e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.950+0000 7fe93c15d700 1 -- 172.21.15.32:0/588121043 >> 172.21.15.32:0/588121043 conn(0x7fe9340f89a0 msgr2=0x7fe9340fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:43.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.951+0000 7fe93c15d700 1 -- 172.21.15.32:0/588121043 shutdown_connections 2024-04-06T14:34:43.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.951+0000 7fe93c15d700 1 -- 172.21.15.32:0/588121043 wait complete. 2024-04-06T14:34:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.951+0000 7fe93c15d700 1 Processor -- start 2024-04-06T14:34:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.951+0000 7fe93c15d700 1 -- start start 2024-04-06T14:34:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe93c15d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 0x7fe934113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe93c15d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9341037b0 0x7fe934113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe93c15d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9341146d0 con 0x7fe934102e10 2024-04-06T14:34:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe93c15d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93410da30 con 0x7fe9341037b0 2024-04-06T14:34:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe939ef9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 0x7fe934113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe939ef9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 0x7fe934113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36422/0 (socket says 172.21.15.32:36422) 2024-04-06T14:34:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe939ef9700 1 -- 172.21.15.32:0/2974793340 learned_addr learned my addr 172.21.15.32:0/2974793340 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe9396f8700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9341037b0 0x7fe934113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe939ef9700 1 -- 172.21.15.32:0/2974793340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9341037b0 msgr2=0x7fe934113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe939ef9700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9341037b0 0x7fe934113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe939ef9700 1 -- 172.21.15.32:0/2974793340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930005040 con 0x7fe934102e10 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe9396f8700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9341037b0 0x7fe934113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe939ef9700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 0x7fe934113ab0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe930009180 tx=0x7fe930009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe92a7fc700 1 -- 172.21.15.32:0/2974793340 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe93000e040 con 0x7fe934102e10 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe92a7fc700 1 -- 172.21.15.32:0/2974793340 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe930004290 con 0x7fe934102e10 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93410dcb0 con 0x7fe934102e10 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe92a7fc700 1 -- 172.21.15.32:0/2974793340 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe93001c3f0 con 0x7fe934102e10 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.952+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93410e270 con 0x7fe934102e10 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.953+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9340fe560 con 0x7fe934102e10 2024-04-06T14:34:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.954+0000 7fe92a7fc700 1 -- 172.21.15.32:0/2974793340 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe930013790 con 0x7fe934102e10 2024-04-06T14:34:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.954+0000 7fe92a7fc700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe92006cc20 0x7fe92006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.954+0000 7fe92a7fc700 1 -- 172.21.15.32:0/2974793340 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe93008d300 con 0x7fe934102e10 2024-04-06T14:34:43.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.954+0000 7fe9396f8700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe92006cc20 0x7fe92006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:43.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.955+0000 7fe9396f8700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe92006cc20 0x7fe92006f0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe924006fd0 tx=0x7fe924008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:43.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:43.959+0000 7fe92a7fc700 1 -- 172.21.15.32:0/2974793340 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe93005add0 con 0x7fe934102e10 2024-04-06T14:34:44.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.209+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe93405f5f0 con 0x7fe934102e10 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.209+0000 7fe92a7fc700 1 -- 172.21.15.32:0/2974793340 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fe930013420 con 0x7fe934102e10 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:34:44.208 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:34:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:34:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:34:44.211 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:34:44.211 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:34:44.211 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:34:44.211 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:34:44.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe92006cc20 msgr2=0x7fe92006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe92006cc20 0x7fe92006f0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe924006fd0 tx=0x7fe924008040 comp rx=0 tx=0).stop 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 msgr2=0x7fe934113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 0x7fe934113ab0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe930009180 tx=0x7fe930009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 shutdown_connections 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe92006cc20 0x7fe92006f0d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe934102e10 0x7fe934113ab0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 --2- 172.21.15.32:0/2974793340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9341037b0 0x7fe934113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 >> 172.21.15.32:0/2974793340 conn(0x7fe9340f89a0 msgr2=0x7fe9340f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 shutdown_connections 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.212+0000 7fe93c15d700 1 -- 172.21.15.32:0/2974793340 wait complete. 2024-04-06T14:34:44.212 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:34:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.317+0000 7f1645504700 1 -- 172.21.15.32:0/880013611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1640101d10 msgr2=0x7f16401020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.317+0000 7f1645504700 1 --2- 172.21.15.32:0/880013611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1640101d10 0x7f16401020e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1630004940 tx=0x7f1630009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.317+0000 7f1645504700 1 -- 172.21.15.32:0/880013611 shutdown_connections 2024-04-06T14:34:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.317+0000 7f1645504700 1 --2- 172.21.15.32:0/880013611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16401026b0 0x7f16400fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.317+0000 7f1645504700 1 --2- 172.21.15.32:0/880013611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1640101d10 0x7f16401020e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.317+0000 7f1645504700 1 -- 172.21.15.32:0/880013611 >> 172.21.15.32:0/880013611 conn(0x7f16400f7880 msgr2=0x7f16400f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.317+0000 7f1645504700 1 -- 172.21.15.32:0/880013611 shutdown_connections 2024-04-06T14:34:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.317+0000 7f1645504700 1 -- 172.21.15.32:0/880013611 wait complete. 2024-04-06T14:34:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.318+0000 7f1645504700 1 Processor -- start 2024-04-06T14:34:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.318+0000 7f1645504700 1 -- start start 2024-04-06T14:34:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.318+0000 7f1645504700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1640101d10 0x7f1640112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.318+0000 7f1645504700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16401026b0 0x7f1640113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.318+0000 7f1645504700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16401138f0 con 0x7f16401026b0 2024-04-06T14:34:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.318+0000 7f1645504700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164010cc30 con 0x7f1640101d10 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.318+0000 7f1637fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16401026b0 0x7f1640113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.318+0000 7f1637fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16401026b0 0x7f1640113210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36440/0 (socket says 172.21.15.32:36440) 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.318+0000 7f1637fff700 1 -- 172.21.15.32:0/3740035635 learned_addr learned my addr 172.21.15.32:0/3740035635 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f163effd700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1640101d10 0x7f1640112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f1637fff700 1 -- 172.21.15.32:0/3740035635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1640101d10 msgr2=0x7f1640112cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f1637fff700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1640101d10 0x7f1640112cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f1637fff700 1 -- 172.21.15.32:0/3740035635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1630005040 con 0x7f16401026b0 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f1637fff700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16401026b0 0x7f1640113210 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f162800daf0 tx=0x7f162800b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f16377fe700 1 -- 172.21.15.32:0/3740035635 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f162800ecf0 con 0x7f16401026b0 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f16377fe700 1 -- 172.21.15.32:0/3740035635 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f162800ee50 con 0x7f16401026b0 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164010ceb0 con 0x7f16401026b0 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f16377fe700 1 -- 172.21.15.32:0/3740035635 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1628016840 con 0x7f16401026b0 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.319+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164010d470 con 0x7f16401026b0 2024-04-06T14:34:44.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.320+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16400fd460 con 0x7f16401026b0 2024-04-06T14:34:44.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.321+0000 7f16377fe700 1 -- 172.21.15.32:0/3740035635 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1628014070 con 0x7f16401026b0 2024-04-06T14:34:44.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.321+0000 7f16377fe700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f162c06cc30 0x7f162c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:44.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.321+0000 7f16377fe700 1 -- 172.21.15.32:0/3740035635 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f162808e160 con 0x7f16401026b0 2024-04-06T14:34:44.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.321+0000 7f163effd700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f162c06cc30 0x7f162c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:44.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.321+0000 7f163effd700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f162c06cc30 0x7f162c06f0e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1630000c00 tx=0x7f1630009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:44.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.326+0000 7f16377fe700 1 -- 172.21.15.32:0/3740035635 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f162805bc30 con 0x7f16401026b0 2024-04-06T14:34:44.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.547+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1640108db0 con 0x7f162c06cc30 2024-04-06T14:34:44.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.548+0000 7f16377fe700 1 -- 172.21.15.32:0/3740035635 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1640108db0 con 0x7f162c06cc30 2024-04-06T14:34:44.546 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:34:44.546 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:34:44.547 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:34:44.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f162c06cc30 msgr2=0x7f162c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f162c06cc30 0x7f162c06f0e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1630000c00 tx=0x7f1630009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:44.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16401026b0 msgr2=0x7f1640113210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16401026b0 0x7f1640113210 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f162800daf0 tx=0x7f162800b4c0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 shutdown_connections 2024-04-06T14:34:44.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f162c06cc30 0x7f162c06f0e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1640101d10 0x7f1640112cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 --2- 172.21.15.32:0/3740035635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16401026b0 0x7f1640113210 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 >> 172.21.15.32:0/3740035635 conn(0x7f16400f7880 msgr2=0x7f16400f8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:44.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 shutdown_connections 2024-04-06T14:34:44.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.550+0000 7f1645504700 1 -- 172.21.15.32:0/3740035635 wait complete. 2024-04-06T14:34:44.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.659+0000 7f06ad063700 1 -- 172.21.15.32:0/723573750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8102710 msgr2=0x7f06a80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.659+0000 7f06ad063700 1 --2- 172.21.15.32:0/723573750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8102710 0x7f06a80fbd80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0690004940 tx=0x7f0690009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:44.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.659+0000 7f06ad063700 1 -- 172.21.15.32:0/723573750 shutdown_connections 2024-04-06T14:34:44.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.659+0000 7f06ad063700 1 --2- 172.21.15.32:0/723573750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8102710 0x7f06a80fbd80 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.659+0000 7f06ad063700 1 --2- 172.21.15.32:0/723573750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06a8101d70 0x7f06a8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.659+0000 7f06ad063700 1 -- 172.21.15.32:0/723573750 >> 172.21.15.32:0/723573750 conn(0x7f06a8072bc0 msgr2=0x7f06a8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:44.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.659+0000 7f06ad063700 1 -- 172.21.15.32:0/723573750 shutdown_connections 2024-04-06T14:34:44.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.659+0000 7f06ad063700 1 -- 172.21.15.32:0/723573750 wait complete. 2024-04-06T14:34:44.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.659+0000 7f06ad063700 1 Processor -- start 2024-04-06T14:34:44.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06ad063700 1 -- start start 2024-04-06T14:34:44.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06ad063700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8101d70 0x7f06a810ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:44.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06ad063700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06a8102710 0x7f06a810efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:44.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06ad063700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a810f6b0 con 0x7f06a8101d70 2024-04-06T14:34:44.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06ad063700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a810f7f0 con 0x7f06a8102710 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06a6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8101d70 0x7f06a810ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06a6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8101d70 0x7f06a810ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46284/0 (socket says 172.21.15.32:46284) 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06a6d9d700 1 -- 172.21.15.32:0/3651088404 learned_addr learned my addr 172.21.15.32:0/3651088404 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06a659c700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06a8102710 0x7f06a810efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06a6d9d700 1 -- 172.21.15.32:0/3651088404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06a8102710 msgr2=0x7f06a810efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06a6d9d700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06a8102710 0x7f06a810efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06a6d9d700 1 -- 172.21.15.32:0/3651088404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0690005040 con 0x7f06a8101d70 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06a6d9d700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8101d70 0x7f06a810ea90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f069800b790 tx=0x7f0698009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f069f7fe700 1 -- 172.21.15.32:0/3651088404 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f069800f770 con 0x7f06a8101d70 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f069f7fe700 1 -- 172.21.15.32:0/3651088404 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f069800fd80 con 0x7f06a8101d70 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.660+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a8117b60 con 0x7f06a8101d70 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.661+0000 7f069f7fe700 1 -- 172.21.15.32:0/3651088404 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f069800d310 con 0x7f06a8101d70 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.661+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a81180f0 con 0x7f06a8101d70 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.661+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a80ff080 con 0x7f06a8101d70 2024-04-06T14:34:44.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.662+0000 7f069f7fe700 1 -- 172.21.15.32:0/3651088404 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0698010420 con 0x7f06a8101d70 2024-04-06T14:34:44.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.662+0000 7f069f7fe700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f069406cc20 0x7f069406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:34:44.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.663+0000 7f069f7fe700 1 -- 172.21.15.32:0/3651088404 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f069808bf00 con 0x7f06a8101d70 2024-04-06T14:34:44.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.663+0000 7f06a659c700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f069406cc20 0x7f069406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:34:44.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.663+0000 7f06a659c700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f069406cc20 0x7f069406f0d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f069000b3b0 tx=0x7f0690009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:34:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.667+0000 7f069f7fe700 1 -- 172.21.15.32:0/3651088404 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0698056000 con 0x7f06a8101d70 2024-04-06T14:34:44.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:44 smithi032 ceph-mon[147414]: pgmap v437: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:44.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:44 smithi032 ceph-mon[147414]: from='client.34840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:44.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:44 smithi032 ceph-mon[147414]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:44.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:44 smithi032 ceph-mon[147414]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:44.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:44 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3174692839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:34:44.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:44 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2974793340' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:34:44.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:44 smithi086 ceph-mon[110975]: pgmap v437: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:44.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:44 smithi086 ceph-mon[110975]: from='client.34840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:44.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:44 smithi086 ceph-mon[110975]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:44.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:44 smithi086 ceph-mon[110975]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:44.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:44 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3174692839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:34:44.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:44 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2974793340' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:34:44.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.950+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f06a80fd4c0 con 0x7f06a8101d70 2024-04-06T14:34:44.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.950+0000 7f069f7fe700 1 -- 172.21.15.32:0/3651088404 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f0698059620 con 0x7f06a8101d70 2024-04-06T14:34:44.948 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:34:44.949 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:34:44.949 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:34:44.949 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:34:44.949 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:34:44.949 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:34:44.949 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:34:44.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f069406cc20 msgr2=0x7f069406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f069406cc20 0x7f069406f0d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f069000b3b0 tx=0x7f0690009e30 comp rx=0 tx=0).stop 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8101d70 msgr2=0x7f06a810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8101d70 0x7f06a810ea90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f069800b790 tx=0x7f0698009250 comp rx=0 tx=0).stop 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 shutdown_connections 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f069406cc20 0x7f069406f0d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a8101d70 0x7f06a810ea90 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 --2- 172.21.15.32:0/3651088404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06a8102710 0x7f06a810efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 >> 172.21.15.32:0/3651088404 conn(0x7f06a8072bc0 msgr2=0x7f06a80fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 shutdown_connections 2024-04-06T14:34:44.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:34:44.952+0000 7f06ad063700 1 -- 172.21.15.32:0/3651088404 wait complete. 2024-04-06T14:34:45.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:45 smithi032 ceph-mon[147414]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:45.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:45 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3651088404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:34:45.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:45 smithi086 ceph-mon[110975]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:34:45.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:45 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3651088404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:34:46.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:46 smithi032 ceph-mon[147414]: pgmap v438: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:46.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:46 smithi086 ceph-mon[110975]: pgmap v438: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:48.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:48 smithi032 ceph-mon[147414]: pgmap v439: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:48.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:48 smithi086 ceph-mon[110975]: pgmap v439: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:50.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:50 smithi032 ceph-mon[147414]: pgmap v440: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:50.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:50 smithi086 ceph-mon[110975]: pgmap v440: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:52.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:52 smithi032 ceph-mon[147414]: pgmap v441: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:52.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:52 smithi086 ceph-mon[110975]: pgmap v441: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:53.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:34:53.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:34:54.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:54 smithi032 ceph-mon[147414]: pgmap v442: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:54.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:54 smithi086 ceph-mon[110975]: pgmap v442: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:56.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:56 smithi032 ceph-mon[147414]: pgmap v443: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:56.876 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:56 smithi086 ceph-mon[110975]: pgmap v443: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:58.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:34:58 smithi032 ceph-mon[147414]: pgmap v444: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:34:58.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:34:58 smithi086 ceph-mon[110975]: pgmap v444: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:00.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:00 smithi032 ceph-mon[147414]: pgmap v445: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:00.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:00 smithi086 ceph-mon[110975]: pgmap v445: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:02.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:02 smithi032 ceph-mon[147414]: pgmap v446: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:02.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:02 smithi086 ceph-mon[110975]: pgmap v446: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:04.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:04 smithi032 ceph-mon[147414]: pgmap v447: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:04.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:04 smithi086 ceph-mon[110975]: pgmap v447: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:06.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:06 smithi032 ceph-mon[147414]: pgmap v448: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:06.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:06 smithi086 ceph-mon[110975]: pgmap v448: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:08.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:08 smithi032 ceph-mon[147414]: pgmap v449: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:08.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:35:08.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:08 smithi086 ceph-mon[110975]: pgmap v449: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:08.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:35:10.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:10 smithi032 ceph-mon[147414]: pgmap v450: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:10.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:10 smithi086 ceph-mon[110975]: pgmap v450: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:12.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:12 smithi032 ceph-mon[147414]: pgmap v451: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:12.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:12 smithi086 ceph-mon[110975]: pgmap v451: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:14.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:14 smithi032 ceph-mon[147414]: pgmap v452: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:14.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:14 smithi086 ceph-mon[110975]: pgmap v452: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.070+0000 7f19a75e1700 1 -- 172.21.15.32:0/2578639340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 msgr2=0x7f19a0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.070+0000 7f19a75e1700 1 --2- 172.21.15.32:0/2578639340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 0x7f19a0102140 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1994004940 tx=0x7f1994009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.070+0000 7f19a75e1700 1 -- 172.21.15.32:0/2578639340 shutdown_connections 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.070+0000 7f19a75e1700 1 --2- 172.21.15.32:0/2578639340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a0102710 0x7f19a00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.070+0000 7f19a75e1700 1 --2- 172.21.15.32:0/2578639340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 0x7f19a0102140 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.070+0000 7f19a75e1700 1 -- 172.21.15.32:0/2578639340 >> 172.21.15.32:0/2578639340 conn(0x7f19a0072bc0 msgr2=0x7f19a0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.070+0000 7f19a75e1700 1 -- 172.21.15.32:0/2578639340 shutdown_connections 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.070+0000 7f19a75e1700 1 -- 172.21.15.32:0/2578639340 wait complete. 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a75e1700 1 Processor -- start 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a75e1700 1 -- start start 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a75e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 0x7f19a0113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a75e1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a0102710 0x7f19a0114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a75e1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a0114650 con 0x7f19a0101d70 2024-04-06T14:35:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a75e1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a010da40 con 0x7f19a0102710 2024-04-06T14:35:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a537d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 0x7f19a0113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a537d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 0x7f19a0113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59590/0 (socket says 172.21.15.32:59590) 2024-04-06T14:35:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a537d700 1 -- 172.21.15.32:0/77847535 learned_addr learned my addr 172.21.15.32:0/77847535 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a4b7c700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a0102710 0x7f19a0114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a537d700 1 -- 172.21.15.32:0/77847535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a0102710 msgr2=0x7f19a0114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a537d700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a0102710 0x7f19a0114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.071+0000 7f19a537d700 1 -- 172.21.15.32:0/77847535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1994005040 con 0x7f19a0101d70 2024-04-06T14:35:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.072+0000 7f19a537d700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 0x7f19a0113ac0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f199400b3b0 tx=0x7f1994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.072+0000 7f1991ffb700 1 -- 172.21.15.32:0/77847535 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f199400e040 con 0x7f19a0101d70 2024-04-06T14:35:15.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.072+0000 7f1991ffb700 1 -- 172.21.15.32:0/77847535 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1994013420 con 0x7f19a0101d70 2024-04-06T14:35:15.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.072+0000 7f1991ffb700 1 -- 172.21.15.32:0/77847535 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f199401c630 con 0x7f19a0101d70 2024-04-06T14:35:15.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.072+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a010dcc0 con 0x7f19a0101d70 2024-04-06T14:35:15.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.072+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a010e280 con 0x7f19a0101d70 2024-04-06T14:35:15.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.073+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a00fd4c0 con 0x7f19a0101d70 2024-04-06T14:35:15.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.074+0000 7f1991ffb700 1 -- 172.21.15.32:0/77847535 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1994013970 con 0x7f19a0101d70 2024-04-06T14:35:15.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.074+0000 7f1991ffb700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f198c06cc20 0x7f198c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:15.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.074+0000 7f1991ffb700 1 -- 172.21.15.32:0/77847535 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f199408d2b0 con 0x7f19a0101d70 2024-04-06T14:35:15.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.074+0000 7f19a4b7c700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f198c06cc20 0x7f198c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:15.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.075+0000 7f19a4b7c700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f198c06cc20 0x7f198c06f0d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f199c007fd0 tx=0x7f199c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:15.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.079+0000 7f1991ffb700 1 -- 172.21.15.32:0/77847535 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f199405ad80 con 0x7f19a0101d70 2024-04-06T14:35:15.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.305+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19a0108d30 con 0x7f198c06cc20 2024-04-06T14:35:15.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.306+0000 7f1991ffb700 1 -- 172.21.15.32:0/77847535 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f19a0108d30 con 0x7f198c06cc20 2024-04-06T14:35:15.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.308+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f198c06cc20 msgr2=0x7f198c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.308+0000 7f19a75e1700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f198c06cc20 0x7f198c06f0d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f199c007fd0 tx=0x7f199c009040 comp rx=0 tx=0).stop 2024-04-06T14:35:15.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.308+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 msgr2=0x7f19a0113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.308+0000 7f19a75e1700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 0x7f19a0113ac0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f199400b3b0 tx=0x7f1994009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:15.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.309+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 shutdown_connections 2024-04-06T14:35:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.309+0000 7f19a75e1700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f198c06cc20 0x7f198c06f0d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.309+0000 7f19a75e1700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19a0101d70 0x7f19a0113ac0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.309+0000 7f19a75e1700 1 --2- 172.21.15.32:0/77847535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a0102710 0x7f19a0114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.309+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 >> 172.21.15.32:0/77847535 conn(0x7f19a0072bc0 msgr2=0x7f19a00659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.309+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 shutdown_connections 2024-04-06T14:35:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.309+0000 7f19a75e1700 1 -- 172.21.15.32:0/77847535 wait complete. 2024-04-06T14:35:15.317 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:35:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.413+0000 7fc2f5083700 1 -- 172.21.15.32:0/2782724790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 msgr2=0x7fc2f0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.413+0000 7fc2f5083700 1 --2- 172.21.15.32:0/2782724790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 0x7fc2f0106f00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8004940 tx=0x7fc2d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.413+0000 7fc2f5083700 1 -- 172.21.15.32:0/2782724790 shutdown_connections 2024-04-06T14:35:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.413+0000 7fc2f5083700 1 --2- 172.21.15.32:0/2782724790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 0x7fc2f0106f00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.413+0000 7fc2f5083700 1 --2- 172.21.15.32:0/2782724790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2f00fde90 0x7fc2f00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.413+0000 7fc2f5083700 1 -- 172.21.15.32:0/2782724790 >> 172.21.15.32:0/2782724790 conn(0x7fc2f0072bc0 msgr2=0x7fc2f0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.413+0000 7fc2f5083700 1 -- 172.21.15.32:0/2782724790 shutdown_connections 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.413+0000 7fc2f5083700 1 -- 172.21.15.32:0/2782724790 wait complete. 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2f5083700 1 Processor -- start 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2f5083700 1 -- start start 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2f5083700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2f00fde90 0x7fc2f00fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2f5083700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 0x7fc2f00fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2f5083700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f00fc820 con 0x7fc2f00fe830 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2f5083700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f00fc960 con 0x7fc2f00fde90 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2ee59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 0x7fc2f00fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2ee59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 0x7fc2f00fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59614/0 (socket says 172.21.15.32:59614) 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2ee59c700 1 -- 172.21.15.32:0/464710390 learned_addr learned my addr 172.21.15.32:0/464710390 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.414+0000 7fc2eed9d700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2f00fde90 0x7fc2f00fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.415+0000 7fc2ee59c700 1 -- 172.21.15.32:0/464710390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2f00fde90 msgr2=0x7fc2f00fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.415+0000 7fc2ee59c700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2f00fde90 0x7fc2f00fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.415+0000 7fc2ee59c700 1 -- 172.21.15.32:0/464710390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8005040 con 0x7fc2f00fe830 2024-04-06T14:35:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.415+0000 7fc2ee59c700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 0x7fc2f00fc290 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800b3b0 tx=0x7fc2d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.415+0000 7fc2e77fe700 1 -- 172.21.15.32:0/464710390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2d800e040 con 0x7fc2f00fe830 2024-04-06T14:35:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.415+0000 7fc2e77fe700 1 -- 172.21.15.32:0/464710390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2d8013420 con 0x7fc2f00fe830 2024-04-06T14:35:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.415+0000 7fc2e77fe700 1 -- 172.21.15.32:0/464710390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2d801c5f0 con 0x7fc2f00fe830 2024-04-06T14:35:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.415+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f00fcbb0 con 0x7fc2f00fe830 2024-04-06T14:35:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.415+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f011a140 con 0x7fc2f00fe830 2024-04-06T14:35:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.416+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0065e30 con 0x7fc2f00fe830 2024-04-06T14:35:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.417+0000 7fc2e77fe700 1 -- 172.21.15.32:0/464710390 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc2d8013590 con 0x7fc2f00fe830 2024-04-06T14:35:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.417+0000 7fc2e77fe700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc2dc06cc20 0x7fc2dc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.417+0000 7fc2e77fe700 1 -- 172.21.15.32:0/464710390 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc2d808d340 con 0x7fc2f00fe830 2024-04-06T14:35:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.417+0000 7fc2eed9d700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc2dc06cc20 0x7fc2dc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.417+0000 7fc2eed9d700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc2dc06cc20 0x7fc2dc06f0d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0007fd0 tx=0x7fc2e0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:15.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.422+0000 7fc2e77fe700 1 -- 172.21.15.32:0/464710390 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc2d8057380 con 0x7fc2f00fe830 2024-04-06T14:35:15.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.642+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2f0109e40 con 0x7fc2dc06cc20 2024-04-06T14:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.643+0000 7fc2e77fe700 1 -- 172.21.15.32:0/464710390 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc2f0109e40 con 0x7fc2dc06cc20 2024-04-06T14:35:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc2dc06cc20 msgr2=0x7fc2dc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc2dc06cc20 0x7fc2dc06f0d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0007fd0 tx=0x7fc2e0009040 comp rx=0 tx=0).stop 2024-04-06T14:35:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 msgr2=0x7fc2f00fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 0x7fc2f00fc290 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800b3b0 tx=0x7fc2d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 shutdown_connections 2024-04-06T14:35:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc2dc06cc20 0x7fc2dc06f0d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2f00fde90 0x7fc2f00fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 --2- 172.21.15.32:0/464710390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2f00fe830 0x7fc2f00fc290 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 >> 172.21.15.32:0/464710390 conn(0x7fc2f0072bc0 msgr2=0x7fc2f0103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 shutdown_connections 2024-04-06T14:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.645+0000 7fc2f5083700 1 -- 172.21.15.32:0/464710390 wait complete. 2024-04-06T14:35:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 -- 172.21.15.32:0/4070105444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 msgr2=0x7f3428103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 --2- 172.21.15.32:0/4070105444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 0x7f3428103240 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3424004940 tx=0x7f3424009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 -- 172.21.15.32:0/4070105444 shutdown_connections 2024-04-06T14:35:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 --2- 172.21.15.32:0/4070105444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34280fb4a0 0x7f34280fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 --2- 172.21.15.32:0/4070105444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 0x7f3428103240 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 -- 172.21.15.32:0/4070105444 >> 172.21.15.32:0/4070105444 conn(0x7f34280714c0 msgr2=0x7f34280718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 -- 172.21.15.32:0/4070105444 shutdown_connections 2024-04-06T14:35:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 -- 172.21.15.32:0/4070105444 wait complete. 2024-04-06T14:35:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 Processor -- start 2024-04-06T14:35:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.749+0000 7f342fe48700 1 -- start start 2024-04-06T14:35:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342fe48700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34280fb4a0 0x7f34281139c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342fe48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 0x7f3428113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342fe48700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3428114550 con 0x7f3428102e70 2024-04-06T14:35:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342fe48700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342810d990 con 0x7f34280fb4a0 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342d3e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 0x7f3428113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342d3e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 0x7f3428113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59620/0 (socket says 172.21.15.32:59620) 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342d3e3700 1 -- 172.21.15.32:0/2822040276 learned_addr learned my addr 172.21.15.32:0/2822040276 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342dbe4700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34280fb4a0 0x7f34281139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342d3e3700 1 -- 172.21.15.32:0/2822040276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34280fb4a0 msgr2=0x7f34281139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342d3e3700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34280fb4a0 0x7f34281139c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342d3e3700 1 -- 172.21.15.32:0/2822040276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3424005040 con 0x7f3428102e70 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.750+0000 7f342dbe4700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34280fb4a0 0x7f34281139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.751+0000 7f342d3e3700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 0x7f3428113f00 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f341800eb40 tx=0x7f341800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.751+0000 7f341e7fc700 1 -- 172.21.15.32:0/2822040276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3418007cc0 con 0x7f3428102e70 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.751+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f342810dc10 con 0x7f3428102e70 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.751+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342810e1d0 con 0x7f3428102e70 2024-04-06T14:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.751+0000 7f341e7fc700 1 -- 172.21.15.32:0/2822040276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f341800f040 con 0x7f3428102e70 2024-04-06T14:35:15.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.751+0000 7f341e7fc700 1 -- 172.21.15.32:0/2822040276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34180094b0 con 0x7f3428102e70 2024-04-06T14:35:15.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.752+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340c0052b0 con 0x7f3428102e70 2024-04-06T14:35:15.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.753+0000 7f341e7fc700 1 -- 172.21.15.32:0/2822040276 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3418009610 con 0x7f3428102e70 2024-04-06T14:35:15.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.753+0000 7f341e7fc700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3414071040 0x7f34140734f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:15.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.753+0000 7f341e7fc700 1 -- 172.21.15.32:0/2822040276 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f341808bce0 con 0x7f3428102e70 2024-04-06T14:35:15.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.753+0000 7f342dbe4700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3414071040 0x7f34140734f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:15.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.753+0000 7f342dbe4700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3414071040 0x7f34140734f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f342400b0d0 tx=0x7f3424009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:15.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.758+0000 7f341e7fc700 1 -- 172.21.15.32:0/2822040276 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3418090070 con 0x7f3428102e70 2024-04-06T14:35:15.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.970+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f340c000bc0 con 0x7f3414071040 2024-04-06T14:35:15.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.980+0000 7f341e7fc700 1 -- 172.21.15.32:0/2822040276 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f340c000bc0 con 0x7f3414071040 2024-04-06T14:35:15.978 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:35:15.978 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (15m) 8m ago 26m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:35:15.978 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (12m) 8m ago 27m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:35:15.978 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (12m) 8m ago 26m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:35:15.978 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (15m) 8m ago 26m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:35:15.978 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (9m) 8m ago 21m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (9m) 8m ago 21m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (8m) 8m ago 21m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (21m) 8m ago 21m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (17m) 8m ago 28m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (16m) 8m ago 26m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (14m) 8m ago 28m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (13m) 8m ago 25m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (16m) 8m ago 27m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (16m) 8m ago 26m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:35:15.979 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (12m) 8m ago 25m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:35:15.980 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 8m ago 24m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:35:15.980 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 8m ago 24m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:35:15.980 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (10m) 8m ago 23m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:35:15.980 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (10m) 8m ago 23m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:35:15.980 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (9m) 8m ago 22m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:35:15.980 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (15m) 8m ago 26m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:35:15.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3414071040 msgr2=0x7f34140734f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3414071040 0x7f34140734f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f342400b0d0 tx=0x7f3424009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 msgr2=0x7f3428113f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 0x7f3428113f00 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f341800eb40 tx=0x7f341800b040 comp rx=0 tx=0).stop 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 shutdown_connections 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3414071040 0x7f34140734f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34280fb4a0 0x7f34281139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 --2- 172.21.15.32:0/2822040276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3428102e70 0x7f3428113f00 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 >> 172.21.15.32:0/2822040276 conn(0x7f34280714c0 msgr2=0x7f34280fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 shutdown_connections 2024-04-06T14:35:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:15.982+0000 7f342fe48700 1 -- 172.21.15.32:0/2822040276 wait complete. 2024-04-06T14:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.085+0000 7f60a8818700 1 -- 172.21.15.32:0/2064005937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fc590 msgr2=0x7f60a41092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.085+0000 7f60a8818700 1 --2- 172.21.15.32:0/2064005937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fc590 0x7f60a41092f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6094004940 tx=0x7f6094009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.085+0000 7f60a8818700 1 -- 172.21.15.32:0/2064005937 shutdown_connections 2024-04-06T14:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.085+0000 7f60a8818700 1 --2- 172.21.15.32:0/2064005937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fc590 0x7f60a41092f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.085+0000 7f60a8818700 1 --2- 172.21.15.32:0/2064005937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a40fbc80 0x7f60a40fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.085+0000 7f60a8818700 1 -- 172.21.15.32:0/2064005937 >> 172.21.15.32:0/2064005937 conn(0x7f60a4072bc0 msgr2=0x7f60a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.085+0000 7f60a8818700 1 -- 172.21.15.32:0/2064005937 shutdown_connections 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.085+0000 7f60a8818700 1 -- 172.21.15.32:0/2064005937 wait complete. 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.085+0000 7f60a8818700 1 Processor -- start 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a8818700 1 -- start start 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a8818700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fbc80 0x7f60a4115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a8818700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a40fc590 0x7f60a41161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a8818700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a4116800 con 0x7f60a40fbc80 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a8818700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a410fbd0 con 0x7f60a40fc590 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fbc80 0x7f60a4115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fbc80 0x7f60a4115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59636/0 (socket says 172.21.15.32:59636) 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a259c700 1 -- 172.21.15.32:0/989714960 learned_addr learned my addr 172.21.15.32:0/989714960 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a1d9b700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a40fc590 0x7f60a41161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a259c700 1 -- 172.21.15.32:0/989714960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a40fc590 msgr2=0x7f60a41161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a259c700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a40fc590 0x7f60a41161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a259c700 1 -- 172.21.15.32:0/989714960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6094005040 con 0x7f60a40fbc80 2024-04-06T14:35:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f60a259c700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fbc80 0x7f60a4115c70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f608c002a50 tx=0x7f608c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.086+0000 7f609affd700 1 -- 172.21.15.32:0/989714960 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f608c007cc0 con 0x7f60a40fbc80 2024-04-06T14:35:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.087+0000 7f609affd700 1 -- 172.21.15.32:0/989714960 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f608c00f040 con 0x7f60a40fbc80 2024-04-06T14:35:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.087+0000 7f609affd700 1 -- 172.21.15.32:0/989714960 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f608c009540 con 0x7f60a40fbc80 2024-04-06T14:35:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.087+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a410feb0 con 0x7f60a40fbc80 2024-04-06T14:35:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.087+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a4110470 con 0x7f60a40fbc80 2024-04-06T14:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.087+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a4065e30 con 0x7f60a40fbc80 2024-04-06T14:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.088+0000 7f609affd700 1 -- 172.21.15.32:0/989714960 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f608c0096a0 con 0x7f60a40fbc80 2024-04-06T14:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.089+0000 7f609affd700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f609006cc30 0x7f609006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.089+0000 7f60a1d9b700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f609006cc30 0x7f609006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.089+0000 7f609affd700 1 -- 172.21.15.32:0/989714960 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f608c08bd70 con 0x7f60a40fbc80 2024-04-06T14:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.089+0000 7f60a1d9b700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f609006cc30 0x7f609006f0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f609400b020 tx=0x7f6094009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:16.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.093+0000 7f609affd700 1 -- 172.21.15.32:0/989714960 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f608c059840 con 0x7f60a40fbc80 2024-04-06T14:35:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.379+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f60a404bed0 con 0x7f60a40fbc80 2024-04-06T14:35:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.380+0000 7f609affd700 1 -- 172.21.15.32:0/989714960 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f608c0593d0 con 0x7f60a40fbc80 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:35:16.378 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:35:16.379 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:35:16.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:35:16.379 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:35:16.379 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:35:16.379 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:35:16.379 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:35:16.379 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f609006cc30 msgr2=0x7f609006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f609006cc30 0x7f609006f0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f609400b020 tx=0x7f6094009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fbc80 msgr2=0x7f60a4115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fbc80 0x7f60a4115c70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f608c002a50 tx=0x7f608c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 shutdown_connections 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f609006cc30 0x7f609006f0e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a40fbc80 0x7f60a4115c70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 --2- 172.21.15.32:0/989714960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a40fc590 0x7f60a41161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 >> 172.21.15.32:0/989714960 conn(0x7f60a4072bc0 msgr2=0x7f60a4105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 shutdown_connections 2024-04-06T14:35:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.382+0000 7f60a8818700 1 -- 172.21.15.32:0/989714960 wait complete. 2024-04-06T14:35:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.486+0000 7f669e5b0700 1 -- 172.21.15.32:0/3400843687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 msgr2=0x7f6698102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.486+0000 7f669e5b0700 1 --2- 172.21.15.32:0/3400843687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 0x7f6698102140 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6688004940 tx=0x7f6688009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.486+0000 7f669e5b0700 1 -- 172.21.15.32:0/3400843687 shutdown_connections 2024-04-06T14:35:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.486+0000 7f669e5b0700 1 --2- 172.21.15.32:0/3400843687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6698102710 0x7f66980fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.486+0000 7f669e5b0700 1 --2- 172.21.15.32:0/3400843687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 0x7f6698102140 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.486+0000 7f669e5b0700 1 -- 172.21.15.32:0/3400843687 >> 172.21.15.32:0/3400843687 conn(0x7f6698072bc0 msgr2=0x7f6698072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.486+0000 7f669e5b0700 1 -- 172.21.15.32:0/3400843687 shutdown_connections 2024-04-06T14:35:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.486+0000 7f669e5b0700 1 -- 172.21.15.32:0/3400843687 wait complete. 2024-04-06T14:35:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.486+0000 7f669e5b0700 1 Processor -- start 2024-04-06T14:35:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f669e5b0700 1 -- start start 2024-04-06T14:35:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f669e5b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 0x7f6698115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f669e5b0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6698102710 0x7f66981161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f669e5b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6698116830 con 0x7f6698101d70 2024-04-06T14:35:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f669e5b0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669810fc00 con 0x7f6698102710 2024-04-06T14:35:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f6697fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 0x7f6698115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f6697fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 0x7f6698115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59664/0 (socket says 172.21.15.32:59664) 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f6697fff700 1 -- 172.21.15.32:0/1148612961 learned_addr learned my addr 172.21.15.32:0/1148612961 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f66977fe700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6698102710 0x7f66981161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f6697fff700 1 -- 172.21.15.32:0/1148612961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6698102710 msgr2=0x7f66981161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f6697fff700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6698102710 0x7f66981161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f6697fff700 1 -- 172.21.15.32:0/1148612961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6688005040 con 0x7f6698101d70 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f66977fe700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6698102710 0x7f66981161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.487+0000 7f6697fff700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 0x7f6698115ca0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6688009390 tx=0x7f6688009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.488+0000 7f6694ff9700 1 -- 172.21.15.32:0/1148612961 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f668800e040 con 0x7f6698101d70 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.488+0000 7f6694ff9700 1 -- 172.21.15.32:0/1148612961 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6688013420 con 0x7f6698101d70 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.488+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669810fe80 con 0x7f6698101d70 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.488+0000 7f6694ff9700 1 -- 172.21.15.32:0/1148612961 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f668801c630 con 0x7f6698101d70 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.488+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6698110440 con 0x7f6698101d70 2024-04-06T14:35:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.489+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66980fd4c0 con 0x7f6698101d70 2024-04-06T14:35:16.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.489+0000 7f6694ff9700 1 -- 172.21.15.32:0/1148612961 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f66880139e0 con 0x7f6698101d70 2024-04-06T14:35:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.489+0000 7f6694ff9700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f668006cc30 0x7f668006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.490+0000 7f6694ff9700 1 -- 172.21.15.32:0/1148612961 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f668808d1a0 con 0x7f6698101d70 2024-04-06T14:35:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.490+0000 7f66977fe700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f668006cc30 0x7f668006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.490+0000 7f66977fe700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f668006cc30 0x7f668006f0e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f668c009fd0 tx=0x7f668c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.494+0000 7f6694ff9700 1 -- 172.21.15.32:0/1148612961 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f668805ade0 con 0x7f6698101d70 2024-04-06T14:35:16.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:16 smithi032 ceph-mon[147414]: pgmap v453: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:16.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:16 smithi032 ceph-mon[147414]: from='client.34868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:16.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:16 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/989714960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:35:16.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.741+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f669804bed0 con 0x7f6698101d70 2024-04-06T14:35:16.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.741+0000 7f6694ff9700 1 -- 172.21.15.32:0/1148612961 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f668805a970 con 0x7f6698101d70 2024-04-06T14:35:16.743 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:35:16.743 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:35:16.743 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:35:16.743 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:35:16.743 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:35:16.743 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:35:16.743 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:35:16.743 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:35:16.743 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:35:16.744 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:35:16.745 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:35:16.746 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:35:16.746 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:35:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.743+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f668006cc30 msgr2=0x7f668006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.743+0000 7f669e5b0700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f668006cc30 0x7f668006f0e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f668c009fd0 tx=0x7f668c00b040 comp rx=0 tx=0).stop 2024-04-06T14:35:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.743+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 msgr2=0x7f6698115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.743+0000 7f669e5b0700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 0x7f6698115ca0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6688009390 tx=0x7f6688009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.744+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 shutdown_connections 2024-04-06T14:35:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.744+0000 7f669e5b0700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f668006cc30 0x7f668006f0e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.744+0000 7f669e5b0700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6698101d70 0x7f6698115ca0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.744+0000 7f669e5b0700 1 --2- 172.21.15.32:0/1148612961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6698102710 0x7f66981161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.744+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 >> 172.21.15.32:0/1148612961 conn(0x7f6698072bc0 msgr2=0x7f66980659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.744+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 shutdown_connections 2024-04-06T14:35:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.744+0000 7f669e5b0700 1 -- 172.21.15.32:0/1148612961 wait complete. 2024-04-06T14:35:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:35:16.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.848+0000 7ff19f903700 1 -- 172.21.15.32:0/3604468048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198102710 msgr2=0x7ff1980fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.848+0000 7ff19f903700 1 --2- 172.21.15.32:0/3604468048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198102710 0x7ff1980fbd80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff194004940 tx=0x7ff194009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.848+0000 7ff19f903700 1 -- 172.21.15.32:0/3604468048 shutdown_connections 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.848+0000 7ff19f903700 1 --2- 172.21.15.32:0/3604468048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198102710 0x7ff1980fbd80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.848+0000 7ff19f903700 1 --2- 172.21.15.32:0/3604468048 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff198101d70 0x7ff198102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.848+0000 7ff19f903700 1 -- 172.21.15.32:0/3604468048 >> 172.21.15.32:0/3604468048 conn(0x7ff198072bc0 msgr2=0x7ff198072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.848+0000 7ff19f903700 1 -- 172.21.15.32:0/3604468048 shutdown_connections 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.848+0000 7ff19f903700 1 -- 172.21.15.32:0/3604468048 wait complete. 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19f903700 1 Processor -- start 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19f903700 1 -- start start 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19f903700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198101d70 0x7ff198112a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19f903700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff198102710 0x7ff198112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19f903700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1981135b0 con 0x7ff198101d70 2024-04-06T14:35:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19f903700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19810c9a0 con 0x7ff198102710 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19d69f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198101d70 0x7ff198112a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19d69f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198101d70 0x7ff198112a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59686/0 (socket says 172.21.15.32:59686) 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19d69f700 1 -- 172.21.15.32:0/3042259909 learned_addr learned my addr 172.21.15.32:0/3042259909 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19ce9e700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff198102710 0x7ff198112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19d69f700 1 -- 172.21.15.32:0/3042259909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff198102710 msgr2=0x7ff198112f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.849+0000 7ff19d69f700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff198102710 0x7ff198112f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.850+0000 7ff19d69f700 1 -- 172.21.15.32:0/3042259909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff194005040 con 0x7ff198101d70 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.850+0000 7ff19d69f700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198101d70 0x7ff198112a20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff18800fce0 tx=0x7ff1880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.850+0000 7ff18dffb700 1 -- 172.21.15.32:0/3042259909 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff18800fe90 con 0x7ff198101d70 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.850+0000 7ff18dffb700 1 -- 172.21.15.32:0/3042259909 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1880090d0 con 0x7ff198101d70 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.850+0000 7ff18dffb700 1 -- 172.21.15.32:0/3042259909 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1880126a0 con 0x7ff198101d70 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.850+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19810cc20 con 0x7ff198101d70 2024-04-06T14:35:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.850+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19810d1e0 con 0x7ff198101d70 2024-04-06T14:35:16.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.851+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1980fd4c0 con 0x7ff198101d70 2024-04-06T14:35:16.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.851+0000 7ff18dffb700 1 -- 172.21.15.32:0/3042259909 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff188009240 con 0x7ff198101d70 2024-04-06T14:35:16.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.852+0000 7ff18dffb700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff18406cc30 0x7ff18406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:16.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.852+0000 7ff18dffb700 1 -- 172.21.15.32:0/3042259909 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff18808e9e0 con 0x7ff198101d70 2024-04-06T14:35:16.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.852+0000 7ff19ce9e700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff18406cc30 0x7ff18406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:16.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.852+0000 7ff19ce9e700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff18406cc30 0x7ff18406f0e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff19400b3b0 tx=0x7ff194009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:16.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:16.856+0000 7ff18dffb700 1 -- 172.21.15.32:0/3042259909 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff18805c3b0 con 0x7ff198101d70 2024-04-06T14:35:16.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:16 smithi086 ceph-mon[110975]: pgmap v453: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:16.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:16 smithi086 ceph-mon[110975]: from='client.34868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:16.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:16 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/989714960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:35:17.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.077+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff198108da0 con 0x7ff18406cc30 2024-04-06T14:35:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.078+0000 7ff18dffb700 1 -- 172.21.15.32:0/3042259909 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff198108da0 con 0x7ff18406cc30 2024-04-06T14:35:17.076 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:35:17.076 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:35:17.076 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:35:17.076 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:35:17.077 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:35:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.080+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff18406cc30 msgr2=0x7ff18406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.080+0000 7ff19f903700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff18406cc30 0x7ff18406f0e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff19400b3b0 tx=0x7ff194009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.080+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198101d70 msgr2=0x7ff198112a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.080+0000 7ff19f903700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198101d70 0x7ff198112a20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff18800fce0 tx=0x7ff1880046b0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.080+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 shutdown_connections 2024-04-06T14:35:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.080+0000 7ff19f903700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff18406cc30 0x7ff18406f0e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.080+0000 7ff19f903700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198101d70 0x7ff198112a20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.080+0000 7ff19f903700 1 --2- 172.21.15.32:0/3042259909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff198102710 0x7ff198112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.081+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 >> 172.21.15.32:0/3042259909 conn(0x7ff198072bc0 msgr2=0x7ff1980fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.081+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 shutdown_connections 2024-04-06T14:35:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.081+0000 7ff19f903700 1 -- 172.21.15.32:0/3042259909 wait complete. 2024-04-06T14:35:17.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.184+0000 7fd7d778f700 1 -- 172.21.15.32:0/1377921677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d01026d0 msgr2=0x7fd7d00fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:17.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.184+0000 7fd7d778f700 1 --2- 172.21.15.32:0/1377921677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d01026d0 0x7fd7d00fbd40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc004940 tx=0x7fd7cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:17.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.184+0000 7fd7d778f700 1 -- 172.21.15.32:0/1377921677 shutdown_connections 2024-04-06T14:35:17.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.184+0000 7fd7d778f700 1 --2- 172.21.15.32:0/1377921677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d01026d0 0x7fd7d00fbd40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.184+0000 7fd7d778f700 1 --2- 172.21.15.32:0/1377921677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7d0101d30 0x7fd7d0102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.184+0000 7fd7d778f700 1 -- 172.21.15.32:0/1377921677 >> 172.21.15.32:0/1377921677 conn(0x7fd7d00f78e0 msgr2=0x7fd7d00f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:17.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.184+0000 7fd7d778f700 1 -- 172.21.15.32:0/1377921677 shutdown_connections 2024-04-06T14:35:17.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.184+0000 7fd7d778f700 1 -- 172.21.15.32:0/1377921677 wait complete. 2024-04-06T14:35:17.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.184+0000 7fd7d778f700 1 Processor -- start 2024-04-06T14:35:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d778f700 1 -- start start 2024-04-06T14:35:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d778f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d0101d30 0x7fd7d0113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d778f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7d01026d0 0x7fd7d0113e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d778f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d0114520 con 0x7fd7d0101d30 2024-04-06T14:35:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d778f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d010da90 con 0x7fd7d01026d0 2024-04-06T14:35:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d552b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d0101d30 0x7fd7d0113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d552b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d0101d30 0x7fd7d0113900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59706/0 (socket says 172.21.15.32:59706) 2024-04-06T14:35:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d552b700 1 -- 172.21.15.32:0/3000994512 learned_addr learned my addr 172.21.15.32:0/3000994512 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d4d2a700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7d01026d0 0x7fd7d0113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d552b700 1 -- 172.21.15.32:0/3000994512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7d01026d0 msgr2=0x7fd7d0113e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d552b700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7d01026d0 0x7fd7d0113e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d552b700 1 -- 172.21.15.32:0/3000994512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc005040 con 0x7fd7d0101d30 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7d552b700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d0101d30 0x7fd7d0113900 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd7c000b790 tx=0x7fd7c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7c5ffb700 1 -- 172.21.15.32:0/3000994512 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7c000f770 con 0x7fd7d0101d30 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.185+0000 7fd7c5ffb700 1 -- 172.21.15.32:0/3000994512 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7c000fd80 con 0x7fd7d0101d30 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.186+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d010dd70 con 0x7fd7d0101d30 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.186+0000 7fd7c5ffb700 1 -- 172.21.15.32:0/3000994512 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7c000d310 con 0x7fd7d0101d30 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.186+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d010e300 con 0x7fd7d0101d30 2024-04-06T14:35:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.187+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d00fd480 con 0x7fd7d0101d30 2024-04-06T14:35:17.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.187+0000 7fd7c5ffb700 1 -- 172.21.15.32:0/3000994512 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd7c0010420 con 0x7fd7d0101d30 2024-04-06T14:35:17.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.187+0000 7fd7c5ffb700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7bc06cc20 0x7fd7bc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:17.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.187+0000 7fd7c5ffb700 1 -- 172.21.15.32:0/3000994512 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd7c008bf30 con 0x7fd7d0101d30 2024-04-06T14:35:17.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.188+0000 7fd7d4d2a700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7bc06cc20 0x7fd7bc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:17.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.188+0000 7fd7d4d2a700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7bc06cc20 0x7fd7bc06f0d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd7d010ed70 tx=0x7fd7cc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:17.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.192+0000 7fd7c5ffb700 1 -- 172.21.15.32:0/3000994512 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd7c00bfab0 con 0x7fd7d0101d30 2024-04-06T14:35:17.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.480+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd7d0108d40 con 0x7fd7d0101d30 2024-04-06T14:35:17.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.481+0000 7fd7c5ffb700 1 -- 172.21.15.32:0/3000994512 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fd7c0059ac0 con 0x7fd7d0101d30 2024-04-06T14:35:17.479 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:35:17.479 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:35:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:35:17.479 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:35:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:35:17.479 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:35:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7bc06cc20 msgr2=0x7fd7bc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7bc06cc20 0x7fd7bc06f0d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd7d010ed70 tx=0x7fd7cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d0101d30 msgr2=0x7fd7d0113900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d0101d30 0x7fd7d0113900 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd7c000b790 tx=0x7fd7c0009250 comp rx=0 tx=0).stop 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 shutdown_connections 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7bc06cc20 0x7fd7bc06f0d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7d0101d30 0x7fd7d0113900 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 --2- 172.21.15.32:0/3000994512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7d01026d0 0x7fd7d0113e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 >> 172.21.15.32:0/3000994512 conn(0x7fd7d00f78e0 msgr2=0x7fd7d00f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 shutdown_connections 2024-04-06T14:35:17.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:17.483+0000 7fd7d778f700 1 -- 172.21.15.32:0/3000994512 wait complete. 2024-04-06T14:35:17.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:17 smithi032 ceph-mon[147414]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:17.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:17 smithi032 ceph-mon[147414]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:17.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:17 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1148612961' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:35:17.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:17 smithi086 ceph-mon[110975]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:17.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:17 smithi086 ceph-mon[110975]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:17.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:17 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1148612961' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:35:18.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:18 smithi032 ceph-mon[147414]: pgmap v454: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:18.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:18 smithi032 ceph-mon[147414]: from='client.34888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:18.728 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:18 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3000994512' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:35:18.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:18 smithi086 ceph-mon[110975]: pgmap v454: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:18.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:18 smithi086 ceph-mon[110975]: from='client.34888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:18.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:18 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3000994512' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:35:20.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:20 smithi032 ceph-mon[147414]: pgmap v455: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:20.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:20 smithi086 ceph-mon[110975]: pgmap v455: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:22.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:22 smithi032 ceph-mon[147414]: pgmap v456: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:22.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:22 smithi086 ceph-mon[110975]: pgmap v456: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:23.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:35:23.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:35:24.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:24 smithi032 ceph-mon[147414]: pgmap v457: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:24.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:24 smithi086 ceph-mon[110975]: pgmap v457: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:26.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:26 smithi032 ceph-mon[147414]: pgmap v458: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:26.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:26 smithi086 ceph-mon[110975]: pgmap v458: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:28.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:28 smithi032 ceph-mon[147414]: pgmap v459: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:28.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:28 smithi086 ceph-mon[110975]: pgmap v459: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:30.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:30 smithi032 ceph-mon[147414]: pgmap v460: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:30.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:30 smithi086 ceph-mon[110975]: pgmap v460: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:32.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:32 smithi032 ceph-mon[147414]: pgmap v461: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:32.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:32 smithi086 ceph-mon[110975]: pgmap v461: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:34.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:34 smithi032 ceph-mon[147414]: pgmap v462: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:34.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:34 smithi086 ceph-mon[110975]: pgmap v462: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:36.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:36 smithi032 ceph-mon[147414]: pgmap v463: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:36.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:36 smithi086 ceph-mon[110975]: pgmap v463: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:38.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:38 smithi032 ceph-mon[147414]: pgmap v464: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:38.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:35:38.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:38 smithi086 ceph-mon[110975]: pgmap v464: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:38.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:35:40.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:40 smithi032 ceph-mon[147414]: pgmap v465: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:40.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:40 smithi086 ceph-mon[110975]: pgmap v465: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:42.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:42 smithi086 ceph-mon[110975]: pgmap v466: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:42.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:42 smithi032 ceph-mon[147414]: pgmap v466: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:44.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:44 smithi086 ceph-mon[110975]: pgmap v467: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:44.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:44 smithi032 ceph-mon[147414]: pgmap v467: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:46.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:46 smithi086 ceph-mon[110975]: pgmap v468: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:46.978 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:46 smithi032 ceph-mon[147414]: pgmap v468: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:47.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.600+0000 7f03edfb1700 1 -- 172.21.15.32:0/444905525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 msgr2=0x7f03e80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:47.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.600+0000 7f03edfb1700 1 --2- 172.21.15.32:0/444905525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 0x7f03e80fbd80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f03d0004940 tx=0x7f03d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:47.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.600+0000 7f03edfb1700 1 -- 172.21.15.32:0/444905525 shutdown_connections 2024-04-06T14:35:47.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.600+0000 7f03edfb1700 1 --2- 172.21.15.32:0/444905525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 0x7f03e80fbd80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.600+0000 7f03edfb1700 1 --2- 172.21.15.32:0/444905525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e8101d70 0x7f03e8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.600+0000 7f03edfb1700 1 -- 172.21.15.32:0/444905525 >> 172.21.15.32:0/444905525 conn(0x7f03e8072bc0 msgr2=0x7f03e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:47.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.600+0000 7f03edfb1700 1 -- 172.21.15.32:0/444905525 shutdown_connections 2024-04-06T14:35:47.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.600+0000 7f03edfb1700 1 -- 172.21.15.32:0/444905525 wait complete. 2024-04-06T14:35:47.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.601+0000 7f03edfb1700 1 Processor -- start 2024-04-06T14:35:47.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.601+0000 7f03edfb1700 1 -- start start 2024-04-06T14:35:47.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.601+0000 7f03edfb1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e8101d70 0x7f03e8115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:47.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.601+0000 7f03edfb1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 0x7f03e81161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:47.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.601+0000 7f03edfb1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e8116830 con 0x7f03e8102710 2024-04-06T14:35:47.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.601+0000 7f03edfb1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e810fc00 con 0x7f03e8101d70 2024-04-06T14:35:47.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.601+0000 7f03e6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 0x7f03e81161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:47.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.601+0000 7f03e6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 0x7f03e81161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36264/0 (socket says 172.21.15.32:36264) 2024-04-06T14:35:47.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.601+0000 7f03e6ffd700 1 -- 172.21.15.32:0/4199798019 learned_addr learned my addr 172.21.15.32:0/4199798019 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:47.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03e77fe700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e8101d70 0x7f03e8115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03e6ffd700 1 -- 172.21.15.32:0/4199798019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e8101d70 msgr2=0x7f03e8115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03e6ffd700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e8101d70 0x7f03e8115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03e6ffd700 1 -- 172.21.15.32:0/4199798019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0005040 con 0x7f03e8102710 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03e77fe700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e8101d70 0x7f03e8115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03e6ffd700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 0x7f03e81161e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f03d00093b0 tx=0x7f03d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03dffff700 1 -- 172.21.15.32:0/4199798019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03d000e040 con 0x7f03e8102710 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e810fe80 con 0x7f03e8102710 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03dffff700 1 -- 172.21.15.32:0/4199798019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03d0013420 con 0x7f03e8102710 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03dffff700 1 -- 172.21.15.32:0/4199798019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03d001c630 con 0x7f03e8102710 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.602+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e8110440 con 0x7f03e8102710 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.603+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e80fd4c0 con 0x7f03e8102710 2024-04-06T14:35:47.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.604+0000 7f03dffff700 1 -- 172.21.15.32:0/4199798019 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f03d00139e0 con 0x7f03e8102710 2024-04-06T14:35:47.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.604+0000 7f03dffff700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03d406cc20 0x7f03d406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:47.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.604+0000 7f03dffff700 1 -- 172.21.15.32:0/4199798019 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f03d008d290 con 0x7f03e8102710 2024-04-06T14:35:47.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.604+0000 7f03e77fe700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03d406cc20 0x7f03d406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:47.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.605+0000 7f03e77fe700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03d406cc20 0x7f03d406f0d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009fd0 tx=0x7f03d8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:47.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.609+0000 7f03dffff700 1 -- 172.21.15.32:0/4199798019 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f03d005ae10 con 0x7f03e8102710 2024-04-06T14:35:47.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.835+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f03e8108d10 con 0x7f03d406cc20 2024-04-06T14:35:47.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.837+0000 7f03dffff700 1 -- 172.21.15.32:0/4199798019 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f03e8108d10 con 0x7f03d406cc20 2024-04-06T14:35:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03d406cc20 msgr2=0x7f03d406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03d406cc20 0x7f03d406f0d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009fd0 tx=0x7f03d8009ee0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 msgr2=0x7f03e81161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 0x7f03e81161e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f03d00093b0 tx=0x7f03d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 shutdown_connections 2024-04-06T14:35:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03d406cc20 0x7f03d406f0d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e8101d70 0x7f03e8115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 --2- 172.21.15.32:0/4199798019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e8102710 0x7f03e81161e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 >> 172.21.15.32:0/4199798019 conn(0x7f03e8072bc0 msgr2=0x7f03e8065a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 shutdown_connections 2024-04-06T14:35:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.839+0000 7f03edfb1700 1 -- 172.21.15.32:0/4199798019 wait complete. 2024-04-06T14:35:47.846 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 -- 172.21.15.32:0/1789746543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 msgr2=0x7f99a8103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 --2- 172.21.15.32:0/1789746543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 0x7f99a8103240 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9990004940 tx=0x7f9990009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 -- 172.21.15.32:0/1789746543 shutdown_connections 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 --2- 172.21.15.32:0/1789746543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a80fb4a0 0x7f99a80fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 --2- 172.21.15.32:0/1789746543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 0x7f99a8103240 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 -- 172.21.15.32:0/1789746543 >> 172.21.15.32:0/1789746543 conn(0x7f99a80714c0 msgr2=0x7f99a80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 -- 172.21.15.32:0/1789746543 shutdown_connections 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 -- 172.21.15.32:0/1789746543 wait complete. 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 Processor -- start 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.943+0000 7f99ae364700 1 -- start start 2024-04-06T14:35:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99ae364700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a80fb4a0 0x7f99a8113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99ae364700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 0x7f99a8113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99ae364700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a8114600 con 0x7f99a8102e70 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99ae364700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a810d9f0 con 0x7f99a80fb4a0 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 0x7f99a8113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 0x7f99a8113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36278/0 (socket says 172.21.15.32:36278) 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a77fe700 1 -- 172.21.15.32:0/2108237614 learned_addr learned my addr 172.21.15.32:0/2108237614 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a7fff700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a80fb4a0 0x7f99a8113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a77fe700 1 -- 172.21.15.32:0/2108237614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a80fb4a0 msgr2=0x7f99a8113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a77fe700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a80fb4a0 0x7f99a8113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a77fe700 1 -- 172.21.15.32:0/2108237614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990005040 con 0x7f99a8102e70 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a77fe700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 0x7f99a8113fb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f999800c830 tx=0x7f9998004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a4ff9700 1 -- 172.21.15.32:0/2108237614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99980106f0 con 0x7f99a8102e70 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a810dcd0 con 0x7f99a8102e70 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a4ff9700 1 -- 172.21.15.32:0/2108237614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9998010d00 con 0x7f99a8102e70 2024-04-06T14:35:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.944+0000 7f99a4ff9700 1 -- 172.21.15.32:0/2108237614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9998012e90 con 0x7f99a8102e70 2024-04-06T14:35:47.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.945+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a810e290 con 0x7f99a8102e70 2024-04-06T14:35:47.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.946+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a8100110 con 0x7f99a8102e70 2024-04-06T14:35:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.946+0000 7f99a4ff9700 1 -- 172.21.15.32:0/2108237614 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9998010850 con 0x7f99a8102e70 2024-04-06T14:35:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.946+0000 7f99a4ff9700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f999406cc20 0x7f999406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.947+0000 7f99a4ff9700 1 -- 172.21.15.32:0/2108237614 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f999808c010 con 0x7f99a8102e70 2024-04-06T14:35:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.947+0000 7f99a7fff700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f999406cc20 0x7f999406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.947+0000 7f99a7fff700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f999406cc20 0x7f999406f0d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9990004f90 tx=0x7f9990009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:47.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:47.951+0000 7f99a4ff9700 1 -- 172.21.15.32:0/2108237614 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9998059ae0 con 0x7f99a8102e70 2024-04-06T14:35:48.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.178+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99a8113740 con 0x7f999406cc20 2024-04-06T14:35:48.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.179+0000 7f99a4ff9700 1 -- 172.21.15.32:0/2108237614 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f99a8113740 con 0x7f999406cc20 2024-04-06T14:35:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f999406cc20 msgr2=0x7f999406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f999406cc20 0x7f999406f0d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9990004f90 tx=0x7f9990009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 msgr2=0x7f99a8113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 0x7f99a8113fb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f999800c830 tx=0x7f9998004470 comp rx=0 tx=0).stop 2024-04-06T14:35:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 shutdown_connections 2024-04-06T14:35:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f999406cc20 0x7f999406f0d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a80fb4a0 0x7f99a8113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 --2- 172.21.15.32:0/2108237614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99a8102e70 0x7f99a8113fb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 >> 172.21.15.32:0/2108237614 conn(0x7f99a80714c0 msgr2=0x7f99a80fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:48.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 shutdown_connections 2024-04-06T14:35:48.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.181+0000 7f99ae364700 1 -- 172.21.15.32:0/2108237614 wait complete. 2024-04-06T14:35:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 -- 172.21.15.32:0/3587652037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c1026b0 msgr2=0x7faf0c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 --2- 172.21.15.32:0/3587652037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c1026b0 0x7faf0c0fbd20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7faf00004940 tx=0x7faf00009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 -- 172.21.15.32:0/3587652037 shutdown_connections 2024-04-06T14:35:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 --2- 172.21.15.32:0/3587652037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c1026b0 0x7faf0c0fbd20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 --2- 172.21.15.32:0/3587652037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf0c101d10 0x7faf0c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 -- 172.21.15.32:0/3587652037 >> 172.21.15.32:0/3587652037 conn(0x7faf0c0f7880 msgr2=0x7faf0c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 -- 172.21.15.32:0/3587652037 shutdown_connections 2024-04-06T14:35:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 -- 172.21.15.32:0/3587652037 wait complete. 2024-04-06T14:35:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 Processor -- start 2024-04-06T14:35:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.284+0000 7faf10cd5700 1 -- start start 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf10cd5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c101d10 0x7faf0c10ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf10cd5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf0c1026b0 0x7faf0c10f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf10cd5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c10f9a0 con 0x7faf0c101d10 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf10cd5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c10fae0 con 0x7faf0c1026b0 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf0a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c101d10 0x7faf0c10ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf0a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c101d10 0x7faf0c10ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36300/0 (socket says 172.21.15.32:36300) 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf0a59c700 1 -- 172.21.15.32:0/2831738481 learned_addr learned my addr 172.21.15.32:0/2831738481 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf09d9b700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf0c1026b0 0x7faf0c10f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf0a59c700 1 -- 172.21.15.32:0/2831738481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf0c1026b0 msgr2=0x7faf0c10f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf0a59c700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf0c1026b0 0x7faf0c10f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf0a59c700 1 -- 172.21.15.32:0/2831738481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf00005040 con 0x7faf0c101d10 2024-04-06T14:35:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf09d9b700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf0c1026b0 0x7faf0c10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:35:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf0a59c700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c101d10 0x7faf0c10ed80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faefc00b790 tx=0x7faefc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faefaffd700 1 -- 172.21.15.32:0/2831738481 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faefc00f770 con 0x7faf0c101d10 2024-04-06T14:35:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faefaffd700 1 -- 172.21.15.32:0/2831738481 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faefc00fd80 con 0x7faf0c101d10 2024-04-06T14:35:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0c117e50 con 0x7faf0c101d10 2024-04-06T14:35:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faefaffd700 1 -- 172.21.15.32:0/2831738481 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faefc00d310 con 0x7faf0c101d10 2024-04-06T14:35:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.285+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0c1183e0 con 0x7faf0c101d10 2024-04-06T14:35:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.286+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0c0fd460 con 0x7faf0c101d10 2024-04-06T14:35:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.287+0000 7faefaffd700 1 -- 172.21.15.32:0/2831738481 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faefc010420 con 0x7faf0c101d10 2024-04-06T14:35:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.287+0000 7faefaffd700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faef406cc30 0x7faef406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.287+0000 7faefaffd700 1 -- 172.21.15.32:0/2831738481 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faefc08bf00 con 0x7faf0c101d10 2024-04-06T14:35:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.287+0000 7faf09d9b700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faef406cc30 0x7faef406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.288+0000 7faf09d9b700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faef406cc30 0x7faef406f0e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faf0000b3b0 tx=0x7faf00009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.292+0000 7faefaffd700 1 -- 172.21.15.32:0/2831738481 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faefc056000 con 0x7faf0c101d10 2024-04-06T14:35:48.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.501+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faf0c066640 con 0x7faef406cc30 2024-04-06T14:35:48.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.511+0000 7faefaffd700 1 -- 172.21.15.32:0/2831738481 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7faf0c066640 con 0x7faef406cc30 2024-04-06T14:35:48.508 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:35:48.508 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (16m) 9m ago 27m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:35:48.509 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (13m) 9m ago 27m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:35:48.509 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (13m) 9m ago 26m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:35:48.509 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (15m) 9m ago 27m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:35:48.509 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (9m) 9m ago 21m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:35:48.509 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (9m) 9m ago 21m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:35:48.509 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (9m) 9m ago 21m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:35:48.509 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (21m) 9m ago 21m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:35:48.509 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (18m) 9m ago 28m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:35:48.509 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (17m) 9m ago 26m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (14m) 9m ago 29m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (14m) 9m ago 26m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (16m) 9m ago 27m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (16m) 9m ago 26m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (13m) 9m ago 25m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (12m) 9m ago 25m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 9m ago 24m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (11m) 9m ago 24m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (10m) 9m ago 23m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (10m) 9m ago 23m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:35:48.510 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (16m) 9m ago 26m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:35:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faef406cc30 msgr2=0x7faef406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faef406cc30 0x7faef406f0e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faf0000b3b0 tx=0x7faf00009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c101d10 msgr2=0x7faf0c10ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c101d10 0x7faf0c10ed80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faefc00b790 tx=0x7faefc009250 comp rx=0 tx=0).stop 2024-04-06T14:35:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 shutdown_connections 2024-04-06T14:35:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faef406cc30 0x7faef406f0e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf0c101d10 0x7faf0c10ed80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 --2- 172.21.15.32:0/2831738481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf0c1026b0 0x7faf0c10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 >> 172.21.15.32:0/2831738481 conn(0x7faf0c0f7880 msgr2=0x7faf0c0f88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 shutdown_connections 2024-04-06T14:35:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.513+0000 7faf10cd5700 1 -- 172.21.15.32:0/2831738481 wait complete. 2024-04-06T14:35:48.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.615+0000 7fcdb5974700 1 -- 172.21.15.32:0/62680429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb0101d20 msgr2=0x7fcdb01020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.615+0000 7fcdb5974700 1 --2- 172.21.15.32:0/62680429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb0101d20 0x7fcdb01020f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcda0004940 tx=0x7fcda0009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:48.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.615+0000 7fcdb5974700 1 -- 172.21.15.32:0/62680429 shutdown_connections 2024-04-06T14:35:48.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.615+0000 7fcdb5974700 1 --2- 172.21.15.32:0/62680429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdb01026c0 0x7fcdb00fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.615+0000 7fcdb5974700 1 --2- 172.21.15.32:0/62680429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb0101d20 0x7fcdb01020f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.615+0000 7fcdb5974700 1 -- 172.21.15.32:0/62680429 >> 172.21.15.32:0/62680429 conn(0x7fcdb00f7850 msgr2=0x7fcdb00f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:48.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.616+0000 7fcdb5974700 1 -- 172.21.15.32:0/62680429 shutdown_connections 2024-04-06T14:35:48.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.616+0000 7fcdb5974700 1 -- 172.21.15.32:0/62680429 wait complete. 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.616+0000 7fcdb5974700 1 Processor -- start 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.616+0000 7fcdb5974700 1 -- start start 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdb5974700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdb0101d20 0x7fcdb010e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdb5974700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb01026c0 0x7fcdb010eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdb5974700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb010f3f0 con 0x7fcdb01026c0 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdb5974700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb010f530 con 0x7fcdb0101d20 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdae7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb01026c0 0x7fcdb010eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdae7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb01026c0 0x7fcdb010eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36326/0 (socket says 172.21.15.32:36326) 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdae7fc700 1 -- 172.21.15.32:0/2611846869 learned_addr learned my addr 172.21.15.32:0/2611846869 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdae7fc700 1 -- 172.21.15.32:0/2611846869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdb0101d20 msgr2=0x7fcdb010e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdaeffd700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdb0101d20 0x7fcdb010e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdae7fc700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdb0101d20 0x7fcdb010e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdae7fc700 1 -- 172.21.15.32:0/2611846869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda0005040 con 0x7fcdb01026c0 2024-04-06T14:35:48.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.617+0000 7fcdae7fc700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb01026c0 0x7fcdb010eeb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcda4002a50 tx=0x7fcda40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:48.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.618+0000 7fcd977fe700 1 -- 172.21.15.32:0/2611846869 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda4007cc0 con 0x7fcdb01026c0 2024-04-06T14:35:48.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.618+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb01179e0 con 0x7fcdb01026c0 2024-04-06T14:35:48.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.618+0000 7fcd977fe700 1 -- 172.21.15.32:0/2611846869 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcda400f040 con 0x7fcdb01026c0 2024-04-06T14:35:48.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.618+0000 7fcd977fe700 1 -- 172.21.15.32:0/2611846869 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda40095d0 con 0x7fcdb01026c0 2024-04-06T14:35:48.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.618+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb0117db0 con 0x7fcdb01026c0 2024-04-06T14:35:48.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.619+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb00fd470 con 0x7fcdb01026c0 2024-04-06T14:35:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.619+0000 7fcd977fe700 1 -- 172.21.15.32:0/2611846869 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcda4009730 con 0x7fcdb01026c0 2024-04-06T14:35:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.619+0000 7fcd977fe700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcd9806cc20 0x7fcd9806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.619+0000 7fcd977fe700 1 -- 172.21.15.32:0/2611846869 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcda408bd40 con 0x7fcdb01026c0 2024-04-06T14:35:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.620+0000 7fcdaeffd700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcd9806cc20 0x7fcd9806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.620+0000 7fcdaeffd700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcd9806cc20 0x7fcd9806f0d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcda000b3b0 tx=0x7fcda0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:48.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.624+0000 7fcd977fe700 1 -- 172.21.15.32:0/2611846869 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcda408f030 con 0x7fcdb01026c0 2024-04-06T14:35:48.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:48 smithi086 ceph-mon[110975]: pgmap v469: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:48.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.911+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcdb0108ce0 con 0x7fcdb01026c0 2024-04-06T14:35:48.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.911+0000 7fcd977fe700 1 -- 172.21.15.32:0/2611846869 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fcda4055d00 con 0x7fcdb01026c0 2024-04-06T14:35:48.908 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:35:48.908 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:35:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:35:48.910 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:35:48.910 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:35:48.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcd9806cc20 msgr2=0x7fcd9806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcd9806cc20 0x7fcd9806f0d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcda000b3b0 tx=0x7fcda0009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb01026c0 msgr2=0x7fcdb010eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb01026c0 0x7fcdb010eeb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcda4002a50 tx=0x7fcda40046b0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 shutdown_connections 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcd9806cc20 0x7fcd9806f0d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdb0101d20 0x7fcdb010e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 --2- 172.21.15.32:0/2611846869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb01026c0 0x7fcdb010eeb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 >> 172.21.15.32:0/2611846869 conn(0x7fcdb00f7850 msgr2=0x7fcdb00f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 shutdown_connections 2024-04-06T14:35:48.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:48.913+0000 7fcdb5974700 1 -- 172.21.15.32:0/2611846869 wait complete. 2024-04-06T14:35:48.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:48 smithi032 ceph-mon[147414]: pgmap v469: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:49.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.016+0000 7fb69e22e700 1 -- 172.21.15.32:0/3315356768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 msgr2=0x7fb6980fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.016+0000 7fb69e22e700 1 --2- 172.21.15.32:0/3315356768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 0x7fb6980fe260 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb68c004940 tx=0x7fb68c009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:49.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.016+0000 7fb69e22e700 1 -- 172.21.15.32:0/3315356768 shutdown_connections 2024-04-06T14:35:49.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.016+0000 7fb69e22e700 1 --2- 172.21.15.32:0/3315356768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6980fe830 0x7fb698106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.016+0000 7fb69e22e700 1 --2- 172.21.15.32:0/3315356768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 0x7fb6980fe260 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.017+0000 7fb69e22e700 1 -- 172.21.15.32:0/3315356768 >> 172.21.15.32:0/3315356768 conn(0x7fb698072bc0 msgr2=0x7fb698072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.017+0000 7fb69e22e700 1 -- 172.21.15.32:0/3315356768 shutdown_connections 2024-04-06T14:35:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.017+0000 7fb69e22e700 1 -- 172.21.15.32:0/3315356768 wait complete. 2024-04-06T14:35:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.017+0000 7fb69e22e700 1 Processor -- start 2024-04-06T14:35:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.017+0000 7fb69e22e700 1 -- start start 2024-04-06T14:35:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb69e22e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 0x7fb698113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb69e22e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6980fe830 0x7fb698114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb69e22e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb698114680 con 0x7fb6980fde90 2024-04-06T14:35:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb69e22e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69810da70 con 0x7fb6980fe830 2024-04-06T14:35:49.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb6977fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 0x7fb698113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:49.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb6977fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 0x7fb698113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36348/0 (socket says 172.21.15.32:36348) 2024-04-06T14:35:49.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb6977fe700 1 -- 172.21.15.32:0/639197468 learned_addr learned my addr 172.21.15.32:0/639197468 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:49.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb696ffd700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6980fe830 0x7fb698114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:49.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb6977fe700 1 -- 172.21.15.32:0/639197468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6980fe830 msgr2=0x7fb698114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb6977fe700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6980fe830 0x7fb698114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb6977fe700 1 -- 172.21.15.32:0/639197468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c005040 con 0x7fb6980fde90 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb6977fe700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 0x7fb698113af0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00b3b0 tx=0x7fb68c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb67ffff700 1 -- 172.21.15.32:0/639197468 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb68c00e040 con 0x7fb6980fde90 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb67ffff700 1 -- 172.21.15.32:0/639197468 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb68c013420 con 0x7fb6980fde90 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb67ffff700 1 -- 172.21.15.32:0/639197468 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb68c01c5d0 con 0x7fb6980fde90 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69810dcf0 con 0x7fb6980fde90 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.018+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69810e230 con 0x7fb6980fde90 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.020+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6981019b0 con 0x7fb6980fde90 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.020+0000 7fb67ffff700 1 -- 172.21.15.32:0/639197468 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb68c013590 con 0x7fb6980fde90 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.020+0000 7fb67ffff700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb68006cc20 0x7fb68006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.021+0000 7fb67ffff700 1 -- 172.21.15.32:0/639197468 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb68c08d2c0 con 0x7fb6980fde90 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.021+0000 7fb696ffd700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb68006cc20 0x7fb68006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.021+0000 7fb696ffd700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb68006cc20 0x7fb68006f0d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb69810ec30 tx=0x7fb688009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:49.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.025+0000 7fb67ffff700 1 -- 172.21.15.32:0/639197468 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb68c05ad10 con 0x7fb6980fde90 2024-04-06T14:35:49.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.272+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb69810f140 con 0x7fb6980fde90 2024-04-06T14:35:49.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.272+0000 7fb67ffff700 1 -- 172.21.15.32:0/639197468 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fb68c05a8a0 con 0x7fb6980fde90 2024-04-06T14:35:49.273 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:35:49.273 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:35:49.273 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:35:49.273 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:35:49.273 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:35:49.273 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:35:49.273 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:35:49.273 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:35:49.274 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:35:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:35:49.276 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:35:49.276 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:35:49.276 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:35:49.276 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:35:49.276 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:35:49.276 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:35:49.276 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:35:49.276 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:35:49.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb68006cc20 msgr2=0x7fb68006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb68006cc20 0x7fb68006f0d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb69810ec30 tx=0x7fb688009ee0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 msgr2=0x7fb698113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 0x7fb698113af0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00b3b0 tx=0x7fb68c009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:49.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 shutdown_connections 2024-04-06T14:35:49.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb68006cc20 0x7fb68006f0d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6980fde90 0x7fb698113af0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 --2- 172.21.15.32:0/639197468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6980fe830 0x7fb698114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 >> 172.21.15.32:0/639197468 conn(0x7fb698072bc0 msgr2=0x7fb6980ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:49.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 shutdown_connections 2024-04-06T14:35:49.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.274+0000 7fb69e22e700 1 -- 172.21.15.32:0/639197468 wait complete. 2024-04-06T14:35:49.278 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:35:49.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.379+0000 7fd37dc5d700 1 -- 172.21.15.32:0/193657772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 msgr2=0x7fd3781031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.379+0000 7fd37dc5d700 1 --2- 172.21.15.32:0/193657772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 0x7fd3781031d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd360004940 tx=0x7fd360009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:49.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.379+0000 7fd37dc5d700 1 -- 172.21.15.32:0/193657772 shutdown_connections 2024-04-06T14:35:49.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.379+0000 7fd37dc5d700 1 --2- 172.21.15.32:0/193657772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3781037a0 0x7fd3780fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.379+0000 7fd37dc5d700 1 --2- 172.21.15.32:0/193657772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 0x7fd3781031d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.379+0000 7fd37dc5d700 1 -- 172.21.15.32:0/193657772 >> 172.21.15.32:0/193657772 conn(0x7fd3780f89b0 msgr2=0x7fd3780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.379+0000 7fd37dc5d700 1 -- 172.21.15.32:0/193657772 shutdown_connections 2024-04-06T14:35:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.379+0000 7fd37dc5d700 1 -- 172.21.15.32:0/193657772 wait complete. 2024-04-06T14:35:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.379+0000 7fd37dc5d700 1 Processor -- start 2024-04-06T14:35:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd37dc5d700 1 -- start start 2024-04-06T14:35:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd37dc5d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 0x7fd378113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd37dc5d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3781037a0 0x7fd378113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd37dc5d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3781146a0 con 0x7fd378102e00 2024-04-06T14:35:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd37dc5d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37810da00 con 0x7fd3781037a0 2024-04-06T14:35:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd3777fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 0x7fd378113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd3777fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 0x7fd378113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36372/0 (socket says 172.21.15.32:36372) 2024-04-06T14:35:49.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd3777fe700 1 -- 172.21.15.32:0/922571165 learned_addr learned my addr 172.21.15.32:0/922571165 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:49.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd3777fe700 1 -- 172.21.15.32:0/922571165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3781037a0 msgr2=0x7fd378113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd376ffd700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3781037a0 0x7fd378113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:49.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd3777fe700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3781037a0 0x7fd378113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.380+0000 7fd3777fe700 1 -- 172.21.15.32:0/922571165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360005040 con 0x7fd378102e00 2024-04-06T14:35:49.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.381+0000 7fd3777fe700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 0x7fd378113a80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd360009390 tx=0x7fd360009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.381+0000 7fd36ffff700 1 -- 172.21.15.32:0/922571165 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd36000e040 con 0x7fd378102e00 2024-04-06T14:35:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.381+0000 7fd36ffff700 1 -- 172.21.15.32:0/922571165 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd360013420 con 0x7fd378102e00 2024-04-06T14:35:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.381+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37810dc80 con 0x7fd378102e00 2024-04-06T14:35:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.381+0000 7fd36ffff700 1 -- 172.21.15.32:0/922571165 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd36001c630 con 0x7fd378102e00 2024-04-06T14:35:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.381+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37810e240 con 0x7fd378102e00 2024-04-06T14:35:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.382+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3780fe550 con 0x7fd378102e00 2024-04-06T14:35:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.382+0000 7fd36ffff700 1 -- 172.21.15.32:0/922571165 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd3600139e0 con 0x7fd378102e00 2024-04-06T14:35:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.383+0000 7fd36ffff700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd36406cc30 0x7fd36406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.383+0000 7fd36ffff700 1 -- 172.21.15.32:0/922571165 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd36008d230 con 0x7fd378102e00 2024-04-06T14:35:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.383+0000 7fd376ffd700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd36406cc30 0x7fd36406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.383+0000 7fd376ffd700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd36406cc30 0x7fd36406f0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd368006fd0 tx=0x7fd368008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.387+0000 7fd36ffff700 1 -- 172.21.15.32:0/922571165 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd36005ae70 con 0x7fd378102e00 2024-04-06T14:35:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.609+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd378109ed0 con 0x7fd36406cc30 2024-04-06T14:35:49.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.611+0000 7fd36ffff700 1 -- 172.21.15.32:0/922571165 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd368007dd0 con 0x7fd36406cc30 2024-04-06T14:35:49.609 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:35:49.609 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:35:49.609 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:35:49.609 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:35:49.609 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:35:49.609 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:35:49.610 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:35:49.610 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:35:49.610 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:35:49.610 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:35:49.610 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:35:49.610 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:35:49.610 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:35:49.611 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:35:49.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd36406cc30 msgr2=0x7fd36406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd36406cc30 0x7fd36406f0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd368006fd0 tx=0x7fd368008040 comp rx=0 tx=0).stop 2024-04-06T14:35:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 msgr2=0x7fd378113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 0x7fd378113a80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd360009390 tx=0x7fd360009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 shutdown_connections 2024-04-06T14:35:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd36406cc30 0x7fd36406f0e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd378102e00 0x7fd378113a80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 --2- 172.21.15.32:0/922571165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3781037a0 0x7fd378113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 >> 172.21.15.32:0/922571165 conn(0x7fd3780f89b0 msgr2=0x7fd3780f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 shutdown_connections 2024-04-06T14:35:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.613+0000 7fd37dc5d700 1 -- 172.21.15.32:0/922571165 wait complete. 2024-04-06T14:35:49.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.716+0000 7f2cdda27700 1 -- 172.21.15.32:0/3771867910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd8102e00 msgr2=0x7f2cd81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.716+0000 7f2cdda27700 1 --2- 172.21.15.32:0/3771867910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd8102e00 0x7f2cd81031d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8004940 tx=0x7f2cc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:49.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.716+0000 7f2cdda27700 1 -- 172.21.15.32:0/3771867910 shutdown_connections 2024-04-06T14:35:49.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.716+0000 7f2cdda27700 1 --2- 172.21.15.32:0/3771867910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd81037a0 0x7f2cd80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.716+0000 7f2cdda27700 1 --2- 172.21.15.32:0/3771867910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd8102e00 0x7f2cd81031d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.716+0000 7f2cdda27700 1 -- 172.21.15.32:0/3771867910 >> 172.21.15.32:0/3771867910 conn(0x7f2cd80f89b0 msgr2=0x7f2cd80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:49.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.716+0000 7f2cdda27700 1 -- 172.21.15.32:0/3771867910 shutdown_connections 2024-04-06T14:35:49.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.716+0000 7f2cdda27700 1 -- 172.21.15.32:0/3771867910 wait complete. 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cdda27700 1 Processor -- start 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cdda27700 1 -- start start 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cdda27700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd8102e00 0x7f2cd81138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cdda27700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd81037a0 0x7f2cd8113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cdda27700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81144c0 con 0x7f2cd81037a0 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cdda27700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd810da30 con 0x7f2cd8102e00 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cd67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd81037a0 0x7f2cd8113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cd67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd81037a0 0x7f2cd8113de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36384/0 (socket says 172.21.15.32:36384) 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cd67fc700 1 -- 172.21.15.32:0/931734993 learned_addr learned my addr 172.21.15.32:0/931734993 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cd6ffd700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd8102e00 0x7f2cd81138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:49.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cd67fc700 1 -- 172.21.15.32:0/931734993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd8102e00 msgr2=0x7f2cd81138a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:49.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cd67fc700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd8102e00 0x7f2cd81138a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:49.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.717+0000 7f2cd67fc700 1 -- 172.21.15.32:0/931734993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc8005040 con 0x7f2cd81037a0 2024-04-06T14:35:49.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.718+0000 7f2cd67fc700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd81037a0 0x7f2cd8113de0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00b790 tx=0x7f2ccc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:49.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.718+0000 7f2cbf7fe700 1 -- 172.21.15.32:0/931734993 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ccc00f6f0 con 0x7f2cd81037a0 2024-04-06T14:35:49.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.718+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd810dd10 con 0x7f2cd81037a0 2024-04-06T14:35:49.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.718+0000 7f2cbf7fe700 1 -- 172.21.15.32:0/931734993 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ccc00fd00 con 0x7f2cd81037a0 2024-04-06T14:35:49.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.718+0000 7f2cbf7fe700 1 -- 172.21.15.32:0/931734993 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ccc0103a0 con 0x7f2cd81037a0 2024-04-06T14:35:49.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.718+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd810e2d0 con 0x7f2cd81037a0 2024-04-06T14:35:49.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.719+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd80fe550 con 0x7f2cd81037a0 2024-04-06T14:35:49.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.719+0000 7f2cbf7fe700 1 -- 172.21.15.32:0/931734993 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2ccc00f850 con 0x7f2cd81037a0 2024-04-06T14:35:49.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.720+0000 7f2cbf7fe700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2cc006cc20 0x7f2cc006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:35:49.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.720+0000 7f2cbf7fe700 1 -- 172.21.15.32:0/931734993 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2ccc08be50 con 0x7f2cd81037a0 2024-04-06T14:35:49.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.720+0000 7f2cd6ffd700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2cc006cc20 0x7f2cc006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:35:49.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.720+0000 7f2cd6ffd700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2cc006cc20 0x7f2cc006f0d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8004f90 tx=0x7f2cc8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:35:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:49.724+0000 7f2cbf7fe700 1 -- 172.21.15.32:0/931734993 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2ccc059920 con 0x7f2cd81037a0 2024-04-06T14:35:49.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:49 smithi086 ceph-mon[110975]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:49.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:49 smithi086 ceph-mon[110975]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:49.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:49 smithi086 ceph-mon[110975]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:49.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:49 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2611846869' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:35:49.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:49 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/639197468' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:35:49.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:49 smithi032 ceph-mon[147414]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:49.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:49 smithi032 ceph-mon[147414]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:49.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:49 smithi032 ceph-mon[147414]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:49.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:49 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2611846869' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:35:49.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:49 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/639197468' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:35:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.012+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2cd8109e30 con 0x7f2cd81037a0 2024-04-06T14:35:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.012+0000 7f2cbf7fe700 1 -- 172.21.15.32:0/931734993 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f2ccc0594b0 con 0x7f2cd81037a0 2024-04-06T14:35:50.009 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:35:50.009 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:35:50.009 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:35:50.009 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:35:50.010 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:35:50.010 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:35:50.010 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:35:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2cc006cc20 msgr2=0x7f2cc006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2cc006cc20 0x7f2cc006f0d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8004f90 tx=0x7f2cc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:35:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd81037a0 msgr2=0x7f2cd8113de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:35:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd81037a0 0x7f2cd8113de0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00b790 tx=0x7f2ccc009220 comp rx=0 tx=0).stop 2024-04-06T14:35:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 shutdown_connections 2024-04-06T14:35:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2cc006cc20 0x7f2cc006f0d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd8102e00 0x7f2cd81138a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 --2- 172.21.15.32:0/931734993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd81037a0 0x7f2cd8113de0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:35:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 >> 172.21.15.32:0/931734993 conn(0x7f2cd80f89b0 msgr2=0x7f2cd80f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:35:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 shutdown_connections 2024-04-06T14:35:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:35:50.014+0000 7f2cdda27700 1 -- 172.21.15.32:0/931734993 wait complete. 2024-04-06T14:35:50.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:50 smithi086 ceph-mon[110975]: pgmap v470: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:50.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:50 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/931734993' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:35:50.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:50 smithi032 ceph-mon[147414]: pgmap v470: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:50.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:50 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/931734993' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:35:51.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:51 smithi086 ceph-mon[110975]: from='client.34916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:51.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:51 smithi032 ceph-mon[147414]: from='client.34916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:35:52.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:52 smithi086 ceph-mon[110975]: pgmap v471: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:52.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:52 smithi032 ceph-mon[147414]: pgmap v471: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:53.727 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:35:53.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:35:54.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:54 smithi086 ceph-mon[110975]: pgmap v472: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:54.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:54 smithi032 ceph-mon[147414]: pgmap v472: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:56.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:56 smithi086 ceph-mon[110975]: pgmap v473: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:56.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:56 smithi032 ceph-mon[147414]: pgmap v473: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:58.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:35:58 smithi086 ceph-mon[110975]: pgmap v474: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:35:58.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:35:58 smithi032 ceph-mon[147414]: pgmap v474: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:00.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:00 smithi086 ceph-mon[110975]: pgmap v475: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:00.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:00 smithi032 ceph-mon[147414]: pgmap v475: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:02.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:02 smithi086 ceph-mon[110975]: pgmap v476: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:02.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:02 smithi032 ceph-mon[147414]: pgmap v476: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:04.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:04 smithi086 ceph-mon[110975]: pgmap v477: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:04.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:04 smithi032 ceph-mon[147414]: pgmap v477: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:06.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:06 smithi086 ceph-mon[110975]: pgmap v478: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:06.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:06 smithi032 ceph-mon[147414]: pgmap v478: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:08.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:08 smithi086 ceph-mon[110975]: pgmap v479: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:08.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:36:08.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:08 smithi032 ceph-mon[147414]: pgmap v479: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:08.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:36:10.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:10 smithi086 ceph-mon[110975]: pgmap v480: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:10.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:10 smithi032 ceph-mon[147414]: pgmap v480: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:12.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:12 smithi086 ceph-mon[110975]: pgmap v481: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:12.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:12 smithi032 ceph-mon[147414]: pgmap v481: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:14.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:14 smithi086 ceph-mon[110975]: pgmap v482: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:14.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:14 smithi032 ceph-mon[147414]: pgmap v482: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:16.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:16 smithi086 ceph-mon[110975]: pgmap v483: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:16.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:16 smithi032 ceph-mon[147414]: pgmap v483: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:18.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:18 smithi086 ceph-mon[110975]: pgmap v484: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:18.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:18 smithi032 ceph-mon[147414]: pgmap v484: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:20.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.132+0000 7fd1ad777700 1 -- 172.21.15.32:0/2060924448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 msgr2=0x7fd1a80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.132+0000 7fd1ad777700 1 --2- 172.21.15.32:0/2060924448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 0x7fd1a80fce10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd19c004940 tx=0x7fd19c009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.133+0000 7fd1ad777700 1 -- 172.21.15.32:0/2060924448 shutdown_connections 2024-04-06T14:36:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.133+0000 7fd1ad777700 1 --2- 172.21.15.32:0/2060924448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 0x7fd1a80fce10 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.133+0000 7fd1ad777700 1 --2- 172.21.15.32:0/2060924448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a8102e00 0x7fd1a81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.133+0000 7fd1ad777700 1 -- 172.21.15.32:0/2060924448 >> 172.21.15.32:0/2060924448 conn(0x7fd1a80f89b0 msgr2=0x7fd1a80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.133+0000 7fd1ad777700 1 -- 172.21.15.32:0/2060924448 shutdown_connections 2024-04-06T14:36:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.133+0000 7fd1ad777700 1 -- 172.21.15.32:0/2060924448 wait complete. 2024-04-06T14:36:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.133+0000 7fd1ad777700 1 Processor -- start 2024-04-06T14:36:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.133+0000 7fd1ad777700 1 -- start start 2024-04-06T14:36:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1ad777700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a8102e00 0x7fd1a80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1ad777700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 0x7fd1a8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1ad777700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80722a0 con 0x7fd1a81037a0 2024-04-06T14:36:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1ad777700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a8070d40 con 0x7fd1a8102e00 2024-04-06T14:36:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1a67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 0x7fd1a8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1a67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 0x7fd1a8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34710/0 (socket says 172.21.15.32:34710) 2024-04-06T14:36:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1a67fc700 1 -- 172.21.15.32:0/2708886871 learned_addr learned my addr 172.21.15.32:0/2708886871 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1a6ffd700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a8102e00 0x7fd1a80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1a67fc700 1 -- 172.21.15.32:0/2708886871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a8102e00 msgr2=0x7fd1a80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1a67fc700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a8102e00 0x7fd1a80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1a67fc700 1 -- 172.21.15.32:0/2708886871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c005040 con 0x7fd1a81037a0 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd1a67fc700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 0x7fd1a8070800 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009380 tx=0x7fd19c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.134+0000 7fd18f7fe700 1 -- 172.21.15.32:0/2708886871 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd19c00e040 con 0x7fd1a81037a0 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.135+0000 7fd18f7fe700 1 -- 172.21.15.32:0/2708886871 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd19c013420 con 0x7fd1a81037a0 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.135+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a8070fc0 con 0x7fd1a81037a0 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.135+0000 7fd18f7fe700 1 -- 172.21.15.32:0/2708886871 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd19c01c630 con 0x7fd1a81037a0 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.135+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a811a0c0 con 0x7fd1a81037a0 2024-04-06T14:36:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.136+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a80fe550 con 0x7fd1a81037a0 2024-04-06T14:36:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.136+0000 7fd18f7fe700 1 -- 172.21.15.32:0/2708886871 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd19c0139e0 con 0x7fd1a81037a0 2024-04-06T14:36:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.137+0000 7fd18f7fe700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd19006cc20 0x7fd19006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.137+0000 7fd18f7fe700 1 -- 172.21.15.32:0/2708886871 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd19c08d100 con 0x7fd1a81037a0 2024-04-06T14:36:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.137+0000 7fd1a6ffd700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd19006cc20 0x7fd19006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.137+0000 7fd1a6ffd700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd19006cc20 0x7fd19006f0d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd198006fd0 tx=0x7fd198008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:20.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.141+0000 7fd18f7fe700 1 -- 172.21.15.32:0/2708886871 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd19c057200 con 0x7fd1a81037a0 2024-04-06T14:36:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.364+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1a80fa0b0 con 0x7fd19006cc20 2024-04-06T14:36:20.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.365+0000 7fd18f7fe700 1 -- 172.21.15.32:0/2708886871 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd1a80fa0b0 con 0x7fd19006cc20 2024-04-06T14:36:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd19006cc20 msgr2=0x7fd19006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd19006cc20 0x7fd19006f0d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd198006fd0 tx=0x7fd198008040 comp rx=0 tx=0).stop 2024-04-06T14:36:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 msgr2=0x7fd1a8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 0x7fd1a8070800 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009380 tx=0x7fd19c009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 shutdown_connections 2024-04-06T14:36:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd19006cc20 0x7fd19006f0d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a8102e00 0x7fd1a80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 --2- 172.21.15.32:0/2708886871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a81037a0 0x7fd1a8070800 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 >> 172.21.15.32:0/2708886871 conn(0x7fd1a80f89b0 msgr2=0x7fd1a80f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:20.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 shutdown_connections 2024-04-06T14:36:20.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.367+0000 7fd1ad777700 1 -- 172.21.15.32:0/2708886871 wait complete. 2024-04-06T14:36:20.374 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:36:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.472+0000 7f038b733700 1 -- 172.21.15.32:0/1385186750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 msgr2=0x7f03840fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.472+0000 7f038b733700 1 --2- 172.21.15.32:0/1385186750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 0x7f03840fbd80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0374004940 tx=0x7f0374009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.472+0000 7f038b733700 1 -- 172.21.15.32:0/1385186750 shutdown_connections 2024-04-06T14:36:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.472+0000 7f038b733700 1 --2- 172.21.15.32:0/1385186750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 0x7f03840fbd80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.472+0000 7f038b733700 1 --2- 172.21.15.32:0/1385186750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0384101d70 0x7f0384102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.472+0000 7f038b733700 1 -- 172.21.15.32:0/1385186750 >> 172.21.15.32:0/1385186750 conn(0x7f0384072bc0 msgr2=0x7f0384072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.472+0000 7f038b733700 1 -- 172.21.15.32:0/1385186750 shutdown_connections 2024-04-06T14:36:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.472+0000 7f038b733700 1 -- 172.21.15.32:0/1385186750 wait complete. 2024-04-06T14:36:20.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.472+0000 7f038b733700 1 Processor -- start 2024-04-06T14:36:20.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f038b733700 1 -- start start 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f038b733700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0384101d70 0x7f03841129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f038b733700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 0x7f0384112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f038b733700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0384113550 con 0x7f0384102710 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f038b733700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038410c940 con 0x7f0384101d70 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f0388cce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 0x7f0384112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f0388cce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 0x7f0384112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34728/0 (socket says 172.21.15.32:34728) 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f0388cce700 1 -- 172.21.15.32:0/2052063291 learned_addr learned my addr 172.21.15.32:0/2052063291 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f03894cf700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0384101d70 0x7f03841129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f0388cce700 1 -- 172.21.15.32:0/2052063291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0384101d70 msgr2=0x7f03841129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f0388cce700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0384101d70 0x7f03841129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f0388cce700 1 -- 172.21.15.32:0/2052063291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374005040 con 0x7f0384102710 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.473+0000 7f03894cf700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0384101d70 0x7f03841129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.474+0000 7f0388cce700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 0x7f0384112f00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0374009380 tx=0x7f0374009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:20.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.474+0000 7f0379ffb700 1 -- 172.21.15.32:0/2052063291 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f037400e040 con 0x7f0384102710 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.474+0000 7f0379ffb700 1 -- 172.21.15.32:0/2052063291 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0374013420 con 0x7f0384102710 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.474+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038410cbc0 con 0x7f0384102710 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.474+0000 7f0379ffb700 1 -- 172.21.15.32:0/2052063291 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f037401c630 con 0x7f0384102710 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.474+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038410d180 con 0x7f0384102710 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.475+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03840ff080 con 0x7f0384102710 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.475+0000 7f0379ffb700 1 -- 172.21.15.32:0/2052063291 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f03740139e0 con 0x7f0384102710 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.476+0000 7f0379ffb700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f037006cc20 0x7f037006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.476+0000 7f0379ffb700 1 -- 172.21.15.32:0/2052063291 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f037408d230 con 0x7f0384102710 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.476+0000 7f03894cf700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f037006cc20 0x7f037006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.476+0000 7f03894cf700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f037006cc20 0x7f037006f0d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0380007fd0 tx=0x7f0380009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.480+0000 7f0379ffb700 1 -- 172.21.15.32:0/2052063291 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f037405ad00 con 0x7f0384102710 2024-04-06T14:36:20.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.702+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f038410e010 con 0x7f037006cc20 2024-04-06T14:36:20.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.704+0000 7f0379ffb700 1 -- 172.21.15.32:0/2052063291 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f038410e010 con 0x7f037006cc20 2024-04-06T14:36:20.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f037006cc20 msgr2=0x7f037006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f037006cc20 0x7f037006f0d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0380007fd0 tx=0x7f0380009040 comp rx=0 tx=0).stop 2024-04-06T14:36:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 msgr2=0x7f0384112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 0x7f0384112f00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0374009380 tx=0x7f0374009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 shutdown_connections 2024-04-06T14:36:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f037006cc20 0x7f037006f0d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0384101d70 0x7f03841129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 --2- 172.21.15.32:0/2052063291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0384102710 0x7f0384112f00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 >> 172.21.15.32:0/2052063291 conn(0x7f0384072bc0 msgr2=0x7f03840fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 shutdown_connections 2024-04-06T14:36:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.706+0000 7f038b733700 1 -- 172.21.15.32:0/2052063291 wait complete. 2024-04-06T14:36:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.809+0000 7f791a27a700 1 -- 172.21.15.32:0/201729095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fde90 msgr2=0x7f79140fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.809+0000 7f791a27a700 1 --2- 172.21.15.32:0/201729095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fde90 0x7f79140fe260 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7904004940 tx=0x7f7904009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.809+0000 7f791a27a700 1 -- 172.21.15.32:0/201729095 shutdown_connections 2024-04-06T14:36:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.809+0000 7f791a27a700 1 --2- 172.21.15.32:0/201729095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79140fe830 0x7f7914106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.809+0000 7f791a27a700 1 --2- 172.21.15.32:0/201729095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fde90 0x7f79140fe260 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.809+0000 7f791a27a700 1 -- 172.21.15.32:0/201729095 >> 172.21.15.32:0/201729095 conn(0x7f7914072bc0 msgr2=0x7f7914072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.809+0000 7f791a27a700 1 -- 172.21.15.32:0/201729095 shutdown_connections 2024-04-06T14:36:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.809+0000 7f791a27a700 1 -- 172.21.15.32:0/201729095 wait complete. 2024-04-06T14:36:20.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.810+0000 7f791a27a700 1 Processor -- start 2024-04-06T14:36:20.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.810+0000 7f791a27a700 1 -- start start 2024-04-06T14:36:20.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.810+0000 7f791a27a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79140fde90 0x7f7914113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:20.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.810+0000 7f791a27a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fe830 0x7f7914114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:20.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f791a27a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7914114710 con 0x7f79140fe830 2024-04-06T14:36:20.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f791a27a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791410da70 con 0x7f79140fde90 2024-04-06T14:36:20.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f79137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fe830 0x7f7914114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f79137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fe830 0x7f7914114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34738/0 (socket says 172.21.15.32:34738) 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f79137fe700 1 -- 172.21.15.32:0/2245324841 learned_addr learned my addr 172.21.15.32:0/2245324841 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f7913fff700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79140fde90 0x7f7914113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f79137fe700 1 -- 172.21.15.32:0/2245324841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79140fde90 msgr2=0x7f7914113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f79137fe700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79140fde90 0x7f7914113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f79137fe700 1 -- 172.21.15.32:0/2245324841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904005040 con 0x7f79140fe830 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f79137fe700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fe830 0x7f7914114030 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f790800b790 tx=0x7f7908009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f7910ff9700 1 -- 172.21.15.32:0/2245324841 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f790800f6f0 con 0x7f79140fe830 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f7910ff9700 1 -- 172.21.15.32:0/2245324841 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f790800fd00 con 0x7f79140fe830 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791410dd50 con 0x7f79140fe830 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f7910ff9700 1 -- 172.21.15.32:0/2245324841 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79080103a0 con 0x7f79140fe830 2024-04-06T14:36:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.811+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791410e290 con 0x7f79140fe830 2024-04-06T14:36:20.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.812+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7914065e30 con 0x7f79140fe830 2024-04-06T14:36:20.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.813+0000 7f7910ff9700 1 -- 172.21.15.32:0/2245324841 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f790800f850 con 0x7f79140fe830 2024-04-06T14:36:20.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.813+0000 7f7910ff9700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78fc06cc20 0x7f78fc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:20.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.813+0000 7f7910ff9700 1 -- 172.21.15.32:0/2245324841 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f790808bdb0 con 0x7f79140fe830 2024-04-06T14:36:20.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.813+0000 7f7913fff700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78fc06cc20 0x7f78fc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:20.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.814+0000 7f7913fff700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78fc06cc20 0x7f78fc06f0d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f790400b7e0 tx=0x7f7904009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:20.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:20.818+0000 7f7910ff9700 1 -- 172.21.15.32:0/2245324841 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7908059940 con 0x7f79140fe830 2024-04-06T14:36:20.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:20 smithi086 ceph-mon[110975]: pgmap v485: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:20.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:20 smithi032 ceph-mon[147414]: pgmap v485: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:21.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.028+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7914109e40 con 0x7f78fc06cc20 2024-04-06T14:36:21.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.038+0000 7f7910ff9700 1 -- 172.21.15.32:0/2245324841 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7914109e40 con 0x7f78fc06cc20 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (16m) 10m ago 27m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (14m) 10m ago 28m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (13m) 9m ago 27m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (16m) 10m ago 27m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (10m) 10m ago 22m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (10m) 10m ago 22m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (9m) 9m ago 22m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (22m) 9m ago 22m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (18m) 10m ago 29m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (17m) 9m ago 27m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (15m) 10m ago 29m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:36:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (14m) 9m ago 26m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:36:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (17m) 10m ago 28m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:36:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (17m) 9m ago 27m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:36:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (13m) 10m ago 26m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:36:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (13m) 10m ago 25m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:36:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (12m) 10m ago 25m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:36:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (11m) 9m ago 24m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:36:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (11m) 9m ago 24m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:36:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (10m) 9m ago 23m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:36:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (17m) 10m ago 27m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78fc06cc20 msgr2=0x7f78fc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78fc06cc20 0x7f78fc06f0d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f790400b7e0 tx=0x7f7904009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fe830 msgr2=0x7f7914114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fe830 0x7f7914114030 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f790800b790 tx=0x7f7908009250 comp rx=0 tx=0).stop 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 shutdown_connections 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78fc06cc20 0x7f78fc06f0d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79140fde90 0x7f7914113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 --2- 172.21.15.32:0/2245324841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79140fe830 0x7f7914114030 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 >> 172.21.15.32:0/2245324841 conn(0x7f7914072bc0 msgr2=0x7f79141037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 shutdown_connections 2024-04-06T14:36:21.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.040+0000 7f791a27a700 1 -- 172.21.15.32:0/2245324841 wait complete. 2024-04-06T14:36:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.142+0000 7f221e3ce700 1 -- 172.21.15.32:0/417154503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 msgr2=0x7f22180fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.142+0000 7f221e3ce700 1 --2- 172.21.15.32:0/417154503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 0x7f22180fbd80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2200004940 tx=0x7f2200009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.142+0000 7f221e3ce700 1 -- 172.21.15.32:0/417154503 shutdown_connections 2024-04-06T14:36:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.142+0000 7f221e3ce700 1 --2- 172.21.15.32:0/417154503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 0x7f22180fbd80 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.142+0000 7f221e3ce700 1 --2- 172.21.15.32:0/417154503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2218101d70 0x7f2218102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.142+0000 7f221e3ce700 1 -- 172.21.15.32:0/417154503 >> 172.21.15.32:0/417154503 conn(0x7f2218072bc0 msgr2=0x7f2218072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.142+0000 7f221e3ce700 1 -- 172.21.15.32:0/417154503 shutdown_connections 2024-04-06T14:36:21.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.142+0000 7f221e3ce700 1 -- 172.21.15.32:0/417154503 wait complete. 2024-04-06T14:36:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f221e3ce700 1 Processor -- start 2024-04-06T14:36:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f221e3ce700 1 -- start start 2024-04-06T14:36:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f221e3ce700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2218101d70 0x7f2218113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f221e3ce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 0x7f2218114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f221e3ce700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2218114650 con 0x7f2218102710 2024-04-06T14:36:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f221e3ce700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221810da40 con 0x7f2218101d70 2024-04-06T14:36:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f22177fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 0x7f2218114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f22177fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 0x7f2218114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34768/0 (socket says 172.21.15.32:34768) 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f22177fe700 1 -- 172.21.15.32:0/714608279 learned_addr learned my addr 172.21.15.32:0/714608279 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.143+0000 7f2217fff700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2218101d70 0x7f2218113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.144+0000 7f22177fe700 1 -- 172.21.15.32:0/714608279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2218101d70 msgr2=0x7f2218113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.144+0000 7f22177fe700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2218101d70 0x7f2218113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.144+0000 7f22177fe700 1 -- 172.21.15.32:0/714608279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2200005040 con 0x7f2218102710 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.144+0000 7f22177fe700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 0x7f2218114000 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f22000093b0 tx=0x7f2200009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.144+0000 7f2214ff9700 1 -- 172.21.15.32:0/714608279 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220000e040 con 0x7f2218102710 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.144+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221810dcc0 con 0x7f2218102710 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.144+0000 7f2214ff9700 1 -- 172.21.15.32:0/714608279 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2200013420 con 0x7f2218102710 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.144+0000 7f2214ff9700 1 -- 172.21.15.32:0/714608279 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220001c630 con 0x7f2218102710 2024-04-06T14:36:21.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.144+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221810e280 con 0x7f2218102710 2024-04-06T14:36:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.145+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22180fd4c0 con 0x7f2218102710 2024-04-06T14:36:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.145+0000 7f2214ff9700 1 -- 172.21.15.32:0/714608279 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f220000ba80 con 0x7f2218102710 2024-04-06T14:36:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.146+0000 7f2214ff9700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f220406cc20 0x7f220406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.146+0000 7f2214ff9700 1 -- 172.21.15.32:0/714608279 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f220008d1c0 con 0x7f2218102710 2024-04-06T14:36:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.146+0000 7f2217fff700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f220406cc20 0x7f220406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.146+0000 7f2217fff700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f220406cc20 0x7f220406f0d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2208004f40 tx=0x7f220800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:21.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.150+0000 7f2214ff9700 1 -- 172.21.15.32:0/714608279 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f220005ac90 con 0x7f2218102710 2024-04-06T14:36:21.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.436+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2218108d10 con 0x7f2218102710 2024-04-06T14:36:21.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.437+0000 7f2214ff9700 1 -- 172.21.15.32:0/714608279 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f220005a820 con 0x7f2218102710 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:36:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:36:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:36:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:36:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:36:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:36:21.435 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f220406cc20 msgr2=0x7f220406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f220406cc20 0x7f220406f0d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2208004f40 tx=0x7f220800b040 comp rx=0 tx=0).stop 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 msgr2=0x7f2218114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 0x7f2218114000 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f22000093b0 tx=0x7f2200009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 shutdown_connections 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f220406cc20 0x7f220406f0d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2218101d70 0x7f2218113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 --2- 172.21.15.32:0/714608279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2218102710 0x7f2218114000 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 >> 172.21.15.32:0/714608279 conn(0x7f2218072bc0 msgr2=0x7f22180659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 shutdown_connections 2024-04-06T14:36:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.439+0000 7f221e3ce700 1 -- 172.21.15.32:0/714608279 wait complete. 2024-04-06T14:36:21.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.541+0000 7fe345083700 1 -- 172.21.15.32:0/3045869683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 msgr2=0x7fe3400fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.541+0000 7fe345083700 1 --2- 172.21.15.32:0/3045869683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 0x7fe3400fbd80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe330004940 tx=0x7fe330009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:21.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.542+0000 7fe345083700 1 -- 172.21.15.32:0/3045869683 shutdown_connections 2024-04-06T14:36:21.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.542+0000 7fe345083700 1 --2- 172.21.15.32:0/3045869683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 0x7fe3400fbd80 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.542+0000 7fe345083700 1 --2- 172.21.15.32:0/3045869683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe340101d70 0x7fe340102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.542+0000 7fe345083700 1 -- 172.21.15.32:0/3045869683 >> 172.21.15.32:0/3045869683 conn(0x7fe340072bc0 msgr2=0x7fe340072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:21.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.542+0000 7fe345083700 1 -- 172.21.15.32:0/3045869683 shutdown_connections 2024-04-06T14:36:21.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.542+0000 7fe345083700 1 -- 172.21.15.32:0/3045869683 wait complete. 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.542+0000 7fe345083700 1 Processor -- start 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.542+0000 7fe345083700 1 -- start start 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.542+0000 7fe345083700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe340101d70 0x7fe3400fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe345083700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 0x7fe3400fdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe345083700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3400fe4c0 con 0x7fe340102710 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe345083700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3400fe600 con 0x7fe340101d70 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe33e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 0x7fe3400fdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe33e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 0x7fe3400fdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34790/0 (socket says 172.21.15.32:34790) 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe33e59c700 1 -- 172.21.15.32:0/4124205311 learned_addr learned my addr 172.21.15.32:0/4124205311 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe33ed9d700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe340101d70 0x7fe3400fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe33e59c700 1 -- 172.21.15.32:0/4124205311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe340101d70 msgr2=0x7fe3400fd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe33e59c700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe340101d70 0x7fe3400fd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe33e59c700 1 -- 172.21.15.32:0/4124205311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe330005040 con 0x7fe340102710 2024-04-06T14:36:21.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe33ed9d700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe340101d70 0x7fe3400fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:36:21.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe33e59c700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 0x7fe3400fdf30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe33000b3b0 tx=0x7fe330009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:21.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe3277fe700 1 -- 172.21.15.32:0/4124205311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33000e040 con 0x7fe340102710 2024-04-06T14:36:21.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3400fe850 con 0x7fe340102710 2024-04-06T14:36:21.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe3277fe700 1 -- 172.21.15.32:0/4124205311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe330013420 con 0x7fe340102710 2024-04-06T14:36:21.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe3277fe700 1 -- 172.21.15.32:0/4124205311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33001c630 con 0x7fe340102710 2024-04-06T14:36:21.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.543+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34011a210 con 0x7fe340102710 2024-04-06T14:36:21.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.544+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3400fd4c0 con 0x7fe340102710 2024-04-06T14:36:21.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.545+0000 7fe3277fe700 1 -- 172.21.15.32:0/4124205311 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe330013590 con 0x7fe340102710 2024-04-06T14:36:21.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.545+0000 7fe3277fe700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe32806cc30 0x7fe32806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:21.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.545+0000 7fe3277fe700 1 -- 172.21.15.32:0/4124205311 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe33008d150 con 0x7fe340102710 2024-04-06T14:36:21.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.545+0000 7fe33ed9d700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe32806cc30 0x7fe32806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:21.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.546+0000 7fe33ed9d700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe32806cc30 0x7fe32806f0e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe334009fd0 tx=0x7fe334009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:21.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.550+0000 7fe3277fe700 1 -- 172.21.15.32:0/4124205311 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe330057190 con 0x7fe340102710 2024-04-06T14:36:21.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.796+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe34004bed0 con 0x7fe340102710 2024-04-06T14:36:21.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.796+0000 7fe3277fe700 1 -- 172.21.15.32:0/4124205311 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fe33005a7b0 con 0x7fe340102710 2024-04-06T14:36:21.793 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:36:21.793 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:36:21.793 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:36:21.793 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:36:21.793 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:36:21.794 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:36:21.795 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:36:21.796 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:36:21.796 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:36:21.796 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:36:21.796 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:36:21.796 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:36:21.796 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:36:21.796 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe32806cc30 msgr2=0x7fe32806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe32806cc30 0x7fe32806f0e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe334009fd0 tx=0x7fe334009f20 comp rx=0 tx=0).stop 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 msgr2=0x7fe3400fdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 0x7fe3400fdf30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe33000b3b0 tx=0x7fe330009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 shutdown_connections 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe32806cc30 0x7fe32806f0e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe340101d70 0x7fe3400fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 --2- 172.21.15.32:0/4124205311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe340102710 0x7fe3400fdf30 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 >> 172.21.15.32:0/4124205311 conn(0x7fe340072bc0 msgr2=0x7fe340065a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 shutdown_connections 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.799+0000 7fe345083700 1 -- 172.21.15.32:0/4124205311 wait complete. 2024-04-06T14:36:21.797 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:36:21.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:21 smithi086 ceph-mon[110975]: from='client.34924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:21.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:21 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/714608279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:36:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.903+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/3469434226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 msgr2=0x7fc1a40fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.903+0000 7fc1a8d2f700 1 --2- 172.21.15.32:0/3469434226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 0x7fc1a40fbd90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc198004940 tx=0x7fc198009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.903+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/3469434226 shutdown_connections 2024-04-06T14:36:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.903+0000 7fc1a8d2f700 1 --2- 172.21.15.32:0/3469434226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 0x7fc1a40fbd90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.903+0000 7fc1a8d2f700 1 --2- 172.21.15.32:0/3469434226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a4101d80 0x7fc1a4102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.903+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/3469434226 >> 172.21.15.32:0/3469434226 conn(0x7fc1a40f78f0 msgr2=0x7fc1a40f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.903+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/3469434226 shutdown_connections 2024-04-06T14:36:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.903+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/3469434226 wait complete. 2024-04-06T14:36:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.904+0000 7fc1a8d2f700 1 Processor -- start 2024-04-06T14:36:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.904+0000 7fc1a8d2f700 1 -- start start 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.904+0000 7fc1a8d2f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a4101d80 0x7fc1a410c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.904+0000 7fc1a8d2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 0x7fc1a410cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.904+0000 7fc1a8d2f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40670a0 con 0x7fc1a4102720 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.904+0000 7fc1a8d2f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a410d240 con 0x7fc1a4101d80 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 0x7fc1a410cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 0x7fc1a410cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34802/0 (socket says 172.21.15.32:34802) 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a1d9b700 1 -- 172.21.15.32:0/319512155 learned_addr learned my addr 172.21.15.32:0/319512155 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a259c700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a4101d80 0x7fc1a410c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a1d9b700 1 -- 172.21.15.32:0/319512155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a4101d80 msgr2=0x7fc1a410c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a1d9b700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a4101d80 0x7fc1a410c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a1d9b700 1 -- 172.21.15.32:0/319512155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198005040 con 0x7fc1a4102720 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a1d9b700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 0x7fc1a410cd00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc198009390 tx=0x7fc198009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc192ffd700 1 -- 172.21.15.32:0/319512155 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19800e040 con 0x7fc1a4102720 2024-04-06T14:36:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc192ffd700 1 -- 172.21.15.32:0/319512155 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc198013420 con 0x7fc1a4102720 2024-04-06T14:36:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a410d4c0 con 0x7fc1a4102720 2024-04-06T14:36:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc192ffd700 1 -- 172.21.15.32:0/319512155 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19801c630 con 0x7fc1a4102720 2024-04-06T14:36:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.905+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a4119ff0 con 0x7fc1a4102720 2024-04-06T14:36:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.906+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a40ff090 con 0x7fc1a4102720 2024-04-06T14:36:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.907+0000 7fc192ffd700 1 -- 172.21.15.32:0/319512155 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc198013a00 con 0x7fc1a4102720 2024-04-06T14:36:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.907+0000 7fc192ffd700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc18c06cc20 0x7fc18c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.907+0000 7fc192ffd700 1 -- 172.21.15.32:0/319512155 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc19808d140 con 0x7fc1a4102720 2024-04-06T14:36:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.907+0000 7fc1a259c700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc18c06cc20 0x7fc18c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.908+0000 7fc1a259c700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc18c06cc20 0x7fc18c06f0d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc194007b20 tx=0x7fc194009e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:21.912+0000 7fc192ffd700 1 -- 172.21.15.32:0/319512155 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc19805ac10 con 0x7fc1a4102720 2024-04-06T14:36:21.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:21 smithi032 ceph-mon[147414]: from='client.34924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:21.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:21 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/714608279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:36:22.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.133+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1a4108d20 con 0x7fc18c06cc20 2024-04-06T14:36:22.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.134+0000 7fc192ffd700 1 -- 172.21.15.32:0/319512155 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc1a4108d20 con 0x7fc18c06cc20 2024-04-06T14:36:22.130 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:36:22.131 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:36:22.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc18c06cc20 msgr2=0x7fc18c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:22.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc18c06cc20 0x7fc18c06f0d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc194007b20 tx=0x7fc194009e20 comp rx=0 tx=0).stop 2024-04-06T14:36:22.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 msgr2=0x7fc1a410cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 0x7fc1a410cd00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc198009390 tx=0x7fc198009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 shutdown_connections 2024-04-06T14:36:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc18c06cc20 0x7fc18c06f0d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a4101d80 0x7fc1a410c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 --2- 172.21.15.32:0/319512155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1a4102720 0x7fc1a410cd00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 >> 172.21.15.32:0/319512155 conn(0x7fc1a40f78f0 msgr2=0x7fc1a4106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 shutdown_connections 2024-04-06T14:36:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.136+0000 7fc1a8d2f700 1 -- 172.21.15.32:0/319512155 wait complete. 2024-04-06T14:36:22.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.239+0000 7f8ab4215700 1 -- 172.21.15.32:0/379066747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 msgr2=0x7f8aac0fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:22.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.239+0000 7f8ab4215700 1 --2- 172.21.15.32:0/379066747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 0x7f8aac0fc050 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8004940 tx=0x7f8aa8009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:22.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.239+0000 7f8ab4215700 1 -- 172.21.15.32:0/379066747 shutdown_connections 2024-04-06T14:36:22.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.239+0000 7f8ab4215700 1 --2- 172.21.15.32:0/379066747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aac0fc590 0x7f8aac1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:22.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.239+0000 7f8ab4215700 1 --2- 172.21.15.32:0/379066747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 0x7f8aac0fc050 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:22.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.239+0000 7f8ab4215700 1 -- 172.21.15.32:0/379066747 >> 172.21.15.32:0/379066747 conn(0x7f8aac072bc0 msgr2=0x7f8aac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:22.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.239+0000 7f8ab4215700 1 -- 172.21.15.32:0/379066747 shutdown_connections 2024-04-06T14:36:22.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.239+0000 7f8ab4215700 1 -- 172.21.15.32:0/379066747 wait complete. 2024-04-06T14:36:22.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab4215700 1 Processor -- start 2024-04-06T14:36:22.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab4215700 1 -- start start 2024-04-06T14:36:22.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab4215700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 0x7f8aac115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab4215700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aac0fc590 0x7f8aac116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab4215700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac1168c0 con 0x7f8aac0fbc80 2024-04-06T14:36:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab4215700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac10fc90 con 0x7f8aac0fc590 2024-04-06T14:36:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab1fb1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 0x7f8aac115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab1fb1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 0x7f8aac115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34810/0 (socket says 172.21.15.32:34810) 2024-04-06T14:36:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab1fb1700 1 -- 172.21.15.32:0/4223512390 learned_addr learned my addr 172.21.15.32:0/4223512390 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab17b0700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aac0fc590 0x7f8aac116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab1fb1700 1 -- 172.21.15.32:0/4223512390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aac0fc590 msgr2=0x7f8aac116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab1fb1700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aac0fc590 0x7f8aac116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab1fb1700 1 -- 172.21.15.32:0/4223512390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8005040 con 0x7f8aac0fbc80 2024-04-06T14:36:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.240+0000 7f8ab17b0700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aac0fc590 0x7f8aac116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:36:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.241+0000 7f8ab1fb1700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 0x7f8aac115d30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8009360 tx=0x7f8aa8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.241+0000 7f8aa27fc700 1 -- 172.21.15.32:0/4223512390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8aa800e040 con 0x7f8aac0fbc80 2024-04-06T14:36:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.241+0000 7f8aa27fc700 1 -- 172.21.15.32:0/4223512390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8aa8013420 con 0x7f8aac0fbc80 2024-04-06T14:36:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.241+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac10ff10 con 0x7f8aac0fbc80 2024-04-06T14:36:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.241+0000 7f8aa27fc700 1 -- 172.21.15.32:0/4223512390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8aa801c5f0 con 0x7f8aac0fbc80 2024-04-06T14:36:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.241+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac110450 con 0x7f8aac0fbc80 2024-04-06T14:36:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.242+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac065e30 con 0x7f8aac0fbc80 2024-04-06T14:36:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.242+0000 7f8aa27fc700 1 -- 172.21.15.32:0/4223512390 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8aa8013590 con 0x7f8aac0fbc80 2024-04-06T14:36:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.243+0000 7f8aa27fc700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a9806cc30 0x7f8a9806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.243+0000 7f8aa27fc700 1 -- 172.21.15.32:0/4223512390 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8aa808d2d0 con 0x7f8aac0fbc80 2024-04-06T14:36:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.243+0000 7f8ab17b0700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a9806cc30 0x7f8a9806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.243+0000 7f8ab17b0700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a9806cc30 0x7f8a9806f0e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c007fd0 tx=0x7f8a9c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.247+0000 7f8aa27fc700 1 -- 172.21.15.32:0/4223512390 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8aa805ada0 con 0x7f8aac0fbc80 2024-04-06T14:36:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.535+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8aac05f5f0 con 0x7f8aac0fbc80 2024-04-06T14:36:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.536+0000 7f8aa27fc700 1 -- 172.21.15.32:0/4223512390 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f8aa805a930 con 0x7f8aac0fbc80 2024-04-06T14:36:22.533 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:36:22.533 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:36:22.533 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:36:22.533 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:36:22.533 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:36:22.533 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:36:22.533 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:36:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a9806cc30 msgr2=0x7f8a9806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a9806cc30 0x7f8a9806f0e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c007fd0 tx=0x7f8a9c009040 comp rx=0 tx=0).stop 2024-04-06T14:36:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 msgr2=0x7f8aac115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 0x7f8aac115d30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8009360 tx=0x7f8aa8009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 shutdown_connections 2024-04-06T14:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8a9806cc30 0x7f8a9806f0e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aac0fbc80 0x7f8aac115d30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 --2- 172.21.15.32:0/4223512390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aac0fc590 0x7f8aac116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 >> 172.21.15.32:0/4223512390 conn(0x7f8aac072bc0 msgr2=0x7f8aac105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 shutdown_connections 2024-04-06T14:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:22.538+0000 7f8ab4215700 1 -- 172.21.15.32:0/4223512390 wait complete. 2024-04-06T14:36:22.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:22 smithi086 ceph-mon[110975]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:22.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:22 smithi086 ceph-mon[110975]: pgmap v486: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:22.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:22 smithi086 ceph-mon[110975]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:22.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:22 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4124205311' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:36:22.875 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:22 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4223512390' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:36:22.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:22 smithi032 ceph-mon[147414]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:22.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:22 smithi032 ceph-mon[147414]: pgmap v486: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:22.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:22 smithi032 ceph-mon[147414]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:22.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:22 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4124205311' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:36:22.977 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:22 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4223512390' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:36:23.726 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:23 smithi032 ceph-mon[147414]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:23.726 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:36:23.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:23 smithi086 ceph-mon[110975]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:23.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:36:24.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:24 smithi086 ceph-mon[110975]: pgmap v487: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:24.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:24 smithi032 ceph-mon[147414]: pgmap v487: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:26.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:26 smithi086 ceph-mon[110975]: pgmap v488: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:26.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:26 smithi032 ceph-mon[147414]: pgmap v488: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:28.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:28 smithi086 ceph-mon[110975]: pgmap v489: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:28.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:28 smithi032 ceph-mon[147414]: pgmap v489: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:30.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:30 smithi086 ceph-mon[110975]: pgmap v490: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:30.980 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:30 smithi032 ceph-mon[147414]: pgmap v490: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:32.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:32 smithi086 ceph-mon[110975]: pgmap v491: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:32.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:32 smithi032 ceph-mon[147414]: pgmap v491: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:34.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:34 smithi086 ceph-mon[110975]: pgmap v492: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:34.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:34 smithi032 ceph-mon[147414]: pgmap v492: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:36.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:36 smithi086 ceph-mon[110975]: pgmap v493: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:36.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:36 smithi032 ceph-mon[147414]: pgmap v493: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:38.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:38 smithi086 ceph-mon[110975]: pgmap v494: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:38.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:36:38.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:38 smithi032 ceph-mon[147414]: pgmap v494: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:38.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:36:40.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:40 smithi032 ceph-mon[147414]: pgmap v495: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:40.624 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:40 smithi086 ceph-mon[110975]: pgmap v495: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:42.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:42 smithi032 ceph-mon[147414]: pgmap v496: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:42.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:42 smithi086 ceph-mon[110975]: pgmap v496: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:44.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:44 smithi032 ceph-mon[147414]: pgmap v497: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:44.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:44 smithi086 ceph-mon[110975]: pgmap v497: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:46.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:46 smithi032 ceph-mon[147414]: pgmap v498: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:46.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:46 smithi086 ceph-mon[110975]: pgmap v498: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:48.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:48 smithi032 ceph-mon[147414]: pgmap v499: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:48.624 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:48 smithi086 ceph-mon[110975]: pgmap v499: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:50.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:50 smithi032 ceph-mon[147414]: pgmap v500: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:50.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:50 smithi086 ceph-mon[110975]: pgmap v500: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:52.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:52 smithi032 ceph-mon[147414]: pgmap v501: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:52.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:52 smithi086 ceph-mon[110975]: pgmap v501: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:52.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.656+0000 7faa61ae9700 1 -- 172.21.15.32:0/1118782295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c102e70 msgr2=0x7faa5c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:52.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.656+0000 7faa61ae9700 1 --2- 172.21.15.32:0/1118782295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c102e70 0x7faa5c103240 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7faa44004940 tx=0x7faa44009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:52.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.656+0000 7faa61ae9700 1 -- 172.21.15.32:0/1118782295 shutdown_connections 2024-04-06T14:36:52.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.656+0000 7faa61ae9700 1 --2- 172.21.15.32:0/1118782295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa5c0fb4a0 0x7faa5c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:52.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.656+0000 7faa61ae9700 1 --2- 172.21.15.32:0/1118782295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c102e70 0x7faa5c103240 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:52.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.656+0000 7faa61ae9700 1 -- 172.21.15.32:0/1118782295 >> 172.21.15.32:0/1118782295 conn(0x7faa5c0714c0 msgr2=0x7faa5c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:52.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.656+0000 7faa61ae9700 1 -- 172.21.15.32:0/1118782295 shutdown_connections 2024-04-06T14:36:52.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.656+0000 7faa61ae9700 1 -- 172.21.15.32:0/1118782295 wait complete. 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa61ae9700 1 Processor -- start 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa61ae9700 1 -- start start 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa61ae9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c0fb4a0 0x7faa5c0fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa61ae9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa5c102e70 0x7faa5c0fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa61ae9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c0fd100 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa61ae9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c0fd240 con 0x7faa5c102e70 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa5b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c0fb4a0 0x7faa5c0fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa5b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c0fb4a0 0x7faa5c0fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56156/0 (socket says 172.21.15.32:56156) 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa5b7fe700 1 -- 172.21.15.32:0/1951977511 learned_addr learned my addr 172.21.15.32:0/1951977511 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.657+0000 7faa5affd700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa5c102e70 0x7faa5c0fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.658+0000 7faa5b7fe700 1 -- 172.21.15.32:0/1951977511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa5c102e70 msgr2=0x7faa5c0fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.658+0000 7faa5b7fe700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa5c102e70 0x7faa5c0fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.658+0000 7faa5b7fe700 1 -- 172.21.15.32:0/1951977511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa44005040 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.658+0000 7faa5b7fe700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c0fb4a0 0x7faa5c0fc680 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faa44000c00 tx=0x7faa44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.658+0000 7faa53fff700 1 -- 172.21.15.32:0/1951977511 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa44021070 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.658+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5c0fd490 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.658+0000 7faa53fff700 1 -- 172.21.15.32:0/1951977511 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa44013de0 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.658+0000 7faa53fff700 1 -- 172.21.15.32:0/1951977511 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa4400e070 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.658+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5c11a090 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.659+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5c100110 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.660+0000 7faa53fff700 1 -- 172.21.15.32:0/1951977511 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faa4401c600 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.660+0000 7faa53fff700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa4806cc20 0x7faa4806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:52.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.660+0000 7faa53fff700 1 -- 172.21.15.32:0/1951977511 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faa4408d130 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.660+0000 7faa5affd700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa4806cc20 0x7faa4806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:52.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.661+0000 7faa5affd700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa4806cc20 0x7faa4806f0d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa4c004600 tx=0x7faa4c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:52.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.665+0000 7faa53fff700 1 -- 172.21.15.32:0/1951977511 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faa44057170 con 0x7faa5c0fb4a0 2024-04-06T14:36:52.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.887+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faa5c04bed0 con 0x7faa4806cc20 2024-04-06T14:36:52.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.888+0000 7faa53fff700 1 -- 172.21.15.32:0/1951977511 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7faa5c04bed0 con 0x7faa4806cc20 2024-04-06T14:36:52.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa4806cc20 msgr2=0x7faa4806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:52.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa4806cc20 0x7faa4806f0d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa4c004600 tx=0x7faa4c00b040 comp rx=0 tx=0).stop 2024-04-06T14:36:52.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c0fb4a0 msgr2=0x7faa5c0fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:52.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c0fb4a0 0x7faa5c0fc680 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faa44000c00 tx=0x7faa44009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:52.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 shutdown_connections 2024-04-06T14:36:52.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa4806cc20 0x7faa4806f0d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:52.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa5c0fb4a0 0x7faa5c0fc680 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:52.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 --2- 172.21.15.32:0/1951977511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa5c102e70 0x7faa5c0fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:52.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 >> 172.21.15.32:0/1951977511 conn(0x7faa5c0714c0 msgr2=0x7faa5c0fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:52.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 shutdown_connections 2024-04-06T14:36:52.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.890+0000 7faa61ae9700 1 -- 172.21.15.32:0/1951977511 wait complete. 2024-04-06T14:36:52.896 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:36:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.994+0000 7f5fb9a94700 1 -- 172.21.15.32:0/4139623959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb4102e70 msgr2=0x7f5fb4103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.994+0000 7f5fb9a94700 1 --2- 172.21.15.32:0/4139623959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb4102e70 0x7f5fb4103240 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4004940 tx=0x7f5fa4009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.994+0000 7f5fb9a94700 1 -- 172.21.15.32:0/4139623959 shutdown_connections 2024-04-06T14:36:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.994+0000 7f5fb9a94700 1 --2- 172.21.15.32:0/4139623959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb40fb4a0 0x7f5fb40fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.994+0000 7f5fb9a94700 1 --2- 172.21.15.32:0/4139623959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb4102e70 0x7f5fb4103240 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.994+0000 7f5fb9a94700 1 -- 172.21.15.32:0/4139623959 >> 172.21.15.32:0/4139623959 conn(0x7f5fb40714c0 msgr2=0x7f5fb40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.995+0000 7f5fb9a94700 1 -- 172.21.15.32:0/4139623959 shutdown_connections 2024-04-06T14:36:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.995+0000 7f5fb9a94700 1 -- 172.21.15.32:0/4139623959 wait complete. 2024-04-06T14:36:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.995+0000 7f5fb9a94700 1 Processor -- start 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.995+0000 7f5fb9a94700 1 -- start start 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.995+0000 7f5fb9a94700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb40fb4a0 0x7f5fb4113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.995+0000 7f5fb9a94700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb4102e70 0x7f5fb4113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.995+0000 7f5fb9a94700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb4114600 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.995+0000 7f5fb9a94700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb410d9f0 con 0x7f5fb4102e70 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb40fb4a0 0x7f5fb4113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb40fb4a0 0x7f5fb4113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56160/0 (socket says 172.21.15.32:56160) 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb37fe700 1 -- 172.21.15.32:0/1224384158 learned_addr learned my addr 172.21.15.32:0/1224384158 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb2ffd700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb4102e70 0x7f5fb4113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb37fe700 1 -- 172.21.15.32:0/1224384158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb4102e70 msgr2=0x7f5fb4113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb37fe700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb4102e70 0x7f5fb4113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb37fe700 1 -- 172.21.15.32:0/1224384158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa4005040 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb37fe700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb40fb4a0 0x7f5fb4113a70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009180 tx=0x7f5fa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5f9bfff700 1 -- 172.21.15.32:0/1224384158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fa400e040 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb410dc70 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5f9bfff700 1 -- 172.21.15.32:0/1224384158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fa4004290 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5f9bfff700 1 -- 172.21.15.32:0/1224384158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fa4013e60 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.996+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb410e230 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.997+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb4100110 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.998+0000 7f5f9bfff700 1 -- 172.21.15.32:0/1224384158 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5fa40136b0 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.998+0000 7f5f9bfff700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f9c06cbb0 0x7f5f9c06f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:52.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.998+0000 7f5f9bfff700 1 -- 172.21.15.32:0/1224384158 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5fa408d320 con 0x7f5fb40fb4a0 2024-04-06T14:36:52.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.998+0000 7f5fb2ffd700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f9c06cbb0 0x7f5f9c06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:52.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:52.999+0000 7f5fb2ffd700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f9c06cbb0 0x7f5f9c06f060 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5fb410ebb0 tx=0x7f5fa8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:52.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.003+0000 7f5f9bfff700 1 -- 172.21.15.32:0/1224384158 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5fa405ae20 con 0x7f5fb40fb4a0 2024-04-06T14:36:53.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.229+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fb410f130 con 0x7f5f9c06cbb0 2024-04-06T14:36:53.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.230+0000 7f5f9bfff700 1 -- 172.21.15.32:0/1224384158 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5fb410f130 con 0x7f5f9c06cbb0 2024-04-06T14:36:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f9c06cbb0 msgr2=0x7f5f9c06f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f9c06cbb0 0x7f5f9c06f060 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5fb410ebb0 tx=0x7f5fa8009040 comp rx=0 tx=0).stop 2024-04-06T14:36:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb40fb4a0 msgr2=0x7f5fb4113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb40fb4a0 0x7f5fb4113a70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009180 tx=0x7f5fa4009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 shutdown_connections 2024-04-06T14:36:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f9c06cbb0 0x7f5f9c06f060 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fb40fb4a0 0x7f5fb4113a70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 --2- 172.21.15.32:0/1224384158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb4102e70 0x7f5fb4113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 >> 172.21.15.32:0/1224384158 conn(0x7f5fb40714c0 msgr2=0x7f5fb40fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:53.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 shutdown_connections 2024-04-06T14:36:53.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.232+0000 7f5fb9a94700 1 -- 172.21.15.32:0/1224384158 wait complete. 2024-04-06T14:36:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.336+0000 7f1271d6a700 1 -- 172.21.15.32:0/3747528066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 msgr2=0x7f126c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.336+0000 7f1271d6a700 1 --2- 172.21.15.32:0/3747528066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 0x7f126c1091b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1260004940 tx=0x7f1260009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.336+0000 7f1271d6a700 1 -- 172.21.15.32:0/3747528066 shutdown_connections 2024-04-06T14:36:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.336+0000 7f1271d6a700 1 --2- 172.21.15.32:0/3747528066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 0x7f126c1091b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.336+0000 7f1271d6a700 1 --2- 172.21.15.32:0/3747528066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c0702e0 0x7f126c0706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.336+0000 7f1271d6a700 1 -- 172.21.15.32:0/3747528066 >> 172.21.15.32:0/3747528066 conn(0x7f126c0f8970 msgr2=0x7f126c0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.336+0000 7f1271d6a700 1 -- 172.21.15.32:0/3747528066 shutdown_connections 2024-04-06T14:36:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.336+0000 7f1271d6a700 1 -- 172.21.15.32:0/3747528066 wait complete. 2024-04-06T14:36:53.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.337+0000 7f1271d6a700 1 Processor -- start 2024-04-06T14:36:53.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.337+0000 7f1271d6a700 1 -- start start 2024-04-06T14:36:53.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.337+0000 7f1271d6a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c0702e0 0x7f126c115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:53.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.337+0000 7f1271d6a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 0x7f126c116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:53.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.337+0000 7f1271d6a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c1168c0 con 0x7f126c070bf0 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.337+0000 7f1271d6a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c10fc90 con 0x7f126c0702e0 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.337+0000 7f126affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 0x7f126c116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.337+0000 7f126affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 0x7f126c116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56180/0 (socket says 172.21.15.32:56180) 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.337+0000 7f126affd700 1 -- 172.21.15.32:0/882143152 learned_addr learned my addr 172.21.15.32:0/882143152 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f126b7fe700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c0702e0 0x7f126c115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f126affd700 1 -- 172.21.15.32:0/882143152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c0702e0 msgr2=0x7f126c115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f126affd700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c0702e0 0x7f126c115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f126affd700 1 -- 172.21.15.32:0/882143152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1260005040 con 0x7f126c070bf0 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f126b7fe700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c0702e0 0x7f126c115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f126affd700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 0x7f126c116270 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f126000b3b0 tx=0x7f1260009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f1253fff700 1 -- 172.21.15.32:0/882143152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f126000e040 con 0x7f126c070bf0 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126c10ff10 con 0x7f126c070bf0 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f1253fff700 1 -- 172.21.15.32:0/882143152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1260013420 con 0x7f126c070bf0 2024-04-06T14:36:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f1253fff700 1 -- 172.21.15.32:0/882143152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f126001c630 con 0x7f126c070bf0 2024-04-06T14:36:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.338+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126c110450 con 0x7f126c070bf0 2024-04-06T14:36:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.339+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c071a20 con 0x7f126c070bf0 2024-04-06T14:36:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.339+0000 7f1253fff700 1 -- 172.21.15.32:0/882143152 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1260013970 con 0x7f126c070bf0 2024-04-06T14:36:53.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.340+0000 7f1253fff700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f125406cc20 0x7f125406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:53.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.340+0000 7f1253fff700 1 -- 172.21.15.32:0/882143152 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f126008d2b0 con 0x7f126c070bf0 2024-04-06T14:36:53.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.340+0000 7f126b7fe700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f125406cc20 0x7f125406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:53.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.340+0000 7f126b7fe700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f125406cc20 0x7f125406f0d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f125c007fd0 tx=0x7f125c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:53.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.344+0000 7f1253fff700 1 -- 172.21.15.32:0/882143152 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f126005ad80 con 0x7f126c070bf0 2024-04-06T14:36:53.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:36:53.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.559+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f126c10bfc0 con 0x7f125406cc20 2024-04-06T14:36:53.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.569+0000 7f1253fff700 1 -- 172.21.15.32:0/882143152 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f126c10bfc0 con 0x7f125406cc20 2024-04-06T14:36:53.565 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:36:53.565 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (17m) 10m ago 28m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:36:53.565 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (14m) 10m ago 28m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:36:53.565 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (14m) 10m ago 27m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (16m) 10m ago 28m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (10m) 10m ago 22m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (10m) 10m ago 22m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (10m) 10m ago 22m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (22m) 10m ago 22m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (19m) 10m ago 29m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (18m) 10m ago 27m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (15m) 10m ago 30m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (15m) 10m ago 27m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (17m) 10m ago 28m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (17m) 10m ago 27m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (14m) 10m ago 26m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:36:53.566 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (13m) 10m ago 26m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:36:53.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (12m) 10m ago 25m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:36:53.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (12m) 10m ago 25m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:36:53.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (11m) 10m ago 24m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:36:53.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (11m) 10m ago 24m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:36:53.567 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (17m) 10m ago 28m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:36:53.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f125406cc20 msgr2=0x7f125406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f125406cc20 0x7f125406f0d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f125c007fd0 tx=0x7f125c009040 comp rx=0 tx=0).stop 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 msgr2=0x7f126c116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 0x7f126c116270 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f126000b3b0 tx=0x7f1260009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 shutdown_connections 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f125406cc20 0x7f125406f0d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c0702e0 0x7f126c115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 --2- 172.21.15.32:0/882143152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126c070bf0 0x7f126c116270 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 >> 172.21.15.32:0/882143152 conn(0x7f126c0f8970 msgr2=0x7f126c105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 shutdown_connections 2024-04-06T14:36:53.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.571+0000 7f1271d6a700 1 -- 172.21.15.32:0/882143152 wait complete. 2024-04-06T14:36:53.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:36:53.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 -- 172.21.15.32:0/360428078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 msgr2=0x7fe1400fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 --2- 172.21.15.32:0/360428078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 0x7fe1400fce10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe128004940 tx=0x7fe128009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:53.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 -- 172.21.15.32:0/360428078 shutdown_connections 2024-04-06T14:36:53.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 --2- 172.21.15.32:0/360428078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 0x7fe1400fce10 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 --2- 172.21.15.32:0/360428078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe140102e00 0x7fe1401031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 -- 172.21.15.32:0/360428078 >> 172.21.15.32:0/360428078 conn(0x7fe1400f89b0 msgr2=0x7fe1400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:53.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 -- 172.21.15.32:0/360428078 shutdown_connections 2024-04-06T14:36:53.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 -- 172.21.15.32:0/360428078 wait complete. 2024-04-06T14:36:53.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 Processor -- start 2024-04-06T14:36:53.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.674+0000 7fe145265700 1 -- start start 2024-04-06T14:36:53.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe145265700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe140102e00 0x7fe1400702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe145265700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 0x7fe140070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe145265700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe140072210 con 0x7fe1401037a0 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe145265700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe140070d40 con 0x7fe140102e00 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe1367fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 0x7fe140070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe1367fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 0x7fe140070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56204/0 (socket says 172.21.15.32:56204) 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe1367fc700 1 -- 172.21.15.32:0/2683755951 learned_addr learned my addr 172.21.15.32:0/2683755951 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe1367fc700 1 -- 172.21.15.32:0/2683755951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe140102e00 msgr2=0x7fe1400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe13effd700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe140102e00 0x7fe1400702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe1367fc700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe140102e00 0x7fe1400702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe1367fc700 1 -- 172.21.15.32:0/2683755951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe128005040 con 0x7fe1401037a0 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe13effd700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe140102e00 0x7fe1400702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe1367fc700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 0x7fe140070800 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe128000c00 tx=0x7fe128009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.675+0000 7fe137fff700 1 -- 172.21.15.32:0/2683755951 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe12800e070 con 0x7fe1401037a0 2024-04-06T14:36:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.676+0000 7fe137fff700 1 -- 172.21.15.32:0/2683755951 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe128013de0 con 0x7fe1401037a0 2024-04-06T14:36:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.676+0000 7fe145265700 1 -- 172.21.15.32:0/2683755951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe140070fc0 con 0x7fe1401037a0 2024-04-06T14:36:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.676+0000 7fe137fff700 1 -- 172.21.15.32:0/2683755951 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe12801cde0 con 0x7fe1401037a0 2024-04-06T14:36:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.676+0000 7fe145265700 1 -- 172.21.15.32:0/2683755951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe140119fa0 con 0x7fe1401037a0 2024-04-06T14:36:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.676+0000 7fe1357fa700 1 -- 172.21.15.32:0/2683755951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1240052b0 con 0x7fe1401037a0 2024-04-06T14:36:53.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.677+0000 7fe137fff700 1 -- 172.21.15.32:0/2683755951 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe12801c600 con 0x7fe1401037a0 2024-04-06T14:36:53.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.677+0000 7fe137fff700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe12c06cc20 0x7fe12c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:53.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.678+0000 7fe137fff700 1 -- 172.21.15.32:0/2683755951 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe12808d4f0 con 0x7fe1401037a0 2024-04-06T14:36:53.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.678+0000 7fe13effd700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe12c06cc20 0x7fe12c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:53.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.678+0000 7fe13effd700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe12c06cc20 0x7fe12c06f0d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe130004600 tx=0x7fe13000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:53.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.682+0000 7fe137fff700 1 -- 172.21.15.32:0/2683755951 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe128057530 con 0x7fe1401037a0 2024-04-06T14:36:53.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.975+0000 7fe1357fa700 1 -- 172.21.15.32:0/2683755951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe124005d70 con 0x7fe1401037a0 2024-04-06T14:36:53.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.975+0000 7fe137fff700 1 -- 172.21.15.32:0/2683755951 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fe12805ab50 con 0x7fe1401037a0 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:36:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:36:53.973 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:36:53.973 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:36:53.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:36:53.973 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:36:53.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.977+0000 7fe1357fa700 1 -- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe12c06cc20 msgr2=0x7fe12c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.977+0000 7fe1357fa700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe12c06cc20 0x7fe12c06f0d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe130004600 tx=0x7fe13000b040 comp rx=0 tx=0).stop 2024-04-06T14:36:53.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.977+0000 7fe1357fa700 1 -- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 msgr2=0x7fe140070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.977+0000 7fe1357fa700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 0x7fe140070800 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe128000c00 tx=0x7fe128009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.977+0000 7fe1357fa700 1 -- 172.21.15.32:0/2683755951 shutdown_connections 2024-04-06T14:36:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.977+0000 7fe1357fa700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe12c06cc20 0x7fe12c06f0d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.977+0000 7fe1357fa700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe140102e00 0x7fe1400702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.977+0000 7fe1357fa700 1 --2- 172.21.15.32:0/2683755951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1401037a0 0x7fe140070800 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.977+0000 7fe1357fa700 1 -- 172.21.15.32:0/2683755951 >> 172.21.15.32:0/2683755951 conn(0x7fe1400f89b0 msgr2=0x7fe1400f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.978+0000 7fe1357fa700 1 -- 172.21.15.32:0/2683755951 shutdown_connections 2024-04-06T14:36:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:53.978+0000 7fe1357fa700 1 -- 172.21.15.32:0/2683755951 wait complete. 2024-04-06T14:36:54.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 -- 172.21.15.32:0/4067225588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0102710 msgr2=0x7f0fc00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 --2- 172.21.15.32:0/4067225588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0102710 0x7f0fc00fbd80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0004940 tx=0x7f0fb0009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:54.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 -- 172.21.15.32:0/4067225588 shutdown_connections 2024-04-06T14:36:54.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 --2- 172.21.15.32:0/4067225588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0102710 0x7f0fc00fbd80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 --2- 172.21.15.32:0/4067225588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fc0101d70 0x7f0fc0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 -- 172.21.15.32:0/4067225588 >> 172.21.15.32:0/4067225588 conn(0x7f0fc0072bc0 msgr2=0x7f0fc0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 -- 172.21.15.32:0/4067225588 shutdown_connections 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 -- 172.21.15.32:0/4067225588 wait complete. 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 Processor -- start 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.081+0000 7f0fc827a700 1 -- start start 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc827a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0101d70 0x7f0fc0113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc827a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fc0102710 0x7f0fc0114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc827a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc0114650 con 0x7f0fc0101d70 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc827a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc010da40 con 0x7f0fc0102710 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc6016700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0101d70 0x7f0fc0113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc6016700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0101d70 0x7f0fc0113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56216/0 (socket says 172.21.15.32:56216) 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc6016700 1 -- 172.21.15.32:0/3278227500 learned_addr learned my addr 172.21.15.32:0/3278227500 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc5815700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fc0102710 0x7f0fc0114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:54.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc6016700 1 -- 172.21.15.32:0/3278227500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fc0102710 msgr2=0x7f0fc0114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc6016700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fc0102710 0x7f0fc0114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc6016700 1 -- 172.21.15.32:0/3278227500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb0005040 con 0x7f0fc0101d70 2024-04-06T14:36:54.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc6016700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0101d70 0x7f0fc0113ac0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc002a50 tx=0x7f0fbc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:54.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fb6ffd700 1 -- 172.21.15.32:0/3278227500 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fbc007cc0 con 0x7f0fc0101d70 2024-04-06T14:36:54.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fb6ffd700 1 -- 172.21.15.32:0/3278227500 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fbc00f040 con 0x7f0fc0101d70 2024-04-06T14:36:54.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fb6ffd700 1 -- 172.21.15.32:0/3278227500 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fbc00d9c0 con 0x7f0fc0101d70 2024-04-06T14:36:54.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc010dd20 con 0x7f0fc0101d70 2024-04-06T14:36:54.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.082+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc010e2b0 con 0x7f0fc0101d70 2024-04-06T14:36:54.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.083+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc00fd4c0 con 0x7f0fc0101d70 2024-04-06T14:36:54.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.084+0000 7f0fb6ffd700 1 -- 172.21.15.32:0/3278227500 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0fbc00db20 con 0x7f0fc0101d70 2024-04-06T14:36:54.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.084+0000 7f0fb6ffd700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0fac06cc20 0x7f0fac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:54.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.084+0000 7f0fb6ffd700 1 -- 172.21.15.32:0/3278227500 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0fbc08bcb0 con 0x7f0fc0101d70 2024-04-06T14:36:54.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.084+0000 7f0fc5815700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0fac06cc20 0x7f0fac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:54.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.085+0000 7f0fc5815700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0fac06cc20 0x7f0fac06f0d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0004890 tx=0x7f0fb0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:54.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.089+0000 7f0fb6ffd700 1 -- 172.21.15.32:0/3278227500 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0fbc059780 con 0x7f0fc0101d70 2024-04-06T14:36:54.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.335+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0fc0108d60 con 0x7f0fc0101d70 2024-04-06T14:36:54.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.335+0000 7f0fb6ffd700 1 -- 172.21.15.32:0/3278227500 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f0fbc059310 con 0x7f0fc0101d70 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:36:54.332 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:36:54.333 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:36:54.334 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:36:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.337+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0fac06cc20 msgr2=0x7f0fac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.337+0000 7f0fc827a700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0fac06cc20 0x7f0fac06f0d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0004890 tx=0x7f0fb0009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.337+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0101d70 msgr2=0x7f0fc0113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.337+0000 7f0fc827a700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0101d70 0x7f0fc0113ac0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc002a50 tx=0x7f0fbc0046b0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.337+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 shutdown_connections 2024-04-06T14:36:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.337+0000 7f0fc827a700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0fac06cc20 0x7f0fac06f0d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.337+0000 7f0fc827a700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fc0101d70 0x7f0fc0113ac0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.337+0000 7f0fc827a700 1 --2- 172.21.15.32:0/3278227500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fc0102710 0x7f0fc0114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.337+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 >> 172.21.15.32:0/3278227500 conn(0x7f0fc0072bc0 msgr2=0x7f0fc00659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.338+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 shutdown_connections 2024-04-06T14:36:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.338+0000 7f0fc827a700 1 -- 172.21.15.32:0/3278227500 wait complete. 2024-04-06T14:36:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:36:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.442+0000 7fe74a444700 1 -- 172.21.15.32:0/1516768516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7441026b0 msgr2=0x7fe7440fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.442+0000 7fe74a444700 1 --2- 172.21.15.32:0/1516768516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7441026b0 0x7fe7440fbd20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe738004940 tx=0x7fe738009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.442+0000 7fe74a444700 1 -- 172.21.15.32:0/1516768516 shutdown_connections 2024-04-06T14:36:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.442+0000 7fe74a444700 1 --2- 172.21.15.32:0/1516768516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7441026b0 0x7fe7440fbd20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.442+0000 7fe74a444700 1 --2- 172.21.15.32:0/1516768516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe744101d10 0x7fe7441020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.442+0000 7fe74a444700 1 -- 172.21.15.32:0/1516768516 >> 172.21.15.32:0/1516768516 conn(0x7fe7440f7880 msgr2=0x7fe7440f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.442+0000 7fe74a444700 1 -- 172.21.15.32:0/1516768516 shutdown_connections 2024-04-06T14:36:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.442+0000 7fe74a444700 1 -- 172.21.15.32:0/1516768516 wait complete. 2024-04-06T14:36:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.443+0000 7fe74a444700 1 Processor -- start 2024-04-06T14:36:54.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.443+0000 7fe74a444700 1 -- start start 2024-04-06T14:36:54.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.443+0000 7fe74a444700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe744101d10 0x7fe74410ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:54.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.443+0000 7fe74a444700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7441026b0 0x7fe74410f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:54.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.443+0000 7fe74a444700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74410f8a0 con 0x7fe744101d10 2024-04-06T14:36:54.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.443+0000 7fe74a444700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74410f9e0 con 0x7fe7441026b0 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.443+0000 7fe743fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe744101d10 0x7fe74410ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.443+0000 7fe743fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe744101d10 0x7fe74410ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53100/0 (socket says 172.21.15.32:53100) 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.443+0000 7fe743fff700 1 -- 172.21.15.32:0/1441840268 learned_addr learned my addr 172.21.15.32:0/1441840268 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe7437fe700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7441026b0 0x7fe74410f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe743fff700 1 -- 172.21.15.32:0/1441840268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7441026b0 msgr2=0x7fe74410f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe743fff700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7441026b0 0x7fe74410f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe743fff700 1 -- 172.21.15.32:0/1441840268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738005040 con 0x7fe744101d10 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe7437fe700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7441026b0 0x7fe74410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe743fff700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe744101d10 0x7fe74410ed10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe734002a80 tx=0x7fe7340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe740ff9700 1 -- 172.21.15.32:0/1441840268 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe734007cc0 con 0x7fe744101d10 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe744117d80 con 0x7fe744101d10 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe740ff9700 1 -- 172.21.15.32:0/1441840268 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe73400f040 con 0x7fe744101d10 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe740ff9700 1 -- 172.21.15.32:0/1441840268 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe734009640 con 0x7fe744101d10 2024-04-06T14:36:54.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.444+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7441182e0 con 0x7fe744101d10 2024-04-06T14:36:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.445+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7440fd460 con 0x7fe744101d10 2024-04-06T14:36:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.446+0000 7fe740ff9700 1 -- 172.21.15.32:0/1441840268 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe7340097a0 con 0x7fe744101d10 2024-04-06T14:36:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.446+0000 7fe740ff9700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe72c06cc20 0x7fe72c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.446+0000 7fe740ff9700 1 -- 172.21.15.32:0/1441840268 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe73408ba10 con 0x7fe744101d10 2024-04-06T14:36:54.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.446+0000 7fe7437fe700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe72c06cc20 0x7fe72c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:54.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.447+0000 7fe7437fe700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe72c06cc20 0x7fe72c06f0d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe73800b3b0 tx=0x7fe738009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.450+0000 7fe740ff9700 1 -- 172.21.15.32:0/1441840268 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe734055a50 con 0x7fe744101d10 2024-04-06T14:36:54.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:54 smithi032 ceph-mon[147414]: pgmap v502: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:54.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:54 smithi032 ceph-mon[147414]: from='client.34952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:54.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:54 smithi032 ceph-mon[147414]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:54.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:54 smithi032 ceph-mon[147414]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:54.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:54 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2683755951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:36:54.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:54 smithi086 ceph-mon[110975]: pgmap v502: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:54.624 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:54 smithi086 ceph-mon[110975]: from='client.34952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:54.624 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:54 smithi086 ceph-mon[110975]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:54.624 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:54 smithi086 ceph-mon[110975]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:54.624 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:54 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2683755951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:36:54.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.669+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe744066640 con 0x7fe72c06cc20 2024-04-06T14:36:54.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.670+0000 7fe740ff9700 1 -- 172.21.15.32:0/1441840268 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe744066640 con 0x7fe72c06cc20 2024-04-06T14:36:54.667 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:36:54.667 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:36:54.667 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:36:54.668 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:36:54.668 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:36:54.668 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:36:54.668 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:36:54.668 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:36:54.668 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:36:54.669 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:36:54.669 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:36:54.669 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:36:54.669 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:36:54.669 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:36:54.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe72c06cc20 msgr2=0x7fe72c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe72c06cc20 0x7fe72c06f0d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe73800b3b0 tx=0x7fe738009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:54.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe744101d10 msgr2=0x7fe74410ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe744101d10 0x7fe74410ed10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe734002a80 tx=0x7fe7340046b0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 shutdown_connections 2024-04-06T14:36:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe72c06cc20 0x7fe72c06f0d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe744101d10 0x7fe74410ed10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 --2- 172.21.15.32:0/1441840268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7441026b0 0x7fe74410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 >> 172.21.15.32:0/1441840268 conn(0x7fe7440f7880 msgr2=0x7fe7440f88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 shutdown_connections 2024-04-06T14:36:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.673+0000 7fe74a444700 1 -- 172.21.15.32:0/1441840268 wait complete. 2024-04-06T14:36:54.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.776+0000 7feda132b700 1 -- 172.21.15.32:0/3897884152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 msgr2=0x7fed9c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.776+0000 7feda132b700 1 --2- 172.21.15.32:0/3897884152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 0x7fed9c0fce20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fed84004940 tx=0x7fed84009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:54.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.776+0000 7feda132b700 1 -- 172.21.15.32:0/3897884152 shutdown_connections 2024-04-06T14:36:54.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.776+0000 7feda132b700 1 --2- 172.21.15.32:0/3897884152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 0x7fed9c0fce20 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.776+0000 7feda132b700 1 --2- 172.21.15.32:0/3897884152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed9c102e10 0x7fed9c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.776+0000 7feda132b700 1 -- 172.21.15.32:0/3897884152 >> 172.21.15.32:0/3897884152 conn(0x7fed9c0f89a0 msgr2=0x7fed9c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:54.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.776+0000 7feda132b700 1 -- 172.21.15.32:0/3897884152 shutdown_connections 2024-04-06T14:36:54.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.776+0000 7feda132b700 1 -- 172.21.15.32:0/3897884152 wait complete. 2024-04-06T14:36:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7feda132b700 1 Processor -- start 2024-04-06T14:36:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7feda132b700 1 -- start start 2024-04-06T14:36:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7feda132b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed9c102e10 0x7fed9c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7feda132b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 0x7fed9c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7feda132b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c070d90 con 0x7fed9c1037b0 2024-04-06T14:36:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7feda132b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c070ed0 con 0x7fed9c102e10 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7fed9a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 0x7fed9c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7fed9a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 0x7fed9c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53122/0 (socket says 172.21.15.32:53122) 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7fed9a7fc700 1 -- 172.21.15.32:0/610009706 learned_addr learned my addr 172.21.15.32:0/610009706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7fed9affd700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed9c102e10 0x7fed9c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7fed9a7fc700 1 -- 172.21.15.32:0/610009706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed9c102e10 msgr2=0x7fed9c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7fed9a7fc700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed9c102e10 0x7fed9c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.777+0000 7fed9a7fc700 1 -- 172.21.15.32:0/610009706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84005040 con 0x7fed9c1037b0 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.778+0000 7fed9a7fc700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 0x7fed9c070800 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fed84009390 tx=0x7fed84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.778+0000 7fed937fe700 1 -- 172.21.15.32:0/610009706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed8400e040 con 0x7fed9c1037b0 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.778+0000 7fed937fe700 1 -- 172.21.15.32:0/610009706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed84013420 con 0x7fed9c1037b0 2024-04-06T14:36:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.778+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9c071120 con 0x7fed9c1037b0 2024-04-06T14:36:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.778+0000 7fed937fe700 1 -- 172.21.15.32:0/610009706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed8401c630 con 0x7fed9c1037b0 2024-04-06T14:36:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.778+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9c11a200 con 0x7fed9c1037b0 2024-04-06T14:36:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.779+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9c0fe560 con 0x7fed9c1037b0 2024-04-06T14:36:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.779+0000 7fed937fe700 1 -- 172.21.15.32:0/610009706 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fed840139e0 con 0x7fed9c1037b0 2024-04-06T14:36:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.780+0000 7fed937fe700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed8806cc30 0x7fed8806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:36:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.780+0000 7fed937fe700 1 -- 172.21.15.32:0/610009706 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fed8408d210 con 0x7fed9c1037b0 2024-04-06T14:36:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.780+0000 7fed9affd700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed8806cc30 0x7fed8806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:36:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.780+0000 7fed9affd700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed8806cc30 0x7fed8806f0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fed8c007fd0 tx=0x7fed8c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:36:54.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:54.784+0000 7fed937fe700 1 -- 172.21.15.32:0/610009706 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fed84057310 con 0x7fed9c1037b0 2024-04-06T14:36:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.072+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fed9c109de0 con 0x7fed9c1037b0 2024-04-06T14:36:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.072+0000 7fed937fe700 1 -- 172.21.15.32:0/610009706 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fed8405a930 con 0x7fed9c1037b0 2024-04-06T14:36:55.069 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:36:55.069 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:36:55.069 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:36:55.069 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:36:55.069 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:36:55.069 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:36:55.069 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:36:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.074+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed8806cc30 msgr2=0x7fed8806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.074+0000 7feda132b700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed8806cc30 0x7fed8806f0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fed8c007fd0 tx=0x7fed8c009040 comp rx=0 tx=0).stop 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.074+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 msgr2=0x7fed9c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.074+0000 7feda132b700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 0x7fed9c070800 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fed84009390 tx=0x7fed84009e30 comp rx=0 tx=0).stop 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.074+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 shutdown_connections 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.074+0000 7feda132b700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed8806cc30 0x7fed8806f0e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.074+0000 7feda132b700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed9c102e10 0x7fed9c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.074+0000 7feda132b700 1 --2- 172.21.15.32:0/610009706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed9c1037b0 0x7fed9c070800 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.074+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 >> 172.21.15.32:0/610009706 conn(0x7fed9c0f89a0 msgr2=0x7fed9c0f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.075+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 shutdown_connections 2024-04-06T14:36:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:36:55.075+0000 7feda132b700 1 -- 172.21.15.32:0/610009706 wait complete. 2024-04-06T14:36:55.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:55 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3278227500' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:36:55.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:55 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/610009706' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:36:55.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:55 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3278227500' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:36:55.624 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:55 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/610009706' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:36:56.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:56 smithi032 ceph-mon[147414]: from='client.34972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:56.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:56 smithi032 ceph-mon[147414]: pgmap v503: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:56.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:56 smithi086 ceph-mon[110975]: from='client.34972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:36:56.624 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:56 smithi086 ceph-mon[110975]: pgmap v503: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:58.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:36:58 smithi032 ceph-mon[147414]: pgmap v504: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:36:58.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:36:58 smithi086 ceph-mon[110975]: pgmap v504: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:00.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:00 smithi032 ceph-mon[147414]: pgmap v505: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:00.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:00 smithi086 ceph-mon[110975]: pgmap v505: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:02.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:02 smithi032 ceph-mon[147414]: pgmap v506: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:02.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:02 smithi086 ceph-mon[110975]: pgmap v506: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:04.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:04 smithi032 ceph-mon[147414]: pgmap v507: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:04.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:04 smithi086 ceph-mon[110975]: pgmap v507: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:06.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:06 smithi032 ceph-mon[147414]: pgmap v508: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:06.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:06 smithi086 ceph-mon[110975]: pgmap v508: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:08.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:08 smithi032 ceph-mon[147414]: pgmap v509: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:08.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:37:08.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:08 smithi086 ceph-mon[110975]: pgmap v509: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:08.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:37:10.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:10 smithi032 ceph-mon[147414]: pgmap v510: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:10.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:10 smithi086 ceph-mon[110975]: pgmap v510: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:12.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:12 smithi032 ceph-mon[147414]: pgmap v511: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:12.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:12 smithi086 ceph-mon[110975]: pgmap v511: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:14.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:14 smithi032 ceph-mon[147414]: pgmap v512: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:14.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:14 smithi086 ceph-mon[110975]: pgmap v512: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:16.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:16 smithi032 ceph-mon[147414]: pgmap v513: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:16.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:16 smithi086 ceph-mon[110975]: pgmap v513: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:18.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:18 smithi032 ceph-mon[147414]: pgmap v514: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:18.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:18 smithi086 ceph-mon[110975]: pgmap v514: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:20.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:20 smithi032 ceph-mon[147414]: pgmap v515: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:20.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:20 smithi086 ceph-mon[110975]: pgmap v515: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:22.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:22 smithi032 ceph-mon[147414]: pgmap v516: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:22.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:22 smithi086 ceph-mon[110975]: pgmap v516: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:23.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:37:23.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:37:24.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:24 smithi032 ceph-mon[147414]: pgmap v517: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:24.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:24 smithi086 ceph-mon[110975]: pgmap v517: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.193+0000 7f9a6fb79700 1 -- 172.21.15.32:0/915118712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 msgr2=0x7f9a680fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.193+0000 7f9a6fb79700 1 --2- 172.21.15.32:0/915118712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fb910 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9a58004940 tx=0x7f9a58009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.193+0000 7f9a6fb79700 1 -- 172.21.15.32:0/915118712 shutdown_connections 2024-04-06T14:37:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.193+0000 7f9a6fb79700 1 --2- 172.21.15.32:0/915118712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fb910 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.193+0000 7f9a6fb79700 1 --2- 172.21.15.32:0/915118712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a68103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.193+0000 7f9a6fb79700 1 -- 172.21.15.32:0/915118712 >> 172.21.15.32:0/915118712 conn(0x7f9a680714c0 msgr2=0x7f9a680718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.193+0000 7f9a6fb79700 1 -- 172.21.15.32:0/915118712 shutdown_connections 2024-04-06T14:37:25.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.193+0000 7f9a6fb79700 1 -- 172.21.15.32:0/915118712 wait complete. 2024-04-06T14:37:25.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6fb79700 1 Processor -- start 2024-04-06T14:37:25.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6fb79700 1 -- start start 2024-04-06T14:37:25.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6fb79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:25.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6fb79700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a680fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:25.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6fb79700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a680fd150 con 0x7f9a680fb4a0 2024-04-06T14:37:25.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6fb79700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a680fd290 con 0x7f9a68102e70 2024-04-06T14:37:25.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6d915700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:25.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6d915700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60892/0 (socket says 172.21.15.32:60892) 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6d915700 1 -- 172.21.15.32:0/2794987288 learned_addr learned my addr 172.21.15.32:0/2794987288 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.194+0000 7f9a6d114700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a680fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a6d915700 1 -- 172.21.15.32:0/2794987288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 msgr2=0x7f9a680fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a6d915700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a680fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a6d915700 1 -- 172.21.15.32:0/2794987288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a58005040 con 0x7f9a680fb4a0 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a6d114700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a680fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a6d915700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fc680 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9a64002a50 tx=0x7f9a640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/2794987288 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a64007cc0 con 0x7f9a680fb4a0 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a68119e50 con 0x7f9a680fb4a0 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/2794987288 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a6400f040 con 0x7f9a680fb4a0 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/2794987288 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a64009540 con 0x7f9a680fb4a0 2024-04-06T14:37:25.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.195+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6811a250 con 0x7f9a680fb4a0 2024-04-06T14:37:25.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.196+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a68100170 con 0x7f9a680fb4a0 2024-04-06T14:37:25.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.197+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/2794987288 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9a640096a0 con 0x7f9a680fb4a0 2024-04-06T14:37:25.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.197+0000 7f9a5e7fc700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:25.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.197+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/2794987288 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9a6408be60 con 0x7f9a680fb4a0 2024-04-06T14:37:25.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.197+0000 7f9a6d114700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:25.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.198+0000 7f9a6d114700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9a58004f90 tx=0x7f9a58009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:25.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.202+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/2794987288 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9a64055e20 con 0x7f9a680fb4a0 2024-04-06T14:37:25.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.423+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a6804bed0 con 0x7f9a5406cc20 2024-04-06T14:37:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.424+0000 7f9a5e7fc700 1 -- 172.21.15.32:0/2794987288 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9a6804bed0 con 0x7f9a5406cc20 2024-04-06T14:37:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 msgr2=0x7f9a5406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9a58004f90 tx=0x7f9a58009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 msgr2=0x7f9a680fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fc680 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9a64002a50 tx=0x7f9a640046b0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 shutdown_connections 2024-04-06T14:37:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a5406cc20 0x7f9a5406f0d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a680fb4a0 0x7f9a680fc680 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 --2- 172.21.15.32:0/2794987288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a68102e70 0x7f9a680fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 >> 172.21.15.32:0/2794987288 conn(0x7f9a680714c0 msgr2=0x7f9a680fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:25.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 shutdown_connections 2024-04-06T14:37:25.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.426+0000 7f9a6fb79700 1 -- 172.21.15.32:0/2794987288 wait complete. 2024-04-06T14:37:25.432 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:37:25.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.530+0000 7f00acb1c700 1 -- 172.21.15.32:0/2204900429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8102710 msgr2=0x7f00a80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.530+0000 7f00acb1c700 1 --2- 172.21.15.32:0/2204900429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8102710 0x7f00a80fbd80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0098004940 tx=0x7f0098009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.530+0000 7f00acb1c700 1 -- 172.21.15.32:0/2204900429 shutdown_connections 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.530+0000 7f00acb1c700 1 --2- 172.21.15.32:0/2204900429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8102710 0x7f00a80fbd80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.530+0000 7f00acb1c700 1 --2- 172.21.15.32:0/2204900429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00a8101d70 0x7f00a8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.530+0000 7f00acb1c700 1 -- 172.21.15.32:0/2204900429 >> 172.21.15.32:0/2204900429 conn(0x7f00a8072bc0 msgr2=0x7f00a8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.530+0000 7f00acb1c700 1 -- 172.21.15.32:0/2204900429 shutdown_connections 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.530+0000 7f00acb1c700 1 -- 172.21.15.32:0/2204900429 wait complete. 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.530+0000 7f00acb1c700 1 Processor -- start 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00acb1c700 1 -- start start 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00acb1c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8101d70 0x7f00a8102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00acb1c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00a8102710 0x7f00a8103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00acb1c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a81036a0 con 0x7f00a8101d70 2024-04-06T14:37:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00acb1c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a81037e0 con 0x7f00a8102710 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00a659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8101d70 0x7f00a8102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00a659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8101d70 0x7f00a8102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60910/0 (socket says 172.21.15.32:60910) 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00a659c700 1 -- 172.21.15.32:0/3856227840 learned_addr learned my addr 172.21.15.32:0/3856227840 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00a5d9b700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00a8102710 0x7f00a8103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00a659c700 1 -- 172.21.15.32:0/3856227840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00a8102710 msgr2=0x7f00a8103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00a659c700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00a8102710 0x7f00a8103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00a659c700 1 -- 172.21.15.32:0/3856227840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0098005040 con 0x7f00a8101d70 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f00a659c700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8101d70 0x7f00a8102bd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0090002a50 tx=0x7f00900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.531+0000 7f009effd700 1 -- 172.21.15.32:0/3856227840 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0090007cc0 con 0x7f00a8101d70 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.532+0000 7f009effd700 1 -- 172.21.15.32:0/3856227840 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f009000f040 con 0x7f00a8101d70 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.532+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a8119dc0 con 0x7f00a8101d70 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.532+0000 7f009effd700 1 -- 172.21.15.32:0/3856227840 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f009000d9c0 con 0x7f00a8101d70 2024-04-06T14:37:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.532+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a811a240 con 0x7f00a8101d70 2024-04-06T14:37:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.532+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a80fd4c0 con 0x7f00a8101d70 2024-04-06T14:37:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.533+0000 7f009effd700 1 -- 172.21.15.32:0/3856227840 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f009000db20 con 0x7f00a8101d70 2024-04-06T14:37:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.534+0000 7f009effd700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f009406cc30 0x7f009406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.534+0000 7f009effd700 1 -- 172.21.15.32:0/3856227840 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f009008c370 con 0x7f00a8101d70 2024-04-06T14:37:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.534+0000 7f00a5d9b700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f009406cc30 0x7f009406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.534+0000 7f00a5d9b700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f009406cc30 0x7f009406f0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f00a8104510 tx=0x7f0098009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:25.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.538+0000 7f009effd700 1 -- 172.21.15.32:0/3856227840 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0090056460 con 0x7f00a8101d70 2024-04-06T14:37:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.758+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f00a81042c0 con 0x7f009406cc30 2024-04-06T14:37:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.760+0000 7f009effd700 1 -- 172.21.15.32:0/3856227840 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f00a81042c0 con 0x7f009406cc30 2024-04-06T14:37:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f009406cc30 msgr2=0x7f009406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f009406cc30 0x7f009406f0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f00a8104510 tx=0x7f0098009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8101d70 msgr2=0x7f00a8102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8101d70 0x7f00a8102bd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0090002a50 tx=0x7f00900046b0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 shutdown_connections 2024-04-06T14:37:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f009406cc30 0x7f009406f0e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a8101d70 0x7f00a8102bd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 --2- 172.21.15.32:0/3856227840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00a8102710 0x7f00a8103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 >> 172.21.15.32:0/3856227840 conn(0x7f00a8072bc0 msgr2=0x7f00a8065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 shutdown_connections 2024-04-06T14:37:25.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.762+0000 7f00acb1c700 1 -- 172.21.15.32:0/3856227840 wait complete. 2024-04-06T14:37:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.865+0000 7fe32e398700 1 -- 172.21.15.32:0/4117432681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070300 msgr2=0x7fe3280706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.865+0000 7fe32e398700 1 --2- 172.21.15.32:0/4117432681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070300 0x7fe3280706d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe318004970 tx=0x7fe318009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.865+0000 7fe32e398700 1 -- 172.21.15.32:0/4117432681 shutdown_connections 2024-04-06T14:37:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.865+0000 7fe32e398700 1 --2- 172.21.15.32:0/4117432681 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328070c10 0x7fe328109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.865+0000 7fe32e398700 1 --2- 172.21.15.32:0/4117432681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070300 0x7fe3280706d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.865+0000 7fe32e398700 1 -- 172.21.15.32:0/4117432681 >> 172.21.15.32:0/4117432681 conn(0x7fe3280f89b0 msgr2=0x7fe3280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.865+0000 7fe32e398700 1 -- 172.21.15.32:0/4117432681 shutdown_connections 2024-04-06T14:37:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.865+0000 7fe32e398700 1 -- 172.21.15.32:0/4117432681 wait complete. 2024-04-06T14:37:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.865+0000 7fe32e398700 1 Processor -- start 2024-04-06T14:37:25.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe32e398700 1 -- start start 2024-04-06T14:37:25.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe32e398700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328070300 0x7fe328115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:25.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe32e398700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070c10 0x7fe328115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:25.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe32e398700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe328116600 con 0x7fe328070c10 2024-04-06T14:37:25.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe32e398700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32810fbe0 con 0x7fe328070300 2024-04-06T14:37:25.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe3277fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070c10 0x7fe328115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe3277fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070c10 0x7fe328115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60936/0 (socket says 172.21.15.32:60936) 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe3277fe700 1 -- 172.21.15.32:0/1022418265 learned_addr learned my addr 172.21.15.32:0/1022418265 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe327fff700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328070300 0x7fe328115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe3277fe700 1 -- 172.21.15.32:0/1022418265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328070300 msgr2=0x7fe328115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe3277fe700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328070300 0x7fe328115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe3277fe700 1 -- 172.21.15.32:0/1022418265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318005040 con 0x7fe328070c10 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe327fff700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328070300 0x7fe328115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.866+0000 7fe3277fe700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070c10 0x7fe328115fb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe310002a50 tx=0x7fe3100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.867+0000 7fe324ff9700 1 -- 172.21.15.32:0/1022418265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe310007cc0 con 0x7fe328070c10 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.867+0000 7fe324ff9700 1 -- 172.21.15.32:0/1022418265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe31000f040 con 0x7fe328070c10 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.867+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32810fec0 con 0x7fe328070c10 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.867+0000 7fe324ff9700 1 -- 172.21.15.32:0/1022418265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe310009540 con 0x7fe328070c10 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.867+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe328110400 con 0x7fe328070c10 2024-04-06T14:37:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.868+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe328103bf0 con 0x7fe328070c10 2024-04-06T14:37:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.868+0000 7fe324ff9700 1 -- 172.21.15.32:0/1022418265 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe3100096a0 con 0x7fe328070c10 2024-04-06T14:37:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.869+0000 7fe324ff9700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe31406cc20 0x7fe31406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.869+0000 7fe327fff700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe31406cc20 0x7fe31406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.869+0000 7fe324ff9700 1 -- 172.21.15.32:0/1022418265 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe31008bd70 con 0x7fe328070c10 2024-04-06T14:37:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.869+0000 7fe327fff700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe31406cc20 0x7fe31406f0d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe318009fd0 tx=0x7fe318009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:25.873+0000 7fe324ff9700 1 -- 172.21.15.32:0/1022418265 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe310059840 con 0x7fe328070c10 2024-04-06T14:37:26.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.084+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe328111130 con 0x7fe31406cc20 2024-04-06T14:37:26.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.094+0000 7fe324ff9700 1 -- 172.21.15.32:0/1022418265 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe328111130 con 0x7fe31406cc20 2024-04-06T14:37:26.090 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:37:26.090 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (17m) 11m ago 29m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:37:26.090 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (15m) 11m ago 29m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:37:26.090 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (15m) 10m ago 28m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:37:26.090 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (17m) 11m ago 28m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:37:26.090 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (11m) 11m ago 23m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:37:26.090 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (11m) 11m ago 23m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (10m) 10m ago 23m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (23m) 10m ago 23m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (19m) 11m ago 30m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (18m) 10m ago 28m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (16m) 11m ago 30m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (15m) 10m ago 28m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (18m) 11m ago 29m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (18m) 10m ago 28m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (14m) 11m ago 27m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (14m) 11m ago 26m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (13m) 11m ago 26m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (12m) 10m ago 25m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (12m) 10m ago 25m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:37:26.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (11m) 10m ago 24m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:37:26.092 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (18m) 11m ago 28m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:37:26.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe31406cc20 msgr2=0x7fe31406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe31406cc20 0x7fe31406f0d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe318009fd0 tx=0x7fe318009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:26.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070c10 msgr2=0x7fe328115fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070c10 0x7fe328115fb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe310002a50 tx=0x7fe3100046b0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 shutdown_connections 2024-04-06T14:37:26.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe31406cc20 0x7fe31406f0d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328070300 0x7fe328115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 --2- 172.21.15.32:0/1022418265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe328070c10 0x7fe328115fb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 >> 172.21.15.32:0/1022418265 conn(0x7fe3280f89b0 msgr2=0x7fe328101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:26.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 shutdown_connections 2024-04-06T14:37:26.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.096+0000 7fe32e398700 1 -- 172.21.15.32:0/1022418265 wait complete. 2024-04-06T14:37:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.198+0000 7fdefebc6700 1 -- 172.21.15.32:0/3702112367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 msgr2=0x7fdef80fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.198+0000 7fdefebc6700 1 --2- 172.21.15.32:0/3702112367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 0x7fdef80fce20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdee4004940 tx=0x7fdee4009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.198+0000 7fdefebc6700 1 -- 172.21.15.32:0/3702112367 shutdown_connections 2024-04-06T14:37:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.198+0000 7fdefebc6700 1 --2- 172.21.15.32:0/3702112367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 0x7fdef80fce20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.198+0000 7fdefebc6700 1 --2- 172.21.15.32:0/3702112367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef8102e10 0x7fdef81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.198+0000 7fdefebc6700 1 -- 172.21.15.32:0/3702112367 >> 172.21.15.32:0/3702112367 conn(0x7fdef80f8980 msgr2=0x7fdef80fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.199+0000 7fdefebc6700 1 -- 172.21.15.32:0/3702112367 shutdown_connections 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.199+0000 7fdefebc6700 1 -- 172.21.15.32:0/3702112367 wait complete. 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.199+0000 7fdefebc6700 1 Processor -- start 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.199+0000 7fdefebc6700 1 -- start start 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.199+0000 7fdefebc6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef8102e10 0x7fdef80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.199+0000 7fdefebc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 0x7fdef8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.199+0000 7fdefebc6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef8070d90 con 0x7fdef81037b0 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.199+0000 7fdefebc6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef8070ed0 con 0x7fdef8102e10 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 0x7fdef8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 0x7fdef8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60948/0 (socket says 172.21.15.32:60948) 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef7fff700 1 -- 172.21.15.32:0/3917889615 learned_addr learned my addr 172.21.15.32:0/3917889615 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdefc962700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef8102e10 0x7fdef80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef7fff700 1 -- 172.21.15.32:0/3917889615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef8102e10 msgr2=0x7fdef80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef7fff700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef8102e10 0x7fdef80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef7fff700 1 -- 172.21.15.32:0/3917889615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee4005040 con 0x7fdef81037b0 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef7fff700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 0x7fdef8070800 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009390 tx=0x7fdee4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef57fa700 1 -- 172.21.15.32:0/3917889615 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdee400e040 con 0x7fdef81037b0 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef57fa700 1 -- 172.21.15.32:0/3917889615 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdee4013420 con 0x7fdef81037b0 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdef57fa700 1 -- 172.21.15.32:0/3917889615 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdee401c630 con 0x7fdef81037b0 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef8071120 con 0x7fdef81037b0 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.200+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef811a210 con 0x7fdef81037b0 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.201+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef80fe560 con 0x7fdef81037b0 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.202+0000 7fdef57fa700 1 -- 172.21.15.32:0/3917889615 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdee4013a00 con 0x7fdef81037b0 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.202+0000 7fdef57fa700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdee806cc30 0x7fdee806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.202+0000 7fdef57fa700 1 -- 172.21.15.32:0/3917889615 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdee408d030 con 0x7fdef81037b0 2024-04-06T14:37:26.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.202+0000 7fdefc962700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdee806cc30 0x7fdee806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:26.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.203+0000 7fdefc962700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdee806cc30 0x7fdee806f0e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdeec007fd0 tx=0x7fdeec009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:26.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.207+0000 7fdef57fa700 1 -- 172.21.15.32:0/3917889615 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdee4057130 con 0x7fdef81037b0 2024-04-06T14:37:26.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:26 smithi032 ceph-mon[147414]: pgmap v518: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:26.475 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:26 smithi032 ceph-mon[147414]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:26.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.495+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdef804bed0 con 0x7fdef81037b0 2024-04-06T14:37:26.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.496+0000 7fdef57fa700 1 -- 172.21.15.32:0/3917889615 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fdee405a750 con 0x7fdef81037b0 2024-04-06T14:37:26.491 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:37:26.491 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:37:26.491 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:37:26.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:37:26.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:37:26.493 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:37:26.493 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:37:26.493 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:37:26.493 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:37:26.493 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:37:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdee806cc30 msgr2=0x7fdee806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdee806cc30 0x7fdee806f0e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdeec007fd0 tx=0x7fdeec009040 comp rx=0 tx=0).stop 2024-04-06T14:37:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 msgr2=0x7fdef8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 0x7fdef8070800 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009390 tx=0x7fdee4009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 shutdown_connections 2024-04-06T14:37:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdee806cc30 0x7fdee806f0e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef8102e10 0x7fdef80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 --2- 172.21.15.32:0/3917889615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdef81037b0 0x7fdef8070800 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 >> 172.21.15.32:0/3917889615 conn(0x7fdef80f8980 msgr2=0x7fdef80f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 shutdown_connections 2024-04-06T14:37:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.498+0000 7fdefebc6700 1 -- 172.21.15.32:0/3917889615 wait complete. 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 -- 172.21.15.32:0/4115340201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154102710 msgr2=0x7f01540fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 --2- 172.21.15.32:0/4115340201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154102710 0x7f01540fbd80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0140004940 tx=0x7f0140009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 -- 172.21.15.32:0/4115340201 shutdown_connections 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 --2- 172.21.15.32:0/4115340201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154102710 0x7f01540fbd80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 --2- 172.21.15.32:0/4115340201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0154101d70 0x7f0154102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 -- 172.21.15.32:0/4115340201 >> 172.21.15.32:0/4115340201 conn(0x7f0154072bc0 msgr2=0x7f0154072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 -- 172.21.15.32:0/4115340201 shutdown_connections 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 -- 172.21.15.32:0/4115340201 wait complete. 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 Processor -- start 2024-04-06T14:37:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.601+0000 7f015ab19700 1 -- start start 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f015ab19700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154101d70 0x7f015410ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f015ab19700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0154102710 0x7f015410efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f015ab19700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015410f6b0 con 0x7f0154101d70 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f015ab19700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015410f7f0 con 0x7f0154102710 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01588b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154101d70 0x7f015410ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01588b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154101d70 0x7f015410ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60974/0 (socket says 172.21.15.32:60974) 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01588b5700 1 -- 172.21.15.32:0/3117102451 learned_addr learned my addr 172.21.15.32:0/3117102451 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01588b5700 1 -- 172.21.15.32:0/3117102451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0154102710 msgr2=0x7f015410efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01588b5700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0154102710 0x7f015410efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01588b5700 1 -- 172.21.15.32:0/3117102451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140005040 con 0x7f0154101d70 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01588b5700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154101d70 0x7f015410ea90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f014800b790 tx=0x7f0148009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01517fa700 1 -- 172.21.15.32:0/3117102451 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f014800f770 con 0x7f0154101d70 2024-04-06T14:37:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0154117b60 con 0x7f0154101d70 2024-04-06T14:37:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01517fa700 1 -- 172.21.15.32:0/3117102451 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f014800fd80 con 0x7f0154101d70 2024-04-06T14:37:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f01517fa700 1 -- 172.21.15.32:0/3117102451 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f014800d310 con 0x7f0154101d70 2024-04-06T14:37:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.602+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01541180f0 con 0x7f0154101d70 2024-04-06T14:37:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.603+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01540ff080 con 0x7f0154101d70 2024-04-06T14:37:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.604+0000 7f01517fa700 1 -- 172.21.15.32:0/3117102451 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0148010420 con 0x7f0154101d70 2024-04-06T14:37:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.604+0000 7f01517fa700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f014406cc20 0x7f014406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.604+0000 7f01517fa700 1 -- 172.21.15.32:0/3117102451 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f014808c0f0 con 0x7f0154101d70 2024-04-06T14:37:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.604+0000 7f0153fff700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f014406cc20 0x7f014406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.605+0000 7f0153fff700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f014406cc20 0x7f014406f0d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0140004f90 tx=0x7f0140009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:26.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.609+0000 7f01517fa700 1 -- 172.21.15.32:0/3117102451 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f01480560b0 con 0x7f0154101d70 2024-04-06T14:37:26.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:26 smithi086 ceph-mon[110975]: pgmap v518: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:26.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:26 smithi086 ceph-mon[110975]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:26.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.855+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0154108d80 con 0x7f0154101d70 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.855+0000 7f01517fa700 1 -- 172.21.15.32:0/3117102451 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f01480596d0 con 0x7f0154101d70 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:37:26.851 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:37:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:37:26.853 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:37:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f014406cc20 msgr2=0x7f014406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f014406cc20 0x7f014406f0d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0140004f90 tx=0x7f0140009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154101d70 msgr2=0x7f015410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154101d70 0x7f015410ea90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f014800b790 tx=0x7f0148009250 comp rx=0 tx=0).stop 2024-04-06T14:37:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 shutdown_connections 2024-04-06T14:37:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f014406cc20 0x7f014406f0d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0154101d70 0x7f015410ea90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 --2- 172.21.15.32:0/3117102451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0154102710 0x7f015410efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 >> 172.21.15.32:0/3117102451 conn(0x7f0154072bc0 msgr2=0x7f01540fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 shutdown_connections 2024-04-06T14:37:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.857+0000 7f015ab19700 1 -- 172.21.15.32:0/3117102451 wait complete. 2024-04-06T14:37:26.855 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:37:26.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.961+0000 7f41c5e8d700 1 -- 172.21.15.32:0/1026895932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 msgr2=0x7f41c01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.961+0000 7f41c5e8d700 1 --2- 172.21.15.32:0/1026895932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 0x7f41c01031e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f41a8004940 tx=0x7f41a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:26.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.961+0000 7f41c5e8d700 1 -- 172.21.15.32:0/1026895932 shutdown_connections 2024-04-06T14:37:26.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.961+0000 7f41c5e8d700 1 --2- 172.21.15.32:0/1026895932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c01037b0 0x7f41c00fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.961+0000 7f41c5e8d700 1 --2- 172.21.15.32:0/1026895932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 0x7f41c01031e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.961+0000 7f41c5e8d700 1 -- 172.21.15.32:0/1026895932 >> 172.21.15.32:0/1026895932 conn(0x7f41c00f89a0 msgr2=0x7f41c00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:26.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.961+0000 7f41c5e8d700 1 -- 172.21.15.32:0/1026895932 shutdown_connections 2024-04-06T14:37:26.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.961+0000 7f41c5e8d700 1 -- 172.21.15.32:0/1026895932 wait complete. 2024-04-06T14:37:26.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.962+0000 7f41c5e8d700 1 Processor -- start 2024-04-06T14:37:26.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.962+0000 7f41c5e8d700 1 -- start start 2024-04-06T14:37:26.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.962+0000 7f41c5e8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 0x7f41c0113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:26.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.962+0000 7f41c5e8d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c01037b0 0x7f41c0114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:26.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.962+0000 7f41c5e8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0114680 con 0x7f41c0102e10 2024-04-06T14:37:26.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.962+0000 7f41c5e8d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c010da70 con 0x7f41c01037b0 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.962+0000 7f41bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 0x7f41c0113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.962+0000 7f41bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 0x7f41c0113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60984/0 (socket says 172.21.15.32:60984) 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.962+0000 7f41bf7fe700 1 -- 172.21.15.32:0/3729663150 learned_addr learned my addr 172.21.15.32:0/3729663150 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41bf7fe700 1 -- 172.21.15.32:0/3729663150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c01037b0 msgr2=0x7f41c0114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41beffd700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c01037b0 0x7f41c0114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41bf7fe700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c01037b0 0x7f41c0114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41bf7fe700 1 -- 172.21.15.32:0/3729663150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a8005040 con 0x7f41c0102e10 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41bf7fe700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 0x7f41c0113af0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f41a800b3b0 tx=0x7f41a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41b7fff700 1 -- 172.21.15.32:0/3729663150 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41a800e040 con 0x7f41c0102e10 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41b7fff700 1 -- 172.21.15.32:0/3729663150 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41a8013420 con 0x7f41c0102e10 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c010dcf0 con 0x7f41c0102e10 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41b7fff700 1 -- 172.21.15.32:0/3729663150 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41a801c630 con 0x7f41c0102e10 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.963+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c010e2b0 con 0x7f41c0102e10 2024-04-06T14:37:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.964+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c00fe560 con 0x7f41c0102e10 2024-04-06T14:37:26.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.964+0000 7f41b7fff700 1 -- 172.21.15.32:0/3729663150 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f41a8013970 con 0x7f41c0102e10 2024-04-06T14:37:26.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.965+0000 7f41b7fff700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41ac06cc20 0x7f41ac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:26.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.965+0000 7f41b7fff700 1 -- 172.21.15.32:0/3729663150 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f41a808d2f0 con 0x7f41c0102e10 2024-04-06T14:37:26.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.965+0000 7f41beffd700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41ac06cc20 0x7f41ac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:26.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.965+0000 7f41beffd700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41ac06cc20 0x7f41ac06f0d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f41c010ec30 tx=0x7f41b000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:26.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:26.969+0000 7f41b7fff700 1 -- 172.21.15.32:0/3729663150 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f41a805ad80 con 0x7f41c0102e10 2024-04-06T14:37:27.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.191+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c0109e40 con 0x7f41ac06cc20 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.192+0000 7f41b7fff700 1 -- 172.21.15.32:0/3729663150 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f41c0109e40 con 0x7f41ac06cc20 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:37:27.188 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:37:27.189 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:37:27.189 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:37:27.189 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41ac06cc20 msgr2=0x7f41ac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41ac06cc20 0x7f41ac06f0d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f41c010ec30 tx=0x7f41b000b040 comp rx=0 tx=0).stop 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 msgr2=0x7f41c0113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 0x7f41c0113af0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f41a800b3b0 tx=0x7f41a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 shutdown_connections 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41ac06cc20 0x7f41ac06f0d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41c0102e10 0x7f41c0113af0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 --2- 172.21.15.32:0/3729663150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c01037b0 0x7f41c0114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 >> 172.21.15.32:0/3729663150 conn(0x7f41c00f89a0 msgr2=0x7f41c00f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 shutdown_connections 2024-04-06T14:37:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.194+0000 7f41c5e8d700 1 -- 172.21.15.32:0/3729663150 wait complete. 2024-04-06T14:37:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 -- 172.21.15.32:0/1971834670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638102710 msgr2=0x7f56380fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 --2- 172.21.15.32:0/1971834670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638102710 0x7f56380fbd80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5628004940 tx=0x7f5628009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 -- 172.21.15.32:0/1971834670 shutdown_connections 2024-04-06T14:37:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 --2- 172.21.15.32:0/1971834670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638102710 0x7f56380fbd80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 --2- 172.21.15.32:0/1971834670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5638101d70 0x7f5638102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 -- 172.21.15.32:0/1971834670 >> 172.21.15.32:0/1971834670 conn(0x7f5638072bc0 msgr2=0x7f5638072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 -- 172.21.15.32:0/1971834670 shutdown_connections 2024-04-06T14:37:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 -- 172.21.15.32:0/1971834670 wait complete. 2024-04-06T14:37:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 Processor -- start 2024-04-06T14:37:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 -- start start 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.299+0000 7f564047c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638101d70 0x7f563810ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f564047c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5638102710 0x7f563810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f564047c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563810f5f0 con 0x7f5638101d70 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f564047c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563810f730 con 0x7f5638102710 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f563e218700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638101d70 0x7f563810ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f563e218700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638101d70 0x7f563810ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32768/0 (socket says 172.21.15.32:32768) 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f563e218700 1 -- 172.21.15.32:0/1876623801 learned_addr learned my addr 172.21.15.32:0/1876623801 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f563da17700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5638102710 0x7f563810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f563e218700 1 -- 172.21.15.32:0/1876623801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5638102710 msgr2=0x7f563810efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f563e218700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5638102710 0x7f563810efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f563e218700 1 -- 172.21.15.32:0/1876623801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5628005040 con 0x7f5638101d70 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f563e218700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638101d70 0x7f563810ea60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f563400c8c0 tx=0x7f56340044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f562effd700 1 -- 172.21.15.32:0/1876623801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563400a980 con 0x7f5638101d70 2024-04-06T14:37:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f562effd700 1 -- 172.21.15.32:0/1876623801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f563400ce60 con 0x7f5638101d70 2024-04-06T14:37:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5638117ad0 con 0x7f5638101d70 2024-04-06T14:37:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f562effd700 1 -- 172.21.15.32:0/1876623801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56340113a0 con 0x7f5638101d70 2024-04-06T14:37:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.300+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5638118030 con 0x7f5638101d70 2024-04-06T14:37:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.301+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56380ff080 con 0x7f5638101d70 2024-04-06T14:37:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.302+0000 7f562effd700 1 -- 172.21.15.32:0/1876623801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5634012400 con 0x7f5638101d70 2024-04-06T14:37:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.302+0000 7f562effd700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f562406cc20 0x7f562406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.302+0000 7f562effd700 1 -- 172.21.15.32:0/1876623801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f563408c100 con 0x7f5638101d70 2024-04-06T14:37:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.302+0000 7f563da17700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f562406cc20 0x7f562406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.303+0000 7f563da17700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f562406cc20 0x7f562406f0d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f562800b3b0 tx=0x7f5628009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:27.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.307+0000 7f562effd700 1 -- 172.21.15.32:0/1876623801 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5634056140 con 0x7f5638101d70 2024-04-06T14:37:27.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.594+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f56380fd4c0 con 0x7f5638101d70 2024-04-06T14:37:27.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.594+0000 7f562effd700 1 -- 172.21.15.32:0/1876623801 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f5634059760 con 0x7f5638101d70 2024-04-06T14:37:27.590 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:37:27.590 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:37:27.590 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:37:27.590 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:37:27.590 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:37:27.590 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:37:27.590 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f562406cc20 msgr2=0x7f562406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f562406cc20 0x7f562406f0d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f562800b3b0 tx=0x7f5628009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638101d70 msgr2=0x7f563810ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638101d70 0x7f563810ea60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f563400c8c0 tx=0x7f56340044c0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 shutdown_connections 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f562406cc20 0x7f562406f0d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5638101d70 0x7f563810ea60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 --2- 172.21.15.32:0/1876623801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5638102710 0x7f563810efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 >> 172.21.15.32:0/1876623801 conn(0x7f5638072bc0 msgr2=0x7f56380fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 shutdown_connections 2024-04-06T14:37:27.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:27.596+0000 7f564047c700 1 -- 172.21.15.32:0/1876623801 wait complete. 2024-04-06T14:37:27.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:27 smithi086 ceph-mon[110975]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:27.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:27 smithi086 ceph-mon[110975]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:27.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:27 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3917889615' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:37:27.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:27 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3117102451' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:37:27.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:27 smithi032 ceph-mon[147414]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:27.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:27 smithi032 ceph-mon[147414]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:27.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:27 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3917889615' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:37:27.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:27 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3117102451' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:37:28.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:28 smithi086 ceph-mon[110975]: pgmap v519: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:28.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:28 smithi086 ceph-mon[110975]: from='client.35000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:28.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:28 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1876623801' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:37:28.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:28 smithi032 ceph-mon[147414]: pgmap v519: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:28.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:28 smithi032 ceph-mon[147414]: from='client.35000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:28.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:28 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1876623801' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:37:30.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:30 smithi086 ceph-mon[110975]: pgmap v520: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:30.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:30 smithi032 ceph-mon[147414]: pgmap v520: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:32.625 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:32 smithi086 ceph-mon[110975]: pgmap v521: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:32.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:32 smithi032 ceph-mon[147414]: pgmap v521: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:34.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:34 smithi086 ceph-mon[110975]: pgmap v522: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:34.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:34 smithi032 ceph-mon[147414]: pgmap v522: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:36.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:36 smithi086 ceph-mon[110975]: pgmap v523: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:36.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:36 smithi032 ceph-mon[147414]: pgmap v523: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:38.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:38 smithi086 ceph-mon[110975]: pgmap v524: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:38.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:37:38.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:38 smithi032 ceph-mon[147414]: pgmap v524: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:38.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:37:40.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:40 smithi086 ceph-mon[110975]: pgmap v525: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:40.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:40 smithi032 ceph-mon[147414]: pgmap v525: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:42.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:42 smithi086 ceph-mon[110975]: pgmap v526: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:42.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:42 smithi032 ceph-mon[147414]: pgmap v526: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:44.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:44 smithi086 ceph-mon[110975]: pgmap v527: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:44.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:44 smithi032 ceph-mon[147414]: pgmap v527: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:46.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:46 smithi086 ceph-mon[110975]: pgmap v528: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:46.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:46 smithi032 ceph-mon[147414]: pgmap v528: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:48.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:48 smithi086 ceph-mon[110975]: pgmap v529: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:48.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:48 smithi032 ceph-mon[147414]: pgmap v529: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:50.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:50 smithi086 ceph-mon[110975]: pgmap v530: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:50.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:50 smithi032 ceph-mon[147414]: pgmap v530: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:52.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:52 smithi086 ceph-mon[110975]: pgmap v531: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:52.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:52 smithi032 ceph-mon[147414]: pgmap v531: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:53.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:37:53.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:37:54.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:54 smithi086 ceph-mon[110975]: pgmap v532: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:54.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:54 smithi032 ceph-mon[147414]: pgmap v532: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:56.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:56 smithi086 ceph-mon[110975]: pgmap v533: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:56.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:56 smithi032 ceph-mon[147414]: pgmap v533: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.714+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1806940375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e01014f0 msgr2=0x7fa1e01018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.714+0000 7fa1e7ec7700 1 --2- 172.21.15.32:0/1806940375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e01014f0 0x7fa1e01018c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc004940 tx=0x7fa1dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.714+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1806940375 shutdown_connections 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.714+0000 7fa1e7ec7700 1 --2- 172.21.15.32:0/1806940375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e00fb4a0 0x7fa1e00fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.714+0000 7fa1e7ec7700 1 --2- 172.21.15.32:0/1806940375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e01014f0 0x7fa1e01018c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.714+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1806940375 >> 172.21.15.32:0/1806940375 conn(0x7fa1e00714c0 msgr2=0x7fa1e00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.714+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1806940375 shutdown_connections 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.714+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1806940375 wait complete. 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e7ec7700 1 Processor -- start 2024-04-06T14:37:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e7ec7700 1 -- start start 2024-04-06T14:37:57.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e7ec7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e00fb4a0 0x7fa1e0111880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:57.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e7ec7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e01014f0 0x7fa1e0111dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:57.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e7ec7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e0112410 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e7ec7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e010b800 con 0x7fa1e01014f0 2024-04-06T14:37:57.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e5c63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e00fb4a0 0x7fa1e0111880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:57.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e5c63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e00fb4a0 0x7fa1e0111880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37032/0 (socket says 172.21.15.32:37032) 2024-04-06T14:37:57.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e5c63700 1 -- 172.21.15.32:0/1418659172 learned_addr learned my addr 172.21.15.32:0/1418659172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.715+0000 7fa1e5462700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e01014f0 0x7fa1e0111dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.716+0000 7fa1e5c63700 1 -- 172.21.15.32:0/1418659172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e01014f0 msgr2=0x7fa1e0111dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.716+0000 7fa1e5c63700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e01014f0 0x7fa1e0111dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.716+0000 7fa1e5c63700 1 -- 172.21.15.32:0/1418659172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc005040 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.716+0000 7fa1e5c63700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e00fb4a0 0x7fa1e0111880 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00b3b0 tx=0x7fa1dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.716+0000 7fa1d67fc700 1 -- 172.21.15.32:0/1418659172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc00e040 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.716+0000 7fa1d67fc700 1 -- 172.21.15.32:0/1418659172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1dc004290 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.716+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e010ba80 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.716+0000 7fa1d67fc700 1 -- 172.21.15.32:0/1418659172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc013660 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.716+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e010bfe0 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.717+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e00fcbd0 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.718+0000 7fa1d67fc700 1 -- 172.21.15.32:0/1418659172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa1dc01e960 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.718+0000 7fa1d67fc700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1cc06cc20 0x7fa1cc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:57.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.718+0000 7fa1d67fc700 1 -- 172.21.15.32:0/1418659172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa1dc08fd90 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.718+0000 7fa1e5462700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1cc06cc20 0x7fa1cc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:57.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.719+0000 7fa1e5462700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1cc06cc20 0x7fa1cc06f0d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa1e010c9c0 tx=0x7fa1d000af90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:57.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.723+0000 7fa1d67fc700 1 -- 172.21.15.32:0/1418659172 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa1dc05d860 con 0x7fa1e00fb4a0 2024-04-06T14:37:57.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.945+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1e0107b90 con 0x7fa1cc06cc20 2024-04-06T14:37:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.946+0000 7fa1d67fc700 1 -- 172.21.15.32:0/1418659172 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa1e0107b90 con 0x7fa1cc06cc20 2024-04-06T14:37:57.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1cc06cc20 msgr2=0x7fa1cc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:57.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1cc06cc20 0x7fa1cc06f0d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa1e010c9c0 tx=0x7fa1d000af90 comp rx=0 tx=0).stop 2024-04-06T14:37:57.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e00fb4a0 msgr2=0x7fa1e0111880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e00fb4a0 0x7fa1e0111880 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00b3b0 tx=0x7fa1dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 shutdown_connections 2024-04-06T14:37:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1cc06cc20 0x7fa1cc06f0d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e00fb4a0 0x7fa1e0111880 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 --2- 172.21.15.32:0/1418659172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e01014f0 0x7fa1e0111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 >> 172.21.15.32:0/1418659172 conn(0x7fa1e00714c0 msgr2=0x7fa1e00f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 shutdown_connections 2024-04-06T14:37:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:57.948+0000 7fa1e7ec7700 1 -- 172.21.15.32:0/1418659172 wait complete. 2024-04-06T14:37:57.954 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:37:58.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 -- 172.21.15.32:0/1736568967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 msgr2=0x7f602c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 --2- 172.21.15.32:0/1736568967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 0x7f602c106f00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6020004940 tx=0x7f6020009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:58.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 -- 172.21.15.32:0/1736568967 shutdown_connections 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 --2- 172.21.15.32:0/1736568967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 0x7f602c106f00 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 --2- 172.21.15.32:0/1736568967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f602c0fde90 0x7f602c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 -- 172.21.15.32:0/1736568967 >> 172.21.15.32:0/1736568967 conn(0x7f602c072bc0 msgr2=0x7f602c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 -- 172.21.15.32:0/1736568967 shutdown_connections 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 -- 172.21.15.32:0/1736568967 wait complete. 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 Processor -- start 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.053+0000 7f6030a02700 1 -- start start 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6030a02700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f602c0fde90 0x7f602c113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6030a02700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 0x7f602c113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6030a02700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c114630 con 0x7f602c0fe830 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6030a02700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c10da20 con 0x7f602c0fde90 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6029d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 0x7f602c113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6029d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 0x7f602c113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37056/0 (socket says 172.21.15.32:37056) 2024-04-06T14:37:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6029d9b700 1 -- 172.21.15.32:0/4139684975 learned_addr learned my addr 172.21.15.32:0/4139684975 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f602a59c700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f602c0fde90 0x7f602c113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6029d9b700 1 -- 172.21.15.32:0/4139684975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f602c0fde90 msgr2=0x7f602c113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6029d9b700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f602c0fde90 0x7f602c113aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6029d9b700 1 -- 172.21.15.32:0/4139684975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6020005040 con 0x7f602c0fe830 2024-04-06T14:37:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6029d9b700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 0x7f602c113fe0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6020000c00 tx=0x7f6020009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f601affd700 1 -- 172.21.15.32:0/4139684975 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6020021070 con 0x7f602c0fe830 2024-04-06T14:37:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c10dca0 con 0x7f602c0fe830 2024-04-06T14:37:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f601affd700 1 -- 172.21.15.32:0/4139684975 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6020013e00 con 0x7f602c0fe830 2024-04-06T14:37:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f601affd700 1 -- 172.21.15.32:0/4139684975 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f602000e070 con 0x7f602c0fe830 2024-04-06T14:37:58.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.054+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c10e1e0 con 0x7f602c0fe830 2024-04-06T14:37:58.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.055+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c101980 con 0x7f602c0fe830 2024-04-06T14:37:58.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.056+0000 7f601affd700 1 -- 172.21.15.32:0/4139684975 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f602001c590 con 0x7f602c0fe830 2024-04-06T14:37:58.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.056+0000 7f601affd700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f601406cc20 0x7f601406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:58.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.056+0000 7f601affd700 1 -- 172.21.15.32:0/4139684975 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f602008d0d0 con 0x7f602c0fe830 2024-04-06T14:37:58.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.056+0000 7f602a59c700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f601406cc20 0x7f601406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:58.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.057+0000 7f602a59c700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f601406cc20 0x7f601406f0d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f601c009b80 tx=0x7f601c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:58.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.061+0000 7f601affd700 1 -- 172.21.15.32:0/4139684975 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f602005aba0 con 0x7f602c0fe830 2024-04-06T14:37:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.282+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f602c10f0f0 con 0x7f601406cc20 2024-04-06T14:37:58.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.283+0000 7f601affd700 1 -- 172.21.15.32:0/4139684975 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f602c10f0f0 con 0x7f601406cc20 2024-04-06T14:37:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f601406cc20 msgr2=0x7f601406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f601406cc20 0x7f601406f0d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f601c009b80 tx=0x7f601c00b040 comp rx=0 tx=0).stop 2024-04-06T14:37:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 msgr2=0x7f602c113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 0x7f602c113fe0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6020000c00 tx=0x7f6020009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 shutdown_connections 2024-04-06T14:37:58.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f601406cc20 0x7f601406f0d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f602c0fde90 0x7f602c113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 --2- 172.21.15.32:0/4139684975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f602c0fe830 0x7f602c113fe0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 >> 172.21.15.32:0/4139684975 conn(0x7f602c072bc0 msgr2=0x7f602c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 shutdown_connections 2024-04-06T14:37:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.285+0000 7f6030a02700 1 -- 172.21.15.32:0/4139684975 wait complete. 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.388+0000 7fc3363f1700 1 -- 172.21.15.32:0/1129028393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330101d70 msgr2=0x7fc330102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.388+0000 7fc3363f1700 1 --2- 172.21.15.32:0/1129028393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330101d70 0x7fc330102140 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc320004940 tx=0x7fc320009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.388+0000 7fc3363f1700 1 -- 172.21.15.32:0/1129028393 shutdown_connections 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.388+0000 7fc3363f1700 1 --2- 172.21.15.32:0/1129028393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc330102710 0x7fc3300fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.388+0000 7fc3363f1700 1 --2- 172.21.15.32:0/1129028393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330101d70 0x7fc330102140 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.388+0000 7fc3363f1700 1 -- 172.21.15.32:0/1129028393 >> 172.21.15.32:0/1129028393 conn(0x7fc330072bc0 msgr2=0x7fc330072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.388+0000 7fc3363f1700 1 -- 172.21.15.32:0/1129028393 shutdown_connections 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.388+0000 7fc3363f1700 1 -- 172.21.15.32:0/1129028393 wait complete. 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc3363f1700 1 Processor -- start 2024-04-06T14:37:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc3363f1700 1 -- start start 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc3363f1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc330101d70 0x7fc330113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc3363f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330102710 0x7fc330113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc3363f1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc330114620 con 0x7fc330102710 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc3363f1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33010da60 con 0x7fc330101d70 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc32f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330102710 0x7fc330113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc32f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330102710 0x7fc330113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37076/0 (socket says 172.21.15.32:37076) 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc32f7fe700 1 -- 172.21.15.32:0/175301428 learned_addr learned my addr 172.21.15.32:0/175301428 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.389+0000 7fc32ffff700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc330101d70 0x7fc330113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.390+0000 7fc32f7fe700 1 -- 172.21.15.32:0/175301428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc330101d70 msgr2=0x7fc330113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.390+0000 7fc32f7fe700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc330101d70 0x7fc330113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.390+0000 7fc32f7fe700 1 -- 172.21.15.32:0/175301428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc320005040 con 0x7fc330102710 2024-04-06T14:37:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.390+0000 7fc32f7fe700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330102710 0x7fc330113fd0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc31800c800 tx=0x7fc318004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.390+0000 7fc32cff9700 1 -- 172.21.15.32:0/175301428 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc31800a760 con 0x7fc330102710 2024-04-06T14:37:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.390+0000 7fc32cff9700 1 -- 172.21.15.32:0/175301428 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc31800ad70 con 0x7fc330102710 2024-04-06T14:37:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.390+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33010dd40 con 0x7fc330102710 2024-04-06T14:37:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.390+0000 7fc32cff9700 1 -- 172.21.15.32:0/175301428 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3180113a0 con 0x7fc330102710 2024-04-06T14:37:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.390+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33010e300 con 0x7fc330102710 2024-04-06T14:37:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.391+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3300fd4c0 con 0x7fc330102710 2024-04-06T14:37:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.391+0000 7fc32cff9700 1 -- 172.21.15.32:0/175301428 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc318012400 con 0x7fc330102710 2024-04-06T14:37:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.392+0000 7fc32cff9700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc31c06cc20 0x7fc31c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.392+0000 7fc32cff9700 1 -- 172.21.15.32:0/175301428 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc31808bfa0 con 0x7fc330102710 2024-04-06T14:37:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.392+0000 7fc32ffff700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc31c06cc20 0x7fc31c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.392+0000 7fc32ffff700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc31c06cc20 0x7fc31c06f0d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc320004f90 tx=0x7fc320009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:58.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.397+0000 7fc32cff9700 1 -- 172.21.15.32:0/175301428 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc318059b30 con 0x7fc330102710 2024-04-06T14:37:58.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.608+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc330108d30 con 0x7fc31c06cc20 2024-04-06T14:37:58.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.618+0000 7fc32cff9700 1 -- 172.21.15.32:0/175301428 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc330108d30 con 0x7fc31c06cc20 2024-04-06T14:37:58.613 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:37:58.613 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (18m) 11m ago 29m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:37:58.613 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (15m) 11m ago 29m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:37:58.613 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (15m) 11m ago 28m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:37:58.613 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (18m) 11m ago 29m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:37:58.613 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (12m) 11m ago 23m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:37:58.613 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (11m) 11m ago 23m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (11m) 11m ago 23m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (23m) 11m ago 23m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (20m) 11m ago 31m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (19m) 11m ago 28m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (16m) 11m ago 31m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (16m) 11m ago 28m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (19m) 11m ago 29m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (18m) 11m ago 28m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (15m) 11m ago 27m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (14m) 11m ago 27m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (14m) 11m ago 26m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (13m) 11m ago 26m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (12m) 11m ago 25m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (12m) 11m ago 25m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:37:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (18m) 11m ago 29m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:37:58.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc31c06cc20 msgr2=0x7fc31c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc31c06cc20 0x7fc31c06f0d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc320004f90 tx=0x7fc320009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:58.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330102710 msgr2=0x7fc330113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330102710 0x7fc330113fd0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc31800c800 tx=0x7fc318004470 comp rx=0 tx=0).stop 2024-04-06T14:37:58.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 shutdown_connections 2024-04-06T14:37:58.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc31c06cc20 0x7fc31c06f0d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc330101d70 0x7fc330113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 --2- 172.21.15.32:0/175301428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc330102710 0x7fc330113fd0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 >> 172.21.15.32:0/175301428 conn(0x7fc330072bc0 msgr2=0x7fc330065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:58.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 shutdown_connections 2024-04-06T14:37:58.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.620+0000 7fc3363f1700 1 -- 172.21.15.32:0/175301428 wait complete. 2024-04-06T14:37:58.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:58 smithi086 ceph-mon[110975]: pgmap v534: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.723+0000 7f57458d0700 1 -- 172.21.15.32:0/3916416326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 msgr2=0x7f5740102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.723+0000 7f57458d0700 1 --2- 172.21.15.32:0/3916416326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 0x7f5740102140 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5730004940 tx=0x7f5730009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.723+0000 7f57458d0700 1 -- 172.21.15.32:0/3916416326 shutdown_connections 2024-04-06T14:37:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.723+0000 7f57458d0700 1 --2- 172.21.15.32:0/3916416326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5740102710 0x7f57400fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.723+0000 7f57458d0700 1 --2- 172.21.15.32:0/3916416326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 0x7f5740102140 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.723+0000 7f57458d0700 1 -- 172.21.15.32:0/3916416326 >> 172.21.15.32:0/3916416326 conn(0x7f5740072bc0 msgr2=0x7f5740072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.723+0000 7f57458d0700 1 -- 172.21.15.32:0/3916416326 shutdown_connections 2024-04-06T14:37:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.723+0000 7f57458d0700 1 -- 172.21.15.32:0/3916416326 wait complete. 2024-04-06T14:37:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.723+0000 7f57458d0700 1 Processor -- start 2024-04-06T14:37:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f57458d0700 1 -- start start 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f57458d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 0x7f5740102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f57458d0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5740102710 0x7f5740103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f57458d0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57401036a0 con 0x7f5740101d70 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f57458d0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57401037e0 con 0x7f5740102710 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f573effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 0x7f5740102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f573effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 0x7f5740102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37086/0 (socket says 172.21.15.32:37086) 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f573effd700 1 -- 172.21.15.32:0/3366998883 learned_addr learned my addr 172.21.15.32:0/3366998883 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f573effd700 1 -- 172.21.15.32:0/3366998883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5740102710 msgr2=0x7f5740103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f573effd700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5740102710 0x7f5740103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f573effd700 1 -- 172.21.15.32:0/3366998883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730005040 con 0x7f5740101d70 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f573effd700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 0x7f5740102bd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f573000b3b0 tx=0x7f5730009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.724+0000 7f57377fe700 1 -- 172.21.15.32:0/3366998883 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f573000e040 con 0x7f5740101d70 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.725+0000 7f57377fe700 1 -- 172.21.15.32:0/3366998883 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5730013420 con 0x7f5740101d70 2024-04-06T14:37:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.725+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5740103a30 con 0x7f5740101d70 2024-04-06T14:37:58.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.725+0000 7f57377fe700 1 -- 172.21.15.32:0/3366998883 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f573001c630 con 0x7f5740101d70 2024-04-06T14:37:58.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.725+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574011a210 con 0x7f5740101d70 2024-04-06T14:37:58.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.725+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57400fd4c0 con 0x7f5740101d70 2024-04-06T14:37:58.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.726+0000 7f57377fe700 1 -- 172.21.15.32:0/3366998883 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5730013970 con 0x7f5740101d70 2024-04-06T14:37:58.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.727+0000 7f57377fe700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f572c06cc20 0x7f572c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:58.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.727+0000 7f57377fe700 1 -- 172.21.15.32:0/3366998883 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f573008d1b0 con 0x7f5740101d70 2024-04-06T14:37:58.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.727+0000 7f573e7fc700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f572c06cc20 0x7f572c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:58.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.727+0000 7f573e7fc700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f572c06cc20 0x7f572c06f0d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5740104510 tx=0x7f5728009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:58.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:58 smithi032 ceph-mon[147414]: pgmap v534: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:37:58.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:58.731+0000 7f57377fe700 1 -- 172.21.15.32:0/3366998883 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f57300572b0 con 0x7f5740101d70 2024-04-06T14:37:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.024+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f574004bed0 con 0x7f5740101d70 2024-04-06T14:37:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.024+0000 7f57377fe700 1 -- 172.21.15.32:0/3366998883 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f573005a8d0 con 0x7f5740101d70 2024-04-06T14:37:59.019 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:37:59.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:37:59.021 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:37:59.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:37:59.021 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.026+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f572c06cc20 msgr2=0x7f572c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.026+0000 7f57458d0700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f572c06cc20 0x7f572c06f0d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5740104510 tx=0x7f5728009ce0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.026+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 msgr2=0x7f5740102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.027+0000 7f57458d0700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 0x7f5740102bd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f573000b3b0 tx=0x7f5730009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.027+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 shutdown_connections 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.027+0000 7f57458d0700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f572c06cc20 0x7f572c06f0d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.027+0000 7f57458d0700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5740101d70 0x7f5740102bd0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.027+0000 7f57458d0700 1 --2- 172.21.15.32:0/3366998883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5740102710 0x7f5740103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.027+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 >> 172.21.15.32:0/3366998883 conn(0x7f5740072bc0 msgr2=0x7f57400659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.027+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 shutdown_connections 2024-04-06T14:37:59.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.027+0000 7f57458d0700 1 -- 172.21.15.32:0/3366998883 wait complete. 2024-04-06T14:37:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.129+0000 7fa324104700 1 -- 172.21.15.32:0/1605398321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c101d10 msgr2=0x7fa31c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.129+0000 7fa324104700 1 --2- 172.21.15.32:0/1605398321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c101d10 0x7fa31c1020e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa318004940 tx=0x7fa318009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.129+0000 7fa324104700 1 -- 172.21.15.32:0/1605398321 shutdown_connections 2024-04-06T14:37:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.129+0000 7fa324104700 1 --2- 172.21.15.32:0/1605398321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c1026b0 0x7fa31c0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.129+0000 7fa324104700 1 --2- 172.21.15.32:0/1605398321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c101d10 0x7fa31c1020e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.129+0000 7fa324104700 1 -- 172.21.15.32:0/1605398321 >> 172.21.15.32:0/1605398321 conn(0x7fa31c0f7880 msgr2=0x7fa31c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.129+0000 7fa324104700 1 -- 172.21.15.32:0/1605398321 shutdown_connections 2024-04-06T14:37:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.129+0000 7fa324104700 1 -- 172.21.15.32:0/1605398321 wait complete. 2024-04-06T14:37:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa324104700 1 Processor -- start 2024-04-06T14:37:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa324104700 1 -- start start 2024-04-06T14:37:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa324104700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c101d10 0x7fa31c112d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa324104700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c1026b0 0x7fa31c113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa324104700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c1138c0 con 0x7fa31c1026b0 2024-04-06T14:37:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa324104700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c10cc90 con 0x7fa31c101d10 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa32169f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c1026b0 0x7fa31c113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa32169f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c1026b0 0x7fa31c113270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37098/0 (socket says 172.21.15.32:37098) 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa32169f700 1 -- 172.21.15.32:0/598958612 learned_addr learned my addr 172.21.15.32:0/598958612 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.130+0000 7fa321ea0700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c101d10 0x7fa31c112d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.131+0000 7fa32169f700 1 -- 172.21.15.32:0/598958612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c101d10 msgr2=0x7fa31c112d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.131+0000 7fa32169f700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c101d10 0x7fa31c112d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.131+0000 7fa32169f700 1 -- 172.21.15.32:0/598958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa318005040 con 0x7fa31c1026b0 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.131+0000 7fa32169f700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c1026b0 0x7fa31c113270 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa30c002a50 tx=0x7fa30c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.131+0000 7fa3127fc700 1 -- 172.21.15.32:0/598958612 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa30c007cc0 con 0x7fa31c1026b0 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.131+0000 7fa3127fc700 1 -- 172.21.15.32:0/598958612 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa30c00f040 con 0x7fa31c1026b0 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.131+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c10cf70 con 0x7fa31c1026b0 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.131+0000 7fa3127fc700 1 -- 172.21.15.32:0/598958612 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa30c00d920 con 0x7fa31c1026b0 2024-04-06T14:37:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.131+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c10d530 con 0x7fa31c1026b0 2024-04-06T14:37:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.132+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31c0fd460 con 0x7fa31c1026b0 2024-04-06T14:37:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.132+0000 7fa3127fc700 1 -- 172.21.15.32:0/598958612 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa30c009680 con 0x7fa31c1026b0 2024-04-06T14:37:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.133+0000 7fa3127fc700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa30806cc20 0x7fa30806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.133+0000 7fa3127fc700 1 -- 172.21.15.32:0/598958612 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa30c08bd40 con 0x7fa31c1026b0 2024-04-06T14:37:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.133+0000 7fa321ea0700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa30806cc20 0x7fa30806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.133+0000 7fa321ea0700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa30806cc20 0x7fa30806f0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa318004f90 tx=0x7fa318009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:59.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.137+0000 7fa3127fc700 1 -- 172.21.15.32:0/598958612 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa30c08f030 con 0x7fa31c1026b0 2024-04-06T14:37:59.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.390+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa31c04bed0 con 0x7fa31c1026b0 2024-04-06T14:37:59.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.390+0000 7fa3127fc700 1 -- 172.21.15.32:0/598958612 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fa30c059500 con 0x7fa31c1026b0 2024-04-06T14:37:59.385 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:37:59.386 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:37:59.387 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:37:59.388 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:37:59.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa30806cc20 msgr2=0x7fa30806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa30806cc20 0x7fa30806f0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa318004f90 tx=0x7fa318009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:59.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c1026b0 msgr2=0x7fa31c113270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c1026b0 0x7fa31c113270 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa30c002a50 tx=0x7fa30c004600 comp rx=0 tx=0).stop 2024-04-06T14:37:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 shutdown_connections 2024-04-06T14:37:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa30806cc20 0x7fa30806f0d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c101d10 0x7fa31c112d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 --2- 172.21.15.32:0/598958612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa31c1026b0 0x7fa31c113270 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 >> 172.21.15.32:0/598958612 conn(0x7fa31c0f7880 msgr2=0x7fa31c0f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.392+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 shutdown_connections 2024-04-06T14:37:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.393+0000 7fa324104700 1 -- 172.21.15.32:0/598958612 wait complete. 2024-04-06T14:37:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:37:59.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.497+0000 7fa21dbe0700 1 -- 172.21.15.32:0/155416638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 msgr2=0x7fa218106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.497+0000 7fa21dbe0700 1 --2- 172.21.15.32:0/155416638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 0x7fa218106df0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa208004940 tx=0x7fa208009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:59.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.497+0000 7fa21dbe0700 1 -- 172.21.15.32:0/155416638 shutdown_connections 2024-04-06T14:37:59.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.497+0000 7fa21dbe0700 1 --2- 172.21.15.32:0/155416638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 0x7fa218106df0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.497+0000 7fa21dbe0700 1 --2- 172.21.15.32:0/155416638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2180fde10 0x7fa2180fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.497+0000 7fa21dbe0700 1 -- 172.21.15.32:0/155416638 >> 172.21.15.32:0/155416638 conn(0x7fa2180f7880 msgr2=0x7fa2180f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:59.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.497+0000 7fa21dbe0700 1 -- 172.21.15.32:0/155416638 shutdown_connections 2024-04-06T14:37:59.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.497+0000 7fa21dbe0700 1 -- 172.21.15.32:0/155416638 wait complete. 2024-04-06T14:37:59.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.497+0000 7fa21dbe0700 1 Processor -- start 2024-04-06T14:37:59.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa21dbe0700 1 -- start start 2024-04-06T14:37:59.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa21dbe0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2180fde10 0x7fa218113da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:59.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa21dbe0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 0x7fa2181142e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:59.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa21dbe0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa218114930 con 0x7fa2180fe7b0 2024-04-06T14:37:59.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa21dbe0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21810dd00 con 0x7fa2180fde10 2024-04-06T14:37:59.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa216ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 0x7fa2181142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:59.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa216ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 0x7fa2181142e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37124/0 (socket says 172.21.15.32:37124) 2024-04-06T14:37:59.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa216ffd700 1 -- 172.21.15.32:0/16298668 learned_addr learned my addr 172.21.15.32:0/16298668 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa2177fe700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2180fde10 0x7fa218113da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa216ffd700 1 -- 172.21.15.32:0/16298668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2180fde10 msgr2=0x7fa218113da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa216ffd700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2180fde10 0x7fa218113da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.498+0000 7fa216ffd700 1 -- 172.21.15.32:0/16298668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa208005040 con 0x7fa2180fe7b0 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.499+0000 7fa216ffd700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 0x7fa2181142e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa208009380 tx=0x7fa208009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.499+0000 7fa1fffff700 1 -- 172.21.15.32:0/16298668 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa20800e040 con 0x7fa2180fe7b0 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.499+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21810df80 con 0x7fa2180fe7b0 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.499+0000 7fa1fffff700 1 -- 172.21.15.32:0/16298668 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa208004290 con 0x7fa2180fe7b0 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.499+0000 7fa1fffff700 1 -- 172.21.15.32:0/16298668 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa20801c3f0 con 0x7fa2180fe7b0 2024-04-06T14:37:59.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.499+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21810e4c0 con 0x7fa2180fe7b0 2024-04-06T14:37:59.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.500+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa218066770 con 0x7fa2180fe7b0 2024-04-06T14:37:59.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.500+0000 7fa1fffff700 1 -- 172.21.15.32:0/16298668 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa208013790 con 0x7fa2180fe7b0 2024-04-06T14:37:59.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.501+0000 7fa1fffff700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa20006cc20 0x7fa20006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:59.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.501+0000 7fa1fffff700 1 -- 172.21.15.32:0/16298668 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa20808d290 con 0x7fa2180fe7b0 2024-04-06T14:37:59.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.501+0000 7fa2177fe700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa20006cc20 0x7fa20006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.501+0000 7fa2177fe700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa20006cc20 0x7fa20006f0d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa20c004c40 tx=0x7fa20c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:59.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.505+0000 7fa1fffff700 1 -- 172.21.15.32:0/16298668 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa20805ae20 con 0x7fa2180fe7b0 2024-04-06T14:37:59.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:59 smithi086 ceph-mon[110975]: from='client.35008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:59.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:59 smithi086 ceph-mon[110975]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:59.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:37:59 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3366998883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:37:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.727+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa218109d60 con 0x7fa20006cc20 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.729+0000 7fa1fffff700 1 -- 172.21.15.32:0/16298668 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa218109d60 con 0x7fa20006cc20 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:37:59.724 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:37:59.725 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:37:59.725 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:37:59.725 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:37:59.725 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:37:59.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:59 smithi032 ceph-mon[147414]: from='client.35008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:59.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:59 smithi032 ceph-mon[147414]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:37:59.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:37:59 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3366998883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:37:59.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa20006cc20 msgr2=0x7fa20006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa20006cc20 0x7fa20006f0d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa20c004c40 tx=0x7fa20c009040 comp rx=0 tx=0).stop 2024-04-06T14:37:59.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 msgr2=0x7fa2181142e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 0x7fa2181142e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa208009380 tx=0x7fa208009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:59.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 shutdown_connections 2024-04-06T14:37:59.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa20006cc20 0x7fa20006f0d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2180fde10 0x7fa218113da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 --2- 172.21.15.32:0/16298668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2180fe7b0 0x7fa2181142e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 >> 172.21.15.32:0/16298668 conn(0x7fa2180f7880 msgr2=0x7fa2180f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:59.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 shutdown_connections 2024-04-06T14:37:59.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.731+0000 7fa21dbe0700 1 -- 172.21.15.32:0/16298668 wait complete. 2024-04-06T14:37:59.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.834+0000 7f9bf2053700 1 -- 172.21.15.32:0/3363433407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 msgr2=0x7f9bec106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.834+0000 7f9bf2053700 1 --2- 172.21.15.32:0/3363433407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 0x7f9bec106f20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9be0004970 tx=0x7f9be0009e30 comp rx=0 tx=0).stop 2024-04-06T14:37:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.834+0000 7f9bf2053700 1 -- 172.21.15.32:0/3363433407 shutdown_connections 2024-04-06T14:37:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.834+0000 7f9bf2053700 1 --2- 172.21.15.32:0/3363433407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 0x7f9bec106f20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.834+0000 7f9bf2053700 1 --2- 172.21.15.32:0/3363433407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bec0fdeb0 0x7f9bec0fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.834+0000 7f9bf2053700 1 -- 172.21.15.32:0/3363433407 >> 172.21.15.32:0/3363433407 conn(0x7f9bec072bc0 msgr2=0x7f9bec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:37:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.834+0000 7f9bf2053700 1 -- 172.21.15.32:0/3363433407 shutdown_connections 2024-04-06T14:37:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.834+0000 7f9bf2053700 1 -- 172.21.15.32:0/3363433407 wait complete. 2024-04-06T14:37:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9bf2053700 1 Processor -- start 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9bf2053700 1 -- start start 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9bf2053700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bec0fdeb0 0x7f9bec0fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9bf2053700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 0x7f9bec0fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9bf2053700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec0fc8d0 con 0x7f9bec0fe850 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9bf2053700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec0fca10 con 0x7f9bec0fdeb0 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9beaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 0x7f9bec0fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9beaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 0x7f9bec0fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37156/0 (socket says 172.21.15.32:37156) 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9beaffd700 1 -- 172.21.15.32:0/2384800915 learned_addr learned my addr 172.21.15.32:0/2384800915 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9beb7fe700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bec0fdeb0 0x7f9bec0fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9beaffd700 1 -- 172.21.15.32:0/2384800915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bec0fdeb0 msgr2=0x7f9bec0fbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:37:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9beaffd700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bec0fdeb0 0x7f9bec0fbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:37:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.835+0000 7f9beaffd700 1 -- 172.21.15.32:0/2384800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be0005040 con 0x7f9bec0fe850 2024-04-06T14:37:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.836+0000 7f9beaffd700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 0x7f9bec0fc2b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009360 tx=0x7f9be0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.836+0000 7f9bd3fff700 1 -- 172.21.15.32:0/2384800915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9be000e040 con 0x7f9bec0fe850 2024-04-06T14:37:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.836+0000 7f9bd3fff700 1 -- 172.21.15.32:0/2384800915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9be0004290 con 0x7f9bec0fe850 2024-04-06T14:37:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.836+0000 7f9bd3fff700 1 -- 172.21.15.32:0/2384800915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9be001c3f0 con 0x7f9bec0fe850 2024-04-06T14:37:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.836+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec119d90 con 0x7f9bec0fe850 2024-04-06T14:37:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.836+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec11a1f0 con 0x7f9bec0fe850 2024-04-06T14:37:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.837+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec065e10 con 0x7f9bec0fe850 2024-04-06T14:37:59.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.837+0000 7f9bd3fff700 1 -- 172.21.15.32:0/2384800915 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9be0013420 con 0x7f9bec0fe850 2024-04-06T14:37:59.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.838+0000 7f9bd3fff700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bd406cc30 0x7f9bd406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:37:59.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.838+0000 7f9bd3fff700 1 -- 172.21.15.32:0/2384800915 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9be008d420 con 0x7f9bec0fe850 2024-04-06T14:37:59.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.838+0000 7f9beb7fe700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bd406cc30 0x7f9bd406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:37:59.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.838+0000 7f9beb7fe700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bd406cc30 0x7f9bd406f0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc006fd0 tx=0x7f9bdc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:37:59.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:37:59.842+0000 7f9bd3fff700 1 -- 172.21.15.32:0/2384800915 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9be0057460 con 0x7f9bec0fe850 2024-04-06T14:38:00.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.128+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9bec05f5f0 con 0x7f9bec0fe850 2024-04-06T14:38:00.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.128+0000 7f9bd3fff700 1 -- 172.21.15.32:0/2384800915 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f9be005aa80 con 0x7f9bec0fe850 2024-04-06T14:38:00.123 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:38:00.123 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:38:00.123 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:38:00.124 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:38:00.124 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:38:00.124 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:38:00.124 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:38:00.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bd406cc30 msgr2=0x7f9bd406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:00.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bd406cc30 0x7f9bd406f0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc006fd0 tx=0x7f9bdc008040 comp rx=0 tx=0).stop 2024-04-06T14:38:00.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 msgr2=0x7f9bec0fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:00.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 0x7f9bec0fc2b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009360 tx=0x7f9be0009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:00.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 shutdown_connections 2024-04-06T14:38:00.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bd406cc30 0x7f9bd406f0e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:00.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bec0fdeb0 0x7f9bec0fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:00.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 --2- 172.21.15.32:0/2384800915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bec0fe850 0x7f9bec0fc2b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:00.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 >> 172.21.15.32:0/2384800915 conn(0x7f9bec072bc0 msgr2=0x7f9bec103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:00.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 shutdown_connections 2024-04-06T14:38:00.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:00.130+0000 7f9bf2053700 1 -- 172.21.15.32:0/2384800915 wait complete. 2024-04-06T14:38:00.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:00 smithi086 ceph-mon[110975]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:00.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:00 smithi086 ceph-mon[110975]: pgmap v535: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:00.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:00 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/598958612' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:38:00.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:00 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2384800915' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:38:00.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:00 smithi032 ceph-mon[147414]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:00.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:00 smithi032 ceph-mon[147414]: pgmap v535: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:00.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:00 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/598958612' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:38:00.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:00 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2384800915' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:38:01.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:01 smithi086 ceph-mon[110975]: from='client.35028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:01.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:01 smithi032 ceph-mon[147414]: from='client.35028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:02.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:02 smithi086 ceph-mon[110975]: pgmap v536: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:02.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:02 smithi032 ceph-mon[147414]: pgmap v536: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:04.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:04 smithi086 ceph-mon[110975]: pgmap v537: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:04.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:04 smithi032 ceph-mon[147414]: pgmap v537: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:06.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:06 smithi086 ceph-mon[110975]: pgmap v538: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:06.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:06 smithi032 ceph-mon[147414]: pgmap v538: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:08.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:08 smithi086 ceph-mon[110975]: pgmap v539: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:08.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:38:08.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:08 smithi032 ceph-mon[147414]: pgmap v539: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:08.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:38:10.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:10 smithi086 ceph-mon[110975]: pgmap v540: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:10.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:10 smithi032 ceph-mon[147414]: pgmap v540: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:12.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:12 smithi086 ceph-mon[110975]: pgmap v541: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:12.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:12 smithi032 ceph-mon[147414]: pgmap v541: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:14.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:14 smithi086 ceph-mon[110975]: pgmap v542: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:14.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:14 smithi032 ceph-mon[147414]: pgmap v542: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:16.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:16 smithi086 ceph-mon[110975]: pgmap v543: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:16.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:16 smithi032 ceph-mon[147414]: pgmap v543: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:18.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:18 smithi086 ceph-mon[110975]: pgmap v544: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:18.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:18 smithi032 ceph-mon[147414]: pgmap v544: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:20.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:20 smithi086 ceph-mon[110975]: pgmap v545: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:20.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:20 smithi032 ceph-mon[147414]: pgmap v545: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:22.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:22 smithi086 ceph-mon[110975]: pgmap v546: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:22.726 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:22 smithi032 ceph-mon[147414]: pgmap v546: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:23.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:38:23.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:38:24.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:24 smithi086 ceph-mon[110975]: pgmap v547: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:24.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:24 smithi032 ceph-mon[147414]: pgmap v547: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:26.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:26 smithi086 ceph-mon[110975]: pgmap v548: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:26.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:26 smithi032 ceph-mon[147414]: pgmap v548: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:28.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:28 smithi086 ceph-mon[110975]: pgmap v549: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:28.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:28 smithi032 ceph-mon[147414]: pgmap v549: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 -- 172.21.15.32:0/3180880444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 msgr2=0x7f7028103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 --2- 172.21.15.32:0/3180880444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 0x7f7028103240 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f701c004940 tx=0x7f701c009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 -- 172.21.15.32:0/3180880444 shutdown_connections 2024-04-06T14:38:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 --2- 172.21.15.32:0/3180880444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70280fb4a0 0x7f70280fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 --2- 172.21.15.32:0/3180880444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 0x7f7028103240 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 -- 172.21.15.32:0/3180880444 >> 172.21.15.32:0/3180880444 conn(0x7f70280714c0 msgr2=0x7f70280718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:30.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 -- 172.21.15.32:0/3180880444 shutdown_connections 2024-04-06T14:38:30.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 -- 172.21.15.32:0/3180880444 wait complete. 2024-04-06T14:38:30.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 Processor -- start 2024-04-06T14:38:30.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.248+0000 7f702eed7700 1 -- start start 2024-04-06T14:38:30.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f702eed7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70280fb4a0 0x7f70281139c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:30.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f702eed7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 0x7f7028113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:30.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f702eed7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7028114550 con 0x7f7028102e70 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f702eed7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702810d990 con 0x7f70280fb4a0 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f7027fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 0x7f7028113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f7027fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 0x7f7028113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43816/0 (socket says 172.21.15.32:43816) 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f7027fff700 1 -- 172.21.15.32:0/946700571 learned_addr learned my addr 172.21.15.32:0/946700571 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f702cc73700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70280fb4a0 0x7f70281139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f7027fff700 1 -- 172.21.15.32:0/946700571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70280fb4a0 msgr2=0x7f70281139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f7027fff700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70280fb4a0 0x7f70281139c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.249+0000 7f7027fff700 1 -- 172.21.15.32:0/946700571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c005040 con 0x7f7028102e70 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.250+0000 7f7027fff700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 0x7f7028113f00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7014002a80 tx=0x7f70140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.250+0000 7f70257fa700 1 -- 172.21.15.32:0/946700571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7014007cc0 con 0x7f7028102e70 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.250+0000 7f70257fa700 1 -- 172.21.15.32:0/946700571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f701400f040 con 0x7f7028102e70 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.250+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702810dc70 con 0x7f7028102e70 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.250+0000 7f70257fa700 1 -- 172.21.15.32:0/946700571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7014009640 con 0x7f7028102e70 2024-04-06T14:38:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.250+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702810e230 con 0x7f7028102e70 2024-04-06T14:38:30.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.251+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7028100110 con 0x7f7028102e70 2024-04-06T14:38:30.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.251+0000 7f70257fa700 1 -- 172.21.15.32:0/946700571 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f70140097a0 con 0x7f7028102e70 2024-04-06T14:38:30.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.252+0000 7f70257fa700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701806cc20 0x7f701806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:30.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.252+0000 7f70257fa700 1 -- 172.21.15.32:0/946700571 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f701408b7f0 con 0x7f7028102e70 2024-04-06T14:38:30.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.252+0000 7f702cc73700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701806cc20 0x7f701806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:30.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.252+0000 7f702cc73700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701806cc20 0x7f701806f0d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f701c004560 tx=0x7f701c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:30.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.256+0000 7f70257fa700 1 -- 172.21.15.32:0/946700571 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7014059240 con 0x7f7028102e70 2024-04-06T14:38:30.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.478+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f702810f010 con 0x7f701806cc20 2024-04-06T14:38:30.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.479+0000 7f70257fa700 1 -- 172.21.15.32:0/946700571 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f702810f010 con 0x7f701806cc20 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701806cc20 msgr2=0x7f701806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701806cc20 0x7f701806f0d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f701c004560 tx=0x7f701c009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 msgr2=0x7f7028113f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 0x7f7028113f00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7014002a80 tx=0x7f70140046b0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 shutdown_connections 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f701806cc20 0x7f701806f0d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70280fb4a0 0x7f70281139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 --2- 172.21.15.32:0/946700571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7028102e70 0x7f7028113f00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 >> 172.21.15.32:0/946700571 conn(0x7f70280714c0 msgr2=0x7f70280fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 shutdown_connections 2024-04-06T14:38:30.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.481+0000 7f702eed7700 1 -- 172.21.15.32:0/946700571 wait complete. 2024-04-06T14:38:30.486 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:38:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.585+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/1385561208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 msgr2=0x7fd1f80fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.585+0000 7fd1fcfd6700 1 --2- 172.21.15.32:0/1385561208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 0x7fd1f80fbd20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0004940 tx=0x7fd1e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.585+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/1385561208 shutdown_connections 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.585+0000 7fd1fcfd6700 1 --2- 172.21.15.32:0/1385561208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 0x7fd1f80fbd20 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.585+0000 7fd1fcfd6700 1 --2- 172.21.15.32:0/1385561208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1f8101d10 0x7fd1f81020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.585+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/1385561208 >> 172.21.15.32:0/1385561208 conn(0x7fd1f80f7880 msgr2=0x7fd1f80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.585+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/1385561208 shutdown_connections 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.585+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/1385561208 wait complete. 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1fcfd6700 1 Processor -- start 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1fcfd6700 1 -- start start 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1fcfd6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1f8101d10 0x7fd1f8107160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1fcfd6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 0x7fd1f81076a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1fcfd6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f8107c30 con 0x7fd1f81026b0 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1fcfd6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f8107d70 con 0x7fd1f8101d10 2024-04-06T14:38:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1f5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 0x7fd1f81076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1f5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 0x7fd1f81076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43836/0 (socket says 172.21.15.32:43836) 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1f5d9b700 1 -- 172.21.15.32:0/3083198599 learned_addr learned my addr 172.21.15.32:0/3083198599 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1f659c700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1f8101d10 0x7fd1f8107160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1f5d9b700 1 -- 172.21.15.32:0/3083198599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1f8101d10 msgr2=0x7fd1f8107160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.586+0000 7fd1f5d9b700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1f8101d10 0x7fd1f8107160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.587+0000 7fd1f5d9b700 1 -- 172.21.15.32:0/3083198599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e0005040 con 0x7fd1f81026b0 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.587+0000 7fd1f5d9b700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 0x7fd1f81076a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009380 tx=0x7fd1e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.587+0000 7fd1eeffd700 1 -- 172.21.15.32:0/3083198599 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e000e040 con 0x7fd1f81026b0 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.587+0000 7fd1eeffd700 1 -- 172.21.15.32:0/3083198599 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1e0013420 con 0x7fd1f81026b0 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.587+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f8107fc0 con 0x7fd1f81026b0 2024-04-06T14:38:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.587+0000 7fd1eeffd700 1 -- 172.21.15.32:0/3083198599 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e001c630 con 0x7fd1f81026b0 2024-04-06T14:38:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.587+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f811a520 con 0x7fd1f81026b0 2024-04-06T14:38:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.588+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f80fd460 con 0x7fd1f81026b0 2024-04-06T14:38:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.588+0000 7fd1eeffd700 1 -- 172.21.15.32:0/3083198599 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd1e00139b0 con 0x7fd1f81026b0 2024-04-06T14:38:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.589+0000 7fd1eeffd700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1e406cc20 0x7fd1e406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.589+0000 7fd1eeffd700 1 -- 172.21.15.32:0/3083198599 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd1e008d1b0 con 0x7fd1f81026b0 2024-04-06T14:38:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.589+0000 7fd1f659c700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1e406cc20 0x7fd1e406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.589+0000 7fd1f659c700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1e406cc20 0x7fd1e406f0d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8004f40 tx=0x7fd1e800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.593+0000 7fd1eeffd700 1 -- 172.21.15.32:0/3083198599 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd1e00571f0 con 0x7fd1f81026b0 2024-04-06T14:38:30.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:30 smithi086 ceph-mon[110975]: pgmap v550: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:30.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:30 smithi032 ceph-mon[147414]: pgmap v550: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.812+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1f8109e10 con 0x7fd1e406cc20 2024-04-06T14:38:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.813+0000 7fd1eeffd700 1 -- 172.21.15.32:0/3083198599 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd1f8109e10 con 0x7fd1e406cc20 2024-04-06T14:38:30.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.815+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1e406cc20 msgr2=0x7fd1e406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.815+0000 7fd1fcfd6700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1e406cc20 0x7fd1e406f0d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8004f40 tx=0x7fd1e800b040 comp rx=0 tx=0).stop 2024-04-06T14:38:30.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.815+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 msgr2=0x7fd1f81076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.815+0000 7fd1fcfd6700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 0x7fd1f81076a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009380 tx=0x7fd1e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.816+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 shutdown_connections 2024-04-06T14:38:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.816+0000 7fd1fcfd6700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1e406cc20 0x7fd1e406f0d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.816+0000 7fd1fcfd6700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1f8101d10 0x7fd1f8107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.816+0000 7fd1fcfd6700 1 --2- 172.21.15.32:0/3083198599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1f81026b0 0x7fd1f81076a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.816+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 >> 172.21.15.32:0/3083198599 conn(0x7fd1f80f7880 msgr2=0x7fd1f80f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.816+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 shutdown_connections 2024-04-06T14:38:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.816+0000 7fd1fcfd6700 1 -- 172.21.15.32:0/3083198599 wait complete. 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.918+0000 7fbab9466700 1 -- 172.21.15.32:0/3498647688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 msgr2=0x7fbab40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.918+0000 7fbab9466700 1 --2- 172.21.15.32:0/3498647688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 0x7fbab40fce10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4004940 tx=0x7fbaa4009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.919+0000 7fbab9466700 1 -- 172.21.15.32:0/3498647688 shutdown_connections 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.919+0000 7fbab9466700 1 --2- 172.21.15.32:0/3498647688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 0x7fbab40fce10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.919+0000 7fbab9466700 1 --2- 172.21.15.32:0/3498647688 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbab4102e00 0x7fbab41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.919+0000 7fbab9466700 1 -- 172.21.15.32:0/3498647688 >> 172.21.15.32:0/3498647688 conn(0x7fbab40f89b0 msgr2=0x7fbab40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.919+0000 7fbab9466700 1 -- 172.21.15.32:0/3498647688 shutdown_connections 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.919+0000 7fbab9466700 1 -- 172.21.15.32:0/3498647688 wait complete. 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.919+0000 7fbab9466700 1 Processor -- start 2024-04-06T14:38:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.919+0000 7fbab9466700 1 -- start start 2024-04-06T14:38:30.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab9466700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbab4102e00 0x7fbab41138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:30.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab9466700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 0x7fbab4113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:30.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab9466700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab41144e0 con 0x7fbab41037a0 2024-04-06T14:38:30.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab9466700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab410da50 con 0x7fbab4102e00 2024-04-06T14:38:30.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 0x7fbab4113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:30.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 0x7fbab4113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43858/0 (socket says 172.21.15.32:43858) 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab27fc700 1 -- 172.21.15.32:0/2157182020 learned_addr learned my addr 172.21.15.32:0/2157182020 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab2ffd700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbab4102e00 0x7fbab41138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab27fc700 1 -- 172.21.15.32:0/2157182020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbab4102e00 msgr2=0x7fbab41138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab27fc700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbab4102e00 0x7fbab41138c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab27fc700 1 -- 172.21.15.32:0/2157182020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa4005040 con 0x7fbab41037a0 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab27fc700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 0x7fbab4113e00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40093b0 tx=0x7fbaa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fba9b7fe700 1 -- 172.21.15.32:0/2157182020 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbaa400e040 con 0x7fbab41037a0 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fba9b7fe700 1 -- 172.21.15.32:0/2157182020 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbaa4013420 con 0x7fbab41037a0 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab410dcd0 con 0x7fbab41037a0 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fba9b7fe700 1 -- 172.21.15.32:0/2157182020 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbaa401c630 con 0x7fbab41037a0 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.920+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab410e290 con 0x7fbab41037a0 2024-04-06T14:38:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.921+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab40fe550 con 0x7fbab41037a0 2024-04-06T14:38:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.922+0000 7fba9b7fe700 1 -- 172.21.15.32:0/2157182020 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbaa40139e0 con 0x7fbab41037a0 2024-04-06T14:38:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.922+0000 7fba9b7fe700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fba9c06cc20 0x7fba9c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.922+0000 7fba9b7fe700 1 -- 172.21.15.32:0/2157182020 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbaa408d110 con 0x7fbab41037a0 2024-04-06T14:38:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.923+0000 7fbab2ffd700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fba9c06cc20 0x7fba9c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.923+0000 7fbab2ffd700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fba9c06cc20 0x7fba9c06f0d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8006fd0 tx=0x7fbaa8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:30.927+0000 7fba9b7fe700 1 -- 172.21.15.32:0/2157182020 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbaa405ad50 con 0x7fbab41037a0 2024-04-06T14:38:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.139+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbab4109e80 con 0x7fba9c06cc20 2024-04-06T14:38:31.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.149+0000 7fba9b7fe700 1 -- 172.21.15.32:0/2157182020 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbab4109e80 con 0x7fba9c06cc20 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (18m) 12m ago 30m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (16m) 12m ago 30m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (16m) 11m ago 29m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (18m) 12m ago 29m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (12m) 12m ago 24m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (12m) 12m ago 24m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (12m) 11m ago 24m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (24m) 11m ago 24m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:38:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (20m) 12m ago 31m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (19m) 11m ago 29m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (17m) 12m ago 31m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (16m) 11m ago 29m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (19m) 12m ago 30m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (19m) 11m ago 29m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (15m) 12m ago 28m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (15m) 12m ago 27m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (14m) 12m ago 27m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (13m) 11m ago 26m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (13m) 11m ago 26m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:38:31.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (12m) 11m ago 26m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:38:31.146 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (19m) 12m ago 29m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:38:31.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fba9c06cc20 msgr2=0x7fba9c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fba9c06cc20 0x7fba9c06f0d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8006fd0 tx=0x7fbaa8008040 comp rx=0 tx=0).stop 2024-04-06T14:38:31.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 msgr2=0x7fbab4113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 0x7fbab4113e00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40093b0 tx=0x7fbaa4009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 shutdown_connections 2024-04-06T14:38:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fba9c06cc20 0x7fba9c06f0d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbab4102e00 0x7fbab41138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 --2- 172.21.15.32:0/2157182020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbab41037a0 0x7fbab4113e00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 >> 172.21.15.32:0/2157182020 conn(0x7fbab40f89b0 msgr2=0x7fbab40f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 shutdown_connections 2024-04-06T14:38:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.151+0000 7fbab9466700 1 -- 172.21.15.32:0/2157182020 wait complete. 2024-04-06T14:38:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.253+0000 7fe3cdea0700 1 -- 172.21.15.32:0/2923085541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 msgr2=0x7fe3c80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.253+0000 7fe3cdea0700 1 --2- 172.21.15.32:0/2923085541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 0x7fe3c80fbd80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0004940 tx=0x7fe3b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.253+0000 7fe3cdea0700 1 -- 172.21.15.32:0/2923085541 shutdown_connections 2024-04-06T14:38:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.253+0000 7fe3cdea0700 1 --2- 172.21.15.32:0/2923085541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 0x7fe3c80fbd80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.253+0000 7fe3cdea0700 1 --2- 172.21.15.32:0/2923085541 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c8101d70 0x7fe3c8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.253+0000 7fe3cdea0700 1 -- 172.21.15.32:0/2923085541 >> 172.21.15.32:0/2923085541 conn(0x7fe3c8072bc0 msgr2=0x7fe3c8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.253+0000 7fe3cdea0700 1 -- 172.21.15.32:0/2923085541 shutdown_connections 2024-04-06T14:38:31.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.253+0000 7fe3cdea0700 1 -- 172.21.15.32:0/2923085541 wait complete. 2024-04-06T14:38:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3cdea0700 1 Processor -- start 2024-04-06T14:38:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3cdea0700 1 -- start start 2024-04-06T14:38:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3cdea0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c8101d70 0x7fe3c8113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3cdea0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 0x7fe3c8113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3cdea0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c8114590 con 0x7fe3c8102710 2024-04-06T14:38:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3cdea0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c810d980 con 0x7fe3c8101d70 2024-04-06T14:38:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 0x7fe3c8113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 0x7fe3c8113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43878/0 (socket says 172.21.15.32:43878) 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3c6ffd700 1 -- 172.21.15.32:0/4125880575 learned_addr learned my addr 172.21.15.32:0/4125880575 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3c77fe700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c8101d70 0x7fe3c8113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3c6ffd700 1 -- 172.21.15.32:0/4125880575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c8101d70 msgr2=0x7fe3c8113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3c6ffd700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c8101d70 0x7fe3c8113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.254+0000 7fe3c6ffd700 1 -- 172.21.15.32:0/4125880575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b0005040 con 0x7fe3c8102710 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.255+0000 7fe3c6ffd700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 0x7fe3c8113f40 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0000c00 tx=0x7fe3b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.255+0000 7fe3bffff700 1 -- 172.21.15.32:0/4125880575 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3b000e070 con 0x7fe3c8102710 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.255+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c810dc00 con 0x7fe3c8102710 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.255+0000 7fe3bffff700 1 -- 172.21.15.32:0/4125880575 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3b00093b0 con 0x7fe3c8102710 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.255+0000 7fe3bffff700 1 -- 172.21.15.32:0/4125880575 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3b001ccb0 con 0x7fe3c8102710 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.255+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c810e160 con 0x7fe3c8102710 2024-04-06T14:38:31.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.256+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c80fd4c0 con 0x7fe3c8102710 2024-04-06T14:38:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.256+0000 7fe3bffff700 1 -- 172.21.15.32:0/4125880575 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe3b001c490 con 0x7fe3c8102710 2024-04-06T14:38:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.257+0000 7fe3bffff700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe3b406cc20 0x7fe3b406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.257+0000 7fe3bffff700 1 -- 172.21.15.32:0/4125880575 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe3b008d4a0 con 0x7fe3c8102710 2024-04-06T14:38:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.257+0000 7fe3c77fe700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe3b406cc20 0x7fe3b406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.257+0000 7fe3c77fe700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe3b406cc20 0x7fe3b406f0d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009b80 tx=0x7fe3b800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:31.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.261+0000 7fe3bffff700 1 -- 172.21.15.32:0/4125880575 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe3b005af70 con 0x7fe3c8102710 2024-04-06T14:38:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.547+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe3c805f5f0 con 0x7fe3c8102710 2024-04-06T14:38:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.548+0000 7fe3bffff700 1 -- 172.21.15.32:0/4125880575 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fe3b005ab00 con 0x7fe3c8102710 2024-04-06T14:38:31.543 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:38:31.543 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:38:31.543 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:38:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:38:31.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:38:31.545 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:38:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe3b406cc20 msgr2=0x7fe3b406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe3b406cc20 0x7fe3b406f0d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009b80 tx=0x7fe3b800b040 comp rx=0 tx=0).stop 2024-04-06T14:38:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 msgr2=0x7fe3c8113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 0x7fe3c8113f40 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0000c00 tx=0x7fe3b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 shutdown_connections 2024-04-06T14:38:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe3b406cc20 0x7fe3b406f0d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c8101d70 0x7fe3c8113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 --2- 172.21.15.32:0/4125880575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c8102710 0x7fe3c8113f40 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 >> 172.21.15.32:0/4125880575 conn(0x7fe3c8072bc0 msgr2=0x7fe3c8065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 shutdown_connections 2024-04-06T14:38:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.550+0000 7fe3cdea0700 1 -- 172.21.15.32:0/4125880575 wait complete. 2024-04-06T14:38:31.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:31 smithi086 ceph-mon[110975]: from='client.35036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.653+0000 7f9caa3d7700 1 -- 172.21.15.32:0/312839801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 msgr2=0x7f9ca41020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.653+0000 7f9caa3d7700 1 --2- 172.21.15.32:0/312839801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 0x7f9ca41020e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9c94004940 tx=0x7f9c94009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.653+0000 7f9caa3d7700 1 -- 172.21.15.32:0/312839801 shutdown_connections 2024-04-06T14:38:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.653+0000 7f9caa3d7700 1 --2- 172.21.15.32:0/312839801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ca41026b0 0x7f9ca40fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.653+0000 7f9caa3d7700 1 --2- 172.21.15.32:0/312839801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 0x7f9ca41020e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.653+0000 7f9caa3d7700 1 -- 172.21.15.32:0/312839801 >> 172.21.15.32:0/312839801 conn(0x7f9ca40f7880 msgr2=0x7f9ca40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.653+0000 7f9caa3d7700 1 -- 172.21.15.32:0/312839801 shutdown_connections 2024-04-06T14:38:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.653+0000 7f9caa3d7700 1 -- 172.21.15.32:0/312839801 wait complete. 2024-04-06T14:38:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9caa3d7700 1 Processor -- start 2024-04-06T14:38:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9caa3d7700 1 -- start start 2024-04-06T14:38:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9caa3d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 0x7f9ca4115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9caa3d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ca41026b0 0x7f9ca4116510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9caa3d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca4116b60 con 0x7f9ca4101d10 2024-04-06T14:38:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9caa3d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca410ff30 con 0x7f9ca41026b0 2024-04-06T14:38:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9ca3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 0x7f9ca4115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9ca3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 0x7f9ca4115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43890/0 (socket says 172.21.15.32:43890) 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9ca3fff700 1 -- 172.21.15.32:0/255173534 learned_addr learned my addr 172.21.15.32:0/255173534 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9ca3fff700 1 -- 172.21.15.32:0/255173534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ca41026b0 msgr2=0x7f9ca4116510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9ca37fe700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ca41026b0 0x7f9ca4116510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9ca3fff700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ca41026b0 0x7f9ca4116510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9ca3fff700 1 -- 172.21.15.32:0/255173534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c94005040 con 0x7f9ca4101d10 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.654+0000 7f9ca37fe700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ca41026b0 0x7f9ca4116510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.655+0000 7f9ca3fff700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 0x7f9ca4115fd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400b3b0 tx=0x7f9c94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.655+0000 7f9ca0ff9700 1 -- 172.21.15.32:0/255173534 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c9400e040 con 0x7f9ca4101d10 2024-04-06T14:38:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.655+0000 7f9ca0ff9700 1 -- 172.21.15.32:0/255173534 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c94013420 con 0x7f9ca4101d10 2024-04-06T14:38:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.655+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca41101b0 con 0x7f9ca4101d10 2024-04-06T14:38:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.655+0000 7f9ca0ff9700 1 -- 172.21.15.32:0/255173534 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c9401c630 con 0x7f9ca4101d10 2024-04-06T14:38:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.655+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca4110770 con 0x7f9ca4101d10 2024-04-06T14:38:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.656+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca40fd460 con 0x7f9ca4101d10 2024-04-06T14:38:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.656+0000 7f9ca0ff9700 1 -- 172.21.15.32:0/255173534 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9c94013970 con 0x7f9ca4101d10 2024-04-06T14:38:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.657+0000 7f9ca0ff9700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c8c06cc20 0x7f9c8c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.657+0000 7f9ca37fe700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c8c06cc20 0x7f9c8c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.657+0000 7f9ca0ff9700 1 -- 172.21.15.32:0/255173534 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c9408d180 con 0x7f9ca4101d10 2024-04-06T14:38:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.657+0000 7f9ca37fe700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c8c06cc20 0x7f9c8c06f0d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9c98007fd0 tx=0x7f9c98009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:31.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.661+0000 7f9ca0ff9700 1 -- 172.21.15.32:0/255173534 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9c9405ad10 con 0x7f9ca4101d10 2024-04-06T14:38:31.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:31 smithi032 ceph-mon[147414]: from='client.35036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:31.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.908+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9ca404bed0 con 0x7f9ca4101d10 2024-04-06T14:38:31.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.908+0000 7f9ca0ff9700 1 -- 172.21.15.32:0/255173534 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f9c9405a8a0 con 0x7f9ca4101d10 2024-04-06T14:38:31.903 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:38:31.903 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:38:31.903 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:38:31.903 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:38:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:38:31.903 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:38:31.904 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:38:31.905 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:38:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:38:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:38:31.906 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:38:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:38:31.906 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:38:31.906 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:38:31.906 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:38:31.906 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:38:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c8c06cc20 msgr2=0x7f9c8c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c8c06cc20 0x7f9c8c06f0d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9c98007fd0 tx=0x7f9c98009040 comp rx=0 tx=0).stop 2024-04-06T14:38:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 msgr2=0x7f9ca4115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 0x7f9ca4115fd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400b3b0 tx=0x7f9c94009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 shutdown_connections 2024-04-06T14:38:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c8c06cc20 0x7f9c8c06f0d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ca4101d10 0x7f9ca4115fd0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 --2- 172.21.15.32:0/255173534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ca41026b0 0x7f9ca4116510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 >> 172.21.15.32:0/255173534 conn(0x7f9ca40f7880 msgr2=0x7f9ca40f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:31.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 shutdown_connections 2024-04-06T14:38:31.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:31.910+0000 7f9caa3d7700 1 -- 172.21.15.32:0/255173534 wait complete. 2024-04-06T14:38:31.908 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:38:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.014+0000 7f9c93ee2700 1 -- 172.21.15.32:0/658258217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 msgr2=0x7f9c8c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.014+0000 7f9c93ee2700 1 --2- 172.21.15.32:0/658258217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 0x7f9c8c0fe260 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9c80004940 tx=0x7f9c80009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.014+0000 7f9c93ee2700 1 -- 172.21.15.32:0/658258217 shutdown_connections 2024-04-06T14:38:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.014+0000 7f9c93ee2700 1 --2- 172.21.15.32:0/658258217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c8c0fe830 0x7f9c8c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.014+0000 7f9c93ee2700 1 --2- 172.21.15.32:0/658258217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 0x7f9c8c0fe260 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.014+0000 7f9c93ee2700 1 -- 172.21.15.32:0/658258217 >> 172.21.15.32:0/658258217 conn(0x7f9c8c072bc0 msgr2=0x7f9c8c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.014+0000 7f9c93ee2700 1 -- 172.21.15.32:0/658258217 shutdown_connections 2024-04-06T14:38:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.014+0000 7f9c93ee2700 1 -- 172.21.15.32:0/658258217 wait complete. 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c93ee2700 1 Processor -- start 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c93ee2700 1 -- start start 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c93ee2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 0x7f9c8c113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c93ee2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c8c0fe830 0x7f9c8c114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c93ee2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c114690 con 0x7f9c8c0fde90 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c93ee2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c10da80 con 0x7f9c8c0fe830 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c91c7e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 0x7f9c8c113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c91c7e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 0x7f9c8c113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43904/0 (socket says 172.21.15.32:43904) 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c91c7e700 1 -- 172.21.15.32:0/1883337005 learned_addr learned my addr 172.21.15.32:0/1883337005 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c9147d700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c8c0fe830 0x7f9c8c114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c91c7e700 1 -- 172.21.15.32:0/1883337005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c8c0fe830 msgr2=0x7f9c8c114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.015+0000 7f9c91c7e700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c8c0fe830 0x7f9c8c114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.016+0000 7f9c91c7e700 1 -- 172.21.15.32:0/1883337005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c80005040 con 0x7f9c8c0fde90 2024-04-06T14:38:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.016+0000 7f9c9147d700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c8c0fe830 0x7f9c8c114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:38:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.016+0000 7f9c91c7e700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 0x7f9c8c113b00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009380 tx=0x7f9c80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.016+0000 7f9c7e7fc700 1 -- 172.21.15.32:0/1883337005 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c8000e040 con 0x7f9c8c0fde90 2024-04-06T14:38:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.016+0000 7f9c7e7fc700 1 -- 172.21.15.32:0/1883337005 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c80004290 con 0x7f9c8c0fde90 2024-04-06T14:38:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.016+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8c10dd00 con 0x7f9c8c0fde90 2024-04-06T14:38:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.016+0000 7f9c7e7fc700 1 -- 172.21.15.32:0/1883337005 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c8001c3f0 con 0x7f9c8c0fde90 2024-04-06T14:38:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.016+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8c10e240 con 0x7f9c8c0fde90 2024-04-06T14:38:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.017+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8c065e30 con 0x7f9c8c0fde90 2024-04-06T14:38:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.018+0000 7f9c7e7fc700 1 -- 172.21.15.32:0/1883337005 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9c8000bb70 con 0x7f9c8c0fde90 2024-04-06T14:38:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.018+0000 7f9c7e7fc700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c7806cc30 0x7f9c7806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.018+0000 7f9c7e7fc700 1 -- 172.21.15.32:0/1883337005 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c8008d1f0 con 0x7f9c8c0fde90 2024-04-06T14:38:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.018+0000 7f9c9147d700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c7806cc30 0x7f9c7806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.019+0000 7f9c9147d700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c7806cc30 0x7f9c7806f0e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009fd0 tx=0x7f9c88009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:32.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.023+0000 7f9c7e7fc700 1 -- 172.21.15.32:0/1883337005 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9c8005ac40 con 0x7f9c8c0fde90 2024-04-06T14:38:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.245+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c8c109df0 con 0x7f9c7806cc30 2024-04-06T14:38:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.247+0000 7f9c7e7fc700 1 -- 172.21.15.32:0/1883337005 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9c8c109df0 con 0x7f9c7806cc30 2024-04-06T14:38:32.241 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:38:32.241 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:38:32.241 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:38:32.241 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:38:32.241 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:38:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:38:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:38:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:38:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:38:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:38:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:38:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:38:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:38:32.242 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:38:32.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c7806cc30 msgr2=0x7f9c7806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:32.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c7806cc30 0x7f9c7806f0e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009fd0 tx=0x7f9c88009f20 comp rx=0 tx=0).stop 2024-04-06T14:38:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 msgr2=0x7f9c8c113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 0x7f9c8c113b00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009380 tx=0x7f9c80009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 shutdown_connections 2024-04-06T14:38:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c7806cc30 0x7f9c7806f0e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c8c0fde90 0x7f9c8c113b00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 --2- 172.21.15.32:0/1883337005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c8c0fe830 0x7f9c8c114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 >> 172.21.15.32:0/1883337005 conn(0x7f9c8c072bc0 msgr2=0x7f9c8c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 shutdown_connections 2024-04-06T14:38:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.249+0000 7f9c93ee2700 1 -- 172.21.15.32:0/1883337005 wait complete. 2024-04-06T14:38:32.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.352+0000 7fc9c6983700 1 -- 172.21.15.32:0/1600874213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0102710 msgr2=0x7fc9c00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:32.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.352+0000 7fc9c6983700 1 --2- 172.21.15.32:0/1600874213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0102710 0x7fc9c00fbd80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4004940 tx=0x7fc9b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:32.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.352+0000 7fc9c6983700 1 -- 172.21.15.32:0/1600874213 shutdown_connections 2024-04-06T14:38:32.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.352+0000 7fc9c6983700 1 --2- 172.21.15.32:0/1600874213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0102710 0x7fc9c00fbd80 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.352+0000 7fc9c6983700 1 --2- 172.21.15.32:0/1600874213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c0101d70 0x7fc9c0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.352+0000 7fc9c6983700 1 -- 172.21.15.32:0/1600874213 >> 172.21.15.32:0/1600874213 conn(0x7fc9c0072bc0 msgr2=0x7fc9c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:32.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.352+0000 7fc9c6983700 1 -- 172.21.15.32:0/1600874213 shutdown_connections 2024-04-06T14:38:32.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.352+0000 7fc9c6983700 1 -- 172.21.15.32:0/1600874213 wait complete. 2024-04-06T14:38:32.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.352+0000 7fc9c6983700 1 Processor -- start 2024-04-06T14:38:32.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9c6983700 1 -- start start 2024-04-06T14:38:32.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9c6983700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0101d70 0x7fc9c0112980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:32.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9c6983700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c0102710 0x7fc9c0112ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:32.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9c6983700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c0113510 con 0x7fc9c0101d70 2024-04-06T14:38:32.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9c6983700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c010c900 con 0x7fc9c0102710 2024-04-06T14:38:32.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0101d70 0x7fc9c0112980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:32.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0101d70 0x7fc9c0112980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43918/0 (socket says 172.21.15.32:43918) 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9bffff700 1 -- 172.21.15.32:0/2043399481 learned_addr learned my addr 172.21.15.32:0/2043399481 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9bf7fe700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c0102710 0x7fc9c0112ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9bffff700 1 -- 172.21.15.32:0/2043399481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c0102710 msgr2=0x7fc9c0112ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9bffff700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c0102710 0x7fc9c0112ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9bffff700 1 -- 172.21.15.32:0/2043399481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4005040 con 0x7fc9c0101d70 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.353+0000 7fc9bffff700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0101d70 0x7fc9c0112980 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000eb70 tx=0x7fc9b000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.354+0000 7fc9bcff9700 1 -- 172.21.15.32:0/2043399481 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9b0007cc0 con 0x7fc9c0101d70 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.354+0000 7fc9bcff9700 1 -- 172.21.15.32:0/2043399481 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9b000f040 con 0x7fc9c0101d70 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.354+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c010cb80 con 0x7fc9c0101d70 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.354+0000 7fc9bcff9700 1 -- 172.21.15.32:0/2043399481 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9b000c9f0 con 0x7fc9c0101d70 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.354+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c010d110 con 0x7fc9c0101d70 2024-04-06T14:38:32.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.355+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c0104240 con 0x7fc9c0101d70 2024-04-06T14:38:32.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.355+0000 7fc9bcff9700 1 -- 172.21.15.32:0/2043399481 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc9b000cb50 con 0x7fc9c0101d70 2024-04-06T14:38:32.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.356+0000 7fc9bcff9700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc9a806cc30 0x7fc9a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:38:32.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.356+0000 7fc9bcff9700 1 -- 172.21.15.32:0/2043399481 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc9b008bb70 con 0x7fc9c0101d70 2024-04-06T14:38:32.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.356+0000 7fc9bf7fe700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc9a806cc30 0x7fc9a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:38:32.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.356+0000 7fc9bf7fe700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc9a806cc30 0x7fc9a806f0e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc9c010dbe0 tx=0x7fc9b4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:38:32.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.360+0000 7fc9bcff9700 1 -- 172.21.15.32:0/2043399481 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc9b0059640 con 0x7fc9c0101d70 2024-04-06T14:38:32.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:32 smithi086 ceph-mon[110975]: pgmap v551: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:32.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:32 smithi086 ceph-mon[110975]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:32.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:32 smithi086 ceph-mon[110975]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:32.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:32 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4125880575' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:38:32.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:32 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/255173534' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:38:32.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.655+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc9c010dfd0 con 0x7fc9c0101d70 2024-04-06T14:38:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.655+0000 7fc9bcff9700 1 -- 172.21.15.32:0/2043399481 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fc9b00591d0 con 0x7fc9c0101d70 2024-04-06T14:38:32.651 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:38:32.651 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:38:32.651 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:38:32.652 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:38:32.652 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:38:32.652 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:38:32.652 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:38:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc9a806cc30 msgr2=0x7fc9a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:32.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc9a806cc30 0x7fc9a806f0e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc9c010dbe0 tx=0x7fc9b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:38:32.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0101d70 msgr2=0x7fc9c0112980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:38:32.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0101d70 0x7fc9c0112980 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000eb70 tx=0x7fc9b000b040 comp rx=0 tx=0).stop 2024-04-06T14:38:32.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 shutdown_connections 2024-04-06T14:38:32.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc9a806cc30 0x7fc9a806f0e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c0101d70 0x7fc9c0112980 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 --2- 172.21.15.32:0/2043399481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c0102710 0x7fc9c0112ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:38:32.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 >> 172.21.15.32:0/2043399481 conn(0x7fc9c0072bc0 msgr2=0x7fc9c00fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:38:32.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 shutdown_connections 2024-04-06T14:38:32.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:38:32.657+0000 7fc9c6983700 1 -- 172.21.15.32:0/2043399481 wait complete. 2024-04-06T14:38:32.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:32 smithi032 ceph-mon[147414]: pgmap v551: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:32.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:32 smithi032 ceph-mon[147414]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:32.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:32 smithi032 ceph-mon[147414]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:32.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:32 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4125880575' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:38:32.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:32 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/255173534' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:38:33.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:33 smithi086 ceph-mon[110975]: from='client.35056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:33.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:33 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2043399481' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:38:33.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:33 smithi032 ceph-mon[147414]: from='client.35056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:38:33.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:33 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2043399481' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:38:34.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:34 smithi086 ceph-mon[110975]: pgmap v552: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:34.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:34 smithi032 ceph-mon[147414]: pgmap v552: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:36.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:36 smithi086 ceph-mon[110975]: pgmap v553: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:36.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:36 smithi032 ceph-mon[147414]: pgmap v553: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:38.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:38 smithi086 ceph-mon[110975]: pgmap v554: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:38.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:38:38.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:38 smithi032 ceph-mon[147414]: pgmap v554: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:38.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:38:40.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:40 smithi086 ceph-mon[110975]: pgmap v555: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:40.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:40 smithi032 ceph-mon[147414]: pgmap v555: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:42.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:42 smithi086 ceph-mon[110975]: pgmap v556: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:42.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:42 smithi032 ceph-mon[147414]: pgmap v556: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:44.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:44 smithi086 ceph-mon[110975]: pgmap v557: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:44.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:44 smithi032 ceph-mon[147414]: pgmap v557: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:46.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:46 smithi086 ceph-mon[110975]: pgmap v558: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:46.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:46 smithi032 ceph-mon[147414]: pgmap v558: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:48.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:48 smithi086 ceph-mon[110975]: pgmap v559: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:48.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:48 smithi032 ceph-mon[147414]: pgmap v559: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:50.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:50 smithi086 ceph-mon[110975]: pgmap v560: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:50.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:50 smithi032 ceph-mon[147414]: pgmap v560: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:52.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:52 smithi086 ceph-mon[110975]: pgmap v561: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:52.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:52 smithi032 ceph-mon[147414]: pgmap v561: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:53.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:38:53.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:38:54.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:54 smithi032 ceph-mon[147414]: pgmap v562: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:54.873 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:54 smithi086 ceph-mon[110975]: pgmap v562: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:56.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:56 smithi032 ceph-mon[147414]: pgmap v563: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:56.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:56 smithi086 ceph-mon[110975]: pgmap v563: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:58.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:38:58 smithi032 ceph-mon[147414]: pgmap v564: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:38:58.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:38:58 smithi086 ceph-mon[110975]: pgmap v564: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:00.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:00 smithi032 ceph-mon[147414]: pgmap v565: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:00.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:00 smithi086 ceph-mon[110975]: pgmap v565: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:02.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:02 smithi032 ceph-mon[147414]: pgmap v566: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:02.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.775+0000 7f21e68ac700 1 -- 172.21.15.32:0/1144578905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 msgr2=0x7f21e0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.775+0000 7f21e68ac700 1 --2- 172.21.15.32:0/1144578905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 0x7f21e0106f00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f21d4004940 tx=0x7f21d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.776+0000 7f21e68ac700 1 -- 172.21.15.32:0/1144578905 shutdown_connections 2024-04-06T14:39:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.776+0000 7f21e68ac700 1 --2- 172.21.15.32:0/1144578905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 0x7f21e0106f00 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.776+0000 7f21e68ac700 1 --2- 172.21.15.32:0/1144578905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e00fde90 0x7f21e00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.776+0000 7f21e68ac700 1 -- 172.21.15.32:0/1144578905 >> 172.21.15.32:0/1144578905 conn(0x7f21e0072bc0 msgr2=0x7f21e0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.776+0000 7f21e68ac700 1 -- 172.21.15.32:0/1144578905 shutdown_connections 2024-04-06T14:39:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.776+0000 7f21e68ac700 1 -- 172.21.15.32:0/1144578905 wait complete. 2024-04-06T14:39:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.776+0000 7f21e68ac700 1 Processor -- start 2024-04-06T14:39:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.776+0000 7f21e68ac700 1 -- start start 2024-04-06T14:39:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21e68ac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e00fde90 0x7f21e00fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21e68ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 0x7f21e00fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21e68ac700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00fc820 con 0x7f21e00fe830 2024-04-06T14:39:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21e68ac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00fc960 con 0x7f21e00fde90 2024-04-06T14:39:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21df7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 0x7f21e00fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21df7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 0x7f21e00fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56764/0 (socket says 172.21.15.32:56764) 2024-04-06T14:39:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21df7fe700 1 -- 172.21.15.32:0/2197621730 learned_addr learned my addr 172.21.15.32:0/2197621730 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21df7fe700 1 -- 172.21.15.32:0/2197621730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e00fde90 msgr2=0x7f21e00fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21df7fe700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e00fde90 0x7f21e00fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21df7fe700 1 -- 172.21.15.32:0/2197621730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d4005040 con 0x7f21e00fe830 2024-04-06T14:39:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21df7fe700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 0x7f21e00fc290 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f21d40093b0 tx=0x7f21d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.777+0000 7f21dcff9700 1 -- 172.21.15.32:0/2197621730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d400e040 con 0x7f21e00fe830 2024-04-06T14:39:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.778+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e00fcbb0 con 0x7f21e00fe830 2024-04-06T14:39:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.778+0000 7f21dcff9700 1 -- 172.21.15.32:0/2197621730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21d4004290 con 0x7f21e00fe830 2024-04-06T14:39:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.778+0000 7f21dcff9700 1 -- 172.21.15.32:0/2197621730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d401c3f0 con 0x7f21e00fe830 2024-04-06T14:39:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.778+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e011a190 con 0x7f21e00fe830 2024-04-06T14:39:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.779+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e0065e30 con 0x7f21e00fe830 2024-04-06T14:39:02.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.779+0000 7f21dcff9700 1 -- 172.21.15.32:0/2197621730 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f21d4013790 con 0x7f21e00fe830 2024-04-06T14:39:02.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.780+0000 7f21dcff9700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f21c806cc30 0x7f21c806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:02.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.780+0000 7f21dcff9700 1 -- 172.21.15.32:0/2197621730 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f21d408d2f0 con 0x7f21e00fe830 2024-04-06T14:39:02.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.780+0000 7f21dffff700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f21c806cc30 0x7f21c806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:02.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.780+0000 7f21dffff700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f21c806cc30 0x7f21c806f0e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f21d0004b70 tx=0x7f21d0007cc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:02.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:02.785+0000 7f21dcff9700 1 -- 172.21.15.32:0/2197621730 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f21d4057330 con 0x7f21e00fe830 2024-04-06T14:39:02.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:02 smithi086 ceph-mon[110975]: pgmap v566: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.008+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21e0109e40 con 0x7f21c806cc30 2024-04-06T14:39:03.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.009+0000 7f21dcff9700 1 -- 172.21.15.32:0/2197621730 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f21e0109e40 con 0x7f21c806cc30 2024-04-06T14:39:03.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.011+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f21c806cc30 msgr2=0x7f21c806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.011+0000 7f21e68ac700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f21c806cc30 0x7f21c806f0e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f21d0004b70 tx=0x7f21d0007cc0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.012+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 msgr2=0x7f21e00fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.012+0000 7f21e68ac700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 0x7f21e00fc290 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f21d40093b0 tx=0x7f21d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:03.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.012+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 shutdown_connections 2024-04-06T14:39:03.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.012+0000 7f21e68ac700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f21c806cc30 0x7f21c806f0e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.012+0000 7f21e68ac700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e00fde90 0x7f21e00fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.012+0000 7f21e68ac700 1 --2- 172.21.15.32:0/2197621730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21e00fe830 0x7f21e00fc290 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.012+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 >> 172.21.15.32:0/2197621730 conn(0x7f21e0072bc0 msgr2=0x7f21e0103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:03.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.012+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 shutdown_connections 2024-04-06T14:39:03.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.012+0000 7f21e68ac700 1 -- 172.21.15.32:0/2197621730 wait complete. 2024-04-06T14:39:03.016 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:39:03.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.115+0000 7fc374efc700 1 -- 172.21.15.32:0/1797952436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 msgr2=0x7fc3700fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.115+0000 7fc374efc700 1 --2- 172.21.15.32:0/1797952436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 0x7fc3700fe1d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc360004940 tx=0x7fc360009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:03.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.115+0000 7fc374efc700 1 -- 172.21.15.32:0/1797952436 shutdown_connections 2024-04-06T14:39:03.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.115+0000 7fc374efc700 1 --2- 172.21.15.32:0/1797952436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3700fe7a0 0x7fc370106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.115+0000 7fc374efc700 1 --2- 172.21.15.32:0/1797952436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 0x7fc3700fe1d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.115+0000 7fc374efc700 1 -- 172.21.15.32:0/1797952436 >> 172.21.15.32:0/1797952436 conn(0x7fc3700f78d0 msgr2=0x7fc3700f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc374efc700 1 -- 172.21.15.32:0/1797952436 shutdown_connections 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc374efc700 1 -- 172.21.15.32:0/1797952436 wait complete. 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc374efc700 1 Processor -- start 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc374efc700 1 -- start start 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc374efc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 0x7fc3700fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc374efc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3700fe7a0 0x7fc3700fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc374efc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700fc700 con 0x7fc3700fde00 2024-04-06T14:39:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc374efc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700fc840 con 0x7fc3700fe7a0 2024-04-06T14:39:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc36e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 0x7fc3700fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc36e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 0x7fc3700fbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56774/0 (socket says 172.21.15.32:56774) 2024-04-06T14:39:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.116+0000 7fc36e59c700 1 -- 172.21.15.32:0/823487640 learned_addr learned my addr 172.21.15.32:0/823487640 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc36dd9b700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3700fe7a0 0x7fc3700fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc36e59c700 1 -- 172.21.15.32:0/823487640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3700fe7a0 msgr2=0x7fc3700fc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc36e59c700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3700fe7a0 0x7fc3700fc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc36e59c700 1 -- 172.21.15.32:0/823487640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc360005040 con 0x7fc3700fde00 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc36dd9b700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3700fe7a0 0x7fc3700fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc36e59c700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 0x7fc3700fbc30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc360004890 tx=0x7fc360009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc35effd700 1 -- 172.21.15.32:0/823487640 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc36000e040 con 0x7fc3700fde00 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc35effd700 1 -- 172.21.15.32:0/823487640 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc360009af0 con 0x7fc3700fde00 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3700fca90 con 0x7fc3700fde00 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc35effd700 1 -- 172.21.15.32:0/823487640 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc36001c3f0 con 0x7fc3700fde00 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.117+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37011a180 con 0x7fc3700fde00 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.118+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc370066770 con 0x7fc3700fde00 2024-04-06T14:39:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.119+0000 7fc35effd700 1 -- 172.21.15.32:0/823487640 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc360013420 con 0x7fc3700fde00 2024-04-06T14:39:03.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.119+0000 7fc35effd700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc35806cc20 0x7fc35806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:03.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.119+0000 7fc35effd700 1 -- 172.21.15.32:0/823487640 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc36008d2d0 con 0x7fc3700fde00 2024-04-06T14:39:03.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.119+0000 7fc36dd9b700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc35806cc20 0x7fc35806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:03.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.120+0000 7fc36dd9b700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc35806cc20 0x7fc35806f0d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc364007fd0 tx=0x7fc364009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:03.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.124+0000 7fc35effd700 1 -- 172.21.15.32:0/823487640 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc3600573d0 con 0x7fc3700fde00 2024-04-06T14:39:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.345+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc37004bf70 con 0x7fc35806cc20 2024-04-06T14:39:03.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.346+0000 7fc35effd700 1 -- 172.21.15.32:0/823487640 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc364004dd0 con 0x7fc35806cc20 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc35806cc20 msgr2=0x7fc35806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc35806cc20 0x7fc35806f0d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc364007fd0 tx=0x7fc364009040 comp rx=0 tx=0).stop 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 msgr2=0x7fc3700fbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 0x7fc3700fbc30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc360004890 tx=0x7fc360009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 shutdown_connections 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc35806cc20 0x7fc35806f0d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3700fde00 0x7fc3700fbc30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 --2- 172.21.15.32:0/823487640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3700fe7a0 0x7fc3700fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 >> 172.21.15.32:0/823487640 conn(0x7fc3700f78d0 msgr2=0x7fc3700f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 shutdown_connections 2024-04-06T14:39:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.348+0000 7fc374efc700 1 -- 172.21.15.32:0/823487640 wait complete. 2024-04-06T14:39:03.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.452+0000 7f6eec3f4700 1 -- 172.21.15.32:0/1468236688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fe7b0 msgr2=0x7f6ee4106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.452+0000 7f6eec3f4700 1 --2- 172.21.15.32:0/1468236688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fe7b0 0x7f6ee4106df0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4004940 tx=0x7f6ed4009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 -- 172.21.15.32:0/1468236688 shutdown_connections 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 --2- 172.21.15.32:0/1468236688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fe7b0 0x7f6ee4106df0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 --2- 172.21.15.32:0/1468236688 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ee40fde10 0x7f6ee40fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 -- 172.21.15.32:0/1468236688 >> 172.21.15.32:0/1468236688 conn(0x7f6ee40f7880 msgr2=0x7f6ee40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 -- 172.21.15.32:0/1468236688 shutdown_connections 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 -- 172.21.15.32:0/1468236688 wait complete. 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 Processor -- start 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 -- start start 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fde10 0x7f6ee40fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ee40fe7b0 0x7f6ee40fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40fc750 con 0x7f6ee40fde10 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.453+0000 7f6eec3f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40fc890 con 0x7f6ee40fe7b0 2024-04-06T14:39:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6eea190700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fde10 0x7f6ee40fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6eea190700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fde10 0x7f6ee40fbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56798/0 (socket says 172.21.15.32:56798) 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6eea190700 1 -- 172.21.15.32:0/2541757002 learned_addr learned my addr 172.21.15.32:0/2541757002 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6ee998f700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ee40fe7b0 0x7f6ee40fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6eea190700 1 -- 172.21.15.32:0/2541757002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ee40fe7b0 msgr2=0x7f6ee40fc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6eea190700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ee40fe7b0 0x7f6ee40fc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6eea190700 1 -- 172.21.15.32:0/2541757002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed4005040 con 0x7f6ee40fde10 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6eea190700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fde10 0x7f6ee40fbc40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000b790 tx=0x7f6ee0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6edaffd700 1 -- 172.21.15.32:0/2541757002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ee000f770 con 0x7f6ee40fde10 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6edaffd700 1 -- 172.21.15.32:0/2541757002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ee000fd80 con 0x7f6ee40fde10 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee411a0d0 con 0x7f6ee40fde10 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6edaffd700 1 -- 172.21.15.32:0/2541757002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ee000d310 con 0x7f6ee40fde10 2024-04-06T14:39:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.454+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee411a530 con 0x7f6ee40fde10 2024-04-06T14:39:03.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.455+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee4066770 con 0x7f6ee40fde10 2024-04-06T14:39:03.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.456+0000 7f6edaffd700 1 -- 172.21.15.32:0/2541757002 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6ee0010420 con 0x7f6ee40fde10 2024-04-06T14:39:03.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.456+0000 7f6edaffd700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6ed006cc20 0x7f6ed006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:03.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.456+0000 7f6edaffd700 1 -- 172.21.15.32:0/2541757002 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6ee008bf30 con 0x7f6ee40fde10 2024-04-06T14:39:03.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.456+0000 7f6ee998f700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6ed006cc20 0x7f6ed006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:03.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.457+0000 7f6ee998f700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6ed006cc20 0x7f6ed006f0d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400b3b0 tx=0x7f6ed4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:03.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.461+0000 7f6edaffd700 1 -- 172.21.15.32:0/2541757002 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6ee00bfab0 con 0x7f6ee40fde10 2024-04-06T14:39:03.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.671+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ee40fd2e0 con 0x7f6ed006cc20 2024-04-06T14:39:03.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.681+0000 7f6edaffd700 1 -- 172.21.15.32:0/2541757002 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6ee40fd2e0 con 0x7f6ed006cc20 2024-04-06T14:39:03.675 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:39:03.675 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (19m) 12m ago 30m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (16m) 12m ago 30m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (16m) 12m ago 29m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (19m) 12m ago 30m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (13m) 12m ago 24m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (12m) 12m ago 24m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (12m) 12m ago 24m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (24m) 12m ago 24m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (21m) 12m ago 32m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (20m) 12m ago 29m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (17m) 12m ago 32m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (17m) 12m ago 29m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (20m) 12m ago 30m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (20m) 12m ago 29m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:39:03.676 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (16m) 12m ago 28m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:39:03.677 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (15m) 12m ago 28m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:39:03.677 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (15m) 12m ago 27m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:39:03.677 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (14m) 12m ago 27m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:39:03.677 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (13m) 12m ago 27m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:39:03.677 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (13m) 12m ago 26m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:39:03.677 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (19m) 12m ago 30m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6ed006cc20 msgr2=0x7f6ed006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6ed006cc20 0x7f6ed006f0d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400b3b0 tx=0x7f6ed4009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fde10 msgr2=0x7f6ee40fbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fde10 0x7f6ee40fbc40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000b790 tx=0x7f6ee0009250 comp rx=0 tx=0).stop 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 shutdown_connections 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6ed006cc20 0x7f6ed006f0d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee40fde10 0x7f6ee40fbc40 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 --2- 172.21.15.32:0/2541757002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ee40fe7b0 0x7f6ee40fc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 >> 172.21.15.32:0/2541757002 conn(0x7f6ee40f7880 msgr2=0x7f6ee40f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 shutdown_connections 2024-04-06T14:39:03.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.683+0000 7f6eec3f4700 1 -- 172.21.15.32:0/2541757002 wait complete. 2024-04-06T14:39:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.785+0000 7f77a3e26700 1 -- 172.21.15.32:0/1852973406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c0fb4a0 msgr2=0x7f779c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.785+0000 7f77a3e26700 1 --2- 172.21.15.32:0/1852973406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c0fb4a0 0x7f779c0fb910 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7798004940 tx=0x7f7798009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.785+0000 7f77a3e26700 1 -- 172.21.15.32:0/1852973406 shutdown_connections 2024-04-06T14:39:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.785+0000 7f77a3e26700 1 --2- 172.21.15.32:0/1852973406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c0fb4a0 0x7f779c0fb910 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.785+0000 7f77a3e26700 1 --2- 172.21.15.32:0/1852973406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f779c102e70 0x7f779c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.785+0000 7f77a3e26700 1 -- 172.21.15.32:0/1852973406 >> 172.21.15.32:0/1852973406 conn(0x7f779c0714c0 msgr2=0x7f779c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.785+0000 7f77a3e26700 1 -- 172.21.15.32:0/1852973406 shutdown_connections 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.785+0000 7f77a3e26700 1 -- 172.21.15.32:0/1852973406 wait complete. 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a3e26700 1 Processor -- start 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a3e26700 1 -- start start 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a3e26700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f779c0fb4a0 0x7f779c115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a3e26700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c102e70 0x7f779c1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a3e26700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c116880 con 0x7f779c102e70 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a3e26700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c10fbc0 con 0x7f779c0fb4a0 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a13c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c102e70 0x7f779c1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a13c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c102e70 0x7f779c1161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56824/0 (socket says 172.21.15.32:56824) 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a13c1700 1 -- 172.21.15.32:0/2561179513 learned_addr learned my addr 172.21.15.32:0/2561179513 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a1bc2700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f779c0fb4a0 0x7f779c115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:03.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a13c1700 1 -- 172.21.15.32:0/2561179513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f779c0fb4a0 msgr2=0x7f779c115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a13c1700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f779c0fb4a0 0x7f779c115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a13c1700 1 -- 172.21.15.32:0/2561179513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798005040 con 0x7f779c102e70 2024-04-06T14:39:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.786+0000 7f77a1bc2700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f779c0fb4a0 0x7f779c115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:39:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.787+0000 7f77a13c1700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c102e70 0x7f779c1161a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7798009180 tx=0x7f7798009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.787+0000 7f77927fc700 1 -- 172.21.15.32:0/2561179513 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f779800e040 con 0x7f779c102e70 2024-04-06T14:39:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.787+0000 7f77927fc700 1 -- 172.21.15.32:0/2561179513 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7798004290 con 0x7f779c102e70 2024-04-06T14:39:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.787+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c10fde0 con 0x7f779c102e70 2024-04-06T14:39:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.787+0000 7f77927fc700 1 -- 172.21.15.32:0/2561179513 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f779801c3f0 con 0x7f779c102e70 2024-04-06T14:39:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.787+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779c1103a0 con 0x7f779c102e70 2024-04-06T14:39:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.788+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779c107290 con 0x7f779c102e70 2024-04-06T14:39:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.788+0000 7f77927fc700 1 -- 172.21.15.32:0/2561179513 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7798013790 con 0x7f779c102e70 2024-04-06T14:39:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.789+0000 7f77927fc700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f778806cc20 0x7f778806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.789+0000 7f77927fc700 1 -- 172.21.15.32:0/2561179513 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f779808d230 con 0x7f779c102e70 2024-04-06T14:39:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.789+0000 7f77a1bc2700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f778806cc20 0x7f778806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.789+0000 7f77a1bc2700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f778806cc20 0x7f778806f0d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f778c006fd0 tx=0x7f778c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:03.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:03.793+0000 7f77927fc700 1 -- 172.21.15.32:0/2561179513 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f779805adc0 con 0x7f779c102e70 2024-04-06T14:39:04.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.079+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f779c065380 con 0x7f779c102e70 2024-04-06T14:39:04.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.079+0000 7f77927fc700 1 -- 172.21.15.32:0/2561179513 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f7798013420 con 0x7f779c102e70 2024-04-06T14:39:04.074 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:39:04.074 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:39:04.074 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:39:04.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:39:04.074 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:39:04.074 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:39:04.075 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:39:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.081+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f778806cc20 msgr2=0x7f778806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.081+0000 7f77a3e26700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f778806cc20 0x7f778806f0d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f778c006fd0 tx=0x7f778c008040 comp rx=0 tx=0).stop 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.081+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c102e70 msgr2=0x7f779c1161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.081+0000 7f77a3e26700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c102e70 0x7f779c1161a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7798009180 tx=0x7f7798009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.082+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 shutdown_connections 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.082+0000 7f77a3e26700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f778806cc20 0x7f778806f0d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.082+0000 7f77a3e26700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f779c0fb4a0 0x7f779c115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.082+0000 7f77a3e26700 1 --2- 172.21.15.32:0/2561179513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f779c102e70 0x7f779c1161a0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.082+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 >> 172.21.15.32:0/2561179513 conn(0x7f779c0714c0 msgr2=0x7f779c0fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.082+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 shutdown_connections 2024-04-06T14:39:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.082+0000 7f77a3e26700 1 -- 172.21.15.32:0/2561179513 wait complete. 2024-04-06T14:39:04.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 -- 172.21.15.32:0/1482795377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c102e70 msgr2=0x7fd89c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 --2- 172.21.15.32:0/1482795377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c102e70 0x7fd89c103240 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd88c004940 tx=0x7fd88c009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:04.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 -- 172.21.15.32:0/1482795377 shutdown_connections 2024-04-06T14:39:04.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 --2- 172.21.15.32:0/1482795377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd89c0fb4a0 0x7fd89c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 --2- 172.21.15.32:0/1482795377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c102e70 0x7fd89c103240 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 -- 172.21.15.32:0/1482795377 >> 172.21.15.32:0/1482795377 conn(0x7fd89c0714c0 msgr2=0x7fd89c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 -- 172.21.15.32:0/1482795377 shutdown_connections 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 -- 172.21.15.32:0/1482795377 wait complete. 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 Processor -- start 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.185+0000 7fd8a0898700 1 -- start start 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd8a0898700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c0fb4a0 0x7fd89c113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd8a0898700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd89c102e70 0x7fd89c114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd8a0898700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c1146e0 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd8a0898700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c10dad0 con 0x7fd89c102e70 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd89a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c0fb4a0 0x7fd89c113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd89a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c0fb4a0 0x7fd89c113b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56840/0 (socket says 172.21.15.32:56840) 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd89a59c700 1 -- 172.21.15.32:0/2691464726 learned_addr learned my addr 172.21.15.32:0/2691464726 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:04.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd89a59c700 1 -- 172.21.15.32:0/2691464726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd89c102e70 msgr2=0x7fd89c114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd899d9b700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd89c102e70 0x7fd89c114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd89a59c700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd89c102e70 0x7fd89c114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd89a59c700 1 -- 172.21.15.32:0/2691464726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c005040 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd899d9b700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd89c102e70 0x7fd89c114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd89a59c700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c0fb4a0 0x7fd89c113b50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00b3b0 tx=0x7fd88c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd892ffd700 1 -- 172.21.15.32:0/2691464726 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd88c00e040 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd892ffd700 1 -- 172.21.15.32:0/2691464726 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd88c013420 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd892ffd700 1 -- 172.21.15.32:0/2691464726 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd88c01c630 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89c10dd50 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.186+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89c10e290 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.187+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89c105320 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.188+0000 7fd892ffd700 1 -- 172.21.15.32:0/2691464726 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd88c013970 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.188+0000 7fd892ffd700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd88806cc20 0x7fd88806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:04.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.188+0000 7fd892ffd700 1 -- 172.21.15.32:0/2691464726 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd88c08d180 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.188+0000 7fd899d9b700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd88806cc20 0x7fd88806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:04.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.189+0000 7fd899d9b700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd88806cc20 0x7fd88806f0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd884009fd0 tx=0x7fd88400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:04.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.194+0000 7fd892ffd700 1 -- 172.21.15.32:0/2691464726 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd88c05ad10 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.440+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd89c10f1a0 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.441+0000 7fd892ffd700 1 -- 172.21.15.32:0/2691464726 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fd88c05a8a0 con 0x7fd89c0fb4a0 2024-04-06T14:39:04.435 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:39:04.435 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:39:04.436 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:39:04.437 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:39:04.438 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:39:04.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd88806cc20 msgr2=0x7fd88806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd88806cc20 0x7fd88806f0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd884009fd0 tx=0x7fd88400b040 comp rx=0 tx=0).stop 2024-04-06T14:39:04.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c0fb4a0 msgr2=0x7fd89c113b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c0fb4a0 0x7fd89c113b50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00b3b0 tx=0x7fd88c009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:04.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 shutdown_connections 2024-04-06T14:39:04.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd88806cc20 0x7fd88806f0d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd89c0fb4a0 0x7fd89c113b50 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 --2- 172.21.15.32:0/2691464726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd89c102e70 0x7fd89c114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 >> 172.21.15.32:0/2691464726 conn(0x7fd89c0714c0 msgr2=0x7fd89c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:04.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 shutdown_connections 2024-04-06T14:39:04.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.443+0000 7fd8a0898700 1 -- 172.21.15.32:0/2691464726 wait complete. 2024-04-06T14:39:04.440 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:39:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 -- 172.21.15.32:0/2292062633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08102710 msgr2=0x7fbf080fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 --2- 172.21.15.32:0/2292062633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08102710 0x7fbf080fbd80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbefc004940 tx=0x7fbefc009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 -- 172.21.15.32:0/2292062633 shutdown_connections 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 --2- 172.21.15.32:0/2292062633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08102710 0x7fbf080fbd80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 --2- 172.21.15.32:0/2292062633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf08101d70 0x7fbf08102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 -- 172.21.15.32:0/2292062633 >> 172.21.15.32:0/2292062633 conn(0x7fbf08072bc0 msgr2=0x7fbf08072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 -- 172.21.15.32:0/2292062633 shutdown_connections 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 -- 172.21.15.32:0/2292062633 wait complete. 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 Processor -- start 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.548+0000 7fbf0fadb700 1 -- start start 2024-04-06T14:39:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0fadb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08101d70 0x7fbf08113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0fadb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf08102710 0x7fbf08114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0fadb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf08114670 con 0x7fbf08101d70 2024-04-06T14:39:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0fadb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0810da60 con 0x7fbf08102710 2024-04-06T14:39:04.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0d877700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08101d70 0x7fbf08113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:04.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0d877700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08101d70 0x7fbf08113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42884/0 (socket says 172.21.15.32:42884) 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0d877700 1 -- 172.21.15.32:0/4268129460 learned_addr learned my addr 172.21.15.32:0/4268129460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0d076700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf08102710 0x7fbf08114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0d877700 1 -- 172.21.15.32:0/4268129460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf08102710 msgr2=0x7fbf08114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0d877700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf08102710 0x7fbf08114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0d877700 1 -- 172.21.15.32:0/4268129460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc005040 con 0x7fbf08101d70 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0d076700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf08102710 0x7fbf08114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0d877700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08101d70 0x7fbf08113ae0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c830 tx=0x7fbf04004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbefa7fc700 1 -- 172.21.15.32:0/4268129460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf0400a760 con 0x7fbf08101d70 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0810dd40 con 0x7fbf08101d70 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbefa7fc700 1 -- 172.21.15.32:0/4268129460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf0400ad70 con 0x7fbf08101d70 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbefa7fc700 1 -- 172.21.15.32:0/4268129460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf040113a0 con 0x7fbf08101d70 2024-04-06T14:39:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.549+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0810e300 con 0x7fbf08101d70 2024-04-06T14:39:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.550+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf080fd4c0 con 0x7fbf08101d70 2024-04-06T14:39:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.551+0000 7fbefa7fc700 1 -- 172.21.15.32:0/4268129460 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbf04012400 con 0x7fbf08101d70 2024-04-06T14:39:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.551+0000 7fbefa7fc700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbef406cc20 0x7fbef406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.551+0000 7fbefa7fc700 1 -- 172.21.15.32:0/4268129460 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbf0408c0b0 con 0x7fbf08101d70 2024-04-06T14:39:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.551+0000 7fbf0d076700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbef406cc20 0x7fbef406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.552+0000 7fbf0d076700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbef406cc20 0x7fbef406f0d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00b3b0 tx=0x7fbefc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.556+0000 7fbefa7fc700 1 -- 172.21.15.32:0/4268129460 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbf04059b80 con 0x7fbf08101d70 2024-04-06T14:39:04.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:04 smithi032 ceph-mon[147414]: pgmap v567: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:04.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:04 smithi032 ceph-mon[147414]: from='client.35064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:04.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:04 smithi032 ceph-mon[147414]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:04.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:04 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2561179513' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:39:04.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.776+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf08108d10 con 0x7fbef406cc20 2024-04-06T14:39:04.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.777+0000 7fbefa7fc700 1 -- 172.21.15.32:0/4268129460 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbf08108d10 con 0x7fbef406cc20 2024-04-06T14:39:04.770 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:39:04.770 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:39:04.770 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:39:04.771 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:39:04.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbef406cc20 msgr2=0x7fbef406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbef406cc20 0x7fbef406f0d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00b3b0 tx=0x7fbefc009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08101d70 msgr2=0x7fbf08113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08101d70 0x7fbf08113ae0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c830 tx=0x7fbf04004470 comp rx=0 tx=0).stop 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 shutdown_connections 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbef406cc20 0x7fbef406f0d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf08101d70 0x7fbf08113ae0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 --2- 172.21.15.32:0/4268129460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf08102710 0x7fbf08114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 >> 172.21.15.32:0/4268129460 conn(0x7fbf08072bc0 msgr2=0x7fbf08065a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 shutdown_connections 2024-04-06T14:39:04.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.779+0000 7fbf0fadb700 1 -- 172.21.15.32:0/4268129460 wait complete. 2024-04-06T14:39:04.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:04 smithi086 ceph-mon[110975]: pgmap v567: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:04.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:04 smithi086 ceph-mon[110975]: from='client.35064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:04.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:04 smithi086 ceph-mon[110975]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:04.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:04 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2561179513' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:39:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.883+0000 7f7395978700 1 -- 172.21.15.32:0/532830357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 msgr2=0x7f73900fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.883+0000 7f7395978700 1 --2- 172.21.15.32:0/532830357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 0x7f73900fce20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7380004940 tx=0x7f7380009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.883+0000 7f7395978700 1 -- 172.21.15.32:0/532830357 shutdown_connections 2024-04-06T14:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.883+0000 7f7395978700 1 --2- 172.21.15.32:0/532830357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 0x7f73900fce20 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.883+0000 7f7395978700 1 --2- 172.21.15.32:0/532830357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7390102e10 0x7f73901031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.883+0000 7f7395978700 1 -- 172.21.15.32:0/532830357 >> 172.21.15.32:0/532830357 conn(0x7f73900f89a0 msgr2=0x7f73900fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.883+0000 7f7395978700 1 -- 172.21.15.32:0/532830357 shutdown_connections 2024-04-06T14:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.883+0000 7f7395978700 1 -- 172.21.15.32:0/532830357 wait complete. 2024-04-06T14:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.883+0000 7f7395978700 1 Processor -- start 2024-04-06T14:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f7395978700 1 -- start start 2024-04-06T14:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f7395978700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7390102e10 0x7f7390113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f7395978700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 0x7f7390114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f7395978700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7390114660 con 0x7f73901037b0 2024-04-06T14:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f7395978700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739010da50 con 0x7f7390102e10 2024-04-06T14:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f738e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 0x7f7390114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f738e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 0x7f7390114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42898/0 (socket says 172.21.15.32:42898) 2024-04-06T14:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f738e7fc700 1 -- 172.21.15.32:0/91585121 learned_addr learned my addr 172.21.15.32:0/91585121 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f738effd700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7390102e10 0x7f7390113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f738e7fc700 1 -- 172.21.15.32:0/91585121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7390102e10 msgr2=0x7f7390113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f738e7fc700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7390102e10 0x7f7390113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f738e7fc700 1 -- 172.21.15.32:0/91585121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7380005040 con 0x7f73901037b0 2024-04-06T14:39:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.884+0000 7f738e7fc700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 0x7f7390114010 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f73800093b0 tx=0x7f7380009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.885+0000 7f73777fe700 1 -- 172.21.15.32:0/91585121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f738000e040 con 0x7f73901037b0 2024-04-06T14:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.885+0000 7f73777fe700 1 -- 172.21.15.32:0/91585121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7380013420 con 0x7f73901037b0 2024-04-06T14:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.885+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739010dcd0 con 0x7f73901037b0 2024-04-06T14:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.885+0000 7f73777fe700 1 -- 172.21.15.32:0/91585121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f738001c630 con 0x7f73901037b0 2024-04-06T14:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.885+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739010e290 con 0x7f73901037b0 2024-04-06T14:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.886+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73900fe560 con 0x7f73901037b0 2024-04-06T14:39:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.886+0000 7f73777fe700 1 -- 172.21.15.32:0/91585121 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f73800139e0 con 0x7f73901037b0 2024-04-06T14:39:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.887+0000 7f73777fe700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f737806cc20 0x7f737806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.887+0000 7f73777fe700 1 -- 172.21.15.32:0/91585121 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f738008d060 con 0x7f73901037b0 2024-04-06T14:39:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.887+0000 7f738effd700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f737806cc20 0x7f737806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.887+0000 7f738effd700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f737806cc20 0x7f737806f0d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7384004f40 tx=0x7f738400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:04.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:04.891+0000 7f73777fe700 1 -- 172.21.15.32:0/91585121 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f738005aca0 con 0x7f73901037b0 2024-04-06T14:39:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.177+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7390109db0 con 0x7f73901037b0 2024-04-06T14:39:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.177+0000 7f73777fe700 1 -- 172.21.15.32:0/91585121 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f738005a830 con 0x7f73901037b0 2024-04-06T14:39:05.171 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:39:05.171 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:39:05.171 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:39:05.171 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:39:05.171 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:39:05.171 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:39:05.171 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f737806cc20 msgr2=0x7f737806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f737806cc20 0x7f737806f0d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7384004f40 tx=0x7f738400b040 comp rx=0 tx=0).stop 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 msgr2=0x7f7390114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 0x7f7390114010 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f73800093b0 tx=0x7f7380009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 shutdown_connections 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f737806cc20 0x7f737806f0d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7390102e10 0x7f7390113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 --2- 172.21.15.32:0/91585121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73901037b0 0x7f7390114010 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 >> 172.21.15.32:0/91585121 conn(0x7f73900f89a0 msgr2=0x7f73900f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 shutdown_connections 2024-04-06T14:39:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:05.179+0000 7f7395978700 1 -- 172.21.15.32:0/91585121 wait complete. 2024-04-06T14:39:05.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:05 smithi032 ceph-mon[147414]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:05.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:05 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2691464726' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:39:05.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:05 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/91585121' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:39:05.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:05 smithi086 ceph-mon[110975]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:05.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:05 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2691464726' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:39:05.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:05 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/91585121' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:39:06.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:06 smithi032 ceph-mon[147414]: from='client.35084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:06.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:06 smithi032 ceph-mon[147414]: pgmap v568: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:06.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:06 smithi086 ceph-mon[110975]: from='client.35084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:06.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:06 smithi086 ceph-mon[110975]: pgmap v568: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:08.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:08 smithi032 ceph-mon[147414]: pgmap v569: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:08.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:39:08.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:08 smithi086 ceph-mon[110975]: pgmap v569: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:08.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:39:10.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:10 smithi032 ceph-mon[147414]: pgmap v570: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:10.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:10 smithi086 ceph-mon[110975]: pgmap v570: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:12.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:12 smithi032 ceph-mon[147414]: pgmap v571: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:12.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:12 smithi086 ceph-mon[110975]: pgmap v571: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:14.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:14 smithi032 ceph-mon[147414]: pgmap v572: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:14.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:14 smithi086 ceph-mon[110975]: pgmap v572: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:16.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:16 smithi032 ceph-mon[147414]: pgmap v573: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:16.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:16 smithi086 ceph-mon[110975]: pgmap v573: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:18.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:18 smithi032 ceph-mon[147414]: pgmap v574: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:18.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:18 smithi086 ceph-mon[110975]: pgmap v574: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:20.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:20 smithi032 ceph-mon[147414]: pgmap v575: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:20.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:20 smithi086 ceph-mon[110975]: pgmap v575: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:22.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:22 smithi032 ceph-mon[147414]: pgmap v576: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:22.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:22 smithi086 ceph-mon[110975]: pgmap v576: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:23.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:39:23.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:39:24.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:24 smithi032 ceph-mon[147414]: pgmap v577: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:24.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:24 smithi086 ceph-mon[110975]: pgmap v577: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:26.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:26 smithi032 ceph-mon[147414]: pgmap v578: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:26.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:26 smithi086 ceph-mon[110975]: pgmap v578: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:28.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:28 smithi032 ceph-mon[147414]: pgmap v579: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:28.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:28 smithi086 ceph-mon[110975]: pgmap v579: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:30.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:30 smithi032 ceph-mon[147414]: pgmap v580: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:30.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:30 smithi086 ceph-mon[110975]: pgmap v580: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:32.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:32 smithi032 ceph-mon[147414]: pgmap v581: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:32.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:32 smithi086 ceph-mon[110975]: pgmap v581: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:34.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:34 smithi032 ceph-mon[147414]: pgmap v582: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:34.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:34 smithi086 ceph-mon[110975]: pgmap v582: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:35.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.295+0000 7f03cd542700 1 -- 172.21.15.32:0/3092359868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 msgr2=0x7f03c80fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.295+0000 7f03cd542700 1 --2- 172.21.15.32:0/3092359868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 0x7f03c80fb940 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f03bc004940 tx=0x7f03bc009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:35.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.296+0000 7f03cd542700 1 -- 172.21.15.32:0/3092359868 shutdown_connections 2024-04-06T14:39:35.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.296+0000 7f03cd542700 1 --2- 172.21.15.32:0/3092359868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 0x7f03c80fb940 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.296+0000 7f03cd542700 1 --2- 172.21.15.32:0/3092359868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03c8102f50 0x7f03c8103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.296+0000 7f03cd542700 1 -- 172.21.15.32:0/3092359868 >> 172.21.15.32:0/3092359868 conn(0x7f03c80714c0 msgr2=0x7f03c80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:35.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.296+0000 7f03cd542700 1 -- 172.21.15.32:0/3092359868 shutdown_connections 2024-04-06T14:39:35.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.296+0000 7f03cd542700 1 -- 172.21.15.32:0/3092359868 wait complete. 2024-04-06T14:39:35.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.296+0000 7f03cd542700 1 Processor -- start 2024-04-06T14:39:35.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.296+0000 7f03cd542700 1 -- start start 2024-04-06T14:39:35.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03cd542700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 0x7f03c8113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:35.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03cd542700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03c8102f50 0x7f03c8113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03cd542700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c81144b0 con 0x7f03c80fb4d0 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03cd542700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c810dab0 con 0x7f03c8102f50 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 0x7f03c8113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 0x7f03c8113920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49258/0 (socket says 172.21.15.32:49258) 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03c6ffd700 1 -- 172.21.15.32:0/1915383144 learned_addr learned my addr 172.21.15.32:0/1915383144 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03c67fc700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03c8102f50 0x7f03c8113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03c6ffd700 1 -- 172.21.15.32:0/1915383144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03c8102f50 msgr2=0x7f03c8113e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03c6ffd700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03c8102f50 0x7f03c8113e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03c6ffd700 1 -- 172.21.15.32:0/1915383144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc005040 con 0x7f03c80fb4d0 2024-04-06T14:39:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03c6ffd700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 0x7f03c8113920 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f03b800c8b0 tx=0x7f03b8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.297+0000 7f03af7fe700 1 -- 172.21.15.32:0/1915383144 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03b800a760 con 0x7f03c80fb4d0 2024-04-06T14:39:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.298+0000 7f03af7fe700 1 -- 172.21.15.32:0/1915383144 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03b800ad70 con 0x7f03c80fb4d0 2024-04-06T14:39:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.298+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c810dd90 con 0x7f03c80fb4d0 2024-04-06T14:39:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.298+0000 7f03af7fe700 1 -- 172.21.15.32:0/1915383144 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03b80113a0 con 0x7f03c80fb4d0 2024-04-06T14:39:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.298+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c810e2d0 con 0x7f03c80fb4d0 2024-04-06T14:39:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.298+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c8105330 con 0x7f03c80fb4d0 2024-04-06T14:39:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.299+0000 7f03af7fe700 1 -- 172.21.15.32:0/1915383144 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f03b8012400 con 0x7f03c80fb4d0 2024-04-06T14:39:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.299+0000 7f03af7fe700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03b006cc20 0x7f03b006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.300+0000 7f03af7fe700 1 -- 172.21.15.32:0/1915383144 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f03b808c090 con 0x7f03c80fb4d0 2024-04-06T14:39:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.300+0000 7f03c67fc700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03b006cc20 0x7f03b006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.300+0000 7f03c67fc700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03b006cc20 0x7f03b006f0d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f03c810ec70 tx=0x7f03bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.304+0000 7f03af7fe700 1 -- 172.21.15.32:0/1915383144 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f03b8059b60 con 0x7f03c80fb4d0 2024-04-06T14:39:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.531+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f03c810efe0 con 0x7f03b006cc20 2024-04-06T14:39:35.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.533+0000 7f03af7fe700 1 -- 172.21.15.32:0/1915383144 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f03c810efe0 con 0x7f03b006cc20 2024-04-06T14:39:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03b006cc20 msgr2=0x7f03b006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03b006cc20 0x7f03b006f0d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f03c810ec70 tx=0x7f03bc009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 msgr2=0x7f03c8113920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 0x7f03c8113920 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f03b800c8b0 tx=0x7f03b8004470 comp rx=0 tx=0).stop 2024-04-06T14:39:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 shutdown_connections 2024-04-06T14:39:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f03b006cc20 0x7f03b006f0d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03c80fb4d0 0x7f03c8113920 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 --2- 172.21.15.32:0/1915383144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03c8102f50 0x7f03c8113e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 >> 172.21.15.32:0/1915383144 conn(0x7f03c80714c0 msgr2=0x7f03c80fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 shutdown_connections 2024-04-06T14:39:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.535+0000 7f03cd542700 1 -- 172.21.15.32:0/1915383144 wait complete. 2024-04-06T14:39:35.539 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:39:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 -- 172.21.15.32:0/2590554839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c4102e10 msgr2=0x7fb6c41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 --2- 172.21.15.32:0/2590554839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c4102e10 0x7fb6c41031e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8004940 tx=0x7fb6b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 -- 172.21.15.32:0/2590554839 shutdown_connections 2024-04-06T14:39:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 --2- 172.21.15.32:0/2590554839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c41037b0 0x7fb6c40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 --2- 172.21.15.32:0/2590554839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c4102e10 0x7fb6c41031e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 -- 172.21.15.32:0/2590554839 >> 172.21.15.32:0/2590554839 conn(0x7fb6c40f8980 msgr2=0x7fb6c40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 -- 172.21.15.32:0/2590554839 shutdown_connections 2024-04-06T14:39:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 -- 172.21.15.32:0/2590554839 wait complete. 2024-04-06T14:39:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 Processor -- start 2024-04-06T14:39:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.645+0000 7fb6cad06700 1 -- start start 2024-04-06T14:39:35.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6cad06700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c4102e10 0x7fb6c4113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6cad06700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c41037b0 0x7fb6c4114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6cad06700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c41146f0 con 0x7fb6c41037b0 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6cad06700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c410da50 con 0x7fb6c4102e10 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c41037b0 0x7fb6c4114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c41037b0 0x7fb6c4114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49278/0 (socket says 172.21.15.32:49278) 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c3fff700 1 -- 172.21.15.32:0/168184910 learned_addr learned my addr 172.21.15.32:0/168184910 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c3fff700 1 -- 172.21.15.32:0/168184910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c4102e10 msgr2=0x7fb6c4113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c8aa2700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c4102e10 0x7fb6c4113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c3fff700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c4102e10 0x7fb6c4113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c3fff700 1 -- 172.21.15.32:0/168184910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b8005040 con 0x7fb6c41037b0 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c3fff700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c41037b0 0x7fb6c4114010 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000b790 tx=0x7fb6b0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c17fa700 1 -- 172.21.15.32:0/168184910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6b000f6f0 con 0x7fb6c41037b0 2024-04-06T14:39:35.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c410dd30 con 0x7fb6c41037b0 2024-04-06T14:39:35.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c17fa700 1 -- 172.21.15.32:0/168184910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6b000fd00 con 0x7fb6c41037b0 2024-04-06T14:39:35.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.646+0000 7fb6c17fa700 1 -- 172.21.15.32:0/168184910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6b00103a0 con 0x7fb6c41037b0 2024-04-06T14:39:35.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.647+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c410e2f0 con 0x7fb6c41037b0 2024-04-06T14:39:35.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.647+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c40fe560 con 0x7fb6c41037b0 2024-04-06T14:39:35.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.648+0000 7fb6c17fa700 1 -- 172.21.15.32:0/168184910 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb6b000f850 con 0x7fb6c41037b0 2024-04-06T14:39:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.648+0000 7fb6c17fa700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6b406cc20 0x7fb6b406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.648+0000 7fb6c17fa700 1 -- 172.21.15.32:0/168184910 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb6b008bef0 con 0x7fb6c41037b0 2024-04-06T14:39:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.649+0000 7fb6c8aa2700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6b406cc20 0x7fb6b406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.649+0000 7fb6c8aa2700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6b406cc20 0x7fb6b406f0d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8004560 tx=0x7fb6b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:35.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.653+0000 7fb6c17fa700 1 -- 172.21.15.32:0/168184910 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb6b0059940 con 0x7fb6c41037b0 2024-04-06T14:39:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.875+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6c4109e40 con 0x7fb6b406cc20 2024-04-06T14:39:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.876+0000 7fb6c17fa700 1 -- 172.21.15.32:0/168184910 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb6c4109e40 con 0x7fb6b406cc20 2024-04-06T14:39:35.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6b406cc20 msgr2=0x7fb6b406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6b406cc20 0x7fb6b406f0d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8004560 tx=0x7fb6b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:35.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c41037b0 msgr2=0x7fb6c4114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c41037b0 0x7fb6c4114010 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000b790 tx=0x7fb6b0009220 comp rx=0 tx=0).stop 2024-04-06T14:39:35.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 shutdown_connections 2024-04-06T14:39:35.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6b406cc20 0x7fb6b406f0d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c4102e10 0x7fb6c4113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 --2- 172.21.15.32:0/168184910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6c41037b0 0x7fb6c4114010 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 >> 172.21.15.32:0/168184910 conn(0x7fb6c40f8980 msgr2=0x7fb6c40f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:35.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 shutdown_connections 2024-04-06T14:39:35.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.878+0000 7fb6cad06700 1 -- 172.21.15.32:0/168184910 wait complete. 2024-04-06T14:39:35.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.982+0000 7f7a6c455700 1 -- 172.21.15.32:0/1652915809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a64102e10 msgr2=0x7f7a641031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.982+0000 7f7a6c455700 1 --2- 172.21.15.32:0/1652915809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a64102e10 0x7f7a641031e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7a60004940 tx=0x7f7a60009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:35.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.982+0000 7f7a6c455700 1 -- 172.21.15.32:0/1652915809 shutdown_connections 2024-04-06T14:39:35.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.982+0000 7f7a6c455700 1 --2- 172.21.15.32:0/1652915809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a641037b0 0x7f7a640fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.982+0000 7f7a6c455700 1 --2- 172.21.15.32:0/1652915809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a64102e10 0x7f7a641031e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.982+0000 7f7a6c455700 1 -- 172.21.15.32:0/1652915809 >> 172.21.15.32:0/1652915809 conn(0x7f7a640f8980 msgr2=0x7f7a640fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.982+0000 7f7a6c455700 1 -- 172.21.15.32:0/1652915809 shutdown_connections 2024-04-06T14:39:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.982+0000 7f7a6c455700 1 -- 172.21.15.32:0/1652915809 wait complete. 2024-04-06T14:39:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a6c455700 1 Processor -- start 2024-04-06T14:39:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a6c455700 1 -- start start 2024-04-06T14:39:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a6c455700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64102e10 0x7f7a64113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a6c455700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a641037b0 0x7f7a64114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a6c455700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641146a0 con 0x7f7a641037b0 2024-04-06T14:39:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a6c455700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6410da90 con 0x7f7a64102e10 2024-04-06T14:39:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a699f0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a641037b0 0x7f7a64114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a699f0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a641037b0 0x7f7a64114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49288/0 (socket says 172.21.15.32:49288) 2024-04-06T14:39:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a699f0700 1 -- 172.21.15.32:0/4004317799 learned_addr learned my addr 172.21.15.32:0/4004317799 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a6a1f1700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64102e10 0x7f7a64113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a699f0700 1 -- 172.21.15.32:0/4004317799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64102e10 msgr2=0x7f7a64113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:35.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a699f0700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64102e10 0x7f7a64113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:35.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.983+0000 7f7a699f0700 1 -- 172.21.15.32:0/4004317799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a60005040 con 0x7f7a641037b0 2024-04-06T14:39:35.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.984+0000 7f7a699f0700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a641037b0 0x7f7a64114050 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7a58002a50 tx=0x7f7a58004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:35.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.984+0000 7f7a56ffd700 1 -- 172.21.15.32:0/4004317799 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a58007cc0 con 0x7f7a641037b0 2024-04-06T14:39:35.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.984+0000 7f7a56ffd700 1 -- 172.21.15.32:0/4004317799 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a5800f040 con 0x7f7a641037b0 2024-04-06T14:39:35.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.984+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6410dd70 con 0x7f7a641037b0 2024-04-06T14:39:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.984+0000 7f7a56ffd700 1 -- 172.21.15.32:0/4004317799 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a5800d990 con 0x7f7a641037b0 2024-04-06T14:39:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.984+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6410e330 con 0x7f7a641037b0 2024-04-06T14:39:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.985+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a640fe560 con 0x7f7a641037b0 2024-04-06T14:39:35.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.985+0000 7f7a56ffd700 1 -- 172.21.15.32:0/4004317799 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7a58009710 con 0x7f7a641037b0 2024-04-06T14:39:35.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.986+0000 7f7a56ffd700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a5006cc20 0x7f7a5006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:35.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.986+0000 7f7a6a1f1700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a5006cc20 0x7f7a5006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:35.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.986+0000 7f7a56ffd700 1 -- 172.21.15.32:0/4004317799 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7a5808be30 con 0x7f7a641037b0 2024-04-06T14:39:35.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.986+0000 7f7a6a1f1700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a5006cc20 0x7f7a5006f0d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a60004f90 tx=0x7f7a60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:35.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:35.990+0000 7f7a56ffd700 1 -- 172.21.15.32:0/4004317799 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7a580595f0 con 0x7f7a641037b0 2024-04-06T14:39:36.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.203+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a64109dd0 con 0x7f7a5006cc20 2024-04-06T14:39:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.213+0000 7f7a56ffd700 1 -- 172.21.15.32:0/4004317799 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7a64109dd0 con 0x7f7a5006cc20 2024-04-06T14:39:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:39:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (19m) 13m ago 31m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:39:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (17m) 13m ago 31m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:39:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (17m) 13m ago 30m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:39:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (19m) 13m ago 30m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:39:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (13m) 13m ago 25m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:39:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (13m) 13m ago 25m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (13m) 13m ago 25m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (25m) 13m ago 25m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (21m) 13m ago 32m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (20m) 13m ago 30m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (18m) 13m ago 32m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (17m) 13m ago 30m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (20m) 13m ago 31m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (20m) 13m ago 30m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (16m) 13m ago 29m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (16m) 13m ago 28m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (15m) 13m ago 28m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:39:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (15m) 13m ago 27m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:39:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (14m) 13m ago 27m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:39:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (13m) 13m ago 27m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:39:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (20m) 13m ago 30m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:39:36.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a5006cc20 msgr2=0x7f7a5006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a5006cc20 0x7f7a5006f0d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a60004f90 tx=0x7f7a60009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a641037b0 msgr2=0x7f7a64114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a641037b0 0x7f7a64114050 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7a58002a50 tx=0x7f7a58004600 comp rx=0 tx=0).stop 2024-04-06T14:39:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 shutdown_connections 2024-04-06T14:39:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7a5006cc20 0x7f7a5006f0d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64102e10 0x7f7a64113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 --2- 172.21.15.32:0/4004317799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a641037b0 0x7f7a64114050 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 >> 172.21.15.32:0/4004317799 conn(0x7f7a640f8980 msgr2=0x7f7a640f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 shutdown_connections 2024-04-06T14:39:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.215+0000 7f7a6c455700 1 -- 172.21.15.32:0/4004317799 wait complete. 2024-04-06T14:39:36.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.317+0000 7f86c0831700 1 -- 172.21.15.32:0/2108375019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 msgr2=0x7f86bc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.317+0000 7f86c0831700 1 --2- 172.21.15.32:0/2108375019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 0x7f86bc0fbd80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f86b0004940 tx=0x7f86b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:36.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.317+0000 7f86c0831700 1 -- 172.21.15.32:0/2108375019 shutdown_connections 2024-04-06T14:39:36.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.317+0000 7f86c0831700 1 --2- 172.21.15.32:0/2108375019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 0x7f86bc0fbd80 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.317+0000 7f86c0831700 1 --2- 172.21.15.32:0/2108375019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86bc101d70 0x7f86bc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.317+0000 7f86c0831700 1 -- 172.21.15.32:0/2108375019 >> 172.21.15.32:0/2108375019 conn(0x7f86bc072bc0 msgr2=0x7f86bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.317+0000 7f86c0831700 1 -- 172.21.15.32:0/2108375019 shutdown_connections 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86c0831700 1 -- 172.21.15.32:0/2108375019 wait complete. 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86c0831700 1 Processor -- start 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86c0831700 1 -- start start 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86c0831700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86bc101d70 0x7f86bc113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86c0831700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 0x7f86bc114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86c0831700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc114700 con 0x7f86bc102710 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86c0831700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc10dab0 con 0x7f86bc101d70 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86b9d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 0x7f86bc114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86b9d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 0x7f86bc114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49300/0 (socket says 172.21.15.32:49300) 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.318+0000 7f86b9d9b700 1 -- 172.21.15.32:0/537337029 learned_addr learned my addr 172.21.15.32:0/537337029 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:36.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86ba59c700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86bc101d70 0x7f86bc113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86b9d9b700 1 -- 172.21.15.32:0/537337029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86bc101d70 msgr2=0x7f86bc113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86b9d9b700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86bc101d70 0x7f86bc113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86b9d9b700 1 -- 172.21.15.32:0/537337029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b0005040 con 0x7f86bc102710 2024-04-06T14:39:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86b9d9b700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 0x7f86bc114020 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009380 tx=0x7f86b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86aaffd700 1 -- 172.21.15.32:0/537337029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b000e040 con 0x7f86bc102710 2024-04-06T14:39:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86bc10dd30 con 0x7f86bc102710 2024-04-06T14:39:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86aaffd700 1 -- 172.21.15.32:0/537337029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86b0013420 con 0x7f86bc102710 2024-04-06T14:39:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86aaffd700 1 -- 172.21.15.32:0/537337029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b001c630 con 0x7f86bc102710 2024-04-06T14:39:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.319+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86bc10e2f0 con 0x7f86bc102710 2024-04-06T14:39:36.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.320+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86bc0fd4c0 con 0x7f86bc102710 2024-04-06T14:39:36.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.321+0000 7f86aaffd700 1 -- 172.21.15.32:0/537337029 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f86b00139b0 con 0x7f86bc102710 2024-04-06T14:39:36.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.321+0000 7f86aaffd700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86a406cc20 0x7f86a406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:36.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.321+0000 7f86aaffd700 1 -- 172.21.15.32:0/537337029 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f86b008d1b0 con 0x7f86bc102710 2024-04-06T14:39:36.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.321+0000 7f86ba59c700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86a406cc20 0x7f86a406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:36.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.321+0000 7f86ba59c700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86a406cc20 0x7f86a406f0d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009bf0 tx=0x7f86ac0091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:36.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.326+0000 7f86aaffd700 1 -- 172.21.15.32:0/537337029 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f86b005ac80 con 0x7f86bc102710 2024-04-06T14:39:36.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.614+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f86bc04bed0 con 0x7f86bc102710 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.615+0000 7f86aaffd700 1 -- 172.21.15.32:0/537337029 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f86b005a810 con 0x7f86bc102710 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:39:36.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:39:36.610 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:39:36.610 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:39:36.610 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:39:36.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:39:36.610 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:39:36.610 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:39:36.610 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:39:36.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:39:36.610 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:39:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86a406cc20 msgr2=0x7f86a406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86a406cc20 0x7f86a406f0d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009bf0 tx=0x7f86ac0091a0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 msgr2=0x7f86bc114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 0x7f86bc114020 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009380 tx=0x7f86b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 shutdown_connections 2024-04-06T14:39:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f86a406cc20 0x7f86a406f0d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86bc101d70 0x7f86bc113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 --2- 172.21.15.32:0/537337029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86bc102710 0x7f86bc114020 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 >> 172.21.15.32:0/537337029 conn(0x7f86bc072bc0 msgr2=0x7f86bc0659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 shutdown_connections 2024-04-06T14:39:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.617+0000 7f86c0831700 1 -- 172.21.15.32:0/537337029 wait complete. 2024-04-06T14:39:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.720+0000 7fdfb9b27700 1 -- 172.21.15.32:0/2527078359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fe870 msgr2=0x7fdfb4106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.720+0000 7fdfb9b27700 1 --2- 172.21.15.32:0/2527078359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fe870 0x7fdfb4106f40 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4004940 tx=0x7fdfa4009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.720+0000 7fdfb9b27700 1 -- 172.21.15.32:0/2527078359 shutdown_connections 2024-04-06T14:39:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.720+0000 7fdfb9b27700 1 --2- 172.21.15.32:0/2527078359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fe870 0x7fdfb4106f40 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.720+0000 7fdfb9b27700 1 --2- 172.21.15.32:0/2527078359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb40fded0 0x7fdfb40fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.720+0000 7fdfb9b27700 1 -- 172.21.15.32:0/2527078359 >> 172.21.15.32:0/2527078359 conn(0x7fdfb4072bc0 msgr2=0x7fdfb4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.720+0000 7fdfb9b27700 1 -- 172.21.15.32:0/2527078359 shutdown_connections 2024-04-06T14:39:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.720+0000 7fdfb9b27700 1 -- 172.21.15.32:0/2527078359 wait complete. 2024-04-06T14:39:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.720+0000 7fdfb9b27700 1 Processor -- start 2024-04-06T14:39:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb9b27700 1 -- start start 2024-04-06T14:39:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb9b27700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fded0 0x7fdfb4113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb9b27700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb40fe870 0x7fdfb4113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb9b27700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb41145f0 con 0x7fdfb40fded0 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb9b27700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb410d9e0 con 0x7fdfb40fe870 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fded0 0x7fdfb4113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fded0 0x7fdfb4113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49324/0 (socket says 172.21.15.32:49324) 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb37fe700 1 -- 172.21.15.32:0/805421466 learned_addr learned my addr 172.21.15.32:0/805421466 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb2ffd700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb40fe870 0x7fdfb4113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb37fe700 1 -- 172.21.15.32:0/805421466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb40fe870 msgr2=0x7fdfb4113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb37fe700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb40fe870 0x7fdfb4113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb37fe700 1 -- 172.21.15.32:0/805421466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa4005040 con 0x7fdfb40fded0 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.721+0000 7fdfb2ffd700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb40fe870 0x7fdfb4113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:39:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.722+0000 7fdfb37fe700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fded0 0x7fdfb4113a60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdfa800c830 tx=0x7fdfa8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.722+0000 7fdf9bfff700 1 -- 172.21.15.32:0/805421466 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfa800a760 con 0x7fdfb40fded0 2024-04-06T14:39:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.722+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb410dcc0 con 0x7fdfb40fded0 2024-04-06T14:39:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.722+0000 7fdf9bfff700 1 -- 172.21.15.32:0/805421466 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfa800ad70 con 0x7fdfb40fded0 2024-04-06T14:39:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.722+0000 7fdf9bfff700 1 -- 172.21.15.32:0/805421466 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfa80113a0 con 0x7fdfb40fded0 2024-04-06T14:39:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.722+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb410e200 con 0x7fdfb40fded0 2024-04-06T14:39:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.723+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb41019a0 con 0x7fdfb40fded0 2024-04-06T14:39:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.723+0000 7fdf9bfff700 1 -- 172.21.15.32:0/805421466 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdfa8012400 con 0x7fdfb40fded0 2024-04-06T14:39:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.724+0000 7fdf9bfff700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf9c06cc20 0x7fdf9c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.724+0000 7fdf9bfff700 1 -- 172.21.15.32:0/805421466 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdfa808c0f0 con 0x7fdfb40fded0 2024-04-06T14:39:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.724+0000 7fdfb2ffd700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf9c06cc20 0x7fdf9c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.724+0000 7fdfb2ffd700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf9c06cc20 0x7fdf9c06f0d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400b3b0 tx=0x7fdfa4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.728+0000 7fdf9bfff700 1 -- 172.21.15.32:0/805421466 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdfa8059b80 con 0x7fdfb40fded0 2024-04-06T14:39:36.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:36 smithi032 ceph-mon[147414]: pgmap v583: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:36.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:36 smithi032 ceph-mon[147414]: from='client.35092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:36.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:36 smithi086 ceph-mon[110975]: pgmap v583: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:36.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:36 smithi086 ceph-mon[110975]: from='client.35092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:36.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.977+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdfb4109d50 con 0x7fdfb40fded0 2024-04-06T14:39:36.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.978+0000 7fdf9bfff700 1 -- 172.21.15.32:0/805421466 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fdfa8059710 con 0x7fdfb40fded0 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:39:36.972 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:39:36.973 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:39:36.974 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:39:36.975 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:39:36.975 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:39:36.975 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:39:36.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf9c06cc20 msgr2=0x7fdf9c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf9c06cc20 0x7fdf9c06f0d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400b3b0 tx=0x7fdfa4009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fded0 msgr2=0x7fdfb4113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fded0 0x7fdfb4113a60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdfa800c830 tx=0x7fdfa8004470 comp rx=0 tx=0).stop 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 shutdown_connections 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdf9c06cc20 0x7fdf9c06f0d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb40fded0 0x7fdfb4113a60 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 --2- 172.21.15.32:0/805421466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb40fe870 0x7fdfb4113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 >> 172.21.15.32:0/805421466 conn(0x7fdfb4072bc0 msgr2=0x7fdfb40ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 shutdown_connections 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:36.980+0000 7fdfb9b27700 1 -- 172.21.15.32:0/805421466 wait complete. 2024-04-06T14:39:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:39:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.084+0000 7f55a61a9700 1 -- 172.21.15.32:0/4058923774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fde90 msgr2=0x7f55a00fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.084+0000 7f55a61a9700 1 --2- 172.21.15.32:0/4058923774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fde90 0x7f55a00fe260 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5594004940 tx=0x7f5594009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.084+0000 7f55a61a9700 1 -- 172.21.15.32:0/4058923774 shutdown_connections 2024-04-06T14:39:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.084+0000 7f55a61a9700 1 --2- 172.21.15.32:0/4058923774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a00fe830 0x7f55a0106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.084+0000 7f55a61a9700 1 --2- 172.21.15.32:0/4058923774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fde90 0x7f55a00fe260 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.084+0000 7f55a61a9700 1 -- 172.21.15.32:0/4058923774 >> 172.21.15.32:0/4058923774 conn(0x7f55a0072bc0 msgr2=0x7f55a0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.084+0000 7f55a61a9700 1 -- 172.21.15.32:0/4058923774 shutdown_connections 2024-04-06T14:39:37.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.084+0000 7f55a61a9700 1 -- 172.21.15.32:0/4058923774 wait complete. 2024-04-06T14:39:37.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f55a61a9700 1 Processor -- start 2024-04-06T14:39:37.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f55a61a9700 1 -- start start 2024-04-06T14:39:37.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f55a61a9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a00fde90 0x7f55a0113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:37.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f55a61a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fe830 0x7f55a0114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f55a61a9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a0114710 con 0x7f55a00fe830 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f55a61a9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a010da70 con 0x7f55a00fde90 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f559effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fe830 0x7f55a0114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f559effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fe830 0x7f55a0114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49338/0 (socket says 172.21.15.32:49338) 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f559effd700 1 -- 172.21.15.32:0/1613907544 learned_addr learned my addr 172.21.15.32:0/1613907544 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.085+0000 7f559f7fe700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a00fde90 0x7f55a0113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.086+0000 7f559effd700 1 -- 172.21.15.32:0/1613907544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a00fde90 msgr2=0x7f55a0113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.086+0000 7f559effd700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a00fde90 0x7f55a0113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.086+0000 7f559effd700 1 -- 172.21.15.32:0/1613907544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5594005040 con 0x7f55a00fe830 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.086+0000 7f559effd700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fe830 0x7f55a0114030 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f559000b790 tx=0x7f5590009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.086+0000 7f5587fff700 1 -- 172.21.15.32:0/1613907544 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f559000f6f0 con 0x7f55a00fe830 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.086+0000 7f5587fff700 1 -- 172.21.15.32:0/1613907544 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f559000fd00 con 0x7f55a00fe830 2024-04-06T14:39:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.086+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a010dd50 con 0x7f55a00fe830 2024-04-06T14:39:37.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.086+0000 7f5587fff700 1 -- 172.21.15.32:0/1613907544 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55900103a0 con 0x7f55a00fe830 2024-04-06T14:39:37.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.086+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a010e290 con 0x7f55a00fe830 2024-04-06T14:39:37.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.087+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a0065e30 con 0x7f55a00fe830 2024-04-06T14:39:37.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.087+0000 7f5587fff700 1 -- 172.21.15.32:0/1613907544 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f559000f850 con 0x7f55a00fe830 2024-04-06T14:39:37.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.088+0000 7f5587fff700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f558806cc30 0x7f558806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:37.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.088+0000 7f5587fff700 1 -- 172.21.15.32:0/1613907544 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f559008bec0 con 0x7f55a00fe830 2024-04-06T14:39:37.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.088+0000 7f559f7fe700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f558806cc30 0x7f558806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:37.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.088+0000 7f559f7fe700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f558806cc30 0x7f558806f0e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f559400b3b0 tx=0x7f5594009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:37.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.092+0000 7f5587fff700 1 -- 172.21.15.32:0/1613907544 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5590059990 con 0x7f55a00fe830 2024-04-06T14:39:37.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.310+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f55a0109e40 con 0x7f558806cc30 2024-04-06T14:39:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.312+0000 7f5587fff700 1 -- 172.21.15.32:0/1613907544 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f55a0109e40 con 0x7f558806cc30 2024-04-06T14:39:37.305 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:39:37.306 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:39:37.307 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f558806cc30 msgr2=0x7f558806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f558806cc30 0x7f558806f0e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f559400b3b0 tx=0x7f5594009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fe830 msgr2=0x7f55a0114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fe830 0x7f55a0114030 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f559000b790 tx=0x7f5590009250 comp rx=0 tx=0).stop 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 shutdown_connections 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f558806cc30 0x7f558806f0e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a00fde90 0x7f55a0113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 --2- 172.21.15.32:0/1613907544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55a00fe830 0x7f55a0114030 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 >> 172.21.15.32:0/1613907544 conn(0x7f55a0072bc0 msgr2=0x7f55a01037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 shutdown_connections 2024-04-06T14:39:37.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.314+0000 7f55a61a9700 1 -- 172.21.15.32:0/1613907544 wait complete. 2024-04-06T14:39:37.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.417+0000 7fdc4509e700 1 -- 172.21.15.32:0/3902218769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fe830 msgr2=0x7fdc40106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:37.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.417+0000 7fdc4509e700 1 --2- 172.21.15.32:0/3902218769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fe830 0x7fdc40106f00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdc30004940 tx=0x7fdc30009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.417+0000 7fdc4509e700 1 -- 172.21.15.32:0/3902218769 shutdown_connections 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.417+0000 7fdc4509e700 1 --2- 172.21.15.32:0/3902218769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fe830 0x7fdc40106f00 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.417+0000 7fdc4509e700 1 --2- 172.21.15.32:0/3902218769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc400fde90 0x7fdc400fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.417+0000 7fdc4509e700 1 -- 172.21.15.32:0/3902218769 >> 172.21.15.32:0/3902218769 conn(0x7fdc40072bc0 msgr2=0x7fdc40072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.417+0000 7fdc4509e700 1 -- 172.21.15.32:0/3902218769 shutdown_connections 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.417+0000 7fdc4509e700 1 -- 172.21.15.32:0/3902218769 wait complete. 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.417+0000 7fdc4509e700 1 Processor -- start 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc4509e700 1 -- start start 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc4509e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fde90 0x7fdc40113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc4509e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc400fe830 0x7fdc40113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc4509e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40114610 con 0x7fdc400fde90 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc4509e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4010da00 con 0x7fdc400fe830 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc3ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fde90 0x7fdc40113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc3ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fde90 0x7fdc40113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49366/0 (socket says 172.21.15.32:49366) 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc3ed9d700 1 -- 172.21.15.32:0/4206780832 learned_addr learned my addr 172.21.15.32:0/4206780832 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc3ed9d700 1 -- 172.21.15.32:0/4206780832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc400fe830 msgr2=0x7fdc40113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc3ed9d700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc400fe830 0x7fdc40113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc3ed9d700 1 -- 172.21.15.32:0/4206780832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc30005040 con 0x7fdc400fde90 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc3ed9d700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fde90 0x7fdc40113a80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fdc34002a80 tx=0x7fdc340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.418+0000 7fdc277fe700 1 -- 172.21.15.32:0/4206780832 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc34007cc0 con 0x7fdc400fde90 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.419+0000 7fdc277fe700 1 -- 172.21.15.32:0/4206780832 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc3400f040 con 0x7fdc400fde90 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.419+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4010dce0 con 0x7fdc400fde90 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.419+0000 7fdc277fe700 1 -- 172.21.15.32:0/4206780832 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc34009640 con 0x7fdc400fde90 2024-04-06T14:39:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.419+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4010e220 con 0x7fdc400fde90 2024-04-06T14:39:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.420+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc40101980 con 0x7fdc400fde90 2024-04-06T14:39:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.420+0000 7fdc277fe700 1 -- 172.21.15.32:0/4206780832 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdc340097a0 con 0x7fdc400fde90 2024-04-06T14:39:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.421+0000 7fdc277fe700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdc2806cc20 0x7fdc2806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:39:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.421+0000 7fdc3e59c700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdc2806cc20 0x7fdc2806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:39:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.421+0000 7fdc277fe700 1 -- 172.21.15.32:0/4206780832 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdc3408b770 con 0x7fdc400fde90 2024-04-06T14:39:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.421+0000 7fdc3e59c700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdc2806cc20 0x7fdc2806f0d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdc30004f90 tx=0x7fdc30009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:39:37.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.425+0000 7fdc277fe700 1 -- 172.21.15.32:0/4206780832 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdc34059240 con 0x7fdc400fde90 2024-04-06T14:39:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.713+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdc4010f100 con 0x7fdc400fde90 2024-04-06T14:39:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.714+0000 7fdc277fe700 1 -- 172.21.15.32:0/4206780832 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fdc34058dd0 con 0x7fdc400fde90 2024-04-06T14:39:37.708 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:39:37.708 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:39:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:39:37.708 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:39:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:39:37.708 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:39:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:39:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdc2806cc20 msgr2=0x7fdc2806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdc2806cc20 0x7fdc2806f0d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdc30004f90 tx=0x7fdc30009e30 comp rx=0 tx=0).stop 2024-04-06T14:39:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fde90 msgr2=0x7fdc40113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:39:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fde90 0x7fdc40113a80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fdc34002a80 tx=0x7fdc340046b0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 shutdown_connections 2024-04-06T14:39:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdc2806cc20 0x7fdc2806f0d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc400fde90 0x7fdc40113a80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 --2- 172.21.15.32:0/4206780832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc400fe830 0x7fdc40113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:39:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 >> 172.21.15.32:0/4206780832 conn(0x7fdc40072bc0 msgr2=0x7fdc400ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:39:37.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 shutdown_connections 2024-04-06T14:39:37.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:39:37.716+0000 7fdc4509e700 1 -- 172.21.15.32:0/4206780832 wait complete. 2024-04-06T14:39:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:37 smithi032 ceph-mon[147414]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:37 smithi032 ceph-mon[147414]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:37 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/537337029' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:39:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:37 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/805421466' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:39:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:37 smithi086 ceph-mon[110975]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:37 smithi086 ceph-mon[110975]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:37 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/537337029' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:39:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:37 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/805421466' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:39:38.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:38 smithi032 ceph-mon[147414]: pgmap v584: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:38.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:38 smithi032 ceph-mon[147414]: from='client.35112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:38.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:39:38.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:38 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4206780832' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:39:38.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:38 smithi086 ceph-mon[110975]: pgmap v584: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:38.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:38 smithi086 ceph-mon[110975]: from='client.35112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:39:38.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:39:38.872 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:38 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4206780832' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:39:40.724 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:40 smithi032 ceph-mon[147414]: pgmap v585: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:40.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:40 smithi086 ceph-mon[110975]: pgmap v585: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:42.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:42 smithi032 ceph-mon[147414]: pgmap v586: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:42.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:42 smithi086 ceph-mon[110975]: pgmap v586: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:44.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:44 smithi032 ceph-mon[147414]: pgmap v587: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:44.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:44 smithi086 ceph-mon[110975]: pgmap v587: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:46.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:46 smithi032 ceph-mon[147414]: pgmap v588: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:46.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:46 smithi086 ceph-mon[110975]: pgmap v588: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:48.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:48 smithi032 ceph-mon[147414]: pgmap v589: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:48.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:48 smithi086 ceph-mon[110975]: pgmap v589: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:50.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:50 smithi032 ceph-mon[147414]: pgmap v590: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:50.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:50 smithi086 ceph-mon[110975]: pgmap v590: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:52.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:52 smithi032 ceph-mon[147414]: pgmap v591: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:52.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:52 smithi086 ceph-mon[110975]: pgmap v591: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:53.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:39:53.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:39:54.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:54 smithi086 ceph-mon[110975]: pgmap v592: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:54.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:54 smithi032 ceph-mon[147414]: pgmap v592: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:56.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:56 smithi086 ceph-mon[110975]: pgmap v593: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:56.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:56 smithi032 ceph-mon[147414]: pgmap v593: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:58.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:39:58 smithi086 ceph-mon[110975]: pgmap v594: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:39:58.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:39:58 smithi032 ceph-mon[147414]: pgmap v594: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:00.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:39:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:40:00.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:00 smithi086 ceph-mon[110975]: pgmap v595: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:00.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:00 smithi086 ceph-mon[110975]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:40:00.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:00 smithi032 ceph-mon[147414]: pgmap v595: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:00.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:00 smithi032 ceph-mon[147414]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:40:02.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:02 smithi086 ceph-mon[110975]: pgmap v596: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:02.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:02 smithi032 ceph-mon[147414]: pgmap v596: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:04.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:04 smithi086 ceph-mon[110975]: pgmap v597: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:04.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:04 smithi032 ceph-mon[147414]: pgmap v597: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:06.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:06 smithi086 ceph-mon[110975]: pgmap v598: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:06.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:06 smithi032 ceph-mon[147414]: pgmap v598: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:07.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.834+0000 7f8d62e3e700 1 -- 172.21.15.32:0/666057856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 msgr2=0x7f8d5c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.834+0000 7f8d62e3e700 1 --2- 172.21.15.32:0/666057856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 0x7f8d5c102140 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8d50004940 tx=0x7f8d50009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.834+0000 7f8d62e3e700 1 -- 172.21.15.32:0/666057856 shutdown_connections 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.834+0000 7f8d62e3e700 1 --2- 172.21.15.32:0/666057856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d5c102710 0x7f8d5c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.834+0000 7f8d62e3e700 1 --2- 172.21.15.32:0/666057856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 0x7f8d5c102140 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.834+0000 7f8d62e3e700 1 -- 172.21.15.32:0/666057856 >> 172.21.15.32:0/666057856 conn(0x7f8d5c072bc0 msgr2=0x7f8d5c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.835+0000 7f8d62e3e700 1 -- 172.21.15.32:0/666057856 shutdown_connections 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.835+0000 7f8d62e3e700 1 -- 172.21.15.32:0/666057856 wait complete. 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.835+0000 7f8d62e3e700 1 Processor -- start 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.835+0000 7f8d62e3e700 1 -- start start 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d62e3e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 0x7f8d5c10ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d62e3e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d5c102710 0x7f8d5c10ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d62e3e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c10f670 con 0x7f8d5c101d70 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d62e3e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c10f7b0 con 0x7f8d5c102710 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d60bda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 0x7f8d5c10ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d60bda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 0x7f8d5c10ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48380/0 (socket says 172.21.15.32:48380) 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d60bda700 1 -- 172.21.15.32:0/2277500604 learned_addr learned my addr 172.21.15.32:0/2277500604 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d5bfff700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d5c102710 0x7f8d5c10ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d60bda700 1 -- 172.21.15.32:0/2277500604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d5c102710 msgr2=0x7f8d5c10ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d60bda700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d5c102710 0x7f8d5c10ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d60bda700 1 -- 172.21.15.32:0/2277500604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d50005040 con 0x7f8d5c101d70 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d5bfff700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d5c102710 0x7f8d5c10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d60bda700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 0x7f8d5c10ea50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000b3b0 tx=0x7f8d50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.836+0000 7f8d597fa700 1 -- 172.21.15.32:0/2277500604 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d5000e040 con 0x7f8d5c101d70 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.837+0000 7f8d597fa700 1 -- 172.21.15.32:0/2277500604 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d50013420 con 0x7f8d5c101d70 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.837+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c117ac0 con 0x7f8d5c101d70 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.837+0000 7f8d597fa700 1 -- 172.21.15.32:0/2277500604 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d5001c630 con 0x7f8d5c101d70 2024-04-06T14:40:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.837+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c118050 con 0x7f8d5c101d70 2024-04-06T14:40:07.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.838+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c0ff080 con 0x7f8d5c101d70 2024-04-06T14:40:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.838+0000 7f8d597fa700 1 -- 172.21.15.32:0/2277500604 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8d50013970 con 0x7f8d5c101d70 2024-04-06T14:40:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.839+0000 7f8d597fa700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8d4c06cc20 0x7f8d4c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.839+0000 7f8d597fa700 1 -- 172.21.15.32:0/2277500604 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8d5008d180 con 0x7f8d5c101d70 2024-04-06T14:40:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.839+0000 7f8d5bfff700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8d4c06cc20 0x7f8d4c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.839+0000 7f8d5bfff700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8d4c06cc20 0x7f8d4c06f0d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8d48006fd0 tx=0x7f8d48008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:07.843+0000 7f8d597fa700 1 -- 172.21.15.32:0/2277500604 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8d50057280 con 0x7f8d5c101d70 2024-04-06T14:40:08.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.064+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d5c04bed0 con 0x7f8d4c06cc20 2024-04-06T14:40:08.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.066+0000 7f8d597fa700 1 -- 172.21.15.32:0/2277500604 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8d5c04bed0 con 0x7f8d4c06cc20 2024-04-06T14:40:08.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8d4c06cc20 msgr2=0x7f8d4c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8d4c06cc20 0x7f8d4c06f0d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8d48006fd0 tx=0x7f8d48008040 comp rx=0 tx=0).stop 2024-04-06T14:40:08.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 msgr2=0x7f8d5c10ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 0x7f8d5c10ea50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000b3b0 tx=0x7f8d50009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:08.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 shutdown_connections 2024-04-06T14:40:08.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8d4c06cc20 0x7f8d4c06f0d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d5c101d70 0x7f8d5c10ea50 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 --2- 172.21.15.32:0/2277500604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d5c102710 0x7f8d5c10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 >> 172.21.15.32:0/2277500604 conn(0x7f8d5c072bc0 msgr2=0x7f8d5c0faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:08.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 shutdown_connections 2024-04-06T14:40:08.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.068+0000 7f8d62e3e700 1 -- 172.21.15.32:0/2277500604 wait complete. 2024-04-06T14:40:08.071 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:40:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.172+0000 7f380a644700 1 -- 172.21.15.32:0/1877340875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804102e70 msgr2=0x7f3804103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.172+0000 7f380a644700 1 --2- 172.21.15.32:0/1877340875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804102e70 0x7f3804103240 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f37f4004940 tx=0x7f37f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.172+0000 7f380a644700 1 -- 172.21.15.32:0/1877340875 shutdown_connections 2024-04-06T14:40:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.172+0000 7f380a644700 1 --2- 172.21.15.32:0/1877340875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38040fb4a0 0x7f38040fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.172+0000 7f380a644700 1 --2- 172.21.15.32:0/1877340875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804102e70 0x7f3804103240 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.172+0000 7f380a644700 1 -- 172.21.15.32:0/1877340875 >> 172.21.15.32:0/1877340875 conn(0x7f38040714c0 msgr2=0x7f38040718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.172+0000 7f380a644700 1 -- 172.21.15.32:0/1877340875 shutdown_connections 2024-04-06T14:40:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.172+0000 7f380a644700 1 -- 172.21.15.32:0/1877340875 wait complete. 2024-04-06T14:40:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.173+0000 7f380a644700 1 Processor -- start 2024-04-06T14:40:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.173+0000 7f380a644700 1 -- start start 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.173+0000 7f380a644700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040fb4a0 0x7f3804115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.173+0000 7f380a644700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3804102e70 0x7f38041161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.173+0000 7f380a644700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3804116880 con 0x7f38040fb4a0 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.173+0000 7f380a644700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380410fbc0 con 0x7f3804102e70 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3803fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040fb4a0 0x7f3804115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3803fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040fb4a0 0x7f3804115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48406/0 (socket says 172.21.15.32:48406) 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3803fff700 1 -- 172.21.15.32:0/313116201 learned_addr learned my addr 172.21.15.32:0/313116201 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f38037fe700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3804102e70 0x7f38041161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3803fff700 1 -- 172.21.15.32:0/313116201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3804102e70 msgr2=0x7f38041161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3803fff700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3804102e70 0x7f38041161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3803fff700 1 -- 172.21.15.32:0/313116201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f4005040 con 0x7f38040fb4a0 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f38037fe700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3804102e70 0x7f38041161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3803fff700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040fb4a0 0x7f3804115c60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f37f400b3b0 tx=0x7f37f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3800ff9700 1 -- 172.21.15.32:0/313116201 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37f400e040 con 0x7f38040fb4a0 2024-04-06T14:40:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3800ff9700 1 -- 172.21.15.32:0/313116201 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37f4013420 con 0x7f38040fb4a0 2024-04-06T14:40:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380410fde0 con 0x7f38040fb4a0 2024-04-06T14:40:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f3800ff9700 1 -- 172.21.15.32:0/313116201 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37f401c630 con 0x7f38040fb4a0 2024-04-06T14:40:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.174+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38041103a0 con 0x7f38040fb4a0 2024-04-06T14:40:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.175+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3804065380 con 0x7f38040fb4a0 2024-04-06T14:40:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.176+0000 7f3800ff9700 1 -- 172.21.15.32:0/313116201 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f37f4013970 con 0x7f38040fb4a0 2024-04-06T14:40:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.176+0000 7f3800ff9700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f37ec06cc20 0x7f37ec06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.176+0000 7f3800ff9700 1 -- 172.21.15.32:0/313116201 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f37f408d2b0 con 0x7f38040fb4a0 2024-04-06T14:40:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.176+0000 7f38037fe700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f37ec06cc20 0x7f37ec06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.177+0000 7f38037fe700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f37ec06cc20 0x7f37ec06f0d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f37f8006fd0 tx=0x7f37f8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.181+0000 7f3800ff9700 1 -- 172.21.15.32:0/313116201 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f37f405ad80 con 0x7f38040fb4a0 2024-04-06T14:40:08.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.403+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f380404bf70 con 0x7f37ec06cc20 2024-04-06T14:40:08.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.404+0000 7f3800ff9700 1 -- 172.21.15.32:0/313116201 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f37f800c660 con 0x7f37ec06cc20 2024-04-06T14:40:08.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f37ec06cc20 msgr2=0x7f37ec06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f37ec06cc20 0x7f37ec06f0d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f37f8006fd0 tx=0x7f37f8008040 comp rx=0 tx=0).stop 2024-04-06T14:40:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040fb4a0 msgr2=0x7f3804115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040fb4a0 0x7f3804115c60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f37f400b3b0 tx=0x7f37f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 shutdown_connections 2024-04-06T14:40:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f37ec06cc20 0x7f37ec06f0d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38040fb4a0 0x7f3804115c60 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 --2- 172.21.15.32:0/313116201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3804102e70 0x7f38041161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 >> 172.21.15.32:0/313116201 conn(0x7f38040714c0 msgr2=0x7f38040f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 shutdown_connections 2024-04-06T14:40:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.406+0000 7f380a644700 1 -- 172.21.15.32:0/313116201 wait complete. 2024-04-06T14:40:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.509+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3306089637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc102710 msgr2=0x7f4afc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.509+0000 7f4b03dbb700 1 --2- 172.21.15.32:0/3306089637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc102710 0x7f4afc0fbd80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4aec004940 tx=0x7f4aec009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.509+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3306089637 shutdown_connections 2024-04-06T14:40:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.509+0000 7f4b03dbb700 1 --2- 172.21.15.32:0/3306089637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc102710 0x7f4afc0fbd80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.509+0000 7f4b03dbb700 1 --2- 172.21.15.32:0/3306089637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4afc101d70 0x7f4afc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.509+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3306089637 >> 172.21.15.32:0/3306089637 conn(0x7f4afc072bc0 msgr2=0x7f4afc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.509+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3306089637 shutdown_connections 2024-04-06T14:40:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.509+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3306089637 wait complete. 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.510+0000 7f4b03dbb700 1 Processor -- start 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.510+0000 7f4b03dbb700 1 -- start start 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.510+0000 7f4b03dbb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc101d70 0x7f4afc1129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b03dbb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4afc102710 0x7f4afc112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b03dbb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc113550 con 0x7f4afc101d70 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b03dbb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc10c940 con 0x7f4afc102710 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b01b57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc101d70 0x7f4afc1129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b01b57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc101d70 0x7f4afc1129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48438/0 (socket says 172.21.15.32:48438) 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b01b57700 1 -- 172.21.15.32:0/3993964824 learned_addr learned my addr 172.21.15.32:0/3993964824 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b01356700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4afc102710 0x7f4afc112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b01b57700 1 -- 172.21.15.32:0/3993964824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4afc102710 msgr2=0x7f4afc112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b01b57700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4afc102710 0x7f4afc112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b01b57700 1 -- 172.21.15.32:0/3993964824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec005040 con 0x7f4afc101d70 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b01b57700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc101d70 0x7f4afc1129c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4af8002a50 tx=0x7f4af80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4af27fc700 1 -- 172.21.15.32:0/3993964824 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4af8007cc0 con 0x7f4afc101d70 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4af27fc700 1 -- 172.21.15.32:0/3993964824 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4af800f040 con 0x7f4afc101d70 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4af27fc700 1 -- 172.21.15.32:0/3993964824 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4af8009540 con 0x7f4afc101d70 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4afc10cc20 con 0x7f4afc101d70 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.511+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4afc10d1e0 con 0x7f4afc101d70 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.512+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4afc0ff080 con 0x7f4afc101d70 2024-04-06T14:40:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.513+0000 7f4af27fc700 1 -- 172.21.15.32:0/3993964824 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4af80096a0 con 0x7f4afc101d70 2024-04-06T14:40:08.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.513+0000 7f4af27fc700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4ae806cc20 0x7f4ae806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:08.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.513+0000 7f4b01356700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4ae806cc20 0x7f4ae806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:08.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.513+0000 7f4af27fc700 1 -- 172.21.15.32:0/3993964824 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4af808bd70 con 0x7f4afc101d70 2024-04-06T14:40:08.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.514+0000 7f4b01356700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4ae806cc20 0x7f4ae806f0d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4aec009fd0 tx=0x7f4aec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.518+0000 7f4af27fc700 1 -- 172.21.15.32:0/3993964824 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4af8059840 con 0x7f4afc101d70 2024-04-06T14:40:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.734+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4afc10e080 con 0x7f4ae806cc20 2024-04-06T14:40:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.744+0000 7f4af27fc700 1 -- 172.21.15.32:0/3993964824 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4afc10e080 con 0x7f4ae806cc20 2024-04-06T14:40:08.738 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:40:08.738 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (20m) 13m ago 31m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:40:08.738 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (17m) 13m ago 32m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:40:08.738 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (17m) 13m ago 31m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:40:08.738 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (20m) 13m ago 31m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:40:08.738 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (14m) 13m ago 26m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (13m) 13m ago 25m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (13m) 13m ago 26m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (25m) 13m ago 25m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (22m) 13m ago 33m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (21m) 13m ago 30m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (18m) 13m ago 33m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (18m) 13m ago 30m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (21m) 13m ago 31m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (21m) 13m ago 30m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (17m) 13m ago 29m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (16m) 13m ago 29m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (16m) 13m ago 28m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (15m) 13m ago 28m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (15m) 13m ago 28m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:40:08.739 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (14m) 13m ago 27m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:40:08.740 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (20m) 13m ago 31m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:40:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4ae806cc20 msgr2=0x7f4ae806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4ae806cc20 0x7f4ae806f0d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4aec009fd0 tx=0x7f4aec009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc101d70 msgr2=0x7f4afc1129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc101d70 0x7f4afc1129c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4af8002a50 tx=0x7f4af80046b0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 shutdown_connections 2024-04-06T14:40:08.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4ae806cc20 0x7f4ae806f0d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4afc101d70 0x7f4afc1129c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 --2- 172.21.15.32:0/3993964824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4afc102710 0x7f4afc112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 >> 172.21.15.32:0/3993964824 conn(0x7f4afc072bc0 msgr2=0x7f4afc0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:08.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.746+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 shutdown_connections 2024-04-06T14:40:08.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.747+0000 7f4b03dbb700 1 -- 172.21.15.32:0/3993964824 wait complete. 2024-04-06T14:40:08.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 -- 172.21.15.32:0/663792430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 msgr2=0x7f1618106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 --2- 172.21.15.32:0/663792430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 0x7f1618106de0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1608004940 tx=0x7f1608009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:08.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 -- 172.21.15.32:0/663792430 shutdown_connections 2024-04-06T14:40:08.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 --2- 172.21.15.32:0/663792430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 0x7f1618106de0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 --2- 172.21.15.32:0/663792430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16180fde00 0x7f16180fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 -- 172.21.15.32:0/663792430 >> 172.21.15.32:0/663792430 conn(0x7f16180f78d0 msgr2=0x7f16180f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:08.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 -- 172.21.15.32:0/663792430 shutdown_connections 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 -- 172.21.15.32:0/663792430 wait complete. 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 Processor -- start 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.849+0000 7f161fe53700 1 -- start start 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161fe53700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16180fde00 0x7f1618113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161fe53700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 0x7f1618113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161fe53700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1618114600 con 0x7f16180fe7a0 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161fe53700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161810d9f0 con 0x7f16180fde00 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161d3ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 0x7f1618113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161d3ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 0x7f1618113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48448/0 (socket says 172.21.15.32:48448) 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161d3ee700 1 -- 172.21.15.32:0/2320734572 learned_addr learned my addr 172.21.15.32:0/2320734572 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161dbef700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16180fde00 0x7f1618113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161d3ee700 1 -- 172.21.15.32:0/2320734572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16180fde00 msgr2=0x7f1618113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161d3ee700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16180fde00 0x7f1618113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161d3ee700 1 -- 172.21.15.32:0/2320734572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1608005040 con 0x7f16180fe7a0 2024-04-06T14:40:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161dbef700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16180fde00 0x7f1618113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:40:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161d3ee700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 0x7f1618113fb0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1608009380 tx=0x7f1608009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f160e7fc700 1 -- 172.21.15.32:0/2320734572 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160800e040 con 0x7f16180fe7a0 2024-04-06T14:40:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f160e7fc700 1 -- 172.21.15.32:0/2320734572 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1608004290 con 0x7f16180fe7a0 2024-04-06T14:40:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161810dc70 con 0x7f16180fe7a0 2024-04-06T14:40:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f160e7fc700 1 -- 172.21.15.32:0/2320734572 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160801c3f0 con 0x7f16180fe7a0 2024-04-06T14:40:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.850+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161810e1b0 con 0x7f16180fe7a0 2024-04-06T14:40:08.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.851+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1618066770 con 0x7f16180fe7a0 2024-04-06T14:40:08.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.852+0000 7f160e7fc700 1 -- 172.21.15.32:0/2320734572 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1608013790 con 0x7f16180fe7a0 2024-04-06T14:40:08.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.852+0000 7f160e7fc700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:08.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.852+0000 7f160e7fc700 1 -- 172.21.15.32:0/2320734572 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f160808d3a0 con 0x7f16180fe7a0 2024-04-06T14:40:08.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.852+0000 7f161dbef700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:08.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.853+0000 7f161dbef700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1614009fd0 tx=0x7f1614009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:08.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:08.857+0000 7f160e7fc700 1 -- 172.21.15.32:0/2320734572 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f160805ae70 con 0x7f16180fe7a0 2024-04-06T14:40:08.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:08 smithi086 ceph-mon[110975]: pgmap v599: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:08.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:40:08.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:08 smithi032 ceph-mon[147414]: pgmap v599: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:08.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:40:09.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.149+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f161805f5f0 con 0x7f16180fe7a0 2024-04-06T14:40:09.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.149+0000 7f160e7fc700 1 -- 172.21.15.32:0/2320734572 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f1608013420 con 0x7f16180fe7a0 2024-04-06T14:40:09.143 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:40:09.143 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:40:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:40:09.145 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:40:09.145 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:40:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 msgr2=0x7f160406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1614009fd0 tx=0x7f1614009f20 comp rx=0 tx=0).stop 2024-04-06T14:40:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 msgr2=0x7f1618113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 0x7f1618113fb0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1608009380 tx=0x7f1608009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 shutdown_connections 2024-04-06T14:40:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16180fde00 0x7f1618113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 --2- 172.21.15.32:0/2320734572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16180fe7a0 0x7f1618113fb0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 >> 172.21.15.32:0/2320734572 conn(0x7f16180f78d0 msgr2=0x7f16180f8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 shutdown_connections 2024-04-06T14:40:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.152+0000 7f161fe53700 1 -- 172.21.15.32:0/2320734572 wait complete. 2024-04-06T14:40:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 -- 172.21.15.32:0/1727710688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 msgr2=0x7f010c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 --2- 172.21.15.32:0/1727710688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 0x7f010c1031e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0108004940 tx=0x7f0108009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 -- 172.21.15.32:0/1727710688 shutdown_connections 2024-04-06T14:40:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 --2- 172.21.15.32:0/1727710688 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010c1037b0 0x7f010c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 --2- 172.21.15.32:0/1727710688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 0x7f010c1031e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 -- 172.21.15.32:0/1727710688 >> 172.21.15.32:0/1727710688 conn(0x7f010c0f8980 msgr2=0x7f010c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:09.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 -- 172.21.15.32:0/1727710688 shutdown_connections 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 -- 172.21.15.32:0/1727710688 wait complete. 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 Processor -- start 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.255+0000 7f0113bef700 1 -- start start 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f0113bef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 0x7f010c113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f0113bef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010c1037b0 0x7f010c114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f0113bef700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c1146a0 con 0x7f010c102e10 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f0113bef700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c10da90 con 0x7f010c1037b0 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f011198b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 0x7f010c113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f011198b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 0x7f010c113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48464/0 (socket says 172.21.15.32:48464) 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f011198b700 1 -- 172.21.15.32:0/647959525 learned_addr learned my addr 172.21.15.32:0/647959525 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f011118a700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010c1037b0 0x7f010c114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f011198b700 1 -- 172.21.15.32:0/647959525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010c1037b0 msgr2=0x7f010c114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f011198b700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010c1037b0 0x7f010c114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f011198b700 1 -- 172.21.15.32:0/647959525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0108005040 con 0x7f010c102e10 2024-04-06T14:40:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f011198b700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 0x7f010c113b10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0108000c00 tx=0x7f0108009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f01027fc700 1 -- 172.21.15.32:0/647959525 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f010800e040 con 0x7f010c102e10 2024-04-06T14:40:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f01027fc700 1 -- 172.21.15.32:0/647959525 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0108013420 con 0x7f010c102e10 2024-04-06T14:40:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010c10dd10 con 0x7f010c102e10 2024-04-06T14:40:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f01027fc700 1 -- 172.21.15.32:0/647959525 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f010801c630 con 0x7f010c102e10 2024-04-06T14:40:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.256+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010c10e2d0 con 0x7f010c102e10 2024-04-06T14:40:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.257+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010c0fe560 con 0x7f010c102e10 2024-04-06T14:40:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.258+0000 7f01027fc700 1 -- 172.21.15.32:0/647959525 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0108013590 con 0x7f010c102e10 2024-04-06T14:40:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.258+0000 7f01027fc700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f806cc20 0x7f00f806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.258+0000 7f01027fc700 1 -- 172.21.15.32:0/647959525 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f010808d1a0 con 0x7f010c102e10 2024-04-06T14:40:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.258+0000 7f011118a700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f806cc20 0x7f00f806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.259+0000 7f011118a700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f806cc20 0x7f00f806f0d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f010c10ec50 tx=0x7f00fc009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.263+0000 7f01027fc700 1 -- 172.21.15.32:0/647959525 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f010805ade0 con 0x7f010c102e10 2024-04-06T14:40:09.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.511+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f010c04bed0 con 0x7f010c102e10 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.511+0000 7f01027fc700 1 -- 172.21.15.32:0/647959525 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f010805a970 con 0x7f010c102e10 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:40:09.505 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:40:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:40:09.507 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:40:09.508 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:40:09.508 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:40:09.508 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:40:09.508 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:40:09.508 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:40:09.508 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:40:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f806cc20 msgr2=0x7f00f806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f806cc20 0x7f00f806f0d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f010c10ec50 tx=0x7f00fc009ee0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 msgr2=0x7f010c113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 0x7f010c113b10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0108000c00 tx=0x7f0108009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 shutdown_connections 2024-04-06T14:40:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f806cc20 0x7f00f806f0d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f010c102e10 0x7f010c113b10 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 --2- 172.21.15.32:0/647959525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010c1037b0 0x7f010c114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 >> 172.21.15.32:0/647959525 conn(0x7f010c0f8980 msgr2=0x7f010c0f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 shutdown_connections 2024-04-06T14:40:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.513+0000 7f0113bef700 1 -- 172.21.15.32:0/647959525 wait complete. 2024-04-06T14:40:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.618+0000 7f197e1b1700 1 -- 172.21.15.32:0/1879452547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 msgr2=0x7f19780fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.618+0000 7f197e1b1700 1 --2- 172.21.15.32:0/1879452547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 0x7f19780fbd80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f196c004940 tx=0x7f196c009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.618+0000 7f197e1b1700 1 -- 172.21.15.32:0/1879452547 shutdown_connections 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.618+0000 7f197e1b1700 1 --2- 172.21.15.32:0/1879452547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 0x7f19780fbd80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.618+0000 7f197e1b1700 1 --2- 172.21.15.32:0/1879452547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1978101d70 0x7f1978102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.618+0000 7f197e1b1700 1 -- 172.21.15.32:0/1879452547 >> 172.21.15.32:0/1879452547 conn(0x7f1978072bc0 msgr2=0x7f1978072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.618+0000 7f197e1b1700 1 -- 172.21.15.32:0/1879452547 shutdown_connections 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.618+0000 7f197e1b1700 1 -- 172.21.15.32:0/1879452547 wait complete. 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.619+0000 7f197e1b1700 1 Processor -- start 2024-04-06T14:40:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.619+0000 7f197e1b1700 1 -- start start 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.619+0000 7f197e1b1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1978101d70 0x7f197810c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.619+0000 7f197e1b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 0x7f197810cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.619+0000 7f197e1b1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197810d2d0 con 0x7f1978102710 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.619+0000 7f197e1b1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197810d440 con 0x7f1978101d70 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.619+0000 7f1976ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 0x7f197810cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.619+0000 7f1976ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 0x7f197810cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48486/0 (socket says 172.21.15.32:48486) 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f1976ffd700 1 -- 172.21.15.32:0/2130585105 learned_addr learned my addr 172.21.15.32:0/2130585105 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f19777fe700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1978101d70 0x7f197810c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f1976ffd700 1 -- 172.21.15.32:0/2130585105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1978101d70 msgr2=0x7f197810c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f1976ffd700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1978101d70 0x7f197810c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f1976ffd700 1 -- 172.21.15.32:0/2130585105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196c005040 con 0x7f1978102710 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f1976ffd700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 0x7f197810cd90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f196c009390 tx=0x7f196c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:09.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f195ffff700 1 -- 172.21.15.32:0/2130585105 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f196c00e040 con 0x7f1978102710 2024-04-06T14:40:09.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197810d6c0 con 0x7f1978102710 2024-04-06T14:40:09.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f195ffff700 1 -- 172.21.15.32:0/2130585105 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f196c013420 con 0x7f1978102710 2024-04-06T14:40:09.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f195ffff700 1 -- 172.21.15.32:0/2130585105 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f196c01c630 con 0x7f1978102710 2024-04-06T14:40:09.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.620+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197811a240 con 0x7f1978102710 2024-04-06T14:40:09.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.621+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19780ff080 con 0x7f1978102710 2024-04-06T14:40:09.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.622+0000 7f195ffff700 1 -- 172.21.15.32:0/2130585105 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f196c013a00 con 0x7f1978102710 2024-04-06T14:40:09.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.622+0000 7f195ffff700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f196006cc30 0x7f196006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:09.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.622+0000 7f195ffff700 1 -- 172.21.15.32:0/2130585105 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f196c08cfe0 con 0x7f1978102710 2024-04-06T14:40:09.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.622+0000 7f19777fe700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f196006cc30 0x7f196006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:09.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.623+0000 7f19777fe700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f196006cc30 0x7f196006f0e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1968007fd0 tx=0x7f1968009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:09.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.627+0000 7f195ffff700 1 -- 172.21.15.32:0/2130585105 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f196c05aa70 con 0x7f1978102710 2024-04-06T14:40:09.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.848+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1978065e50 con 0x7f196006cc30 2024-04-06T14:40:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.850+0000 7f195ffff700 1 -- 172.21.15.32:0/2130585105 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1978065e50 con 0x7f196006cc30 2024-04-06T14:40:09.843 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:40:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:40:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:40:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:40:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:40:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:40:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:40:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:40:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:40:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:40:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:40:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:40:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:40:09.844 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f196006cc30 msgr2=0x7f196006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f196006cc30 0x7f196006f0e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1968007fd0 tx=0x7f1968009040 comp rx=0 tx=0).stop 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 msgr2=0x7f197810cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 0x7f197810cd90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f196c009390 tx=0x7f196c009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 shutdown_connections 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f196006cc30 0x7f196006f0e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1978101d70 0x7f197810c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 --2- 172.21.15.32:0/2130585105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1978102710 0x7f197810cd90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 >> 172.21.15.32:0/2130585105 conn(0x7f1978072bc0 msgr2=0x7f19780fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 shutdown_connections 2024-04-06T14:40:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.852+0000 7f197e1b1700 1 -- 172.21.15.32:0/2130585105 wait complete. 2024-04-06T14:40:09.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:09 smithi086 ceph-mon[110975]: from='client.35120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:09.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:09 smithi086 ceph-mon[110975]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:09.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:09 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2320734572' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:40:09.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 -- 172.21.15.32:0/92194049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f8102e70 msgr2=0x7f76f8103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 --2- 172.21.15.32:0/92194049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f8102e70 0x7f76f8103240 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f76e8004940 tx=0x7f76e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:09.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 -- 172.21.15.32:0/92194049 shutdown_connections 2024-04-06T14:40:09.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 --2- 172.21.15.32:0/92194049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76f80fb4a0 0x7f76f80fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 --2- 172.21.15.32:0/92194049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f8102e70 0x7f76f8103240 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 -- 172.21.15.32:0/92194049 >> 172.21.15.32:0/92194049 conn(0x7f76f80714c0 msgr2=0x7f76f80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:09.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 -- 172.21.15.32:0/92194049 shutdown_connections 2024-04-06T14:40:09.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 -- 172.21.15.32:0/92194049 wait complete. 2024-04-06T14:40:09.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 Processor -- start 2024-04-06T14:40:09.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 -- start start 2024-04-06T14:40:09.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f80fb4a0 0x7f76f80fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:09.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76f8102e70 0x7f76f80fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:09.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80fd1e0 con 0x7f76f80fb4a0 2024-04-06T14:40:09.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.955+0000 7f76fcb39700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80fd320 con 0x7f76f8102e70 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76f659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f80fb4a0 0x7f76f80fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76f659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f80fb4a0 0x7f76f80fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48510/0 (socket says 172.21.15.32:48510) 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76f659c700 1 -- 172.21.15.32:0/2740705590 learned_addr learned my addr 172.21.15.32:0/2740705590 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76f659c700 1 -- 172.21.15.32:0/2740705590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76f8102e70 msgr2=0x7f76f80fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76f659c700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76f8102e70 0x7f76f80fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76f659c700 1 -- 172.21.15.32:0/2740705590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e8005040 con 0x7f76f80fb4a0 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76f659c700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f80fb4a0 0x7f76f80fc680 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f76e8000c00 tx=0x7f76e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76e6ffd700 1 -- 172.21.15.32:0/2740705590 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e800e040 con 0x7f76f80fb4a0 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76e6ffd700 1 -- 172.21.15.32:0/2740705590 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76e8013420 con 0x7f76f80fb4a0 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f8119d90 con 0x7f76f80fb4a0 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76e6ffd700 1 -- 172.21.15.32:0/2740705590 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e801c630 con 0x7f76f80fb4a0 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.956+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f811a270 con 0x7f76f80fb4a0 2024-04-06T14:40:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.957+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f8100110 con 0x7f76f80fb4a0 2024-04-06T14:40:09.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.958+0000 7f76e6ffd700 1 -- 172.21.15.32:0/2740705590 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f76e8013590 con 0x7f76f80fb4a0 2024-04-06T14:40:09.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.958+0000 7f76e6ffd700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f76e006cc20 0x7f76e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:09.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.958+0000 7f76e6ffd700 1 -- 172.21.15.32:0/2740705590 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f76e808d1b0 con 0x7f76f80fb4a0 2024-04-06T14:40:09.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.958+0000 7f76f5d9b700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f76e006cc20 0x7f76e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:09.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.959+0000 7f76f5d9b700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f76e006cc20 0x7f76e006f0d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006fd0 tx=0x7f76ec006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:09.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:09.963+0000 7f76e6ffd700 1 -- 172.21.15.32:0/2740705590 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f76e80572b0 con 0x7f76f80fb4a0 2024-04-06T14:40:09.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:09 smithi032 ceph-mon[147414]: from='client.35120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:09.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:09 smithi032 ceph-mon[147414]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:09.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:09 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2320734572' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:40:10.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.248+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f76f8065380 con 0x7f76f80fb4a0 2024-04-06T14:40:10.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.249+0000 7f76e6ffd700 1 -- 172.21.15.32:0/2740705590 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f76f8065380 con 0x7f76f80fb4a0 2024-04-06T14:40:10.242 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:40:10.243 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:40:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:40:10.243 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:40:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:40:10.243 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:40:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:40:10.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f76e006cc20 msgr2=0x7f76e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:10.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f76e006cc20 0x7f76e006f0d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006fd0 tx=0x7f76ec006db0 comp rx=0 tx=0).stop 2024-04-06T14:40:10.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f80fb4a0 msgr2=0x7f76f80fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:10.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f80fb4a0 0x7f76f80fc680 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f76e8000c00 tx=0x7f76e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:10.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 shutdown_connections 2024-04-06T14:40:10.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f76e006cc20 0x7f76e006f0d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:10.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f80fb4a0 0x7f76f80fc680 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:10.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 --2- 172.21.15.32:0/2740705590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76f8102e70 0x7f76f80fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:10.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 >> 172.21.15.32:0/2740705590 conn(0x7f76f80714c0 msgr2=0x7f76f80fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:10.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 shutdown_connections 2024-04-06T14:40:10.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:10.251+0000 7f76fcb39700 1 -- 172.21.15.32:0/2740705590 wait complete. 2024-04-06T14:40:10.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:10 smithi086 ceph-mon[110975]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:10.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:10 smithi086 ceph-mon[110975]: pgmap v600: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:10.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:10 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/647959525' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:40:10.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:10 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2740705590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:40:10.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:10 smithi032 ceph-mon[147414]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:10.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:10 smithi032 ceph-mon[147414]: pgmap v600: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:10.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:10 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/647959525' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:40:10.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:10 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2740705590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:40:11.621 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:11 smithi086 ceph-mon[110975]: from='client.35140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:11.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:11 smithi032 ceph-mon[147414]: from='client.35140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:12.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:12 smithi086 ceph-mon[110975]: pgmap v601: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:12.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:12 smithi032 ceph-mon[147414]: pgmap v601: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:14.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:14 smithi086 ceph-mon[110975]: pgmap v602: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:14.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:14 smithi032 ceph-mon[147414]: pgmap v602: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:16.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:16 smithi086 ceph-mon[110975]: pgmap v603: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:16.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:16 smithi032 ceph-mon[147414]: pgmap v603: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:18.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:18 smithi086 ceph-mon[110975]: pgmap v604: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:18.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:18 smithi032 ceph-mon[147414]: pgmap v604: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:20.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:20 smithi086 ceph-mon[110975]: pgmap v605: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:20.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:20 smithi032 ceph-mon[147414]: pgmap v605: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:22.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:22 smithi086 ceph-mon[110975]: pgmap v606: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:22.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:22 smithi032 ceph-mon[147414]: pgmap v606: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:23.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:40:23.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:40:24.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:24 smithi086 ceph-mon[110975]: pgmap v607: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:24.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:24 smithi032 ceph-mon[147414]: pgmap v607: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:26.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:26 smithi086 ceph-mon[110975]: pgmap v608: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:26.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:26 smithi032 ceph-mon[147414]: pgmap v608: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:28.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:28 smithi086 ceph-mon[110975]: pgmap v609: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:28.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:28 smithi032 ceph-mon[147414]: pgmap v609: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:30.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:30 smithi086 ceph-mon[110975]: pgmap v610: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:30.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:30 smithi032 ceph-mon[147414]: pgmap v610: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:32.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:32 smithi086 ceph-mon[110975]: pgmap v611: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:32.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:32 smithi032 ceph-mon[147414]: pgmap v611: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:34.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:34 smithi086 ceph-mon[110975]: pgmap v612: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:34.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:34 smithi032 ceph-mon[147414]: pgmap v612: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:36.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:36 smithi086 ceph-mon[110975]: pgmap v613: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:36.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:36 smithi032 ceph-mon[147414]: pgmap v613: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:38 smithi086 ceph-mon[110975]: pgmap v614: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:38.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:40:38.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:38 smithi032 ceph-mon[147414]: pgmap v614: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:38.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:40:40.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.368+0000 7f9c0e0de700 1 -- 172.21.15.32:0/4064842111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 msgr2=0x7f9c08102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:40.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.368+0000 7f9c0e0de700 1 --2- 172.21.15.32:0/4064842111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 0x7f9c08102140 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8004940 tx=0x7f9bf8009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:40.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.368+0000 7f9c0e0de700 1 -- 172.21.15.32:0/4064842111 shutdown_connections 2024-04-06T14:40:40.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.368+0000 7f9c0e0de700 1 --2- 172.21.15.32:0/4064842111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c08102710 0x7f9c080fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.368+0000 7f9c0e0de700 1 --2- 172.21.15.32:0/4064842111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 0x7f9c08102140 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.368+0000 7f9c0e0de700 1 -- 172.21.15.32:0/4064842111 >> 172.21.15.32:0/4064842111 conn(0x7f9c08072bc0 msgr2=0x7f9c08072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:40.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.368+0000 7f9c0e0de700 1 -- 172.21.15.32:0/4064842111 shutdown_connections 2024-04-06T14:40:40.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.368+0000 7f9c0e0de700 1 -- 172.21.15.32:0/4064842111 wait complete. 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.369+0000 7f9c0e0de700 1 Processor -- start 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.369+0000 7f9c0e0de700 1 -- start start 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.369+0000 7f9c0e0de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 0x7f9c0810ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.369+0000 7f9c0e0de700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c08102710 0x7f9c0810f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.369+0000 7f9c0e0de700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0810f660 con 0x7f9c08101d70 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.369+0000 7f9c0e0de700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0810f7a0 con 0x7f9c08102710 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 0x7f9c0810ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 0x7f9c0810ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59240/0 (socket says 172.21.15.32:59240) 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c077fe700 1 -- 172.21.15.32:0/3623402033 learned_addr learned my addr 172.21.15.32:0/3623402033 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c06ffd700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c08102710 0x7f9c0810f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c077fe700 1 -- 172.21.15.32:0/3623402033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c08102710 msgr2=0x7f9c0810f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c077fe700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c08102710 0x7f9c0810f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c077fe700 1 -- 172.21.15.32:0/3623402033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf8005040 con 0x7f9c08101d70 2024-04-06T14:40:40.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c06ffd700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c08102710 0x7f9c0810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:40:40.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c077fe700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 0x7f9c0810ebe0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009360 tx=0x7f9bf8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:40.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9beffff700 1 -- 172.21.15.32:0/3623402033 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bf800e040 con 0x7f9c08101d70 2024-04-06T14:40:40.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9beffff700 1 -- 172.21.15.32:0/3623402033 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bf8013420 con 0x7f9c08101d70 2024-04-06T14:40:40.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c08117a90 con 0x7f9c08101d70 2024-04-06T14:40:40.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9beffff700 1 -- 172.21.15.32:0/3623402033 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bf801c5f0 con 0x7f9c08101d70 2024-04-06T14:40:40.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.370+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c08118020 con 0x7f9c08101d70 2024-04-06T14:40:40.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.371+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c080ff080 con 0x7f9c08101d70 2024-04-06T14:40:40.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.372+0000 7f9beffff700 1 -- 172.21.15.32:0/3623402033 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9bf8013590 con 0x7f9c08101d70 2024-04-06T14:40:40.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.372+0000 7f9beffff700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bf006cc30 0x7f9bf006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:40.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.373+0000 7f9beffff700 1 -- 172.21.15.32:0/3623402033 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9bf808d2d0 con 0x7f9c08101d70 2024-04-06T14:40:40.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.373+0000 7f9c06ffd700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bf006cc30 0x7f9bf006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:40.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.373+0000 7f9c06ffd700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bf006cc30 0x7f9bf006f0e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9c08066020 tx=0x7f9bfc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:40.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.378+0000 7f9beffff700 1 -- 172.21.15.32:0/3623402033 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9bf8057310 con 0x7f9c08101d70 2024-04-06T14:40:40.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.601+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c0804bed0 con 0x7f9bf006cc30 2024-04-06T14:40:40.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.602+0000 7f9beffff700 1 -- 172.21.15.32:0/3623402033 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9c0804bed0 con 0x7f9bf006cc30 2024-04-06T14:40:40.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bf006cc30 msgr2=0x7f9bf006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:40.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bf006cc30 0x7f9bf006f0e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9c08066020 tx=0x7f9bfc009040 comp rx=0 tx=0).stop 2024-04-06T14:40:40.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 msgr2=0x7f9c0810ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:40.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 0x7f9c0810ebe0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009360 tx=0x7f9bf8009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:40.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 shutdown_connections 2024-04-06T14:40:40.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9bf006cc30 0x7f9bf006f0e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c08101d70 0x7f9c0810ebe0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 --2- 172.21.15.32:0/3623402033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c08102710 0x7f9c0810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 >> 172.21.15.32:0/3623402033 conn(0x7f9c08072bc0 msgr2=0x7f9c080fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:40.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 shutdown_connections 2024-04-06T14:40:40.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.605+0000 7f9c0e0de700 1 -- 172.21.15.32:0/3623402033 wait complete. 2024-04-06T14:40:40.608 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:40:40.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.708+0000 7f124caea700 1 -- 172.21.15.32:0/3398378256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 msgr2=0x7f12480fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:40.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.708+0000 7f124caea700 1 --2- 172.21.15.32:0/3398378256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 0x7f12480fbd80 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1230004940 tx=0x7f1230009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:40.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.708+0000 7f124caea700 1 -- 172.21.15.32:0/3398378256 shutdown_connections 2024-04-06T14:40:40.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.708+0000 7f124caea700 1 --2- 172.21.15.32:0/3398378256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 0x7f12480fbd80 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.708+0000 7f124caea700 1 --2- 172.21.15.32:0/3398378256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1248101d70 0x7f1248102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.708+0000 7f124caea700 1 -- 172.21.15.32:0/3398378256 >> 172.21.15.32:0/3398378256 conn(0x7f1248072bc0 msgr2=0x7f1248072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f124caea700 1 -- 172.21.15.32:0/3398378256 shutdown_connections 2024-04-06T14:40:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f124caea700 1 -- 172.21.15.32:0/3398378256 wait complete. 2024-04-06T14:40:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f124caea700 1 Processor -- start 2024-04-06T14:40:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f124caea700 1 -- start start 2024-04-06T14:40:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f124caea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1248101d70 0x7f124810ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f124caea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 0x7f124810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f124caea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124810f5f0 con 0x7f1248102710 2024-04-06T14:40:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f124caea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124810f730 con 0x7f1248101d70 2024-04-06T14:40:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f1245d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 0x7f124810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f1245d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 0x7f124810efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59264/0 (socket says 172.21.15.32:59264) 2024-04-06T14:40:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.709+0000 7f1245d9b700 1 -- 172.21.15.32:0/3718143053 learned_addr learned my addr 172.21.15.32:0/3718143053 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f124659c700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1248101d70 0x7f124810ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f1245d9b700 1 -- 172.21.15.32:0/3718143053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1248101d70 msgr2=0x7f124810ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f1245d9b700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1248101d70 0x7f124810ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f1245d9b700 1 -- 172.21.15.32:0/3718143053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1230005040 con 0x7f1248102710 2024-04-06T14:40:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f1245d9b700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 0x7f124810efa0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1230009380 tx=0x7f1230009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f123effd700 1 -- 172.21.15.32:0/3718143053 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f123000e040 con 0x7f1248102710 2024-04-06T14:40:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f123effd700 1 -- 172.21.15.32:0/3718143053 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1230013420 con 0x7f1248102710 2024-04-06T14:40:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1248117ad0 con 0x7f1248102710 2024-04-06T14:40:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f123effd700 1 -- 172.21.15.32:0/3718143053 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f123001c630 con 0x7f1248102710 2024-04-06T14:40:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.710+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1248117fe0 con 0x7f1248102710 2024-04-06T14:40:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.711+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12480ff080 con 0x7f1248102710 2024-04-06T14:40:40.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.712+0000 7f123effd700 1 -- 172.21.15.32:0/3718143053 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f12300139e0 con 0x7f1248102710 2024-04-06T14:40:40.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.712+0000 7f123effd700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f123406ccf0 0x7f123406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:40.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.712+0000 7f123effd700 1 -- 172.21.15.32:0/3718143053 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f123008d0f0 con 0x7f1248102710 2024-04-06T14:40:40.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.712+0000 7f124659c700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f123406ccf0 0x7f123406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:40.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.712+0000 7f124659c700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f123406ccf0 0x7f123406f1a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1238007fd0 tx=0x7f1238009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:40.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.717+0000 7f123effd700 1 -- 172.21.15.32:0/3718143053 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f12300571f0 con 0x7f1248102710 2024-04-06T14:40:40.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:40 smithi086 ceph-mon[110975]: pgmap v615: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:40.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.941+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1248065e20 con 0x7f123406ccf0 2024-04-06T14:40:40.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.942+0000 7f123effd700 1 -- 172.21.15.32:0/3718143053 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1248065e20 con 0x7f123406ccf0 2024-04-06T14:40:40.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f123406ccf0 msgr2=0x7f123406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f123406ccf0 0x7f123406f1a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1238007fd0 tx=0x7f1238009040 comp rx=0 tx=0).stop 2024-04-06T14:40:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 msgr2=0x7f124810efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 0x7f124810efa0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1230009380 tx=0x7f1230009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 shutdown_connections 2024-04-06T14:40:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f123406ccf0 0x7f123406f1a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1248101d70 0x7f124810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 --2- 172.21.15.32:0/3718143053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1248102710 0x7f124810efa0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 >> 172.21.15.32:0/3718143053 conn(0x7f1248072bc0 msgr2=0x7f12480fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 shutdown_connections 2024-04-06T14:40:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:40.944+0000 7f124caea700 1 -- 172.21.15.32:0/3718143053 wait complete. 2024-04-06T14:40:40.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:40 smithi032 ceph-mon[147414]: pgmap v615: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:41.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 -- 172.21.15.32:0/1939896340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fe830 msgr2=0x7f807c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 --2- 172.21.15.32:0/1939896340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fe830 0x7f807c106f00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8068004940 tx=0x7f8068009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 -- 172.21.15.32:0/1939896340 shutdown_connections 2024-04-06T14:40:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 --2- 172.21.15.32:0/1939896340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fe830 0x7f807c106f00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 --2- 172.21.15.32:0/1939896340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f807c0fde90 0x7f807c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 -- 172.21.15.32:0/1939896340 >> 172.21.15.32:0/1939896340 conn(0x7f807c072bc0 msgr2=0x7f807c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 -- 172.21.15.32:0/1939896340 shutdown_connections 2024-04-06T14:40:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 -- 172.21.15.32:0/1939896340 wait complete. 2024-04-06T14:40:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 Processor -- start 2024-04-06T14:40:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.048+0000 7f80831ae700 1 -- start start 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f80831ae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fde90 0x7f807c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f80831ae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f807c0fe830 0x7f807c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f80831ae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0fc820 con 0x7f807c0fde90 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f80831ae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0fc960 con 0x7f807c0fe830 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f8080f4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fde90 0x7f807c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f8080f4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fde90 0x7f807c0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59280/0 (socket says 172.21.15.32:59280) 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f8080f4a700 1 -- 172.21.15.32:0/928123568 learned_addr learned my addr 172.21.15.32:0/928123568 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f807bfff700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f807c0fe830 0x7f807c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f8080f4a700 1 -- 172.21.15.32:0/928123568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f807c0fe830 msgr2=0x7f807c0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f8080f4a700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f807c0fe830 0x7f807c0fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f8080f4a700 1 -- 172.21.15.32:0/928123568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8068005040 con 0x7f807c0fde90 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f807bfff700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f807c0fe830 0x7f807c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:40:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.049+0000 7f8080f4a700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fde90 0x7f807c0fbd50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f807000c830 tx=0x7f8070004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.050+0000 7f80797fa700 1 -- 172.21.15.32:0/928123568 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f807000a760 con 0x7f807c0fde90 2024-04-06T14:40:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.050+0000 7f80797fa700 1 -- 172.21.15.32:0/928123568 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f807000ad70 con 0x7f807c0fde90 2024-04-06T14:40:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.050+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c119de0 con 0x7f807c0fde90 2024-04-06T14:40:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.050+0000 7f80797fa700 1 -- 172.21.15.32:0/928123568 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80700113a0 con 0x7f807c0fde90 2024-04-06T14:40:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.050+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c11a1e0 con 0x7f807c0fde90 2024-04-06T14:40:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.051+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c065e30 con 0x7f807c0fde90 2024-04-06T14:40:41.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.051+0000 7f80797fa700 1 -- 172.21.15.32:0/928123568 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8070012400 con 0x7f807c0fde90 2024-04-06T14:40:41.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.051+0000 7f80797fa700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f806c06cc20 0x7f806c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:41.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.052+0000 7f80797fa700 1 -- 172.21.15.32:0/928123568 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f807008c0b0 con 0x7f807c0fde90 2024-04-06T14:40:41.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.052+0000 7f807bfff700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f806c06cc20 0x7f806c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:41.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.052+0000 7f807bfff700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f806c06cc20 0x7f806c06f0d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f807c0fd690 tx=0x7f8068009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:41.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.056+0000 7f80797fa700 1 -- 172.21.15.32:0/928123568 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f80700560f0 con 0x7f807c0fde90 2024-04-06T14:40:41.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.269+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f807c0fd440 con 0x7f806c06cc20 2024-04-06T14:40:41.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.279+0000 7f80797fa700 1 -- 172.21.15.32:0/928123568 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f807c0fd440 con 0x7f806c06cc20 2024-04-06T14:40:41.271 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:40:41.271 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (21m) 14m ago 32m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:40:41.271 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (18m) 14m ago 32m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:40:41.271 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (18m) 14m ago 31m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:40:41.271 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (20m) 14m ago 32m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:40:41.271 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (14m) 14m ago 26m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:40:41.271 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (14m) 14m ago 26m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:40:41.271 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (14m) 14m ago 26m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (26m) 14m ago 26m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (22m) 14m ago 33m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (21m) 14m ago 31m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (19m) 14m ago 33m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (19m) 14m ago 31m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (21m) 14m ago 32m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (21m) 14m ago 31m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (17m) 14m ago 30m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (17m) 14m ago 30m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (16m) 14m ago 29m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (16m) 14m ago 29m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (15m) 14m ago 28m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (15m) 14m ago 28m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:40:41.272 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (21m) 14m ago 31m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:40:41.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f806c06cc20 msgr2=0x7f806c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f806c06cc20 0x7f806c06f0d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f807c0fd690 tx=0x7f8068009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:41.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fde90 msgr2=0x7f807c0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fde90 0x7f807c0fbd50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f807000c830 tx=0x7f8070004470 comp rx=0 tx=0).stop 2024-04-06T14:40:41.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 shutdown_connections 2024-04-06T14:40:41.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f806c06cc20 0x7f806c06f0d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807c0fde90 0x7f807c0fbd50 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 --2- 172.21.15.32:0/928123568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f807c0fe830 0x7f807c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 >> 172.21.15.32:0/928123568 conn(0x7f807c072bc0 msgr2=0x7f807c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:41.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 shutdown_connections 2024-04-06T14:40:41.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.281+0000 7f80831ae700 1 -- 172.21.15.32:0/928123568 wait complete. 2024-04-06T14:40:41.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 -- 172.21.15.32:0/831227215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fc590 msgr2=0x7f14781092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 --2- 172.21.15.32:0/831227215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fc590 0x7f14781092f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f146c004940 tx=0x7f146c009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 -- 172.21.15.32:0/831227215 shutdown_connections 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 --2- 172.21.15.32:0/831227215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fc590 0x7f14781092f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 --2- 172.21.15.32:0/831227215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780fbc80 0x7f14780fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 -- 172.21.15.32:0/831227215 >> 172.21.15.32:0/831227215 conn(0x7f1478072bc0 msgr2=0x7f1478072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 -- 172.21.15.32:0/831227215 shutdown_connections 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 -- 172.21.15.32:0/831227215 wait complete. 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 Processor -- start 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 -- start start 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fbc80 0x7f1478115ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780fc590 0x7f14781160e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478116730 con 0x7f14780fbc80 2024-04-06T14:40:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.383+0000 7f147ca3c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147810fb90 con 0x7f14780fc590 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f147659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fbc80 0x7f1478115ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f147659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fbc80 0x7f1478115ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59304/0 (socket says 172.21.15.32:59304) 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f147659c700 1 -- 172.21.15.32:0/2209504376 learned_addr learned my addr 172.21.15.32:0/2209504376 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f1475d9b700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780fc590 0x7f14781160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f147659c700 1 -- 172.21.15.32:0/2209504376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780fc590 msgr2=0x7f14781160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f147659c700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780fc590 0x7f14781160e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f147659c700 1 -- 172.21.15.32:0/2209504376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c005040 con 0x7f14780fbc80 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f147659c700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fbc80 0x7f1478115ba0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1468002a80 tx=0x7f14680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f1466ffd700 1 -- 172.21.15.32:0/2209504376 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1468007cc0 con 0x7f14780fbc80 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f1466ffd700 1 -- 172.21.15.32:0/2209504376 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f146800f040 con 0x7f14780fbc80 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147810fe70 con 0x7f14780fbc80 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f1466ffd700 1 -- 172.21.15.32:0/2209504376 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1468009640 con 0x7f14780fbc80 2024-04-06T14:40:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.384+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1478110430 con 0x7f14780fbc80 2024-04-06T14:40:41.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.385+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1478065e30 con 0x7f14780fbc80 2024-04-06T14:40:41.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.386+0000 7f1466ffd700 1 -- 172.21.15.32:0/2209504376 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f14680097a0 con 0x7f14780fbc80 2024-04-06T14:40:41.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.386+0000 7f1466ffd700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146006cc20 0x7f146006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:41.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.386+0000 7f1466ffd700 1 -- 172.21.15.32:0/2209504376 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f146808b880 con 0x7f14780fbc80 2024-04-06T14:40:41.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.386+0000 7f1475d9b700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146006cc20 0x7f146006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:41.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.387+0000 7f1475d9b700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146006cc20 0x7f146006f0d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f146c004f90 tx=0x7f146c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:41.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.391+0000 7f1466ffd700 1 -- 172.21.15.32:0/2209504376 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1468055840 con 0x7f14780fbc80 2024-04-06T14:40:41.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.676+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f147804bed0 con 0x7f14780fbc80 2024-04-06T14:40:41.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.676+0000 7f1466ffd700 1 -- 172.21.15.32:0/2209504376 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f1468058e60 con 0x7f14780fbc80 2024-04-06T14:40:41.668 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:40:41.668 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:40:41.669 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:40:41.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:40:41.670 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:40:41.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146006cc20 msgr2=0x7f146006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146006cc20 0x7f146006f0d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f146c004f90 tx=0x7f146c009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:41.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fbc80 msgr2=0x7f1478115ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fbc80 0x7f1478115ba0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1468002a80 tx=0x7f14680046b0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 shutdown_connections 2024-04-06T14:40:41.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f146006cc20 0x7f146006f0d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14780fbc80 0x7f1478115ba0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 --2- 172.21.15.32:0/2209504376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780fc590 0x7f14781160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 >> 172.21.15.32:0/2209504376 conn(0x7f1478072bc0 msgr2=0x7f1478105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:41.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 shutdown_connections 2024-04-06T14:40:41.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.678+0000 7f147ca3c700 1 -- 172.21.15.32:0/2209504376 wait complete. 2024-04-06T14:40:41.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.781+0000 7f80960ad700 1 -- 172.21.15.32:0/3114589246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090101d70 msgr2=0x7f8090102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.781+0000 7f80960ad700 1 --2- 172.21.15.32:0/3114589246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090101d70 0x7f8090102140 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8078004940 tx=0x7f8078009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:41.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.781+0000 7f80960ad700 1 -- 172.21.15.32:0/3114589246 shutdown_connections 2024-04-06T14:40:41.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.781+0000 7f80960ad700 1 --2- 172.21.15.32:0/3114589246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8090102710 0x7f80900fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.781+0000 7f80960ad700 1 --2- 172.21.15.32:0/3114589246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090101d70 0x7f8090102140 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.781+0000 7f80960ad700 1 -- 172.21.15.32:0/3114589246 >> 172.21.15.32:0/3114589246 conn(0x7f8090072bc0 msgr2=0x7f8090072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:41.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.781+0000 7f80960ad700 1 -- 172.21.15.32:0/3114589246 shutdown_connections 2024-04-06T14:40:41.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.781+0000 7f80960ad700 1 -- 172.21.15.32:0/3114589246 wait complete. 2024-04-06T14:40:41.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.781+0000 7f80960ad700 1 Processor -- start 2024-04-06T14:40:41.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f80960ad700 1 -- start start 2024-04-06T14:40:41.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f80960ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8090101d70 0x7f8090115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:41.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f80960ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090102710 0x7f80901160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:41.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f80960ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8090116740 con 0x7f8090102710 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f80960ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809010fba0 con 0x7f8090101d70 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f808effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090102710 0x7f80901160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f808effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090102710 0x7f80901160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59328/0 (socket says 172.21.15.32:59328) 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f808effd700 1 -- 172.21.15.32:0/2470768197 learned_addr learned my addr 172.21.15.32:0/2470768197 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f808f7fe700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8090101d70 0x7f8090115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f808effd700 1 -- 172.21.15.32:0/2470768197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8090101d70 msgr2=0x7f8090115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f808effd700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8090101d70 0x7f8090115bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f808effd700 1 -- 172.21.15.32:0/2470768197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8078005040 con 0x7f8090102710 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f808effd700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090102710 0x7f80901160f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f808000eb40 tx=0x7f808000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f8087fff700 1 -- 172.21.15.32:0/2470768197 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8080007cc0 con 0x7f8090102710 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.782+0000 7f8087fff700 1 -- 172.21.15.32:0/2470768197 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f808000f040 con 0x7f8090102710 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.783+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809010fe20 con 0x7f8090102710 2024-04-06T14:40:41.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.783+0000 7f8087fff700 1 -- 172.21.15.32:0/2470768197 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f808000c8f0 con 0x7f8090102710 2024-04-06T14:40:41.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.783+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80901103e0 con 0x7f8090102710 2024-04-06T14:40:41.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.784+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80900fd4c0 con 0x7f8090102710 2024-04-06T14:40:41.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.784+0000 7f8087fff700 1 -- 172.21.15.32:0/2470768197 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f80800090d0 con 0x7f8090102710 2024-04-06T14:40:41.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.784+0000 7f8087fff700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f807c06cc20 0x7f807c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:41.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.785+0000 7f8087fff700 1 -- 172.21.15.32:0/2470768197 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f808008b9e0 con 0x7f8090102710 2024-04-06T14:40:41.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.785+0000 7f808f7fe700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f807c06cc20 0x7f807c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:41.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.785+0000 7f808f7fe700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f807c06cc20 0x7f807c06f0d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f807800b0d0 tx=0x7f8078009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:41.789+0000 7f8087fff700 1 -- 172.21.15.32:0/2470768197 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8080059560 con 0x7f8090102710 2024-04-06T14:40:41.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:41 smithi086 ceph-mon[110975]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:41.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:41 smithi032 ceph-mon[147414]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:42.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.040+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f809004bed0 con 0x7f8090102710 2024-04-06T14:40:42.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.040+0000 7f8087fff700 1 -- 172.21.15.32:0/2470768197 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f8080019740 con 0x7f8090102710 2024-04-06T14:40:42.033 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:40:42.034 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:40:42.035 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:40:42.036 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:40:42.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.042+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f807c06cc20 msgr2=0x7f807c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.042+0000 7f80960ad700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f807c06cc20 0x7f807c06f0d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f807800b0d0 tx=0x7f8078009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:42.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.042+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090102710 msgr2=0x7f80901160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.042+0000 7f80960ad700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090102710 0x7f80901160f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f808000eb40 tx=0x7f808000b040 comp rx=0 tx=0).stop 2024-04-06T14:40:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.042+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 shutdown_connections 2024-04-06T14:40:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.042+0000 7f80960ad700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f807c06cc20 0x7f807c06f0d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.042+0000 7f80960ad700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8090101d70 0x7f8090115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.042+0000 7f80960ad700 1 --2- 172.21.15.32:0/2470768197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8090102710 0x7f80901160f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.042+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 >> 172.21.15.32:0/2470768197 conn(0x7f8090072bc0 msgr2=0x7f8090065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.043+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 shutdown_connections 2024-04-06T14:40:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.043+0000 7f80960ad700 1 -- 172.21.15.32:0/2470768197 wait complete. 2024-04-06T14:40:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.146+0000 7fefb11b9700 1 -- 172.21.15.32:0/1706637663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac1037a0 msgr2=0x7fefac0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.146+0000 7fefb11b9700 1 --2- 172.21.15.32:0/1706637663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac1037a0 0x7fefac0fce10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fefa0004940 tx=0x7fefa0009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.146+0000 7fefb11b9700 1 -- 172.21.15.32:0/1706637663 shutdown_connections 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.146+0000 7fefb11b9700 1 --2- 172.21.15.32:0/1706637663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac1037a0 0x7fefac0fce10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.146+0000 7fefb11b9700 1 --2- 172.21.15.32:0/1706637663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefac102e00 0x7fefac1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.146+0000 7fefb11b9700 1 -- 172.21.15.32:0/1706637663 >> 172.21.15.32:0/1706637663 conn(0x7fefac0f89b0 msgr2=0x7fefac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.146+0000 7fefb11b9700 1 -- 172.21.15.32:0/1706637663 shutdown_connections 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.146+0000 7fefb11b9700 1 -- 172.21.15.32:0/1706637663 wait complete. 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.147+0000 7fefb11b9700 1 Processor -- start 2024-04-06T14:40:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.147+0000 7fefb11b9700 1 -- start start 2024-04-06T14:40:42.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.147+0000 7fefb11b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac102e00 0x7fefac1138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:42.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.147+0000 7fefb11b9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefac1037a0 0x7fefac113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:42.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.147+0000 7fefb11b9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac1144e0 con 0x7fefac102e00 2024-04-06T14:40:42.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.147+0000 7fefb11b9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac10da50 con 0x7fefac1037a0 2024-04-06T14:40:42.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.147+0000 7fefaad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac102e00 0x7fefac1138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:42.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.147+0000 7fefaad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac102e00 0x7fefac1138c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59344/0 (socket says 172.21.15.32:59344) 2024-04-06T14:40:42.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.147+0000 7fefaad9d700 1 -- 172.21.15.32:0/1476742062 learned_addr learned my addr 172.21.15.32:0/1476742062 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:42.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fefaa59c700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefac1037a0 0x7fefac113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:42.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fefaad9d700 1 -- 172.21.15.32:0/1476742062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefac1037a0 msgr2=0x7fefac113e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fefaad9d700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefac1037a0 0x7fefac113e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fefaad9d700 1 -- 172.21.15.32:0/1476742062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa0005040 con 0x7fefac102e00 2024-04-06T14:40:42.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fefaad9d700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac102e00 0x7fefac1138c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00b790 tx=0x7fef9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:42.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fef937fe700 1 -- 172.21.15.32:0/1476742062 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9c00f770 con 0x7fefac102e00 2024-04-06T14:40:42.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fef937fe700 1 -- 172.21.15.32:0/1476742062 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef9c00fd80 con 0x7fefac102e00 2024-04-06T14:40:42.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac10dd30 con 0x7fefac102e00 2024-04-06T14:40:42.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fef937fe700 1 -- 172.21.15.32:0/1476742062 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9c00d310 con 0x7fefac102e00 2024-04-06T14:40:42.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.148+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac10e2f0 con 0x7fefac102e00 2024-04-06T14:40:42.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.149+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefac0fe550 con 0x7fefac102e00 2024-04-06T14:40:42.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.149+0000 7fef937fe700 1 -- 172.21.15.32:0/1476742062 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fef9c010420 con 0x7fefac102e00 2024-04-06T14:40:42.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.150+0000 7fef937fe700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fef9406cc20 0x7fef9406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:42.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.150+0000 7fefaa59c700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fef9406cc20 0x7fef9406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:42.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.150+0000 7fef937fe700 1 -- 172.21.15.32:0/1476742062 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fef9c08bf30 con 0x7fefac102e00 2024-04-06T14:40:42.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.150+0000 7fefaa59c700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fef9406cc20 0x7fef9406f0d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fefac10ec10 tx=0x7fefa0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:42.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.155+0000 7fef937fe700 1 -- 172.21.15.32:0/1476742062 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fef9c0bfab0 con 0x7fefac102e00 2024-04-06T14:40:42.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.374+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefac109e80 con 0x7fef9406cc20 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.376+0000 7fef937fe700 1 -- 172.21.15.32:0/1476742062 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fefac109e80 con 0x7fef9406cc20 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:40:42.368 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:40:42.369 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:40:42.369 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:40:42.369 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:40:42.369 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:40:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fef9406cc20 msgr2=0x7fef9406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fef9406cc20 0x7fef9406f0d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fefac10ec10 tx=0x7fefa0009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac102e00 msgr2=0x7fefac1138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac102e00 0x7fefac1138c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00b790 tx=0x7fef9c009250 comp rx=0 tx=0).stop 2024-04-06T14:40:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 shutdown_connections 2024-04-06T14:40:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fef9406cc20 0x7fef9406f0d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefac102e00 0x7fefac1138c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 --2- 172.21.15.32:0/1476742062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefac1037a0 0x7fefac113e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 >> 172.21.15.32:0/1476742062 conn(0x7fefac0f89b0 msgr2=0x7fefac0f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:42.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 shutdown_connections 2024-04-06T14:40:42.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.378+0000 7fefb11b9700 1 -- 172.21.15.32:0/1476742062 wait complete. 2024-04-06T14:40:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.481+0000 7fb022f97700 1 -- 172.21.15.32:0/118776028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 msgr2=0x7fb01c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.481+0000 7fb022f97700 1 --2- 172.21.15.32:0/118776028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 0x7fb01c0fb910 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb008004940 tx=0x7fb008009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.481+0000 7fb022f97700 1 -- 172.21.15.32:0/118776028 shutdown_connections 2024-04-06T14:40:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.481+0000 7fb022f97700 1 --2- 172.21.15.32:0/118776028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 0x7fb01c0fb910 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.481+0000 7fb022f97700 1 --2- 172.21.15.32:0/118776028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb01c102e70 0x7fb01c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.481+0000 7fb022f97700 1 -- 172.21.15.32:0/118776028 >> 172.21.15.32:0/118776028 conn(0x7fb01c0714c0 msgr2=0x7fb01c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.481+0000 7fb022f97700 1 -- 172.21.15.32:0/118776028 shutdown_connections 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.481+0000 7fb022f97700 1 -- 172.21.15.32:0/118776028 wait complete. 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.482+0000 7fb022f97700 1 Processor -- start 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.482+0000 7fb022f97700 1 -- start start 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.482+0000 7fb022f97700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 0x7fb01c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.482+0000 7fb022f97700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb01c102e70 0x7fb01c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.482+0000 7fb022f97700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c114600 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.482+0000 7fb022f97700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c10d9f0 con 0x7fb01c102e70 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.482+0000 7fb020d33700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 0x7fb01c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.482+0000 7fb020d33700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 0x7fb01c113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59366/0 (socket says 172.21.15.32:59366) 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.482+0000 7fb020d33700 1 -- 172.21.15.32:0/847774252 learned_addr learned my addr 172.21.15.32:0/847774252 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb01bfff700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb01c102e70 0x7fb01c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb020d33700 1 -- 172.21.15.32:0/847774252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb01c102e70 msgr2=0x7fb01c113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb020d33700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb01c102e70 0x7fb01c113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb020d33700 1 -- 172.21.15.32:0/847774252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008005040 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb01bfff700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb01c102e70 0x7fb01c113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:40:42.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb020d33700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 0x7fb01c113a70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb010002a50 tx=0x7fb0100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:42.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb0197fa700 1 -- 172.21.15.32:0/847774252 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb010007cc0 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb0197fa700 1 -- 172.21.15.32:0/847774252 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb01000f040 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c10dcd0 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb0197fa700 1 -- 172.21.15.32:0/847774252 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb010009540 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.483+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c10e290 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.484+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c100110 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.484+0000 7fb0197fa700 1 -- 172.21.15.32:0/847774252 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb0100096a0 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.485+0000 7fb0197fa700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb00c06cc30 0x7fb00c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:40:42.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.485+0000 7fb0197fa700 1 -- 172.21.15.32:0/847774252 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb01008bd70 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.485+0000 7fb01bfff700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb00c06cc30 0x7fb00c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:40:42.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.485+0000 7fb01bfff700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb00c06cc30 0x7fb00c06f0e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb01c10ebb0 tx=0x7fb008009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:40:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.489+0000 7fb0197fa700 1 -- 172.21.15.32:0/847774252 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb010059840 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.780+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb01c10f0c0 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.780+0000 7fb0197fa700 1 -- 172.21.15.32:0/847774252 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fb0100593d0 con 0x7fb01c0fb4a0 2024-04-06T14:40:42.772 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:40:42.773 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:40:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:40:42.773 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:40:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:40:42.773 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:40:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:40:42.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb00c06cc30 msgr2=0x7fb00c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb00c06cc30 0x7fb00c06f0e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb01c10ebb0 tx=0x7fb008009e30 comp rx=0 tx=0).stop 2024-04-06T14:40:42.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 msgr2=0x7fb01c113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:40:42.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 0x7fb01c113a70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb010002a50 tx=0x7fb0100046b0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 shutdown_connections 2024-04-06T14:40:42.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb00c06cc30 0x7fb00c06f0e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb01c0fb4a0 0x7fb01c113a70 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 --2- 172.21.15.32:0/847774252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb01c102e70 0x7fb01c113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:40:42.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 >> 172.21.15.32:0/847774252 conn(0x7fb01c0714c0 msgr2=0x7fb01c0fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:40:42.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 shutdown_connections 2024-04-06T14:40:42.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:40:42.782+0000 7fb022f97700 1 -- 172.21.15.32:0/847774252 wait complete. 2024-04-06T14:40:42.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:42 smithi086 ceph-mon[110975]: pgmap v616: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:42.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:42 smithi086 ceph-mon[110975]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:42.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:42 smithi086 ceph-mon[110975]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:42.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:42 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2209504376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:40:42.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:42 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2470768197' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:40:42.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:42 smithi032 ceph-mon[147414]: pgmap v616: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:42.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:42 smithi032 ceph-mon[147414]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:42.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:42 smithi032 ceph-mon[147414]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:42.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:42 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2209504376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:40:42.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:42 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2470768197' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:40:43.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:43 smithi032 ceph-mon[147414]: from='client.35168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:43.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:43 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/847774252' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:40:43.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:43 smithi086 ceph-mon[110975]: from='client.35168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:40:43.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:43 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/847774252' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:40:44.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:44 smithi086 ceph-mon[110975]: pgmap v617: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:44.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:44 smithi032 ceph-mon[147414]: pgmap v617: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:46.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:46 smithi086 ceph-mon[110975]: pgmap v618: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:46.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:46 smithi032 ceph-mon[147414]: pgmap v618: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:48.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:48 smithi086 ceph-mon[110975]: pgmap v619: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:48.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:48 smithi032 ceph-mon[147414]: pgmap v619: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:50.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:50 smithi086 ceph-mon[110975]: pgmap v620: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:50.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:50 smithi032 ceph-mon[147414]: pgmap v620: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:52.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:52 smithi086 ceph-mon[110975]: pgmap v621: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:52.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:52 smithi032 ceph-mon[147414]: pgmap v621: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:53.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:40:53.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:40:54.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:54 smithi086 ceph-mon[110975]: pgmap v622: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:54.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:54 smithi032 ceph-mon[147414]: pgmap v622: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:56.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:56 smithi086 ceph-mon[110975]: pgmap v623: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:56.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:56 smithi032 ceph-mon[147414]: pgmap v623: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:58.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:40:58 smithi086 ceph-mon[110975]: pgmap v624: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:40:58.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:40:58 smithi032 ceph-mon[147414]: pgmap v624: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:00.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:00 smithi086 ceph-mon[110975]: pgmap v625: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:00.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:00 smithi032 ceph-mon[147414]: pgmap v625: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:02.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:02 smithi086 ceph-mon[110975]: pgmap v626: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:02.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:02 smithi032 ceph-mon[147414]: pgmap v626: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:04.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:04 smithi086 ceph-mon[110975]: pgmap v627: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:04.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:04 smithi032 ceph-mon[147414]: pgmap v627: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:06.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:06 smithi086 ceph-mon[110975]: pgmap v628: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:06.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:06 smithi032 ceph-mon[147414]: pgmap v628: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:08.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:08 smithi086 ceph-mon[110975]: pgmap v629: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:08.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:41:08.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:08 smithi032 ceph-mon[147414]: pgmap v629: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:08.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:41:10.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:10 smithi086 ceph-mon[110975]: pgmap v630: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:10.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:10 smithi032 ceph-mon[147414]: pgmap v630: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:12.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:12 smithi086 ceph-mon[110975]: pgmap v631: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:12.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.899+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/2204783773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa541037b0 msgr2=0x7ffa540fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:12.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.899+0000 7ffa5ab6d700 1 --2- 172.21.15.32:0/2204783773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa541037b0 0x7ffa540fce20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffa40004940 tx=0x7ffa40009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:12.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.899+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/2204783773 shutdown_connections 2024-04-06T14:41:12.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.899+0000 7ffa5ab6d700 1 --2- 172.21.15.32:0/2204783773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa541037b0 0x7ffa540fce20 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:12.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.899+0000 7ffa5ab6d700 1 --2- 172.21.15.32:0/2204783773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa54102e10 0x7ffa541031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:12.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.899+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/2204783773 >> 172.21.15.32:0/2204783773 conn(0x7ffa540f89a0 msgr2=0x7ffa540fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:12.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.899+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/2204783773 shutdown_connections 2024-04-06T14:41:12.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.899+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/2204783773 wait complete. 2024-04-06T14:41:12.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa5ab6d700 1 Processor -- start 2024-04-06T14:41:12.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa5ab6d700 1 -- start start 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa5ab6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa54102e10 0x7ffa540702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa5ab6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa541037b0 0x7ffa54070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa5ab6d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa54070d90 con 0x7ffa54102e10 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa5ab6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa54070ed0 con 0x7ffa541037b0 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa58909700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa54102e10 0x7ffa540702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa58909700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa54102e10 0x7ffa540702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39034/0 (socket says 172.21.15.32:39034) 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa58909700 1 -- 172.21.15.32:0/3509877489 learned_addr learned my addr 172.21.15.32:0/3509877489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.900+0000 7ffa53fff700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa541037b0 0x7ffa54070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.901+0000 7ffa58909700 1 -- 172.21.15.32:0/3509877489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa541037b0 msgr2=0x7ffa54070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.901+0000 7ffa58909700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa541037b0 0x7ffa54070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.901+0000 7ffa58909700 1 -- 172.21.15.32:0/3509877489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa40005040 con 0x7ffa54102e10 2024-04-06T14:41:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.901+0000 7ffa58909700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa54102e10 0x7ffa540702c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ffa48002a50 tx=0x7ffa480046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.901+0000 7ffa517fa700 1 -- 172.21.15.32:0/3509877489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa48007cc0 con 0x7ffa54102e10 2024-04-06T14:41:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.901+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa54119dd0 con 0x7ffa54102e10 2024-04-06T14:41:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.901+0000 7ffa517fa700 1 -- 172.21.15.32:0/3509877489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa4800f040 con 0x7ffa54102e10 2024-04-06T14:41:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.901+0000 7ffa517fa700 1 -- 172.21.15.32:0/3509877489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa4800d9c0 con 0x7ffa54102e10 2024-04-06T14:41:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.901+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5411a250 con 0x7ffa54102e10 2024-04-06T14:41:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.902+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa540fe560 con 0x7ffa54102e10 2024-04-06T14:41:12.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.903+0000 7ffa517fa700 1 -- 172.21.15.32:0/3509877489 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ffa4800db20 con 0x7ffa54102e10 2024-04-06T14:41:12.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.903+0000 7ffa517fa700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffa4406cc30 0x7ffa4406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:12.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.903+0000 7ffa517fa700 1 -- 172.21.15.32:0/3509877489 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffa4808bdb0 con 0x7ffa54102e10 2024-04-06T14:41:12.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.903+0000 7ffa53fff700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffa4406cc30 0x7ffa4406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:12.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.904+0000 7ffa53fff700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffa4406cc30 0x7ffa4406f0e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffa54071c00 tx=0x7ffa40009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:12.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:12.908+0000 7ffa517fa700 1 -- 172.21.15.32:0/3509877489 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ffa48055d70 con 0x7ffa54102e10 2024-04-06T14:41:12.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:12 smithi032 ceph-mon[147414]: pgmap v631: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:13.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.129+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa540719b0 con 0x7ffa4406cc30 2024-04-06T14:41:13.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.130+0000 7ffa517fa700 1 -- 172.21.15.32:0/3509877489 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ffa540719b0 con 0x7ffa4406cc30 2024-04-06T14:41:13.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.132+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffa4406cc30 msgr2=0x7ffa4406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.132+0000 7ffa5ab6d700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffa4406cc30 0x7ffa4406f0e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffa54071c00 tx=0x7ffa40009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:13.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.132+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa54102e10 msgr2=0x7ffa540702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.132+0000 7ffa5ab6d700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa54102e10 0x7ffa540702c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ffa48002a50 tx=0x7ffa480046b0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.133+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 shutdown_connections 2024-04-06T14:41:13.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.133+0000 7ffa5ab6d700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffa4406cc30 0x7ffa4406f0e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.133+0000 7ffa5ab6d700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa54102e10 0x7ffa540702c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.133+0000 7ffa5ab6d700 1 --2- 172.21.15.32:0/3509877489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa541037b0 0x7ffa54070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.133+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 >> 172.21.15.32:0/3509877489 conn(0x7ffa540f89a0 msgr2=0x7ffa540f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:13.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.133+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 shutdown_connections 2024-04-06T14:41:13.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.133+0000 7ffa5ab6d700 1 -- 172.21.15.32:0/3509877489 wait complete. 2024-04-06T14:41:13.135 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:41:13.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.237+0000 7fa1f03fb700 1 -- 172.21.15.32:0/712153634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 msgr2=0x7fa1e8106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.237+0000 7fa1f03fb700 1 --2- 172.21.15.32:0/712153634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 0x7fa1e8106f40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc004940 tx=0x7fa1dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.237+0000 7fa1f03fb700 1 -- 172.21.15.32:0/712153634 shutdown_connections 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.237+0000 7fa1f03fb700 1 --2- 172.21.15.32:0/712153634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 0x7fa1e8106f40 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.237+0000 7fa1f03fb700 1 --2- 172.21.15.32:0/712153634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e80fded0 0x7fa1e80fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.237+0000 7fa1f03fb700 1 -- 172.21.15.32:0/712153634 >> 172.21.15.32:0/712153634 conn(0x7fa1e8072bc0 msgr2=0x7fa1e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.237+0000 7fa1f03fb700 1 -- 172.21.15.32:0/712153634 shutdown_connections 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.237+0000 7fa1f03fb700 1 -- 172.21.15.32:0/712153634 wait complete. 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1f03fb700 1 Processor -- start 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1f03fb700 1 -- start start 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1f03fb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e80fded0 0x7fa1e8113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1f03fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 0x7fa1e8113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1f03fb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81145f0 con 0x7fa1e80fe870 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1f03fb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e810d9e0 con 0x7fa1e80fded0 2024-04-06T14:41:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1ed996700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 0x7fa1e8113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1ed996700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 0x7fa1e8113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39058/0 (socket says 172.21.15.32:39058) 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1ed996700 1 -- 172.21.15.32:0/1582381793 learned_addr learned my addr 172.21.15.32:0/1582381793 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1ee197700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e80fded0 0x7fa1e8113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1ed996700 1 -- 172.21.15.32:0/1582381793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e80fded0 msgr2=0x7fa1e8113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1ed996700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e80fded0 0x7fa1e8113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1ed996700 1 -- 172.21.15.32:0/1582381793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc005040 con 0x7fa1e80fe870 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.238+0000 7fa1ed996700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 0x7fa1e8113fa0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009380 tx=0x7fa1dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.239+0000 7fa1daffd700 1 -- 172.21.15.32:0/1582381793 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc00e040 con 0x7fa1e80fe870 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.239+0000 7fa1daffd700 1 -- 172.21.15.32:0/1582381793 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1dc004290 con 0x7fa1e80fe870 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.239+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e810dcc0 con 0x7fa1e80fe870 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.239+0000 7fa1daffd700 1 -- 172.21.15.32:0/1582381793 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc013e90 con 0x7fa1e80fe870 2024-04-06T14:41:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.239+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e810e200 con 0x7fa1e80fe870 2024-04-06T14:41:13.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.240+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e81019a0 con 0x7fa1e80fe870 2024-04-06T14:41:13.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.240+0000 7fa1daffd700 1 -- 172.21.15.32:0/1582381793 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa1dc0136b0 con 0x7fa1e80fe870 2024-04-06T14:41:13.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.241+0000 7fa1daffd700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1d406cc20 0x7fa1d406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:13.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.241+0000 7fa1ee197700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1d406cc20 0x7fa1d406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:13.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.241+0000 7fa1daffd700 1 -- 172.21.15.32:0/1582381793 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa1dc08d340 con 0x7fa1e80fe870 2024-04-06T14:41:13.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.241+0000 7fa1ee197700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1d406cc20 0x7fa1d406f0d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4007fd0 tx=0x7fa1e4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:13.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.245+0000 7fa1daffd700 1 -- 172.21.15.32:0/1582381793 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa1dc05ae10 con 0x7fa1e80fe870 2024-04-06T14:41:13.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.468+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1e810f0b0 con 0x7fa1d406cc20 2024-04-06T14:41:13.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.469+0000 7fa1daffd700 1 -- 172.21.15.32:0/1582381793 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa1e810f0b0 con 0x7fa1d406cc20 2024-04-06T14:41:13.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.471+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1d406cc20 msgr2=0x7fa1d406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.471+0000 7fa1f03fb700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1d406cc20 0x7fa1d406f0d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4007fd0 tx=0x7fa1e4009040 comp rx=0 tx=0).stop 2024-04-06T14:41:13.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.471+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 msgr2=0x7fa1e8113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.471+0000 7fa1f03fb700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 0x7fa1e8113fa0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009380 tx=0x7fa1dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.472+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 shutdown_connections 2024-04-06T14:41:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.472+0000 7fa1f03fb700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa1d406cc20 0x7fa1d406f0d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.472+0000 7fa1f03fb700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1e80fded0 0x7fa1e8113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.472+0000 7fa1f03fb700 1 --2- 172.21.15.32:0/1582381793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1e80fe870 0x7fa1e8113fa0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.472+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 >> 172.21.15.32:0/1582381793 conn(0x7fa1e8072bc0 msgr2=0x7fa1e80ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.472+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 shutdown_connections 2024-04-06T14:41:13.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.472+0000 7fa1f03fb700 1 -- 172.21.15.32:0/1582381793 wait complete. 2024-04-06T14:41:13.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.575+0000 7f8c54d4d700 1 -- 172.21.15.32:0/313235138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 msgr2=0x7f8c500fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.575+0000 7f8c54d4d700 1 --2- 172.21.15.32:0/313235138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 0x7f8c500fce20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8c38004940 tx=0x7f8c38009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.575+0000 7f8c54d4d700 1 -- 172.21.15.32:0/313235138 shutdown_connections 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.575+0000 7f8c54d4d700 1 --2- 172.21.15.32:0/313235138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 0x7f8c500fce20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.575+0000 7f8c54d4d700 1 --2- 172.21.15.32:0/313235138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c50102e10 0x7f8c501031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.575+0000 7f8c54d4d700 1 -- 172.21.15.32:0/313235138 >> 172.21.15.32:0/313235138 conn(0x7f8c500f89a0 msgr2=0x7f8c500fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.575+0000 7f8c54d4d700 1 -- 172.21.15.32:0/313235138 shutdown_connections 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.575+0000 7f8c54d4d700 1 -- 172.21.15.32:0/313235138 wait complete. 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c54d4d700 1 Processor -- start 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c54d4d700 1 -- start start 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c54d4d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c50102e10 0x7f8c500702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c54d4d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 0x7f8c50070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c54d4d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c50070e20 con 0x7f8c501037b0 2024-04-06T14:41:13.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c54d4d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c50070f60 con 0x7f8c50102e10 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c4dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 0x7f8c50070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c4dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 0x7f8c50070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39066/0 (socket says 172.21.15.32:39066) 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c4dd9b700 1 -- 172.21.15.32:0/3435785618 learned_addr learned my addr 172.21.15.32:0/3435785618 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c4dd9b700 1 -- 172.21.15.32:0/3435785618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c50102e10 msgr2=0x7f8c500702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c4dd9b700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c50102e10 0x7f8c500702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c4dd9b700 1 -- 172.21.15.32:0/3435785618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c38005040 con 0x7f8c501037b0 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c4dd9b700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 0x7f8c50070800 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009380 tx=0x7f8c38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c46ffd700 1 -- 172.21.15.32:0/3435785618 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3800e040 con 0x7f8c501037b0 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.576+0000 7f8c46ffd700 1 -- 172.21.15.32:0/3435785618 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c38013420 con 0x7f8c501037b0 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.577+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c50119df0 con 0x7f8c501037b0 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.577+0000 7f8c46ffd700 1 -- 172.21.15.32:0/3435785618 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3801c630 con 0x7f8c501037b0 2024-04-06T14:41:13.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.577+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5011a2d0 con 0x7f8c501037b0 2024-04-06T14:41:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.577+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c500fe560 con 0x7f8c501037b0 2024-04-06T14:41:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.578+0000 7f8c46ffd700 1 -- 172.21.15.32:0/3435785618 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8c380139e0 con 0x7f8c501037b0 2024-04-06T14:41:13.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.578+0000 7f8c46ffd700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c3c06cc20 0x7f8c3c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.578+0000 7f8c46ffd700 1 -- 172.21.15.32:0/3435785618 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8c3808d0f0 con 0x7f8c501037b0 2024-04-06T14:41:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.578+0000 7f8c4e59c700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c3c06cc20 0x7f8c3c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:13.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.579+0000 7f8c4e59c700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c3c06cc20 0x7f8c3c06f0d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8c40006fd0 tx=0x7f8c40006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:13.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.583+0000 7f8c46ffd700 1 -- 172.21.15.32:0/3435785618 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8c380571f0 con 0x7f8c501037b0 2024-04-06T14:41:13.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.795+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c50109ec0 con 0x7f8c3c06cc20 2024-04-06T14:41:13.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.805+0000 7f8c46ffd700 1 -- 172.21.15.32:0/3435785618 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8c40009dd0 con 0x7f8c3c06cc20 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (21m) 14m ago 32m 27.8M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (18m) 14m ago 33m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (18m) 14m ago 32m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (21m) 14m ago 32m 84.4M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (15m) 14m ago 27m 12.5M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (15m) 14m ago 27m 14.6M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (14m) 14m ago 27m 10.2M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (26m) 14m ago 26m 17.7M - 17.2.7-819-g6699495c ec300eaa768b 101e8b252a21 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (23m) 14m ago 34m 508M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (22m) 14m ago 31m 451M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:41:13.798 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (20m) 14m ago 34m 54.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (19m) 14m ago 31m 43.0M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (22m) 14m ago 32m 16.8M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (22m) 14m ago 32m 17.6M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (18m) 14m ago 31m 83.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (17m) 14m ago 30m 83.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (17m) 14m ago 30m 79.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (16m) 14m ago 29m 92.5M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (16m) 14m ago 29m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (15m) 14m ago 28m 81.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:41:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (21m) 14m ago 32m 62.4M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c3c06cc20 msgr2=0x7f8c3c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c3c06cc20 0x7f8c3c06f0d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8c40006fd0 tx=0x7f8c40006db0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 msgr2=0x7f8c50070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 0x7f8c50070800 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009380 tx=0x7f8c38009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 shutdown_connections 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c3c06cc20 0x7f8c3c06f0d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c50102e10 0x7f8c500702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 --2- 172.21.15.32:0/3435785618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c501037b0 0x7f8c50070800 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 >> 172.21.15.32:0/3435785618 conn(0x7f8c500f89a0 msgr2=0x7f8c500f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:13.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 shutdown_connections 2024-04-06T14:41:13.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.807+0000 7f8c54d4d700 1 -- 172.21.15.32:0/3435785618 wait complete. 2024-04-06T14:41:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.911+0000 7fae5e317700 1 -- 172.21.15.32:0/2353435715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fde10 msgr2=0x7fae580fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.911+0000 7fae5e317700 1 --2- 172.21.15.32:0/2353435715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fde10 0x7fae580fe1e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fae48004940 tx=0x7fae48009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.911+0000 7fae5e317700 1 -- 172.21.15.32:0/2353435715 shutdown_connections 2024-04-06T14:41:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.911+0000 7fae5e317700 1 --2- 172.21.15.32:0/2353435715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae580fe7b0 0x7fae58106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.911+0000 7fae5e317700 1 --2- 172.21.15.32:0/2353435715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fde10 0x7fae580fe1e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.911+0000 7fae5e317700 1 -- 172.21.15.32:0/2353435715 >> 172.21.15.32:0/2353435715 conn(0x7fae580f78d0 msgr2=0x7fae580f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.911+0000 7fae5e317700 1 -- 172.21.15.32:0/2353435715 shutdown_connections 2024-04-06T14:41:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.911+0000 7fae5e317700 1 -- 172.21.15.32:0/2353435715 wait complete. 2024-04-06T14:41:13.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae5e317700 1 Processor -- start 2024-04-06T14:41:13.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae5e317700 1 -- start start 2024-04-06T14:41:13.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae5e317700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae580fde10 0x7fae58113b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae5e317700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fe7b0 0x7fae58114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae5e317700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae581146d0 con 0x7fae580fe7b0 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae5e317700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5810dac0 con 0x7fae580fde10 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae577fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fe7b0 0x7fae58114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae577fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fe7b0 0x7fae58114080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39072/0 (socket says 172.21.15.32:39072) 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae577fe700 1 -- 172.21.15.32:0/3178595724 learned_addr learned my addr 172.21.15.32:0/3178595724 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae57fff700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae580fde10 0x7fae58113b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae577fe700 1 -- 172.21.15.32:0/3178595724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae580fde10 msgr2=0x7fae58113b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae577fe700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae580fde10 0x7fae58113b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.912+0000 7fae577fe700 1 -- 172.21.15.32:0/3178595724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae48005040 con 0x7fae580fe7b0 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.913+0000 7fae577fe700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fe7b0 0x7fae58114080 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fae4c002a50 tx=0x7fae4c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:13.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.913+0000 7fae54ff9700 1 -- 172.21.15.32:0/3178595724 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae4c007cc0 con 0x7fae580fe7b0 2024-04-06T14:41:13.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.913+0000 7fae54ff9700 1 -- 172.21.15.32:0/3178595724 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae4c00f040 con 0x7fae580fe7b0 2024-04-06T14:41:13.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.913+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5810dda0 con 0x7fae580fe7b0 2024-04-06T14:41:13.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.913+0000 7fae54ff9700 1 -- 172.21.15.32:0/3178595724 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae4c00d9c0 con 0x7fae580fe7b0 2024-04-06T14:41:13.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.913+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5810e2e0 con 0x7fae580fe7b0 2024-04-06T14:41:13.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.914+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae580666f0 con 0x7fae580fe7b0 2024-04-06T14:41:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.914+0000 7fae54ff9700 1 -- 172.21.15.32:0/3178595724 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fae4c00db20 con 0x7fae580fe7b0 2024-04-06T14:41:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.915+0000 7fae54ff9700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae4006cc20 0x7fae4006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.915+0000 7fae54ff9700 1 -- 172.21.15.32:0/3178595724 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fae4c08bcb0 con 0x7fae580fe7b0 2024-04-06T14:41:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.915+0000 7fae57fff700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae4006cc20 0x7fae4006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.915+0000 7fae57fff700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae4006cc20 0x7fae4006f0d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fae48004f90 tx=0x7fae48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:13.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:13.920+0000 7fae54ff9700 1 -- 172.21.15.32:0/3178595724 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fae4c059780 con 0x7fae580fe7b0 2024-04-06T14:41:14.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.207+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fae58109d60 con 0x7fae580fe7b0 2024-04-06T14:41:14.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.209+0000 7fae54ff9700 1 -- 172.21.15.32:0/3178595724 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fae4c059310 con 0x7fae580fe7b0 2024-04-06T14:41:14.201 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:41:14.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:41:14.201 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 3 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:41:14.202 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-819-g6699495c (6699495ce2af437c1ff648f1736d2457f417a1aa) quincy (stable)": 1, 2024-04-06T14:41:14.203 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 13 2024-04-06T14:41:14.203 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:41:14.203 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae4006cc20 msgr2=0x7fae4006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae4006cc20 0x7fae4006f0d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fae48004f90 tx=0x7fae48009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fe7b0 msgr2=0x7fae58114080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fe7b0 0x7fae58114080 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fae4c002a50 tx=0x7fae4c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 shutdown_connections 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fae4006cc20 0x7fae4006f0d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae580fde10 0x7fae58113b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 --2- 172.21.15.32:0/3178595724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae580fe7b0 0x7fae58114080 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 >> 172.21.15.32:0/3178595724 conn(0x7fae580f78d0 msgr2=0x7fae580f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 shutdown_connections 2024-04-06T14:41:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.211+0000 7fae5e317700 1 -- 172.21.15.32:0/3178595724 wait complete. 2024-04-06T14:41:14.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.314+0000 7efe81d99700 1 -- 172.21.15.32:0/3473799305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 msgr2=0x7efe7c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.314+0000 7efe81d99700 1 --2- 172.21.15.32:0/3473799305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 0x7efe7c0fbd80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7efe70004940 tx=0x7efe70009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.315+0000 7efe81d99700 1 -- 172.21.15.32:0/3473799305 shutdown_connections 2024-04-06T14:41:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.315+0000 7efe81d99700 1 --2- 172.21.15.32:0/3473799305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 0x7efe7c0fbd80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.315+0000 7efe81d99700 1 --2- 172.21.15.32:0/3473799305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe7c101d70 0x7efe7c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.315+0000 7efe81d99700 1 -- 172.21.15.32:0/3473799305 >> 172.21.15.32:0/3473799305 conn(0x7efe7c072bc0 msgr2=0x7efe7c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.315+0000 7efe81d99700 1 -- 172.21.15.32:0/3473799305 shutdown_connections 2024-04-06T14:41:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.315+0000 7efe81d99700 1 -- 172.21.15.32:0/3473799305 wait complete. 2024-04-06T14:41:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.315+0000 7efe81d99700 1 Processor -- start 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.315+0000 7efe81d99700 1 -- start start 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.315+0000 7efe81d99700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe7c101d70 0x7efe7c102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe81d99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 0x7efe7c103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe81d99700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c1036a0 con 0x7efe7c102710 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe81d99700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c1037e0 con 0x7efe7c101d70 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe7affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 0x7efe7c103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe7affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 0x7efe7c103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39094/0 (socket says 172.21.15.32:39094) 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe7affd700 1 -- 172.21.15.32:0/3402036854 learned_addr learned my addr 172.21.15.32:0/3402036854 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe7b7fe700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe7c101d70 0x7efe7c102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:14.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe7affd700 1 -- 172.21.15.32:0/3402036854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe7c101d70 msgr2=0x7efe7c102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe7affd700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe7c101d70 0x7efe7c102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe7affd700 1 -- 172.21.15.32:0/3402036854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70005040 con 0x7efe7c102710 2024-04-06T14:41:14.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe7affd700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 0x7efe7c103110 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7efe700093b0 tx=0x7efe70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:14.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe63fff700 1 -- 172.21.15.32:0/3402036854 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe7000e040 con 0x7efe7c102710 2024-04-06T14:41:14.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe63fff700 1 -- 172.21.15.32:0/3402036854 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe70013420 con 0x7efe7c102710 2024-04-06T14:41:14.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c103a30 con 0x7efe7c102710 2024-04-06T14:41:14.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe63fff700 1 -- 172.21.15.32:0/3402036854 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe7001c630 con 0x7efe7c102710 2024-04-06T14:41:14.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.316+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c11a1f0 con 0x7efe7c102710 2024-04-06T14:41:14.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.317+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c0fd4c0 con 0x7efe7c102710 2024-04-06T14:41:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.318+0000 7efe63fff700 1 -- 172.21.15.32:0/3402036854 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efe700139e0 con 0x7efe7c102710 2024-04-06T14:41:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.318+0000 7efe63fff700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe6406cc20 0x7efe6406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.318+0000 7efe63fff700 1 -- 172.21.15.32:0/3402036854 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efe7008d2b0 con 0x7efe7c102710 2024-04-06T14:41:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.318+0000 7efe7b7fe700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe6406cc20 0x7efe6406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.319+0000 7efe7b7fe700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe6406cc20 0x7efe6406f0d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efe6c004d60 tx=0x7efe6c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:14.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.323+0000 7efe63fff700 1 -- 172.21.15.32:0/3402036854 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efe700573a0 con 0x7efe7c102710 2024-04-06T14:41:14.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.570+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efe7c108d40 con 0x7efe7c102710 2024-04-06T14:41:14.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.571+0000 7efe63fff700 1 -- 172.21.15.32:0/3402036854 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1836 (secure 0 0 0) 0x7efe7005a9c0 con 0x7efe7c102710 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:e30 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:41:14.564 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:41:14.565 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:24313} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/2372611428,v1:172.21.15.86:6827/2372611428] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:41:14.566 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe6406cc20 msgr2=0x7efe6406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe6406cc20 0x7efe6406f0d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efe6c004d60 tx=0x7efe6c00b040 comp rx=0 tx=0).stop 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 msgr2=0x7efe7c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 0x7efe7c103110 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7efe700093b0 tx=0x7efe70009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 shutdown_connections 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe6406cc20 0x7efe6406f0d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe7c101d70 0x7efe7c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 --2- 172.21.15.32:0/3402036854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7c102710 0x7efe7c103110 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 >> 172.21.15.32:0/3402036854 conn(0x7efe7c072bc0 msgr2=0x7efe7c065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:14.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 shutdown_connections 2024-04-06T14:41:14.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.573+0000 7efe81d99700 1 -- 172.21.15.32:0/3402036854 wait complete. 2024-04-06T14:41:14.568 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:41:14.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.677+0000 7f0be2b32700 1 -- 172.21.15.32:0/2209181664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc102710 msgr2=0x7f0bdc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.677+0000 7f0be2b32700 1 --2- 172.21.15.32:0/2209181664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc102710 0x7f0bdc0fbd80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8004940 tx=0x7f0bc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.677+0000 7f0be2b32700 1 -- 172.21.15.32:0/2209181664 shutdown_connections 2024-04-06T14:41:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.677+0000 7f0be2b32700 1 --2- 172.21.15.32:0/2209181664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc102710 0x7f0bdc0fbd80 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.677+0000 7f0be2b32700 1 --2- 172.21.15.32:0/2209181664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bdc101d70 0x7f0bdc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.677+0000 7f0be2b32700 1 -- 172.21.15.32:0/2209181664 >> 172.21.15.32:0/2209181664 conn(0x7f0bdc072bc0 msgr2=0x7f0bdc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.677+0000 7f0be2b32700 1 -- 172.21.15.32:0/2209181664 shutdown_connections 2024-04-06T14:41:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.677+0000 7f0be2b32700 1 -- 172.21.15.32:0/2209181664 wait complete. 2024-04-06T14:41:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.678+0000 7f0be2b32700 1 Processor -- start 2024-04-06T14:41:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.678+0000 7f0be2b32700 1 -- start start 2024-04-06T14:41:14.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.678+0000 7f0be2b32700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc101d70 0x7f0bdc102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.678+0000 7f0be2b32700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bdc102710 0x7f0bdc103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.678+0000 7f0be2b32700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc103650 con 0x7f0bdc101d70 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.678+0000 7f0be2b32700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1037c0 con 0x7f0bdc102710 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.678+0000 7f0be08ce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc101d70 0x7f0bdc102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.678+0000 7f0be08ce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc101d70 0x7f0bdc102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60334/0 (socket says 172.21.15.32:60334) 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.678+0000 7f0be08ce700 1 -- 172.21.15.32:0/1460288475 learned_addr learned my addr 172.21.15.32:0/1460288475 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0bdbfff700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bdc102710 0x7f0bdc103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0be08ce700 1 -- 172.21.15.32:0/1460288475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bdc102710 msgr2=0x7f0bdc103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0be08ce700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bdc102710 0x7f0bdc103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0be08ce700 1 -- 172.21.15.32:0/1460288475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8005040 con 0x7f0bdc101d70 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0be08ce700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc101d70 0x7f0bdc102bd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c830 tx=0x7f0bd0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0bd97fa700 1 -- 172.21.15.32:0/1460288475 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd00106f0 con 0x7f0bdc101d70 2024-04-06T14:41:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc119de0 con 0x7f0bdc101d70 2024-04-06T14:41:14.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0bd97fa700 1 -- 172.21.15.32:0/1460288475 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bd0010d00 con 0x7f0bdc101d70 2024-04-06T14:41:14.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc11a1e0 con 0x7f0bdc101d70 2024-04-06T14:41:14.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.679+0000 7f0bd97fa700 1 -- 172.21.15.32:0/1460288475 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd0012e70 con 0x7f0bdc101d70 2024-04-06T14:41:14.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.680+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc0fd4c0 con 0x7f0bdc101d70 2024-04-06T14:41:14.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.681+0000 7f0bd97fa700 1 -- 172.21.15.32:0/1460288475 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0bd0010850 con 0x7f0bdc101d70 2024-04-06T14:41:14.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.681+0000 7f0bd97fa700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bcc06cc30 0x7f0bcc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:14.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.681+0000 7f0bd97fa700 1 -- 172.21.15.32:0/1460288475 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0bd008c030 con 0x7f0bdc101d70 2024-04-06T14:41:14.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.681+0000 7f0bdbfff700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bcc06cc30 0x7f0bcc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:14.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.682+0000 7f0bdbfff700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bcc06cc30 0x7f0bcc06f0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0fa780 tx=0x7f0bc8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.686+0000 7f0bd97fa700 1 -- 172.21.15.32:0/1460288475 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0bd0056070 con 0x7f0bdc101d70 2024-04-06T14:41:14.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:14 smithi086 ceph-mon[110975]: pgmap v632: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:14.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:14 smithi086 ceph-mon[110975]: from='client.35176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:14.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:14 smithi086 ceph-mon[110975]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:14.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:14 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3178595724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:14.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:14 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3402036854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:41:14.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.911+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bdc04bf40 con 0x7f0bcc06cc30 2024-04-06T14:41:14.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.912+0000 7f0bd97fa700 1 -- 172.21.15.32:0/1460288475 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0bdc04bf40 con 0x7f0bcc06cc30 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:41:14.905 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:41:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bcc06cc30 msgr2=0x7f0bcc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bcc06cc30 0x7f0bcc06f0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0fa780 tx=0x7f0bc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc101d70 msgr2=0x7f0bdc102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc101d70 0x7f0bdc102bd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c830 tx=0x7f0bd0004470 comp rx=0 tx=0).stop 2024-04-06T14:41:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 shutdown_connections 2024-04-06T14:41:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0bcc06cc30 0x7f0bcc06f0e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bdc101d70 0x7f0bdc102bd0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 --2- 172.21.15.32:0/1460288475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bdc102710 0x7f0bdc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:14.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 >> 172.21.15.32:0/1460288475 conn(0x7f0bdc072bc0 msgr2=0x7f0bdc065a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:14.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 shutdown_connections 2024-04-06T14:41:14.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:14.914+0000 7f0be2b32700 1 -- 172.21.15.32:0/1460288475 wait complete. 2024-04-06T14:41:14.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:14 smithi032 ceph-mon[147414]: pgmap v632: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:14.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:14 smithi032 ceph-mon[147414]: from='client.35176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:14.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:14 smithi032 ceph-mon[147414]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:14.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:14 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3178595724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:14.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:14 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3402036854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:41:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.017+0000 7f16206db700 1 -- 172.21.15.32:0/1205410107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 msgr2=0x7f16180fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.017+0000 7f16206db700 1 --2- 172.21.15.32:0/1205410107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 0x7f16180fbd80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f160c004940 tx=0x7f160c009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.017+0000 7f16206db700 1 -- 172.21.15.32:0/1205410107 shutdown_connections 2024-04-06T14:41:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.017+0000 7f16206db700 1 --2- 172.21.15.32:0/1205410107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 0x7f16180fbd80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.017+0000 7f16206db700 1 --2- 172.21.15.32:0/1205410107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1618101d70 0x7f1618102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.017+0000 7f16206db700 1 -- 172.21.15.32:0/1205410107 >> 172.21.15.32:0/1205410107 conn(0x7f1618072bc0 msgr2=0x7f1618072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.017+0000 7f16206db700 1 -- 172.21.15.32:0/1205410107 shutdown_connections 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.017+0000 7f16206db700 1 -- 172.21.15.32:0/1205410107 wait complete. 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f16206db700 1 Processor -- start 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f16206db700 1 -- start start 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f16206db700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1618101d70 0x7f16181129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f16206db700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 0x7f1618112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f16206db700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1618113550 con 0x7f1618102710 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f16206db700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161810c940 con 0x7f1618101d70 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f161dc76700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 0x7f1618112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f161dc76700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 0x7f1618112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60342/0 (socket says 172.21.15.32:60342) 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f161dc76700 1 -- 172.21.15.32:0/3132091766 learned_addr learned my addr 172.21.15.32:0/3132091766 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f161dc76700 1 -- 172.21.15.32:0/3132091766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1618101d70 msgr2=0x7f16181129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f161dc76700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1618101d70 0x7f16181129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f161dc76700 1 -- 172.21.15.32:0/3132091766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c005040 con 0x7f1618102710 2024-04-06T14:41:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.018+0000 7f161dc76700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 0x7f1618112f00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f160c0093b0 tx=0x7f160c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.019+0000 7f160affd700 1 -- 172.21.15.32:0/3132091766 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160c00e040 con 0x7f1618102710 2024-04-06T14:41:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.019+0000 7f160affd700 1 -- 172.21.15.32:0/3132091766 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f160c013420 con 0x7f1618102710 2024-04-06T14:41:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.019+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161810cbc0 con 0x7f1618102710 2024-04-06T14:41:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.019+0000 7f160affd700 1 -- 172.21.15.32:0/3132091766 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160c01c630 con 0x7f1618102710 2024-04-06T14:41:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.019+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161810d180 con 0x7f1618102710 2024-04-06T14:41:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.020+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16180ff080 con 0x7f1618102710 2024-04-06T14:41:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.020+0000 7f160affd700 1 -- 172.21.15.32:0/3132091766 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f160c0139e0 con 0x7f1618102710 2024-04-06T14:41:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.021+0000 7f160affd700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.021+0000 7f160affd700 1 -- 172.21.15.32:0/3132091766 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f160c08d110 con 0x7f1618102710 2024-04-06T14:41:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.021+0000 7f161e477700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.021+0000 7f161e477700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1614004ad0 tx=0x7f1614007d40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:15.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.025+0000 7f160affd700 1 -- 172.21.15.32:0/3132091766 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f160c05aca0 con 0x7f1618102710 2024-04-06T14:41:15.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.315+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f161810e010 con 0x7f1618102710 2024-04-06T14:41:15.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.316+0000 7f160affd700 1 -- 172.21.15.32:0/3132091766 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f160c05a830 con 0x7f1618102710 2024-04-06T14:41:15.309 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:41:15.309 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:41:15.309 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:41:15.309 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:41:15.309 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:41:15.309 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:41:15.309 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 msgr2=0x7f160406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1614004ad0 tx=0x7f1614007d40 comp rx=0 tx=0).stop 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 msgr2=0x7f1618112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 0x7f1618112f00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f160c0093b0 tx=0x7f160c009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 shutdown_connections 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f160406cc20 0x7f160406f0d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1618101d70 0x7f16181129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 --2- 172.21.15.32:0/3132091766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1618102710 0x7f1618112f00 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 >> 172.21.15.32:0/3132091766 conn(0x7f1618072bc0 msgr2=0x7f16180fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 shutdown_connections 2024-04-06T14:41:15.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:15.318+0000 7f16206db700 1 -- 172.21.15.32:0/3132091766 wait complete. 2024-04-06T14:41:15.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:15 smithi086 ceph-mon[110975]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:15.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:15 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3132091766' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:41:15.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:15 smithi032 ceph-mon[147414]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:15.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:15 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3132091766' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:41:16.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:16 smithi086 ceph-mon[110975]: pgmap v633: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:16.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:16 smithi086 ceph-mon[110975]: from='client.35196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:16.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:16 smithi032 ceph-mon[147414]: pgmap v633: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:16.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:16 smithi032 ceph-mon[147414]: from='client.35196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:18.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:18 smithi032 ceph-mon[147414]: pgmap v634: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:19.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:18 smithi086 ceph-mon[110975]: pgmap v634: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:20.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:20 smithi032 ceph-mon[147414]: pgmap v635: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:21.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:20 smithi086 ceph-mon[110975]: pgmap v635: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:22.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:22 smithi032 ceph-mon[147414]: pgmap v636: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:22.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:41:23.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:22 smithi086 ceph-mon[110975]: pgmap v636: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:23.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:41:23.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:23 smithi032 ceph-mon[147414]: pgmap v637: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:24.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:23 smithi086 ceph-mon[110975]: pgmap v637: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:26.472 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:26 smithi032 ceph-mon[147414]: pgmap v638: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:26.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:26 smithi086 ceph-mon[110975]: pgmap v638: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:28.472 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:28 smithi032 ceph-mon[147414]: pgmap v639: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:28.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:28 smithi086 ceph-mon[110975]: pgmap v639: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:30.472 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:30 smithi032 ceph-mon[147414]: pgmap v640: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:30.472 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:30 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:41:30.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:30 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:41:30.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:30 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:30.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:30 smithi086 ceph-mon[110975]: pgmap v640: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:30.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:30 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:41:30.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:30 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:41:30.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:30 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:31.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:31 smithi086 ceph-mon[110975]: pgmap v641: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:31.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:31 smithi086 ceph-mon[110975]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-06T14:41:31.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:41:31.370 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:31.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:31.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:31.371 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:31 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:31.472 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:31 smithi032 ceph-mon[147414]: pgmap v641: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:31.472 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:31 smithi032 ceph-mon[147414]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-06T14:41:31.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:41:31.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:31.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:31.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:31.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:31 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:33.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:32 smithi086 ceph-mon[110975]: Upgrade: Updating mds.cephfs.smithi086.ojmchr 2024-04-06T14:41:33.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:32 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:33.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:32 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.ojmchr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:41:33.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:32 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:41:33.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:32 smithi086 ceph-mon[110975]: Deploying daemon mds.cephfs.smithi086.ojmchr on smithi086 2024-04-06T14:41:33.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:32 smithi086 ceph-mon[110975]: pgmap v642: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:33.222 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:32 smithi032 ceph-mon[147414]: Upgrade: Updating mds.cephfs.smithi086.ojmchr 2024-04-06T14:41:33.222 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:32 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:33.222 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:32 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.ojmchr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T14:41:33.222 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:32 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:41:33.223 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:32 smithi032 ceph-mon[147414]: Deploying daemon mds.cephfs.smithi086.ojmchr on smithi086 2024-04-06T14:41:33.223 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:32 smithi032 ceph-mon[147414]: pgmap v642: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:34.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:34 smithi032 ceph-mon[147414]: osdmap e85: 6 total, 6 up, 6 in 2024-04-06T14:41:34.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:34 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-06T14:41:34.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:34 smithi032 ceph-mon[147414]: pgmap v644: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:35.026 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:34 smithi086 ceph-mon[110975]: osdmap e85: 6 total, 6 up, 6 in 2024-04-06T14:41:35.026 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:34 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-06T14:41:35.027 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:34 smithi086 ceph-mon[110975]: pgmap v644: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:37.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: pgmap v645: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] up:boot 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:41:37.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:41:37.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: pgmap v645: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:37.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:41:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] up:boot 2024-04-06T14:41:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:41:37.871 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:41:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:41:38.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:38.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:41:39.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:39 smithi086 ceph-mon[110975]: pgmap v646: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:39.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:39 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:39.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:39 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:39.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:39 smithi032 ceph-mon[147414]: pgmap v646: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:39.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:39 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:39.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:39 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:40.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:40 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:40.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:40 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:40.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:40 smithi032 ceph-mon[147414]: pgmap v647: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:41.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:40 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:41.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:40 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:41.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:40 smithi086 ceph-mon[110975]: pgmap v647: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:42.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:42 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:42.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:42 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:42.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:42 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:42.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:42 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:41:43.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:43 smithi032 ceph-mon[147414]: pgmap v648: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:43.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:43 smithi086 ceph-mon[110975]: pgmap v648: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:45.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.437+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/3030173934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b01037b0 msgr2=0x7fb9b00fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:45.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.437+0000 7fb9b7dfc700 1 --2- 172.21.15.32:0/3030173934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b01037b0 0x7fb9b00fce20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4004940 tx=0x7fb9a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:45.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.437+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/3030173934 shutdown_connections 2024-04-06T14:41:45.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.437+0000 7fb9b7dfc700 1 --2- 172.21.15.32:0/3030173934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b01037b0 0x7fb9b00fce20 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.437+0000 7fb9b7dfc700 1 --2- 172.21.15.32:0/3030173934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9b0102e10 0x7fb9b01031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.437+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/3030173934 >> 172.21.15.32:0/3030173934 conn(0x7fb9b00f89a0 msgr2=0x7fb9b00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:45.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.437+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/3030173934 shutdown_connections 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.437+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/3030173934 wait complete. 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.438+0000 7fb9b7dfc700 1 Processor -- start 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.438+0000 7fb9b7dfc700 1 -- start start 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.438+0000 7fb9b7dfc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b0102e10 0x7fb9b00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.438+0000 7fb9b7dfc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9b01037b0 0x7fb9b0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.438+0000 7fb9b7dfc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b0070d90 con 0x7fb9b0102e10 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.438+0000 7fb9b7dfc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b0070ed0 con 0x7fb9b01037b0 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.438+0000 7fb9b5b98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b0102e10 0x7fb9b00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9b5b98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b0102e10 0x7fb9b00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56854/0 (socket says 172.21.15.32:56854) 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9b5b98700 1 -- 172.21.15.32:0/945419773 learned_addr learned my addr 172.21.15.32:0/945419773 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9b5397700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9b01037b0 0x7fb9b0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9b5b98700 1 -- 172.21.15.32:0/945419773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9b01037b0 msgr2=0x7fb9b0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9b5b98700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9b01037b0 0x7fb9b0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9b5b98700 1 -- 172.21.15.32:0/945419773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a4005040 con 0x7fb9b0102e10 2024-04-06T14:41:45.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9b5b98700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b0102e10 0x7fb9b00702c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00c8c0 tx=0x7fb9ac0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:45.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9a27fc700 1 -- 172.21.15.32:0/945419773 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9ac00a980 con 0x7fb9b0102e10 2024-04-06T14:41:45.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b0119dd0 con 0x7fb9b0102e10 2024-04-06T14:41:45.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9a27fc700 1 -- 172.21.15.32:0/945419773 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9ac00ce60 con 0x7fb9b0102e10 2024-04-06T14:41:45.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9a27fc700 1 -- 172.21.15.32:0/945419773 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9ac0113a0 con 0x7fb9b0102e10 2024-04-06T14:41:45.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.439+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b011a250 con 0x7fb9b0102e10 2024-04-06T14:41:45.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.440+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b00fe560 con 0x7fb9b0102e10 2024-04-06T14:41:45.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.441+0000 7fb9a27fc700 1 -- 172.21.15.32:0/945419773 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb9ac012400 con 0x7fb9b0102e10 2024-04-06T14:41:45.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.441+0000 7fb9a27fc700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:45.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.441+0000 7fb9a27fc700 1 -- 172.21.15.32:0/945419773 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb9ac08c170 con 0x7fb9b0102e10 2024-04-06T14:41:45.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.441+0000 7fb9b5397700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:45.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.442+0000 7fb9b5397700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400b3b0 tx=0x7fb9a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:45.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.446+0000 7fb9a27fc700 1 -- 172.21.15.32:0/945419773 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb9ac0561b0 con 0x7fb9b0102e10 2024-04-06T14:41:45.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.668+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9b00719b0 con 0x7fb99c06cc20 2024-04-06T14:41:45.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.669+0000 7fb9a27fc700 1 -- 172.21.15.32:0/945419773 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb9b00719b0 con 0x7fb99c06cc20 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 msgr2=0x7fb99c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400b3b0 tx=0x7fb9a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b0102e10 msgr2=0x7fb9b00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b0102e10 0x7fb9b00702c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00c8c0 tx=0x7fb9ac0044c0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 shutdown_connections 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb99c06cc20 0x7fb99c06f0d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9b0102e10 0x7fb9b00702c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 --2- 172.21.15.32:0/945419773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9b01037b0 0x7fb9b0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 >> 172.21.15.32:0/945419773 conn(0x7fb9b00f89a0 msgr2=0x7fb9b00f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 shutdown_connections 2024-04-06T14:41:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.671+0000 7fb9b7dfc700 1 -- 172.21.15.32:0/945419773 wait complete. 2024-04-06T14:41:45.674 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:41:45.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.775+0000 7fb6f1a50700 1 -- 172.21.15.32:0/3025772555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec102710 msgr2=0x7fb6ec0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.775+0000 7fb6f1a50700 1 --2- 172.21.15.32:0/3025772555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec102710 0x7fb6ec0fbd80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4004940 tx=0x7fb6d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.775+0000 7fb6f1a50700 1 -- 172.21.15.32:0/3025772555 shutdown_connections 2024-04-06T14:41:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.775+0000 7fb6f1a50700 1 --2- 172.21.15.32:0/3025772555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec102710 0x7fb6ec0fbd80 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.775+0000 7fb6f1a50700 1 --2- 172.21.15.32:0/3025772555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec101d70 0x7fb6ec102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.775+0000 7fb6f1a50700 1 -- 172.21.15.32:0/3025772555 >> 172.21.15.32:0/3025772555 conn(0x7fb6ec072bc0 msgr2=0x7fb6ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.775+0000 7fb6f1a50700 1 -- 172.21.15.32:0/3025772555 shutdown_connections 2024-04-06T14:41:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.775+0000 7fb6f1a50700 1 -- 172.21.15.32:0/3025772555 wait complete. 2024-04-06T14:41:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6f1a50700 1 Processor -- start 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6f1a50700 1 -- start start 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6f1a50700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec101d70 0x7fb6ec1129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6f1a50700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec102710 0x7fb6ec112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6f1a50700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec113550 con 0x7fb6ec101d70 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6f1a50700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec10c940 con 0x7fb6ec102710 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec101d70 0x7fb6ec1129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec101d70 0x7fb6ec1129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56864/0 (socket says 172.21.15.32:56864) 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6eaffd700 1 -- 172.21.15.32:0/472400758 learned_addr learned my addr 172.21.15.32:0/472400758 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6eaffd700 1 -- 172.21.15.32:0/472400758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec102710 msgr2=0x7fb6ec112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6eaffd700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec102710 0x7fb6ec112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6eaffd700 1 -- 172.21.15.32:0/472400758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d4005040 con 0x7fb6ec101d70 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.776+0000 7fb6eaffd700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec101d70 0x7fb6ec1129c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc002a50 tx=0x7fb6dc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.777+0000 7fb6e37fe700 1 -- 172.21.15.32:0/472400758 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6dc007cc0 con 0x7fb6ec101d70 2024-04-06T14:41:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.777+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec10cc20 con 0x7fb6ec101d70 2024-04-06T14:41:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.777+0000 7fb6e37fe700 1 -- 172.21.15.32:0/472400758 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6dc00f040 con 0x7fb6ec101d70 2024-04-06T14:41:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.777+0000 7fb6e37fe700 1 -- 172.21.15.32:0/472400758 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6dc009540 con 0x7fb6ec101d70 2024-04-06T14:41:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.777+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ec10d1e0 con 0x7fb6ec101d70 2024-04-06T14:41:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.778+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ec0ff080 con 0x7fb6ec101d70 2024-04-06T14:41:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.778+0000 7fb6e37fe700 1 -- 172.21.15.32:0/472400758 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb6dc0096a0 con 0x7fb6ec101d70 2024-04-06T14:41:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.779+0000 7fb6e37fe700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.779+0000 7fb6e37fe700 1 -- 172.21.15.32:0/472400758 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb6dc08be70 con 0x7fb6ec101d70 2024-04-06T14:41:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.779+0000 7fb6ea7fc700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.779+0000 7fb6ea7fc700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4004f90 tx=0x7fb6d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:45.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:45.783+0000 7fb6e37fe700 1 -- 172.21.15.32:0/472400758 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb6dc0598c0 con 0x7fb6ec101d70 2024-04-06T14:41:45.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:45 smithi086 ceph-mon[110975]: pgmap v649: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:45.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:45 smithi032 ceph-mon[147414]: pgmap v649: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.004+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6ec10e080 con 0x7fb6d806cc20 2024-04-06T14:41:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.005+0000 7fb6e37fe700 1 -- 172.21.15.32:0/472400758 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb6ec10e080 con 0x7fb6d806cc20 2024-04-06T14:41:45.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 msgr2=0x7fb6d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:45.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4004f90 tx=0x7fb6d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:45.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec101d70 msgr2=0x7fb6ec1129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:45.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec101d70 0x7fb6ec1129c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc002a50 tx=0x7fb6dc0046b0 comp rx=0 tx=0).stop 2024-04-06T14:41:45.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 shutdown_connections 2024-04-06T14:41:45.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec101d70 0x7fb6ec1129c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 --2- 172.21.15.32:0/472400758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec102710 0x7fb6ec112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 >> 172.21.15.32:0/472400758 conn(0x7fb6ec072bc0 msgr2=0x7fb6ec0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:46.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 shutdown_connections 2024-04-06T14:41:46.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.007+0000 7fb6f1a50700 1 -- 172.21.15.32:0/472400758 wait complete. 2024-04-06T14:41:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.110+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/906121443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 msgr2=0x7fdd080fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.110+0000 7fdd0ffa6700 1 --2- 172.21.15.32:0/906121443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 0x7fdd080fe1e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc004940 tx=0x7fdcfc009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.110+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/906121443 shutdown_connections 2024-04-06T14:41:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.110+0000 7fdd0ffa6700 1 --2- 172.21.15.32:0/906121443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd080fe7b0 0x7fdd08106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.110+0000 7fdd0ffa6700 1 --2- 172.21.15.32:0/906121443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 0x7fdd080fe1e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.110+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/906121443 >> 172.21.15.32:0/906121443 conn(0x7fdd080f78d0 msgr2=0x7fdd080f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.110+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/906121443 shutdown_connections 2024-04-06T14:41:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.110+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/906121443 wait complete. 2024-04-06T14:41:46.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0ffa6700 1 Processor -- start 2024-04-06T14:41:46.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0ffa6700 1 -- start start 2024-04-06T14:41:46.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0ffa6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 0x7fdd08113b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:46.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0ffa6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd080fe7b0 0x7fdd08114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:46.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0ffa6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd081146d0 con 0x7fdd080fde10 2024-04-06T14:41:46.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0ffa6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0810dac0 con 0x7fdd080fe7b0 2024-04-06T14:41:46.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0dd42700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 0x7fdd08113b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:46.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0dd42700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 0x7fdd08113b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56886/0 (socket says 172.21.15.32:56886) 2024-04-06T14:41:46.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0dd42700 1 -- 172.21.15.32:0/756368446 learned_addr learned my addr 172.21.15.32:0/756368446 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0d541700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd080fe7b0 0x7fdd08114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0dd42700 1 -- 172.21.15.32:0/756368446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd080fe7b0 msgr2=0x7fdd08114080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0dd42700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd080fe7b0 0x7fdd08114080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0dd42700 1 -- 172.21.15.32:0/756368446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc005040 con 0x7fdd080fde10 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.111+0000 7fdd0d541700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd080fe7b0 0x7fdd08114080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.112+0000 7fdd0dd42700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 0x7fdd08113b40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009390 tx=0x7fdcfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.112+0000 7fdcfa7fc700 1 -- 172.21.15.32:0/756368446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc00e040 con 0x7fdd080fde10 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.112+0000 7fdcfa7fc700 1 -- 172.21.15.32:0/756368446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcfc013420 con 0x7fdd080fde10 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.112+0000 7fdcfa7fc700 1 -- 172.21.15.32:0/756368446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc01c630 con 0x7fdd080fde10 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.112+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0810dd40 con 0x7fdd080fde10 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.112+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0810e230 con 0x7fdd080fde10 2024-04-06T14:41:46.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.113+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd080666f0 con 0x7fdd080fde10 2024-04-06T14:41:46.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.113+0000 7fdcfa7fc700 1 -- 172.21.15.32:0/756368446 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdcfc013a20 con 0x7fdd080fde10 2024-04-06T14:41:46.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.114+0000 7fdcfa7fc700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdcf406cc30 0x7fdcf406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:46.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.114+0000 7fdcfa7fc700 1 -- 172.21.15.32:0/756368446 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdcfc08d290 con 0x7fdd080fde10 2024-04-06T14:41:46.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.114+0000 7fdd0d541700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdcf406cc30 0x7fdcf406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:46.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.114+0000 7fdd0d541700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdcf406cc30 0x7fdcf406f0e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009fd0 tx=0x7fdd0400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:46.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.118+0000 7fdcfa7fc700 1 -- 172.21.15.32:0/756368446 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdcfc05ad60 con 0x7fdd080fde10 2024-04-06T14:41:46.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.331+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd08109d60 con 0x7fdcf406cc30 2024-04-06T14:41:46.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.341+0000 7fdcfa7fc700 1 -- 172.21.15.32:0/756368446 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fdd08109d60 con 0x7fdcf406cc30 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (22m) 4s ago 33m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (19m) 4s ago 33m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (19m) 7s ago 32m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (21m) 4s ago 33m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (15m) 4s ago 27m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (15m) 4s ago 27m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (15m) 7s ago 27m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (10s) 7s ago 27m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:41:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (24m) 4s ago 34m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (23m) 7s ago 32m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (20m) 4s ago 34m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (20m) 7s ago 32m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (22m) 4s ago 33m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (22m) 7s ago 32m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (19m) 4s ago 31m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (18m) 4s ago 31m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (17m) 4s ago 30m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (17m) 7s ago 30m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (16m) 7s ago 29m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (16m) 7s ago 29m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:41:46.335 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (22m) 4s ago 32m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:41:46.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdcf406cc30 msgr2=0x7fdcf406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdcf406cc30 0x7fdcf406f0e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009fd0 tx=0x7fdd0400b040 comp rx=0 tx=0).stop 2024-04-06T14:41:46.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 msgr2=0x7fdd08113b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 0x7fdd08113b40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009390 tx=0x7fdcfc009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:46.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 shutdown_connections 2024-04-06T14:41:46.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdcf406cc30 0x7fdcf406f0e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080fde10 0x7fdd08113b40 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 --2- 172.21.15.32:0/756368446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd080fe7b0 0x7fdd08114080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 >> 172.21.15.32:0/756368446 conn(0x7fdd080f78d0 msgr2=0x7fdd080f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:46.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 shutdown_connections 2024-04-06T14:41:46.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.343+0000 7fdd0ffa6700 1 -- 172.21.15.32:0/756368446 wait complete. 2024-04-06T14:41:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.445+0000 7f46a9433700 1 -- 172.21.15.32:0/3733936413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a41037a0 msgr2=0x7f46a40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.445+0000 7f46a9433700 1 --2- 172.21.15.32:0/3733936413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a41037a0 0x7f46a40fce10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4698004940 tx=0x7f4698009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.445+0000 7f46a9433700 1 -- 172.21.15.32:0/3733936413 shutdown_connections 2024-04-06T14:41:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.445+0000 7f46a9433700 1 --2- 172.21.15.32:0/3733936413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a41037a0 0x7f46a40fce10 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.445+0000 7f46a9433700 1 --2- 172.21.15.32:0/3733936413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a4102e00 0x7f46a41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.445+0000 7f46a9433700 1 -- 172.21.15.32:0/3733936413 >> 172.21.15.32:0/3733936413 conn(0x7f46a40f89b0 msgr2=0x7f46a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.445+0000 7f46a9433700 1 -- 172.21.15.32:0/3733936413 shutdown_connections 2024-04-06T14:41:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.446+0000 7f46a9433700 1 -- 172.21.15.32:0/3733936413 wait complete. 2024-04-06T14:41:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.446+0000 7f46a9433700 1 Processor -- start 2024-04-06T14:41:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.446+0000 7f46a9433700 1 -- start start 2024-04-06T14:41:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a9433700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4102e00 0x7f46a4113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a9433700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a41037a0 0x7f46a4113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a9433700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a41143e0 con 0x7f46a4102e00 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a9433700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a410d9e0 con 0x7f46a41037a0 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4102e00 0x7f46a4113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4102e00 0x7f46a4113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56908/0 (socket says 172.21.15.32:56908) 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a2ffd700 1 -- 172.21.15.32:0/2338551950 learned_addr learned my addr 172.21.15.32:0/2338551950 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a27fc700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a41037a0 0x7f46a4113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a2ffd700 1 -- 172.21.15.32:0/2338551950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a41037a0 msgr2=0x7f46a4113d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a2ffd700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a41037a0 0x7f46a4113d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a2ffd700 1 -- 172.21.15.32:0/2338551950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4698005040 con 0x7f46a4102e00 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a27fc700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a41037a0 0x7f46a4113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a2ffd700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4102e00 0x7f46a4113850 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4694002a80 tx=0x7f46940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f468b7fe700 1 -- 172.21.15.32:0/2338551950 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4694007cc0 con 0x7f46a4102e00 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f468b7fe700 1 -- 172.21.15.32:0/2338551950 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f469400f040 con 0x7f46a4102e00 2024-04-06T14:41:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a410dcc0 con 0x7f46a4102e00 2024-04-06T14:41:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.447+0000 7f468b7fe700 1 -- 172.21.15.32:0/2338551950 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4694009640 con 0x7f46a4102e00 2024-04-06T14:41:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.448+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a410e280 con 0x7f46a4102e00 2024-04-06T14:41:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.448+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a40fe550 con 0x7f46a4102e00 2024-04-06T14:41:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.449+0000 7f468b7fe700 1 -- 172.21.15.32:0/2338551950 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f46940097a0 con 0x7f46a4102e00 2024-04-06T14:41:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.449+0000 7f468b7fe700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f468c06cc20 0x7f468c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.449+0000 7f468b7fe700 1 -- 172.21.15.32:0/2338551950 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f469408b770 con 0x7f46a4102e00 2024-04-06T14:41:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.449+0000 7f46a27fc700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f468c06cc20 0x7f468c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.450+0000 7f46a27fc700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f468c06cc20 0x7f468c06f0d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f469800b3b0 tx=0x7f4698009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.454+0000 7f468b7fe700 1 -- 172.21.15.32:0/2338551950 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4694059240 con 0x7f46a4102e00 2024-04-06T14:41:46.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.741+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f46a4109da0 con 0x7f46a4102e00 2024-04-06T14:41:46.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.742+0000 7f468b7fe700 1 -- 172.21.15.32:0/2338551950 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f4694058dd0 con 0x7f46a4102e00 2024-04-06T14:41:46.734 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:41:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:41:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:41:46.735 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:41:46.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f468c06cc20 msgr2=0x7f468c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f468c06cc20 0x7f468c06f0d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f469800b3b0 tx=0x7f4698009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:46.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4102e00 msgr2=0x7f46a4113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4102e00 0x7f46a4113850 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4694002a80 tx=0x7f46940046b0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 shutdown_connections 2024-04-06T14:41:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f468c06cc20 0x7f468c06f0d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4102e00 0x7f46a4113850 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 --2- 172.21.15.32:0/2338551950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a41037a0 0x7f46a4113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 >> 172.21.15.32:0/2338551950 conn(0x7f46a40f89b0 msgr2=0x7f46a40f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 shutdown_connections 2024-04-06T14:41:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.744+0000 7f46a9433700 1 -- 172.21.15.32:0/2338551950 wait complete. 2024-04-06T14:41:46.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 -- 172.21.15.32:0/2714205697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148101d70 msgr2=0x7fa148102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 --2- 172.21.15.32:0/2714205697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148101d70 0x7fa148102140 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa144004940 tx=0x7fa144009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:46.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 -- 172.21.15.32:0/2714205697 shutdown_connections 2024-04-06T14:41:46.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 --2- 172.21.15.32:0/2714205697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa148102710 0x7fa1480fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 --2- 172.21.15.32:0/2714205697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148101d70 0x7fa148102140 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 -- 172.21.15.32:0/2714205697 >> 172.21.15.32:0/2714205697 conn(0x7fa148072bc0 msgr2=0x7fa148072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 -- 172.21.15.32:0/2714205697 shutdown_connections 2024-04-06T14:41:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 -- 172.21.15.32:0/2714205697 wait complete. 2024-04-06T14:41:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 Processor -- start 2024-04-06T14:41:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.847+0000 7fa14f921700 1 -- start start 2024-04-06T14:41:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14f921700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa148101d70 0x7fa148102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14f921700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148102710 0x7fa148103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14f921700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148103650 con 0x7fa148102710 2024-04-06T14:41:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14f921700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148103790 con 0x7fa148101d70 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14cebc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148102710 0x7fa148103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14cebc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148102710 0x7fa148103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56926/0 (socket says 172.21.15.32:56926) 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14cebc700 1 -- 172.21.15.32:0/301586499 learned_addr learned my addr 172.21.15.32:0/301586499 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14d6bd700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa148101d70 0x7fa148102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14cebc700 1 -- 172.21.15.32:0/301586499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa148101d70 msgr2=0x7fa148102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14cebc700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa148101d70 0x7fa148102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14cebc700 1 -- 172.21.15.32:0/301586499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa144005040 con 0x7fa148102710 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14d6bd700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa148101d70 0x7fa148102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14cebc700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148102710 0x7fa148103110 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa138002a50 tx=0x7fa138004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa13dffb700 1 -- 172.21.15.32:0/301586499 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa138007cc0 con 0x7fa148102710 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148103a40 con 0x7fa148102710 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa13dffb700 1 -- 172.21.15.32:0/301586499 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa13800f040 con 0x7fa148102710 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.848+0000 7fa13dffb700 1 -- 172.21.15.32:0/301586499 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa13800d990 con 0x7fa148102710 2024-04-06T14:41:46.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.849+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14811a210 con 0x7fa148102710 2024-04-06T14:41:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.850+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1480fd4c0 con 0x7fa148102710 2024-04-06T14:41:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.850+0000 7fa13dffb700 1 -- 172.21.15.32:0/301586499 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa138009710 con 0x7fa148102710 2024-04-06T14:41:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.850+0000 7fa13dffb700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa13406cc30 0x7fa13406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.851+0000 7fa13dffb700 1 -- 172.21.15.32:0/301586499 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa13808bcc0 con 0x7fa148102710 2024-04-06T14:41:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.851+0000 7fa14d6bd700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa13406cc30 0x7fa13406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:46.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.851+0000 7fa14d6bd700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa13406cc30 0x7fa13406f0e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa14400b3b0 tx=0x7fa144009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:46.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:46.855+0000 7fa13dffb700 1 -- 172.21.15.32:0/301586499 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa138059710 con 0x7fa148102710 2024-04-06T14:41:47.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.099+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa1481039e0 con 0x7fa148102710 2024-04-06T14:41:47.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.100+0000 7fa13dffb700 1 -- 172.21.15.32:0/301586499 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fa1380592a0 con 0x7fa148102710 2024-04-06T14:41:47.092 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:41:47.092 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:41:47.092 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:41:47.092 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:41:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:41:47.092 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:41:47.093 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:41:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:41:47.095 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:41:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:41:47.095 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:41:47.095 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:41:47.095 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:41:47.095 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa13406cc30 msgr2=0x7fa13406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa13406cc30 0x7fa13406f0e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa14400b3b0 tx=0x7fa144009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148102710 msgr2=0x7fa148103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148102710 0x7fa148103110 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa138002a50 tx=0x7fa138004600 comp rx=0 tx=0).stop 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 shutdown_connections 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa13406cc30 0x7fa13406f0e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa148101d70 0x7fa148102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 --2- 172.21.15.32:0/301586499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa148102710 0x7fa148103110 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 >> 172.21.15.32:0/301586499 conn(0x7fa148072bc0 msgr2=0x7fa148065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 shutdown_connections 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.102+0000 7fa14f921700 1 -- 172.21.15.32:0/301586499 wait complete. 2024-04-06T14:41:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:41:47.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.205+0000 7f3cac219700 1 -- 172.21.15.32:0/3458578989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 msgr2=0x7f3ca40fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.205+0000 7f3cac219700 1 --2- 172.21.15.32:0/3458578989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 0x7f3ca40fe220 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0004940 tx=0x7f3ca0009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:47.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.206+0000 7f3cac219700 1 -- 172.21.15.32:0/3458578989 shutdown_connections 2024-04-06T14:41:47.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.206+0000 7f3cac219700 1 --2- 172.21.15.32:0/3458578989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca40fe7f0 0x7f3ca4106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.206+0000 7f3cac219700 1 --2- 172.21.15.32:0/3458578989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 0x7f3ca40fe220 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.206+0000 7f3cac219700 1 -- 172.21.15.32:0/3458578989 >> 172.21.15.32:0/3458578989 conn(0x7f3ca40f78d0 msgr2=0x7f3ca40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:47.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.206+0000 7f3cac219700 1 -- 172.21.15.32:0/3458578989 shutdown_connections 2024-04-06T14:41:47.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.206+0000 7f3cac219700 1 -- 172.21.15.32:0/3458578989 wait complete. 2024-04-06T14:41:47.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.206+0000 7f3cac219700 1 Processor -- start 2024-04-06T14:41:47.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.206+0000 7f3cac219700 1 -- start start 2024-04-06T14:41:47.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3cac219700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 0x7f3ca4113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:47.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3cac219700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca40fe7f0 0x7f3ca4114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3cac219700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca41146b0 con 0x7f3ca40fde50 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3cac219700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca410daa0 con 0x7f3ca40fe7f0 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3ca9fb5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 0x7f3ca4113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3ca9fb5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 0x7f3ca4113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56942/0 (socket says 172.21.15.32:56942) 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3ca9fb5700 1 -- 172.21.15.32:0/1593095225 learned_addr learned my addr 172.21.15.32:0/1593095225 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3ca97b4700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca40fe7f0 0x7f3ca4114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3ca9fb5700 1 -- 172.21.15.32:0/1593095225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca40fe7f0 msgr2=0x7f3ca4114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3ca9fb5700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca40fe7f0 0x7f3ca4114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3ca9fb5700 1 -- 172.21.15.32:0/1593095225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca0005040 con 0x7f3ca40fde50 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3ca97b4700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca40fe7f0 0x7f3ca4114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3ca9fb5700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 0x7f3ca4113b20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009c90 tx=0x7f3ca0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.207+0000 7f3c9a7fc700 1 -- 172.21.15.32:0/1593095225 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ca000e040 con 0x7f3ca40fde50 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.208+0000 7f3c9a7fc700 1 -- 172.21.15.32:0/1593095225 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ca000bed0 con 0x7f3ca40fde50 2024-04-06T14:41:47.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.208+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca410dd20 con 0x7f3ca40fde50 2024-04-06T14:41:47.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.208+0000 7f3c9a7fc700 1 -- 172.21.15.32:0/1593095225 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ca0013ea0 con 0x7f3ca40fde50 2024-04-06T14:41:47.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.208+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca410e260 con 0x7f3ca40fde50 2024-04-06T14:41:47.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.208+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca40666b0 con 0x7f3ca40fde50 2024-04-06T14:41:47.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.209+0000 7f3c9a7fc700 1 -- 172.21.15.32:0/1593095225 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3ca0013690 con 0x7f3ca40fde50 2024-04-06T14:41:47.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.209+0000 7f3c9a7fc700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c9006cc20 0x7f3c9006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:47.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.209+0000 7f3c9a7fc700 1 -- 172.21.15.32:0/1593095225 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3ca008d270 con 0x7f3ca40fde50 2024-04-06T14:41:47.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.209+0000 7f3ca97b4700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c9006cc20 0x7f3c9006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:47.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.210+0000 7f3ca97b4700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c9006cc20 0x7f3c9006f0d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009fd0 tx=0x7f3c94009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:47.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.214+0000 7f3c9a7fc700 1 -- 172.21.15.32:0/1593095225 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ca005aeb0 con 0x7f3ca40fde50 2024-04-06T14:41:47.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.436+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ca4109d80 con 0x7f3c9006cc20 2024-04-06T14:41:47.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.437+0000 7f3c9a7fc700 1 -- 172.21.15.32:0/1593095225 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3ca4109d80 con 0x7f3c9006cc20 2024-04-06T14:41:47.429 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:41:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:41:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:41:47.430 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:41:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c9006cc20 msgr2=0x7f3c9006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c9006cc20 0x7f3c9006f0d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009fd0 tx=0x7f3c94009ee0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 msgr2=0x7f3ca4113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 0x7f3ca4113b20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009c90 tx=0x7f3ca0009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 shutdown_connections 2024-04-06T14:41:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c9006cc20 0x7f3c9006f0d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ca40fde50 0x7f3ca4113b20 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 --2- 172.21.15.32:0/1593095225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca40fe7f0 0x7f3ca4114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 >> 172.21.15.32:0/1593095225 conn(0x7f3ca40f78d0 msgr2=0x7f3ca40f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 shutdown_connections 2024-04-06T14:41:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.439+0000 7f3cac219700 1 -- 172.21.15.32:0/1593095225 wait complete. 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.543+0000 7fa003a61700 1 -- 172.21.15.32:0/529638559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 msgr2=0x7f9ffc1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.543+0000 7fa003a61700 1 --2- 172.21.15.32:0/529638559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 0x7f9ffc1031f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9fec004940 tx=0x7f9fec009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.543+0000 7fa003a61700 1 -- 172.21.15.32:0/529638559 shutdown_connections 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.543+0000 7fa003a61700 1 --2- 172.21.15.32:0/529638559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ffc1037c0 0x7f9ffc0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.543+0000 7fa003a61700 1 --2- 172.21.15.32:0/529638559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 0x7f9ffc1031f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.543+0000 7fa003a61700 1 -- 172.21.15.32:0/529638559 >> 172.21.15.32:0/529638559 conn(0x7f9ffc0f89b0 msgr2=0x7f9ffc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.543+0000 7fa003a61700 1 -- 172.21.15.32:0/529638559 shutdown_connections 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.543+0000 7fa003a61700 1 -- 172.21.15.32:0/529638559 wait complete. 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa003a61700 1 Processor -- start 2024-04-06T14:41:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa003a61700 1 -- start start 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa003a61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 0x7f9ffc1138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa003a61700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ffc1037c0 0x7f9ffc113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa003a61700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc114460 con 0x7f9ffc102e20 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa003a61700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc10da60 con 0x7f9ffc1037c0 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa0017fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 0x7f9ffc1138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa0017fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 0x7f9ffc1138d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56950/0 (socket says 172.21.15.32:56950) 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa0017fd700 1 -- 172.21.15.32:0/519866078 learned_addr learned my addr 172.21.15.32:0/519866078 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa0017fd700 1 -- 172.21.15.32:0/519866078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ffc1037c0 msgr2=0x7f9ffc113e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa000ffc700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ffc1037c0 0x7f9ffc113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa0017fd700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ffc1037c0 0x7f9ffc113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa0017fd700 1 -- 172.21.15.32:0/519866078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec005040 con 0x7f9ffc102e20 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.544+0000 7fa0017fd700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 0x7f9ffc1138d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9fec000c00 tx=0x7f9fec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.545+0000 7f9ff1ffb700 1 -- 172.21.15.32:0/519866078 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fec00e040 con 0x7f9ffc102e20 2024-04-06T14:41:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.545+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ffc10dce0 con 0x7f9ffc102e20 2024-04-06T14:41:47.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.545+0000 7f9ff1ffb700 1 -- 172.21.15.32:0/519866078 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fec013420 con 0x7f9ffc102e20 2024-04-06T14:41:47.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.545+0000 7f9ff1ffb700 1 -- 172.21.15.32:0/519866078 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fec01c630 con 0x7f9ffc102e20 2024-04-06T14:41:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.545+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ffc10e2a0 con 0x7f9ffc102e20 2024-04-06T14:41:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.546+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ffc0fe570 con 0x7f9ffc102e20 2024-04-06T14:41:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.546+0000 7f9ff1ffb700 1 -- 172.21.15.32:0/519866078 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9fec01c8b0 con 0x7f9ffc102e20 2024-04-06T14:41:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.547+0000 7f9ff1ffb700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9fe806cc20 0x7f9fe806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:41:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.547+0000 7f9ff1ffb700 1 -- 172.21.15.32:0/519866078 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9fec08d280 con 0x7f9ffc102e20 2024-04-06T14:41:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.547+0000 7fa000ffc700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9fe806cc20 0x7f9fe806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:41:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.547+0000 7fa000ffc700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9fe806cc20 0x7f9fe806f0d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc10ec20 tx=0x7f9ff800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:41:47.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.551+0000 7f9ff1ffb700 1 -- 172.21.15.32:0/519866078 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9fec05adc0 con 0x7f9ffc102e20 2024-04-06T14:41:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.839+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9ffc109de0 con 0x7f9ffc102e20 2024-04-06T14:41:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.840+0000 7f9ff1ffb700 1 -- 172.21.15.32:0/519866078 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f9fec05a950 con 0x7f9ffc102e20 2024-04-06T14:41:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:41:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:41:47.834 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:41:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9fe806cc20 msgr2=0x7f9fe806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9fe806cc20 0x7f9fe806f0d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc10ec20 tx=0x7f9ff800b040 comp rx=0 tx=0).stop 2024-04-06T14:41:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 msgr2=0x7f9ffc1138d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:41:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 0x7f9ffc1138d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9fec000c00 tx=0x7f9fec009e30 comp rx=0 tx=0).stop 2024-04-06T14:41:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 shutdown_connections 2024-04-06T14:41:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9fe806cc20 0x7f9fe806f0d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc102e20 0x7f9ffc1138d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 --2- 172.21.15.32:0/519866078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ffc1037c0 0x7f9ffc113e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:41:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 >> 172.21.15.32:0/519866078 conn(0x7f9ffc0f89b0 msgr2=0x7f9ffc0f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:41:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 shutdown_connections 2024-04-06T14:41:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:41:47.842+0000 7fa003a61700 1 -- 172.21.15.32:0/519866078 wait complete. 2024-04-06T14:41:47.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:47 smithi086 ceph-mon[110975]: from='client.35206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:47.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:47 smithi086 ceph-mon[110975]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:47.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:47 smithi086 ceph-mon[110975]: pgmap v650: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:47.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:47 smithi086 ceph-mon[110975]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:47.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:47 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2338551950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:47.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:47 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/301586499' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:41:47.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:47 smithi032 ceph-mon[147414]: from='client.35206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:47.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:47 smithi032 ceph-mon[147414]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:47.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:47 smithi032 ceph-mon[147414]: pgmap v650: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:47.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:47 smithi032 ceph-mon[147414]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:47.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:47 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2338551950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:41:47.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:47 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/301586499' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:41:48.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:48 smithi086 ceph-mon[110975]: from='client.35226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:48.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:48 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/519866078' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:41:48.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:48 smithi032 ceph-mon[147414]: from='client.35226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:41:48.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:48 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/519866078' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:41:49.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:49 smithi086 ceph-mon[110975]: pgmap v651: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:49.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:49 smithi032 ceph-mon[147414]: pgmap v651: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:51.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:51 smithi086 ceph-mon[110975]: pgmap v652: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:51.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:51 smithi032 ceph-mon[147414]: pgmap v652: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:53.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:53 smithi032 ceph-mon[147414]: pgmap v653: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:53.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:41:53.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:53 smithi086 ceph-mon[110975]: pgmap v653: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:53.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:41:55.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:55 smithi086 ceph-mon[110975]: pgmap v654: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:55.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:55 smithi032 ceph-mon[147414]: pgmap v654: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:57.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:57 smithi086 ceph-mon[110975]: pgmap v655: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:57.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:57 smithi032 ceph-mon[147414]: pgmap v655: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:59.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:41:59 smithi086 ceph-mon[110975]: pgmap v656: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:41:59.974 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:41:59 smithi032 ceph-mon[147414]: pgmap v656: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:01.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:01 smithi086 ceph-mon[110975]: pgmap v657: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:01.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:01 smithi032 ceph-mon[147414]: pgmap v657: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:03.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:03 smithi032 ceph-mon[147414]: pgmap v658: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:03.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:03 smithi086 ceph-mon[110975]: pgmap v658: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:05.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:05 smithi086 ceph-mon[110975]: pgmap v659: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:05.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:05 smithi032 ceph-mon[147414]: pgmap v659: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:07.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:07 smithi086 ceph-mon[110975]: pgmap v660: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:07.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:07 smithi032 ceph-mon[147414]: pgmap v660: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:08.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:42:08.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:42:09.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:09 smithi086 ceph-mon[110975]: pgmap v661: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:09.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:09 smithi032 ceph-mon[147414]: pgmap v661: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:11.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:11 smithi086 ceph-mon[110975]: pgmap v662: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:11.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:11 smithi032 ceph-mon[147414]: pgmap v662: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:13.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:13 smithi032 ceph-mon[147414]: pgmap v663: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:13.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:13 smithi086 ceph-mon[110975]: pgmap v663: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:15.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:15 smithi086 ceph-mon[110975]: pgmap v664: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:15.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:15 smithi032 ceph-mon[147414]: pgmap v664: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:17.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:17 smithi086 ceph-mon[110975]: pgmap v665: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.961+0000 7f550df91700 1 -- 172.21.15.32:0/2442427582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fe830 msgr2=0x7f5508106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.961+0000 7f550df91700 1 --2- 172.21.15.32:0/2442427582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fe830 0x7f5508106f00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f54f0004940 tx=0x7f54f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.961+0000 7f550df91700 1 -- 172.21.15.32:0/2442427582 shutdown_connections 2024-04-06T14:42:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.961+0000 7f550df91700 1 --2- 172.21.15.32:0/2442427582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fe830 0x7f5508106f00 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.961+0000 7f550df91700 1 --2- 172.21.15.32:0/2442427582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55080fde90 0x7f55080fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.961+0000 7f550df91700 1 -- 172.21.15.32:0/2442427582 >> 172.21.15.32:0/2442427582 conn(0x7f5508072bc0 msgr2=0x7f5508072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.961+0000 7f550df91700 1 -- 172.21.15.32:0/2442427582 shutdown_connections 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.961+0000 7f550df91700 1 -- 172.21.15.32:0/2442427582 wait complete. 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.961+0000 7f550df91700 1 Processor -- start 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f550df91700 1 -- start start 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f550df91700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fde90 0x7f5508113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f550df91700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55080fe830 0x7f5508114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f550df91700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5508114670 con 0x7f55080fde90 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f550df91700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550810da60 con 0x7f55080fe830 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f55077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fde90 0x7f5508113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f55077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fde90 0x7f5508113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55290/0 (socket says 172.21.15.32:55290) 2024-04-06T14:42:17.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f55077fe700 1 -- 172.21.15.32:0/1649976063 learned_addr learned my addr 172.21.15.32:0/1649976063 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f5506ffd700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55080fe830 0x7f5508114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f55077fe700 1 -- 172.21.15.32:0/1649976063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55080fe830 msgr2=0x7f5508114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f55077fe700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55080fe830 0x7f5508114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.962+0000 7f55077fe700 1 -- 172.21.15.32:0/1649976063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f0005040 con 0x7f55080fde90 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.963+0000 7f55077fe700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fde90 0x7f5508113ae0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f54f8002a50 tx=0x7f54f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.963+0000 7f54fffff700 1 -- 172.21.15.32:0/1649976063 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54f8007cc0 con 0x7f55080fde90 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.963+0000 7f54fffff700 1 -- 172.21.15.32:0/1649976063 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54f800f040 con 0x7f55080fde90 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.963+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550810dd40 con 0x7f55080fde90 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.963+0000 7f54fffff700 1 -- 172.21.15.32:0/1649976063 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54f800d9c0 con 0x7f55080fde90 2024-04-06T14:42:17.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.963+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550810e280 con 0x7f55080fde90 2024-04-06T14:42:17.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.964+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5508065e30 con 0x7f55080fde90 2024-04-06T14:42:17.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.964+0000 7f54fffff700 1 -- 172.21.15.32:0/1649976063 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f54f800db20 con 0x7f55080fde90 2024-04-06T14:42:17.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.965+0000 7f54fffff700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54f406cc20 0x7f54f406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:17.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.965+0000 7f54fffff700 1 -- 172.21.15.32:0/1649976063 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f54f808bd70 con 0x7f55080fde90 2024-04-06T14:42:17.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.965+0000 7f5506ffd700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54f406cc20 0x7f54f406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:17.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.965+0000 7f5506ffd700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54f406cc20 0x7f54f406f0d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f550810ec20 tx=0x7f54f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:17.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:17.970+0000 7f54fffff700 1 -- 172.21.15.32:0/1649976063 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f54f80597c0 con 0x7f55080fde90 2024-04-06T14:42:17.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:17 smithi032 ceph-mon[147414]: pgmap v665: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:18.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.190+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5508109dd0 con 0x7f54f406cc20 2024-04-06T14:42:18.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.191+0000 7f54fffff700 1 -- 172.21.15.32:0/1649976063 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5508109dd0 con 0x7f54f406cc20 2024-04-06T14:42:18.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54f406cc20 msgr2=0x7f54f406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54f406cc20 0x7f54f406f0d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f550810ec20 tx=0x7f54f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:18.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fde90 msgr2=0x7f5508113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fde90 0x7f5508113ae0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f54f8002a50 tx=0x7f54f80046b0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 shutdown_connections 2024-04-06T14:42:18.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54f406cc20 0x7f54f406f0d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55080fde90 0x7f5508113ae0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 --2- 172.21.15.32:0/1649976063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55080fe830 0x7f5508114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 >> 172.21.15.32:0/1649976063 conn(0x7f5508072bc0 msgr2=0x7f5508103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:18.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 shutdown_connections 2024-04-06T14:42:18.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.193+0000 7f550df91700 1 -- 172.21.15.32:0/1649976063 wait complete. 2024-04-06T14:42:18.194 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:42:18.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.297+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3217519640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8101d70 msgr2=0x7fc5c8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.297+0000 7fc5ced6d700 1 --2- 172.21.15.32:0/3217519640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8101d70 0x7fc5c8102140 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc004940 tx=0x7fc5bc009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.297+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3217519640 shutdown_connections 2024-04-06T14:42:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.297+0000 7fc5ced6d700 1 --2- 172.21.15.32:0/3217519640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c8102710 0x7fc5c80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.297+0000 7fc5ced6d700 1 --2- 172.21.15.32:0/3217519640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8101d70 0x7fc5c8102140 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.297+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3217519640 >> 172.21.15.32:0/3217519640 conn(0x7fc5c8072bc0 msgr2=0x7fc5c8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.297+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3217519640 shutdown_connections 2024-04-06T14:42:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.297+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3217519640 wait complete. 2024-04-06T14:42:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5ced6d700 1 Processor -- start 2024-04-06T14:42:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5ced6d700 1 -- start start 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5ced6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c8101d70 0x7fc5c8102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5ced6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8102710 0x7fc5c8103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5ced6d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8103650 con 0x7fc5c8102710 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5ced6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8103790 con 0x7fc5c8101d70 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8102710 0x7fc5c8103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8102710 0x7fc5c8103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55312/0 (socket says 172.21.15.32:55312) 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5c7fff700 1 -- 172.21.15.32:0/3688121458 learned_addr learned my addr 172.21.15.32:0/3688121458 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.298+0000 7fc5ccb09700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c8101d70 0x7fc5c8102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.299+0000 7fc5c7fff700 1 -- 172.21.15.32:0/3688121458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c8101d70 msgr2=0x7fc5c8102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.299+0000 7fc5c7fff700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c8101d70 0x7fc5c8102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.299+0000 7fc5c7fff700 1 -- 172.21.15.32:0/3688121458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc005040 con 0x7fc5c8102710 2024-04-06T14:42:18.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.299+0000 7fc5c7fff700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8102710 0x7fc5c8103110 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4002a50 tx=0x7fc5b4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:18.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.299+0000 7fc5c57fa700 1 -- 172.21.15.32:0/3688121458 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5b4007cc0 con 0x7fc5c8102710 2024-04-06T14:42:18.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.299+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c8103a40 con 0x7fc5c8102710 2024-04-06T14:42:18.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.299+0000 7fc5c57fa700 1 -- 172.21.15.32:0/3688121458 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5b400f040 con 0x7fc5c8102710 2024-04-06T14:42:18.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.299+0000 7fc5c57fa700 1 -- 172.21.15.32:0/3688121458 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5b400d990 con 0x7fc5c8102710 2024-04-06T14:42:18.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.299+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c811a210 con 0x7fc5c8102710 2024-04-06T14:42:18.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.300+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c80fd4c0 con 0x7fc5c8102710 2024-04-06T14:42:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.300+0000 7fc5c57fa700 1 -- 172.21.15.32:0/3688121458 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc5b4009710 con 0x7fc5c8102710 2024-04-06T14:42:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.301+0000 7fc5c57fa700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b806cc20 0x7fc5b806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.301+0000 7fc5c57fa700 1 -- 172.21.15.32:0/3688121458 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc5b408bcc0 con 0x7fc5c8102710 2024-04-06T14:42:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.301+0000 7fc5ccb09700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b806cc20 0x7fc5b806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.301+0000 7fc5ccb09700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b806cc20 0x7fc5b806f0d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc004f90 tx=0x7fc5bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.305+0000 7fc5c57fa700 1 -- 172.21.15.32:0/3688121458 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc5b4059710 con 0x7fc5c8102710 2024-04-06T14:42:18.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.527+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5c8103a40 con 0x7fc5b806cc20 2024-04-06T14:42:18.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.528+0000 7fc5c57fa700 1 -- 172.21.15.32:0/3688121458 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc5c8103a40 con 0x7fc5b806cc20 2024-04-06T14:42:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b806cc20 msgr2=0x7fc5b806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b806cc20 0x7fc5b806f0d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc004f90 tx=0x7fc5bc009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8102710 msgr2=0x7fc5c8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8102710 0x7fc5c8103110 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4002a50 tx=0x7fc5b4004600 comp rx=0 tx=0).stop 2024-04-06T14:42:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 shutdown_connections 2024-04-06T14:42:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b806cc20 0x7fc5b806f0d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c8101d70 0x7fc5c8102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 --2- 172.21.15.32:0/3688121458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8102710 0x7fc5c8103110 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 >> 172.21.15.32:0/3688121458 conn(0x7fc5c8072bc0 msgr2=0x7fc5c8065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 shutdown_connections 2024-04-06T14:42:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.530+0000 7fc5ced6d700 1 -- 172.21.15.32:0/3688121458 wait complete. 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.635+0000 7fe155aae700 1 -- 172.21.15.32:0/757474029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 msgr2=0x7fe1500fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.635+0000 7fe155aae700 1 --2- 172.21.15.32:0/757474029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 0x7fe1500fbd80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe138004940 tx=0x7fe138009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.635+0000 7fe155aae700 1 -- 172.21.15.32:0/757474029 shutdown_connections 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.635+0000 7fe155aae700 1 --2- 172.21.15.32:0/757474029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 0x7fe1500fbd80 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.635+0000 7fe155aae700 1 --2- 172.21.15.32:0/757474029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe150101d70 0x7fe150102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.635+0000 7fe155aae700 1 -- 172.21.15.32:0/757474029 >> 172.21.15.32:0/757474029 conn(0x7fe150072bc0 msgr2=0x7fe150072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.635+0000 7fe155aae700 1 -- 172.21.15.32:0/757474029 shutdown_connections 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.635+0000 7fe155aae700 1 -- 172.21.15.32:0/757474029 wait complete. 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe155aae700 1 Processor -- start 2024-04-06T14:42:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe155aae700 1 -- start start 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe155aae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe150101d70 0x7fe1501129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe155aae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 0x7fe150112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe155aae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe150113550 con 0x7fe150102710 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe155aae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15010c940 con 0x7fe150101d70 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe14effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 0x7fe150112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe14effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 0x7fe150112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55324/0 (socket says 172.21.15.32:55324) 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe14effd700 1 -- 172.21.15.32:0/2535881249 learned_addr learned my addr 172.21.15.32:0/2535881249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe14f7fe700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe150101d70 0x7fe1501129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe14effd700 1 -- 172.21.15.32:0/2535881249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe150101d70 msgr2=0x7fe1501129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe14effd700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe150101d70 0x7fe1501129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.636+0000 7fe14effd700 1 -- 172.21.15.32:0/2535881249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe138005040 con 0x7fe150102710 2024-04-06T14:42:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.637+0000 7fe14effd700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 0x7fe150112f00 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe138009390 tx=0x7fe138009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:18.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.637+0000 7fe147fff700 1 -- 172.21.15.32:0/2535881249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe13800e040 con 0x7fe150102710 2024-04-06T14:42:18.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.637+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15010cbc0 con 0x7fe150102710 2024-04-06T14:42:18.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.637+0000 7fe147fff700 1 -- 172.21.15.32:0/2535881249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe138004290 con 0x7fe150102710 2024-04-06T14:42:18.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.637+0000 7fe147fff700 1 -- 172.21.15.32:0/2535881249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe138013e70 con 0x7fe150102710 2024-04-06T14:42:18.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.637+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15010d180 con 0x7fe150102710 2024-04-06T14:42:18.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.638+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1500ff080 con 0x7fe150102710 2024-04-06T14:42:18.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.638+0000 7fe147fff700 1 -- 172.21.15.32:0/2535881249 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe138013690 con 0x7fe150102710 2024-04-06T14:42:18.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.639+0000 7fe147fff700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe13c06cbc0 0x7fe13c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:18.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.639+0000 7fe147fff700 1 -- 172.21.15.32:0/2535881249 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe13808d2c0 con 0x7fe150102710 2024-04-06T14:42:18.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.639+0000 7fe14f7fe700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe13c06cbc0 0x7fe13c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:18.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.639+0000 7fe14f7fe700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe13c06cbc0 0x7fe13c06f070 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe140004f40 tx=0x7fe14000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.643+0000 7fe147fff700 1 -- 172.21.15.32:0/2535881249 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe13805adc0 con 0x7fe150102710 2024-04-06T14:42:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.855+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe15010e010 con 0x7fe13c06cbc0 2024-04-06T14:42:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.864+0000 7fe147fff700 1 -- 172.21.15.32:0/2535881249 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe15010e010 con 0x7fe13c06cbc0 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (22m) 37s ago 33m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (20m) 37s ago 34m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (19m) 39s ago 33m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (22m) 37s ago 33m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (16m) 37s ago 28m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (16m) 37s ago 28m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (15m) 39s ago 28m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (43s) 39s ago 28m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (24m) 37s ago 35m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (23m) 39s ago 33m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (21m) 37s ago 35m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (20m) 39s ago 32m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (23m) 37s ago 34m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:42:18.856 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (23m) 39s ago 33m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:42:18.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (19m) 37s ago 32m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:42:18.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (18m) 37s ago 31m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:42:18.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (18m) 37s ago 31m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:42:18.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (17m) 39s ago 30m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:42:18.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (17m) 39s ago 30m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:42:18.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (16m) 39s ago 29m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:42:18.857 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (22m) 37s ago 33m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.866+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe13c06cbc0 msgr2=0x7fe13c06f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.866+0000 7fe155aae700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe13c06cbc0 0x7fe13c06f070 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe140004f40 tx=0x7fe14000b040 comp rx=0 tx=0).stop 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.866+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 msgr2=0x7fe150112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.866+0000 7fe155aae700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 0x7fe150112f00 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe138009390 tx=0x7fe138009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.866+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 shutdown_connections 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.866+0000 7fe155aae700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe13c06cbc0 0x7fe13c06f070 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.866+0000 7fe155aae700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe150101d70 0x7fe1501129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.866+0000 7fe155aae700 1 --2- 172.21.15.32:0/2535881249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe150102710 0x7fe150112f00 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.866+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 >> 172.21.15.32:0/2535881249 conn(0x7fe150072bc0 msgr2=0x7fe1500fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.867+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 shutdown_connections 2024-04-06T14:42:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.867+0000 7fe155aae700 1 -- 172.21.15.32:0/2535881249 wait complete. 2024-04-06T14:42:18.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.969+0000 7f0518d82700 1 -- 172.21.15.32:0/4288035553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fde90 msgr2=0x7f05140fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.969+0000 7f0518d82700 1 --2- 172.21.15.32:0/4288035553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fde90 0x7f05140fe260 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0504004940 tx=0x7f0504009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:18.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.969+0000 7f0518d82700 1 -- 172.21.15.32:0/4288035553 shutdown_connections 2024-04-06T14:42:18.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.969+0000 7f0518d82700 1 --2- 172.21.15.32:0/4288035553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05140fe830 0x7f0514106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.969+0000 7f0518d82700 1 --2- 172.21.15.32:0/4288035553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fde90 0x7f05140fe260 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.969+0000 7f0518d82700 1 -- 172.21.15.32:0/4288035553 >> 172.21.15.32:0/4288035553 conn(0x7f0514072bc0 msgr2=0x7f0514072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:18.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.969+0000 7f0518d82700 1 -- 172.21.15.32:0/4288035553 shutdown_connections 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.969+0000 7f0518d82700 1 -- 172.21.15.32:0/4288035553 wait complete. 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f0518d82700 1 Processor -- start 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f0518d82700 1 -- start start 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f0518d82700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05140fde90 0x7f05140fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f0518d82700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fe830 0x7f05140fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f0518d82700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05140fc820 con 0x7f05140fe830 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f0518d82700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05140fc960 con 0x7f05140fde90 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f0511d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fe830 0x7f05140fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f0511d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fe830 0x7f05140fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55340/0 (socket says 172.21.15.32:55340) 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f0511d9b700 1 -- 172.21.15.32:0/3405696874 learned_addr learned my addr 172.21.15.32:0/3405696874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.970+0000 7f051259c700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05140fde90 0x7f05140fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.971+0000 7f0511d9b700 1 -- 172.21.15.32:0/3405696874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05140fde90 msgr2=0x7f05140fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.971+0000 7f0511d9b700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05140fde90 0x7f05140fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.971+0000 7f0511d9b700 1 -- 172.21.15.32:0/3405696874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0504005040 con 0x7f05140fe830 2024-04-06T14:42:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.971+0000 7f0511d9b700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fe830 0x7f05140fc290 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f050800eb70 tx=0x7f050800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:18.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.971+0000 7f0502ffd700 1 -- 172.21.15.32:0/3405696874 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0508007cc0 con 0x7f05140fe830 2024-04-06T14:42:18.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.971+0000 7f0502ffd700 1 -- 172.21.15.32:0/3405696874 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f050800f040 con 0x7f05140fe830 2024-04-06T14:42:18.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.971+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05140fcbb0 con 0x7f05140fe830 2024-04-06T14:42:18.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.971+0000 7f0502ffd700 1 -- 172.21.15.32:0/3405696874 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f050800c9f0 con 0x7f05140fe830 2024-04-06T14:42:18.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.971+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051411a130 con 0x7f05140fe830 2024-04-06T14:42:18.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.972+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0514065e30 con 0x7f05140fe830 2024-04-06T14:42:18.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.973+0000 7f0502ffd700 1 -- 172.21.15.32:0/3405696874 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f050800cb50 con 0x7f05140fe830 2024-04-06T14:42:18.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.973+0000 7f0502ffd700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f04fc06cc20 0x7f04fc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:18.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.974+0000 7f0502ffd700 1 -- 172.21.15.32:0/3405696874 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f050808be80 con 0x7f05140fe830 2024-04-06T14:42:18.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.974+0000 7f051259c700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f04fc06cc20 0x7f04fc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:18.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.974+0000 7f051259c700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f04fc06cc20 0x7f04fc06f0d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f050400b0d0 tx=0x7f0504009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:18.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:18.977+0000 7f0502ffd700 1 -- 172.21.15.32:0/3405696874 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0508055ec0 con 0x7f05140fe830 2024-04-06T14:42:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.263+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f05140fd440 con 0x7f05140fe830 2024-04-06T14:42:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.263+0000 7f0502ffd700 1 -- 172.21.15.32:0/3405696874 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f0508019070 con 0x7f05140fe830 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:42:19.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:42:19.256 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:42:19.256 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:42:19.256 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:42:19.256 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:42:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f04fc06cc20 msgr2=0x7f04fc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f04fc06cc20 0x7f04fc06f0d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f050400b0d0 tx=0x7f0504009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fe830 msgr2=0x7f05140fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fe830 0x7f05140fc290 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f050800eb70 tx=0x7f050800b040 comp rx=0 tx=0).stop 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 shutdown_connections 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f04fc06cc20 0x7f04fc06f0d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05140fde90 0x7f05140fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 --2- 172.21.15.32:0/3405696874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05140fe830 0x7f05140fc290 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 >> 172.21.15.32:0/3405696874 conn(0x7f0514072bc0 msgr2=0x7f05140ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 shutdown_connections 2024-04-06T14:42:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.265+0000 7f0518d82700 1 -- 172.21.15.32:0/3405696874 wait complete. 2024-04-06T14:42:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.368+0000 7f4a8a525700 1 -- 172.21.15.32:0/3395601831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a84102e70 msgr2=0x7f4a84103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.368+0000 7f4a8a525700 1 --2- 172.21.15.32:0/3395601831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a84102e70 0x7f4a84103240 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4a74004940 tx=0x7f4a74009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.368+0000 7f4a8a525700 1 -- 172.21.15.32:0/3395601831 shutdown_connections 2024-04-06T14:42:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.368+0000 7f4a8a525700 1 --2- 172.21.15.32:0/3395601831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a840fb4a0 0x7f4a840fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.368+0000 7f4a8a525700 1 --2- 172.21.15.32:0/3395601831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a84102e70 0x7f4a84103240 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.368+0000 7f4a8a525700 1 -- 172.21.15.32:0/3395601831 >> 172.21.15.32:0/3395601831 conn(0x7f4a840714c0 msgr2=0x7f4a840718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.368+0000 7f4a8a525700 1 -- 172.21.15.32:0/3395601831 shutdown_connections 2024-04-06T14:42:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.368+0000 7f4a8a525700 1 -- 172.21.15.32:0/3395601831 wait complete. 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.369+0000 7f4a8a525700 1 Processor -- start 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.369+0000 7f4a8a525700 1 -- start start 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.369+0000 7f4a8a525700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a840fb4a0 0x7f4a84113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.369+0000 7f4a8a525700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a84102e70 0x7f4a84114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.369+0000 7f4a8a525700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a841146e0 con 0x7f4a840fb4a0 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.369+0000 7f4a8a525700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8410dad0 con 0x7f4a84102e70 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a83fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a840fb4a0 0x7f4a84113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a83fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a840fb4a0 0x7f4a84113b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55368/0 (socket says 172.21.15.32:55368) 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a83fff700 1 -- 172.21.15.32:0/2678215392 learned_addr learned my addr 172.21.15.32:0/2678215392 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a837fe700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a84102e70 0x7f4a84114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a83fff700 1 -- 172.21.15.32:0/2678215392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a84102e70 msgr2=0x7f4a84114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a83fff700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a84102e70 0x7f4a84114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a83fff700 1 -- 172.21.15.32:0/2678215392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a74005040 con 0x7f4a840fb4a0 2024-04-06T14:42:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a837fe700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a84102e70 0x7f4a84114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:42:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a83fff700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a840fb4a0 0x7f4a84113b50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009360 tx=0x7f4a74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a80ff9700 1 -- 172.21.15.32:0/2678215392 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a7400e040 con 0x7f4a840fb4a0 2024-04-06T14:42:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8410dd50 con 0x7f4a840fb4a0 2024-04-06T14:42:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a80ff9700 1 -- 172.21.15.32:0/2678215392 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a74013420 con 0x7f4a840fb4a0 2024-04-06T14:42:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a80ff9700 1 -- 172.21.15.32:0/2678215392 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a7401c5f0 con 0x7f4a840fb4a0 2024-04-06T14:42:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.370+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8410e290 con 0x7f4a840fb4a0 2024-04-06T14:42:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.371+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a840653a0 con 0x7f4a840fb4a0 2024-04-06T14:42:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.372+0000 7f4a80ff9700 1 -- 172.21.15.32:0/2678215392 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4a74013590 con 0x7f4a840fb4a0 2024-04-06T14:42:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.372+0000 7f4a80ff9700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4a6c06cc30 0x7f4a6c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.372+0000 7f4a80ff9700 1 -- 172.21.15.32:0/2678215392 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4a7408d2d0 con 0x7f4a840fb4a0 2024-04-06T14:42:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.372+0000 7f4a837fe700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4a6c06cc30 0x7f4a6c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.373+0000 7f4a837fe700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4a6c06cc30 0x7f4a6c06f0e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009fd0 tx=0x7f4a78009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.377+0000 7f4a80ff9700 1 -- 172.21.15.32:0/2678215392 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4a7405ada0 con 0x7f4a840fb4a0 2024-04-06T14:42:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.627+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4a84105300 con 0x7f4a840fb4a0 2024-04-06T14:42:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.627+0000 7f4a80ff9700 1 -- 172.21.15.32:0/2678215392 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f4a7405a930 con 0x7f4a840fb4a0 2024-04-06T14:42:19.618 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:42:19.618 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:42:19.618 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:42:19.619 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:42:19.620 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:42:19.621 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4a6c06cc30 msgr2=0x7f4a6c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4a6c06cc30 0x7f4a6c06f0e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009fd0 tx=0x7f4a78009ee0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a840fb4a0 msgr2=0x7f4a84113b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a840fb4a0 0x7f4a84113b50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009360 tx=0x7f4a74009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 shutdown_connections 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4a6c06cc30 0x7f4a6c06f0e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a840fb4a0 0x7f4a84113b50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 --2- 172.21.15.32:0/2678215392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a84102e70 0x7f4a84114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 >> 172.21.15.32:0/2678215392 conn(0x7f4a840714c0 msgr2=0x7f4a840fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 shutdown_connections 2024-04-06T14:42:19.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.629+0000 7f4a8a525700 1 -- 172.21.15.32:0/2678215392 wait complete. 2024-04-06T14:42:19.623 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:42:19.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.733+0000 7eff434ab700 1 -- 172.21.15.32:0/2686495862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fde90 msgr2=0x7eff3c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.733+0000 7eff434ab700 1 --2- 172.21.15.32:0/2686495862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fde90 0x7eff3c0fe260 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7eff38004940 tx=0x7eff38009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:19.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.733+0000 7eff434ab700 1 -- 172.21.15.32:0/2686495862 shutdown_connections 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.733+0000 7eff434ab700 1 --2- 172.21.15.32:0/2686495862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff3c0fe830 0x7eff3c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.733+0000 7eff434ab700 1 --2- 172.21.15.32:0/2686495862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fde90 0x7eff3c0fe260 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.733+0000 7eff434ab700 1 -- 172.21.15.32:0/2686495862 >> 172.21.15.32:0/2686495862 conn(0x7eff3c072bc0 msgr2=0x7eff3c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.733+0000 7eff434ab700 1 -- 172.21.15.32:0/2686495862 shutdown_connections 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.733+0000 7eff434ab700 1 -- 172.21.15.32:0/2686495862 wait complete. 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff434ab700 1 Processor -- start 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff434ab700 1 -- start start 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff434ab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff3c0fde90 0x7eff3c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff434ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fe830 0x7eff3c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff434ab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c0fc820 con 0x7eff3c0fe830 2024-04-06T14:42:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff434ab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c0fc960 con 0x7eff3c0fde90 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff40a46700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fe830 0x7eff3c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff40a46700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fe830 0x7eff3c0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55394/0 (socket says 172.21.15.32:55394) 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff40a46700 1 -- 172.21.15.32:0/199493223 learned_addr learned my addr 172.21.15.32:0/199493223 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.734+0000 7eff41247700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff3c0fde90 0x7eff3c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.735+0000 7eff40a46700 1 -- 172.21.15.32:0/199493223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff3c0fde90 msgr2=0x7eff3c0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.735+0000 7eff40a46700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff3c0fde90 0x7eff3c0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.735+0000 7eff40a46700 1 -- 172.21.15.32:0/199493223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff38005040 con 0x7eff3c0fe830 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.735+0000 7eff40a46700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fe830 0x7eff3c0fc290 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7eff30002a50 tx=0x7eff300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.735+0000 7eff2dffb700 1 -- 172.21.15.32:0/199493223 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff30007cc0 con 0x7eff3c0fe830 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.735+0000 7eff2dffb700 1 -- 172.21.15.32:0/199493223 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff3000f040 con 0x7eff3c0fe830 2024-04-06T14:42:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.735+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3c119df0 con 0x7eff3c0fe830 2024-04-06T14:42:19.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.735+0000 7eff2dffb700 1 -- 172.21.15.32:0/199493223 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff3000d9c0 con 0x7eff3c0fe830 2024-04-06T14:42:19.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.735+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3c11a1f0 con 0x7eff3c0fe830 2024-04-06T14:42:19.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.736+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3c065e30 con 0x7eff3c0fe830 2024-04-06T14:42:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.736+0000 7eff2dffb700 1 -- 172.21.15.32:0/199493223 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7eff3000db20 con 0x7eff3c0fe830 2024-04-06T14:42:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.737+0000 7eff2dffb700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7eff2806cc30 0x7eff2806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.737+0000 7eff2dffb700 1 -- 172.21.15.32:0/199493223 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7eff3008bdb0 con 0x7eff3c0fe830 2024-04-06T14:42:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.737+0000 7eff41247700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7eff2806cc30 0x7eff2806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.737+0000 7eff41247700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7eff2806cc30 0x7eff2806f0e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7eff3800b3b0 tx=0x7eff38009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:19.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.741+0000 7eff2dffb700 1 -- 172.21.15.32:0/199493223 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7eff30055d70 con 0x7eff3c0fe830 2024-04-06T14:42:19.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:19 smithi086 ceph-mon[110975]: pgmap v666: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:19.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:19 smithi086 ceph-mon[110975]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:19.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:19 smithi086 ceph-mon[110975]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:19.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:19 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3405696874' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:42:19.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.966+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff3c04bed0 con 0x7eff2806cc30 2024-04-06T14:42:19.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.967+0000 7eff2dffb700 1 -- 172.21.15.32:0/199493223 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7eff3c04bed0 con 0x7eff2806cc30 2024-04-06T14:42:19.958 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:42:19.958 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:42:19.958 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:42:19.958 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:42:19.958 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:42:19.958 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:42:19.959 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:42:19.959 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:42:19.959 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:42:19.959 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:42:19.959 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:42:19.959 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:42:19.959 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:42:19.959 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:42:19.959 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:42:19.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7eff2806cc30 msgr2=0x7eff2806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7eff2806cc30 0x7eff2806f0e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7eff3800b3b0 tx=0x7eff38009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:19.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fe830 msgr2=0x7eff3c0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:19.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fe830 0x7eff3c0fc290 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7eff30002a50 tx=0x7eff300046b0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 shutdown_connections 2024-04-06T14:42:19.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7eff2806cc30 0x7eff2806f0e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff3c0fde90 0x7eff3c0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 --2- 172.21.15.32:0/199493223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff3c0fe830 0x7eff3c0fc290 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:19.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 >> 172.21.15.32:0/199493223 conn(0x7eff3c072bc0 msgr2=0x7eff3c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:19.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 shutdown_connections 2024-04-06T14:42:19.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:19.969+0000 7eff434ab700 1 -- 172.21.15.32:0/199493223 wait complete. 2024-04-06T14:42:19.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:19 smithi032 ceph-mon[147414]: pgmap v666: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:19.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:19 smithi032 ceph-mon[147414]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:19.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:19 smithi032 ceph-mon[147414]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:19.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:19 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3405696874' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:42:20.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.072+0000 7fed2c070700 1 -- 172.21.15.32:0/3741848115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed24102e10 msgr2=0x7fed241031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.072+0000 7fed2c070700 1 --2- 172.21.15.32:0/3741848115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed24102e10 0x7fed241031e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fed20004940 tx=0x7fed20009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.072+0000 7fed2c070700 1 -- 172.21.15.32:0/3741848115 shutdown_connections 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.072+0000 7fed2c070700 1 --2- 172.21.15.32:0/3741848115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed241037b0 0x7fed240fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.072+0000 7fed2c070700 1 --2- 172.21.15.32:0/3741848115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed24102e10 0x7fed241031e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.072+0000 7fed2c070700 1 -- 172.21.15.32:0/3741848115 >> 172.21.15.32:0/3741848115 conn(0x7fed240f89a0 msgr2=0x7fed240fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2c070700 1 -- 172.21.15.32:0/3741848115 shutdown_connections 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2c070700 1 -- 172.21.15.32:0/3741848115 wait complete. 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2c070700 1 Processor -- start 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2c070700 1 -- start start 2024-04-06T14:42:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2c070700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed24102e10 0x7fed240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2c070700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed241037b0 0x7fed24070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2c070700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed24070d90 con 0x7fed241037b0 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2c070700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed24070ed0 con 0x7fed24102e10 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2960b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed241037b0 0x7fed24070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2960b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed241037b0 0x7fed24070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55416/0 (socket says 172.21.15.32:55416) 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.073+0000 7fed2960b700 1 -- 172.21.15.32:0/2824181985 learned_addr learned my addr 172.21.15.32:0/2824181985 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed29e0c700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed24102e10 0x7fed240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed2960b700 1 -- 172.21.15.32:0/2824181985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed24102e10 msgr2=0x7fed240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed2960b700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed24102e10 0x7fed240702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed2960b700 1 -- 172.21.15.32:0/2824181985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20005040 con 0x7fed241037b0 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed2960b700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed241037b0 0x7fed24070800 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fed1800eb40 tx=0x7fed1800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed167fc700 1 -- 172.21.15.32:0/2824181985 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed18007cc0 con 0x7fed241037b0 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed24071120 con 0x7fed241037b0 2024-04-06T14:42:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed167fc700 1 -- 172.21.15.32:0/2824181985 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed1800f040 con 0x7fed241037b0 2024-04-06T14:42:20.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed167fc700 1 -- 172.21.15.32:0/2824181985 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1800c8f0 con 0x7fed241037b0 2024-04-06T14:42:20.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.074+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2411a180 con 0x7fed241037b0 2024-04-06T14:42:20.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.075+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed240fe560 con 0x7fed241037b0 2024-04-06T14:42:20.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.076+0000 7fed167fc700 1 -- 172.21.15.32:0/2824181985 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fed1800cb60 con 0x7fed241037b0 2024-04-06T14:42:20.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.076+0000 7fed167fc700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1006cc20 0x7fed1006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:20.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.077+0000 7fed29e0c700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1006cc20 0x7fed1006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:20.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.077+0000 7fed167fc700 1 -- 172.21.15.32:0/2824181985 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fed1808bb50 con 0x7fed241037b0 2024-04-06T14:42:20.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.077+0000 7fed29e0c700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1006cc20 0x7fed1006f0d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fed2000b0d0 tx=0x7fed20009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:20.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.081+0000 7fed167fc700 1 -- 172.21.15.32:0/2824181985 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fed18055b40 con 0x7fed241037b0 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.373+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fed2405f5f0 con 0x7fed241037b0 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.373+0000 7fed167fc700 1 -- 172.21.15.32:0/2824181985 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7fed18059160 con 0x7fed241037b0 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:42:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:42:20.366 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:42:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:42:20.367 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:42:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:42:20.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.375+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1006cc20 msgr2=0x7fed1006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.375+0000 7fed2c070700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1006cc20 0x7fed1006f0d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fed2000b0d0 tx=0x7fed20009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.375+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed241037b0 msgr2=0x7fed24070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.375+0000 7fed2c070700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed241037b0 0x7fed24070800 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fed1800eb40 tx=0x7fed1800b040 comp rx=0 tx=0).stop 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.376+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 shutdown_connections 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.376+0000 7fed2c070700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1006cc20 0x7fed1006f0d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.376+0000 7fed2c070700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed24102e10 0x7fed240702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.376+0000 7fed2c070700 1 --2- 172.21.15.32:0/2824181985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed241037b0 0x7fed24070800 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.376+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 >> 172.21.15.32:0/2824181985 conn(0x7fed240f89a0 msgr2=0x7fed240f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.376+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 shutdown_connections 2024-04-06T14:42:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:20.376+0000 7fed2c070700 1 -- 172.21.15.32:0/2824181985 wait complete. 2024-04-06T14:42:20.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:20 smithi086 ceph-mon[110975]: from='client.35242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:20.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:20 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2678215392' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:42:20.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:20 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2824181985' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:42:20.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:20 smithi032 ceph-mon[147414]: from='client.35242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:20.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:20 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2678215392' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:42:20.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:20 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2824181985' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:42:21.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:21 smithi086 ceph-mon[110975]: from='client.35254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:21.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:21 smithi086 ceph-mon[110975]: pgmap v667: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:21.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:21 smithi032 ceph-mon[147414]: from='client.35254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:21.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:21 smithi032 ceph-mon[147414]: pgmap v667: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:23.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:23 smithi032 ceph-mon[147414]: pgmap v668: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:23.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:42:23.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:23 smithi086 ceph-mon[110975]: pgmap v668: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:23.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:42:25.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:25 smithi086 ceph-mon[110975]: pgmap v669: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:25.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:25 smithi032 ceph-mon[147414]: pgmap v669: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:27.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:27 smithi086 ceph-mon[110975]: pgmap v670: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:27.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:27 smithi032 ceph-mon[147414]: pgmap v670: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:29.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:29 smithi086 ceph-mon[110975]: pgmap v671: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:29.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:29 smithi032 ceph-mon[147414]: pgmap v671: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:31.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:31 smithi032 ceph-mon[147414]: pgmap v672: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:32.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:31 smithi086 ceph-mon[110975]: pgmap v672: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:33.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:33 smithi032 ceph-mon[147414]: pgmap v673: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:34.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:33 smithi086 ceph-mon[110975]: pgmap v673: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:35.976 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:35 smithi032 ceph-mon[147414]: pgmap v674: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:36.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:35 smithi086 ceph-mon[110975]: pgmap v674: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:37.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:37 smithi032 ceph-mon[147414]: pgmap v675: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:37.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:42:38.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:37 smithi086 ceph-mon[110975]: pgmap v675: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:38.120 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:42:39.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:39 smithi032 ceph-mon[147414]: pgmap v676: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:40.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:39 smithi086 ceph-mon[110975]: pgmap v676: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:41.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:41 smithi032 ceph-mon[147414]: pgmap v677: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:42.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:41 smithi086 ceph-mon[110975]: pgmap v677: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:43.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:43 smithi032 ceph-mon[147414]: pgmap v678: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:44.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:43 smithi086 ceph-mon[110975]: pgmap v678: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:45.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:45 smithi032 ceph-mon[147414]: pgmap v679: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:46.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:45 smithi086 ceph-mon[110975]: pgmap v679: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:46.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:46 smithi032 ceph-mon[147414]: pgmap v680: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:47.119 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:46 smithi086 ceph-mon[110975]: pgmap v680: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:49.471 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:49 smithi032 ceph-mon[147414]: pgmap v681: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:49.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:49 smithi086 ceph-mon[110975]: pgmap v681: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.492+0000 7f11dbeaf700 1 -- 172.21.15.32:0/1272093403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fde90 msgr2=0x7f11d40fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.492+0000 7f11dbeaf700 1 --2- 172.21.15.32:0/1272093403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fde90 0x7f11d40fe260 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f11d0004940 tx=0x7f11d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.492+0000 7f11dbeaf700 1 -- 172.21.15.32:0/1272093403 shutdown_connections 2024-04-06T14:42:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.492+0000 7f11dbeaf700 1 --2- 172.21.15.32:0/1272093403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11d40fe830 0x7f11d4106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.492+0000 7f11dbeaf700 1 --2- 172.21.15.32:0/1272093403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fde90 0x7f11d40fe260 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.492+0000 7f11dbeaf700 1 -- 172.21.15.32:0/1272093403 >> 172.21.15.32:0/1272093403 conn(0x7f11d4072bc0 msgr2=0x7f11d4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.492+0000 7f11dbeaf700 1 -- 172.21.15.32:0/1272093403 shutdown_connections 2024-04-06T14:42:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.492+0000 7f11dbeaf700 1 -- 172.21.15.32:0/1272093403 wait complete. 2024-04-06T14:42:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11dbeaf700 1 Processor -- start 2024-04-06T14:42:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11dbeaf700 1 -- start start 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11dbeaf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11d40fde90 0x7f11d40fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11dbeaf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fe830 0x7f11d40fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11dbeaf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40fc820 con 0x7f11d40fe830 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11dbeaf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40fc960 con 0x7f11d40fde90 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11d944a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fe830 0x7f11d40fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11d944a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fe830 0x7f11d40fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52630/0 (socket says 172.21.15.32:52630) 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11d944a700 1 -- 172.21.15.32:0/123516262 learned_addr learned my addr 172.21.15.32:0/123516262 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.493+0000 7f11d9c4b700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11d40fde90 0x7f11d40fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.494+0000 7f11d944a700 1 -- 172.21.15.32:0/123516262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11d40fde90 msgr2=0x7f11d40fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.494+0000 7f11d944a700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11d40fde90 0x7f11d40fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.494+0000 7f11d944a700 1 -- 172.21.15.32:0/123516262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d0005040 con 0x7f11d40fe830 2024-04-06T14:42:50.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.494+0000 7f11d944a700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fe830 0x7f11d40fc290 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f11c4002a80 tx=0x7f11c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:50.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.494+0000 7f11ca7fc700 1 -- 172.21.15.32:0/123516262 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c4007cc0 con 0x7f11d40fe830 2024-04-06T14:42:50.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.494+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d4119d70 con 0x7f11d40fe830 2024-04-06T14:42:50.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.494+0000 7f11ca7fc700 1 -- 172.21.15.32:0/123516262 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11c400f040 con 0x7f11d40fe830 2024-04-06T14:42:50.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.494+0000 7f11ca7fc700 1 -- 172.21.15.32:0/123516262 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c400da60 con 0x7f11d40fe830 2024-04-06T14:42:50.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.494+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d411a170 con 0x7f11d40fe830 2024-04-06T14:42:50.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.495+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d4065e30 con 0x7f11d40fe830 2024-04-06T14:42:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.496+0000 7f11ca7fc700 1 -- 172.21.15.32:0/123516262 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f11c400dbc0 con 0x7f11d40fe830 2024-04-06T14:42:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.496+0000 7f11ca7fc700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c006cc30 0x7f11c006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.496+0000 7f11ca7fc700 1 -- 172.21.15.32:0/123516262 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f11c408bcf0 con 0x7f11d40fe830 2024-04-06T14:42:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.496+0000 7f11d9c4b700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c006cc30 0x7f11c006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.497+0000 7f11d9c4b700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c006cc30 0x7f11c006f0e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f11d000b3b0 tx=0x7f11d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:50.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.501+0000 7f11ca7fc700 1 -- 172.21.15.32:0/123516262 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f11c4055cb0 con 0x7f11d40fe830 2024-04-06T14:42:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.724+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f11d404bed0 con 0x7f11c006cc30 2024-04-06T14:42:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.726+0000 7f11ca7fc700 1 -- 172.21.15.32:0/123516262 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f11d404bed0 con 0x7f11c006cc30 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c006cc30 msgr2=0x7f11c006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c006cc30 0x7f11c006f0e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f11d000b3b0 tx=0x7f11d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fe830 msgr2=0x7f11d40fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fe830 0x7f11d40fc290 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f11c4002a80 tx=0x7f11c40046b0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 shutdown_connections 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f11c006cc30 0x7f11c006f0e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11d40fde90 0x7f11d40fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 --2- 172.21.15.32:0/123516262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11d40fe830 0x7f11d40fc290 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 >> 172.21.15.32:0/123516262 conn(0x7f11d4072bc0 msgr2=0x7f11d41037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 shutdown_connections 2024-04-06T14:42:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.728+0000 7f11dbeaf700 1 -- 172.21.15.32:0/123516262 wait complete. 2024-04-06T14:42:50.730 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:42:50.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.832+0000 7f7d3793a700 1 -- 172.21.15.32:0/3310187794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 msgr2=0x7f7d300fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.832+0000 7f7d3793a700 1 --2- 172.21.15.32:0/3310187794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 0x7f7d300fbd80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7d20004940 tx=0x7f7d20009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 -- 172.21.15.32:0/3310187794 shutdown_connections 2024-04-06T14:42:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 --2- 172.21.15.32:0/3310187794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 0x7f7d300fbd80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 --2- 172.21.15.32:0/3310187794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d30101d70 0x7f7d30102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 -- 172.21.15.32:0/3310187794 >> 172.21.15.32:0/3310187794 conn(0x7f7d30072bc0 msgr2=0x7f7d30072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 -- 172.21.15.32:0/3310187794 shutdown_connections 2024-04-06T14:42:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 -- 172.21.15.32:0/3310187794 wait complete. 2024-04-06T14:42:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 Processor -- start 2024-04-06T14:42:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 -- start start 2024-04-06T14:42:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d30101d70 0x7f7d3010c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 0x7f7d3010ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3010d210 con 0x7f7d30102710 2024-04-06T14:42:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d3793a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3010d380 con 0x7f7d30101d70 2024-04-06T14:42:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.833+0000 7f7d34ed5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 0x7f7d3010ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d34ed5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 0x7f7d3010ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52650/0 (socket says 172.21.15.32:52650) 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d34ed5700 1 -- 172.21.15.32:0/1719247288 learned_addr learned my addr 172.21.15.32:0/1719247288 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d356d6700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d30101d70 0x7f7d3010c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d34ed5700 1 -- 172.21.15.32:0/1719247288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d30101d70 msgr2=0x7f7d3010c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d34ed5700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d30101d70 0x7f7d3010c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d34ed5700 1 -- 172.21.15.32:0/1719247288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d20005040 con 0x7f7d30102710 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d34ed5700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 0x7f7d3010ccd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7d20000c00 tx=0x7f7d20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d25ffb700 1 -- 172.21.15.32:0/1719247288 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2000e070 con 0x7f7d30102710 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3010d600 con 0x7f7d30102710 2024-04-06T14:42:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d25ffb700 1 -- 172.21.15.32:0/1719247288 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d200093b0 con 0x7f7d30102710 2024-04-06T14:42:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d25ffb700 1 -- 172.21.15.32:0/1719247288 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2001ccb0 con 0x7f7d30102710 2024-04-06T14:42:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.834+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3011a150 con 0x7f7d30102710 2024-04-06T14:42:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.835+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d300ff080 con 0x7f7d30102710 2024-04-06T14:42:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.836+0000 7f7d25ffb700 1 -- 172.21.15.32:0/1719247288 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7d2001c490 con 0x7f7d30102710 2024-04-06T14:42:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.836+0000 7f7d25ffb700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d1c06cc20 0x7f7d1c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.836+0000 7f7d25ffb700 1 -- 172.21.15.32:0/1719247288 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7d2008d2f0 con 0x7f7d30102710 2024-04-06T14:42:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.836+0000 7f7d356d6700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d1c06cc20 0x7f7d1c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:50.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.837+0000 7f7d356d6700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d1c06cc20 0x7f7d1c06f0d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009b80 tx=0x7f7d2c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:50.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:50.841+0000 7f7d25ffb700 1 -- 172.21.15.32:0/1719247288 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7d2005adc0 con 0x7f7d30102710 2024-04-06T14:42:51.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.060+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d3010d600 con 0x7f7d1c06cc20 2024-04-06T14:42:51.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.061+0000 7f7d25ffb700 1 -- 172.21.15.32:0/1719247288 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7d3010d600 con 0x7f7d1c06cc20 2024-04-06T14:42:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d1c06cc20 msgr2=0x7f7d1c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d1c06cc20 0x7f7d1c06f0d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009b80 tx=0x7f7d2c00b040 comp rx=0 tx=0).stop 2024-04-06T14:42:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 msgr2=0x7f7d3010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 0x7f7d3010ccd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7d20000c00 tx=0x7f7d20009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 shutdown_connections 2024-04-06T14:42:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d1c06cc20 0x7f7d1c06f0d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d30101d70 0x7f7d3010c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 --2- 172.21.15.32:0/1719247288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d30102710 0x7f7d3010ccd0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 >> 172.21.15.32:0/1719247288 conn(0x7f7d30072bc0 msgr2=0x7f7d300faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 shutdown_connections 2024-04-06T14:42:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.063+0000 7f7d3793a700 1 -- 172.21.15.32:0/1719247288 wait complete. 2024-04-06T14:42:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 -- 172.21.15.32:0/785886234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 msgr2=0x7f20100fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 --2- 172.21.15.32:0/785886234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 0x7f20100fbd80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f200c004940 tx=0x7f200c009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 -- 172.21.15.32:0/785886234 shutdown_connections 2024-04-06T14:42:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 --2- 172.21.15.32:0/785886234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 0x7f20100fbd80 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 --2- 172.21.15.32:0/785886234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2010101d70 0x7f2010102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 -- 172.21.15.32:0/785886234 >> 172.21.15.32:0/785886234 conn(0x7f2010072bc0 msgr2=0x7f2010072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 -- 172.21.15.32:0/785886234 shutdown_connections 2024-04-06T14:42:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 -- 172.21.15.32:0/785886234 wait complete. 2024-04-06T14:42:51.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 Processor -- start 2024-04-06T14:42:51.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.167+0000 7f2017abc700 1 -- start start 2024-04-06T14:42:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2017abc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2010101d70 0x7f2010113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2017abc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 0x7f2010113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2017abc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2010114630 con 0x7f2010102710 2024-04-06T14:42:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2017abc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201010da20 con 0x7f2010101d70 2024-04-06T14:42:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2015057700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 0x7f2010113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2015057700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 0x7f2010113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52670/0 (socket says 172.21.15.32:52670) 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2015057700 1 -- 172.21.15.32:0/1895067331 learned_addr learned my addr 172.21.15.32:0/1895067331 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2015858700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2010101d70 0x7f2010113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2015057700 1 -- 172.21.15.32:0/1895067331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2010101d70 msgr2=0x7f2010113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2015057700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2010101d70 0x7f2010113aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2015057700 1 -- 172.21.15.32:0/1895067331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c005040 con 0x7f2010102710 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2015057700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 0x7f2010113fe0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f200c0093b0 tx=0x7f200c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f20027fc700 1 -- 172.21.15.32:0/1895067331 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f200c00e040 con 0x7f2010102710 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f20027fc700 1 -- 172.21.15.32:0/1895067331 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f200c013420 con 0x7f2010102710 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201010dca0 con 0x7f2010102710 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f20027fc700 1 -- 172.21.15.32:0/1895067331 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f200c01c630 con 0x7f2010102710 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.168+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201010e260 con 0x7f2010102710 2024-04-06T14:42:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.170+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20100fd4c0 con 0x7f2010102710 2024-04-06T14:42:51.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.170+0000 7f20027fc700 1 -- 172.21.15.32:0/1895067331 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f200c0139e0 con 0x7f2010102710 2024-04-06T14:42:51.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.170+0000 7f20027fc700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ffc06cc20 0x7f1ffc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:51.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.170+0000 7f20027fc700 1 -- 172.21.15.32:0/1895067331 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f200c08d060 con 0x7f2010102710 2024-04-06T14:42:51.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.171+0000 7f2015858700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ffc06cc20 0x7f1ffc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:51.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.171+0000 7f2015858700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ffc06cc20 0x7f1ffc06f0d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2004004f40 tx=0x7f200400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:51.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.175+0000 7f20027fc700 1 -- 172.21.15.32:0/1895067331 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f200c05aca0 con 0x7f2010102710 2024-04-06T14:42:51.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.399+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f20100666a0 con 0x7f1ffc06cc20 2024-04-06T14:42:51.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.408+0000 7f20027fc700 1 -- 172.21.15.32:0/1895067331 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f20100666a0 con 0x7f1ffc06cc20 2024-04-06T14:42:51.400 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (23m) 69s ago 34m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (20m) 69s ago 34m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (20m) 72s ago 33m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (22m) 69s ago 34m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (16m) 69s ago 28m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (16m) 69s ago 28m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (16m) 72s ago 28m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (75s) 72s ago 28m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (25m) 69s ago 35m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (24m) 72s ago 33m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (21m) 69s ago 36m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (21m) 72s ago 33m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:42:51.401 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (23m) 69s ago 34m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:42:51.402 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (23m) 72s ago 33m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:42:51.402 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (20m) 69s ago 32m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:42:51.402 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (19m) 69s ago 32m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:42:51.402 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (18m) 69s ago 31m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:42:51.402 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (18m) 72s ago 31m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:42:51.402 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (17m) 72s ago 30m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:42:51.402 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (17m) 72s ago 30m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:42:51.402 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (23m) 69s ago 34m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.410+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ffc06cc20 msgr2=0x7f1ffc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.410+0000 7f2017abc700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ffc06cc20 0x7f1ffc06f0d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2004004f40 tx=0x7f200400b040 comp rx=0 tx=0).stop 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.411+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 msgr2=0x7f2010113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.411+0000 7f2017abc700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 0x7f2010113fe0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f200c0093b0 tx=0x7f200c009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.411+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 shutdown_connections 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.411+0000 7f2017abc700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ffc06cc20 0x7f1ffc06f0d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.411+0000 7f2017abc700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2010101d70 0x7f2010113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.411+0000 7f2017abc700 1 --2- 172.21.15.32:0/1895067331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2010102710 0x7f2010113fe0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.411+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 >> 172.21.15.32:0/1895067331 conn(0x7f2010072bc0 msgr2=0x7f20101058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.411+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 shutdown_connections 2024-04-06T14:42:51.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.411+0000 7f2017abc700 1 -- 172.21.15.32:0/1895067331 wait complete. 2024-04-06T14:42:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.513+0000 7fa81427b700 1 -- 172.21.15.32:0/1444359163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 msgr2=0x7fa80c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.513+0000 7fa81427b700 1 --2- 172.21.15.32:0/1444359163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 0x7fa80c102140 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa808004940 tx=0x7fa808009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.513+0000 7fa81427b700 1 -- 172.21.15.32:0/1444359163 shutdown_connections 2024-04-06T14:42:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.513+0000 7fa81427b700 1 --2- 172.21.15.32:0/1444359163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c102710 0x7fa80c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.513+0000 7fa81427b700 1 --2- 172.21.15.32:0/1444359163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 0x7fa80c102140 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.513+0000 7fa81427b700 1 -- 172.21.15.32:0/1444359163 >> 172.21.15.32:0/1444359163 conn(0x7fa80c072bc0 msgr2=0x7fa80c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.513+0000 7fa81427b700 1 -- 172.21.15.32:0/1444359163 shutdown_connections 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.513+0000 7fa81427b700 1 -- 172.21.15.32:0/1444359163 wait complete. 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa81427b700 1 Processor -- start 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa81427b700 1 -- start start 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa81427b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 0x7fa80c115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa81427b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c102710 0x7fa80c1161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa81427b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c116820 con 0x7fa80c101d70 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa81427b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c10fc40 con 0x7fa80c102710 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa812017700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 0x7fa80c115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa812017700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 0x7fa80c115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52680/0 (socket says 172.21.15.32:52680) 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa812017700 1 -- 172.21.15.32:0/2556217608 learned_addr learned my addr 172.21.15.32:0/2556217608 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa811816700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c102710 0x7fa80c1161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa812017700 1 -- 172.21.15.32:0/2556217608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c102710 msgr2=0x7fa80c1161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.514+0000 7fa812017700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c102710 0x7fa80c1161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.515+0000 7fa812017700 1 -- 172.21.15.32:0/2556217608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa808005040 con 0x7fa80c101d70 2024-04-06T14:42:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.515+0000 7fa812017700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 0x7fa80c115c90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa80800b3b0 tx=0x7fa808009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.515+0000 7fa7feffd700 1 -- 172.21.15.32:0/2556217608 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa80800e040 con 0x7fa80c101d70 2024-04-06T14:42:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.515+0000 7fa7feffd700 1 -- 172.21.15.32:0/2556217608 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa808013420 con 0x7fa80c101d70 2024-04-06T14:42:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.515+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80c10fec0 con 0x7fa80c101d70 2024-04-06T14:42:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.515+0000 7fa7feffd700 1 -- 172.21.15.32:0/2556217608 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa80801c610 con 0x7fa80c101d70 2024-04-06T14:42:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.515+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80c110480 con 0x7fa80c101d70 2024-04-06T14:42:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.516+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80c0fd4c0 con 0x7fa80c101d70 2024-04-06T14:42:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.516+0000 7fa7feffd700 1 -- 172.21.15.32:0/2556217608 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa808013a00 con 0x7fa80c101d70 2024-04-06T14:42:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.517+0000 7fa7feffd700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa7f806cc20 0x7fa7f806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.517+0000 7fa7feffd700 1 -- 172.21.15.32:0/2556217608 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa80808d150 con 0x7fa80c101d70 2024-04-06T14:42:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.517+0000 7fa811816700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa7f806cc20 0x7fa7f806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.517+0000 7fa811816700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa7f806cc20 0x7fa7f806f0d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa80c111530 tx=0x7fa80000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:51.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.521+0000 7fa7feffd700 1 -- 172.21.15.32:0/2556217608 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa80805ac20 con 0x7fa80c101d70 2024-04-06T14:42:51.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:51 smithi086 ceph-mon[110975]: pgmap v682: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:51.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:51 smithi032 ceph-mon[147414]: pgmap v682: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:51.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.808+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa80c04bed0 con 0x7fa80c101d70 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.808+0000 7fa7feffd700 1 -- 172.21.15.32:0/2556217608 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fa80805a7b0 con 0x7fa80c101d70 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:42:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:42:51.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:42:51.801 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:42:51.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:42:51.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:42:51.801 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:42:51.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:42:51.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:42:51.801 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:42:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa7f806cc20 msgr2=0x7fa7f806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa7f806cc20 0x7fa7f806f0d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa80c111530 tx=0x7fa80000b040 comp rx=0 tx=0).stop 2024-04-06T14:42:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 msgr2=0x7fa80c115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 0x7fa80c115c90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa80800b3b0 tx=0x7fa808009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 shutdown_connections 2024-04-06T14:42:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa7f806cc20 0x7fa7f806f0d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c101d70 0x7fa80c115c90 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 --2- 172.21.15.32:0/2556217608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c102710 0x7fa80c1161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 >> 172.21.15.32:0/2556217608 conn(0x7fa80c072bc0 msgr2=0x7fa80c065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 shutdown_connections 2024-04-06T14:42:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.810+0000 7fa81427b700 1 -- 172.21.15.32:0/2556217608 wait complete. 2024-04-06T14:42:51.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.914+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2853289564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4101d70 msgr2=0x7fc4f4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.914+0000 7fc4fa3a2700 1 --2- 172.21.15.32:0/2853289564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4101d70 0x7fc4f4102140 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc004940 tx=0x7fc4dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:51.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.914+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2853289564 shutdown_connections 2024-04-06T14:42:51.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.914+0000 7fc4fa3a2700 1 --2- 172.21.15.32:0/2853289564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4f4102710 0x7fc4f40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.914+0000 7fc4fa3a2700 1 --2- 172.21.15.32:0/2853289564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4101d70 0x7fc4f4102140 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.914+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2853289564 >> 172.21.15.32:0/2853289564 conn(0x7fc4f4072bc0 msgr2=0x7fc4f4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:51.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.914+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2853289564 shutdown_connections 2024-04-06T14:42:51.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.914+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2853289564 wait complete. 2024-04-06T14:42:51.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.914+0000 7fc4fa3a2700 1 Processor -- start 2024-04-06T14:42:51.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4fa3a2700 1 -- start start 2024-04-06T14:42:51.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4fa3a2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4f4101d70 0x7fc4f4115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:51.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4fa3a2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4102710 0x7fc4f4116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:51.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4fa3a2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f41167d0 con 0x7fc4f4102710 2024-04-06T14:42:51.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4fa3a2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f410fba0 con 0x7fc4f4101d70 2024-04-06T14:42:51.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4f37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4102710 0x7fc4f4116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:51.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4f37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4102710 0x7fc4f4116180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52702/0 (socket says 172.21.15.32:52702) 2024-04-06T14:42:51.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4f37fe700 1 -- 172.21.15.32:0/2482589789 learned_addr learned my addr 172.21.15.32:0/2482589789 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4f3fff700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4f4101d70 0x7fc4f4115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4f37fe700 1 -- 172.21.15.32:0/2482589789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4f4101d70 msgr2=0x7fc4f4115c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4f37fe700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4f4101d70 0x7fc4f4115c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4f37fe700 1 -- 172.21.15.32:0/2482589789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc005040 con 0x7fc4f4102710 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4f37fe700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4102710 0x7fc4f4116180 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc4e400eb40 tx=0x7fc4e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.915+0000 7fc4f0ff9700 1 -- 172.21.15.32:0/2482589789 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4e4007cc0 con 0x7fc4f4102710 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.916+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f410fe20 con 0x7fc4f4102710 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.916+0000 7fc4f0ff9700 1 -- 172.21.15.32:0/2482589789 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4e400f040 con 0x7fc4f4102710 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.916+0000 7fc4f0ff9700 1 -- 172.21.15.32:0/2482589789 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4e4009510 con 0x7fc4f4102710 2024-04-06T14:42:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.916+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f41103e0 con 0x7fc4f4102710 2024-04-06T14:42:51.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.917+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f40fd4c0 con 0x7fc4f4102710 2024-04-06T14:42:51.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.917+0000 7fc4f0ff9700 1 -- 172.21.15.32:0/2482589789 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc4e4009670 con 0x7fc4f4102710 2024-04-06T14:42:51.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.918+0000 7fc4f0ff9700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc4e006cc20 0x7fc4e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:51.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.918+0000 7fc4f0ff9700 1 -- 172.21.15.32:0/2482589789 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc4e408bdc0 con 0x7fc4f4102710 2024-04-06T14:42:51.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.918+0000 7fc4f3fff700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc4e006cc20 0x7fc4e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:51.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.918+0000 7fc4f3fff700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc4e006cc20 0x7fc4e006f0d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00b0d0 tx=0x7fc4dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:51.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:51.922+0000 7fc4f0ff9700 1 -- 172.21.15.32:0/2482589789 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc4e408f030 con 0x7fc4f4102710 2024-04-06T14:42:52.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.171+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc4f404bed0 con 0x7fc4f4102710 2024-04-06T14:42:52.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.171+0000 7fc4f0ff9700 1 -- 172.21.15.32:0/2482589789 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fc4e4059790 con 0x7fc4f4102710 2024-04-06T14:42:52.163 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:42:52.164 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:42:52.164 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:42:52.164 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:42:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:42:52.164 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:42:52.164 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:42:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:42:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:42:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:42:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:42:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:42:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:42:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:42:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:42:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:42:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:42:52.167 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:42:52.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc4e006cc20 msgr2=0x7fc4e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc4e006cc20 0x7fc4e006f0d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00b0d0 tx=0x7fc4dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4102710 msgr2=0x7fc4f4116180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4102710 0x7fc4f4116180 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc4e400eb40 tx=0x7fc4e400b040 comp rx=0 tx=0).stop 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 shutdown_connections 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc4e006cc20 0x7fc4e006f0d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4f4101d70 0x7fc4f4115c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 --2- 172.21.15.32:0/2482589789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4f4102710 0x7fc4f4116180 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 >> 172.21.15.32:0/2482589789 conn(0x7fc4f4072bc0 msgr2=0x7fc4f4065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 shutdown_connections 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.173+0000 7fc4fa3a2700 1 -- 172.21.15.32:0/2482589789 wait complete. 2024-04-06T14:42:52.169 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:42:52.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 -- 172.21.15.32:0/3547277828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070102710 msgr2=0x7fa0700fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 --2- 172.21.15.32:0/3547277828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070102710 0x7fa0700fbd80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa064004940 tx=0x7fa064009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:52.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 -- 172.21.15.32:0/3547277828 shutdown_connections 2024-04-06T14:42:52.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 --2- 172.21.15.32:0/3547277828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070102710 0x7fa0700fbd80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 --2- 172.21.15.32:0/3547277828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa070101d70 0x7fa070102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 -- 172.21.15.32:0/3547277828 >> 172.21.15.32:0/3547277828 conn(0x7fa070072bc0 msgr2=0x7fa070072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:52.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 -- 172.21.15.32:0/3547277828 shutdown_connections 2024-04-06T14:42:52.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 -- 172.21.15.32:0/3547277828 wait complete. 2024-04-06T14:42:52.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 Processor -- start 2024-04-06T14:42:52.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.278+0000 7fa07833d700 1 -- start start 2024-04-06T14:42:52.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa07833d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070101d70 0x7fa07010ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa07833d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa070102710 0x7fa07010efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa07833d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07010f6b0 con 0x7fa070101d70 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa07833d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07010f7f0 con 0x7fa070102710 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa0760d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070101d70 0x7fa07010ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa0760d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070101d70 0x7fa07010ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52730/0 (socket says 172.21.15.32:52730) 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa0760d9700 1 -- 172.21.15.32:0/3500456364 learned_addr learned my addr 172.21.15.32:0/3500456364 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa0760d9700 1 -- 172.21.15.32:0/3500456364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa070102710 msgr2=0x7fa07010efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa0760d9700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa070102710 0x7fa07010efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa0760d9700 1 -- 172.21.15.32:0/3500456364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064005040 con 0x7fa070101d70 2024-04-06T14:42:52.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa0760d9700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070101d70 0x7fa07010ea90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00b790 tx=0x7fa06c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa062ffd700 1 -- 172.21.15.32:0/3500456364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa06c00f770 con 0x7fa070101d70 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa062ffd700 1 -- 172.21.15.32:0/3500456364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa06c00fd80 con 0x7fa070101d70 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa062ffd700 1 -- 172.21.15.32:0/3500456364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa06c00d310 con 0x7fa070101d70 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa070117b60 con 0x7fa070101d70 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.279+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0701180a0 con 0x7fa070101d70 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.280+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0700ff080 con 0x7fa070101d70 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.281+0000 7fa062ffd700 1 -- 172.21.15.32:0/3500456364 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa06c010420 con 0x7fa070101d70 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.281+0000 7fa062ffd700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa05c06cc30 0x7fa05c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.281+0000 7fa062ffd700 1 -- 172.21.15.32:0/3500456364 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa06c08c070 con 0x7fa070101d70 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.281+0000 7fa0758d8700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa05c06cc30 0x7fa05c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.282+0000 7fa0758d8700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa05c06cc30 0x7fa05c06f0e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa064004f90 tx=0x7fa064009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:52.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.286+0000 7fa062ffd700 1 -- 172.21.15.32:0/3500456364 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa06c0560b0 con 0x7fa070101d70 2024-04-06T14:42:52.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.507+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa070108d80 con 0x7fa05c06cc30 2024-04-06T14:42:52.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.508+0000 7fa062ffd700 1 -- 172.21.15.32:0/3500456364 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa070108d80 con 0x7fa05c06cc30 2024-04-06T14:42:52.500 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:42:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:42:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:42:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:42:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:42:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:42:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:42:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:42:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:42:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:42:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:42:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:42:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:42:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:42:52.501 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:42:52.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.510+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa05c06cc30 msgr2=0x7fa05c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.510+0000 7fa07833d700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa05c06cc30 0x7fa05c06f0e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa064004f90 tx=0x7fa064009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:52.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.510+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070101d70 msgr2=0x7fa07010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.510+0000 7fa07833d700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070101d70 0x7fa07010ea90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00b790 tx=0x7fa06c009250 comp rx=0 tx=0).stop 2024-04-06T14:42:52.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.510+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 shutdown_connections 2024-04-06T14:42:52.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.510+0000 7fa07833d700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa05c06cc30 0x7fa05c06f0e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.510+0000 7fa07833d700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa070101d70 0x7fa07010ea90 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.510+0000 7fa07833d700 1 --2- 172.21.15.32:0/3500456364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa070102710 0x7fa07010efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.510+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 >> 172.21.15.32:0/3500456364 conn(0x7fa070072bc0 msgr2=0x7fa0700fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:52.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.511+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 shutdown_connections 2024-04-06T14:42:52.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.511+0000 7fa07833d700 1 -- 172.21.15.32:0/3500456364 wait complete. 2024-04-06T14:42:52.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.613+0000 7fa686353700 1 -- 172.21.15.32:0/4229356488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 msgr2=0x7fa680102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.613+0000 7fa686353700 1 --2- 172.21.15.32:0/4229356488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 0x7fa680102140 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa670004940 tx=0x7fa670009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:52.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.613+0000 7fa686353700 1 -- 172.21.15.32:0/4229356488 shutdown_connections 2024-04-06T14:42:52.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.613+0000 7fa686353700 1 --2- 172.21.15.32:0/4229356488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa680102710 0x7fa6800fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.613+0000 7fa686353700 1 --2- 172.21.15.32:0/4229356488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 0x7fa680102140 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.613+0000 7fa686353700 1 -- 172.21.15.32:0/4229356488 >> 172.21.15.32:0/4229356488 conn(0x7fa680072bc0 msgr2=0x7fa680072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:52.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.613+0000 7fa686353700 1 -- 172.21.15.32:0/4229356488 shutdown_connections 2024-04-06T14:42:52.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.613+0000 7fa686353700 1 -- 172.21.15.32:0/4229356488 wait complete. 2024-04-06T14:42:52.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.613+0000 7fa686353700 1 Processor -- start 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa686353700 1 -- start start 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa686353700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 0x7fa680113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa686353700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa680102710 0x7fa680113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa686353700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa680114510 con 0x7fa680101d70 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa686353700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68010d940 con 0x7fa680102710 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa67ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 0x7fa680113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa67ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 0x7fa680113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52736/0 (socket says 172.21.15.32:52736) 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa67ffff700 1 -- 172.21.15.32:0/3132779963 learned_addr learned my addr 172.21.15.32:0/3132779963 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa67f7fe700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa680102710 0x7fa680113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:52.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa67ffff700 1 -- 172.21.15.32:0/3132779963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa680102710 msgr2=0x7fa680113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa67ffff700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa680102710 0x7fa680113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa67ffff700 1 -- 172.21.15.32:0/3132779963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa670005040 con 0x7fa680101d70 2024-04-06T14:42:52.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.614+0000 7fa67ffff700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 0x7fa680113980 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa670009390 tx=0x7fa670009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:52.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.615+0000 7fa67cff9700 1 -- 172.21.15.32:0/3132779963 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa67000e040 con 0x7fa680101d70 2024-04-06T14:42:52.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.615+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa68010dbc0 con 0x7fa680101d70 2024-04-06T14:42:52.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.615+0000 7fa67cff9700 1 -- 172.21.15.32:0/3132779963 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa670013420 con 0x7fa680101d70 2024-04-06T14:42:52.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.615+0000 7fa67cff9700 1 -- 172.21.15.32:0/3132779963 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa67001c630 con 0x7fa680101d70 2024-04-06T14:42:52.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.615+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68010e180 con 0x7fa680101d70 2024-04-06T14:42:52.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.616+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6800fd4c0 con 0x7fa680101d70 2024-04-06T14:42:52.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.616+0000 7fa67cff9700 1 -- 172.21.15.32:0/3132779963 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa6700139e0 con 0x7fa680101d70 2024-04-06T14:42:52.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.617+0000 7fa67cff9700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa66806cc30 0x7fa66806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:42:52.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.617+0000 7fa67cff9700 1 -- 172.21.15.32:0/3132779963 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa67008d380 con 0x7fa680101d70 2024-04-06T14:42:52.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.617+0000 7fa67f7fe700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa66806cc30 0x7fa66806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:42:52.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.617+0000 7fa67f7fe700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa66806cc30 0x7fa66806f0e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa6800fa600 tx=0x7fa67400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:42:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.621+0000 7fa67cff9700 1 -- 172.21.15.32:0/3132779963 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa67005ae50 con 0x7fa680101d70 2024-04-06T14:42:52.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:52 smithi086 ceph-mon[110975]: from='client.35262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:52.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:52 smithi086 ceph-mon[110975]: from='client.35266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:52.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:52 smithi086 ceph-mon[110975]: from='client.35270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:52.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:52 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2556217608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:42:52.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:52 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2482589789' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:42:52.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:52 smithi032 ceph-mon[147414]: from='client.35262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:52.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:52 smithi032 ceph-mon[147414]: from='client.35266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:52.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:52 smithi032 ceph-mon[147414]: from='client.35270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:52.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:52 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2556217608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:42:52.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:52 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2482589789' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:42:52.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.908+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa68004bed0 con 0x7fa680101d70 2024-04-06T14:42:52.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.909+0000 7fa67cff9700 1 -- 172.21.15.32:0/3132779963 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7fa67005a9e0 con 0x7fa680101d70 2024-04-06T14:42:52.900 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:42:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:42:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:42:52.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.910+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa66806cc30 msgr2=0x7fa66806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.910+0000 7fa686353700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa66806cc30 0x7fa66806f0e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa6800fa600 tx=0x7fa67400b040 comp rx=0 tx=0).stop 2024-04-06T14:42:52.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.911+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 msgr2=0x7fa680113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:42:52.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.911+0000 7fa686353700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 0x7fa680113980 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa670009390 tx=0x7fa670009e30 comp rx=0 tx=0).stop 2024-04-06T14:42:52.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.911+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 shutdown_connections 2024-04-06T14:42:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.911+0000 7fa686353700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa66806cc30 0x7fa66806f0e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.911+0000 7fa686353700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680101d70 0x7fa680113980 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.911+0000 7fa686353700 1 --2- 172.21.15.32:0/3132779963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa680102710 0x7fa680113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:42:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.911+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 >> 172.21.15.32:0/3132779963 conn(0x7fa680072bc0 msgr2=0x7fa680065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:42:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.911+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 shutdown_connections 2024-04-06T14:42:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:42:52.911+0000 7fa686353700 1 -- 172.21.15.32:0/3132779963 wait complete. 2024-04-06T14:42:53.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:53 smithi086 ceph-mon[110975]: pgmap v683: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:53.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:53 smithi086 ceph-mon[110975]: from='client.35282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:53.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:42:53.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:53 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3132779963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:42:53.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:53 smithi032 ceph-mon[147414]: pgmap v683: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:53.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:53 smithi032 ceph-mon[147414]: from='client.35282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:42:53.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:42:53.722 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:53 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3132779963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:42:55.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:55 smithi086 ceph-mon[110975]: pgmap v684: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:55.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:55 smithi032 ceph-mon[147414]: pgmap v684: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:57.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:57 smithi086 ceph-mon[110975]: pgmap v685: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:57.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:57 smithi032 ceph-mon[147414]: pgmap v685: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:59.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:42:59 smithi086 ceph-mon[110975]: pgmap v686: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:42:59.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:42:59 smithi032 ceph-mon[147414]: pgmap v686: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:01.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:01 smithi086 ceph-mon[110975]: pgmap v687: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:01.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:01 smithi032 ceph-mon[147414]: pgmap v687: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:03.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:03 smithi086 ceph-mon[110975]: pgmap v688: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:03.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:03 smithi032 ceph-mon[147414]: pgmap v688: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:05.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:05 smithi086 ceph-mon[110975]: pgmap v689: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:05.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:05 smithi032 ceph-mon[147414]: pgmap v689: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:07.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:07 smithi086 ceph-mon[110975]: pgmap v690: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:07.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:07 smithi032 ceph-mon[147414]: pgmap v690: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:08.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:43:08.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:43:09.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:09 smithi086 ceph-mon[110975]: pgmap v691: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:09.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:09 smithi032 ceph-mon[147414]: pgmap v691: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:11.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:11 smithi086 ceph-mon[110975]: pgmap v692: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:11.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:11 smithi032 ceph-mon[147414]: pgmap v692: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:13.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:13 smithi086 ceph-mon[110975]: pgmap v693: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:13.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:13 smithi032 ceph-mon[147414]: pgmap v693: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:15.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:15 smithi086 ceph-mon[110975]: pgmap v694: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:15.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:15 smithi032 ceph-mon[147414]: pgmap v694: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:17.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:17 smithi086 ceph-mon[110975]: pgmap v695: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:17.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:17 smithi032 ceph-mon[147414]: pgmap v695: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:19.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:19 smithi086 ceph-mon[110975]: pgmap v696: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:19.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:19 smithi032 ceph-mon[147414]: pgmap v696: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:21.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:21 smithi086 ceph-mon[110975]: pgmap v697: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:21.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:21 smithi032 ceph-mon[147414]: pgmap v697: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.028+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3920726903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f81026b0 msgr2=0x7fa3f80fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.028+0000 7fa3fe8d5700 1 --2- 172.21.15.32:0/3920726903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f81026b0 0x7fa3f80fbd20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec004940 tx=0x7fa3ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.028+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3920726903 shutdown_connections 2024-04-06T14:43:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.028+0000 7fa3fe8d5700 1 --2- 172.21.15.32:0/3920726903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f81026b0 0x7fa3f80fbd20 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.028+0000 7fa3fe8d5700 1 --2- 172.21.15.32:0/3920726903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8101d10 0x7fa3f81020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.028+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3920726903 >> 172.21.15.32:0/3920726903 conn(0x7fa3f80f7880 msgr2=0x7fa3f80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.028+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3920726903 shutdown_connections 2024-04-06T14:43:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.028+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3920726903 wait complete. 2024-04-06T14:43:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3fe8d5700 1 Processor -- start 2024-04-06T14:43:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3fe8d5700 1 -- start start 2024-04-06T14:43:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3fe8d5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f8101d10 0x7fa3f8112ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3fe8d5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f81026b0 0x7fa3f8113220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3fe8d5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f8113870 con 0x7fa3f8101d10 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3fe8d5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f810cc40 con 0x7fa3f81026b0 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3f7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f8101d10 0x7fa3f8112ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3f7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f8101d10 0x7fa3f8112ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36370/0 (socket says 172.21.15.32:36370) 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3f7fff700 1 -- 172.21.15.32:0/3105196834 learned_addr learned my addr 172.21.15.32:0/3105196834 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.029+0000 7fa3f77fe700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f81026b0 0x7fa3f8113220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.030+0000 7fa3f7fff700 1 -- 172.21.15.32:0/3105196834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f81026b0 msgr2=0x7fa3f8113220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.030+0000 7fa3f7fff700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f81026b0 0x7fa3f8113220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.030+0000 7fa3f7fff700 1 -- 172.21.15.32:0/3105196834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec005040 con 0x7fa3f8101d10 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.030+0000 7fa3f7fff700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f8101d10 0x7fa3f8112ce0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800c830 tx=0x7fa3e8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.030+0000 7fa3f4ff9700 1 -- 172.21.15.32:0/3105196834 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3e80106f0 con 0x7fa3f8101d10 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.030+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f810cf20 con 0x7fa3f8101d10 2024-04-06T14:43:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.030+0000 7fa3f4ff9700 1 -- 172.21.15.32:0/3105196834 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3e8010d00 con 0x7fa3f8101d10 2024-04-06T14:43:23.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.030+0000 7fa3f4ff9700 1 -- 172.21.15.32:0/3105196834 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3e8012e90 con 0x7fa3f8101d10 2024-04-06T14:43:23.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.030+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f810d4e0 con 0x7fa3f8101d10 2024-04-06T14:43:23.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.031+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f80fd460 con 0x7fa3f8101d10 2024-04-06T14:43:23.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.032+0000 7fa3f4ff9700 1 -- 172.21.15.32:0/3105196834 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa3e8010850 con 0x7fa3f8101d10 2024-04-06T14:43:23.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.032+0000 7fa3f4ff9700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3e006cc30 0x7fa3e006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:23.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.032+0000 7fa3f4ff9700 1 -- 172.21.15.32:0/3105196834 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa3e808c010 con 0x7fa3f8101d10 2024-04-06T14:43:23.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.032+0000 7fa3f77fe700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3e006cc30 0x7fa3e006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:23.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.033+0000 7fa3f77fe700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3e006cc30 0x7fa3e006f0e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa3f810e580 tx=0x7fa3ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:23.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.037+0000 7fa3f4ff9700 1 -- 172.21.15.32:0/3105196834 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa3e8059ae0 con 0x7fa3f8101d10 2024-04-06T14:43:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.257+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3f804bed0 con 0x7fa3e006cc30 2024-04-06T14:43:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.258+0000 7fa3f4ff9700 1 -- 172.21.15.32:0/3105196834 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa3f804bed0 con 0x7fa3e006cc30 2024-04-06T14:43:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3e006cc30 msgr2=0x7fa3e006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3e006cc30 0x7fa3e006f0e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa3f810e580 tx=0x7fa3ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f8101d10 msgr2=0x7fa3f8112ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f8101d10 0x7fa3f8112ce0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800c830 tx=0x7fa3e8004470 comp rx=0 tx=0).stop 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 shutdown_connections 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3e006cc30 0x7fa3e006f0e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3f8101d10 0x7fa3f8112ce0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 --2- 172.21.15.32:0/3105196834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f81026b0 0x7fa3f8113220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 >> 172.21.15.32:0/3105196834 conn(0x7fa3f80f7880 msgr2=0x7fa3f80f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 shutdown_connections 2024-04-06T14:43:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.260+0000 7fa3fe8d5700 1 -- 172.21.15.32:0/3105196834 wait complete. 2024-04-06T14:43:23.262 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:43:23.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.365+0000 7f54fa0e4700 1 -- 172.21.15.32:0/2643527818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fde10 msgr2=0x7f54f40fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.365+0000 7f54fa0e4700 1 --2- 172.21.15.32:0/2643527818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fde10 0x7f54f40fe1e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f54dc004940 tx=0x7f54dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.365+0000 7f54fa0e4700 1 -- 172.21.15.32:0/2643527818 shutdown_connections 2024-04-06T14:43:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.365+0000 7f54fa0e4700 1 --2- 172.21.15.32:0/2643527818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f40fe7b0 0x7f54f4106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.365+0000 7f54fa0e4700 1 --2- 172.21.15.32:0/2643527818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fde10 0x7f54f40fe1e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.365+0000 7f54fa0e4700 1 -- 172.21.15.32:0/2643527818 >> 172.21.15.32:0/2643527818 conn(0x7f54f40f7880 msgr2=0x7f54f40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.365+0000 7f54fa0e4700 1 -- 172.21.15.32:0/2643527818 shutdown_connections 2024-04-06T14:43:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.365+0000 7f54fa0e4700 1 -- 172.21.15.32:0/2643527818 wait complete. 2024-04-06T14:43:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.365+0000 7f54fa0e4700 1 Processor -- start 2024-04-06T14:43:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54fa0e4700 1 -- start start 2024-04-06T14:43:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54fa0e4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f40fde10 0x7f54f4113d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54fa0e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fe7b0 0x7f54f4114290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54fa0e4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f41148e0 con 0x7f54f40fe7b0 2024-04-06T14:43:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54fa0e4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f410dcb0 con 0x7f54f40fde10 2024-04-06T14:43:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54f2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fe7b0 0x7f54f4114290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54f2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fe7b0 0x7f54f4114290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36388/0 (socket says 172.21.15.32:36388) 2024-04-06T14:43:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54f2ffd700 1 -- 172.21.15.32:0/204805382 learned_addr learned my addr 172.21.15.32:0/204805382 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54f37fe700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f40fde10 0x7f54f4113d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54f2ffd700 1 -- 172.21.15.32:0/204805382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f40fde10 msgr2=0x7f54f4113d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54f2ffd700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f40fde10 0x7f54f4113d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54f2ffd700 1 -- 172.21.15.32:0/204805382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc005040 con 0x7f54f40fe7b0 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54f37fe700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f40fde10 0x7f54f4113d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54f2ffd700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fe7b0 0x7f54f4114290 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f54e400eb40 tx=0x7f54e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.366+0000 7f54ebfff700 1 -- 172.21.15.32:0/204805382 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e4007cc0 con 0x7f54f40fe7b0 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.367+0000 7f54ebfff700 1 -- 172.21.15.32:0/204805382 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54e400f040 con 0x7f54f40fe7b0 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.367+0000 7f54ebfff700 1 -- 172.21.15.32:0/204805382 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e4009570 con 0x7f54f40fe7b0 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.367+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f410df30 con 0x7f54f40fe7b0 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.367+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f410e470 con 0x7f54f40fe7b0 2024-04-06T14:43:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.367+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f4066770 con 0x7f54f40fe7b0 2024-04-06T14:43:23.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.368+0000 7f54ebfff700 1 -- 172.21.15.32:0/204805382 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f54e40096d0 con 0x7f54f40fe7b0 2024-04-06T14:43:23.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.369+0000 7f54ebfff700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54e006cc80 0x7f54e006f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:23.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.369+0000 7f54ebfff700 1 -- 172.21.15.32:0/204805382 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f54e408bdb0 con 0x7f54f40fe7b0 2024-04-06T14:43:23.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.369+0000 7f54f37fe700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54e006cc80 0x7f54e006f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:23.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.369+0000 7f54f37fe700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54e006cc80 0x7f54e006f130 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f54dc000c00 tx=0x7f54dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:23.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.373+0000 7f54ebfff700 1 -- 172.21.15.32:0/204805382 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f54e4059830 con 0x7f54f40fe7b0 2024-04-06T14:43:23.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.596+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f54f4109dd0 con 0x7f54e006cc80 2024-04-06T14:43:23.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.597+0000 7f54ebfff700 1 -- 172.21.15.32:0/204805382 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f54f4109dd0 con 0x7f54e006cc80 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.599+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54e006cc80 msgr2=0x7f54e006f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.599+0000 7f54fa0e4700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54e006cc80 0x7f54e006f130 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f54dc000c00 tx=0x7f54dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.599+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fe7b0 msgr2=0x7f54f4114290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.599+0000 7f54fa0e4700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fe7b0 0x7f54f4114290 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f54e400eb40 tx=0x7f54e400b040 comp rx=0 tx=0).stop 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.599+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 shutdown_connections 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.599+0000 7f54fa0e4700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f54e006cc80 0x7f54e006f130 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.599+0000 7f54fa0e4700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f40fde10 0x7f54f4113d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.599+0000 7f54fa0e4700 1 --2- 172.21.15.32:0/204805382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54f40fe7b0 0x7f54f4114290 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.599+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 >> 172.21.15.32:0/204805382 conn(0x7f54f40f7880 msgr2=0x7f54f40f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.600+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 shutdown_connections 2024-04-06T14:43:23.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.600+0000 7f54fa0e4700 1 -- 172.21.15.32:0/204805382 wait complete. 2024-04-06T14:43:23.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:23 smithi086 ceph-mon[110975]: pgmap v698: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:23.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:43:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.703+0000 7f5b39e99700 1 -- 172.21.15.32:0/2942583896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b34102e70 msgr2=0x7f5b34103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.703+0000 7f5b39e99700 1 --2- 172.21.15.32:0/2942583896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b34102e70 0x7f5b34103240 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5b24004940 tx=0x7f5b24009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.703+0000 7f5b39e99700 1 -- 172.21.15.32:0/2942583896 shutdown_connections 2024-04-06T14:43:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.703+0000 7f5b39e99700 1 --2- 172.21.15.32:0/2942583896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b340fb4a0 0x7f5b340fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.703+0000 7f5b39e99700 1 --2- 172.21.15.32:0/2942583896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b34102e70 0x7f5b34103240 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.703+0000 7f5b39e99700 1 -- 172.21.15.32:0/2942583896 >> 172.21.15.32:0/2942583896 conn(0x7f5b340714c0 msgr2=0x7f5b340718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.703+0000 7f5b39e99700 1 -- 172.21.15.32:0/2942583896 shutdown_connections 2024-04-06T14:43:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.703+0000 7f5b39e99700 1 -- 172.21.15.32:0/2942583896 wait complete. 2024-04-06T14:43:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.704+0000 7f5b39e99700 1 Processor -- start 2024-04-06T14:43:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.704+0000 7f5b39e99700 1 -- start start 2024-04-06T14:43:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.704+0000 7f5b39e99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b340fb4a0 0x7f5b34113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.704+0000 7f5b39e99700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b34102e70 0x7f5b34114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b39e99700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b34114730 con 0x7f5b340fb4a0 2024-04-06T14:43:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b39e99700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3410da90 con 0x7f5b34102e70 2024-04-06T14:43:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b337fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b340fb4a0 0x7f5b34113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b337fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b340fb4a0 0x7f5b34113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36408/0 (socket says 172.21.15.32:36408) 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b337fe700 1 -- 172.21.15.32:0/1194387845 learned_addr learned my addr 172.21.15.32:0/1194387845 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b32ffd700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b34102e70 0x7f5b34114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b337fe700 1 -- 172.21.15.32:0/1194387845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b34102e70 msgr2=0x7f5b34114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b337fe700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b34102e70 0x7f5b34114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b337fe700 1 -- 172.21.15.32:0/1194387845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24005040 con 0x7f5b340fb4a0 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b337fe700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b340fb4a0 0x7f5b34113b10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009180 tx=0x7f5b24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b1bfff700 1 -- 172.21.15.32:0/1194387845 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b2400e040 con 0x7f5b340fb4a0 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b1bfff700 1 -- 172.21.15.32:0/1194387845 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b24004290 con 0x7f5b340fb4a0 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3410dd10 con 0x7f5b340fb4a0 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b1bfff700 1 -- 172.21.15.32:0/1194387845 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b2401c3f0 con 0x7f5b340fb4a0 2024-04-06T14:43:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.705+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3410e250 con 0x7f5b340fb4a0 2024-04-06T14:43:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.706+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b34100170 con 0x7f5b340fb4a0 2024-04-06T14:43:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.707+0000 7f5b1bfff700 1 -- 172.21.15.32:0/1194387845 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5b24013790 con 0x7f5b340fb4a0 2024-04-06T14:43:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.707+0000 7f5b1bfff700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b1c06cc20 0x7f5b1c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.707+0000 7f5b1bfff700 1 -- 172.21.15.32:0/1194387845 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5b2408d300 con 0x7f5b340fb4a0 2024-04-06T14:43:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.707+0000 7f5b32ffd700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b1c06cc20 0x7f5b1c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.708+0000 7f5b32ffd700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b1c06cc20 0x7f5b1c06f0d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5b28006fd0 tx=0x7f5b28008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:23.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.712+0000 7f5b1bfff700 1 -- 172.21.15.32:0/1194387845 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5b2405add0 con 0x7f5b340fb4a0 2024-04-06T14:43:23.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:23 smithi032 ceph-mon[147414]: pgmap v698: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:23.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:43:23.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.928+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b3410f1d0 con 0x7f5b1c06cc20 2024-04-06T14:43:23.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.938+0000 7f5b1bfff700 1 -- 172.21.15.32:0/1194387845 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f5b3410f1d0 con 0x7f5b1c06cc20 2024-04-06T14:43:23.929 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:43:23.929 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (23m) 102s ago 34m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:43:23.929 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (21m) 102s ago 35m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:43:23.929 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (21m) 104s ago 34m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (23m) 102s ago 34m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (17m) 102s ago 29m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (17m) 102s ago 29m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (16m) 104s ago 29m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (108s) 104s ago 29m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (25m) 102s ago 36m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (24m) 104s ago 34m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (22m) 102s ago 36m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (21m) 104s ago 34m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (24m) 102s ago 35m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (24m) 104s ago 34m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (20m) 102s ago 33m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (20m) 102s ago 32m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (19m) 102s ago 32m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:43:23.930 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (18m) 104s ago 31m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:43:23.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (18m) 104s ago 31m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:43:23.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (17m) 104s ago 30m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:43:23.931 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (24m) 102s ago 34m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:43:23.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b1c06cc20 msgr2=0x7f5b1c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b1c06cc20 0x7f5b1c06f0d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5b28006fd0 tx=0x7f5b28008040 comp rx=0 tx=0).stop 2024-04-06T14:43:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b340fb4a0 msgr2=0x7f5b34113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b340fb4a0 0x7f5b34113b10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009180 tx=0x7f5b24009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 shutdown_connections 2024-04-06T14:43:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b1c06cc20 0x7f5b1c06f0d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b340fb4a0 0x7f5b34113b10 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 --2- 172.21.15.32:0/1194387845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b34102e70 0x7f5b34114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 >> 172.21.15.32:0/1194387845 conn(0x7f5b340714c0 msgr2=0x7f5b340fa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 shutdown_connections 2024-04-06T14:43:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:23.940+0000 7f5b39e99700 1 -- 172.21.15.32:0/1194387845 wait complete. 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 -- 172.21.15.32:0/4119158531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fe830 msgr2=0x7f344c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 --2- 172.21.15.32:0/4119158531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fe830 0x7f344c106f00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3440004940 tx=0x7f3440009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 -- 172.21.15.32:0/4119158531 shutdown_connections 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 --2- 172.21.15.32:0/4119158531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fe830 0x7f344c106f00 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 --2- 172.21.15.32:0/4119158531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f344c0fde90 0x7f344c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 -- 172.21.15.32:0/4119158531 >> 172.21.15.32:0/4119158531 conn(0x7f344c072bc0 msgr2=0x7f344c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 -- 172.21.15.32:0/4119158531 shutdown_connections 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 -- 172.21.15.32:0/4119158531 wait complete. 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 Processor -- start 2024-04-06T14:43:24.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.042+0000 7f3454182700 1 -- start start 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3454182700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fde90 0x7f344c113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3454182700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f344c0fe830 0x7f344c114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3454182700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c114670 con 0x7f344c0fde90 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3454182700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c10da60 con 0x7f344c0fe830 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3451f1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fde90 0x7f344c113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3451f1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fde90 0x7f344c113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36438/0 (socket says 172.21.15.32:36438) 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3451f1e700 1 -- 172.21.15.32:0/2026864187 learned_addr learned my addr 172.21.15.32:0/2026864187 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f345171d700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f344c0fe830 0x7f344c114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3451f1e700 1 -- 172.21.15.32:0/2026864187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f344c0fe830 msgr2=0x7f344c114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3451f1e700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f344c0fe830 0x7f344c114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3451f1e700 1 -- 172.21.15.32:0/2026864187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3440005040 con 0x7f344c0fde90 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3451f1e700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fde90 0x7f344c113ae0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3448002a50 tx=0x7f34480046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f343e7fc700 1 -- 172.21.15.32:0/2026864187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3448007cc0 con 0x7f344c0fde90 2024-04-06T14:43:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f343e7fc700 1 -- 172.21.15.32:0/2026864187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f344800f040 con 0x7f344c0fde90 2024-04-06T14:43:24.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344c10dd40 con 0x7f344c0fde90 2024-04-06T14:43:24.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f343e7fc700 1 -- 172.21.15.32:0/2026864187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3448009540 con 0x7f344c0fde90 2024-04-06T14:43:24.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.043+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344c10e250 con 0x7f344c0fde90 2024-04-06T14:43:24.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.044+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344c065e30 con 0x7f344c0fde90 2024-04-06T14:43:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.045+0000 7f343e7fc700 1 -- 172.21.15.32:0/2026864187 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f34480096a0 con 0x7f344c0fde90 2024-04-06T14:43:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.045+0000 7f343e7fc700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f343806cc30 0x7f343806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.045+0000 7f345171d700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f343806cc30 0x7f343806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.045+0000 7f343e7fc700 1 -- 172.21.15.32:0/2026864187 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f344808bd70 con 0x7f344c0fde90 2024-04-06T14:43:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.046+0000 7f345171d700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f343806cc30 0x7f343806f0e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f344c10ed40 tx=0x7f3440009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.050+0000 7f343e7fc700 1 -- 172.21.15.32:0/2026864187 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3448059840 con 0x7f344c0fde90 2024-04-06T14:43:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.338+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f344c109dd0 con 0x7f344c0fde90 2024-04-06T14:43:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.338+0000 7f343e7fc700 1 -- 172.21.15.32:0/2026864187 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f34480593d0 con 0x7f344c0fde90 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:43:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:43:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:43:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:43:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:43:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:43:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:43:24.331 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f343806cc30 msgr2=0x7f343806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f343806cc30 0x7f343806f0e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f344c10ed40 tx=0x7f3440009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fde90 msgr2=0x7f344c113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fde90 0x7f344c113ae0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3448002a50 tx=0x7f34480046b0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 shutdown_connections 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f343806cc30 0x7f343806f0e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f344c0fde90 0x7f344c113ae0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 --2- 172.21.15.32:0/2026864187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f344c0fe830 0x7f344c114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 >> 172.21.15.32:0/2026864187 conn(0x7f344c072bc0 msgr2=0x7f344c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 shutdown_connections 2024-04-06T14:43:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.340+0000 7f3454182700 1 -- 172.21.15.32:0/2026864187 wait complete. 2024-04-06T14:43:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.443+0000 7fb76245c700 1 -- 172.21.15.32:0/1935268274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c1037a0 msgr2=0x7fb75c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.443+0000 7fb76245c700 1 --2- 172.21.15.32:0/1935268274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c1037a0 0x7fb75c0fce10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb750004940 tx=0x7fb750009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.443+0000 7fb76245c700 1 -- 172.21.15.32:0/1935268274 shutdown_connections 2024-04-06T14:43:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.443+0000 7fb76245c700 1 --2- 172.21.15.32:0/1935268274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c1037a0 0x7fb75c0fce10 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.443+0000 7fb76245c700 1 --2- 172.21.15.32:0/1935268274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75c102e00 0x7fb75c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.443+0000 7fb76245c700 1 -- 172.21.15.32:0/1935268274 >> 172.21.15.32:0/1935268274 conn(0x7fb75c0f89b0 msgr2=0x7fb75c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.443+0000 7fb76245c700 1 -- 172.21.15.32:0/1935268274 shutdown_connections 2024-04-06T14:43:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.443+0000 7fb76245c700 1 -- 172.21.15.32:0/1935268274 wait complete. 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb76245c700 1 Processor -- start 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb76245c700 1 -- start start 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb76245c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c102e00 0x7fb75c113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb76245c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75c1037a0 0x7fb75c114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb76245c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c114650 con 0x7fb75c102e00 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb76245c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c10da40 con 0x7fb75c1037a0 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb75bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c102e00 0x7fb75c113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb75bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c102e00 0x7fb75c113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39544/0 (socket says 172.21.15.32:39544) 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb75bfff700 1 -- 172.21.15.32:0/902713203 learned_addr learned my addr 172.21.15.32:0/902713203 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb75b7fe700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75c1037a0 0x7fb75c114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.444+0000 7fb75bfff700 1 -- 172.21.15.32:0/902713203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75c1037a0 msgr2=0x7fb75c114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.445+0000 7fb75bfff700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75c1037a0 0x7fb75c114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.445+0000 7fb75bfff700 1 -- 172.21.15.32:0/902713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb750005040 con 0x7fb75c102e00 2024-04-06T14:43:24.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.445+0000 7fb75bfff700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c102e00 0x7fb75c113ac0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb74c00c8c0 tx=0x7fb74c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:24.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.445+0000 7fb758ff9700 1 -- 172.21.15.32:0/902713203 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb74c00a980 con 0x7fb75c102e00 2024-04-06T14:43:24.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.445+0000 7fb758ff9700 1 -- 172.21.15.32:0/902713203 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb74c00ce60 con 0x7fb75c102e00 2024-04-06T14:43:24.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.445+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c10dd20 con 0x7fb75c102e00 2024-04-06T14:43:24.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.445+0000 7fb758ff9700 1 -- 172.21.15.32:0/902713203 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb74c0113a0 con 0x7fb75c102e00 2024-04-06T14:43:24.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.445+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c10e2e0 con 0x7fb75c102e00 2024-04-06T14:43:24.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.446+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75c0fe550 con 0x7fb75c102e00 2024-04-06T14:43:24.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.446+0000 7fb758ff9700 1 -- 172.21.15.32:0/902713203 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb74c012400 con 0x7fb75c102e00 2024-04-06T14:43:24.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.447+0000 7fb758ff9700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb74406cc20 0x7fb74406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:24.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.447+0000 7fb758ff9700 1 -- 172.21.15.32:0/902713203 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb74c08c160 con 0x7fb75c102e00 2024-04-06T14:43:24.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.447+0000 7fb75b7fe700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb74406cc20 0x7fb74406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:24.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.447+0000 7fb75b7fe700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb74406cc20 0x7fb74406f0d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb75000b3b0 tx=0x7fb750009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:24.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.451+0000 7fb758ff9700 1 -- 172.21.15.32:0/902713203 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb74c059c30 con 0x7fb75c102e00 2024-04-06T14:43:24.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:24 smithi086 ceph-mon[110975]: from='client.35290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:24.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:24 smithi086 ceph-mon[110975]: from='client.35294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:24.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.698+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb75c109da0 con 0x7fb75c102e00 2024-04-06T14:43:24.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.698+0000 7fb758ff9700 1 -- 172.21.15.32:0/902713203 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fb74c0597c0 con 0x7fb75c102e00 2024-04-06T14:43:24.690 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:43:24.690 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:43:24.690 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:43:24.690 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:43:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:43:24.690 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:43:24.690 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:43:24.690 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:43:24.691 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:43:24.692 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:43:24.693 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:43:24.693 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:43:24.693 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:43:24.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb74406cc20 msgr2=0x7fb74406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb74406cc20 0x7fb74406f0d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb75000b3b0 tx=0x7fb750009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c102e00 msgr2=0x7fb75c113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c102e00 0x7fb75c113ac0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb74c00c8c0 tx=0x7fb74c0044c0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 shutdown_connections 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb74406cc20 0x7fb74406f0d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75c102e00 0x7fb75c113ac0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 --2- 172.21.15.32:0/902713203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75c1037a0 0x7fb75c114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 >> 172.21.15.32:0/902713203 conn(0x7fb75c0f89b0 msgr2=0x7fb75c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 shutdown_connections 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.700+0000 7fb76245c700 1 -- 172.21.15.32:0/902713203 wait complete. 2024-04-06T14:43:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:43:24.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:24 smithi032 ceph-mon[147414]: from='client.35290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:24.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:24 smithi032 ceph-mon[147414]: from='client.35294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:24.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.804+0000 7fad28fa6700 1 -- 172.21.15.32:0/568193309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad24102e20 msgr2=0x7fad241031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.804+0000 7fad28fa6700 1 --2- 172.21.15.32:0/568193309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad24102e20 0x7fad241031f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fad14004940 tx=0x7fad14009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.804+0000 7fad28fa6700 1 -- 172.21.15.32:0/568193309 shutdown_connections 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.804+0000 7fad28fa6700 1 --2- 172.21.15.32:0/568193309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad241037c0 0x7fad240fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.804+0000 7fad28fa6700 1 --2- 172.21.15.32:0/568193309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad24102e20 0x7fad241031f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.804+0000 7fad28fa6700 1 -- 172.21.15.32:0/568193309 >> 172.21.15.32:0/568193309 conn(0x7fad240f89d0 msgr2=0x7fad240fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.804+0000 7fad28fa6700 1 -- 172.21.15.32:0/568193309 shutdown_connections 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.804+0000 7fad28fa6700 1 -- 172.21.15.32:0/568193309 wait complete. 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad28fa6700 1 Processor -- start 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad28fa6700 1 -- start start 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad28fa6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad24102e20 0x7fad24113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad28fa6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad241037c0 0x7fad24114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad28fa6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad24114670 con 0x7fad241037c0 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad28fa6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2410da60 con 0x7fad24102e20 2024-04-06T14:43:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad21d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad241037c0 0x7fad24114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad21d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad241037c0 0x7fad24114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39558/0 (socket says 172.21.15.32:39558) 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad21d9b700 1 -- 172.21.15.32:0/200290879 learned_addr learned my addr 172.21.15.32:0/200290879 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.805+0000 7fad2259c700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad24102e20 0x7fad24113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad21d9b700 1 -- 172.21.15.32:0/200290879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad24102e20 msgr2=0x7fad24113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad21d9b700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad24102e20 0x7fad24113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad21d9b700 1 -- 172.21.15.32:0/200290879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad14005040 con 0x7fad241037c0 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad2259c700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad24102e20 0x7fad24113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad21d9b700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad241037c0 0x7fad24114020 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00c890 tx=0x7fad0c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad1affd700 1 -- 172.21.15.32:0/200290879 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad0c00a7d0 con 0x7fad241037c0 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad1affd700 1 -- 172.21.15.32:0/200290879 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad0c00ade0 con 0x7fad241037c0 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2410dd40 con 0x7fad241037c0 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad1affd700 1 -- 172.21.15.32:0/200290879 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad0c0113a0 con 0x7fad241037c0 2024-04-06T14:43:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.806+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2410e300 con 0x7fad241037c0 2024-04-06T14:43:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.807+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad240fe570 con 0x7fad241037c0 2024-04-06T14:43:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.807+0000 7fad1affd700 1 -- 172.21.15.32:0/200290879 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fad0c012400 con 0x7fad241037c0 2024-04-06T14:43:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.808+0000 7fad1affd700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fad1006cc30 0x7fad1006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.808+0000 7fad1affd700 1 -- 172.21.15.32:0/200290879 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fad0c08c050 con 0x7fad241037c0 2024-04-06T14:43:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.808+0000 7fad2259c700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fad1006cc30 0x7fad1006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.808+0000 7fad2259c700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fad1006cc30 0x7fad1006f0e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fad14000c00 tx=0x7fad14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:24.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:24.812+0000 7fad1affd700 1 -- 172.21.15.32:0/200290879 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fad0c059be0 con 0x7fad241037c0 2024-04-06T14:43:25.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.034+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fad24109de0 con 0x7fad1006cc30 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.036+0000 7fad1affd700 1 -- 172.21.15.32:0/200290879 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fad24109de0 con 0x7fad1006cc30 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:43:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:43:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:43:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:43:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:43:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:43:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:43:25.028 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:43:25.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fad1006cc30 msgr2=0x7fad1006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:25.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fad1006cc30 0x7fad1006f0e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fad14000c00 tx=0x7fad14009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:25.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad241037c0 msgr2=0x7fad24114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:25.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad241037c0 0x7fad24114020 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00c890 tx=0x7fad0c0044c0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 shutdown_connections 2024-04-06T14:43:25.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fad1006cc30 0x7fad1006f0e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad24102e20 0x7fad24113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 --2- 172.21.15.32:0/200290879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad241037c0 0x7fad24114020 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 >> 172.21.15.32:0/200290879 conn(0x7fad240f89d0 msgr2=0x7fad240f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:25.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 shutdown_connections 2024-04-06T14:43:25.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.038+0000 7fad28fa6700 1 -- 172.21.15.32:0/200290879 wait complete. 2024-04-06T14:43:25.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.141+0000 7f0275ad7700 1 -- 172.21.15.32:0/3867961000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02701037c0 msgr2=0x7f02700fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:25.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.141+0000 7f0275ad7700 1 --2- 172.21.15.32:0/3867961000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02701037c0 0x7f02700fce30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0260004940 tx=0x7f0260009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:25.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.141+0000 7f0275ad7700 1 -- 172.21.15.32:0/3867961000 shutdown_connections 2024-04-06T14:43:25.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.141+0000 7f0275ad7700 1 --2- 172.21.15.32:0/3867961000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02701037c0 0x7f02700fce30 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.141+0000 7f0275ad7700 1 --2- 172.21.15.32:0/3867961000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0270102e20 0x7f02701031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.141+0000 7f0275ad7700 1 -- 172.21.15.32:0/3867961000 >> 172.21.15.32:0/3867961000 conn(0x7f02700f89d0 msgr2=0x7f02700fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:25.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.141+0000 7f0275ad7700 1 -- 172.21.15.32:0/3867961000 shutdown_connections 2024-04-06T14:43:25.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.141+0000 7f0275ad7700 1 -- 172.21.15.32:0/3867961000 wait complete. 2024-04-06T14:43:25.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.141+0000 7f0275ad7700 1 Processor -- start 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f0275ad7700 1 -- start start 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f0275ad7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0270102e20 0x7f0270113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f0275ad7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02701037c0 0x7f0270114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f0275ad7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0270114660 con 0x7f0270102e20 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f0275ad7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027010da50 con 0x7f02701037c0 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f026f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0270102e20 0x7f0270113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f026f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0270102e20 0x7f0270113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39568/0 (socket says 172.21.15.32:39568) 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f026f7fe700 1 -- 172.21.15.32:0/1854858241 learned_addr learned my addr 172.21.15.32:0/1854858241 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f026effd700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02701037c0 0x7f0270114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f026f7fe700 1 -- 172.21.15.32:0/1854858241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02701037c0 msgr2=0x7f0270114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f026f7fe700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02701037c0 0x7f0270114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f026f7fe700 1 -- 172.21.15.32:0/1854858241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0260005040 con 0x7f0270102e20 2024-04-06T14:43:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f026f7fe700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0270102e20 0x7f0270113ad0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0264002a50 tx=0x7f02640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.142+0000 7f0257fff700 1 -- 172.21.15.32:0/1854858241 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0264007cc0 con 0x7f0270102e20 2024-04-06T14:43:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.143+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027010dd30 con 0x7f0270102e20 2024-04-06T14:43:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.143+0000 7f0257fff700 1 -- 172.21.15.32:0/1854858241 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f026400f040 con 0x7f0270102e20 2024-04-06T14:43:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.143+0000 7f0257fff700 1 -- 172.21.15.32:0/1854858241 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f026400d9c0 con 0x7f0270102e20 2024-04-06T14:43:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.143+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027010e2c0 con 0x7f0270102e20 2024-04-06T14:43:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.143+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02700fe570 con 0x7f0270102e20 2024-04-06T14:43:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.144+0000 7f0257fff700 1 -- 172.21.15.32:0/1854858241 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f026400db20 con 0x7f0270102e20 2024-04-06T14:43:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.145+0000 7f0257fff700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f025806cc20 0x7f025806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.145+0000 7f0257fff700 1 -- 172.21.15.32:0/1854858241 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f026408c460 con 0x7f0270102e20 2024-04-06T14:43:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.145+0000 7f026effd700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f025806cc20 0x7f025806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.145+0000 7f026effd700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f025806cc20 0x7f025806f0d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0260004890 tx=0x7f0260009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:25.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.149+0000 7f0257fff700 1 -- 172.21.15.32:0/1854858241 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0264059f60 con 0x7f0270102e20 2024-04-06T14:43:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.434+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0270109df0 con 0x7f0270102e20 2024-04-06T14:43:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.435+0000 7f0257fff700 1 -- 172.21.15.32:0/1854858241 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f0264059af0 con 0x7f0270102e20 2024-04-06T14:43:25.426 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:43:25.427 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:43:25.427 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:43:25.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:43:25.427 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:43:25.427 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:43:25.427 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:43:25.427 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:43:25.427 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:43:25.428 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:43:25.428 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:43:25.428 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:43:25.428 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:43:25.428 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:43:25.428 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:43:25.428 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:43:25.428 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:43:25.429 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:43:25.429 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:43:25.429 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:43:25.429 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:43:25.429 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:43:25.429 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:43:25.429 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:43:25.429 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:43:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f025806cc20 msgr2=0x7f025806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f025806cc20 0x7f025806f0d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0260004890 tx=0x7f0260009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0270102e20 msgr2=0x7f0270113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0270102e20 0x7f0270113ad0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0264002a50 tx=0x7f02640046b0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 shutdown_connections 2024-04-06T14:43:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f025806cc20 0x7f025806f0d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0270102e20 0x7f0270113ad0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 --2- 172.21.15.32:0/1854858241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02701037c0 0x7f0270114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 >> 172.21.15.32:0/1854858241 conn(0x7f02700f89d0 msgr2=0x7f02700f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 shutdown_connections 2024-04-06T14:43:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:25.437+0000 7f0275ad7700 1 -- 172.21.15.32:0/1854858241 wait complete. 2024-04-06T14:43:25.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:25 smithi086 ceph-mon[110975]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:25.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:25 smithi086 ceph-mon[110975]: pgmap v699: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:25.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:25 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2026864187' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:43:25.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:25 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/902713203' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:43:25.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:25 smithi032 ceph-mon[147414]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:25.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:25 smithi032 ceph-mon[147414]: pgmap v699: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:25.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:25 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2026864187' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:43:25.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:25 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/902713203' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:43:26.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:26 smithi086 ceph-mon[110975]: from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:26.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:26 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1854858241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:43:26.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:26 smithi032 ceph-mon[147414]: from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:26.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:26 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1854858241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:43:27.622 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:27 smithi086 ceph-mon[110975]: pgmap v700: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:27.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:27 smithi032 ceph-mon[147414]: pgmap v700: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:29.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:29 smithi086 ceph-mon[110975]: pgmap v701: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:29.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:29 smithi032 ceph-mon[147414]: pgmap v701: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:31.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:31 smithi086 ceph-mon[110975]: pgmap v702: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:31.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:31 smithi032 ceph-mon[147414]: pgmap v702: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:33.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:33 smithi086 ceph-mon[110975]: pgmap v703: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:33.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:33 smithi032 ceph-mon[147414]: pgmap v703: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:35.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:35 smithi086 ceph-mon[110975]: pgmap v704: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:35.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:35 smithi032 ceph-mon[147414]: pgmap v704: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:37.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:37 smithi086 ceph-mon[110975]: pgmap v705: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:37.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:37 smithi032 ceph-mon[147414]: pgmap v705: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:38.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:43:38.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:43:39.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:39 smithi086 ceph-mon[110975]: pgmap v706: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:39.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:39 smithi032 ceph-mon[147414]: pgmap v706: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:41.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:41 smithi086 ceph-mon[110975]: pgmap v707: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:41.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:41 smithi032 ceph-mon[147414]: pgmap v707: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:43.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:43 smithi086 ceph-mon[110975]: pgmap v708: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:43.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:43 smithi032 ceph-mon[147414]: pgmap v708: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:45.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:45 smithi086 ceph-mon[110975]: pgmap v709: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:45.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:45 smithi032 ceph-mon[147414]: pgmap v709: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:47.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:47 smithi086 ceph-mon[110975]: pgmap v710: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:47.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:47 smithi032 ceph-mon[147414]: pgmap v710: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:49.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:49 smithi086 ceph-mon[110975]: pgmap v711: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:49.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:49 smithi032 ceph-mon[147414]: pgmap v711: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:51.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:51 smithi086 ceph-mon[110975]: pgmap v712: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:51.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:51 smithi032 ceph-mon[147414]: pgmap v712: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:53.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:53 smithi086 ceph-mon[110975]: pgmap v713: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:53.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:43:53.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:53 smithi032 ceph-mon[147414]: pgmap v713: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:53.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:43:55.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.556+0000 7f806a5d4700 1 -- 172.21.15.32:0/3645753801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 msgr2=0x7f80640fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:55.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.556+0000 7f806a5d4700 1 --2- 172.21.15.32:0/3645753801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 0x7f80640fb910 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8058004940 tx=0x7f8058009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:55.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.556+0000 7f806a5d4700 1 -- 172.21.15.32:0/3645753801 shutdown_connections 2024-04-06T14:43:55.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.556+0000 7f806a5d4700 1 --2- 172.21.15.32:0/3645753801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 0x7f80640fb910 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.556+0000 7f806a5d4700 1 --2- 172.21.15.32:0/3645753801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8064102e70 0x7f8064103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.556+0000 7f806a5d4700 1 -- 172.21.15.32:0/3645753801 >> 172.21.15.32:0/3645753801 conn(0x7f80640714c0 msgr2=0x7f80640718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:55.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.556+0000 7f806a5d4700 1 -- 172.21.15.32:0/3645753801 shutdown_connections 2024-04-06T14:43:55.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.556+0000 7f806a5d4700 1 -- 172.21.15.32:0/3645753801 wait complete. 2024-04-06T14:43:55.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f806a5d4700 1 Processor -- start 2024-04-06T14:43:55.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f806a5d4700 1 -- start start 2024-04-06T14:43:55.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f806a5d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 0x7f8064113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f806a5d4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8064102e70 0x7f8064114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f806a5d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80641146e0 con 0x7f80640fb4a0 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f806a5d4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806410dad0 con 0x7f8064102e70 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f8063fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 0x7f8064113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f8063fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 0x7f8064113b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51740/0 (socket says 172.21.15.32:51740) 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f8063fff700 1 -- 172.21.15.32:0/389888728 learned_addr learned my addr 172.21.15.32:0/389888728 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.557+0000 7f80637fe700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8064102e70 0x7f8064114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.558+0000 7f8063fff700 1 -- 172.21.15.32:0/389888728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8064102e70 msgr2=0x7f8064114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.558+0000 7f8063fff700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8064102e70 0x7f8064114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.558+0000 7f8063fff700 1 -- 172.21.15.32:0/389888728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8058005040 con 0x7f80640fb4a0 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.558+0000 7f8063fff700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 0x7f8064113b50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8054002a50 tx=0x7f80540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.558+0000 7f8060ff9700 1 -- 172.21.15.32:0/389888728 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8054007cc0 con 0x7f80640fb4a0 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.558+0000 7f8060ff9700 1 -- 172.21.15.32:0/389888728 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f805400f040 con 0x7f80640fb4a0 2024-04-06T14:43:55.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.558+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806410ddb0 con 0x7f80640fb4a0 2024-04-06T14:43:55.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.558+0000 7f8060ff9700 1 -- 172.21.15.32:0/389888728 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f805400d9c0 con 0x7f80640fb4a0 2024-04-06T14:43:55.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.558+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806410e2f0 con 0x7f80640fb4a0 2024-04-06T14:43:55.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.559+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8064100170 con 0x7f80640fb4a0 2024-04-06T14:43:55.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.560+0000 7f8060ff9700 1 -- 172.21.15.32:0/389888728 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f805400db20 con 0x7f80640fb4a0 2024-04-06T14:43:55.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.560+0000 7f8060ff9700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f804c06cc20 0x7f804c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:55.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.560+0000 7f8060ff9700 1 -- 172.21.15.32:0/389888728 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f805408bcb0 con 0x7f80640fb4a0 2024-04-06T14:43:55.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.560+0000 7f80637fe700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f804c06cc20 0x7f804c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:55.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.561+0000 7f80637fe700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f804c06cc20 0x7f804c06f0d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f806410ec90 tx=0x7f8058009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.565+0000 7f8060ff9700 1 -- 172.21.15.32:0/389888728 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8054059780 con 0x7f80640fb4a0 2024-04-06T14:43:55.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:55 smithi086 ceph-mon[110975]: pgmap v714: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:55.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:55 smithi032 ceph-mon[147414]: pgmap v714: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:55.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.790+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f806410f1a0 con 0x7f804c06cc20 2024-04-06T14:43:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.791+0000 7f8060ff9700 1 -- 172.21.15.32:0/389888728 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f806410f1a0 con 0x7f804c06cc20 2024-04-06T14:43:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.793+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f804c06cc20 msgr2=0x7f804c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.793+0000 7f806a5d4700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f804c06cc20 0x7f804c06f0d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f806410ec90 tx=0x7f8058009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.793+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 msgr2=0x7f8064113b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.793+0000 7f806a5d4700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 0x7f8064113b50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8054002a50 tx=0x7f80540046b0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.793+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 shutdown_connections 2024-04-06T14:43:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.793+0000 7f806a5d4700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f804c06cc20 0x7f804c06f0d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.793+0000 7f806a5d4700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80640fb4a0 0x7f8064113b50 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.793+0000 7f806a5d4700 1 --2- 172.21.15.32:0/389888728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8064102e70 0x7f8064114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.793+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 >> 172.21.15.32:0/389888728 conn(0x7f80640714c0 msgr2=0x7f80640fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.794+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 shutdown_connections 2024-04-06T14:43:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.794+0000 7f806a5d4700 1 -- 172.21.15.32:0/389888728 wait complete. 2024-04-06T14:43:55.794 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:43:55.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.898+0000 7f055368a700 1 -- 172.21.15.32:0/237723904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 msgr2=0x7f054c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.898+0000 7f055368a700 1 --2- 172.21.15.32:0/237723904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 0x7f054c0fbd20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f053c0049c0 tx=0x7f053c009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.898+0000 7f055368a700 1 -- 172.21.15.32:0/237723904 shutdown_connections 2024-04-06T14:43:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.898+0000 7f055368a700 1 --2- 172.21.15.32:0/237723904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 0x7f054c0fbd20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.898+0000 7f055368a700 1 --2- 172.21.15.32:0/237723904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c101d10 0x7f054c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.898+0000 7f055368a700 1 -- 172.21.15.32:0/237723904 >> 172.21.15.32:0/237723904 conn(0x7f054c0f7880 msgr2=0x7f054c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.898+0000 7f055368a700 1 -- 172.21.15.32:0/237723904 shutdown_connections 2024-04-06T14:43:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.898+0000 7f055368a700 1 -- 172.21.15.32:0/237723904 wait complete. 2024-04-06T14:43:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f055368a700 1 Processor -- start 2024-04-06T14:43:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f055368a700 1 -- start start 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f055368a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c101d10 0x7f054c111b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f055368a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 0x7f054c1120a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f055368a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c1126f0 con 0x7f054c1026b0 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f055368a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c10bae0 con 0x7f054c101d10 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f0550c25700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 0x7f054c1120a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f0550c25700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 0x7f054c1120a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51772/0 (socket says 172.21.15.32:51772) 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f0550c25700 1 -- 172.21.15.32:0/1211546940 learned_addr learned my addr 172.21.15.32:0/1211546940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f0551426700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c101d10 0x7f054c111b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f0550c25700 1 -- 172.21.15.32:0/1211546940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c101d10 msgr2=0x7f054c111b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f0550c25700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c101d10 0x7f054c111b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:55.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.899+0000 7f0550c25700 1 -- 172.21.15.32:0/1211546940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c005040 con 0x7f054c1026b0 2024-04-06T14:43:55.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.900+0000 7f0550c25700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 0x7f054c1120a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f053c00b840 tx=0x7f053c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:55.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.900+0000 7f0541ffb700 1 -- 172.21.15.32:0/1211546940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f053c009ae0 con 0x7f054c1026b0 2024-04-06T14:43:55.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.900+0000 7f0541ffb700 1 -- 172.21.15.32:0/1211546940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f053c004290 con 0x7f054c1026b0 2024-04-06T14:43:55.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.900+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c10bd60 con 0x7f054c1026b0 2024-04-06T14:43:55.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.900+0000 7f0541ffb700 1 -- 172.21.15.32:0/1211546940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f053c011d50 con 0x7f054c1026b0 2024-04-06T14:43:55.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.900+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c10c320 con 0x7f054c1026b0 2024-04-06T14:43:55.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.901+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c0fd460 con 0x7f054c1026b0 2024-04-06T14:43:55.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.902+0000 7f0541ffb700 1 -- 172.21.15.32:0/1211546940 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f053c021070 con 0x7f054c1026b0 2024-04-06T14:43:55.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.903+0000 7f0541ffb700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f053806cc30 0x7f053806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:55.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.903+0000 7f0541ffb700 1 -- 172.21.15.32:0/1211546940 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f053c08fc70 con 0x7f054c1026b0 2024-04-06T14:43:55.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.903+0000 7f0551426700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f053806cc30 0x7f053806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:55.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.903+0000 7f0551426700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f053806cc30 0x7f053806f0e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0548004f40 tx=0x7f054800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:55.907+0000 7f0541ffb700 1 -- 172.21.15.32:0/1211546940 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f053c05d740 con 0x7f054c1026b0 2024-04-06T14:43:56.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.129+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f054c107b30 con 0x7f053806cc30 2024-04-06T14:43:56.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.131+0000 7f0541ffb700 1 -- 172.21.15.32:0/1211546940 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f054c107b30 con 0x7f053806cc30 2024-04-06T14:43:56.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f053806cc30 msgr2=0x7f053806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f053806cc30 0x7f053806f0e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0548004f40 tx=0x7f054800b040 comp rx=0 tx=0).stop 2024-04-06T14:43:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 msgr2=0x7f054c1120a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 0x7f054c1120a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f053c00b840 tx=0x7f053c009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 shutdown_connections 2024-04-06T14:43:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f053806cc30 0x7f053806f0e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c101d10 0x7f054c111b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 --2- 172.21.15.32:0/1211546940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c1026b0 0x7f054c1120a0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 >> 172.21.15.32:0/1211546940 conn(0x7f054c0f7880 msgr2=0x7f054c0f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 shutdown_connections 2024-04-06T14:43:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.133+0000 7f055368a700 1 -- 172.21.15.32:0/1211546940 wait complete. 2024-04-06T14:43:56.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 -- 172.21.15.32:0/3399881247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 msgr2=0x7f4e8c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 --2- 172.21.15.32:0/3399881247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 0x7f4e8c1031e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4e80004940 tx=0x7f4e80009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:56.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 -- 172.21.15.32:0/3399881247 shutdown_connections 2024-04-06T14:43:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 --2- 172.21.15.32:0/3399881247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e8c1037b0 0x7f4e8c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 --2- 172.21.15.32:0/3399881247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 0x7f4e8c1031e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 -- 172.21.15.32:0/3399881247 >> 172.21.15.32:0/3399881247 conn(0x7f4e8c0f89a0 msgr2=0x7f4e8c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 -- 172.21.15.32:0/3399881247 shutdown_connections 2024-04-06T14:43:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 -- 172.21.15.32:0/3399881247 wait complete. 2024-04-06T14:43:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 Processor -- start 2024-04-06T14:43:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.236+0000 7f4e92c1b700 1 -- start start 2024-04-06T14:43:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e92c1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 0x7f4e8c113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e92c1b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e8c1037b0 0x7f4e8c113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e92c1b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c1145e0 con 0x7f4e8c102e10 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e92c1b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c10d9d0 con 0x7f4e8c1037b0 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e909b7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 0x7f4e8c113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e909b7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 0x7f4e8c113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51794/0 (socket says 172.21.15.32:51794) 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e909b7700 1 -- 172.21.15.32:0/1616255902 learned_addr learned my addr 172.21.15.32:0/1616255902 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e909b7700 1 -- 172.21.15.32:0/1616255902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e8c1037b0 msgr2=0x7f4e8c113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e909b7700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e8c1037b0 0x7f4e8c113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e909b7700 1 -- 172.21.15.32:0/1616255902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e80005040 con 0x7f4e8c102e10 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.237+0000 7f4e909b7700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 0x7f4e8c113a50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000c00 tx=0x7f4e80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.238+0000 7f4e897fa700 1 -- 172.21.15.32:0/1616255902 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e80021070 con 0x7f4e8c102e10 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.238+0000 7f4e897fa700 1 -- 172.21.15.32:0/1616255902 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e80013de0 con 0x7f4e8c102e10 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.238+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c10dc50 con 0x7f4e8c102e10 2024-04-06T14:43:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.238+0000 7f4e897fa700 1 -- 172.21.15.32:0/1616255902 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8000e070 con 0x7f4e8c102e10 2024-04-06T14:43:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.238+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c10e1b0 con 0x7f4e8c102e10 2024-04-06T14:43:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.239+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8c0fe560 con 0x7f4e8c102e10 2024-04-06T14:43:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.239+0000 7f4e897fa700 1 -- 172.21.15.32:0/1616255902 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4e8001c600 con 0x7f4e8c102e10 2024-04-06T14:43:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.239+0000 7f4e897fa700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4e7c06cc20 0x7f4e7c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.240+0000 7f4e897fa700 1 -- 172.21.15.32:0/1616255902 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4e8008d130 con 0x7f4e8c102e10 2024-04-06T14:43:56.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.240+0000 7f4e8bfff700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4e7c06cc20 0x7f4e7c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:56.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.240+0000 7f4e8bfff700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4e7c06cc20 0x7f4e7c06f0d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c10eb90 tx=0x7f4e7800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.244+0000 7f4e897fa700 1 -- 172.21.15.32:0/1616255902 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4e8005ac00 con 0x7f4e8c102e10 2024-04-06T14:43:56.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.455+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e8c109e70 con 0x7f4e7c06cc20 2024-04-06T14:43:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.465+0000 7f4e897fa700 1 -- 172.21.15.32:0/1616255902 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4e7800f040 con 0x7f4e7c06cc20 2024-04-06T14:43:56.455 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (24m) 2m ago 35m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (21m) 2m ago 35m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (21m) 2m ago 34m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (24m) 2m ago 35m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (18m) 2m ago 29m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (17m) 2m ago 29m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (17m) 2m ago 29m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (2m) 2m ago 29m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (26m) 2m ago 36m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (25m) 2m ago 34m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (22m) 2m ago 37m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:43:56.456 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (22m) 2m ago 34m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:43:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (24m) 2m ago 35m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:43:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (24m) 2m ago 34m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:43:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (21m) 2m ago 33m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:43:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (20m) 2m ago 33m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:43:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (19m) 2m ago 32m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:43:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (19m) 2m ago 32m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:43:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (18m) 2m ago 31m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:43:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (18m) 2m ago 31m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:43:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (24m) 2m ago 35m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4e7c06cc20 msgr2=0x7f4e7c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4e7c06cc20 0x7f4e7c06f0d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c10eb90 tx=0x7f4e7800b040 comp rx=0 tx=0).stop 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 msgr2=0x7f4e8c113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 0x7f4e8c113a50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000c00 tx=0x7f4e80009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 shutdown_connections 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4e7c06cc20 0x7f4e7c06f0d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e8c102e10 0x7f4e8c113a50 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 --2- 172.21.15.32:0/1616255902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e8c1037b0 0x7f4e8c113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 >> 172.21.15.32:0/1616255902 conn(0x7f4e8c0f89a0 msgr2=0x7f4e8c0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 shutdown_connections 2024-04-06T14:43:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.467+0000 7f4e92c1b700 1 -- 172.21.15.32:0/1616255902 wait complete. 2024-04-06T14:43:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.569+0000 7f46456f3700 1 -- 172.21.15.32:0/952580211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 msgr2=0x7f4640102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.569+0000 7f46456f3700 1 --2- 172.21.15.32:0/952580211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 0x7f4640102140 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4630004940 tx=0x7f4630009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.569+0000 7f46456f3700 1 -- 172.21.15.32:0/952580211 shutdown_connections 2024-04-06T14:43:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.569+0000 7f46456f3700 1 --2- 172.21.15.32:0/952580211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4640102710 0x7f46400fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.569+0000 7f46456f3700 1 --2- 172.21.15.32:0/952580211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 0x7f4640102140 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.569+0000 7f46456f3700 1 -- 172.21.15.32:0/952580211 >> 172.21.15.32:0/952580211 conn(0x7f4640072bc0 msgr2=0x7f4640072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.569+0000 7f46456f3700 1 -- 172.21.15.32:0/952580211 shutdown_connections 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.569+0000 7f46456f3700 1 -- 172.21.15.32:0/952580211 wait complete. 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f46456f3700 1 Processor -- start 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f46456f3700 1 -- start start 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f46456f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 0x7f4640103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f46456f3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4640102710 0x7f46401037c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f46456f3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4640103d70 con 0x7f4640101d70 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f46456f3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4640103ee0 con 0x7f4640102710 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f463effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 0x7f4640103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f463effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 0x7f4640103280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51806/0 (socket says 172.21.15.32:51806) 2024-04-06T14:43:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f463effd700 1 -- 172.21.15.32:0/3063167560 learned_addr learned my addr 172.21.15.32:0/3063167560 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f463effd700 1 -- 172.21.15.32:0/3063167560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4640102710 msgr2=0x7f46401037c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f463effd700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4640102710 0x7f46401037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f463effd700 1 -- 172.21.15.32:0/3063167560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4630005040 con 0x7f4640101d70 2024-04-06T14:43:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.570+0000 7f463effd700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 0x7f4640103280 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4630000c00 tx=0x7f4630009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.571+0000 7f46377fe700 1 -- 172.21.15.32:0/3063167560 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4630021070 con 0x7f4640101d70 2024-04-06T14:43:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.571+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4640119d20 con 0x7f4640101d70 2024-04-06T14:43:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.571+0000 7f46377fe700 1 -- 172.21.15.32:0/3063167560 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4630013de0 con 0x7f4640101d70 2024-04-06T14:43:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.571+0000 7f46377fe700 1 -- 172.21.15.32:0/3063167560 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f463000e070 con 0x7f4640101d70 2024-04-06T14:43:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.571+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464011a130 con 0x7f4640101d70 2024-04-06T14:43:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.572+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46400fd4c0 con 0x7f4640101d70 2024-04-06T14:43:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.572+0000 7f46377fe700 1 -- 172.21.15.32:0/3063167560 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f463001c600 con 0x7f4640101d70 2024-04-06T14:43:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.572+0000 7f46377fe700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f462c06cc20 0x7f462c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.573+0000 7f46377fe700 1 -- 172.21.15.32:0/3063167560 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f463008d100 con 0x7f4640101d70 2024-04-06T14:43:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.573+0000 7f463e7fc700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f462c06cc20 0x7f462c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.573+0000 7f463e7fc700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f462c06cc20 0x7f462c06f0d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4640104bc0 tx=0x7f462800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:56.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.577+0000 7f46377fe700 1 -- 172.21.15.32:0/3063167560 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4630057140 con 0x7f4640101d70 2024-04-06T14:43:56.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.867+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f464004bed0 con 0x7f4640101d70 2024-04-06T14:43:56.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.868+0000 7f46377fe700 1 -- 172.21.15.32:0/3063167560 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f463005a760 con 0x7f4640101d70 2024-04-06T14:43:56.858 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:43:56.858 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:43:56.859 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:43:56.860 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:43:56.860 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:43:56.860 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:43:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f462c06cc20 msgr2=0x7f462c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f462c06cc20 0x7f462c06f0d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4640104bc0 tx=0x7f462800b040 comp rx=0 tx=0).stop 2024-04-06T14:43:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 msgr2=0x7f4640103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 0x7f4640103280 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4630000c00 tx=0x7f4630009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 shutdown_connections 2024-04-06T14:43:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f462c06cc20 0x7f462c06f0d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4640101d70 0x7f4640103280 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 --2- 172.21.15.32:0/3063167560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4640102710 0x7f46401037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 >> 172.21.15.32:0/3063167560 conn(0x7f4640072bc0 msgr2=0x7f4640065b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 shutdown_connections 2024-04-06T14:43:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.870+0000 7f46456f3700 1 -- 172.21.15.32:0/3063167560 wait complete. 2024-04-06T14:43:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 -- 172.21.15.32:0/3242122979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 msgr2=0x7fbd141020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 --2- 172.21.15.32:0/3242122979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 0x7fbd141020e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fbd08004940 tx=0x7fbd08009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 -- 172.21.15.32:0/3242122979 shutdown_connections 2024-04-06T14:43:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 --2- 172.21.15.32:0/3242122979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd141026b0 0x7fbd140fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 --2- 172.21.15.32:0/3242122979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 0x7fbd141020e0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 -- 172.21.15.32:0/3242122979 >> 172.21.15.32:0/3242122979 conn(0x7fbd140f7880 msgr2=0x7fbd140f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:56.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 -- 172.21.15.32:0/3242122979 shutdown_connections 2024-04-06T14:43:56.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 -- 172.21.15.32:0/3242122979 wait complete. 2024-04-06T14:43:56.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 Processor -- start 2024-04-06T14:43:56.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.973+0000 7fbd1abf5700 1 -- start start 2024-04-06T14:43:56.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd1abf5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 0x7fbd14113dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:56.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd1abf5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd141026b0 0x7fbd14114300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:56.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd1abf5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd14114950 con 0x7fbd14101d10 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd1abf5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1410dd20 con 0x7fbd141026b0 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd18991700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 0x7fbd14113dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd18991700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 0x7fbd14113dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51828/0 (socket says 172.21.15.32:51828) 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd18991700 1 -- 172.21.15.32:0/4029538273 learned_addr learned my addr 172.21.15.32:0/4029538273 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd13fff700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd141026b0 0x7fbd14114300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd18991700 1 -- 172.21.15.32:0/4029538273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd141026b0 msgr2=0x7fbd14114300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd18991700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd141026b0 0x7fbd14114300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd18991700 1 -- 172.21.15.32:0/4029538273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd08005040 con 0x7fbd14101d10 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd18991700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 0x7fbd14113dc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009360 tx=0x7fbd08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd117fa700 1 -- 172.21.15.32:0/4029538273 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd0800e040 con 0x7fbd14101d10 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd117fa700 1 -- 172.21.15.32:0/4029538273 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd08013420 con 0x7fbd14101d10 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1410dfa0 con 0x7fbd14101d10 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd117fa700 1 -- 172.21.15.32:0/4029538273 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd0801c5f0 con 0x7fbd14101d10 2024-04-06T14:43:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.974+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1410e560 con 0x7fbd14101d10 2024-04-06T14:43:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.975+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd140fd460 con 0x7fbd14101d10 2024-04-06T14:43:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.976+0000 7fbd117fa700 1 -- 172.21.15.32:0/4029538273 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbd08013590 con 0x7fbd14101d10 2024-04-06T14:43:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.976+0000 7fbd117fa700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc30 0x7fbd0406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.976+0000 7fbd117fa700 1 -- 172.21.15.32:0/4029538273 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbd0808d2d0 con 0x7fbd14101d10 2024-04-06T14:43:56.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.976+0000 7fbd13fff700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc30 0x7fbd0406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:56.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.977+0000 7fbd13fff700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc30 0x7fbd0406f0e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbd00007fd0 tx=0x7fbd00009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:56.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:56.981+0000 7fbd117fa700 1 -- 172.21.15.32:0/4029538273 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbd0805ada0 con 0x7fbd14101d10 2024-04-06T14:43:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.225+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbd1404bed0 con 0x7fbd14101d10 2024-04-06T14:43:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.225+0000 7fbd117fa700 1 -- 172.21.15.32:0/4029538273 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fbd0805a930 con 0x7fbd14101d10 2024-04-06T14:43:57.215 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:43:57.215 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:43:57.216 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:43:57.217 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:43:57.218 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:43:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc30 msgr2=0x7fbd0406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc30 0x7fbd0406f0e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbd00007fd0 tx=0x7fbd00009040 comp rx=0 tx=0).stop 2024-04-06T14:43:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 msgr2=0x7fbd14113dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 0x7fbd14113dc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009360 tx=0x7fbd08009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 shutdown_connections 2024-04-06T14:43:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc30 0x7fbd0406f0e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14101d10 0x7fbd14113dc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 --2- 172.21.15.32:0/4029538273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd141026b0 0x7fbd14114300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 >> 172.21.15.32:0/4029538273 conn(0x7fbd140f7880 msgr2=0x7fbd140f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 shutdown_connections 2024-04-06T14:43:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.227+0000 7fbd1abf5700 1 -- 172.21.15.32:0/4029538273 wait complete. 2024-04-06T14:43:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:43:57.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.331+0000 7f63d16f4700 1 -- 172.21.15.32:0/73233452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 msgr2=0x7f63cc1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.331+0000 7f63d16f4700 1 --2- 172.21.15.32:0/73233452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 0x7f63cc1092f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f63bc004940 tx=0x7f63bc009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:57.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.331+0000 7f63d16f4700 1 -- 172.21.15.32:0/73233452 shutdown_connections 2024-04-06T14:43:57.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.331+0000 7f63d16f4700 1 --2- 172.21.15.32:0/73233452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 0x7f63cc1092f0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.331+0000 7f63d16f4700 1 --2- 172.21.15.32:0/73233452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63cc0fbc80 0x7f63cc0fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.331+0000 7f63d16f4700 1 -- 172.21.15.32:0/73233452 >> 172.21.15.32:0/73233452 conn(0x7f63cc072bc0 msgr2=0x7f63cc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:57.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.331+0000 7f63d16f4700 1 -- 172.21.15.32:0/73233452 shutdown_connections 2024-04-06T14:43:57.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.331+0000 7f63d16f4700 1 -- 172.21.15.32:0/73233452 wait complete. 2024-04-06T14:43:57.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63d16f4700 1 Processor -- start 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63d16f4700 1 -- start start 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63d16f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63cc0fbc80 0x7f63cc115ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63d16f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 0x7f63cc1160e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63d16f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc116730 con 0x7f63cc0fc590 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63d16f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc10fb90 con 0x7f63cc0fbc80 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63ca7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 0x7f63cc1160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63ca7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 0x7f63cc1160e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51852/0 (socket says 172.21.15.32:51852) 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63ca7fc700 1 -- 172.21.15.32:0/678364082 learned_addr learned my addr 172.21.15.32:0/678364082 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.332+0000 7f63caffd700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63cc0fbc80 0x7f63cc115ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.333+0000 7f63ca7fc700 1 -- 172.21.15.32:0/678364082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63cc0fbc80 msgr2=0x7f63cc115ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.333+0000 7f63ca7fc700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63cc0fbc80 0x7f63cc115ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.333+0000 7f63ca7fc700 1 -- 172.21.15.32:0/678364082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc005040 con 0x7f63cc0fc590 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.333+0000 7f63ca7fc700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 0x7f63cc1160e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009fd0 tx=0x7f63bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.333+0000 7f63b37fe700 1 -- 172.21.15.32:0/678364082 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63bc021070 con 0x7f63cc0fc590 2024-04-06T14:43:57.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.333+0000 7f63b37fe700 1 -- 172.21.15.32:0/678364082 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63bc013e00 con 0x7f63cc0fc590 2024-04-06T14:43:57.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.333+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc10fe10 con 0x7f63cc0fc590 2024-04-06T14:43:57.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.333+0000 7f63b37fe700 1 -- 172.21.15.32:0/678364082 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63bc00e070 con 0x7f63cc0fc590 2024-04-06T14:43:57.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.333+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc110370 con 0x7f63cc0fc590 2024-04-06T14:43:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.334+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc065e30 con 0x7f63cc0fc590 2024-04-06T14:43:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.335+0000 7f63b37fe700 1 -- 172.21.15.32:0/678364082 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f63bc01c500 con 0x7f63cc0fc590 2024-04-06T14:43:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.335+0000 7f63b37fe700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f63b406cc20 0x7f63b406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.335+0000 7f63b37fe700 1 -- 172.21.15.32:0/678364082 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f63bc08d100 con 0x7f63cc0fc590 2024-04-06T14:43:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.335+0000 7f63caffd700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f63b406cc20 0x7f63b406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.335+0000 7f63caffd700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f63b406cc20 0x7f63b406f0d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009b80 tx=0x7f63c000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:57.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.339+0000 7f63b37fe700 1 -- 172.21.15.32:0/678364082 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f63bc057140 con 0x7f63cc0fc590 2024-04-06T14:43:57.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.560+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f63cc10c020 con 0x7f63b406cc20 2024-04-06T14:43:57.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.561+0000 7f63b37fe700 1 -- 172.21.15.32:0/678364082 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f63cc10c020 con 0x7f63b406cc20 2024-04-06T14:43:57.551 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:43:57.551 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:43:57.551 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:43:57.551 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:43:57.551 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:43:57.551 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:43:57.551 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:43:57.552 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:43:57.552 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:43:57.552 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:43:57.552 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:43:57.552 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:43:57.552 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:43:57.552 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:43:57.552 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:43:57.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f63b406cc20 msgr2=0x7f63b406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f63b406cc20 0x7f63b406f0d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009b80 tx=0x7f63c000b040 comp rx=0 tx=0).stop 2024-04-06T14:43:57.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 msgr2=0x7f63cc1160e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 0x7f63cc1160e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009fd0 tx=0x7f63bc009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:57.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 shutdown_connections 2024-04-06T14:43:57.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f63b406cc20 0x7f63b406f0d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63cc0fbc80 0x7f63cc115ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 --2- 172.21.15.32:0/678364082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63cc0fc590 0x7f63cc1160e0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 >> 172.21.15.32:0/678364082 conn(0x7f63cc072bc0 msgr2=0x7f63cc105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:57.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 shutdown_connections 2024-04-06T14:43:57.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.563+0000 7f63d16f4700 1 -- 172.21.15.32:0/678364082 wait complete. 2024-04-06T14:43:57.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:57 smithi086 ceph-mon[110975]: from='client.35318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:57.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:57 smithi086 ceph-mon[110975]: pgmap v715: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:57.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:57 smithi086 ceph-mon[110975]: from='client.35322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:57.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:57 smithi086 ceph-mon[110975]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:57.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:57 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3063167560' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:43:57.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:57 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4029538273' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:43:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.666+0000 7f066cb6c700 1 -- 172.21.15.32:0/689979914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668102710 msgr2=0x7f06680fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.666+0000 7f066cb6c700 1 --2- 172.21.15.32:0/689979914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668102710 0x7f06680fbd80 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0650004940 tx=0x7f0650009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.666+0000 7f066cb6c700 1 -- 172.21.15.32:0/689979914 shutdown_connections 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.666+0000 7f066cb6c700 1 --2- 172.21.15.32:0/689979914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668102710 0x7f06680fbd80 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.666+0000 7f066cb6c700 1 --2- 172.21.15.32:0/689979914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0668101d70 0x7f0668102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.666+0000 7f066cb6c700 1 -- 172.21.15.32:0/689979914 >> 172.21.15.32:0/689979914 conn(0x7f0668072bc0 msgr2=0x7f0668072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066cb6c700 1 -- 172.21.15.32:0/689979914 shutdown_connections 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066cb6c700 1 -- 172.21.15.32:0/689979914 wait complete. 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066cb6c700 1 Processor -- start 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066cb6c700 1 -- start start 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066cb6c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668101d70 0x7f0668115b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066cb6c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0668102710 0x7f06681160d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066cb6c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0668116720 con 0x7f0668101d70 2024-04-06T14:43:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066cb6c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066810fba0 con 0x7f0668102710 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668101d70 0x7f0668115b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668101d70 0x7f0668115b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51868/0 (socket says 172.21.15.32:51868) 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.667+0000 7f066659c700 1 -- 172.21.15.32:0/2481943310 learned_addr learned my addr 172.21.15.32:0/2481943310 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f0665d9b700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0668102710 0x7f06681160d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f066659c700 1 -- 172.21.15.32:0/2481943310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0668102710 msgr2=0x7f06681160d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f066659c700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0668102710 0x7f06681160d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f066659c700 1 -- 172.21.15.32:0/2481943310 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0650005040 con 0x7f0668101d70 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f066659c700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668101d70 0x7f0668115b90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f065800eb40 tx=0x7f065800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f065effd700 1 -- 172.21.15.32:0/2481943310 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0658007cc0 con 0x7f0668101d70 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f065effd700 1 -- 172.21.15.32:0/2481943310 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f065800f040 con 0x7f0668101d70 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066810fe20 con 0x7f0668101d70 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f065effd700 1 -- 172.21.15.32:0/2481943310 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0658009570 con 0x7f0668101d70 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.668+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06681103e0 con 0x7f0668101d70 2024-04-06T14:43:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.669+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06680fd4c0 con 0x7f0668101d70 2024-04-06T14:43:57.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.670+0000 7f065effd700 1 -- 172.21.15.32:0/2481943310 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f06580096d0 con 0x7f0668101d70 2024-04-06T14:43:57.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.670+0000 7f065effd700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f065406cc20 0x7f065406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:43:57.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.671+0000 7f065effd700 1 -- 172.21.15.32:0/2481943310 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f065808be70 con 0x7f0668101d70 2024-04-06T14:43:57.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.671+0000 7f0665d9b700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f065406cc20 0x7f065406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:43:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.671+0000 7f0665d9b700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f065406cc20 0x7f065406f0d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f065000b0d0 tx=0x7f0650009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:43:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.674+0000 7f065effd700 1 -- 172.21.15.32:0/2481943310 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0658056020 con 0x7f0668101d70 2024-04-06T14:43:57.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:57 smithi032 ceph-mon[147414]: from='client.35318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:57.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:57 smithi032 ceph-mon[147414]: pgmap v715: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:57.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:57 smithi032 ceph-mon[147414]: from='client.35322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:57.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:57 smithi032 ceph-mon[147414]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:57.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:57 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3063167560' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:43:57.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:57 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4029538273' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:43:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.961+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f066805f5f0 con 0x7f0668101d70 2024-04-06T14:43:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.961+0000 7f065effd700 1 -- 172.21.15.32:0/2481943310 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f0658059640 con 0x7f0668101d70 2024-04-06T14:43:57.952 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:43:57.952 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:43:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:43:57.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:43:57.952 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:43:57.952 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:43:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:43:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:43:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:43:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:43:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:43:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:43:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:43:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:43:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:43:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:43:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:43:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:43:57.954 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:43:57.954 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:43:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:43:57.954 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:43:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:43:57.955 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:43:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:43:57.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f065406cc20 msgr2=0x7f065406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f065406cc20 0x7f065406f0d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f065000b0d0 tx=0x7f0650009e30 comp rx=0 tx=0).stop 2024-04-06T14:43:57.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668101d70 msgr2=0x7f0668115b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:43:57.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668101d70 0x7f0668115b90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f065800eb40 tx=0x7f065800b040 comp rx=0 tx=0).stop 2024-04-06T14:43:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 shutdown_connections 2024-04-06T14:43:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f065406cc20 0x7f065406f0d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0668101d70 0x7f0668115b90 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 --2- 172.21.15.32:0/2481943310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0668102710 0x7f06681160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:43:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 >> 172.21.15.32:0/2481943310 conn(0x7f0668072bc0 msgr2=0x7f0668065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:43:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 shutdown_connections 2024-04-06T14:43:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:43:57.963+0000 7f066cb6c700 1 -- 172.21.15.32:0/2481943310 wait complete. 2024-04-06T14:43:58.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:58 smithi086 ceph-mon[110975]: from='client.35338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:58.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:58 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2481943310' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:43:58.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:58 smithi032 ceph-mon[147414]: from='client.35338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:43:58.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:58 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2481943310' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:43:59.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:43:59 smithi086 ceph-mon[110975]: pgmap v716: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:43:59.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:43:59 smithi032 ceph-mon[147414]: pgmap v716: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:01.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:01 smithi086 ceph-mon[110975]: pgmap v717: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:01.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:01 smithi032 ceph-mon[147414]: pgmap v717: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:03.619 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:03 smithi086 ceph-mon[110975]: pgmap v718: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:03.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:03 smithi032 ceph-mon[147414]: pgmap v718: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:05.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:05 smithi086 ceph-mon[110975]: pgmap v719: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:05.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:05 smithi032 ceph-mon[147414]: pgmap v719: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:07.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:07 smithi086 ceph-mon[110975]: pgmap v720: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:07.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:07 smithi032 ceph-mon[147414]: pgmap v720: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:08.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:44:08.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:44:09.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:09 smithi032 ceph-mon[147414]: pgmap v721: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:09.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:09 smithi086 ceph-mon[110975]: pgmap v721: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:11.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:11 smithi032 ceph-mon[147414]: pgmap v722: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:11.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:11 smithi086 ceph-mon[110975]: pgmap v722: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:13.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:13 smithi032 ceph-mon[147414]: pgmap v723: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:13.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:13 smithi086 ceph-mon[110975]: pgmap v723: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:15.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:15 smithi032 ceph-mon[147414]: pgmap v724: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:15.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:15 smithi086 ceph-mon[110975]: pgmap v724: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:17.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:17 smithi032 ceph-mon[147414]: pgmap v725: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:17.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:17 smithi086 ceph-mon[110975]: pgmap v725: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:19.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:19 smithi032 ceph-mon[147414]: pgmap v726: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:19.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:19 smithi086 ceph-mon[110975]: pgmap v726: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:21.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:21 smithi032 ceph-mon[147414]: pgmap v727: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:21.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:21 smithi086 ceph-mon[110975]: pgmap v727: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:23.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:23 smithi032 ceph-mon[147414]: pgmap v728: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:23.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:44:23.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:23 smithi086 ceph-mon[110975]: pgmap v728: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:23.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:44:25.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:25 smithi032 ceph-mon[147414]: pgmap v729: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:25.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:25 smithi086 ceph-mon[110975]: pgmap v729: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:27.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:27 smithi032 ceph-mon[147414]: pgmap v730: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:27.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:27 smithi086 ceph-mon[110975]: pgmap v730: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.080+0000 7fa3ede28700 1 -- 172.21.15.32:0/3714075066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fe830 msgr2=0x7fa3e8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.080+0000 7fa3ede28700 1 --2- 172.21.15.32:0/3714075066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fe830 0x7fa3e8106f00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8004940 tx=0x7fa3d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.080+0000 7fa3ede28700 1 -- 172.21.15.32:0/3714075066 shutdown_connections 2024-04-06T14:44:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.080+0000 7fa3ede28700 1 --2- 172.21.15.32:0/3714075066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fe830 0x7fa3e8106f00 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.080+0000 7fa3ede28700 1 --2- 172.21.15.32:0/3714075066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3e80fde90 0x7fa3e80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.080+0000 7fa3ede28700 1 -- 172.21.15.32:0/3714075066 >> 172.21.15.32:0/3714075066 conn(0x7fa3e8072bc0 msgr2=0x7fa3e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.080+0000 7fa3ede28700 1 -- 172.21.15.32:0/3714075066 shutdown_connections 2024-04-06T14:44:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.080+0000 7fa3ede28700 1 -- 172.21.15.32:0/3714075066 wait complete. 2024-04-06T14:44:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.081+0000 7fa3ede28700 1 Processor -- start 2024-04-06T14:44:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.081+0000 7fa3ede28700 1 -- start start 2024-04-06T14:44:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.081+0000 7fa3ede28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fde90 0x7fa3e8113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.081+0000 7fa3ede28700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3e80fe830 0x7fa3e8113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.081+0000 7fa3ede28700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e8114610 con 0x7fa3e80fde90 2024-04-06T14:44:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.081+0000 7fa3ede28700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e810da00 con 0x7fa3e80fe830 2024-04-06T14:44:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.081+0000 7fa3e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fde90 0x7fa3e8113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.081+0000 7fa3e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fde90 0x7fa3e8113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38442/0 (socket says 172.21.15.32:38442) 2024-04-06T14:44:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.081+0000 7fa3e77fe700 1 -- 172.21.15.32:0/1362907654 learned_addr learned my addr 172.21.15.32:0/1362907654 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:44:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3e6ffd700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3e80fe830 0x7fa3e8113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3e77fe700 1 -- 172.21.15.32:0/1362907654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3e80fe830 msgr2=0x7fa3e8113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3e77fe700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3e80fe830 0x7fa3e8113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3e77fe700 1 -- 172.21.15.32:0/1362907654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d8005040 con 0x7fa3e80fde90 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3e77fe700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fde90 0x7fa3e8113a80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000eb40 tx=0x7fa3d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3dffff700 1 -- 172.21.15.32:0/1362907654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d0007cc0 con 0x7fa3e80fde90 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e810dc80 con 0x7fa3e80fde90 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3dffff700 1 -- 172.21.15.32:0/1362907654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3d000f040 con 0x7fa3e80fde90 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3dffff700 1 -- 172.21.15.32:0/1362907654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d000c990 con 0x7fa3e80fde90 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.082+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e810e1c0 con 0x7fa3e80fde90 2024-04-06T14:44:28.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.083+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e8101980 con 0x7fa3e80fde90 2024-04-06T14:44:28.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.084+0000 7fa3dffff700 1 -- 172.21.15.32:0/1362907654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa3d000caf0 con 0x7fa3e80fde90 2024-04-06T14:44:28.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.084+0000 7fa3dffff700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3d406cc20 0x7fa3d406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:28.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.084+0000 7fa3dffff700 1 -- 172.21.15.32:0/1362907654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa3d008bb90 con 0x7fa3e80fde90 2024-04-06T14:44:28.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.084+0000 7fa3e6ffd700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3d406cc20 0x7fa3d406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:28.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.085+0000 7fa3e6ffd700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3d406cc20 0x7fa3d406f0d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa3e810ebc0 tx=0x7fa3d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:28.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.089+0000 7fa3dffff700 1 -- 172.21.15.32:0/1362907654 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa3d0059710 con 0x7fa3e80fde90 2024-04-06T14:44:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.316+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3e810f0d0 con 0x7fa3d406cc20 2024-04-06T14:44:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.317+0000 7fa3dffff700 1 -- 172.21.15.32:0/1362907654 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa3e810f0d0 con 0x7fa3d406cc20 2024-04-06T14:44:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3d406cc20 msgr2=0x7fa3d406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3d406cc20 0x7fa3d406f0d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa3e810ebc0 tx=0x7fa3d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fde90 msgr2=0x7fa3e8113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fde90 0x7fa3e8113a80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000eb40 tx=0x7fa3d000b040 comp rx=0 tx=0).stop 2024-04-06T14:44:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 shutdown_connections 2024-04-06T14:44:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa3d406cc20 0x7fa3d406f0d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e80fde90 0x7fa3e8113a80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 --2- 172.21.15.32:0/1362907654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3e80fe830 0x7fa3e8113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 >> 172.21.15.32:0/1362907654 conn(0x7fa3e8072bc0 msgr2=0x7fa3e80ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 shutdown_connections 2024-04-06T14:44:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.319+0000 7fa3ede28700 1 -- 172.21.15.32:0/1362907654 wait complete. 2024-04-06T14:44:28.321 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:44:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.424+0000 7f14f3746700 1 -- 172.21.15.32:0/1577897083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fbbf0 msgr2=0x7f14ec0fbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.424+0000 7f14f3746700 1 --2- 172.21.15.32:0/1577897083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fbbf0 0x7f14ec0fbfc0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f14e8004940 tx=0x7f14e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.424+0000 7f14f3746700 1 -- 172.21.15.32:0/1577897083 shutdown_connections 2024-04-06T14:44:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.424+0000 7f14f3746700 1 --2- 172.21.15.32:0/1577897083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ec0fc590 0x7f14ec100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.424+0000 7f14f3746700 1 --2- 172.21.15.32:0/1577897083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fbbf0 0x7f14ec0fbfc0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.424+0000 7f14f3746700 1 -- 172.21.15.32:0/1577897083 >> 172.21.15.32:0/1577897083 conn(0x7f14ec0f78d0 msgr2=0x7f14ec0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.425+0000 7f14f3746700 1 -- 172.21.15.32:0/1577897083 shutdown_connections 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.425+0000 7f14f3746700 1 -- 172.21.15.32:0/1577897083 wait complete. 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.425+0000 7f14f3746700 1 Processor -- start 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.425+0000 7f14f3746700 1 -- start start 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.425+0000 7f14f3746700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ec0fbbf0 0x7f14ec11a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.425+0000 7f14f3746700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fc590 0x7f14ec11a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.425+0000 7f14f3746700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec11ad90 con 0x7f14ec0fc590 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.425+0000 7f14f3746700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec1141a0 con 0x7f14ec0fbbf0 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f0ce1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fc590 0x7f14ec11a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f14e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ec0fbbf0 0x7f14ec11a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f0ce1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fc590 0x7f14ec11a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38466/0 (socket says 172.21.15.32:38466) 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f0ce1700 1 -- 172.21.15.32:0/3272769642 learned_addr learned my addr 172.21.15.32:0/3272769642 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f0ce1700 1 -- 172.21.15.32:0/3272769642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ec0fbbf0 msgr2=0x7f14ec11a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f0ce1700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ec0fbbf0 0x7f14ec11a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f0ce1700 1 -- 172.21.15.32:0/3272769642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e8005040 con 0x7f14ec0fc590 2024-04-06T14:44:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f0ce1700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fc590 0x7f14ec11a740 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00c830 tx=0x7f14dc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14e1ffb700 1 -- 172.21.15.32:0/3272769642 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14dc0106f0 con 0x7f14ec0fc590 2024-04-06T14:44:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14e1ffb700 1 -- 172.21.15.32:0/3272769642 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14dc010d00 con 0x7f14ec0fc590 2024-04-06T14:44:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec114480 con 0x7f14ec0fc590 2024-04-06T14:44:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14e1ffb700 1 -- 172.21.15.32:0/3272769642 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14dc012e90 con 0x7f14ec0fc590 2024-04-06T14:44:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.426+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec1149c0 con 0x7f14ec0fc590 2024-04-06T14:44:28.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.427+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec0fe2d0 con 0x7f14ec0fc590 2024-04-06T14:44:28.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.428+0000 7f14e1ffb700 1 -- 172.21.15.32:0/3272769642 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f14dc010850 con 0x7f14ec0fc590 2024-04-06T14:44:28.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.428+0000 7f14e1ffb700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14d806cc30 0x7f14d806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:28.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.428+0000 7f14e1ffb700 1 -- 172.21.15.32:0/3272769642 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f14dc08bf60 con 0x7f14ec0fc590 2024-04-06T14:44:28.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.428+0000 7f14f14e2700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14d806cc30 0x7f14d806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:28.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.429+0000 7f14f14e2700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14d806cc30 0x7f14d806f0e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f14e800b840 tx=0x7f14e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.433+0000 7f14e1ffb700 1 -- 172.21.15.32:0/3272769642 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f14dc0bfab0 con 0x7f14ec0fc590 2024-04-06T14:44:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.653+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f14ec115890 con 0x7f14d806cc30 2024-04-06T14:44:28.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.655+0000 7f14e1ffb700 1 -- 172.21.15.32:0/3272769642 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f14ec115890 con 0x7f14d806cc30 2024-04-06T14:44:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14d806cc30 msgr2=0x7f14d806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14d806cc30 0x7f14d806f0e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f14e800b840 tx=0x7f14e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fc590 msgr2=0x7f14ec11a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fc590 0x7f14ec11a740 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00c830 tx=0x7f14dc004470 comp rx=0 tx=0).stop 2024-04-06T14:44:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 shutdown_connections 2024-04-06T14:44:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f14d806cc30 0x7f14d806f0e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ec0fbbf0 0x7f14ec11a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 --2- 172.21.15.32:0/3272769642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14ec0fc590 0x7f14ec11a740 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 >> 172.21.15.32:0/3272769642 conn(0x7f14ec0f78d0 msgr2=0x7f14ec0f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 shutdown_connections 2024-04-06T14:44:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.657+0000 7f14f3746700 1 -- 172.21.15.32:0/3272769642 wait complete. 2024-04-06T14:44:28.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.760+0000 7fb489bb4700 1 -- 172.21.15.32:0/1586248864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4841037b0 msgr2=0x7fb4840fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.760+0000 7fb489bb4700 1 --2- 172.21.15.32:0/1586248864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4841037b0 0x7fb4840fce20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb478004940 tx=0x7fb478009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:28.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.760+0000 7fb489bb4700 1 -- 172.21.15.32:0/1586248864 shutdown_connections 2024-04-06T14:44:28.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.760+0000 7fb489bb4700 1 --2- 172.21.15.32:0/1586248864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4841037b0 0x7fb4840fce20 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.760+0000 7fb489bb4700 1 --2- 172.21.15.32:0/1586248864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb484102e10 0x7fb4841031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.760+0000 7fb489bb4700 1 -- 172.21.15.32:0/1586248864 >> 172.21.15.32:0/1586248864 conn(0x7fb4840f8980 msgr2=0x7fb4840fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:28.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.760+0000 7fb489bb4700 1 -- 172.21.15.32:0/1586248864 shutdown_connections 2024-04-06T14:44:28.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.760+0000 7fb489bb4700 1 -- 172.21.15.32:0/1586248864 wait complete. 2024-04-06T14:44:28.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.761+0000 7fb489bb4700 1 Processor -- start 2024-04-06T14:44:28.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.761+0000 7fb489bb4700 1 -- start start 2024-04-06T14:44:28.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.761+0000 7fb489bb4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb484102e10 0x7fb484113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.761+0000 7fb489bb4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4841037b0 0x7fb484114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.761+0000 7fb489bb4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb484114670 con 0x7fb484102e10 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.761+0000 7fb489bb4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48410da60 con 0x7fb4841037b0 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb4837fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb484102e10 0x7fb484113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb4837fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb484102e10 0x7fb484113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38484/0 (socket says 172.21.15.32:38484) 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb4837fe700 1 -- 172.21.15.32:0/3548860405 learned_addr learned my addr 172.21.15.32:0/3548860405 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb482ffd700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4841037b0 0x7fb484114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb4837fe700 1 -- 172.21.15.32:0/3548860405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4841037b0 msgr2=0x7fb484114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb4837fe700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4841037b0 0x7fb484114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb4837fe700 1 -- 172.21.15.32:0/3548860405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb478005040 con 0x7fb484102e10 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb482ffd700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4841037b0 0x7fb484114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb4837fe700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb484102e10 0x7fb484113ae0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb47400c830 tx=0x7fb474004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb46bfff700 1 -- 172.21.15.32:0/3548860405 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb47400a760 con 0x7fb484102e10 2024-04-06T14:44:28.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48410dd40 con 0x7fb484102e10 2024-04-06T14:44:28.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb46bfff700 1 -- 172.21.15.32:0/3548860405 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb47400ad70 con 0x7fb484102e10 2024-04-06T14:44:28.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb46bfff700 1 -- 172.21.15.32:0/3548860405 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4740113a0 con 0x7fb484102e10 2024-04-06T14:44:28.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.762+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48410e300 con 0x7fb484102e10 2024-04-06T14:44:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.763+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4840fe560 con 0x7fb484102e10 2024-04-06T14:44:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.764+0000 7fb46bfff700 1 -- 172.21.15.32:0/3548860405 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb474012400 con 0x7fb484102e10 2024-04-06T14:44:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.764+0000 7fb46bfff700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb46c06cc20 0x7fb46c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.764+0000 7fb46bfff700 1 -- 172.21.15.32:0/3548860405 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb47408c130 con 0x7fb484102e10 2024-04-06T14:44:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.764+0000 7fb482ffd700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb46c06cc20 0x7fb46c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.765+0000 7fb482ffd700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb46c06cc20 0x7fb46c06f0d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb48410ec20 tx=0x7fb478009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:28.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.769+0000 7fb46bfff700 1 -- 172.21.15.32:0/3548860405 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb474059b80 con 0x7fb484102e10 2024-04-06T14:44:28.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.981+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb484109db0 con 0x7fb46c06cc20 2024-04-06T14:44:28.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.991+0000 7fb46bfff700 1 -- 172.21.15.32:0/3548860405 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb484109db0 con 0x7fb46c06cc20 2024-04-06T14:44:28.982 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:44:28.982 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (24m) 2m ago 36m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:44:28.982 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (22m) 2m ago 36m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (22m) 2m ago 35m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (24m) 2m ago 35m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (18m) 2m ago 30m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (18m) 2m ago 30m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (17m) 2m ago 30m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (2m) 2m ago 30m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (26m) 2m ago 37m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (25m) 2m ago 35m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (23m) 2m ago 37m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (22m) 2m ago 35m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (25m) 2m ago 36m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (25m) 2m ago 35m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (21m) 2m ago 34m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:44:28.983 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (21m) 2m ago 33m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:44:28.984 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (20m) 2m ago 33m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:44:28.984 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (19m) 2m ago 32m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:44:28.984 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (19m) 2m ago 32m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:44:28.984 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (18m) 2m ago 31m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:44:28.984 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (25m) 2m ago 35m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:44:28.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb46c06cc20 msgr2=0x7fb46c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb46c06cc20 0x7fb46c06f0d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb48410ec20 tx=0x7fb478009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb484102e10 msgr2=0x7fb484113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb484102e10 0x7fb484113ae0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb47400c830 tx=0x7fb474004470 comp rx=0 tx=0).stop 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 shutdown_connections 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb46c06cc20 0x7fb46c06f0d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb484102e10 0x7fb484113ae0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 --2- 172.21.15.32:0/3548860405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4841037b0 0x7fb484114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 >> 172.21.15.32:0/3548860405 conn(0x7fb4840f8980 msgr2=0x7fb4840f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.993+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 shutdown_connections 2024-04-06T14:44:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:28.994+0000 7fb489bb4700 1 -- 172.21.15.32:0/3548860405 wait complete. 2024-04-06T14:44:29.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.096+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/1026845881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b441037b0 msgr2=0x7f5b440fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.096+0000 7f5b4a9a6700 1 --2- 172.21.15.32:0/1026845881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b441037b0 0x7f5b440fce20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5b38004940 tx=0x7f5b38009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:29.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.096+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/1026845881 shutdown_connections 2024-04-06T14:44:29.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.096+0000 7f5b4a9a6700 1 --2- 172.21.15.32:0/1026845881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b441037b0 0x7f5b440fce20 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.096+0000 7f5b4a9a6700 1 --2- 172.21.15.32:0/1026845881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44102e10 0x7f5b441031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.096+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/1026845881 >> 172.21.15.32:0/1026845881 conn(0x7f5b440f8980 msgr2=0x7f5b440fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:29.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.096+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/1026845881 shutdown_connections 2024-04-06T14:44:29.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.096+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/1026845881 wait complete. 2024-04-06T14:44:29.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b4a9a6700 1 Processor -- start 2024-04-06T14:44:29.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b4a9a6700 1 -- start start 2024-04-06T14:44:29.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b4a9a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b44102e10 0x7f5b44113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:29.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b4a9a6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b441037b0 0x7f5b44114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b4a9a6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b44114670 con 0x7f5b44102e10 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b4a9a6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4410da60 con 0x7f5b441037b0 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b43fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b44102e10 0x7f5b44113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b43fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b44102e10 0x7f5b44113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38502/0 (socket says 172.21.15.32:38502) 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b43fff700 1 -- 172.21.15.32:0/529219655 learned_addr learned my addr 172.21.15.32:0/529219655 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b437fe700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b441037b0 0x7f5b44114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b43fff700 1 -- 172.21.15.32:0/529219655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b441037b0 msgr2=0x7f5b44114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b43fff700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b441037b0 0x7f5b44114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.097+0000 7f5b43fff700 1 -- 172.21.15.32:0/529219655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b38005040 con 0x7f5b44102e10 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.098+0000 7f5b43fff700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b44102e10 0x7f5b44113ae0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5b34002a50 tx=0x7f5b340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.098+0000 7f5b40ff9700 1 -- 172.21.15.32:0/529219655 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b34007cc0 con 0x7f5b44102e10 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.098+0000 7f5b40ff9700 1 -- 172.21.15.32:0/529219655 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b3400f040 con 0x7f5b44102e10 2024-04-06T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.098+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4410dd40 con 0x7f5b44102e10 2024-04-06T14:44:29.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.098+0000 7f5b40ff9700 1 -- 172.21.15.32:0/529219655 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b34009540 con 0x7f5b44102e10 2024-04-06T14:44:29.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.098+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4410e300 con 0x7f5b44102e10 2024-04-06T14:44:29.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.099+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b440fe560 con 0x7f5b44102e10 2024-04-06T14:44:29.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.100+0000 7f5b40ff9700 1 -- 172.21.15.32:0/529219655 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5b340096a0 con 0x7f5b44102e10 2024-04-06T14:44:29.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.100+0000 7f5b40ff9700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b2c06cc20 0x7f5b2c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:29.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.100+0000 7f5b40ff9700 1 -- 172.21.15.32:0/529219655 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5b3408c510 con 0x7f5b44102e10 2024-04-06T14:44:29.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.100+0000 7f5b437fe700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b2c06cc20 0x7f5b2c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:29.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.101+0000 7f5b437fe700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b2c06cc20 0x7f5b2c06f0d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5b4410ec20 tx=0x7f5b38009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:29.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.104+0000 7f5b40ff9700 1 -- 172.21.15.32:0/529219655 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5b34059f60 con 0x7f5b44102e10 2024-04-06T14:44:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.391+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5b44109de0 con 0x7f5b44102e10 2024-04-06T14:44:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.392+0000 7f5b40ff9700 1 -- 172.21.15.32:0/529219655 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f5b34059af0 con 0x7f5b44102e10 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:44:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:44:29.384 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:44:29.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:44:29.384 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:44:29.384 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:44:29.384 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:44:29.384 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:44:29.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b2c06cc20 msgr2=0x7f5b2c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b2c06cc20 0x7f5b2c06f0d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5b4410ec20 tx=0x7f5b38009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b44102e10 msgr2=0x7f5b44113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b44102e10 0x7f5b44113ae0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5b34002a50 tx=0x7f5b340046b0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 shutdown_connections 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5b2c06cc20 0x7f5b2c06f0d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b44102e10 0x7f5b44113ae0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 --2- 172.21.15.32:0/529219655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b441037b0 0x7f5b44114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 >> 172.21.15.32:0/529219655 conn(0x7f5b440f8980 msgr2=0x7f5b440f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 shutdown_connections 2024-04-06T14:44:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.394+0000 7f5b4a9a6700 1 -- 172.21.15.32:0/529219655 wait complete. 2024-04-06T14:44:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.496+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2133663826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 msgr2=0x7f77f4103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.496+0000 7f77fb9a5700 1 --2- 172.21.15.32:0/2133663826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 0x7f77f4103240 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f77f0004940 tx=0x7f77f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.496+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2133663826 shutdown_connections 2024-04-06T14:44:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.496+0000 7f77fb9a5700 1 --2- 172.21.15.32:0/2133663826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f40fb4a0 0x7f77f40fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.496+0000 7f77fb9a5700 1 --2- 172.21.15.32:0/2133663826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 0x7f77f4103240 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.496+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2133663826 >> 172.21.15.32:0/2133663826 conn(0x7f77f40714c0 msgr2=0x7f77f40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:29.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.496+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2133663826 shutdown_connections 2024-04-06T14:44:29.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.496+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2133663826 wait complete. 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.497+0000 7f77fb9a5700 1 Processor -- start 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.497+0000 7f77fb9a5700 1 -- start start 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.497+0000 7f77fb9a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f40fb4a0 0x7f77f4113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.497+0000 7f77fb9a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 0x7f77f4113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.497+0000 7f77fb9a5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f41146b0 con 0x7f77f4102e70 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.497+0000 7f77fb9a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f410da10 con 0x7f77f40fb4a0 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.497+0000 7f77f8f40700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 0x7f77f4113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.497+0000 7f77f8f40700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 0x7f77f4113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38508/0 (socket says 172.21.15.32:38508) 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.497+0000 7f77f8f40700 1 -- 172.21.15.32:0/2807644449 learned_addr learned my addr 172.21.15.32:0/2807644449 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77f9741700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f40fb4a0 0x7f77f4113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77f8f40700 1 -- 172.21.15.32:0/2807644449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f40fb4a0 msgr2=0x7f77f4113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77f8f40700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f40fb4a0 0x7f77f4113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77f8f40700 1 -- 172.21.15.32:0/2807644449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f0005040 con 0x7f77f4102e70 2024-04-06T14:44:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77f8f40700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 0x7f77f4113fd0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f77e400b790 tx=0x7f77e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77e9ffb700 1 -- 172.21.15.32:0/2807644449 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77e400f770 con 0x7f77f4102e70 2024-04-06T14:44:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77e9ffb700 1 -- 172.21.15.32:0/2807644449 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77e400fd80 con 0x7f77f4102e70 2024-04-06T14:44:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f410dcf0 con 0x7f77f4102e70 2024-04-06T14:44:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77e9ffb700 1 -- 172.21.15.32:0/2807644449 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77e400d310 con 0x7f77f4102e70 2024-04-06T14:44:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.498+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f410e2b0 con 0x7f77f4102e70 2024-04-06T14:44:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.499+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f4100110 con 0x7f77f4102e70 2024-04-06T14:44:29.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.500+0000 7f77e9ffb700 1 -- 172.21.15.32:0/2807644449 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f77e4010420 con 0x7f77f4102e70 2024-04-06T14:44:29.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.500+0000 7f77e9ffb700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f77e006cc30 0x7f77e006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:29.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.500+0000 7f77e9ffb700 1 -- 172.21.15.32:0/2807644449 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f77e408c6f0 con 0x7f77f4102e70 2024-04-06T14:44:29.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.500+0000 7f77f9741700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f77e006cc30 0x7f77e006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:29.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.501+0000 7f77f9741700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f77e006cc30 0x7f77e006f0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f77f0000c00 tx=0x7f77f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.504+0000 7f77e9ffb700 1 -- 172.21.15.32:0/2807644449 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f77e405a280 con 0x7f77f4102e70 2024-04-06T14:44:29.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:29 smithi032 ceph-mon[147414]: pgmap v731: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:29.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:29 smithi032 ceph-mon[147414]: from='client.35346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:44:29.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:29 smithi032 ceph-mon[147414]: from='client.35350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:44:29.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:29 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/529219655' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:44:29.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.751+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f77f4065380 con 0x7f77f4102e70 2024-04-06T14:44:29.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.752+0000 7f77e9ffb700 1 -- 172.21.15.32:0/2807644449 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f77e4059e10 con 0x7f77f4102e70 2024-04-06T14:44:29.743 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:44:29.743 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:44:29.743 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:44:29.743 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:44:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:44:29.743 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:44:29.743 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:44:29.743 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:44:29.743 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:44:29.744 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:44:29.745 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:44:29.746 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:44:29.746 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f77e006cc30 msgr2=0x7f77e006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f77e006cc30 0x7f77e006f0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f77f0000c00 tx=0x7f77f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 msgr2=0x7f77f4113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 0x7f77f4113fd0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f77e400b790 tx=0x7f77e4009250 comp rx=0 tx=0).stop 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 shutdown_connections 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f77e006cc30 0x7f77e006f0e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f40fb4a0 0x7f77f4113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 --2- 172.21.15.32:0/2807644449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f4102e70 0x7f77f4113fd0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 >> 172.21.15.32:0/2807644449 conn(0x7f77f40714c0 msgr2=0x7f77f40fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 shutdown_connections 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.754+0000 7f77fb9a5700 1 -- 172.21.15.32:0/2807644449 wait complete. 2024-04-06T14:44:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:44:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1123411124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fe7c0 msgr2=0x7f5e48106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 --2- 172.21.15.32:0/1123411124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fe7c0 0x7f5e48106e00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5e44004940 tx=0x7f5e44009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1123411124 shutdown_connections 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 --2- 172.21.15.32:0/1123411124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fe7c0 0x7f5e48106e00 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 --2- 172.21.15.32:0/1123411124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e480fde20 0x7f5e480fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1123411124 >> 172.21.15.32:0/1123411124 conn(0x7f5e480f78d0 msgr2=0x7f5e480f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1123411124 shutdown_connections 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1123411124 wait complete. 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 Processor -- start 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.859+0000 7f5e4fc09700 1 -- start start 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4fc09700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fde20 0x7f5e480fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4fc09700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e480fe7c0 0x7f5e480fc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4fc09700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480fc720 con 0x7f5e480fde20 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4fc09700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480fc860 con 0x7f5e480fe7c0 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4d9a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fde20 0x7f5e480fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4d9a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fde20 0x7f5e480fbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38538/0 (socket says 172.21.15.32:38538) 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4d9a5700 1 -- 172.21.15.32:0/1565579830 learned_addr learned my addr 172.21.15.32:0/1565579830 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:44:29.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4d1a4700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e480fe7c0 0x7f5e480fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4d9a5700 1 -- 172.21.15.32:0/1565579830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e480fe7c0 msgr2=0x7f5e480fc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4d9a5700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e480fe7c0 0x7f5e480fc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4d9a5700 1 -- 172.21.15.32:0/1565579830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44005040 con 0x7f5e480fde20 2024-04-06T14:44:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4d9a5700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fde20 0x7f5e480fbc50 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800c8c0 tx=0x7f5e380044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e3e7fc700 1 -- 172.21.15.32:0/1565579830 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e3800a980 con 0x7f5e480fde20 2024-04-06T14:44:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e3e7fc700 1 -- 172.21.15.32:0/1565579830 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e3800ce60 con 0x7f5e480fde20 2024-04-06T14:44:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e48119df0 con 0x7f5e480fde20 2024-04-06T14:44:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.860+0000 7f5e3e7fc700 1 -- 172.21.15.32:0/1565579830 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e380113a0 con 0x7f5e480fde20 2024-04-06T14:44:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.861+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4811a1f0 con 0x7f5e480fde20 2024-04-06T14:44:29.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.861+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e48066770 con 0x7f5e480fde20 2024-04-06T14:44:29.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.862+0000 7f5e3e7fc700 1 -- 172.21.15.32:0/1565579830 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5e38012400 con 0x7f5e480fde20 2024-04-06T14:44:29.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.862+0000 7f5e3e7fc700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5e3406cc20 0x7f5e3406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:29.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.863+0000 7f5e3e7fc700 1 -- 172.21.15.32:0/1565579830 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5e3808c170 con 0x7f5e480fde20 2024-04-06T14:44:29.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.863+0000 7f5e4d1a4700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5e3406cc20 0x7f5e3406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:29.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.863+0000 7f5e4d1a4700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5e3406cc20 0x7f5e3406f0d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400b3b0 tx=0x7f5e44009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:29.867+0000 7f5e3e7fc700 1 -- 172.21.15.32:0/1565579830 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5e380561b0 con 0x7f5e480fde20 2024-04-06T14:44:29.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:29 smithi086 ceph-mon[110975]: pgmap v731: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:29.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:29 smithi086 ceph-mon[110975]: from='client.35346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:44:29.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:29 smithi086 ceph-mon[110975]: from='client.35350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:44:29.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:29 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/529219655' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:44:30.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.090+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e480fd340 con 0x7f5e3406cc20 2024-04-06T14:44:30.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.091+0000 7f5e3e7fc700 1 -- 172.21.15.32:0/1565579830 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5e480fd340 con 0x7f5e3406cc20 2024-04-06T14:44:30.082 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:44:30.082 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:44:30.082 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:44:30.082 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:44:30.082 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:44:30.082 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:44:30.082 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:44:30.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:44:30.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:44:30.083 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:44:30.083 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:44:30.083 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:44:30.083 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:44:30.083 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:44:30.083 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:44:30.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5e3406cc20 msgr2=0x7f5e3406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:30.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5e3406cc20 0x7f5e3406f0d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400b3b0 tx=0x7f5e44009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:30.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fde20 msgr2=0x7f5e480fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:30.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fde20 0x7f5e480fbc50 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800c8c0 tx=0x7f5e380044c0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 shutdown_connections 2024-04-06T14:44:30.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5e3406cc20 0x7f5e3406f0d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e480fde20 0x7f5e480fbc50 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 --2- 172.21.15.32:0/1565579830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e480fe7c0 0x7f5e480fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 >> 172.21.15.32:0/1565579830 conn(0x7f5e480f78d0 msgr2=0x7f5e480f8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:30.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 shutdown_connections 2024-04-06T14:44:30.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.093+0000 7f5e4fc09700 1 -- 172.21.15.32:0/1565579830 wait complete. 2024-04-06T14:44:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.196+0000 7f8adcd5c700 1 -- 172.21.15.32:0/881654922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 msgr2=0x7f8ad8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.196+0000 7f8adcd5c700 1 --2- 172.21.15.32:0/881654922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 0x7f8ad8102140 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8acc004940 tx=0x7f8acc009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.196+0000 7f8adcd5c700 1 -- 172.21.15.32:0/881654922 shutdown_connections 2024-04-06T14:44:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.196+0000 7f8adcd5c700 1 --2- 172.21.15.32:0/881654922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad8102710 0x7f8ad80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.196+0000 7f8adcd5c700 1 --2- 172.21.15.32:0/881654922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 0x7f8ad8102140 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.196+0000 7f8adcd5c700 1 -- 172.21.15.32:0/881654922 >> 172.21.15.32:0/881654922 conn(0x7f8ad8072bc0 msgr2=0x7f8ad8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.196+0000 7f8adcd5c700 1 -- 172.21.15.32:0/881654922 shutdown_connections 2024-04-06T14:44:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.196+0000 7f8adcd5c700 1 -- 172.21.15.32:0/881654922 wait complete. 2024-04-06T14:44:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.197+0000 7f8adcd5c700 1 Processor -- start 2024-04-06T14:44:30.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.197+0000 7f8adcd5c700 1 -- start start 2024-04-06T14:44:30.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.197+0000 7f8adcd5c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 0x7f8ad810ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:30.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8adcd5c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad8102710 0x7f8ad810ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:30.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8adcd5c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad810f670 con 0x7f8ad8101d70 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8adcd5c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad810f7b0 con 0x7f8ad8102710 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ad659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 0x7f8ad810ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ad659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 0x7f8ad810ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38552/0 (socket says 172.21.15.32:38552) 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ad659c700 1 -- 172.21.15.32:0/1769896706 learned_addr learned my addr 172.21.15.32:0/1769896706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ad5d9b700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad8102710 0x7f8ad810ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ad659c700 1 -- 172.21.15.32:0/1769896706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad8102710 msgr2=0x7f8ad810ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ad659c700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad8102710 0x7f8ad810ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ad659c700 1 -- 172.21.15.32:0/1769896706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc005040 con 0x7f8ad8101d70 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ad5d9b700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad8102710 0x7f8ad810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ad659c700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 0x7f8ad810ea50 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009180 tx=0x7f8acc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ac6ffd700 1 -- 172.21.15.32:0/1769896706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8acc00e040 con 0x7f8ad8101d70 2024-04-06T14:44:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ac6ffd700 1 -- 172.21.15.32:0/1769896706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8acc004290 con 0x7f8ad8101d70 2024-04-06T14:44:30.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad8117ac0 con 0x7f8ad8101d70 2024-04-06T14:44:30.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8ac6ffd700 1 -- 172.21.15.32:0/1769896706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8acc01c3f0 con 0x7f8ad8101d70 2024-04-06T14:44:30.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.198+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad8118050 con 0x7f8ad8101d70 2024-04-06T14:44:30.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.199+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad80ff080 con 0x7f8ad8101d70 2024-04-06T14:44:30.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.200+0000 7f8ac6ffd700 1 -- 172.21.15.32:0/1769896706 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8acc013790 con 0x7f8ad8101d70 2024-04-06T14:44:30.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.200+0000 7f8ac6ffd700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8ac006cc20 0x7f8ac006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:44:30.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.200+0000 7f8ac6ffd700 1 -- 172.21.15.32:0/1769896706 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8acc08d200 con 0x7f8ad8101d70 2024-04-06T14:44:30.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.200+0000 7f8ad5d9b700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8ac006cc20 0x7f8ac006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:44:30.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.201+0000 7f8ad5d9b700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8ac006cc20 0x7f8ac006f0d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8066000 tx=0x7f8ac8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:44:30.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.205+0000 7f8ac6ffd700 1 -- 172.21.15.32:0/1769896706 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8acc057300 con 0x7f8ad8101d70 2024-04-06T14:44:30.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.489+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8ad804bed0 con 0x7f8ad8101d70 2024-04-06T14:44:30.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.490+0000 7f8ac6ffd700 1 -- 172.21.15.32:0/1769896706 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f8acc013420 con 0x7f8ad8101d70 2024-04-06T14:44:30.481 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:44:30.481 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:44:30.481 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:44:30.481 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:44:30.481 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:44:30.481 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:44:30.481 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:44:30.481 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:44:30.482 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:44:30.483 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:44:30.483 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:44:30.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8ac006cc20 msgr2=0x7f8ac006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:30.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8ac006cc20 0x7f8ac006f0d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8066000 tx=0x7f8ac8008040 comp rx=0 tx=0).stop 2024-04-06T14:44:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 msgr2=0x7f8ad810ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:44:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 0x7f8ad810ea50 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009180 tx=0x7f8acc009e30 comp rx=0 tx=0).stop 2024-04-06T14:44:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 shutdown_connections 2024-04-06T14:44:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8ac006cc20 0x7f8ac006f0d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad8101d70 0x7f8ad810ea50 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 --2- 172.21.15.32:0/1769896706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad8102710 0x7f8ad810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:44:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 >> 172.21.15.32:0/1769896706 conn(0x7f8ad8072bc0 msgr2=0x7f8ad80faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:44:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 shutdown_connections 2024-04-06T14:44:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:44:30.492+0000 7f8adcd5c700 1 -- 172.21.15.32:0/1769896706 wait complete. 2024-04-06T14:44:30.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:30 smithi032 ceph-mon[147414]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:44:30.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:30 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2807644449' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:44:30.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:30 smithi086 ceph-mon[110975]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:44:30.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:30 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2807644449' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:44:31.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:31 smithi032 ceph-mon[147414]: from='client.35366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:44:31.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:31 smithi032 ceph-mon[147414]: pgmap v732: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:31.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:31 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1769896706' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:44:31.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:31 smithi086 ceph-mon[110975]: from='client.35366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:44:31.722 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:31 smithi086 ceph-mon[110975]: pgmap v732: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:31.722 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:31 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1769896706' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:44:33.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:33 smithi032 ceph-mon[147414]: pgmap v733: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:33.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:33 smithi086 ceph-mon[110975]: pgmap v733: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:35.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:35 smithi032 ceph-mon[147414]: pgmap v734: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:35.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:35 smithi086 ceph-mon[110975]: pgmap v734: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:37.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:37 smithi032 ceph-mon[147414]: pgmap v735: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:37.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:37 smithi086 ceph-mon[110975]: pgmap v735: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:38.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:44:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:44:39.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:39 smithi032 ceph-mon[147414]: pgmap v736: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:39.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:39 smithi086 ceph-mon[110975]: pgmap v736: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:41.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:41 smithi032 ceph-mon[147414]: pgmap v737: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:41.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:41 smithi086 ceph-mon[110975]: pgmap v737: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:43.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:43 smithi032 ceph-mon[147414]: pgmap v738: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:43.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:43 smithi086 ceph-mon[110975]: pgmap v738: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:45.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:45 smithi032 ceph-mon[147414]: pgmap v739: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:45.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:45 smithi086 ceph-mon[110975]: pgmap v739: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:47.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:47 smithi032 ceph-mon[147414]: pgmap v740: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:47.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:47 smithi086 ceph-mon[110975]: pgmap v740: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:49.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:49 smithi032 ceph-mon[147414]: pgmap v741: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:49.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:49 smithi086 ceph-mon[110975]: pgmap v741: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:51.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:51 smithi032 ceph-mon[147414]: pgmap v742: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:51.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:51 smithi086 ceph-mon[110975]: pgmap v742: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:53.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:53 smithi032 ceph-mon[147414]: pgmap v743: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:53.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:44:53.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:53 smithi086 ceph-mon[110975]: pgmap v743: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:53.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:44:55.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:55 smithi032 ceph-mon[147414]: pgmap v744: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:55.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:55 smithi086 ceph-mon[110975]: pgmap v744: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:57.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:57 smithi032 ceph-mon[147414]: pgmap v745: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:57.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:57 smithi086 ceph-mon[110975]: pgmap v745: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:59.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:44:59 smithi032 ceph-mon[147414]: pgmap v746: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:44:59.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:44:59 smithi086 ceph-mon[110975]: pgmap v746: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:00.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.609+0000 7f1beddc1700 1 -- 172.21.15.32:0/4204897963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 msgr2=0x7f1be80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:00.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.609+0000 7f1beddc1700 1 --2- 172.21.15.32:0/4204897963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 0x7f1be80fbd80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc004940 tx=0x7f1bdc009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:00.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.609+0000 7f1beddc1700 1 -- 172.21.15.32:0/4204897963 shutdown_connections 2024-04-06T14:45:00.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.609+0000 7f1beddc1700 1 --2- 172.21.15.32:0/4204897963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 0x7f1be80fbd80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.609+0000 7f1beddc1700 1 --2- 172.21.15.32:0/4204897963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1be8101d70 0x7f1be8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.609+0000 7f1beddc1700 1 -- 172.21.15.32:0/4204897963 >> 172.21.15.32:0/4204897963 conn(0x7f1be8072bc0 msgr2=0x7f1be8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:00.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.609+0000 7f1beddc1700 1 -- 172.21.15.32:0/4204897963 shutdown_connections 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.609+0000 7f1beddc1700 1 -- 172.21.15.32:0/4204897963 wait complete. 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.610+0000 7f1beddc1700 1 Processor -- start 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.610+0000 7f1beddc1700 1 -- start start 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.610+0000 7f1beddc1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1be8101d70 0x7f1be81129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.610+0000 7f1beddc1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 0x7f1be8112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.610+0000 7f1beddc1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be8113550 con 0x7f1be8102710 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.610+0000 7f1beddc1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be810c940 con 0x7f1be8101d70 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.610+0000 7f1be6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 0x7f1be8112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.610+0000 7f1be6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 0x7f1be8112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35268/0 (socket says 172.21.15.32:35268) 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.610+0000 7f1be6ffd700 1 -- 172.21.15.32:0/764631294 learned_addr learned my addr 172.21.15.32:0/764631294 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1be77fe700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1be8101d70 0x7f1be81129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1be6ffd700 1 -- 172.21.15.32:0/764631294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1be8101d70 msgr2=0x7f1be81129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1be6ffd700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1be8101d70 0x7f1be81129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1be6ffd700 1 -- 172.21.15.32:0/764631294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc005040 con 0x7f1be8102710 2024-04-06T14:45:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1be6ffd700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 0x7f1be8112f00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc009390 tx=0x7f1bdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1bcffff700 1 -- 172.21.15.32:0/764631294 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bdc00e040 con 0x7f1be8102710 2024-04-06T14:45:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be810cbc0 con 0x7f1be8102710 2024-04-06T14:45:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1bcffff700 1 -- 172.21.15.32:0/764631294 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bdc013420 con 0x7f1be8102710 2024-04-06T14:45:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1bcffff700 1 -- 172.21.15.32:0/764631294 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bdc01c630 con 0x7f1be8102710 2024-04-06T14:45:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.611+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be810d180 con 0x7f1be8102710 2024-04-06T14:45:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.612+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be80ff080 con 0x7f1be8102710 2024-04-06T14:45:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.613+0000 7f1bcffff700 1 -- 172.21.15.32:0/764631294 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1bdc013a00 con 0x7f1be8102710 2024-04-06T14:45:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.613+0000 7f1bcffff700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bd006cc30 0x7f1bd006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.613+0000 7f1bcffff700 1 -- 172.21.15.32:0/764631294 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1bdc08d320 con 0x7f1be8102710 2024-04-06T14:45:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.613+0000 7f1be77fe700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bd006cc30 0x7f1bd006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.614+0000 7f1be77fe700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bd006cc30 0x7f1bd006f0e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8004990 tx=0x7f1bd8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:00.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.618+0000 7f1bcffff700 1 -- 172.21.15.32:0/764631294 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1bdc05adf0 con 0x7f1be8102710 2024-04-06T14:45:00.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.838+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1be805f690 con 0x7f1bd006cc30 2024-04-06T14:45:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.839+0000 7f1bcffff700 1 -- 172.21.15.32:0/764631294 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1be805f690 con 0x7f1bd006cc30 2024-04-06T14:45:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.841+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bd006cc30 msgr2=0x7f1bd006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.841+0000 7f1beddc1700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bd006cc30 0x7f1bd006f0e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8004990 tx=0x7f1bd8009ee0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.841+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 msgr2=0x7f1be8112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.841+0000 7f1beddc1700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 0x7f1be8112f00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc009390 tx=0x7f1bdc009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.841+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 shutdown_connections 2024-04-06T14:45:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.841+0000 7f1beddc1700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bd006cc30 0x7f1bd006f0e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.841+0000 7f1beddc1700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1be8101d70 0x7f1be81129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.841+0000 7f1beddc1700 1 --2- 172.21.15.32:0/764631294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1be8102710 0x7f1be8112f00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.841+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 >> 172.21.15.32:0/764631294 conn(0x7f1be8072bc0 msgr2=0x7f1be80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.842+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 shutdown_connections 2024-04-06T14:45:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.842+0000 7f1beddc1700 1 -- 172.21.15.32:0/764631294 wait complete. 2024-04-06T14:45:00.842 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:45:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.946+0000 7fb401805700 1 -- 172.21.15.32:0/1678685637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 msgr2=0x7fb3fc0fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.946+0000 7fb401805700 1 --2- 172.21.15.32:0/1678685637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 0x7fb3fc0fe1d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0004940 tx=0x7fb3f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.946+0000 7fb401805700 1 -- 172.21.15.32:0/1678685637 shutdown_connections 2024-04-06T14:45:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.946+0000 7fb401805700 1 --2- 172.21.15.32:0/1678685637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc0fe7a0 0x7fb3fc106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.946+0000 7fb401805700 1 --2- 172.21.15.32:0/1678685637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 0x7fb3fc0fe1d0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.946+0000 7fb401805700 1 -- 172.21.15.32:0/1678685637 >> 172.21.15.32:0/1678685637 conn(0x7fb3fc0f7880 msgr2=0x7fb3fc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.946+0000 7fb401805700 1 -- 172.21.15.32:0/1678685637 shutdown_connections 2024-04-06T14:45:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.946+0000 7fb401805700 1 -- 172.21.15.32:0/1678685637 wait complete. 2024-04-06T14:45:00.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb401805700 1 Processor -- start 2024-04-06T14:45:00.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb401805700 1 -- start start 2024-04-06T14:45:00.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb401805700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 0x7fb3fc0fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:00.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb401805700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc0fe7a0 0x7fb3fc0fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:00.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb401805700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0fc790 con 0x7fb3fc0fde00 2024-04-06T14:45:00.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb401805700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0fc8d0 con 0x7fb3fc0fe7a0 2024-04-06T14:45:00.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb3faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 0x7fb3fc0fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb3faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 0x7fb3fc0fbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35290/0 (socket says 172.21.15.32:35290) 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb3faffd700 1 -- 172.21.15.32:0/736036339 learned_addr learned my addr 172.21.15.32:0/736036339 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb3fa7fc700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc0fe7a0 0x7fb3fc0fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb3faffd700 1 -- 172.21.15.32:0/736036339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc0fe7a0 msgr2=0x7fb3fc0fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb3faffd700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc0fe7a0 0x7fb3fc0fc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb3faffd700 1 -- 172.21.15.32:0/736036339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f0005040 con 0x7fb3fc0fde00 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.947+0000 7fb3faffd700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 0x7fb3fc0fbcc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009380 tx=0x7fb3f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.948+0000 7fb3e37fe700 1 -- 172.21.15.32:0/736036339 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3f000e040 con 0x7fb3fc0fde00 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.948+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc0fcb20 con 0x7fb3fc0fde00 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.948+0000 7fb3e37fe700 1 -- 172.21.15.32:0/736036339 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3f0004290 con 0x7fb3fc0fde00 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.948+0000 7fb3e37fe700 1 -- 172.21.15.32:0/736036339 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3f001c3f0 con 0x7fb3fc0fde00 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.948+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc11a4a0 con 0x7fb3fc0fde00 2024-04-06T14:45:00.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.949+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc0666f0 con 0x7fb3fc0fde00 2024-04-06T14:45:00.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.949+0000 7fb3e37fe700 1 -- 172.21.15.32:0/736036339 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb3f0013790 con 0x7fb3fc0fde00 2024-04-06T14:45:00.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.949+0000 7fb3e37fe700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3e406cc20 0x7fb3e406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:00.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.950+0000 7fb3e37fe700 1 -- 172.21.15.32:0/736036339 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb3f008d3a0 con 0x7fb3fc0fde00 2024-04-06T14:45:00.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.950+0000 7fb3fa7fc700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3e406cc20 0x7fb3e406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:00.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.950+0000 7fb3fa7fc700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3e406cc20 0x7fb3e406f0d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0fd600 tx=0x7fb3ec009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:00.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:00.954+0000 7fb3e37fe700 1 -- 172.21.15.32:0/736036339 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb3f00573e0 con 0x7fb3fc0fde00 2024-04-06T14:45:01.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.176+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3fc04bf70 con 0x7fb3e406cc20 2024-04-06T14:45:01.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.177+0000 7fb3e37fe700 1 -- 172.21.15.32:0/736036339 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb3ec004e30 con 0x7fb3e406cc20 2024-04-06T14:45:01.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.179+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3e406cc20 msgr2=0x7fb3e406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.179+0000 7fb401805700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3e406cc20 0x7fb3e406f0d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0fd600 tx=0x7fb3ec009ee0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.179+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 msgr2=0x7fb3fc0fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.179+0000 7fb401805700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 0x7fb3fc0fbcc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009380 tx=0x7fb3f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:01.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.179+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 shutdown_connections 2024-04-06T14:45:01.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.179+0000 7fb401805700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb3e406cc20 0x7fb3e406f0d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.179+0000 7fb401805700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3fc0fde00 0x7fb3fc0fbcc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.179+0000 7fb401805700 1 --2- 172.21.15.32:0/736036339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc0fe7a0 0x7fb3fc0fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.179+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 >> 172.21.15.32:0/736036339 conn(0x7fb3fc0f7880 msgr2=0x7fb3fc0f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:01.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.180+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 shutdown_connections 2024-04-06T14:45:01.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.180+0000 7fb401805700 1 -- 172.21.15.32:0/736036339 wait complete. 2024-04-06T14:45:01.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.282+0000 7f4415782700 1 -- 172.21.15.32:0/2053835473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 msgr2=0x7f44100fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.282+0000 7f4415782700 1 --2- 172.21.15.32:0/2053835473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 0x7f44100fce10 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f43f8004940 tx=0x7f43f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:01.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.282+0000 7f4415782700 1 -- 172.21.15.32:0/2053835473 shutdown_connections 2024-04-06T14:45:01.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.282+0000 7f4415782700 1 --2- 172.21.15.32:0/2053835473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 0x7f44100fce10 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.282+0000 7f4415782700 1 --2- 172.21.15.32:0/2053835473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4410102e00 0x7f44101031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.282+0000 7f4415782700 1 -- 172.21.15.32:0/2053835473 >> 172.21.15.32:0/2053835473 conn(0x7f44100f89b0 msgr2=0x7f44100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:01.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.282+0000 7f4415782700 1 -- 172.21.15.32:0/2053835473 shutdown_connections 2024-04-06T14:45:01.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.282+0000 7f4415782700 1 -- 172.21.15.32:0/2053835473 wait complete. 2024-04-06T14:45:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.283+0000 7f4415782700 1 Processor -- start 2024-04-06T14:45:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.283+0000 7f4415782700 1 -- start start 2024-04-06T14:45:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f4415782700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4410102e00 0x7f44101137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f4415782700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 0x7f4410113d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f4415782700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4410114380 con 0x7f44101037a0 2024-04-06T14:45:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f4415782700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441010d9c0 con 0x7f4410102e00 2024-04-06T14:45:01.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f440e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 0x7f4410113d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f440e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 0x7f4410113d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35310/0 (socket says 172.21.15.32:35310) 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f440e7fc700 1 -- 172.21.15.32:0/203857 learned_addr learned my addr 172.21.15.32:0/203857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f440effd700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4410102e00 0x7f44101137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f440e7fc700 1 -- 172.21.15.32:0/203857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4410102e00 msgr2=0x7f44101137f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f440e7fc700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4410102e00 0x7f44101137f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f440e7fc700 1 -- 172.21.15.32:0/203857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f8005040 con 0x7f44101037a0 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f440e7fc700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 0x7f4410113d30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f43f8000c00 tx=0x7f43f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f44077fe700 1 -- 172.21.15.32:0/203857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43f800e070 con 0x7f44101037a0 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f4415782700 1 -- 172.21.15.32:0/203857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441010dc40 con 0x7f44101037a0 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f44077fe700 1 -- 172.21.15.32:0/203857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43f80093b0 con 0x7f44101037a0 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f44077fe700 1 -- 172.21.15.32:0/203857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43f801ccb0 con 0x7f44101037a0 2024-04-06T14:45:01.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.284+0000 7f4415782700 1 -- 172.21.15.32:0/203857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441010e1a0 con 0x7f44101037a0 2024-04-06T14:45:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.285+0000 7f4415782700 1 -- 172.21.15.32:0/203857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44100fe550 con 0x7f44101037a0 2024-04-06T14:45:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.286+0000 7f44077fe700 1 -- 172.21.15.32:0/203857 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f43f801c490 con 0x7f44101037a0 2024-04-06T14:45:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.286+0000 7f44077fe700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f43fc06cc20 0x7f43fc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:01.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.286+0000 7f44077fe700 1 -- 172.21.15.32:0/203857 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f43f808d520 con 0x7f44101037a0 2024-04-06T14:45:01.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.286+0000 7f440effd700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f43fc06cc20 0x7f43fc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:01.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.287+0000 7f440effd700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f43fc06cc20 0x7f43fc06f0d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4400009b80 tx=0x7f440000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:01.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.291+0000 7f44077fe700 1 -- 172.21.15.32:0/203857 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f43f805af70 con 0x7f44101037a0 2024-04-06T14:45:01.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.502+0000 7f4415782700 1 -- 172.21.15.32:0/203857 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4410109dc0 con 0x7f43fc06cc20 2024-04-06T14:45:01.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.512+0000 7f44077fe700 1 -- 172.21.15.32:0/203857 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4410109dc0 con 0x7f43fc06cc20 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (25m) 3m ago 36m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (22m) 3m ago 36m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (22m) 3m ago 35m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (25m) 3m ago 36m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (19m) 3m ago 30m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (18m) 3m ago 30m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (18m) 3m ago 30m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (3m) 3m ago 30m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (27m) 3m ago 38m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (26m) 3m ago 35m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:45:01.503 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (23m) 3m ago 38m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (23m) 3m ago 35m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (26m) 3m ago 36m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (26m) 3m ago 35m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (22m) 3m ago 34m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (21m) 3m ago 34m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (21m) 3m ago 33m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (20m) 3m ago 33m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (19m) 3m ago 32m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (19m) 3m ago 32m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:45:01.504 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (25m) 3m ago 36m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 -- 172.21.15.32:0/203857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f43fc06cc20 msgr2=0x7f43fc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f43fc06cc20 0x7f43fc06f0d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4400009b80 tx=0x7f440000b040 comp rx=0 tx=0).stop 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 -- 172.21.15.32:0/203857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 msgr2=0x7f4410113d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 0x7f4410113d30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f43f8000c00 tx=0x7f43f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 -- 172.21.15.32:0/203857 shutdown_connections 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f43fc06cc20 0x7f43fc06f0d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4410102e00 0x7f44101137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 --2- 172.21.15.32:0/203857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44101037a0 0x7f4410113d30 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 -- 172.21.15.32:0/203857 >> 172.21.15.32:0/203857 conn(0x7f44100f89b0 msgr2=0x7f44100f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 -- 172.21.15.32:0/203857 shutdown_connections 2024-04-06T14:45:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.514+0000 7f4415782700 1 -- 172.21.15.32:0/203857 wait complete. 2024-04-06T14:45:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.616+0000 7f96f38ac700 1 -- 172.21.15.32:0/2881830526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 msgr2=0x7f96ec102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.616+0000 7f96f38ac700 1 --2- 172.21.15.32:0/2881830526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 0x7f96ec102140 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f96e8004940 tx=0x7f96e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.616+0000 7f96f38ac700 1 -- 172.21.15.32:0/2881830526 shutdown_connections 2024-04-06T14:45:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.616+0000 7f96f38ac700 1 --2- 172.21.15.32:0/2881830526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec102710 0x7f96ec0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.616+0000 7f96f38ac700 1 --2- 172.21.15.32:0/2881830526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 0x7f96ec102140 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.616+0000 7f96f38ac700 1 -- 172.21.15.32:0/2881830526 >> 172.21.15.32:0/2881830526 conn(0x7f96ec072bc0 msgr2=0x7f96ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.616+0000 7f96f38ac700 1 -- 172.21.15.32:0/2881830526 shutdown_connections 2024-04-06T14:45:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.616+0000 7f96f38ac700 1 -- 172.21.15.32:0/2881830526 wait complete. 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f38ac700 1 Processor -- start 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f38ac700 1 -- start start 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f38ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 0x7f96ec1129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f38ac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec102710 0x7f96ec112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f38ac700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec113530 con 0x7f96ec101d70 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f38ac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec10c920 con 0x7f96ec102710 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f1648700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 0x7f96ec1129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f1648700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 0x7f96ec1129a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35330/0 (socket says 172.21.15.32:35330) 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f1648700 1 -- 172.21.15.32:0/3813141221 learned_addr learned my addr 172.21.15.32:0/3813141221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f1648700 1 -- 172.21.15.32:0/3813141221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec102710 msgr2=0x7f96ec112ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.617+0000 7f96f0e47700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec102710 0x7f96ec112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:01.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.618+0000 7f96f1648700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec102710 0x7f96ec112ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.618+0000 7f96f1648700 1 -- 172.21.15.32:0/3813141221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e8005040 con 0x7f96ec101d70 2024-04-06T14:45:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.618+0000 7f96f0e47700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec102710 0x7f96ec112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:45:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.618+0000 7f96f1648700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 0x7f96ec1129a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f96e800b3b0 tx=0x7f96e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.618+0000 7f96ddffb700 1 -- 172.21.15.32:0/3813141221 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e800e040 con 0x7f96ec101d70 2024-04-06T14:45:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.618+0000 7f96ddffb700 1 -- 172.21.15.32:0/3813141221 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96e8013420 con 0x7f96ec101d70 2024-04-06T14:45:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.618+0000 7f96ddffb700 1 -- 172.21.15.32:0/3813141221 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e801c630 con 0x7f96ec101d70 2024-04-06T14:45:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.618+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec10cba0 con 0x7f96ec101d70 2024-04-06T14:45:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.618+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec10d160 con 0x7f96ec101d70 2024-04-06T14:45:01.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.619+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec0ff080 con 0x7f96ec101d70 2024-04-06T14:45:01.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.620+0000 7f96ddffb700 1 -- 172.21.15.32:0/3813141221 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f96e8013970 con 0x7f96ec101d70 2024-04-06T14:45:01.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.620+0000 7f96ddffb700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f96d806cc20 0x7f96d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:01.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.620+0000 7f96ddffb700 1 -- 172.21.15.32:0/3813141221 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f96e808d1a0 con 0x7f96ec101d70 2024-04-06T14:45:01.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.620+0000 7f96f0e47700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f96d806cc20 0x7f96d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:01.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.621+0000 7f96f0e47700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f96d806cc20 0x7f96d806f0d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f96e0007fd0 tx=0x7f96e0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.625+0000 7f96ddffb700 1 -- 172.21.15.32:0/3813141221 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f96e805ad30 con 0x7f96ec101d70 2024-04-06T14:45:01.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:01 smithi032 ceph-mon[147414]: pgmap v747: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:01.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:01 smithi086 ceph-mon[110975]: pgmap v747: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.915+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f96ec10e040 con 0x7f96ec101d70 2024-04-06T14:45:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.915+0000 7f96ddffb700 1 -- 172.21.15.32:0/3813141221 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f96e805a8c0 con 0x7f96ec101d70 2024-04-06T14:45:01.906 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:45:01.906 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:45:01.906 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:45:01.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:45:01.906 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:45:01.906 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:45:01.907 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:45:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f96d806cc20 msgr2=0x7f96d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f96d806cc20 0x7f96d806f0d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f96e0007fd0 tx=0x7f96e0009040 comp rx=0 tx=0).stop 2024-04-06T14:45:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 msgr2=0x7f96ec1129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 0x7f96ec1129a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f96e800b3b0 tx=0x7f96e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 shutdown_connections 2024-04-06T14:45:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f96d806cc20 0x7f96d806f0d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec101d70 0x7f96ec1129a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 --2- 172.21.15.32:0/3813141221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec102710 0x7f96ec112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:01.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 >> 172.21.15.32:0/3813141221 conn(0x7f96ec072bc0 msgr2=0x7f96ec0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:01.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 shutdown_connections 2024-04-06T14:45:01.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:01.917+0000 7f96f38ac700 1 -- 172.21.15.32:0/3813141221 wait complete. 2024-04-06T14:45:02.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.021+0000 7f6137db4700 1 -- 172.21.15.32:0/1812499935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130101d70 msgr2=0x7f6130102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.021+0000 7f6137db4700 1 --2- 172.21.15.32:0/1812499935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130101d70 0x7f6130102140 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6124004940 tx=0x7f6124009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.021+0000 7f6137db4700 1 -- 172.21.15.32:0/1812499935 shutdown_connections 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.021+0000 7f6137db4700 1 --2- 172.21.15.32:0/1812499935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6130102710 0x7f61300fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.021+0000 7f6137db4700 1 --2- 172.21.15.32:0/1812499935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130101d70 0x7f6130102140 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.021+0000 7f6137db4700 1 -- 172.21.15.32:0/1812499935 >> 172.21.15.32:0/1812499935 conn(0x7f6130072bc0 msgr2=0x7f6130072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.021+0000 7f6137db4700 1 -- 172.21.15.32:0/1812499935 shutdown_connections 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.021+0000 7f6137db4700 1 -- 172.21.15.32:0/1812499935 wait complete. 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f6137db4700 1 Processor -- start 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f6137db4700 1 -- start start 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f6137db4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6130101d70 0x7f6130113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f6137db4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130102710 0x7f6130113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f6137db4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61301145f0 con 0x7f6130102710 2024-04-06T14:45:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f6137db4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613010d9e0 con 0x7f6130101d70 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f613534f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130102710 0x7f6130113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f613534f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130102710 0x7f6130113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35346/0 (socket says 172.21.15.32:35346) 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f613534f700 1 -- 172.21.15.32:0/2356510733 learned_addr learned my addr 172.21.15.32:0/2356510733 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f6135b50700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6130101d70 0x7f6130113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f613534f700 1 -- 172.21.15.32:0/2356510733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6130101d70 msgr2=0x7f6130113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f613534f700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6130101d70 0x7f6130113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.022+0000 7f613534f700 1 -- 172.21.15.32:0/2356510733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124005040 con 0x7f6130102710 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.023+0000 7f613534f700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130102710 0x7f6130113fa0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f612c002a50 tx=0x7f612c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.023+0000 7f61227fc700 1 -- 172.21.15.32:0/2356510733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f612c007cc0 con 0x7f6130102710 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.023+0000 7f61227fc700 1 -- 172.21.15.32:0/2356510733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f612c00f040 con 0x7f6130102710 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.023+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613010dcc0 con 0x7f6130102710 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.023+0000 7f61227fc700 1 -- 172.21.15.32:0/2356510733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f612c00d9a0 con 0x7f6130102710 2024-04-06T14:45:02.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.023+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613010e280 con 0x7f6130102710 2024-04-06T14:45:02.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.024+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61300fd4c0 con 0x7f6130102710 2024-04-06T14:45:02.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.024+0000 7f61227fc700 1 -- 172.21.15.32:0/2356510733 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f612c00dc60 con 0x7f6130102710 2024-04-06T14:45:02.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.025+0000 7f61227fc700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f611c06cc20 0x7f611c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:02.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.025+0000 7f61227fc700 1 -- 172.21.15.32:0/2356510733 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f612c08bf00 con 0x7f6130102710 2024-04-06T14:45:02.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.025+0000 7f6135b50700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f611c06cc20 0x7f611c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:02.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.025+0000 7f6135b50700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f611c06cc20 0x7f611c06f0d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6124004f90 tx=0x7f6124009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:02.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.029+0000 7f61227fc700 1 -- 172.21.15.32:0/2356510733 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f612c0598d0 con 0x7f6130102710 2024-04-06T14:45:02.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.281+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6130108d30 con 0x7f6130102710 2024-04-06T14:45:02.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.282+0000 7f61227fc700 1 -- 172.21.15.32:0/2356510733 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f612c059460 con 0x7f6130102710 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:45:02.273 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:45:02.274 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:45:02.275 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:45:02.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f611c06cc20 msgr2=0x7f611c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f611c06cc20 0x7f611c06f0d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6124004f90 tx=0x7f6124009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:02.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130102710 msgr2=0x7f6130113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130102710 0x7f6130113fa0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f612c002a50 tx=0x7f612c004600 comp rx=0 tx=0).stop 2024-04-06T14:45:02.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 shutdown_connections 2024-04-06T14:45:02.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f611c06cc20 0x7f611c06f0d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6130101d70 0x7f6130113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 --2- 172.21.15.32:0/2356510733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6130102710 0x7f6130113fa0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 >> 172.21.15.32:0/2356510733 conn(0x7f6130072bc0 msgr2=0x7f61300659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:02.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 shutdown_connections 2024-04-06T14:45:02.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.284+0000 7f6137db4700 1 -- 172.21.15.32:0/2356510733 wait complete. 2024-04-06T14:45:02.277 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:45:02.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 -- 172.21.15.32:0/804831072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec102710 msgr2=0x7f57ec0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 --2- 172.21.15.32:0/804831072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec102710 0x7f57ec0fbd80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f57e8004940 tx=0x7f57e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:02.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 -- 172.21.15.32:0/804831072 shutdown_connections 2024-04-06T14:45:02.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 --2- 172.21.15.32:0/804831072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec102710 0x7f57ec0fbd80 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 --2- 172.21.15.32:0/804831072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec101d70 0x7f57ec102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 -- 172.21.15.32:0/804831072 >> 172.21.15.32:0/804831072 conn(0x7f57ec072bc0 msgr2=0x7f57ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 -- 172.21.15.32:0/804831072 shutdown_connections 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 -- 172.21.15.32:0/804831072 wait complete. 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 Processor -- start 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.388+0000 7f57f415d700 1 -- start start 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f415d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec101d70 0x7f57ec10ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f415d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec102710 0x7f57ec10f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f415d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec10f640 con 0x7f57ec101d70 2024-04-06T14:45:02.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f415d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec10f780 con 0x7f57ec102710 2024-04-06T14:45:02.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f1ef9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec101d70 0x7f57ec10ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:02.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f1ef9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec101d70 0x7f57ec10ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35354/0 (socket says 172.21.15.32:35354) 2024-04-06T14:45:02.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f1ef9700 1 -- 172.21.15.32:0/3539621708 learned_addr learned my addr 172.21.15.32:0/3539621708 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:02.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f16f8700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec102710 0x7f57ec10f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:02.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f1ef9700 1 -- 172.21.15.32:0/3539621708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec102710 msgr2=0x7f57ec10f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f1ef9700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec102710 0x7f57ec10f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f1ef9700 1 -- 172.21.15.32:0/3539621708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e8005040 con 0x7f57ec101d70 2024-04-06T14:45:02.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f1ef9700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec101d70 0x7f57ec10ebc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c830 tx=0x7f57dc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:02.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57e27fc700 1 -- 172.21.15.32:0/3539621708 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57dc00a760 con 0x7f57ec101d70 2024-04-06T14:45:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57e27fc700 1 -- 172.21.15.32:0/3539621708 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57dc00ad70 con 0x7f57ec101d70 2024-04-06T14:45:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ec117a70 con 0x7f57ec101d70 2024-04-06T14:45:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57e27fc700 1 -- 172.21.15.32:0/3539621708 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57dc0113a0 con 0x7f57ec101d70 2024-04-06T14:45:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.389+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ec118030 con 0x7f57ec101d70 2024-04-06T14:45:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.390+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57ec0ff080 con 0x7f57ec101d70 2024-04-06T14:45:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.391+0000 7f57e27fc700 1 -- 172.21.15.32:0/3539621708 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f57dc012400 con 0x7f57ec101d70 2024-04-06T14:45:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.391+0000 7f57e27fc700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f57d806cc30 0x7f57d806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.391+0000 7f57e27fc700 1 -- 172.21.15.32:0/3539621708 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f57dc08c0b0 con 0x7f57ec101d70 2024-04-06T14:45:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.391+0000 7f57f16f8700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f57d806cc30 0x7f57d806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:02.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.392+0000 7f57f16f8700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f57d806cc30 0x7f57d806f0e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f57e800b020 tx=0x7f57e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:02.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.396+0000 7f57e27fc700 1 -- 172.21.15.32:0/3539621708 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f57dc0560f0 con 0x7f57ec101d70 2024-04-06T14:45:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.618+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f57ec04bed0 con 0x7f57d806cc30 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.619+0000 7f57e27fc700 1 -- 172.21.15.32:0/3539621708 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f57ec04bed0 con 0x7f57d806cc30 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:45:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:45:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:45:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:45:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:45:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:45:02.611 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:45:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.621+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f57d806cc30 msgr2=0x7f57d806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.621+0000 7f57f415d700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f57d806cc30 0x7f57d806f0e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f57e800b020 tx=0x7f57e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.621+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec101d70 msgr2=0x7f57ec10ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.621+0000 7f57f415d700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec101d70 0x7f57ec10ebc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c830 tx=0x7f57dc004470 comp rx=0 tx=0).stop 2024-04-06T14:45:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.621+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 shutdown_connections 2024-04-06T14:45:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.621+0000 7f57f415d700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f57d806cc30 0x7f57d806f0e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.621+0000 7f57f415d700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec101d70 0x7f57ec10ebc0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.621+0000 7f57f415d700 1 --2- 172.21.15.32:0/3539621708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec102710 0x7f57ec10f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.622+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 >> 172.21.15.32:0/3539621708 conn(0x7f57ec072bc0 msgr2=0x7f57ec0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.622+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 shutdown_connections 2024-04-06T14:45:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.622+0000 7f57f415d700 1 -- 172.21.15.32:0/3539621708 wait complete. 2024-04-06T14:45:02.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.724+0000 7f31e8504700 1 -- 172.21.15.32:0/390438861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 msgr2=0x7f31e01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.724+0000 7f31e8504700 1 --2- 172.21.15.32:0/390438861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 0x7f31e01031e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f31dc004940 tx=0x7f31dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:02.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.725+0000 7f31e8504700 1 -- 172.21.15.32:0/390438861 shutdown_connections 2024-04-06T14:45:02.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.725+0000 7f31e8504700 1 --2- 172.21.15.32:0/390438861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e01037b0 0x7f31e00fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.725+0000 7f31e8504700 1 --2- 172.21.15.32:0/390438861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 0x7f31e01031e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.725+0000 7f31e8504700 1 -- 172.21.15.32:0/390438861 >> 172.21.15.32:0/390438861 conn(0x7f31e00f8980 msgr2=0x7f31e00fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:02.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.725+0000 7f31e8504700 1 -- 172.21.15.32:0/390438861 shutdown_connections 2024-04-06T14:45:02.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.725+0000 7f31e8504700 1 -- 172.21.15.32:0/390438861 wait complete. 2024-04-06T14:45:02.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.725+0000 7f31e8504700 1 Processor -- start 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.725+0000 7f31e8504700 1 -- start start 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e8504700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 0x7f31e0113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e8504700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e01037b0 0x7f31e0114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e8504700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e01146a0 con 0x7f31e0102e10 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e8504700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e010da90 con 0x7f31e01037b0 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e62a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 0x7f31e0113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e62a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 0x7f31e0113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35378/0 (socket says 172.21.15.32:35378) 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e62a0700 1 -- 172.21.15.32:0/3833027004 learned_addr learned my addr 172.21.15.32:0/3833027004 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e5a9f700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e01037b0 0x7f31e0114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e62a0700 1 -- 172.21.15.32:0/3833027004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e01037b0 msgr2=0x7f31e0114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e62a0700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e01037b0 0x7f31e0114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e62a0700 1 -- 172.21.15.32:0/3833027004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc005040 con 0x7f31e0102e10 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e62a0700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 0x7f31e0113b10 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f31dc009360 tx=0x7f31dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31d6ffd700 1 -- 172.21.15.32:0/3833027004 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31dc00e040 con 0x7f31e0102e10 2024-04-06T14:45:02.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31d6ffd700 1 -- 172.21.15.32:0/3833027004 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31dc013420 con 0x7f31e0102e10 2024-04-06T14:45:02.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e010dd10 con 0x7f31e0102e10 2024-04-06T14:45:02.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.726+0000 7f31d6ffd700 1 -- 172.21.15.32:0/3833027004 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31dc01c5f0 con 0x7f31e0102e10 2024-04-06T14:45:02.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.727+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e010e2d0 con 0x7f31e0102e10 2024-04-06T14:45:02.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.728+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e00fe560 con 0x7f31e0102e10 2024-04-06T14:45:02.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.728+0000 7f31d6ffd700 1 -- 172.21.15.32:0/3833027004 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f31dc013590 con 0x7f31e0102e10 2024-04-06T14:45:02.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.728+0000 7f31d6ffd700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31cc06cc30 0x7f31cc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:02.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.729+0000 7f31e5a9f700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31cc06cc30 0x7f31cc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:02.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.729+0000 7f31d6ffd700 1 -- 172.21.15.32:0/3833027004 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f31dc08d2d0 con 0x7f31e0102e10 2024-04-06T14:45:02.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.729+0000 7f31e5a9f700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31cc06cc30 0x7f31cc06f0e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f31e010ec50 tx=0x7f31d0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:02.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:02 smithi032 ceph-mon[147414]: from='client.35374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:02.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:02 smithi032 ceph-mon[147414]: from='client.35378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:02.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:02 smithi032 ceph-mon[147414]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:02.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:02 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3813141221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:45:02.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:02 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2356510733' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:45:02.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:02.733+0000 7f31d6ffd700 1 -- 172.21.15.32:0/3833027004 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f31dc05ada0 con 0x7f31e0102e10 2024-04-06T14:45:02.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:02 smithi086 ceph-mon[110975]: from='client.35374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:02.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:02 smithi086 ceph-mon[110975]: from='client.35378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:02.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:02 smithi086 ceph-mon[110975]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:02.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:02 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3813141221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:45:02.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:02 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2356510733' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:45:03.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.020+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f31e004bed0 con 0x7f31e0102e10 2024-04-06T14:45:03.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.021+0000 7f31d6ffd700 1 -- 172.21.15.32:0/3833027004 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f31dc05a930 con 0x7f31e0102e10 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:45:03.012 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:45:03.013 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:45:03.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31cc06cc30 msgr2=0x7f31cc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:03.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31cc06cc30 0x7f31cc06f0e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f31e010ec50 tx=0x7f31d0009ee0 comp rx=0 tx=0).stop 2024-04-06T14:45:03.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 msgr2=0x7f31e0113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:03.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 0x7f31e0113b10 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f31dc009360 tx=0x7f31dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:03.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 shutdown_connections 2024-04-06T14:45:03.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f31cc06cc30 0x7f31cc06f0e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:03.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e0102e10 0x7f31e0113b10 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:03.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 --2- 172.21.15.32:0/3833027004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e01037b0 0x7f31e0114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:03.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 >> 172.21.15.32:0/3833027004 conn(0x7f31e00f8980 msgr2=0x7f31e00f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:03.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 shutdown_connections 2024-04-06T14:45:03.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:03.023+0000 7f31e8504700 1 -- 172.21.15.32:0/3833027004 wait complete. 2024-04-06T14:45:03.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:03 smithi032 ceph-mon[147414]: pgmap v748: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:03.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:03 smithi032 ceph-mon[147414]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:03.721 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:03 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3833027004' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:45:03.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:03 smithi086 ceph-mon[110975]: pgmap v748: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:03.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:03 smithi086 ceph-mon[110975]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:03.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:03 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3833027004' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:45:05.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:05 smithi032 ceph-mon[147414]: pgmap v749: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:05.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:05 smithi086 ceph-mon[110975]: pgmap v749: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:07.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:07 smithi032 ceph-mon[147414]: pgmap v750: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:07.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:07 smithi086 ceph-mon[110975]: pgmap v750: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:08.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:45:08.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:45:09.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:09 smithi032 ceph-mon[147414]: pgmap v751: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:09.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:09 smithi086 ceph-mon[110975]: pgmap v751: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:11.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:11 smithi032 ceph-mon[147414]: pgmap v752: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:11.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:11 smithi086 ceph-mon[110975]: pgmap v752: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:13.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:13 smithi032 ceph-mon[147414]: pgmap v753: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:13.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:13 smithi086 ceph-mon[110975]: pgmap v753: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:15.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:15 smithi086 ceph-mon[110975]: pgmap v754: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:15.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:15 smithi032 ceph-mon[147414]: pgmap v754: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:17.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:17 smithi086 ceph-mon[110975]: pgmap v755: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:17.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:17 smithi032 ceph-mon[147414]: pgmap v755: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:19.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:19 smithi086 ceph-mon[110975]: pgmap v756: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:19.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:19 smithi032 ceph-mon[147414]: pgmap v756: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:21.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:21 smithi086 ceph-mon[110975]: pgmap v757: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:21.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:21 smithi032 ceph-mon[147414]: pgmap v757: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:23.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:23 smithi032 ceph-mon[147414]: pgmap v758: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:23.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:45:23.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:23 smithi086 ceph-mon[110975]: pgmap v758: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:23.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:45:25.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:25 smithi086 ceph-mon[110975]: pgmap v759: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:25.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:25 smithi032 ceph-mon[147414]: pgmap v759: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:27.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:27 smithi086 ceph-mon[110975]: pgmap v760: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:27.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:27 smithi032 ceph-mon[147414]: pgmap v760: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:29.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:29 smithi086 ceph-mon[110975]: pgmap v761: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:29.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:29 smithi032 ceph-mon[147414]: pgmap v761: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:31.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:31 smithi086 ceph-mon[110975]: pgmap v762: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:31.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:31 smithi032 ceph-mon[147414]: pgmap v762: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:33.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.141+0000 7fcc99df2700 1 -- 172.21.15.32:0/1489375318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 msgr2=0x7fcc940fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.141+0000 7fcc99df2700 1 --2- 172.21.15.32:0/1489375318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 0x7fcc940fbd80 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fcc88004940 tx=0x7fcc88009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:33.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.141+0000 7fcc99df2700 1 -- 172.21.15.32:0/1489375318 shutdown_connections 2024-04-06T14:45:33.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.141+0000 7fcc99df2700 1 --2- 172.21.15.32:0/1489375318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 0x7fcc940fbd80 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.141+0000 7fcc99df2700 1 --2- 172.21.15.32:0/1489375318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc94101d70 0x7fcc94102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.141+0000 7fcc99df2700 1 -- 172.21.15.32:0/1489375318 >> 172.21.15.32:0/1489375318 conn(0x7fcc94072bc0 msgr2=0x7fcc94072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:33.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.141+0000 7fcc99df2700 1 -- 172.21.15.32:0/1489375318 shutdown_connections 2024-04-06T14:45:33.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.141+0000 7fcc99df2700 1 -- 172.21.15.32:0/1489375318 wait complete. 2024-04-06T14:45:33.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.142+0000 7fcc99df2700 1 Processor -- start 2024-04-06T14:45:33.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.142+0000 7fcc99df2700 1 -- start start 2024-04-06T14:45:33.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.142+0000 7fcc99df2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc94101d70 0x7fcc941032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:33.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.142+0000 7fcc99df2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 0x7fcc94103810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:33.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.142+0000 7fcc99df2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc94103da0 con 0x7fcc94102710 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.142+0000 7fcc99df2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc94103ee0 con 0x7fcc94101d70 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc92ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 0x7fcc94103810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc92ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 0x7fcc94103810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38424/0 (socket says 172.21.15.32:38424) 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc92ffd700 1 -- 172.21.15.32:0/2633399438 learned_addr learned my addr 172.21.15.32:0/2633399438 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc937fe700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc94101d70 0x7fcc941032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc92ffd700 1 -- 172.21.15.32:0/2633399438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc94101d70 msgr2=0x7fcc941032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc92ffd700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc94101d70 0x7fcc941032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc92ffd700 1 -- 172.21.15.32:0/2633399438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc88005040 con 0x7fcc94102710 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc92ffd700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 0x7fcc94103810 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fcc880093b0 tx=0x7fcc88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc7bfff700 1 -- 172.21.15.32:0/2633399438 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc8800e040 con 0x7fcc94102710 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc7bfff700 1 -- 172.21.15.32:0/2633399438 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc88013420 con 0x7fcc94102710 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc94104130 con 0x7fcc94102710 2024-04-06T14:45:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc7bfff700 1 -- 172.21.15.32:0/2633399438 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc8801c630 con 0x7fcc94102710 2024-04-06T14:45:33.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.143+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9411a140 con 0x7fcc94102710 2024-04-06T14:45:33.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.144+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc940fd4c0 con 0x7fcc94102710 2024-04-06T14:45:33.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.145+0000 7fcc7bfff700 1 -- 172.21.15.32:0/2633399438 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcc880139e0 con 0x7fcc94102710 2024-04-06T14:45:33.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.145+0000 7fcc7bfff700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7c06cc20 0x7fcc7c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:33.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.145+0000 7fcc7bfff700 1 -- 172.21.15.32:0/2633399438 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcc8808d170 con 0x7fcc94102710 2024-04-06T14:45:33.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.145+0000 7fcc937fe700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7c06cc20 0x7fcc7c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:33.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.146+0000 7fcc937fe700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7c06cc20 0x7fcc7c06f0d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009fd0 tx=0x7fcc8400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:33.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.150+0000 7fcc7bfff700 1 -- 172.21.15.32:0/2633399438 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcc88057260 con 0x7fcc94102710 2024-04-06T14:45:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.374+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc94108db0 con 0x7fcc7c06cc20 2024-04-06T14:45:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.375+0000 7fcc7bfff700 1 -- 172.21.15.32:0/2633399438 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcc8400f040 con 0x7fcc7c06cc20 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.377+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7c06cc20 msgr2=0x7fcc7c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.377+0000 7fcc99df2700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7c06cc20 0x7fcc7c06f0d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009fd0 tx=0x7fcc8400b040 comp rx=0 tx=0).stop 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.377+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 msgr2=0x7fcc94103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.377+0000 7fcc99df2700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 0x7fcc94103810 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fcc880093b0 tx=0x7fcc88009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.377+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 shutdown_connections 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.377+0000 7fcc99df2700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcc7c06cc20 0x7fcc7c06f0d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.377+0000 7fcc99df2700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc94101d70 0x7fcc941032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.377+0000 7fcc99df2700 1 --2- 172.21.15.32:0/2633399438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc94102710 0x7fcc94103810 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.377+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 >> 172.21.15.32:0/2633399438 conn(0x7fcc94072bc0 msgr2=0x7fcc941056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.378+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 shutdown_connections 2024-04-06T14:45:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.378+0000 7fcc99df2700 1 -- 172.21.15.32:0/2633399438 wait complete. 2024-04-06T14:45:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:45:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 -- 172.21.15.32:0/2057058032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 msgr2=0x7f95240fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 --2- 172.21.15.32:0/2057058032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 0x7f95240fbd80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9514004940 tx=0x7f9514009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 -- 172.21.15.32:0/2057058032 shutdown_connections 2024-04-06T14:45:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 --2- 172.21.15.32:0/2057058032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 0x7f95240fbd80 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 --2- 172.21.15.32:0/2057058032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9524101d70 0x7f9524102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 -- 172.21.15.32:0/2057058032 >> 172.21.15.32:0/2057058032 conn(0x7f9524072bc0 msgr2=0x7f9524072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 -- 172.21.15.32:0/2057058032 shutdown_connections 2024-04-06T14:45:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 -- 172.21.15.32:0/2057058032 wait complete. 2024-04-06T14:45:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 Processor -- start 2024-04-06T14:45:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.482+0000 7f952958a700 1 -- start start 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f952958a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9524101d70 0x7f952410ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f952958a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 0x7f952410f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f952958a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952410f680 con 0x7f9524102710 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f952958a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952410f7c0 con 0x7f9524101d70 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f95227fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 0x7f952410f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f95227fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 0x7f952410f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38438/0 (socket says 172.21.15.32:38438) 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f95227fc700 1 -- 172.21.15.32:0/1667360001 learned_addr learned my addr 172.21.15.32:0/1667360001 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f9522ffd700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9524101d70 0x7f952410ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f95227fc700 1 -- 172.21.15.32:0/1667360001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9524101d70 msgr2=0x7f952410ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f95227fc700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9524101d70 0x7f952410ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f95227fc700 1 -- 172.21.15.32:0/1667360001 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9514005040 con 0x7f9524102710 2024-04-06T14:45:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f95227fc700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 0x7f952410f140 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f95140093b0 tx=0x7f9514009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:33.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f950b7fe700 1 -- 172.21.15.32:0/1667360001 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951400e040 con 0x7f9524102710 2024-04-06T14:45:33.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f950b7fe700 1 -- 172.21.15.32:0/1667360001 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9514013420 con 0x7f9524102710 2024-04-06T14:45:33.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.483+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9524117ab0 con 0x7f9524102710 2024-04-06T14:45:33.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.484+0000 7f950b7fe700 1 -- 172.21.15.32:0/1667360001 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951401c630 con 0x7f9524102710 2024-04-06T14:45:33.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.484+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9524118040 con 0x7f9524102710 2024-04-06T14:45:33.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.485+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95240ff080 con 0x7f9524102710 2024-04-06T14:45:33.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.485+0000 7f950b7fe700 1 -- 172.21.15.32:0/1667360001 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f95140139e0 con 0x7f9524102710 2024-04-06T14:45:33.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.485+0000 7f950b7fe700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f950c06cc20 0x7f950c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:33.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.486+0000 7f950b7fe700 1 -- 172.21.15.32:0/1667360001 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f951408d140 con 0x7f9524102710 2024-04-06T14:45:33.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.486+0000 7f9522ffd700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f950c06cc20 0x7f950c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:33.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.486+0000 7f9522ffd700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f950c06cc20 0x7f950c06f0d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9518004f40 tx=0x7f951800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:33.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.490+0000 7f950b7fe700 1 -- 172.21.15.32:0/1667360001 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9514057180 con 0x7f9524102710 2024-04-06T14:45:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.713+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9524065e40 con 0x7f950c06cc20 2024-04-06T14:45:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.714+0000 7f950b7fe700 1 -- 172.21.15.32:0/1667360001 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9524065e40 con 0x7f950c06cc20 2024-04-06T14:45:33.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.716+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f950c06cc20 msgr2=0x7f950c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.716+0000 7f952958a700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f950c06cc20 0x7f950c06f0d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9518004f40 tx=0x7f951800b040 comp rx=0 tx=0).stop 2024-04-06T14:45:33.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.717+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 msgr2=0x7f952410f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.717+0000 7f952958a700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 0x7f952410f140 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f95140093b0 tx=0x7f9514009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:33.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.717+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 shutdown_connections 2024-04-06T14:45:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.717+0000 7f952958a700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f950c06cc20 0x7f950c06f0d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.717+0000 7f952958a700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9524101d70 0x7f952410ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.717+0000 7f952958a700 1 --2- 172.21.15.32:0/1667360001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9524102710 0x7f952410f140 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.717+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 >> 172.21.15.32:0/1667360001 conn(0x7f9524072bc0 msgr2=0x7f95240fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.717+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 shutdown_connections 2024-04-06T14:45:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.717+0000 7f952958a700 1 -- 172.21.15.32:0/1667360001 wait complete. 2024-04-06T14:45:33.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:33 smithi032 ceph-mon[147414]: pgmap v763: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:33.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.820+0000 7f540d5ab700 1 -- 172.21.15.32:0/1365440002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 msgr2=0x7f54081031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.820+0000 7f540d5ab700 1 --2- 172.21.15.32:0/1365440002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 0x7f54081031e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f53f8004940 tx=0x7f53f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:33.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.820+0000 7f540d5ab700 1 -- 172.21.15.32:0/1365440002 shutdown_connections 2024-04-06T14:45:33.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.820+0000 7f540d5ab700 1 --2- 172.21.15.32:0/1365440002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54081037b0 0x7f54080fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.820+0000 7f540d5ab700 1 --2- 172.21.15.32:0/1365440002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 0x7f54081031e0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.820+0000 7f540d5ab700 1 -- 172.21.15.32:0/1365440002 >> 172.21.15.32:0/1365440002 conn(0x7f54080f89a0 msgr2=0x7f54080fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.820+0000 7f540d5ab700 1 -- 172.21.15.32:0/1365440002 shutdown_connections 2024-04-06T14:45:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.820+0000 7f540d5ab700 1 -- 172.21.15.32:0/1365440002 wait complete. 2024-04-06T14:45:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.821+0000 7f540d5ab700 1 Processor -- start 2024-04-06T14:45:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.821+0000 7f540d5ab700 1 -- start start 2024-04-06T14:45:33.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.821+0000 7f540d5ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 0x7f54080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:33.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.821+0000 7f540d5ab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54081037b0 0x7f5408070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:33.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.821+0000 7f540d5ab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5408070e20 con 0x7f5408102e10 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.821+0000 7f540d5ab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5408070f60 con 0x7f54081037b0 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.821+0000 7f5406ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 0x7f54080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.821+0000 7f5406ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 0x7f54080702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38466/0 (socket says 172.21.15.32:38466) 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.821+0000 7f5406ffd700 1 -- 172.21.15.32:0/4066456767 learned_addr learned my addr 172.21.15.32:0/4066456767 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f54067fc700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54081037b0 0x7f5408070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f5406ffd700 1 -- 172.21.15.32:0/4066456767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54081037b0 msgr2=0x7f5408070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f5406ffd700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54081037b0 0x7f5408070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f5406ffd700 1 -- 172.21.15.32:0/4066456767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f8005040 con 0x7f5408102e10 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f5406ffd700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 0x7f54080702c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009180 tx=0x7f53f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f53ef7fe700 1 -- 172.21.15.32:0/4066456767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53f800e040 con 0x7f5408102e10 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5408119db0 con 0x7f5408102e10 2024-04-06T14:45:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f53ef7fe700 1 -- 172.21.15.32:0/4066456767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53f8004290 con 0x7f5408102e10 2024-04-06T14:45:33.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f53ef7fe700 1 -- 172.21.15.32:0/4066456767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53f801c3f0 con 0x7f5408102e10 2024-04-06T14:45:33.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.822+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540811a290 con 0x7f5408102e10 2024-04-06T14:45:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.823+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54080fe560 con 0x7f5408102e10 2024-04-06T14:45:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.824+0000 7f53ef7fe700 1 -- 172.21.15.32:0/4066456767 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f53f8013790 con 0x7f5408102e10 2024-04-06T14:45:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.824+0000 7f53ef7fe700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53f006cc20 0x7f53f006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.824+0000 7f53ef7fe700 1 -- 172.21.15.32:0/4066456767 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f53f808d2f0 con 0x7f5408102e10 2024-04-06T14:45:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.824+0000 7f54067fc700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53f006cc20 0x7f53f006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.825+0000 7f54067fc700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53f006cc20 0x7f53f006f0d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f53fc006fd0 tx=0x7f53fc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:33.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:33.829+0000 7f53ef7fe700 1 -- 172.21.15.32:0/4066456767 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f53f8057330 con 0x7f5408102e10 2024-04-06T14:45:33.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:33 smithi086 ceph-mon[110975]: pgmap v763: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.040+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f54080710f0 con 0x7f53f006cc20 2024-04-06T14:45:34.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.050+0000 7f53ef7fe700 1 -- 172.21.15.32:0/4066456767 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f54080710f0 con 0x7f53f006cc20 2024-04-06T14:45:34.040 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:45:34.040 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (25m) 3m ago 37m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:45:34.040 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (23m) 3m ago 37m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:45:34.040 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (23m) 3m ago 36m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:45:34.040 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (25m) 3m ago 36m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (19m) 3m ago 31m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (19m) 3m ago 31m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (19m) 3m ago 31m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (3m) 3m ago 31m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (27m) 3m ago 38m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (26m) 3m ago 36m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (24m) 3m ago 38m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (23m) 3m ago 36m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (26m) 3m ago 37m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (26m) 3m ago 36m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (22m) 3m ago 35m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (22m) 3m ago 34m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (21m) 3m ago 34m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:45:34.041 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (21m) 3m ago 33m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:45:34.042 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (20m) 3m ago 33m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:45:34.042 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (19m) 3m ago 33m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:45:34.042 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (26m) 3m ago 36m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:45:34.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53f006cc20 msgr2=0x7f53f006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53f006cc20 0x7f53f006f0d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f53fc006fd0 tx=0x7f53fc008040 comp rx=0 tx=0).stop 2024-04-06T14:45:34.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 msgr2=0x7f54080702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 0x7f54080702c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009180 tx=0x7f53f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:34.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 shutdown_connections 2024-04-06T14:45:34.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53f006cc20 0x7f53f006f0d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5408102e10 0x7f54080702c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 --2- 172.21.15.32:0/4066456767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54081037b0 0x7f5408070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 >> 172.21.15.32:0/4066456767 conn(0x7f54080f89a0 msgr2=0x7f54080f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 shutdown_connections 2024-04-06T14:45:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.052+0000 7f540d5ab700 1 -- 172.21.15.32:0/4066456767 wait complete. 2024-04-06T14:45:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.153+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/276610689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 msgr2=0x7ff7e4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.153+0000 7ff7e9fc3700 1 --2- 172.21.15.32:0/276610689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 0x7ff7e4102140 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4004940 tx=0x7ff7d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.153+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/276610689 shutdown_connections 2024-04-06T14:45:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.153+0000 7ff7e9fc3700 1 --2- 172.21.15.32:0/276610689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e4102710 0x7ff7e40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.153+0000 7ff7e9fc3700 1 --2- 172.21.15.32:0/276610689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 0x7ff7e4102140 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.153+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/276610689 >> 172.21.15.32:0/276610689 conn(0x7ff7e4072bc0 msgr2=0x7ff7e4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.154+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/276610689 shutdown_connections 2024-04-06T14:45:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.154+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/276610689 wait complete. 2024-04-06T14:45:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.154+0000 7ff7e9fc3700 1 Processor -- start 2024-04-06T14:45:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.154+0000 7ff7e9fc3700 1 -- start start 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.154+0000 7ff7e9fc3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 0x7ff7e4113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e9fc3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e4102710 0x7ff7e4114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e9fc3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e41146f0 con 0x7ff7e4101d70 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e9fc3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e410da50 con 0x7ff7e4102710 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 0x7ff7e4113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 0x7ff7e4113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38496/0 (socket says 172.21.15.32:38496) 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e37fe700 1 -- 172.21.15.32:0/4260580156 learned_addr learned my addr 172.21.15.32:0/4260580156 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e2ffd700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e4102710 0x7ff7e4114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e37fe700 1 -- 172.21.15.32:0/4260580156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e4102710 msgr2=0x7ff7e4114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e37fe700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e4102710 0x7ff7e4114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e37fe700 1 -- 172.21.15.32:0/4260580156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d4005040 con 0x7ff7e4101d70 2024-04-06T14:45:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e2ffd700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e4102710 0x7ff7e4114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:45:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e37fe700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 0x7ff7e4113ad0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009360 tx=0x7ff7d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7cbfff700 1 -- 172.21.15.32:0/4260580156 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7d400e040 con 0x7ff7e4101d70 2024-04-06T14:45:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e410dcd0 con 0x7ff7e4101d70 2024-04-06T14:45:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7cbfff700 1 -- 172.21.15.32:0/4260580156 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7d4013420 con 0x7ff7e4101d70 2024-04-06T14:45:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7cbfff700 1 -- 172.21.15.32:0/4260580156 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7d401c5f0 con 0x7ff7e4101d70 2024-04-06T14:45:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.155+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e410e290 con 0x7ff7e4101d70 2024-04-06T14:45:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.156+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e40fd4c0 con 0x7ff7e4101d70 2024-04-06T14:45:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.158+0000 7ff7cbfff700 1 -- 172.21.15.32:0/4260580156 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff7d4013590 con 0x7ff7e4101d70 2024-04-06T14:45:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.158+0000 7ff7cbfff700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7cc06cc30 0x7ff7cc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.158+0000 7ff7cbfff700 1 -- 172.21.15.32:0/4260580156 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff7d408dcd0 con 0x7ff7e4101d70 2024-04-06T14:45:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.158+0000 7ff7e2ffd700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7cc06cc30 0x7ff7cc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.159+0000 7ff7e2ffd700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7cc06cc30 0x7ff7cc06f0e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8006fd0 tx=0x7ff7d8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.162+0000 7ff7cbfff700 1 -- 172.21.15.32:0/4260580156 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff7d405b860 con 0x7ff7e4101d70 2024-04-06T14:45:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.449+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff7e4108e30 con 0x7ff7e4101d70 2024-04-06T14:45:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.449+0000 7ff7cbfff700 1 -- 172.21.15.32:0/4260580156 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7ff7d4011370 con 0x7ff7e4101d70 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:45:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:45:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:45:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:45:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:45:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:45:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:45:34.440 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7cc06cc30 msgr2=0x7ff7cc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7cc06cc30 0x7ff7cc06f0e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8006fd0 tx=0x7ff7d8008040 comp rx=0 tx=0).stop 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 msgr2=0x7ff7e4113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 0x7ff7e4113ad0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009360 tx=0x7ff7d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 shutdown_connections 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7cc06cc30 0x7ff7cc06f0e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e4101d70 0x7ff7e4113ad0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 --2- 172.21.15.32:0/4260580156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e4102710 0x7ff7e4114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 >> 172.21.15.32:0/4260580156 conn(0x7ff7e4072bc0 msgr2=0x7ff7e41056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 shutdown_connections 2024-04-06T14:45:34.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.451+0000 7ff7e9fc3700 1 -- 172.21.15.32:0/4260580156 wait complete. 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 -- 172.21.15.32:0/1997752016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 msgr2=0x7f977c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 --2- 172.21.15.32:0/1997752016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 0x7f977c0fe260 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9778004940 tx=0x7f9778009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 -- 172.21.15.32:0/1997752016 shutdown_connections 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 --2- 172.21.15.32:0/1997752016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f977c0fe830 0x7f977c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 --2- 172.21.15.32:0/1997752016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 0x7f977c0fe260 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 -- 172.21.15.32:0/1997752016 >> 172.21.15.32:0/1997752016 conn(0x7f977c072bc0 msgr2=0x7f977c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 -- 172.21.15.32:0/1997752016 shutdown_connections 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 -- 172.21.15.32:0/1997752016 wait complete. 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 Processor -- start 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 -- start start 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.557+0000 7f9783c70700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 0x7f977c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9783c70700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f977c0fe830 0x7f977c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9783c70700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0fc820 con 0x7f977c0fde90 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9783c70700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0fc960 con 0x7f977c0fe830 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9781a0c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 0x7f977c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9781a0c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 0x7f977c0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34308/0 (socket says 172.21.15.32:34308) 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9781a0c700 1 -- 172.21.15.32:0/915300317 learned_addr learned my addr 172.21.15.32:0/915300317 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f978120b700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f977c0fe830 0x7f977c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9781a0c700 1 -- 172.21.15.32:0/915300317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f977c0fe830 msgr2=0x7f977c0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9781a0c700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f977c0fe830 0x7f977c0fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9781a0c700 1 -- 172.21.15.32:0/915300317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778005040 con 0x7f977c0fde90 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f978120b700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f977c0fe830 0x7f977c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9781a0c700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 0x7f977c0fbd50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f97780093b0 tx=0x7f9778009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f97727fc700 1 -- 172.21.15.32:0/915300317 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f977800e040 con 0x7f977c0fde90 2024-04-06T14:45:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f97727fc700 1 -- 172.21.15.32:0/915300317 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9778004290 con 0x7f977c0fde90 2024-04-06T14:45:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c0fcbb0 con 0x7f977c0fde90 2024-04-06T14:45:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f97727fc700 1 -- 172.21.15.32:0/915300317 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f977801c3f0 con 0x7f977c0fde90 2024-04-06T14:45:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.558+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c11a1a0 con 0x7f977c0fde90 2024-04-06T14:45:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.559+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c065e30 con 0x7f977c0fde90 2024-04-06T14:45:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.560+0000 7f97727fc700 1 -- 172.21.15.32:0/915300317 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9778013790 con 0x7f977c0fde90 2024-04-06T14:45:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.560+0000 7f97727fc700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f976806cc30 0x7f976806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.561+0000 7f97727fc700 1 -- 172.21.15.32:0/915300317 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f977808dd40 con 0x7f977c0fde90 2024-04-06T14:45:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.561+0000 7f978120b700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f976806cc30 0x7f976806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.561+0000 7f978120b700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f976806cc30 0x7f976806f0e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f976c009fd0 tx=0x7f976c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.565+0000 7f97727fc700 1 -- 172.21.15.32:0/915300317 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9778057e40 con 0x7f977c0fde90 2024-04-06T14:45:34.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.810+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f977c0fd440 con 0x7f977c0fde90 2024-04-06T14:45:34.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.810+0000 7f97727fc700 1 -- 172.21.15.32:0/915300317 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f9778013420 con 0x7f977c0fde90 2024-04-06T14:45:34.801 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:45:34.801 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:45:34.801 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:45:34.801 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:45:34.801 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:45:34.801 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:45:34.801 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:45:34.802 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:45:34.803 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:45:34.804 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:45:34.804 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:45:34.804 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:45:34.804 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.812+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f976806cc30 msgr2=0x7f976806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.812+0000 7f9783c70700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f976806cc30 0x7f976806f0e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f976c009fd0 tx=0x7f976c009ee0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.812+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 msgr2=0x7f977c0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.812+0000 7f9783c70700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 0x7f977c0fbd50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f97780093b0 tx=0x7f9778009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.812+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 shutdown_connections 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.813+0000 7f9783c70700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f976806cc30 0x7f976806f0e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.813+0000 7f9783c70700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f977c0fde90 0x7f977c0fbd50 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.813+0000 7f9783c70700 1 --2- 172.21.15.32:0/915300317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f977c0fe830 0x7f977c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.813+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 >> 172.21.15.32:0/915300317 conn(0x7f977c072bc0 msgr2=0x7f977c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.813+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 shutdown_connections 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.813+0000 7f9783c70700 1 -- 172.21.15.32:0/915300317 wait complete. 2024-04-06T14:45:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:45:34.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:34 smithi086 ceph-mon[110975]: from='client.35402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:34.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:34 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4260580156' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:45:34.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.917+0000 7ff7eead3700 1 -- 172.21.15.32:0/559863400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 msgr2=0x7ff7e80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.917+0000 7ff7eead3700 1 --2- 172.21.15.32:0/559863400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 0x7ff7e80fe260 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc004940 tx=0x7ff7dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:34.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.917+0000 7ff7eead3700 1 -- 172.21.15.32:0/559863400 shutdown_connections 2024-04-06T14:45:34.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.917+0000 7ff7eead3700 1 --2- 172.21.15.32:0/559863400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e80fe830 0x7ff7e8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.917+0000 7ff7eead3700 1 --2- 172.21.15.32:0/559863400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 0x7ff7e80fe260 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.917+0000 7ff7eead3700 1 -- 172.21.15.32:0/559863400 >> 172.21.15.32:0/559863400 conn(0x7ff7e8072bc0 msgr2=0x7ff7e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:34.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.917+0000 7ff7eead3700 1 -- 172.21.15.32:0/559863400 shutdown_connections 2024-04-06T14:45:34.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.917+0000 7ff7eead3700 1 -- 172.21.15.32:0/559863400 wait complete. 2024-04-06T14:45:34.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.917+0000 7ff7eead3700 1 Processor -- start 2024-04-06T14:45:34.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7eead3700 1 -- start start 2024-04-06T14:45:34.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7eead3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 0x7ff7e80fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:34.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7eead3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e80fe830 0x7ff7e80fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:34.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7eead3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80fc820 con 0x7ff7e80fde90 2024-04-06T14:45:34.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7eead3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80fc960 con 0x7ff7e80fe830 2024-04-06T14:45:34.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7ec86f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 0x7ff7e80fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:34.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7ec86f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 0x7ff7e80fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34326/0 (socket says 172.21.15.32:34326) 2024-04-06T14:45:34.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7ec86f700 1 -- 172.21.15.32:0/3168377616 learned_addr learned my addr 172.21.15.32:0/3168377616 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:34.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7e7fff700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e80fe830 0x7ff7e80fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:34.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7ec86f700 1 -- 172.21.15.32:0/3168377616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e80fe830 msgr2=0x7ff7e80fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7ec86f700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e80fe830 0x7ff7e80fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7ec86f700 1 -- 172.21.15.32:0/3168377616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc005040 con 0x7ff7e80fde90 2024-04-06T14:45:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.918+0000 7ff7ec86f700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 0x7ff7e80fbd50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0093b0 tx=0x7ff7dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.919+0000 7ff7e57fa700 1 -- 172.21.15.32:0/3168377616 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7dc00e040 con 0x7ff7e80fde90 2024-04-06T14:45:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.919+0000 7ff7e57fa700 1 -- 172.21.15.32:0/3168377616 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7dc00bed0 con 0x7ff7e80fde90 2024-04-06T14:45:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.919+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e80fcbb0 con 0x7ff7e80fde90 2024-04-06T14:45:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.919+0000 7ff7e57fa700 1 -- 172.21.15.32:0/3168377616 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7dc01c3f0 con 0x7ff7e80fde90 2024-04-06T14:45:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.919+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e811a1a0 con 0x7ff7e80fde90 2024-04-06T14:45:34.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.920+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e8065e30 con 0x7ff7e80fde90 2024-04-06T14:45:34.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.920+0000 7ff7e57fa700 1 -- 172.21.15.32:0/3168377616 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff7dc013790 con 0x7ff7e80fde90 2024-04-06T14:45:34.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.921+0000 7ff7e57fa700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7d806cc30 0x7ff7d806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:34.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.921+0000 7ff7e57fa700 1 -- 172.21.15.32:0/3168377616 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff7dc08d3a0 con 0x7ff7e80fde90 2024-04-06T14:45:34.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.921+0000 7ff7e7fff700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7d806cc30 0x7ff7d806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:34.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.921+0000 7ff7e7fff700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7d806cc30 0x7ff7d806f0e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80fd690 tx=0x7ff7d400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:34.925+0000 7ff7e57fa700 1 -- 172.21.15.32:0/3168377616 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff7dc0573e0 con 0x7ff7e80fde90 2024-04-06T14:45:34.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:34 smithi032 ceph-mon[147414]: from='client.35402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:34.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:34 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4260580156' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:45:35.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.146+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7e80fd490 con 0x7ff7d806cc30 2024-04-06T14:45:35.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.148+0000 7ff7e57fa700 1 -- 172.21.15.32:0/3168377616 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff7e80fd490 con 0x7ff7d806cc30 2024-04-06T14:45:35.138 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:45:35.138 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:45:35.138 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:45:35.138 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:45:35.138 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:45:35.138 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:45:35.138 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:45:35.139 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:45:35.139 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:45:35.139 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:45:35.139 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:45:35.139 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:45:35.139 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:45:35.139 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:45:35.139 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:45:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7d806cc30 msgr2=0x7ff7d806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7d806cc30 0x7ff7d806f0e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80fd690 tx=0x7ff7d400b040 comp rx=0 tx=0).stop 2024-04-06T14:45:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 msgr2=0x7ff7e80fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 0x7ff7e80fbd50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0093b0 tx=0x7ff7dc009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 shutdown_connections 2024-04-06T14:45:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff7d806cc30 0x7ff7d806f0e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7e80fde90 0x7ff7e80fbd50 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 --2- 172.21.15.32:0/3168377616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7e80fe830 0x7ff7e80fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 >> 172.21.15.32:0/3168377616 conn(0x7ff7e8072bc0 msgr2=0x7ff7e80ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 shutdown_connections 2024-04-06T14:45:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.150+0000 7ff7eead3700 1 -- 172.21.15.32:0/3168377616 wait complete. 2024-04-06T14:45:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.252+0000 7fe35837d700 1 -- 172.21.15.32:0/3669055336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 msgr2=0x7fe3500fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.252+0000 7fe35837d700 1 --2- 172.21.15.32:0/3669055336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 0x7fe3500fbd80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe340004940 tx=0x7fe340009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.252+0000 7fe35837d700 1 -- 172.21.15.32:0/3669055336 shutdown_connections 2024-04-06T14:45:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 --2- 172.21.15.32:0/3669055336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 0x7fe3500fbd80 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 --2- 172.21.15.32:0/3669055336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe350101d70 0x7fe350102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 -- 172.21.15.32:0/3669055336 >> 172.21.15.32:0/3669055336 conn(0x7fe350072bc0 msgr2=0x7fe350072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 -- 172.21.15.32:0/3669055336 shutdown_connections 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 -- 172.21.15.32:0/3669055336 wait complete. 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 Processor -- start 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 -- start start 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe350101d70 0x7fe350113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 0x7fe350113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3501146c0 con 0x7fe350102710 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe35837d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35010da20 con 0x7fe350101d70 2024-04-06T14:45:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe355918700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 0x7fe350113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe355918700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 0x7fe350113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34336/0 (socket says 172.21.15.32:34336) 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.253+0000 7fe355918700 1 -- 172.21.15.32:0/92062483 learned_addr learned my addr 172.21.15.32:0/92062483 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe356119700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe350101d70 0x7fe350113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe355918700 1 -- 172.21.15.32:0/92062483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe350101d70 msgr2=0x7fe350113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe355918700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe350101d70 0x7fe350113aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe355918700 1 -- 172.21.15.32:0/92062483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe340005040 con 0x7fe350102710 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe355918700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 0x7fe350113fe0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe3400093b0 tx=0x7fe340009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe346ffd700 1 -- 172.21.15.32:0/92062483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe34000e040 con 0x7fe350102710 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35010dca0 con 0x7fe350102710 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe346ffd700 1 -- 172.21.15.32:0/92062483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe340013420 con 0x7fe350102710 2024-04-06T14:45:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe346ffd700 1 -- 172.21.15.32:0/92062483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe34001c630 con 0x7fe350102710 2024-04-06T14:45:35.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.254+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35010e260 con 0x7fe350102710 2024-04-06T14:45:35.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.255+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3500fd4c0 con 0x7fe350102710 2024-04-06T14:45:35.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.256+0000 7fe346ffd700 1 -- 172.21.15.32:0/92062483 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe3400139e0 con 0x7fe350102710 2024-04-06T14:45:35.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.256+0000 7fe346ffd700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe33c06cc20 0x7fe33c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:45:35.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.256+0000 7fe346ffd700 1 -- 172.21.15.32:0/92062483 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe34008d190 con 0x7fe350102710 2024-04-06T14:45:35.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.256+0000 7fe356119700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe33c06cc20 0x7fe33c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:45:35.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.257+0000 7fe356119700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe33c06cc20 0x7fe33c06f0d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009bf0 tx=0x7fe34c0091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:45:35.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.261+0000 7fe346ffd700 1 -- 172.21.15.32:0/92062483 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe34005ad10 con 0x7fe350102710 2024-04-06T14:45:35.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.549+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe35004bed0 con 0x7fe350102710 2024-04-06T14:45:35.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.549+0000 7fe346ffd700 1 -- 172.21.15.32:0/92062483 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7fe34005a8a0 con 0x7fe350102710 2024-04-06T14:45:35.539 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:45:35.540 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:45:35.541 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:45:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe33c06cc20 msgr2=0x7fe33c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe33c06cc20 0x7fe33c06f0d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009bf0 tx=0x7fe34c0091a0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 msgr2=0x7fe350113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:45:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 0x7fe350113fe0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe3400093b0 tx=0x7fe340009e30 comp rx=0 tx=0).stop 2024-04-06T14:45:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 shutdown_connections 2024-04-06T14:45:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe33c06cc20 0x7fe33c06f0d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe350101d70 0x7fe350113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 --2- 172.21.15.32:0/92062483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe350102710 0x7fe350113fe0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:45:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 >> 172.21.15.32:0/92062483 conn(0x7fe350072bc0 msgr2=0x7fe3501057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:45:35.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 shutdown_connections 2024-04-06T14:45:35.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:45:35.551+0000 7fe35837d700 1 -- 172.21.15.32:0/92062483 wait complete. 2024-04-06T14:45:35.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:35 smithi086 ceph-mon[110975]: from='client.35406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:35.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:35 smithi086 ceph-mon[110975]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:35.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:35 smithi086 ceph-mon[110975]: pgmap v764: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:35.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:35 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/915300317' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:45:35.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:35 smithi032 ceph-mon[147414]: from='client.35406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:35.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:35 smithi032 ceph-mon[147414]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:35.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:35 smithi032 ceph-mon[147414]: pgmap v764: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:35.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:35 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/915300317' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:45:36.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:36 smithi086 ceph-mon[110975]: from='client.35422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:36.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:36 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/92062483' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:45:36.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:36 smithi032 ceph-mon[147414]: from='client.35422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:45:36.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:36 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/92062483' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:45:37.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:37 smithi086 ceph-mon[110975]: pgmap v765: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:37.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:37 smithi032 ceph-mon[147414]: pgmap v765: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:45:38.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:45:39.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:39 smithi086 ceph-mon[110975]: pgmap v766: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:39.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:39 smithi032 ceph-mon[147414]: pgmap v766: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:41.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:41 smithi086 ceph-mon[110975]: pgmap v767: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:41.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:41 smithi032 ceph-mon[147414]: pgmap v767: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:43.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:43 smithi032 ceph-mon[147414]: pgmap v768: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:43.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:43 smithi086 ceph-mon[110975]: pgmap v768: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:45.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:45 smithi086 ceph-mon[110975]: pgmap v769: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:45.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:45 smithi032 ceph-mon[147414]: pgmap v769: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:47.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:47 smithi086 ceph-mon[110975]: pgmap v770: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:47.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:47 smithi032 ceph-mon[147414]: pgmap v770: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:49.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:49 smithi086 ceph-mon[110975]: pgmap v771: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:49.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:49 smithi032 ceph-mon[147414]: pgmap v771: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:51.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:51 smithi086 ceph-mon[110975]: pgmap v772: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:51.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:51 smithi032 ceph-mon[147414]: pgmap v772: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:53.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:53 smithi032 ceph-mon[147414]: pgmap v773: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:53.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:45:53.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:53 smithi086 ceph-mon[110975]: pgmap v773: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:53.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:45:55.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:55 smithi086 ceph-mon[110975]: pgmap v774: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:55.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:55 smithi032 ceph-mon[147414]: pgmap v774: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:57.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:57 smithi086 ceph-mon[110975]: pgmap v775: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:57.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:57 smithi032 ceph-mon[147414]: pgmap v775: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:59.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:45:59 smithi086 ceph-mon[110975]: pgmap v776: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:45:59.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:45:59 smithi032 ceph-mon[147414]: pgmap v776: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:01.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:01 smithi086 ceph-mon[110975]: pgmap v777: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:01.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:01 smithi032 ceph-mon[147414]: pgmap v777: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:03.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:03 smithi032 ceph-mon[147414]: pgmap v778: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:03.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:03 smithi086 ceph-mon[110975]: pgmap v778: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:05.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.671+0000 7f322ef48700 1 -- 172.21.15.32:0/4144900853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fe7b0 msgr2=0x7f3228106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:05.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.671+0000 7f322ef48700 1 --2- 172.21.15.32:0/4144900853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fe7b0 0x7f3228106df0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3214004940 tx=0x7f3214009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:05.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.671+0000 7f322ef48700 1 -- 172.21.15.32:0/4144900853 shutdown_connections 2024-04-06T14:46:05.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.671+0000 7f322ef48700 1 --2- 172.21.15.32:0/4144900853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fe7b0 0x7f3228106df0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:05.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.671+0000 7f322ef48700 1 --2- 172.21.15.32:0/4144900853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32280fde10 0x7f32280fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:05.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.671+0000 7f322ef48700 1 -- 172.21.15.32:0/4144900853 >> 172.21.15.32:0/4144900853 conn(0x7f32280f7860 msgr2=0x7f32280f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:05.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.671+0000 7f322ef48700 1 -- 172.21.15.32:0/4144900853 shutdown_connections 2024-04-06T14:46:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.671+0000 7f322ef48700 1 -- 172.21.15.32:0/4144900853 wait complete. 2024-04-06T14:46:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f322ef48700 1 Processor -- start 2024-04-06T14:46:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f322ef48700 1 -- start start 2024-04-06T14:46:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f322ef48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fde10 0x7f3228113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f322ef48700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32280fe7b0 0x7f3228113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f322ef48700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32281145f0 con 0x7f32280fde10 2024-04-06T14:46:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f322ef48700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322810d9e0 con 0x7f32280fe7b0 2024-04-06T14:46:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f322cce4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fde10 0x7f3228113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f322cce4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fde10 0x7f3228113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59102/0 (socket says 172.21.15.32:59102) 2024-04-06T14:46:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f322cce4700 1 -- 172.21.15.32:0/495461182 learned_addr learned my addr 172.21.15.32:0/495461182 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.672+0000 7f3227fff700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32280fe7b0 0x7f3228113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f322cce4700 1 -- 172.21.15.32:0/495461182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32280fe7b0 msgr2=0x7f3228113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f322cce4700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32280fe7b0 0x7f3228113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f322cce4700 1 -- 172.21.15.32:0/495461182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214005040 con 0x7f32280fde10 2024-04-06T14:46:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f3227fff700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32280fe7b0 0x7f3228113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:46:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f322cce4700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fde10 0x7f3228113a60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f321c00eb40 tx=0x7f321c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f32257fa700 1 -- 172.21.15.32:0/495461182 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f321c007cc0 con 0x7f32280fde10 2024-04-06T14:46:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322810dc60 con 0x7f32280fde10 2024-04-06T14:46:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f32257fa700 1 -- 172.21.15.32:0/495461182 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f321c00f040 con 0x7f32280fde10 2024-04-06T14:46:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f32257fa700 1 -- 172.21.15.32:0/495461182 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f321c009570 con 0x7f32280fde10 2024-04-06T14:46:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.673+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322810e1a0 con 0x7f32280fde10 2024-04-06T14:46:05.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.674+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3228066770 con 0x7f32280fde10 2024-04-06T14:46:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.675+0000 7f32257fa700 1 -- 172.21.15.32:0/495461182 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f321c0096d0 con 0x7f32280fde10 2024-04-06T14:46:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.675+0000 7f32257fa700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321806cc30 0x7f321806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.675+0000 7f32257fa700 1 -- 172.21.15.32:0/495461182 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f321c08bd40 con 0x7f32280fde10 2024-04-06T14:46:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.675+0000 7f3227fff700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321806cc30 0x7f321806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.676+0000 7f3227fff700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321806cc30 0x7f321806f0e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f321400b0d0 tx=0x7f3214009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:05.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.680+0000 7f32257fa700 1 -- 172.21.15.32:0/495461182 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f321c08f030 con 0x7f32280fde10 2024-04-06T14:46:05.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:05 smithi086 ceph-mon[110975]: pgmap v779: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:05.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.903+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3228109d60 con 0x7f321806cc30 2024-04-06T14:46:05.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.904+0000 7f32257fa700 1 -- 172.21.15.32:0/495461182 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3228109d60 con 0x7f321806cc30 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321806cc30 msgr2=0x7f321806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321806cc30 0x7f321806f0e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f321400b0d0 tx=0x7f3214009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fde10 msgr2=0x7f3228113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fde10 0x7f3228113a60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f321c00eb40 tx=0x7f321c00b040 comp rx=0 tx=0).stop 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 shutdown_connections 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321806cc30 0x7f321806f0e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32280fde10 0x7f3228113a60 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 --2- 172.21.15.32:0/495461182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32280fe7b0 0x7f3228113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 >> 172.21.15.32:0/495461182 conn(0x7f32280f7860 msgr2=0x7f32280f8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 shutdown_connections 2024-04-06T14:46:05.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:05.906+0000 7f322ef48700 1 -- 172.21.15.32:0/495461182 wait complete. 2024-04-06T14:46:05.907 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:46:05.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:05 smithi032 ceph-mon[147414]: pgmap v779: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.011+0000 7f94f769c700 1 -- 172.21.15.32:0/3073926896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 msgr2=0x7f94f00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.011+0000 7f94f769c700 1 --2- 172.21.15.32:0/3073926896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 0x7f94f00fce10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f94e0004940 tx=0x7f94e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.011+0000 7f94f769c700 1 -- 172.21.15.32:0/3073926896 shutdown_connections 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.011+0000 7f94f769c700 1 --2- 172.21.15.32:0/3073926896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 0x7f94f00fce10 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.011+0000 7f94f769c700 1 --2- 172.21.15.32:0/3073926896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f0102e00 0x7f94f01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.011+0000 7f94f769c700 1 -- 172.21.15.32:0/3073926896 >> 172.21.15.32:0/3073926896 conn(0x7f94f00f89b0 msgr2=0x7f94f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.011+0000 7f94f769c700 1 -- 172.21.15.32:0/3073926896 shutdown_connections 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.011+0000 7f94f769c700 1 -- 172.21.15.32:0/3073926896 wait complete. 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.011+0000 7f94f769c700 1 Processor -- start 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f769c700 1 -- start start 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f769c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f0102e00 0x7f94f00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f769c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 0x7f94f0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f769c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0070d90 con 0x7f94f01037a0 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f769c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0070ed0 con 0x7f94f0102e00 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f4c37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 0x7f94f0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f4c37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 0x7f94f0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59124/0 (socket says 172.21.15.32:59124) 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f4c37700 1 -- 172.21.15.32:0/2935226052 learned_addr learned my addr 172.21.15.32:0/2935226052 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f5438700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f0102e00 0x7f94f00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f4c37700 1 -- 172.21.15.32:0/2935226052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f0102e00 msgr2=0x7f94f00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f4c37700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f0102e00 0x7f94f00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f4c37700 1 -- 172.21.15.32:0/2935226052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0005040 con 0x7f94f01037a0 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f5438700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f0102e00 0x7f94f00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94f4c37700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 0x7f94f0070800 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009380 tx=0x7f94e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.012+0000 7f94e5ffb700 1 -- 172.21.15.32:0/2935226052 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94e000e040 con 0x7f94f01037a0 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.013+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f0071120 con 0x7f94f01037a0 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.013+0000 7f94e5ffb700 1 -- 172.21.15.32:0/2935226052 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94e0013420 con 0x7f94f01037a0 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.013+0000 7f94e5ffb700 1 -- 172.21.15.32:0/2935226052 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94e001c630 con 0x7f94f01037a0 2024-04-06T14:46:06.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.013+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f011a230 con 0x7f94f01037a0 2024-04-06T14:46:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.014+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f00fe550 con 0x7f94f01037a0 2024-04-06T14:46:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.014+0000 7f94e5ffb700 1 -- 172.21.15.32:0/2935226052 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f94e00139e0 con 0x7f94f01037a0 2024-04-06T14:46:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.015+0000 7f94e5ffb700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f94dc06cc20 0x7f94dc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.015+0000 7f94e5ffb700 1 -- 172.21.15.32:0/2935226052 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f94e008d290 con 0x7f94f01037a0 2024-04-06T14:46:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.015+0000 7f94f5438700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f94dc06cc20 0x7f94dc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.015+0000 7f94f5438700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f94dc06cc20 0x7f94dc06f0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f94ec007fd0 tx=0x7f94ec009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:06.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.019+0000 7f94e5ffb700 1 -- 172.21.15.32:0/2935226052 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f94e0057250 con 0x7f94f01037a0 2024-04-06T14:46:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.241+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f94f004bed0 con 0x7f94dc06cc20 2024-04-06T14:46:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.242+0000 7f94e5ffb700 1 -- 172.21.15.32:0/2935226052 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f94f004bed0 con 0x7f94dc06cc20 2024-04-06T14:46:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f94dc06cc20 msgr2=0x7f94dc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f94dc06cc20 0x7f94dc06f0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f94ec007fd0 tx=0x7f94ec009040 comp rx=0 tx=0).stop 2024-04-06T14:46:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 msgr2=0x7f94f0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 0x7f94f0070800 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009380 tx=0x7f94e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 shutdown_connections 2024-04-06T14:46:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f94dc06cc20 0x7f94dc06f0d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f0102e00 0x7f94f00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 --2- 172.21.15.32:0/2935226052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f01037a0 0x7f94f0070800 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 >> 172.21.15.32:0/2935226052 conn(0x7f94f00f89b0 msgr2=0x7f94f00f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 shutdown_connections 2024-04-06T14:46:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.244+0000 7f94f769c700 1 -- 172.21.15.32:0/2935226052 wait complete. 2024-04-06T14:46:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.347+0000 7f9f72199700 1 -- 172.21.15.32:0/3489264053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 msgr2=0x7f9f6c0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.347+0000 7f9f72199700 1 --2- 172.21.15.32:0/3489264053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 0x7f9f6c0fbd30 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c004940 tx=0x7f9f5c009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.347+0000 7f9f72199700 1 -- 172.21.15.32:0/3489264053 shutdown_connections 2024-04-06T14:46:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.347+0000 7f9f72199700 1 --2- 172.21.15.32:0/3489264053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 0x7f9f6c0fbd30 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.347+0000 7f9f72199700 1 --2- 172.21.15.32:0/3489264053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c101d20 0x7f9f6c1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.347+0000 7f9f72199700 1 -- 172.21.15.32:0/3489264053 >> 172.21.15.32:0/3489264053 conn(0x7f9f6c0f78d0 msgr2=0x7f9f6c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.347+0000 7f9f72199700 1 -- 172.21.15.32:0/3489264053 shutdown_connections 2024-04-06T14:46:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.347+0000 7f9f72199700 1 -- 172.21.15.32:0/3489264053 wait complete. 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f72199700 1 Processor -- start 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f72199700 1 -- start start 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f72199700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c101d20 0x7f9f6c10ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f72199700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 0x7f9f6c10ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f72199700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c10f5a0 con 0x7f9f6c1026c0 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f72199700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c10f6e0 con 0x7f9f6c101d20 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f6affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 0x7f9f6c10ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f6affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 0x7f9f6c10ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59136/0 (socket says 172.21.15.32:59136) 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f6affd700 1 -- 172.21.15.32:0/3557376804 learned_addr learned my addr 172.21.15.32:0/3557376804 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f6b7fe700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c101d20 0x7f9f6c10ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f6affd700 1 -- 172.21.15.32:0/3557376804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c101d20 msgr2=0x7f9f6c10ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f6affd700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c101d20 0x7f9f6c10ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.348+0000 7f9f6affd700 1 -- 172.21.15.32:0/3557376804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c005040 con 0x7f9f6c1026c0 2024-04-06T14:46:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.349+0000 7f9f6affd700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 0x7f9f6c10ef50 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c000c00 tx=0x7f9f5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.349+0000 7f9f63fff700 1 -- 172.21.15.32:0/3557376804 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f5c021070 con 0x7f9f6c1026c0 2024-04-06T14:46:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.349+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c117a80 con 0x7f9f6c1026c0 2024-04-06T14:46:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.349+0000 7f9f63fff700 1 -- 172.21.15.32:0/3557376804 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f5c013de0 con 0x7f9f6c1026c0 2024-04-06T14:46:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.349+0000 7f9f63fff700 1 -- 172.21.15.32:0/3557376804 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f5c00e070 con 0x7f9f6c1026c0 2024-04-06T14:46:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.349+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c117f60 con 0x7f9f6c1026c0 2024-04-06T14:46:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.350+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c0fd470 con 0x7f9f6c1026c0 2024-04-06T14:46:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.351+0000 7f9f63fff700 1 -- 172.21.15.32:0/3557376804 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9f5c01c600 con 0x7f9f6c1026c0 2024-04-06T14:46:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.351+0000 7f9f63fff700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9f5806cc20 0x7f9f5806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.351+0000 7f9f63fff700 1 -- 172.21.15.32:0/3557376804 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9f5c08d8e0 con 0x7f9f6c1026c0 2024-04-06T14:46:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.351+0000 7f9f6b7fe700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9f5806cc20 0x7f9f5806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.352+0000 7f9f6b7fe700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9f5806cc20 0x7f9f5806f0d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009b80 tx=0x7f9f5400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:06.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.355+0000 7f9f63fff700 1 -- 172.21.15.32:0/3557376804 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9f5c0579e0 con 0x7f9f6c1026c0 2024-04-06T14:46:06.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.565+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f6c04bed0 con 0x7f9f5806cc20 2024-04-06T14:46:06.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.575+0000 7f9f63fff700 1 -- 172.21.15.32:0/3557376804 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9f6c04bed0 con 0x7f9f5806cc20 2024-04-06T14:46:06.566 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:46:06.566 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (26m) 4m ago 37m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:46:06.566 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (23m) 4m ago 37m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:46:06.566 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (23m) 4m ago 36m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:46:06.566 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (26m) 4m ago 37m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:46:06.567 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (20m) 4m ago 32m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:46:06.567 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (19m) 4m ago 31m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:46:06.567 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (19m) 4m ago 31m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:46:06.567 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (4m) 4m ago 31m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:46:06.567 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (28m) 4m ago 39m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:46:06.567 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (27m) 4m ago 36m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:46:06.568 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (24m) 4m ago 39m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:46:06.568 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (24m) 4m ago 36m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:46:06.568 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (27m) 4m ago 37m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:46:06.568 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (27m) 4m ago 36m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:46:06.568 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (23m) 4m ago 35m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:46:06.568 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (22m) 4m ago 35m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:46:06.569 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (22m) 4m ago 34m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:46:06.569 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (21m) 4m ago 34m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:46:06.569 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (20m) 4m ago 34m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:46:06.569 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (20m) 4m ago 33m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:46:06.569 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (26m) 4m ago 37m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.577+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9f5806cc20 msgr2=0x7f9f5806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.577+0000 7f9f72199700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9f5806cc20 0x7f9f5806f0d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009b80 tx=0x7f9f5400b040 comp rx=0 tx=0).stop 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.577+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 msgr2=0x7f9f6c10ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.577+0000 7f9f72199700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 0x7f9f6c10ef50 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c000c00 tx=0x7f9f5c009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.577+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 shutdown_connections 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.577+0000 7f9f72199700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9f5806cc20 0x7f9f5806f0d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.577+0000 7f9f72199700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c101d20 0x7f9f6c10ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.577+0000 7f9f72199700 1 --2- 172.21.15.32:0/3557376804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1026c0 0x7f9f6c10ef50 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.577+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 >> 172.21.15.32:0/3557376804 conn(0x7f9f6c0f78d0 msgr2=0x7f9f6c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.578+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 shutdown_connections 2024-04-06T14:46:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.578+0000 7f9f72199700 1 -- 172.21.15.32:0/3557376804 wait complete. 2024-04-06T14:46:06.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.679+0000 7f17fdf60700 1 -- 172.21.15.32:0/1986720089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 msgr2=0x7f17f80fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.679+0000 7f17fdf60700 1 --2- 172.21.15.32:0/1986720089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 0x7f17f80fce20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f17e0004940 tx=0x7f17e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.679+0000 7f17fdf60700 1 -- 172.21.15.32:0/1986720089 shutdown_connections 2024-04-06T14:46:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.679+0000 7f17fdf60700 1 --2- 172.21.15.32:0/1986720089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 0x7f17f80fce20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.679+0000 7f17fdf60700 1 --2- 172.21.15.32:0/1986720089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8102e10 0x7f17f81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.679+0000 7f17fdf60700 1 -- 172.21.15.32:0/1986720089 >> 172.21.15.32:0/1986720089 conn(0x7f17f80f89a0 msgr2=0x7f17f80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.679+0000 7f17fdf60700 1 -- 172.21.15.32:0/1986720089 shutdown_connections 2024-04-06T14:46:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.679+0000 7f17fdf60700 1 -- 172.21.15.32:0/1986720089 wait complete. 2024-04-06T14:46:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.679+0000 7f17fdf60700 1 Processor -- start 2024-04-06T14:46:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17fdf60700 1 -- start start 2024-04-06T14:46:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17fdf60700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8102e10 0x7f17f8113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17fdf60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 0x7f17f8114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17fdf60700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8114660 con 0x7f17f81037b0 2024-04-06T14:46:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17fdf60700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f810da50 con 0x7f17f8102e10 2024-04-06T14:46:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17f6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 0x7f17f8114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17f6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 0x7f17f8114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59144/0 (socket says 172.21.15.32:59144) 2024-04-06T14:46:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17f6ffd700 1 -- 172.21.15.32:0/964176028 learned_addr learned my addr 172.21.15.32:0/964176028 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17f77fe700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8102e10 0x7f17f8113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17f6ffd700 1 -- 172.21.15.32:0/964176028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8102e10 msgr2=0x7f17f8113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17f6ffd700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8102e10 0x7f17f8113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17f6ffd700 1 -- 172.21.15.32:0/964176028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e0005040 con 0x7f17f81037b0 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.680+0000 7f17f6ffd700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 0x7f17f8114010 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009380 tx=0x7f17e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.681+0000 7f17effff700 1 -- 172.21.15.32:0/964176028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17e000e040 con 0x7f17f81037b0 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.681+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f810dcd0 con 0x7f17f81037b0 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.681+0000 7f17effff700 1 -- 172.21.15.32:0/964176028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17e0013420 con 0x7f17f81037b0 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.681+0000 7f17effff700 1 -- 172.21.15.32:0/964176028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17e001c630 con 0x7f17f81037b0 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.681+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f810e290 con 0x7f17f81037b0 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.682+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f80fe560 con 0x7f17f81037b0 2024-04-06T14:46:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.682+0000 7f17effff700 1 -- 172.21.15.32:0/964176028 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f17e000bb50 con 0x7f17f81037b0 2024-04-06T14:46:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.683+0000 7f17effff700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f17e406cc20 0x7f17e406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.683+0000 7f17effff700 1 -- 172.21.15.32:0/964176028 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f17e008d240 con 0x7f17f81037b0 2024-04-06T14:46:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.683+0000 7f17f77fe700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f17e406cc20 0x7f17e406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.683+0000 7f17f77fe700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f17e406cc20 0x7f17e406f0d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f17e8004990 tx=0x7f17e8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:06.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.687+0000 7f17effff700 1 -- 172.21.15.32:0/964176028 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f17e005ad10 con 0x7f17f81037b0 2024-04-06T14:46:06.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.974+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f17f8109db0 con 0x7f17f81037b0 2024-04-06T14:46:06.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.974+0000 7f17effff700 1 -- 172.21.15.32:0/964176028 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f17e005a8a0 con 0x7f17f81037b0 2024-04-06T14:46:06.965 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:46:06.965 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:46:06.965 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:46:06.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:46:06.965 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:46:06.965 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:46:06.966 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:46:06.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f17e406cc20 msgr2=0x7f17e406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f17e406cc20 0x7f17e406f0d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f17e8004990 tx=0x7f17e8009ee0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 msgr2=0x7f17f8114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:06.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 0x7f17f8114010 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009380 tx=0x7f17e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:06.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 shutdown_connections 2024-04-06T14:46:06.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f17e406cc20 0x7f17e406f0d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8102e10 0x7f17f8113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 --2- 172.21.15.32:0/964176028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f81037b0 0x7f17f8114010 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:06.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 >> 172.21.15.32:0/964176028 conn(0x7f17f80f89a0 msgr2=0x7f17f80f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:06.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 shutdown_connections 2024-04-06T14:46:06.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:06.976+0000 7f17fdf60700 1 -- 172.21.15.32:0/964176028 wait complete. 2024-04-06T14:46:07.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.079+0000 7f6538a46700 1 -- 172.21.15.32:0/3879014148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65341026b0 msgr2=0x7f65340fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.079+0000 7f6538a46700 1 --2- 172.21.15.32:0/3879014148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65341026b0 0x7f65340fbd20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6524004940 tx=0x7f6524009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:07.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.079+0000 7f6538a46700 1 -- 172.21.15.32:0/3879014148 shutdown_connections 2024-04-06T14:46:07.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.079+0000 7f6538a46700 1 --2- 172.21.15.32:0/3879014148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65341026b0 0x7f65340fbd20 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.079+0000 7f6538a46700 1 --2- 172.21.15.32:0/3879014148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6534101d10 0x7f65341020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.079+0000 7f6538a46700 1 -- 172.21.15.32:0/3879014148 >> 172.21.15.32:0/3879014148 conn(0x7f65340f7880 msgr2=0x7f65340f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.079+0000 7f6538a46700 1 -- 172.21.15.32:0/3879014148 shutdown_connections 2024-04-06T14:46:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.079+0000 7f6538a46700 1 -- 172.21.15.32:0/3879014148 wait complete. 2024-04-06T14:46:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f6538a46700 1 Processor -- start 2024-04-06T14:46:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f6538a46700 1 -- start start 2024-04-06T14:46:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f6538a46700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6534101d10 0x7f653410eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f6538a46700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65341026b0 0x7f653410f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f6538a46700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653410f950 con 0x7f6534101d10 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f6538a46700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653410fa90 con 0x7f65341026b0 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f653259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6534101d10 0x7f653410eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f653259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6534101d10 0x7f653410eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59160/0 (socket says 172.21.15.32:59160) 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f653259c700 1 -- 172.21.15.32:0/280263857 learned_addr learned my addr 172.21.15.32:0/280263857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f6531d9b700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65341026b0 0x7f653410f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f653259c700 1 -- 172.21.15.32:0/280263857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65341026b0 msgr2=0x7f653410f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f653259c700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65341026b0 0x7f653410f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f653259c700 1 -- 172.21.15.32:0/280263857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6524005040 con 0x7f6534101d10 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.080+0000 7f653259c700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6534101d10 0x7f653410eed0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f652800eb70 tx=0x7f652800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.081+0000 7f6522ffd700 1 -- 172.21.15.32:0/280263857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6528007cc0 con 0x7f6534101d10 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.081+0000 7f6522ffd700 1 -- 172.21.15.32:0/280263857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f652800f040 con 0x7f6534101d10 2024-04-06T14:46:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.081+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6534117d80 con 0x7f6534101d10 2024-04-06T14:46:07.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.081+0000 7f6522ffd700 1 -- 172.21.15.32:0/280263857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f652800c9f0 con 0x7f6534101d10 2024-04-06T14:46:07.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.081+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65341182b0 con 0x7f6534101d10 2024-04-06T14:46:07.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.082+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65340fd460 con 0x7f6534101d10 2024-04-06T14:46:07.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.082+0000 7f6522ffd700 1 -- 172.21.15.32:0/280263857 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f652800cb50 con 0x7f6534101d10 2024-04-06T14:46:07.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.083+0000 7f6522ffd700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f651c06cc20 0x7f651c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:07.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.083+0000 7f6522ffd700 1 -- 172.21.15.32:0/280263857 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f652808bb70 con 0x7f6534101d10 2024-04-06T14:46:07.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.083+0000 7f6531d9b700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f651c06cc20 0x7f651c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:07.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.083+0000 7f6531d9b700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f651c06cc20 0x7f651c06f0d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f652400b0d0 tx=0x7f6524009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.087+0000 7f6522ffd700 1 -- 172.21.15.32:0/280263857 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6528055bb0 con 0x7f6534101d10 2024-04-06T14:46:07.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.336+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f653405f5f0 con 0x7f6534101d10 2024-04-06T14:46:07.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.337+0000 7f6522ffd700 1 -- 172.21.15.32:0/280263857 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f65280591d0 con 0x7f6534101d10 2024-04-06T14:46:07.327 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:46:07.327 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:46:07.327 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:46:07.327 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:46:07.327 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:46:07.328 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:46:07.329 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:46:07.330 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:46:07.330 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:46:07.330 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:46:07.330 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:46:07.330 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:46:07.330 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f651c06cc20 msgr2=0x7f651c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f651c06cc20 0x7f651c06f0d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f652400b0d0 tx=0x7f6524009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6534101d10 msgr2=0x7f653410eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6534101d10 0x7f653410eed0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f652800eb70 tx=0x7f652800b040 comp rx=0 tx=0).stop 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 shutdown_connections 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f651c06cc20 0x7f651c06f0d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6534101d10 0x7f653410eed0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 --2- 172.21.15.32:0/280263857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65341026b0 0x7f653410f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 >> 172.21.15.32:0/280263857 conn(0x7f65340f7880 msgr2=0x7f65340f88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 shutdown_connections 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.339+0000 7f6538a46700 1 -- 172.21.15.32:0/280263857 wait complete. 2024-04-06T14:46:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:46:07.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 -- 172.21.15.32:0/1544925819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 msgr2=0x7fa1040fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 --2- 172.21.15.32:0/1544925819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 0x7fa1040fbd20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4004940 tx=0x7fa0f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:07.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 -- 172.21.15.32:0/1544925819 shutdown_connections 2024-04-06T14:46:07.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 --2- 172.21.15.32:0/1544925819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 0x7fa1040fbd20 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 --2- 172.21.15.32:0/1544925819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa104101d10 0x7fa1041020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 -- 172.21.15.32:0/1544925819 >> 172.21.15.32:0/1544925819 conn(0x7fa1040f7880 msgr2=0x7fa1040f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 -- 172.21.15.32:0/1544925819 shutdown_connections 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 -- 172.21.15.32:0/1544925819 wait complete. 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 Processor -- start 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.443+0000 7fa108847700 1 -- start start 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa108847700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa104101d10 0x7fa10410ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa108847700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 0x7fa10410f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa108847700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10410f910 con 0x7fa1041026b0 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa108847700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10410fa50 con 0x7fa104101d10 2024-04-06T14:46:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa101d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 0x7fa10410f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa101d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 0x7fa10410f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59176/0 (socket says 172.21.15.32:59176) 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa101d9b700 1 -- 172.21.15.32:0/1549860304 learned_addr learned my addr 172.21.15.32:0/1549860304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa10259c700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa104101d10 0x7fa10410ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa101d9b700 1 -- 172.21.15.32:0/1549860304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa104101d10 msgr2=0x7fa10410ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa101d9b700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa104101d10 0x7fa10410ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa101d9b700 1 -- 172.21.15.32:0/1549860304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f4005040 con 0x7fa1041026b0 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa101d9b700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 0x7fa10410f2c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009390 tx=0x7fa0f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa0f2ffd700 1 -- 172.21.15.32:0/1549860304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f400e040 con 0x7fa1041026b0 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa0f2ffd700 1 -- 172.21.15.32:0/1549860304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0f4013420 con 0x7fa1041026b0 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa0f2ffd700 1 -- 172.21.15.32:0/1549860304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f401c630 con 0x7fa1041026b0 2024-04-06T14:46:07.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.444+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa104117df0 con 0x7fa1041026b0 2024-04-06T14:46:07.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.445+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa104118300 con 0x7fa1041026b0 2024-04-06T14:46:07.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.445+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1040fd460 con 0x7fa1041026b0 2024-04-06T14:46:07.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.446+0000 7fa0f2ffd700 1 -- 172.21.15.32:0/1549860304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa0f40139e0 con 0x7fa1041026b0 2024-04-06T14:46:07.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.446+0000 7fa0f2ffd700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa0ec06cc30 0x7fa0ec06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:07.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.447+0000 7fa0f2ffd700 1 -- 172.21.15.32:0/1549860304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa0f408d2c0 con 0x7fa1041026b0 2024-04-06T14:46:07.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.447+0000 7fa10259c700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa0ec06cc30 0x7fa0ec06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:07.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.447+0000 7fa10259c700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa0ec06cc30 0x7fa0ec06f0e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8004990 tx=0x7fa0f8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:07.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.451+0000 7fa0f2ffd700 1 -- 172.21.15.32:0/1549860304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa0f4057300 con 0x7fa1041026b0 2024-04-06T14:46:07.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.683+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa10404bed0 con 0x7fa0ec06cc30 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.684+0000 7fa0f2ffd700 1 -- 172.21.15.32:0/1549860304 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa10404bed0 con 0x7fa0ec06cc30 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:46:07.674 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:46:07.675 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:46:07.675 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:46:07.675 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:46:07.675 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:46:07.675 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:46:07.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa0ec06cc30 msgr2=0x7fa0ec06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa0ec06cc30 0x7fa0ec06f0e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8004990 tx=0x7fa0f8009ee0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 msgr2=0x7fa10410f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 0x7fa10410f2c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009390 tx=0x7fa0f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:07.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 shutdown_connections 2024-04-06T14:46:07.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa0ec06cc30 0x7fa0ec06f0e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa104101d10 0x7fa10410ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 --2- 172.21.15.32:0/1549860304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1041026b0 0x7fa10410f2c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 >> 172.21.15.32:0/1549860304 conn(0x7fa1040f7880 msgr2=0x7fa1040f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:07.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 shutdown_connections 2024-04-06T14:46:07.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.686+0000 7fa108847700 1 -- 172.21.15.32:0/1549860304 wait complete. 2024-04-06T14:46:07.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.789+0000 7f0d93e15700 1 -- 172.21.15.32:0/3249373496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c101d10 msgr2=0x7f0d8c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.789+0000 7f0d93e15700 1 --2- 172.21.15.32:0/3249373496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c101d10 0x7f0d8c1020e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0d80004940 tx=0x7f0d80009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:07.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.789+0000 7f0d93e15700 1 -- 172.21.15.32:0/3249373496 shutdown_connections 2024-04-06T14:46:07.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.789+0000 7f0d93e15700 1 --2- 172.21.15.32:0/3249373496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8c1026b0 0x7f0d8c0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.789+0000 7f0d93e15700 1 --2- 172.21.15.32:0/3249373496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c101d10 0x7f0d8c1020e0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.789+0000 7f0d93e15700 1 -- 172.21.15.32:0/3249373496 >> 172.21.15.32:0/3249373496 conn(0x7f0d8c0f7880 msgr2=0x7f0d8c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.789+0000 7f0d93e15700 1 -- 172.21.15.32:0/3249373496 shutdown_connections 2024-04-06T14:46:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.789+0000 7f0d93e15700 1 -- 172.21.15.32:0/3249373496 wait complete. 2024-04-06T14:46:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d93e15700 1 Processor -- start 2024-04-06T14:46:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d93e15700 1 -- start start 2024-04-06T14:46:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d93e15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8c101d10 0x7f0d8c10eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d93e15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c1026b0 0x7f0d8c10f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d93e15700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c10f940 con 0x7f0d8c1026b0 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d93e15700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c10fa80 con 0x7f0d8c101d10 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d913b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c1026b0 0x7f0d8c10f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d913b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c1026b0 0x7f0d8c10f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59192/0 (socket says 172.21.15.32:59192) 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d913b0700 1 -- 172.21.15.32:0/2901623711 learned_addr learned my addr 172.21.15.32:0/2901623711 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.790+0000 7f0d91bb1700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8c101d10 0x7f0d8c10eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.791+0000 7f0d913b0700 1 -- 172.21.15.32:0/2901623711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8c101d10 msgr2=0x7f0d8c10eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.791+0000 7f0d913b0700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8c101d10 0x7f0d8c10eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.791+0000 7f0d913b0700 1 -- 172.21.15.32:0/2901623711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80005040 con 0x7f0d8c1026b0 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.791+0000 7f0d913b0700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c1026b0 0x7f0d8c10f400 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0d88002a50 tx=0x7f0d880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.791+0000 7f0d7e7fc700 1 -- 172.21.15.32:0/2901623711 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d88007cc0 con 0x7f0d8c1026b0 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.791+0000 7f0d7e7fc700 1 -- 172.21.15.32:0/2901623711 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d8800f040 con 0x7f0d8c1026b0 2024-04-06T14:46:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.791+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c117d70 con 0x7f0d8c1026b0 2024-04-06T14:46:07.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.791+0000 7f0d7e7fc700 1 -- 172.21.15.32:0/2901623711 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8800da30 con 0x7f0d8c1026b0 2024-04-06T14:46:07.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.791+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c1182a0 con 0x7f0d8c1026b0 2024-04-06T14:46:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.792+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c0fd460 con 0x7f0d8c1026b0 2024-04-06T14:46:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.793+0000 7f0d7e7fc700 1 -- 172.21.15.32:0/2901623711 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0d8800dbd0 con 0x7f0d8c1026b0 2024-04-06T14:46:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.793+0000 7f0d7e7fc700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0d7806cc20 0x7f0d7806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.793+0000 7f0d7e7fc700 1 -- 172.21.15.32:0/2901623711 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0d8808bc20 con 0x7f0d8c1026b0 2024-04-06T14:46:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.793+0000 7f0d91bb1700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0d7806cc20 0x7f0d7806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.794+0000 7f0d91bb1700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0d7806cc20 0x7f0d7806f0d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0d80004f90 tx=0x7f0d80009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:07.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:07.798+0000 7f0d7e7fc700 1 -- 172.21.15.32:0/2901623711 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0d88055c60 con 0x7f0d8c1026b0 2024-04-06T14:46:07.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:07 smithi086 ceph-mon[110975]: from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:07.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:07 smithi086 ceph-mon[110975]: pgmap v780: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:07.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:07 smithi086 ceph-mon[110975]: from='client.35434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:07.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:07 smithi086 ceph-mon[110975]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:07.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:07 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/964176028' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:46:07.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:07 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/280263857' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:46:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:07 smithi032 ceph-mon[147414]: from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:07 smithi032 ceph-mon[147414]: pgmap v780: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:07 smithi032 ceph-mon[147414]: from='client.35434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:07 smithi032 ceph-mon[147414]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:07.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:07 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/964176028' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:46:07.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:07 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/280263857' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:46:08.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.082+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0d8c04bed0 con 0x7f0d8c1026b0 2024-04-06T14:46:08.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.082+0000 7f0d7e7fc700 1 -- 172.21.15.32:0/2901623711 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f0d88059280 con 0x7f0d8c1026b0 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:46:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:46:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:46:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0d7806cc20 msgr2=0x7f0d7806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0d7806cc20 0x7f0d7806f0d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0d80004f90 tx=0x7f0d80009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c1026b0 msgr2=0x7f0d8c10f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c1026b0 0x7f0d8c10f400 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0d88002a50 tx=0x7f0d880046b0 comp rx=0 tx=0).stop 2024-04-06T14:46:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 shutdown_connections 2024-04-06T14:46:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0d7806cc20 0x7f0d7806f0d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8c101d10 0x7f0d8c10eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 --2- 172.21.15.32:0/2901623711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8c1026b0 0x7f0d8c10f400 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:08.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 >> 172.21.15.32:0/2901623711 conn(0x7f0d8c0f7880 msgr2=0x7f0d8c0f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 shutdown_connections 2024-04-06T14:46:08.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:08.085+0000 7f0d93e15700 1 -- 172.21.15.32:0/2901623711 wait complete. 2024-04-06T14:46:08.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:46:08.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:08 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2901623711' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:46:08.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:46:08.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:08 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2901623711' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:46:09.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:09 smithi086 ceph-mon[110975]: from='client.35450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:09.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:09 smithi086 ceph-mon[110975]: pgmap v781: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:09.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:09 smithi032 ceph-mon[147414]: from='client.35450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:09.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:09 smithi032 ceph-mon[147414]: pgmap v781: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:11.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:11 smithi086 ceph-mon[110975]: pgmap v782: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:11.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:11 smithi032 ceph-mon[147414]: pgmap v782: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:13.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:13 smithi032 ceph-mon[147414]: pgmap v783: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:13.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:13 smithi086 ceph-mon[110975]: pgmap v783: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:15.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:15 smithi086 ceph-mon[110975]: pgmap v784: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:15.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:15 smithi032 ceph-mon[147414]: pgmap v784: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:17.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:17 smithi086 ceph-mon[110975]: pgmap v785: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:17.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:17 smithi032 ceph-mon[147414]: pgmap v785: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:19.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:19 smithi086 ceph-mon[110975]: pgmap v786: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:19.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:19 smithi032 ceph-mon[147414]: pgmap v786: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:21.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:21 smithi086 ceph-mon[110975]: pgmap v787: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:21.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:21 smithi032 ceph-mon[147414]: pgmap v787: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:23.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:23 smithi032 ceph-mon[147414]: pgmap v788: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:23.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:46:23.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:23 smithi086 ceph-mon[110975]: pgmap v788: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:23.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:46:25.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:25 smithi086 ceph-mon[110975]: pgmap v789: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:25.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:25 smithi032 ceph-mon[147414]: pgmap v789: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:27.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:27 smithi086 ceph-mon[110975]: pgmap v790: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:27.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:27 smithi032 ceph-mon[147414]: pgmap v790: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:29.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:29 smithi086 ceph-mon[110975]: pgmap v791: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:29.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:29 smithi032 ceph-mon[147414]: pgmap v791: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:31.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:31 smithi086 ceph-mon[110975]: pgmap v792: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:31.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:31 smithi032 ceph-mon[147414]: pgmap v792: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:33.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:33 smithi032 ceph-mon[147414]: pgmap v793: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:33.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:33 smithi086 ceph-mon[110975]: pgmap v793: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:35.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:35 smithi086 ceph-mon[110975]: pgmap v794: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:35.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:35 smithi032 ceph-mon[147414]: pgmap v794: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:37.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:37 smithi086 ceph-mon[110975]: pgmap v795: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:37.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:37 smithi032 ceph-mon[147414]: pgmap v795: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:38.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.202+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2998580552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd4102e10 msgr2=0x7f1cd41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.202+0000 7f1cdb2d5700 1 --2- 172.21.15.32:0/2998580552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd4102e10 0x7f1cd41031e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0004940 tx=0x7f1cd0009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.202+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2998580552 shutdown_connections 2024-04-06T14:46:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.202+0000 7f1cdb2d5700 1 --2- 172.21.15.32:0/2998580552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cd41037b0 0x7f1cd40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.202+0000 7f1cdb2d5700 1 --2- 172.21.15.32:0/2998580552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd4102e10 0x7f1cd41031e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.202+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2998580552 >> 172.21.15.32:0/2998580552 conn(0x7f1cd40f8980 msgr2=0x7f1cd40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.202+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2998580552 shutdown_connections 2024-04-06T14:46:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.202+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2998580552 wait complete. 2024-04-06T14:46:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.203+0000 7f1cdb2d5700 1 Processor -- start 2024-04-06T14:46:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.203+0000 7f1cdb2d5700 1 -- start start 2024-04-06T14:46:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.203+0000 7f1cdb2d5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cd4102e10 0x7f1cd4113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cdb2d5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd41037b0 0x7f1cd4114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cdb2d5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd41146a0 con 0x7f1cd41037b0 2024-04-06T14:46:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cdb2d5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd410da90 con 0x7f1cd4102e10 2024-04-06T14:46:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cd8870700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd41037b0 0x7f1cd4114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cd8870700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd41037b0 0x7f1cd4114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39936/0 (socket says 172.21.15.32:39936) 2024-04-06T14:46:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cd8870700 1 -- 172.21.15.32:0/2360727278 learned_addr learned my addr 172.21.15.32:0/2360727278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cd9071700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cd4102e10 0x7f1cd4113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cd8870700 1 -- 172.21.15.32:0/2360727278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cd4102e10 msgr2=0x7f1cd4113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cd8870700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cd4102e10 0x7f1cd4113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cd8870700 1 -- 172.21.15.32:0/2360727278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd0005040 con 0x7f1cd41037b0 2024-04-06T14:46:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cd8870700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd41037b0 0x7f1cd4114050 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8002a50 tx=0x7f1cc8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:38.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cc5ffb700 1 -- 172.21.15.32:0/2360727278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cc8007cc0 con 0x7f1cd41037b0 2024-04-06T14:46:38.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd410dd70 con 0x7f1cd41037b0 2024-04-06T14:46:38.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cc5ffb700 1 -- 172.21.15.32:0/2360727278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cc800f040 con 0x7f1cd41037b0 2024-04-06T14:46:38.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.204+0000 7f1cc5ffb700 1 -- 172.21.15.32:0/2360727278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cc800d920 con 0x7f1cd41037b0 2024-04-06T14:46:38.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.205+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd410e330 con 0x7f1cd41037b0 2024-04-06T14:46:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.206+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd40fe560 con 0x7f1cd41037b0 2024-04-06T14:46:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.207+0000 7f1cc5ffb700 1 -- 172.21.15.32:0/2360727278 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1cc8009680 con 0x7f1cd41037b0 2024-04-06T14:46:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.207+0000 7f1cc5ffb700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1cc006cc30 0x7f1cc006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.207+0000 7f1cc5ffb700 1 -- 172.21.15.32:0/2360727278 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1cc808c440 con 0x7f1cd41037b0 2024-04-06T14:46:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.207+0000 7f1cd9071700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1cc006cc30 0x7f1cc006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.208+0000 7f1cd9071700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1cc006cc30 0x7f1cc006f0e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000b840 tx=0x7f1cd0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:38.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.212+0000 7f1cc5ffb700 1 -- 172.21.15.32:0/2360727278 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1cc8059f10 con 0x7f1cd41037b0 2024-04-06T14:46:38.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.432+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cd4109e70 con 0x7f1cc006cc30 2024-04-06T14:46:38.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.433+0000 7f1cc5ffb700 1 -- 172.21.15.32:0/2360727278 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1cd0008ee0 con 0x7f1cc006cc30 2024-04-06T14:46:38.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1cc006cc30 msgr2=0x7f1cc006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1cc006cc30 0x7f1cc006f0e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000b840 tx=0x7f1cd0009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:38.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd41037b0 msgr2=0x7f1cd4114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd41037b0 0x7f1cd4114050 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8002a50 tx=0x7f1cc8004600 comp rx=0 tx=0).stop 2024-04-06T14:46:38.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 shutdown_connections 2024-04-06T14:46:38.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1cc006cc30 0x7f1cc006f0e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cd4102e10 0x7f1cd4113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 --2- 172.21.15.32:0/2360727278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cd41037b0 0x7f1cd4114050 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 >> 172.21.15.32:0/2360727278 conn(0x7f1cd40f8980 msgr2=0x7f1cd40f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:38.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.435+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 shutdown_connections 2024-04-06T14:46:38.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.436+0000 7f1cdb2d5700 1 -- 172.21.15.32:0/2360727278 wait complete. 2024-04-06T14:46:38.435 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:46:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.539+0000 7fa882d92700 1 -- 172.21.15.32:0/3622981158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 msgr2=0x7fa87c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.539+0000 7fa882d92700 1 --2- 172.21.15.32:0/3622981158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 0x7fa87c102140 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa870004940 tx=0x7fa870009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.540+0000 7fa882d92700 1 -- 172.21.15.32:0/3622981158 shutdown_connections 2024-04-06T14:46:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.540+0000 7fa882d92700 1 --2- 172.21.15.32:0/3622981158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa87c102710 0x7fa87c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.540+0000 7fa882d92700 1 --2- 172.21.15.32:0/3622981158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 0x7fa87c102140 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.540+0000 7fa882d92700 1 -- 172.21.15.32:0/3622981158 >> 172.21.15.32:0/3622981158 conn(0x7fa87c072bc0 msgr2=0x7fa87c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.540+0000 7fa882d92700 1 -- 172.21.15.32:0/3622981158 shutdown_connections 2024-04-06T14:46:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.540+0000 7fa882d92700 1 -- 172.21.15.32:0/3622981158 wait complete. 2024-04-06T14:46:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.540+0000 7fa882d92700 1 Processor -- start 2024-04-06T14:46:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.540+0000 7fa882d92700 1 -- start start 2024-04-06T14:46:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa882d92700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 0x7fa87c112760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa882d92700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa87c102710 0x7fa87c112ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa882d92700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c1132f0 con 0x7fa87c101d70 2024-04-06T14:46:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa882d92700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c10c6e0 con 0x7fa87c102710 2024-04-06T14:46:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa880b2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 0x7fa87c112760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa880b2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 0x7fa87c112760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39956/0 (socket says 172.21.15.32:39956) 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa880b2e700 1 -- 172.21.15.32:0/1328540929 learned_addr learned my addr 172.21.15.32:0/1328540929 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa87bfff700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa87c102710 0x7fa87c112ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa880b2e700 1 -- 172.21.15.32:0/1328540929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa87c102710 msgr2=0x7fa87c112ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa880b2e700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa87c102710 0x7fa87c112ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa880b2e700 1 -- 172.21.15.32:0/1328540929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870005040 con 0x7fa87c101d70 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.541+0000 7fa87bfff700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa87c102710 0x7fa87c112ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.542+0000 7fa880b2e700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 0x7fa87c112760 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa87000b3b0 tx=0x7fa870009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.542+0000 7fa8797fa700 1 -- 172.21.15.32:0/1328540929 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa87000e040 con 0x7fa87c101d70 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.542+0000 7fa8797fa700 1 -- 172.21.15.32:0/1328540929 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa87000bcd0 con 0x7fa87c101d70 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.542+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87c10c960 con 0x7fa87c101d70 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.542+0000 7fa8797fa700 1 -- 172.21.15.32:0/1328540929 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa87001c630 con 0x7fa87c101d70 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.542+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87c10cf20 con 0x7fa87c101d70 2024-04-06T14:46:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.543+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87c0ff080 con 0x7fa87c101d70 2024-04-06T14:46:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.544+0000 7fa8797fa700 1 -- 172.21.15.32:0/1328540929 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa870013960 con 0x7fa87c101d70 2024-04-06T14:46:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.544+0000 7fa8797fa700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86c06cc20 0x7fa86c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.544+0000 7fa8797fa700 1 -- 172.21.15.32:0/1328540929 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa87008d820 con 0x7fa87c101d70 2024-04-06T14:46:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.544+0000 7fa87bfff700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86c06cc20 0x7fa86c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:38.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.545+0000 7fa87bfff700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86c06cc20 0x7fa86c06f0d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa87c10d8a0 tx=0x7fa868009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.548+0000 7fa8797fa700 1 -- 172.21.15.32:0/1328540929 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa87005b2f0 con 0x7fa87c101d70 2024-04-06T14:46:38.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.769+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa87c10ddb0 con 0x7fa86c06cc20 2024-04-06T14:46:38.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.770+0000 7fa8797fa700 1 -- 172.21.15.32:0/1328540929 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa87c10ddb0 con 0x7fa86c06cc20 2024-04-06T14:46:38.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86c06cc20 msgr2=0x7fa86c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86c06cc20 0x7fa86c06f0d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa87c10d8a0 tx=0x7fa868009040 comp rx=0 tx=0).stop 2024-04-06T14:46:38.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 msgr2=0x7fa87c112760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 0x7fa87c112760 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa87000b3b0 tx=0x7fa870009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:38.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 shutdown_connections 2024-04-06T14:46:38.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86c06cc20 0x7fa86c06f0d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa87c101d70 0x7fa87c112760 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 --2- 172.21.15.32:0/1328540929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa87c102710 0x7fa87c112ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 >> 172.21.15.32:0/1328540929 conn(0x7fa87c072bc0 msgr2=0x7fa87c0faa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:38.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 shutdown_connections 2024-04-06T14:46:38.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.772+0000 7fa882d92700 1 -- 172.21.15.32:0/1328540929 wait complete. 2024-04-06T14:46:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.875+0000 7fc45ecae700 1 -- 172.21.15.32:0/1999313485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 msgr2=0x7fc4580fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.875+0000 7fc45ecae700 1 --2- 172.21.15.32:0/1999313485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 0x7fc4580fce10 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fc444004940 tx=0x7fc444009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.875+0000 7fc45ecae700 1 -- 172.21.15.32:0/1999313485 shutdown_connections 2024-04-06T14:46:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.875+0000 7fc45ecae700 1 --2- 172.21.15.32:0/1999313485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 0x7fc4580fce10 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.875+0000 7fc45ecae700 1 --2- 172.21.15.32:0/1999313485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc458102e00 0x7fc4581031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.875+0000 7fc45ecae700 1 -- 172.21.15.32:0/1999313485 >> 172.21.15.32:0/1999313485 conn(0x7fc4580f89b0 msgr2=0x7fc4580fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.875+0000 7fc45ecae700 1 -- 172.21.15.32:0/1999313485 shutdown_connections 2024-04-06T14:46:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.875+0000 7fc45ecae700 1 -- 172.21.15.32:0/1999313485 wait complete. 2024-04-06T14:46:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.875+0000 7fc45ecae700 1 Processor -- start 2024-04-06T14:46:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc45ecae700 1 -- start start 2024-04-06T14:46:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc45ecae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc458102e00 0x7fc458113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc45ecae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 0x7fc458114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc45ecae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458114690 con 0x7fc4581037a0 2024-04-06T14:46:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc45ecae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45810da80 con 0x7fc458102e00 2024-04-06T14:46:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc457fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 0x7fc458114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc457fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 0x7fc458114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39974/0 (socket says 172.21.15.32:39974) 2024-04-06T14:46:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc457fff700 1 -- 172.21.15.32:0/1054605792 learned_addr learned my addr 172.21.15.32:0/1054605792 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc45ca4a700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc458102e00 0x7fc458113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc457fff700 1 -- 172.21.15.32:0/1054605792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc458102e00 msgr2=0x7fc458113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc457fff700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc458102e00 0x7fc458113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.876+0000 7fc457fff700 1 -- 172.21.15.32:0/1054605792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444005040 con 0x7fc4581037a0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.877+0000 7fc457fff700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 0x7fc458114040 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc4440093b0 tx=0x7fc444009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.877+0000 7fc4557fa700 1 -- 172.21.15.32:0/1054605792 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc44400e040 con 0x7fc4581037a0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.877+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45810dd00 con 0x7fc4581037a0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.877+0000 7fc4557fa700 1 -- 172.21.15.32:0/1054605792 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc444013420 con 0x7fc4581037a0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.877+0000 7fc4557fa700 1 -- 172.21.15.32:0/1054605792 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc44401c630 con 0x7fc4581037a0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.877+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45810e2c0 con 0x7fc4581037a0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.878+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4580fe550 con 0x7fc4581037a0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.878+0000 7fc4557fa700 1 -- 172.21.15.32:0/1054605792 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc4440139e0 con 0x7fc4581037a0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.879+0000 7fc4557fa700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc44806cc20 0x7fc44806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.879+0000 7fc4557fa700 1 -- 172.21.15.32:0/1054605792 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc44408d290 con 0x7fc4581037a0 2024-04-06T14:46:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.879+0000 7fc45ca4a700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc44806cc20 0x7fc44806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.879+0000 7fc45ca4a700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc44806cc20 0x7fc44806f0d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc44c004990 tx=0x7fc44c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:38.883+0000 7fc4557fa700 1 -- 172.21.15.32:0/1054605792 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc44405ae10 con 0x7fc4581037a0 2024-04-06T14:46:38.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:46:39.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.095+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc458109da0 con 0x7fc44806cc20 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.105+0000 7fc4557fa700 1 -- 172.21.15.32:0/1054605792 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc458109da0 con 0x7fc44806cc20 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (27m) 4m ago 38m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (24m) 4m ago 38m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (24m) 5m ago 37m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (26m) 4m ago 38m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (20m) 4m ago 32m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (20m) 4m ago 32m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (20m) 5m ago 32m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:46:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (5m) 5m ago 32m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (28m) 4m ago 39m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (27m) 5m ago 37m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (25m) 4m ago 39m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (25m) 5m ago 37m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (27m) 4m ago 38m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (27m) 5m ago 37m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (23m) 4m ago 36m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (23m) 4m ago 35m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (22m) 4m ago 35m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (22m) 5m ago 35m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (21m) 5m ago 34m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (20m) 5m ago 34m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:46:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (27m) 4m ago 37m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:46:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc44806cc20 msgr2=0x7fc44806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc44806cc20 0x7fc44806f0d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc44c004990 tx=0x7fc44c009ee0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 msgr2=0x7fc458114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 0x7fc458114040 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc4440093b0 tx=0x7fc444009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 shutdown_connections 2024-04-06T14:46:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc44806cc20 0x7fc44806f0d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc458102e00 0x7fc458113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 --2- 172.21.15.32:0/1054605792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4581037a0 0x7fc458114040 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 >> 172.21.15.32:0/1054605792 conn(0x7fc4580f89b0 msgr2=0x7fc4580f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 shutdown_connections 2024-04-06T14:46:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.107+0000 7fc45ecae700 1 -- 172.21.15.32:0/1054605792 wait complete. 2024-04-06T14:46:39.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:46:39.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.209+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/1526005882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8101d70 msgr2=0x7fd1e8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.209+0000 7fd1ee9f0700 1 --2- 172.21.15.32:0/1526005882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8101d70 0x7fd1e8102140 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8004940 tx=0x7fd1d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:39.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.209+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/1526005882 shutdown_connections 2024-04-06T14:46:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.209+0000 7fd1ee9f0700 1 --2- 172.21.15.32:0/1526005882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8102710 0x7fd1e80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.209+0000 7fd1ee9f0700 1 --2- 172.21.15.32:0/1526005882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8101d70 0x7fd1e8102140 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.209+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/1526005882 >> 172.21.15.32:0/1526005882 conn(0x7fd1e8072bc0 msgr2=0x7fd1e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.209+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/1526005882 shutdown_connections 2024-04-06T14:46:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.209+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/1526005882 wait complete. 2024-04-06T14:46:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1ee9f0700 1 Processor -- start 2024-04-06T14:46:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1ee9f0700 1 -- start start 2024-04-06T14:46:39.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1ee9f0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8101d70 0x7fd1e8103320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1ee9f0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8102710 0x7fd1e8103860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1ee9f0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e8103df0 con 0x7fd1e8102710 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1ee9f0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e8103f30 con 0x7fd1e8101d70 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8102710 0x7fd1e8103860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8102710 0x7fd1e8103860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39998/0 (socket says 172.21.15.32:39998) 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1e77fe700 1 -- 172.21.15.32:0/641002462 learned_addr learned my addr 172.21.15.32:0/641002462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1e7fff700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8101d70 0x7fd1e8103320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1e77fe700 1 -- 172.21.15.32:0/641002462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8101d70 msgr2=0x7fd1e8103320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1e77fe700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8101d70 0x7fd1e8103320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.210+0000 7fd1e77fe700 1 -- 172.21.15.32:0/641002462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8005040 con 0x7fd1e8102710 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.211+0000 7fd1e77fe700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8102710 0x7fd1e8103860 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0002a50 tx=0x7fd1d0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.211+0000 7fd1e4ff9700 1 -- 172.21.15.32:0/641002462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d0007cc0 con 0x7fd1e8102710 2024-04-06T14:46:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.211+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e8119d80 con 0x7fd1e8102710 2024-04-06T14:46:39.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.211+0000 7fd1e4ff9700 1 -- 172.21.15.32:0/641002462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1d000f040 con 0x7fd1e8102710 2024-04-06T14:46:39.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.211+0000 7fd1e4ff9700 1 -- 172.21.15.32:0/641002462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d000d990 con 0x7fd1e8102710 2024-04-06T14:46:39.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.211+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e811a200 con 0x7fd1e8102710 2024-04-06T14:46:39.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.212+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e80fd4c0 con 0x7fd1e8102710 2024-04-06T14:46:39.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.212+0000 7fd1e4ff9700 1 -- 172.21.15.32:0/641002462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd1d0009710 con 0x7fd1e8102710 2024-04-06T14:46:39.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.213+0000 7fd1e4ff9700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1d406cc30 0x7fd1d406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:39.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.213+0000 7fd1e4ff9700 1 -- 172.21.15.32:0/641002462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd1d008be60 con 0x7fd1e8102710 2024-04-06T14:46:39.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.213+0000 7fd1e7fff700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1d406cc30 0x7fd1d406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:39.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.213+0000 7fd1e7fff700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1d406cc30 0x7fd1d406f0e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800b3b0 tx=0x7fd1d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:39.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.217+0000 7fd1e4ff9700 1 -- 172.21.15.32:0/641002462 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd1d00598b0 con 0x7fd1e8102710 2024-04-06T14:46:39.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.505+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd1e804bed0 con 0x7fd1e8102710 2024-04-06T14:46:39.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.505+0000 7fd1e4ff9700 1 -- 172.21.15.32:0/641002462 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fd1d0059440 con 0x7fd1e8102710 2024-04-06T14:46:39.495 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:46:39.496 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:46:39.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:46:39.497 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:46:39.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1d406cc30 msgr2=0x7fd1d406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1d406cc30 0x7fd1d406f0e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800b3b0 tx=0x7fd1d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:39.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8102710 msgr2=0x7fd1e8103860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8102710 0x7fd1e8103860 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0002a50 tx=0x7fd1d0004600 comp rx=0 tx=0).stop 2024-04-06T14:46:39.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 shutdown_connections 2024-04-06T14:46:39.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd1d406cc30 0x7fd1d406f0e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8101d70 0x7fd1e8103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 --2- 172.21.15.32:0/641002462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e8102710 0x7fd1e8103860 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 >> 172.21.15.32:0/641002462 conn(0x7fd1e8072bc0 msgr2=0x7fd1e8105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:39.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 shutdown_connections 2024-04-06T14:46:39.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.507+0000 7fd1ee9f0700 1 -- 172.21.15.32:0/641002462 wait complete. 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.610+0000 7f9a82dcb700 1 -- 172.21.15.32:0/1508054139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fde90 msgr2=0x7f9a7c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.610+0000 7f9a82dcb700 1 --2- 172.21.15.32:0/1508054139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fde90 0x7f9a7c0fe260 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9a70004940 tx=0x7f9a70009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.610+0000 7f9a82dcb700 1 -- 172.21.15.32:0/1508054139 shutdown_connections 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.610+0000 7f9a82dcb700 1 --2- 172.21.15.32:0/1508054139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c0fe830 0x7f9a7c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.610+0000 7f9a82dcb700 1 --2- 172.21.15.32:0/1508054139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fde90 0x7f9a7c0fe260 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.610+0000 7f9a82dcb700 1 -- 172.21.15.32:0/1508054139 >> 172.21.15.32:0/1508054139 conn(0x7f9a7c072bc0 msgr2=0x7f9a7c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.610+0000 7f9a82dcb700 1 -- 172.21.15.32:0/1508054139 shutdown_connections 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.610+0000 7f9a82dcb700 1 -- 172.21.15.32:0/1508054139 wait complete. 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.611+0000 7f9a82dcb700 1 Processor -- start 2024-04-06T14:46:39.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.611+0000 7f9a82dcb700 1 -- start start 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.611+0000 7f9a82dcb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c0fde90 0x7f9a7c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.611+0000 7f9a82dcb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fe830 0x7f9a7c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.611+0000 7f9a82dcb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0fc820 con 0x7f9a7c0fe830 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.611+0000 7f9a82dcb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0fc960 con 0x7f9a7c0fde90 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.611+0000 7f9a7bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fe830 0x7f9a7c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a7bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fe830 0x7f9a7c0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40026/0 (socket says 172.21.15.32:40026) 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a7bfff700 1 -- 172.21.15.32:0/65702705 learned_addr learned my addr 172.21.15.32:0/65702705 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a80b67700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c0fde90 0x7f9a7c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a7bfff700 1 -- 172.21.15.32:0/65702705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c0fde90 msgr2=0x7f9a7c0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a7bfff700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c0fde90 0x7f9a7c0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a7bfff700 1 -- 172.21.15.32:0/65702705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a70005040 con 0x7f9a7c0fe830 2024-04-06T14:46:39.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a7bfff700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fe830 0x7f9a7c0fc290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800c8c0 tx=0x7f9a680044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:39.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a797fa700 1 -- 172.21.15.32:0/65702705 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a6800a760 con 0x7f9a7c0fe830 2024-04-06T14:46:39.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a797fa700 1 -- 172.21.15.32:0/65702705 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a6800ad70 con 0x7f9a7c0fe830 2024-04-06T14:46:39.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c119df0 con 0x7f9a7c0fe830 2024-04-06T14:46:39.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a797fa700 1 -- 172.21.15.32:0/65702705 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a680113a0 con 0x7f9a7c0fe830 2024-04-06T14:46:39.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.612+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c11a1f0 con 0x7f9a7c0fe830 2024-04-06T14:46:39.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.613+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c065e30 con 0x7f9a7c0fe830 2024-04-06T14:46:39.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.614+0000 7f9a797fa700 1 -- 172.21.15.32:0/65702705 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9a68012400 con 0x7f9a7c0fe830 2024-04-06T14:46:39.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.614+0000 7f9a797fa700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a6c06cc30 0x7f9a6c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:39.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.614+0000 7f9a797fa700 1 -- 172.21.15.32:0/65702705 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9a6808bfc0 con 0x7f9a7c0fe830 2024-04-06T14:46:39.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.614+0000 7f9a80b67700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a6c06cc30 0x7f9a6c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:39.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.615+0000 7f9a80b67700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a6c06cc30 0x7f9a6c06f0e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000b3b0 tx=0x7f9a70009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:39.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.619+0000 7f9a797fa700 1 -- 172.21.15.32:0/65702705 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9a680560c0 con 0x7f9a7c0fe830 2024-04-06T14:46:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.866+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9a7c04bed0 con 0x7f9a7c0fe830 2024-04-06T14:46:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.866+0000 7f9a797fa700 1 -- 172.21.15.32:0/65702705 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f9a680596e0 con 0x7f9a7c0fe830 2024-04-06T14:46:39.856 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:46:39.856 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:46:39.856 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:46:39.856 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:46:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:46:39.856 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:46:39.857 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:46:39.858 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:46:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:46:39.859 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:46:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:46:39.859 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:46:39.859 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:46:39.859 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:46:39.859 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a6c06cc30 msgr2=0x7f9a6c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a6c06cc30 0x7f9a6c06f0e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000b3b0 tx=0x7f9a70009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fe830 msgr2=0x7f9a7c0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fe830 0x7f9a7c0fc290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800c8c0 tx=0x7f9a680044c0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 shutdown_connections 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9a6c06cc30 0x7f9a6c06f0e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c0fde90 0x7f9a7c0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 --2- 172.21.15.32:0/65702705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0fe830 0x7f9a7c0fc290 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 >> 172.21.15.32:0/65702705 conn(0x7f9a7c072bc0 msgr2=0x7f9a7c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 shutdown_connections 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.868+0000 7f9a82dcb700 1 -- 172.21.15.32:0/65702705 wait complete. 2024-04-06T14:46:39.860 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:46:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.972+0000 7fdb58926700 1 -- 172.21.15.32:0/930388023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb54102e10 msgr2=0x7fdb541031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.972+0000 7fdb58926700 1 --2- 172.21.15.32:0/930388023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb54102e10 0x7fdb541031e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fdb44004940 tx=0x7fdb44009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.972+0000 7fdb58926700 1 -- 172.21.15.32:0/930388023 shutdown_connections 2024-04-06T14:46:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.972+0000 7fdb58926700 1 --2- 172.21.15.32:0/930388023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb541037b0 0x7fdb540fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.972+0000 7fdb58926700 1 --2- 172.21.15.32:0/930388023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb54102e10 0x7fdb541031e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.972+0000 7fdb58926700 1 -- 172.21.15.32:0/930388023 >> 172.21.15.32:0/930388023 conn(0x7fdb540f8980 msgr2=0x7fdb540fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.972+0000 7fdb58926700 1 -- 172.21.15.32:0/930388023 shutdown_connections 2024-04-06T14:46:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.972+0000 7fdb58926700 1 -- 172.21.15.32:0/930388023 wait complete. 2024-04-06T14:46:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb58926700 1 Processor -- start 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb58926700 1 -- start start 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb58926700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb54102e10 0x7fdb54113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb58926700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb541037b0 0x7fdb54114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb58926700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb54114660 con 0x7fdb541037b0 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb58926700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5410da50 con 0x7fdb54102e10 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb51d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb541037b0 0x7fdb54114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb51d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb541037b0 0x7fdb54114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40052/0 (socket says 172.21.15.32:40052) 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb51d9b700 1 -- 172.21.15.32:0/2902717256 learned_addr learned my addr 172.21.15.32:0/2902717256 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.973+0000 7fdb5259c700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb54102e10 0x7fdb54113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.974+0000 7fdb51d9b700 1 -- 172.21.15.32:0/2902717256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb54102e10 msgr2=0x7fdb54113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.974+0000 7fdb51d9b700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb54102e10 0x7fdb54113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.974+0000 7fdb51d9b700 1 -- 172.21.15.32:0/2902717256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb44005040 con 0x7fdb541037b0 2024-04-06T14:46:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.974+0000 7fdb51d9b700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb541037b0 0x7fdb54114010 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c002a50 tx=0x7fdb3c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.974+0000 7fdb4affd700 1 -- 172.21.15.32:0/2902717256 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3c007cc0 con 0x7fdb541037b0 2024-04-06T14:46:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.974+0000 7fdb4affd700 1 -- 172.21.15.32:0/2902717256 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb3c00f040 con 0x7fdb541037b0 2024-04-06T14:46:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.974+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5410dd30 con 0x7fdb541037b0 2024-04-06T14:46:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.974+0000 7fdb4affd700 1 -- 172.21.15.32:0/2902717256 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3c00d920 con 0x7fdb541037b0 2024-04-06T14:46:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.974+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5410e2f0 con 0x7fdb541037b0 2024-04-06T14:46:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.975+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb540fe560 con 0x7fdb541037b0 2024-04-06T14:46:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.975+0000 7fdb4affd700 1 -- 172.21.15.32:0/2902717256 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdb3c009680 con 0x7fdb541037b0 2024-04-06T14:46:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.976+0000 7fdb4affd700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdb4006cc30 0x7fdb4006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.976+0000 7fdb4affd700 1 -- 172.21.15.32:0/2902717256 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdb3c08bde0 con 0x7fdb541037b0 2024-04-06T14:46:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.976+0000 7fdb5259c700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdb4006cc30 0x7fdb4006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.976+0000 7fdb5259c700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdb4006cc30 0x7fdb4006f0e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400b840 tx=0x7fdb44009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:39.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:39 smithi032 ceph-mon[147414]: pgmap v796: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:39.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:39 smithi032 ceph-mon[147414]: from='client.35458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:39.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:39 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/641002462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:46:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:39.980+0000 7fdb4affd700 1 -- 172.21.15.32:0/2902717256 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdb3c059830 con 0x7fdb541037b0 2024-04-06T14:46:40.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:39 smithi086 ceph-mon[110975]: pgmap v796: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:40.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:39 smithi086 ceph-mon[110975]: from='client.35458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:40.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:39 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/641002462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:46:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.205+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb54109dd0 con 0x7fdb4006cc30 2024-04-06T14:46:40.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.206+0000 7fdb4affd700 1 -- 172.21.15.32:0/2902717256 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdb54109dd0 con 0x7fdb4006cc30 2024-04-06T14:46:40.196 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:46:40.196 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:46:40.196 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:46:40.196 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:46:40.196 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:46:40.196 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:46:40.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:46:40.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:46:40.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:46:40.197 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:46:40.197 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:46:40.197 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:46:40.197 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:46:40.197 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:46:40.197 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdb4006cc30 msgr2=0x7fdb4006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdb4006cc30 0x7fdb4006f0e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400b840 tx=0x7fdb44009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb541037b0 msgr2=0x7fdb54114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb541037b0 0x7fdb54114010 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c002a50 tx=0x7fdb3c004600 comp rx=0 tx=0).stop 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 shutdown_connections 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fdb4006cc30 0x7fdb4006f0e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb54102e10 0x7fdb54113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 --2- 172.21.15.32:0/2902717256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb541037b0 0x7fdb54114010 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 >> 172.21.15.32:0/2902717256 conn(0x7fdb540f8980 msgr2=0x7fdb540f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 shutdown_connections 2024-04-06T14:46:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.208+0000 7fdb58926700 1 -- 172.21.15.32:0/2902717256 wait complete. 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.311+0000 7f91b9da2700 1 -- 172.21.15.32:0/3307558356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b4102e10 msgr2=0x7f91b41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.311+0000 7f91b9da2700 1 --2- 172.21.15.32:0/3307558356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b4102e10 0x7f91b41031e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f91a4004940 tx=0x7f91a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.311+0000 7f91b9da2700 1 -- 172.21.15.32:0/3307558356 shutdown_connections 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.311+0000 7f91b9da2700 1 --2- 172.21.15.32:0/3307558356 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91b41037b0 0x7f91b40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.311+0000 7f91b9da2700 1 --2- 172.21.15.32:0/3307558356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b4102e10 0x7f91b41031e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.311+0000 7f91b9da2700 1 -- 172.21.15.32:0/3307558356 >> 172.21.15.32:0/3307558356 conn(0x7f91b40f89a0 msgr2=0x7f91b40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.311+0000 7f91b9da2700 1 -- 172.21.15.32:0/3307558356 shutdown_connections 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.311+0000 7f91b9da2700 1 -- 172.21.15.32:0/3307558356 wait complete. 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.311+0000 7f91b9da2700 1 Processor -- start 2024-04-06T14:46:40.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b9da2700 1 -- start start 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b9da2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91b4102e10 0x7f91b4113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b9da2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b41037b0 0x7f91b4114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b9da2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b41146a0 con 0x7f91b41037b0 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b9da2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b410da90 con 0x7f91b4102e10 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b41037b0 0x7f91b4114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b41037b0 0x7f91b4114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40074/0 (socket says 172.21.15.32:40074) 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b2ffd700 1 -- 172.21.15.32:0/3668334132 learned_addr learned my addr 172.21.15.32:0/3668334132 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b37fe700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91b4102e10 0x7f91b4113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b2ffd700 1 -- 172.21.15.32:0/3668334132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91b4102e10 msgr2=0x7f91b4113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b2ffd700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91b4102e10 0x7f91b4113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b2ffd700 1 -- 172.21.15.32:0/3668334132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a4005040 con 0x7f91b41037b0 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.312+0000 7f91b37fe700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91b4102e10 0x7f91b4113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:46:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.313+0000 7f91b2ffd700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b41037b0 0x7f91b4114050 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f919c002a80 tx=0x7f919c004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.313+0000 7f91abfff700 1 -- 172.21.15.32:0/3668334132 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919c007cc0 con 0x7f91b41037b0 2024-04-06T14:46:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.313+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b410dd70 con 0x7f91b41037b0 2024-04-06T14:46:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.313+0000 7f91abfff700 1 -- 172.21.15.32:0/3668334132 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f919c00f040 con 0x7f91b41037b0 2024-04-06T14:46:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.313+0000 7f91abfff700 1 -- 172.21.15.32:0/3668334132 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919c00da40 con 0x7f91b41037b0 2024-04-06T14:46:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.313+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b410e330 con 0x7f91b41037b0 2024-04-06T14:46:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.314+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b40fe560 con 0x7f91b41037b0 2024-04-06T14:46:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.314+0000 7f91abfff700 1 -- 172.21.15.32:0/3668334132 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f919c00dbe0 con 0x7f91b41037b0 2024-04-06T14:46:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.315+0000 7f91abfff700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f91a006cc30 0x7f91a006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:46:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.315+0000 7f91abfff700 1 -- 172.21.15.32:0/3668334132 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f919c08b780 con 0x7f91b41037b0 2024-04-06T14:46:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.315+0000 7f91b37fe700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f91a006cc30 0x7f91a006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:46:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.315+0000 7f91b37fe700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f91a006cc30 0x7f91a006f0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f91a400b840 tx=0x7f91a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:46:40.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.319+0000 7f91abfff700 1 -- 172.21.15.32:0/3668334132 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f919c0591d0 con 0x7f91b41037b0 2024-04-06T14:46:40.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.609+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f91b405f5f0 con 0x7f91b41037b0 2024-04-06T14:46:40.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.609+0000 7f91abfff700 1 -- 172.21.15.32:0/3668334132 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f919c017ed0 con 0x7f91b41037b0 2024-04-06T14:46:40.599 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:46:40.599 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:46:40.599 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:46:40.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:46:40.599 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:46:40.599 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:46:40.600 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:46:40.601 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:46:40.601 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:46:40.601 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:46:40.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f91a006cc30 msgr2=0x7f91a006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:40.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f91a006cc30 0x7f91a006f0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f91a400b840 tx=0x7f91a4009e30 comp rx=0 tx=0).stop 2024-04-06T14:46:40.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b41037b0 msgr2=0x7f91b4114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:46:40.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b41037b0 0x7f91b4114050 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f919c002a80 tx=0x7f919c004630 comp rx=0 tx=0).stop 2024-04-06T14:46:40.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 shutdown_connections 2024-04-06T14:46:40.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f91a006cc30 0x7f91a006f0e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:40.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91b4102e10 0x7f91b4113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:40.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 --2- 172.21.15.32:0/3668334132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91b41037b0 0x7f91b4114050 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:46:40.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 >> 172.21.15.32:0/3668334132 conn(0x7f91b40f89a0 msgr2=0x7f91b40f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:46:40.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 shutdown_connections 2024-04-06T14:46:40.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:46:40.611+0000 7f91b9da2700 1 -- 172.21.15.32:0/3668334132 wait complete. 2024-04-06T14:46:40.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:40 smithi032 ceph-mon[147414]: from='client.35462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:40.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:40 smithi032 ceph-mon[147414]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:40.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:40 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/65702705' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:46:40.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:40 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3668334132' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:46:41.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:40 smithi086 ceph-mon[110975]: from='client.35462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:41.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:40 smithi086 ceph-mon[110975]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:41.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:40 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/65702705' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:46:41.118 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:40 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3668334132' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:46:41.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:41 smithi032 ceph-mon[147414]: pgmap v797: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:41.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:41 smithi032 ceph-mon[147414]: from='client.35478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:42.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:41 smithi086 ceph-mon[110975]: pgmap v797: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:42.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:41 smithi086 ceph-mon[110975]: from='client.35478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:46:43.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:43 smithi032 ceph-mon[147414]: pgmap v798: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:44.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:43 smithi086 ceph-mon[110975]: pgmap v798: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:45.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:45 smithi032 ceph-mon[147414]: pgmap v799: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:46.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:45 smithi086 ceph-mon[110975]: pgmap v799: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:47.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:47 smithi032 ceph-mon[147414]: pgmap v800: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:48.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:47 smithi086 ceph-mon[110975]: pgmap v800: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:49.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:49 smithi032 ceph-mon[147414]: pgmap v801: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:50.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:49 smithi086 ceph-mon[110975]: pgmap v801: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:51.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:51 smithi032 ceph-mon[147414]: pgmap v802: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:52.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:51 smithi086 ceph-mon[110975]: pgmap v802: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:52.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:46:53.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:46:53.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:53 smithi032 ceph-mon[147414]: pgmap v803: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:54.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:53 smithi086 ceph-mon[110975]: pgmap v803: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:55.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:55 smithi032 ceph-mon[147414]: pgmap v804: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:56.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:55 smithi086 ceph-mon[110975]: pgmap v804: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:57.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:57 smithi032 ceph-mon[147414]: pgmap v805: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:58.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:57 smithi086 ceph-mon[110975]: pgmap v805: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:46:59.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:46:59 smithi032 ceph-mon[147414]: pgmap v806: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:00.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:46:59 smithi086 ceph-mon[110975]: pgmap v806: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:01.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:01 smithi032 ceph-mon[147414]: pgmap v807: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:02.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:01 smithi086 ceph-mon[110975]: pgmap v807: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:03.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:03 smithi032 ceph-mon[147414]: pgmap v808: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:04.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:03 smithi086 ceph-mon[110975]: pgmap v808: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:05.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:05 smithi032 ceph-mon[147414]: pgmap v809: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:06.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:05 smithi086 ceph-mon[110975]: pgmap v809: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:07.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:07 smithi032 ceph-mon[147414]: pgmap v810: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:07.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:47:08.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:07 smithi086 ceph-mon[110975]: pgmap v810: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:08.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:47:09.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:09 smithi032 ceph-mon[147414]: pgmap v811: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:10.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:09 smithi086 ceph-mon[110975]: pgmap v811: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:10.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 -- 172.21.15.32:0/4193972851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c102710 msgr2=0x7f820c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:10.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 --2- 172.21.15.32:0/4193972851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c102710 0x7f820c0fbd80 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f81fc004940 tx=0x7f81fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:10.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 -- 172.21.15.32:0/4193972851 shutdown_connections 2024-04-06T14:47:10.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 --2- 172.21.15.32:0/4193972851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c102710 0x7f820c0fbd80 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:10.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 --2- 172.21.15.32:0/4193972851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f820c101d70 0x7f820c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 -- 172.21.15.32:0/4193972851 >> 172.21.15.32:0/4193972851 conn(0x7f820c072bc0 msgr2=0x7f820c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 -- 172.21.15.32:0/4193972851 shutdown_connections 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 -- 172.21.15.32:0/4193972851 wait complete. 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 Processor -- start 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.730+0000 7f8213d3b700 1 -- start start 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8213d3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c101d70 0x7f820c10e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8213d3b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f820c102710 0x7f820c10eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8213d3b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c10f530 con 0x7f820c101d70 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8213d3b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c10f670 con 0x7f820c102710 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8211ad7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c101d70 0x7f820c10e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8211ad7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c101d70 0x7f820c10e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50836/0 (socket says 172.21.15.32:50836) 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8211ad7700 1 -- 172.21.15.32:0/670518494 learned_addr learned my addr 172.21.15.32:0/670518494 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:10.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f82112d6700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f820c102710 0x7f820c10eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:10.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8211ad7700 1 -- 172.21.15.32:0/670518494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f820c102710 msgr2=0x7f820c10eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:10.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8211ad7700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f820c102710 0x7f820c10eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:10.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8211ad7700 1 -- 172.21.15.32:0/670518494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc005040 con 0x7f820c101d70 2024-04-06T14:47:10.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8211ad7700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c101d70 0x7f820c10e9a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f820800eb40 tx=0x7f820800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:10.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f82027fc700 1 -- 172.21.15.32:0/670518494 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8208007cc0 con 0x7f820c101d70 2024-04-06T14:47:10.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f82027fc700 1 -- 172.21.15.32:0/670518494 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f820800f040 con 0x7f820c101d70 2024-04-06T14:47:10.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.731+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c117a10 con 0x7f820c101d70 2024-04-06T14:47:10.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.732+0000 7f82027fc700 1 -- 172.21.15.32:0/670518494 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8208009570 con 0x7f820c101d70 2024-04-06T14:47:10.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.732+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c117f20 con 0x7f820c101d70 2024-04-06T14:47:10.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.732+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c0ff080 con 0x7f820c101d70 2024-04-06T14:47:10.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.733+0000 7f82027fc700 1 -- 172.21.15.32:0/670518494 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f82080096d0 con 0x7f820c101d70 2024-04-06T14:47:10.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.733+0000 7f82027fc700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81f806cc20 0x7f81f806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:10.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.734+0000 7f82027fc700 1 -- 172.21.15.32:0/670518494 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f820808bc30 con 0x7f820c101d70 2024-04-06T14:47:10.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.734+0000 7f82112d6700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81f806cc20 0x7f81f806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:10.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.734+0000 7f82112d6700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81f806cc20 0x7f81f806f0d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f820c066000 tx=0x7f81fc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:10.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.738+0000 7f82027fc700 1 -- 172.21.15.32:0/670518494 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8208055d20 con 0x7f820c101d70 2024-04-06T14:47:10.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.960+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f820c04bed0 con 0x7f81f806cc20 2024-04-06T14:47:10.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.961+0000 7f82027fc700 1 -- 172.21.15.32:0/670518494 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f820c04bed0 con 0x7f81f806cc20 2024-04-06T14:47:10.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.963+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81f806cc20 msgr2=0x7f81f806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:10.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.963+0000 7f8213d3b700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81f806cc20 0x7f81f806f0d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f820c066000 tx=0x7f81fc009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:10.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.963+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c101d70 msgr2=0x7f820c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:10.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.963+0000 7f8213d3b700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c101d70 0x7f820c10e9a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f820800eb40 tx=0x7f820800b040 comp rx=0 tx=0).stop 2024-04-06T14:47:10.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.963+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 shutdown_connections 2024-04-06T14:47:10.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.963+0000 7f8213d3b700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f81f806cc20 0x7f81f806f0d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:10.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.963+0000 7f8213d3b700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820c101d70 0x7f820c10e9a0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:10.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.963+0000 7f8213d3b700 1 --2- 172.21.15.32:0/670518494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f820c102710 0x7f820c10eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:10.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.963+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 >> 172.21.15.32:0/670518494 conn(0x7f820c072bc0 msgr2=0x7f820c0faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:10.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.964+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 shutdown_connections 2024-04-06T14:47:10.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:10.964+0000 7f8213d3b700 1 -- 172.21.15.32:0/670518494 wait complete. 2024-04-06T14:47:10.962 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:47:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 -- 172.21.15.32:0/3444277219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384101d70 msgr2=0x7f9384102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 --2- 172.21.15.32:0/3444277219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384101d70 0x7f9384102140 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f9380004940 tx=0x7f9380009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 -- 172.21.15.32:0/3444277219 shutdown_connections 2024-04-06T14:47:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 --2- 172.21.15.32:0/3444277219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9384102710 0x7f93840fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 --2- 172.21.15.32:0/3444277219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384101d70 0x7f9384102140 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 -- 172.21.15.32:0/3444277219 >> 172.21.15.32:0/3444277219 conn(0x7f9384072bc0 msgr2=0x7f9384072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 -- 172.21.15.32:0/3444277219 shutdown_connections 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 -- 172.21.15.32:0/3444277219 wait complete. 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 Processor -- start 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.068+0000 7f938bd6d700 1 -- start start 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f938bd6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9384101d70 0x7f9384113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f938bd6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384102710 0x7f9384113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f938bd6d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9384114620 con 0x7f9384102710 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f938bd6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938410da60 con 0x7f9384101d70 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f9389308700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384102710 0x7f9384113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f9389308700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384102710 0x7f9384113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50854/0 (socket says 172.21.15.32:50854) 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f9389308700 1 -- 172.21.15.32:0/2653118978 learned_addr learned my addr 172.21.15.32:0/2653118978 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f9389b09700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9384101d70 0x7f9384113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f9389308700 1 -- 172.21.15.32:0/2653118978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9384101d70 msgr2=0x7f9384113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f9389308700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9384101d70 0x7f9384113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f9389308700 1 -- 172.21.15.32:0/2653118978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9380005040 con 0x7f9384102710 2024-04-06T14:47:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f9389308700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384102710 0x7f9384113fd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9374002a50 tx=0x7f9374004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f937a7fc700 1 -- 172.21.15.32:0/2653118978 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9374007cc0 con 0x7f9384102710 2024-04-06T14:47:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938410dd40 con 0x7f9384102710 2024-04-06T14:47:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f937a7fc700 1 -- 172.21.15.32:0/2653118978 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f937400f040 con 0x7f9384102710 2024-04-06T14:47:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f937a7fc700 1 -- 172.21.15.32:0/2653118978 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f937400d990 con 0x7f9384102710 2024-04-06T14:47:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.069+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938410e300 con 0x7f9384102710 2024-04-06T14:47:11.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.070+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93840fd4c0 con 0x7f9384102710 2024-04-06T14:47:11.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.071+0000 7f937a7fc700 1 -- 172.21.15.32:0/2653118978 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9374009710 con 0x7f9384102710 2024-04-06T14:47:11.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.071+0000 7f937a7fc700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f937006cc20 0x7f937006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:11.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.071+0000 7f937a7fc700 1 -- 172.21.15.32:0/2653118978 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f937408be60 con 0x7f9384102710 2024-04-06T14:47:11.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.071+0000 7f9389b09700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f937006cc20 0x7f937006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:11.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.072+0000 7f9389b09700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f937006cc20 0x7f937006f0d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9380004f90 tx=0x7f9380009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:11.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.076+0000 7f937a7fc700 1 -- 172.21.15.32:0/2653118978 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9374059620 con 0x7f9384102710 2024-04-06T14:47:11.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.299+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9384108d30 con 0x7f937006cc20 2024-04-06T14:47:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.301+0000 7f937a7fc700 1 -- 172.21.15.32:0/2653118978 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9384108d30 con 0x7f937006cc20 2024-04-06T14:47:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f937006cc20 msgr2=0x7f937006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f937006cc20 0x7f937006f0d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9380004f90 tx=0x7f9380009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384102710 msgr2=0x7f9384113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384102710 0x7f9384113fd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9374002a50 tx=0x7f9374004600 comp rx=0 tx=0).stop 2024-04-06T14:47:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 shutdown_connections 2024-04-06T14:47:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f937006cc20 0x7f937006f0d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9384101d70 0x7f9384113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 --2- 172.21.15.32:0/2653118978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9384102710 0x7f9384113fd0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 >> 172.21.15.32:0/2653118978 conn(0x7f9384072bc0 msgr2=0x7f9384065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 shutdown_connections 2024-04-06T14:47:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.303+0000 7f938bd6d700 1 -- 172.21.15.32:0/2653118978 wait complete. 2024-04-06T14:47:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.404+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/1440594422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 msgr2=0x7f6d980fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.404+0000 7f6d9f8b2700 1 --2- 172.21.15.32:0/1440594422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 0x7f6d980fbd80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c004940 tx=0x7f6d8c009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.404+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/1440594422 shutdown_connections 2024-04-06T14:47:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.404+0000 7f6d9f8b2700 1 --2- 172.21.15.32:0/1440594422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 0x7f6d980fbd80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.404+0000 7f6d9f8b2700 1 --2- 172.21.15.32:0/1440594422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d98101d70 0x7f6d98102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.404+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/1440594422 >> 172.21.15.32:0/1440594422 conn(0x7f6d98072bc0 msgr2=0x7f6d98072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.405+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/1440594422 shutdown_connections 2024-04-06T14:47:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.405+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/1440594422 wait complete. 2024-04-06T14:47:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.405+0000 7f6d9f8b2700 1 Processor -- start 2024-04-06T14:47:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.405+0000 7f6d9f8b2700 1 -- start start 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.405+0000 7f6d9f8b2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d98101d70 0x7f6d9810ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.405+0000 7f6d9f8b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 0x7f6d9810ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.405+0000 7f6d9f8b2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9810f5b0 con 0x7f6d98102710 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.405+0000 7f6d9f8b2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9810f6f0 con 0x7f6d98101d70 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.405+0000 7f6d9ce4d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 0x7f6d9810ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d9ce4d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 0x7f6d9810ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50864/0 (socket says 172.21.15.32:50864) 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d9ce4d700 1 -- 172.21.15.32:0/3504848670 learned_addr learned my addr 172.21.15.32:0/3504848670 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d9d64e700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d98101d70 0x7f6d9810ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d9ce4d700 1 -- 172.21.15.32:0/3504848670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d98101d70 msgr2=0x7f6d9810ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d9ce4d700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d98101d70 0x7f6d9810ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d9ce4d700 1 -- 172.21.15.32:0/3504848670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c005040 con 0x7f6d98102710 2024-04-06T14:47:11.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d9ce4d700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 0x7f6d9810ef60 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c000c00 tx=0x7f6d8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:11.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d89ffb700 1 -- 172.21.15.32:0/3504848670 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d8c00e070 con 0x7f6d98102710 2024-04-06T14:47:11.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d98117a90 con 0x7f6d98102710 2024-04-06T14:47:11.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d89ffb700 1 -- 172.21.15.32:0/3504848670 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d8c0093b0 con 0x7f6d98102710 2024-04-06T14:47:11.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d89ffb700 1 -- 172.21.15.32:0/3504848670 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d8c01ccb0 con 0x7f6d98102710 2024-04-06T14:47:11.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.406+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d98117f70 con 0x7f6d98102710 2024-04-06T14:47:11.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.407+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d98104240 con 0x7f6d98102710 2024-04-06T14:47:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.408+0000 7f6d89ffb700 1 -- 172.21.15.32:0/3504848670 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6d8c01c490 con 0x7f6d98102710 2024-04-06T14:47:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.408+0000 7f6d89ffb700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6d8406cc20 0x7f6d8406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.408+0000 7f6d9d64e700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6d8406cc20 0x7f6d8406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.408+0000 7f6d89ffb700 1 -- 172.21.15.32:0/3504848670 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6d8c08d520 con 0x7f6d98102710 2024-04-06T14:47:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.409+0000 7f6d9d64e700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6d8406cc20 0x7f6d8406f0d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009b80 tx=0x7f6d9400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:11.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.413+0000 7f6d89ffb700 1 -- 172.21.15.32:0/3504848670 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6d8c0574e0 con 0x7f6d98102710 2024-04-06T14:47:11.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.625+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d98108d30 con 0x7f6d8406cc20 2024-04-06T14:47:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.635+0000 7f6d89ffb700 1 -- 172.21.15.32:0/3504848670 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6d98108d30 con 0x7f6d8406cc20 2024-04-06T14:47:11.625 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (27m) 5m ago 38m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (24m) 5m ago 39m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (24m) 5m ago 38m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (27m) 5m ago 38m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (21m) 5m ago 33m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (20m) 5m ago 32m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (20m) 5m ago 33m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (5m) 5m ago 32m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (29m) 5m ago 40m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (28m) 5m ago 37m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (25m) 5m ago 40m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:47:11.626 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (25m) 5m ago 37m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:47:11.627 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (28m) 5m ago 38m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:47:11.627 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (28m) 5m ago 37m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:47:11.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (24m) 5m ago 37m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:47:11.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (23m) 5m ago 36m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:47:11.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (23m) 5m ago 36m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:47:11.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (22m) 5m ago 35m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:47:11.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (22m) 5m ago 35m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:47:11.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (21m) 5m ago 34m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:47:11.627 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (27m) 5m ago 38m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6d8406cc20 msgr2=0x7f6d8406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6d8406cc20 0x7f6d8406f0d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009b80 tx=0x7f6d9400b040 comp rx=0 tx=0).stop 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 msgr2=0x7f6d9810ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 0x7f6d9810ef60 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c000c00 tx=0x7f6d8c009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 shutdown_connections 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6d8406cc20 0x7f6d8406f0d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d98101d70 0x7f6d9810ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 --2- 172.21.15.32:0/3504848670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d98102710 0x7f6d9810ef60 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 >> 172.21.15.32:0/3504848670 conn(0x7f6d98072bc0 msgr2=0x7f6d980fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 shutdown_connections 2024-04-06T14:47:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.638+0000 7f6d9f8b2700 1 -- 172.21.15.32:0/3504848670 wait complete. 2024-04-06T14:47:11.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 -- 172.21.15.32:0/1987970479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 msgr2=0x7faaac0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 --2- 172.21.15.32:0/1987970479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 0x7faaac0fce30 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7faa9c004940 tx=0x7faa9c009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 -- 172.21.15.32:0/1987970479 shutdown_connections 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 --2- 172.21.15.32:0/1987970479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 0x7faaac0fce30 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 --2- 172.21.15.32:0/1987970479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaac102e20 0x7faaac1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 -- 172.21.15.32:0/1987970479 >> 172.21.15.32:0/1987970479 conn(0x7faaac0f89d0 msgr2=0x7faaac0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 -- 172.21.15.32:0/1987970479 shutdown_connections 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 -- 172.21.15.32:0/1987970479 wait complete. 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 Processor -- start 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.741+0000 7faab33d1700 1 -- start start 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab33d1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaac102e20 0x7faaac113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab33d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 0x7faaac114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab33d1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaac114660 con 0x7faaac1037c0 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab33d1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaac10da50 con 0x7faaac102e20 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab096c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 0x7faaac114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab096c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 0x7faaac114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50886/0 (socket says 172.21.15.32:50886) 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab096c700 1 -- 172.21.15.32:0/3791119548 learned_addr learned my addr 172.21.15.32:0/3791119548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab116d700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaac102e20 0x7faaac113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab096c700 1 -- 172.21.15.32:0/3791119548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaac102e20 msgr2=0x7faaac113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab096c700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaac102e20 0x7faaac113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab096c700 1 -- 172.21.15.32:0/3791119548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c005040 con 0x7faaac1037c0 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab116d700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaac102e20 0x7faaac113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab096c700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 0x7faaac114010 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0093b0 tx=0x7faa9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faaa1ffb700 1 -- 172.21.15.32:0/3791119548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa9c00e040 con 0x7faaac1037c0 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaac10dcd0 con 0x7faaac1037c0 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.742+0000 7faaa1ffb700 1 -- 172.21.15.32:0/3791119548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa9c013420 con 0x7faaac1037c0 2024-04-06T14:47:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.743+0000 7faaa1ffb700 1 -- 172.21.15.32:0/3791119548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa9c01c630 con 0x7faaac1037c0 2024-04-06T14:47:11.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.743+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaac10e290 con 0x7faaac1037c0 2024-04-06T14:47:11.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.744+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaac0fe570 con 0x7faaac1037c0 2024-04-06T14:47:11.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.744+0000 7faaa1ffb700 1 -- 172.21.15.32:0/3791119548 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faa9c0139e0 con 0x7faaac1037c0 2024-04-06T14:47:11.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.744+0000 7faaa1ffb700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa9806cc20 0x7faa9806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:11.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.745+0000 7faaa1ffb700 1 -- 172.21.15.32:0/3791119548 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faa9c08d310 con 0x7faaac1037c0 2024-04-06T14:47:11.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.745+0000 7faab116d700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa9806cc20 0x7faa9806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:11.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.745+0000 7faab116d700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa9806cc20 0x7faa9806f0d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009fd0 tx=0x7faaa8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:11.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:11.749+0000 7faaa1ffb700 1 -- 172.21.15.32:0/3791119548 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faa9c05ad60 con 0x7faaac1037c0 2024-04-06T14:47:11.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:11 smithi032 ceph-mon[147414]: pgmap v812: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.039+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faaac109dc0 con 0x7faaac1037c0 2024-04-06T14:47:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.040+0000 7faaa1ffb700 1 -- 172.21.15.32:0/3791119548 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7faa9c05a8f0 con 0x7faaac1037c0 2024-04-06T14:47:12.030 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:47:12.030 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:47:12.030 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:47:12.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:47:12.030 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:47:12.030 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:47:12.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:47:12.030 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:47:12.031 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:47:12.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:47:12.031 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:47:12.031 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:47:12.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:47:12.031 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:47:12.031 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:47:12.031 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:47:12.031 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:47:12.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa9806cc20 msgr2=0x7faa9806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa9806cc20 0x7faa9806f0d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009fd0 tx=0x7faaa8009ee0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 msgr2=0x7faaac114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 0x7faaac114010 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0093b0 tx=0x7faa9c009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:12.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 shutdown_connections 2024-04-06T14:47:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7faa9806cc20 0x7faa9806f0d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaac102e20 0x7faaac113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 --2- 172.21.15.32:0/3791119548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaac1037c0 0x7faaac114010 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 >> 172.21.15.32:0/3791119548 conn(0x7faaac0f89d0 msgr2=0x7faaac0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 shutdown_connections 2024-04-06T14:47:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.042+0000 7faab33d1700 1 -- 172.21.15.32:0/3791119548 wait complete. 2024-04-06T14:47:12.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:11 smithi086 ceph-mon[110975]: pgmap v812: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:12.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.145+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2827435306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480702e0 msgr2=0x7f7c480706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.145+0000 7f7c4e2b9700 1 --2- 172.21.15.32:0/2827435306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480702e0 0x7f7c480706b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7c38004940 tx=0x7f7c38009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.145+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2827435306 shutdown_connections 2024-04-06T14:47:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.145+0000 7f7c4e2b9700 1 --2- 172.21.15.32:0/2827435306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c48070bf0 0x7f7c481091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.145+0000 7f7c4e2b9700 1 --2- 172.21.15.32:0/2827435306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480702e0 0x7f7c480706b0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.145+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2827435306 >> 172.21.15.32:0/2827435306 conn(0x7f7c480f8980 msgr2=0x7f7c480fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.145+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2827435306 shutdown_connections 2024-04-06T14:47:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.145+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2827435306 wait complete. 2024-04-06T14:47:12.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.145+0000 7f7c4e2b9700 1 Processor -- start 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c4e2b9700 1 -- start start 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c4e2b9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c480702e0 0x7f7c48115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c4e2b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c48070bf0 0x7f7c48116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c4e2b9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48116870 con 0x7f7c48070bf0 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c4e2b9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4810fc40 con 0x7f7c480702e0 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c48070bf0 0x7f7c48116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c48070bf0 0x7f7c48116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50902/0 (socket says 172.21.15.32:50902) 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c477fe700 1 -- 172.21.15.32:0/2809936191 learned_addr learned my addr 172.21.15.32:0/2809936191 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c47fff700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c480702e0 0x7f7c48115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c477fe700 1 -- 172.21.15.32:0/2809936191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c480702e0 msgr2=0x7f7c48115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c477fe700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c480702e0 0x7f7c48115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.146+0000 7f7c477fe700 1 -- 172.21.15.32:0/2809936191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c38005040 con 0x7f7c48070bf0 2024-04-06T14:47:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.147+0000 7f7c477fe700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c48070bf0 0x7f7c48116220 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c002a50 tx=0x7f7c3c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.147+0000 7f7c44ff9700 1 -- 172.21.15.32:0/2809936191 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c3c007cc0 con 0x7f7c48070bf0 2024-04-06T14:47:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.147+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4810ff20 con 0x7f7c48070bf0 2024-04-06T14:47:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.147+0000 7f7c44ff9700 1 -- 172.21.15.32:0/2809936191 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c3c00f040 con 0x7f7c48070bf0 2024-04-06T14:47:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.147+0000 7f7c44ff9700 1 -- 172.21.15.32:0/2809936191 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c3c009540 con 0x7f7c48070bf0 2024-04-06T14:47:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.147+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c48110460 con 0x7f7c48070bf0 2024-04-06T14:47:12.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.148+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c48071a20 con 0x7f7c48070bf0 2024-04-06T14:47:12.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.148+0000 7f7c44ff9700 1 -- 172.21.15.32:0/2809936191 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7c3c0096a0 con 0x7f7c48070bf0 2024-04-06T14:47:12.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.149+0000 7f7c44ff9700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c3006cc30 0x7f7c3006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:12.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.149+0000 7f7c44ff9700 1 -- 172.21.15.32:0/2809936191 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7c3c08bef0 con 0x7f7c48070bf0 2024-04-06T14:47:12.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.149+0000 7f7c47fff700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c3006cc30 0x7f7c3006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:12.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.149+0000 7f7c47fff700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c3006cc30 0x7f7c3006f0e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800b3b0 tx=0x7f7c38009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:12.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.153+0000 7f7c44ff9700 1 -- 172.21.15.32:0/2809936191 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7c3c0598c0 con 0x7f7c48070bf0 2024-04-06T14:47:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.402+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7c4804bed0 con 0x7f7c48070bf0 2024-04-06T14:47:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.403+0000 7f7c44ff9700 1 -- 172.21.15.32:0/2809936191 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f7c3c059450 con 0x7f7c48070bf0 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:47:12.393 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:47:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:47:12.395 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:47:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c3006cc30 msgr2=0x7f7c3006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c3006cc30 0x7f7c3006f0e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800b3b0 tx=0x7f7c38009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c48070bf0 msgr2=0x7f7c48116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c48070bf0 0x7f7c48116220 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c002a50 tx=0x7f7c3c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 shutdown_connections 2024-04-06T14:47:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7c3006cc30 0x7f7c3006f0e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c480702e0 0x7f7c48115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 --2- 172.21.15.32:0/2809936191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c48070bf0 0x7f7c48116220 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 >> 172.21.15.32:0/2809936191 conn(0x7f7c480f8980 msgr2=0x7f7c48105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:12.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 shutdown_connections 2024-04-06T14:47:12.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.405+0000 7f7c4e2b9700 1 -- 172.21.15.32:0/2809936191 wait complete. 2024-04-06T14:47:12.397 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:47:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.508+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/969612630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 msgr2=0x7f1bd81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.508+0000 7f1bdd8b5700 1 --2- 172.21.15.32:0/969612630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 0x7f1bd81031d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8004940 tx=0x7f1bc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.508+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/969612630 shutdown_connections 2024-04-06T14:47:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.508+0000 7f1bdd8b5700 1 --2- 172.21.15.32:0/969612630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd81037a0 0x7f1bd80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.508+0000 7f1bdd8b5700 1 --2- 172.21.15.32:0/969612630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 0x7f1bd81031d0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.508+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/969612630 >> 172.21.15.32:0/969612630 conn(0x7f1bd80f89b0 msgr2=0x7f1bd80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:12.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.508+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/969612630 shutdown_connections 2024-04-06T14:47:12.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.508+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/969612630 wait complete. 2024-04-06T14:47:12.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.509+0000 7f1bdd8b5700 1 Processor -- start 2024-04-06T14:47:12.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.509+0000 7f1bdd8b5700 1 -- start start 2024-04-06T14:47:12.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.509+0000 7f1bdd8b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 0x7f1bd81138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.509+0000 7f1bdd8b5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd81037a0 0x7f1bd8113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.509+0000 7f1bdd8b5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd8114440 con 0x7f1bd8102e00 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.509+0000 7f1bdd8b5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd810da40 con 0x7f1bd81037a0 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bd6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 0x7f1bd81138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bd6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 0x7f1bd81138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50914/0 (socket says 172.21.15.32:50914) 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bd6ffd700 1 -- 172.21.15.32:0/776194464 learned_addr learned my addr 172.21.15.32:0/776194464 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bd67fc700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd81037a0 0x7f1bd8113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bd6ffd700 1 -- 172.21.15.32:0/776194464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd81037a0 msgr2=0x7f1bd8113df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bd6ffd700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd81037a0 0x7f1bd8113df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bd6ffd700 1 -- 172.21.15.32:0/776194464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc8005040 con 0x7f1bd8102e00 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bd6ffd700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 0x7f1bd81138b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8000c00 tx=0x7f1bc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bbf7fe700 1 -- 172.21.15.32:0/776194464 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bc800e040 con 0x7f1bd8102e00 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bbf7fe700 1 -- 172.21.15.32:0/776194464 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bc8013420 con 0x7f1bd8102e00 2024-04-06T14:47:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd810dcc0 con 0x7f1bd8102e00 2024-04-06T14:47:12.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bbf7fe700 1 -- 172.21.15.32:0/776194464 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bc801c630 con 0x7f1bd8102e00 2024-04-06T14:47:12.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.510+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd810e280 con 0x7f1bd8102e00 2024-04-06T14:47:12.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.511+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd80fe550 con 0x7f1bd8102e00 2024-04-06T14:47:12.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.512+0000 7f1bbf7fe700 1 -- 172.21.15.32:0/776194464 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1bc8013590 con 0x7f1bd8102e00 2024-04-06T14:47:12.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.512+0000 7f1bbf7fe700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bc006cc20 0x7f1bc006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:12.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.512+0000 7f1bbf7fe700 1 -- 172.21.15.32:0/776194464 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1bc808d190 con 0x7f1bd8102e00 2024-04-06T14:47:12.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.512+0000 7f1bd67fc700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bc006cc20 0x7f1bc006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:12.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.513+0000 7f1bd67fc700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bc006cc20 0x7f1bc006f0d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1bd810ec00 tx=0x7f1bcc009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:12.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.517+0000 7f1bbf7fe700 1 -- 172.21.15.32:0/776194464 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1bc805ad20 con 0x7f1bd8102e00 2024-04-06T14:47:12.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.739+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bd8109e30 con 0x7f1bc006cc20 2024-04-06T14:47:12.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.740+0000 7f1bbf7fe700 1 -- 172.21.15.32:0/776194464 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1bd8109e30 con 0x7f1bc006cc20 2024-04-06T14:47:12.730 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:47:12.730 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:47:12.730 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:47:12.730 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:47:12.730 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:47:12.730 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:47:12.731 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:47:12.731 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:47:12.731 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:47:12.731 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:47:12.731 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:47:12.731 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:47:12.731 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:47:12.731 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:47:12.731 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:47:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.742+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bc006cc20 msgr2=0x7f1bc006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.742+0000 7f1bdd8b5700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bc006cc20 0x7f1bc006f0d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1bd810ec00 tx=0x7f1bcc009ee0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.742+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 msgr2=0x7f1bd81138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.742+0000 7f1bdd8b5700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 0x7f1bd81138b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8000c00 tx=0x7f1bc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.742+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 shutdown_connections 2024-04-06T14:47:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.742+0000 7f1bdd8b5700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1bc006cc20 0x7f1bc006f0d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.742+0000 7f1bdd8b5700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bd8102e00 0x7f1bd81138b0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.742+0000 7f1bdd8b5700 1 --2- 172.21.15.32:0/776194464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd81037a0 0x7f1bd8113df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.742+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 >> 172.21.15.32:0/776194464 conn(0x7f1bd80f89b0 msgr2=0x7f1bd80f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.743+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 shutdown_connections 2024-04-06T14:47:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.743+0000 7f1bdd8b5700 1 -- 172.21.15.32:0/776194464 wait complete. 2024-04-06T14:47:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 -- 172.21.15.32:0/3357296998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 msgr2=0x7f27b00fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 --2- 172.21.15.32:0/3357296998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 0x7f27b00fb910 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f279c004940 tx=0x7f279c009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 -- 172.21.15.32:0/3357296998 shutdown_connections 2024-04-06T14:47:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 --2- 172.21.15.32:0/3357296998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 0x7f27b00fb910 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 --2- 172.21.15.32:0/3357296998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b0102e70 0x7f27b0103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 -- 172.21.15.32:0/3357296998 >> 172.21.15.32:0/3357296998 conn(0x7f27b00714c0 msgr2=0x7f27b00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 -- 172.21.15.32:0/3357296998 shutdown_connections 2024-04-06T14:47:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 -- 172.21.15.32:0/3357296998 wait complete. 2024-04-06T14:47:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 Processor -- start 2024-04-06T14:47:12.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.846+0000 7f27b7236700 1 -- start start 2024-04-06T14:47:12.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b7236700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 0x7f27b0113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:12.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b7236700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b0102e70 0x7f27b0113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:12.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b7236700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01145c0 con 0x7f27b00fb4a0 2024-04-06T14:47:12.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b7236700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b010d9b0 con 0x7f27b0102e70 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b4fd2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 0x7f27b0113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b4fd2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 0x7f27b0113a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50932/0 (socket says 172.21.15.32:50932) 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b4fd2700 1 -- 172.21.15.32:0/1312954029 learned_addr learned my addr 172.21.15.32:0/1312954029 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27affff700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b0102e70 0x7f27b0113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b4fd2700 1 -- 172.21.15.32:0/1312954029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b0102e70 msgr2=0x7f27b0113f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b4fd2700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b0102e70 0x7f27b0113f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b4fd2700 1 -- 172.21.15.32:0/1312954029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c005040 con 0x7f27b00fb4a0 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27affff700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b0102e70 0x7f27b0113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27b4fd2700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 0x7f27b0113a30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f27a4002a80 tx=0x7f27a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.847+0000 7f27ad7fa700 1 -- 172.21.15.32:0/1312954029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27a4007cc0 con 0x7f27b00fb4a0 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.848+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b010dc90 con 0x7f27b00fb4a0 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.848+0000 7f27ad7fa700 1 -- 172.21.15.32:0/1312954029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27a400f040 con 0x7f27b00fb4a0 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.848+0000 7f27ad7fa700 1 -- 172.21.15.32:0/1312954029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27a4009640 con 0x7f27b00fb4a0 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.848+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b010e250 con 0x7f27b00fb4a0 2024-04-06T14:47:12.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.849+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b0100110 con 0x7f27b00fb4a0 2024-04-06T14:47:12.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.849+0000 7f27ad7fa700 1 -- 172.21.15.32:0/1312954029 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f27a40097a0 con 0x7f27b00fb4a0 2024-04-06T14:47:12.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.850+0000 7f27ad7fa700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27a006cc30 0x7f27a006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:12.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.850+0000 7f27ad7fa700 1 -- 172.21.15.32:0/1312954029 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f27a408b900 con 0x7f27b00fb4a0 2024-04-06T14:47:12.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.850+0000 7f27affff700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27a006cc30 0x7f27a006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:12.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.850+0000 7f27affff700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27a006cc30 0x7f27a006f0e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f27b010eb70 tx=0x7f279c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:12.854+0000 7f27ad7fa700 1 -- 172.21.15.32:0/1312954029 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f27a40592d0 con 0x7f27b00fb4a0 2024-04-06T14:47:12.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:12 smithi032 ceph-mon[147414]: from='client.35486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:12.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:12 smithi032 ceph-mon[147414]: from='client.35490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:12.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:12 smithi032 ceph-mon[147414]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:12.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:12 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3791119548' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:47:12.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:12 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2809936191' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:47:13.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:12 smithi086 ceph-mon[110975]: from='client.35486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:13.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:12 smithi086 ceph-mon[110975]: from='client.35490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:13.118 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:12 smithi086 ceph-mon[110975]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:13.118 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:12 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3791119548' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:47:13.118 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:12 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2809936191' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:47:13.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.141+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f27b010f080 con 0x7f27b00fb4a0 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.141+0000 7f27ad7fa700 1 -- 172.21.15.32:0/1312954029 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f27a4058e60 con 0x7f27b00fb4a0 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:47:13.131 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:47:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:47:13.133 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:47:13.133 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:47:13.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27a006cc30 msgr2=0x7f27a006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:13.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27a006cc30 0x7f27a006f0e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f27b010eb70 tx=0x7f279c009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:13.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 msgr2=0x7f27b0113a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:13.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 0x7f27b0113a30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f27a4002a80 tx=0x7f27a40046b0 comp rx=0 tx=0).stop 2024-04-06T14:47:13.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 shutdown_connections 2024-04-06T14:47:13.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f27a006cc30 0x7f27a006f0e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:13.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27b00fb4a0 0x7f27b0113a30 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:13.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 --2- 172.21.15.32:0/1312954029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b0102e70 0x7f27b0113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:13.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 >> 172.21.15.32:0/1312954029 conn(0x7f27b00714c0 msgr2=0x7f27b00fa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:13.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 shutdown_connections 2024-04-06T14:47:13.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:13.143+0000 7f27b7236700 1 -- 172.21.15.32:0/1312954029 wait complete. 2024-04-06T14:47:13.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:13 smithi032 ceph-mon[147414]: pgmap v813: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:13.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:13 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1312954029' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:47:14.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:13 smithi086 ceph-mon[110975]: pgmap v813: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:14.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:13 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1312954029' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:47:14.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:14 smithi032 ceph-mon[147414]: from='client.35506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:15.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:14 smithi086 ceph-mon[110975]: from='client.35506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:15.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:15 smithi032 ceph-mon[147414]: pgmap v814: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:16.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:15 smithi086 ceph-mon[110975]: pgmap v814: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:16.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:16 smithi032 ceph-mon[147414]: pgmap v815: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:17.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:16 smithi086 ceph-mon[110975]: pgmap v815: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:19.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:19 smithi032 ceph-mon[147414]: pgmap v816: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:19.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:19 smithi086 ceph-mon[110975]: pgmap v816: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:21.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:21 smithi086 ceph-mon[110975]: pgmap v817: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:21.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:21 smithi032 ceph-mon[147414]: pgmap v817: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:23.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:23 smithi086 ceph-mon[110975]: pgmap v818: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:23.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:47:23.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:23 smithi032 ceph-mon[147414]: pgmap v818: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:23.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:47:25.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:25 smithi086 ceph-mon[110975]: pgmap v819: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:25.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:25 smithi032 ceph-mon[147414]: pgmap v819: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:27.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:27 smithi086 ceph-mon[110975]: pgmap v820: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:27.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:27 smithi032 ceph-mon[147414]: pgmap v820: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:29.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:29 smithi086 ceph-mon[110975]: pgmap v821: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:29.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:29 smithi032 ceph-mon[147414]: pgmap v821: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:31.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:31 smithi086 ceph-mon[110975]: pgmap v822: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:31.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:31 smithi032 ceph-mon[147414]: pgmap v822: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:33.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:33 smithi086 ceph-mon[110975]: pgmap v823: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:33.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:33 smithi032 ceph-mon[147414]: pgmap v823: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:35.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:35 smithi086 ceph-mon[110975]: pgmap v824: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:35.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:35 smithi032 ceph-mon[147414]: pgmap v824: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:37.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:37 smithi086 ceph-mon[110975]: pgmap v825: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:37.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:37 smithi032 ceph-mon[147414]: pgmap v825: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:38.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:47:38.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:47:39.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:39 smithi086 ceph-mon[110975]: pgmap v826: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:39.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:39 smithi032 ceph-mon[147414]: pgmap v826: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:41.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:41 smithi086 ceph-mon[110975]: pgmap v827: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:41.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:41 smithi032 ceph-mon[147414]: pgmap v827: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.261+0000 7fbf69150700 1 -- 172.21.15.32:0/3391677280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 msgr2=0x7fbf641031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.261+0000 7fbf69150700 1 --2- 172.21.15.32:0/3391677280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 0x7fbf641031d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fbf54004940 tx=0x7fbf54009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.261+0000 7fbf69150700 1 -- 172.21.15.32:0/3391677280 shutdown_connections 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.261+0000 7fbf69150700 1 --2- 172.21.15.32:0/3391677280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf641037a0 0x7fbf640fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.261+0000 7fbf69150700 1 --2- 172.21.15.32:0/3391677280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 0x7fbf641031d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.261+0000 7fbf69150700 1 -- 172.21.15.32:0/3391677280 >> 172.21.15.32:0/3391677280 conn(0x7fbf640f89b0 msgr2=0x7fbf640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.261+0000 7fbf69150700 1 -- 172.21.15.32:0/3391677280 shutdown_connections 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.261+0000 7fbf69150700 1 -- 172.21.15.32:0/3391677280 wait complete. 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.262+0000 7fbf69150700 1 Processor -- start 2024-04-06T14:47:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.262+0000 7fbf69150700 1 -- start start 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.262+0000 7fbf69150700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 0x7fbf64113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.262+0000 7fbf69150700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf641037a0 0x7fbf64113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.262+0000 7fbf69150700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf641145d0 con 0x7fbf64102e00 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.262+0000 7fbf69150700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6410d9c0 con 0x7fbf641037a0 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.262+0000 7fbf62d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 0x7fbf64113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.262+0000 7fbf62d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 0x7fbf64113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49244/0 (socket says 172.21.15.32:49244) 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.262+0000 7fbf62d9d700 1 -- 172.21.15.32:0/1340707086 learned_addr learned my addr 172.21.15.32:0/1340707086 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf6259c700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf641037a0 0x7fbf64113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf62d9d700 1 -- 172.21.15.32:0/1340707086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf641037a0 msgr2=0x7fbf64113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf62d9d700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf641037a0 0x7fbf64113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf62d9d700 1 -- 172.21.15.32:0/1340707086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf54005040 con 0x7fbf64102e00 2024-04-06T14:47:43.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf62d9d700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 0x7fbf64113a40 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009180 tx=0x7fbf54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:43.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf5b7fe700 1 -- 172.21.15.32:0/1340707086 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf5400e040 con 0x7fbf64102e00 2024-04-06T14:47:43.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6410dc40 con 0x7fbf64102e00 2024-04-06T14:47:43.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf5b7fe700 1 -- 172.21.15.32:0/1340707086 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf54004290 con 0x7fbf64102e00 2024-04-06T14:47:43.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf5b7fe700 1 -- 172.21.15.32:0/1340707086 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf54013e60 con 0x7fbf64102e00 2024-04-06T14:47:43.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.263+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6410e200 con 0x7fbf64102e00 2024-04-06T14:47:43.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.264+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf640fe550 con 0x7fbf64102e00 2024-04-06T14:47:43.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.265+0000 7fbf5b7fe700 1 -- 172.21.15.32:0/1340707086 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbf540136b0 con 0x7fbf64102e00 2024-04-06T14:47:43.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.265+0000 7fbf5b7fe700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf5006cbb0 0x7fbf5006f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:43.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.265+0000 7fbf5b7fe700 1 -- 172.21.15.32:0/1340707086 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbf5408d2a0 con 0x7fbf64102e00 2024-04-06T14:47:43.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.265+0000 7fbf6259c700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf5006cbb0 0x7fbf5006f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:43.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.266+0000 7fbf6259c700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf5006cbb0 0x7fbf5006f060 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbf6410eb80 tx=0x7fbf4c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:43.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.270+0000 7fbf5b7fe700 1 -- 172.21.15.32:0/1340707086 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbf5405ad70 con 0x7fbf64102e00 2024-04-06T14:47:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.496+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf64109e30 con 0x7fbf5006cbb0 2024-04-06T14:47:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.497+0000 7fbf5b7fe700 1 -- 172.21.15.32:0/1340707086 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbf64109e30 con 0x7fbf5006cbb0 2024-04-06T14:47:43.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf5006cbb0 msgr2=0x7fbf5006f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf5006cbb0 0x7fbf5006f060 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbf6410eb80 tx=0x7fbf4c00b040 comp rx=0 tx=0).stop 2024-04-06T14:47:43.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 msgr2=0x7fbf64113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 0x7fbf64113a40 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009180 tx=0x7fbf54009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:43.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 shutdown_connections 2024-04-06T14:47:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf5006cbb0 0x7fbf5006f060 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf64102e00 0x7fbf64113a40 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 --2- 172.21.15.32:0/1340707086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf641037a0 0x7fbf64113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 >> 172.21.15.32:0/1340707086 conn(0x7fbf640f89b0 msgr2=0x7fbf640f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 shutdown_connections 2024-04-06T14:47:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.499+0000 7fbf69150700 1 -- 172.21.15.32:0/1340707086 wait complete. 2024-04-06T14:47:43.498 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:47:43.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.603+0000 7fca1f6f6700 1 -- 172.21.15.32:0/4196446282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca18102e10 msgr2=0x7fca181031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.603+0000 7fca1f6f6700 1 --2- 172.21.15.32:0/4196446282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca18102e10 0x7fca181031e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fca08004940 tx=0x7fca08009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:43.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.603+0000 7fca1f6f6700 1 -- 172.21.15.32:0/4196446282 shutdown_connections 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.603+0000 7fca1f6f6700 1 --2- 172.21.15.32:0/4196446282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca181037b0 0x7fca180fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.603+0000 7fca1f6f6700 1 --2- 172.21.15.32:0/4196446282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca18102e10 0x7fca181031e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.603+0000 7fca1f6f6700 1 -- 172.21.15.32:0/4196446282 >> 172.21.15.32:0/4196446282 conn(0x7fca180f89a0 msgr2=0x7fca180fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.603+0000 7fca1f6f6700 1 -- 172.21.15.32:0/4196446282 shutdown_connections 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.603+0000 7fca1f6f6700 1 -- 172.21.15.32:0/4196446282 wait complete. 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1f6f6700 1 Processor -- start 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1f6f6700 1 -- start start 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1f6f6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca18102e10 0x7fca18113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1f6f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca181037b0 0x7fca18114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1f6f6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca18114680 con 0x7fca181037b0 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1f6f6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1810da70 con 0x7fca18102e10 2024-04-06T14:47:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1cc91700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca181037b0 0x7fca18114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1cc91700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca181037b0 0x7fca18114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49262/0 (socket says 172.21.15.32:49262) 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1cc91700 1 -- 172.21.15.32:0/2018455727 learned_addr learned my addr 172.21.15.32:0/2018455727 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1d492700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca18102e10 0x7fca18113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1cc91700 1 -- 172.21.15.32:0/2018455727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca18102e10 msgr2=0x7fca18113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1cc91700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca18102e10 0x7fca18113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1cc91700 1 -- 172.21.15.32:0/2018455727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca08005040 con 0x7fca181037b0 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.604+0000 7fca1cc91700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca181037b0 0x7fca18114030 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fca14002a50 tx=0x7fca14004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.605+0000 7fca0dffb700 1 -- 172.21.15.32:0/2018455727 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca14007cc0 con 0x7fca181037b0 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.605+0000 7fca0dffb700 1 -- 172.21.15.32:0/2018455727 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca1400f040 con 0x7fca181037b0 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.605+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1810dd50 con 0x7fca181037b0 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.605+0000 7fca0dffb700 1 -- 172.21.15.32:0/2018455727 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca1400d920 con 0x7fca181037b0 2024-04-06T14:47:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.605+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1810e310 con 0x7fca181037b0 2024-04-06T14:47:43.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.606+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca180fe560 con 0x7fca181037b0 2024-04-06T14:47:43.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.606+0000 7fca0dffb700 1 -- 172.21.15.32:0/2018455727 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fca14009680 con 0x7fca181037b0 2024-04-06T14:47:43.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.607+0000 7fca0dffb700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca0406cc30 0x7fca0406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:43.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.607+0000 7fca0dffb700 1 -- 172.21.15.32:0/2018455727 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fca1408bde0 con 0x7fca181037b0 2024-04-06T14:47:43.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.607+0000 7fca1d492700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca0406cc30 0x7fca0406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:43.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.607+0000 7fca1d492700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca0406cc30 0x7fca0406f0e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fca0800b840 tx=0x7fca08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:43.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.611+0000 7fca0dffb700 1 -- 172.21.15.32:0/2018455727 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fca14059830 con 0x7fca181037b0 2024-04-06T14:47:43.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:43 smithi086 ceph-mon[110975]: pgmap v828: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:43.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:43 smithi032 ceph-mon[147414]: pgmap v828: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:43.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.834+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fca18109dd0 con 0x7fca0406cc30 2024-04-06T14:47:43.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.835+0000 7fca0dffb700 1 -- 172.21.15.32:0/2018455727 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fca18109dd0 con 0x7fca0406cc30 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca0406cc30 msgr2=0x7fca0406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca0406cc30 0x7fca0406f0e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fca0800b840 tx=0x7fca08009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca181037b0 msgr2=0x7fca18114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca181037b0 0x7fca18114030 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fca14002a50 tx=0x7fca14004600 comp rx=0 tx=0).stop 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 shutdown_connections 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca0406cc30 0x7fca0406f0e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca18102e10 0x7fca18113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 --2- 172.21.15.32:0/2018455727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca181037b0 0x7fca18114030 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 >> 172.21.15.32:0/2018455727 conn(0x7fca180f89a0 msgr2=0x7fca180f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 shutdown_connections 2024-04-06T14:47:43.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.837+0000 7fca1f6f6700 1 -- 172.21.15.32:0/2018455727 wait complete. 2024-04-06T14:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.940+0000 7fb73a471700 1 -- 172.21.15.32:0/1038221192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734102710 msgr2=0x7fb7340fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.940+0000 7fb73a471700 1 --2- 172.21.15.32:0/1038221192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734102710 0x7fb7340fbd80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb728004940 tx=0x7fb728009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.940+0000 7fb73a471700 1 -- 172.21.15.32:0/1038221192 shutdown_connections 2024-04-06T14:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.940+0000 7fb73a471700 1 --2- 172.21.15.32:0/1038221192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734102710 0x7fb7340fbd80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.940+0000 7fb73a471700 1 --2- 172.21.15.32:0/1038221192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb734101d70 0x7fb734102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.940+0000 7fb73a471700 1 -- 172.21.15.32:0/1038221192 >> 172.21.15.32:0/1038221192 conn(0x7fb734072bc0 msgr2=0x7fb734072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.940+0000 7fb73a471700 1 -- 172.21.15.32:0/1038221192 shutdown_connections 2024-04-06T14:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.941+0000 7fb73a471700 1 -- 172.21.15.32:0/1038221192 wait complete. 2024-04-06T14:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.941+0000 7fb73a471700 1 Processor -- start 2024-04-06T14:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.941+0000 7fb73a471700 1 -- start start 2024-04-06T14:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.941+0000 7fb73a471700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734101d70 0x7fb734112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.941+0000 7fb73a471700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb734102710 0x7fb734112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.941+0000 7fb73a471700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb734113490 con 0x7fb734101d70 2024-04-06T14:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.941+0000 7fb73a471700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73410c880 con 0x7fb734102710 2024-04-06T14:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb733fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734101d70 0x7fb734112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb733fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734101d70 0x7fb734112900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49282/0 (socket says 172.21.15.32:49282) 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb733fff700 1 -- 172.21.15.32:0/51658778 learned_addr learned my addr 172.21.15.32:0/51658778 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb7337fe700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb734102710 0x7fb734112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb733fff700 1 -- 172.21.15.32:0/51658778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb734102710 msgr2=0x7fb734112e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb733fff700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb734102710 0x7fb734112e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb733fff700 1 -- 172.21.15.32:0/51658778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb728005040 con 0x7fb734101d70 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb733fff700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734101d70 0x7fb734112900 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb72400eb40 tx=0x7fb72400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb730ff9700 1 -- 172.21.15.32:0/51658778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb724007cc0 con 0x7fb734101d70 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb730ff9700 1 -- 172.21.15.32:0/51658778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb72400f040 con 0x7fb734101d70 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73410cb00 con 0x7fb734101d70 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb730ff9700 1 -- 172.21.15.32:0/51658778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb72400c990 con 0x7fb734101d70 2024-04-06T14:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.942+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73410d090 con 0x7fb734101d70 2024-04-06T14:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.943+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7340ff080 con 0x7fb734101d70 2024-04-06T14:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.944+0000 7fb730ff9700 1 -- 172.21.15.32:0/51658778 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb72400cc00 con 0x7fb734101d70 2024-04-06T14:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.944+0000 7fb730ff9700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb71c06cc20 0x7fb71c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.944+0000 7fb730ff9700 1 -- 172.21.15.32:0/51658778 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb72408bce0 con 0x7fb734101d70 2024-04-06T14:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.944+0000 7fb7337fe700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb71c06cc20 0x7fb71c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.945+0000 7fb7337fe700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb71c06cc20 0x7fb71c06f0d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb73410db60 tx=0x7fb728009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:43.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:43.949+0000 7fb730ff9700 1 -- 172.21.15.32:0/51658778 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb724059760 con 0x7fb734101d70 2024-04-06T14:47:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.159+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb73410dfc0 con 0x7fb71c06cc20 2024-04-06T14:47:44.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.169+0000 7fb730ff9700 1 -- 172.21.15.32:0/51658778 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb73410dfc0 con 0x7fb71c06cc20 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (28m) 6m ago 39m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (25m) 6m ago 39m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (25m) 6m ago 38m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (27m) 6m ago 39m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (21m) 6m ago 33m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (21m) 6m ago 33m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (21m) 6m ago 33m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (6m) 6m ago 33m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (29m) 6m ago 40m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:47:44.159 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (28m) 6m ago 38m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (26m) 6m ago 40m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (26m) 6m ago 38m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (28m) 6m ago 39m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (28m) 6m ago 38m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (25m) 6m ago 37m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (24m) 6m ago 37m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (23m) 6m ago 36m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (23m) 6m ago 36m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (22m) 6m ago 35m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (22m) 6m ago 35m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:47:44.160 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (28m) 6m ago 38m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb71c06cc20 msgr2=0x7fb71c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb71c06cc20 0x7fb71c06f0d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb73410db60 tx=0x7fb728009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734101d70 msgr2=0x7fb734112900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734101d70 0x7fb734112900 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb72400eb40 tx=0x7fb72400b040 comp rx=0 tx=0).stop 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 shutdown_connections 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb71c06cc20 0x7fb71c06f0d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb734101d70 0x7fb734112900 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 --2- 172.21.15.32:0/51658778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb734102710 0x7fb734112e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 >> 172.21.15.32:0/51658778 conn(0x7fb734072bc0 msgr2=0x7fb7340faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 shutdown_connections 2024-04-06T14:47:44.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.171+0000 7fb73a471700 1 -- 172.21.15.32:0/51658778 wait complete. 2024-04-06T14:47:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 -- 172.21.15.32:0/2115932574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 msgr2=0x7f01440fb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 --2- 172.21.15.32:0/2115932574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 0x7f01440fb5a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0130004940 tx=0x7f0130009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 -- 172.21.15.32:0/2115932574 shutdown_connections 2024-04-06T14:47:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 --2- 172.21.15.32:0/2115932574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 0x7f01440fb5a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 --2- 172.21.15.32:0/2115932574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0144102c90 0x7f0144103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 -- 172.21.15.32:0/2115932574 >> 172.21.15.32:0/2115932574 conn(0x7f0144075090 msgr2=0x7f01440f92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 -- 172.21.15.32:0/2115932574 shutdown_connections 2024-04-06T14:47:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 -- 172.21.15.32:0/2115932574 wait complete. 2024-04-06T14:47:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 Processor -- start 2024-04-06T14:47:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.274+0000 7f0149da2700 1 -- start start 2024-04-06T14:47:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0149da2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 0x7f0144113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0149da2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0144102c90 0x7f0144113da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0149da2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01441143f0 con 0x7f01440fb130 2024-04-06T14:47:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0149da2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014410d830 con 0x7f0144102c90 2024-04-06T14:47:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0148da0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 0x7f0144113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0148da0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 0x7f0144113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49312/0 (socket says 172.21.15.32:49312) 2024-04-06T14:47:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0148da0700 1 -- 172.21.15.32:0/1641069997 learned_addr learned my addr 172.21.15.32:0/1641069997 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0143fff700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0144102c90 0x7f0144113da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0148da0700 1 -- 172.21.15.32:0/1641069997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0144102c90 msgr2=0x7f0144113da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0148da0700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0144102c90 0x7f0144113da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0148da0700 1 -- 172.21.15.32:0/1641069997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0130005040 con 0x7f01440fb130 2024-04-06T14:47:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0143fff700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0144102c90 0x7f0144113da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:47:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0148da0700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 0x7f0144113860 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f013800c8c0 tx=0x7f01380044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f01417fa700 1 -- 172.21.15.32:0/1641069997 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f013800a980 con 0x7f01440fb130 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f01417fa700 1 -- 172.21.15.32:0/1641069997 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f013800ce60 con 0x7f01440fb130 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014410db10 con 0x7f01440fb130 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.275+0000 7f01417fa700 1 -- 172.21.15.32:0/1641069997 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01380113a0 con 0x7f01440fb130 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.276+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014410e0d0 con 0x7f01440fb130 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.276+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01440fff40 con 0x7f01440fb130 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.277+0000 7f01417fa700 1 -- 172.21.15.32:0/1641069997 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0138012400 con 0x7f01440fb130 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.277+0000 7f01417fa700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f013406cc30 0x7f013406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.277+0000 7f01417fa700 1 -- 172.21.15.32:0/1641069997 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f013808c040 con 0x7f01440fb130 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.278+0000 7f0143fff700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f013406cc30 0x7f013406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:44.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.278+0000 7f0143fff700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f013406cc30 0x7f013406f0e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0130009fd0 tx=0x7f0130009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:44.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.282+0000 7f01417fa700 1 -- 172.21.15.32:0/1641069997 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0138059bd0 con 0x7f01440fb130 2024-04-06T14:47:44.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.573+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f014410eeb0 con 0x7f01440fb130 2024-04-06T14:47:44.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.573+0000 7f01417fa700 1 -- 172.21.15.32:0/1641069997 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f0138059760 con 0x7f01440fb130 2024-04-06T14:47:44.563 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:47:44.563 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:47:44.563 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:47:44.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:47:44.563 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:47:44.563 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:47:44.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:47:44.565 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:47:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f013406cc30 msgr2=0x7f013406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f013406cc30 0x7f013406f0e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0130009fd0 tx=0x7f0130009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 msgr2=0x7f0144113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 0x7f0144113860 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f013800c8c0 tx=0x7f01380044c0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 shutdown_connections 2024-04-06T14:47:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f013406cc30 0x7f013406f0e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01440fb130 0x7f0144113860 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 --2- 172.21.15.32:0/1641069997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0144102c90 0x7f0144113da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 >> 172.21.15.32:0/1641069997 conn(0x7f0144075090 msgr2=0x7f0144107160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 shutdown_connections 2024-04-06T14:47:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.575+0000 7f0149da2700 1 -- 172.21.15.32:0/1641069997 wait complete. 2024-04-06T14:47:44.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:44 smithi086 ceph-mon[110975]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.679+0000 7fcac6005700 1 -- 172.21.15.32:0/3889603325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac00fb4a0 msgr2=0x7fcac00fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.679+0000 7fcac6005700 1 --2- 172.21.15.32:0/3889603325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac00fb4a0 0x7fcac00fb910 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fcab0004940 tx=0x7fcab0009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.679+0000 7fcac6005700 1 -- 172.21.15.32:0/3889603325 shutdown_connections 2024-04-06T14:47:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.679+0000 7fcac6005700 1 --2- 172.21.15.32:0/3889603325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac00fb4a0 0x7fcac00fb910 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.679+0000 7fcac6005700 1 --2- 172.21.15.32:0/3889603325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcac01014f0 0x7fcac01018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.679+0000 7fcac6005700 1 -- 172.21.15.32:0/3889603325 >> 172.21.15.32:0/3889603325 conn(0x7fcac00714c0 msgr2=0x7fcac00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.679+0000 7fcac6005700 1 -- 172.21.15.32:0/3889603325 shutdown_connections 2024-04-06T14:47:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.679+0000 7fcac6005700 1 -- 172.21.15.32:0/3889603325 wait complete. 2024-04-06T14:47:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.679+0000 7fcac6005700 1 Processor -- start 2024-04-06T14:47:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcac6005700 1 -- start start 2024-04-06T14:47:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcac6005700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcac00fb4a0 0x7fcac010ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcac6005700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac01014f0 0x7fcac010efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcac6005700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac010f610 con 0x7fcac01014f0 2024-04-06T14:47:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcac6005700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac010f750 con 0x7fcac00fb4a0 2024-04-06T14:47:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcabeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac01014f0 0x7fcac010efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcabeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac01014f0 0x7fcac010efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48736/0 (socket says 172.21.15.32:48736) 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcabeffd700 1 -- 172.21.15.32:0/2857616922 learned_addr learned my addr 172.21.15.32:0/2857616922 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcabf7fe700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcac00fb4a0 0x7fcac010ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcabeffd700 1 -- 172.21.15.32:0/2857616922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcac00fb4a0 msgr2=0x7fcac010ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcabeffd700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcac00fb4a0 0x7fcac010ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcabeffd700 1 -- 172.21.15.32:0/2857616922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab0005040 con 0x7fcac01014f0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.680+0000 7fcabeffd700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac01014f0 0x7fcac010efc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009360 tx=0x7fcab0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.681+0000 7fcab7fff700 1 -- 172.21.15.32:0/2857616922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcab000e040 con 0x7fcac01014f0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.681+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac0117af0 con 0x7fcac01014f0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.681+0000 7fcab7fff700 1 -- 172.21.15.32:0/2857616922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcab0013420 con 0x7fcac01014f0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.681+0000 7fcab7fff700 1 -- 172.21.15.32:0/2857616922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcab001c5f0 con 0x7fcac01014f0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.681+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac0118000 con 0x7fcac01014f0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.682+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac00fcbd0 con 0x7fcac01014f0 2024-04-06T14:47:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.682+0000 7fcab7fff700 1 -- 172.21.15.32:0/2857616922 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcab0013590 con 0x7fcac01014f0 2024-04-06T14:47:44.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.682+0000 7fcab7fff700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcaac06cc30 0x7fcaac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:44.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.683+0000 7fcab7fff700 1 -- 172.21.15.32:0/2857616922 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcab008d300 con 0x7fcac01014f0 2024-04-06T14:47:44.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.683+0000 7fcabf7fe700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcaac06cc30 0x7fcaac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:44.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.683+0000 7fcabf7fe700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcaac06cc30 0x7fcaac06f0e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8007fd0 tx=0x7fcaa8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:44.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.687+0000 7fcab7fff700 1 -- 172.21.15.32:0/2857616922 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcab0057340 con 0x7fcac01014f0 2024-04-06T14:47:44.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:44 smithi032 ceph-mon[147414]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.933+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcac004bed0 con 0x7fcac01014f0 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.933+0000 7fcab7fff700 1 -- 172.21.15.32:0/2857616922 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fcab005a960 con 0x7fcac01014f0 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:47:44.923 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:47:44.924 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:47:44.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:47:44.926 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:47:44.926 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:47:44.926 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:47:44.926 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:47:44.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcaac06cc30 msgr2=0x7fcaac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcaac06cc30 0x7fcaac06f0e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8007fd0 tx=0x7fcaa8009040 comp rx=0 tx=0).stop 2024-04-06T14:47:44.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac01014f0 msgr2=0x7fcac010efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:44.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac01014f0 0x7fcac010efc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009360 tx=0x7fcab0009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:44.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 shutdown_connections 2024-04-06T14:47:44.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcaac06cc30 0x7fcaac06f0e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcac00fb4a0 0x7fcac010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 --2- 172.21.15.32:0/2857616922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcac01014f0 0x7fcac010efc0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:44.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 >> 172.21.15.32:0/2857616922 conn(0x7fcac00714c0 msgr2=0x7fcac00f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:44.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 shutdown_connections 2024-04-06T14:47:44.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:44.935+0000 7fcac6005700 1 -- 172.21.15.32:0/2857616922 wait complete. 2024-04-06T14:47:44.929 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 -- 172.21.15.32:0/1936100713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780102710 msgr2=0x7f27800fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 --2- 172.21.15.32:0/1936100713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780102710 0x7f27800fbd80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2770004940 tx=0x7f2770009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 -- 172.21.15.32:0/1936100713 shutdown_connections 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 --2- 172.21.15.32:0/1936100713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780102710 0x7f27800fbd80 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 --2- 172.21.15.32:0/1936100713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2780101d70 0x7f2780102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 -- 172.21.15.32:0/1936100713 >> 172.21.15.32:0/1936100713 conn(0x7f2780072bc0 msgr2=0x7f2780072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 -- 172.21.15.32:0/1936100713 shutdown_connections 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 -- 172.21.15.32:0/1936100713 wait complete. 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 Processor -- start 2024-04-06T14:47:45.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.039+0000 7f27874aa700 1 -- start start 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f27874aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780101d70 0x7f2780103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f27874aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2780102710 0x7f2780103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f27874aa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2780103e10 con 0x7f2780101d70 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f27874aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2780103f50 con 0x7f2780102710 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2785246700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780101d70 0x7f2780103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2785246700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780101d70 0x7f2780103340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48748/0 (socket says 172.21.15.32:48748) 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2785246700 1 -- 172.21.15.32:0/3681471427 learned_addr learned my addr 172.21.15.32:0/3681471427 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2784a45700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2780102710 0x7f2780103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2785246700 1 -- 172.21.15.32:0/3681471427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2780102710 msgr2=0x7f2780103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2785246700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2780102710 0x7f2780103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2785246700 1 -- 172.21.15.32:0/3681471427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2770005040 con 0x7f2780101d70 2024-04-06T14:47:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2785246700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780101d70 0x7f2780103340 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f277c00c8c0 tx=0x7f277c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:45.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2775ffb700 1 -- 172.21.15.32:0/3681471427 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277c00a980 con 0x7f2780101d70 2024-04-06T14:47:45.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2775ffb700 1 -- 172.21.15.32:0/3681471427 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f277c00ce60 con 0x7f2780101d70 2024-04-06T14:47:45.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2780119da0 con 0x7f2780101d70 2024-04-06T14:47:45.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.040+0000 7f2775ffb700 1 -- 172.21.15.32:0/3681471427 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277c0113a0 con 0x7f2780101d70 2024-04-06T14:47:45.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.041+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278011a220 con 0x7f2780101d70 2024-04-06T14:47:45.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.042+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27800fd4c0 con 0x7f2780101d70 2024-04-06T14:47:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.042+0000 7f2775ffb700 1 -- 172.21.15.32:0/3681471427 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f277c012400 con 0x7f2780101d70 2024-04-06T14:47:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.043+0000 7f2775ffb700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f276c06cc20 0x7f276c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.043+0000 7f2775ffb700 1 -- 172.21.15.32:0/3681471427 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f277c08cbf0 con 0x7f2780101d70 2024-04-06T14:47:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.043+0000 7f2784a45700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f276c06cc20 0x7f276c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.043+0000 7f2784a45700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f276c06cc20 0x7f276c06f0d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f277000b3b0 tx=0x7f2770009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.047+0000 7f2775ffb700 1 -- 172.21.15.32:0/3681471427 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f277c056c70 con 0x7f2780101d70 2024-04-06T14:47:45.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.271+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2780104a30 con 0x7f276c06cc20 2024-04-06T14:47:45.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.272+0000 7f2775ffb700 1 -- 172.21.15.32:0/3681471427 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2780104a30 con 0x7f276c06cc20 2024-04-06T14:47:45.262 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:47:45.262 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:47:45.262 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:47:45.263 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:47:45.263 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:47:45.263 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:47:45.263 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:47:45.263 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:47:45.263 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:47:45.263 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:47:45.263 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:47:45.263 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:47:45.264 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:47:45.264 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:47:45.264 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:47:45.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.274+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f276c06cc20 msgr2=0x7f276c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:45.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.274+0000 7f27874aa700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f276c06cc20 0x7f276c06f0d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f277000b3b0 tx=0x7f2770009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:45.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.274+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780101d70 msgr2=0x7f2780103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:45.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.274+0000 7f27874aa700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780101d70 0x7f2780103340 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f277c00c8c0 tx=0x7f277c0044c0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.275+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 shutdown_connections 2024-04-06T14:47:45.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.275+0000 7f27874aa700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f276c06cc20 0x7f276c06f0d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.275+0000 7f27874aa700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2780101d70 0x7f2780103340 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.275+0000 7f27874aa700 1 --2- 172.21.15.32:0/3681471427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2780102710 0x7f2780103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.275+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 >> 172.21.15.32:0/3681471427 conn(0x7f2780072bc0 msgr2=0x7f27801058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:45.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.275+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 shutdown_connections 2024-04-06T14:47:45.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.275+0000 7f27874aa700 1 -- 172.21.15.32:0/3681471427 wait complete. 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 -- 172.21.15.32:0/1806686318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc840fb4a0 msgr2=0x7efc840fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 --2- 172.21.15.32:0/1806686318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc840fb4a0 0x7efc840fb910 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efc78004940 tx=0x7efc78009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 -- 172.21.15.32:0/1806686318 shutdown_connections 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 --2- 172.21.15.32:0/1806686318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc840fb4a0 0x7efc840fb910 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 --2- 172.21.15.32:0/1806686318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc84102e70 0x7efc84103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 -- 172.21.15.32:0/1806686318 >> 172.21.15.32:0/1806686318 conn(0x7efc840714c0 msgr2=0x7efc840718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 -- 172.21.15.32:0/1806686318 shutdown_connections 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 -- 172.21.15.32:0/1806686318 wait complete. 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 Processor -- start 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 -- start start 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.378+0000 7efc88b89700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840fb4a0 0x7efc840fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc88b89700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc84102e70 0x7efc840fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc88b89700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc840fd170 con 0x7efc84102e70 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc88b89700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc840fd2e0 con 0x7efc840fb4a0 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc81d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc84102e70 0x7efc840fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc81d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc84102e70 0x7efc840fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48758/0 (socket says 172.21.15.32:48758) 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc81d9b700 1 -- 172.21.15.32:0/1898625355 learned_addr learned my addr 172.21.15.32:0/1898625355 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc8259c700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840fb4a0 0x7efc840fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc81d9b700 1 -- 172.21.15.32:0/1898625355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840fb4a0 msgr2=0x7efc840fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc81d9b700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840fb4a0 0x7efc840fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc81d9b700 1 -- 172.21.15.32:0/1898625355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc78005040 con 0x7efc84102e70 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc81d9b700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc84102e70 0x7efc840fcbc0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7efc7800b3b0 tx=0x7efc78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc72ffd700 1 -- 172.21.15.32:0/1898625355 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc7800e040 con 0x7efc84102e70 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc72ffd700 1 -- 172.21.15.32:0/1898625355 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc78013420 con 0x7efc84102e70 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc84119d70 con 0x7efc84102e70 2024-04-06T14:47:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc72ffd700 1 -- 172.21.15.32:0/1898625355 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc7801c610 con 0x7efc84102e70 2024-04-06T14:47:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.379+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8411a180 con 0x7efc84102e70 2024-04-06T14:47:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.380+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc84100110 con 0x7efc84102e70 2024-04-06T14:47:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.381+0000 7efc72ffd700 1 -- 172.21.15.32:0/1898625355 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efc78013a00 con 0x7efc84102e70 2024-04-06T14:47:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.381+0000 7efc72ffd700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efc6c06cc20 0x7efc6c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:47:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.381+0000 7efc72ffd700 1 -- 172.21.15.32:0/1898625355 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efc7808d170 con 0x7efc84102e70 2024-04-06T14:47:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.381+0000 7efc8259c700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efc6c06cc20 0x7efc6c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:47:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.382+0000 7efc8259c700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efc6c06cc20 0x7efc6c06f0d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efc74004f40 tx=0x7efc7400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:47:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.386+0000 7efc72ffd700 1 -- 172.21.15.32:0/1898625355 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efc780571b0 con 0x7efc84102e70 2024-04-06T14:47:45.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:45 smithi086 ceph-mon[110975]: from='client.35518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:45.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:45 smithi086 ceph-mon[110975]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:45.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:45 smithi086 ceph-mon[110975]: pgmap v829: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:45.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:45 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1641069997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:47:45.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:45 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2857616922' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:47:45.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.670+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efc840fdd70 con 0x7efc84102e70 2024-04-06T14:47:45.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.670+0000 7efc72ffd700 1 -- 172.21.15.32:0/1898625355 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7efc7805a7d0 con 0x7efc84102e70 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:47:45.660 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:47:45.661 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:47:45.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efc6c06cc20 msgr2=0x7efc6c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:45.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efc6c06cc20 0x7efc6c06f0d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efc74004f40 tx=0x7efc7400b040 comp rx=0 tx=0).stop 2024-04-06T14:47:45.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc84102e70 msgr2=0x7efc840fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:47:45.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc84102e70 0x7efc840fcbc0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7efc7800b3b0 tx=0x7efc78009e30 comp rx=0 tx=0).stop 2024-04-06T14:47:45.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 shutdown_connections 2024-04-06T14:47:45.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efc6c06cc20 0x7efc6c06f0d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840fb4a0 0x7efc840fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 --2- 172.21.15.32:0/1898625355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc84102e70 0x7efc840fcbc0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:47:45.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 >> 172.21.15.32:0/1898625355 conn(0x7efc840714c0 msgr2=0x7efc840fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:47:45.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.672+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 shutdown_connections 2024-04-06T14:47:45.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:47:45.673+0000 7efc88b89700 1 -- 172.21.15.32:0/1898625355 wait complete. 2024-04-06T14:47:45.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:45 smithi032 ceph-mon[147414]: from='client.35518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:45.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:45 smithi032 ceph-mon[147414]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:45.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:45 smithi032 ceph-mon[147414]: pgmap v829: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:45.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:45 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1641069997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:47:45.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:45 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2857616922' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:47:46.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:46 smithi086 ceph-mon[110975]: from='client.35534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:46.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:46 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1898625355' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:47:46.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:46 smithi032 ceph-mon[147414]: from='client.35534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:47:46.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:46 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1898625355' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:47:47.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:47 smithi086 ceph-mon[110975]: pgmap v830: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:47.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:47 smithi032 ceph-mon[147414]: pgmap v830: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:49.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:49 smithi086 ceph-mon[110975]: pgmap v831: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:49.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:49 smithi032 ceph-mon[147414]: pgmap v831: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:51.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:51 smithi086 ceph-mon[110975]: pgmap v832: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:51.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:51 smithi032 ceph-mon[147414]: pgmap v832: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:53.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:53 smithi086 ceph-mon[110975]: pgmap v833: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:53.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:47:53.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:53 smithi032 ceph-mon[147414]: pgmap v833: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:53.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:47:55.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:55 smithi086 ceph-mon[110975]: pgmap v834: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:55.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:55 smithi032 ceph-mon[147414]: pgmap v834: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:57.616 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:57 smithi086 ceph-mon[110975]: pgmap v835: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:57.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:57 smithi032 ceph-mon[147414]: pgmap v835: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:59.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:47:59 smithi086 ceph-mon[110975]: pgmap v836: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:47:59.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:47:59 smithi032 ceph-mon[147414]: pgmap v836: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:01.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:01 smithi086 ceph-mon[110975]: pgmap v837: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:01.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:01 smithi032 ceph-mon[147414]: pgmap v837: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:03.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:03 smithi086 ceph-mon[110975]: pgmap v838: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:03.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:03 smithi032 ceph-mon[147414]: pgmap v838: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:05.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:05 smithi086 ceph-mon[110975]: pgmap v839: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:05.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:05 smithi032 ceph-mon[147414]: pgmap v839: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:07.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:07 smithi086 ceph-mon[110975]: pgmap v840: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:07.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:07 smithi032 ceph-mon[147414]: pgmap v840: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:08.617 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:48:08.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:48:09.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:09 smithi032 ceph-mon[147414]: pgmap v841: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:09.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:09 smithi086 ceph-mon[110975]: pgmap v841: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:11.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:11 smithi032 ceph-mon[147414]: pgmap v842: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:11.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:11 smithi086 ceph-mon[110975]: pgmap v842: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:13.375 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:13 smithi086 ceph-mon[110975]: pgmap v843: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:13.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:13 smithi032 ceph-mon[147414]: pgmap v843: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:15.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:15 smithi032 ceph-mon[147414]: pgmap v844: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:15.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.790+0000 7fb7deea3700 1 -- 172.21.15.32:0/2750742815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 msgr2=0x7fb7d80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.790+0000 7fb7deea3700 1 --2- 172.21.15.32:0/2750742815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 0x7fb7d80fe260 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc004940 tx=0x7fb7cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.790+0000 7fb7deea3700 1 -- 172.21.15.32:0/2750742815 shutdown_connections 2024-04-06T14:48:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.790+0000 7fb7deea3700 1 --2- 172.21.15.32:0/2750742815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d80fe830 0x7fb7d8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.790+0000 7fb7deea3700 1 --2- 172.21.15.32:0/2750742815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 0x7fb7d80fe260 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.790+0000 7fb7deea3700 1 -- 172.21.15.32:0/2750742815 >> 172.21.15.32:0/2750742815 conn(0x7fb7d8072bc0 msgr2=0x7fb7d8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.790+0000 7fb7deea3700 1 -- 172.21.15.32:0/2750742815 shutdown_connections 2024-04-06T14:48:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.790+0000 7fb7deea3700 1 -- 172.21.15.32:0/2750742815 wait complete. 2024-04-06T14:48:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7deea3700 1 Processor -- start 2024-04-06T14:48:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7deea3700 1 -- start start 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7deea3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 0x7fb7d8113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7deea3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d80fe830 0x7fb7d8113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7deea3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8114610 con 0x7fb7d80fde90 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7deea3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d810da00 con 0x7fb7d80fe830 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7dcc3f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 0x7fb7d8113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7dcc3f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 0x7fb7d8113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58998/0 (socket says 172.21.15.32:58998) 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7dcc3f700 1 -- 172.21.15.32:0/2107382389 learned_addr learned my addr 172.21.15.32:0/2107382389 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.791+0000 7fb7d7fff700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d80fe830 0x7fb7d8113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.792+0000 7fb7dcc3f700 1 -- 172.21.15.32:0/2107382389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d80fe830 msgr2=0x7fb7d8113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.792+0000 7fb7dcc3f700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d80fe830 0x7fb7d8113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.792+0000 7fb7dcc3f700 1 -- 172.21.15.32:0/2107382389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc005040 con 0x7fb7d80fde90 2024-04-06T14:48:15.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.792+0000 7fb7dcc3f700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 0x7fb7d8113a80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc000c00 tx=0x7fb7cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:15.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.792+0000 7fb7d57fa700 1 -- 172.21.15.32:0/2107382389 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7cc021070 con 0x7fb7d80fde90 2024-04-06T14:48:15.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.792+0000 7fb7d57fa700 1 -- 172.21.15.32:0/2107382389 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7cc013e00 con 0x7fb7d80fde90 2024-04-06T14:48:15.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.792+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d810dc80 con 0x7fb7d80fde90 2024-04-06T14:48:15.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.792+0000 7fb7d57fa700 1 -- 172.21.15.32:0/2107382389 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7cc00e070 con 0x7fb7d80fde90 2024-04-06T14:48:15.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.792+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d810e1c0 con 0x7fb7d80fde90 2024-04-06T14:48:15.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.793+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d81019c0 con 0x7fb7d80fde90 2024-04-06T14:48:15.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.793+0000 7fb7d57fa700 1 -- 172.21.15.32:0/2107382389 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb7cc013950 con 0x7fb7d80fde90 2024-04-06T14:48:15.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.794+0000 7fb7d57fa700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb7c806cc20 0x7fb7c806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:15.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.794+0000 7fb7d57fa700 1 -- 172.21.15.32:0/2107382389 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb7cc08d170 con 0x7fb7d80fde90 2024-04-06T14:48:15.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.794+0000 7fb7d7fff700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb7c806cc20 0x7fb7c806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:15.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.794+0000 7fb7d7fff700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb7c806cc20 0x7fb7c806f0d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4004600 tx=0x7fb7c400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:15.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:15.799+0000 7fb7d57fa700 1 -- 172.21.15.32:0/2107382389 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb7cc05ac40 con 0x7fb7d80fde90 2024-04-06T14:48:15.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:15 smithi086 ceph-mon[110975]: pgmap v844: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.018+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7d810f0d0 con 0x7fb7c806cc20 2024-04-06T14:48:16.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.019+0000 7fb7d57fa700 1 -- 172.21.15.32:0/2107382389 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb7d810f0d0 con 0x7fb7c806cc20 2024-04-06T14:48:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb7c806cc20 msgr2=0x7fb7c806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb7c806cc20 0x7fb7c806f0d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4004600 tx=0x7fb7c400b040 comp rx=0 tx=0).stop 2024-04-06T14:48:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 msgr2=0x7fb7d8113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 0x7fb7d8113a80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc000c00 tx=0x7fb7cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 shutdown_connections 2024-04-06T14:48:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb7c806cc20 0x7fb7c806f0d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7d80fde90 0x7fb7d8113a80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 --2- 172.21.15.32:0/2107382389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d80fe830 0x7fb7d8113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 >> 172.21.15.32:0/2107382389 conn(0x7fb7d8072bc0 msgr2=0x7fb7d80ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 shutdown_connections 2024-04-06T14:48:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.021+0000 7fb7deea3700 1 -- 172.21.15.32:0/2107382389 wait complete. 2024-04-06T14:48:16.021 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:48:16.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.125+0000 7f9926629700 1 -- 172.21.15.32:0/1981350173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99200fb4a0 msgr2=0x7f99200fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.125+0000 7f9926629700 1 --2- 172.21.15.32:0/1981350173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99200fb4a0 0x7f99200fb910 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9914004940 tx=0x7f9914009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.125+0000 7f9926629700 1 -- 172.21.15.32:0/1981350173 shutdown_connections 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.125+0000 7f9926629700 1 --2- 172.21.15.32:0/1981350173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99200fb4a0 0x7f99200fb910 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.125+0000 7f9926629700 1 --2- 172.21.15.32:0/1981350173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9920102e70 0x7f9920103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.125+0000 7f9926629700 1 -- 172.21.15.32:0/1981350173 >> 172.21.15.32:0/1981350173 conn(0x7f99200714c0 msgr2=0x7f99200718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.125+0000 7f9926629700 1 -- 172.21.15.32:0/1981350173 shutdown_connections 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.126+0000 7f9926629700 1 -- 172.21.15.32:0/1981350173 wait complete. 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.126+0000 7f9926629700 1 Processor -- start 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.126+0000 7f9926629700 1 -- start start 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.126+0000 7f9926629700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99200fb4a0 0x7f9920113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.126+0000 7f9926629700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9920102e70 0x7f9920113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.126+0000 7f9926629700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920114600 con 0x7f9920102e70 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.126+0000 7f9926629700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992010d9f0 con 0x7f99200fb4a0 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9920102e70 0x7f9920113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9920102e70 0x7f9920113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59016/0 (socket says 172.21.15.32:59016) 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991f7fe700 1 -- 172.21.15.32:0/1109755113 learned_addr learned my addr 172.21.15.32:0/1109755113 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991ffff700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99200fb4a0 0x7f9920113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991f7fe700 1 -- 172.21.15.32:0/1109755113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99200fb4a0 msgr2=0x7f9920113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991f7fe700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99200fb4a0 0x7f9920113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991f7fe700 1 -- 172.21.15.32:0/1109755113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9914005040 con 0x7f9920102e70 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991f7fe700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9920102e70 0x7f9920113fb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f991400b3b0 tx=0x7f9914009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991cff9700 1 -- 172.21.15.32:0/1109755113 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991400e040 con 0x7f9920102e70 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992010dc70 con 0x7f9920102e70 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991cff9700 1 -- 172.21.15.32:0/1109755113 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9914013420 con 0x7f9920102e70 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f991cff9700 1 -- 172.21.15.32:0/1109755113 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991401c630 con 0x7f9920102e70 2024-04-06T14:48:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.127+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992010e230 con 0x7f9920102e70 2024-04-06T14:48:16.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.128+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9920100110 con 0x7f9920102e70 2024-04-06T14:48:16.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.129+0000 7f991cff9700 1 -- 172.21.15.32:0/1109755113 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9914013970 con 0x7f9920102e70 2024-04-06T14:48:16.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.129+0000 7f991cff9700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f990806cc20 0x7f990806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:16.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.129+0000 7f991cff9700 1 -- 172.21.15.32:0/1109755113 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f991408d2b0 con 0x7f9920102e70 2024-04-06T14:48:16.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.129+0000 7f991ffff700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f990806cc20 0x7f990806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:16.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.130+0000 7f991ffff700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f990806cc20 0x7f990806f0d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9910007fd0 tx=0x7f9910009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:16.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.134+0000 7f991cff9700 1 -- 172.21.15.32:0/1109755113 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f991405ad80 con 0x7f9920102e70 2024-04-06T14:48:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.353+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f992010f0c0 con 0x7f990806cc20 2024-04-06T14:48:16.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.354+0000 7f991cff9700 1 -- 172.21.15.32:0/1109755113 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f992010f0c0 con 0x7f990806cc20 2024-04-06T14:48:16.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f990806cc20 msgr2=0x7f990806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f990806cc20 0x7f990806f0d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9910007fd0 tx=0x7f9910009040 comp rx=0 tx=0).stop 2024-04-06T14:48:16.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9920102e70 msgr2=0x7f9920113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9920102e70 0x7f9920113fb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f991400b3b0 tx=0x7f9914009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 shutdown_connections 2024-04-06T14:48:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f990806cc20 0x7f990806f0d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99200fb4a0 0x7f9920113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 --2- 172.21.15.32:0/1109755113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9920102e70 0x7f9920113fb0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 >> 172.21.15.32:0/1109755113 conn(0x7f99200714c0 msgr2=0x7f99200fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 shutdown_connections 2024-04-06T14:48:16.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.356+0000 7f9926629700 1 -- 172.21.15.32:0/1109755113 wait complete. 2024-04-06T14:48:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.459+0000 7fa87d97f700 1 -- 172.21.15.32:0/1322407165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 msgr2=0x7fa8781031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.459+0000 7fa87d97f700 1 --2- 172.21.15.32:0/1322407165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 0x7fa8781031d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa868004940 tx=0x7fa868009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.459+0000 7fa87d97f700 1 -- 172.21.15.32:0/1322407165 shutdown_connections 2024-04-06T14:48:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.459+0000 7fa87d97f700 1 --2- 172.21.15.32:0/1322407165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8781037a0 0x7fa8780fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.459+0000 7fa87d97f700 1 --2- 172.21.15.32:0/1322407165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 0x7fa8781031d0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.459+0000 7fa87d97f700 1 -- 172.21.15.32:0/1322407165 >> 172.21.15.32:0/1322407165 conn(0x7fa8780f89b0 msgr2=0x7fa8780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.459+0000 7fa87d97f700 1 -- 172.21.15.32:0/1322407165 shutdown_connections 2024-04-06T14:48:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.459+0000 7fa87d97f700 1 -- 172.21.15.32:0/1322407165 wait complete. 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.460+0000 7fa87d97f700 1 Processor -- start 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.460+0000 7fa87d97f700 1 -- start start 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.460+0000 7fa87d97f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 0x7fa8781138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa87d97f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8781037a0 0x7fa878113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa87d97f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa878114440 con 0x7fa878102e00 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa87d97f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87810da40 con 0x7fa8781037a0 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa876ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 0x7fa8781138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa876ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 0x7fa8781138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59036/0 (socket says 172.21.15.32:59036) 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa876ffd700 1 -- 172.21.15.32:0/3160311731 learned_addr learned my addr 172.21.15.32:0/3160311731 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa876ffd700 1 -- 172.21.15.32:0/3160311731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8781037a0 msgr2=0x7fa878113df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa876ffd700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8781037a0 0x7fa878113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa876ffd700 1 -- 172.21.15.32:0/3160311731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868005040 con 0x7fa878102e00 2024-04-06T14:48:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa876ffd700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 0x7fa8781138b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa86800b3b0 tx=0x7fa868009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa85f7fe700 1 -- 172.21.15.32:0/3160311731 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa86800e040 con 0x7fa878102e00 2024-04-06T14:48:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa85f7fe700 1 -- 172.21.15.32:0/3160311731 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa868013420 con 0x7fa878102e00 2024-04-06T14:48:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87810dcc0 con 0x7fa878102e00 2024-04-06T14:48:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa85f7fe700 1 -- 172.21.15.32:0/3160311731 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa86801c630 con 0x7fa878102e00 2024-04-06T14:48:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.461+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87810e280 con 0x7fa878102e00 2024-04-06T14:48:16.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.462+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8780fe550 con 0x7fa878102e00 2024-04-06T14:48:16.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.463+0000 7fa85f7fe700 1 -- 172.21.15.32:0/3160311731 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa868013970 con 0x7fa878102e00 2024-04-06T14:48:16.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.463+0000 7fa85f7fe700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86006cc20 0x7fa86006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:16.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.463+0000 7fa85f7fe700 1 -- 172.21.15.32:0/3160311731 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa86808d180 con 0x7fa878102e00 2024-04-06T14:48:16.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.463+0000 7fa8767fc700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86006cc20 0x7fa86006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:16.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.464+0000 7fa8767fc700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86006cc20 0x7fa86006f0d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa87810ec00 tx=0x7fa86c007cc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.468+0000 7fa85f7fe700 1 -- 172.21.15.32:0/3160311731 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa86805ad10 con 0x7fa878102e00 2024-04-06T14:48:16.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.683+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa878109e60 con 0x7fa86006cc20 2024-04-06T14:48:16.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.692+0000 7fa85f7fe700 1 -- 172.21.15.32:0/3160311731 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa86c0047f0 con 0x7fa86006cc20 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (28m) 6m ago 39m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (25m) 6m ago 40m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (25m) 6m ago 39m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (28m) 6m ago 39m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (22m) 6m ago 34m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (22m) 6m ago 34m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (21m) 6m ago 34m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (6m) 6m ago 34m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:48:16.682 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (30m) 6m ago 41m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (29m) 6m ago 39m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (27m) 6m ago 41m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (26m) 6m ago 38m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (29m) 6m ago 40m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (29m) 6m ago 39m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (25m) 6m ago 38m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (24m) 6m ago 37m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (24m) 6m ago 37m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (23m) 6m ago 36m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (23m) 6m ago 36m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (22m) 6m ago 35m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:48:16.683 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (28m) 6m ago 39m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:48:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.694+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86006cc20 msgr2=0x7fa86006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.694+0000 7fa87d97f700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86006cc20 0x7fa86006f0d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa87810ec00 tx=0x7fa86c007cc0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.694+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 msgr2=0x7fa8781138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.694+0000 7fa87d97f700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 0x7fa8781138b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa86800b3b0 tx=0x7fa868009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.694+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 shutdown_connections 2024-04-06T14:48:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.694+0000 7fa87d97f700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa86006cc20 0x7fa86006f0d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.694+0000 7fa87d97f700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa878102e00 0x7fa8781138b0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.694+0000 7fa87d97f700 1 --2- 172.21.15.32:0/3160311731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8781037a0 0x7fa878113df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.695+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 >> 172.21.15.32:0/3160311731 conn(0x7fa8780f89b0 msgr2=0x7fa8780f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:16.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.695+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 shutdown_connections 2024-04-06T14:48:16.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.695+0000 7fa87d97f700 1 -- 172.21.15.32:0/3160311731 wait complete. 2024-04-06T14:48:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.796+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2491361857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 msgr2=0x7ffbb80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.796+0000 7ffbbcc50700 1 --2- 172.21.15.32:0/2491361857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 0x7ffbb80fbd80 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ffba0004940 tx=0x7ffba0009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.796+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2491361857 shutdown_connections 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.796+0000 7ffbbcc50700 1 --2- 172.21.15.32:0/2491361857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 0x7ffbb80fbd80 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.796+0000 7ffbbcc50700 1 --2- 172.21.15.32:0/2491361857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb8101d70 0x7ffbb8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.796+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2491361857 >> 172.21.15.32:0/2491361857 conn(0x7ffbb8072bc0 msgr2=0x7ffbb8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.796+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2491361857 shutdown_connections 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.796+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2491361857 wait complete. 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbbcc50700 1 Processor -- start 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbbcc50700 1 -- start start 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbbcc50700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb8101d70 0x7ffbb810ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbbcc50700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 0x7ffbb810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbbcc50700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb810f5f0 con 0x7ffbb8102710 2024-04-06T14:48:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbbcc50700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb810f730 con 0x7ffbb8101d70 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbb5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 0x7ffbb810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbb5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 0x7ffbb810efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59044/0 (socket says 172.21.15.32:59044) 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbb5d9b700 1 -- 172.21.15.32:0/2901771719 learned_addr learned my addr 172.21.15.32:0/2901771719 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbb659c700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb8101d70 0x7ffbb810ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbb5d9b700 1 -- 172.21.15.32:0/2901771719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb8101d70 msgr2=0x7ffbb810ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbb5d9b700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb8101d70 0x7ffbb810ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbb5d9b700 1 -- 172.21.15.32:0/2901771719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba0005040 con 0x7ffbb8102710 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbb659c700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb8101d70 0x7ffbb810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.797+0000 7ffbb5d9b700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 0x7ffbb810efa0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009380 tx=0x7ffba0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.798+0000 7ffbaeffd700 1 -- 172.21.15.32:0/2901771719 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba000e040 con 0x7ffbb8102710 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.798+0000 7ffbaeffd700 1 -- 172.21.15.32:0/2901771719 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffba0013420 con 0x7ffbb8102710 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.798+0000 7ffbaeffd700 1 -- 172.21.15.32:0/2901771719 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba001c630 con 0x7ffbb8102710 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.798+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb8117ad0 con 0x7ffbb8102710 2024-04-06T14:48:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.798+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb8117fe0 con 0x7ffbb8102710 2024-04-06T14:48:16.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.799+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb80ff080 con 0x7ffbb8102710 2024-04-06T14:48:16.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.799+0000 7ffbaeffd700 1 -- 172.21.15.32:0/2901771719 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ffba00139e0 con 0x7ffbb8102710 2024-04-06T14:48:16.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.800+0000 7ffbaeffd700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffba406cc20 0x7ffba406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:16.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.800+0000 7ffbaeffd700 1 -- 172.21.15.32:0/2901771719 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffba008d130 con 0x7ffbb8102710 2024-04-06T14:48:16.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.800+0000 7ffbb659c700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffba406cc20 0x7ffba406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:16.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.800+0000 7ffbb659c700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffba406cc20 0x7ffba406f0d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009fd0 tx=0x7ffba8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:16.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:16.804+0000 7ffbaeffd700 1 -- 172.21.15.32:0/2901771719 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ffba0057230 con 0x7ffbb8102710 2024-04-06T14:48:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.089+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffbb8108d10 con 0x7ffbb8102710 2024-04-06T14:48:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.090+0000 7ffbaeffd700 1 -- 172.21.15.32:0/2901771719 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7ffba005a850 con 0x7ffbb8102710 2024-04-06T14:48:17.079 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:48:17.079 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:48:17.079 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:48:17.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:48:17.080 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:48:17.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffba406cc20 msgr2=0x7ffba406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffba406cc20 0x7ffba406f0d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009fd0 tx=0x7ffba8009ee0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 msgr2=0x7ffbb810efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 0x7ffbb810efa0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009380 tx=0x7ffba0009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:17.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 shutdown_connections 2024-04-06T14:48:17.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffba406cc20 0x7ffba406f0d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb8101d70 0x7ffbb810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 --2- 172.21.15.32:0/2901771719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb8102710 0x7ffbb810efa0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 >> 172.21.15.32:0/2901771719 conn(0x7ffbb8072bc0 msgr2=0x7ffbb80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:17.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 shutdown_connections 2024-04-06T14:48:17.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.092+0000 7ffbbcc50700 1 -- 172.21.15.32:0/2901771719 wait complete. 2024-04-06T14:48:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 -- 172.21.15.32:0/3871670860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 msgr2=0x7fac1c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 --2- 172.21.15.32:0/3871670860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 0x7fac1c103240 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fac10004940 tx=0x7fac10009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:17.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 -- 172.21.15.32:0/3871670860 shutdown_connections 2024-04-06T14:48:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 --2- 172.21.15.32:0/3871670860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac1c0fb4a0 0x7fac1c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 --2- 172.21.15.32:0/3871670860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 0x7fac1c103240 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 -- 172.21.15.32:0/3871670860 >> 172.21.15.32:0/3871670860 conn(0x7fac1c0714c0 msgr2=0x7fac1c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 -- 172.21.15.32:0/3871670860 shutdown_connections 2024-04-06T14:48:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 -- 172.21.15.32:0/3871670860 wait complete. 2024-04-06T14:48:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 Processor -- start 2024-04-06T14:48:17.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.195+0000 7fac22ae7700 1 -- start start 2024-04-06T14:48:17.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac22ae7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac1c0fb4a0 0x7fac1c1139c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:17.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac22ae7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 0x7fac1c113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac22ae7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c114550 con 0x7fac1c102e70 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac22ae7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c10d990 con 0x7fac1c0fb4a0 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac1bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 0x7fac1c113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac1bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 0x7fac1c113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59068/0 (socket says 172.21.15.32:59068) 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac1bfff700 1 -- 172.21.15.32:0/1103240161 learned_addr learned my addr 172.21.15.32:0/1103240161 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac20883700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac1c0fb4a0 0x7fac1c1139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac1bfff700 1 -- 172.21.15.32:0/1103240161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac1c0fb4a0 msgr2=0x7fac1c1139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac1bfff700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac1c0fb4a0 0x7fac1c1139c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac1bfff700 1 -- 172.21.15.32:0/1103240161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac10005040 con 0x7fac1c102e70 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac1bfff700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 0x7fac1c113f00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fac0800eb40 tx=0x7fac0800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac197fa700 1 -- 172.21.15.32:0/1103240161 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac08007cc0 con 0x7fac1c102e70 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac197fa700 1 -- 172.21.15.32:0/1103240161 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac0800f040 con 0x7fac1c102e70 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c10dc10 con 0x7fac1c102e70 2024-04-06T14:48:17.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.196+0000 7fac197fa700 1 -- 172.21.15.32:0/1103240161 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac08009570 con 0x7fac1c102e70 2024-04-06T14:48:17.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.197+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c10e1d0 con 0x7fac1c102e70 2024-04-06T14:48:17.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.197+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c100110 con 0x7fac1c102e70 2024-04-06T14:48:17.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.198+0000 7fac197fa700 1 -- 172.21.15.32:0/1103240161 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fac080096d0 con 0x7fac1c102e70 2024-04-06T14:48:17.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.198+0000 7fac197fa700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac0c06cc30 0x7fac0c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:17.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.198+0000 7fac197fa700 1 -- 172.21.15.32:0/1103240161 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fac0808bc40 con 0x7fac1c102e70 2024-04-06T14:48:17.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.199+0000 7fac20883700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac0c06cc30 0x7fac0c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:17.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.199+0000 7fac20883700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac0c06cc30 0x7fac0c06f0e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fac1000b0d0 tx=0x7fac10009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:17.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.203+0000 7fac197fa700 1 -- 172.21.15.32:0/1103240161 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fac08091020 con 0x7fac1c102e70 2024-04-06T14:48:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.450+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fac1c10f010 con 0x7fac1c102e70 2024-04-06T14:48:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.450+0000 7fac197fa700 1 -- 172.21.15.32:0/1103240161 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fac08019740 con 0x7fac1c102e70 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:48:17.440 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:48:17.441 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:48:17.442 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:48:17.443 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:48:17.443 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:48:17.443 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:48:17.443 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:48:17.443 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:48:17.443 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.452+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac0c06cc30 msgr2=0x7fac0c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.452+0000 7fac22ae7700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac0c06cc30 0x7fac0c06f0e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fac1000b0d0 tx=0x7fac10009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.452+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 msgr2=0x7fac1c113f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.452+0000 7fac22ae7700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 0x7fac1c113f00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fac0800eb40 tx=0x7fac0800b040 comp rx=0 tx=0).stop 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.452+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 shutdown_connections 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.452+0000 7fac22ae7700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fac0c06cc30 0x7fac0c06f0e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.452+0000 7fac22ae7700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac1c0fb4a0 0x7fac1c1139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.452+0000 7fac22ae7700 1 --2- 172.21.15.32:0/1103240161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac1c102e70 0x7fac1c113f00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.452+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 >> 172.21.15.32:0/1103240161 conn(0x7fac1c0714c0 msgr2=0x7fac1c0fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.453+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 shutdown_connections 2024-04-06T14:48:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.453+0000 7fac22ae7700 1 -- 172.21.15.32:0/1103240161 wait complete. 2024-04-06T14:48:17.445 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:48:17.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 -- 172.21.15.32:0/2458175945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e81037a0 msgr2=0x7f80e80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 --2- 172.21.15.32:0/2458175945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e81037a0 0x7f80e80fce10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f80d8004940 tx=0x7f80d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:17.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 -- 172.21.15.32:0/2458175945 shutdown_connections 2024-04-06T14:48:17.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 --2- 172.21.15.32:0/2458175945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e81037a0 0x7f80e80fce10 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 --2- 172.21.15.32:0/2458175945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e8102e00 0x7f80e81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 -- 172.21.15.32:0/2458175945 >> 172.21.15.32:0/2458175945 conn(0x7f80e80f89b0 msgr2=0x7f80e80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:17.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 -- 172.21.15.32:0/2458175945 shutdown_connections 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 -- 172.21.15.32:0/2458175945 wait complete. 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 Processor -- start 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.557+0000 7f80efd2c700 1 -- start start 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80efd2c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e8102e00 0x7f80e80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80efd2c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e81037a0 0x7f80e8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80efd2c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e8070d90 con 0x7f80e8102e00 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80efd2c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e8070ed0 con 0x7f80e81037a0 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80edac8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e8102e00 0x7f80e80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80edac8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e8102e00 0x7f80e80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59082/0 (socket says 172.21.15.32:59082) 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80edac8700 1 -- 172.21.15.32:0/763067014 learned_addr learned my addr 172.21.15.32:0/763067014 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80edac8700 1 -- 172.21.15.32:0/763067014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e81037a0 msgr2=0x7f80e8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80ed2c7700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e81037a0 0x7f80e8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80edac8700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e81037a0 0x7f80e8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80edac8700 1 -- 172.21.15.32:0/763067014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d8005040 con 0x7f80e8102e00 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80edac8700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e8102e00 0x7f80e80702c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f80e4002a50 tx=0x7f80e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80de7fc700 1 -- 172.21.15.32:0/763067014 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80e4007cc0 con 0x7f80e8102e00 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80de7fc700 1 -- 172.21.15.32:0/763067014 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80e400f040 con 0x7f80e8102e00 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e8119e00 con 0x7f80e8102e00 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80de7fc700 1 -- 172.21.15.32:0/763067014 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80e4009540 con 0x7f80e8102e00 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.558+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e811a280 con 0x7f80e8102e00 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.559+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e80fe550 con 0x7f80e8102e00 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.560+0000 7f80de7fc700 1 -- 172.21.15.32:0/763067014 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f80e40096a0 con 0x7f80e8102e00 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.560+0000 7f80de7fc700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80d406cc30 0x7f80d406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.560+0000 7f80de7fc700 1 -- 172.21.15.32:0/763067014 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f80e408be10 con 0x7f80e8102e00 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.560+0000 7f80ed2c7700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80d406cc30 0x7f80d406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.561+0000 7f80ed2c7700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80d406cc30 0x7f80d406f0e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f80d800b3b0 tx=0x7f80d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:17.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.565+0000 7f80de7fc700 1 -- 172.21.15.32:0/763067014 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f80e4055dd0 con 0x7f80e8102e00 2024-04-06T14:48:17.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:17 smithi032 ceph-mon[147414]: from='client.35542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:17.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:17 smithi032 ceph-mon[147414]: pgmap v845: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:17.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:17 smithi032 ceph-mon[147414]: from='client.35546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:17.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:17 smithi032 ceph-mon[147414]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:17.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:17 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2901771719' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:48:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.791+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f80e80719b0 con 0x7f80d406cc30 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.793+0000 7f80de7fc700 1 -- 172.21.15.32:0/763067014 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f80e80719b0 con 0x7f80d406cc30 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:48:17.782 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:48:17.783 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:48:17.783 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:48:17.783 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:48:17.783 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:48:17.783 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:48:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80d406cc30 msgr2=0x7f80d406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80d406cc30 0x7f80d406f0e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f80d800b3b0 tx=0x7f80d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e8102e00 msgr2=0x7f80e80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e8102e00 0x7f80e80702c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f80e4002a50 tx=0x7f80e40046b0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 shutdown_connections 2024-04-06T14:48:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80d406cc30 0x7f80d406f0e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e8102e00 0x7f80e80702c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 --2- 172.21.15.32:0/763067014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e81037a0 0x7f80e8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 >> 172.21.15.32:0/763067014 conn(0x7f80e80f89b0 msgr2=0x7f80e80f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:17.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 shutdown_connections 2024-04-06T14:48:17.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.795+0000 7f80efd2c700 1 -- 172.21.15.32:0/763067014 wait complete. 2024-04-06T14:48:17.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:17 smithi086 ceph-mon[110975]: from='client.35542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:17.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:17 smithi086 ceph-mon[110975]: pgmap v845: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:17.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:17 smithi086 ceph-mon[110975]: from='client.35546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:17.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:17 smithi086 ceph-mon[110975]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:17.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:17 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2901771719' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:48:17.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.898+0000 7f2479fe0700 1 -- 172.21.15.32:0/7050004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 msgr2=0x7f24741031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.898+0000 7f2479fe0700 1 --2- 172.21.15.32:0/7050004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 0x7f24741031e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2464004940 tx=0x7f2464009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:17.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.898+0000 7f2479fe0700 1 -- 172.21.15.32:0/7050004 shutdown_connections 2024-04-06T14:48:17.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.898+0000 7f2479fe0700 1 --2- 172.21.15.32:0/7050004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24741037b0 0x7f24740fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.898+0000 7f2479fe0700 1 --2- 172.21.15.32:0/7050004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 0x7f24741031e0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.898+0000 7f2479fe0700 1 -- 172.21.15.32:0/7050004 >> 172.21.15.32:0/7050004 conn(0x7f24740f89a0 msgr2=0x7f24740fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.898+0000 7f2479fe0700 1 -- 172.21.15.32:0/7050004 shutdown_connections 2024-04-06T14:48:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.898+0000 7f2479fe0700 1 -- 172.21.15.32:0/7050004 wait complete. 2024-04-06T14:48:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f2479fe0700 1 Processor -- start 2024-04-06T14:48:17.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f2479fe0700 1 -- start start 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f2479fe0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 0x7f24740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f2479fe0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24741037b0 0x7f2474070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f2479fe0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2474070d90 con 0x7f2474102e10 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f2479fe0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2474070ed0 con 0x7f24741037b0 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f24737fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 0x7f24740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f24737fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 0x7f24740702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59090/0 (socket says 172.21.15.32:59090) 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f24737fe700 1 -- 172.21.15.32:0/3695877536 learned_addr learned my addr 172.21.15.32:0/3695877536 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.899+0000 7f2472ffd700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24741037b0 0x7f2474070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.900+0000 7f24737fe700 1 -- 172.21.15.32:0/3695877536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24741037b0 msgr2=0x7f2474070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.900+0000 7f24737fe700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24741037b0 0x7f2474070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:17.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.900+0000 7f24737fe700 1 -- 172.21.15.32:0/3695877536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2464005040 con 0x7f2474102e10 2024-04-06T14:48:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.900+0000 7f24737fe700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 0x7f24740702c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2464009180 tx=0x7f2464009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.900+0000 7f246bfff700 1 -- 172.21.15.32:0/3695877536 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f246400e040 con 0x7f2474102e10 2024-04-06T14:48:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.900+0000 7f246bfff700 1 -- 172.21.15.32:0/3695877536 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2464004290 con 0x7f2474102e10 2024-04-06T14:48:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.900+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2474071120 con 0x7f2474102e10 2024-04-06T14:48:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.900+0000 7f246bfff700 1 -- 172.21.15.32:0/3695877536 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f246401c3f0 con 0x7f2474102e10 2024-04-06T14:48:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.900+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247411a220 con 0x7f2474102e10 2024-04-06T14:48:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.901+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24740fe560 con 0x7f2474102e10 2024-04-06T14:48:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.901+0000 7f246bfff700 1 -- 172.21.15.32:0/3695877536 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2464013790 con 0x7f2474102e10 2024-04-06T14:48:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.902+0000 7f246bfff700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f246006cc20 0x7f246006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.902+0000 7f246bfff700 1 -- 172.21.15.32:0/3695877536 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f246408d300 con 0x7f2474102e10 2024-04-06T14:48:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.902+0000 7f2472ffd700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f246006cc20 0x7f246006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.902+0000 7f2472ffd700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f246006cc20 0x7f246006f0d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2474071c00 tx=0x7f245c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:17.906+0000 7f246bfff700 1 -- 172.21.15.32:0/3695877536 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2464057340 con 0x7f2474102e10 2024-04-06T14:48:18.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.192+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f247405f5f0 con 0x7f2474102e10 2024-04-06T14:48:18.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.193+0000 7f246bfff700 1 -- 172.21.15.32:0/3695877536 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f2464013420 con 0x7f2474102e10 2024-04-06T14:48:18.182 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:48:18.182 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:48:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:48:18.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:48:18.182 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:48:18.182 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:48:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:48:18.184 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:48:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:48:18.184 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:48:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:48:18.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.194+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f246006cc20 msgr2=0x7f246006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:18.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.194+0000 7f2479fe0700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f246006cc20 0x7f246006f0d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2474071c00 tx=0x7f245c009ee0 comp rx=0 tx=0).stop 2024-04-06T14:48:18.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.194+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 msgr2=0x7f24740702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:18.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.194+0000 7f2479fe0700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 0x7f24740702c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2464009180 tx=0x7f2464009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:18.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.195+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 shutdown_connections 2024-04-06T14:48:18.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.195+0000 7f2479fe0700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f246006cc20 0x7f246006f0d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:18.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.195+0000 7f2479fe0700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2474102e10 0x7f24740702c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:18.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.195+0000 7f2479fe0700 1 --2- 172.21.15.32:0/3695877536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24741037b0 0x7f2474070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:18.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.195+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 >> 172.21.15.32:0/3695877536 conn(0x7f24740f89a0 msgr2=0x7f24740f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:18.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.195+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 shutdown_connections 2024-04-06T14:48:18.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:18.195+0000 7f2479fe0700 1 -- 172.21.15.32:0/3695877536 wait complete. 2024-04-06T14:48:18.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:18 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1103240161' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:48:18.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:18 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3695877536' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:48:18.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:18 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1103240161' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:48:18.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:18 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3695877536' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:48:19.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:19 smithi032 ceph-mon[147414]: from='client.35562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:19.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:19 smithi032 ceph-mon[147414]: pgmap v846: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:19.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:19 smithi086 ceph-mon[110975]: from='client.35562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:19.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:19 smithi086 ceph-mon[110975]: pgmap v846: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:21.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:21 smithi032 ceph-mon[147414]: pgmap v847: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:21.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:21 smithi086 ceph-mon[110975]: pgmap v847: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:23.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:23 smithi032 ceph-mon[147414]: pgmap v848: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:23.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:48:23.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:23 smithi086 ceph-mon[110975]: pgmap v848: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:23.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:48:25.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:25 smithi032 ceph-mon[147414]: pgmap v849: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:25.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:25 smithi086 ceph-mon[110975]: pgmap v849: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:27.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:27 smithi032 ceph-mon[147414]: pgmap v850: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:27.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:27 smithi086 ceph-mon[110975]: pgmap v850: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:29.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:29 smithi032 ceph-mon[147414]: pgmap v851: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:29.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:29 smithi086 ceph-mon[110975]: pgmap v851: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:31.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:31 smithi032 ceph-mon[147414]: pgmap v852: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:31.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:31 smithi086 ceph-mon[110975]: pgmap v852: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:33.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:33 smithi032 ceph-mon[147414]: pgmap v853: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:33.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:33 smithi086 ceph-mon[110975]: pgmap v853: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:35.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:35 smithi032 ceph-mon[147414]: pgmap v854: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:35.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:35 smithi086 ceph-mon[110975]: pgmap v854: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:37.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:37 smithi032 ceph-mon[147414]: pgmap v855: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:37.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:37 smithi086 ceph-mon[110975]: pgmap v855: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:38.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:48:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:48:39.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:39 smithi032 ceph-mon[147414]: pgmap v856: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:39.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:39 smithi086 ceph-mon[110975]: pgmap v856: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:41.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:41 smithi032 ceph-mon[147414]: pgmap v857: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:41.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:41 smithi086 ceph-mon[110975]: pgmap v857: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:43.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:43 smithi032 ceph-mon[147414]: pgmap v858: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:43.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:43 smithi086 ceph-mon[110975]: pgmap v858: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:45.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:45 smithi032 ceph-mon[147414]: pgmap v859: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:45.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:45 smithi086 ceph-mon[110975]: pgmap v859: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:47.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:47 smithi032 ceph-mon[147414]: pgmap v860: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:47.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:47 smithi086 ceph-mon[110975]: pgmap v860: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:48.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.311+0000 7f3832213700 1 -- 172.21.15.32:0/1279570880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c102e10 msgr2=0x7f382c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.311+0000 7f3832213700 1 --2- 172.21.15.32:0/1279570880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c102e10 0x7f382c1031e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f381c004940 tx=0x7f381c009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:48.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.311+0000 7f3832213700 1 -- 172.21.15.32:0/1279570880 shutdown_connections 2024-04-06T14:48:48.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.311+0000 7f3832213700 1 --2- 172.21.15.32:0/1279570880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1037b0 0x7f382c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.311+0000 7f3832213700 1 --2- 172.21.15.32:0/1279570880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c102e10 0x7f382c1031e0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.311+0000 7f3832213700 1 -- 172.21.15.32:0/1279570880 >> 172.21.15.32:0/1279570880 conn(0x7f382c0f89a0 msgr2=0x7f382c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:48.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.312+0000 7f3832213700 1 -- 172.21.15.32:0/1279570880 shutdown_connections 2024-04-06T14:48:48.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.312+0000 7f3832213700 1 -- 172.21.15.32:0/1279570880 wait complete. 2024-04-06T14:48:48.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.312+0000 7f3832213700 1 Processor -- start 2024-04-06T14:48:48.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.312+0000 7f3832213700 1 -- start start 2024-04-06T14:48:48.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f3832213700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c102e10 0x7f382c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:48.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f3832213700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c1037b0 0x7f382c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:48.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f3832213700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c070d90 con 0x7f382c1037b0 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f3832213700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c070ed0 con 0x7f382c102e10 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f382affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c1037b0 0x7f382c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f382affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c1037b0 0x7f382c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60818/0 (socket says 172.21.15.32:60818) 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f382affd700 1 -- 172.21.15.32:0/2787813627 learned_addr learned my addr 172.21.15.32:0/2787813627 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f382b7fe700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c102e10 0x7f382c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f382affd700 1 -- 172.21.15.32:0/2787813627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c102e10 msgr2=0x7f382c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f382affd700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c102e10 0x7f382c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f382affd700 1 -- 172.21.15.32:0/2787813627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c005040 con 0x7f382c1037b0 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f382b7fe700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c102e10 0x7f382c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f382affd700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c1037b0 0x7f382c070800 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3820002a50 tx=0x7f38200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f3813fff700 1 -- 172.21.15.32:0/2787813627 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3820007cc0 con 0x7f382c1037b0 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f3813fff700 1 -- 172.21.15.32:0/2787813627 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f382000f040 con 0x7f382c1037b0 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382c119d50 con 0x7f382c1037b0 2024-04-06T14:48:48.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f3813fff700 1 -- 172.21.15.32:0/2787813627 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38200095d0 con 0x7f382c1037b0 2024-04-06T14:48:48.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.313+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382c11a1d0 con 0x7f382c1037b0 2024-04-06T14:48:48.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.314+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382c0fe560 con 0x7f382c1037b0 2024-04-06T14:48:48.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.315+0000 7f3813fff700 1 -- 172.21.15.32:0/2787813627 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3820009730 con 0x7f382c1037b0 2024-04-06T14:48:48.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.315+0000 7f3813fff700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f381406cc30 0x7f381406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:48.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.316+0000 7f3813fff700 1 -- 172.21.15.32:0/2787813627 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f382008bd40 con 0x7f382c1037b0 2024-04-06T14:48:48.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.316+0000 7f382b7fe700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f381406cc30 0x7f381406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:48.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.316+0000 7f382b7fe700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f381406cc30 0x7f381406f0e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f381c00b840 tx=0x7f381c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:48.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.320+0000 7f3813fff700 1 -- 172.21.15.32:0/2787813627 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f382008f030 con 0x7f382c1037b0 2024-04-06T14:48:48.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.540+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f382c04bed0 con 0x7f381406cc30 2024-04-06T14:48:48.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.541+0000 7f3813fff700 1 -- 172.21.15.32:0/2787813627 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f382c04bed0 con 0x7f381406cc30 2024-04-06T14:48:48.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f381406cc30 msgr2=0x7f381406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f381406cc30 0x7f381406f0e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f381c00b840 tx=0x7f381c009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:48.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c1037b0 msgr2=0x7f382c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c1037b0 0x7f382c070800 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3820002a50 tx=0x7f38200046b0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 shutdown_connections 2024-04-06T14:48:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f381406cc30 0x7f381406f0e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c102e10 0x7f382c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 --2- 172.21.15.32:0/2787813627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382c1037b0 0x7f382c070800 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 >> 172.21.15.32:0/2787813627 conn(0x7f382c0f89a0 msgr2=0x7f382c0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.543+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 shutdown_connections 2024-04-06T14:48:48.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.544+0000 7f3832213700 1 -- 172.21.15.32:0/2787813627 wait complete. 2024-04-06T14:48:48.543 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:48:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.647+0000 7f50f981c700 1 -- 172.21.15.32:0/3756046095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 msgr2=0x7f50f40fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.647+0000 7f50f981c700 1 --2- 172.21.15.32:0/3756046095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 0x7f50f40fbd30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f50e8004940 tx=0x7f50e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.647+0000 7f50f981c700 1 -- 172.21.15.32:0/3756046095 shutdown_connections 2024-04-06T14:48:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.647+0000 7f50f981c700 1 --2- 172.21.15.32:0/3756046095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 0x7f50f40fbd30 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.647+0000 7f50f981c700 1 --2- 172.21.15.32:0/3756046095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50f4101d20 0x7f50f41020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.647+0000 7f50f981c700 1 -- 172.21.15.32:0/3756046095 >> 172.21.15.32:0/3756046095 conn(0x7f50f40f7870 msgr2=0x7f50f40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.647+0000 7f50f981c700 1 -- 172.21.15.32:0/3756046095 shutdown_connections 2024-04-06T14:48:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.647+0000 7f50f981c700 1 -- 172.21.15.32:0/3756046095 wait complete. 2024-04-06T14:48:48.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.648+0000 7f50f981c700 1 Processor -- start 2024-04-06T14:48:48.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.648+0000 7f50f981c700 1 -- start start 2024-04-06T14:48:48.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f981c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50f4101d20 0x7f50f410ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f981c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 0x7f50f410efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f981c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f410f4f0 con 0x7f50f41026c0 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f981c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f410f630 con 0x7f50f4101d20 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 0x7f50f410efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 0x7f50f410efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60840/0 (socket says 172.21.15.32:60840) 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f27fc700 1 -- 172.21.15.32:0/1203410396 learned_addr learned my addr 172.21.15.32:0/1203410396 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f2ffd700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50f4101d20 0x7f50f410ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f27fc700 1 -- 172.21.15.32:0/1203410396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50f4101d20 msgr2=0x7f50f410ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f27fc700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50f4101d20 0x7f50f410ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f27fc700 1 -- 172.21.15.32:0/1203410396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e8005040 con 0x7f50f41026c0 2024-04-06T14:48:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f27fc700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 0x7f50f410efb0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f50e8009380 tx=0x7f50e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50db7fe700 1 -- 172.21.15.32:0/1203410396 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50e800e040 con 0x7f50f41026c0 2024-04-06T14:48:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f410f8b0 con 0x7f50f41026c0 2024-04-06T14:48:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50db7fe700 1 -- 172.21.15.32:0/1203410396 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50e8013420 con 0x7f50f41026c0 2024-04-06T14:48:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50db7fe700 1 -- 172.21.15.32:0/1203410396 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50e801c630 con 0x7f50f41026c0 2024-04-06T14:48:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.649+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f4117de0 con 0x7f50f41026c0 2024-04-06T14:48:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.650+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f40fd470 con 0x7f50f41026c0 2024-04-06T14:48:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.651+0000 7f50db7fe700 1 -- 172.21.15.32:0/1203410396 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f50e80139e0 con 0x7f50f41026c0 2024-04-06T14:48:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.652+0000 7f50db7fe700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f50dc06cc20 0x7f50dc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.652+0000 7f50db7fe700 1 -- 172.21.15.32:0/1203410396 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f50e808dc70 con 0x7f50f41026c0 2024-04-06T14:48:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.652+0000 7f50f2ffd700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f50dc06cc20 0x7f50dc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.652+0000 7f50f2ffd700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f50dc06cc20 0x7f50dc06f0d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f50e4007fd0 tx=0x7f50e4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:48.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.656+0000 7f50db7fe700 1 -- 172.21.15.32:0/1203410396 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f50e8057d70 con 0x7f50f41026c0 2024-04-06T14:48:48.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.877+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50f4108d60 con 0x7f50dc06cc20 2024-04-06T14:48:48.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.878+0000 7f50db7fe700 1 -- 172.21.15.32:0/1203410396 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f50e4004db0 con 0x7f50dc06cc20 2024-04-06T14:48:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.880+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f50dc06cc20 msgr2=0x7f50dc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.880+0000 7f50f981c700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f50dc06cc20 0x7f50dc06f0d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f50e4007fd0 tx=0x7f50e4009040 comp rx=0 tx=0).stop 2024-04-06T14:48:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.880+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 msgr2=0x7f50f410efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.880+0000 7f50f981c700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 0x7f50f410efb0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f50e8009380 tx=0x7f50e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.880+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 shutdown_connections 2024-04-06T14:48:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.880+0000 7f50f981c700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f50dc06cc20 0x7f50dc06f0d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.880+0000 7f50f981c700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50f4101d20 0x7f50f410ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.880+0000 7f50f981c700 1 --2- 172.21.15.32:0/1203410396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50f41026c0 0x7f50f410efb0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.880+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 >> 172.21.15.32:0/1203410396 conn(0x7f50f40f7870 msgr2=0x7f50f40f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.881+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 shutdown_connections 2024-04-06T14:48:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.881+0000 7f50f981c700 1 -- 172.21.15.32:0/1203410396 wait complete. 2024-04-06T14:48:48.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.984+0000 7fb2bd88c700 1 -- 172.21.15.32:0/3102395635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fde60 msgr2=0x7fb2b80fe230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.984+0000 7fb2bd88c700 1 --2- 172.21.15.32:0/3102395635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fde60 0x7fb2b80fe230 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8004940 tx=0x7fb2a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:48.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.984+0000 7fb2bd88c700 1 -- 172.21.15.32:0/3102395635 shutdown_connections 2024-04-06T14:48:48.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.984+0000 7fb2bd88c700 1 --2- 172.21.15.32:0/3102395635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80fe800 0x7fb2b8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.984+0000 7fb2bd88c700 1 --2- 172.21.15.32:0/3102395635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fde60 0x7fb2b80fe230 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.984+0000 7fb2bd88c700 1 -- 172.21.15.32:0/3102395635 >> 172.21.15.32:0/3102395635 conn(0x7fb2b80f7850 msgr2=0x7fb2b80f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:48.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.984+0000 7fb2bd88c700 1 -- 172.21.15.32:0/3102395635 shutdown_connections 2024-04-06T14:48:48.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.984+0000 7fb2bd88c700 1 -- 172.21.15.32:0/3102395635 wait complete. 2024-04-06T14:48:48.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.985+0000 7fb2bd88c700 1 Processor -- start 2024-04-06T14:48:48.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.985+0000 7fb2bd88c700 1 -- start start 2024-04-06T14:48:48.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.985+0000 7fb2bd88c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80fde60 0x7fb2b80fbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:48.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.985+0000 7fb2bd88c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fe800 0x7fb2b80fc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.985+0000 7fb2bd88c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b80fdbe0 con 0x7fb2b80fe800 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.985+0000 7fb2bd88c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b80fc710 con 0x7fb2b80fde60 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.985+0000 7fb2b67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fe800 0x7fb2b80fc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.985+0000 7fb2b67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fe800 0x7fb2b80fc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60860/0 (socket says 172.21.15.32:60860) 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.985+0000 7fb2b67fc700 1 -- 172.21.15.32:0/191452090 learned_addr learned my addr 172.21.15.32:0/191452090 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb2b6ffd700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80fde60 0x7fb2b80fbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb2b67fc700 1 -- 172.21.15.32:0/191452090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80fde60 msgr2=0x7fb2b80fbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb2b67fc700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80fde60 0x7fb2b80fbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb2b67fc700 1 -- 172.21.15.32:0/191452090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a8005040 con 0x7fb2b80fe800 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb2b6ffd700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80fde60 0x7fb2b80fbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb2b67fc700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fe800 0x7fb2b80fc1d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac002a80 tx=0x7fb2ac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:48.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb29f7fe700 1 -- 172.21.15.32:0/191452090 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2ac007cc0 con 0x7fb2b80fe800 2024-04-06T14:48:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb29f7fe700 1 -- 172.21.15.32:0/191452090 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2ac00f040 con 0x7fb2b80fe800 2024-04-06T14:48:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b80fc9f0 con 0x7fb2b80fe800 2024-04-06T14:48:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb29f7fe700 1 -- 172.21.15.32:0/191452090 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2ac009640 con 0x7fb2b80fe800 2024-04-06T14:48:48.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.986+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b8119eb0 con 0x7fb2b80fe800 2024-04-06T14:48:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.987+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b8066b30 con 0x7fb2b80fe800 2024-04-06T14:48:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.987+0000 7fb29f7fe700 1 -- 172.21.15.32:0/191452090 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb2ac0097a0 con 0x7fb2b80fe800 2024-04-06T14:48:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.988+0000 7fb29f7fe700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb2a006cc20 0x7fb2a006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.988+0000 7fb29f7fe700 1 -- 172.21.15.32:0/191452090 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb2ac08b760 con 0x7fb2b80fe800 2024-04-06T14:48:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.988+0000 7fb2b6ffd700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb2a006cc20 0x7fb2a006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:48.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.988+0000 7fb2b6ffd700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb2a006cc20 0x7fb2a006f0d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb2a800b3b0 tx=0x7fb2a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:48.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:48.992+0000 7fb29f7fe700 1 -- 172.21.15.32:0/191452090 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb2ac0557a0 con 0x7fb2b80fe800 2024-04-06T14:48:49.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.206+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2b80f9630 con 0x7fb2a006cc20 2024-04-06T14:48:49.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.216+0000 7fb29f7fe700 1 -- 172.21.15.32:0/191452090 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb2b80f9630 con 0x7fb2a006cc20 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (29m) 7m ago 40m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (26m) 7m ago 40m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (26m) 7m ago 39m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (28m) 7m ago 40m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (22m) 7m ago 34m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (22m) 7m ago 34m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (22m) 7m ago 34m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (7m) 7m ago 34m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (31m) 7m ago 41m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:48:49.205 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (30m) 7m ago 39m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (27m) 7m ago 42m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (27m) 7m ago 39m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (29m) 7m ago 40m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (29m) 7m ago 39m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (26m) 7m ago 38m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (25m) 7m ago 38m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (24m) 7m ago 37m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (24m) 7m ago 37m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (23m) 7m ago 36m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (23m) 7m ago 36m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:48:49.206 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (29m) 7m ago 39m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:48:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb2a006cc20 msgr2=0x7fb2a006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb2a006cc20 0x7fb2a006f0d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb2a800b3b0 tx=0x7fb2a8009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fe800 msgr2=0x7fb2b80fc1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fe800 0x7fb2b80fc1d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac002a80 tx=0x7fb2ac0046b0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 shutdown_connections 2024-04-06T14:48:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb2a006cc20 0x7fb2a006f0d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80fde60 0x7fb2b80fbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 --2- 172.21.15.32:0/191452090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2b80fe800 0x7fb2b80fc1d0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 >> 172.21.15.32:0/191452090 conn(0x7fb2b80f7850 msgr2=0x7fb2b80f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 shutdown_connections 2024-04-06T14:48:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.218+0000 7fb2bd88c700 1 -- 172.21.15.32:0/191452090 wait complete. 2024-04-06T14:48:49.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.321+0000 7efea0ba7700 1 -- 172.21.15.32:0/2475288433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fbc80 msgr2=0x7efe9c0fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.321+0000 7efea0ba7700 1 --2- 172.21.15.32:0/2475288433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fbc80 0x7efe9c0fc050 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7efe8c004940 tx=0x7efe8c009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:49.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 -- 172.21.15.32:0/2475288433 shutdown_connections 2024-04-06T14:48:49.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 --2- 172.21.15.32:0/2475288433 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe9c0fc590 0x7efe9c1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 --2- 172.21.15.32:0/2475288433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fbc80 0x7efe9c0fc050 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 -- 172.21.15.32:0/2475288433 >> 172.21.15.32:0/2475288433 conn(0x7efe9c072bc0 msgr2=0x7efe9c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:49.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 -- 172.21.15.32:0/2475288433 shutdown_connections 2024-04-06T14:48:49.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 -- 172.21.15.32:0/2475288433 wait complete. 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 Processor -- start 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 -- start start 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe9c0fbc80 0x7efe9c115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fc590 0x7efe9c116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c1168c0 con 0x7efe9c0fc590 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.322+0000 7efea0ba7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c10fc90 con 0x7efe9c0fbc80 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe99d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fc590 0x7efe9c116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe99d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fc590 0x7efe9c116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60880/0 (socket says 172.21.15.32:60880) 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe99d9b700 1 -- 172.21.15.32:0/3550577880 learned_addr learned my addr 172.21.15.32:0/3550577880 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe9a59c700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe9c0fbc80 0x7efe9c115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe99d9b700 1 -- 172.21.15.32:0/3550577880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe9c0fbc80 msgr2=0x7efe9c115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe99d9b700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe9c0fbc80 0x7efe9c115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe99d9b700 1 -- 172.21.15.32:0/3550577880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c005040 con 0x7efe9c0fc590 2024-04-06T14:48:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe9a59c700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe9c0fbc80 0x7efe9c115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:48:49.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe99d9b700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fc590 0x7efe9c116270 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7efe90002a50 tx=0x7efe900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:49.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe8affd700 1 -- 172.21.15.32:0/3550577880 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe90007cc0 con 0x7efe9c0fc590 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c10ff70 con 0x7efe9c0fc590 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe8affd700 1 -- 172.21.15.32:0/3550577880 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe9000f040 con 0x7efe9c0fc590 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efe8affd700 1 -- 172.21.15.32:0/3550577880 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe90009540 con 0x7efe9c0fc590 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.323+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c1104b0 con 0x7efe9c0fc590 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.325+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c065e30 con 0x7efe9c0fc590 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.325+0000 7efe8affd700 1 -- 172.21.15.32:0/3550577880 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efe900096a0 con 0x7efe9c0fc590 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.325+0000 7efe8affd700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe8406cc20 0x7efe8406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.325+0000 7efe8affd700 1 -- 172.21.15.32:0/3550577880 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efe9008be70 con 0x7efe9c0fc590 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.325+0000 7efe9a59c700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe8406cc20 0x7efe8406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.326+0000 7efe9a59c700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe8406cc20 0x7efe8406f0d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7efe8c004560 tx=0x7efe8c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:49.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.330+0000 7efe8affd700 1 -- 172.21.15.32:0/3550577880 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efe900598c0 con 0x7efe9c0fc590 2024-04-06T14:48:49.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.624+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efe9c10c020 con 0x7efe9c0fc590 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.625+0000 7efe8affd700 1 -- 172.21.15.32:0/3550577880 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7efe90059450 con 0x7efe9c0fc590 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:48:49.614 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:48:49.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:48:49.615 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:48:49.615 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:48:49.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:48:49.615 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:48:49.615 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:48:49.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:48:49.615 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:48:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe8406cc20 msgr2=0x7efe8406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe8406cc20 0x7efe8406f0d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7efe8c004560 tx=0x7efe8c009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fc590 msgr2=0x7efe9c116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fc590 0x7efe9c116270 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7efe90002a50 tx=0x7efe900046b0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 shutdown_connections 2024-04-06T14:48:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe8406cc20 0x7efe8406f0d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe9c0fbc80 0x7efe9c115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 --2- 172.21.15.32:0/3550577880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe9c0fc590 0x7efe9c116270 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 >> 172.21.15.32:0/3550577880 conn(0x7efe9c072bc0 msgr2=0x7efe9c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 shutdown_connections 2024-04-06T14:48:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.627+0000 7efea0ba7700 1 -- 172.21.15.32:0/3550577880 wait complete. 2024-04-06T14:48:49.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:49 smithi032 ceph-mon[147414]: pgmap v861: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:49.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:49 smithi032 ceph-mon[147414]: from='client.35570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.730+0000 7f99cdc11700 1 -- 172.21.15.32:0/1055891218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 msgr2=0x7f99c8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.730+0000 7f99cdc11700 1 --2- 172.21.15.32:0/1055891218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 0x7f99c8102140 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f99b8004940 tx=0x7f99b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.730+0000 7f99cdc11700 1 -- 172.21.15.32:0/1055891218 shutdown_connections 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.730+0000 7f99cdc11700 1 --2- 172.21.15.32:0/1055891218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99c8102710 0x7f99c80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.730+0000 7f99cdc11700 1 --2- 172.21.15.32:0/1055891218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 0x7f99c8102140 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.730+0000 7f99cdc11700 1 -- 172.21.15.32:0/1055891218 >> 172.21.15.32:0/1055891218 conn(0x7f99c8072bc0 msgr2=0x7f99c8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.730+0000 7f99cdc11700 1 -- 172.21.15.32:0/1055891218 shutdown_connections 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.730+0000 7f99cdc11700 1 -- 172.21.15.32:0/1055891218 wait complete. 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99cdc11700 1 Processor -- start 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99cdc11700 1 -- start start 2024-04-06T14:48:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99cdc11700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 0x7f99c81129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99cdc11700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99c8102710 0x7f99c8112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99cdc11700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c8113530 con 0x7f99c8101d70 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99cdc11700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c810c920 con 0x7f99c8102710 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99c77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 0x7f99c81129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99c77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 0x7f99c81129a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60904/0 (socket says 172.21.15.32:60904) 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99c77fe700 1 -- 172.21.15.32:0/3105613834 learned_addr learned my addr 172.21.15.32:0/3105613834 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.731+0000 7f99c6ffd700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99c8102710 0x7f99c8112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99c77fe700 1 -- 172.21.15.32:0/3105613834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99c8102710 msgr2=0x7f99c8112ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99c77fe700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99c8102710 0x7f99c8112ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99c77fe700 1 -- 172.21.15.32:0/3105613834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b8005040 con 0x7f99c8101d70 2024-04-06T14:48:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99c6ffd700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99c8102710 0x7f99c8112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:48:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99c77fe700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 0x7f99c81129a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f99b8009360 tx=0x7f99b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99affff700 1 -- 172.21.15.32:0/3105613834 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b800e040 con 0x7f99c8101d70 2024-04-06T14:48:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c810cba0 con 0x7f99c8101d70 2024-04-06T14:48:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99affff700 1 -- 172.21.15.32:0/3105613834 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99b8013420 con 0x7f99c8101d70 2024-04-06T14:48:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99affff700 1 -- 172.21.15.32:0/3105613834 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b801c630 con 0x7f99c8101d70 2024-04-06T14:48:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.732+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c810d160 con 0x7f99c8101d70 2024-04-06T14:48:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.733+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c80ff080 con 0x7f99c8101d70 2024-04-06T14:48:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.733+0000 7f99affff700 1 -- 172.21.15.32:0/3105613834 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f99b8013590 con 0x7f99c8101d70 2024-04-06T14:48:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.734+0000 7f99affff700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99b006cc30 0x7f99b006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.734+0000 7f99affff700 1 -- 172.21.15.32:0/3105613834 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f99b808d360 con 0x7f99c8101d70 2024-04-06T14:48:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.734+0000 7f99c6ffd700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99b006cc30 0x7f99b006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.734+0000 7f99c6ffd700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99b006cc30 0x7f99b006f0e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009fd0 tx=0x7f99bc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:49.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.738+0000 7f99affff700 1 -- 172.21.15.32:0/3105613834 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f99b805aee0 con 0x7f99c8101d70 2024-04-06T14:48:49.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:49 smithi086 ceph-mon[110975]: pgmap v861: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:49.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:49 smithi086 ceph-mon[110975]: from='client.35570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:49.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.985+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f99c810e020 con 0x7f99c8101d70 2024-04-06T14:48:49.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.985+0000 7f99affff700 1 -- 172.21.15.32:0/3105613834 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f99b805aa70 con 0x7f99c8101d70 2024-04-06T14:48:49.975 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:48:49.975 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:48:49.975 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:48:49.975 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:48:49.975 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:48:49.975 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:48:49.975 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:48:49.975 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:48:49.976 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:48:49.977 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:48:49.978 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:48:49.978 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:48:49.978 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:48:49.978 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:48:49.978 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:48:49.978 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:48:49.978 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:48:49.978 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:48:49.978 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:48:49.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.987+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99b006cc30 msgr2=0x7f99b006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.987+0000 7f99cdc11700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99b006cc30 0x7f99b006f0e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009fd0 tx=0x7f99bc00b040 comp rx=0 tx=0).stop 2024-04-06T14:48:49.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.987+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 msgr2=0x7f99c81129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:49.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.987+0000 7f99cdc11700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 0x7f99c81129a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f99b8009360 tx=0x7f99b8009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:49.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.988+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 shutdown_connections 2024-04-06T14:48:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.988+0000 7f99cdc11700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99b006cc30 0x7f99b006f0e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.988+0000 7f99cdc11700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8101d70 0x7f99c81129a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.988+0000 7f99cdc11700 1 --2- 172.21.15.32:0/3105613834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99c8102710 0x7f99c8112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.988+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 >> 172.21.15.32:0/3105613834 conn(0x7f99c8072bc0 msgr2=0x7f99c80fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.988+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 shutdown_connections 2024-04-06T14:48:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:49.988+0000 7f99cdc11700 1 -- 172.21.15.32:0/3105613834 wait complete. 2024-04-06T14:48:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.091+0000 7f45c458b700 1 -- 172.21.15.32:0/2227868823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fe830 msgr2=0x7f45bc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.091+0000 7f45c458b700 1 --2- 172.21.15.32:0/2227868823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fe830 0x7f45bc106f00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f45ac004940 tx=0x7f45ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.091+0000 7f45c458b700 1 -- 172.21.15.32:0/2227868823 shutdown_connections 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.091+0000 7f45c458b700 1 --2- 172.21.15.32:0/2227868823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fe830 0x7f45bc106f00 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.091+0000 7f45c458b700 1 --2- 172.21.15.32:0/2227868823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45bc0fde90 0x7f45bc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.091+0000 7f45c458b700 1 -- 172.21.15.32:0/2227868823 >> 172.21.15.32:0/2227868823 conn(0x7f45bc072bc0 msgr2=0x7f45bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.091+0000 7f45c458b700 1 -- 172.21.15.32:0/2227868823 shutdown_connections 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.091+0000 7f45c458b700 1 -- 172.21.15.32:0/2227868823 wait complete. 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c458b700 1 Processor -- start 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c458b700 1 -- start start 2024-04-06T14:48:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c458b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fde90 0x7f45bc113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c458b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45bc0fe830 0x7f45bc114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c458b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc114670 con 0x7f45bc0fde90 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c458b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc10da60 con 0x7f45bc0fe830 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c2327700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fde90 0x7f45bc113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c2327700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fde90 0x7f45bc113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60918/0 (socket says 172.21.15.32:60918) 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c2327700 1 -- 172.21.15.32:0/1072723732 learned_addr learned my addr 172.21.15.32:0/1072723732 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c2327700 1 -- 172.21.15.32:0/1072723732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45bc0fe830 msgr2=0x7f45bc114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c2327700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45bc0fe830 0x7f45bc114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c2327700 1 -- 172.21.15.32:0/1072723732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac005040 con 0x7f45bc0fde90 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.092+0000 7f45c2327700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fde90 0x7f45bc113ae0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f45b800c8c0 tx=0x7f45b80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.093+0000 7f45b2ffd700 1 -- 172.21.15.32:0/1072723732 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45b800a980 con 0x7f45bc0fde90 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.093+0000 7f45b2ffd700 1 -- 172.21.15.32:0/1072723732 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45b800ce60 con 0x7f45bc0fde90 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.093+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45bc10dd40 con 0x7f45bc0fde90 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.093+0000 7f45b2ffd700 1 -- 172.21.15.32:0/1072723732 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45b80113a0 con 0x7f45bc0fde90 2024-04-06T14:48:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.093+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc10e280 con 0x7f45bc0fde90 2024-04-06T14:48:50.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.094+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45bc065e30 con 0x7f45bc0fde90 2024-04-06T14:48:50.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.094+0000 7f45b2ffd700 1 -- 172.21.15.32:0/1072723732 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f45b8012400 con 0x7f45bc0fde90 2024-04-06T14:48:50.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.095+0000 7f45b2ffd700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45a806cc30 0x7f45a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:50.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.095+0000 7f45b2ffd700 1 -- 172.21.15.32:0/1072723732 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f45b808c1e0 con 0x7f45bc0fde90 2024-04-06T14:48:50.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.095+0000 7f45c1b26700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45a806cc30 0x7f45a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:50.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.095+0000 7f45c1b26700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45a806cc30 0x7f45a806f0e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f45ac004890 tx=0x7f45ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.099+0000 7f45b2ffd700 1 -- 172.21.15.32:0/1072723732 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f45b8059c30 con 0x7f45bc0fde90 2024-04-06T14:48:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.320+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45bc109e20 con 0x7f45a806cc30 2024-04-06T14:48:50.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.321+0000 7f45b2ffd700 1 -- 172.21.15.32:0/1072723732 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f45bc109e20 con 0x7f45a806cc30 2024-04-06T14:48:50.310 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:48:50.310 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:48:50.310 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:48:50.310 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:48:50.310 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:48:50.310 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:48:50.310 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:48:50.310 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:48:50.311 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:48:50.311 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:48:50.311 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:48:50.311 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:48:50.311 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:48:50.311 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:48:50.311 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:48:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45a806cc30 msgr2=0x7f45a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45a806cc30 0x7f45a806f0e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f45ac004890 tx=0x7f45ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fde90 msgr2=0x7f45bc113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fde90 0x7f45bc113ae0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f45b800c8c0 tx=0x7f45b80044c0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 shutdown_connections 2024-04-06T14:48:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f45a806cc30 0x7f45a806f0e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45bc0fde90 0x7f45bc113ae0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 --2- 172.21.15.32:0/1072723732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45bc0fe830 0x7f45bc114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 >> 172.21.15.32:0/1072723732 conn(0x7f45bc072bc0 msgr2=0x7f45bc103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 shutdown_connections 2024-04-06T14:48:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.323+0000 7f45c458b700 1 -- 172.21.15.32:0/1072723732 wait complete. 2024-04-06T14:48:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.425+0000 7fc836be5700 1 -- 172.21.15.32:0/790339934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8301037b0 msgr2=0x7fc8300fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.425+0000 7fc836be5700 1 --2- 172.21.15.32:0/790339934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8301037b0 0x7fc8300fce20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc81c004940 tx=0x7fc81c009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.426+0000 7fc836be5700 1 -- 172.21.15.32:0/790339934 shutdown_connections 2024-04-06T14:48:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.426+0000 7fc836be5700 1 --2- 172.21.15.32:0/790339934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8301037b0 0x7fc8300fce20 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.426+0000 7fc836be5700 1 --2- 172.21.15.32:0/790339934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc830102e10 0x7fc8301031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.426+0000 7fc836be5700 1 -- 172.21.15.32:0/790339934 >> 172.21.15.32:0/790339934 conn(0x7fc8300f8980 msgr2=0x7fc8300fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:50.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.426+0000 7fc836be5700 1 -- 172.21.15.32:0/790339934 shutdown_connections 2024-04-06T14:48:50.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.426+0000 7fc836be5700 1 -- 172.21.15.32:0/790339934 wait complete. 2024-04-06T14:48:50.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.426+0000 7fc836be5700 1 Processor -- start 2024-04-06T14:48:50.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.426+0000 7fc836be5700 1 -- start start 2024-04-06T14:48:50.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc836be5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc830102e10 0x7fc8300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:50.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc836be5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8301037b0 0x7fc830070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:50.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc836be5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc830070d90 con 0x7fc830102e10 2024-04-06T14:48:50.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc836be5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc830070ed0 con 0x7fc8301037b0 2024-04-06T14:48:50.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc834981700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc830102e10 0x7fc8300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:50.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc834981700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc830102e10 0x7fc8300702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60940/0 (socket says 172.21.15.32:60940) 2024-04-06T14:48:50.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc834981700 1 -- 172.21.15.32:0/3689518585 learned_addr learned my addr 172.21.15.32:0/3689518585 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc82ffff700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8301037b0 0x7fc830070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc834981700 1 -- 172.21.15.32:0/3689518585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8301037b0 msgr2=0x7fc830070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc834981700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8301037b0 0x7fc830070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc834981700 1 -- 172.21.15.32:0/3689518585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c005040 con 0x7fc830102e10 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc82ffff700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8301037b0 0x7fc830070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc834981700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc830102e10 0x7fc8300702c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc824002a50 tx=0x7fc8240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc82d7fa700 1 -- 172.21.15.32:0/3689518585 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc824007cc0 con 0x7fc830102e10 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc830119df0 con 0x7fc830102e10 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc82d7fa700 1 -- 172.21.15.32:0/3689518585 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc82400f040 con 0x7fc830102e10 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc82d7fa700 1 -- 172.21.15.32:0/3689518585 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc824009540 con 0x7fc830102e10 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.427+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83011a270 con 0x7fc830102e10 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.428+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8300fe560 con 0x7fc830102e10 2024-04-06T14:48:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.429+0000 7fc82d7fa700 1 -- 172.21.15.32:0/3689518585 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc8240096a0 con 0x7fc830102e10 2024-04-06T14:48:50.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.429+0000 7fc82d7fa700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc82006cc20 0x7fc82006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:48:50.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.429+0000 7fc82d7fa700 1 -- 172.21.15.32:0/3689518585 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc82408be60 con 0x7fc830102e10 2024-04-06T14:48:50.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.429+0000 7fc82ffff700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc82006cc20 0x7fc82006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:48:50.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.430+0000 7fc82ffff700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc82006cc20 0x7fc82006f0d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00b3b0 tx=0x7fc81c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:48:50.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.434+0000 7fc82d7fa700 1 -- 172.21.15.32:0/3689518585 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc824055e20 con 0x7fc830102e10 2024-04-06T14:48:50.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.726+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc83005f5f0 con 0x7fc830102e10 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.726+0000 7fc82d7fa700 1 -- 172.21.15.32:0/3689518585 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7fc824059440 con 0x7fc830102e10 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:48:50.716 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:48:50.717 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:48:50.718 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:48:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc82006cc20 msgr2=0x7fc82006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc82006cc20 0x7fc82006f0d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00b3b0 tx=0x7fc81c009e30 comp rx=0 tx=0).stop 2024-04-06T14:48:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc830102e10 msgr2=0x7fc8300702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:48:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc830102e10 0x7fc8300702c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc824002a50 tx=0x7fc8240046b0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 shutdown_connections 2024-04-06T14:48:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc82006cc20 0x7fc82006f0d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc830102e10 0x7fc8300702c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 --2- 172.21.15.32:0/3689518585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8301037b0 0x7fc830070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:48:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 >> 172.21.15.32:0/3689518585 conn(0x7fc8300f8980 msgr2=0x7fc8300f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:48:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 shutdown_connections 2024-04-06T14:48:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:48:50.728+0000 7fc836be5700 1 -- 172.21.15.32:0/3689518585 wait complete. 2024-04-06T14:48:50.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:50 smithi032 ceph-mon[147414]: from='client.35574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:50.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:50 smithi032 ceph-mon[147414]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:50.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:50 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3550577880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:48:50.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:50 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3105613834' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:48:50.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:50 smithi086 ceph-mon[110975]: from='client.35574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:50.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:50 smithi086 ceph-mon[110975]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:50.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:50 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3550577880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:48:50.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:50 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3105613834' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:48:51.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:51 smithi032 ceph-mon[147414]: pgmap v862: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:51.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:51 smithi032 ceph-mon[147414]: from='client.35590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:51.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:51 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3689518585' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:48:51.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:51 smithi086 ceph-mon[110975]: pgmap v862: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:51.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:51 smithi086 ceph-mon[110975]: from='client.35590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:48:51.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:51 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3689518585' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:48:53.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:53 smithi032 ceph-mon[147414]: pgmap v863: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:53.720 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:48:53.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:53 smithi086 ceph-mon[110975]: pgmap v863: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:53.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:48:55.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:55 smithi032 ceph-mon[147414]: pgmap v864: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:55.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:55 smithi086 ceph-mon[110975]: pgmap v864: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:57.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:57 smithi032 ceph-mon[147414]: pgmap v865: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:57.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:57 smithi086 ceph-mon[110975]: pgmap v865: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:59.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:48:59 smithi032 ceph-mon[147414]: pgmap v866: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:48:59.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:48:59 smithi086 ceph-mon[110975]: pgmap v866: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:01.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:01 smithi032 ceph-mon[147414]: pgmap v867: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:01.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:01 smithi086 ceph-mon[110975]: pgmap v867: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:03.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:03 smithi032 ceph-mon[147414]: pgmap v868: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:03.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:03 smithi086 ceph-mon[110975]: pgmap v868: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:05.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:05 smithi032 ceph-mon[147414]: pgmap v869: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:05.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:05 smithi086 ceph-mon[110975]: pgmap v869: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:07.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:07 smithi032 ceph-mon[147414]: pgmap v870: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:07.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:07 smithi086 ceph-mon[110975]: pgmap v870: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:08.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:49:08.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:49:09.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:09 smithi032 ceph-mon[147414]: pgmap v871: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:09.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:09 smithi086 ceph-mon[110975]: pgmap v871: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:11.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:11 smithi086 ceph-mon[110975]: pgmap v872: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:11.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:11 smithi032 ceph-mon[147414]: pgmap v872: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:13.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:13 smithi032 ceph-mon[147414]: pgmap v873: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:13.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:13 smithi086 ceph-mon[110975]: pgmap v873: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:15.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:15 smithi086 ceph-mon[110975]: pgmap v874: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:15.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:15 smithi032 ceph-mon[147414]: pgmap v874: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:17.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:17 smithi086 ceph-mon[110975]: pgmap v875: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:17.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:17 smithi032 ceph-mon[147414]: pgmap v875: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:19.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:19 smithi086 ceph-mon[110975]: pgmap v876: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:19.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:19 smithi032 ceph-mon[147414]: pgmap v876: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:20.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.846+0000 7ff1155e1700 1 -- 172.21.15.32:0/1814871563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110102f50 msgr2=0x7ff110103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:20.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.846+0000 7ff1155e1700 1 --2- 172.21.15.32:0/1814871563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110102f50 0x7ff110103320 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ff100004940 tx=0x7ff100009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:20.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.846+0000 7ff1155e1700 1 -- 172.21.15.32:0/1814871563 shutdown_connections 2024-04-06T14:49:20.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.846+0000 7ff1155e1700 1 --2- 172.21.15.32:0/1814871563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1100fb4d0 0x7ff1100fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:20.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.846+0000 7ff1155e1700 1 --2- 172.21.15.32:0/1814871563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110102f50 0x7ff110103320 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:20.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.846+0000 7ff1155e1700 1 -- 172.21.15.32:0/1814871563 >> 172.21.15.32:0/1814871563 conn(0x7ff1100714c0 msgr2=0x7ff1100718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:20.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.846+0000 7ff1155e1700 1 -- 172.21.15.32:0/1814871563 shutdown_connections 2024-04-06T14:49:20.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.846+0000 7ff1155e1700 1 -- 172.21.15.32:0/1814871563 wait complete. 2024-04-06T14:49:20.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff1155e1700 1 Processor -- start 2024-04-06T14:49:20.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff1155e1700 1 -- start start 2024-04-06T14:49:20.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff1155e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1100fb4d0 0x7ff1100fc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff1155e1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110102f50 0x7ff1100fcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff1155e1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100fe600 con 0x7ff1100fb4d0 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff1155e1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100fd130 con 0x7ff110102f50 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff10effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1100fb4d0 0x7ff1100fc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff10effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1100fb4d0 0x7ff1100fc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42298/0 (socket says 172.21.15.32:42298) 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff10effd700 1 -- 172.21.15.32:0/1271327289 learned_addr learned my addr 172.21.15.32:0/1271327289 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.847+0000 7ff10e7fc700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110102f50 0x7ff1100fcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.848+0000 7ff10effd700 1 -- 172.21.15.32:0/1271327289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110102f50 msgr2=0x7ff1100fcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.848+0000 7ff10effd700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110102f50 0x7ff1100fcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.848+0000 7ff10effd700 1 -- 172.21.15.32:0/1271327289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff100005040 con 0x7ff1100fb4d0 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.848+0000 7ff10effd700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1100fb4d0 0x7ff1100fc6b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff100000c00 tx=0x7ff100009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.848+0000 7ff0f77fe700 1 -- 172.21.15.32:0/1271327289 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff10000e070 con 0x7ff1100fb4d0 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.848+0000 7ff0f77fe700 1 -- 172.21.15.32:0/1271327289 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1000093b0 con 0x7ff1100fb4d0 2024-04-06T14:49:20.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.848+0000 7ff0f77fe700 1 -- 172.21.15.32:0/1271327289 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff10001ccb0 con 0x7ff1100fb4d0 2024-04-06T14:49:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.848+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1100fd3b0 con 0x7ff1100fb4d0 2024-04-06T14:49:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.848+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11011a010 con 0x7ff1100fb4d0 2024-04-06T14:49:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.849+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff110100200 con 0x7ff1100fb4d0 2024-04-06T14:49:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.850+0000 7ff0f77fe700 1 -- 172.21.15.32:0/1271327289 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff10001c490 con 0x7ff1100fb4d0 2024-04-06T14:49:20.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.850+0000 7ff0f77fe700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0f806cc20 0x7ff0f806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:20.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.850+0000 7ff0f77fe700 1 -- 172.21.15.32:0/1271327289 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff10008d4a0 con 0x7ff1100fb4d0 2024-04-06T14:49:20.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.850+0000 7ff10e7fc700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0f806cc20 0x7ff0f806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:20.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.851+0000 7ff10e7fc700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0f806cc20 0x7ff0f806f0d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff1100fdde0 tx=0x7ff10400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:20.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:20.855+0000 7ff0f77fe700 1 -- 172.21.15.32:0/1271327289 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff1000574e0 con 0x7ff1100fb4d0 2024-04-06T14:49:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.078+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1100fe880 con 0x7ff0f806cc20 2024-04-06T14:49:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.079+0000 7ff0f77fe700 1 -- 172.21.15.32:0/1271327289 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff10400f040 con 0x7ff0f806cc20 2024-04-06T14:49:21.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0f806cc20 msgr2=0x7ff0f806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0f806cc20 0x7ff0f806f0d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff1100fdde0 tx=0x7ff10400b040 comp rx=0 tx=0).stop 2024-04-06T14:49:21.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1100fb4d0 msgr2=0x7ff1100fc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1100fb4d0 0x7ff1100fc6b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff100000c00 tx=0x7ff100009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 shutdown_connections 2024-04-06T14:49:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff0f806cc20 0x7ff0f806f0d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1100fb4d0 0x7ff1100fc6b0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 --2- 172.21.15.32:0/1271327289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110102f50 0x7ff1100fcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 >> 172.21.15.32:0/1271327289 conn(0x7ff1100714c0 msgr2=0x7ff1100fa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 shutdown_connections 2024-04-06T14:49:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.081+0000 7ff1155e1700 1 -- 172.21.15.32:0/1271327289 wait complete. 2024-04-06T14:49:21.080 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:49:21.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 -- 172.21.15.32:0/2597362231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 msgr2=0x7f19c00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 --2- 172.21.15.32:0/2597362231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 0x7f19c00fce10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f19b0004940 tx=0x7f19b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:21.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 -- 172.21.15.32:0/2597362231 shutdown_connections 2024-04-06T14:49:21.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 --2- 172.21.15.32:0/2597362231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 0x7f19c00fce10 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 --2- 172.21.15.32:0/2597362231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c0102e00 0x7f19c01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 -- 172.21.15.32:0/2597362231 >> 172.21.15.32:0/2597362231 conn(0x7f19c00f89b0 msgr2=0x7f19c00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:21.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 -- 172.21.15.32:0/2597362231 shutdown_connections 2024-04-06T14:49:21.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 -- 172.21.15.32:0/2597362231 wait complete. 2024-04-06T14:49:21.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 Processor -- start 2024-04-06T14:49:21.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.186+0000 7f19c56d1700 1 -- start start 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19c56d1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c0102e00 0x7f19c00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19c56d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 0x7f19c0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19c56d1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0072210 con 0x7f19c01037a0 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19c56d1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0070d40 con 0x7f19c0102e00 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19be7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 0x7f19c0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19be7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 0x7f19c0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42326/0 (socket says 172.21.15.32:42326) 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19be7fc700 1 -- 172.21.15.32:0/788390091 learned_addr learned my addr 172.21.15.32:0/788390091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19beffd700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c0102e00 0x7f19c00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19be7fc700 1 -- 172.21.15.32:0/788390091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c0102e00 msgr2=0x7f19c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19be7fc700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c0102e00 0x7f19c00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19be7fc700 1 -- 172.21.15.32:0/788390091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b0005040 con 0x7f19c01037a0 2024-04-06T14:49:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19beffd700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c0102e00 0x7f19c00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:49:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19be7fc700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 0x7f19c0070800 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f19b0009380 tx=0x7f19b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19a77fe700 1 -- 172.21.15.32:0/788390091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19b000e040 con 0x7f19c01037a0 2024-04-06T14:49:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19a77fe700 1 -- 172.21.15.32:0/788390091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19b0013420 con 0x7f19c01037a0 2024-04-06T14:49:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c0070fc0 con 0x7f19c01037a0 2024-04-06T14:49:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.187+0000 7f19a77fe700 1 -- 172.21.15.32:0/788390091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19b001c630 con 0x7f19c01037a0 2024-04-06T14:49:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.188+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c011a010 con 0x7f19c01037a0 2024-04-06T14:49:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.188+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c00fe550 con 0x7f19c01037a0 2024-04-06T14:49:21.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.189+0000 7f19a77fe700 1 -- 172.21.15.32:0/788390091 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f19b00139e0 con 0x7f19c01037a0 2024-04-06T14:49:21.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.189+0000 7f19a77fe700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19a806cc20 0x7f19a806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:21.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.189+0000 7f19a77fe700 1 -- 172.21.15.32:0/788390091 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f19b008d220 con 0x7f19c01037a0 2024-04-06T14:49:21.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.189+0000 7f19beffd700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19a806cc20 0x7f19a806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:21.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.190+0000 7f19beffd700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19a806cc20 0x7f19a806f0d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f19b4007fd0 tx=0x7f19b4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:21.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.194+0000 7f19a77fe700 1 -- 172.21.15.32:0/788390091 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f19b0057260 con 0x7f19c01037a0 2024-04-06T14:49:21.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.413+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19c00fa140 con 0x7f19a806cc20 2024-04-06T14:49:21.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.414+0000 7f19a77fe700 1 -- 172.21.15.32:0/788390091 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f19c00fa140 con 0x7f19a806cc20 2024-04-06T14:49:21.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.416+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19a806cc20 msgr2=0x7f19a806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.416+0000 7f19c56d1700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19a806cc20 0x7f19a806f0d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f19b4007fd0 tx=0x7f19b4009040 comp rx=0 tx=0).stop 2024-04-06T14:49:21.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.416+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 msgr2=0x7f19c0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.416+0000 7f19c56d1700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 0x7f19c0070800 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f19b0009380 tx=0x7f19b0009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:21.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.416+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 shutdown_connections 2024-04-06T14:49:21.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.416+0000 7f19c56d1700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19a806cc20 0x7f19a806f0d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.416+0000 7f19c56d1700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c0102e00 0x7f19c00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.416+0000 7f19c56d1700 1 --2- 172.21.15.32:0/788390091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19c01037a0 0x7f19c0070800 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.416+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 >> 172.21.15.32:0/788390091 conn(0x7f19c00f89b0 msgr2=0x7f19c00f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:21.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.417+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 shutdown_connections 2024-04-06T14:49:21.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.417+0000 7f19c56d1700 1 -- 172.21.15.32:0/788390091 wait complete. 2024-04-06T14:49:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 -- 172.21.15.32:0/1581039290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9268070bf0 msgr2=0x7f9268109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 --2- 172.21.15.32:0/1581039290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9268070bf0 0x7f9268109130 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9258004940 tx=0x7f9258009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 -- 172.21.15.32:0/1581039290 shutdown_connections 2024-04-06T14:49:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 --2- 172.21.15.32:0/1581039290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9268070bf0 0x7f9268109130 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 --2- 172.21.15.32:0/1581039290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92680702e0 0x7f92680706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 -- 172.21.15.32:0/1581039290 >> 172.21.15.32:0/1581039290 conn(0x7f92680f8970 msgr2=0x7f92680fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 -- 172.21.15.32:0/1581039290 shutdown_connections 2024-04-06T14:49:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 -- 172.21.15.32:0/1581039290 wait complete. 2024-04-06T14:49:21.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 Processor -- start 2024-04-06T14:49:21.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.521+0000 7f926d794700 1 -- start start 2024-04-06T14:49:21.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f926d794700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92680702e0 0x7f9268115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:21.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f926d794700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9268070bf0 0x7f92681161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:21.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f926d794700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9268116830 con 0x7f92680702e0 2024-04-06T14:49:21.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f926d794700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926810fc00 con 0x7f9268070bf0 2024-04-06T14:49:21.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f9266ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92680702e0 0x7f9268115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f9266ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92680702e0 0x7f9268115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42336/0 (socket says 172.21.15.32:42336) 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f9266ffd700 1 -- 172.21.15.32:0/2792990159 learned_addr learned my addr 172.21.15.32:0/2792990159 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f92667fc700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9268070bf0 0x7f92681161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f9266ffd700 1 -- 172.21.15.32:0/2792990159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9268070bf0 msgr2=0x7f92681161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f9266ffd700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9268070bf0 0x7f92681161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f9266ffd700 1 -- 172.21.15.32:0/2792990159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9258005040 con 0x7f92680702e0 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f92667fc700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9268070bf0 0x7f92681161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f9266ffd700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92680702e0 0x7f9268115ca0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9250002a50 tx=0x7f92500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.522+0000 7f925f7fe700 1 -- 172.21.15.32:0/2792990159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9250007cc0 con 0x7f92680702e0 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.523+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926810fee0 con 0x7f92680702e0 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.523+0000 7f925f7fe700 1 -- 172.21.15.32:0/2792990159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f925000f040 con 0x7f92680702e0 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.523+0000 7f925f7fe700 1 -- 172.21.15.32:0/2792990159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9250009540 con 0x7f92680702e0 2024-04-06T14:49:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.523+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9268110420 con 0x7f92680702e0 2024-04-06T14:49:21.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.523+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9268071a20 con 0x7f92680702e0 2024-04-06T14:49:21.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.524+0000 7f925f7fe700 1 -- 172.21.15.32:0/2792990159 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f92500096a0 con 0x7f92680702e0 2024-04-06T14:49:21.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.524+0000 7f925f7fe700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f925406cc20 0x7f925406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:21.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.525+0000 7f925f7fe700 1 -- 172.21.15.32:0/2792990159 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f925008be70 con 0x7f92680702e0 2024-04-06T14:49:21.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.525+0000 7f92667fc700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f925406cc20 0x7f925406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:21.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.525+0000 7f92667fc700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f925406cc20 0x7f925406f0d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9258004f90 tx=0x7f9258009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:21.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.529+0000 7f925f7fe700 1 -- 172.21.15.32:0/2792990159 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f92500598c0 con 0x7f92680702e0 2024-04-06T14:49:21.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.745+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f926804bf40 con 0x7f925406cc20 2024-04-06T14:49:21.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.755+0000 7f925f7fe700 1 -- 172.21.15.32:0/2792990159 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f926804bf40 con 0x7f925406cc20 2024-04-06T14:49:21.744 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:49:21.744 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (29m) 7m ago 40m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:49:21.744 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (27m) 7m ago 41m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (26m) 7m ago 40m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (29m) 7m ago 40m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (23m) 7m ago 35m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (23m) 7m ago 35m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (22m) 7m ago 35m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (7m) 7m ago 35m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (31m) 7m ago 42m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (30m) 7m ago 40m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (28m) 7m ago 42m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (27m) 7m ago 40m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (30m) 7m ago 41m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (30m) 7m ago 40m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (26m) 7m ago 39m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:49:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (26m) 7m ago 38m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:49:21.746 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (25m) 7m ago 38m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:49:21.746 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (24m) 7m ago 37m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:49:21.746 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (24m) 7m ago 37m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:49:21.746 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (23m) 7m ago 36m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:49:21.746 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (30m) 7m ago 40m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:49:21.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f925406cc20 msgr2=0x7f925406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f925406cc20 0x7f925406f0d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9258004f90 tx=0x7f9258009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92680702e0 msgr2=0x7f9268115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92680702e0 0x7f9268115ca0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9250002a50 tx=0x7f92500046b0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 shutdown_connections 2024-04-06T14:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f925406cc20 0x7f925406f0d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92680702e0 0x7f9268115ca0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 --2- 172.21.15.32:0/2792990159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9268070bf0 0x7f92681161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 >> 172.21.15.32:0/2792990159 conn(0x7f92680f8970 msgr2=0x7f92680f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 shutdown_connections 2024-04-06T14:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.758+0000 7f926d794700 1 -- 172.21.15.32:0/2792990159 wait complete. 2024-04-06T14:49:21.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.859+0000 7fb746f93700 1 -- 172.21.15.32:0/3866757357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740102710 msgr2=0x7fb7400fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.859+0000 7fb746f93700 1 --2- 172.21.15.32:0/3866757357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740102710 0x7fb7400fbd80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb72c004940 tx=0x7fb72c009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.859+0000 7fb746f93700 1 -- 172.21.15.32:0/3866757357 shutdown_connections 2024-04-06T14:49:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.859+0000 7fb746f93700 1 --2- 172.21.15.32:0/3866757357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740102710 0x7fb7400fbd80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.859+0000 7fb746f93700 1 --2- 172.21.15.32:0/3866757357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740101d70 0x7fb740102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.859+0000 7fb746f93700 1 -- 172.21.15.32:0/3866757357 >> 172.21.15.32:0/3866757357 conn(0x7fb740072bc0 msgr2=0x7fb740072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.859+0000 7fb746f93700 1 -- 172.21.15.32:0/3866757357 shutdown_connections 2024-04-06T14:49:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.859+0000 7fb746f93700 1 -- 172.21.15.32:0/3866757357 wait complete. 2024-04-06T14:49:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.860+0000 7fb746f93700 1 Processor -- start 2024-04-06T14:49:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.860+0000 7fb746f93700 1 -- start start 2024-04-06T14:49:21.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.860+0000 7fb746f93700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb7401139c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:21.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.860+0000 7fb746f93700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 0x7fb740113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:21.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.860+0000 7fb746f93700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb740114550 con 0x7fb740101d70 2024-04-06T14:49:21.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.860+0000 7fb746f93700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74010d980 con 0x7fb740102710 2024-04-06T14:49:21.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.860+0000 7fb744d2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb7401139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:21.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb744d2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb7401139c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42360/0 (socket says 172.21.15.32:42360) 2024-04-06T14:49:21.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb744d2f700 1 -- 172.21.15.32:0/2784777053 learned_addr learned my addr 172.21.15.32:0/2784777053 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb73ffff700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 0x7fb740113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb744d2f700 1 -- 172.21.15.32:0/2784777053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 msgr2=0x7fb740113f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb744d2f700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 0x7fb740113f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb744d2f700 1 -- 172.21.15.32:0/2784777053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c005040 con 0x7fb740101d70 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb744d2f700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb7401139c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb734002a80 tx=0x7fb7340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb73d7fa700 1 -- 172.21.15.32:0/2784777053 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb734007cc0 con 0x7fb740101d70 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb73d7fa700 1 -- 172.21.15.32:0/2784777053 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb73400f040 con 0x7fb740101d70 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74010dc60 con 0x7fb740101d70 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb73d7fa700 1 -- 172.21.15.32:0/2784777053 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb73400da60 con 0x7fb740101d70 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.861+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74010e1f0 con 0x7fb740101d70 2024-04-06T14:49:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.862+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7400fd4c0 con 0x7fb740101d70 2024-04-06T14:49:21.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.863+0000 7fb73d7fa700 1 -- 172.21.15.32:0/2784777053 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb73400dbc0 con 0x7fb740101d70 2024-04-06T14:49:21.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.863+0000 7fb73d7fa700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb73006cc30 0x7fb73006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:21.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.863+0000 7fb73d7fa700 1 -- 172.21.15.32:0/2784777053 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb73408bcb0 con 0x7fb740101d70 2024-04-06T14:49:21.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.863+0000 7fb73ffff700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb73006cc30 0x7fb73006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:21.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.864+0000 7fb73ffff700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb73006cc30 0x7fb73006f0e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb74010f2a0 tx=0x7fb72c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:21.868+0000 7fb73d7fa700 1 -- 172.21.15.32:0/2784777053 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb734059700 con 0x7fb740101d70 2024-04-06T14:49:21.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:21 smithi086 ceph-mon[110975]: pgmap v877: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:21.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:21 smithi032 ceph-mon[147414]: pgmap v877: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:22.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.155+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb74004bed0 con 0x7fb740101d70 2024-04-06T14:49:22.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.156+0000 7fb73d7fa700 1 -- 172.21.15.32:0/2784777053 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fb734059290 con 0x7fb740101d70 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:49:22.145 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:49:22.146 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:49:22.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:49:22.146 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:49:22.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb73006cc30 msgr2=0x7fb73006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb73006cc30 0x7fb73006f0e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb74010f2a0 tx=0x7fb72c009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 msgr2=0x7fb7401139c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb7401139c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb734002a80 tx=0x7fb7340046b0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 shutdown_connections 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb73006cc30 0x7fb73006f0e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb7401139c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 --2- 172.21.15.32:0/2784777053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 0x7fb740113f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 >> 172.21.15.32:0/2784777053 conn(0x7fb740072bc0 msgr2=0x7fb740065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 shutdown_connections 2024-04-06T14:49:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.158+0000 7fb746f93700 1 -- 172.21.15.32:0/2784777053 wait complete. 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.260+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/3042159502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 msgr2=0x7f9d540fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.260+0000 7f9d5c2a6700 1 --2- 172.21.15.32:0/3042159502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 0x7f9d540fbd80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9d48004940 tx=0x7f9d48009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.260+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/3042159502 shutdown_connections 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.260+0000 7f9d5c2a6700 1 --2- 172.21.15.32:0/3042159502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 0x7f9d540fbd80 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.260+0000 7f9d5c2a6700 1 --2- 172.21.15.32:0/3042159502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d54101d70 0x7f9d54102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.260+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/3042159502 >> 172.21.15.32:0/3042159502 conn(0x7f9d54072bc0 msgr2=0x7f9d54072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.260+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/3042159502 shutdown_connections 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.260+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/3042159502 wait complete. 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d5c2a6700 1 Processor -- start 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d5c2a6700 1 -- start start 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d5c2a6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d54101d70 0x7f9d5410ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d5c2a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 0x7f9d5410efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d5c2a6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5410f5f0 con 0x7f9d54102710 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d5c2a6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5410f730 con 0x7f9d54101d70 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d59841700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 0x7f9d5410efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d59841700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 0x7f9d5410efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42380/0 (socket says 172.21.15.32:42380) 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d59841700 1 -- 172.21.15.32:0/356871387 learned_addr learned my addr 172.21.15.32:0/356871387 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d5a042700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d54101d70 0x7f9d5410ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d59841700 1 -- 172.21.15.32:0/356871387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d54101d70 msgr2=0x7f9d5410ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d59841700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d54101d70 0x7f9d5410ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d59841700 1 -- 172.21.15.32:0/356871387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d48005040 con 0x7f9d54102710 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.261+0000 7f9d59841700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 0x7f9d5410efa0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009380 tx=0x7f9d48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.262+0000 7f9d46ffd700 1 -- 172.21.15.32:0/356871387 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d4800e040 con 0x7f9d54102710 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.262+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d54117ad0 con 0x7f9d54102710 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.262+0000 7f9d46ffd700 1 -- 172.21.15.32:0/356871387 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d48013420 con 0x7f9d54102710 2024-04-06T14:49:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.262+0000 7f9d46ffd700 1 -- 172.21.15.32:0/356871387 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d4801c630 con 0x7f9d54102710 2024-04-06T14:49:22.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.262+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d54117fe0 con 0x7f9d54102710 2024-04-06T14:49:22.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.263+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d540ff080 con 0x7f9d54102710 2024-04-06T14:49:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.263+0000 7f9d46ffd700 1 -- 172.21.15.32:0/356871387 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9d480139e0 con 0x7f9d54102710 2024-04-06T14:49:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.264+0000 7f9d46ffd700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d4006cc20 0x7f9d4006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.264+0000 7f9d46ffd700 1 -- 172.21.15.32:0/356871387 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9d4808d240 con 0x7f9d54102710 2024-04-06T14:49:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.264+0000 7f9d5a042700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d4006cc20 0x7f9d4006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.264+0000 7f9d5a042700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d4006cc20 0x7f9d4006f0d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9d50004d60 tx=0x7f9d5000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:22.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.268+0000 7f9d46ffd700 1 -- 172.21.15.32:0/356871387 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9d48057280 con 0x7f9d54102710 2024-04-06T14:49:22.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.521+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9d54065db0 con 0x7f9d54102710 2024-04-06T14:49:22.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.521+0000 7f9d46ffd700 1 -- 172.21.15.32:0/356871387 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f9d4805a8a0 con 0x7f9d54102710 2024-04-06T14:49:22.510 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:49:22.510 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:49:22.510 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:49:22.510 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:49:22.510 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:49:22.511 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:49:22.512 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:49:22.513 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:49:22.513 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:49:22.513 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:49:22.513 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:49:22.513 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d4006cc20 msgr2=0x7f9d4006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d4006cc20 0x7f9d4006f0d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9d50004d60 tx=0x7f9d5000b040 comp rx=0 tx=0).stop 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 msgr2=0x7f9d5410efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 0x7f9d5410efa0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009380 tx=0x7f9d48009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 shutdown_connections 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9d4006cc20 0x7f9d4006f0d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d54101d70 0x7f9d5410ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 --2- 172.21.15.32:0/356871387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d54102710 0x7f9d5410efa0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 >> 172.21.15.32:0/356871387 conn(0x7f9d54072bc0 msgr2=0x7f9d540fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 shutdown_connections 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.523+0000 7f9d5c2a6700 1 -- 172.21.15.32:0/356871387 wait complete. 2024-04-06T14:49:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:49:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.628+0000 7f23c27a2700 1 -- 172.21.15.32:0/1488124881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc101d70 msgr2=0x7f23bc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.628+0000 7f23c27a2700 1 --2- 172.21.15.32:0/1488124881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc101d70 0x7f23bc102140 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f23ac004940 tx=0x7f23ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.628+0000 7f23c27a2700 1 -- 172.21.15.32:0/1488124881 shutdown_connections 2024-04-06T14:49:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.628+0000 7f23c27a2700 1 --2- 172.21.15.32:0/1488124881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23bc102710 0x7f23bc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.628+0000 7f23c27a2700 1 --2- 172.21.15.32:0/1488124881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc101d70 0x7f23bc102140 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.628+0000 7f23c27a2700 1 -- 172.21.15.32:0/1488124881 >> 172.21.15.32:0/1488124881 conn(0x7f23bc072bc0 msgr2=0x7f23bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.628+0000 7f23c27a2700 1 -- 172.21.15.32:0/1488124881 shutdown_connections 2024-04-06T14:49:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.628+0000 7f23c27a2700 1 -- 172.21.15.32:0/1488124881 wait complete. 2024-04-06T14:49:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.628+0000 7f23c27a2700 1 Processor -- start 2024-04-06T14:49:22.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23c27a2700 1 -- start start 2024-04-06T14:49:22.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23c27a2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23bc101d70 0x7f23bc102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:22.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23c27a2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc102710 0x7f23bc103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:22.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23c27a2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc103650 con 0x7f23bc102710 2024-04-06T14:49:22.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23c27a2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc1037c0 con 0x7f23bc101d70 2024-04-06T14:49:22.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23bb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc102710 0x7f23bc103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:22.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23bb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc102710 0x7f23bc103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42408/0 (socket says 172.21.15.32:42408) 2024-04-06T14:49:22.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23bb7fe700 1 -- 172.21.15.32:0/1327998010 learned_addr learned my addr 172.21.15.32:0/1327998010 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23bbfff700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23bc101d70 0x7f23bc102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23bb7fe700 1 -- 172.21.15.32:0/1327998010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23bc101d70 msgr2=0x7f23bc102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23bb7fe700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23bc101d70 0x7f23bc102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.629+0000 7f23bb7fe700 1 -- 172.21.15.32:0/1327998010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac005040 con 0x7f23bc102710 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.630+0000 7f23bb7fe700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc102710 0x7f23bc103110 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f23b0002a50 tx=0x7f23b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.630+0000 7f23b8ff9700 1 -- 172.21.15.32:0/1327998010 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23b0007cc0 con 0x7f23bc102710 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.630+0000 7f23b8ff9700 1 -- 172.21.15.32:0/1327998010 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23b000f040 con 0x7f23bc102710 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.630+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc119d20 con 0x7f23bc102710 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.630+0000 7f23b8ff9700 1 -- 172.21.15.32:0/1327998010 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23b000da30 con 0x7f23bc102710 2024-04-06T14:49:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.630+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23bc11a120 con 0x7f23bc102710 2024-04-06T14:49:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.631+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23bc0fd4c0 con 0x7f23bc102710 2024-04-06T14:49:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.631+0000 7f23b8ff9700 1 -- 172.21.15.32:0/1327998010 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f23b000db90 con 0x7f23bc102710 2024-04-06T14:49:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.632+0000 7f23b8ff9700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f23a406cc20 0x7f23a406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.632+0000 7f23b8ff9700 1 -- 172.21.15.32:0/1327998010 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f23b008bc80 con 0x7f23bc102710 2024-04-06T14:49:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.632+0000 7f23bbfff700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f23a406cc20 0x7f23a406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.632+0000 7f23bbfff700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f23a406cc20 0x7f23a406f0d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f23ac004f90 tx=0x7f23ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:22.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.636+0000 7f23b8ff9700 1 -- 172.21.15.32:0/1327998010 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f23b0055c40 con 0x7f23bc102710 2024-04-06T14:49:22.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.863+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f23bc04bf70 con 0x7f23a406cc20 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.864+0000 7f23b8ff9700 1 -- 172.21.15.32:0/1327998010 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f23ac007d70 con 0x7f23a406cc20 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:49:22.853 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:49:22.854 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:49:22.854 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:49:22.854 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:49:22.854 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:49:22.854 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:49:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f23a406cc20 msgr2=0x7f23a406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f23a406cc20 0x7f23a406f0d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f23ac004f90 tx=0x7f23ac009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc102710 msgr2=0x7f23bc103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc102710 0x7f23bc103110 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f23b0002a50 tx=0x7f23b00046b0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 shutdown_connections 2024-04-06T14:49:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f23a406cc20 0x7f23a406f0d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23bc101d70 0x7f23bc102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 --2- 172.21.15.32:0/1327998010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc102710 0x7f23bc103110 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 >> 172.21.15.32:0/1327998010 conn(0x7f23bc072bc0 msgr2=0x7f23bc065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:22.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 shutdown_connections 2024-04-06T14:49:22.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.866+0000 7f23c27a2700 1 -- 172.21.15.32:0/1327998010 wait complete. 2024-04-06T14:49:22.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:22 smithi086 ceph-mon[110975]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:22.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:22 smithi086 ceph-mon[110975]: from='client.35602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:22.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:22 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2784777053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:49:22.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.968+0000 7fb9f3014700 1 -- 172.21.15.32:0/217204695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec1026b0 msgr2=0x7fb9ec0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.968+0000 7fb9f3014700 1 --2- 172.21.15.32:0/217204695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec1026b0 0x7fb9ec0fbd20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8004940 tx=0x7fb9d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.968+0000 7fb9f3014700 1 -- 172.21.15.32:0/217204695 shutdown_connections 2024-04-06T14:49:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.968+0000 7fb9f3014700 1 --2- 172.21.15.32:0/217204695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec1026b0 0x7fb9ec0fbd20 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.968+0000 7fb9f3014700 1 --2- 172.21.15.32:0/217204695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ec101d10 0x7fb9ec1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.968+0000 7fb9f3014700 1 -- 172.21.15.32:0/217204695 >> 172.21.15.32:0/217204695 conn(0x7fb9ec0f7880 msgr2=0x7fb9ec0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.968+0000 7fb9f3014700 1 -- 172.21.15.32:0/217204695 shutdown_connections 2024-04-06T14:49:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.968+0000 7fb9f3014700 1 -- 172.21.15.32:0/217204695 wait complete. 2024-04-06T14:49:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.969+0000 7fb9f3014700 1 Processor -- start 2024-04-06T14:49:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.969+0000 7fb9f3014700 1 -- start start 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.969+0000 7fb9f3014700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec101d10 0x7fb9ec113da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.969+0000 7fb9f3014700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ec1026b0 0x7fb9ec1142e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.969+0000 7fb9f3014700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec114930 con 0x7fb9ec101d10 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.969+0000 7fb9f3014700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec10dd00 con 0x7fb9ec1026b0 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.969+0000 7fb9f0db0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec101d10 0x7fb9ec113da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9f0db0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec101d10 0x7fb9ec113da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42416/0 (socket says 172.21.15.32:42416) 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9f0db0700 1 -- 172.21.15.32:0/30316018 learned_addr learned my addr 172.21.15.32:0/30316018 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9ebfff700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ec1026b0 0x7fb9ec1142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9f0db0700 1 -- 172.21.15.32:0/30316018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ec1026b0 msgr2=0x7fb9ec1142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9f0db0700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ec1026b0 0x7fb9ec1142e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9f0db0700 1 -- 172.21.15.32:0/30316018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d8005040 con 0x7fb9ec101d10 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9f0db0700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec101d10 0x7fb9ec113da0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000c8c0 tx=0x7fb9e00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9e97fa700 1 -- 172.21.15.32:0/30316018 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e000a760 con 0x7fb9ec101d10 2024-04-06T14:49:22.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ec10dfe0 con 0x7fb9ec101d10 2024-04-06T14:49:22.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9e97fa700 1 -- 172.21.15.32:0/30316018 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9e000ad70 con 0x7fb9ec101d10 2024-04-06T14:49:22.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9e97fa700 1 -- 172.21.15.32:0/30316018 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e00113a0 con 0x7fb9ec101d10 2024-04-06T14:49:22.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.970+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec10e5a0 con 0x7fb9ec101d10 2024-04-06T14:49:22.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.971+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ec0fd460 con 0x7fb9ec101d10 2024-04-06T14:49:22.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.972+0000 7fb9e97fa700 1 -- 172.21.15.32:0/30316018 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb9e0012400 con 0x7fb9ec101d10 2024-04-06T14:49:22.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.972+0000 7fb9e97fa700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:22.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.972+0000 7fb9e97fa700 1 -- 172.21.15.32:0/30316018 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb9e008c090 con 0x7fb9ec101d10 2024-04-06T14:49:22.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.972+0000 7fb9ebfff700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:22.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.973+0000 7fb9ebfff700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec10f640 tx=0x7fb9d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:22.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:22.977+0000 7fb9e97fa700 1 -- 172.21.15.32:0/30316018 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb9e0059b60 con 0x7fb9ec101d10 2024-04-06T14:49:22.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:22 smithi032 ceph-mon[147414]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:22.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:22 smithi032 ceph-mon[147414]: from='client.35602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:22.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:22 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2784777053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.261+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb9ec05f5f0 con 0x7fb9ec101d10 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.262+0000 7fb9e97fa700 1 -- 172.21.15.32:0/30316018 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7fb9e00596f0 con 0x7fb9ec101d10 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:49:23.251 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:49:23.252 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:49:23.253 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:49:23.253 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:49:23.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9dc06cc30 msgr2=0x7fb9dc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:23.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec10f640 tx=0x7fb9d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:23.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec101d10 msgr2=0x7fb9ec113da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:23.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec101d10 0x7fb9ec113da0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000c8c0 tx=0x7fb9e00044c0 comp rx=0 tx=0).stop 2024-04-06T14:49:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 shutdown_connections 2024-04-06T14:49:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec101d10 0x7fb9ec113da0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 --2- 172.21.15.32:0/30316018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9ec1026b0 0x7fb9ec1142e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 >> 172.21.15.32:0/30316018 conn(0x7fb9ec0f7880 msgr2=0x7fb9ec0f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 shutdown_connections 2024-04-06T14:49:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:23.264+0000 7fb9f3014700 1 -- 172.21.15.32:0/30316018 wait complete. 2024-04-06T14:49:23.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:23 smithi032 ceph-mon[147414]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:23.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:23 smithi032 ceph-mon[147414]: pgmap v878: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:23.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:23 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/356871387' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:49:23.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:49:23.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:23 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/30316018' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:49:23.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:23 smithi086 ceph-mon[110975]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:23.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:23 smithi086 ceph-mon[110975]: pgmap v878: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:23.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:23 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/356871387' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:49:23.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:49:23.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:23 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/30316018' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:49:24.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:24 smithi086 ceph-mon[110975]: from='client.35618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:24.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:24 smithi032 ceph-mon[147414]: from='client.35618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:25.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:25 smithi086 ceph-mon[110975]: pgmap v879: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:25.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:25 smithi032 ceph-mon[147414]: pgmap v879: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:27.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:27 smithi086 ceph-mon[110975]: pgmap v880: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:27.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:27 smithi032 ceph-mon[147414]: pgmap v880: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:29.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:29 smithi086 ceph-mon[110975]: pgmap v881: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:29.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:29 smithi032 ceph-mon[147414]: pgmap v881: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:31.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:31 smithi086 ceph-mon[110975]: pgmap v882: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:31.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:31 smithi032 ceph-mon[147414]: pgmap v882: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:33.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:33 smithi032 ceph-mon[147414]: pgmap v883: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:33.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:33 smithi086 ceph-mon[110975]: pgmap v883: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:35.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:35 smithi086 ceph-mon[110975]: pgmap v884: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:35.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:35 smithi032 ceph-mon[147414]: pgmap v884: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:37.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:37 smithi086 ceph-mon[110975]: pgmap v885: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:37.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:37 smithi032 ceph-mon[147414]: pgmap v885: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:49:38.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:49:39.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:39 smithi086 ceph-mon[110975]: pgmap v886: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:39.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:39 smithi032 ceph-mon[147414]: pgmap v886: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:41.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:41 smithi086 ceph-mon[110975]: pgmap v887: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:41.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:41 smithi032 ceph-mon[147414]: pgmap v887: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:43.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:43 smithi032 ceph-mon[147414]: pgmap v888: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:43.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:43 smithi086 ceph-mon[110975]: pgmap v888: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:45.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:45 smithi086 ceph-mon[110975]: pgmap v889: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:45.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:45 smithi032 ceph-mon[147414]: pgmap v889: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:47.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:47 smithi086 ceph-mon[110975]: pgmap v890: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:47.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:47 smithi032 ceph-mon[147414]: pgmap v890: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:49.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:49 smithi086 ceph-mon[110975]: pgmap v891: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:49.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:49 smithi032 ceph-mon[147414]: pgmap v891: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:51.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:51 smithi086 ceph-mon[110975]: pgmap v892: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:51.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:51 smithi032 ceph-mon[147414]: pgmap v892: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.381+0000 7f0eaa227700 1 -- 172.21.15.32:0/2045482567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea4102e10 msgr2=0x7f0ea41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.381+0000 7f0eaa227700 1 --2- 172.21.15.32:0/2045482567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea4102e10 0x7f0ea41031e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0e94004940 tx=0x7f0e94009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.381+0000 7f0eaa227700 1 -- 172.21.15.32:0/2045482567 shutdown_connections 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.381+0000 7f0eaa227700 1 --2- 172.21.15.32:0/2045482567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ea41037b0 0x7f0ea40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.381+0000 7f0eaa227700 1 --2- 172.21.15.32:0/2045482567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea4102e10 0x7f0ea41031e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.381+0000 7f0eaa227700 1 -- 172.21.15.32:0/2045482567 >> 172.21.15.32:0/2045482567 conn(0x7f0ea40f8980 msgr2=0x7f0ea40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.382+0000 7f0eaa227700 1 -- 172.21.15.32:0/2045482567 shutdown_connections 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.382+0000 7f0eaa227700 1 -- 172.21.15.32:0/2045482567 wait complete. 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.382+0000 7f0eaa227700 1 Processor -- start 2024-04-06T14:49:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.382+0000 7f0eaa227700 1 -- start start 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0eaa227700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ea4102e10 0x7f0ea40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0eaa227700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea41037b0 0x7f0ea4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0eaa227700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea4070d90 con 0x7f0ea41037b0 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0eaa227700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea4070ed0 con 0x7f0ea4102e10 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0ea2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea41037b0 0x7f0ea4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0ea2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea41037b0 0x7f0ea4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42850/0 (socket says 172.21.15.32:42850) 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0ea2ffd700 1 -- 172.21.15.32:0/3754400153 learned_addr learned my addr 172.21.15.32:0/3754400153 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0ea37fe700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ea4102e10 0x7f0ea40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0ea2ffd700 1 -- 172.21.15.32:0/3754400153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ea4102e10 msgr2=0x7f0ea40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0ea2ffd700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ea4102e10 0x7f0ea40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0ea2ffd700 1 -- 172.21.15.32:0/3754400153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e94005040 con 0x7f0ea41037b0 2024-04-06T14:49:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0ea37fe700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ea4102e10 0x7f0ea40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:49:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0ea2ffd700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea41037b0 0x7f0ea4070800 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f0e98002a50 tx=0x7f0e98004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0e8bfff700 1 -- 172.21.15.32:0/3754400153 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e98007cc0 con 0x7f0ea41037b0 2024-04-06T14:49:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.383+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea4119e10 con 0x7f0ea41037b0 2024-04-06T14:49:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.384+0000 7f0e8bfff700 1 -- 172.21.15.32:0/3754400153 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e9800f040 con 0x7f0ea41037b0 2024-04-06T14:49:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.384+0000 7f0e8bfff700 1 -- 172.21.15.32:0/3754400153 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e9800d9a0 con 0x7f0ea41037b0 2024-04-06T14:49:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.384+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea411a290 con 0x7f0ea41037b0 2024-04-06T14:49:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.384+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea40fe560 con 0x7f0ea41037b0 2024-04-06T14:49:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.386+0000 7f0e8bfff700 1 -- 172.21.15.32:0/3754400153 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0e9800dc60 con 0x7f0ea41037b0 2024-04-06T14:49:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.386+0000 7f0e8bfff700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0e8c06cc20 0x7f0e8c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.386+0000 7f0e8bfff700 1 -- 172.21.15.32:0/3754400153 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0e9808c500 con 0x7f0ea41037b0 2024-04-06T14:49:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.387+0000 7f0ea37fe700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0e8c06cc20 0x7f0e8c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.387+0000 7f0ea37fe700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0e8c06cc20 0x7f0e8c06f0d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0e94004f90 tx=0x7f0e94009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:53.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.390+0000 7f0e8bfff700 1 -- 172.21.15.32:0/3754400153 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0e98056540 con 0x7f0ea41037b0 2024-04-06T14:49:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.622+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ea4109e70 con 0x7f0e8c06cc20 2024-04-06T14:49:53.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.624+0000 7f0e8bfff700 1 -- 172.21.15.32:0/3754400153 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0e94007cc0 con 0x7f0e8c06cc20 2024-04-06T14:49:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0e8c06cc20 msgr2=0x7f0e8c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0e8c06cc20 0x7f0e8c06f0d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0e94004f90 tx=0x7f0e94009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea41037b0 msgr2=0x7f0ea4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea41037b0 0x7f0ea4070800 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f0e98002a50 tx=0x7f0e98004600 comp rx=0 tx=0).stop 2024-04-06T14:49:53.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 shutdown_connections 2024-04-06T14:49:53.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0e8c06cc20 0x7f0e8c06f0d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ea4102e10 0x7f0ea40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 --2- 172.21.15.32:0/3754400153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ea41037b0 0x7f0ea4070800 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 >> 172.21.15.32:0/3754400153 conn(0x7f0ea40f8980 msgr2=0x7f0ea40f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:53.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 shutdown_connections 2024-04-06T14:49:53.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.626+0000 7f0eaa227700 1 -- 172.21.15.32:0/3754400153 wait complete. 2024-04-06T14:49:53.625 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:49:53.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:53 smithi032 ceph-mon[147414]: pgmap v893: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:53.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:49:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.730+0000 7f400d568700 1 -- 172.21.15.32:0/1246041104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 msgr2=0x7f4008106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.730+0000 7f400d568700 1 --2- 172.21.15.32:0/1246041104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 0x7f4008106e80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc004940 tx=0x7f3ffc009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.730+0000 7f400d568700 1 -- 172.21.15.32:0/1246041104 shutdown_connections 2024-04-06T14:49:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.730+0000 7f400d568700 1 --2- 172.21.15.32:0/1246041104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 0x7f4008106e80 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.730+0000 7f400d568700 1 --2- 172.21.15.32:0/1246041104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40080fde10 0x7f40080fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.730+0000 7f400d568700 1 -- 172.21.15.32:0/1246041104 >> 172.21.15.32:0/1246041104 conn(0x7f40080f7870 msgr2=0x7f40080f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.730+0000 7f400d568700 1 -- 172.21.15.32:0/1246041104 shutdown_connections 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.730+0000 7f400d568700 1 -- 172.21.15.32:0/1246041104 wait complete. 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.731+0000 7f400d568700 1 Processor -- start 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.731+0000 7f400d568700 1 -- start start 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.731+0000 7f400d568700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40080fde10 0x7f4008115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.731+0000 7f400d568700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 0x7f4008115fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.731+0000 7f400d568700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4008116630 con 0x7f40080fe7b0 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.731+0000 7f400d568700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400810fc10 con 0x7f40080fde10 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.731+0000 7f40067fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 0x7f4008115fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f40067fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 0x7f4008115fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42872/0 (socket says 172.21.15.32:42872) 2024-04-06T14:49:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f40067fc700 1 -- 172.21.15.32:0/1121190634 learned_addr learned my addr 172.21.15.32:0/1121190634 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f4006ffd700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40080fde10 0x7f4008115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f40067fc700 1 -- 172.21.15.32:0/1121190634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40080fde10 msgr2=0x7f4008115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f40067fc700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40080fde10 0x7f4008115aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f40067fc700 1 -- 172.21.15.32:0/1121190634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc005040 con 0x7f40080fe7b0 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f40067fc700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 0x7f4008115fe0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009380 tx=0x7f3ffc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f3fef7fe700 1 -- 172.21.15.32:0/1121190634 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ffc00e040 con 0x7f40080fe7b0 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400810fe90 con 0x7f40080fe7b0 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f3fef7fe700 1 -- 172.21.15.32:0/1121190634 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ffc004290 con 0x7f40080fe7b0 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f3fef7fe700 1 -- 172.21.15.32:0/1121190634 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ffc013e90 con 0x7f40080fe7b0 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.732+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40081103d0 con 0x7f40080fe7b0 2024-04-06T14:49:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.733+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4008066ad0 con 0x7f40080fe7b0 2024-04-06T14:49:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.734+0000 7f3fef7fe700 1 -- 172.21.15.32:0/1121190634 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3ffc0136b0 con 0x7f40080fe7b0 2024-04-06T14:49:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.734+0000 7f3fef7fe700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ff006cc20 0x7f3ff006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.734+0000 7f4006ffd700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ff006cc20 0x7f3ff006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.734+0000 7f3fef7fe700 1 -- 172.21.15.32:0/1121190634 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3ffc08d370 con 0x7f40080fe7b0 2024-04-06T14:49:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.735+0000 7f4006ffd700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ff006cc20 0x7f3ff006f0d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8007fd0 tx=0x7f3ff8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:53.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.739+0000 7f3fef7fe700 1 -- 172.21.15.32:0/1121190634 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ffc05ae40 con 0x7f40080fe7b0 2024-04-06T14:49:53.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:53 smithi086 ceph-mon[110975]: pgmap v893: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:53.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:49:53.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.965+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4008109d60 con 0x7f3ff006cc20 2024-04-06T14:49:53.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.966+0000 7f3fef7fe700 1 -- 172.21.15.32:0/1121190634 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4008109d60 con 0x7f3ff006cc20 2024-04-06T14:49:53.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ff006cc20 msgr2=0x7f3ff006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ff006cc20 0x7f3ff006f0d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8007fd0 tx=0x7f3ff8009040 comp rx=0 tx=0).stop 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 msgr2=0x7f4008115fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 0x7f4008115fe0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009380 tx=0x7f3ffc009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 shutdown_connections 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3ff006cc20 0x7f3ff006f0d0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40080fde10 0x7f4008115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 --2- 172.21.15.32:0/1121190634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40080fe7b0 0x7f4008115fe0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 >> 172.21.15.32:0/1121190634 conn(0x7f40080f7870 msgr2=0x7f40080fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 shutdown_connections 2024-04-06T14:49:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:53.968+0000 7f400d568700 1 -- 172.21.15.32:0/1121190634 wait complete. 2024-04-06T14:49:54.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.071+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1497535535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 msgr2=0x7f5a381031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.071+0000 7f5a3d6e3700 1 --2- 172.21.15.32:0/1497535535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 0x7f5a381031e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5a28004940 tx=0x7f5a28009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:54.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.071+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1497535535 shutdown_connections 2024-04-06T14:49:54.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.071+0000 7f5a3d6e3700 1 --2- 172.21.15.32:0/1497535535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a381037b0 0x7f5a380fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.071+0000 7f5a3d6e3700 1 --2- 172.21.15.32:0/1497535535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 0x7f5a381031e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.071+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1497535535 >> 172.21.15.32:0/1497535535 conn(0x7f5a380f89a0 msgr2=0x7f5a380fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:54.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.071+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1497535535 shutdown_connections 2024-04-06T14:49:54.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.071+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1497535535 wait complete. 2024-04-06T14:49:54.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a3d6e3700 1 Processor -- start 2024-04-06T14:49:54.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a3d6e3700 1 -- start start 2024-04-06T14:49:54.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a3d6e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 0x7f5a380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:54.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a3d6e3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a381037b0 0x7f5a38070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:54.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a3d6e3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a38070d90 con 0x7f5a38102e10 2024-04-06T14:49:54.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a3d6e3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a38070ed0 con 0x7f5a381037b0 2024-04-06T14:49:54.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a36ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 0x7f5a380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:54.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a36ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 0x7f5a380702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42880/0 (socket says 172.21.15.32:42880) 2024-04-06T14:49:54.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a36ffd700 1 -- 172.21.15.32:0/1335173401 learned_addr learned my addr 172.21.15.32:0/1335173401 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:54.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a367fc700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a381037b0 0x7f5a38070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:54.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a36ffd700 1 -- 172.21.15.32:0/1335173401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a381037b0 msgr2=0x7f5a38070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a36ffd700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a381037b0 0x7f5a38070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a36ffd700 1 -- 172.21.15.32:0/1335173401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a28005040 con 0x7f5a38102e10 2024-04-06T14:49:54.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a367fc700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a381037b0 0x7f5a38070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:49:54.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.072+0000 7f5a36ffd700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 0x7f5a380702c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009180 tx=0x7f5a28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:54.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.073+0000 7f5a2f7fe700 1 -- 172.21.15.32:0/1335173401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a2800e040 con 0x7f5a38102e10 2024-04-06T14:49:54.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.073+0000 7f5a2f7fe700 1 -- 172.21.15.32:0/1335173401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a28004290 con 0x7f5a38102e10 2024-04-06T14:49:54.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.073+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a38071120 con 0x7f5a38102e10 2024-04-06T14:49:54.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.073+0000 7f5a2f7fe700 1 -- 172.21.15.32:0/1335173401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a2801c3f0 con 0x7f5a38102e10 2024-04-06T14:49:54.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.073+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3811a220 con 0x7f5a38102e10 2024-04-06T14:49:54.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.074+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a380fe560 con 0x7f5a38102e10 2024-04-06T14:49:54.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.074+0000 7f5a2f7fe700 1 -- 172.21.15.32:0/1335173401 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5a28013790 con 0x7f5a38102e10 2024-04-06T14:49:54.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.075+0000 7f5a2f7fe700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a2406cc20 0x7f5a2406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:54.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.075+0000 7f5a2f7fe700 1 -- 172.21.15.32:0/1335173401 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5a2808d300 con 0x7f5a38102e10 2024-04-06T14:49:54.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.075+0000 7f5a367fc700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a2406cc20 0x7f5a2406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:54.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.075+0000 7f5a367fc700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a2406cc20 0x7f5a2406f0d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5a20007fd0 tx=0x7f5a20009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:54.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.079+0000 7f5a2f7fe700 1 -- 172.21.15.32:0/1335173401 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5a28057340 con 0x7f5a38102e10 2024-04-06T14:49:54.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.291+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a380719b0 con 0x7f5a2406cc20 2024-04-06T14:49:54.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.301+0000 7f5a2f7fe700 1 -- 172.21.15.32:0/1335173401 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5a380719b0 con 0x7f5a2406cc20 2024-04-06T14:49:54.290 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:49:54.290 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (30m) 8m ago 41m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:49:54.290 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (27m) 8m ago 41m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:49:54.290 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (27m) 8m ago 40m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:49:54.290 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (29m) 8m ago 41m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:49:54.290 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (24m) 8m ago 35m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:49:54.290 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (23m) 8m ago 35m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (23m) 8m ago 35m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (8m) 8m ago 35m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (32m) 8m ago 42m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (31m) 8m ago 40m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (28m) 8m ago 43m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (28m) 8m ago 40m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (30m) 8m ago 41m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (30m) 8m ago 40m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (27m) 8m ago 39m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (26m) 8m ago 39m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (25m) 8m ago 38m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (25m) 8m ago 38m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (24m) 8m ago 37m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (24m) 8m ago 37m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:49:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (30m) 8m ago 41m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:49:54.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a2406cc20 msgr2=0x7f5a2406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a2406cc20 0x7f5a2406f0d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5a20007fd0 tx=0x7f5a20009040 comp rx=0 tx=0).stop 2024-04-06T14:49:54.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 msgr2=0x7f5a380702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 0x7f5a380702c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009180 tx=0x7f5a28009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:54.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 shutdown_connections 2024-04-06T14:49:54.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5a2406cc20 0x7f5a2406f0d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a38102e10 0x7f5a380702c0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 --2- 172.21.15.32:0/1335173401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a381037b0 0x7f5a38070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 >> 172.21.15.32:0/1335173401 conn(0x7f5a380f89a0 msgr2=0x7f5a380f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 shutdown_connections 2024-04-06T14:49:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.303+0000 7f5a3d6e3700 1 -- 172.21.15.32:0/1335173401 wait complete. 2024-04-06T14:49:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.407+0000 7f8eb0c07700 1 -- 172.21.15.32:0/725741503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 msgr2=0x7f8eac0fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.407+0000 7f8eb0c07700 1 --2- 172.21.15.32:0/725741503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 0x7f8eac0fe1d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c004940 tx=0x7f8e9c009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.407+0000 7f8eb0c07700 1 -- 172.21.15.32:0/725741503 shutdown_connections 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.407+0000 7f8eb0c07700 1 --2- 172.21.15.32:0/725741503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eac0fe7a0 0x7f8eac106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.407+0000 7f8eb0c07700 1 --2- 172.21.15.32:0/725741503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 0x7f8eac0fe1d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.407+0000 7f8eb0c07700 1 -- 172.21.15.32:0/725741503 >> 172.21.15.32:0/725741503 conn(0x7f8eac0f7880 msgr2=0x7f8eac0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.407+0000 7f8eb0c07700 1 -- 172.21.15.32:0/725741503 shutdown_connections 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.407+0000 7f8eb0c07700 1 -- 172.21.15.32:0/725741503 wait complete. 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eb0c07700 1 Processor -- start 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eb0c07700 1 -- start start 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eb0c07700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 0x7f8eac115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eb0c07700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eac0fe7a0 0x7f8eac116500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eb0c07700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac116b50 con 0x7f8eac0fde00 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eb0c07700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac10ff20 con 0x7f8eac0fe7a0 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eaa59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 0x7f8eac115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eaa59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 0x7f8eac115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51296/0 (socket says 172.21.15.32:51296) 2024-04-06T14:49:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eaa59c700 1 -- 172.21.15.32:0/3636574654 learned_addr learned my addr 172.21.15.32:0/3636574654 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eaa59c700 1 -- 172.21.15.32:0/3636574654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eac0fe7a0 msgr2=0x7f8eac116500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8ea9d9b700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eac0fe7a0 0x7f8eac116500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eaa59c700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eac0fe7a0 0x7f8eac116500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eaa59c700 1 -- 172.21.15.32:0/3636574654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c005040 con 0x7f8eac0fde00 2024-04-06T14:49:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8ea9d9b700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eac0fe7a0 0x7f8eac116500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:49:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.408+0000 7f8eaa59c700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 0x7f8eac115fc0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00b3b0 tx=0x7f8e9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.409+0000 7f8ea2ffd700 1 -- 172.21.15.32:0/3636574654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9c00e040 con 0x7f8eac0fde00 2024-04-06T14:49:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.409+0000 7f8ea2ffd700 1 -- 172.21.15.32:0/3636574654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e9c013420 con 0x7f8eac0fde00 2024-04-06T14:49:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.409+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac1101a0 con 0x7f8eac0fde00 2024-04-06T14:49:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.409+0000 7f8ea2ffd700 1 -- 172.21.15.32:0/3636574654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9c01c630 con 0x7f8eac0fde00 2024-04-06T14:49:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.409+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac1106e0 con 0x7f8eac0fde00 2024-04-06T14:49:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.410+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eac0666f0 con 0x7f8eac0fde00 2024-04-06T14:49:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.410+0000 7f8ea2ffd700 1 -- 172.21.15.32:0/3636574654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8e9c013590 con 0x7f8eac0fde00 2024-04-06T14:49:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.411+0000 7f8ea2ffd700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e9806cc30 0x7f8e9806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.411+0000 7f8ea2ffd700 1 -- 172.21.15.32:0/3636574654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8e9c08d290 con 0x7f8eac0fde00 2024-04-06T14:49:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.411+0000 7f8ea9d9b700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e9806cc30 0x7f8e9806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.411+0000 7f8ea9d9b700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e9806cc30 0x7f8e9806f0e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8e94007fd0 tx=0x7f8e94009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:54.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.415+0000 7f8ea2ffd700 1 -- 172.21.15.32:0/3636574654 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8e9c05ace0 con 0x7f8eac0fde00 2024-04-06T14:49:54.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.703+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8eac04bed0 con 0x7f8eac0fde00 2024-04-06T14:49:54.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.703+0000 7f8ea2ffd700 1 -- 172.21.15.32:0/3636574654 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f8e9c05a870 con 0x7f8eac0fde00 2024-04-06T14:49:54.692 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:49:54.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:49:54.692 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:49:54.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:49:54.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:49:54.693 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:49:54.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:49:54.693 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:49:54.693 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:49:54.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:49:54.693 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:49:54.693 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:49:54.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:49:54.694 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:49:54.694 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:49:54.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:49:54.694 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:49:54.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e9806cc30 msgr2=0x7f8e9806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e9806cc30 0x7f8e9806f0e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8e94007fd0 tx=0x7f8e94009040 comp rx=0 tx=0).stop 2024-04-06T14:49:54.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 msgr2=0x7f8eac115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 0x7f8eac115fc0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00b3b0 tx=0x7f8e9c009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:54.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 shutdown_connections 2024-04-06T14:49:54.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e9806cc30 0x7f8e9806f0e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eac0fde00 0x7f8eac115fc0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 --2- 172.21.15.32:0/3636574654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eac0fe7a0 0x7f8eac116500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 >> 172.21.15.32:0/3636574654 conn(0x7f8eac0f7880 msgr2=0x7f8eac0f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:54.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 shutdown_connections 2024-04-06T14:49:54.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.705+0000 7f8eb0c07700 1 -- 172.21.15.32:0/3636574654 wait complete. 2024-04-06T14:49:54.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.809+0000 7f987899e700 1 -- 172.21.15.32:0/692436735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 msgr2=0x7f9874101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.809+0000 7f987899e700 1 --2- 172.21.15.32:0/692436735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 0x7f9874101f70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9864004940 tx=0x7f9864009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:54.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.809+0000 7f987899e700 1 -- 172.21.15.32:0/692436735 shutdown_connections 2024-04-06T14:49:54.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.809+0000 7f987899e700 1 --2- 172.21.15.32:0/692436735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98740fbb70 0x7f98740fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.809+0000 7f987899e700 1 --2- 172.21.15.32:0/692436735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 0x7f9874101f70 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.809+0000 7f987899e700 1 -- 172.21.15.32:0/692436735 >> 172.21.15.32:0/692436735 conn(0x7f9874071b70 msgr2=0x7f9874071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:54.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.809+0000 7f987899e700 1 -- 172.21.15.32:0/692436735 shutdown_connections 2024-04-06T14:49:54.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.809+0000 7f987899e700 1 -- 172.21.15.32:0/692436735 wait complete. 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f987899e700 1 Processor -- start 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f987899e700 1 -- start start 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f987899e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98740fbb70 0x7f9874102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f987899e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 0x7f9874102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f987899e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9874103420 con 0x7f9874101ba0 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f987899e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9874103560 con 0x7f98740fbb70 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f9872ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 0x7f9874102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f9872ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 0x7f9874102e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51314/0 (socket says 172.21.15.32:51314) 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f9872ffd700 1 -- 172.21.15.32:0/3985482848 learned_addr learned my addr 172.21.15.32:0/3985482848 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f98737fe700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98740fbb70 0x7f9874102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f9872ffd700 1 -- 172.21.15.32:0/3985482848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98740fbb70 msgr2=0x7f9874102950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f9872ffd700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98740fbb70 0x7f9874102950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:54.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.810+0000 7f9872ffd700 1 -- 172.21.15.32:0/3985482848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9864005040 con 0x7f9874101ba0 2024-04-06T14:49:54.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.811+0000 7f9872ffd700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 0x7f9874102e90 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f985c002a50 tx=0x7f985c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:54.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.811+0000 7f986bfff700 1 -- 172.21.15.32:0/3985482848 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f985c007cc0 con 0x7f9874101ba0 2024-04-06T14:49:54.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.811+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9874119bc0 con 0x7f9874101ba0 2024-04-06T14:49:54.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.811+0000 7f986bfff700 1 -- 172.21.15.32:0/3985482848 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f985c00f040 con 0x7f9874101ba0 2024-04-06T14:49:54.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.811+0000 7f986bfff700 1 -- 172.21.15.32:0/3985482848 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f985c009540 con 0x7f9874101ba0 2024-04-06T14:49:54.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.811+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987411a040 con 0x7f9874101ba0 2024-04-06T14:49:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.812+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98740fd200 con 0x7f9874101ba0 2024-04-06T14:49:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.812+0000 7f986bfff700 1 -- 172.21.15.32:0/3985482848 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f985c0096a0 con 0x7f9874101ba0 2024-04-06T14:49:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.813+0000 7f986bfff700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f986006cc20 0x7f986006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.813+0000 7f986bfff700 1 -- 172.21.15.32:0/3985482848 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f985c08beb0 con 0x7f9874101ba0 2024-04-06T14:49:54.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.813+0000 7f98737fe700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f986006cc20 0x7f986006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:54.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.813+0000 7f98737fe700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f986006cc20 0x7f986006f0d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9864005010 tx=0x7f9864009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:54.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:54.817+0000 7f986bfff700 1 -- 172.21.15.32:0/3985482848 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f985c055e30 con 0x7f9874101ba0 2024-04-06T14:49:54.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:54 smithi086 ceph-mon[110975]: from='client.35626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:54.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:54 smithi032 ceph-mon[147414]: from='client.35626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.066+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f987405f5f0 con 0x7f9874101ba0 2024-04-06T14:49:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.067+0000 7f986bfff700 1 -- 172.21.15.32:0/3985482848 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f985c059450 con 0x7f9874101ba0 2024-04-06T14:49:55.056 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:49:55.056 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:49:55.056 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:49:55.056 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:49:55.057 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:49:55.057 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:49:55.057 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:49:55.057 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:49:55.057 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:49:55.057 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:49:55.057 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:49:55.057 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:49:55.058 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:49:55.058 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:49:55.058 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:49:55.058 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:49:55.058 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:49:55.058 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:49:55.058 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:49:55.058 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:49:55.059 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:49:55.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:49:55.059 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:49:55.059 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:49:55.059 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:49:55.060 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:49:55.060 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:49:55.060 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:49:55.060 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:49:55.061 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:49:55.061 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:49:55.061 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:49:55.061 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:49:55.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:49:55.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:49:55.061 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:49:55.062 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:49:55.062 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:49:55.062 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:49:55.062 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:49:55.062 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:49:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f986006cc20 msgr2=0x7f986006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f986006cc20 0x7f986006f0d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9864005010 tx=0x7f9864009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 msgr2=0x7f9874102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 0x7f9874102e90 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f985c002a50 tx=0x7f985c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 shutdown_connections 2024-04-06T14:49:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f986006cc20 0x7f986006f0d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98740fbb70 0x7f9874102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 --2- 172.21.15.32:0/3985482848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9874101ba0 0x7f9874102e90 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 >> 172.21.15.32:0/3985482848 conn(0x7f9874071b70 msgr2=0x7f98740fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 shutdown_connections 2024-04-06T14:49:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.069+0000 7f987899e700 1 -- 172.21.15.32:0/3985482848 wait complete. 2024-04-06T14:49:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:49:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 -- 172.21.15.32:0/3442899679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 msgr2=0x7f0f640fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 --2- 172.21.15.32:0/3442899679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 0x7f0f640fce10 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0f54004940 tx=0x7f0f54009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 -- 172.21.15.32:0/3442899679 shutdown_connections 2024-04-06T14:49:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 --2- 172.21.15.32:0/3442899679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 0x7f0f640fce10 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 --2- 172.21.15.32:0/3442899679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f64102e00 0x7f0f641031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 -- 172.21.15.32:0/3442899679 >> 172.21.15.32:0/3442899679 conn(0x7f0f640f89b0 msgr2=0x7f0f640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 -- 172.21.15.32:0/3442899679 shutdown_connections 2024-04-06T14:49:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 -- 172.21.15.32:0/3442899679 wait complete. 2024-04-06T14:49:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 Processor -- start 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.173+0000 7f0f69660700 1 -- start start 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f69660700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f64102e00 0x7f0f64113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f69660700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 0x7f0f64113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f69660700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f641143b0 con 0x7f0f641037a0 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f69660700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6410d9b0 con 0x7f0f64102e00 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f627fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 0x7f0f64113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f627fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 0x7f0f64113d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51338/0 (socket says 172.21.15.32:51338) 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f627fc700 1 -- 172.21.15.32:0/1984946914 learned_addr learned my addr 172.21.15.32:0/1984946914 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f62ffd700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f64102e00 0x7f0f64113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f627fc700 1 -- 172.21.15.32:0/1984946914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f64102e00 msgr2=0x7f0f64113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f627fc700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f64102e00 0x7f0f64113820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f627fc700 1 -- 172.21.15.32:0/1984946914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f54005040 con 0x7f0f641037a0 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f627fc700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 0x7f0f64113d60 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0f54008d50 tx=0x7f0f54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f4b7fe700 1 -- 172.21.15.32:0/1984946914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f5400e030 con 0x7f0f641037a0 2024-04-06T14:49:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6410dc30 con 0x7f0f641037a0 2024-04-06T14:49:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f4b7fe700 1 -- 172.21.15.32:0/1984946914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f540093b0 con 0x7f0f641037a0 2024-04-06T14:49:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f4b7fe700 1 -- 172.21.15.32:0/1984946914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f5401cbb0 con 0x7f0f641037a0 2024-04-06T14:49:55.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.174+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6410e1f0 con 0x7f0f641037a0 2024-04-06T14:49:55.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.176+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f640fe550 con 0x7f0f641037a0 2024-04-06T14:49:55.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.176+0000 7f0f4b7fe700 1 -- 172.21.15.32:0/1984946914 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0f54013930 con 0x7f0f641037a0 2024-04-06T14:49:55.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.176+0000 7f0f4b7fe700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0f4c06cc20 0x7f0f4c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:55.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.177+0000 7f0f4b7fe700 1 -- 172.21.15.32:0/1984946914 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0f5408d870 con 0x7f0f641037a0 2024-04-06T14:49:55.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.177+0000 7f0f62ffd700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0f4c06cc20 0x7f0f4c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:55.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.177+0000 7f0f62ffd700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0f4c06cc20 0x7f0f4c06f0d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009b80 tx=0x7f0f5800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:55.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.181+0000 7f0f4b7fe700 1 -- 172.21.15.32:0/1984946914 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0f5405b2c0 con 0x7f0f641037a0 2024-04-06T14:49:55.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.401+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f641134f0 con 0x7f0f4c06cc20 2024-04-06T14:49:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.402+0000 7f0f4b7fe700 1 -- 172.21.15.32:0/1984946914 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0f641134f0 con 0x7f0f4c06cc20 2024-04-06T14:49:55.391 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:49:55.391 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:49:55.391 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:49:55.391 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:49:55.391 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:49:55.391 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:49:55.392 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:49:55.392 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:49:55.392 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:49:55.392 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:49:55.392 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:49:55.392 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:49:55.393 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:49:55.393 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:49:55.393 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:49:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0f4c06cc20 msgr2=0x7f0f4c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0f4c06cc20 0x7f0f4c06f0d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009b80 tx=0x7f0f5800b040 comp rx=0 tx=0).stop 2024-04-06T14:49:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 msgr2=0x7f0f64113d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 0x7f0f64113d60 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0f54008d50 tx=0x7f0f54009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 shutdown_connections 2024-04-06T14:49:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0f4c06cc20 0x7f0f4c06f0d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f64102e00 0x7f0f64113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 --2- 172.21.15.32:0/1984946914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f641037a0 0x7f0f64113d60 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 >> 172.21.15.32:0/1984946914 conn(0x7f0f640f89b0 msgr2=0x7f0f640f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 shutdown_connections 2024-04-06T14:49:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.404+0000 7f0f69660700 1 -- 172.21.15.32:0/1984946914 wait complete. 2024-04-06T14:49:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 -- 172.21.15.32:0/3052193070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 msgr2=0x7f65d00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 --2- 172.21.15.32:0/3052193070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 0x7f65d00fbd80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f65c0004940 tx=0x7f65c0009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 -- 172.21.15.32:0/3052193070 shutdown_connections 2024-04-06T14:49:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 --2- 172.21.15.32:0/3052193070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 0x7f65d00fbd80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 --2- 172.21.15.32:0/3052193070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d0101d70 0x7f65d0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 -- 172.21.15.32:0/3052193070 >> 172.21.15.32:0/3052193070 conn(0x7f65d0072bc0 msgr2=0x7f65d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 -- 172.21.15.32:0/3052193070 shutdown_connections 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 -- 172.21.15.32:0/3052193070 wait complete. 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 Processor -- start 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 -- start start 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d0101d70 0x7f65d010eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.508+0000 7f65d73d3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 0x7f65d010f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d73d3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d010f650 con 0x7f65d0102710 2024-04-06T14:49:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d73d3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d010f790 con 0x7f65d0101d70 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d496e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 0x7f65d010f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d496e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 0x7f65d010f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51342/0 (socket says 172.21.15.32:51342) 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d496e700 1 -- 172.21.15.32:0/1604468925 learned_addr learned my addr 172.21.15.32:0/1604468925 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d516f700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d0101d70 0x7f65d010eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d496e700 1 -- 172.21.15.32:0/1604468925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d0101d70 msgr2=0x7f65d010eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d496e700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d0101d70 0x7f65d010eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d496e700 1 -- 172.21.15.32:0/1604468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c0005040 con 0x7f65d0102710 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d496e700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 0x7f65d010f000 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009380 tx=0x7f65c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65c5ffb700 1 -- 172.21.15.32:0/1604468925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c000e040 con 0x7f65d0102710 2024-04-06T14:49:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65c5ffb700 1 -- 172.21.15.32:0/1604468925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65c0013420 con 0x7f65d0102710 2024-04-06T14:49:55.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65c5ffb700 1 -- 172.21.15.32:0/1604468925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c001c630 con 0x7f65d0102710 2024-04-06T14:49:55.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.509+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d0117b30 con 0x7f65d0102710 2024-04-06T14:49:55.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.510+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d0118010 con 0x7f65d0102710 2024-04-06T14:49:55.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.511+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d0104240 con 0x7f65d0102710 2024-04-06T14:49:55.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.511+0000 7f65c5ffb700 1 -- 172.21.15.32:0/1604468925 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f65c00139b0 con 0x7f65d0102710 2024-04-06T14:49:55.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.511+0000 7f65c5ffb700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f65bc06cc20 0x7f65bc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:49:55.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.512+0000 7f65c5ffb700 1 -- 172.21.15.32:0/1604468925 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f65c008d1b0 con 0x7f65d0102710 2024-04-06T14:49:55.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.512+0000 7f65d516f700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f65bc06cc20 0x7f65bc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:49:55.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.512+0000 7f65d516f700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f65bc06cc20 0x7f65bc06f0d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f65cc004f60 tx=0x7f65cc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:49:55.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.516+0000 7f65c5ffb700 1 -- 172.21.15.32:0/1604468925 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f65c00571f0 con 0x7f65d0102710 2024-04-06T14:49:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.806+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f65d0065db0 con 0x7f65d0102710 2024-04-06T14:49:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.806+0000 7f65c5ffb700 1 -- 172.21.15.32:0/1604468925 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f65c005a810 con 0x7f65d0102710 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:49:55.795 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:49:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f65bc06cc20 msgr2=0x7f65bc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f65bc06cc20 0x7f65bc06f0d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f65cc004f60 tx=0x7f65cc00b040 comp rx=0 tx=0).stop 2024-04-06T14:49:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 msgr2=0x7f65d010f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:49:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 0x7f65d010f000 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009380 tx=0x7f65c0009e30 comp rx=0 tx=0).stop 2024-04-06T14:49:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 shutdown_connections 2024-04-06T14:49:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f65bc06cc20 0x7f65bc06f0d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d0101d70 0x7f65d010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 --2- 172.21.15.32:0/1604468925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d0102710 0x7f65d010f000 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:49:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 >> 172.21.15.32:0/1604468925 conn(0x7f65d0072bc0 msgr2=0x7f65d00fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:49:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 shutdown_connections 2024-04-06T14:49:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:49:55.808+0000 7f65d73d3700 1 -- 172.21.15.32:0/1604468925 wait complete. 2024-04-06T14:49:55.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:55 smithi086 ceph-mon[110975]: from='client.35630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:55.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:55 smithi086 ceph-mon[110975]: pgmap v894: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:55.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:55 smithi086 ceph-mon[110975]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:55.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:55 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3636574654' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:49:55.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:55 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3985482848' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:49:55.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:55 smithi032 ceph-mon[147414]: from='client.35630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:55.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:55 smithi032 ceph-mon[147414]: pgmap v894: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:55.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:55 smithi032 ceph-mon[147414]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:55.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:55 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3636574654' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:49:55.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:55 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3985482848' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:49:56.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:56 smithi086 ceph-mon[110975]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:56.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:56 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1604468925' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:49:56.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:56 smithi032 ceph-mon[147414]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:49:56.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:56 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1604468925' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:49:57.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:57 smithi086 ceph-mon[110975]: pgmap v895: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:57.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:57 smithi032 ceph-mon[147414]: pgmap v895: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:59.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:49:59 smithi086 ceph-mon[110975]: pgmap v896: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:49:59.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:49:59 smithi032 ceph-mon[147414]: pgmap v896: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : "__main__", mod_spec) 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:50:00.469 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:50:00.470 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T14:49:59.999+0000 7fa223c88700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-06T14:50:00.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:50:00.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:50:00.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: /usr/bin/podman: timeout after 895 seconds 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: Traceback (most recent call last): 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: "__main__", mod_spec) 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: exec(code, run_globals) 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:50:00.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:50:00.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:50:00.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:50:00.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:50:00.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:50:00.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: fs cephfs is degraded 2024-04-06T14:50:00.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:50:00.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:50:00.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:50:00.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:00 smithi086 ceph-mon[110975]: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:50:00.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: /usr/bin/podman: timeout after 895 seconds 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: Traceback (most recent call last): 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: "__main__", mod_spec) 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: exec(code, run_globals) 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:50:00.969 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: fs cephfs is degraded 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:50:00.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:00 smithi032 ceph-mon[147414]: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:50:01.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:01 smithi086 ceph-mon[110975]: pgmap v897: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:01.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:01 smithi032 ceph-mon[147414]: pgmap v897: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:03.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:03 smithi032 ceph-mon[147414]: pgmap v898: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:03.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:03 smithi086 ceph-mon[110975]: pgmap v898: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:05.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:05 smithi086 ceph-mon[110975]: pgmap v899: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:05.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:05 smithi032 ceph-mon[147414]: pgmap v899: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:07.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:07 smithi086 ceph-mon[110975]: pgmap v900: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:07.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:07 smithi032 ceph-mon[147414]: pgmap v900: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:08.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:50:08.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:50:09.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:09 smithi086 ceph-mon[110975]: pgmap v901: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:09.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:09 smithi032 ceph-mon[147414]: pgmap v901: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:11.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:11 smithi086 ceph-mon[110975]: pgmap v902: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:11.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:11 smithi032 ceph-mon[147414]: pgmap v902: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:13.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:13 smithi032 ceph-mon[147414]: pgmap v903: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:13.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:13 smithi086 ceph-mon[110975]: pgmap v903: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:15.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:15 smithi086 ceph-mon[110975]: pgmap v904: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:15.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:15 smithi032 ceph-mon[147414]: pgmap v904: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:17.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:17 smithi086 ceph-mon[110975]: pgmap v905: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:17.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:17 smithi032 ceph-mon[147414]: pgmap v905: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:19.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:19 smithi086 ceph-mon[110975]: pgmap v906: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:19.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:19 smithi032 ceph-mon[147414]: pgmap v906: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:21.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:21 smithi086 ceph-mon[110975]: pgmap v907: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:21.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:21 smithi032 ceph-mon[147414]: pgmap v907: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:23.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:23 smithi032 ceph-mon[147414]: pgmap v908: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:23.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:50:23.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:23 smithi086 ceph-mon[110975]: pgmap v908: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:23.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:50:25.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:25 smithi086 ceph-mon[110975]: pgmap v909: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.926+0000 7fe199329700 1 -- 172.21.15.32:0/1400242026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 msgr2=0x7fe1940fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.926+0000 7fe199329700 1 --2- 172.21.15.32:0/1400242026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 0x7fe1940fbd90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe184004940 tx=0x7fe184009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.926+0000 7fe199329700 1 -- 172.21.15.32:0/1400242026 shutdown_connections 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.926+0000 7fe199329700 1 --2- 172.21.15.32:0/1400242026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 0x7fe1940fbd90 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.926+0000 7fe199329700 1 --2- 172.21.15.32:0/1400242026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe194101d80 0x7fe194102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.926+0000 7fe199329700 1 -- 172.21.15.32:0/1400242026 >> 172.21.15.32:0/1400242026 conn(0x7fe1940f78f0 msgr2=0x7fe1940f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.926+0000 7fe199329700 1 -- 172.21.15.32:0/1400242026 shutdown_connections 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.926+0000 7fe199329700 1 -- 172.21.15.32:0/1400242026 wait complete. 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe199329700 1 Processor -- start 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe199329700 1 -- start start 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe199329700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe194101d80 0x7fe194102be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe199329700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 0x7fe194103120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe199329700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe194104b30 con 0x7fe194102720 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe199329700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe194103660 con 0x7fe194101d80 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe1927fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 0x7fe194103120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe1927fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 0x7fe194103120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50890/0 (socket says 172.21.15.32:50890) 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe1927fc700 1 -- 172.21.15.32:0/1397373785 learned_addr learned my addr 172.21.15.32:0/1397373785 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe192ffd700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe194101d80 0x7fe194102be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe1927fc700 1 -- 172.21.15.32:0/1397373785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe194101d80 msgr2=0x7fe194102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe1927fc700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe194101d80 0x7fe194102be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.927+0000 7fe1927fc700 1 -- 172.21.15.32:0/1397373785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe184005040 con 0x7fe194102720 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.928+0000 7fe1927fc700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 0x7fe194103120 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe184009380 tx=0x7fe184009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.928+0000 7fe17b7fe700 1 -- 172.21.15.32:0/1397373785 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18400e040 con 0x7fe194102720 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.928+0000 7fe17b7fe700 1 -- 172.21.15.32:0/1397373785 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe184013420 con 0x7fe194102720 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.928+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1941038e0 con 0x7fe194102720 2024-04-06T14:50:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.928+0000 7fe17b7fe700 1 -- 172.21.15.32:0/1397373785 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18401c630 con 0x7fe194102720 2024-04-06T14:50:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.928+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19411a010 con 0x7fe194102720 2024-04-06T14:50:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.929+0000 7fe17b7fe700 1 -- 172.21.15.32:0/1397373785 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe1840139e0 con 0x7fe194102720 2024-04-06T14:50:25.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.930+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1940fd4d0 con 0x7fe194102720 2024-04-06T14:50:25.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.930+0000 7fe17b7fe700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe17c06cc20 0x7fe17c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:25.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.930+0000 7fe17b7fe700 1 -- 172.21.15.32:0/1397373785 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe18408d220 con 0x7fe194102720 2024-04-06T14:50:25.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.930+0000 7fe192ffd700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe17c06cc20 0x7fe17c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:25.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.930+0000 7fe192ffd700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe17c06cc20 0x7fe17c06f0d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe188004990 tx=0x7fe188009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:25.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:25.935+0000 7fe17b7fe700 1 -- 172.21.15.32:0/1397373785 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe184057260 con 0x7fe194102720 2024-04-06T14:50:25.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:25 smithi032 ceph-mon[147414]: pgmap v909: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.158+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe194105d30 con 0x7fe17c06cc20 2024-04-06T14:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.159+0000 7fe17b7fe700 1 -- 172.21.15.32:0/1397373785 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe194105d30 con 0x7fe17c06cc20 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe17c06cc20 msgr2=0x7fe17c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe17c06cc20 0x7fe17c06f0d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe188004990 tx=0x7fe188009ee0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 msgr2=0x7fe194103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 0x7fe194103120 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe184009380 tx=0x7fe184009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 shutdown_connections 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe17c06cc20 0x7fe17c06f0d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe194101d80 0x7fe194102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 --2- 172.21.15.32:0/1397373785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe194102720 0x7fe194103120 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 >> 172.21.15.32:0/1397373785 conn(0x7fe1940f78f0 msgr2=0x7fe1940f96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.161+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 shutdown_connections 2024-04-06T14:50:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.162+0000 7fe199329700 1 -- 172.21.15.32:0/1397373785 wait complete. 2024-04-06T14:50:26.160 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:50:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.265+0000 7f82abd17700 1 -- 172.21.15.32:0/3357326762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 msgr2=0x7f82a41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.265+0000 7f82abd17700 1 --2- 172.21.15.32:0/3357326762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 0x7f82a41020f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f82a0004940 tx=0x7f82a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.265+0000 7f82abd17700 1 -- 172.21.15.32:0/3357326762 shutdown_connections 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.265+0000 7f82abd17700 1 --2- 172.21.15.32:0/3357326762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41026c0 0x7f82a40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.265+0000 7f82abd17700 1 --2- 172.21.15.32:0/3357326762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 0x7f82a41020f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.265+0000 7f82abd17700 1 -- 172.21.15.32:0/3357326762 >> 172.21.15.32:0/3357326762 conn(0x7f82a40f78d0 msgr2=0x7f82a40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82abd17700 1 -- 172.21.15.32:0/3357326762 shutdown_connections 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82abd17700 1 -- 172.21.15.32:0/3357326762 wait complete. 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82abd17700 1 Processor -- start 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82abd17700 1 -- start start 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82abd17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 0x7f82a4102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82abd17700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41026c0 0x7f82a41030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82abd17700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a4103600 con 0x7f82a4101d20 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82abd17700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a4103740 con 0x7f82a41026c0 2024-04-06T14:50:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82a9ab3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 0x7f82a4102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82a9ab3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 0x7f82a4102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50910/0 (socket says 172.21.15.32:50910) 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.266+0000 7f82a9ab3700 1 -- 172.21.15.32:0/1391040644 learned_addr learned my addr 172.21.15.32:0/1391040644 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f82a92b2700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41026c0 0x7f82a41030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f82a9ab3700 1 -- 172.21.15.32:0/1391040644 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41026c0 msgr2=0x7f82a41030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f82a9ab3700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41026c0 0x7f82a41030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f82a9ab3700 1 -- 172.21.15.32:0/1391040644 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a0005040 con 0x7f82a4101d20 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f82a9ab3700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 0x7f82a4102b80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009360 tx=0x7f82a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f829a7fc700 1 -- 172.21.15.32:0/1391040644 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82a000e040 con 0x7f82a4101d20 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f829a7fc700 1 -- 172.21.15.32:0/1391040644 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82a0013420 con 0x7f82a4101d20 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a4103990 con 0x7f82a4101d20 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f829a7fc700 1 -- 172.21.15.32:0/1391040644 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82a001c5d0 con 0x7f82a4101d20 2024-04-06T14:50:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.267+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a411a220 con 0x7f82a4101d20 2024-04-06T14:50:26.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.268+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a40fd470 con 0x7f82a4101d20 2024-04-06T14:50:26.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.269+0000 7f829a7fc700 1 -- 172.21.15.32:0/1391040644 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f82a0013590 con 0x7f82a4101d20 2024-04-06T14:50:26.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.269+0000 7f829a7fc700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f829006cc30 0x7f829006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:26.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.269+0000 7f829a7fc700 1 -- 172.21.15.32:0/1391040644 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f82a008d160 con 0x7f82a4101d20 2024-04-06T14:50:26.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.269+0000 7f82a92b2700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f829006cc30 0x7f829006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:26.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.270+0000 7f82a92b2700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f829006cc30 0x7f829006f0e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f82a4104470 tx=0x7f829400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.274+0000 7f829a7fc700 1 -- 172.21.15.32:0/1391040644 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f82a0057120 con 0x7f82a4101d20 2024-04-06T14:50:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.494+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f82a4104220 con 0x7f829006cc30 2024-04-06T14:50:26.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.495+0000 7f829a7fc700 1 -- 172.21.15.32:0/1391040644 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f82a4104220 con 0x7f829006cc30 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f829006cc30 msgr2=0x7f829006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f829006cc30 0x7f829006f0e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f82a4104470 tx=0x7f829400b040 comp rx=0 tx=0).stop 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 msgr2=0x7f82a4102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 0x7f82a4102b80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009360 tx=0x7f82a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 shutdown_connections 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f829006cc30 0x7f829006f0e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a4101d20 0x7f82a4102b80 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 --2- 172.21.15.32:0/1391040644 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41026c0 0x7f82a41030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 >> 172.21.15.32:0/1391040644 conn(0x7f82a40f78d0 msgr2=0x7f82a40f89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 shutdown_connections 2024-04-06T14:50:26.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.497+0000 7f82abd17700 1 -- 172.21.15.32:0/1391040644 wait complete. 2024-04-06T14:50:26.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.600+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/2626734868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 msgr2=0x7f6c780fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.600+0000 7f6c7fb9f700 1 --2- 172.21.15.32:0/2626734868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 0x7f6c780fbd90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6c68004940 tx=0x7f6c68009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.600+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/2626734868 shutdown_connections 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.600+0000 7f6c7fb9f700 1 --2- 172.21.15.32:0/2626734868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 0x7f6c780fbd90 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.600+0000 7f6c7fb9f700 1 --2- 172.21.15.32:0/2626734868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c78101d80 0x7f6c78102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.600+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/2626734868 >> 172.21.15.32:0/2626734868 conn(0x7f6c780f78f0 msgr2=0x7f6c780f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/2626734868 shutdown_connections 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/2626734868 wait complete. 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7fb9f700 1 Processor -- start 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7fb9f700 1 -- start start 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7fb9f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c78101d80 0x7f6c78113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7fb9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 0x7f6c78113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7fb9f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78114590 con 0x7f6c78102720 2024-04-06T14:50:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7fb9f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7810db00 con 0x7f6c78101d80 2024-04-06T14:50:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7d13a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 0x7f6c78113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7d13a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 0x7f6c78113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50940/0 (socket says 172.21.15.32:50940) 2024-04-06T14:50:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.601+0000 7f6c7d13a700 1 -- 172.21.15.32:0/3092971886 learned_addr learned my addr 172.21.15.32:0/3092971886 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c7d93b700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c78101d80 0x7f6c78113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c7d13a700 1 -- 172.21.15.32:0/3092971886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c78101d80 msgr2=0x7f6c78113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c7d13a700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c78101d80 0x7f6c78113970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c7d13a700 1 -- 172.21.15.32:0/3092971886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c68005040 con 0x7f6c78102720 2024-04-06T14:50:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c7d93b700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c78101d80 0x7f6c78113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:50:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c7d13a700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 0x7f6c78113eb0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009390 tx=0x7f6c68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c6e7fc700 1 -- 172.21.15.32:0/3092971886 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c6800e040 con 0x7f6c78102720 2024-04-06T14:50:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c6e7fc700 1 -- 172.21.15.32:0/3092971886 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c68013420 con 0x7f6c78102720 2024-04-06T14:50:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7810dd80 con 0x7f6c78102720 2024-04-06T14:50:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c6e7fc700 1 -- 172.21.15.32:0/3092971886 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c6801c630 con 0x7f6c78102720 2024-04-06T14:50:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.602+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7810e2c0 con 0x7f6c78102720 2024-04-06T14:50:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.603+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c780fd4d0 con 0x7f6c78102720 2024-04-06T14:50:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.604+0000 7f6c6e7fc700 1 -- 172.21.15.32:0/3092971886 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6c68013a00 con 0x7f6c78102720 2024-04-06T14:50:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.604+0000 7f6c6e7fc700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6c6406cc20 0x7f6c6406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.604+0000 7f6c6e7fc700 1 -- 172.21.15.32:0/3092971886 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6c6808d320 con 0x7f6c78102720 2024-04-06T14:50:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.604+0000 7f6c7d93b700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6c6406cc20 0x7f6c6406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.605+0000 7f6c7d93b700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6c6406cc20 0x7f6c6406f0d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009b10 tx=0x7f6c74008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.609+0000 7f6c6e7fc700 1 -- 172.21.15.32:0/3092971886 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6c6805adf0 con 0x7f6c78102720 2024-04-06T14:50:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.823+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c78108e00 con 0x7f6c6406cc20 2024-04-06T14:50:26.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.832+0000 7f6c6e7fc700 1 -- 172.21.15.32:0/3092971886 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6c78108e00 con 0x7f6c6406cc20 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (30m) 8m ago 42m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (28m) 8m ago 42m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (28m) 8m ago 41m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (30m) 8m ago 41m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (24m) 8m ago 36m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (24m) 8m ago 36m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (23m) 8m ago 36m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (8m) 8m ago 36m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (32m) 8m ago 43m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:50:26.822 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (31m) 8m ago 41m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (29m) 8m ago 43m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (28m) 8m ago 41m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (31m) 8m ago 42m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (31m) 8m ago 41m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (27m) 8m ago 40m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (27m) 8m ago 39m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (26m) 8m ago 39m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (25m) 8m ago 38m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (25m) 8m ago 38m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (24m) 8m ago 37m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:50:26.823 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (31m) 8m ago 41m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6c6406cc20 msgr2=0x7f6c6406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6c6406cc20 0x7f6c6406f0d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009b10 tx=0x7f6c74008040 comp rx=0 tx=0).stop 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 msgr2=0x7f6c78113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 0x7f6c78113eb0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009390 tx=0x7f6c68009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 shutdown_connections 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6c6406cc20 0x7f6c6406f0d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c78101d80 0x7f6c78113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 --2- 172.21.15.32:0/3092971886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78102720 0x7f6c78113eb0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 >> 172.21.15.32:0/3092971886 conn(0x7f6c780f78f0 msgr2=0x7f6c78066650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 shutdown_connections 2024-04-06T14:50:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.835+0000 7f6c7fb9f700 1 -- 172.21.15.32:0/3092971886 wait complete. 2024-04-06T14:50:26.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.937+0000 7fbebba8d700 1 -- 172.21.15.32:0/3249009996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 msgr2=0x7fbeb4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.937+0000 7fbebba8d700 1 --2- 172.21.15.32:0/3249009996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 0x7fbeb4102140 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009ce0 tx=0x7fbeb0004d10 comp rx=0 tx=0).stop 2024-04-06T14:50:26.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.937+0000 7fbebba8d700 1 -- 172.21.15.32:0/3249009996 shutdown_connections 2024-04-06T14:50:26.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.937+0000 7fbebba8d700 1 --2- 172.21.15.32:0/3249009996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbeb4102710 0x7fbeb40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.937+0000 7fbebba8d700 1 --2- 172.21.15.32:0/3249009996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 0x7fbeb4102140 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.937+0000 7fbebba8d700 1 -- 172.21.15.32:0/3249009996 >> 172.21.15.32:0/3249009996 conn(0x7fbeb4072bc0 msgr2=0x7fbeb4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:26.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.937+0000 7fbebba8d700 1 -- 172.21.15.32:0/3249009996 shutdown_connections 2024-04-06T14:50:26.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.937+0000 7fbebba8d700 1 -- 172.21.15.32:0/3249009996 wait complete. 2024-04-06T14:50:26.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbebba8d700 1 Processor -- start 2024-04-06T14:50:26.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbebba8d700 1 -- start start 2024-04-06T14:50:26.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbebba8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 0x7fbeb4113c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:26.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbebba8d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbeb4102710 0x7fbeb4114160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:26.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbebba8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb41147b0 con 0x7fbeb4101d70 2024-04-06T14:50:26.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbebba8d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb410dba0 con 0x7fbeb4102710 2024-04-06T14:50:26.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbeb9829700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 0x7fbeb4113c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbeb9829700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 0x7fbeb4113c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50960/0 (socket says 172.21.15.32:50960) 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbeb9829700 1 -- 172.21.15.32:0/663211003 learned_addr learned my addr 172.21.15.32:0/663211003 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.938+0000 7fbeb9028700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbeb4102710 0x7fbeb4114160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.939+0000 7fbeb9829700 1 -- 172.21.15.32:0/663211003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbeb4102710 msgr2=0x7fbeb4114160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.939+0000 7fbeb9829700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbeb4102710 0x7fbeb4114160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.939+0000 7fbeb9829700 1 -- 172.21.15.32:0/663211003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb0009940 con 0x7fbeb4101d70 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.939+0000 7fbeb9829700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 0x7fbeb4113c20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0005010 tx=0x7fbeb0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.939+0000 7fbeaa7fc700 1 -- 172.21.15.32:0/663211003 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb000e040 con 0x7fbeb4101d70 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.939+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb410de20 con 0x7fbeb4101d70 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.939+0000 7fbeaa7fc700 1 -- 172.21.15.32:0/663211003 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbeb0007d10 con 0x7fbeb4101d70 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.939+0000 7fbeaa7fc700 1 -- 172.21.15.32:0/663211003 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb0012650 con 0x7fbeb4101d70 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.939+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb410e3e0 con 0x7fbeb4101d70 2024-04-06T14:50:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.940+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb40fd4c0 con 0x7fbeb4101d70 2024-04-06T14:50:26.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.940+0000 7fbeaa7fc700 1 -- 172.21.15.32:0/663211003 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbeb001c440 con 0x7fbeb4101d70 2024-04-06T14:50:26.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.941+0000 7fbeaa7fc700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbea006cc30 0x7fbea006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:26.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.941+0000 7fbeaa7fc700 1 -- 172.21.15.32:0/663211003 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbeb008dd20 con 0x7fbeb4101d70 2024-04-06T14:50:26.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.941+0000 7fbeb9028700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbea006cc30 0x7fbea006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:26.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.941+0000 7fbeb9028700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbea006cc30 0x7fbea006f0e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbeb410f450 tx=0x7fbea400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:26.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:26.945+0000 7fbeaa7fc700 1 -- 172.21.15.32:0/663211003 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbeb005b770 con 0x7fbeb4101d70 2024-04-06T14:50:27.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.234+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbeb404bf60 con 0x7fbeb4101d70 2024-04-06T14:50:27.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.234+0000 7fbeaa7fc700 1 -- 172.21.15.32:0/663211003 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fbeb0004e10 con 0x7fbeb4101d70 2024-04-06T14:50:27.227 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:50:27.228 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:50:27.228 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:50:27.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:50:27.228 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:50:27.228 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:50:27.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:50:27.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:50:27.229 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:50:27.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:50:27.229 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:50:27.229 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:50:27.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:50:27.230 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:50:27.230 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:50:27.230 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:50:27.230 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:50:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.236+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbea006cc30 msgr2=0x7fbea006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.236+0000 7fbebba8d700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbea006cc30 0x7fbea006f0e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbeb410f450 tx=0x7fbea400b040 comp rx=0 tx=0).stop 2024-04-06T14:50:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.236+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 msgr2=0x7fbeb4113c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.237+0000 7fbebba8d700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 0x7fbeb4113c20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0005010 tx=0x7fbeb0004d10 comp rx=0 tx=0).stop 2024-04-06T14:50:27.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.237+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 shutdown_connections 2024-04-06T14:50:27.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.237+0000 7fbebba8d700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbea006cc30 0x7fbea006f0e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.237+0000 7fbebba8d700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb4101d70 0x7fbeb4113c20 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.237+0000 7fbebba8d700 1 --2- 172.21.15.32:0/663211003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbeb4102710 0x7fbeb4114160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.237+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 >> 172.21.15.32:0/663211003 conn(0x7fbeb4072bc0 msgr2=0x7fbeb40659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:27.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.237+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 shutdown_connections 2024-04-06T14:50:27.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.237+0000 7fbebba8d700 1 -- 172.21.15.32:0/663211003 wait complete. 2024-04-06T14:50:27.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.339+0000 7f3c97bbc700 1 -- 172.21.15.32:0/2768915715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90102710 msgr2=0x7f3c900fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.339+0000 7f3c97bbc700 1 --2- 172.21.15.32:0/2768915715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90102710 0x7f3c900fbd80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3c80004940 tx=0x7f3c80009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:27.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 -- 172.21.15.32:0/2768915715 shutdown_connections 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 --2- 172.21.15.32:0/2768915715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90102710 0x7f3c900fbd80 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 --2- 172.21.15.32:0/2768915715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c90101d70 0x7f3c90102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 -- 172.21.15.32:0/2768915715 >> 172.21.15.32:0/2768915715 conn(0x7f3c90072bc0 msgr2=0x7f3c90072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 -- 172.21.15.32:0/2768915715 shutdown_connections 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 -- 172.21.15.32:0/2768915715 wait complete. 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 Processor -- start 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 -- start start 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90101d70 0x7f3c90113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c90102710 0x7f3c90114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90114700 con 0x7f3c90101d70 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.340+0000 7f3c97bbc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9010dab0 con 0x7f3c90102710 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c95958700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90101d70 0x7f3c90113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c95958700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90101d70 0x7f3c90113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50986/0 (socket says 172.21.15.32:50986) 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c95958700 1 -- 172.21.15.32:0/4023913304 learned_addr learned my addr 172.21.15.32:0/4023913304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c95157700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c90102710 0x7f3c90114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c95958700 1 -- 172.21.15.32:0/4023913304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c90102710 msgr2=0x7f3c90114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c95958700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c90102710 0x7f3c90114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c95958700 1 -- 172.21.15.32:0/4023913304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80005040 con 0x7f3c90101d70 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c95958700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90101d70 0x7f3c90113ae0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00b790 tx=0x7f3c8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c867fc700 1 -- 172.21.15.32:0/4023913304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c8c00f6f0 con 0x7f3c90101d70 2024-04-06T14:50:27.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c867fc700 1 -- 172.21.15.32:0/4023913304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c8c00fd00 con 0x7f3c90101d70 2024-04-06T14:50:27.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9010dd90 con 0x7f3c90101d70 2024-04-06T14:50:27.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c867fc700 1 -- 172.21.15.32:0/4023913304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c8c0103a0 con 0x7f3c90101d70 2024-04-06T14:50:27.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.341+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9010e320 con 0x7f3c90101d70 2024-04-06T14:50:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.342+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c900fd4c0 con 0x7f3c90101d70 2024-04-06T14:50:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.343+0000 7f3c867fc700 1 -- 172.21.15.32:0/4023913304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3c8c00f850 con 0x7f3c90101d70 2024-04-06T14:50:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.343+0000 7f3c867fc700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c7c06cc20 0x7f3c7c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.343+0000 7f3c867fc700 1 -- 172.21.15.32:0/4023913304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3c8c08be70 con 0x7f3c90101d70 2024-04-06T14:50:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.343+0000 7f3c95157700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c7c06cc20 0x7f3c7c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.344+0000 7f3c95157700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c7c06cc20 0x7f3c7c06f0d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3c9010ed40 tx=0x7f3c80009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:27.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.348+0000 7f3c867fc700 1 -- 172.21.15.32:0/4023913304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3c8c059940 con 0x7f3c90101d70 2024-04-06T14:50:27.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.594+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3c9004bed0 con 0x7f3c90101d70 2024-04-06T14:50:27.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.595+0000 7f3c867fc700 1 -- 172.21.15.32:0/4023913304 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f3c8c0594d0 con 0x7f3c90101d70 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:50:27.584 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:50:27.585 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:50:27.586 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:50:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c7c06cc20 msgr2=0x7f3c7c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c7c06cc20 0x7f3c7c06f0d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3c9010ed40 tx=0x7f3c80009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90101d70 msgr2=0x7f3c90113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90101d70 0x7f3c90113ae0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00b790 tx=0x7f3c8c009250 comp rx=0 tx=0).stop 2024-04-06T14:50:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 shutdown_connections 2024-04-06T14:50:27.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3c7c06cc20 0x7f3c7c06f0d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c90101d70 0x7f3c90113ae0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 --2- 172.21.15.32:0/4023913304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c90102710 0x7f3c90114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 >> 172.21.15.32:0/4023913304 conn(0x7f3c90072bc0 msgr2=0x7f3c90065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 shutdown_connections 2024-04-06T14:50:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.597+0000 7f3c97bbc700 1 -- 172.21.15.32:0/4023913304 wait complete. 2024-04-06T14:50:27.588 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:50:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.701+0000 7fa305f1e700 1 -- 172.21.15.32:0/3772684589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 msgr2=0x7fa3000fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.701+0000 7fa305f1e700 1 --2- 172.21.15.32:0/3772684589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 0x7fa3000fbd80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0004940 tx=0x7fa2f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.702+0000 7fa305f1e700 1 -- 172.21.15.32:0/3772684589 shutdown_connections 2024-04-06T14:50:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.702+0000 7fa305f1e700 1 --2- 172.21.15.32:0/3772684589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 0x7fa3000fbd80 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.702+0000 7fa305f1e700 1 --2- 172.21.15.32:0/3772684589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa300101d70 0x7fa300102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.702+0000 7fa305f1e700 1 -- 172.21.15.32:0/3772684589 >> 172.21.15.32:0/3772684589 conn(0x7fa300072bc0 msgr2=0x7fa300072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.702+0000 7fa305f1e700 1 -- 172.21.15.32:0/3772684589 shutdown_connections 2024-04-06T14:50:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.702+0000 7fa305f1e700 1 -- 172.21.15.32:0/3772684589 wait complete. 2024-04-06T14:50:27.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.702+0000 7fa305f1e700 1 Processor -- start 2024-04-06T14:50:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.702+0000 7fa305f1e700 1 -- start start 2024-04-06T14:50:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa305f1e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa300101d70 0x7fa300113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa305f1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 0x7fa300114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa305f1e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa300114650 con 0x7fa300102710 2024-04-06T14:50:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa305f1e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30010da40 con 0x7fa300101d70 2024-04-06T14:50:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2feffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 0x7fa300114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2feffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 0x7fa300114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51004/0 (socket says 172.21.15.32:51004) 2024-04-06T14:50:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2feffd700 1 -- 172.21.15.32:0/2362066590 learned_addr learned my addr 172.21.15.32:0/2362066590 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2feffd700 1 -- 172.21.15.32:0/2362066590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa300101d70 msgr2=0x7fa300113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2ff7fe700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa300101d70 0x7fa300113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2feffd700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa300101d70 0x7fa300113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2feffd700 1 -- 172.21.15.32:0/2362066590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f0005040 con 0x7fa300102710 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2ff7fe700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa300101d70 0x7fa300113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2feffd700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 0x7fa300114000 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009380 tx=0x7fa2f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2e7fff700 1 -- 172.21.15.32:0/2362066590 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2f000e040 con 0x7fa300102710 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30010dcc0 con 0x7fa300102710 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2e7fff700 1 -- 172.21.15.32:0/2362066590 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2f0013420 con 0x7fa300102710 2024-04-06T14:50:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa2e7fff700 1 -- 172.21.15.32:0/2362066590 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2f001c630 con 0x7fa300102710 2024-04-06T14:50:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.703+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30010e280 con 0x7fa300102710 2024-04-06T14:50:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.704+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3000fd4c0 con 0x7fa300102710 2024-04-06T14:50:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.705+0000 7fa2e7fff700 1 -- 172.21.15.32:0/2362066590 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa2f00139e0 con 0x7fa300102710 2024-04-06T14:50:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.705+0000 7fa2e7fff700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa2e806cc20 0x7fa2e806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.705+0000 7fa2e7fff700 1 -- 172.21.15.32:0/2362066590 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa2f008d220 con 0x7fa300102710 2024-04-06T14:50:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.705+0000 7fa2ff7fe700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa2e806cc20 0x7fa2e806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.706+0000 7fa2ff7fe700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa2e806cc20 0x7fa2e806f0d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4007fd0 tx=0x7fa2f4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:27.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.710+0000 7fa2e7fff700 1 -- 172.21.15.32:0/2362066590 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa2f005acf0 con 0x7fa300102710 2024-04-06T14:50:27.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:27 smithi086 ceph-mon[110975]: from='client.35654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:50:27.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:27 smithi086 ceph-mon[110975]: pgmap v910: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:27.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:27 smithi086 ceph-mon[110975]: from='client.35658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:50:27.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:27 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/663211003' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:50:27.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:27 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4023913304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:50:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.932+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa300108d10 con 0x7fa2e806cc20 2024-04-06T14:50:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.933+0000 7fa2e7fff700 1 -- 172.21.15.32:0/2362066590 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa300108d10 con 0x7fa2e806cc20 2024-04-06T14:50:27.922 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:50:27.922 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:50:27.922 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:50:27.922 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:50:27.923 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:50:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.935+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa2e806cc20 msgr2=0x7fa2e806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.935+0000 7fa305f1e700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa2e806cc20 0x7fa2e806f0d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4007fd0 tx=0x7fa2f4009040 comp rx=0 tx=0).stop 2024-04-06T14:50:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.935+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 msgr2=0x7fa300114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.935+0000 7fa305f1e700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 0x7fa300114000 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009380 tx=0x7fa2f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.935+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 shutdown_connections 2024-04-06T14:50:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.935+0000 7fa305f1e700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa2e806cc20 0x7fa2e806f0d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.935+0000 7fa305f1e700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa300101d70 0x7fa300113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.935+0000 7fa305f1e700 1 --2- 172.21.15.32:0/2362066590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa300102710 0x7fa300114000 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.935+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 >> 172.21.15.32:0/2362066590 conn(0x7fa300072bc0 msgr2=0x7fa3000659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.936+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 shutdown_connections 2024-04-06T14:50:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:27.936+0000 7fa305f1e700 1 -- 172.21.15.32:0/2362066590 wait complete. 2024-04-06T14:50:27.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:27 smithi032 ceph-mon[147414]: from='client.35654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:50:27.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:27 smithi032 ceph-mon[147414]: pgmap v910: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:27.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:27 smithi032 ceph-mon[147414]: from='client.35658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:50:27.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:27 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/663211003' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:50:27.972 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:27 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4023913304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:50:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.038+0000 7f810010d700 1 -- 172.21.15.32:0/3273807755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8102710 msgr2=0x7f80f80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.038+0000 7f810010d700 1 --2- 172.21.15.32:0/3273807755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8102710 0x7f80f80fbd80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f80ec004940 tx=0x7f80ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.038+0000 7f810010d700 1 -- 172.21.15.32:0/3273807755 shutdown_connections 2024-04-06T14:50:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.038+0000 7f810010d700 1 --2- 172.21.15.32:0/3273807755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8102710 0x7f80f80fbd80 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.038+0000 7f810010d700 1 --2- 172.21.15.32:0/3273807755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80f8101d70 0x7f80f8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:28.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.038+0000 7f810010d700 1 -- 172.21.15.32:0/3273807755 >> 172.21.15.32:0/3273807755 conn(0x7f80f8072bc0 msgr2=0x7f80f8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:28.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.038+0000 7f810010d700 1 -- 172.21.15.32:0/3273807755 shutdown_connections 2024-04-06T14:50:28.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.038+0000 7f810010d700 1 -- 172.21.15.32:0/3273807755 wait complete. 2024-04-06T14:50:28.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.039+0000 7f810010d700 1 Processor -- start 2024-04-06T14:50:28.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.039+0000 7f810010d700 1 -- start start 2024-04-06T14:50:28.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.039+0000 7f810010d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8101d70 0x7f80f8115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:28.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.039+0000 7f810010d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80f8102710 0x7f80f81161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:28.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.039+0000 7f810010d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f8116830 con 0x7f80f8101d70 2024-04-06T14:50:28.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.039+0000 7f810010d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f810fc00 con 0x7f80f8102710 2024-04-06T14:50:28.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.039+0000 7f80fdea9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8101d70 0x7f80f8115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:28.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.039+0000 7f80fdea9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8101d70 0x7f80f8115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51028/0 (socket says 172.21.15.32:51028) 2024-04-06T14:50:28.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.039+0000 7f80fdea9700 1 -- 172.21.15.32:0/498508591 learned_addr learned my addr 172.21.15.32:0/498508591 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:28.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f80fd6a8700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80f8102710 0x7f80f81161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:28.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f80fdea9700 1 -- 172.21.15.32:0/498508591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80f8102710 msgr2=0x7f80f81161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:28.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f80fdea9700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80f8102710 0x7f80f81161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:28.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f80fdea9700 1 -- 172.21.15.32:0/498508591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec005040 con 0x7f80f8101d70 2024-04-06T14:50:28.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f80fd6a8700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80f8102710 0x7f80f81161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:50:28.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f80fdea9700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8101d70 0x7f80f8115ca0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f80f400c8c0 tx=0x7f80f40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:28.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f80ea7fc700 1 -- 172.21.15.32:0/498508591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80f400a980 con 0x7f80f8101d70 2024-04-06T14:50:28.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f80ea7fc700 1 -- 172.21.15.32:0/498508591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80f400ce60 con 0x7f80f8101d70 2024-04-06T14:50:28.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f810fee0 con 0x7f80f8101d70 2024-04-06T14:50:28.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f80ea7fc700 1 -- 172.21.15.32:0/498508591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80f40113a0 con 0x7f80f8101d70 2024-04-06T14:50:28.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.040+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f81104a0 con 0x7f80f8101d70 2024-04-06T14:50:28.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.041+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f80fd4c0 con 0x7f80f8101d70 2024-04-06T14:50:28.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.041+0000 7f80ea7fc700 1 -- 172.21.15.32:0/498508591 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f80f4012400 con 0x7f80f8101d70 2024-04-06T14:50:28.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.042+0000 7f80ea7fc700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80e406cc20 0x7f80e406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:28.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.042+0000 7f80ea7fc700 1 -- 172.21.15.32:0/498508591 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f80f408c040 con 0x7f80f8101d70 2024-04-06T14:50:28.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.042+0000 7f80fd6a8700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80e406cc20 0x7f80e406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:28.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.042+0000 7f80fd6a8700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80e406cc20 0x7f80e406f0d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009fd0 tx=0x7f80ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:28.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.046+0000 7f80ea7fc700 1 -- 172.21.15.32:0/498508591 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f80f4059bd0 con 0x7f80f8101d70 2024-04-06T14:50:28.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.334+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f80f805f5f0 con 0x7f80f8101d70 2024-04-06T14:50:28.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.335+0000 7f80ea7fc700 1 -- 172.21.15.32:0/498508591 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f80f4059760 con 0x7f80f8101d70 2024-04-06T14:50:28.323 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:50:28.323 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:50:28.323 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:50:28.324 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:50:28.325 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:50:28.325 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:50:28.325 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:50:28.325 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:50:28.325 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:50:28.325 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:50:28.325 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:50:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80e406cc20 msgr2=0x7f80e406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80e406cc20 0x7f80e406f0d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009fd0 tx=0x7f80ec009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8101d70 msgr2=0x7f80f8115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8101d70 0x7f80f8115ca0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f80f400c8c0 tx=0x7f80f40044c0 comp rx=0 tx=0).stop 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 shutdown_connections 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f80e406cc20 0x7f80e406f0d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f8101d70 0x7f80f8115ca0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 --2- 172.21.15.32:0/498508591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80f8102710 0x7f80f81161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 >> 172.21.15.32:0/498508591 conn(0x7f80f8072bc0 msgr2=0x7f80f80659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 shutdown_connections 2024-04-06T14:50:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:28.337+0000 7f810010d700 1 -- 172.21.15.32:0/498508591 wait complete. 2024-04-06T14:50:28.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:28 smithi086 ceph-mon[110975]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:50:28.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:28 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/498508591' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:50:28.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:28 smithi032 ceph-mon[147414]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:50:28.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:28 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/498508591' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:50:29.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:29 smithi032 ceph-mon[147414]: from='client.35674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:50:29.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:29 smithi032 ceph-mon[147414]: pgmap v911: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:30.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:29 smithi086 ceph-mon[110975]: from='client.35674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:50:30.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:29 smithi086 ceph-mon[110975]: pgmap v911: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:31.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:31 smithi032 ceph-mon[147414]: pgmap v912: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:32.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:31 smithi086 ceph-mon[110975]: pgmap v912: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:33.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:33 smithi032 ceph-mon[147414]: pgmap v913: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:34.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:33 smithi086 ceph-mon[110975]: pgmap v913: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:35.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:35 smithi032 ceph-mon[147414]: pgmap v914: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:36.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:35 smithi086 ceph-mon[110975]: pgmap v914: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:37.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:37 smithi032 ceph-mon[147414]: pgmap v915: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:37.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:50:38.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:37 smithi086 ceph-mon[110975]: pgmap v915: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:38.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:50:39.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:39 smithi032 ceph-mon[147414]: pgmap v916: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:40.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:39 smithi086 ceph-mon[110975]: pgmap v916: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:41.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:41 smithi032 ceph-mon[147414]: pgmap v917: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:42.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:41 smithi086 ceph-mon[110975]: pgmap v917: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:43.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:43 smithi032 ceph-mon[147414]: pgmap v918: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:44.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:43 smithi086 ceph-mon[110975]: pgmap v918: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:45.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:45 smithi032 ceph-mon[147414]: pgmap v919: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:46.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:45 smithi086 ceph-mon[110975]: pgmap v919: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:47.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:47 smithi032 ceph-mon[147414]: pgmap v920: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:48.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:47 smithi086 ceph-mon[110975]: pgmap v920: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:49.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:49 smithi032 ceph-mon[147414]: pgmap v921: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:50.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:49 smithi086 ceph-mon[110975]: pgmap v921: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:51.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:51 smithi032 ceph-mon[147414]: pgmap v922: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:52.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:51 smithi086 ceph-mon[110975]: pgmap v922: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:52.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:50:53.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:50:53.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:53 smithi032 ceph-mon[147414]: pgmap v923: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:54.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:53 smithi086 ceph-mon[110975]: pgmap v923: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:55.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:55 smithi032 ceph-mon[147414]: pgmap v924: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:56.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:55 smithi086 ceph-mon[110975]: pgmap v924: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:57.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:57 smithi032 ceph-mon[147414]: pgmap v925: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:58.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:57 smithi086 ceph-mon[110975]: pgmap v925: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:58.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.453+0000 7fd815014700 1 -- 172.21.15.32:0/147020872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 msgr2=0x7fd8100fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:58.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.453+0000 7fd815014700 1 --2- 172.21.15.32:0/147020872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 0x7fd8100fb910 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd804004940 tx=0x7fd804009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:58.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.454+0000 7fd815014700 1 -- 172.21.15.32:0/147020872 shutdown_connections 2024-04-06T14:50:58.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.454+0000 7fd815014700 1 --2- 172.21.15.32:0/147020872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 0x7fd8100fb910 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.454+0000 7fd815014700 1 --2- 172.21.15.32:0/147020872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd810102e70 0x7fd810103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.454+0000 7fd815014700 1 -- 172.21.15.32:0/147020872 >> 172.21.15.32:0/147020872 conn(0x7fd8100714c0 msgr2=0x7fd8100718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:58.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.454+0000 7fd815014700 1 -- 172.21.15.32:0/147020872 shutdown_connections 2024-04-06T14:50:58.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.454+0000 7fd815014700 1 -- 172.21.15.32:0/147020872 wait complete. 2024-04-06T14:50:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd815014700 1 Processor -- start 2024-04-06T14:50:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd815014700 1 -- start start 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd815014700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 0x7fd8100fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd815014700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd810102e70 0x7fd8100fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd815014700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100fd150 con 0x7fd8100fb4a0 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd815014700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100fd290 con 0x7fd810102e70 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd80ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 0x7fd8100fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd80ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 0x7fd8100fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51722/0 (socket says 172.21.15.32:51722) 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd80ed9d700 1 -- 172.21.15.32:0/1768072268 learned_addr learned my addr 172.21.15.32:0/1768072268 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd80e59c700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd810102e70 0x7fd8100fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd80ed9d700 1 -- 172.21.15.32:0/1768072268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd810102e70 msgr2=0x7fd8100fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd80ed9d700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd810102e70 0x7fd8100fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.455+0000 7fd80ed9d700 1 -- 172.21.15.32:0/1768072268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd804005040 con 0x7fd8100fb4a0 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.456+0000 7fd80ed9d700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 0x7fd8100fc680 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd800002a50 tx=0x7fd8000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.456+0000 7fd7f77fe700 1 -- 172.21.15.32:0/1768072268 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd800007cc0 con 0x7fd8100fb4a0 2024-04-06T14:50:58.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.456+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd810119d70 con 0x7fd8100fb4a0 2024-04-06T14:50:58.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.456+0000 7fd7f77fe700 1 -- 172.21.15.32:0/1768072268 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd80000f040 con 0x7fd8100fb4a0 2024-04-06T14:50:58.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.456+0000 7fd7f77fe700 1 -- 172.21.15.32:0/1768072268 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80000da70 con 0x7fd8100fb4a0 2024-04-06T14:50:58.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.456+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81011a1f0 con 0x7fd8100fb4a0 2024-04-06T14:50:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.457+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd810100110 con 0x7fd8100fb4a0 2024-04-06T14:50:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.457+0000 7fd7f77fe700 1 -- 172.21.15.32:0/1768072268 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd80000dc10 con 0x7fd8100fb4a0 2024-04-06T14:50:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.458+0000 7fd7f77fe700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7f806cc20 0x7fd7f806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.458+0000 7fd7f77fe700 1 -- 172.21.15.32:0/1768072268 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd80008b7e0 con 0x7fd8100fb4a0 2024-04-06T14:50:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.458+0000 7fd80e59c700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7f806cc20 0x7fd7f806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.458+0000 7fd80e59c700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7f806cc20 0x7fd7f806f0d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd804004f90 tx=0x7fd804009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:58.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.463+0000 7fd7f77fe700 1 -- 172.21.15.32:0/1768072268 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd8000557a0 con 0x7fd8100fb4a0 2024-04-06T14:50:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.684+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd81004bed0 con 0x7fd7f806cc20 2024-04-06T14:50:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.685+0000 7fd7f77fe700 1 -- 172.21.15.32:0/1768072268 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd81004bed0 con 0x7fd7f806cc20 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7f806cc20 msgr2=0x7fd7f806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7f806cc20 0x7fd7f806f0d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd804004f90 tx=0x7fd804009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 msgr2=0x7fd8100fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 0x7fd8100fc680 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd800002a50 tx=0x7fd8000046b0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 shutdown_connections 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd7f806cc20 0x7fd7f806f0d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8100fb4a0 0x7fd8100fc680 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 --2- 172.21.15.32:0/1768072268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd810102e70 0x7fd8100fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 >> 172.21.15.32:0/1768072268 conn(0x7fd8100714c0 msgr2=0x7fd8100fa900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 shutdown_connections 2024-04-06T14:50:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.687+0000 7fd815014700 1 -- 172.21.15.32:0/1768072268 wait complete. 2024-04-06T14:50:58.686 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:50:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 -- 172.21.15.32:0/1562806856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c102710 msgr2=0x7fde1c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 --2- 172.21.15.32:0/1562806856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c102710 0x7fde1c0fbd80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fde08004940 tx=0x7fde08009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 -- 172.21.15.32:0/1562806856 shutdown_connections 2024-04-06T14:50:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 --2- 172.21.15.32:0/1562806856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c102710 0x7fde1c0fbd80 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 --2- 172.21.15.32:0/1562806856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde1c101d70 0x7fde1c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 -- 172.21.15.32:0/1562806856 >> 172.21.15.32:0/1562806856 conn(0x7fde1c072bc0 msgr2=0x7fde1c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 -- 172.21.15.32:0/1562806856 shutdown_connections 2024-04-06T14:50:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 -- 172.21.15.32:0/1562806856 wait complete. 2024-04-06T14:50:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 Processor -- start 2024-04-06T14:50:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.792+0000 7fde22c0c700 1 -- start start 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde22c0c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c101d70 0x7fde1c10ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde22c0c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde1c102710 0x7fde1c10efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde22c0c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c10f5f0 con 0x7fde1c101d70 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde22c0c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c10f730 con 0x7fde1c102710 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde209a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c101d70 0x7fde1c10ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde209a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c101d70 0x7fde1c10ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51736/0 (socket says 172.21.15.32:51736) 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde209a8700 1 -- 172.21.15.32:0/3103707124 learned_addr learned my addr 172.21.15.32:0/3103707124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde1bfff700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde1c102710 0x7fde1c10efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde209a8700 1 -- 172.21.15.32:0/3103707124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde1c102710 msgr2=0x7fde1c10efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde209a8700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde1c102710 0x7fde1c10efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde209a8700 1 -- 172.21.15.32:0/3103707124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde08005040 con 0x7fde1c101d70 2024-04-06T14:50:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde1bfff700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde1c102710 0x7fde1c10efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:50:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde209a8700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c101d70 0x7fde1c10ea60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c8c0 tx=0x7fde100044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde197fa700 1 -- 172.21.15.32:0/3103707124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde1000a980 con 0x7fde1c101d70 2024-04-06T14:50:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde197fa700 1 -- 172.21.15.32:0/3103707124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde1000ce60 con 0x7fde1c101d70 2024-04-06T14:50:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c117ad0 con 0x7fde1c101d70 2024-04-06T14:50:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde197fa700 1 -- 172.21.15.32:0/3103707124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde100113a0 con 0x7fde1c101d70 2024-04-06T14:50:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.793+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c118030 con 0x7fde1c101d70 2024-04-06T14:50:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.794+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c0ff080 con 0x7fde1c101d70 2024-04-06T14:50:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.795+0000 7fde197fa700 1 -- 172.21.15.32:0/3103707124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fde10012400 con 0x7fde1c101d70 2024-04-06T14:50:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.796+0000 7fde197fa700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde0c06cc20 0x7fde0c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.796+0000 7fde197fa700 1 -- 172.21.15.32:0/3103707124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fde1008cc70 con 0x7fde1c101d70 2024-04-06T14:50:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.796+0000 7fde1bfff700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde0c06cc20 0x7fde0c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.796+0000 7fde1bfff700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde0c06cc20 0x7fde0c06f0d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fde0800b3b0 tx=0x7fde08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:58.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:58.800+0000 7fde197fa700 1 -- 172.21.15.32:0/3103707124 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fde10056cb0 con 0x7fde1c101d70 2024-04-06T14:50:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.022+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde1c108db0 con 0x7fde0c06cc20 2024-04-06T14:50:59.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.023+0000 7fde197fa700 1 -- 172.21.15.32:0/3103707124 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fde08007d50 con 0x7fde0c06cc20 2024-04-06T14:50:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde0c06cc20 msgr2=0x7fde0c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde0c06cc20 0x7fde0c06f0d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fde0800b3b0 tx=0x7fde08009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c101d70 msgr2=0x7fde1c10ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c101d70 0x7fde1c10ea60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c8c0 tx=0x7fde100044c0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 shutdown_connections 2024-04-06T14:50:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fde0c06cc20 0x7fde0c06f0d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde1c101d70 0x7fde1c10ea60 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 --2- 172.21.15.32:0/3103707124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde1c102710 0x7fde1c10efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 >> 172.21.15.32:0/3103707124 conn(0x7fde1c072bc0 msgr2=0x7fde1c0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 shutdown_connections 2024-04-06T14:50:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.025+0000 7fde22c0c700 1 -- 172.21.15.32:0/3103707124 wait complete. 2024-04-06T14:50:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.128+0000 7f505bcdb700 1 -- 172.21.15.32:0/2007282793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 msgr2=0x7f50540fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.128+0000 7f505bcdb700 1 --2- 172.21.15.32:0/2007282793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 0x7f50540fbd30 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f5048004940 tx=0x7f5048009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.128+0000 7f505bcdb700 1 -- 172.21.15.32:0/2007282793 shutdown_connections 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.128+0000 7f505bcdb700 1 --2- 172.21.15.32:0/2007282793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 0x7f50540fbd30 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.128+0000 7f505bcdb700 1 --2- 172.21.15.32:0/2007282793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5054101d20 0x7f50541020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.128+0000 7f505bcdb700 1 -- 172.21.15.32:0/2007282793 >> 172.21.15.32:0/2007282793 conn(0x7f50540f7870 msgr2=0x7f50540f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.128+0000 7f505bcdb700 1 -- 172.21.15.32:0/2007282793 shutdown_connections 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.128+0000 7f505bcdb700 1 -- 172.21.15.32:0/2007282793 wait complete. 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.128+0000 7f505bcdb700 1 Processor -- start 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f505bcdb700 1 -- start start 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f505bcdb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5054101d20 0x7f505410c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f505bcdb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 0x7f505410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f505bcdb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5054067120 con 0x7f50541026c0 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f505bcdb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505410d250 con 0x7f5054101d20 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f5059276700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 0x7f505410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f5059276700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 0x7f505410cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51766/0 (socket says 172.21.15.32:51766) 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f5059276700 1 -- 172.21.15.32:0/445025945 learned_addr learned my addr 172.21.15.32:0/445025945 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f5059a77700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5054101d20 0x7f505410c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f5059276700 1 -- 172.21.15.32:0/445025945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5054101d20 msgr2=0x7f505410c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f5059276700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5054101d20 0x7f505410c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f5059276700 1 -- 172.21.15.32:0/445025945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048005040 con 0x7f50541026c0 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f5059a77700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5054101d20 0x7f505410c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f5059276700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 0x7f505410cd10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5048008d50 tx=0x7f5048009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.129+0000 7f50467fc700 1 -- 172.21.15.32:0/445025945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504800e030 con 0x7f50541026c0 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.130+0000 7f50467fc700 1 -- 172.21.15.32:0/445025945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5048013de0 con 0x7f50541026c0 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.130+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505410d4d0 con 0x7f50541026c0 2024-04-06T14:50:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.130+0000 7f50467fc700 1 -- 172.21.15.32:0/445025945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504801cd90 con 0x7f50541026c0 2024-04-06T14:50:59.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.130+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505411a000 con 0x7f50541026c0 2024-04-06T14:50:59.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.130+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50540fd470 con 0x7f50541026c0 2024-04-06T14:50:59.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.131+0000 7f50467fc700 1 -- 172.21.15.32:0/445025945 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5048013930 con 0x7f50541026c0 2024-04-06T14:50:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.131+0000 7f50467fc700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f504006cc20 0x7f504006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.131+0000 7f50467fc700 1 -- 172.21.15.32:0/445025945 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f504808d800 con 0x7f50541026c0 2024-04-06T14:50:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.131+0000 7f5059a77700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f504006cc20 0x7f504006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.132+0000 7f5059a77700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f504006cc20 0x7f504006f0d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5050004600 tx=0x7f505000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:59.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.136+0000 7f50467fc700 1 -- 172.21.15.32:0/445025945 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f504805b390 con 0x7f50541026c0 2024-04-06T14:50:59.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.346+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f50540f9000 con 0x7f504006cc20 2024-04-06T14:50:59.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.355+0000 7f50467fc700 1 -- 172.21.15.32:0/445025945 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f50540f9000 con 0x7f504006cc20 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (31m) 9m ago 42m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (28m) 9m ago 42m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (28m) 9m ago 41m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (31m) 9m ago 42m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (25m) 9m ago 36m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (24m) 9m ago 36m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (24m) 9m ago 36m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (9m) 9m ago 36m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (33m) 9m ago 44m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (32m) 9m ago 41m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:50:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (29m) 9m ago 44m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (29m) 9m ago 41m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (32m) 9m ago 42m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (31m) 9m ago 41m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (28m) 9m ago 40m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (27m) 9m ago 40m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (27m) 9m ago 39m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (26m) 9m ago 39m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (25m) 9m ago 38m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (25m) 9m ago 38m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:50:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (31m) 9m ago 42m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:50:59.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.357+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f504006cc20 msgr2=0x7f504006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.357+0000 7f505bcdb700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f504006cc20 0x7f504006f0d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5050004600 tx=0x7f505000b040 comp rx=0 tx=0).stop 2024-04-06T14:50:59.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.357+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 msgr2=0x7f505410cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.357+0000 7f505bcdb700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 0x7f505410cd10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5048008d50 tx=0x7f5048009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:59.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.357+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 shutdown_connections 2024-04-06T14:50:59.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.357+0000 7f505bcdb700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f504006cc20 0x7f504006f0d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.357+0000 7f505bcdb700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5054101d20 0x7f505410c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.357+0000 7f505bcdb700 1 --2- 172.21.15.32:0/445025945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50541026c0 0x7f505410cd10 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.357+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 >> 172.21.15.32:0/445025945 conn(0x7f50540f7870 msgr2=0x7f50540f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:59.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.358+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 shutdown_connections 2024-04-06T14:50:59.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.358+0000 7f505bcdb700 1 -- 172.21.15.32:0/445025945 wait complete. 2024-04-06T14:50:59.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.459+0000 7fecc82bd700 1 -- 172.21.15.32:0/2312719120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc01037b0 msgr2=0x7fecc00fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.459+0000 7fecc82bd700 1 --2- 172.21.15.32:0/2312719120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc01037b0 0x7fecc00fce20 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fecb4004940 tx=0x7fecb4009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:59.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.459+0000 7fecc82bd700 1 -- 172.21.15.32:0/2312719120 shutdown_connections 2024-04-06T14:50:59.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.459+0000 7fecc82bd700 1 --2- 172.21.15.32:0/2312719120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc01037b0 0x7fecc00fce20 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.459+0000 7fecc82bd700 1 --2- 172.21.15.32:0/2312719120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc0102e10 0x7fecc01031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.459+0000 7fecc82bd700 1 -- 172.21.15.32:0/2312719120 >> 172.21.15.32:0/2312719120 conn(0x7fecc00f89a0 msgr2=0x7fecc00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:59.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.460+0000 7fecc82bd700 1 -- 172.21.15.32:0/2312719120 shutdown_connections 2024-04-06T14:50:59.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.460+0000 7fecc82bd700 1 -- 172.21.15.32:0/2312719120 wait complete. 2024-04-06T14:50:59.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.460+0000 7fecc82bd700 1 Processor -- start 2024-04-06T14:50:59.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.460+0000 7fecc82bd700 1 -- start start 2024-04-06T14:50:59.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc82bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc0102e10 0x7fecc0113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc82bd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc01037b0 0x7fecc0114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc82bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0114660 con 0x7fecc0102e10 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc82bd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc010da50 con 0x7fecc01037b0 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc6059700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc0102e10 0x7fecc0113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc6059700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc0102e10 0x7fecc0113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51788/0 (socket says 172.21.15.32:51788) 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc6059700 1 -- 172.21.15.32:0/2750028446 learned_addr learned my addr 172.21.15.32:0/2750028446 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc5858700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc01037b0 0x7fecc0114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc6059700 1 -- 172.21.15.32:0/2750028446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc01037b0 msgr2=0x7fecc0114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc6059700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc01037b0 0x7fecc0114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc6059700 1 -- 172.21.15.32:0/2750028446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb4005040 con 0x7fecc0102e10 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc6059700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc0102e10 0x7fecc0113ad0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fecbc002a50 tx=0x7fecbc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecb2ffd700 1 -- 172.21.15.32:0/2750028446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecbc007cc0 con 0x7fecc0102e10 2024-04-06T14:50:59.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecb2ffd700 1 -- 172.21.15.32:0/2750028446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecbc00f040 con 0x7fecc0102e10 2024-04-06T14:50:59.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc010dd30 con 0x7fecc0102e10 2024-04-06T14:50:59.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecb2ffd700 1 -- 172.21.15.32:0/2750028446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecbc00d9c0 con 0x7fecc0102e10 2024-04-06T14:50:59.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.461+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc010e2c0 con 0x7fecc0102e10 2024-04-06T14:50:59.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.462+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc00fe560 con 0x7fecc0102e10 2024-04-06T14:50:59.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.463+0000 7fecb2ffd700 1 -- 172.21.15.32:0/2750028446 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fecbc00db20 con 0x7fecc0102e10 2024-04-06T14:50:59.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.463+0000 7fecb2ffd700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecac06cc30 0x7fecac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:59.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.464+0000 7fecb2ffd700 1 -- 172.21.15.32:0/2750028446 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fecbc08bcb0 con 0x7fecc0102e10 2024-04-06T14:50:59.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.464+0000 7fecc5858700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecac06cc30 0x7fecac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:59.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.464+0000 7fecc5858700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecac06cc30 0x7fecac06f0e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fecc010ed30 tx=0x7fecb4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:59.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.468+0000 7fecb2ffd700 1 -- 172.21.15.32:0/2750028446 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fecbc059780 con 0x7fecc0102e10 2024-04-06T14:50:59.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.764+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fecc0109e00 con 0x7fecc0102e10 2024-04-06T14:50:59.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.764+0000 7fecb2ffd700 1 -- 172.21.15.32:0/2750028446 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fecbc059310 con 0x7fecc0102e10 2024-04-06T14:50:59.752 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:50:59.753 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:50:59.754 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:50:59.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:50:59.754 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:50:59.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.766+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecac06cc30 msgr2=0x7fecac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.766+0000 7fecc82bd700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecac06cc30 0x7fecac06f0e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fecc010ed30 tx=0x7fecb4009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.766+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc0102e10 msgr2=0x7fecc0113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.766+0000 7fecc82bd700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc0102e10 0x7fecc0113ad0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fecbc002a50 tx=0x7fecbc0046b0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.766+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 shutdown_connections 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.766+0000 7fecc82bd700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecac06cc30 0x7fecac06f0e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.766+0000 7fecc82bd700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc0102e10 0x7fecc0113ad0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.766+0000 7fecc82bd700 1 --2- 172.21.15.32:0/2750028446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc01037b0 0x7fecc0114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.766+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 >> 172.21.15.32:0/2750028446 conn(0x7fecc00f89a0 msgr2=0x7fecc00f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.767+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 shutdown_connections 2024-04-06T14:50:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.767+0000 7fecc82bd700 1 -- 172.21.15.32:0/2750028446 wait complete. 2024-04-06T14:50:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.870+0000 7f4513bba700 1 -- 172.21.15.32:0/3418526012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 msgr2=0x7f450c0fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.870+0000 7f4513bba700 1 --2- 172.21.15.32:0/3418526012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 0x7f450c0fbd90 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4500004940 tx=0x7f4500009e30 comp rx=0 tx=0).stop 2024-04-06T14:50:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.871+0000 7f4513bba700 1 -- 172.21.15.32:0/3418526012 shutdown_connections 2024-04-06T14:50:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.871+0000 7f4513bba700 1 --2- 172.21.15.32:0/3418526012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 0x7f450c0fbd90 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.871+0000 7f4513bba700 1 --2- 172.21.15.32:0/3418526012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c101d80 0x7f450c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.871+0000 7f4513bba700 1 -- 172.21.15.32:0/3418526012 >> 172.21.15.32:0/3418526012 conn(0x7f450c0f78f0 msgr2=0x7f450c0f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:50:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.871+0000 7f4513bba700 1 -- 172.21.15.32:0/3418526012 shutdown_connections 2024-04-06T14:50:59.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.871+0000 7f4513bba700 1 -- 172.21.15.32:0/3418526012 wait complete. 2024-04-06T14:50:59.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.871+0000 7f4513bba700 1 Processor -- start 2024-04-06T14:50:59.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.871+0000 7f4513bba700 1 -- start start 2024-04-06T14:50:59.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4513bba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c101d80 0x7f450c103380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:59.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4513bba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 0x7f450c1038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4513bba700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c105320 con 0x7f450c102720 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4513bba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c103e00 con 0x7f450c101d80 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4511155700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 0x7f450c1038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4511155700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 0x7f450c1038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51814/0 (socket says 172.21.15.32:51814) 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4511155700 1 -- 172.21.15.32:0/2236976167 learned_addr learned my addr 172.21.15.32:0/2236976167 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4511956700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c101d80 0x7f450c103380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4511155700 1 -- 172.21.15.32:0/2236976167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c101d80 msgr2=0x7f450c103380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4511155700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c101d80 0x7f450c103380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4511155700 1 -- 172.21.15.32:0/2236976167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4500005040 con 0x7f450c102720 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4511155700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 0x7f450c1038c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f45000093b0 tx=0x7f4500009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f44fe7fc700 1 -- 172.21.15.32:0/2236976167 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f450000e040 con 0x7f450c102720 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f44fe7fc700 1 -- 172.21.15.32:0/2236976167 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4500013420 con 0x7f450c102720 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c104050 con 0x7f450c102720 2024-04-06T14:50:59.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f44fe7fc700 1 -- 172.21.15.32:0/2236976167 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f450001c630 con 0x7f450c102720 2024-04-06T14:50:59.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.872+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c119fd0 con 0x7f450c102720 2024-04-06T14:50:59.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.873+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c0fd4d0 con 0x7f450c102720 2024-04-06T14:50:59.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.874+0000 7f44fe7fc700 1 -- 172.21.15.32:0/2236976167 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f45000139e0 con 0x7f450c102720 2024-04-06T14:50:59.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.874+0000 7f44fe7fc700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44f806cc20 0x7f44f806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:50:59.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.874+0000 7f44fe7fc700 1 -- 172.21.15.32:0/2236976167 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f450008d130 con 0x7f450c102720 2024-04-06T14:50:59.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.874+0000 7f4511956700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44f806cc20 0x7f44f806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:50:59.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.875+0000 7f4511956700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44f806cc20 0x7f44f806f0d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4508004f40 tx=0x7f450800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:50:59.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:50:59.879+0000 7f44fe7fc700 1 -- 172.21.15.32:0/2236976167 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4500057170 con 0x7f450c102720 2024-04-06T14:50:59.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:59 smithi032 ceph-mon[147414]: pgmap v926: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:50:59.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:50:59 smithi032 ceph-mon[147414]: from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:00.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:59 smithi086 ceph-mon[110975]: pgmap v926: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:00.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:50:59 smithi086 ceph-mon[110975]: from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.129+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f450c04c710 con 0x7f450c102720 2024-04-06T14:51:00.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.130+0000 7f44fe7fc700 1 -- 172.21.15.32:0/2236976167 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f450005a790 con 0x7f450c102720 2024-04-06T14:51:00.118 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:51:00.118 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:51:00.118 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:51:00.118 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:51:00.118 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:51:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:51:00.120 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:51:00.121 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:51:00.121 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:51:00.121 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:51:00.121 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44f806cc20 msgr2=0x7f44f806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44f806cc20 0x7f44f806f0d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4508004f40 tx=0x7f450800b040 comp rx=0 tx=0).stop 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 msgr2=0x7f450c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 0x7f450c1038c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f45000093b0 tx=0x7f4500009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 shutdown_connections 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44f806cc20 0x7f44f806f0d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c101d80 0x7f450c103380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 --2- 172.21.15.32:0/2236976167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c102720 0x7f450c1038c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 >> 172.21.15.32:0/2236976167 conn(0x7f450c0f78f0 msgr2=0x7f450c106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 shutdown_connections 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.132+0000 7f4513bba700 1 -- 172.21.15.32:0/2236976167 wait complete. 2024-04-06T14:51:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.237+0000 7f405dfda700 1 -- 172.21.15.32:0/498657420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058101d70 msgr2=0x7f4058102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.237+0000 7f405dfda700 1 --2- 172.21.15.32:0/498657420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058101d70 0x7f4058102140 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f404c004940 tx=0x7f404c009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.237+0000 7f405dfda700 1 -- 172.21.15.32:0/498657420 shutdown_connections 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.237+0000 7f405dfda700 1 --2- 172.21.15.32:0/498657420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4058102710 0x7f40580fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.237+0000 7f405dfda700 1 --2- 172.21.15.32:0/498657420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058101d70 0x7f4058102140 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.237+0000 7f405dfda700 1 -- 172.21.15.32:0/498657420 >> 172.21.15.32:0/498657420 conn(0x7f4058072bc0 msgr2=0x7f4058072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.237+0000 7f405dfda700 1 -- 172.21.15.32:0/498657420 shutdown_connections 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.237+0000 7f405dfda700 1 -- 172.21.15.32:0/498657420 wait complete. 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.237+0000 7f405dfda700 1 Processor -- start 2024-04-06T14:51:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f405dfda700 1 -- start start 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f405dfda700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4058101d70 0x7f4058112920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f405dfda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058102710 0x7f4058112e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f405dfda700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40581134b0 con 0x7f4058102710 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f405dfda700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405810c8a0 con 0x7f4058101d70 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f4056ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058102710 0x7f4058112e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f4056ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058102710 0x7f4058112e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51830/0 (socket says 172.21.15.32:51830) 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f4056ffd700 1 -- 172.21.15.32:0/1462811791 learned_addr learned my addr 172.21.15.32:0/1462811791 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f40577fe700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4058101d70 0x7f4058112920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f4056ffd700 1 -- 172.21.15.32:0/1462811791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4058101d70 msgr2=0x7f4058112920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f4056ffd700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4058101d70 0x7f4058112920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f4056ffd700 1 -- 172.21.15.32:0/1462811791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c005040 con 0x7f4058102710 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.238+0000 7f4056ffd700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058102710 0x7f4058112e60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f404800eb40 tx=0x7f404800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.239+0000 7f403ffff700 1 -- 172.21.15.32:0/1462811791 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4048007cc0 con 0x7f4058102710 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.239+0000 7f403ffff700 1 -- 172.21.15.32:0/1462811791 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f404800f040 con 0x7f4058102710 2024-04-06T14:51:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.239+0000 7f403ffff700 1 -- 172.21.15.32:0/1462811791 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f404800c8f0 con 0x7f4058102710 2024-04-06T14:51:00.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.239+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405810cb20 con 0x7f4058102710 2024-04-06T14:51:00.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.239+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405810d0e0 con 0x7f4058102710 2024-04-06T14:51:00.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.240+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40580ff080 con 0x7f4058102710 2024-04-06T14:51:00.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.240+0000 7f403ffff700 1 -- 172.21.15.32:0/1462811791 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f404800cb60 con 0x7f4058102710 2024-04-06T14:51:00.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.241+0000 7f403ffff700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f404006cc20 0x7f404006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:00.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.241+0000 7f403ffff700 1 -- 172.21.15.32:0/1462811791 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f404808bae0 con 0x7f4058102710 2024-04-06T14:51:00.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.241+0000 7f40577fe700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f404006cc20 0x7f404006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:00.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.241+0000 7f40577fe700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f404006cc20 0x7f404006f0d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f404c00b0d0 tx=0x7f404c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:00.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.245+0000 7f403ffff700 1 -- 172.21.15.32:0/1462811791 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f40480595e0 con 0x7f4058102710 2024-04-06T14:51:00.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.465+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f405810dfe0 con 0x7f404006cc20 2024-04-06T14:51:00.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.466+0000 7f403ffff700 1 -- 172.21.15.32:0/1462811791 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f405810dfe0 con 0x7f404006cc20 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:51:00.455 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:51:00.456 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:51:00.456 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.468+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f404006cc20 msgr2=0x7f404006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.468+0000 7f405dfda700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f404006cc20 0x7f404006f0d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f404c00b0d0 tx=0x7f404c009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.468+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058102710 msgr2=0x7f4058112e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.468+0000 7f405dfda700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058102710 0x7f4058112e60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f404800eb40 tx=0x7f404800b040 comp rx=0 tx=0).stop 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.469+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 shutdown_connections 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.469+0000 7f405dfda700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f404006cc20 0x7f404006f0d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.469+0000 7f405dfda700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4058101d70 0x7f4058112920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.469+0000 7f405dfda700 1 --2- 172.21.15.32:0/1462811791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4058102710 0x7f4058112e60 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.469+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 >> 172.21.15.32:0/1462811791 conn(0x7f4058072bc0 msgr2=0x7f40580faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.469+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 shutdown_connections 2024-04-06T14:51:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.469+0000 7f405dfda700 1 -- 172.21.15.32:0/1462811791 wait complete. 2024-04-06T14:51:00.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.572+0000 7fc1f70ef700 1 -- 172.21.15.32:0/3507056934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 msgr2=0x7fc1f0103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.572+0000 7fc1f70ef700 1 --2- 172.21.15.32:0/3507056934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 0x7fc1f0103240 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4004940 tx=0x7fc1e4009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:00.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.572+0000 7fc1f70ef700 1 -- 172.21.15.32:0/3507056934 shutdown_connections 2024-04-06T14:51:00.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.572+0000 7fc1f70ef700 1 --2- 172.21.15.32:0/3507056934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1f00fb4a0 0x7fc1f00fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.572+0000 7fc1f70ef700 1 --2- 172.21.15.32:0/3507056934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 0x7fc1f0103240 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.572+0000 7fc1f70ef700 1 -- 172.21.15.32:0/3507056934 >> 172.21.15.32:0/3507056934 conn(0x7fc1f00714c0 msgr2=0x7fc1f00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:00.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.572+0000 7fc1f70ef700 1 -- 172.21.15.32:0/3507056934 shutdown_connections 2024-04-06T14:51:00.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.572+0000 7fc1f70ef700 1 -- 172.21.15.32:0/3507056934 wait complete. 2024-04-06T14:51:00.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.572+0000 7fc1f70ef700 1 Processor -- start 2024-04-06T14:51:00.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1f70ef700 1 -- start start 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1f70ef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1f00fb4a0 0x7fc1f0113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1f70ef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 0x7fc1f0113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1f70ef700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0114600 con 0x7fc1f0102e70 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1f70ef700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f010d9f0 con 0x7fc1f00fb4a0 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 0x7fc1f0113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 0x7fc1f0113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51850/0 (socket says 172.21.15.32:51850) 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1effff700 1 -- 172.21.15.32:0/2705264257 learned_addr learned my addr 172.21.15.32:0/2705264257 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1f4e8b700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1f00fb4a0 0x7fc1f0113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1effff700 1 -- 172.21.15.32:0/2705264257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1f00fb4a0 msgr2=0x7fc1f0113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1effff700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1f00fb4a0 0x7fc1f0113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1effff700 1 -- 172.21.15.32:0/2705264257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e4005040 con 0x7fc1f0102e70 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.573+0000 7fc1effff700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 0x7fc1f0113fb0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc002a50 tx=0x7fc1dc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:00.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.574+0000 7fc1ed7fa700 1 -- 172.21.15.32:0/2705264257 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1dc007cc0 con 0x7fc1f0102e70 2024-04-06T14:51:00.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.574+0000 7fc1ed7fa700 1 -- 172.21.15.32:0/2705264257 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1dc00f040 con 0x7fc1f0102e70 2024-04-06T14:51:00.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.574+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f010dcd0 con 0x7fc1f0102e70 2024-04-06T14:51:00.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.574+0000 7fc1ed7fa700 1 -- 172.21.15.32:0/2705264257 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1dc009540 con 0x7fc1f0102e70 2024-04-06T14:51:00.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.574+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f010e290 con 0x7fc1f0102e70 2024-04-06T14:51:00.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.575+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f0100110 con 0x7fc1f0102e70 2024-04-06T14:51:00.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.575+0000 7fc1ed7fa700 1 -- 172.21.15.32:0/2705264257 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc1dc0096a0 con 0x7fc1f0102e70 2024-04-06T14:51:00.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.575+0000 7fc1ed7fa700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1e006cc30 0x7fc1e006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:00.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.576+0000 7fc1ed7fa700 1 -- 172.21.15.32:0/2705264257 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc1dc08be70 con 0x7fc1f0102e70 2024-04-06T14:51:00.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.576+0000 7fc1f4e8b700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1e006cc30 0x7fc1e006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:00.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.576+0000 7fc1f4e8b700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1e006cc30 0x7fc1e006f0e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400b3b0 tx=0x7fc1e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:00.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.580+0000 7fc1ed7fa700 1 -- 172.21.15.32:0/2705264257 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc1dc0598c0 con 0x7fc1f0102e70 2024-04-06T14:51:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.866+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc1f010f0c0 con 0x7fc1f0102e70 2024-04-06T14:51:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.866+0000 7fc1ed7fa700 1 -- 172.21.15.32:0/2705264257 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7fc1dc059450 con 0x7fc1f0102e70 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:51:00.855 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:51:00.856 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:51:00.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1e006cc30 msgr2=0x7fc1e006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1e006cc30 0x7fc1e006f0e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400b3b0 tx=0x7fc1e4009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:00.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 msgr2=0x7fc1f0113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:00.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 0x7fc1f0113fb0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc002a50 tx=0x7fc1dc0046b0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 shutdown_connections 2024-04-06T14:51:00.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc1e006cc30 0x7fc1e006f0e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1f00fb4a0 0x7fc1f0113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 --2- 172.21.15.32:0/2705264257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1f0102e70 0x7fc1f0113fb0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 >> 172.21.15.32:0/2705264257 conn(0x7fc1f00714c0 msgr2=0x7fc1f00fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 shutdown_connections 2024-04-06T14:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:00.868+0000 7fc1f70ef700 1 -- 172.21.15.32:0/2705264257 wait complete. 2024-04-06T14:51:00.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:00 smithi032 ceph-mon[147414]: from='client.35686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:00.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:00 smithi032 ceph-mon[147414]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:00.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:00 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2750028446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:51:00.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:00 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2236976167' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:51:01.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:00 smithi086 ceph-mon[110975]: from='client.35686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:01.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:00 smithi086 ceph-mon[110975]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:01.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:00 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2750028446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:51:01.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:00 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2236976167' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:51:01.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:01 smithi032 ceph-mon[147414]: pgmap v927: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:01.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:01 smithi032 ceph-mon[147414]: from='client.35702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:01.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:01 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2705264257' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:51:02.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:01 smithi086 ceph-mon[110975]: pgmap v927: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:02.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:01 smithi086 ceph-mon[110975]: from='client.35702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:02.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:01 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2705264257' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:51:03.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:03 smithi032 ceph-mon[147414]: pgmap v928: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:04.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:03 smithi086 ceph-mon[110975]: pgmap v928: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:05.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:05 smithi032 ceph-mon[147414]: pgmap v929: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:06.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:05 smithi086 ceph-mon[110975]: pgmap v929: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:07.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:07 smithi032 ceph-mon[147414]: pgmap v930: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:07.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:51:08.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:07 smithi086 ceph-mon[110975]: pgmap v930: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:08.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:51:09.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:09 smithi032 ceph-mon[147414]: pgmap v931: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:10.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:09 smithi086 ceph-mon[110975]: pgmap v931: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:11.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:11 smithi032 ceph-mon[147414]: pgmap v932: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:12.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:11 smithi086 ceph-mon[110975]: pgmap v932: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:13.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:13 smithi032 ceph-mon[147414]: pgmap v933: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:14.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:13 smithi086 ceph-mon[110975]: pgmap v933: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:15.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:15 smithi032 ceph-mon[147414]: pgmap v934: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:16.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:15 smithi086 ceph-mon[110975]: pgmap v934: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:17.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:17 smithi032 ceph-mon[147414]: pgmap v935: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:18.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:17 smithi086 ceph-mon[110975]: pgmap v935: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:19.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:19 smithi032 ceph-mon[147414]: pgmap v936: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:20.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:19 smithi086 ceph-mon[110975]: pgmap v936: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:21.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:21 smithi032 ceph-mon[147414]: pgmap v937: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:22.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:21 smithi086 ceph-mon[110975]: pgmap v937: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:22.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:51:23.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:51:23.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:23 smithi032 ceph-mon[147414]: pgmap v938: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:24.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:23 smithi086 ceph-mon[110975]: pgmap v938: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:25.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:25 smithi032 ceph-mon[147414]: pgmap v939: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:26.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:25 smithi086 ceph-mon[110975]: pgmap v939: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:27.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:27 smithi032 ceph-mon[147414]: pgmap v940: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:28.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:27 smithi086 ceph-mon[110975]: pgmap v940: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:29.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:29 smithi032 ceph-mon[147414]: pgmap v941: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:30.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:29 smithi086 ceph-mon[110975]: pgmap v941: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.985+0000 7fcee271b700 1 -- 172.21.15.32:0/3407241481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 msgr2=0x7fcedc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.985+0000 7fcee271b700 1 --2- 172.21.15.32:0/3407241481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 0x7fcedc106f00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fced0004940 tx=0x7fced0009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.985+0000 7fcee271b700 1 -- 172.21.15.32:0/3407241481 shutdown_connections 2024-04-06T14:51:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.985+0000 7fcee271b700 1 --2- 172.21.15.32:0/3407241481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 0x7fcedc106f00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.985+0000 7fcee271b700 1 --2- 172.21.15.32:0/3407241481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcedc0fde90 0x7fcedc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.985+0000 7fcee271b700 1 -- 172.21.15.32:0/3407241481 >> 172.21.15.32:0/3407241481 conn(0x7fcedc072bc0 msgr2=0x7fcedc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.985+0000 7fcee271b700 1 -- 172.21.15.32:0/3407241481 shutdown_connections 2024-04-06T14:51:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.986+0000 7fcee271b700 1 -- 172.21.15.32:0/3407241481 wait complete. 2024-04-06T14:51:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.986+0000 7fcee271b700 1 Processor -- start 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.986+0000 7fcee271b700 1 -- start start 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.986+0000 7fcee271b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcedc0fde90 0x7fcedc115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.986+0000 7fcee271b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 0x7fcedc116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.986+0000 7fcee271b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc1168c0 con 0x7fcedc0fe830 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.986+0000 7fcee271b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc10fc90 con 0x7fcedc0fde90 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcedb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 0x7fcedc116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcedb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 0x7fcedc116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36018/0 (socket says 172.21.15.32:36018) 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcedb7fe700 1 -- 172.21.15.32:0/1531601129 learned_addr learned my addr 172.21.15.32:0/1531601129 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcedbfff700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcedc0fde90 0x7fcedc115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcedb7fe700 1 -- 172.21.15.32:0/1531601129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcedc0fde90 msgr2=0x7fcedc115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcedb7fe700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcedc0fde90 0x7fcedc115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcedb7fe700 1 -- 172.21.15.32:0/1531601129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced0005040 con 0x7fcedc0fe830 2024-04-06T14:51:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcedb7fe700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 0x7fcedc116270 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fced0000c00 tx=0x7fced0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fced8ff9700 1 -- 172.21.15.32:0/1531601129 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced000e040 con 0x7fcedc0fe830 2024-04-06T14:51:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcedc10ff10 con 0x7fcedc0fe830 2024-04-06T14:51:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fced8ff9700 1 -- 172.21.15.32:0/1531601129 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fced0013420 con 0x7fcedc0fe830 2024-04-06T14:51:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fced8ff9700 1 -- 172.21.15.32:0/1531601129 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced001c630 con 0x7fcedc0fe830 2024-04-06T14:51:30.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.987+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcedc110450 con 0x7fcedc0fe830 2024-04-06T14:51:30.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.988+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcedc106a20 con 0x7fcedc0fe830 2024-04-06T14:51:30.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.989+0000 7fced8ff9700 1 -- 172.21.15.32:0/1531601129 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fced0013970 con 0x7fcedc0fe830 2024-04-06T14:51:30.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.989+0000 7fced8ff9700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcec406cc30 0x7fcec406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:30.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.989+0000 7fced8ff9700 1 -- 172.21.15.32:0/1531601129 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fced008d270 con 0x7fcedc0fe830 2024-04-06T14:51:30.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.989+0000 7fcedbfff700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcec406cc30 0x7fcec406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:30.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.990+0000 7fcedbfff700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcec406cc30 0x7fcec406f0e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcecc004990 tx=0x7fcecc009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:30.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:30.994+0000 7fced8ff9700 1 -- 172.21.15.32:0/1531601129 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fced005ad40 con 0x7fcedc0fe830 2024-04-06T14:51:31.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.215+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcedc111380 con 0x7fcec406cc30 2024-04-06T14:51:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.216+0000 7fced8ff9700 1 -- 172.21.15.32:0/1531601129 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcedc111380 con 0x7fcec406cc30 2024-04-06T14:51:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcec406cc30 msgr2=0x7fcec406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcec406cc30 0x7fcec406f0e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcecc004990 tx=0x7fcecc009ee0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 msgr2=0x7fcedc116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 0x7fcedc116270 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fced0000c00 tx=0x7fced0009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 shutdown_connections 2024-04-06T14:51:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcec406cc30 0x7fcec406f0e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcedc0fde90 0x7fcedc115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 --2- 172.21.15.32:0/1531601129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcedc0fe830 0x7fcedc116270 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 >> 172.21.15.32:0/1531601129 conn(0x7fcedc072bc0 msgr2=0x7fcedc0fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 shutdown_connections 2024-04-06T14:51:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.218+0000 7fcee271b700 1 -- 172.21.15.32:0/1531601129 wait complete. 2024-04-06T14:51:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:51:31.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.322+0000 7f44855c0700 1 -- 172.21.15.32:0/2877995848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 msgr2=0x7f4480102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.322+0000 7f44855c0700 1 --2- 172.21.15.32:0/2877995848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 0x7f4480102140 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f4474004940 tx=0x7f4474009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:31.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.322+0000 7f44855c0700 1 -- 172.21.15.32:0/2877995848 shutdown_connections 2024-04-06T14:51:31.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.322+0000 7f44855c0700 1 --2- 172.21.15.32:0/2877995848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4480102710 0x7f44800fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.322+0000 7f44855c0700 1 --2- 172.21.15.32:0/2877995848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 0x7f4480102140 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.322+0000 7f44855c0700 1 -- 172.21.15.32:0/2877995848 >> 172.21.15.32:0/2877995848 conn(0x7f4480072bc0 msgr2=0x7f4480072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:31.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.322+0000 7f44855c0700 1 -- 172.21.15.32:0/2877995848 shutdown_connections 2024-04-06T14:51:31.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.322+0000 7f44855c0700 1 -- 172.21.15.32:0/2877995848 wait complete. 2024-04-06T14:51:31.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.322+0000 7f44855c0700 1 Processor -- start 2024-04-06T14:51:31.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f44855c0700 1 -- start start 2024-04-06T14:51:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f44855c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 0x7f448010ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f44855c0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4480102710 0x7f448010ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f44855c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448010f5d0 con 0x7f4480101d70 2024-04-06T14:51:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f44855c0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448010f710 con 0x7f4480102710 2024-04-06T14:51:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f447effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 0x7f448010ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f447effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 0x7f448010ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36032/0 (socket says 172.21.15.32:36032) 2024-04-06T14:51:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f447effd700 1 -- 172.21.15.32:0/1010124648 learned_addr learned my addr 172.21.15.32:0/1010124648 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:51:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f447effd700 1 -- 172.21.15.32:0/1010124648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4480102710 msgr2=0x7f448010ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f447e7fc700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4480102710 0x7f448010ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f447effd700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4480102710 0x7f448010ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f447effd700 1 -- 172.21.15.32:0/1010124648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4474005040 con 0x7f4480101d70 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f447e7fc700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4480102710 0x7f448010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.323+0000 7f447effd700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 0x7f448010ea40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f4474009180 tx=0x7f4474009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.324+0000 7f44677fe700 1 -- 172.21.15.32:0/1010124648 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447400e040 con 0x7f4480101d70 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.324+0000 7f44677fe700 1 -- 172.21.15.32:0/1010124648 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4474004290 con 0x7f4480101d70 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.324+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4480117ab0 con 0x7f4480101d70 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.324+0000 7f44677fe700 1 -- 172.21.15.32:0/1010124648 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447401c3f0 con 0x7f4480101d70 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.324+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4480117fc0 con 0x7f4480101d70 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.325+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44800ff080 con 0x7f4480101d70 2024-04-06T14:51:31.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.325+0000 7f44677fe700 1 -- 172.21.15.32:0/1010124648 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4474013790 con 0x7f4480101d70 2024-04-06T14:51:31.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.325+0000 7f44677fe700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f446806cc20 0x7f446806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:31.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.326+0000 7f44677fe700 1 -- 172.21.15.32:0/1010124648 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f447408d330 con 0x7f4480101d70 2024-04-06T14:51:31.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.326+0000 7f447e7fc700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f446806cc20 0x7f446806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:31.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.326+0000 7f447e7fc700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f446806cc20 0x7f446806f0d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4470007fd0 tx=0x7f4470009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:31.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.330+0000 7f44677fe700 1 -- 172.21.15.32:0/1010124648 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4474057370 con 0x7f4480101d70 2024-04-06T14:51:31.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.552+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4480108dd0 con 0x7f446806cc20 2024-04-06T14:51:31.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.553+0000 7f44677fe700 1 -- 172.21.15.32:0/1010124648 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4470004920 con 0x7f446806cc20 2024-04-06T14:51:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f446806cc20 msgr2=0x7f446806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f446806cc20 0x7f446806f0d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4470007fd0 tx=0x7f4470009040 comp rx=0 tx=0).stop 2024-04-06T14:51:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 msgr2=0x7f448010ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 0x7f448010ea40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f4474009180 tx=0x7f4474009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 shutdown_connections 2024-04-06T14:51:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f446806cc20 0x7f446806f0d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4480101d70 0x7f448010ea40 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 --2- 172.21.15.32:0/1010124648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4480102710 0x7f448010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 >> 172.21.15.32:0/1010124648 conn(0x7f4480072bc0 msgr2=0x7f44800fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 shutdown_connections 2024-04-06T14:51:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.555+0000 7f44855c0700 1 -- 172.21.15.32:0/1010124648 wait complete. 2024-04-06T14:51:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.658+0000 7f41f3e50700 1 -- 172.21.15.32:0/1171390591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 msgr2=0x7f41ec0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.658+0000 7f41f3e50700 1 --2- 172.21.15.32:0/1171390591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 0x7f41ec0fbd80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f41e0004940 tx=0x7f41e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.658+0000 7f41f3e50700 1 -- 172.21.15.32:0/1171390591 shutdown_connections 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.658+0000 7f41f3e50700 1 --2- 172.21.15.32:0/1171390591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 0x7f41ec0fbd80 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.658+0000 7f41f3e50700 1 --2- 172.21.15.32:0/1171390591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41ec101d70 0x7f41ec102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.658+0000 7f41f3e50700 1 -- 172.21.15.32:0/1171390591 >> 172.21.15.32:0/1171390591 conn(0x7f41ec072bc0 msgr2=0x7f41ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.658+0000 7f41f3e50700 1 -- 172.21.15.32:0/1171390591 shutdown_connections 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.658+0000 7f41f3e50700 1 -- 172.21.15.32:0/1171390591 wait complete. 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f3e50700 1 Processor -- start 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f3e50700 1 -- start start 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f3e50700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41ec101d70 0x7f41ec10ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f3e50700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 0x7f41ec10f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f3e50700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec10f640 con 0x7f41ec102710 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f3e50700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec10f780 con 0x7f41ec101d70 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f13eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 0x7f41ec10f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f13eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 0x7f41ec10f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36050/0 (socket says 172.21.15.32:36050) 2024-04-06T14:51:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f13eb700 1 -- 172.21.15.32:0/800436248 learned_addr learned my addr 172.21.15.32:0/800436248 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:51:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.659+0000 7f41f1bec700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41ec101d70 0x7f41ec10ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.660+0000 7f41f13eb700 1 -- 172.21.15.32:0/800436248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41ec101d70 msgr2=0x7f41ec10ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.660+0000 7f41f13eb700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41ec101d70 0x7f41ec10ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.660+0000 7f41f13eb700 1 -- 172.21.15.32:0/800436248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e0005040 con 0x7f41ec102710 2024-04-06T14:51:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.660+0000 7f41f13eb700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 0x7f41ec10f100 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009390 tx=0x7f41e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.660+0000 7f41de7fc700 1 -- 172.21.15.32:0/800436248 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41e000e040 con 0x7f41ec102710 2024-04-06T14:51:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.660+0000 7f41de7fc700 1 -- 172.21.15.32:0/800436248 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41e0013420 con 0x7f41ec102710 2024-04-06T14:51:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.660+0000 7f41de7fc700 1 -- 172.21.15.32:0/800436248 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41e001c630 con 0x7f41ec102710 2024-04-06T14:51:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.660+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec117a70 con 0x7f41ec102710 2024-04-06T14:51:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.660+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec117fd0 con 0x7f41ec102710 2024-04-06T14:51:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.661+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ec0ff080 con 0x7f41ec102710 2024-04-06T14:51:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.661+0000 7f41de7fc700 1 -- 172.21.15.32:0/800436248 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f41e0013a00 con 0x7f41ec102710 2024-04-06T14:51:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.662+0000 7f41de7fc700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41d806cc30 0x7f41d806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.662+0000 7f41de7fc700 1 -- 172.21.15.32:0/800436248 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f41e008d030 con 0x7f41ec102710 2024-04-06T14:51:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.662+0000 7f41f1bec700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41d806cc30 0x7f41d806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:31.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.662+0000 7f41f1bec700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41d806cc30 0x7f41d806f0e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f41e8004f40 tx=0x7f41e800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:31.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.666+0000 7f41de7fc700 1 -- 172.21.15.32:0/800436248 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f41e0057130 con 0x7f41ec102710 2024-04-06T14:51:31.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.879+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f41ec108d10 con 0x7f41d806cc30 2024-04-06T14:51:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.889+0000 7f41de7fc700 1 -- 172.21.15.32:0/800436248 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f41ec108d10 con 0x7f41d806cc30 2024-04-06T14:51:31.877 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:51:31.877 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (31m) 9m ago 43m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:51:31.877 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (29m) 9m ago 43m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:51:31.877 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (29m) 9m ago 42m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:51:31.877 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (31m) 9m ago 42m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:51:31.877 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (25m) 9m ago 37m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:51:31.877 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (25m) 9m ago 37m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:51:31.877 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (25m) 9m ago 37m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:51:31.877 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (9m) 9m ago 37m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (33m) 9m ago 44m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (32m) 9m ago 42m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (30m) 9m ago 44m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (29m) 9m ago 42m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (32m) 9m ago 43m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (32m) 9m ago 42m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (28m) 9m ago 41m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (28m) 9m ago 40m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (27m) 9m ago 40m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (26m) 9m ago 39m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (26m) 9m ago 39m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (25m) 9m ago 39m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:51:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (32m) 9m ago 42m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:51:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41d806cc30 msgr2=0x7f41d806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41d806cc30 0x7f41d806f0e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f41e8004f40 tx=0x7f41e800b040 comp rx=0 tx=0).stop 2024-04-06T14:51:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 msgr2=0x7f41ec10f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 0x7f41ec10f100 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009390 tx=0x7f41e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 shutdown_connections 2024-04-06T14:51:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f41d806cc30 0x7f41d806f0e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41ec101d70 0x7f41ec10ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 --2- 172.21.15.32:0/800436248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ec102710 0x7f41ec10f100 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 >> 172.21.15.32:0/800436248 conn(0x7f41ec072bc0 msgr2=0x7f41ec0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 shutdown_connections 2024-04-06T14:51:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.891+0000 7f41f3e50700 1 -- 172.21.15.32:0/800436248 wait complete. 2024-04-06T14:51:31.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:31 smithi032 ceph-mon[147414]: pgmap v942: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:31.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.993+0000 7fa96340c700 1 -- 172.21.15.32:0/2310319184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fe830 msgr2=0x7fa95c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.993+0000 7fa96340c700 1 --2- 172.21.15.32:0/2310319184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fe830 0x7fa95c106f00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa950004940 tx=0x7fa950009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:31.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.993+0000 7fa96340c700 1 -- 172.21.15.32:0/2310319184 shutdown_connections 2024-04-06T14:51:31.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.993+0000 7fa96340c700 1 --2- 172.21.15.32:0/2310319184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fe830 0x7fa95c106f00 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.993+0000 7fa96340c700 1 --2- 172.21.15.32:0/2310319184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95c0fde90 0x7fa95c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.993+0000 7fa96340c700 1 -- 172.21.15.32:0/2310319184 >> 172.21.15.32:0/2310319184 conn(0x7fa95c072bc0 msgr2=0x7fa95c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:31.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.993+0000 7fa96340c700 1 -- 172.21.15.32:0/2310319184 shutdown_connections 2024-04-06T14:51:31.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.993+0000 7fa96340c700 1 -- 172.21.15.32:0/2310319184 wait complete. 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa96340c700 1 Processor -- start 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa96340c700 1 -- start start 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa96340c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fde90 0x7fa95c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa96340c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95c0fe830 0x7fa95c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa96340c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0fc820 con 0x7fa95c0fde90 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa96340c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0fc960 con 0x7fa95c0fe830 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa9611a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fde90 0x7fa95c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa9611a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fde90 0x7fa95c0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36062/0 (socket says 172.21.15.32:36062) 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa9611a8700 1 -- 172.21.15.32:0/729237032 learned_addr learned my addr 172.21.15.32:0/729237032 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:51:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa9611a8700 1 -- 172.21.15.32:0/729237032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95c0fe830 msgr2=0x7fa95c0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:31.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa9611a8700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95c0fe830 0x7fa95c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:31.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa9611a8700 1 -- 172.21.15.32:0/729237032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa950005040 con 0x7fa95c0fde90 2024-04-06T14:51:31.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.994+0000 7fa9611a8700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fde90 0x7fa95c0fbd50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa95800eb40 tx=0x7fa95800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:31.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.995+0000 7fa94dffb700 1 -- 172.21.15.32:0/729237032 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa958007cc0 con 0x7fa95c0fde90 2024-04-06T14:51:31.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.995+0000 7fa94dffb700 1 -- 172.21.15.32:0/729237032 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa95800f040 con 0x7fa95c0fde90 2024-04-06T14:51:31.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.995+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c0fcbb0 con 0x7fa95c0fde90 2024-04-06T14:51:31.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.995+0000 7fa94dffb700 1 -- 172.21.15.32:0/729237032 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa958009570 con 0x7fa95c0fde90 2024-04-06T14:51:31.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.995+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c11a130 con 0x7fa95c0fde90 2024-04-06T14:51:31.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.996+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c065e30 con 0x7fa95c0fde90 2024-04-06T14:51:31.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.996+0000 7fa94dffb700 1 -- 172.21.15.32:0/729237032 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa9580096d0 con 0x7fa95c0fde90 2024-04-06T14:51:31.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.996+0000 7fa94dffb700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa94806cc30 0x7fa94806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:31.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.997+0000 7fa94dffb700 1 -- 172.21.15.32:0/729237032 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa95808bc10 con 0x7fa95c0fde90 2024-04-06T14:51:31.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.997+0000 7fa9609a7700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa94806cc30 0x7fa94806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:31.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:31.997+0000 7fa9609a7700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa94806cc30 0x7fa94806f0e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa95000b0d0 tx=0x7fa950009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:31.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.001+0000 7fa94dffb700 1 -- 172.21.15.32:0/729237032 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa958055d00 con 0x7fa95c0fde90 2024-04-06T14:51:32.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:31 smithi086 ceph-mon[110975]: pgmap v942: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.294+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa95c04bed0 con 0x7fa95c0fde90 2024-04-06T14:51:32.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.294+0000 7fa94dffb700 1 -- 172.21.15.32:0/729237032 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fa958059320 con 0x7fa95c0fde90 2024-04-06T14:51:32.282 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:51:32.283 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:51:32.284 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:51:32.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa94806cc30 msgr2=0x7fa94806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa94806cc30 0x7fa94806f0e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa95000b0d0 tx=0x7fa950009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fde90 msgr2=0x7fa95c0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fde90 0x7fa95c0fbd50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa95800eb40 tx=0x7fa95800b040 comp rx=0 tx=0).stop 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 shutdown_connections 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa94806cc30 0x7fa94806f0e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95c0fde90 0x7fa95c0fbd50 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 --2- 172.21.15.32:0/729237032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95c0fe830 0x7fa95c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 >> 172.21.15.32:0/729237032 conn(0x7fa95c072bc0 msgr2=0x7fa95c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 shutdown_connections 2024-04-06T14:51:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.296+0000 7fa96340c700 1 -- 172.21.15.32:0/729237032 wait complete. 2024-04-06T14:51:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.399+0000 7f9cde9e8700 1 -- 172.21.15.32:0/93158987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 msgr2=0x7f9cd80fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.400+0000 7f9cde9e8700 1 --2- 172.21.15.32:0/93158987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 0x7f9cd80fce20 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc004940 tx=0x7f9ccc009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.400+0000 7f9cde9e8700 1 -- 172.21.15.32:0/93158987 shutdown_connections 2024-04-06T14:51:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.400+0000 7f9cde9e8700 1 --2- 172.21.15.32:0/93158987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 0x7f9cd80fce20 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.400+0000 7f9cde9e8700 1 --2- 172.21.15.32:0/93158987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd8102e10 0x7f9cd81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.400+0000 7f9cde9e8700 1 -- 172.21.15.32:0/93158987 >> 172.21.15.32:0/93158987 conn(0x7f9cd80f8980 msgr2=0x7f9cd80fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.400+0000 7f9cde9e8700 1 -- 172.21.15.32:0/93158987 shutdown_connections 2024-04-06T14:51:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.400+0000 7f9cde9e8700 1 -- 172.21.15.32:0/93158987 wait complete. 2024-04-06T14:51:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.400+0000 7f9cde9e8700 1 Processor -- start 2024-04-06T14:51:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.400+0000 7f9cde9e8700 1 -- start start 2024-04-06T14:51:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cde9e8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd8102e10 0x7f9cd8113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cde9e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 0x7f9cd8114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cde9e8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd8114680 con 0x7f9cd81037b0 2024-04-06T14:51:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cde9e8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd810da70 con 0x7f9cd8102e10 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 0x7f9cd8114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 0x7f9cd8114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36082/0 (socket says 172.21.15.32:36082) 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd77fe700 1 -- 172.21.15.32:0/1356979014 learned_addr learned my addr 172.21.15.32:0/1356979014 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd7fff700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd8102e10 0x7f9cd8113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd77fe700 1 -- 172.21.15.32:0/1356979014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd8102e10 msgr2=0x7f9cd8113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd77fe700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd8102e10 0x7f9cd8113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd77fe700 1 -- 172.21.15.32:0/1356979014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc005040 con 0x7f9cd81037b0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd77fe700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 0x7f9cd8114030 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009380 tx=0x7f9ccc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd4ff9700 1 -- 172.21.15.32:0/1356979014 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ccc00e040 con 0x7f9cd81037b0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd810dcf0 con 0x7f9cd81037b0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd4ff9700 1 -- 172.21.15.32:0/1356979014 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ccc013420 con 0x7f9cd81037b0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cd4ff9700 1 -- 172.21.15.32:0/1356979014 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ccc01c630 con 0x7f9cd81037b0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.401+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd810e2b0 con 0x7f9cd81037b0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.402+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd80fe560 con 0x7f9cd81037b0 2024-04-06T14:51:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.403+0000 7f9cd4ff9700 1 -- 172.21.15.32:0/1356979014 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9ccc0139e0 con 0x7f9cd81037b0 2024-04-06T14:51:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.403+0000 7f9cd4ff9700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9cc006cc20 0x7f9cc006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.403+0000 7f9cd4ff9700 1 -- 172.21.15.32:0/1356979014 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9ccc08d220 con 0x7f9cd81037b0 2024-04-06T14:51:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.403+0000 7f9cd7fff700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9cc006cc20 0x7f9cc006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.404+0000 7f9cd7fff700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9cc006cc20 0x7f9cc006f0d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8007fd0 tx=0x7f9cc8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.408+0000 7f9cd4ff9700 1 -- 172.21.15.32:0/1356979014 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9ccc05acf0 con 0x7f9cd81037b0 2024-04-06T14:51:32.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.657+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9cd8109db0 con 0x7f9cd81037b0 2024-04-06T14:51:32.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.658+0000 7f9cd4ff9700 1 -- 172.21.15.32:0/1356979014 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f9ccc05a880 con 0x7f9cd81037b0 2024-04-06T14:51:32.646 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:51:32.646 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:51:32.646 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:51:32.646 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:51:32.646 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:51:32.646 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:51:32.646 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:51:32.647 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:51:32.648 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:51:32.649 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:51:32.649 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:51:32.649 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:51:32.649 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:51:32.649 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:51:32.649 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:51:32.649 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:51:32.649 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:51:32.649 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:51:32.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9cc006cc20 msgr2=0x7f9cc006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9cc006cc20 0x7f9cc006f0d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8007fd0 tx=0x7f9cc8009040 comp rx=0 tx=0).stop 2024-04-06T14:51:32.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 msgr2=0x7f9cd8114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 0x7f9cd8114030 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009380 tx=0x7f9ccc009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 shutdown_connections 2024-04-06T14:51:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9cc006cc20 0x7f9cc006f0d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd8102e10 0x7f9cd8113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 --2- 172.21.15.32:0/1356979014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cd81037b0 0x7f9cd8114030 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 >> 172.21.15.32:0/1356979014 conn(0x7f9cd80f8980 msgr2=0x7f9cd80f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 shutdown_connections 2024-04-06T14:51:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.660+0000 7f9cde9e8700 1 -- 172.21.15.32:0/1356979014 wait complete. 2024-04-06T14:51:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:51:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.764+0000 7eff093b5700 1 -- 172.21.15.32:0/3654218724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff04102e00 msgr2=0x7eff041031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.764+0000 7eff093b5700 1 --2- 172.21.15.32:0/3654218724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff04102e00 0x7eff041031d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7efef4004940 tx=0x7efef4009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.764+0000 7eff093b5700 1 -- 172.21.15.32:0/3654218724 shutdown_connections 2024-04-06T14:51:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.764+0000 7eff093b5700 1 --2- 172.21.15.32:0/3654218724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff041037a0 0x7eff040fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.764+0000 7eff093b5700 1 --2- 172.21.15.32:0/3654218724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff04102e00 0x7eff041031d0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.764+0000 7eff093b5700 1 -- 172.21.15.32:0/3654218724 >> 172.21.15.32:0/3654218724 conn(0x7eff040f89b0 msgr2=0x7eff040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.764+0000 7eff093b5700 1 -- 172.21.15.32:0/3654218724 shutdown_connections 2024-04-06T14:51:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.764+0000 7eff093b5700 1 -- 172.21.15.32:0/3654218724 wait complete. 2024-04-06T14:51:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff093b5700 1 Processor -- start 2024-04-06T14:51:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff093b5700 1 -- start start 2024-04-06T14:51:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff093b5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff04102e00 0x7eff041138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff093b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff041037a0 0x7eff04113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff093b5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff04114440 con 0x7eff041037a0 2024-04-06T14:51:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff093b5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0410da40 con 0x7eff04102e00 2024-04-06T14:51:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff027fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff041037a0 0x7eff04113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff027fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff041037a0 0x7eff04113df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36096/0 (socket says 172.21.15.32:36096) 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff02ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff04102e00 0x7eff041138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.765+0000 7eff027fc700 1 -- 172.21.15.32:0/492757405 learned_addr learned my addr 172.21.15.32:0/492757405 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.766+0000 7eff027fc700 1 -- 172.21.15.32:0/492757405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff04102e00 msgr2=0x7eff041138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.766+0000 7eff027fc700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff04102e00 0x7eff041138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.766+0000 7eff027fc700 1 -- 172.21.15.32:0/492757405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef4005040 con 0x7eff041037a0 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.766+0000 7eff027fc700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff041037a0 0x7eff04113df0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7efef8002a50 tx=0x7efef8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.766+0000 7efeeb7fe700 1 -- 172.21.15.32:0/492757405 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efef8007cc0 con 0x7eff041037a0 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.766+0000 7efeeb7fe700 1 -- 172.21.15.32:0/492757405 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efef800f040 con 0x7eff041037a0 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.766+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0410dd20 con 0x7eff041037a0 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.766+0000 7efeeb7fe700 1 -- 172.21.15.32:0/492757405 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efef800d920 con 0x7eff041037a0 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.766+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0410e2e0 con 0x7eff041037a0 2024-04-06T14:51:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.767+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff040fe550 con 0x7eff041037a0 2024-04-06T14:51:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.768+0000 7efeeb7fe700 1 -- 172.21.15.32:0/492757405 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efef8009680 con 0x7eff041037a0 2024-04-06T14:51:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.768+0000 7efeeb7fe700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efeec06cc20 0x7efeec06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.768+0000 7efeeb7fe700 1 -- 172.21.15.32:0/492757405 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efef808c370 con 0x7eff041037a0 2024-04-06T14:51:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.768+0000 7eff02ffd700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efeec06cc20 0x7efeec06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.769+0000 7eff02ffd700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efeec06cc20 0x7efeec06f0d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7efef4004f90 tx=0x7efef4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.772+0000 7efeeb7fe700 1 -- 172.21.15.32:0/492757405 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efef8059f00 con 0x7eff041037a0 2024-04-06T14:51:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.993+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff04109e60 con 0x7efeec06cc20 2024-04-06T14:51:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.994+0000 7efeeb7fe700 1 -- 172.21.15.32:0/492757405 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7efef4007d50 con 0x7efeec06cc20 2024-04-06T14:51:32.982 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:51:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:51:32.984 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:51:32.984 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:51:32.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efeec06cc20 msgr2=0x7efeec06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efeec06cc20 0x7efeec06f0d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7efef4004f90 tx=0x7efef4009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:32.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff041037a0 msgr2=0x7eff04113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:32.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff041037a0 0x7eff04113df0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7efef8002a50 tx=0x7efef8004600 comp rx=0 tx=0).stop 2024-04-06T14:51:32.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 shutdown_connections 2024-04-06T14:51:32.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efeec06cc20 0x7efeec06f0d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff04102e00 0x7eff041138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 --2- 172.21.15.32:0/492757405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff041037a0 0x7eff04113df0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:32.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 >> 172.21.15.32:0/492757405 conn(0x7eff040f89b0 msgr2=0x7eff040f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:32.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 shutdown_connections 2024-04-06T14:51:32.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:32.996+0000 7eff093b5700 1 -- 172.21.15.32:0/492757405 wait complete. 2024-04-06T14:51:33.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.099+0000 7f53f3def700 1 -- 172.21.15.32:0/2141020801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec1037b0 msgr2=0x7f53ec0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:33.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.099+0000 7f53f3def700 1 --2- 172.21.15.32:0/2141020801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec1037b0 0x7f53ec0fce20 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f53e0004940 tx=0x7f53e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:33.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.099+0000 7f53f3def700 1 -- 172.21.15.32:0/2141020801 shutdown_connections 2024-04-06T14:51:33.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.099+0000 7f53f3def700 1 --2- 172.21.15.32:0/2141020801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec1037b0 0x7f53ec0fce20 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.099+0000 7f53f3def700 1 --2- 172.21.15.32:0/2141020801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53ec102e10 0x7f53ec1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.099+0000 7f53f3def700 1 -- 172.21.15.32:0/2141020801 >> 172.21.15.32:0/2141020801 conn(0x7f53ec0f8980 msgr2=0x7f53ec0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.099+0000 7f53f3def700 1 -- 172.21.15.32:0/2141020801 shutdown_connections 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.099+0000 7f53f3def700 1 -- 172.21.15.32:0/2141020801 wait complete. 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f3def700 1 Processor -- start 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f3def700 1 -- start start 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f3def700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec102e10 0x7f53ec113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f3def700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53ec1037b0 0x7f53ec114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f3def700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec114670 con 0x7f53ec102e10 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f3def700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec10da60 con 0x7f53ec1037b0 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f1b8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec102e10 0x7f53ec113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f1b8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec102e10 0x7f53ec113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36106/0 (socket says 172.21.15.32:36106) 2024-04-06T14:51:33.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f1b8b700 1 -- 172.21.15.32:0/3408912557 learned_addr learned my addr 172.21.15.32:0/3408912557 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.100+0000 7f53f138a700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53ec1037b0 0x7f53ec114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53f1b8b700 1 -- 172.21.15.32:0/3408912557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53ec1037b0 msgr2=0x7f53ec114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53f1b8b700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53ec1037b0 0x7f53ec114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53f1b8b700 1 -- 172.21.15.32:0/3408912557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e0005040 con 0x7f53ec102e10 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53f138a700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53ec1037b0 0x7f53ec114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53f1b8b700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec102e10 0x7f53ec113ae0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f53e800c830 tx=0x7f53e8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53de7fc700 1 -- 172.21.15.32:0/3408912557 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e800a760 con 0x7f53ec102e10 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53de7fc700 1 -- 172.21.15.32:0/3408912557 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53e800ad70 con 0x7f53ec102e10 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ec10dd40 con 0x7f53ec102e10 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53de7fc700 1 -- 172.21.15.32:0/3408912557 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e80113a0 con 0x7f53ec102e10 2024-04-06T14:51:33.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.101+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ec10e300 con 0x7f53ec102e10 2024-04-06T14:51:33.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.102+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ec0fe560 con 0x7f53ec102e10 2024-04-06T14:51:33.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.102+0000 7f53de7fc700 1 -- 172.21.15.32:0/3408912557 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f53e8012400 con 0x7f53ec102e10 2024-04-06T14:51:33.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.103+0000 7f53de7fc700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53d806cc20 0x7f53d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:51:33.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.103+0000 7f53de7fc700 1 -- 172.21.15.32:0/3408912557 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f53e808c0b0 con 0x7f53ec102e10 2024-04-06T14:51:33.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.103+0000 7f53f138a700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53d806cc20 0x7f53d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:51:33.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.103+0000 7f53f138a700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53d806cc20 0x7f53d806f0d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f53ec10ec20 tx=0x7f53e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:51:33.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.107+0000 7f53de7fc700 1 -- 172.21.15.32:0/3408912557 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f53e8059b80 con 0x7f53ec102e10 2024-04-06T14:51:33.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:32 smithi086 ceph-mon[110975]: from='client.35710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:33.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:32 smithi086 ceph-mon[110975]: from='client.35714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:33.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:32 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/729237032' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:51:33.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:32 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1356979014' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:51:33.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:32 smithi032 ceph-mon[147414]: from='client.35710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:33.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:32 smithi032 ceph-mon[147414]: from='client.35714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:33.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:32 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/729237032' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:51:33.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:32 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1356979014' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:51:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.397+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f53ec109db0 con 0x7f53ec102e10 2024-04-06T14:51:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.397+0000 7f53de7fc700 1 -- 172.21.15.32:0/3408912557 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f53e8059710 con 0x7f53ec102e10 2024-04-06T14:51:33.385 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:51:33.385 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:51:33.386 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:51:33.387 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:51:33.388 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:51:33.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53d806cc20 msgr2=0x7f53d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:33.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53d806cc20 0x7f53d806f0d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f53ec10ec20 tx=0x7f53e0009e30 comp rx=0 tx=0).stop 2024-04-06T14:51:33.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec102e10 msgr2=0x7f53ec113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:51:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec102e10 0x7f53ec113ae0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f53e800c830 tx=0x7f53e8004470 comp rx=0 tx=0).stop 2024-04-06T14:51:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 shutdown_connections 2024-04-06T14:51:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f53d806cc20 0x7f53d806f0d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53ec102e10 0x7f53ec113ae0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 --2- 172.21.15.32:0/3408912557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53ec1037b0 0x7f53ec114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:51:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 >> 172.21.15.32:0/3408912557 conn(0x7f53ec0f8980 msgr2=0x7f53ec0f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:51:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 shutdown_connections 2024-04-06T14:51:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:51:33.400+0000 7f53f3def700 1 -- 172.21.15.32:0/3408912557 wait complete. 2024-04-06T14:51:34.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:33 smithi086 ceph-mon[110975]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:34.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:33 smithi086 ceph-mon[110975]: pgmap v943: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:34.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:33 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3408912557' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:51:34.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:33 smithi032 ceph-mon[147414]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:34.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:33 smithi032 ceph-mon[147414]: pgmap v943: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:34.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:33 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3408912557' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:51:35.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:34 smithi086 ceph-mon[110975]: from='client.35730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:35.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:34 smithi032 ceph-mon[147414]: from='client.35730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:51:36.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:35 smithi086 ceph-mon[110975]: pgmap v944: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:36.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:35 smithi032 ceph-mon[147414]: pgmap v944: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:37.116 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:36 smithi086 ceph-mon[110975]: pgmap v945: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:37.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:36 smithi032 ceph-mon[147414]: pgmap v945: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:38.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:51:38.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:51:39.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:38 smithi086 ceph-mon[110975]: pgmap v946: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:39.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:38 smithi032 ceph-mon[147414]: pgmap v946: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:41.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:41 smithi086 ceph-mon[110975]: pgmap v947: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:41.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:41 smithi032 ceph-mon[147414]: pgmap v947: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:43.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:43 smithi086 ceph-mon[110975]: pgmap v948: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:43.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:43 smithi032 ceph-mon[147414]: pgmap v948: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:45.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:45 smithi086 ceph-mon[110975]: pgmap v949: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:45.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:45 smithi032 ceph-mon[147414]: pgmap v949: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:47.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:47 smithi086 ceph-mon[110975]: pgmap v950: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:47.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:47 smithi032 ceph-mon[147414]: pgmap v950: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:49.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:49 smithi086 ceph-mon[110975]: pgmap v951: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:49.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:49 smithi032 ceph-mon[147414]: pgmap v951: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:51.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:51 smithi086 ceph-mon[110975]: pgmap v952: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:51.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:51 smithi032 ceph-mon[147414]: pgmap v952: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:53.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:53 smithi086 ceph-mon[110975]: pgmap v953: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:53.616 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:51:53.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:53 smithi032 ceph-mon[147414]: pgmap v953: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:53.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:51:55.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:55 smithi086 ceph-mon[110975]: pgmap v954: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:55.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:55 smithi032 ceph-mon[147414]: pgmap v954: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:57.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:57 smithi086 ceph-mon[110975]: pgmap v955: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:57.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:57 smithi032 ceph-mon[147414]: pgmap v955: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:59.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:51:59 smithi032 ceph-mon[147414]: pgmap v956: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:51:59.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:51:59 smithi086 ceph-mon[110975]: pgmap v956: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:01.719 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:01 smithi032 ceph-mon[147414]: pgmap v957: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:01.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:01 smithi086 ceph-mon[110975]: pgmap v957: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:03.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.517+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3111024917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24101d70 msgr2=0x7f2c24102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:03.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.517+0000 7f2c2ab4d700 1 --2- 172.21.15.32:0/3111024917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24101d70 0x7f2c24102140 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2c18004940 tx=0x7f2c18009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:03.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.517+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3111024917 shutdown_connections 2024-04-06T14:52:03.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.517+0000 7f2c2ab4d700 1 --2- 172.21.15.32:0/3111024917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c24102710 0x7f2c240fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:03.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.517+0000 7f2c2ab4d700 1 --2- 172.21.15.32:0/3111024917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24101d70 0x7f2c24102140 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.517+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3111024917 >> 172.21.15.32:0/3111024917 conn(0x7f2c24072bc0 msgr2=0x7f2c24072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.517+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3111024917 shutdown_connections 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.517+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3111024917 wait complete. 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c2ab4d700 1 Processor -- start 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c2ab4d700 1 -- start start 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c2ab4d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c24101d70 0x7f2c241129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c2ab4d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24102710 0x7f2c24112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c2ab4d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c24113530 con 0x7f2c24102710 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c2ab4d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2410c920 con 0x7f2c24101d70 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c23fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24102710 0x7f2c24112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c23fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24102710 0x7f2c24112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36594/0 (socket says 172.21.15.32:36594) 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c23fff700 1 -- 172.21.15.32:0/3124380323 learned_addr learned my addr 172.21.15.32:0/3124380323 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.518+0000 7f2c288e9700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c24101d70 0x7f2c241129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.519+0000 7f2c23fff700 1 -- 172.21.15.32:0/3124380323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c24101d70 msgr2=0x7f2c241129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:03.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.519+0000 7f2c23fff700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c24101d70 0x7f2c241129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:03.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.519+0000 7f2c23fff700 1 -- 172.21.15.32:0/3124380323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c18005040 con 0x7f2c24102710 2024-04-06T14:52:03.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.519+0000 7f2c23fff700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24102710 0x7f2c24112ee0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000ed20 tx=0x7f2c10007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:03.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.519+0000 7f2c217fa700 1 -- 172.21.15.32:0/3124380323 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1000cd10 con 0x7f2c24102710 2024-04-06T14:52:03.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.519+0000 7f2c217fa700 1 -- 172.21.15.32:0/3124380323 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c1000ce70 con 0x7f2c24102710 2024-04-06T14:52:03.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.519+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2410cba0 con 0x7f2c24102710 2024-04-06T14:52:03.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.519+0000 7f2c217fa700 1 -- 172.21.15.32:0/3124380323 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c100107d0 con 0x7f2c24102710 2024-04-06T14:52:03.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.519+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2410d160 con 0x7f2c24102710 2024-04-06T14:52:03.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.520+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c240fd4c0 con 0x7f2c24102710 2024-04-06T14:52:03.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.521+0000 7f2c217fa700 1 -- 172.21.15.32:0/3124380323 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2c10010990 con 0x7f2c24102710 2024-04-06T14:52:03.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.521+0000 7f2c217fa700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2c1406cc20 0x7f2c1406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:03.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.521+0000 7f2c217fa700 1 -- 172.21.15.32:0/3124380323 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2c1008e900 con 0x7f2c24102710 2024-04-06T14:52:03.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.521+0000 7f2c288e9700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2c1406cc20 0x7f2c1406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:03.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.522+0000 7f2c288e9700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2c1406cc20 0x7f2c1406f0d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2c18004f90 tx=0x7f2c18009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:03.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.526+0000 7f2c217fa700 1 -- 172.21.15.32:0/3124380323 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2c1005c3d0 con 0x7f2c24102710 2024-04-06T14:52:03.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:03 smithi032 ceph-mon[147414]: pgmap v958: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:03.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.751+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c24112670 con 0x7f2c1406cc20 2024-04-06T14:52:03.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.753+0000 7f2c217fa700 1 -- 172.21.15.32:0/3124380323 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2c24112670 con 0x7f2c1406cc20 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2c1406cc20 msgr2=0x7f2c1406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2c1406cc20 0x7f2c1406f0d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2c18004f90 tx=0x7f2c18009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24102710 msgr2=0x7f2c24112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24102710 0x7f2c24112ee0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000ed20 tx=0x7f2c10007f90 comp rx=0 tx=0).stop 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 shutdown_connections 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2c1406cc20 0x7f2c1406f0d0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c24101d70 0x7f2c241129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 --2- 172.21.15.32:0/3124380323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c24102710 0x7f2c24112ee0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 >> 172.21.15.32:0/3124380323 conn(0x7f2c24072bc0 msgr2=0x7f2c240fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 shutdown_connections 2024-04-06T14:52:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.755+0000 7f2c2ab4d700 1 -- 172.21.15.32:0/3124380323 wait complete. 2024-04-06T14:52:03.753 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:52:03.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 -- 172.21.15.32:0/1349614633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 msgr2=0x7f8180102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:03.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 --2- 172.21.15.32:0/1349614633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 0x7f8180102140 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8170004940 tx=0x7f8170009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:03.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 -- 172.21.15.32:0/1349614633 shutdown_connections 2024-04-06T14:52:03.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 --2- 172.21.15.32:0/1349614633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8180102710 0x7f81800fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 --2- 172.21.15.32:0/1349614633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 0x7f8180102140 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 -- 172.21.15.32:0/1349614633 >> 172.21.15.32:0/1349614633 conn(0x7f8180072bc0 msgr2=0x7f8180072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 -- 172.21.15.32:0/1349614633 shutdown_connections 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 -- 172.21.15.32:0/1349614633 wait complete. 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 Processor -- start 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.862+0000 7f81862fc700 1 -- start start 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f81862fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 0x7f8180113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f81862fc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8180102710 0x7f8180113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f81862fc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81801145a0 con 0x7f8180101d70 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f81862fc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818010d990 con 0x7f8180102710 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 0x7f8180113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 0x7f8180113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36626/0 (socket says 172.21.15.32:36626) 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817ffff700 1 -- 172.21.15.32:0/3283318158 learned_addr learned my addr 172.21.15.32:0/3283318158 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f8177fff700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8180102710 0x7f8180113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817ffff700 1 -- 172.21.15.32:0/3283318158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8180102710 msgr2=0x7f8180113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817ffff700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8180102710 0x7f8180113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817ffff700 1 -- 172.21.15.32:0/3283318158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8170005040 con 0x7f8180101d70 2024-04-06T14:52:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817ffff700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 0x7f8180113a10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8170000c00 tx=0x7f8170009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817d7fa700 1 -- 172.21.15.32:0/3283318158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8170021070 con 0x7f8180101d70 2024-04-06T14:52:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817d7fa700 1 -- 172.21.15.32:0/3283318158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81700093b0 con 0x7f8180101d70 2024-04-06T14:52:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818010dc10 con 0x7f8180101d70 2024-04-06T14:52:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f817d7fa700 1 -- 172.21.15.32:0/3283318158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f817000e070 con 0x7f8180101d70 2024-04-06T14:52:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.863+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818010e1d0 con 0x7f8180101d70 2024-04-06T14:52:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.864+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81800fd4c0 con 0x7f8180101d70 2024-04-06T14:52:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.865+0000 7f817d7fa700 1 -- 172.21.15.32:0/3283318158 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f817001c490 con 0x7f8180101d70 2024-04-06T14:52:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.865+0000 7f817d7fa700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f816c06cc20 0x7f816c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.865+0000 7f817d7fa700 1 -- 172.21.15.32:0/3283318158 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f817008d000 con 0x7f8180101d70 2024-04-06T14:52:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.865+0000 7f8177fff700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f816c06cc20 0x7f816c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.866+0000 7f8177fff700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f816c06cc20 0x7f816c06f0d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f818010eb50 tx=0x7f816800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:03.870+0000 7f817d7fa700 1 -- 172.21.15.32:0/3283318158 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f817005ab90 con 0x7f8180101d70 2024-04-06T14:52:03.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:03 smithi086 ceph-mon[110975]: pgmap v958: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.096+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8180108da0 con 0x7f816c06cc20 2024-04-06T14:52:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.097+0000 7f817d7fa700 1 -- 172.21.15.32:0/3283318158 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8180108da0 con 0x7f816c06cc20 2024-04-06T14:52:04.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f816c06cc20 msgr2=0x7f816c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f816c06cc20 0x7f816c06f0d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f818010eb50 tx=0x7f816800b040 comp rx=0 tx=0).stop 2024-04-06T14:52:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 msgr2=0x7f8180113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 0x7f8180113a10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8170000c00 tx=0x7f8170009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 shutdown_connections 2024-04-06T14:52:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f816c06cc20 0x7f816c06f0d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8180101d70 0x7f8180113a10 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 --2- 172.21.15.32:0/3283318158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8180102710 0x7f8180113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 >> 172.21.15.32:0/3283318158 conn(0x7f8180072bc0 msgr2=0x7f8180065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 shutdown_connections 2024-04-06T14:52:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.099+0000 7f81862fc700 1 -- 172.21.15.32:0/3283318158 wait complete. 2024-04-06T14:52:04.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 -- 172.21.15.32:0/3654586516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 msgr2=0x7f32280fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 --2- 172.21.15.32:0/3654586516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 0x7f32280fce20 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3218004940 tx=0x7f3218009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:04.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 -- 172.21.15.32:0/3654586516 shutdown_connections 2024-04-06T14:52:04.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 --2- 172.21.15.32:0/3654586516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 0x7f32280fce20 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 --2- 172.21.15.32:0/3654586516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3228102e10 0x7f32281031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 -- 172.21.15.32:0/3654586516 >> 172.21.15.32:0/3654586516 conn(0x7f32280f8980 msgr2=0x7f32280fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:04.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 -- 172.21.15.32:0/3654586516 shutdown_connections 2024-04-06T14:52:04.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 -- 172.21.15.32:0/3654586516 wait complete. 2024-04-06T14:52:04.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 Processor -- start 2024-04-06T14:52:04.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.204+0000 7f322fc68700 1 -- start start 2024-04-06T14:52:04.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322fc68700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3228102e10 0x7f32280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:04.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322fc68700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 0x7f3228070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:04.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322fc68700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3228070d90 con 0x7f32281037b0 2024-04-06T14:52:04.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322fc68700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3228070ed0 con 0x7f3228102e10 2024-04-06T14:52:04.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322d203700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 0x7f3228070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:04.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322d203700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 0x7f3228070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36650/0 (socket says 172.21.15.32:36650) 2024-04-06T14:52:04.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322d203700 1 -- 172.21.15.32:0/46333693 learned_addr learned my addr 172.21.15.32:0/46333693 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322da04700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3228102e10 0x7f32280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322d203700 1 -- 172.21.15.32:0/46333693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3228102e10 msgr2=0x7f32280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322d203700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3228102e10 0x7f32280702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322d203700 1 -- 172.21.15.32:0/46333693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3218005040 con 0x7f32281037b0 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f322d203700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 0x7f3228070800 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f32180093b0 tx=0x7f3218009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.205+0000 7f321e7fc700 1 -- 172.21.15.32:0/46333693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f321800e040 con 0x7f32281037b0 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.206+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3228071120 con 0x7f32281037b0 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.206+0000 7f321e7fc700 1 -- 172.21.15.32:0/46333693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3218013420 con 0x7f32281037b0 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.206+0000 7f321e7fc700 1 -- 172.21.15.32:0/46333693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f321801c630 con 0x7f32281037b0 2024-04-06T14:52:04.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.206+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322811a190 con 0x7f32281037b0 2024-04-06T14:52:04.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.207+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32280fe560 con 0x7f32281037b0 2024-04-06T14:52:04.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.207+0000 7f321e7fc700 1 -- 172.21.15.32:0/46333693 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f32180139e0 con 0x7f32281037b0 2024-04-06T14:52:04.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.208+0000 7f321e7fc700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321406cc20 0x7f321406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:04.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.208+0000 7f322da04700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321406cc20 0x7f321406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:04.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.208+0000 7f321e7fc700 1 -- 172.21.15.32:0/46333693 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f321808d170 con 0x7f32281037b0 2024-04-06T14:52:04.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.208+0000 7f322da04700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321406cc20 0x7f321406f0d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3224009fd0 tx=0x7f322400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:04.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.212+0000 7f321e7fc700 1 -- 172.21.15.32:0/46333693 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3218057260 con 0x7f32281037b0 2024-04-06T14:52:04.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.426+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3228109e50 con 0x7f321406cc20 2024-04-06T14:52:04.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.436+0000 7f321e7fc700 1 -- 172.21.15.32:0/46333693 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f322400f040 con 0x7f321406cc20 2024-04-06T14:52:04.424 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:52:04.424 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (32m) 10m ago 43m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:52:04.424 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (29m) 10m ago 43m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (29m) 10m ago 42m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (32m) 10m ago 43m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (26m) 10m ago 37m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (25m) 10m ago 37m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (25m) 10m ago 37m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (10m) 10m ago 37m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (34m) 10m ago 45m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (33m) 10m ago 42m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (30m) 10m ago 45m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (30m) 10m ago 42m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (33m) 10m ago 43m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (33m) 10m ago 42m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (29m) 10m ago 41m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:52:04.425 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (28m) 10m ago 41m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:52:04.426 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (28m) 10m ago 40m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:52:04.426 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (27m) 10m ago 40m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:52:04.426 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (26m) 10m ago 40m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:52:04.426 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (26m) 10m ago 39m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:52:04.426 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (32m) 10m ago 43m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:52:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321406cc20 msgr2=0x7f321406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321406cc20 0x7f321406f0d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3224009fd0 tx=0x7f322400b040 comp rx=0 tx=0).stop 2024-04-06T14:52:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 msgr2=0x7f3228070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 0x7f3228070800 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f32180093b0 tx=0x7f3218009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 shutdown_connections 2024-04-06T14:52:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f321406cc20 0x7f321406f0d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3228102e10 0x7f32280702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 --2- 172.21.15.32:0/46333693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32281037b0 0x7f3228070800 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 >> 172.21.15.32:0/46333693 conn(0x7f32280f8980 msgr2=0x7f32280f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 shutdown_connections 2024-04-06T14:52:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.438+0000 7f322fc68700 1 -- 172.21.15.32:0/46333693 wait complete. 2024-04-06T14:52:04.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.540+0000 7fc5d1af6700 1 -- 172.21.15.32:0/3793887658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc101d10 msgr2=0x7fc5cc1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.540+0000 7fc5d1af6700 1 --2- 172.21.15.32:0/3793887658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc101d10 0x7fc5cc1020e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0004940 tx=0x7fc5c0009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:04.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.540+0000 7fc5d1af6700 1 -- 172.21.15.32:0/3793887658 shutdown_connections 2024-04-06T14:52:04.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.540+0000 7fc5d1af6700 1 --2- 172.21.15.32:0/3793887658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5cc1026b0 0x7fc5cc0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.540+0000 7fc5d1af6700 1 --2- 172.21.15.32:0/3793887658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc101d10 0x7fc5cc1020e0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.540+0000 7fc5d1af6700 1 -- 172.21.15.32:0/3793887658 >> 172.21.15.32:0/3793887658 conn(0x7fc5cc0f7880 msgr2=0x7fc5cc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.540+0000 7fc5d1af6700 1 -- 172.21.15.32:0/3793887658 shutdown_connections 2024-04-06T14:52:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.540+0000 7fc5d1af6700 1 -- 172.21.15.32:0/3793887658 wait complete. 2024-04-06T14:52:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.540+0000 7fc5d1af6700 1 Processor -- start 2024-04-06T14:52:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5d1af6700 1 -- start start 2024-04-06T14:52:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5d1af6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5cc101d10 0x7fc5cc113d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5d1af6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc1026b0 0x7fc5cc1142b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5d1af6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc114900 con 0x7fc5cc1026b0 2024-04-06T14:52:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5d1af6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc10dcd0 con 0x7fc5cc101d10 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5caffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc1026b0 0x7fc5cc1142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5caffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc1026b0 0x7fc5cc1142b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42666/0 (socket says 172.21.15.32:42666) 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5caffd700 1 -- 172.21.15.32:0/2373323509 learned_addr learned my addr 172.21.15.32:0/2373323509 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5cb7fe700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5cc101d10 0x7fc5cc113d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5caffd700 1 -- 172.21.15.32:0/2373323509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5cc101d10 msgr2=0x7fc5cc113d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5caffd700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5cc101d10 0x7fc5cc113d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5caffd700 1 -- 172.21.15.32:0/2373323509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c0005040 con 0x7fc5cc1026b0 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5caffd700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc1026b0 0x7fc5cc1142b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00eb40 tx=0x7fc5bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.541+0000 7fc5b3fff700 1 -- 172.21.15.32:0/2373323509 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5bc007cc0 con 0x7fc5cc1026b0 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.542+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc10df50 con 0x7fc5cc1026b0 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.542+0000 7fc5b3fff700 1 -- 172.21.15.32:0/2373323509 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5bc00f040 con 0x7fc5cc1026b0 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.542+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc10e510 con 0x7fc5cc1026b0 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.542+0000 7fc5b3fff700 1 -- 172.21.15.32:0/2373323509 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5bc00c8f0 con 0x7fc5cc1026b0 2024-04-06T14:52:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.543+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc0fd460 con 0x7fc5cc1026b0 2024-04-06T14:52:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.543+0000 7fc5b3fff700 1 -- 172.21.15.32:0/2373323509 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc5bc00cb60 con 0x7fc5cc1026b0 2024-04-06T14:52:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.543+0000 7fc5b3fff700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b406cc20 0x7fc5b406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.544+0000 7fc5cb7fe700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b406cc20 0x7fc5b406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.544+0000 7fc5b3fff700 1 -- 172.21.15.32:0/2373323509 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc5bc08bb00 con 0x7fc5cc1026b0 2024-04-06T14:52:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.544+0000 7fc5cb7fe700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b406cc20 0x7fc5b406f0d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc5c000b0d0 tx=0x7fc5c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:04.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.548+0000 7fc5b3fff700 1 -- 172.21.15.32:0/2373323509 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc5bc059600 con 0x7fc5cc1026b0 2024-04-06T14:52:04.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.838+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc5cc10f420 con 0x7fc5cc1026b0 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.839+0000 7fc5b3fff700 1 -- 172.21.15.32:0/2373323509 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc5bc059190 con 0x7fc5cc1026b0 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:52:04.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:52:04.828 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:52:04.828 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:52:04.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:52:04.828 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:52:04.828 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:52:04.828 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:52:04.828 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b406cc20 msgr2=0x7fc5b406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b406cc20 0x7fc5b406f0d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc5c000b0d0 tx=0x7fc5c0009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc1026b0 msgr2=0x7fc5cc1142b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc1026b0 0x7fc5cc1142b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00eb40 tx=0x7fc5bc00b040 comp rx=0 tx=0).stop 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 shutdown_connections 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc5b406cc20 0x7fc5b406f0d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5cc101d10 0x7fc5cc113d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 --2- 172.21.15.32:0/2373323509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5cc1026b0 0x7fc5cc1142b0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 >> 172.21.15.32:0/2373323509 conn(0x7fc5cc0f7880 msgr2=0x7fc5cc0f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 shutdown_connections 2024-04-06T14:52:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.841+0000 7fc5d1af6700 1 -- 172.21.15.32:0/2373323509 wait complete. 2024-04-06T14:52:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 -- 172.21.15.32:0/1160110759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c1037b0 msgr2=0x7f6b3c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 --2- 172.21.15.32:0/1160110759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c1037b0 0x7f6b3c0fce20 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c004940 tx=0x7f6b2c009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 -- 172.21.15.32:0/1160110759 shutdown_connections 2024-04-06T14:52:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 --2- 172.21.15.32:0/1160110759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c1037b0 0x7f6b3c0fce20 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 --2- 172.21.15.32:0/1160110759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c102e10 0x7f6b3c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 -- 172.21.15.32:0/1160110759 >> 172.21.15.32:0/1160110759 conn(0x7f6b3c0f8980 msgr2=0x7f6b3c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 -- 172.21.15.32:0/1160110759 shutdown_connections 2024-04-06T14:52:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 -- 172.21.15.32:0/1160110759 wait complete. 2024-04-06T14:52:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 Processor -- start 2024-04-06T14:52:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.944+0000 7f6b44748700 1 -- start start 2024-04-06T14:52:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b44748700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c102e10 0x7f6b3c113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b44748700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c1037b0 0x7f6b3c114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b44748700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c114670 con 0x7f6b3c102e10 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b44748700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c10da60 con 0x7f6b3c1037b0 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b424e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c102e10 0x7f6b3c113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b424e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c102e10 0x7f6b3c113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42688/0 (socket says 172.21.15.32:42688) 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b424e4700 1 -- 172.21.15.32:0/1515744447 learned_addr learned my addr 172.21.15.32:0/1515744447 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b41ce3700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c1037b0 0x7f6b3c114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b424e4700 1 -- 172.21.15.32:0/1515744447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c1037b0 msgr2=0x7f6b3c114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b424e4700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c1037b0 0x7f6b3c114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b424e4700 1 -- 172.21.15.32:0/1515744447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c005040 con 0x7f6b3c102e10 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b424e4700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c102e10 0x7f6b3c113ae0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6b38002a50 tx=0x7f6b380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b32ffd700 1 -- 172.21.15.32:0/1515744447 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b38007cc0 con 0x7f6b3c102e10 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c10dd40 con 0x7f6b3c102e10 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b32ffd700 1 -- 172.21.15.32:0/1515744447 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b3800f040 con 0x7f6b3c102e10 2024-04-06T14:52:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b32ffd700 1 -- 172.21.15.32:0/1515744447 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3800d9c0 con 0x7f6b3c102e10 2024-04-06T14:52:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.945+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c10e2d0 con 0x7f6b3c102e10 2024-04-06T14:52:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.946+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c0fe560 con 0x7f6b3c102e10 2024-04-06T14:52:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.947+0000 7f6b32ffd700 1 -- 172.21.15.32:0/1515744447 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6b3800db20 con 0x7f6b3c102e10 2024-04-06T14:52:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.948+0000 7f6b32ffd700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b2806cc20 0x7f6b2806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.948+0000 7f6b32ffd700 1 -- 172.21.15.32:0/1515744447 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6b3808c460 con 0x7f6b3c102e10 2024-04-06T14:52:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.948+0000 7f6b41ce3700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b2806cc20 0x7f6b2806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.948+0000 7f6b41ce3700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b2806cc20 0x7f6b2806f0d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c004890 tx=0x7f6b2c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:04.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:04.951+0000 7f6b32ffd700 1 -- 172.21.15.32:0/1515744447 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6b38059f60 con 0x7f6b3c102e10 2024-04-06T14:52:05.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.200+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6b3c109de0 con 0x7f6b3c102e10 2024-04-06T14:52:05.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.201+0000 7f6b32ffd700 1 -- 172.21.15.32:0/1515744447 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f6b38059af0 con 0x7f6b3c102e10 2024-04-06T14:52:05.189 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:52:05.190 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:52:05.191 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:52:05.192 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:52:05.192 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:52:05.192 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:52:05.192 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:52:05.192 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:52:05.192 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:52:05.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:52:05.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:52:05.194 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:52:05.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:52:05.194 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:52:05.194 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:52:05.195 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:52:05.195 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:52:05.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b2806cc20 msgr2=0x7f6b2806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b2806cc20 0x7f6b2806f0d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c004890 tx=0x7f6b2c009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c102e10 msgr2=0x7f6b3c113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c102e10 0x7f6b3c113ae0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6b38002a50 tx=0x7f6b380046b0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 shutdown_connections 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b2806cc20 0x7f6b2806f0d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b3c102e10 0x7f6b3c113ae0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 --2- 172.21.15.32:0/1515744447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c1037b0 0x7f6b3c114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 >> 172.21.15.32:0/1515744447 conn(0x7f6b3c0f8980 msgr2=0x7f6b3c0f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 shutdown_connections 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.203+0000 7f6b44748700 1 -- 172.21.15.32:0/1515744447 wait complete. 2024-04-06T14:52:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:52:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.307+0000 7fca4d8a6700 1 -- 172.21.15.32:0/4073461392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 msgr2=0x7fca48106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.307+0000 7fca4d8a6700 1 --2- 172.21.15.32:0/4073461392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 0x7fca48106ec0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fca3c004940 tx=0x7fca3c009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.307+0000 7fca4d8a6700 1 -- 172.21.15.32:0/4073461392 shutdown_connections 2024-04-06T14:52:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.307+0000 7fca4d8a6700 1 --2- 172.21.15.32:0/4073461392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 0x7fca48106ec0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.307+0000 7fca4d8a6700 1 --2- 172.21.15.32:0/4073461392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca480fdee0 0x7fca480fe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.307+0000 7fca4d8a6700 1 -- 172.21.15.32:0/4073461392 >> 172.21.15.32:0/4073461392 conn(0x7fca480f78f0 msgr2=0x7fca480f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.307+0000 7fca4d8a6700 1 -- 172.21.15.32:0/4073461392 shutdown_connections 2024-04-06T14:52:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.308+0000 7fca4d8a6700 1 -- 172.21.15.32:0/4073461392 wait complete. 2024-04-06T14:52:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.308+0000 7fca4d8a6700 1 Processor -- start 2024-04-06T14:52:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.308+0000 7fca4d8a6700 1 -- start start 2024-04-06T14:52:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.308+0000 7fca4d8a6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca480fdee0 0x7fca480fbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.308+0000 7fca4d8a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 0x7fca480fc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.308+0000 7fca4d8a6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca480fdc60 con 0x7fca480fe880 2024-04-06T14:52:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.308+0000 7fca4d8a6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca480fc790 con 0x7fca480fdee0 2024-04-06T14:52:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca467fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 0x7fca480fc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca467fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 0x7fca480fc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42704/0 (socket says 172.21.15.32:42704) 2024-04-06T14:52:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca467fc700 1 -- 172.21.15.32:0/2710019764 learned_addr learned my addr 172.21.15.32:0/2710019764 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca46ffd700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca480fdee0 0x7fca480fbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca467fc700 1 -- 172.21.15.32:0/2710019764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca480fdee0 msgr2=0x7fca480fbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca467fc700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca480fdee0 0x7fca480fbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca467fc700 1 -- 172.21.15.32:0/2710019764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c005040 con 0x7fca480fe880 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca467fc700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 0x7fca480fc250 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fca3c00b3b0 tx=0x7fca3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca2f7fe700 1 -- 172.21.15.32:0/2710019764 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca3c00e040 con 0x7fca480fe880 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca480fca70 con 0x7fca480fe880 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca2f7fe700 1 -- 172.21.15.32:0/2710019764 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca3c013420 con 0x7fca480fe880 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca2f7fe700 1 -- 172.21.15.32:0/2710019764 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca3c01c630 con 0x7fca480fe880 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.309+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca48119f30 con 0x7fca480fe880 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.310+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca48066a90 con 0x7fca480fe880 2024-04-06T14:52:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.311+0000 7fca2f7fe700 1 -- 172.21.15.32:0/2710019764 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fca3c013590 con 0x7fca480fe880 2024-04-06T14:52:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.311+0000 7fca2f7fe700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca3006cc20 0x7fca3006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.311+0000 7fca2f7fe700 1 -- 172.21.15.32:0/2710019764 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fca3c08d2e0 con 0x7fca480fe880 2024-04-06T14:52:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.311+0000 7fca46ffd700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca3006cc20 0x7fca3006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.312+0000 7fca46ffd700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca3006cc20 0x7fca3006f0d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fca38004990 tx=0x7fca38009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.316+0000 7fca2f7fe700 1 -- 172.21.15.32:0/2710019764 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fca3c0572a0 con 0x7fca480fe880 2024-04-06T14:52:05.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.537+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fca481021c0 con 0x7fca3006cc20 2024-04-06T14:52:05.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.538+0000 7fca2f7fe700 1 -- 172.21.15.32:0/2710019764 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fca481021c0 con 0x7fca3006cc20 2024-04-06T14:52:05.526 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:52:05.526 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:52:05.526 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:52:05.526 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:52:05.526 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:52:05.527 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:52:05.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca3006cc20 msgr2=0x7fca3006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca3006cc20 0x7fca3006f0d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fca38004990 tx=0x7fca38009ee0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 msgr2=0x7fca480fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 0x7fca480fc250 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fca3c00b3b0 tx=0x7fca3c009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:05.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 shutdown_connections 2024-04-06T14:52:05.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fca3006cc20 0x7fca3006f0d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca480fdee0 0x7fca480fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 --2- 172.21.15.32:0/2710019764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca480fe880 0x7fca480fc250 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 >> 172.21.15.32:0/2710019764 conn(0x7fca480f78f0 msgr2=0x7fca481037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:05.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.540+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 shutdown_connections 2024-04-06T14:52:05.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.541+0000 7fca4d8a6700 1 -- 172.21.15.32:0/2710019764 wait complete. 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.643+0000 7f1fdf895700 1 -- 172.21.15.32:0/4216441398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd80fb4a0 msgr2=0x7f1fd80fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.643+0000 7f1fdf895700 1 --2- 172.21.15.32:0/4216441398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd80fb4a0 0x7f1fd80fb910 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8004940 tx=0x7f1fc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.643+0000 7f1fdf895700 1 -- 172.21.15.32:0/4216441398 shutdown_connections 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.643+0000 7f1fdf895700 1 --2- 172.21.15.32:0/4216441398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd80fb4a0 0x7f1fd80fb910 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.643+0000 7f1fdf895700 1 --2- 172.21.15.32:0/4216441398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fd8102e70 0x7f1fd8103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.643+0000 7f1fdf895700 1 -- 172.21.15.32:0/4216441398 >> 172.21.15.32:0/4216441398 conn(0x7f1fd80714c0 msgr2=0x7f1fd80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.643+0000 7f1fdf895700 1 -- 172.21.15.32:0/4216441398 shutdown_connections 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.643+0000 7f1fdf895700 1 -- 172.21.15.32:0/4216441398 wait complete. 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdf895700 1 Processor -- start 2024-04-06T14:52:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdf895700 1 -- start start 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdf895700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fd80fb4a0 0x7f1fd8113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdf895700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8102e70 0x7f1fd8113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdf895700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8114640 con 0x7f1fd8102e70 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdf895700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd810da30 con 0x7f1fd80fb4a0 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdce30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8102e70 0x7f1fd8113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdce30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8102e70 0x7f1fd8113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42730/0 (socket says 172.21.15.32:42730) 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdce30700 1 -- 172.21.15.32:0/1996210266 learned_addr learned my addr 172.21.15.32:0/1996210266 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.644+0000 7f1fdd631700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fd80fb4a0 0x7f1fd8113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.645+0000 7f1fdce30700 1 -- 172.21.15.32:0/1996210266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fd80fb4a0 msgr2=0x7f1fd8113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.645+0000 7f1fdce30700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fd80fb4a0 0x7f1fd8113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.645+0000 7f1fdce30700 1 -- 172.21.15.32:0/1996210266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc8005040 con 0x7f1fd8102e70 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.645+0000 7f1fdce30700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8102e70 0x7f1fd8113ff0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8008d80 tx=0x7f1fc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.645+0000 7f1fcdffb700 1 -- 172.21.15.32:0/1996210266 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fc800e030 con 0x7f1fd8102e70 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.645+0000 7f1fcdffb700 1 -- 172.21.15.32:0/1996210266 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fc8013de0 con 0x7f1fd8102e70 2024-04-06T14:52:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.645+0000 7f1fcdffb700 1 -- 172.21.15.32:0/1996210266 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fc801cd00 con 0x7f1fd8102e70 2024-04-06T14:52:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.645+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd810dcb0 con 0x7f1fd8102e70 2024-04-06T14:52:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.645+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd810e1f0 con 0x7f1fd8102e70 2024-04-06T14:52:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.646+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd8100170 con 0x7f1fd8102e70 2024-04-06T14:52:05.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.647+0000 7f1fcdffb700 1 -- 172.21.15.32:0/1996210266 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1fc8013930 con 0x7f1fd8102e70 2024-04-06T14:52:05.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.647+0000 7f1fcdffb700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fc406cc30 0x7f1fc406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:05.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.647+0000 7f1fcdffb700 1 -- 172.21.15.32:0/1996210266 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1fc808d8d0 con 0x7f1fd8102e70 2024-04-06T14:52:05.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.647+0000 7f1fdd631700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fc406cc30 0x7f1fc406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:05.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.648+0000 7f1fdd631700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fc406cc30 0x7f1fc406f0e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009b80 tx=0x7f1fd400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:05.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.651+0000 7f1fcdffb700 1 -- 172.21.15.32:0/1996210266 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1fc805b3a0 con 0x7f1fd8102e70 2024-04-06T14:52:05.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:05 smithi032 ceph-mon[147414]: from='client.35738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:05.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:05 smithi032 ceph-mon[147414]: from='client.35742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:05.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:05 smithi032 ceph-mon[147414]: pgmap v959: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:05.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:05 smithi032 ceph-mon[147414]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:05.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:05 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2373323509' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:52:05.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:05 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1515744447' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:52:05.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:05 smithi086 ceph-mon[110975]: from='client.35738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:05.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:05 smithi086 ceph-mon[110975]: from='client.35742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:05.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:05 smithi086 ceph-mon[110975]: pgmap v959: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:05.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:05 smithi086 ceph-mon[110975]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:05.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:05 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2373323509' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:52:05.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:05 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1515744447' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:52:05.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.941+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1fd80653a0 con 0x7f1fd8102e70 2024-04-06T14:52:05.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.942+0000 7f1fcdffb700 1 -- 172.21.15.32:0/1996210266 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f1fc801ce60 con 0x7f1fd8102e70 2024-04-06T14:52:05.930 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:52:05.930 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:52:05.930 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:52:05.930 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:52:05.930 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:52:05.930 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:52:05.930 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:52:05.931 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:52:05.932 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:52:05.932 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:52:05.932 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:52:05.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fc406cc30 msgr2=0x7f1fc406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fc406cc30 0x7f1fc406f0e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009b80 tx=0x7f1fd400b040 comp rx=0 tx=0).stop 2024-04-06T14:52:05.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8102e70 msgr2=0x7f1fd8113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:05.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8102e70 0x7f1fd8113ff0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8008d80 tx=0x7f1fc8009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:05.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 shutdown_connections 2024-04-06T14:52:05.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fc406cc30 0x7f1fc406f0e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fd80fb4a0 0x7f1fd8113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 --2- 172.21.15.32:0/1996210266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8102e70 0x7f1fd8113ff0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:05.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 >> 172.21.15.32:0/1996210266 conn(0x7f1fd80714c0 msgr2=0x7f1fd80fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:05.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 shutdown_connections 2024-04-06T14:52:05.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:05.944+0000 7f1fdf895700 1 -- 172.21.15.32:0/1996210266 wait complete. 2024-04-06T14:52:06.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:06 smithi032 ceph-mon[147414]: from='client.35758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:06.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:06 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1996210266' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:52:06.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:06 smithi086 ceph-mon[110975]: from='client.35758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:06.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:06 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1996210266' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:52:07.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:07 smithi032 ceph-mon[147414]: pgmap v960: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:07.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:07 smithi086 ceph-mon[110975]: pgmap v960: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:08.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:52:08.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:52:09.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:09 smithi032 ceph-mon[147414]: pgmap v961: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:09.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:09 smithi086 ceph-mon[110975]: pgmap v961: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:11.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:11 smithi032 ceph-mon[147414]: pgmap v962: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:11.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:11 smithi086 ceph-mon[110975]: pgmap v962: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:13.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:13 smithi032 ceph-mon[147414]: pgmap v963: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:13.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:13 smithi086 ceph-mon[110975]: pgmap v963: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:15.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:15 smithi032 ceph-mon[147414]: pgmap v964: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:15.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:15 smithi086 ceph-mon[110975]: pgmap v964: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:17.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:17 smithi032 ceph-mon[147414]: pgmap v965: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:17.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:17 smithi086 ceph-mon[110975]: pgmap v965: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:19.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:19 smithi032 ceph-mon[147414]: pgmap v966: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:19.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:19 smithi086 ceph-mon[110975]: pgmap v966: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:21.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:21 smithi032 ceph-mon[147414]: pgmap v967: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:21.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:21 smithi086 ceph-mon[110975]: pgmap v967: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:23.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:23 smithi032 ceph-mon[147414]: pgmap v968: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:23.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:52:23.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:23 smithi086 ceph-mon[110975]: pgmap v968: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:23.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:52:25.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:25 smithi032 ceph-mon[147414]: pgmap v969: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:25.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:25 smithi086 ceph-mon[110975]: pgmap v969: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:27.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:27 smithi032 ceph-mon[147414]: pgmap v970: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:27.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:27 smithi086 ceph-mon[110975]: pgmap v970: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:29.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:29 smithi032 ceph-mon[147414]: pgmap v971: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:29.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:29 smithi086 ceph-mon[110975]: pgmap v971: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:31.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:31 smithi032 ceph-mon[147414]: pgmap v972: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:31.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:31 smithi086 ceph-mon[110975]: pgmap v972: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:33.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:33 smithi032 ceph-mon[147414]: pgmap v973: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:33.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:33 smithi086 ceph-mon[110975]: pgmap v973: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:35.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:35 smithi032 ceph-mon[147414]: pgmap v974: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:35.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:35 smithi086 ceph-mon[110975]: pgmap v974: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:36.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.061+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2967018111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac101d70 msgr2=0x7fbfac102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.061+0000 7fbfb3bb0700 1 --2- 172.21.15.32:0/2967018111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac101d70 0x7fbfac102140 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004940 tx=0x7fbfa8009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.061+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2967018111 shutdown_connections 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.061+0000 7fbfb3bb0700 1 --2- 172.21.15.32:0/2967018111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfac102710 0x7fbfac0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.061+0000 7fbfb3bb0700 1 --2- 172.21.15.32:0/2967018111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac101d70 0x7fbfac102140 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.061+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2967018111 >> 172.21.15.32:0/2967018111 conn(0x7fbfac072bc0 msgr2=0x7fbfac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.061+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2967018111 shutdown_connections 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.061+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2967018111 wait complete. 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.062+0000 7fbfb3bb0700 1 Processor -- start 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.062+0000 7fbfb3bb0700 1 -- start start 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.062+0000 7fbfb3bb0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfac101d70 0x7fbfac113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.062+0000 7fbfb3bb0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac102710 0x7fbfac113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.062+0000 7fbfb3bb0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac1145f0 con 0x7fbfac102710 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.062+0000 7fbfb3bb0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac10d9e0 con 0x7fbfac101d70 2024-04-06T14:52:36.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.062+0000 7fbfb114b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac102710 0x7fbfac113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.062+0000 7fbfb114b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac102710 0x7fbfac113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34814/0 (socket says 172.21.15.32:34814) 2024-04-06T14:52:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.062+0000 7fbfb114b700 1 -- 172.21.15.32:0/2452811487 learned_addr learned my addr 172.21.15.32:0/2452811487 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbfb194c700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfac101d70 0x7fbfac113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbfb114b700 1 -- 172.21.15.32:0/2452811487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfac101d70 msgr2=0x7fbfac113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbfb114b700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfac101d70 0x7fbfac113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbfb114b700 1 -- 172.21.15.32:0/2452811487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8005040 con 0x7fbfac102710 2024-04-06T14:52:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbfb194c700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfac101d70 0x7fbfac113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:52:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbfb114b700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac102710 0x7fbfac113fa0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000c800 tx=0x7fbfa0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:36.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbf9e7fc700 1 -- 172.21.15.32:0/2452811487 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfa000a760 con 0x7fbfac102710 2024-04-06T14:52:36.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbf9e7fc700 1 -- 172.21.15.32:0/2452811487 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfa000ad70 con 0x7fbfac102710 2024-04-06T14:52:36.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfac10dcc0 con 0x7fbfac102710 2024-04-06T14:52:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbf9e7fc700 1 -- 172.21.15.32:0/2452811487 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfa00113a0 con 0x7fbfac102710 2024-04-06T14:52:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.063+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfac10e280 con 0x7fbfac102710 2024-04-06T14:52:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.064+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfac0fd4c0 con 0x7fbfac102710 2024-04-06T14:52:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.065+0000 7fbf9e7fc700 1 -- 172.21.15.32:0/2452811487 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbfa0012400 con 0x7fbfac102710 2024-04-06T14:52:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.065+0000 7fbf9e7fc700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf9806cc30 0x7fbf9806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.065+0000 7fbf9e7fc700 1 -- 172.21.15.32:0/2452811487 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbfa008c060 con 0x7fbfac102710 2024-04-06T14:52:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.065+0000 7fbfb194c700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf9806cc30 0x7fbf9806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.066+0000 7fbfb194c700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf9806cc30 0x7fbf9806f0e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800b3b0 tx=0x7fbfa8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:36.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.070+0000 7fbf9e7fc700 1 -- 172.21.15.32:0/2452811487 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbfa0059b30 con 0x7fbfac102710 2024-04-06T14:52:36.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.294+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfac108d30 con 0x7fbf9806cc30 2024-04-06T14:52:36.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.295+0000 7fbf9e7fc700 1 -- 172.21.15.32:0/2452811487 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbfac108d30 con 0x7fbf9806cc30 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf9806cc30 msgr2=0x7fbf9806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf9806cc30 0x7fbf9806f0e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800b3b0 tx=0x7fbfa8009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac102710 msgr2=0x7fbfac113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac102710 0x7fbfac113fa0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000c800 tx=0x7fbfa0004470 comp rx=0 tx=0).stop 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 shutdown_connections 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbf9806cc30 0x7fbf9806f0e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfac101d70 0x7fbfac113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 --2- 172.21.15.32:0/2452811487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfac102710 0x7fbfac113fa0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 >> 172.21.15.32:0/2452811487 conn(0x7fbfac072bc0 msgr2=0x7fbfac0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 shutdown_connections 2024-04-06T14:52:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.297+0000 7fbfb3bb0700 1 -- 172.21.15.32:0/2452811487 wait complete. 2024-04-06T14:52:36.295 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:52:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 -- 172.21.15.32:0/1239807884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94102710 msgr2=0x7f8e940fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 --2- 172.21.15.32:0/1239807884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94102710 0x7f8e940fbd80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8e84004940 tx=0x7f8e84009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 -- 172.21.15.32:0/1239807884 shutdown_connections 2024-04-06T14:52:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 --2- 172.21.15.32:0/1239807884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94102710 0x7f8e940fbd80 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 --2- 172.21.15.32:0/1239807884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e94101d70 0x7f8e94102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 -- 172.21.15.32:0/1239807884 >> 172.21.15.32:0/1239807884 conn(0x7f8e94072bc0 msgr2=0x7f8e94072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 -- 172.21.15.32:0/1239807884 shutdown_connections 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 -- 172.21.15.32:0/1239807884 wait complete. 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 Processor -- start 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.401+0000 7f8e990ab700 1 -- start start 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e990ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94101d70 0x7f8e94115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e990ab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e94102710 0x7f8e941161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e990ab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e94116830 con 0x7f8e94101d70 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e990ab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9410fc00 con 0x7f8e94102710 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e92d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94101d70 0x7f8e94115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e92d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94101d70 0x7f8e94115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34842/0 (socket says 172.21.15.32:34842) 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e92d9d700 1 -- 172.21.15.32:0/3923223270 learned_addr learned my addr 172.21.15.32:0/3923223270 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e9259c700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e94102710 0x7f8e941161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e92d9d700 1 -- 172.21.15.32:0/3923223270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e94102710 msgr2=0x7f8e941161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e92d9d700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e94102710 0x7f8e941161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e92d9d700 1 -- 172.21.15.32:0/3923223270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84005040 con 0x7f8e94101d70 2024-04-06T14:52:36.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e92d9d700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94101d70 0x7f8e94115ca0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8e88002a50 tx=0x7f8e880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:36.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.402+0000 7f8e7b7fe700 1 -- 172.21.15.32:0/3923223270 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e88007cc0 con 0x7f8e94101d70 2024-04-06T14:52:36.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.403+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9410fee0 con 0x7f8e94101d70 2024-04-06T14:52:36.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.403+0000 7f8e7b7fe700 1 -- 172.21.15.32:0/3923223270 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e8800f040 con 0x7f8e94101d70 2024-04-06T14:52:36.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.403+0000 7f8e7b7fe700 1 -- 172.21.15.32:0/3923223270 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e8800d9c0 con 0x7f8e94101d70 2024-04-06T14:52:36.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.403+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e941104a0 con 0x7f8e94101d70 2024-04-06T14:52:36.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.404+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e940fd4c0 con 0x7f8e94101d70 2024-04-06T14:52:36.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.404+0000 7f8e7b7fe700 1 -- 172.21.15.32:0/3923223270 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8e8800db20 con 0x7f8e94101d70 2024-04-06T14:52:36.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.405+0000 7f8e7b7fe700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e7c06cc30 0x7f8e7c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:36.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.405+0000 7f8e7b7fe700 1 -- 172.21.15.32:0/3923223270 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8e8808bd70 con 0x7f8e94101d70 2024-04-06T14:52:36.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.405+0000 7f8e9259c700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e7c06cc30 0x7f8e7c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:36.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.405+0000 7f8e9259c700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e7c06cc30 0x7f8e7c06f0e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8e94111540 tx=0x7f8e84009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:36.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.409+0000 7f8e7b7fe700 1 -- 172.21.15.32:0/3923223270 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8e880597c0 con 0x7f8e94101d70 2024-04-06T14:52:36.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.633+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e9404bf40 con 0x7f8e7c06cc30 2024-04-06T14:52:36.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.634+0000 7f8e7b7fe700 1 -- 172.21.15.32:0/3923223270 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8e9404bf40 con 0x7f8e7c06cc30 2024-04-06T14:52:36.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e7c06cc30 msgr2=0x7f8e7c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e7c06cc30 0x7f8e7c06f0e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8e94111540 tx=0x7f8e84009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94101d70 msgr2=0x7f8e94115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94101d70 0x7f8e94115ca0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8e88002a50 tx=0x7f8e880046b0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 shutdown_connections 2024-04-06T14:52:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8e7c06cc30 0x7f8e7c06f0e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e94101d70 0x7f8e94115ca0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 --2- 172.21.15.32:0/3923223270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e94102710 0x7f8e941161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 >> 172.21.15.32:0/3923223270 conn(0x7f8e94072bc0 msgr2=0x7f8e940659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 shutdown_connections 2024-04-06T14:52:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.637+0000 7f8e990ab700 1 -- 172.21.15.32:0/3923223270 wait complete. 2024-04-06T14:52:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 -- 172.21.15.32:0/390382018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 msgr2=0x7f2688103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 --2- 172.21.15.32:0/390382018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 0x7f2688103240 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2684004940 tx=0x7f2684009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 -- 172.21.15.32:0/390382018 shutdown_connections 2024-04-06T14:52:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 --2- 172.21.15.32:0/390382018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26880fb4a0 0x7f26880fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 --2- 172.21.15.32:0/390382018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 0x7f2688103240 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 -- 172.21.15.32:0/390382018 >> 172.21.15.32:0/390382018 conn(0x7f26880714c0 msgr2=0x7f26880718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 -- 172.21.15.32:0/390382018 shutdown_connections 2024-04-06T14:52:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 -- 172.21.15.32:0/390382018 wait complete. 2024-04-06T14:52:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 Processor -- start 2024-04-06T14:52:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.740+0000 7f26903f6700 1 -- start start 2024-04-06T14:52:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f26903f6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26880fb4a0 0x7f26880fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f26903f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 0x7f26880fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f26903f6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26880fd150 con 0x7f2688102e70 2024-04-06T14:52:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f26903f6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26880fd290 con 0x7f26880fb4a0 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f268d991700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 0x7f26880fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f268d991700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 0x7f26880fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34872/0 (socket says 172.21.15.32:34872) 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f268d991700 1 -- 172.21.15.32:0/2578698385 learned_addr learned my addr 172.21.15.32:0/2578698385 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f268e192700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26880fb4a0 0x7f26880fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f268d991700 1 -- 172.21.15.32:0/2578698385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26880fb4a0 msgr2=0x7f26880fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f268d991700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26880fb4a0 0x7f26880fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f268d991700 1 -- 172.21.15.32:0/2578698385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684005040 con 0x7f2688102e70 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f268e192700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26880fb4a0 0x7f26880fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f268d991700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 0x7f26880fcbc0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f267800c8c0 tx=0x7f26780044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f267effd700 1 -- 172.21.15.32:0/2578698385 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f267800a980 con 0x7f2688102e70 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f267effd700 1 -- 172.21.15.32:0/2578698385 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f267800ce60 con 0x7f2688102e70 2024-04-06T14:52:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2688119d70 con 0x7f2688102e70 2024-04-06T14:52:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f267effd700 1 -- 172.21.15.32:0/2578698385 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26780113a0 con 0x7f2688102e70 2024-04-06T14:52:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.741+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268811a1f0 con 0x7f2688102e70 2024-04-06T14:52:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.742+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2688100110 con 0x7f2688102e70 2024-04-06T14:52:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.743+0000 7f267effd700 1 -- 172.21.15.32:0/2578698385 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2678012400 con 0x7f2688102e70 2024-04-06T14:52:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.743+0000 7f267effd700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f267406cc20 0x7f267406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.743+0000 7f267effd700 1 -- 172.21.15.32:0/2578698385 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f267808c100 con 0x7f2688102e70 2024-04-06T14:52:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.743+0000 7f268e192700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f267406cc20 0x7f267406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.744+0000 7f268e192700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f267406cc20 0x7f267406f0d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2684004560 tx=0x7f2684009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.748+0000 7f267effd700 1 -- 172.21.15.32:0/2578698385 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2678056140 con 0x7f2688102e70 2024-04-06T14:52:36.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.961+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f26880fdda0 con 0x7f267406cc20 2024-04-06T14:52:36.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.971+0000 7f267effd700 1 -- 172.21.15.32:0/2578698385 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f26880fdda0 con 0x7f267406cc20 2024-04-06T14:52:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:52:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (32m) 10m ago 44m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:52:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (30m) 10m ago 44m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:52:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (30m) 10m ago 43m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:52:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (32m) 10m ago 43m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:52:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (26m) 10m ago 38m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:52:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (26m) 10m ago 38m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:52:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (26m) 10m ago 38m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:52:36.959 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (11m) 10m ago 38m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (34m) 10m ago 45m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (33m) 10m ago 43m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (31m) 10m ago 45m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (30m) 10m ago 43m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (33m) 10m ago 44m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (33m) 10m ago 43m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (29m) 10m ago 42m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (29m) 10m ago 41m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (28m) 10m ago 41m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (28m) 10m ago 41m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (27m) 10m ago 40m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (26m) 10m ago 40m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:52:36.960 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (33m) 10m ago 43m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:52:36.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f267406cc20 msgr2=0x7f267406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f267406cc20 0x7f267406f0d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2684004560 tx=0x7f2684009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:36.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 msgr2=0x7f26880fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:36.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 0x7f26880fcbc0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f267800c8c0 tx=0x7f26780044c0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 shutdown_connections 2024-04-06T14:52:36.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f267406cc20 0x7f267406f0d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26880fb4a0 0x7f26880fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 --2- 172.21.15.32:0/2578698385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2688102e70 0x7f26880fcbc0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:36.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 >> 172.21.15.32:0/2578698385 conn(0x7f26880714c0 msgr2=0x7f26880fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:36.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 shutdown_connections 2024-04-06T14:52:36.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:36.973+0000 7f26903f6700 1 -- 172.21.15.32:0/2578698385 wait complete. 2024-04-06T14:52:37.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.075+0000 7f1d04717700 1 -- 172.21.15.32:0/165392596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc102e10 msgr2=0x7f1cfc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.075+0000 7f1d04717700 1 --2- 172.21.15.32:0/165392596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc102e10 0x7f1cfc1031e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1cec004940 tx=0x7f1cec009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:37.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.075+0000 7f1d04717700 1 -- 172.21.15.32:0/165392596 shutdown_connections 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.075+0000 7f1d04717700 1 --2- 172.21.15.32:0/165392596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cfc1037b0 0x7f1cfc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.075+0000 7f1d04717700 1 --2- 172.21.15.32:0/165392596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc102e10 0x7f1cfc1031e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.075+0000 7f1d04717700 1 -- 172.21.15.32:0/165392596 >> 172.21.15.32:0/165392596 conn(0x7f1cfc0f8980 msgr2=0x7f1cfc0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.075+0000 7f1d04717700 1 -- 172.21.15.32:0/165392596 shutdown_connections 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.075+0000 7f1d04717700 1 -- 172.21.15.32:0/165392596 wait complete. 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.075+0000 7f1d04717700 1 Processor -- start 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d04717700 1 -- start start 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d04717700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cfc102e10 0x7f1cfc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d04717700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc1037b0 0x7f1cfc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d04717700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc070d90 con 0x7f1cfc1037b0 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d04717700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc070ed0 con 0x7f1cfc102e10 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d01cb2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc1037b0 0x7f1cfc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d01cb2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc1037b0 0x7f1cfc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34894/0 (socket says 172.21.15.32:34894) 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d01cb2700 1 -- 172.21.15.32:0/2171592067 learned_addr learned my addr 172.21.15.32:0/2171592067 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:37.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d024b3700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cfc102e10 0x7f1cfc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d01cb2700 1 -- 172.21.15.32:0/2171592067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cfc102e10 msgr2=0x7f1cfc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d01cb2700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cfc102e10 0x7f1cfc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d01cb2700 1 -- 172.21.15.32:0/2171592067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec005040 con 0x7f1cfc1037b0 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d024b3700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cfc102e10 0x7f1cfc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.076+0000 7f1d01cb2700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc1037b0 0x7f1cfc070800 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800c800 tx=0x7f1cf8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.077+0000 7f1cf2ffd700 1 -- 172.21.15.32:0/2171592067 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cf800a760 con 0x7f1cfc1037b0 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.077+0000 7f1cf2ffd700 1 -- 172.21.15.32:0/2171592067 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cf800ad70 con 0x7f1cfc1037b0 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.077+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cfc119dd0 con 0x7f1cfc1037b0 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.077+0000 7f1cf2ffd700 1 -- 172.21.15.32:0/2171592067 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cf80113a0 con 0x7f1cfc1037b0 2024-04-06T14:52:37.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.077+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cfc11a250 con 0x7f1cfc1037b0 2024-04-06T14:52:37.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.077+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cfc0fe560 con 0x7f1cfc1037b0 2024-04-06T14:52:37.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.078+0000 7f1cf2ffd700 1 -- 172.21.15.32:0/2171592067 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1cf8012400 con 0x7f1cfc1037b0 2024-04-06T14:52:37.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.079+0000 7f1cf2ffd700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ce806cc20 0x7f1ce806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:37.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.079+0000 7f1cf2ffd700 1 -- 172.21.15.32:0/2171592067 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1cf808bf60 con 0x7f1cfc1037b0 2024-04-06T14:52:37.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.079+0000 7f1d024b3700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ce806cc20 0x7f1ce806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:37.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.079+0000 7f1d024b3700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ce806cc20 0x7f1ce806f0d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1cec004560 tx=0x7f1cec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:37.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.083+0000 7f1cf2ffd700 1 -- 172.21.15.32:0/2171592067 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1cf80bfab0 con 0x7f1cfc1037b0 2024-04-06T14:52:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.371+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1cfc04bed0 con 0x7f1cfc1037b0 2024-04-06T14:52:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.371+0000 7f1cf2ffd700 1 -- 172.21.15.32:0/2171592067 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f1cf8056060 con 0x7f1cfc1037b0 2024-04-06T14:52:37.359 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:52:37.359 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:52:37.360 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:52:37.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.373+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ce806cc20 msgr2=0x7f1ce806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.373+0000 7f1d04717700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ce806cc20 0x7f1ce806f0d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1cec004560 tx=0x7f1cec009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:37.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.373+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc1037b0 msgr2=0x7f1cfc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.373+0000 7f1d04717700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc1037b0 0x7f1cfc070800 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800c800 tx=0x7f1cf8004470 comp rx=0 tx=0).stop 2024-04-06T14:52:37.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.373+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 shutdown_connections 2024-04-06T14:52:37.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.373+0000 7f1d04717700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1ce806cc20 0x7f1ce806f0d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.373+0000 7f1d04717700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cfc102e10 0x7f1cfc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.373+0000 7f1d04717700 1 --2- 172.21.15.32:0/2171592067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cfc1037b0 0x7f1cfc070800 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.373+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 >> 172.21.15.32:0/2171592067 conn(0x7f1cfc0f8980 msgr2=0x7f1cfc0f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:37.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.374+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 shutdown_connections 2024-04-06T14:52:37.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.374+0000 7f1d04717700 1 -- 172.21.15.32:0/2171592067 wait complete. 2024-04-06T14:52:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.477+0000 7f28fba64700 1 -- 172.21.15.32:0/281283969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 msgr2=0x7f28f40fe1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.477+0000 7f28fba64700 1 --2- 172.21.15.32:0/281283969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 0x7f28f40fe1a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f28f0004940 tx=0x7f28f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.477+0000 7f28fba64700 1 -- 172.21.15.32:0/281283969 shutdown_connections 2024-04-06T14:52:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.477+0000 7f28fba64700 1 --2- 172.21.15.32:0/281283969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28f40fe770 0x7f28f4106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.477+0000 7f28fba64700 1 --2- 172.21.15.32:0/281283969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 0x7f28f40fe1a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.477+0000 7f28fba64700 1 -- 172.21.15.32:0/281283969 >> 172.21.15.32:0/281283969 conn(0x7f28f40f7870 msgr2=0x7f28f40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.477+0000 7f28fba64700 1 -- 172.21.15.32:0/281283969 shutdown_connections 2024-04-06T14:52:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.477+0000 7f28fba64700 1 -- 172.21.15.32:0/281283969 wait complete. 2024-04-06T14:52:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28fba64700 1 Processor -- start 2024-04-06T14:52:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28fba64700 1 -- start start 2024-04-06T14:52:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28fba64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 0x7f28f40fbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28fba64700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28f40fe770 0x7f28f40fc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28fba64700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40fdbe0 con 0x7f28f40fddd0 2024-04-06T14:52:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28fba64700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40fc710 con 0x7f28f40fe770 2024-04-06T14:52:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28f9800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 0x7f28f40fbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28f9800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 0x7f28f40fbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34916/0 (socket says 172.21.15.32:34916) 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28f9800700 1 -- 172.21.15.32:0/1102372287 learned_addr learned my addr 172.21.15.32:0/1102372287 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.478+0000 7f28f8fff700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28f40fe770 0x7f28f40fc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28f9800700 1 -- 172.21.15.32:0/1102372287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28f40fe770 msgr2=0x7f28f40fc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28f9800700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28f40fe770 0x7f28f40fc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28f9800700 1 -- 172.21.15.32:0/1102372287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f0005040 con 0x7f28f40fddd0 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28f8fff700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28f40fe770 0x7f28f40fc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28f9800700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 0x7f28f40fbc90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f28f0004890 tx=0x7f28f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28e9ffb700 1 -- 172.21.15.32:0/1102372287 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28f000e040 con 0x7f28f40fddd0 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28e9ffb700 1 -- 172.21.15.32:0/1102372287 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28f0009af0 con 0x7f28f40fddd0 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f40fc990 con 0x7f28f40fddd0 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28e9ffb700 1 -- 172.21.15.32:0/1102372287 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28f001c3f0 con 0x7f28f40fddd0 2024-04-06T14:52:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.479+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f4119f30 con 0x7f28f40fddd0 2024-04-06T14:52:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.480+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f4066b50 con 0x7f28f40fddd0 2024-04-06T14:52:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.481+0000 7f28e9ffb700 1 -- 172.21.15.32:0/1102372287 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f28f0013420 con 0x7f28f40fddd0 2024-04-06T14:52:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.481+0000 7f28e9ffb700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f28e006cc20 0x7f28e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.481+0000 7f28e9ffb700 1 -- 172.21.15.32:0/1102372287 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f28f008d400 con 0x7f28f40fddd0 2024-04-06T14:52:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.481+0000 7f28f8fff700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f28e006cc20 0x7f28e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.481+0000 7f28f8fff700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f28e006cc20 0x7f28e006f0d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009fd0 tx=0x7f28e400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:37.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.486+0000 7f28e9ffb700 1 -- 172.21.15.32:0/1102372287 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f28f0057440 con 0x7f28f40fddd0 2024-04-06T14:52:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:37 smithi032 ceph-mon[147414]: pgmap v975: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:37 smithi032 ceph-mon[147414]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:37 smithi032 ceph-mon[147414]: from='client.35770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:37 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2171592067' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:52:37.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.731+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f28f411a120 con 0x7f28f40fddd0 2024-04-06T14:52:37.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.732+0000 7f28e9ffb700 1 -- 172.21.15.32:0/1102372287 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f28f005aa60 con 0x7f28f40fddd0 2024-04-06T14:52:37.719 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:52:37.719 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:52:37.719 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:52:37.719 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:52:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:52:37.720 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:52:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:52:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:52:37.722 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:52:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:52:37.722 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:52:37.722 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:52:37.722 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:52:37.722 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f28e006cc20 msgr2=0x7f28e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f28e006cc20 0x7f28e006f0d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009fd0 tx=0x7f28e400b040 comp rx=0 tx=0).stop 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 msgr2=0x7f28f40fbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 0x7f28f40fbc90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f28f0004890 tx=0x7f28f0009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 shutdown_connections 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f28e006cc20 0x7f28e006f0d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f40fddd0 0x7f28f40fbc90 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 --2- 172.21.15.32:0/1102372287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28f40fe770 0x7f28f40fc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 >> 172.21.15.32:0/1102372287 conn(0x7f28f40f7870 msgr2=0x7f28f40f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 shutdown_connections 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.734+0000 7f28fba64700 1 -- 172.21.15.32:0/1102372287 wait complete. 2024-04-06T14:52:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 -- 172.21.15.32:0/1333452021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc101d70 msgr2=0x7f3ccc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 --2- 172.21.15.32:0/1333452021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc101d70 0x7f3ccc102140 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc004940 tx=0x7f3cbc009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 -- 172.21.15.32:0/1333452021 shutdown_connections 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 --2- 172.21.15.32:0/1333452021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc102710 0x7f3ccc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 --2- 172.21.15.32:0/1333452021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc101d70 0x7f3ccc102140 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 -- 172.21.15.32:0/1333452021 >> 172.21.15.32:0/1333452021 conn(0x7f3ccc072bc0 msgr2=0x7f3ccc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 -- 172.21.15.32:0/1333452021 shutdown_connections 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 -- 172.21.15.32:0/1333452021 wait complete. 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 Processor -- start 2024-04-06T14:52:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.838+0000 7f3cd1222700 1 -- start start 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cd1222700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc101d70 0x7f3ccc102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cd1222700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc102710 0x7f3ccc103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cd1222700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc103650 con 0x7f3ccc102710 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cd1222700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc1037c0 con 0x7f3ccc101d70 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cca59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc102710 0x7f3ccc103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cca59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc102710 0x7f3ccc103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34942/0 (socket says 172.21.15.32:34942) 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cca59c700 1 -- 172.21.15.32:0/1438998641 learned_addr learned my addr 172.21.15.32:0/1438998641 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3ccad9d700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc101d70 0x7f3ccc102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cca59c700 1 -- 172.21.15.32:0/1438998641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc101d70 msgr2=0x7f3ccc102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cca59c700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc101d70 0x7f3ccc102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cca59c700 1 -- 172.21.15.32:0/1438998641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc005040 con 0x7f3ccc102710 2024-04-06T14:52:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cca59c700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc102710 0x7f3ccc103110 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0002a50 tx=0x7f3cc00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cb37fe700 1 -- 172.21.15.32:0/1438998641 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cc0007cc0 con 0x7f3ccc102710 2024-04-06T14:52:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ccc119d20 con 0x7f3ccc102710 2024-04-06T14:52:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cb37fe700 1 -- 172.21.15.32:0/1438998641 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cc000f040 con 0x7f3ccc102710 2024-04-06T14:52:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cb37fe700 1 -- 172.21.15.32:0/1438998641 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cc000da30 con 0x7f3ccc102710 2024-04-06T14:52:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.839+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc11a120 con 0x7f3ccc102710 2024-04-06T14:52:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.840+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc0fd4c0 con 0x7f3ccc102710 2024-04-06T14:52:37.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.841+0000 7f3cb37fe700 1 -- 172.21.15.32:0/1438998641 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3cc000db90 con 0x7f3ccc102710 2024-04-06T14:52:37.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.841+0000 7f3cb37fe700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cb406cc30 0x7f3cb406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:37.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.841+0000 7f3cb37fe700 1 -- 172.21.15.32:0/1438998641 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3cc008bd00 con 0x7f3ccc102710 2024-04-06T14:52:37.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.842+0000 7f3ccad9d700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cb406cc30 0x7f3cb406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:37.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.842+0000 7f3ccad9d700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cb406cc30 0x7f3cb406f0e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc009fd0 tx=0x7f3cbc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:37.846+0000 7f3cb37fe700 1 -- 172.21.15.32:0/1438998641 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3cc0055c40 con 0x7f3ccc102710 2024-04-06T14:52:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:37 smithi086 ceph-mon[110975]: pgmap v975: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:37 smithi086 ceph-mon[110975]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:37 smithi086 ceph-mon[110975]: from='client.35770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:37 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2171592067' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:52:38.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.067+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ccc04bf70 con 0x7f3cb406cc30 2024-04-06T14:52:38.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.068+0000 7f3cb37fe700 1 -- 172.21.15.32:0/1438998641 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3cbc0096f0 con 0x7f3cb406cc30 2024-04-06T14:52:38.055 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:52:38.056 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:52:38.057 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:52:38.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cb406cc30 msgr2=0x7f3cb406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:38.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cb406cc30 0x7f3cb406f0e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc009fd0 tx=0x7f3cbc009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:38.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc102710 msgr2=0x7f3ccc103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:38.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc102710 0x7f3ccc103110 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0002a50 tx=0x7f3cc00046b0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 shutdown_connections 2024-04-06T14:52:38.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3cb406cc30 0x7f3cb406f0e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc101d70 0x7f3ccc102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 --2- 172.21.15.32:0/1438998641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ccc102710 0x7f3ccc103110 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 >> 172.21.15.32:0/1438998641 conn(0x7f3ccc072bc0 msgr2=0x7f3ccc065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:38.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 shutdown_connections 2024-04-06T14:52:38.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.070+0000 7f3cd1222700 1 -- 172.21.15.32:0/1438998641 wait complete. 2024-04-06T14:52:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.172+0000 7fa3581c7700 1 -- 172.21.15.32:0/4217573721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fe7a0 msgr2=0x7fa350106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 --2- 172.21.15.32:0/4217573721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fe7a0 0x7fa350106de0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa340004940 tx=0x7fa340009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 -- 172.21.15.32:0/4217573721 shutdown_connections 2024-04-06T14:52:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 --2- 172.21.15.32:0/4217573721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fe7a0 0x7fa350106de0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 --2- 172.21.15.32:0/4217573721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3500fde00 0x7fa3500fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 -- 172.21.15.32:0/4217573721 >> 172.21.15.32:0/4217573721 conn(0x7fa3500f78d0 msgr2=0x7fa3500f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 -- 172.21.15.32:0/4217573721 shutdown_connections 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 -- 172.21.15.32:0/4217573721 wait complete. 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 Processor -- start 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 -- start start 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.173+0000 7fa3581c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fde00 0x7fa3500fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa3581c7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3500fe7a0 0x7fa3500fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa3581c7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500fc700 con 0x7fa3500fde00 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa3581c7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500fc840 con 0x7fa3500fe7a0 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa355f63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fde00 0x7fa3500fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa355f63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fde00 0x7fa3500fbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34952/0 (socket says 172.21.15.32:34952) 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa355f63700 1 -- 172.21.15.32:0/532200189 learned_addr learned my addr 172.21.15.32:0/532200189 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa355762700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3500fe7a0 0x7fa3500fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa355f63700 1 -- 172.21.15.32:0/532200189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3500fe7a0 msgr2=0x7fa3500fc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa355f63700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3500fe7a0 0x7fa3500fc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa355f63700 1 -- 172.21.15.32:0/532200189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa340005040 con 0x7fa3500fde00 2024-04-06T14:52:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa355f63700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fde00 0x7fa3500fbc30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00c8c0 tx=0x7fa34c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa3467fc700 1 -- 172.21.15.32:0/532200189 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa34c00a980 con 0x7fa3500fde00 2024-04-06T14:52:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa350119d50 con 0x7fa3500fde00 2024-04-06T14:52:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa3467fc700 1 -- 172.21.15.32:0/532200189 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa34c00ce60 con 0x7fa3500fde00 2024-04-06T14:52:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa3467fc700 1 -- 172.21.15.32:0/532200189 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa34c0113a0 con 0x7fa3500fde00 2024-04-06T14:52:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.174+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35011a150 con 0x7fa3500fde00 2024-04-06T14:52:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.175+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa350066770 con 0x7fa3500fde00 2024-04-06T14:52:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.176+0000 7fa3467fc700 1 -- 172.21.15.32:0/532200189 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa34c012400 con 0x7fa3500fde00 2024-04-06T14:52:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.176+0000 7fa3467fc700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa33c06cc30 0x7fa33c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:52:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.176+0000 7fa3467fc700 1 -- 172.21.15.32:0/532200189 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa34c08c060 con 0x7fa3500fde00 2024-04-06T14:52:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.176+0000 7fa355762700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa33c06cc30 0x7fa33c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:52:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.177+0000 7fa355762700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa33c06cc30 0x7fa33c06f0e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa3500fd570 tx=0x7fa340009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:52:38.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.181+0000 7fa3467fc700 1 -- 172.21.15.32:0/532200189 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa34c056160 con 0x7fa3500fde00 2024-04-06T14:52:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.468+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa35005f5f0 con 0x7fa3500fde00 2024-04-06T14:52:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.468+0000 7fa3467fc700 1 -- 172.21.15.32:0/532200189 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7fa35005f5f0 con 0x7fa3500fde00 2024-04-06T14:52:38.468 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:52:38.468 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:52:38.468 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:52:38.469 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:52:38.470 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:52:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa33c06cc30 msgr2=0x7fa33c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa33c06cc30 0x7fa33c06f0e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa3500fd570 tx=0x7fa340009e30 comp rx=0 tx=0).stop 2024-04-06T14:52:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fde00 msgr2=0x7fa3500fbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:52:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fde00 0x7fa3500fbc30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00c8c0 tx=0x7fa34c0044c0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 shutdown_connections 2024-04-06T14:52:38.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa33c06cc30 0x7fa33c06f0e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3500fde00 0x7fa3500fbc30 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 --2- 172.21.15.32:0/532200189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3500fe7a0 0x7fa3500fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:52:38.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 >> 172.21.15.32:0/532200189 conn(0x7fa3500f78d0 msgr2=0x7fa3500f8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:52:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 shutdown_connections 2024-04-06T14:52:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:52:38.471+0000 7fa3581c7700 1 -- 172.21.15.32:0/532200189 wait complete. 2024-04-06T14:52:38.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:38 smithi032 ceph-mon[147414]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:38.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:52:38.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:38 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1102372287' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:52:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:38 smithi086 ceph-mon[110975]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:52:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:38 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1102372287' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:52:39.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:39 smithi032 ceph-mon[147414]: from='client.35786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:39.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:39 smithi032 ceph-mon[147414]: pgmap v976: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:39.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:39 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/532200189' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:52:39.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:39 smithi086 ceph-mon[110975]: from='client.35786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:52:39.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:39 smithi086 ceph-mon[110975]: pgmap v976: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:39.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:39 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/532200189' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:52:41.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:41 smithi032 ceph-mon[147414]: pgmap v977: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:41.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:41 smithi086 ceph-mon[110975]: pgmap v977: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:43.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:43 smithi032 ceph-mon[147414]: pgmap v978: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:43.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:43 smithi086 ceph-mon[110975]: pgmap v978: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:45.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:45 smithi032 ceph-mon[147414]: pgmap v979: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:45.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:45 smithi086 ceph-mon[110975]: pgmap v979: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:47.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:47 smithi032 ceph-mon[147414]: pgmap v980: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:47.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:47 smithi086 ceph-mon[110975]: pgmap v980: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:49.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:49 smithi032 ceph-mon[147414]: pgmap v981: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:49.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:49 smithi086 ceph-mon[110975]: pgmap v981: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:51.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:51 smithi032 ceph-mon[147414]: pgmap v982: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:51.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:51 smithi086 ceph-mon[110975]: pgmap v982: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:53.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:53 smithi032 ceph-mon[147414]: pgmap v983: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:53.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:52:53.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:53 smithi086 ceph-mon[110975]: pgmap v983: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:53.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:52:55.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:55 smithi032 ceph-mon[147414]: pgmap v984: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:55.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:55 smithi086 ceph-mon[110975]: pgmap v984: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:57.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:57 smithi032 ceph-mon[147414]: pgmap v985: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:57.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:57 smithi086 ceph-mon[110975]: pgmap v985: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:59.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:52:59 smithi032 ceph-mon[147414]: pgmap v986: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:52:59.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:52:59 smithi086 ceph-mon[110975]: pgmap v986: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:01.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:01 smithi086 ceph-mon[110975]: pgmap v987: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:01.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:01 smithi032 ceph-mon[147414]: pgmap v987: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:03.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:03 smithi032 ceph-mon[147414]: pgmap v988: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:03.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:03 smithi086 ceph-mon[110975]: pgmap v988: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:05.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:05 smithi086 ceph-mon[110975]: pgmap v989: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:05.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:05 smithi032 ceph-mon[147414]: pgmap v989: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:07.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:07 smithi086 ceph-mon[110975]: pgmap v990: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:07.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:07 smithi032 ceph-mon[147414]: pgmap v990: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.589+0000 7f46accd8700 1 -- 172.21.15.32:0/3515261789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a8102e00 msgr2=0x7f46a81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.589+0000 7f46accd8700 1 --2- 172.21.15.32:0/3515261789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a8102e00 0x7f46a81031d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f4698004940 tx=0x7f4698009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.589+0000 7f46accd8700 1 -- 172.21.15.32:0/3515261789 shutdown_connections 2024-04-06T14:53:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.589+0000 7f46accd8700 1 --2- 172.21.15.32:0/3515261789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a81037a0 0x7f46a80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.589+0000 7f46accd8700 1 --2- 172.21.15.32:0/3515261789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a8102e00 0x7f46a81031d0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.589+0000 7f46accd8700 1 -- 172.21.15.32:0/3515261789 >> 172.21.15.32:0/3515261789 conn(0x7f46a80f89b0 msgr2=0x7f46a80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.589+0000 7f46accd8700 1 -- 172.21.15.32:0/3515261789 shutdown_connections 2024-04-06T14:53:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.589+0000 7f46accd8700 1 -- 172.21.15.32:0/3515261789 wait complete. 2024-04-06T14:53:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.589+0000 7f46accd8700 1 Processor -- start 2024-04-06T14:53:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46accd8700 1 -- start start 2024-04-06T14:53:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46accd8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a8102e00 0x7f46a80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46accd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a81037a0 0x7f46a8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46accd8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8072210 con 0x7f46a81037a0 2024-04-06T14:53:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46accd8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8070d40 con 0x7f46a8102e00 2024-04-06T14:53:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46a5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a81037a0 0x7f46a8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46a5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a81037a0 0x7f46a8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54762/0 (socket says 172.21.15.32:54762) 2024-04-06T14:53:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46a5d9b700 1 -- 172.21.15.32:0/2359567856 learned_addr learned my addr 172.21.15.32:0/2359567856 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46a659c700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a8102e00 0x7f46a80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46a5d9b700 1 -- 172.21.15.32:0/2359567856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a8102e00 msgr2=0x7f46a80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46a5d9b700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a8102e00 0x7f46a80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46a5d9b700 1 -- 172.21.15.32:0/2359567856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4698005040 con 0x7f46a81037a0 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.590+0000 7f46a5d9b700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a81037a0 0x7f46a8070800 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f469c002a50 tx=0x7f469c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.591+0000 7f4696ffd700 1 -- 172.21.15.32:0/2359567856 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f469c007cc0 con 0x7f46a81037a0 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.591+0000 7f4696ffd700 1 -- 172.21.15.32:0/2359567856 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f469c00f040 con 0x7f46a81037a0 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.591+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a8071020 con 0x7f46a81037a0 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.591+0000 7f4696ffd700 1 -- 172.21.15.32:0/2359567856 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f469c00d920 con 0x7f46a81037a0 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.591+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a811a060 con 0x7f46a81037a0 2024-04-06T14:53:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.592+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a80fe550 con 0x7f46a81037a0 2024-04-06T14:53:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.592+0000 7f4696ffd700 1 -- 172.21.15.32:0/2359567856 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f469c009680 con 0x7f46a81037a0 2024-04-06T14:53:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.593+0000 7f4696ffd700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f469006cc20 0x7f469006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.593+0000 7f4696ffd700 1 -- 172.21.15.32:0/2359567856 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f469c08be80 con 0x7f46a81037a0 2024-04-06T14:53:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.593+0000 7f46a659c700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f469006cc20 0x7f469006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.593+0000 7f46a659c700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f469006cc20 0x7f469006f0d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4698004f90 tx=0x7f4698009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.598+0000 7f4696ffd700 1 -- 172.21.15.32:0/2359567856 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f469c055dc0 con 0x7f46a81037a0 2024-04-06T14:53:08.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.822+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f46a80fa180 con 0x7f469006cc20 2024-04-06T14:53:08.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.823+0000 7f4696ffd700 1 -- 172.21.15.32:0/2359567856 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f46a80fa180 con 0x7f469006cc20 2024-04-06T14:53:08.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.825+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f469006cc20 msgr2=0x7f469006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:08.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.825+0000 7f46accd8700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f469006cc20 0x7f469006f0d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4698004f90 tx=0x7f4698009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:08.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.825+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a81037a0 msgr2=0x7f46a8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:08.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.825+0000 7f46accd8700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a81037a0 0x7f46a8070800 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f469c002a50 tx=0x7f469c004600 comp rx=0 tx=0).stop 2024-04-06T14:53:08.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.825+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 shutdown_connections 2024-04-06T14:53:08.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.825+0000 7f46accd8700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f469006cc20 0x7f469006f0d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:08.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.825+0000 7f46accd8700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46a8102e00 0x7f46a80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:08.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.825+0000 7f46accd8700 1 --2- 172.21.15.32:0/2359567856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a81037a0 0x7f46a8070800 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:08.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.825+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 >> 172.21.15.32:0/2359567856 conn(0x7f46a80f89b0 msgr2=0x7f46a80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:08.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.826+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 shutdown_connections 2024-04-06T14:53:08.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.826+0000 7f46accd8700 1 -- 172.21.15.32:0/2359567856 wait complete. 2024-04-06T14:53:08.823 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:53:08.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:53:08.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.930+0000 7f297f71d700 1 -- 172.21.15.32:0/1460947942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 msgr2=0x7f2978102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.930+0000 7f297f71d700 1 --2- 172.21.15.32:0/1460947942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 0x7f2978102140 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2974004940 tx=0x7f2974009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.930+0000 7f297f71d700 1 -- 172.21.15.32:0/1460947942 shutdown_connections 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.930+0000 7f297f71d700 1 --2- 172.21.15.32:0/1460947942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2978102710 0x7f29780fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.930+0000 7f297f71d700 1 --2- 172.21.15.32:0/1460947942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 0x7f2978102140 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.930+0000 7f297f71d700 1 -- 172.21.15.32:0/1460947942 >> 172.21.15.32:0/1460947942 conn(0x7f2978072bc0 msgr2=0x7f2978072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.930+0000 7f297f71d700 1 -- 172.21.15.32:0/1460947942 shutdown_connections 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.930+0000 7f297f71d700 1 -- 172.21.15.32:0/1460947942 wait complete. 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297f71d700 1 Processor -- start 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297f71d700 1 -- start start 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297f71d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 0x7f2978102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297f71d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2978102710 0x7f2978103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297f71d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29781036a0 con 0x7f2978101d70 2024-04-06T14:53:08.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297f71d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29781037e0 con 0x7f2978102710 2024-04-06T14:53:08.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297d4b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 0x7f2978102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:08.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297d4b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 0x7f2978102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54782/0 (socket says 172.21.15.32:54782) 2024-04-06T14:53:08.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297d4b9700 1 -- 172.21.15.32:0/4110004685 learned_addr learned my addr 172.21.15.32:0/4110004685 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:08.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297d4b9700 1 -- 172.21.15.32:0/4110004685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2978102710 msgr2=0x7f2978103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:08.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297d4b9700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2978102710 0x7f2978103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:08.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297d4b9700 1 -- 172.21.15.32:0/4110004685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2974005040 con 0x7f2978101d70 2024-04-06T14:53:08.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.931+0000 7f297d4b9700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 0x7f2978102bd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2974009360 tx=0x7f2974009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:08.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.932+0000 7f296dffb700 1 -- 172.21.15.32:0/4110004685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f297400e040 con 0x7f2978101d70 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.932+0000 7f296dffb700 1 -- 172.21.15.32:0/4110004685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2974013420 con 0x7f2978101d70 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.932+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2978103a30 con 0x7f2978101d70 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.932+0000 7f296dffb700 1 -- 172.21.15.32:0/4110004685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f297401c5f0 con 0x7f2978101d70 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.932+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297811a210 con 0x7f2978101d70 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.933+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29780fd4c0 con 0x7f2978101d70 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.933+0000 7f296dffb700 1 -- 172.21.15.32:0/4110004685 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2974013590 con 0x7f2978101d70 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.934+0000 7f296dffb700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f296406cc30 0x7f296406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.934+0000 7f296dffb700 1 -- 172.21.15.32:0/4110004685 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f297408d240 con 0x7f2978101d70 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.934+0000 7f297ccb8700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f296406cc30 0x7f296406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.934+0000 7f297ccb8700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f296406cc30 0x7f296406f0e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2978104510 tx=0x7f2968009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:08.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:08.938+0000 7f296dffb700 1 -- 172.21.15.32:0/4110004685 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f29740572c0 con 0x7f2978101d70 2024-04-06T14:53:08.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:53:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.159+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f297804bed0 con 0x7f296406cc30 2024-04-06T14:53:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.160+0000 7f296dffb700 1 -- 172.21.15.32:0/4110004685 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f297804bed0 con 0x7f296406cc30 2024-04-06T14:53:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f296406cc30 msgr2=0x7f296406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f296406cc30 0x7f296406f0e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2978104510 tx=0x7f2968009ce0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 msgr2=0x7f2978102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 0x7f2978102bd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2974009360 tx=0x7f2974009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 shutdown_connections 2024-04-06T14:53:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f296406cc30 0x7f296406f0e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2978101d70 0x7f2978102bd0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 --2- 172.21.15.32:0/4110004685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2978102710 0x7f2978103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 >> 172.21.15.32:0/4110004685 conn(0x7f2978072bc0 msgr2=0x7f29780659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 shutdown_connections 2024-04-06T14:53:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.162+0000 7f297f71d700 1 -- 172.21.15.32:0/4110004685 wait complete. 2024-04-06T14:53:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.265+0000 7f05732ab700 1 -- 172.21.15.32:0/2942397554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 msgr2=0x7f056c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.265+0000 7f05732ab700 1 --2- 172.21.15.32:0/2942397554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 0x7f056c0fce20 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0560004940 tx=0x7f0560009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.265+0000 7f05732ab700 1 -- 172.21.15.32:0/2942397554 shutdown_connections 2024-04-06T14:53:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.265+0000 7f05732ab700 1 --2- 172.21.15.32:0/2942397554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 0x7f056c0fce20 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.265+0000 7f05732ab700 1 --2- 172.21.15.32:0/2942397554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f056c102e10 0x7f056c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.265+0000 7f05732ab700 1 -- 172.21.15.32:0/2942397554 >> 172.21.15.32:0/2942397554 conn(0x7f056c0f89a0 msgr2=0x7f056c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.265+0000 7f05732ab700 1 -- 172.21.15.32:0/2942397554 shutdown_connections 2024-04-06T14:53:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.265+0000 7f05732ab700 1 -- 172.21.15.32:0/2942397554 wait complete. 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f05732ab700 1 Processor -- start 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f05732ab700 1 -- start start 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f05732ab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f056c102e10 0x7f056c113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f05732ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 0x7f056c114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f05732ab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c114660 con 0x7f056c1037b0 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f05732ab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c10da50 con 0x7f056c102e10 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f0570846700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 0x7f056c114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f0570846700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 0x7f056c114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54798/0 (socket says 172.21.15.32:54798) 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f0570846700 1 -- 172.21.15.32:0/3230651071 learned_addr learned my addr 172.21.15.32:0/3230651071 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.266+0000 7f0571047700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f056c102e10 0x7f056c113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.267+0000 7f0570846700 1 -- 172.21.15.32:0/3230651071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f056c102e10 msgr2=0x7f056c113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.267+0000 7f0570846700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f056c102e10 0x7f056c113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.267+0000 7f0570846700 1 -- 172.21.15.32:0/3230651071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0560005040 con 0x7f056c1037b0 2024-04-06T14:53:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.267+0000 7f0570846700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 0x7f056c114010 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0560009380 tx=0x7f0560009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.267+0000 7f055dffb700 1 -- 172.21.15.32:0/3230651071 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056000e040 con 0x7f056c1037b0 2024-04-06T14:53:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.267+0000 7f055dffb700 1 -- 172.21.15.32:0/3230651071 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0560013420 con 0x7f056c1037b0 2024-04-06T14:53:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.267+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c10dcd0 con 0x7f056c1037b0 2024-04-06T14:53:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.267+0000 7f055dffb700 1 -- 172.21.15.32:0/3230651071 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056001c630 con 0x7f056c1037b0 2024-04-06T14:53:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.267+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c10e290 con 0x7f056c1037b0 2024-04-06T14:53:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.268+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c0fe560 con 0x7f056c1037b0 2024-04-06T14:53:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.269+0000 7f055dffb700 1 -- 172.21.15.32:0/3230651071 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f05600139b0 con 0x7f056c1037b0 2024-04-06T14:53:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.269+0000 7f055dffb700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f055806cc20 0x7f055806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.269+0000 7f055dffb700 1 -- 172.21.15.32:0/3230651071 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f056008d1b0 con 0x7f056c1037b0 2024-04-06T14:53:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.269+0000 7f0571047700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f055806cc20 0x7f055806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.269+0000 7f0571047700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f055806cc20 0x7f055806f0d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0568004f40 tx=0x7f056800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:09.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.274+0000 7f055dffb700 1 -- 172.21.15.32:0/3230651071 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f056005ac80 con 0x7f056c1037b0 2024-04-06T14:53:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.486+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f056c109db0 con 0x7f055806cc20 2024-04-06T14:53:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.496+0000 7f055dffb700 1 -- 172.21.15.32:0/3230651071 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f056c109db0 con 0x7f055806cc20 2024-04-06T14:53:09.483 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (33m) 11m ago 44m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (30m) 11m ago 45m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (30m) 11m ago 44m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (33m) 11m ago 44m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (27m) 11m ago 39m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (26m) 11m ago 38m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (26m) 11m ago 39m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (11m) 11m ago 38m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (35m) 11m ago 46m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (34m) 11m ago 43m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (31m) 11m ago 46m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (31m) 11m ago 43m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (34m) 11m ago 44m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:53:09.484 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (34m) 11m ago 43m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:53:09.485 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (30m) 11m ago 43m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:53:09.485 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (29m) 11m ago 42m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:53:09.485 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (29m) 11m ago 41m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:53:09.485 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (28m) 11m ago 41m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:53:09.485 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (28m) 11m ago 41m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:53:09.485 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (27m) 11m ago 40m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:53:09.485 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (33m) 11m ago 44m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f055806cc20 msgr2=0x7f055806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f055806cc20 0x7f055806f0d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0568004f40 tx=0x7f056800b040 comp rx=0 tx=0).stop 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 msgr2=0x7f056c114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 0x7f056c114010 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0560009380 tx=0x7f0560009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 shutdown_connections 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f055806cc20 0x7f055806f0d0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f056c102e10 0x7f056c113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 --2- 172.21.15.32:0/3230651071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f056c1037b0 0x7f056c114010 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 >> 172.21.15.32:0/3230651071 conn(0x7f056c0f89a0 msgr2=0x7f056c0f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 shutdown_connections 2024-04-06T14:53:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.498+0000 7f05732ab700 1 -- 172.21.15.32:0/3230651071 wait complete. 2024-04-06T14:53:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 -- 172.21.15.32:0/3186324933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 msgr2=0x7f59a80fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 --2- 172.21.15.32:0/3186324933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 0x7f59a80fbd90 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5998004940 tx=0x7f5998009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 -- 172.21.15.32:0/3186324933 shutdown_connections 2024-04-06T14:53:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 --2- 172.21.15.32:0/3186324933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 0x7f59a80fbd90 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 --2- 172.21.15.32:0/3186324933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59a8101d80 0x7f59a8102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 -- 172.21.15.32:0/3186324933 >> 172.21.15.32:0/3186324933 conn(0x7f59a80f78f0 msgr2=0x7f59a80f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 -- 172.21.15.32:0/3186324933 shutdown_connections 2024-04-06T14:53:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 -- 172.21.15.32:0/3186324933 wait complete. 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 Processor -- start 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.600+0000 7f59ad796700 1 -- start start 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59ad796700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59a8101d80 0x7f59a8115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59ad796700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 0x7f59a8116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59ad796700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8116650 con 0x7f59a8102720 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59ad796700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a810fc30 con 0x7f59a8101d80 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59a67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 0x7f59a8116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59a67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 0x7f59a8116000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54810/0 (socket says 172.21.15.32:54810) 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59a67fc700 1 -- 172.21.15.32:0/608710318 learned_addr learned my addr 172.21.15.32:0/608710318 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59a6ffd700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59a8101d80 0x7f59a8115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59a67fc700 1 -- 172.21.15.32:0/608710318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59a8101d80 msgr2=0x7f59a8115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59a67fc700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59a8101d80 0x7f59a8115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59a67fc700 1 -- 172.21.15.32:0/608710318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5998005040 con 0x7f59a8102720 2024-04-06T14:53:09.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59a67fc700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 0x7f59a8116000 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5998009390 tx=0x7f5998009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:09.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f598f7fe700 1 -- 172.21.15.32:0/608710318 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f599800e040 con 0x7f59a8102720 2024-04-06T14:53:09.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a810feb0 con 0x7f59a8102720 2024-04-06T14:53:09.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f598f7fe700 1 -- 172.21.15.32:0/608710318 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5998013420 con 0x7f59a8102720 2024-04-06T14:53:09.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f598f7fe700 1 -- 172.21.15.32:0/608710318 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f599801c630 con 0x7f59a8102720 2024-04-06T14:53:09.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.601+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a8110470 con 0x7f59a8102720 2024-04-06T14:53:09.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.602+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a80fd4d0 con 0x7f59a8102720 2024-04-06T14:53:09.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.603+0000 7f598f7fe700 1 -- 172.21.15.32:0/608710318 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f59980139e0 con 0x7f59a8102720 2024-04-06T14:53:09.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.603+0000 7f598f7fe700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f599006cc20 0x7f599006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:09.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.603+0000 7f598f7fe700 1 -- 172.21.15.32:0/608710318 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f599808d2f0 con 0x7f59a8102720 2024-04-06T14:53:09.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.603+0000 7f59a6ffd700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f599006cc20 0x7f599006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:09.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.604+0000 7f59a6ffd700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f599006cc20 0x7f599006f0d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f599c004d60 tx=0x7f599c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:09.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.608+0000 7f598f7fe700 1 -- 172.21.15.32:0/608710318 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f599805adc0 con 0x7f59a8102720 2024-04-06T14:53:09.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:09 smithi086 ceph-mon[110975]: pgmap v991: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.898+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f59a8108d20 con 0x7f59a8102720 2024-04-06T14:53:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.899+0000 7f598f7fe700 1 -- 172.21.15.32:0/608710318 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f599805a950 con 0x7f59a8102720 2024-04-06T14:53:09.887 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:53:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:53:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:53:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:53:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:53:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:53:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:53:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:53:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:53:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:53:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:53:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:53:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:53:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:53:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:53:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:53:09.888 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f599006cc20 msgr2=0x7f599006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f599006cc20 0x7f599006f0d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f599c004d60 tx=0x7f599c00b040 comp rx=0 tx=0).stop 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 msgr2=0x7f59a8116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 0x7f59a8116000 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5998009390 tx=0x7f5998009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 shutdown_connections 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f599006cc20 0x7f599006f0d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59a8101d80 0x7f59a8115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 --2- 172.21.15.32:0/608710318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59a8102720 0x7f59a8116000 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 >> 172.21.15.32:0/608710318 conn(0x7f59a80f78f0 msgr2=0x7f59a80665f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 shutdown_connections 2024-04-06T14:53:09.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:09.901+0000 7f59ad796700 1 -- 172.21.15.32:0/608710318 wait complete. 2024-04-06T14:53:09.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:09 smithi032 ceph-mon[147414]: pgmap v991: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.004+0000 7f86ac445700 1 -- 172.21.15.32:0/1545256839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a41037a0 msgr2=0x7f86a40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.004+0000 7f86ac445700 1 --2- 172.21.15.32:0/1545256839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a41037a0 0x7f86a40fce10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f86a0004940 tx=0x7f86a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.004+0000 7f86ac445700 1 -- 172.21.15.32:0/1545256839 shutdown_connections 2024-04-06T14:53:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.004+0000 7f86ac445700 1 --2- 172.21.15.32:0/1545256839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a41037a0 0x7f86a40fce10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.004+0000 7f86ac445700 1 --2- 172.21.15.32:0/1545256839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a4102e00 0x7f86a41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.004+0000 7f86ac445700 1 -- 172.21.15.32:0/1545256839 >> 172.21.15.32:0/1545256839 conn(0x7f86a40f89b0 msgr2=0x7f86a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.004+0000 7f86ac445700 1 -- 172.21.15.32:0/1545256839 shutdown_connections 2024-04-06T14:53:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.004+0000 7f86ac445700 1 -- 172.21.15.32:0/1545256839 wait complete. 2024-04-06T14:53:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86ac445700 1 Processor -- start 2024-04-06T14:53:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86ac445700 1 -- start start 2024-04-06T14:53:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86ac445700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a4102e00 0x7f86a40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86ac445700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a41037a0 0x7f86a4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86ac445700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4072210 con 0x7f86a4102e00 2024-04-06T14:53:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86ac445700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4070d40 con 0x7f86a41037a0 2024-04-06T14:53:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86aa1e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a4102e00 0x7f86a40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86aa1e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a4102e00 0x7f86a40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54824/0 (socket says 172.21.15.32:54824) 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86aa1e1700 1 -- 172.21.15.32:0/4189887198 learned_addr learned my addr 172.21.15.32:0/4189887198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86aa1e1700 1 -- 172.21.15.32:0/4189887198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a41037a0 msgr2=0x7f86a4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86a99e0700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a41037a0 0x7f86a4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86aa1e1700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a41037a0 0x7f86a4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.005+0000 7f86aa1e1700 1 -- 172.21.15.32:0/4189887198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0005040 con 0x7f86a4102e00 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.006+0000 7f86aa1e1700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a4102e00 0x7f86a40702c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f8694002a50 tx=0x7f86940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.006+0000 7f869affd700 1 -- 172.21.15.32:0/4189887198 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8694007cc0 con 0x7f86a4102e00 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.006+0000 7f869affd700 1 -- 172.21.15.32:0/4189887198 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f869400f040 con 0x7f86a4102e00 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.006+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a4071020 con 0x7f86a4102e00 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.006+0000 7f869affd700 1 -- 172.21.15.32:0/4189887198 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8694009540 con 0x7f86a4102e00 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.006+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a411a070 con 0x7f86a4102e00 2024-04-06T14:53:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.007+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a40fe550 con 0x7f86a4102e00 2024-04-06T14:53:09.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.007+0000 7f869affd700 1 -- 172.21.15.32:0/4189887198 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f86940096a0 con 0x7f86a4102e00 2024-04-06T14:53:09.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.008+0000 7f869affd700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f869006cc20 0x7f869006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:09.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.008+0000 7f869affd700 1 -- 172.21.15.32:0/4189887198 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f869408be60 con 0x7f86a4102e00 2024-04-06T14:53:09.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.008+0000 7f86a99e0700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f869006cc20 0x7f869006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:09.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.008+0000 7f86a99e0700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f869006cc20 0x7f869006f0d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f86a0000c00 tx=0x7f86a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:09.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.012+0000 7f869affd700 1 -- 172.21.15.32:0/4189887198 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8694055e20 con 0x7f86a4102e00 2024-04-06T14:53:10.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.261+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f86a411a260 con 0x7f86a4102e00 2024-04-06T14:53:10.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.261+0000 7f869affd700 1 -- 172.21.15.32:0/4189887198 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f8694059440 con 0x7f86a4102e00 2024-04-06T14:53:10.249 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:53:10.249 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:53:10.249 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:53:10.249 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:53:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:53:10.249 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:53:10.249 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:53:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:53:10.251 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:53:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:53:10.252 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:53:10.252 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:53:10.252 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:53:10.252 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:53:10.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f869006cc20 msgr2=0x7f869006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f869006cc20 0x7f869006f0d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f86a0000c00 tx=0x7f86a0009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a4102e00 msgr2=0x7f86a40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a4102e00 0x7f86a40702c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f8694002a50 tx=0x7f86940046b0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 shutdown_connections 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f869006cc20 0x7f869006f0d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a4102e00 0x7f86a40702c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 --2- 172.21.15.32:0/4189887198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a41037a0 0x7f86a4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 >> 172.21.15.32:0/4189887198 conn(0x7f86a40f89b0 msgr2=0x7f86a40f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 shutdown_connections 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.263+0000 7f86ac445700 1 -- 172.21.15.32:0/4189887198 wait complete. 2024-04-06T14:53:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:53:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1428397108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534101d70 msgr2=0x7fa534102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 --2- 172.21.15.32:0/1428397108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534101d70 0x7fa534102140 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fa524004940 tx=0x7fa524009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1428397108 shutdown_connections 2024-04-06T14:53:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 --2- 172.21.15.32:0/1428397108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534102710 0x7fa5340fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 --2- 172.21.15.32:0/1428397108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534101d70 0x7fa534102140 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1428397108 >> 172.21.15.32:0/1428397108 conn(0x7fa534072bc0 msgr2=0x7fa534072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1428397108 shutdown_connections 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1428397108 wait complete. 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 Processor -- start 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.367+0000 7fa53bf2a700 1 -- start start 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa53bf2a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 0x7fa534113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa53bf2a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa53bf2a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa534114610 con 0x7fa534102710 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa53bf2a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53410da00 con 0x7fa534101d70 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa5394c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa5394c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54830/0 (socket says 172.21.15.32:54830) 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa5394c5700 1 -- 172.21.15.32:0/1077141359 learned_addr learned my addr 172.21.15.32:0/1077141359 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa539cc6700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 0x7fa534113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa5394c5700 1 -- 172.21.15.32:0/1077141359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 msgr2=0x7fa534113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa5394c5700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 0x7fa534113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa5394c5700 1 -- 172.21.15.32:0/1077141359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa524005040 con 0x7fa534102710 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa5394c5700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534113fc0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa530002a50 tx=0x7fa530004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa52a7fc700 1 -- 172.21.15.32:0/1077141359 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa530007cc0 con 0x7fa534102710 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa52a7fc700 1 -- 172.21.15.32:0/1077141359 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa53000f040 con 0x7fa534102710 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.368+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53410dce0 con 0x7fa534102710 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.369+0000 7fa52a7fc700 1 -- 172.21.15.32:0/1077141359 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa53000d990 con 0x7fa534102710 2024-04-06T14:53:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.369+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53410e2a0 con 0x7fa534102710 2024-04-06T14:53:10.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.369+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5340fd4c0 con 0x7fa534102710 2024-04-06T14:53:10.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.370+0000 7fa52a7fc700 1 -- 172.21.15.32:0/1077141359 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa530009710 con 0x7fa534102710 2024-04-06T14:53:10.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.370+0000 7fa52a7fc700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52006cd00 0x7fa52006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:10.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.371+0000 7fa52a7fc700 1 -- 172.21.15.32:0/1077141359 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa53008be30 con 0x7fa534102710 2024-04-06T14:53:10.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.371+0000 7fa539cc6700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52006cd00 0x7fa52006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:10.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.371+0000 7fa539cc6700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52006cd00 0x7fa52006f1b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa52400b3b0 tx=0x7fa524009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:10.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.375+0000 7fa52a7fc700 1 -- 172.21.15.32:0/1077141359 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa5300595f0 con 0x7fa534102710 2024-04-06T14:53:10.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.596+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5340666a0 con 0x7fa52006cd00 2024-04-06T14:53:10.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.597+0000 7fa52a7fc700 1 -- 172.21.15.32:0/1077141359 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa5340666a0 con 0x7fa52006cd00 2024-04-06T14:53:10.585 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:53:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:53:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:53:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:53:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:53:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:53:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:53:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:53:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:53:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:53:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:53:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:53:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:53:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:53:10.586 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:53:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.599+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52006cd00 msgr2=0x7fa52006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.599+0000 7fa53bf2a700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52006cd00 0x7fa52006f1b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa52400b3b0 tx=0x7fa524009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.599+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 msgr2=0x7fa534113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.599+0000 7fa53bf2a700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534113fc0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa530002a50 tx=0x7fa530004600 comp rx=0 tx=0).stop 2024-04-06T14:53:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.599+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 shutdown_connections 2024-04-06T14:53:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.599+0000 7fa53bf2a700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fa52006cd00 0x7fa52006f1b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.599+0000 7fa53bf2a700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa534101d70 0x7fa534113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.599+0000 7fa53bf2a700 1 --2- 172.21.15.32:0/1077141359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102710 0x7fa534113fc0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.600+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 >> 172.21.15.32:0/1077141359 conn(0x7fa534072bc0 msgr2=0x7fa534105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:10.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.600+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 shutdown_connections 2024-04-06T14:53:10.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.600+0000 7fa53bf2a700 1 -- 172.21.15.32:0/1077141359 wait complete. 2024-04-06T14:53:10.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.703+0000 7f165fc3c700 1 -- 172.21.15.32:0/3444444245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658102710 msgr2=0x7f16580fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.703+0000 7f165fc3c700 1 --2- 172.21.15.32:0/3444444245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658102710 0x7f16580fbd80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f1648004940 tx=0x7f1648009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:10.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.703+0000 7f165fc3c700 1 -- 172.21.15.32:0/3444444245 shutdown_connections 2024-04-06T14:53:10.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.703+0000 7f165fc3c700 1 --2- 172.21.15.32:0/3444444245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658102710 0x7f16580fbd80 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.703+0000 7f165fc3c700 1 --2- 172.21.15.32:0/3444444245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1658101d70 0x7f1658102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.703+0000 7f165fc3c700 1 -- 172.21.15.32:0/3444444245 >> 172.21.15.32:0/3444444245 conn(0x7f1658072bc0 msgr2=0x7f1658072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:10.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.703+0000 7f165fc3c700 1 -- 172.21.15.32:0/3444444245 shutdown_connections 2024-04-06T14:53:10.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.703+0000 7f165fc3c700 1 -- 172.21.15.32:0/3444444245 wait complete. 2024-04-06T14:53:10.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.703+0000 7f165fc3c700 1 Processor -- start 2024-04-06T14:53:10.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165fc3c700 1 -- start start 2024-04-06T14:53:10.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165fc3c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658101d70 0x7f1658115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:10.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165fc3c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1658102710 0x7f1658116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165fc3c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1658116790 con 0x7f1658101d70 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165fc3c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165810fb80 con 0x7f1658102710 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165d9d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658101d70 0x7f1658115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165d9d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658101d70 0x7f1658115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54842/0 (socket says 172.21.15.32:54842) 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165d9d8700 1 -- 172.21.15.32:0/508557877 learned_addr learned my addr 172.21.15.32:0/508557877 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165d1d7700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1658102710 0x7f1658116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165d9d8700 1 -- 172.21.15.32:0/508557877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1658102710 msgr2=0x7f1658116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165d9d8700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1658102710 0x7f1658116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165d9d8700 1 -- 172.21.15.32:0/508557877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648005040 con 0x7f1658101d70 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f165d9d8700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658101d70 0x7f1658115c00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f165400eb40 tx=0x7f165400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.704+0000 7f164e7fc700 1 -- 172.21.15.32:0/508557877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1654007cc0 con 0x7f1658101d70 2024-04-06T14:53:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.705+0000 7f164e7fc700 1 -- 172.21.15.32:0/508557877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f165400f040 con 0x7f1658101d70 2024-04-06T14:53:10.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.705+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165810fe00 con 0x7f1658101d70 2024-04-06T14:53:10.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.705+0000 7f164e7fc700 1 -- 172.21.15.32:0/508557877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f165400c990 con 0x7f1658101d70 2024-04-06T14:53:10.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.705+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1658110390 con 0x7f1658101d70 2024-04-06T14:53:10.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.706+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16580fd4c0 con 0x7f1658101d70 2024-04-06T14:53:10.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.706+0000 7f164e7fc700 1 -- 172.21.15.32:0/508557877 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f165400cc00 con 0x7f1658101d70 2024-04-06T14:53:10.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.707+0000 7f164e7fc700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f164406cc20 0x7f164406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:10.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.707+0000 7f164e7fc700 1 -- 172.21.15.32:0/508557877 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f165408bc60 con 0x7f1658101d70 2024-04-06T14:53:10.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.707+0000 7f165d1d7700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f164406cc20 0x7f164406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:10.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.707+0000 7f165d1d7700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f164406cc20 0x7f164406f0d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1658110e60 tx=0x7f1648009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.711+0000 7f164e7fc700 1 -- 172.21.15.32:0/508557877 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1654059760 con 0x7f1658101d70 2024-04-06T14:53:10.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:10 smithi086 ceph-mon[110975]: from='client.35794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:10.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:10 smithi086 ceph-mon[110975]: from='client.35798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:10.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:10 smithi086 ceph-mon[110975]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:10.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:10 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/608710318' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:53:10.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:10 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4189887198' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:53:10.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:10 smithi032 ceph-mon[147414]: from='client.35794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:10.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:10 smithi032 ceph-mon[147414]: from='client.35798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:10.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:10 smithi032 ceph-mon[147414]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:10.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:10 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/608710318' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:53:10.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:10 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4189887198' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:53:10.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.998+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f165804bed0 con 0x7f1658101d70 2024-04-06T14:53:10.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:10.998+0000 7f164e7fc700 1 -- 172.21.15.32:0/508557877 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f16540592f0 con 0x7f1658101d70 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:53:10.986 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:53:10.987 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:53:10.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f164406cc20 msgr2=0x7f164406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f164406cc20 0x7f164406f0d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1658110e60 tx=0x7f1648009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:10.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658101d70 msgr2=0x7f1658115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:10.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658101d70 0x7f1658115c00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f165400eb40 tx=0x7f165400b040 comp rx=0 tx=0).stop 2024-04-06T14:53:10.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 shutdown_connections 2024-04-06T14:53:10.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f164406cc20 0x7f164406f0d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1658101d70 0x7f1658115c00 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 --2- 172.21.15.32:0/508557877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1658102710 0x7f1658116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:10.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 >> 172.21.15.32:0/508557877 conn(0x7f1658072bc0 msgr2=0x7f16580658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:10.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 shutdown_connections 2024-04-06T14:53:10.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:11.000+0000 7f165fc3c700 1 -- 172.21.15.32:0/508557877 wait complete. 2024-04-06T14:53:11.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:11 smithi086 ceph-mon[110975]: pgmap v992: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:11.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:11 smithi086 ceph-mon[110975]: from='client.35814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:11.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:11 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/508557877' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:53:11.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:11 smithi032 ceph-mon[147414]: pgmap v992: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:11.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:11 smithi032 ceph-mon[147414]: from='client.35814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:11.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:11 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/508557877' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:53:13.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:13 smithi032 ceph-mon[147414]: pgmap v993: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:13.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:13 smithi086 ceph-mon[110975]: pgmap v993: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:15.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:15 smithi086 ceph-mon[110975]: pgmap v994: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:15 smithi032 ceph-mon[147414]: pgmap v994: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:17.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:17 smithi086 ceph-mon[110975]: pgmap v995: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:17.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:17 smithi032 ceph-mon[147414]: pgmap v995: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:19.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:19 smithi086 ceph-mon[110975]: pgmap v996: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:19.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:19 smithi032 ceph-mon[147414]: pgmap v996: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:21.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:21 smithi086 ceph-mon[110975]: pgmap v997: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:21.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:21 smithi032 ceph-mon[147414]: pgmap v997: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:23.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:23 smithi032 ceph-mon[147414]: pgmap v998: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:23.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:53:23.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:23 smithi086 ceph-mon[110975]: pgmap v998: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:23.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:53:25.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:25 smithi086 ceph-mon[110975]: pgmap v999: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:25.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:25 smithi032 ceph-mon[147414]: pgmap v999: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:27.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:27 smithi086 ceph-mon[110975]: pgmap v1000: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:27.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:27 smithi032 ceph-mon[147414]: pgmap v1000: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:29.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:29 smithi086 ceph-mon[110975]: pgmap v1001: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:29.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:29 smithi032 ceph-mon[147414]: pgmap v1001: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:31.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:31 smithi086 ceph-mon[110975]: pgmap v1002: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:31.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:31 smithi032 ceph-mon[147414]: pgmap v1002: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:33.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:33 smithi032 ceph-mon[147414]: pgmap v1003: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:33.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:33 smithi086 ceph-mon[110975]: pgmap v1003: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:35.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:35 smithi086 ceph-mon[110975]: pgmap v1004: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:35.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:35 smithi032 ceph-mon[147414]: pgmap v1004: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:37 smithi086 ceph-mon[110975]: pgmap v1005: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:37.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:37 smithi032 ceph-mon[147414]: pgmap v1005: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:53:38.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:53:39.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:39 smithi086 ceph-mon[110975]: pgmap v1006: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:39.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:39 smithi032 ceph-mon[147414]: pgmap v1006: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:41.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.117+0000 7f7ecedb3700 1 -- 172.21.15.32:0/914308085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 msgr2=0x7f7ec80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.117+0000 7f7ecedb3700 1 --2- 172.21.15.32:0/914308085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 0x7f7ec80fe260 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc004940 tx=0x7f7ebc009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:41.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.117+0000 7f7ecedb3700 1 -- 172.21.15.32:0/914308085 shutdown_connections 2024-04-06T14:53:41.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.117+0000 7f7ecedb3700 1 --2- 172.21.15.32:0/914308085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ec80fe830 0x7f7ec8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.117+0000 7f7ecedb3700 1 --2- 172.21.15.32:0/914308085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 0x7f7ec80fe260 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.117+0000 7f7ecedb3700 1 -- 172.21.15.32:0/914308085 >> 172.21.15.32:0/914308085 conn(0x7f7ec8072bc0 msgr2=0x7f7ec8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:41.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.117+0000 7f7ecedb3700 1 -- 172.21.15.32:0/914308085 shutdown_connections 2024-04-06T14:53:41.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.117+0000 7f7ecedb3700 1 -- 172.21.15.32:0/914308085 wait complete. 2024-04-06T14:53:41.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.118+0000 7f7ecedb3700 1 Processor -- start 2024-04-06T14:53:41.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.118+0000 7f7ecedb3700 1 -- start start 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ecedb3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 0x7f7ec8115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ecedb3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ec80fe830 0x7f7ec81161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ecedb3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8116810 con 0x7f7ec80fde90 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ecedb3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec810fbe0 con 0x7f7ec80fe830 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7eccb4f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 0x7f7ec8115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7eccb4f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 0x7f7ec8115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47768/0 (socket says 172.21.15.32:47768) 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7eccb4f700 1 -- 172.21.15.32:0/3523458417 learned_addr learned my addr 172.21.15.32:0/3523458417 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ec7fff700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ec80fe830 0x7f7ec81161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7eccb4f700 1 -- 172.21.15.32:0/3523458417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ec80fe830 msgr2=0x7f7ec81161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7eccb4f700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ec80fe830 0x7f7ec81161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7eccb4f700 1 -- 172.21.15.32:0/3523458417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc005040 con 0x7f7ec80fde90 2024-04-06T14:53:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7eccb4f700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 0x7f7ec8115c80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000c00 tx=0x7f7ebc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:41.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ec57fa700 1 -- 172.21.15.32:0/3523458417 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ebc021070 con 0x7f7ec80fde90 2024-04-06T14:53:41.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ec57fa700 1 -- 172.21.15.32:0/3523458417 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ebc013e00 con 0x7f7ec80fde90 2024-04-06T14:53:41.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec810fe60 con 0x7f7ec80fde90 2024-04-06T14:53:41.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ec57fa700 1 -- 172.21.15.32:0/3523458417 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ebc00e070 con 0x7f7ec80fde90 2024-04-06T14:53:41.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.119+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec81103a0 con 0x7f7ec80fde90 2024-04-06T14:53:41.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.120+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8106a20 con 0x7f7ec80fde90 2024-04-06T14:53:41.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.121+0000 7f7ec57fa700 1 -- 172.21.15.32:0/3523458417 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7ebc01c590 con 0x7f7ec80fde90 2024-04-06T14:53:41.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.121+0000 7f7ec57fa700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7eb806cc20 0x7f7eb806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:41.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.122+0000 7f7ec57fa700 1 -- 172.21.15.32:0/3523458417 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7ebc08d0d0 con 0x7f7ec80fde90 2024-04-06T14:53:41.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.122+0000 7f7ec7fff700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7eb806cc20 0x7f7eb806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:41.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.122+0000 7f7ec7fff700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7eb806cc20 0x7f7eb806f0d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8111520 tx=0x7f7eb400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:41.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.126+0000 7f7ec57fa700 1 -- 172.21.15.32:0/3523458417 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7ebc05aba0 con 0x7f7ec80fde90 2024-04-06T14:53:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.349+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ec8109e60 con 0x7f7eb806cc20 2024-04-06T14:53:41.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.350+0000 7f7ec57fa700 1 -- 172.21.15.32:0/3523458417 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7ec8109e60 con 0x7f7eb806cc20 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7eb806cc20 msgr2=0x7f7eb806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7eb806cc20 0x7f7eb806f0d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8111520 tx=0x7f7eb400b040 comp rx=0 tx=0).stop 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 msgr2=0x7f7ec8115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 0x7f7ec8115c80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000c00 tx=0x7f7ebc009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 shutdown_connections 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7eb806cc20 0x7f7eb806f0d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec80fde90 0x7f7ec8115c80 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 --2- 172.21.15.32:0/3523458417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ec80fe830 0x7f7ec81161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 >> 172.21.15.32:0/3523458417 conn(0x7f7ec8072bc0 msgr2=0x7f7ec80fb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 shutdown_connections 2024-04-06T14:53:41.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.352+0000 7f7ecedb3700 1 -- 172.21.15.32:0/3523458417 wait complete. 2024-04-06T14:53:41.350 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:53:41.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.456+0000 7f8c8a384700 1 -- 172.21.15.32:0/1952716885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c84102e70 msgr2=0x7f8c84103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.456+0000 7f8c8a384700 1 --2- 172.21.15.32:0/1952716885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c84102e70 0x7f8c84103240 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8c74004940 tx=0x7f8c74009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.456+0000 7f8c8a384700 1 -- 172.21.15.32:0/1952716885 shutdown_connections 2024-04-06T14:53:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.456+0000 7f8c8a384700 1 --2- 172.21.15.32:0/1952716885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c840fb4a0 0x7f8c840fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.456+0000 7f8c8a384700 1 --2- 172.21.15.32:0/1952716885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c84102e70 0x7f8c84103240 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.456+0000 7f8c8a384700 1 -- 172.21.15.32:0/1952716885 >> 172.21.15.32:0/1952716885 conn(0x7f8c840714c0 msgr2=0x7f8c840718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.456+0000 7f8c8a384700 1 -- 172.21.15.32:0/1952716885 shutdown_connections 2024-04-06T14:53:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.456+0000 7f8c8a384700 1 -- 172.21.15.32:0/1952716885 wait complete. 2024-04-06T14:53:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.457+0000 7f8c8a384700 1 Processor -- start 2024-04-06T14:53:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.457+0000 7f8c8a384700 1 -- start start 2024-04-06T14:53:41.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.457+0000 7f8c8a384700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c840fb4a0 0x7f8c840fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:41.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.457+0000 7f8c8a384700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c840fcbc0 0x7f8c840fd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:41.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.457+0000 7f8c8a384700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84119f70 con 0x7f8c840fcbc0 2024-04-06T14:53:41.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.457+0000 7f8c8a384700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8411a0b0 con 0x7f8c840fb4a0 2024-04-06T14:53:41.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.457+0000 7f8c837fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c840fcbc0 0x7f8c840fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c837fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c840fcbc0 0x7f8c840fd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47786/0 (socket says 172.21.15.32:47786) 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c837fe700 1 -- 172.21.15.32:0/1529065943 learned_addr learned my addr 172.21.15.32:0/1529065943 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c83fff700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c840fb4a0 0x7f8c840fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c837fe700 1 -- 172.21.15.32:0/1529065943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c840fb4a0 msgr2=0x7f8c840fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c837fe700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c840fb4a0 0x7f8c840fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c837fe700 1 -- 172.21.15.32:0/1529065943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74005040 con 0x7f8c840fcbc0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c837fe700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c840fcbc0 0x7f8c840fd030 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8c7800b790 tx=0x7f8c78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c80ff9700 1 -- 172.21.15.32:0/1529065943 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c7800f6f0 con 0x7f8c840fcbc0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8411a340 con 0x7f8c840fcbc0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c80ff9700 1 -- 172.21.15.32:0/1529065943 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c7800fd00 con 0x7f8c840fcbc0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c80ff9700 1 -- 172.21.15.32:0/1529065943 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c78011e00 con 0x7f8c840fcbc0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.458+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8411a880 con 0x7f8c840fcbc0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.459+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c84065380 con 0x7f8c840fcbc0 2024-04-06T14:53:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.460+0000 7f8c80ff9700 1 -- 172.21.15.32:0/1529065943 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8c7800f850 con 0x7f8c840fcbc0 2024-04-06T14:53:41.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.460+0000 7f8c80ff9700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c6c06cbc0 0x7f8c6c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:41.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.461+0000 7f8c80ff9700 1 -- 172.21.15.32:0/1529065943 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8c7808c930 con 0x7f8c840fcbc0 2024-04-06T14:53:41.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.461+0000 7f8c83fff700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c6c06cbc0 0x7f8c6c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:41.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.461+0000 7f8c83fff700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c6c06cbc0 0x7f8c6c06f070 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400b3b0 tx=0x7f8c74009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:41.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.464+0000 7f8c80ff9700 1 -- 172.21.15.32:0/1529065943 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8c78056a30 con 0x7f8c840fcbc0 2024-04-06T14:53:41.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.689+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c8404bed0 con 0x7f8c6c06cbc0 2024-04-06T14:53:41.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.691+0000 7f8c80ff9700 1 -- 172.21.15.32:0/1529065943 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8c8404bed0 con 0x7f8c6c06cbc0 2024-04-06T14:53:41.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c6c06cbc0 msgr2=0x7f8c6c06f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c6c06cbc0 0x7f8c6c06f070 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400b3b0 tx=0x7f8c74009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:41.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c840fcbc0 msgr2=0x7f8c840fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c840fcbc0 0x7f8c840fd030 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8c7800b790 tx=0x7f8c78009250 comp rx=0 tx=0).stop 2024-04-06T14:53:41.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 shutdown_connections 2024-04-06T14:53:41.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c6c06cbc0 0x7f8c6c06f070 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c840fb4a0 0x7f8c840fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 --2- 172.21.15.32:0/1529065943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c840fcbc0 0x7f8c840fd030 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 >> 172.21.15.32:0/1529065943 conn(0x7f8c840714c0 msgr2=0x7f8c840fa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:41.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 shutdown_connections 2024-04-06T14:53:41.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.693+0000 7f8c8a384700 1 -- 172.21.15.32:0/1529065943 wait complete. 2024-04-06T14:53:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 -- 172.21.15.32:0/18071099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928102710 msgr2=0x7f59280fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 --2- 172.21.15.32:0/18071099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928102710 0x7f59280fbd80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f5918004940 tx=0x7f5918009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 -- 172.21.15.32:0/18071099 shutdown_connections 2024-04-06T14:53:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 --2- 172.21.15.32:0/18071099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928102710 0x7f59280fbd80 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 --2- 172.21.15.32:0/18071099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5928101d70 0x7f5928102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 -- 172.21.15.32:0/18071099 >> 172.21.15.32:0/18071099 conn(0x7f5928072bc0 msgr2=0x7f5928072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 -- 172.21.15.32:0/18071099 shutdown_connections 2024-04-06T14:53:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 -- 172.21.15.32:0/18071099 wait complete. 2024-04-06T14:53:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 Processor -- start 2024-04-06T14:53:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.796+0000 7f592d3d3700 1 -- start start 2024-04-06T14:53:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f592d3d3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928101d70 0x7f59280fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f592d3d3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5928102710 0x7f59280fdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f592d3d3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59280fe4c0 con 0x7f5928101d70 2024-04-06T14:53:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f592d3d3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59280fe600 con 0x7f5928102710 2024-04-06T14:53:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f5926ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928101d70 0x7f59280fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f5926ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928101d70 0x7f59280fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47792/0 (socket says 172.21.15.32:47792) 2024-04-06T14:53:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f5926ffd700 1 -- 172.21.15.32:0/152691896 learned_addr learned my addr 172.21.15.32:0/152691896 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f59267fc700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5928102710 0x7f59280fdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f5926ffd700 1 -- 172.21.15.32:0/152691896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5928102710 msgr2=0x7f59280fdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f5926ffd700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5928102710 0x7f59280fdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f5926ffd700 1 -- 172.21.15.32:0/152691896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918005040 con 0x7f5928101d70 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f59267fc700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5928102710 0x7f59280fdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f5926ffd700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928101d70 0x7f59280fd9f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f591c00c810 tx=0x7f591c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f590f7fe700 1 -- 172.21.15.32:0/152691896 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f591c012810 con 0x7f5928101d70 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f590f7fe700 1 -- 172.21.15.32:0/152691896 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f591c012e20 con 0x7f5928101d70 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5928119e00 con 0x7f5928101d70 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.797+0000 7f590f7fe700 1 -- 172.21.15.32:0/152691896 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f591c00a370 con 0x7f5928101d70 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.798+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592811a280 con 0x7f5928101d70 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.798+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59280fd4c0 con 0x7f5928101d70 2024-04-06T14:53:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.799+0000 7f590f7fe700 1 -- 172.21.15.32:0/152691896 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f591c011780 con 0x7f5928101d70 2024-04-06T14:53:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.799+0000 7f590f7fe700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591006cc20 0x7f591006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.799+0000 7f590f7fe700 1 -- 172.21.15.32:0/152691896 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f591c08c4d0 con 0x7f5928101d70 2024-04-06T14:53:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.799+0000 7f59267fc700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591006cc20 0x7f591006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.800+0000 7f59267fc700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591006cc20 0x7f591006f0d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5918009fd0 tx=0x7f5918009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:41.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:41.804+0000 7f590f7fe700 1 -- 172.21.15.32:0/152691896 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f591c056490 con 0x7f5928101d70 2024-04-06T14:53:41.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:41 smithi086 ceph-mon[110975]: pgmap v1007: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:41.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:41 smithi032 ceph-mon[147414]: pgmap v1007: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.016+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f59280ff0e0 con 0x7f591006cc20 2024-04-06T14:53:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.026+0000 7f590f7fe700 1 -- 172.21.15.32:0/152691896 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f59280ff0e0 con 0x7f591006cc20 2024-04-06T14:53:42.013 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (34m) 12m ago 45m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (31m) 12m ago 45m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (31m) 12m ago 44m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (33m) 12m ago 45m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (27m) 12m ago 39m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (27m) 12m ago 39m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (27m) 12m ago 39m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (12m) 12m ago 39m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (35m) 12m ago 46m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (34m) 12m ago 44m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (32m) 12m ago 46m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (32m) 12m ago 44m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:53:42.014 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (34m) 12m ago 45m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:53:42.015 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (34m) 12m ago 44m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:53:42.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (30m) 12m ago 43m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:53:42.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (30m) 12m ago 43m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:53:42.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (29m) 12m ago 42m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:53:42.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (29m) 12m ago 42m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:53:42.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (28m) 12m ago 41m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:53:42.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (28m) 12m ago 41m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:53:42.015 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (34m) 12m ago 44m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:53:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591006cc20 msgr2=0x7f591006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591006cc20 0x7f591006f0d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5918009fd0 tx=0x7f5918009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928101d70 msgr2=0x7f59280fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928101d70 0x7f59280fd9f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f591c00c810 tx=0x7f591c004470 comp rx=0 tx=0).stop 2024-04-06T14:53:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 shutdown_connections 2024-04-06T14:53:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f591006cc20 0x7f591006f0d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5928101d70 0x7f59280fd9f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 --2- 172.21.15.32:0/152691896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5928102710 0x7f59280fdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 >> 172.21.15.32:0/152691896 conn(0x7f5928072bc0 msgr2=0x7f5928065a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 shutdown_connections 2024-04-06T14:53:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.028+0000 7f592d3d3700 1 -- 172.21.15.32:0/152691896 wait complete. 2024-04-06T14:53:42.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.130+0000 7f6627d5b700 1 -- 172.21.15.32:0/268980320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 msgr2=0x7f66201053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.130+0000 7f6627d5b700 1 --2- 172.21.15.32:0/268980320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 0x7f66201053d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f6614004940 tx=0x7f6614009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:42.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.130+0000 7f6627d5b700 1 -- 172.21.15.32:0/268980320 shutdown_connections 2024-04-06T14:53:42.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.130+0000 7f6627d5b700 1 --2- 172.21.15.32:0/268980320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66201059a0 0x7f66200ff010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.130+0000 7f6627d5b700 1 --2- 172.21.15.32:0/268980320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 0x7f66201053d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.130+0000 7f6627d5b700 1 -- 172.21.15.32:0/268980320 >> 172.21.15.32:0/268980320 conn(0x7f66200fabd0 msgr2=0x7f66200fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.130+0000 7f6627d5b700 1 -- 172.21.15.32:0/268980320 shutdown_connections 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.130+0000 7f6627d5b700 1 -- 172.21.15.32:0/268980320 wait complete. 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6627d5b700 1 Processor -- start 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6627d5b700 1 -- start start 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6627d5b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 0x7f6620115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6627d5b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66201059a0 0x7f6620116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6627d5b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6620116880 con 0x7f6620105000 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6627d5b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662010fc50 con 0x7f66201059a0 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6625af7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 0x7f6620115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6625af7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 0x7f6620115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47816/0 (socket says 172.21.15.32:47816) 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6625af7700 1 -- 172.21.15.32:0/3440521830 learned_addr learned my addr 172.21.15.32:0/3440521830 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:42.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6625af7700 1 -- 172.21.15.32:0/3440521830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66201059a0 msgr2=0x7f6620116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6625af7700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66201059a0 0x7f6620116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6625af7700 1 -- 172.21.15.32:0/3440521830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6614005040 con 0x7f6620105000 2024-04-06T14:53:42.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f6625af7700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 0x7f6620115cf0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f6614009180 tx=0x7f6614009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:42.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.131+0000 7f66127fc700 1 -- 172.21.15.32:0/3440521830 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f661400e040 con 0x7f6620105000 2024-04-06T14:53:42.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.132+0000 7f66127fc700 1 -- 172.21.15.32:0/3440521830 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6614004290 con 0x7f6620105000 2024-04-06T14:53:42.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.132+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662010fed0 con 0x7f6620105000 2024-04-06T14:53:42.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.132+0000 7f66127fc700 1 -- 172.21.15.32:0/3440521830 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f661401c3f0 con 0x7f6620105000 2024-04-06T14:53:42.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.132+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6620110490 con 0x7f6620105000 2024-04-06T14:53:42.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.132+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6620102310 con 0x7f6620105000 2024-04-06T14:53:42.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.133+0000 7f66127fc700 1 -- 172.21.15.32:0/3440521830 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6614013790 con 0x7f6620105000 2024-04-06T14:53:42.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.134+0000 7f66127fc700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f660c06cc20 0x7f660c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:42.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.134+0000 7f66252f6700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f660c06cc20 0x7f660c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:42.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.134+0000 7f66127fc700 1 -- 172.21.15.32:0/3440521830 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f661408d300 con 0x7f6620105000 2024-04-06T14:53:42.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.134+0000 7f66252f6700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f660c06cc20 0x7f660c06f0d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6620111590 tx=0x7f661c007d40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:42.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.138+0000 7f66127fc700 1 -- 172.21.15.32:0/3440521830 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f661405add0 con 0x7f6620105000 2024-04-06T14:53:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.424+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6620111340 con 0x7f6620105000 2024-04-06T14:53:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.425+0000 7f66127fc700 1 -- 172.21.15.32:0/3440521830 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6614013420 con 0x7f6620105000 2024-04-06T14:53:42.412 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:53:42.412 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:53:42.412 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:53:42.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:53:42.412 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:53:42.412 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:53:42.413 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:53:42.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.426+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f660c06cc20 msgr2=0x7f660c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.426+0000 7f6627d5b700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f660c06cc20 0x7f660c06f0d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6620111590 tx=0x7f661c007d40 comp rx=0 tx=0).stop 2024-04-06T14:53:42.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.427+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 msgr2=0x7f6620115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.427+0000 7f6627d5b700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 0x7f6620115cf0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f6614009180 tx=0x7f6614009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:42.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.427+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 shutdown_connections 2024-04-06T14:53:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.427+0000 7f6627d5b700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f660c06cc20 0x7f660c06f0d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.427+0000 7f6627d5b700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620105000 0x7f6620115cf0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.427+0000 7f6627d5b700 1 --2- 172.21.15.32:0/3440521830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66201059a0 0x7f6620116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.427+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 >> 172.21.15.32:0/3440521830 conn(0x7f66200fabd0 msgr2=0x7f66200fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.427+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 shutdown_connections 2024-04-06T14:53:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.427+0000 7f6627d5b700 1 -- 172.21.15.32:0/3440521830 wait complete. 2024-04-06T14:53:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 -- 172.21.15.32:0/3441289343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63640fb4a0 msgr2=0x7f63640fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 --2- 172.21.15.32:0/3441289343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63640fb4a0 0x7f63640fb910 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6358004940 tx=0x7f6358009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 -- 172.21.15.32:0/3441289343 shutdown_connections 2024-04-06T14:53:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 --2- 172.21.15.32:0/3441289343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63640fb4a0 0x7f63640fb910 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 --2- 172.21.15.32:0/3441289343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6364102e70 0x7f6364103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 -- 172.21.15.32:0/3441289343 >> 172.21.15.32:0/3441289343 conn(0x7f63640714c0 msgr2=0x7f63640718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 -- 172.21.15.32:0/3441289343 shutdown_connections 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 -- 172.21.15.32:0/3441289343 wait complete. 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 Processor -- start 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 -- start start 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63640fb4a0 0x7f6364113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6364102e70 0x7f6364113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6364114600 con 0x7f6364102e70 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.530+0000 7f6368ed7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636410d9f0 con 0x7f63640fb4a0 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6361d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6364102e70 0x7f6364113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6361d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6364102e70 0x7f6364113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47826/0 (socket says 172.21.15.32:47826) 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6361d9b700 1 -- 172.21.15.32:0/3882107401 learned_addr learned my addr 172.21.15.32:0/3882107401 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f636259c700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63640fb4a0 0x7f6364113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6361d9b700 1 -- 172.21.15.32:0/3882107401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63640fb4a0 msgr2=0x7f6364113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6361d9b700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63640fb4a0 0x7f6364113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6361d9b700 1 -- 172.21.15.32:0/3882107401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6358005040 con 0x7f6364102e70 2024-04-06T14:53:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f636259c700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63640fb4a0 0x7f6364113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:53:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6361d9b700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6364102e70 0x7f6364113fb0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f635800b3b0 tx=0x7f6358009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6352ffd700 1 -- 172.21.15.32:0/3882107401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635800e040 con 0x7f6364102e70 2024-04-06T14:53:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6352ffd700 1 -- 172.21.15.32:0/3882107401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6358013420 con 0x7f6364102e70 2024-04-06T14:53:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636410dc70 con 0x7f6364102e70 2024-04-06T14:53:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6352ffd700 1 -- 172.21.15.32:0/3882107401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635801c630 con 0x7f6364102e70 2024-04-06T14:53:42.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.531+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636410e230 con 0x7f6364102e70 2024-04-06T14:53:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.532+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6364100110 con 0x7f6364102e70 2024-04-06T14:53:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.533+0000 7f6352ffd700 1 -- 172.21.15.32:0/3882107401 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6358013970 con 0x7f6364102e70 2024-04-06T14:53:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.533+0000 7f6352ffd700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f634c06cc20 0x7f634c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.533+0000 7f6352ffd700 1 -- 172.21.15.32:0/3882107401 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f635808d180 con 0x7f6364102e70 2024-04-06T14:53:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.533+0000 7f636259c700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f634c06cc20 0x7f634c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:42.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.534+0000 7f636259c700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f634c06cc20 0x7f634c06f0d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6354007fd0 tx=0x7f6354009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:42.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.538+0000 7f6352ffd700 1 -- 172.21.15.32:0/3882107401 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f635805ad10 con 0x7f6364102e70 2024-04-06T14:53:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.784+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f636410f0c0 con 0x7f6364102e70 2024-04-06T14:53:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.784+0000 7f6352ffd700 1 -- 172.21.15.32:0/3882107401 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f635805a8a0 con 0x7f6364102e70 2024-04-06T14:53:42.773 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:53:42.773 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:53:42.773 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:53:42.773 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:53:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:53:42.773 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:53:42.774 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:53:42.774 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:53:42.774 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:53:42.774 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:53:42.774 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:53:42.774 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:53:42.775 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:53:42.775 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:53:42.775 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:53:42.775 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:53:42.775 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:53:42.775 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:53:42.776 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:53:42.776 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:53:42.776 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:53:42.776 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:53:42.776 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:53:42.776 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:53:42.776 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:53:42.777 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:53:42.777 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:53:42.777 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:53:42.777 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:53:42.777 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:53:42.777 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:53:42.778 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:53:42.778 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:53:42.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:53:42.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:53:42.778 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:53:42.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:53:42.779 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:53:42.779 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:53:42.779 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:53:42.779 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:53:42.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f634c06cc20 msgr2=0x7f634c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f634c06cc20 0x7f634c06f0d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6354007fd0 tx=0x7f6354009040 comp rx=0 tx=0).stop 2024-04-06T14:53:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6364102e70 msgr2=0x7f6364113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6364102e70 0x7f6364113fb0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f635800b3b0 tx=0x7f6358009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 shutdown_connections 2024-04-06T14:53:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f634c06cc20 0x7f634c06f0d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63640fb4a0 0x7f6364113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 --2- 172.21.15.32:0/3882107401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6364102e70 0x7f6364113fb0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 >> 172.21.15.32:0/3882107401 conn(0x7f63640714c0 msgr2=0x7f63640fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 shutdown_connections 2024-04-06T14:53:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.786+0000 7f6368ed7700 1 -- 172.21.15.32:0/3882107401 wait complete. 2024-04-06T14:53:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:53:42.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:42 smithi086 ceph-mon[110975]: from='client.35822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:42.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:42 smithi086 ceph-mon[110975]: from='client.35826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:42.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:42 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3440521830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:53:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.890+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/401622625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 msgr2=0x7f3fc0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.890+0000 7f3fc7ae2700 1 --2- 172.21.15.32:0/401622625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 0x7f3fc0102140 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0004940 tx=0x7f3fb0009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.890+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/401622625 shutdown_connections 2024-04-06T14:53:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.890+0000 7f3fc7ae2700 1 --2- 172.21.15.32:0/401622625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc0102710 0x7f3fc00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.890+0000 7f3fc7ae2700 1 --2- 172.21.15.32:0/401622625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 0x7f3fc0102140 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.890+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/401622625 >> 172.21.15.32:0/401622625 conn(0x7f3fc0072bc0 msgr2=0x7f3fc0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.890+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/401622625 shutdown_connections 2024-04-06T14:53:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.890+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/401622625 wait complete. 2024-04-06T14:53:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc7ae2700 1 Processor -- start 2024-04-06T14:53:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc7ae2700 1 -- start start 2024-04-06T14:53:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc7ae2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 0x7f3fc01129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc7ae2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc0102710 0x7f3fc0112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc7ae2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0113550 con 0x7f3fc0101d70 2024-04-06T14:53:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc7ae2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc010c940 con 0x7f3fc0102710 2024-04-06T14:53:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc587e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 0x7f3fc01129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc587e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 0x7f3fc01129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47840/0 (socket says 172.21.15.32:47840) 2024-04-06T14:53:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc587e700 1 -- 172.21.15.32:0/2594878981 learned_addr learned my addr 172.21.15.32:0/2594878981 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc507d700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc0102710 0x7f3fc0112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc587e700 1 -- 172.21.15.32:0/2594878981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc0102710 msgr2=0x7f3fc0112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.891+0000 7f3fc587e700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc0102710 0x7f3fc0112f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.892+0000 7f3fc587e700 1 -- 172.21.15.32:0/2594878981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb0005040 con 0x7f3fc0101d70 2024-04-06T14:53:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.892+0000 7f3fc587e700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 0x7f3fc01129c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000b3b0 tx=0x7f3fb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.892+0000 7f3fb67fc700 1 -- 172.21.15.32:0/2594878981 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb000e040 con 0x7f3fc0101d70 2024-04-06T14:53:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.892+0000 7f3fb67fc700 1 -- 172.21.15.32:0/2594878981 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fb0004290 con 0x7f3fc0101d70 2024-04-06T14:53:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.892+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc010cbc0 con 0x7f3fc0101d70 2024-04-06T14:53:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.892+0000 7f3fb67fc700 1 -- 172.21.15.32:0/2594878981 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb0013660 con 0x7f3fc0101d70 2024-04-06T14:53:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.892+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc010d120 con 0x7f3fc0101d70 2024-04-06T14:53:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.893+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc00fd4c0 con 0x7f3fc0101d70 2024-04-06T14:53:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.893+0000 7f3fb67fc700 1 -- 172.21.15.32:0/2594878981 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3fb000bd60 con 0x7f3fc0101d70 2024-04-06T14:53:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.894+0000 7f3fb67fc700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fac06cc20 0x7f3fac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.894+0000 7f3fb67fc700 1 -- 172.21.15.32:0/2594878981 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3fb008fe20 con 0x7f3fc0101d70 2024-04-06T14:53:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.894+0000 7f3fc507d700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fac06cc20 0x7f3fac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.894+0000 7f3fc507d700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fac06cc20 0x7f3fac06f0d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009fd0 tx=0x7f3fbc009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:42.898+0000 7f3fb67fc700 1 -- 172.21.15.32:0/2594878981 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3fb005d8f0 con 0x7f3fc0101d70 2024-04-06T14:53:42.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:42 smithi032 ceph-mon[147414]: from='client.35822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:42.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:42 smithi032 ceph-mon[147414]: from='client.35826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:42.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:42 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3440521830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:53:43.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.120+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fc0108dc0 con 0x7f3fac06cc20 2024-04-06T14:53:43.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.121+0000 7f3fb67fc700 1 -- 172.21.15.32:0/2594878981 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3fc0108dc0 con 0x7f3fac06cc20 2024-04-06T14:53:43.108 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:53:43.108 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:53:43.108 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:53:43.108 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:53:43.109 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:53:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fac06cc20 msgr2=0x7f3fac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fac06cc20 0x7f3fac06f0d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009fd0 tx=0x7f3fbc009f20 comp rx=0 tx=0).stop 2024-04-06T14:53:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 msgr2=0x7f3fc01129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 0x7f3fc01129c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000b3b0 tx=0x7f3fb0009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 shutdown_connections 2024-04-06T14:53:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fac06cc20 0x7f3fac06f0d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc0101d70 0x7f3fc01129c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 --2- 172.21.15.32:0/2594878981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc0102710 0x7f3fc0112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 >> 172.21.15.32:0/2594878981 conn(0x7f3fc0072bc0 msgr2=0x7f3fc00fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 shutdown_connections 2024-04-06T14:53:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.123+0000 7f3fc7ae2700 1 -- 172.21.15.32:0/2594878981 wait complete. 2024-04-06T14:53:43.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.225+0000 7fab2f574700 1 -- 172.21.15.32:0/3076633719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 msgr2=0x7fab28103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:43.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.225+0000 7fab2f574700 1 --2- 172.21.15.32:0/3076633719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 0x7fab28103240 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fab24004940 tx=0x7fab24009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:43.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.225+0000 7fab2f574700 1 -- 172.21.15.32:0/3076633719 shutdown_connections 2024-04-06T14:53:43.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.225+0000 7fab2f574700 1 --2- 172.21.15.32:0/3076633719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab280fb4a0 0x7fab280fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.225+0000 7fab2f574700 1 --2- 172.21.15.32:0/3076633719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 0x7fab28103240 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.225+0000 7fab2f574700 1 -- 172.21.15.32:0/3076633719 >> 172.21.15.32:0/3076633719 conn(0x7fab280714c0 msgr2=0x7fab280718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2f574700 1 -- 172.21.15.32:0/3076633719 shutdown_connections 2024-04-06T14:53:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2f574700 1 -- 172.21.15.32:0/3076633719 wait complete. 2024-04-06T14:53:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2f574700 1 Processor -- start 2024-04-06T14:53:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2f574700 1 -- start start 2024-04-06T14:53:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2f574700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab280fb4a0 0x7fab28113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2f574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 0x7fab28113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2f574700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab28114620 con 0x7fab28102e70 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2f574700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2810da10 con 0x7fab280fb4a0 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2cb0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 0x7fab28113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2cb0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 0x7fab28113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47862/0 (socket says 172.21.15.32:47862) 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.226+0000 7fab2cb0f700 1 -- 172.21.15.32:0/1445440648 learned_addr learned my addr 172.21.15.32:0/1445440648 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab2d310700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab280fb4a0 0x7fab28113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab2cb0f700 1 -- 172.21.15.32:0/1445440648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab280fb4a0 msgr2=0x7fab28113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab2cb0f700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab280fb4a0 0x7fab28113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab2cb0f700 1 -- 172.21.15.32:0/1445440648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab24005040 con 0x7fab28102e70 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab2cb0f700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 0x7fab28113fd0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fab1c002a80 tx=0x7fab1c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab19ffb700 1 -- 172.21.15.32:0/1445440648 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab1c007cc0 con 0x7fab28102e70 2024-04-06T14:53:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab19ffb700 1 -- 172.21.15.32:0/1445440648 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab1c00f040 con 0x7fab28102e70 2024-04-06T14:53:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2810dcf0 con 0x7fab28102e70 2024-04-06T14:53:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab19ffb700 1 -- 172.21.15.32:0/1445440648 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab1c00da60 con 0x7fab28102e70 2024-04-06T14:53:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.227+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2810e230 con 0x7fab28102e70 2024-04-06T14:53:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.228+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab28100170 con 0x7fab28102e70 2024-04-06T14:53:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.228+0000 7fab19ffb700 1 -- 172.21.15.32:0/1445440648 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fab1c00dbc0 con 0x7fab28102e70 2024-04-06T14:53:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.229+0000 7fab19ffb700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab1406cc20 0x7fab1406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:53:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.229+0000 7fab19ffb700 1 -- 172.21.15.32:0/1445440648 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fab1c08bcb0 con 0x7fab28102e70 2024-04-06T14:53:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.229+0000 7fab2d310700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab1406cc20 0x7fab1406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:53:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.229+0000 7fab2d310700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab1406cc20 0x7fab1406f0d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fab24004f90 tx=0x7fab24009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:53:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.234+0000 7fab19ffb700 1 -- 172.21.15.32:0/1445440648 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fab1c059700 con 0x7fab28102e70 2024-04-06T14:53:43.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.519+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fab2810f0e0 con 0x7fab28102e70 2024-04-06T14:53:43.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.519+0000 7fab19ffb700 1 -- 172.21.15.32:0/1445440648 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7fab1c059290 con 0x7fab28102e70 2024-04-06T14:53:43.507 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:53:43.507 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:53:43.507 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:53:43.507 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:53:43.507 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:53:43.507 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:53:43.507 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:53:43.508 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:53:43.509 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:53:43.509 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:53:43.509 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:53:43.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab1406cc20 msgr2=0x7fab1406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:43.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab1406cc20 0x7fab1406f0d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fab24004f90 tx=0x7fab24009e30 comp rx=0 tx=0).stop 2024-04-06T14:53:43.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 msgr2=0x7fab28113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:53:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 0x7fab28113fd0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fab1c002a80 tx=0x7fab1c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 shutdown_connections 2024-04-06T14:53:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fab1406cc20 0x7fab1406f0d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab280fb4a0 0x7fab28113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 --2- 172.21.15.32:0/1445440648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab28102e70 0x7fab28113fd0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:53:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 >> 172.21.15.32:0/1445440648 conn(0x7fab280714c0 msgr2=0x7fab280fa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:53:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 shutdown_connections 2024-04-06T14:53:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:53:43.522+0000 7fab2f574700 1 -- 172.21.15.32:0/1445440648 wait complete. 2024-04-06T14:53:43.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:43 smithi032 ceph-mon[147414]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:43.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:43 smithi032 ceph-mon[147414]: pgmap v1008: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:43.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:43 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3882107401' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:53:43.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:43 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1445440648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:53:43.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:43 smithi086 ceph-mon[110975]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:43.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:43 smithi086 ceph-mon[110975]: pgmap v1008: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:43.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:43 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3882107401' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:53:43.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:43 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1445440648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:53:44.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:44 smithi086 ceph-mon[110975]: from='client.35842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:44.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:44 smithi032 ceph-mon[147414]: from='client.35842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:53:45.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:45 smithi086 ceph-mon[110975]: pgmap v1009: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:45.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:45 smithi032 ceph-mon[147414]: pgmap v1009: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:47.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:47 smithi086 ceph-mon[110975]: pgmap v1010: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:47.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:47 smithi032 ceph-mon[147414]: pgmap v1010: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:49.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:49 smithi086 ceph-mon[110975]: pgmap v1011: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:49.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:49 smithi032 ceph-mon[147414]: pgmap v1011: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:51.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:51 smithi086 ceph-mon[110975]: pgmap v1012: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:51.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:51 smithi032 ceph-mon[147414]: pgmap v1012: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:53.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:53 smithi032 ceph-mon[147414]: pgmap v1013: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:53.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:53:53.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:53 smithi086 ceph-mon[110975]: pgmap v1013: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:53.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:53:55.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:55 smithi086 ceph-mon[110975]: pgmap v1014: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:55.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:55 smithi032 ceph-mon[147414]: pgmap v1014: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:57.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:57 smithi086 ceph-mon[110975]: pgmap v1015: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:57.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:57 smithi032 ceph-mon[147414]: pgmap v1015: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:59.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:53:59 smithi086 ceph-mon[110975]: pgmap v1016: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:53:59.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:53:59 smithi032 ceph-mon[147414]: pgmap v1016: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:01.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:01 smithi086 ceph-mon[110975]: pgmap v1017: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:01.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:01 smithi032 ceph-mon[147414]: pgmap v1017: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:03.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:03 smithi032 ceph-mon[147414]: pgmap v1018: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:03.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:03 smithi086 ceph-mon[110975]: pgmap v1018: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:05.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:05 smithi086 ceph-mon[110975]: pgmap v1019: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:05.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:05 smithi032 ceph-mon[147414]: pgmap v1019: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:07.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:07 smithi086 ceph-mon[110975]: pgmap v1020: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:07.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:07 smithi032 ceph-mon[147414]: pgmap v1020: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:08.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:54:08.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:54:09.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:09 smithi086 ceph-mon[110975]: pgmap v1021: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:09.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:09 smithi032 ceph-mon[147414]: pgmap v1021: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:11.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:11 smithi086 ceph-mon[110975]: pgmap v1022: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:11.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:11 smithi032 ceph-mon[147414]: pgmap v1022: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:13.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.639+0000 7f99f6265700 1 -- 172.21.15.32:0/2971284458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 msgr2=0x7f99f00fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.639+0000 7f99f6265700 1 --2- 172.21.15.32:0/2971284458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 0x7f99f00fe1e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f99d8004940 tx=0x7f99d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.639+0000 7f99f6265700 1 -- 172.21.15.32:0/2971284458 shutdown_connections 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.639+0000 7f99f6265700 1 --2- 172.21.15.32:0/2971284458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f00fe7b0 0x7f99f0106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.639+0000 7f99f6265700 1 --2- 172.21.15.32:0/2971284458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 0x7f99f00fe1e0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.639+0000 7f99f6265700 1 -- 172.21.15.32:0/2971284458 >> 172.21.15.32:0/2971284458 conn(0x7f99f00f7880 msgr2=0x7f99f00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.639+0000 7f99f6265700 1 -- 172.21.15.32:0/2971284458 shutdown_connections 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.640+0000 7f99f6265700 1 -- 172.21.15.32:0/2971284458 wait complete. 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.640+0000 7f99f6265700 1 Processor -- start 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.640+0000 7f99f6265700 1 -- start start 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.640+0000 7f99f6265700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 0x7f99f0113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99f6265700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f00fe7b0 0x7f99f0114240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99f6265700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0114890 con 0x7f99f00fde10 2024-04-06T14:54:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99f6265700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f010dc60 con 0x7f99f00fe7b0 2024-04-06T14:54:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 0x7f99f0113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 0x7f99f0113d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45684/0 (socket says 172.21.15.32:45684) 2024-04-06T14:54:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99effff700 1 -- 172.21.15.32:0/2412280408 learned_addr learned my addr 172.21.15.32:0/2412280408 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:13.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99ef7fe700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f00fe7b0 0x7f99f0114240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:13.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99effff700 1 -- 172.21.15.32:0/2412280408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f00fe7b0 msgr2=0x7f99f0114240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99effff700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f00fe7b0 0x7f99f0114240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99effff700 1 -- 172.21.15.32:0/2412280408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d8005040 con 0x7f99f00fde10 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99ef7fe700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f00fe7b0 0x7f99f0114240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99effff700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 0x7f99f0113d00 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f99d8004890 tx=0x7f99d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99ecff9700 1 -- 172.21.15.32:0/2412280408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d800e040 con 0x7f99f00fde10 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f010dee0 con 0x7f99f00fde10 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99ecff9700 1 -- 172.21.15.32:0/2412280408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99d8009af0 con 0x7f99f00fde10 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99ecff9700 1 -- 172.21.15.32:0/2412280408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d801c3f0 con 0x7f99f00fde10 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.641+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f010e420 con 0x7f99f00fde10 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.642+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f0066770 con 0x7f99f00fde10 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.643+0000 7f99ecff9700 1 -- 172.21.15.32:0/2412280408 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f99d8013420 con 0x7f99f00fde10 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.643+0000 7f99ecff9700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99dc06cc20 0x7f99dc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.643+0000 7f99ecff9700 1 -- 172.21.15.32:0/2412280408 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f99d808d440 con 0x7f99f00fde10 2024-04-06T14:54:13.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.644+0000 7f99ef7fe700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99dc06cc20 0x7f99dc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:13.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.644+0000 7f99ef7fe700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99dc06cc20 0x7f99dc06f0d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f99e0007fd0 tx=0x7f99e0009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:13.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.648+0000 7f99ecff9700 1 -- 172.21.15.32:0/2412280408 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f99d805af10 con 0x7f99f00fde10 2024-04-06T14:54:13.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:13 smithi032 ceph-mon[147414]: pgmap v1023: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:13.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.870+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99f004bf40 con 0x7f99dc06cc20 2024-04-06T14:54:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.871+0000 7f99ecff9700 1 -- 172.21.15.32:0/2412280408 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f99f004bf40 con 0x7f99dc06cc20 2024-04-06T14:54:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99dc06cc20 msgr2=0x7f99dc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99dc06cc20 0x7f99dc06f0d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f99e0007fd0 tx=0x7f99e0009f60 comp rx=0 tx=0).stop 2024-04-06T14:54:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 msgr2=0x7f99f0113d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 0x7f99f0113d00 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f99d8004890 tx=0x7f99d8009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 shutdown_connections 2024-04-06T14:54:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f99dc06cc20 0x7f99dc06f0d0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde10 0x7f99f0113d00 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 --2- 172.21.15.32:0/2412280408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f00fe7b0 0x7f99f0114240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 >> 172.21.15.32:0/2412280408 conn(0x7f99f00f7880 msgr2=0x7f99f00f8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 shutdown_connections 2024-04-06T14:54:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.873+0000 7f99f6265700 1 -- 172.21.15.32:0/2412280408 wait complete. 2024-04-06T14:54:13.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:13 smithi086 ceph-mon[110975]: pgmap v1023: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:13.870 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:54:13.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.977+0000 7f34d0d93700 1 -- 172.21.15.32:0/1731166574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fde90 msgr2=0x7f34cc0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:13.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.977+0000 7f34d0d93700 1 --2- 172.21.15.32:0/1731166574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fde90 0x7f34cc0fe260 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f34bc004940 tx=0x7f34bc009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:13.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.978+0000 7f34d0d93700 1 -- 172.21.15.32:0/1731166574 shutdown_connections 2024-04-06T14:54:13.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.978+0000 7f34d0d93700 1 --2- 172.21.15.32:0/1731166574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34cc0fe830 0x7f34cc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:13.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.978+0000 7f34d0d93700 1 --2- 172.21.15.32:0/1731166574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fde90 0x7f34cc0fe260 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:13.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.978+0000 7f34d0d93700 1 -- 172.21.15.32:0/1731166574 >> 172.21.15.32:0/1731166574 conn(0x7f34cc072bc0 msgr2=0x7f34cc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.978+0000 7f34d0d93700 1 -- 172.21.15.32:0/1731166574 shutdown_connections 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.978+0000 7f34d0d93700 1 -- 172.21.15.32:0/1731166574 wait complete. 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.978+0000 7f34d0d93700 1 Processor -- start 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.978+0000 7f34d0d93700 1 -- start start 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34d0d93700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34cc0fde90 0x7f34cc113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34d0d93700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fe830 0x7f34cc114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34d0d93700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc114680 con 0x7f34cc0fe830 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34d0d93700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc10da70 con 0x7f34cc0fde90 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34c9d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fe830 0x7f34cc114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34c9d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fe830 0x7f34cc114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45698/0 (socket says 172.21.15.32:45698) 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34c9d9b700 1 -- 172.21.15.32:0/2930216990 learned_addr learned my addr 172.21.15.32:0/2930216990 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:13.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34ca59c700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34cc0fde90 0x7f34cc113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34c9d9b700 1 -- 172.21.15.32:0/2930216990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34cc0fde90 msgr2=0x7f34cc113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34c9d9b700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34cc0fde90 0x7f34cc113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34c9d9b700 1 -- 172.21.15.32:0/2930216990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc005040 con 0x7f34cc0fe830 2024-04-06T14:54:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34c9d9b700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fe830 0x7f34cc114030 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f34c0002a50 tx=0x7f34c00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.979+0000 7f34baffd700 1 -- 172.21.15.32:0/2930216990 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34c0007cc0 con 0x7f34cc0fe830 2024-04-06T14:54:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.980+0000 7f34baffd700 1 -- 172.21.15.32:0/2930216990 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34c000f040 con 0x7f34cc0fe830 2024-04-06T14:54:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.980+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc10dd50 con 0x7f34cc0fe830 2024-04-06T14:54:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.980+0000 7f34baffd700 1 -- 172.21.15.32:0/2930216990 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34c000d9c0 con 0x7f34cc0fe830 2024-04-06T14:54:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.980+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc10e290 con 0x7f34cc0fe830 2024-04-06T14:54:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.981+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc1019b0 con 0x7f34cc0fe830 2024-04-06T14:54:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.981+0000 7f34baffd700 1 -- 172.21.15.32:0/2930216990 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f34c000db20 con 0x7f34cc0fe830 2024-04-06T14:54:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.982+0000 7f34baffd700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f34b406cc30 0x7f34b406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.982+0000 7f34ca59c700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f34b406cc30 0x7f34b406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.982+0000 7f34baffd700 1 -- 172.21.15.32:0/2930216990 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f34c008bd70 con 0x7f34cc0fe830 2024-04-06T14:54:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.982+0000 7f34ca59c700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f34b406cc30 0x7f34b406f0e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00b3b0 tx=0x7f34bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:13.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:13.986+0000 7f34baffd700 1 -- 172.21.15.32:0/2930216990 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f34c00597c0 con 0x7f34cc0fe830 2024-04-06T14:54:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.208+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f34cc1137c0 con 0x7f34b406cc30 2024-04-06T14:54:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.209+0000 7f34baffd700 1 -- 172.21.15.32:0/2930216990 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f34cc1137c0 con 0x7f34b406cc30 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f34b406cc30 msgr2=0x7f34b406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f34b406cc30 0x7f34b406f0e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00b3b0 tx=0x7f34bc009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fe830 msgr2=0x7f34cc114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fe830 0x7f34cc114030 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f34c0002a50 tx=0x7f34c00046b0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 shutdown_connections 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f34b406cc30 0x7f34b406f0e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34cc0fde90 0x7f34cc113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 --2- 172.21.15.32:0/2930216990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34cc0fe830 0x7f34cc114030 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 >> 172.21.15.32:0/2930216990 conn(0x7f34cc072bc0 msgr2=0x7f34cc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 shutdown_connections 2024-04-06T14:54:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.211+0000 7f34d0d93700 1 -- 172.21.15.32:0/2930216990 wait complete. 2024-04-06T14:54:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.314+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/1976878785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d541037a0 msgr2=0x7f1d540fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.314+0000 7f1d5c1ba700 1 --2- 172.21.15.32:0/1976878785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d541037a0 0x7f1d540fce10 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f1d48004940 tx=0x7f1d48009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.314+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/1976878785 shutdown_connections 2024-04-06T14:54:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.314+0000 7f1d5c1ba700 1 --2- 172.21.15.32:0/1976878785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d541037a0 0x7f1d540fce10 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.314+0000 7f1d5c1ba700 1 --2- 172.21.15.32:0/1976878785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d54102e00 0x7f1d541031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.314+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/1976878785 >> 172.21.15.32:0/1976878785 conn(0x7f1d540f89b0 msgr2=0x7f1d540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.314+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/1976878785 shutdown_connections 2024-04-06T14:54:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.314+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/1976878785 wait complete. 2024-04-06T14:54:14.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d5c1ba700 1 Processor -- start 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d5c1ba700 1 -- start start 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d5c1ba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d54102e00 0x7f1d54113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d5c1ba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d541037a0 0x7f1d54114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d5c1ba700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d54114690 con 0x7f1d54102e00 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d5c1ba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5410da80 con 0x7f1d541037a0 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d59f56700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d54102e00 0x7f1d54113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d59f56700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d54102e00 0x7f1d54113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45716/0 (socket says 172.21.15.32:45716) 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d59f56700 1 -- 172.21.15.32:0/2032316228 learned_addr learned my addr 172.21.15.32:0/2032316228 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d59755700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d541037a0 0x7f1d54114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d59f56700 1 -- 172.21.15.32:0/2032316228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d541037a0 msgr2=0x7f1d54114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d59f56700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d541037a0 0x7f1d54114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.315+0000 7f1d59f56700 1 -- 172.21.15.32:0/2032316228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d48005040 con 0x7f1d54102e00 2024-04-06T14:54:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.316+0000 7f1d59f56700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d54102e00 0x7f1d54113b00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1d50002a50 tx=0x7f1d500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.316+0000 7f1d467fc700 1 -- 172.21.15.32:0/2032316228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d50007cc0 con 0x7f1d54102e00 2024-04-06T14:54:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.316+0000 7f1d467fc700 1 -- 172.21.15.32:0/2032316228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d5000f040 con 0x7f1d54102e00 2024-04-06T14:54:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.316+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5410dd60 con 0x7f1d54102e00 2024-04-06T14:54:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.316+0000 7f1d467fc700 1 -- 172.21.15.32:0/2032316228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d50009540 con 0x7f1d54102e00 2024-04-06T14:54:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.316+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5410e320 con 0x7f1d54102e00 2024-04-06T14:54:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.317+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d540fe550 con 0x7f1d54102e00 2024-04-06T14:54:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.317+0000 7f1d467fc700 1 -- 172.21.15.32:0/2032316228 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1d500096a0 con 0x7f1d54102e00 2024-04-06T14:54:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.318+0000 7f1d467fc700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1d4006cc30 0x7f1d4006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.318+0000 7f1d467fc700 1 -- 172.21.15.32:0/2032316228 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1d5008be70 con 0x7f1d54102e00 2024-04-06T14:54:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.318+0000 7f1d59755700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1d4006cc30 0x7f1d4006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.318+0000 7f1d59755700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1d4006cc30 0x7f1d4006f0e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1d5410ec40 tx=0x7f1d48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:14.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.322+0000 7f1d467fc700 1 -- 172.21.15.32:0/2032316228 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1d500598c0 con 0x7f1d54102e00 2024-04-06T14:54:14.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.536+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d54109da0 con 0x7f1d4006cc30 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.546+0000 7f1d467fc700 1 -- 172.21.15.32:0/2032316228 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1d54109da0 con 0x7f1d4006cc30 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (34m) 12m ago 45m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (31m) 12m ago 46m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (31m) 12m ago 45m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (34m) 12m ago 45m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (28m) 12m ago 40m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (28m) 12m ago 40m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (27m) 12m ago 40m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:54:14.533 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (12m) 12m ago 39m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (36m) 12m ago 47m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (35m) 12m ago 44m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (33m) 12m ago 47m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (32m) 12m ago 44m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (35m) 12m ago 46m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (35m) 12m ago 45m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (31m) 12m ago 44m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (30m) 12m ago 43m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (30m) 12m ago 43m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (29m) 12m ago 42m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (29m) 12m ago 42m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (28m) 12m ago 41m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:54:14.534 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (34m) 12m ago 45m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:54:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.547+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1d4006cc30 msgr2=0x7f1d4006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.547+0000 7f1d5c1ba700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1d4006cc30 0x7f1d4006f0e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1d5410ec40 tx=0x7f1d48009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.547+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d54102e00 msgr2=0x7f1d54113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.547+0000 7f1d5c1ba700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d54102e00 0x7f1d54113b00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1d50002a50 tx=0x7f1d500046b0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.548+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 shutdown_connections 2024-04-06T14:54:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.548+0000 7f1d5c1ba700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1d4006cc30 0x7f1d4006f0e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.548+0000 7f1d5c1ba700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d54102e00 0x7f1d54113b00 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.548+0000 7f1d5c1ba700 1 --2- 172.21.15.32:0/2032316228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d541037a0 0x7f1d54114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.548+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 >> 172.21.15.32:0/2032316228 conn(0x7f1d540f89b0 msgr2=0x7f1d540f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:14.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.548+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 shutdown_connections 2024-04-06T14:54:14.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.548+0000 7f1d5c1ba700 1 -- 172.21.15.32:0/2032316228 wait complete. 2024-04-06T14:54:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.650+0000 7f8c5dac4700 1 -- 172.21.15.32:0/824201455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fe810 msgr2=0x7f8c58106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.650+0000 7f8c5dac4700 1 --2- 172.21.15.32:0/824201455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fe810 0x7f8c58106e50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8c40004940 tx=0x7f8c40009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.650+0000 7f8c5dac4700 1 -- 172.21.15.32:0/824201455 shutdown_connections 2024-04-06T14:54:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.650+0000 7f8c5dac4700 1 --2- 172.21.15.32:0/824201455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fe810 0x7f8c58106e50 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.650+0000 7f8c5dac4700 1 --2- 172.21.15.32:0/824201455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580fde70 0x7f8c580fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.650+0000 7f8c5dac4700 1 -- 172.21.15.32:0/824201455 >> 172.21.15.32:0/824201455 conn(0x7f8c580f7880 msgr2=0x7f8c580f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.650+0000 7f8c5dac4700 1 -- 172.21.15.32:0/824201455 shutdown_connections 2024-04-06T14:54:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.650+0000 7f8c5dac4700 1 -- 172.21.15.32:0/824201455 wait complete. 2024-04-06T14:54:14.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.650+0000 7f8c5dac4700 1 Processor -- start 2024-04-06T14:54:14.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c5dac4700 1 -- start start 2024-04-06T14:54:14.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c5dac4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fde70 0x7f8c58113c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:14.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c5dac4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580fe810 0x7f8c581141d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:14.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c5dac4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c58114820 con 0x7f8c580fde70 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c5dac4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5810dc40 con 0x7f8c580fe810 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c577fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fde70 0x7f8c58113c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c577fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fde70 0x7f8c58113c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41774/0 (socket says 172.21.15.32:41774) 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c577fe700 1 -- 172.21.15.32:0/388401559 learned_addr learned my addr 172.21.15.32:0/388401559 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c577fe700 1 -- 172.21.15.32:0/388401559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580fe810 msgr2=0x7f8c581141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c577fe700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580fe810 0x7f8c581141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c577fe700 1 -- 172.21.15.32:0/388401559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c40005040 con 0x7f8c580fde70 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.651+0000 7f8c577fe700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fde70 0x7f8c58113c90 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800eb40 tx=0x7f8c4800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.652+0000 7f8c4ffff700 1 -- 172.21.15.32:0/388401559 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c48007cc0 con 0x7f8c580fde70 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.652+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5810dec0 con 0x7f8c580fde70 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.652+0000 7f8c4ffff700 1 -- 172.21.15.32:0/388401559 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c4800f040 con 0x7f8c580fde70 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.652+0000 7f8c4ffff700 1 -- 172.21.15.32:0/388401559 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c48009570 con 0x7f8c580fde70 2024-04-06T14:54:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.652+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5810e400 con 0x7f8c580fde70 2024-04-06T14:54:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.653+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c58066710 con 0x7f8c580fde70 2024-04-06T14:54:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.653+0000 7f8c4ffff700 1 -- 172.21.15.32:0/388401559 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8c480096d0 con 0x7f8c580fde70 2024-04-06T14:54:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.653+0000 7f8c4ffff700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c4406cc20 0x7f8c4406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.654+0000 7f8c4ffff700 1 -- 172.21.15.32:0/388401559 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8c4808bd30 con 0x7f8c580fde70 2024-04-06T14:54:14.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.654+0000 7f8c56ffd700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c4406cc20 0x7f8c4406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:14.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.654+0000 7f8c56ffd700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c4406cc20 0x7f8c4406f0d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000b0d0 tx=0x7f8c40009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:14.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.658+0000 7f8c4ffff700 1 -- 172.21.15.32:0/388401559 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8c48059830 con 0x7f8c580fde70 2024-04-06T14:54:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.947+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8c5804bed0 con 0x7f8c580fde70 2024-04-06T14:54:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.948+0000 7f8c4ffff700 1 -- 172.21.15.32:0/388401559 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f8c480593c0 con 0x7f8c580fde70 2024-04-06T14:54:14.935 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:54:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:54:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:54:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:54:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:54:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:54:14.936 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:54:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c4406cc20 msgr2=0x7f8c4406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c4406cc20 0x7f8c4406f0d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000b0d0 tx=0x7f8c40009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fde70 msgr2=0x7f8c58113c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fde70 0x7f8c58113c90 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800eb40 tx=0x7f8c4800b040 comp rx=0 tx=0).stop 2024-04-06T14:54:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 shutdown_connections 2024-04-06T14:54:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f8c4406cc20 0x7f8c4406f0d0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580fde70 0x7f8c58113c90 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 --2- 172.21.15.32:0/388401559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580fe810 0x7f8c581141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 >> 172.21.15.32:0/388401559 conn(0x7f8c580f7880 msgr2=0x7f8c580f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:14.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 shutdown_connections 2024-04-06T14:54:14.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:14.950+0000 7f8c5dac4700 1 -- 172.21.15.32:0/388401559 wait complete. 2024-04-06T14:54:15.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/3757966450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a481037b0 msgr2=0x7f0a480fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 --2- 172.21.15.32:0/3757966450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a481037b0 0x7f0a480fce20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f0a38004940 tx=0x7f0a38009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:15.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/3757966450 shutdown_connections 2024-04-06T14:54:15.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 --2- 172.21.15.32:0/3757966450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a481037b0 0x7f0a480fce20 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 --2- 172.21.15.32:0/3757966450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a48102e10 0x7f0a481031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/3757966450 >> 172.21.15.32:0/3757966450 conn(0x7f0a480f8980 msgr2=0x7f0a480fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:15.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/3757966450 shutdown_connections 2024-04-06T14:54:15.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/3757966450 wait complete. 2024-04-06T14:54:15.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 Processor -- start 2024-04-06T14:54:15.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.053+0000 7f0a4d1e4700 1 -- start start 2024-04-06T14:54:15.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a4d1e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a48102e10 0x7f0a48113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a4d1e4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a481037b0 0x7f0a48114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a4d1e4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a48114670 con 0x7f0a48102e10 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a4d1e4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4810da60 con 0x7f0a481037b0 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a46d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a48102e10 0x7f0a48113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a46d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a48102e10 0x7f0a48113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41792/0 (socket says 172.21.15.32:41792) 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a46d9d700 1 -- 172.21.15.32:0/1693467560 learned_addr learned my addr 172.21.15.32:0/1693467560 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a4659c700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a481037b0 0x7f0a48114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a46d9d700 1 -- 172.21.15.32:0/1693467560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a481037b0 msgr2=0x7f0a48114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a46d9d700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a481037b0 0x7f0a48114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a46d9d700 1 -- 172.21.15.32:0/1693467560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a38005040 con 0x7f0a48102e10 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a4659c700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a481037b0 0x7f0a48114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a46d9d700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a48102e10 0x7f0a48113ae0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00c8c0 tx=0x7f0a3c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:15.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a2f7fe700 1 -- 172.21.15.32:0/1693467560 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a3c00a980 con 0x7f0a48102e10 2024-04-06T14:54:15.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a2f7fe700 1 -- 172.21.15.32:0/1693467560 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a3c00ce60 con 0x7f0a48102e10 2024-04-06T14:54:15.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4810dd40 con 0x7f0a48102e10 2024-04-06T14:54:15.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.054+0000 7f0a2f7fe700 1 -- 172.21.15.32:0/1693467560 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a3c0113a0 con 0x7f0a48102e10 2024-04-06T14:54:15.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.055+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4810e300 con 0x7f0a48102e10 2024-04-06T14:54:15.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.055+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a480fe560 con 0x7f0a48102e10 2024-04-06T14:54:15.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.056+0000 7f0a2f7fe700 1 -- 172.21.15.32:0/1693467560 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0a3c012400 con 0x7f0a48102e10 2024-04-06T14:54:15.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.056+0000 7f0a2f7fe700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a3006cc30 0x7f0a3006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:15.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.057+0000 7f0a4659c700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a3006cc30 0x7f0a3006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:15.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.057+0000 7f0a2f7fe700 1 -- 172.21.15.32:0/1693467560 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0a3c08c160 con 0x7f0a48102e10 2024-04-06T14:54:15.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.057+0000 7f0a4659c700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a3006cc30 0x7f0a3006f0e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800b3b0 tx=0x7f0a38009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:15.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.061+0000 7f0a2f7fe700 1 -- 172.21.15.32:0/1693467560 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0a3c059c30 con 0x7f0a48102e10 2024-04-06T14:54:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.306+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0a48109db0 con 0x7f0a48102e10 2024-04-06T14:54:15.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.307+0000 7f0a2f7fe700 1 -- 172.21.15.32:0/1693467560 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f0a3c0597c0 con 0x7f0a48102e10 2024-04-06T14:54:15.294 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:54:15.294 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:54:15.294 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:54:15.294 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:54:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:54:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:54:15.297 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:54:15.297 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:54:15.297 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:54:15.297 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:54:15.297 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:54:15.297 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:54:15.297 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:54:15.297 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:54:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a3006cc30 msgr2=0x7f0a3006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a3006cc30 0x7f0a3006f0e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800b3b0 tx=0x7f0a38009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a48102e10 msgr2=0x7f0a48113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a48102e10 0x7f0a48113ae0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00c8c0 tx=0x7f0a3c0044c0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 shutdown_connections 2024-04-06T14:54:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f0a3006cc30 0x7f0a3006f0e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a48102e10 0x7f0a48113ae0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 --2- 172.21.15.32:0/1693467560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a481037b0 0x7f0a48114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 >> 172.21.15.32:0/1693467560 conn(0x7f0a480f8980 msgr2=0x7f0a480f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:15.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 shutdown_connections 2024-04-06T14:54:15.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.309+0000 7f0a4d1e4700 1 -- 172.21.15.32:0/1693467560 wait complete. 2024-04-06T14:54:15.299 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.413+0000 7f2db0069700 1 -- 172.21.15.32:0/276623723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 msgr2=0x7f2da81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.413+0000 7f2db0069700 1 --2- 172.21.15.32:0/276623723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 0x7f2da81031e0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f2da4004940 tx=0x7f2da4009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.413+0000 7f2db0069700 1 -- 172.21.15.32:0/276623723 shutdown_connections 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.413+0000 7f2db0069700 1 --2- 172.21.15.32:0/276623723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2da81037b0 0x7f2da80fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.413+0000 7f2db0069700 1 --2- 172.21.15.32:0/276623723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 0x7f2da81031e0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.413+0000 7f2db0069700 1 -- 172.21.15.32:0/276623723 >> 172.21.15.32:0/276623723 conn(0x7f2da80f8980 msgr2=0x7f2da80fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.413+0000 7f2db0069700 1 -- 172.21.15.32:0/276623723 shutdown_connections 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.413+0000 7f2db0069700 1 -- 172.21.15.32:0/276623723 wait complete. 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2db0069700 1 Processor -- start 2024-04-06T14:54:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2db0069700 1 -- start start 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2db0069700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 0x7f2da8113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2db0069700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2da81037b0 0x7f2da8114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2db0069700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da81146a0 con 0x7f2da8102e10 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2db0069700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da810da90 con 0x7f2da81037b0 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2dade05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 0x7f2da8113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2dade05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 0x7f2da8113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41818/0 (socket says 172.21.15.32:41818) 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2dade05700 1 -- 172.21.15.32:0/3936324598 learned_addr learned my addr 172.21.15.32:0/3936324598 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.414+0000 7f2dad604700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2da81037b0 0x7f2da8114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.415+0000 7f2dade05700 1 -- 172.21.15.32:0/3936324598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2da81037b0 msgr2=0x7f2da8114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.415+0000 7f2dade05700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2da81037b0 0x7f2da8114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.415+0000 7f2dade05700 1 -- 172.21.15.32:0/3936324598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da4005040 con 0x7f2da8102e10 2024-04-06T14:54:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.415+0000 7f2dade05700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 0x7f2da8113b10 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2da400b3b0 tx=0x7f2da4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.415+0000 7f2d9a7fc700 1 -- 172.21.15.32:0/3936324598 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2da400e040 con 0x7f2da8102e10 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.415+0000 7f2d9a7fc700 1 -- 172.21.15.32:0/3936324598 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2da4013420 con 0x7f2da8102e10 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.415+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da810dd10 con 0x7f2da8102e10 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.415+0000 7f2d9a7fc700 1 -- 172.21.15.32:0/3936324598 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2da401c610 con 0x7f2da8102e10 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.415+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da810e2d0 con 0x7f2da8102e10 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.416+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da80fe560 con 0x7f2da8102e10 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.416+0000 7f2d9a7fc700 1 -- 172.21.15.32:0/3936324598 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2da4013a00 con 0x7f2da8102e10 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.417+0000 7f2d9a7fc700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d9406cc20 0x7f2d9406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.417+0000 7f2d9a7fc700 1 -- 172.21.15.32:0/3936324598 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2da408d1c0 con 0x7f2da8102e10 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.417+0000 7f2dad604700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d9406cc20 0x7f2d9406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.417+0000 7f2dad604700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d9406cc20 0x7f2d9406f0d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2da810ec50 tx=0x7f2d9c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:15.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.421+0000 7f2d9a7fc700 1 -- 172.21.15.32:0/3936324598 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2da405ac90 con 0x7f2da8102e10 2024-04-06T14:54:15.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.643+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2da8109e40 con 0x7f2d9406cc20 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.645+0000 7f2d9a7fc700 1 -- 172.21.15.32:0/3936324598 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2da8109e40 con 0x7f2d9406cc20 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:54:15.632 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:54:15.633 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:54:15.633 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:54:15.633 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:54:15.633 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:54:15.633 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:54:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d9406cc20 msgr2=0x7f2d9406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d9406cc20 0x7f2d9406f0d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2da810ec50 tx=0x7f2d9c00b040 comp rx=0 tx=0).stop 2024-04-06T14:54:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 msgr2=0x7f2da8113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 0x7f2da8113b10 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2da400b3b0 tx=0x7f2da4009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 shutdown_connections 2024-04-06T14:54:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d9406cc20 0x7f2d9406f0d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8102e10 0x7f2da8113b10 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 --2- 172.21.15.32:0/3936324598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2da81037b0 0x7f2da8114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 >> 172.21.15.32:0/3936324598 conn(0x7f2da80f8980 msgr2=0x7f2da80f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 shutdown_connections 2024-04-06T14:54:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.647+0000 7f2db0069700 1 -- 172.21.15.32:0/3936324598 wait complete. 2024-04-06T14:54:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.750+0000 7f78ca7db700 1 -- 172.21.15.32:0/1420016998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 msgr2=0x7f78c41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.750+0000 7f78ca7db700 1 --2- 172.21.15.32:0/1420016998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 0x7f78c41031e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f78b4004940 tx=0x7f78b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.750+0000 7f78ca7db700 1 -- 172.21.15.32:0/1420016998 shutdown_connections 2024-04-06T14:54:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.750+0000 7f78ca7db700 1 --2- 172.21.15.32:0/1420016998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c41037b0 0x7f78c40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.750+0000 7f78ca7db700 1 --2- 172.21.15.32:0/1420016998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 0x7f78c41031e0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.750+0000 7f78ca7db700 1 -- 172.21.15.32:0/1420016998 >> 172.21.15.32:0/1420016998 conn(0x7f78c40f89a0 msgr2=0x7f78c40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.750+0000 7f78ca7db700 1 -- 172.21.15.32:0/1420016998 shutdown_connections 2024-04-06T14:54:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.750+0000 7f78ca7db700 1 -- 172.21.15.32:0/1420016998 wait complete. 2024-04-06T14:54:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.751+0000 7f78ca7db700 1 Processor -- start 2024-04-06T14:54:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.751+0000 7f78ca7db700 1 -- start start 2024-04-06T14:54:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.751+0000 7f78ca7db700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 0x7f78c4113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.751+0000 7f78ca7db700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c41037b0 0x7f78c4113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.751+0000 7f78ca7db700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c41146d0 con 0x7f78c4102e10 2024-04-06T14:54:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.751+0000 7f78ca7db700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c410da30 con 0x7f78c41037b0 2024-04-06T14:54:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.751+0000 7f78c3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 0x7f78c4113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.751+0000 7f78c3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 0x7f78c4113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41828/0 (socket says 172.21.15.32:41828) 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.751+0000 7f78c3fff700 1 -- 172.21.15.32:0/63131492 learned_addr learned my addr 172.21.15.32:0/63131492 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78c37fe700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c41037b0 0x7f78c4113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78c3fff700 1 -- 172.21.15.32:0/63131492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c41037b0 msgr2=0x7f78c4113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78c3fff700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c41037b0 0x7f78c4113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78c3fff700 1 -- 172.21.15.32:0/63131492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4005040 con 0x7f78c4102e10 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78c3fff700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 0x7f78c4113ab0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009360 tx=0x7f78b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78c0ff9700 1 -- 172.21.15.32:0/63131492 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78b400e040 con 0x7f78c4102e10 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78c0ff9700 1 -- 172.21.15.32:0/63131492 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78b4013420 con 0x7f78c4102e10 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c410dcb0 con 0x7f78c4102e10 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78c0ff9700 1 -- 172.21.15.32:0/63131492 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78b401c5f0 con 0x7f78c4102e10 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.752+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c410e270 con 0x7f78c4102e10 2024-04-06T14:54:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.753+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c40fe560 con 0x7f78c4102e10 2024-04-06T14:54:15.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.754+0000 7f78c0ff9700 1 -- 172.21.15.32:0/63131492 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f78b4013590 con 0x7f78c4102e10 2024-04-06T14:54:15.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.754+0000 7f78c0ff9700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78ac06cc30 0x7f78ac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:15.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.754+0000 7f78c37fe700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78ac06cc30 0x7f78ac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:15.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.754+0000 7f78c0ff9700 1 -- 172.21.15.32:0/63131492 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f78b408d190 con 0x7f78c4102e10 2024-04-06T14:54:15.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.755+0000 7f78c37fe700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78ac06cc30 0x7f78ac06f0e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f78b8006fd0 tx=0x7f78b8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:15.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:15.759+0000 7f78c0ff9700 1 -- 172.21.15.32:0/63131492 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f78b405ad20 con 0x7f78c4102e10 2024-04-06T14:54:15.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:15 smithi086 ceph-mon[110975]: from='client.35850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:15.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:15 smithi086 ceph-mon[110975]: from='client.35854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:15.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:15 smithi086 ceph-mon[110975]: pgmap v1024: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:15.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:15 smithi086 ceph-mon[110975]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:15.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:15 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/388401559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:54:15.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:15 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1693467560' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:54:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:15 smithi032 ceph-mon[147414]: from='client.35850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:15 smithi032 ceph-mon[147414]: from='client.35854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:15 smithi032 ceph-mon[147414]: pgmap v1024: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:15 smithi032 ceph-mon[147414]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:15 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/388401559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:54:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:15 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1693467560' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:54:16.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.048+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f78c404bed0 con 0x7f78c4102e10 2024-04-06T14:54:16.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.048+0000 7f78c0ff9700 1 -- 172.21.15.32:0/63131492 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f78b405a8b0 con 0x7f78c4102e10 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:54:16.036 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:54:16.037 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:54:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78ac06cc30 msgr2=0x7f78ac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78ac06cc30 0x7f78ac06f0e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f78b8006fd0 tx=0x7f78b8008040 comp rx=0 tx=0).stop 2024-04-06T14:54:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 msgr2=0x7f78c4113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 0x7f78c4113ab0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009360 tx=0x7f78b4009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 shutdown_connections 2024-04-06T14:54:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f78ac06cc30 0x7f78ac06f0e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102e10 0x7f78c4113ab0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 --2- 172.21.15.32:0/63131492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c41037b0 0x7f78c4113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:16.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 >> 172.21.15.32:0/63131492 conn(0x7f78c40f89a0 msgr2=0x7f78c40f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:16.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 shutdown_connections 2024-04-06T14:54:16.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:16.050+0000 7f78ca7db700 1 -- 172.21.15.32:0/63131492 wait complete. 2024-04-06T14:54:16.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:16 smithi086 ceph-mon[110975]: from='client.35870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:16.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:16 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/63131492' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:54:16.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:16 smithi032 ceph-mon[147414]: from='client.35870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:16.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:16 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/63131492' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:54:17.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:17 smithi086 ceph-mon[110975]: pgmap v1025: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:17.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:17 smithi032 ceph-mon[147414]: pgmap v1025: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:19.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:19 smithi086 ceph-mon[110975]: pgmap v1026: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:19.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:19 smithi032 ceph-mon[147414]: pgmap v1026: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:21.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:21 smithi086 ceph-mon[110975]: pgmap v1027: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:21.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:21 smithi032 ceph-mon[147414]: pgmap v1027: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:23.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:23 smithi032 ceph-mon[147414]: pgmap v1028: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:23.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:54:23.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:23 smithi086 ceph-mon[110975]: pgmap v1028: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:23.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:54:25.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:25 smithi032 ceph-mon[147414]: pgmap v1029: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:26.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:25 smithi086 ceph-mon[110975]: pgmap v1029: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:27.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:27 smithi032 ceph-mon[147414]: pgmap v1030: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:28.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:27 smithi086 ceph-mon[110975]: pgmap v1030: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:29.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:29 smithi032 ceph-mon[147414]: pgmap v1031: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:30.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:29 smithi086 ceph-mon[110975]: pgmap v1031: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:31.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:31 smithi032 ceph-mon[147414]: pgmap v1032: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:32.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:31 smithi086 ceph-mon[110975]: pgmap v1032: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:33.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:33 smithi032 ceph-mon[147414]: pgmap v1033: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:34.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:33 smithi086 ceph-mon[110975]: pgmap v1033: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:35.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:35 smithi032 ceph-mon[147414]: pgmap v1034: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:36.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:35 smithi086 ceph-mon[110975]: pgmap v1034: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:37.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:37 smithi032 ceph-mon[147414]: pgmap v1035: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:37.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:54:38.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:37 smithi086 ceph-mon[110975]: pgmap v1035: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:38.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:54:39.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:39 smithi032 ceph-mon[147414]: pgmap v1036: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:40.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:39 smithi086 ceph-mon[110975]: pgmap v1036: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:41.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:41 smithi032 ceph-mon[147414]: pgmap v1037: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:42.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:41 smithi086 ceph-mon[110975]: pgmap v1037: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:43.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:43 smithi032 ceph-mon[147414]: pgmap v1038: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:44.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:43 smithi086 ceph-mon[110975]: pgmap v1038: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:45.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:45 smithi032 ceph-mon[147414]: pgmap v1039: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:46.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:45 smithi086 ceph-mon[110975]: pgmap v1039: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:46.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.167+0000 7fb65175f700 1 -- 172.21.15.32:0/3671632874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c101530 msgr2=0x7fb64c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.167+0000 7fb65175f700 1 --2- 172.21.15.32:0/3671632874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c101530 0x7fb64c101900 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb63c004940 tx=0x7fb63c009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:46.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.167+0000 7fb65175f700 1 -- 172.21.15.32:0/3671632874 shutdown_connections 2024-04-06T14:54:46.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.167+0000 7fb65175f700 1 --2- 172.21.15.32:0/3671632874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb64c0fb570 0x7fb64c0fb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.167+0000 7fb65175f700 1 --2- 172.21.15.32:0/3671632874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c101530 0x7fb64c101900 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.167+0000 7fb65175f700 1 -- 172.21.15.32:0/3671632874 >> 172.21.15.32:0/3671632874 conn(0x7fb64c0714c0 msgr2=0x7fb64c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:46.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.167+0000 7fb65175f700 1 -- 172.21.15.32:0/3671632874 shutdown_connections 2024-04-06T14:54:46.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.168+0000 7fb65175f700 1 -- 172.21.15.32:0/3671632874 wait complete. 2024-04-06T14:54:46.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.168+0000 7fb65175f700 1 Processor -- start 2024-04-06T14:54:46.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.168+0000 7fb65175f700 1 -- start start 2024-04-06T14:54:46.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb65175f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c0fb570 0x7fb64c113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:46.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb65175f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb64c101530 0x7fb64c113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb65175f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c1143b0 con 0x7fb64c0fb570 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb65175f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c10d9b0 con 0x7fb64c101530 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb64affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c0fb570 0x7fb64c113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb64affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c0fb570 0x7fb64c113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37710/0 (socket says 172.21.15.32:37710) 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb64affd700 1 -- 172.21.15.32:0/492708324 learned_addr learned my addr 172.21.15.32:0/492708324 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb64a7fc700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb64c101530 0x7fb64c113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb64affd700 1 -- 172.21.15.32:0/492708324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb64c101530 msgr2=0x7fb64c113d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb64affd700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb64c101530 0x7fb64c113d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb64affd700 1 -- 172.21.15.32:0/492708324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c005040 con 0x7fb64c0fb570 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb64a7fc700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb64c101530 0x7fb64c113d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb64affd700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c0fb570 0x7fb64c113820 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb63c008d80 tx=0x7fb63c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb6337fe700 1 -- 172.21.15.32:0/492708324 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63c00e030 con 0x7fb64c0fb570 2024-04-06T14:54:46.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb6337fe700 1 -- 172.21.15.32:0/492708324 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb63c013de0 con 0x7fb64c0fb570 2024-04-06T14:54:46.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64c10dc30 con 0x7fb64c0fb570 2024-04-06T14:54:46.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.169+0000 7fb6337fe700 1 -- 172.21.15.32:0/492708324 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63c01cd60 con 0x7fb64c0fb570 2024-04-06T14:54:46.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.170+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64c10e1f0 con 0x7fb64c0fb570 2024-04-06T14:54:46.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.171+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64c1059f0 con 0x7fb64c0fb570 2024-04-06T14:54:46.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.171+0000 7fb6337fe700 1 -- 172.21.15.32:0/492708324 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb63c01c590 con 0x7fb64c0fb570 2024-04-06T14:54:46.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.171+0000 7fb6337fe700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb63406cc20 0x7fb63406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:46.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.171+0000 7fb6337fe700 1 -- 172.21.15.32:0/492708324 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb63c08d930 con 0x7fb64c0fb570 2024-04-06T14:54:46.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.171+0000 7fb64a7fc700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb63406cc20 0x7fb63406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:46.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.172+0000 7fb64a7fc700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb63406cc20 0x7fb63406f0d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb640004710 tx=0x7fb64000c040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:46.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.176+0000 7fb6337fe700 1 -- 172.21.15.32:0/492708324 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb63c05b400 con 0x7fb64c0fb570 2024-04-06T14:54:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.398+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb64c10ee70 con 0x7fb63406cc20 2024-04-06T14:54:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.399+0000 7fb6337fe700 1 -- 172.21.15.32:0/492708324 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb64c10ee70 con 0x7fb63406cc20 2024-04-06T14:54:46.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb63406cc20 msgr2=0x7fb63406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb63406cc20 0x7fb63406f0d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb640004710 tx=0x7fb64000c040 comp rx=0 tx=0).stop 2024-04-06T14:54:46.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c0fb570 msgr2=0x7fb64c113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c0fb570 0x7fb64c113820 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb63c008d80 tx=0x7fb63c009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:46.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 shutdown_connections 2024-04-06T14:54:46.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb63406cc20 0x7fb63406f0d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb64c0fb570 0x7fb64c113820 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 --2- 172.21.15.32:0/492708324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb64c101530 0x7fb64c113d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 >> 172.21.15.32:0/492708324 conn(0x7fb64c0714c0 msgr2=0x7fb64c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:46.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 shutdown_connections 2024-04-06T14:54:46.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.401+0000 7fb65175f700 1 -- 172.21.15.32:0/492708324 wait complete. 2024-04-06T14:54:46.398 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:54:46.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.505+0000 7efe2a452700 1 -- 172.21.15.32:0/2907539783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fbc80 msgr2=0x7efe240fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.505+0000 7efe2a452700 1 --2- 172.21.15.32:0/2907539783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fbc80 0x7efe240fc050 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7efe0c004940 tx=0x7efe0c009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.505+0000 7efe2a452700 1 -- 172.21.15.32:0/2907539783 shutdown_connections 2024-04-06T14:54:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.505+0000 7efe2a452700 1 --2- 172.21.15.32:0/2907539783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe240fc590 0x7efe241092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.505+0000 7efe2a452700 1 --2- 172.21.15.32:0/2907539783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fbc80 0x7efe240fc050 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.505+0000 7efe2a452700 1 -- 172.21.15.32:0/2907539783 >> 172.21.15.32:0/2907539783 conn(0x7efe24072bc0 msgr2=0x7efe24072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.505+0000 7efe2a452700 1 -- 172.21.15.32:0/2907539783 shutdown_connections 2024-04-06T14:54:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.505+0000 7efe2a452700 1 -- 172.21.15.32:0/2907539783 wait complete. 2024-04-06T14:54:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe2a452700 1 Processor -- start 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe2a452700 1 -- start start 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe2a452700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe240fbc80 0x7efe24115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe2a452700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fc590 0x7efe241161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe2a452700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe24116840 con 0x7efe240fc590 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe2a452700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2410fc10 con 0x7efe240fbc80 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe237fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fc590 0x7efe241161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe237fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fc590 0x7efe241161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37736/0 (socket says 172.21.15.32:37736) 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe237fe700 1 -- 172.21.15.32:0/37326163 learned_addr learned my addr 172.21.15.32:0/37326163 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe23fff700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe240fbc80 0x7efe24115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe237fe700 1 -- 172.21.15.32:0/37326163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe240fbc80 msgr2=0x7efe24115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe237fe700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe240fbc80 0x7efe24115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe237fe700 1 -- 172.21.15.32:0/37326163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c005040 con 0x7efe240fc590 2024-04-06T14:54:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.506+0000 7efe23fff700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe240fbc80 0x7efe24115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:54:46.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.507+0000 7efe237fe700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fc590 0x7efe241161f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7efe14002a50 tx=0x7efe140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:46.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.507+0000 7efe20ff9700 1 -- 172.21.15.32:0/37326163 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe14007cc0 con 0x7efe240fc590 2024-04-06T14:54:46.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.507+0000 7efe20ff9700 1 -- 172.21.15.32:0/37326163 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe1400f040 con 0x7efe240fc590 2024-04-06T14:54:46.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.507+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2410fef0 con 0x7efe240fc590 2024-04-06T14:54:46.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.507+0000 7efe20ff9700 1 -- 172.21.15.32:0/37326163 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe1400da70 con 0x7efe240fc590 2024-04-06T14:54:46.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.507+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe241104b0 con 0x7efe240fc590 2024-04-06T14:54:46.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.508+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe24065e30 con 0x7efe240fc590 2024-04-06T14:54:46.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.509+0000 7efe20ff9700 1 -- 172.21.15.32:0/37326163 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efe1400dc10 con 0x7efe240fc590 2024-04-06T14:54:46.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.509+0000 7efe20ff9700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe1006cc20 0x7efe1006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:46.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.509+0000 7efe20ff9700 1 -- 172.21.15.32:0/37326163 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efe1408b7a0 con 0x7efe240fc590 2024-04-06T14:54:46.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.509+0000 7efe23fff700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe1006cc20 0x7efe1006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:46.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.509+0000 7efe23fff700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe1006cc20 0x7efe1006f0d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7efe0c004560 tx=0x7efe0c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:46.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.513+0000 7efe20ff9700 1 -- 172.21.15.32:0/37326163 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efe140591f0 con 0x7efe240fc590 2024-04-06T14:54:46.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.734+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efe2410c040 con 0x7efe1006cc20 2024-04-06T14:54:46.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.735+0000 7efe20ff9700 1 -- 172.21.15.32:0/37326163 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7efe2410c040 con 0x7efe1006cc20 2024-04-06T14:54:46.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe1006cc20 msgr2=0x7efe1006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe1006cc20 0x7efe1006f0d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7efe0c004560 tx=0x7efe0c009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:46.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fc590 msgr2=0x7efe241161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fc590 0x7efe241161f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7efe14002a50 tx=0x7efe140046b0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 shutdown_connections 2024-04-06T14:54:46.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efe1006cc20 0x7efe1006f0d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe240fbc80 0x7efe24115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 --2- 172.21.15.32:0/37326163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe240fc590 0x7efe241161f0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 >> 172.21.15.32:0/37326163 conn(0x7efe24072bc0 msgr2=0x7efe24105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:46.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 shutdown_connections 2024-04-06T14:54:46.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.737+0000 7efe2a452700 1 -- 172.21.15.32:0/37326163 wait complete. 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.840+0000 7fc898655700 1 -- 172.21.15.32:0/1537185310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fe810 msgr2=0x7fc890106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.840+0000 7fc898655700 1 --2- 172.21.15.32:0/1537185310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fe810 0x7fc890106e50 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc88c004940 tx=0x7fc88c009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.840+0000 7fc898655700 1 -- 172.21.15.32:0/1537185310 shutdown_connections 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.840+0000 7fc898655700 1 --2- 172.21.15.32:0/1537185310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fe810 0x7fc890106e50 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.840+0000 7fc898655700 1 --2- 172.21.15.32:0/1537185310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900fde70 0x7fc8900fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.840+0000 7fc898655700 1 -- 172.21.15.32:0/1537185310 >> 172.21.15.32:0/1537185310 conn(0x7fc8900f7880 msgr2=0x7fc8900f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.840+0000 7fc898655700 1 -- 172.21.15.32:0/1537185310 shutdown_connections 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.840+0000 7fc898655700 1 -- 172.21.15.32:0/1537185310 wait complete. 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc898655700 1 Processor -- start 2024-04-06T14:54:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc898655700 1 -- start start 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc898655700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fde70 0x7fc8900fbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc898655700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900fe810 0x7fc8900fc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc898655700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8900fc770 con 0x7fc8900fde70 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc898655700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8900fc8b0 con 0x7fc8900fe810 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc8963f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fde70 0x7fc8900fbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc8963f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fde70 0x7fc8900fbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37756/0 (socket says 172.21.15.32:37756) 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc8963f1700 1 -- 172.21.15.32:0/16665298 learned_addr learned my addr 172.21.15.32:0/16665298 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc895bf0700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900fe810 0x7fc8900fc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc8963f1700 1 -- 172.21.15.32:0/16665298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900fe810 msgr2=0x7fc8900fc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc8963f1700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900fe810 0x7fc8900fc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc8963f1700 1 -- 172.21.15.32:0/16665298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c005040 con 0x7fc8900fde70 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.841+0000 7fc8963f1700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fde70 0x7fc8900fbca0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc880002a50 tx=0x7fc8800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.842+0000 7fc886ffd700 1 -- 172.21.15.32:0/16665298 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc880007cc0 con 0x7fc8900fde70 2024-04-06T14:54:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.842+0000 7fc886ffd700 1 -- 172.21.15.32:0/16665298 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc88000f040 con 0x7fc8900fde70 2024-04-06T14:54:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.842+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89011a040 con 0x7fc8900fde70 2024-04-06T14:54:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.842+0000 7fc886ffd700 1 -- 172.21.15.32:0/16665298 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc88000d9c0 con 0x7fc8900fde70 2024-04-06T14:54:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.842+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89011a440 con 0x7fc8900fde70 2024-04-06T14:54:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.843+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc890066710 con 0x7fc8900fde70 2024-04-06T14:54:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.843+0000 7fc886ffd700 1 -- 172.21.15.32:0/16665298 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc88000db60 con 0x7fc8900fde70 2024-04-06T14:54:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.844+0000 7fc886ffd700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc87c06cc30 0x7fc87c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.844+0000 7fc886ffd700 1 -- 172.21.15.32:0/16665298 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc88008b610 con 0x7fc8900fde70 2024-04-06T14:54:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.844+0000 7fc895bf0700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc87c06cc30 0x7fc87c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.844+0000 7fc895bf0700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc87c06cc30 0x7fc87c06f0e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc8900fd5e0 tx=0x7fc88c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:46.848+0000 7fc886ffd700 1 -- 172.21.15.32:0/16665298 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc880055650 con 0x7fc8900fde70 2024-04-06T14:54:47.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.060+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8900fd390 con 0x7fc87c06cc30 2024-04-06T14:54:47.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.070+0000 7fc886ffd700 1 -- 172.21.15.32:0/16665298 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc8900fd390 con 0x7fc87c06cc30 2024-04-06T14:54:47.057 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (35m) 13m ago 46m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (32m) 13m ago 46m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (32m) 13m ago 45m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (34m) 13m ago 46m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (28m) 13m ago 40m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (28m) 13m ago 40m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (28m) 13m ago 40m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (13m) 13m ago 40m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (37m) 13m ago 47m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (36m) 13m ago 45m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (33m) 13m ago 47m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (33m) 13m ago 45m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:54:47.058 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (35m) 13m ago 46m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:54:47.059 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (35m) 13m ago 45m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:54:47.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (32m) 13m ago 44m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:54:47.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (31m) 13m ago 44m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:54:47.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (30m) 13m ago 43m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:54:47.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (30m) 13m ago 43m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:54:47.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (29m) 13m ago 42m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:54:47.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (29m) 13m ago 42m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:54:47.059 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (35m) 13m ago 45m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.072+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc87c06cc30 msgr2=0x7fc87c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.072+0000 7fc898655700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc87c06cc30 0x7fc87c06f0e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc8900fd5e0 tx=0x7fc88c009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.072+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fde70 msgr2=0x7fc8900fbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.072+0000 7fc898655700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fde70 0x7fc8900fbca0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc880002a50 tx=0x7fc8800046b0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.072+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 shutdown_connections 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.072+0000 7fc898655700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc87c06cc30 0x7fc87c06f0e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.072+0000 7fc898655700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8900fde70 0x7fc8900fbca0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.072+0000 7fc898655700 1 --2- 172.21.15.32:0/16665298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900fe810 0x7fc8900fc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.072+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 >> 172.21.15.32:0/16665298 conn(0x7fc8900f7880 msgr2=0x7fc8900f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.073+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 shutdown_connections 2024-04-06T14:54:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.073+0000 7fc898655700 1 -- 172.21.15.32:0/16665298 wait complete. 2024-04-06T14:54:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.175+0000 7efe02025700 1 -- 172.21.15.32:0/2593691507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 msgr2=0x7efdfc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.175+0000 7efe02025700 1 --2- 172.21.15.32:0/2593691507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 0x7efdfc1031e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7efdf0004940 tx=0x7efdf0009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.175+0000 7efe02025700 1 -- 172.21.15.32:0/2593691507 shutdown_connections 2024-04-06T14:54:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.175+0000 7efe02025700 1 --2- 172.21.15.32:0/2593691507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdfc1037b0 0x7efdfc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.175+0000 7efe02025700 1 --2- 172.21.15.32:0/2593691507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 0x7efdfc1031e0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.175+0000 7efe02025700 1 -- 172.21.15.32:0/2593691507 >> 172.21.15.32:0/2593691507 conn(0x7efdfc0f8980 msgr2=0x7efdfc0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.175+0000 7efe02025700 1 -- 172.21.15.32:0/2593691507 shutdown_connections 2024-04-06T14:54:47.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.175+0000 7efe02025700 1 -- 172.21.15.32:0/2593691507 wait complete. 2024-04-06T14:54:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.175+0000 7efe02025700 1 Processor -- start 2024-04-06T14:54:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efe02025700 1 -- start start 2024-04-06T14:54:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efe02025700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 0x7efdfc113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efe02025700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdfc1037b0 0x7efdfc114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efe02025700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1146a0 con 0x7efdfc102e10 2024-04-06T14:54:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efe02025700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc10da90 con 0x7efdfc1037b0 2024-04-06T14:54:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efdfb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 0x7efdfc113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efdfb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 0x7efdfc113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37772/0 (socket says 172.21.15.32:37772) 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efdfb7fe700 1 -- 172.21.15.32:0/3904686825 learned_addr learned my addr 172.21.15.32:0/3904686825 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efdfaffd700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdfc1037b0 0x7efdfc114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efdfb7fe700 1 -- 172.21.15.32:0/3904686825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdfc1037b0 msgr2=0x7efdfc114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efdfb7fe700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdfc1037b0 0x7efdfc114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.176+0000 7efdfb7fe700 1 -- 172.21.15.32:0/3904686825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf0005040 con 0x7efdfc102e10 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.177+0000 7efdfb7fe700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 0x7efdfc113b10 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009360 tx=0x7efdf0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.177+0000 7efde3fff700 1 -- 172.21.15.32:0/3904686825 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdf000e040 con 0x7efdfc102e10 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.177+0000 7efde3fff700 1 -- 172.21.15.32:0/3904686825 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdf0013420 con 0x7efdfc102e10 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.177+0000 7efde3fff700 1 -- 172.21.15.32:0/3904686825 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdf001c5d0 con 0x7efdfc102e10 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.177+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc10dd10 con 0x7efdfc102e10 2024-04-06T14:54:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.177+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc10e2d0 con 0x7efdfc102e10 2024-04-06T14:54:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.178+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdfc0fe560 con 0x7efdfc102e10 2024-04-06T14:54:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.178+0000 7efde3fff700 1 -- 172.21.15.32:0/3904686825 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efdf0013590 con 0x7efdfc102e10 2024-04-06T14:54:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.179+0000 7efde3fff700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efde406cc30 0x7efde406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.179+0000 7efde3fff700 1 -- 172.21.15.32:0/3904686825 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efdf008d180 con 0x7efdfc102e10 2024-04-06T14:54:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.179+0000 7efdfaffd700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efde406cc30 0x7efde406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:47.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.179+0000 7efdfaffd700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efde406cc30 0x7efde406f0e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7efdfc10ec50 tx=0x7efdec00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:47.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.183+0000 7efde3fff700 1 -- 172.21.15.32:0/3904686825 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efdf005ac50 con 0x7efdfc102e10 2024-04-06T14:54:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.473+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efdfc04bed0 con 0x7efdfc102e10 2024-04-06T14:54:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.474+0000 7efde3fff700 1 -- 172.21.15.32:0/3904686825 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7efdf005a7e0 con 0x7efdfc102e10 2024-04-06T14:54:47.461 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:54:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:54:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:54:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:54:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:54:47.462 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:54:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efde406cc30 msgr2=0x7efde406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efde406cc30 0x7efde406f0e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7efdfc10ec50 tx=0x7efdec00b040 comp rx=0 tx=0).stop 2024-04-06T14:54:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 msgr2=0x7efdfc113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 0x7efdfc113b10 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009360 tx=0x7efdf0009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 shutdown_connections 2024-04-06T14:54:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7efde406cc30 0x7efde406f0e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdfc102e10 0x7efdfc113b10 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 --2- 172.21.15.32:0/3904686825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdfc1037b0 0x7efdfc114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 >> 172.21.15.32:0/3904686825 conn(0x7efdfc0f8980 msgr2=0x7efdfc0f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 shutdown_connections 2024-04-06T14:54:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.476+0000 7efe02025700 1 -- 172.21.15.32:0/3904686825 wait complete. 2024-04-06T14:54:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.579+0000 7fe66fdc9700 1 -- 172.21.15.32:0/4052656663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fde70 msgr2=0x7fe6680fe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.579+0000 7fe66fdc9700 1 --2- 172.21.15.32:0/4052656663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fde70 0x7fe6680fe240 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe658004940 tx=0x7fe658009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.579+0000 7fe66fdc9700 1 -- 172.21.15.32:0/4052656663 shutdown_connections 2024-04-06T14:54:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.579+0000 7fe66fdc9700 1 --2- 172.21.15.32:0/4052656663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6680fe810 0x7fe668106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.579+0000 7fe66fdc9700 1 --2- 172.21.15.32:0/4052656663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fde70 0x7fe6680fe240 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.579+0000 7fe66fdc9700 1 -- 172.21.15.32:0/4052656663 >> 172.21.15.32:0/4052656663 conn(0x7fe6680f7880 msgr2=0x7fe6680f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.579+0000 7fe66fdc9700 1 -- 172.21.15.32:0/4052656663 shutdown_connections 2024-04-06T14:54:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.579+0000 7fe66fdc9700 1 -- 172.21.15.32:0/4052656663 wait complete. 2024-04-06T14:54:47.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66fdc9700 1 Processor -- start 2024-04-06T14:54:47.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66fdc9700 1 -- start start 2024-04-06T14:54:47.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66fdc9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6680fde70 0x7fe668113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:47.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66fdc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fe810 0x7fe6681142d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:47.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66fdc9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe668114920 con 0x7fe6680fe810 2024-04-06T14:54:47.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66fdc9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66810dcf0 con 0x7fe6680fde70 2024-04-06T14:54:47.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66d364700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fe810 0x7fe6681142d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66d364700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fe810 0x7fe6681142d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37790/0 (socket says 172.21.15.32:37790) 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66d364700 1 -- 172.21.15.32:0/3322416229 learned_addr learned my addr 172.21.15.32:0/3322416229 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66db65700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6680fde70 0x7fe668113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66d364700 1 -- 172.21.15.32:0/3322416229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6680fde70 msgr2=0x7fe668113d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66d364700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6680fde70 0x7fe668113d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.580+0000 7fe66d364700 1 -- 172.21.15.32:0/3322416229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe658005040 con 0x7fe6680fe810 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.581+0000 7fe66db65700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6680fde70 0x7fe668113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.581+0000 7fe66d364700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fe810 0x7fe6681142d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe664002a50 tx=0x7fe6640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.581+0000 7fe65e7fc700 1 -- 172.21.15.32:0/3322416229 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe664007cc0 con 0x7fe6680fe810 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.581+0000 7fe65e7fc700 1 -- 172.21.15.32:0/3322416229 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe66400f040 con 0x7fe6680fe810 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.581+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66810dfd0 con 0x7fe6680fe810 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.581+0000 7fe65e7fc700 1 -- 172.21.15.32:0/3322416229 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe664009540 con 0x7fe6680fe810 2024-04-06T14:54:47.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.581+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66810e510 con 0x7fe6680fe810 2024-04-06T14:54:47.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.582+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe668066710 con 0x7fe6680fe810 2024-04-06T14:54:47.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.582+0000 7fe65e7fc700 1 -- 172.21.15.32:0/3322416229 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe6640096a0 con 0x7fe6680fe810 2024-04-06T14:54:47.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.583+0000 7fe65e7fc700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe65406cc20 0x7fe65406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:47.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.583+0000 7fe65e7fc700 1 -- 172.21.15.32:0/3322416229 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe66408bc60 con 0x7fe6680fe810 2024-04-06T14:54:47.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.583+0000 7fe66db65700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe65406cc20 0x7fe65406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:47.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.583+0000 7fe66db65700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe65406cc20 0x7fe65406f0d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe658000c00 tx=0x7fe658009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:47.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.587+0000 7fe65e7fc700 1 -- 172.21.15.32:0/3322416229 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe6640596b0 con 0x7fe6680fe810 2024-04-06T14:54:47.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.843+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe66804bed0 con 0x7fe6680fe810 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.844+0000 7fe65e7fc700 1 -- 172.21.15.32:0/3322416229 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fe664059240 con 0x7fe6680fe810 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:54:47.831 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:54:47.832 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:54:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:54:47.834 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe65406cc20 msgr2=0x7fe65406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe65406cc20 0x7fe65406f0d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe658000c00 tx=0x7fe658009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fe810 msgr2=0x7fe6681142d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fe810 0x7fe6681142d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe664002a50 tx=0x7fe6640046b0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 shutdown_connections 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe65406cc20 0x7fe65406f0d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6680fde70 0x7fe668113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 --2- 172.21.15.32:0/3322416229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6680fe810 0x7fe6681142d0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 >> 172.21.15.32:0/3322416229 conn(0x7fe6680f7880 msgr2=0x7fe6680f97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 shutdown_connections 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.846+0000 7fe66fdc9700 1 -- 172.21.15.32:0/3322416229 wait complete. 2024-04-06T14:54:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:54:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 -- 172.21.15.32:0/3645793911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 msgr2=0x7f1fc80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 --2- 172.21.15.32:0/3645793911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 0x7f1fc80fe260 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8004940 tx=0x7f1fb8009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 -- 172.21.15.32:0/3645793911 shutdown_connections 2024-04-06T14:54:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 --2- 172.21.15.32:0/3645793911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc80fe830 0x7f1fc8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 --2- 172.21.15.32:0/3645793911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 0x7f1fc80fe260 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 -- 172.21.15.32:0/3645793911 >> 172.21.15.32:0/3645793911 conn(0x7f1fc8072bc0 msgr2=0x7f1fc8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 -- 172.21.15.32:0/3645793911 shutdown_connections 2024-04-06T14:54:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 -- 172.21.15.32:0/3645793911 wait complete. 2024-04-06T14:54:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 Processor -- start 2024-04-06T14:54:47.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.951+0000 7f1fce00b700 1 -- start start 2024-04-06T14:54:47.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fce00b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 0x7f1fc8113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:47.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fce00b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc80fe830 0x7f1fc8114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:47.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fce00b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8114680 con 0x7f1fc80fde90 2024-04-06T14:54:47.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fce00b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc810da70 con 0x7f1fc80fe830 2024-04-06T14:54:47.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fc77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 0x7f1fc8113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fc77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 0x7f1fc8113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37800/0 (socket says 172.21.15.32:37800) 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fc77fe700 1 -- 172.21.15.32:0/1574111354 learned_addr learned my addr 172.21.15.32:0/1574111354 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fc6ffd700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc80fe830 0x7f1fc8114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fc77fe700 1 -- 172.21.15.32:0/1574111354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc80fe830 msgr2=0x7f1fc8114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fc77fe700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc80fe830 0x7f1fc8114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fc77fe700 1 -- 172.21.15.32:0/1574111354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb8005040 con 0x7f1fc80fde90 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fc77fe700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 0x7f1fc8113af0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800b3b0 tx=0x7f1fb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1faffff700 1 -- 172.21.15.32:0/1574111354 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fb800e040 con 0x7f1fc80fde90 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1faffff700 1 -- 172.21.15.32:0/1574111354 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fb8013420 con 0x7f1fc80fde90 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1faffff700 1 -- 172.21.15.32:0/1574111354 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fb801c5f0 con 0x7f1fc80fde90 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc810dcf0 con 0x7f1fc80fde90 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.952+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc810e1e0 con 0x7f1fc80fde90 2024-04-06T14:54:47.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.953+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc81019b0 con 0x7f1fc80fde90 2024-04-06T14:54:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.954+0000 7f1faffff700 1 -- 172.21.15.32:0/1574111354 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1fb8013590 con 0x7f1fc80fde90 2024-04-06T14:54:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.954+0000 7f1faffff700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fb006cc20 0x7f1fb006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.954+0000 7f1faffff700 1 -- 172.21.15.32:0/1574111354 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1fb808d340 con 0x7f1fc80fde90 2024-04-06T14:54:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.954+0000 7f1fc6ffd700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fb006cc20 0x7f1fb006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.955+0000 7f1fc6ffd700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fb006cc20 0x7f1fb006f0d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1fc810ec30 tx=0x7f1fbc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:47.959+0000 7f1faffff700 1 -- 172.21.15.32:0/1574111354 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1fb805ae10 con 0x7f1fc80fde90 2024-04-06T14:54:47.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:47 smithi032 ceph-mon[147414]: pgmap v1040: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:47.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:47 smithi032 ceph-mon[147414]: from='client.35878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:47.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:47 smithi032 ceph-mon[147414]: from='client.35882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:47.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:47 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3904686825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:54:48.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:47 smithi086 ceph-mon[110975]: pgmap v1040: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:48.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:47 smithi086 ceph-mon[110975]: from='client.35878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:48.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:47 smithi086 ceph-mon[110975]: from='client.35882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:48.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:47 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3904686825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:54:48.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.178+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fc810f140 con 0x7f1fb006cc20 2024-04-06T14:54:48.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.180+0000 7f1faffff700 1 -- 172.21.15.32:0/1574111354 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1fc810f140 con 0x7f1fb006cc20 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:54:48.167 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:54:48.168 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:54:48.168 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:54:48.168 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:54:48.168 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fb006cc20 msgr2=0x7f1fb006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fb006cc20 0x7f1fb006f0d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1fc810ec30 tx=0x7f1fbc00b040 comp rx=0 tx=0).stop 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 msgr2=0x7f1fc8113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 0x7f1fc8113af0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800b3b0 tx=0x7f1fb8009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 shutdown_connections 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f1fb006cc20 0x7f1fb006f0d0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fc80fde90 0x7f1fc8113af0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 --2- 172.21.15.32:0/1574111354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc80fe830 0x7f1fc8114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 >> 172.21.15.32:0/1574111354 conn(0x7f1fc8072bc0 msgr2=0x7f1fc80ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 shutdown_connections 2024-04-06T14:54:48.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.182+0000 7f1fce00b700 1 -- 172.21.15.32:0/1574111354 wait complete. 2024-04-06T14:54:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 -- 172.21.15.32:0/1438037959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4820101d10 msgr2=0x7f48201020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 --2- 172.21.15.32:0/1438037959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4820101d10 0x7f48201020e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4810004940 tx=0x7f4810009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 -- 172.21.15.32:0/1438037959 shutdown_connections 2024-04-06T14:54:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 --2- 172.21.15.32:0/1438037959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48201026b0 0x7f48200fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 --2- 172.21.15.32:0/1438037959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4820101d10 0x7f48201020e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 -- 172.21.15.32:0/1438037959 >> 172.21.15.32:0/1438037959 conn(0x7f48200f7880 msgr2=0x7f48200f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 -- 172.21.15.32:0/1438037959 shutdown_connections 2024-04-06T14:54:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 -- 172.21.15.32:0/1438037959 wait complete. 2024-04-06T14:54:48.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 Processor -- start 2024-04-06T14:54:48.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.285+0000 7f4827f2e700 1 -- start start 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f4827f2e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4820101d10 0x7f4820112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f4827f2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48201026b0 0x7f4820113260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f4827f2e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48201138b0 con 0x7f48201026b0 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f4827f2e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482010cc80 con 0x7f4820101d10 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48254c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48201026b0 0x7f4820113260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48254c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48201026b0 0x7f4820113260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37810/0 (socket says 172.21.15.32:37810) 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48254c9700 1 -- 172.21.15.32:0/3972468213 learned_addr learned my addr 172.21.15.32:0/3972468213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f4825cca700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4820101d10 0x7f4820112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48254c9700 1 -- 172.21.15.32:0/3972468213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4820101d10 msgr2=0x7f4820112d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48254c9700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4820101d10 0x7f4820112d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48254c9700 1 -- 172.21.15.32:0/3972468213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4810005040 con 0x7f48201026b0 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f4825cca700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4820101d10 0x7f4820112d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48254c9700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48201026b0 0x7f4820113260 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f481c00fce0 tx=0x7f481c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48167fc700 1 -- 172.21.15.32:0/3972468213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f481c00fe90 con 0x7f48201026b0 2024-04-06T14:54:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482010cf00 con 0x7f48201026b0 2024-04-06T14:54:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48167fc700 1 -- 172.21.15.32:0/3972468213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f481c0090d0 con 0x7f48201026b0 2024-04-06T14:54:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f48167fc700 1 -- 172.21.15.32:0/3972468213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f481c0126a0 con 0x7f48201026b0 2024-04-06T14:54:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.286+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482010d4c0 con 0x7f48201026b0 2024-04-06T14:54:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.287+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48200fd460 con 0x7f48201026b0 2024-04-06T14:54:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.288+0000 7f48167fc700 1 -- 172.21.15.32:0/3972468213 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f481c009690 con 0x7f48201026b0 2024-04-06T14:54:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.288+0000 7f48167fc700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f480c06cc30 0x7f480c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:54:48.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.288+0000 7f48167fc700 1 -- 172.21.15.32:0/3972468213 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f481c08e850 con 0x7f48201026b0 2024-04-06T14:54:48.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.288+0000 7f4825cca700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f480c06cc30 0x7f480c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:54:48.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.289+0000 7f4825cca700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f480c06cc30 0x7f480c06f0e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f481000b840 tx=0x7f4810009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:54:48.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.293+0000 7f48167fc700 1 -- 172.21.15.32:0/3972468213 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f481c05c2a0 con 0x7f48201026b0 2024-04-06T14:54:48.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.580+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f482004bed0 con 0x7f48201026b0 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.580+0000 7f48167fc700 1 -- 172.21.15.32:0/3972468213 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f481c05be30 con 0x7f48201026b0 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:54:48.568 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:54:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:54:48.570 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:54:48.570 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:54:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f480c06cc30 msgr2=0x7f480c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f480c06cc30 0x7f480c06f0e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f481000b840 tx=0x7f4810009e30 comp rx=0 tx=0).stop 2024-04-06T14:54:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48201026b0 msgr2=0x7f4820113260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:54:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48201026b0 0x7f4820113260 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f481c00fce0 tx=0x7f481c004600 comp rx=0 tx=0).stop 2024-04-06T14:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 shutdown_connections 2024-04-06T14:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f480c06cc30 0x7f480c06f0e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4820101d10 0x7f4820112d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 --2- 172.21.15.32:0/3972468213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48201026b0 0x7f4820113260 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:54:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 >> 172.21.15.32:0/3972468213 conn(0x7f48200f7880 msgr2=0x7f48200f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:54:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 shutdown_connections 2024-04-06T14:54:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:54:48.583+0000 7f4827f2e700 1 -- 172.21.15.32:0/3972468213 wait complete. 2024-04-06T14:54:48.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:48 smithi032 ceph-mon[147414]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:48.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:48 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3322416229' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:54:48.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:48 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3972468213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:54:49.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:48 smithi086 ceph-mon[110975]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:49.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:48 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3322416229' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:54:49.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:48 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3972468213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:54:49.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:49 smithi032 ceph-mon[147414]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:49.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:49 smithi032 ceph-mon[147414]: pgmap v1041: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:50.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:49 smithi086 ceph-mon[110975]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:54:50.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:49 smithi086 ceph-mon[110975]: pgmap v1041: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:51.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:51 smithi032 ceph-mon[147414]: pgmap v1042: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:52.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:51 smithi086 ceph-mon[110975]: pgmap v1042: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:52.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:54:53.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:54:53.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:53 smithi032 ceph-mon[147414]: pgmap v1043: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:54.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:53 smithi086 ceph-mon[110975]: pgmap v1043: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:55.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:55 smithi032 ceph-mon[147414]: pgmap v1044: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:56.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:55 smithi086 ceph-mon[110975]: pgmap v1044: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:57.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:57 smithi032 ceph-mon[147414]: pgmap v1045: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:58.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:57 smithi086 ceph-mon[110975]: pgmap v1045: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:54:59.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:54:59 smithi032 ceph-mon[147414]: pgmap v1046: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:00.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:54:59 smithi086 ceph-mon[110975]: pgmap v1046: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:01.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:01 smithi032 ceph-mon[147414]: pgmap v1047: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:02.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:01 smithi086 ceph-mon[110975]: pgmap v1047: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:03.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:03 smithi032 ceph-mon[147414]: pgmap v1048: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:04.605 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:03 smithi086 ceph-mon[110975]: pgmap v1048: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:05.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:05 smithi032 ceph-mon[147414]: pgmap v1049: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:06.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:05 smithi086 ceph-mon[110975]: pgmap v1049: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:07.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:07 smithi032 ceph-mon[147414]: pgmap v1050: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:07.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:55:08.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:07 smithi086 ceph-mon[110975]: pgmap v1050: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:08.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:55:09.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:09 smithi032 ceph-mon[147414]: pgmap v1051: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:10.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:09 smithi086 ceph-mon[110975]: pgmap v1051: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:11.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:11 smithi032 ceph-mon[147414]: pgmap v1052: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:12.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:11 smithi086 ceph-mon[110975]: pgmap v1052: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:13.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:13 smithi032 ceph-mon[147414]: pgmap v1053: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:14.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:13 smithi086 ceph-mon[110975]: pgmap v1053: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:15 smithi032 ceph-mon[147414]: pgmap v1054: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:16.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:15 smithi086 ceph-mon[110975]: pgmap v1054: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:17.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:17 smithi032 ceph-mon[147414]: pgmap v1055: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:18.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:17 smithi086 ceph-mon[110975]: pgmap v1055: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 -- 172.21.15.32:0/3735563926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fdeb0 msgr2=0x7f178c0fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 --2- 172.21.15.32:0/3735563926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fdeb0 0x7f178c0fe280 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1788004970 tx=0x7f1788009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 -- 172.21.15.32:0/3735563926 shutdown_connections 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 --2- 172.21.15.32:0/3735563926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f178c0fe850 0x7f178c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 --2- 172.21.15.32:0/3735563926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fdeb0 0x7f178c0fe280 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 -- 172.21.15.32:0/3735563926 >> 172.21.15.32:0/3735563926 conn(0x7f178c072bc0 msgr2=0x7f178c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 -- 172.21.15.32:0/3735563926 shutdown_connections 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 -- 172.21.15.32:0/3735563926 wait complete. 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 Processor -- start 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.701+0000 7f1794531700 1 -- start start 2024-04-06T14:55:18.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1794531700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f178c0fdeb0 0x7f178c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1794531700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fe850 0x7f178c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1794531700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c116800 con 0x7f178c0fe850 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1794531700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c10fc10 con 0x7f178c0fdeb0 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1791acc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fe850 0x7f178c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1791acc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fe850 0x7f178c1161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51240/0 (socket says 172.21.15.32:51240) 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1791acc700 1 -- 172.21.15.32:0/1957947256 learned_addr learned my addr 172.21.15.32:0/1957947256 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f17922cd700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f178c0fdeb0 0x7f178c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1791acc700 1 -- 172.21.15.32:0/1957947256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f178c0fdeb0 msgr2=0x7f178c115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1791acc700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f178c0fdeb0 0x7f178c115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1791acc700 1 -- 172.21.15.32:0/1957947256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1788005040 con 0x7f178c0fe850 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1791acc700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fe850 0x7f178c1161b0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f177c002a80 tx=0x7f177c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1782ffd700 1 -- 172.21.15.32:0/1957947256 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f177c007cc0 con 0x7f178c0fe850 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1782ffd700 1 -- 172.21.15.32:0/1957947256 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f177c00f040 con 0x7f178c0fe850 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c10fef0 con 0x7f178c0fe850 2024-04-06T14:55:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.702+0000 7f1782ffd700 1 -- 172.21.15.32:0/1957947256 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f177c00da60 con 0x7f178c0fe850 2024-04-06T14:55:18.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.703+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c110430 con 0x7f178c0fe850 2024-04-06T14:55:18.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.703+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c1069f0 con 0x7f178c0fe850 2024-04-06T14:55:18.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.704+0000 7f1782ffd700 1 -- 172.21.15.32:0/1957947256 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f177c00dbc0 con 0x7f178c0fe850 2024-04-06T14:55:18.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.704+0000 7f1782ffd700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f177806cc20 0x7f177806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:18.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.704+0000 7f1782ffd700 1 -- 172.21.15.32:0/1957947256 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f177c08bcb0 con 0x7f178c0fe850 2024-04-06T14:55:18.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.705+0000 7f17922cd700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f177806cc20 0x7f177806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:18.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.705+0000 7f17922cd700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f177806cc20 0x7f177806f0d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1788004f50 tx=0x7f1788009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:18.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.709+0000 7f1782ffd700 1 -- 172.21.15.32:0/1957947256 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f177c055c70 con 0x7f178c0fe850 2024-04-06T14:55:18.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.930+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f178c111300 con 0x7f177806cc20 2024-04-06T14:55:18.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.931+0000 7f1782ffd700 1 -- 172.21.15.32:0/1957947256 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f178c111300 con 0x7f177806cc20 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f177806cc20 msgr2=0x7f177806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f177806cc20 0x7f177806f0d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1788004f50 tx=0x7f1788009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fe850 msgr2=0x7f178c1161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fe850 0x7f178c1161b0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f177c002a80 tx=0x7f177c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 shutdown_connections 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f177806cc20 0x7f177806f0d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f178c0fdeb0 0x7f178c115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 --2- 172.21.15.32:0/1957947256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178c0fe850 0x7f178c1161b0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 >> 172.21.15.32:0/1957947256 conn(0x7f178c072bc0 msgr2=0x7f178c0fb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 shutdown_connections 2024-04-06T14:55:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:18.933+0000 7f1794531700 1 -- 172.21.15.32:0/1957947256 wait complete. 2024-04-06T14:55:18.930 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:55:19.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.037+0000 7f546415a700 1 -- 172.21.15.32:0/81609925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c101d70 msgr2=0x7f545c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.037+0000 7f546415a700 1 --2- 172.21.15.32:0/81609925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c101d70 0x7f545c102140 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5458004940 tx=0x7f5458009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:19.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.037+0000 7f546415a700 1 -- 172.21.15.32:0/81609925 shutdown_connections 2024-04-06T14:55:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.037+0000 7f546415a700 1 --2- 172.21.15.32:0/81609925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f545c102710 0x7f545c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.037+0000 7f546415a700 1 --2- 172.21.15.32:0/81609925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c101d70 0x7f545c102140 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.037+0000 7f546415a700 1 -- 172.21.15.32:0/81609925 >> 172.21.15.32:0/81609925 conn(0x7f545c072bc0 msgr2=0x7f545c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.037+0000 7f546415a700 1 -- 172.21.15.32:0/81609925 shutdown_connections 2024-04-06T14:55:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.037+0000 7f546415a700 1 -- 172.21.15.32:0/81609925 wait complete. 2024-04-06T14:55:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f546415a700 1 Processor -- start 2024-04-06T14:55:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f546415a700 1 -- start start 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f546415a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f545c101d70 0x7f545c102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f546415a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c102710 0x7f545c103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f546415a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c1036a0 con 0x7f545c102710 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f546415a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c1037e0 con 0x7f545c101d70 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f54616f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c102710 0x7f545c103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f54616f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c102710 0x7f545c103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51264/0 (socket says 172.21.15.32:51264) 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f54616f5700 1 -- 172.21.15.32:0/684296321 learned_addr learned my addr 172.21.15.32:0/684296321 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.038+0000 7f5461ef6700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f545c101d70 0x7f545c102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.039+0000 7f54616f5700 1 -- 172.21.15.32:0/684296321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f545c101d70 msgr2=0x7f545c102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.039+0000 7f54616f5700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f545c101d70 0x7f545c102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.039+0000 7f54616f5700 1 -- 172.21.15.32:0/684296321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5458005040 con 0x7f545c102710 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.039+0000 7f54616f5700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c102710 0x7f545c103110 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f544c002a50 tx=0x7f544c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.039+0000 7f54527fc700 1 -- 172.21.15.32:0/684296321 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f544c007cc0 con 0x7f545c102710 2024-04-06T14:55:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.039+0000 7f54527fc700 1 -- 172.21.15.32:0/684296321 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f544c00f040 con 0x7f545c102710 2024-04-06T14:55:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.039+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545c119d10 con 0x7f545c102710 2024-04-06T14:55:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.039+0000 7f54527fc700 1 -- 172.21.15.32:0/684296321 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f544c00da30 con 0x7f545c102710 2024-04-06T14:55:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.039+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545c11a190 con 0x7f545c102710 2024-04-06T14:55:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.040+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545c0fd4c0 con 0x7f545c102710 2024-04-06T14:55:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.040+0000 7f54527fc700 1 -- 172.21.15.32:0/684296321 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f544c00db90 con 0x7f545c102710 2024-04-06T14:55:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.041+0000 7f54527fc700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f544806cc30 0x7f544806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.041+0000 7f54527fc700 1 -- 172.21.15.32:0/684296321 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f544c08bc80 con 0x7f545c102710 2024-04-06T14:55:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.041+0000 7f5461ef6700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f544806cc30 0x7f544806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.041+0000 7f5461ef6700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f544806cc30 0x7f544806f0e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5458009fd0 tx=0x7f5458009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.045+0000 7f54527fc700 1 -- 172.21.15.32:0/684296321 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f544c055c40 con 0x7f545c102710 2024-04-06T14:55:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.266+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f545c04bed0 con 0x7f544806cc30 2024-04-06T14:55:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.267+0000 7f54527fc700 1 -- 172.21.15.32:0/684296321 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f545c04bed0 con 0x7f544806cc30 2024-04-06T14:55:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f544806cc30 msgr2=0x7f544806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f544806cc30 0x7f544806f0e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5458009fd0 tx=0x7f5458009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c102710 msgr2=0x7f545c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c102710 0x7f545c103110 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f544c002a50 tx=0x7f544c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 shutdown_connections 2024-04-06T14:55:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f544806cc30 0x7f544806f0e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f545c101d70 0x7f545c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 --2- 172.21.15.32:0/684296321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f545c102710 0x7f545c103110 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 >> 172.21.15.32:0/684296321 conn(0x7f545c072bc0 msgr2=0x7f545c065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 shutdown_connections 2024-04-06T14:55:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.269+0000 7f546415a700 1 -- 172.21.15.32:0/684296321 wait complete. 2024-04-06T14:55:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.372+0000 7fbfd500d700 1 -- 172.21.15.32:0/1178346641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 msgr2=0x7fbfd00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.372+0000 7fbfd500d700 1 --2- 172.21.15.32:0/1178346641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 0x7fbfd00fbd80 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8004940 tx=0x7fbfb8009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.372+0000 7fbfd500d700 1 -- 172.21.15.32:0/1178346641 shutdown_connections 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.372+0000 7fbfd500d700 1 --2- 172.21.15.32:0/1178346641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 0x7fbfd00fbd80 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.372+0000 7fbfd500d700 1 --2- 172.21.15.32:0/1178346641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfd0101d70 0x7fbfd0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.372+0000 7fbfd500d700 1 -- 172.21.15.32:0/1178346641 >> 172.21.15.32:0/1178346641 conn(0x7fbfd0072bc0 msgr2=0x7fbfd0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.372+0000 7fbfd500d700 1 -- 172.21.15.32:0/1178346641 shutdown_connections 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.372+0000 7fbfd500d700 1 -- 172.21.15.32:0/1178346641 wait complete. 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfd500d700 1 Processor -- start 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfd500d700 1 -- start start 2024-04-06T14:55:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfd500d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfd0101d70 0x7fbfd0113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfd500d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 0x7fbfd0113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfd500d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0114590 con 0x7fbfd0102710 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfd500d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd010d980 con 0x7fbfd0101d70 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfce59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 0x7fbfd0113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfce59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 0x7fbfd0113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51276/0 (socket says 172.21.15.32:51276) 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfce59c700 1 -- 172.21.15.32:0/229162061 learned_addr learned my addr 172.21.15.32:0/229162061 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.373+0000 7fbfced9d700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfd0101d70 0x7fbfd0113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.374+0000 7fbfce59c700 1 -- 172.21.15.32:0/229162061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfd0101d70 msgr2=0x7fbfd0113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.374+0000 7fbfce59c700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfd0101d70 0x7fbfd0113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.374+0000 7fbfce59c700 1 -- 172.21.15.32:0/229162061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb8005040 con 0x7fbfd0102710 2024-04-06T14:55:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.374+0000 7fbfce59c700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 0x7fbfd0113f40 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8000c00 tx=0x7fbfb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.374+0000 7fbfc77fe700 1 -- 172.21.15.32:0/229162061 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfb800e070 con 0x7fbfd0102710 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.374+0000 7fbfc77fe700 1 -- 172.21.15.32:0/229162061 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfb80093b0 con 0x7fbfd0102710 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.374+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd010dc00 con 0x7fbfd0102710 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.374+0000 7fbfc77fe700 1 -- 172.21.15.32:0/229162061 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfb801ccb0 con 0x7fbfd0102710 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.374+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd010e160 con 0x7fbfd0102710 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.375+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd00fd4c0 con 0x7fbfd0102710 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.375+0000 7fbfc77fe700 1 -- 172.21.15.32:0/229162061 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbfb8013930 con 0x7fbfd0102710 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.376+0000 7fbfc77fe700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbfbc06cc20 0x7fbfbc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.376+0000 7fbfc77fe700 1 -- 172.21.15.32:0/229162061 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbfb808d3d0 con 0x7fbfd0102710 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.376+0000 7fbfced9d700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbfbc06cc20 0x7fbfbc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.376+0000 7fbfced9d700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbfbc06cc20 0x7fbfbc06f0d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009b80 tx=0x7fbfc000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.381+0000 7fbfc77fe700 1 -- 172.21.15.32:0/229162061 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbfb805af60 con 0x7fbfd0102710 2024-04-06T14:55:19.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.593+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfd0108d10 con 0x7fbfbc06cc20 2024-04-06T14:55:19.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.603+0000 7fbfc77fe700 1 -- 172.21.15.32:0/229162061 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbfd0108d10 con 0x7fbfbc06cc20 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (35m) 13m ago 46m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (33m) 13m ago 47m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (32m) 13m ago 46m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (35m) 13m ago 46m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (29m) 13m ago 41m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (29m) 13m ago 41m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (28m) 13m ago 41m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (13m) 13m ago 41m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (37m) 13m ago 48m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:55:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (36m) 13m ago 46m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (34m) 13m ago 48m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (33m) 13m ago 45m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (36m) 13m ago 47m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (36m) 13m ago 46m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (32m) 13m ago 45m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (31m) 13m ago 44m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (31m) 13m ago 44m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (30m) 13m ago 43m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (30m) 13m ago 43m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (29m) 13m ago 42m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:55:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (36m) 13m ago 46m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:55:19.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbfbc06cc20 msgr2=0x7fbfbc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbfbc06cc20 0x7fbfbc06f0d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009b80 tx=0x7fbfc000b040 comp rx=0 tx=0).stop 2024-04-06T14:55:19.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 msgr2=0x7fbfd0113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 0x7fbfd0113f40 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8000c00 tx=0x7fbfb8009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:19.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 shutdown_connections 2024-04-06T14:55:19.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbfbc06cc20 0x7fbfbc06f0d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfd0101d70 0x7fbfd0113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 --2- 172.21.15.32:0/229162061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfd0102710 0x7fbfd0113f40 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 >> 172.21.15.32:0/229162061 conn(0x7fbfd0072bc0 msgr2=0x7fbfd0065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 shutdown_connections 2024-04-06T14:55:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.605+0000 7fbfd500d700 1 -- 172.21.15.32:0/229162061 wait complete. 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.707+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/2074379329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 msgr2=0x7fbe481031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.707+0000 7fbe4cb5a700 1 --2- 172.21.15.32:0/2074379329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 0x7fbe481031e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fbe38004940 tx=0x7fbe38009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.707+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/2074379329 shutdown_connections 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.707+0000 7fbe4cb5a700 1 --2- 172.21.15.32:0/2074379329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481037b0 0x7fbe480fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.707+0000 7fbe4cb5a700 1 --2- 172.21.15.32:0/2074379329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 0x7fbe481031e0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.707+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/2074379329 >> 172.21.15.32:0/2074379329 conn(0x7fbe480f8980 msgr2=0x7fbe480fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.707+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/2074379329 shutdown_connections 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/2074379329 wait complete. 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4cb5a700 1 Processor -- start 2024-04-06T14:55:19.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4cb5a700 1 -- start start 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4cb5a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 0x7fbe48115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4cb5a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481037b0 0x7fbe48116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4cb5a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48116850 con 0x7fbe48102e10 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4cb5a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4810fc20 con 0x7fbe481037b0 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 0x7fbe48115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 0x7fbe48115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51298/0 (socket says 172.21.15.32:51298) 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe4659c700 1 -- 172.21.15.32:0/3720625348 learned_addr learned my addr 172.21.15.32:0/3720625348 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.708+0000 7fbe45d9b700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481037b0 0x7fbe48116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe4659c700 1 -- 172.21.15.32:0/3720625348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481037b0 msgr2=0x7fbe48116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe4659c700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481037b0 0x7fbe48116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe4659c700 1 -- 172.21.15.32:0/3720625348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe38005040 con 0x7fbe48102e10 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe45d9b700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481037b0 0x7fbe48116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe4659c700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 0x7fbe48115cc0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800b3b0 tx=0x7fbe38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:19.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe3effd700 1 -- 172.21.15.32:0/3720625348 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe3800e040 con 0x7fbe48102e10 2024-04-06T14:55:19.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe3effd700 1 -- 172.21.15.32:0/3720625348 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe38013420 con 0x7fbe48102e10 2024-04-06T14:55:19.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4810fea0 con 0x7fbe48102e10 2024-04-06T14:55:19.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe3effd700 1 -- 172.21.15.32:0/3720625348 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe3801c630 con 0x7fbe48102e10 2024-04-06T14:55:19.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.709+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe48110460 con 0x7fbe48102e10 2024-04-06T14:55:19.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.710+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe480fe560 con 0x7fbe48102e10 2024-04-06T14:55:19.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.711+0000 7fbe3effd700 1 -- 172.21.15.32:0/3720625348 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbe38013970 con 0x7fbe48102e10 2024-04-06T14:55:19.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.711+0000 7fbe3effd700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbe3406cc20 0x7fbe3406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:19.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.711+0000 7fbe3effd700 1 -- 172.21.15.32:0/3720625348 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbe3808d290 con 0x7fbe48102e10 2024-04-06T14:55:19.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.711+0000 7fbe45d9b700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbe3406cc20 0x7fbe3406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:19.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.711+0000 7fbe45d9b700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbe3406cc20 0x7fbe3406f0d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009fd0 tx=0x7fbe30009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:19.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:19.716+0000 7fbe3effd700 1 -- 172.21.15.32:0/3720625348 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbe3805ad60 con 0x7fbe48102e10 2024-04-06T14:55:19.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:19 smithi032 ceph-mon[147414]: pgmap v1056: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.009+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbe4804bed0 con 0x7fbe48102e10 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.010+0000 7fbe3effd700 1 -- 172.21.15.32:0/3720625348 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fbe3805a8f0 con 0x7fbe48102e10 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:55:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:55:19.998 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:55:19.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:55:19.998 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:55:19.998 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:55:19.998 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:55:19.998 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:55:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbe3406cc20 msgr2=0x7fbe3406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbe3406cc20 0x7fbe3406f0d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009fd0 tx=0x7fbe30009ee0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 msgr2=0x7fbe48115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 0x7fbe48115cc0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800b3b0 tx=0x7fbe38009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 shutdown_connections 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbe3406cc20 0x7fbe3406f0d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe48102e10 0x7fbe48115cc0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 --2- 172.21.15.32:0/3720625348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481037b0 0x7fbe48116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 >> 172.21.15.32:0/3720625348 conn(0x7fbe480f8980 msgr2=0x7fbe480f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 shutdown_connections 2024-04-06T14:55:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.012+0000 7fbe4cb5a700 1 -- 172.21.15.32:0/3720625348 wait complete. 2024-04-06T14:55:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.115+0000 7fee31819700 1 -- 172.21.15.32:0/121460996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c102680 msgr2=0x7fee2c0fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.115+0000 7fee31819700 1 --2- 172.21.15.32:0/121460996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c102680 0x7fee2c0fbcf0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fee20004940 tx=0x7fee20009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.116+0000 7fee31819700 1 -- 172.21.15.32:0/121460996 shutdown_connections 2024-04-06T14:55:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.116+0000 7fee31819700 1 --2- 172.21.15.32:0/121460996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c102680 0x7fee2c0fbcf0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.116+0000 7fee31819700 1 --2- 172.21.15.32:0/121460996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee2c101ce0 0x7fee2c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.116+0000 7fee31819700 1 -- 172.21.15.32:0/121460996 >> 172.21.15.32:0/121460996 conn(0x7fee2c0f7870 msgr2=0x7fee2c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.116+0000 7fee31819700 1 -- 172.21.15.32:0/121460996 shutdown_connections 2024-04-06T14:55:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.116+0000 7fee31819700 1 -- 172.21.15.32:0/121460996 wait complete. 2024-04-06T14:55:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.116+0000 7fee31819700 1 Processor -- start 2024-04-06T14:55:20.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.116+0000 7fee31819700 1 -- start start 2024-04-06T14:55:20.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.116+0000 7fee31819700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c101ce0 0x7fee2c102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:20.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee31819700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee2c102680 0x7fee2c103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:20.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee31819700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c104a90 con 0x7fee2c101ce0 2024-04-06T14:55:20.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee31819700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c1035c0 con 0x7fee2c102680 2024-04-06T14:55:20.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee2affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c101ce0 0x7fee2c102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:20.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee2affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c101ce0 0x7fee2c102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51328/0 (socket says 172.21.15.32:51328) 2024-04-06T14:55:20.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee2affd700 1 -- 172.21.15.32:0/1085550836 learned_addr learned my addr 172.21.15.32:0/1085550836 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee2a7fc700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee2c102680 0x7fee2c103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee2affd700 1 -- 172.21.15.32:0/1085550836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee2c102680 msgr2=0x7fee2c103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee2affd700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee2c102680 0x7fee2c103080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee2affd700 1 -- 172.21.15.32:0/1085550836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20005040 con 0x7fee2c101ce0 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee2affd700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c101ce0 0x7fee2c102b40 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fee1c002a80 tx=0x7fee1c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee137fe700 1 -- 172.21.15.32:0/1085550836 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee1c007cc0 con 0x7fee2c101ce0 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c1038a0 con 0x7fee2c101ce0 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee137fe700 1 -- 172.21.15.32:0/1085550836 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee1c00f040 con 0x7fee2c101ce0 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee137fe700 1 -- 172.21.15.32:0/1085550836 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee1c00da60 con 0x7fee2c101ce0 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.117+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c119fc0 con 0x7fee2c101ce0 2024-04-06T14:55:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.118+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c0fd430 con 0x7fee2c101ce0 2024-04-06T14:55:20.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.119+0000 7fee137fe700 1 -- 172.21.15.32:0/1085550836 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fee1c00dbc0 con 0x7fee2c101ce0 2024-04-06T14:55:20.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.119+0000 7fee137fe700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fee1406cc20 0x7fee1406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:20.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.119+0000 7fee137fe700 1 -- 172.21.15.32:0/1085550836 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fee1c08bd70 con 0x7fee2c101ce0 2024-04-06T14:55:20.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.119+0000 7fee2a7fc700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fee1406cc20 0x7fee1406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:20.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.120+0000 7fee2a7fc700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fee1406cc20 0x7fee1406f0d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fee2c104270 tx=0x7fee20009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:20.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.124+0000 7fee137fe700 1 -- 172.21.15.32:0/1085550836 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fee1c055cb0 con 0x7fee2c101ce0 2024-04-06T14:55:20.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:19 smithi086 ceph-mon[110975]: pgmap v1056: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.369+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fee2c11a1b0 con 0x7fee2c101ce0 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.370+0000 7fee137fe700 1 -- 172.21.15.32:0/1085550836 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fee1c0592d0 con 0x7fee2c101ce0 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:55:20.357 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:55:20.358 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:55:20.359 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:55:20.360 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:55:20.360 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:55:20.360 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:55:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fee1406cc20 msgr2=0x7fee1406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fee1406cc20 0x7fee1406f0d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fee2c104270 tx=0x7fee20009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c101ce0 msgr2=0x7fee2c102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c101ce0 0x7fee2c102b40 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fee1c002a80 tx=0x7fee1c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 shutdown_connections 2024-04-06T14:55:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fee1406cc20 0x7fee1406f0d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c101ce0 0x7fee2c102b40 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 --2- 172.21.15.32:0/1085550836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee2c102680 0x7fee2c103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 >> 172.21.15.32:0/1085550836 conn(0x7fee2c0f7870 msgr2=0x7fee2c066320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 shutdown_connections 2024-04-06T14:55:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.372+0000 7fee31819700 1 -- 172.21.15.32:0/1085550836 wait complete. 2024-04-06T14:55:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:55:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.476+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3934649394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 msgr2=0x7fc0a40fbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.476+0000 7fc0a88d4700 1 --2- 172.21.15.32:0/3934649394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 0x7fc0a40fbd10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc08c004940 tx=0x7fc08c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.476+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3934649394 shutdown_connections 2024-04-06T14:55:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.476+0000 7fc0a88d4700 1 --2- 172.21.15.32:0/3934649394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 0x7fc0a40fbd10 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.476+0000 7fc0a88d4700 1 --2- 172.21.15.32:0/3934649394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a4101d00 0x7fc0a41020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.476+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3934649394 >> 172.21.15.32:0/3934649394 conn(0x7fc0a40f78d0 msgr2=0x7fc0a40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:20.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.476+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3934649394 shutdown_connections 2024-04-06T14:55:20.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.476+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3934649394 wait complete. 2024-04-06T14:55:20.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.477+0000 7fc0a88d4700 1 Processor -- start 2024-04-06T14:55:20.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.477+0000 7fc0a88d4700 1 -- start start 2024-04-06T14:55:20.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.477+0000 7fc0a88d4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a4101d00 0x7fc0a4112a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:20.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.477+0000 7fc0a88d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 0x7fc0a4112f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:20.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.477+0000 7fc0a88d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a4113590 con 0x7fc0a41026a0 2024-04-06T14:55:20.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.477+0000 7fc0a88d4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a410c980 con 0x7fc0a4101d00 2024-04-06T14:55:20.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.477+0000 7fc0a1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 0x7fc0a4112f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.477+0000 7fc0a1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 0x7fc0a4112f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51348/0 (socket says 172.21.15.32:51348) 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.477+0000 7fc0a1d9b700 1 -- 172.21.15.32:0/3941356611 learned_addr learned my addr 172.21.15.32:0/3941356611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc0a259c700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a4101d00 0x7fc0a4112a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc0a1d9b700 1 -- 172.21.15.32:0/3941356611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a4101d00 msgr2=0x7fc0a4112a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc0a1d9b700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a4101d00 0x7fc0a4112a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc0a1d9b700 1 -- 172.21.15.32:0/3941356611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c005040 con 0x7fc0a41026a0 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc0a1d9b700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 0x7fc0a4112f40 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc08c004f90 tx=0x7fc08c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc09affd700 1 -- 172.21.15.32:0/3941356611 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc08c00e040 con 0x7fc0a41026a0 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc09affd700 1 -- 172.21.15.32:0/3941356611 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc08c00bc90 con 0x7fc0a41026a0 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a410cc00 con 0x7fc0a41026a0 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc09affd700 1 -- 172.21.15.32:0/3941356611 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc08c01a630 con 0x7fc0a41026a0 2024-04-06T14:55:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.478+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a410d1c0 con 0x7fc0a41026a0 2024-04-06T14:55:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.479+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a40fd450 con 0x7fc0a41026a0 2024-04-06T14:55:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.480+0000 7fc09affd700 1 -- 172.21.15.32:0/3941356611 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc08c020020 con 0x7fc0a41026a0 2024-04-06T14:55:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.480+0000 7fc09affd700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc09006cc20 0x7fc09006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.480+0000 7fc09affd700 1 -- 172.21.15.32:0/3941356611 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc08c08eee0 con 0x7fc0a41026a0 2024-04-06T14:55:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.480+0000 7fc0a259c700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc09006cc20 0x7fc09006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.480+0000 7fc0a259c700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc09006cc20 0x7fc09006f0d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc094007fd0 tx=0x7fc094009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:20.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.485+0000 7fc09affd700 1 -- 172.21.15.32:0/3941356611 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc08c05c9b0 con 0x7fc0a41026a0 2024-04-06T14:55:20.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.705+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0a4108ca0 con 0x7fc09006cc20 2024-04-06T14:55:20.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.706+0000 7fc09affd700 1 -- 172.21.15.32:0/3941356611 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc0a4108ca0 con 0x7fc09006cc20 2024-04-06T14:55:20.693 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:55:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:55:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:55:20.693 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:55:20.694 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:55:20.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.708+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc09006cc20 msgr2=0x7fc09006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.708+0000 7fc0a88d4700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc09006cc20 0x7fc09006f0d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc094007fd0 tx=0x7fc094009040 comp rx=0 tx=0).stop 2024-04-06T14:55:20.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.708+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 msgr2=0x7fc0a4112f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.708+0000 7fc0a88d4700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 0x7fc0a4112f40 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc08c004f90 tx=0x7fc08c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:20.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.708+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 shutdown_connections 2024-04-06T14:55:20.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.708+0000 7fc0a88d4700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc09006cc20 0x7fc09006f0d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.708+0000 7fc0a88d4700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a4101d00 0x7fc0a4112a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.708+0000 7fc0a88d4700 1 --2- 172.21.15.32:0/3941356611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a41026a0 0x7fc0a4112f40 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.709+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 >> 172.21.15.32:0/3941356611 conn(0x7fc0a40f78d0 msgr2=0x7fc0a40f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:20.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.709+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 shutdown_connections 2024-04-06T14:55:20.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.709+0000 7fc0a88d4700 1 -- 172.21.15.32:0/3941356611 wait complete. 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.812+0000 7ff989ad9700 1 -- 172.21.15.32:0/740423380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9841037b0 msgr2=0x7ff9840fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.812+0000 7ff989ad9700 1 --2- 172.21.15.32:0/740423380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9841037b0 0x7ff9840fce20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff96c004940 tx=0x7ff96c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.812+0000 7ff989ad9700 1 -- 172.21.15.32:0/740423380 shutdown_connections 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.812+0000 7ff989ad9700 1 --2- 172.21.15.32:0/740423380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9841037b0 0x7ff9840fce20 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.812+0000 7ff989ad9700 1 --2- 172.21.15.32:0/740423380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff984102e10 0x7ff9841031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.812+0000 7ff989ad9700 1 -- 172.21.15.32:0/740423380 >> 172.21.15.32:0/740423380 conn(0x7ff9840f8980 msgr2=0x7ff9840fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.812+0000 7ff989ad9700 1 -- 172.21.15.32:0/740423380 shutdown_connections 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.812+0000 7ff989ad9700 1 -- 172.21.15.32:0/740423380 wait complete. 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff989ad9700 1 Processor -- start 2024-04-06T14:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff989ad9700 1 -- start start 2024-04-06T14:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff989ad9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff984102e10 0x7ff9840702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff989ad9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9841037b0 0x7ff984070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff989ad9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff984070d90 con 0x7ff984102e10 2024-04-06T14:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff989ad9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff984070ed0 con 0x7ff9841037b0 2024-04-06T14:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff9837fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff984102e10 0x7ff9840702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff9837fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff984102e10 0x7ff9840702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51374/0 (socket says 172.21.15.32:51374) 2024-04-06T14:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff9837fe700 1 -- 172.21.15.32:0/2778794401 learned_addr learned my addr 172.21.15.32:0/2778794401 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.813+0000 7ff982ffd700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9841037b0 0x7ff984070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.814+0000 7ff9837fe700 1 -- 172.21.15.32:0/2778794401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9841037b0 msgr2=0x7ff984070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.814+0000 7ff9837fe700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9841037b0 0x7ff984070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.814+0000 7ff9837fe700 1 -- 172.21.15.32:0/2778794401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c005040 con 0x7ff984102e10 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.814+0000 7ff9837fe700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff984102e10 0x7ff9840702c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff974002a50 tx=0x7ff9740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.814+0000 7ff97bfff700 1 -- 172.21.15.32:0/2778794401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff974007cc0 con 0x7ff984102e10 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.814+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff984119dc0 con 0x7ff984102e10 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.814+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98411a240 con 0x7ff984102e10 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.814+0000 7ff97bfff700 1 -- 172.21.15.32:0/2778794401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff97400f040 con 0x7ff984102e10 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.814+0000 7ff97bfff700 1 -- 172.21.15.32:0/2778794401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff97400d8d0 con 0x7ff984102e10 2024-04-06T14:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.815+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9640052b0 con 0x7ff984102e10 2024-04-06T14:55:20.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.816+0000 7ff97bfff700 1 -- 172.21.15.32:0/2778794401 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff97400da30 con 0x7ff984102e10 2024-04-06T14:55:20.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.816+0000 7ff97bfff700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff97006cc20 0x7ff97006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:20.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.816+0000 7ff97bfff700 1 -- 172.21.15.32:0/2778794401 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff97408bcd0 con 0x7ff984102e10 2024-04-06T14:55:20.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.816+0000 7ff982ffd700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff97006cc20 0x7ff97006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:20.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.817+0000 7ff982ffd700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff97006cc20 0x7ff97006f0d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff96c004890 tx=0x7ff96c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:20.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:20.821+0000 7ff97bfff700 1 -- 172.21.15.32:0/2778794401 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff974055c90 con 0x7ff984102e10 2024-04-06T14:55:20.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:20 smithi032 ceph-mon[147414]: from='client.35906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:20.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:20 smithi032 ceph-mon[147414]: from='client.35910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:20.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:20 smithi032 ceph-mon[147414]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:20.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:20 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3720625348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:55:20.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:20 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1085550836' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:55:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.104+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff964005560 con 0x7ff984102e10 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.104+0000 7ff97bfff700 1 -- 172.21.15.32:0/2778794401 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7ff9740592b0 con 0x7ff984102e10 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:55:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:55:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:55:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.106+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff97006cc20 msgr2=0x7ff97006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.106+0000 7ff989ad9700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff97006cc20 0x7ff97006f0d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff96c004890 tx=0x7ff96c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.106+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff984102e10 msgr2=0x7ff9840702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.106+0000 7ff989ad9700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff984102e10 0x7ff9840702c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff974002a50 tx=0x7ff9740046b0 comp rx=0 tx=0).stop 2024-04-06T14:55:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.107+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 shutdown_connections 2024-04-06T14:55:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.107+0000 7ff989ad9700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff97006cc20 0x7ff97006f0d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.107+0000 7ff989ad9700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff984102e10 0x7ff9840702c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.107+0000 7ff989ad9700 1 --2- 172.21.15.32:0/2778794401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9841037b0 0x7ff984070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.107+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 >> 172.21.15.32:0/2778794401 conn(0x7ff9840f8980 msgr2=0x7ff9840f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.107+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 shutdown_connections 2024-04-06T14:55:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:21.107+0000 7ff989ad9700 1 -- 172.21.15.32:0/2778794401 wait complete. 2024-04-06T14:55:21.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:20 smithi086 ceph-mon[110975]: from='client.35906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:21.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:20 smithi086 ceph-mon[110975]: from='client.35910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:21.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:20 smithi086 ceph-mon[110975]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:21.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:20 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3720625348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:55:21.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:20 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1085550836' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:55:21.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:21 smithi032 ceph-mon[147414]: pgmap v1057: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:21.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:21 smithi032 ceph-mon[147414]: from='client.35926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:21.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:21 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2778794401' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:55:22.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:21 smithi086 ceph-mon[110975]: pgmap v1057: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:22.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:21 smithi086 ceph-mon[110975]: from='client.35926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:22.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:21 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2778794401' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:55:22.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:55:23.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:55:23.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:23 smithi032 ceph-mon[147414]: pgmap v1058: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:24.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:23 smithi086 ceph-mon[110975]: pgmap v1058: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:26.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:25 smithi086 ceph-mon[110975]: pgmap v1059: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:26.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:25 smithi032 ceph-mon[147414]: pgmap v1059: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:28.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:27 smithi086 ceph-mon[110975]: pgmap v1060: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:28.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:27 smithi032 ceph-mon[147414]: pgmap v1060: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:30.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:29 smithi086 ceph-mon[110975]: pgmap v1061: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:30.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:29 smithi032 ceph-mon[147414]: pgmap v1061: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:32.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:31 smithi086 ceph-mon[110975]: pgmap v1062: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:32.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:31 smithi032 ceph-mon[147414]: pgmap v1062: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:34.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:33 smithi086 ceph-mon[110975]: pgmap v1063: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:34.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:33 smithi032 ceph-mon[147414]: pgmap v1063: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:36.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:35 smithi086 ceph-mon[110975]: pgmap v1064: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:36.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:35 smithi032 ceph-mon[147414]: pgmap v1064: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:38.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:37 smithi086 ceph-mon[110975]: pgmap v1065: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:38.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:55:38.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:37 smithi032 ceph-mon[147414]: pgmap v1065: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:38.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:55:40.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:39 smithi086 ceph-mon[110975]: pgmap v1066: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:40.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:39 smithi032 ceph-mon[147414]: pgmap v1066: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:42.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:41 smithi086 ceph-mon[110975]: pgmap v1067: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:42.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:41 smithi032 ceph-mon[147414]: pgmap v1067: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:44.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:43 smithi086 ceph-mon[110975]: pgmap v1068: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:44.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:43 smithi032 ceph-mon[147414]: pgmap v1068: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:46.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:45 smithi086 ceph-mon[110975]: pgmap v1069: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:46.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:45 smithi032 ceph-mon[147414]: pgmap v1069: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:48.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:47 smithi086 ceph-mon[110975]: pgmap v1070: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:48.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:47 smithi032 ceph-mon[147414]: pgmap v1070: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:50.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:49 smithi086 ceph-mon[110975]: pgmap v1071: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:50.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:49 smithi032 ceph-mon[147414]: pgmap v1071: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.224+0000 7f9c52cc6700 1 -- 172.21.15.32:0/318502512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 msgr2=0x7f9c4c0fc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.224+0000 7f9c52cc6700 1 --2- 172.21.15.32:0/318502512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 0x7f9c4c0fc5b0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9c38004940 tx=0x7f9c38009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.224+0000 7f9c52cc6700 1 -- 172.21.15.32:0/318502512 shutdown_connections 2024-04-06T14:55:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.224+0000 7f9c52cc6700 1 --2- 172.21.15.32:0/318502512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 0x7f9c4c0fc5b0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.224+0000 7f9c52cc6700 1 --2- 172.21.15.32:0/318502512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c101d50 0x7f9c4c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.224+0000 7f9c52cc6700 1 -- 172.21.15.32:0/318502512 >> 172.21.15.32:0/318502512 conn(0x7f9c4c0714c0 msgr2=0x7f9c4c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.224+0000 7f9c52cc6700 1 -- 172.21.15.32:0/318502512 shutdown_connections 2024-04-06T14:55:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.224+0000 7f9c52cc6700 1 -- 172.21.15.32:0/318502512 wait complete. 2024-04-06T14:55:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.225+0000 7f9c52cc6700 1 Processor -- start 2024-04-06T14:55:51.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.225+0000 7f9c52cc6700 1 -- start start 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c52cc6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c101d50 0x7f9c4c112a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c52cc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 0x7f9c4c112f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c52cc6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c1135a0 con 0x7f9c4c102660 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c52cc6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c10c9e0 con 0x7f9c4c101d50 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c4bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 0x7f9c4c112f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c4bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 0x7f9c4c112f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50516/0 (socket says 172.21.15.32:50516) 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c4bfff700 1 -- 172.21.15.32:0/3481528262 learned_addr learned my addr 172.21.15.32:0/3481528262 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c50a62700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c101d50 0x7f9c4c112a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c4bfff700 1 -- 172.21.15.32:0/3481528262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c101d50 msgr2=0x7f9c4c112a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c4bfff700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c101d50 0x7f9c4c112a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c4bfff700 1 -- 172.21.15.32:0/3481528262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c38005040 con 0x7f9c4c102660 2024-04-06T14:55:51.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c4bfff700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 0x7f9c4c112f50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800b3b0 tx=0x7f9c38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c497fa700 1 -- 172.21.15.32:0/3481528262 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c3800e040 con 0x7f9c4c102660 2024-04-06T14:55:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.226+0000 7f9c497fa700 1 -- 172.21.15.32:0/3481528262 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c38013420 con 0x7f9c4c102660 2024-04-06T14:55:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.227+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4c10cc60 con 0x7f9c4c102660 2024-04-06T14:55:51.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.227+0000 7f9c497fa700 1 -- 172.21.15.32:0/3481528262 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c3801c610 con 0x7f9c4c102660 2024-04-06T14:55:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.227+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4c10d1a0 con 0x7f9c4c102660 2024-04-06T14:55:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.228+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4c0fd5c0 con 0x7f9c4c102660 2024-04-06T14:55:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.228+0000 7f9c497fa700 1 -- 172.21.15.32:0/3481528262 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9c38013a00 con 0x7f9c4c102660 2024-04-06T14:55:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.228+0000 7f9c497fa700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c3c06cc20 0x7f9c3c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.229+0000 7f9c497fa700 1 -- 172.21.15.32:0/3481528262 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c3808d240 con 0x7f9c4c102660 2024-04-06T14:55:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.229+0000 7f9c50a62700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c3c06cc20 0x7f9c3c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.229+0000 7f9c50a62700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c3c06cc20 0x7f9c3c06f0d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9c400041d0 tx=0x7f9c400043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:51.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.233+0000 7f9c497fa700 1 -- 172.21.15.32:0/3481528262 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9c3805ad10 con 0x7f9c4c102660 2024-04-06T14:55:51.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.454+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c4c108cc0 con 0x7f9c3c06cc20 2024-04-06T14:55:51.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.456+0000 7f9c497fa700 1 -- 172.21.15.32:0/3481528262 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9c4c108cc0 con 0x7f9c3c06cc20 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c3c06cc20 msgr2=0x7f9c3c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c3c06cc20 0x7f9c3c06f0d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9c400041d0 tx=0x7f9c400043c0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 msgr2=0x7f9c4c112f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 0x7f9c4c112f50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800b3b0 tx=0x7f9c38009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 shutdown_connections 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f9c3c06cc20 0x7f9c3c06f0d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c101d50 0x7f9c4c112a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 --2- 172.21.15.32:0/3481528262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c4c102660 0x7f9c4c112f50 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 >> 172.21.15.32:0/3481528262 conn(0x7f9c4c0714c0 msgr2=0x7f9c4c104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 shutdown_connections 2024-04-06T14:55:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.458+0000 7f9c52cc6700 1 -- 172.21.15.32:0/3481528262 wait complete. 2024-04-06T14:55:51.455 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:55:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.561+0000 7f705e320700 1 -- 172.21.15.32:0/1317374090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 msgr2=0x7f70580fbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.561+0000 7f705e320700 1 --2- 172.21.15.32:0/1317374090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 0x7f70580fbfc0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f7040004940 tx=0x7f7040009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.562+0000 7f705e320700 1 -- 172.21.15.32:0/1317374090 shutdown_connections 2024-04-06T14:55:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.562+0000 7f705e320700 1 --2- 172.21.15.32:0/1317374090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70580fc500 0x7f7058109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.562+0000 7f705e320700 1 --2- 172.21.15.32:0/1317374090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 0x7f70580fbfc0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.562+0000 7f705e320700 1 -- 172.21.15.32:0/1317374090 >> 172.21.15.32:0/1317374090 conn(0x7f70580f7880 msgr2=0x7f70580f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.562+0000 7f705e320700 1 -- 172.21.15.32:0/1317374090 shutdown_connections 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.562+0000 7f705e320700 1 -- 172.21.15.32:0/1317374090 wait complete. 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.562+0000 7f705e320700 1 Processor -- start 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.562+0000 7f705e320700 1 -- start start 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f705e320700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 0x7f7058115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f705e320700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70580fc500 0x7f7058116500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f705e320700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7058116b50 con 0x7f70580fbbf0 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f705e320700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705810ff20 con 0x7f70580fc500 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7057fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 0x7f7058115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7057fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 0x7f7058115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50542/0 (socket says 172.21.15.32:50542) 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7057fff700 1 -- 172.21.15.32:0/3133227472 learned_addr learned my addr 172.21.15.32:0/3133227472 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f70577fe700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70580fc500 0x7f7058116500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7057fff700 1 -- 172.21.15.32:0/3133227472 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70580fc500 msgr2=0x7f7058116500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7057fff700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70580fc500 0x7f7058116500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7057fff700 1 -- 172.21.15.32:0/3133227472 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040005040 con 0x7f70580fbbf0 2024-04-06T14:55:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f70577fe700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70580fc500 0x7f7058116500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:55:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7057fff700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 0x7f7058115fc0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f704000b3b0 tx=0x7f7040009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7054ff9700 1 -- 172.21.15.32:0/3133227472 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f704000e040 con 0x7f70580fbbf0 2024-04-06T14:55:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70581101a0 con 0x7f70580fbbf0 2024-04-06T14:55:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7054ff9700 1 -- 172.21.15.32:0/3133227472 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f704000bed0 con 0x7f70580fbbf0 2024-04-06T14:55:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f7054ff9700 1 -- 172.21.15.32:0/3133227472 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f704001c3f0 con 0x7f70580fbbf0 2024-04-06T14:55:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.563+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7058110760 con 0x7f70580fbbf0 2024-04-06T14:55:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.564+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7058108f00 con 0x7f70580fbbf0 2024-04-06T14:55:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.565+0000 7f7054ff9700 1 -- 172.21.15.32:0/3133227472 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f704000ba20 con 0x7f70580fbbf0 2024-04-06T14:55:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.565+0000 7f7054ff9700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f704406cc20 0x7f704406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.566+0000 7f7054ff9700 1 -- 172.21.15.32:0/3133227472 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f704008d3d0 con 0x7f70580fbbf0 2024-04-06T14:55:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.566+0000 7f70577fe700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f704406cc20 0x7f704406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.566+0000 7f70577fe700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f704406cc20 0x7f704406f0d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7058111860 tx=0x7f7048009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:51.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.570+0000 7f7054ff9700 1 -- 172.21.15.32:0/3133227472 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f704005aea0 con 0x7f70580fbbf0 2024-04-06T14:55:51.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.793+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f705810c000 con 0x7f704406cc20 2024-04-06T14:55:51.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.795+0000 7f7054ff9700 1 -- 172.21.15.32:0/3133227472 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7048004940 con 0x7f704406cc20 2024-04-06T14:55:51.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f704406cc20 msgr2=0x7f704406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f704406cc20 0x7f704406f0d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7058111860 tx=0x7f7048009040 comp rx=0 tx=0).stop 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 msgr2=0x7f7058115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 0x7f7058115fc0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f704000b3b0 tx=0x7f7040009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 shutdown_connections 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f704406cc20 0x7f704406f0d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70580fbbf0 0x7f7058115fc0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 --2- 172.21.15.32:0/3133227472 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70580fc500 0x7f7058116500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 >> 172.21.15.32:0/3133227472 conn(0x7f70580f7880 msgr2=0x7f70580f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 shutdown_connections 2024-04-06T14:55:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.797+0000 7f705e320700 1 -- 172.21.15.32:0/3133227472 wait complete. 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.899+0000 7f3b575ae700 1 -- 172.21.15.32:0/252888916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 msgr2=0x7f3b50102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.899+0000 7f3b575ae700 1 --2- 172.21.15.32:0/252888916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 0x7f3b50102140 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c004940 tx=0x7f3b4c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.899+0000 7f3b575ae700 1 -- 172.21.15.32:0/252888916 shutdown_connections 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.899+0000 7f3b575ae700 1 --2- 172.21.15.32:0/252888916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b50102710 0x7f3b500fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.899+0000 7f3b575ae700 1 --2- 172.21.15.32:0/252888916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 0x7f3b50102140 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.899+0000 7f3b575ae700 1 -- 172.21.15.32:0/252888916 >> 172.21.15.32:0/252888916 conn(0x7f3b50072bc0 msgr2=0x7f3b50072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b575ae700 1 -- 172.21.15.32:0/252888916 shutdown_connections 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b575ae700 1 -- 172.21.15.32:0/252888916 wait complete. 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b575ae700 1 Processor -- start 2024-04-06T14:55:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b575ae700 1 -- start start 2024-04-06T14:55:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b575ae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 0x7f3b50113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b575ae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b50102710 0x7f3b50114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b575ae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b50114670 con 0x7f3b50101d70 2024-04-06T14:55:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b575ae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5010da60 con 0x7f3b50102710 2024-04-06T14:55:51.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b5534a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 0x7f3b50113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:51.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.900+0000 7f3b5534a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 0x7f3b50113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50568/0 (socket says 172.21.15.32:50568) 2024-04-06T14:55:51.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b5534a700 1 -- 172.21.15.32:0/210726935 learned_addr learned my addr 172.21.15.32:0/210726935 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b54b49700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b50102710 0x7f3b50114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b5534a700 1 -- 172.21.15.32:0/210726935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b50102710 msgr2=0x7f3b50114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b5534a700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b50102710 0x7f3b50114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b5534a700 1 -- 172.21.15.32:0/210726935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c005040 con 0x7f3b50101d70 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b5534a700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 0x7f3b50113ae0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009390 tx=0x7f3b4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b45ffb700 1 -- 172.21.15.32:0/210726935 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b4c00e040 con 0x7f3b50101d70 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b45ffb700 1 -- 172.21.15.32:0/210726935 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b4c013420 con 0x7f3b50101d70 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5010dce0 con 0x7f3b50101d70 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b45ffb700 1 -- 172.21.15.32:0/210726935 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b4c01c630 con 0x7f3b50101d70 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.901+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5010e2a0 con 0x7f3b50101d70 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.902+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b500fd4c0 con 0x7f3b50101d70 2024-04-06T14:55:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.903+0000 7f3b45ffb700 1 -- 172.21.15.32:0/210726935 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3b4c0139e0 con 0x7f3b50101d70 2024-04-06T14:55:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.903+0000 7f3b45ffb700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b3c06cc30 0x7f3b3c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.903+0000 7f3b45ffb700 1 -- 172.21.15.32:0/210726935 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3b4c08d3e0 con 0x7f3b50101d70 2024-04-06T14:55:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.903+0000 7f3b54b49700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b3c06cc30 0x7f3b3c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.904+0000 7f3b54b49700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b3c06cc30 0x7f3b3c06f0e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3b5010ec20 tx=0x7f3b4000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:51.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:51.908+0000 7f3b45ffb700 1 -- 172.21.15.32:0/210726935 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3b4c05ae30 con 0x7f3b50101d70 2024-04-06T14:55:52.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.121+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b50108da0 con 0x7f3b3c06cc30 2024-04-06T14:55:52.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:51 smithi086 ceph-mon[110975]: pgmap v1072: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:52.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.131+0000 7f3b45ffb700 1 -- 172.21.15.32:0/210726935 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3b50108da0 con 0x7f3b3c06cc30 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (36m) 14m ago 47m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (33m) 14m ago 47m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (33m) 14m ago 46m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (35m) 14m ago 47m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (29m) 14m ago 41m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (29m) 14m ago 41m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (29m) 14m ago 41m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (14m) 14m ago 41m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (38m) 14m ago 48m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (37m) 14m ago 46m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:55:52.118 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (34m) 14m ago 49m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (34m) 14m ago 46m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (36m) 14m ago 47m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (36m) 14m ago 46m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (33m) 14m ago 45m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (32m) 14m ago 45m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (31m) 14m ago 44m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (31m) 14m ago 44m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (30m) 14m ago 43m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (30m) 14m ago 43m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:55:52.119 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (36m) 14m ago 47m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b3c06cc30 msgr2=0x7f3b3c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b3c06cc30 0x7f3b3c06f0e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3b5010ec20 tx=0x7f3b4000b040 comp rx=0 tx=0).stop 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 msgr2=0x7f3b50113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 0x7f3b50113ae0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009390 tx=0x7f3b4c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 shutdown_connections 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b3c06cc30 0x7f3b3c06f0e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d70 0x7f3b50113ae0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 --2- 172.21.15.32:0/210726935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b50102710 0x7f3b50114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 >> 172.21.15.32:0/210726935 conn(0x7f3b50072bc0 msgr2=0x7f3b500659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 shutdown_connections 2024-04-06T14:55:52.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.133+0000 7f3b575ae700 1 -- 172.21.15.32:0/210726935 wait complete. 2024-04-06T14:55:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:51 smithi032 ceph-mon[147414]: pgmap v1072: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:52.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.234+0000 7fc95d850700 1 -- 172.21.15.32:0/2468505924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958101d20 msgr2=0x7fc9581020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.234+0000 7fc95d850700 1 --2- 172.21.15.32:0/2468505924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958101d20 0x7fc9581020f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc948004940 tx=0x7fc948009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:52.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.235+0000 7fc95d850700 1 -- 172.21.15.32:0/2468505924 shutdown_connections 2024-04-06T14:55:52.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.235+0000 7fc95d850700 1 --2- 172.21.15.32:0/2468505924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9581026c0 0x7fc9580fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.235+0000 7fc95d850700 1 --2- 172.21.15.32:0/2468505924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958101d20 0x7fc9581020f0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.235+0000 7fc95d850700 1 -- 172.21.15.32:0/2468505924 >> 172.21.15.32:0/2468505924 conn(0x7fc9580f7870 msgr2=0x7fc9580f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:52.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.235+0000 7fc95d850700 1 -- 172.21.15.32:0/2468505924 shutdown_connections 2024-04-06T14:55:52.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.235+0000 7fc95d850700 1 -- 172.21.15.32:0/2468505924 wait complete. 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.235+0000 7fc95d850700 1 Processor -- start 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.235+0000 7fc95d850700 1 -- start start 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc95d850700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc958101d20 0x7fc9581127d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc95d850700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9581026c0 0x7fc958112d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc95d850700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc958113360 con 0x7fc9581026c0 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc95d850700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95810c960 con 0x7fc958101d20 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc9567fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9581026c0 0x7fc958112d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc9567fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9581026c0 0x7fc958112d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50588/0 (socket says 172.21.15.32:50588) 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc9567fc700 1 -- 172.21.15.32:0/1000499092 learned_addr learned my addr 172.21.15.32:0/1000499092 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc956ffd700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc958101d20 0x7fc9581127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc9567fc700 1 -- 172.21.15.32:0/1000499092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc958101d20 msgr2=0x7fc9581127d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc9567fc700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc958101d20 0x7fc9581127d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc9567fc700 1 -- 172.21.15.32:0/1000499092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948005040 con 0x7fc9581026c0 2024-04-06T14:55:52.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc9567fc700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9581026c0 0x7fc958112d10 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc94c002a50 tx=0x7fc94c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:52.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc93f7fe700 1 -- 172.21.15.32:0/1000499092 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc94c007cc0 con 0x7fc9581026c0 2024-04-06T14:55:52.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc93f7fe700 1 -- 172.21.15.32:0/1000499092 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc94c00f040 con 0x7fc9581026c0 2024-04-06T14:55:52.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95810cc40 con 0x7fc9581026c0 2024-04-06T14:55:52.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc93f7fe700 1 -- 172.21.15.32:0/1000499092 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc94c00d920 con 0x7fc9581026c0 2024-04-06T14:55:52.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.236+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95810d200 con 0x7fc9581026c0 2024-04-06T14:55:52.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.237+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9580fd470 con 0x7fc9581026c0 2024-04-06T14:55:52.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.238+0000 7fc93f7fe700 1 -- 172.21.15.32:0/1000499092 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc94c009680 con 0x7fc9581026c0 2024-04-06T14:55:52.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.238+0000 7fc93f7fe700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc94006cc20 0x7fc94006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:52.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.238+0000 7fc93f7fe700 1 -- 172.21.15.32:0/1000499092 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc94c08bde0 con 0x7fc9581026c0 2024-04-06T14:55:52.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.238+0000 7fc956ffd700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc94006cc20 0x7fc94006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:52.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.239+0000 7fc956ffd700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc94006cc20 0x7fc94006f0d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc948004f90 tx=0x7fc948009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:52.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.243+0000 7fc93f7fe700 1 -- 172.21.15.32:0/1000499092 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc94c059830 con 0x7fc9581026c0 2024-04-06T14:55:52.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.533+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc958108ce0 con 0x7fc9581026c0 2024-04-06T14:55:52.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.533+0000 7fc93f7fe700 1 -- 172.21.15.32:0/1000499092 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc94c0593c0 con 0x7fc9581026c0 2024-04-06T14:55:52.520 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:55:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:55:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:55:52.521 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:55:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc94006cc20 msgr2=0x7fc94006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc94006cc20 0x7fc94006f0d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc948004f90 tx=0x7fc948009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9581026c0 msgr2=0x7fc958112d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9581026c0 0x7fc958112d10 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc94c002a50 tx=0x7fc94c004600 comp rx=0 tx=0).stop 2024-04-06T14:55:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 shutdown_connections 2024-04-06T14:55:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc94006cc20 0x7fc94006f0d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc958101d20 0x7fc9581127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 --2- 172.21.15.32:0/1000499092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9581026c0 0x7fc958112d10 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 >> 172.21.15.32:0/1000499092 conn(0x7fc9580f7870 msgr2=0x7fc9580f8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:52.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 shutdown_connections 2024-04-06T14:55:52.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.535+0000 7fc95d850700 1 -- 172.21.15.32:0/1000499092 wait complete. 2024-04-06T14:55:52.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.638+0000 7f010b8f9700 1 -- 172.21.15.32:0/3017888629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01040fc5b0 msgr2=0x7f01040fca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.638+0000 7f010b8f9700 1 --2- 172.21.15.32:0/3017888629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01040fc5b0 0x7f01040fca20 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f00f4004940 tx=0x7f00f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:52.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.638+0000 7f010b8f9700 1 -- 172.21.15.32:0/3017888629 shutdown_connections 2024-04-06T14:55:52.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.638+0000 7f010b8f9700 1 --2- 172.21.15.32:0/3017888629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01040fc5b0 0x7f01040fca20 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.638+0000 7f010b8f9700 1 --2- 172.21.15.32:0/3017888629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0104101d70 0x7f0104102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.638+0000 7f010b8f9700 1 -- 172.21.15.32:0/3017888629 >> 172.21.15.32:0/3017888629 conn(0x7f01040714c0 msgr2=0x7f01040718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:52.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.638+0000 7f010b8f9700 1 -- 172.21.15.32:0/3017888629 shutdown_connections 2024-04-06T14:55:52.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.638+0000 7f010b8f9700 1 -- 172.21.15.32:0/3017888629 wait complete. 2024-04-06T14:55:52.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.639+0000 7f010b8f9700 1 Processor -- start 2024-04-06T14:55:52.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.639+0000 7f010b8f9700 1 -- start start 2024-04-06T14:55:52.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.639+0000 7f010b8f9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01040fc5b0 0x7f01041129b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:52.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f010b8f9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0104101d70 0x7f0104112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:52.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f010b8f9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0104113540 con 0x7f0104101d70 2024-04-06T14:55:52.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f010b8f9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010410c930 con 0x7f01040fc5b0 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f0108e94700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0104101d70 0x7f0104112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f0108e94700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0104101d70 0x7f0104112ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50618/0 (socket says 172.21.15.32:50618) 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f0108e94700 1 -- 172.21.15.32:0/3907195142 learned_addr learned my addr 172.21.15.32:0/3907195142 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f0109695700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01040fc5b0 0x7f01041129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f0108e94700 1 -- 172.21.15.32:0/3907195142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01040fc5b0 msgr2=0x7f01041129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f0108e94700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01040fc5b0 0x7f01041129b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f0108e94700 1 -- 172.21.15.32:0/3907195142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f4005040 con 0x7f0104101d70 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f0108e94700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0104101d70 0x7f0104112ef0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f00f400b3b0 tx=0x7f00f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f00f9ffb700 1 -- 172.21.15.32:0/3907195142 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00f400e040 con 0x7f0104101d70 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f00f9ffb700 1 -- 172.21.15.32:0/3907195142 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00f4013420 con 0x7f0104101d70 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010410cbb0 con 0x7f0104101d70 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.640+0000 7f00f9ffb700 1 -- 172.21.15.32:0/3907195142 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00f401c630 con 0x7f0104101d70 2024-04-06T14:55:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.641+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010410d170 con 0x7f0104101d70 2024-04-06T14:55:52.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.641+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01040fd5c0 con 0x7f0104101d70 2024-04-06T14:55:52.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.643+0000 7f00f9ffb700 1 -- 172.21.15.32:0/3907195142 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f00f4013970 con 0x7f0104101d70 2024-04-06T14:55:52.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.643+0000 7f00f9ffb700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f006cc20 0x7f00f006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:52.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.643+0000 7f00f9ffb700 1 -- 172.21.15.32:0/3907195142 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f00f408dd20 con 0x7f0104101d70 2024-04-06T14:55:52.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.643+0000 7f0109695700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f006cc20 0x7f00f006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:52.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.644+0000 7f0109695700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f006cc20 0x7f00f006f0d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0100009fd0 tx=0x7f0100009f00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:52.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.647+0000 7f00f9ffb700 1 -- 172.21.15.32:0/3907195142 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f00f405b8b0 con 0x7f0104101d70 2024-04-06T14:55:52.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.897+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f010405f5f0 con 0x7f0104101d70 2024-04-06T14:55:52.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.897+0000 7f00f9ffb700 1 -- 172.21.15.32:0/3907195142 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f00f4011370 con 0x7f0104101d70 2024-04-06T14:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:55:52.885 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:55:52.886 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:55:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:55:52.887 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:55:52.887 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:55:52.887 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:55:52.887 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:55:52.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.899+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f006cc20 msgr2=0x7f00f006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.899+0000 7f010b8f9700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f006cc20 0x7f00f006f0d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0100009fd0 tx=0x7f0100009f00 comp rx=0 tx=0).stop 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.899+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0104101d70 msgr2=0x7f0104112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.899+0000 7f010b8f9700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0104101d70 0x7f0104112ef0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f00f400b3b0 tx=0x7f00f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.899+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 shutdown_connections 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.899+0000 7f010b8f9700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f00f006cc20 0x7f00f006f0d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.899+0000 7f010b8f9700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01040fc5b0 0x7f01041129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.899+0000 7f010b8f9700 1 --2- 172.21.15.32:0/3907195142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0104101d70 0x7f0104112ef0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.899+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 >> 172.21.15.32:0/3907195142 conn(0x7f01040714c0 msgr2=0x7f0104104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.900+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 shutdown_connections 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:52.900+0000 7f010b8f9700 1 -- 172.21.15.32:0/3907195142 wait complete. 2024-04-06T14:55:52.888 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:55:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 -- 172.21.15.32:0/3495685476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6900fb4a0 msgr2=0x7fd6900fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 --2- 172.21.15.32:0/3495685476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6900fb4a0 0x7fd6900fb910 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd68c004940 tx=0x7fd68c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 -- 172.21.15.32:0/3495685476 shutdown_connections 2024-04-06T14:55:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 --2- 172.21.15.32:0/3495685476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6900fb4a0 0x7fd6900fb910 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 --2- 172.21.15.32:0/3495685476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd690102e70 0x7fd690103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 -- 172.21.15.32:0/3495685476 >> 172.21.15.32:0/3495685476 conn(0x7fd6900714c0 msgr2=0x7fd6900718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 -- 172.21.15.32:0/3495685476 shutdown_connections 2024-04-06T14:55:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 -- 172.21.15.32:0/3495685476 wait complete. 2024-04-06T14:55:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 Processor -- start 2024-04-06T14:55:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.003+0000 7fd697626700 1 -- start start 2024-04-06T14:55:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd697626700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6900fb4a0 0x7fd690113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd697626700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd690102e70 0x7fd690113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd697626700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6901145c0 con 0x7fd690102e70 2024-04-06T14:55:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd697626700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69010d9b0 con 0x7fd6900fb4a0 2024-04-06T14:55:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd694bc1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd690102e70 0x7fd690113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd694bc1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd690102e70 0x7fd690113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50636/0 (socket says 172.21.15.32:50636) 2024-04-06T14:55:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd694bc1700 1 -- 172.21.15.32:0/634927038 learned_addr learned my addr 172.21.15.32:0/634927038 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd6953c2700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6900fb4a0 0x7fd690113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd694bc1700 1 -- 172.21.15.32:0/634927038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6900fb4a0 msgr2=0x7fd690113a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd694bc1700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6900fb4a0 0x7fd690113a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd694bc1700 1 -- 172.21.15.32:0/634927038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c005040 con 0x7fd690102e70 2024-04-06T14:55:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd694bc1700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd690102e70 0x7fd690113f70 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd68c000c00 tx=0x7fd68c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd685ffb700 1 -- 172.21.15.32:0/634927038 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68c021070 con 0x7fd690102e70 2024-04-06T14:55:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69010dc30 con 0x7fd690102e70 2024-04-06T14:55:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd685ffb700 1 -- 172.21.15.32:0/634927038 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd68c013de0 con 0x7fd690102e70 2024-04-06T14:55:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.004+0000 7fd685ffb700 1 -- 172.21.15.32:0/634927038 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68c00e070 con 0x7fd690102e70 2024-04-06T14:55:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.005+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69010e190 con 0x7fd690102e70 2024-04-06T14:55:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.005+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd690100110 con 0x7fd690102e70 2024-04-06T14:55:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.006+0000 7fd685ffb700 1 -- 172.21.15.32:0/634927038 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd68c01c600 con 0x7fd690102e70 2024-04-06T14:55:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.006+0000 7fd685ffb700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd67c06cc20 0x7fd67c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.007+0000 7fd685ffb700 1 -- 172.21.15.32:0/634927038 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd68c08d130 con 0x7fd690102e70 2024-04-06T14:55:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.007+0000 7fd6953c2700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd67c06cc20 0x7fd67c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.007+0000 7fd6953c2700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd67c06cc20 0x7fd67c06f0d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd680004600 tx=0x7fd68000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:52.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.011+0000 7fd685ffb700 1 -- 172.21.15.32:0/634927038 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd68c05ac00 con 0x7fd690102e70 2024-04-06T14:55:53.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:52 smithi086 ceph-mon[110975]: from='client.35934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:53.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:52 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1000499092' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:55:53.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:55:53.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:52 smithi032 ceph-mon[147414]: from='client.35934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:53.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:52 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1000499092' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:55:53.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:55:53.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.232+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd69010f080 con 0x7fd67c06cc20 2024-04-06T14:55:53.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.233+0000 7fd685ffb700 1 -- 172.21.15.32:0/634927038 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd69010f080 con 0x7fd67c06cc20 2024-04-06T14:55:53.219 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:55:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:55:53.220 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd67c06cc20 msgr2=0x7fd67c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd67c06cc20 0x7fd67c06f0d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd680004600 tx=0x7fd68000b040 comp rx=0 tx=0).stop 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd690102e70 msgr2=0x7fd690113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd690102e70 0x7fd690113f70 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd68c000c00 tx=0x7fd68c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 shutdown_connections 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd67c06cc20 0x7fd67c06f0d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6900fb4a0 0x7fd690113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 --2- 172.21.15.32:0/634927038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd690102e70 0x7fd690113f70 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 >> 172.21.15.32:0/634927038 conn(0x7fd6900714c0 msgr2=0x7fd6900fa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 shutdown_connections 2024-04-06T14:55:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.235+0000 7fd697626700 1 -- 172.21.15.32:0/634927038 wait complete. 2024-04-06T14:55:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 -- 172.21.15.32:0/4118136939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 msgr2=0x7f7da00fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 --2- 172.21.15.32:0/4118136939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 0x7f7da00fe1d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c004940 tx=0x7f7d9c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 -- 172.21.15.32:0/4118136939 shutdown_connections 2024-04-06T14:55:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 --2- 172.21.15.32:0/4118136939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7da00fe7a0 0x7f7da0106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 --2- 172.21.15.32:0/4118136939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 0x7f7da00fe1d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 -- 172.21.15.32:0/4118136939 >> 172.21.15.32:0/4118136939 conn(0x7f7da00f7880 msgr2=0x7f7da00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 -- 172.21.15.32:0/4118136939 shutdown_connections 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 -- 172.21.15.32:0/4118136939 wait complete. 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 Processor -- start 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 -- start start 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.338+0000 7f7da7899700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 0x7f7da00fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da7899700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7da00fe7a0 0x7f7da00fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da7899700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da00fc790 con 0x7f7da00fde00 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da7899700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da00fc8d0 con 0x7f7da00fe7a0 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da5635700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 0x7f7da00fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da5635700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 0x7f7da00fbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50658/0 (socket says 172.21.15.32:50658) 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da5635700 1 -- 172.21.15.32:0/3465913221 learned_addr learned my addr 172.21.15.32:0/3465913221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da4e34700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7da00fe7a0 0x7f7da00fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da5635700 1 -- 172.21.15.32:0/3465913221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7da00fe7a0 msgr2=0x7f7da00fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da5635700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7da00fe7a0 0x7f7da00fc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da5635700 1 -- 172.21.15.32:0/3465913221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c005040 con 0x7f7da00fde00 2024-04-06T14:55:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da4e34700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7da00fe7a0 0x7f7da00fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:55:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da5635700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 0x7f7da00fbcc0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009380 tx=0x7f7d9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7d95ffb700 1 -- 172.21.15.32:0/3465913221 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d9c00e040 con 0x7f7da00fde00 2024-04-06T14:55:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7d95ffb700 1 -- 172.21.15.32:0/3465913221 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d9c004290 con 0x7f7da00fde00 2024-04-06T14:55:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da00fcb20 con 0x7f7da00fde00 2024-04-06T14:55:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7d95ffb700 1 -- 172.21.15.32:0/3465913221 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d9c01c3f0 con 0x7f7da00fde00 2024-04-06T14:55:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.339+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da011a4a0 con 0x7f7da00fde00 2024-04-06T14:55:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.340+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da00666f0 con 0x7f7da00fde00 2024-04-06T14:55:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.341+0000 7f7d95ffb700 1 -- 172.21.15.32:0/3465913221 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7d9c013790 con 0x7f7da00fde00 2024-04-06T14:55:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.341+0000 7f7d95ffb700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d8c06cc20 0x7f7d8c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:55:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.341+0000 7f7d95ffb700 1 -- 172.21.15.32:0/3465913221 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7d9c08d3a0 con 0x7f7da00fde00 2024-04-06T14:55:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.341+0000 7f7da4e34700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d8c06cc20 0x7f7d8c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:55:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.342+0000 7f7da4e34700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d8c06cc20 0x7f7d8c06f0d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009fd0 tx=0x7f7d90009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:55:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.346+0000 7f7d95ffb700 1 -- 172.21.15.32:0/3465913221 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7d9c0573e0 con 0x7f7da00fde00 2024-04-06T14:55:53.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.634+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7da004bf00 con 0x7f7da00fde00 2024-04-06T14:55:53.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.635+0000 7f7d95ffb700 1 -- 172.21.15.32:0/3465913221 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7f7d9c013420 con 0x7f7da00fde00 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:55:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:55:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:55:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d8c06cc20 msgr2=0x7f7d8c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d8c06cc20 0x7f7d8c06f0d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009fd0 tx=0x7f7d90009ee0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 msgr2=0x7f7da00fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:55:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 0x7f7da00fbcc0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009380 tx=0x7f7d9c009e30 comp rx=0 tx=0).stop 2024-04-06T14:55:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 shutdown_connections 2024-04-06T14:55:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d8c06cc20 0x7f7d8c06f0d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00fde00 0x7f7da00fbcc0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 --2- 172.21.15.32:0/3465913221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7da00fe7a0 0x7f7da00fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:55:53.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 >> 172.21.15.32:0/3465913221 conn(0x7f7da00f7880 msgr2=0x7f7da00f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:55:53.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 shutdown_connections 2024-04-06T14:55:53.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:55:53.637+0000 7f7da7899700 1 -- 172.21.15.32:0/3465913221 wait complete. 2024-04-06T14:55:54.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:53 smithi086 ceph-mon[110975]: from='client.35938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:54.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:53 smithi086 ceph-mon[110975]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:54.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:53 smithi086 ceph-mon[110975]: pgmap v1073: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:54.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:53 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3907195142' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:55:54.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:53 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3465913221' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:55:54.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:53 smithi032 ceph-mon[147414]: from='client.35938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:54.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:53 smithi032 ceph-mon[147414]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:54.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:53 smithi032 ceph-mon[147414]: pgmap v1073: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:54.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:53 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3907195142' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:55:54.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:53 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3465913221' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:55:55.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:54 smithi086 ceph-mon[110975]: from='client.35954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:55.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:54 smithi032 ceph-mon[147414]: from='client.35954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:55:56.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:55 smithi086 ceph-mon[110975]: pgmap v1074: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:56.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:55 smithi032 ceph-mon[147414]: pgmap v1074: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:57.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:56 smithi086 ceph-mon[110975]: pgmap v1075: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:57.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:56 smithi032 ceph-mon[147414]: pgmap v1075: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:59.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:55:59 smithi032 ceph-mon[147414]: pgmap v1076: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:55:59.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:55:59 smithi086 ceph-mon[110975]: pgmap v1076: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:01.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:01 smithi032 ceph-mon[147414]: pgmap v1077: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:01.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:01 smithi086 ceph-mon[110975]: pgmap v1077: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:03.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:03 smithi032 ceph-mon[147414]: pgmap v1078: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:03.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:03 smithi086 ceph-mon[110975]: pgmap v1078: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:05.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:05 smithi032 ceph-mon[147414]: pgmap v1079: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:05.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:05 smithi086 ceph-mon[110975]: pgmap v1079: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:07.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:07 smithi032 ceph-mon[147414]: pgmap v1080: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:07.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:07 smithi086 ceph-mon[110975]: pgmap v1080: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:08.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:56:08.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:56:09.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:09 smithi032 ceph-mon[147414]: pgmap v1081: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:09.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:09 smithi086 ceph-mon[110975]: pgmap v1081: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:11.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:11 smithi032 ceph-mon[147414]: pgmap v1082: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:11.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:11 smithi086 ceph-mon[110975]: pgmap v1082: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:13.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:13 smithi032 ceph-mon[147414]: pgmap v1083: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:13.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:13 smithi086 ceph-mon[110975]: pgmap v1083: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:15.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:15 smithi032 ceph-mon[147414]: pgmap v1084: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:15.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:15 smithi086 ceph-mon[110975]: pgmap v1084: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:17.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:17 smithi032 ceph-mon[147414]: pgmap v1085: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:17.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:17 smithi086 ceph-mon[110975]: pgmap v1085: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:19.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:19 smithi032 ceph-mon[147414]: pgmap v1086: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:19.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:19 smithi086 ceph-mon[110975]: pgmap v1086: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:21.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:21 smithi032 ceph-mon[147414]: pgmap v1087: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:21.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:21 smithi086 ceph-mon[110975]: pgmap v1087: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:23.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:23 smithi032 ceph-mon[147414]: pgmap v1088: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:23.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:56:23.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.754+0000 7f3b57094700 1 -- 172.21.15.32:0/217892239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b501026b0 msgr2=0x7f3b500fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:23.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.754+0000 7f3b57094700 1 --2- 172.21.15.32:0/217892239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b501026b0 0x7f3b500fbd20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c004940 tx=0x7f3b3c009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:23.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.754+0000 7f3b57094700 1 -- 172.21.15.32:0/217892239 shutdown_connections 2024-04-06T14:56:23.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.754+0000 7f3b57094700 1 --2- 172.21.15.32:0/217892239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b501026b0 0x7f3b500fbd20 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:23.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.754+0000 7f3b57094700 1 --2- 172.21.15.32:0/217892239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b50101d10 0x7f3b501020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:23.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.754+0000 7f3b57094700 1 -- 172.21.15.32:0/217892239 >> 172.21.15.32:0/217892239 conn(0x7f3b500f7880 msgr2=0x7f3b500f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:23.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.754+0000 7f3b57094700 1 -- 172.21.15.32:0/217892239 shutdown_connections 2024-04-06T14:56:23.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.754+0000 7f3b57094700 1 -- 172.21.15.32:0/217892239 wait complete. 2024-04-06T14:56:23.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.755+0000 7f3b57094700 1 Processor -- start 2024-04-06T14:56:23.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.755+0000 7f3b57094700 1 -- start start 2024-04-06T14:56:23.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b57094700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d10 0x7f3b5010ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:23.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b57094700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b501026b0 0x7f3b5010f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b57094700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5010f9c0 con 0x7f3b50101d10 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b57094700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5010fb00 con 0x7f3b501026b0 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b54e30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d10 0x7f3b5010ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b54e30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d10 0x7f3b5010ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57798/0 (socket says 172.21.15.32:57798) 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b54e30700 1 -- 172.21.15.32:0/343699696 learned_addr learned my addr 172.21.15.32:0/343699696 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b4ffff700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b501026b0 0x7f3b5010f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b54e30700 1 -- 172.21.15.32:0/343699696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b501026b0 msgr2=0x7f3b5010f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b54e30700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b501026b0 0x7f3b5010f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b54e30700 1 -- 172.21.15.32:0/343699696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c005040 con 0x7f3b50101d10 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b54e30700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d10 0x7f3b5010ef40 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f3b44002a50 tx=0x7f3b440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b4d7fa700 1 -- 172.21.15.32:0/343699696 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b44007cc0 con 0x7f3b50101d10 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b4d7fa700 1 -- 172.21.15.32:0/343699696 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b4400f040 con 0x7f3b50101d10 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.756+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b50117df0 con 0x7f3b50101d10 2024-04-06T14:56:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.757+0000 7f3b4d7fa700 1 -- 172.21.15.32:0/343699696 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b4400d9c0 con 0x7f3b50101d10 2024-04-06T14:56:23.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.757+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b50118330 con 0x7f3b50101d10 2024-04-06T14:56:23.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.758+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b500fd460 con 0x7f3b50101d10 2024-04-06T14:56:23.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.758+0000 7f3b4d7fa700 1 -- 172.21.15.32:0/343699696 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3b4400db60 con 0x7f3b50101d10 2024-04-06T14:56:23.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.758+0000 7f3b4d7fa700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b4006cc20 0x7f3b4006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:23.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.759+0000 7f3b4d7fa700 1 -- 172.21.15.32:0/343699696 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3b4408bd80 con 0x7f3b50101d10 2024-04-06T14:56:23.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.759+0000 7f3b4ffff700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b4006cc20 0x7f3b4006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:23.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.759+0000 7f3b4ffff700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b4006cc20 0x7f3b4006f0d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3b500668b0 tx=0x7f3b3c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:23.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.763+0000 7f3b4d7fa700 1 -- 172.21.15.32:0/343699696 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3b44055d40 con 0x7f3b50101d10 2024-04-06T14:56:23.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:23 smithi086 ceph-mon[110975]: pgmap v1088: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:23.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:56:23.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.985+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b50066660 con 0x7f3b4006cc20 2024-04-06T14:56:23.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.986+0000 7f3b4d7fa700 1 -- 172.21.15.32:0/343699696 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3b50066660 con 0x7f3b4006cc20 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b4006cc20 msgr2=0x7f3b4006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b4006cc20 0x7f3b4006f0d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3b500668b0 tx=0x7f3b3c009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d10 msgr2=0x7f3b5010ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d10 0x7f3b5010ef40 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f3b44002a50 tx=0x7f3b440046b0 comp rx=0 tx=0).stop 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 shutdown_connections 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3b4006cc20 0x7f3b4006f0d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b50101d10 0x7f3b5010ef40 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 --2- 172.21.15.32:0/343699696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b501026b0 0x7f3b5010f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 >> 172.21.15.32:0/343699696 conn(0x7f3b500f7880 msgr2=0x7f3b500f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 shutdown_connections 2024-04-06T14:56:23.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:23.988+0000 7f3b57094700 1 -- 172.21.15.32:0/343699696 wait complete. 2024-04-06T14:56:23.985 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:56:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.092+0000 7fecdb34b700 1 -- 172.21.15.32:0/2010536027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4102710 msgr2=0x7fecd40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.092+0000 7fecdb34b700 1 --2- 172.21.15.32:0/2010536027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4102710 0x7fecd40fbd80 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fecc4004940 tx=0x7fecc4009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.092+0000 7fecdb34b700 1 -- 172.21.15.32:0/2010536027 shutdown_connections 2024-04-06T14:56:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.092+0000 7fecdb34b700 1 --2- 172.21.15.32:0/2010536027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4102710 0x7fecd40fbd80 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.092+0000 7fecdb34b700 1 --2- 172.21.15.32:0/2010536027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4101d70 0x7fecd4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.092+0000 7fecdb34b700 1 -- 172.21.15.32:0/2010536027 >> 172.21.15.32:0/2010536027 conn(0x7fecd4072bc0 msgr2=0x7fecd4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecdb34b700 1 -- 172.21.15.32:0/2010536027 shutdown_connections 2024-04-06T14:56:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecdb34b700 1 -- 172.21.15.32:0/2010536027 wait complete. 2024-04-06T14:56:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecdb34b700 1 Processor -- start 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecdb34b700 1 -- start start 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecdb34b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4101d70 0x7fecd4113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecdb34b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4102710 0x7fecd4114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecdb34b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd4114650 con 0x7fecd4101d70 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecdb34b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd410da40 con 0x7fecd4102710 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecd90e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4101d70 0x7fecd4113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecd90e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4101d70 0x7fecd4113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57818/0 (socket says 172.21.15.32:57818) 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.093+0000 7fecd90e7700 1 -- 172.21.15.32:0/2050516996 learned_addr learned my addr 172.21.15.32:0/2050516996 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecd88e6700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4102710 0x7fecd4114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecd90e7700 1 -- 172.21.15.32:0/2050516996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4102710 msgr2=0x7fecd4114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecd90e7700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4102710 0x7fecd4114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecd90e7700 1 -- 172.21.15.32:0/2050516996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc4005040 con 0x7fecd4101d70 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecd88e6700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4102710 0x7fecd4114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecd90e7700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4101d70 0x7fecd4113ac0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fecd000c8c0 tx=0x7fecd00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecc9ffb700 1 -- 172.21.15.32:0/2050516996 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecd000a760 con 0x7fecd4101d70 2024-04-06T14:56:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecc9ffb700 1 -- 172.21.15.32:0/2050516996 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecd000ad70 con 0x7fecd4101d70 2024-04-06T14:56:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd410dd20 con 0x7fecd4101d70 2024-04-06T14:56:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecc9ffb700 1 -- 172.21.15.32:0/2050516996 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecd00113a0 con 0x7fecd4101d70 2024-04-06T14:56:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.094+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd410e2b0 con 0x7fecd4101d70 2024-04-06T14:56:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.095+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd40fd4c0 con 0x7fecd4101d70 2024-04-06T14:56:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.096+0000 7fecc9ffb700 1 -- 172.21.15.32:0/2050516996 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fecd0012400 con 0x7fecd4101d70 2024-04-06T14:56:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.096+0000 7fecc9ffb700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecc006cc20 0x7fecc006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.096+0000 7fecc9ffb700 1 -- 172.21.15.32:0/2050516996 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fecd008bf70 con 0x7fecd4101d70 2024-04-06T14:56:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.096+0000 7fecd88e6700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecc006cc20 0x7fecc006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.097+0000 7fecd88e6700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecc006cc20 0x7fecc006f0d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fecd410ed20 tx=0x7fecc4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.101+0000 7fecc9ffb700 1 -- 172.21.15.32:0/2050516996 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fecd00bfab0 con 0x7fecd4101d70 2024-04-06T14:56:24.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.320+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fecd4108d10 con 0x7fecc006cc20 2024-04-06T14:56:24.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.321+0000 7fecc9ffb700 1 -- 172.21.15.32:0/2050516996 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fecd4108d10 con 0x7fecc006cc20 2024-04-06T14:56:24.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.323+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecc006cc20 msgr2=0x7fecc006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.323+0000 7fecdb34b700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecc006cc20 0x7fecc006f0d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fecd410ed20 tx=0x7fecc4009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:24.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.323+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4101d70 msgr2=0x7fecd4113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.323+0000 7fecdb34b700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4101d70 0x7fecd4113ac0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fecd000c8c0 tx=0x7fecd00044c0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.323+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 shutdown_connections 2024-04-06T14:56:24.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.323+0000 7fecdb34b700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fecc006cc20 0x7fecc006f0d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.323+0000 7fecdb34b700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd4101d70 0x7fecd4113ac0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.323+0000 7fecdb34b700 1 --2- 172.21.15.32:0/2050516996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4102710 0x7fecd4114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.323+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 >> 172.21.15.32:0/2050516996 conn(0x7fecd4072bc0 msgr2=0x7fecd40659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:24.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.324+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 shutdown_connections 2024-04-06T14:56:24.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.324+0000 7fecdb34b700 1 -- 172.21.15.32:0/2050516996 wait complete. 2024-04-06T14:56:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.427+0000 7f5550f13700 1 -- 172.21.15.32:0/2381436571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fe830 msgr2=0x7f554c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.427+0000 7f5550f13700 1 --2- 172.21.15.32:0/2381436571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fe830 0x7f554c106f00 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f5540004940 tx=0x7f5540009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.427+0000 7f5550f13700 1 -- 172.21.15.32:0/2381436571 shutdown_connections 2024-04-06T14:56:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.427+0000 7f5550f13700 1 --2- 172.21.15.32:0/2381436571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fe830 0x7f554c106f00 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.427+0000 7f5550f13700 1 --2- 172.21.15.32:0/2381436571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f554c0fde90 0x7f554c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.427+0000 7f5550f13700 1 -- 172.21.15.32:0/2381436571 >> 172.21.15.32:0/2381436571 conn(0x7f554c072bc0 msgr2=0x7f554c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.427+0000 7f5550f13700 1 -- 172.21.15.32:0/2381436571 shutdown_connections 2024-04-06T14:56:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.427+0000 7f5550f13700 1 -- 172.21.15.32:0/2381436571 wait complete. 2024-04-06T14:56:24.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f5550f13700 1 Processor -- start 2024-04-06T14:56:24.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f5550f13700 1 -- start start 2024-04-06T14:56:24.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f5550f13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fde90 0x7f554c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:24.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f5550f13700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f554c0fe830 0x7f554c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:24.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f5550f13700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c0fc820 con 0x7f554c0fde90 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f5550f13700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c0fc960 con 0x7f554c0fe830 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f554a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fde90 0x7f554c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f554a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fde90 0x7f554c0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43910/0 (socket says 172.21.15.32:43910) 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f554a59c700 1 -- 172.21.15.32:0/3881309891 learned_addr learned my addr 172.21.15.32:0/3881309891 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f554a59c700 1 -- 172.21.15.32:0/3881309891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f554c0fe830 msgr2=0x7f554c0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f554a59c700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f554c0fe830 0x7f554c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.428+0000 7f554a59c700 1 -- 172.21.15.32:0/3881309891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5540005040 con 0x7f554c0fde90 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.429+0000 7f554a59c700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fde90 0x7f554c0fbd50 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f553c002a80 tx=0x7f553c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.429+0000 7f553affd700 1 -- 172.21.15.32:0/3881309891 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f553c007cc0 con 0x7f554c0fde90 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.429+0000 7f553affd700 1 -- 172.21.15.32:0/3881309891 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f553c00f040 con 0x7f554c0fde90 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.429+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554c119d80 con 0x7f554c0fde90 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.429+0000 7f553affd700 1 -- 172.21.15.32:0/3881309891 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f553c009640 con 0x7f554c0fde90 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.429+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554c11a180 con 0x7f554c0fde90 2024-04-06T14:56:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.430+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554c065e30 con 0x7f554c0fde90 2024-04-06T14:56:24.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.430+0000 7f553affd700 1 -- 172.21.15.32:0/3881309891 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f553c0097a0 con 0x7f554c0fde90 2024-04-06T14:56:24.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.431+0000 7f553affd700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f553406cc30 0x7f553406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:24.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.431+0000 7f553affd700 1 -- 172.21.15.32:0/3881309891 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f553c08b770 con 0x7f554c0fde90 2024-04-06T14:56:24.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.431+0000 7f5549d9b700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f553406cc30 0x7f553406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:24.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.431+0000 7f5549d9b700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f553406cc30 0x7f553406f0e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5540004f90 tx=0x7f5540009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:24.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.435+0000 7f553affd700 1 -- 172.21.15.32:0/3881309891 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f553c0557b0 con 0x7f554c0fde90 2024-04-06T14:56:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.649+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f554c04bf60 con 0x7f553406cc30 2024-04-06T14:56:24.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.659+0000 7f553affd700 1 -- 172.21.15.32:0/3881309891 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f554c04bf60 con 0x7f553406cc30 2024-04-06T14:56:24.646 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:56:24.646 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (36m) 14m ago 48m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:56:24.646 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (34m) 14m ago 48m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:56:24.646 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (34m) 14m ago 47m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:56:24.646 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (36m) 14m ago 47m 89.3M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:56:24.646 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (30m) 14m ago 42m 18.2M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:56:24.646 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (30m) 14m ago 42m 20.2M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:56:24.646 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (29m) 14m ago 42m 17.0M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (14m) 14m ago 42m 11.2M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (38m) 14m ago 49m 512M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (37m) 14m ago 47m 456M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (35m) 14m ago 49m 70.3M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (34m) 14m ago 47m 49.9M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (37m) 14m ago 48m 16.9M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (37m) 14m ago 47m 18.4M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (33m) 14m ago 46m 87.2M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (33m) 14m ago 45m 87.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (32m) 14m ago 45m 82.6M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (31m) 14m ago 44m 96.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (31m) 14m ago 44m 95.1M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:56:24.647 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (30m) 14m ago 43m 85.8M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:56:24.648 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (37m) 14m ago 47m 76.3M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:56:24.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f553406cc30 msgr2=0x7f553406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f553406cc30 0x7f553406f0e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5540004f90 tx=0x7f5540009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:24.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fde90 msgr2=0x7f554c0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fde90 0x7f554c0fbd50 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f553c002a80 tx=0x7f553c0046b0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 shutdown_connections 2024-04-06T14:56:24.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f553406cc30 0x7f553406f0e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c0fde90 0x7f554c0fbd50 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 --2- 172.21.15.32:0/3881309891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f554c0fe830 0x7f554c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 >> 172.21.15.32:0/3881309891 conn(0x7f554c072bc0 msgr2=0x7f554c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:24.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 shutdown_connections 2024-04-06T14:56:24.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.661+0000 7f5550f13700 1 -- 172.21.15.32:0/3881309891 wait complete. 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.763+0000 7f38e9f90700 1 -- 172.21.15.32:0/1224041605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e41037a0 msgr2=0x7f38e40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.763+0000 7f38e9f90700 1 --2- 172.21.15.32:0/1224041605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e41037a0 0x7f38e40fce10 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f38cc004940 tx=0x7f38cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.763+0000 7f38e9f90700 1 -- 172.21.15.32:0/1224041605 shutdown_connections 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.763+0000 7f38e9f90700 1 --2- 172.21.15.32:0/1224041605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e41037a0 0x7f38e40fce10 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.763+0000 7f38e9f90700 1 --2- 172.21.15.32:0/1224041605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e4102e00 0x7f38e41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.763+0000 7f38e9f90700 1 -- 172.21.15.32:0/1224041605 >> 172.21.15.32:0/1224041605 conn(0x7f38e40f89b0 msgr2=0x7f38e40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.763+0000 7f38e9f90700 1 -- 172.21.15.32:0/1224041605 shutdown_connections 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.763+0000 7f38e9f90700 1 -- 172.21.15.32:0/1224041605 wait complete. 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.764+0000 7f38e9f90700 1 Processor -- start 2024-04-06T14:56:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.764+0000 7f38e9f90700 1 -- start start 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.764+0000 7f38e9f90700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e4102e00 0x7f38e4113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.764+0000 7f38e9f90700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e41037a0 0x7f38e4114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.764+0000 7f38e9f90700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e4114700 con 0x7f38e4102e00 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.764+0000 7f38e9f90700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e410da60 con 0x7f38e41037a0 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.764+0000 7f38e37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e4102e00 0x7f38e4113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.764+0000 7f38e37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e4102e00 0x7f38e4113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43922/0 (socket says 172.21.15.32:43922) 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.764+0000 7f38e37fe700 1 -- 172.21.15.32:0/127420145 learned_addr learned my addr 172.21.15.32:0/127420145 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38e2ffd700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e41037a0 0x7f38e4114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38e37fe700 1 -- 172.21.15.32:0/127420145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e41037a0 msgr2=0x7f38e4114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38e37fe700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e41037a0 0x7f38e4114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38e37fe700 1 -- 172.21.15.32:0/127420145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc005040 con 0x7f38e4102e00 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38e37fe700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e4102e00 0x7f38e4113ae0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00b790 tx=0x7f38dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38dbfff700 1 -- 172.21.15.32:0/127420145 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38dc00f770 con 0x7f38e4102e00 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38dbfff700 1 -- 172.21.15.32:0/127420145 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38dc00fd80 con 0x7f38e4102e00 2024-04-06T14:56:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e410dd40 con 0x7f38e4102e00 2024-04-06T14:56:24.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38dbfff700 1 -- 172.21.15.32:0/127420145 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38dc00d310 con 0x7f38e4102e00 2024-04-06T14:56:24.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.765+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e410e300 con 0x7f38e4102e00 2024-04-06T14:56:24.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.766+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e40fe550 con 0x7f38e4102e00 2024-04-06T14:56:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.766+0000 7f38dbfff700 1 -- 172.21.15.32:0/127420145 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f38dc010420 con 0x7f38e4102e00 2024-04-06T14:56:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.767+0000 7f38dbfff700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38d006cc20 0x7f38d006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.767+0000 7f38dbfff700 1 -- 172.21.15.32:0/127420145 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f38dc08c050 con 0x7f38e4102e00 2024-04-06T14:56:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.767+0000 7f38e2ffd700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38d006cc20 0x7f38d006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.767+0000 7f38e2ffd700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38d006cc20 0x7f38d006f0d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f38e410ec20 tx=0x7f38cc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:24.772+0000 7f38dbfff700 1 -- 172.21.15.32:0/127420145 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f38dc059b20 con 0x7f38e4102e00 2024-04-06T14:56:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.059+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f38e404bed0 con 0x7f38e4102e00 2024-04-06T14:56:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.059+0000 7f38dbfff700 1 -- 172.21.15.32:0/127420145 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f38dc0596b0 con 0x7f38e4102e00 2024-04-06T14:56:25.046 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:56:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:56:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:56:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:56:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:56:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:56:25.047 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38d006cc20 msgr2=0x7f38d006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38d006cc20 0x7f38d006f0d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f38e410ec20 tx=0x7f38cc009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e4102e00 msgr2=0x7f38e4113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e4102e00 0x7f38e4113ae0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00b790 tx=0x7f38dc009250 comp rx=0 tx=0).stop 2024-04-06T14:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 shutdown_connections 2024-04-06T14:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f38d006cc20 0x7f38d006f0d0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38e4102e00 0x7f38e4113ae0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 --2- 172.21.15.32:0/127420145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e41037a0 0x7f38e4114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 >> 172.21.15.32:0/127420145 conn(0x7f38e40f89b0 msgr2=0x7f38e40f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 shutdown_connections 2024-04-06T14:56:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.061+0000 7f38e9f90700 1 -- 172.21.15.32:0/127420145 wait complete. 2024-04-06T14:56:25.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.164+0000 7f3634796700 1 -- 172.21.15.32:0/1062716315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 msgr2=0x7f362c106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.164+0000 7f3634796700 1 --2- 172.21.15.32:0/1062716315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 0x7f362c106e70 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3620004940 tx=0x7f3620009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.164+0000 7f3634796700 1 -- 172.21.15.32:0/1062716315 shutdown_connections 2024-04-06T14:56:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.164+0000 7f3634796700 1 --2- 172.21.15.32:0/1062716315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 0x7f362c106e70 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.164+0000 7f3634796700 1 --2- 172.21.15.32:0/1062716315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f362c0fde00 0x7f362c0fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.164+0000 7f3634796700 1 -- 172.21.15.32:0/1062716315 >> 172.21.15.32:0/1062716315 conn(0x7f362c0f7880 msgr2=0x7f362c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.164+0000 7f3634796700 1 -- 172.21.15.32:0/1062716315 shutdown_connections 2024-04-06T14:56:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.164+0000 7f3634796700 1 -- 172.21.15.32:0/1062716315 wait complete. 2024-04-06T14:56:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.165+0000 7f3634796700 1 Processor -- start 2024-04-06T14:56:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.165+0000 7f3634796700 1 -- start start 2024-04-06T14:56:25.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.165+0000 7f3634796700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f362c0fde00 0x7f362c113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:25.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3634796700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 0x7f362c114260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:25.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3634796700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c1148b0 con 0x7f362c0fe7a0 2024-04-06T14:56:25.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3634796700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c10dcd0 con 0x7f362c0fde00 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3631d31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 0x7f362c114260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3631d31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 0x7f362c114260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43926/0 (socket says 172.21.15.32:43926) 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3631d31700 1 -- 172.21.15.32:0/137141510 learned_addr learned my addr 172.21.15.32:0/137141510 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3632532700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f362c0fde00 0x7f362c113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3631d31700 1 -- 172.21.15.32:0/137141510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f362c0fde00 msgr2=0x7f362c113d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3631d31700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f362c0fde00 0x7f362c113d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3631d31700 1 -- 172.21.15.32:0/137141510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3620005040 con 0x7f362c0fe7a0 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3631d31700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 0x7f362c114260 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3620000c00 tx=0x7f3620009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f361effd700 1 -- 172.21.15.32:0/137141510 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3620021070 con 0x7f362c0fe7a0 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f361effd700 1 -- 172.21.15.32:0/137141510 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3620013e00 con 0x7f362c0fe7a0 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362c10df50 con 0x7f362c0fe7a0 2024-04-06T14:56:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f361effd700 1 -- 172.21.15.32:0/137141510 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f362000e070 con 0x7f362c0fe7a0 2024-04-06T14:56:25.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.166+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362c10e490 con 0x7f362c0fe7a0 2024-04-06T14:56:25.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.167+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362c0666f0 con 0x7f362c0fe7a0 2024-04-06T14:56:25.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.168+0000 7f361effd700 1 -- 172.21.15.32:0/137141510 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f362001c590 con 0x7f362c0fe7a0 2024-04-06T14:56:25.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.168+0000 7f361effd700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f361806cc20 0x7f361806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:25.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.168+0000 7f361effd700 1 -- 172.21.15.32:0/137141510 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f362008d120 con 0x7f362c0fe7a0 2024-04-06T14:56:25.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.169+0000 7f3632532700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f361806cc20 0x7f361806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:25.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.169+0000 7f3632532700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f361806cc20 0x7f361806f0d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3628009b80 tx=0x7f362800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:25.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.173+0000 7f361effd700 1 -- 172.21.15.32:0/137141510 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f362005abf0 con 0x7f362c0fe7a0 2024-04-06T14:56:25.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.417+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f362c109d50 con 0x7f362c0fe7a0 2024-04-06T14:56:25.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.418+0000 7f361effd700 1 -- 172.21.15.32:0/137141510 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1836 (secure 0 0 0) 0x7f362005a780 con 0x7f362c0fe7a0 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:e32 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 23 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:25:48.052931+0000 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:56:25.405 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:56:25.406 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{-1:34382} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:56:25.407 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:56:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f361806cc20 msgr2=0x7f361806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f361806cc20 0x7f361806f0d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3628009b80 tx=0x7f362800b040 comp rx=0 tx=0).stop 2024-04-06T14:56:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 msgr2=0x7f362c114260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 0x7f362c114260 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3620000c00 tx=0x7f3620009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 shutdown_connections 2024-04-06T14:56:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f361806cc20 0x7f361806f0d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f362c0fde00 0x7f362c113d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 --2- 172.21.15.32:0/137141510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0fe7a0 0x7f362c114260 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 >> 172.21.15.32:0/137141510 conn(0x7f362c0f7880 msgr2=0x7f362c0f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 shutdown_connections 2024-04-06T14:56:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.420+0000 7f3634796700 1 -- 172.21.15.32:0/137141510 wait complete. 2024-04-06T14:56:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:56:25.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 -- 172.21.15.32:0/277323683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24102710 msgr2=0x7f5f240fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 --2- 172.21.15.32:0/277323683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24102710 0x7f5f240fbd80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f5f14004940 tx=0x7f5f14009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:25.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 -- 172.21.15.32:0/277323683 shutdown_connections 2024-04-06T14:56:25.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 --2- 172.21.15.32:0/277323683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24102710 0x7f5f240fbd80 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 --2- 172.21.15.32:0/277323683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f24101d70 0x7f5f24102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 -- 172.21.15.32:0/277323683 >> 172.21.15.32:0/277323683 conn(0x7f5f24072bc0 msgr2=0x7f5f24072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:25.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 -- 172.21.15.32:0/277323683 shutdown_connections 2024-04-06T14:56:25.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 -- 172.21.15.32:0/277323683 wait complete. 2024-04-06T14:56:25.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 Processor -- start 2024-04-06T14:56:25.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.525+0000 7f5f28cfd700 1 -- start start 2024-04-06T14:56:25.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f28cfd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24101d70 0x7f5f2410ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:25.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f28cfd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f24102710 0x7f5f2410efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:25.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f28cfd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2410f6b0 con 0x7f5f24101d70 2024-04-06T14:56:25.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f28cfd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2410f7f0 con 0x7f5f24102710 2024-04-06T14:56:25.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f2259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24101d70 0x7f5f2410ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f2259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24101d70 0x7f5f2410ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43942/0 (socket says 172.21.15.32:43942) 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f2259c700 1 -- 172.21.15.32:0/2715009321 learned_addr learned my addr 172.21.15.32:0/2715009321 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f21d9b700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f24102710 0x7f5f2410efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f2259c700 1 -- 172.21.15.32:0/2715009321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f24102710 msgr2=0x7f5f2410efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f2259c700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f24102710 0x7f5f2410efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f2259c700 1 -- 172.21.15.32:0/2715009321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f14005040 con 0x7f5f24101d70 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f21d9b700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f24102710 0x7f5f2410efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f2259c700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24101d70 0x7f5f2410ea90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800b790 tx=0x7f5f18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f12ffd700 1 -- 172.21.15.32:0/2715009321 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f1800f770 con 0x7f5f24101d70 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f12ffd700 1 -- 172.21.15.32:0/2715009321 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f1800fd80 con 0x7f5f24101d70 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f24117b60 con 0x7f5f24101d70 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.526+0000 7f5f12ffd700 1 -- 172.21.15.32:0/2715009321 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f1800d310 con 0x7f5f24101d70 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.527+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f241180f0 con 0x7f5f24101d70 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.527+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f240ff080 con 0x7f5f24101d70 2024-04-06T14:56:25.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.528+0000 7f5f12ffd700 1 -- 172.21.15.32:0/2715009321 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5f18010420 con 0x7f5f24101d70 2024-04-06T14:56:25.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.528+0000 7f5f12ffd700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f0c06cc30 0x7f5f0c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:25.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.528+0000 7f5f12ffd700 1 -- 172.21.15.32:0/2715009321 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5f1808c070 con 0x7f5f24101d70 2024-04-06T14:56:25.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.528+0000 7f5f21d9b700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f0c06cc30 0x7f5f0c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:25.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.529+0000 7f5f21d9b700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f0c06cc30 0x7f5f0c06f0e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5f24066000 tx=0x7f5f14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:25.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.533+0000 7f5f12ffd700 1 -- 172.21.15.32:0/2715009321 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5f180560b0 con 0x7f5f24101d70 2024-04-06T14:56:25.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:25 smithi032 ceph-mon[147414]: from='client.35962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:25.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:25 smithi032 ceph-mon[147414]: from='client.35966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:25.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:25 smithi032 ceph-mon[147414]: pgmap v1089: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:25.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:25 smithi032 ceph-mon[147414]: from='client.35970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:25.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:25 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/127420145' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:25.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:25 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/137141510' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:56:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.754+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f2404bed0 con 0x7f5f0c06cc30 2024-04-06T14:56:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.755+0000 7f5f12ffd700 1 -- 172.21.15.32:0/2715009321 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5f2404bed0 con 0x7f5f0c06cc30 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e", 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "mds", 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-06T14:56:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-06T14:56:25.743 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-06T14:56:25.743 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T14:56:25.743 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:56:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f0c06cc30 msgr2=0x7f5f0c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f0c06cc30 0x7f5f0c06f0e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5f24066000 tx=0x7f5f14009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24101d70 msgr2=0x7f5f2410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24101d70 0x7f5f2410ea90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800b790 tx=0x7f5f18009250 comp rx=0 tx=0).stop 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 shutdown_connections 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f5f0c06cc30 0x7f5f0c06f0e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f24101d70 0x7f5f2410ea90 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 --2- 172.21.15.32:0/2715009321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f24102710 0x7f5f2410efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 >> 172.21.15.32:0/2715009321 conn(0x7f5f24072bc0 msgr2=0x7f5f240fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 shutdown_connections 2024-04-06T14:56:25.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.757+0000 7f5f28cfd700 1 -- 172.21.15.32:0/2715009321 wait complete. 2024-04-06T14:56:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.860+0000 7fc8a9017700 1 -- 172.21.15.32:0/890934471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a41037b0 msgr2=0x7fc8a40fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.860+0000 7fc8a9017700 1 --2- 172.21.15.32:0/890934471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a41037b0 0x7fc8a40fce20 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fc898004940 tx=0x7fc898009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.860+0000 7fc8a9017700 1 -- 172.21.15.32:0/890934471 shutdown_connections 2024-04-06T14:56:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.860+0000 7fc8a9017700 1 --2- 172.21.15.32:0/890934471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a41037b0 0x7fc8a40fce20 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 --2- 172.21.15.32:0/890934471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a4102e10 0x7fc8a41031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 -- 172.21.15.32:0/890934471 >> 172.21.15.32:0/890934471 conn(0x7fc8a40f89a0 msgr2=0x7fc8a40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 -- 172.21.15.32:0/890934471 shutdown_connections 2024-04-06T14:56:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 -- 172.21.15.32:0/890934471 wait complete. 2024-04-06T14:56:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 Processor -- start 2024-04-06T14:56:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 -- start start 2024-04-06T14:56:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a4102e10 0x7fc8a40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a41037b0 0x7fc8a4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a4070d90 con 0x7fc8a4102e10 2024-04-06T14:56:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.861+0000 7fc8a9017700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a4070ed0 con 0x7fc8a41037b0 2024-04-06T14:56:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a4102e10 0x7fc8a40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a4102e10 0x7fc8a40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43966/0 (socket says 172.21.15.32:43966) 2024-04-06T14:56:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a2d9d700 1 -- 172.21.15.32:0/3228539470 learned_addr learned my addr 172.21.15.32:0/3228539470 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a259c700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a41037b0 0x7fc8a4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a2d9d700 1 -- 172.21.15.32:0/3228539470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a41037b0 msgr2=0x7fc8a4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a2d9d700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a41037b0 0x7fc8a4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a2d9d700 1 -- 172.21.15.32:0/3228539470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc898005040 con 0x7fc8a4102e10 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a259c700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a41037b0 0x7fc8a4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a2d9d700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a4102e10 0x7fc8a40702c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc894002a50 tx=0x7fc8940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc88b7fe700 1 -- 172.21.15.32:0/3228539470 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc894007cc0 con 0x7fc8a4102e10 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a4119dd0 con 0x7fc8a4102e10 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc88b7fe700 1 -- 172.21.15.32:0/3228539470 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc89400f040 con 0x7fc8a4102e10 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc88b7fe700 1 -- 172.21.15.32:0/3228539470 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc894009540 con 0x7fc8a4102e10 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.862+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a411a250 con 0x7fc8a4102e10 2024-04-06T14:56:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.863+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a40fe560 con 0x7fc8a4102e10 2024-04-06T14:56:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.864+0000 7fc88b7fe700 1 -- 172.21.15.32:0/3228539470 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc8940096a0 con 0x7fc8a4102e10 2024-04-06T14:56:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.864+0000 7fc88b7fe700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc88c06cc20 0x7fc88c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.864+0000 7fc88b7fe700 1 -- 172.21.15.32:0/3228539470 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc89408be60 con 0x7fc8a4102e10 2024-04-06T14:56:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.864+0000 7fc8a259c700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc88c06cc20 0x7fc88c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:25.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.865+0000 7fc8a259c700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc88c06cc20 0x7fc88c06f0d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc89800b3b0 tx=0x7fc898009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:25.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:25.869+0000 7fc88b7fe700 1 -- 172.21.15.32:0/3228539470 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc894055e20 con 0x7fc8a4102e10 2024-04-06T14:56:25.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:25 smithi086 ceph-mon[110975]: from='client.35962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:25.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:25 smithi086 ceph-mon[110975]: from='client.35966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:25.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:25 smithi086 ceph-mon[110975]: pgmap v1089: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:25.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:25 smithi086 ceph-mon[110975]: from='client.35970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:25.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:25 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/127420145' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:25.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:25 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/137141510' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:56:26.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.157+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc8a405f5f0 con 0x7fc8a4102e10 2024-04-06T14:56:26.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.158+0000 7fc88b7fe700 1 -- 172.21.15.32:0/3228539470 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3773 (secure 0 0 0) 0x7fc8a405f5f0 con 0x7fc8a4102e10 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: host smithi086 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi086/config 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout:Traceback (most recent call last): 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "__main__", mod_spec) 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: exec(code, run_globals) 2024-04-06T14:56:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10889, in 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 10877, in main 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: File "/tmp/tmpmhzvx2_d.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnxqqnfa9:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:45cbf7e6d38f79ab874661480485370918610a7e9a2761c03c5b2e736405702e inventory --format=json-pretty --filter-for-batch 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:56:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:56:26.147 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-06T14:56:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-06T14:56:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc88c06cc20 msgr2=0x7fc88c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc88c06cc20 0x7fc88c06f0d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc89800b3b0 tx=0x7fc898009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a4102e10 msgr2=0x7fc8a40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a4102e10 0x7fc8a40702c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc894002a50 tx=0x7fc8940046b0 comp rx=0 tx=0).stop 2024-04-06T14:56:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 shutdown_connections 2024-04-06T14:56:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc88c06cc20 0x7fc88c06f0d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a4102e10 0x7fc8a40702c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 --2- 172.21.15.32:0/3228539470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a41037b0 0x7fc8a4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 >> 172.21.15.32:0/3228539470 conn(0x7fc8a40f89a0 msgr2=0x7fc8a40f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 shutdown_connections 2024-04-06T14:56:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:26.160+0000 7fc8a9017700 1 -- 172.21.15.32:0/3228539470 wait complete. 2024-04-06T14:56:26.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:26 smithi032 ceph-mon[147414]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:26.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:26 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3228539470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:56:26.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:26 smithi086 ceph-mon[110975]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:26.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:26 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3228539470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:56:27.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:27 smithi032 ceph-mon[147414]: pgmap v1090: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:27.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:27 smithi086 ceph-mon[110975]: pgmap v1090: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:29.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:29 smithi032 ceph-mon[147414]: pgmap v1091: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:29.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:29 smithi086 ceph-mon[110975]: pgmap v1091: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:31.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:31 smithi032 ceph-mon[147414]: pgmap v1092: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:31.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:31 smithi086 ceph-mon[110975]: pgmap v1092: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:33.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:33 smithi032 ceph-mon[147414]: pgmap v1093: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:33.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:33 smithi086 ceph-mon[110975]: pgmap v1093: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:35.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:35 smithi032 ceph-mon[147414]: pgmap v1094: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:35.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:35 smithi086 ceph-mon[110975]: pgmap v1094: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:36.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:36 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:56:36.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:36 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:56:36.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:36 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:36.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:36 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:56:36.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:36 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:56:36.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:36 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:37.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: pgmap v1095: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: pgmap v1096: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: pgmap v1097: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all mds 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.gcjdjo"}]': finished 2024-04-06T14:56:37.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:56:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.mqoaof"}]': finished 2024-04-06T14:56:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:56:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.nikfjj"}]': finished 2024-04-06T14:56:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:56:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.ojmchr"}]': finished 2024-04-06T14:56:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: Upgrade: Setting filesystem cephfs Joinable 2024-04-06T14:56:37.718 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-06T14:56:37.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: pgmap v1095: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:37.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: pgmap v1096: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:37.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: pgmap v1097: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:56:37.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:56:37.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all mds 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.gcjdjo"}]: dispatch 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.gcjdjo"}]': finished 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.mqoaof"}]: dispatch 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.mqoaof"}]': finished 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.nikfjj"}]: dispatch 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.nikfjj"}]': finished 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.ojmchr"}]: dispatch 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.ojmchr"}]': finished 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: Upgrade: Setting filesystem cephfs Joinable 2024-04-06T14:56:37.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-06T14:56:38.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-06T14:56:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Standby daemon mds.cephfs.smithi032.gcjdjo assigned to filesystem cephfs as rank 0 2024-04-06T14:56:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Standby daemon mds.cephfs.smithi032.mqoaof assigned to filesystem cephfs as rank 1 2024-04-06T14:56:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T14:56:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:56:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:replay,1=cephfs.smithi032.mqoaof=up:replay} 2 up:standby 2024-04-06T14:56:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all rgw 2024-04-06T14:56:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all rbd-mirror 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all ceph-exporter 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all iscsi 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all nfs 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.866 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Upgrade: Setting container_image for all nvmeof 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Upgrade: Finalizing container_image settings 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-06T14:56:38.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-06T14:56:38.868 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-06T14:56:38.869 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-06T14:56:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: Upgrade: Complete! 2024-04-06T14:56:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-06T14:56:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-06T14:56:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:56:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.870 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:38 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.929 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-06T14:56:38.929 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Standby daemon mds.cephfs.smithi032.gcjdjo assigned to filesystem cephfs as rank 0 2024-04-06T14:56:38.929 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Standby daemon mds.cephfs.smithi032.mqoaof assigned to filesystem cephfs as rank 1 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:replay,1=cephfs.smithi032.mqoaof=up:replay} 2 up:standby 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all rgw 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all rbd-mirror 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all ceph-exporter 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.930 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all iscsi 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all nfs 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Upgrade: Setting container_image for all nvmeof 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.931 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.932 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.932 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:56:38.932 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Upgrade: Finalizing container_image settings 2024-04-06T14:56:38.932 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-06T14:56:38.932 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-06T14:56:38.932 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.932 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-06T14:56:38.932 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-06T14:56:38.933 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: Upgrade: Complete! 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-06T14:56:38.934 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:56:38.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:38.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:38 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:39.835 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:39 smithi032 ceph-mon[147414]: pgmap v1098: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 0 op/s 2024-04-06T14:56:39.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:39 smithi086 ceph-mon[110975]: pgmap v1098: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 0 op/s 2024-04-06T14:56:41.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:41.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:41 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:41.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:41 smithi086 ceph-mon[110975]: pgmap v1099: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 5 op/s 2024-04-06T14:56:41.466 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:41.466 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:41 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:41.466 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:41 smithi032 ceph-mon[147414]: pgmap v1099: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 5 op/s 2024-04-06T14:56:43.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:43 smithi032 ceph-mon[147414]: pgmap v1100: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 8 op/s 2024-04-06T14:56:43.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:43 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:43.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:43 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:43.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:43 smithi086 ceph-mon[110975]: pgmap v1100: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 8 op/s 2024-04-06T14:56:43.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:43 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:43.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:43 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:45.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:45 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:45.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:45 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:45.364 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:45 smithi086 ceph-mon[110975]: pgmap v1101: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 32 MiB/s rd, 11 op/s 2024-04-06T14:56:45.466 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:45 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:45.466 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:45 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:45.466 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:45 smithi032 ceph-mon[147414]: pgmap v1101: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 32 MiB/s rd, 11 op/s 2024-04-06T14:56:46.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:46 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:resolve 2024-04-06T14:56:46.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:46 smithi032 ceph-mon[147414]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:resolve,1=cephfs.smithi032.mqoaof=up:replay} 2 up:standby 2024-04-06T14:56:46.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:46 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:resolve 2024-04-06T14:56:46.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:46 smithi086 ceph-mon[110975]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:resolve,1=cephfs.smithi032.mqoaof=up:replay} 2 up:standby 2024-04-06T14:56:47.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:47 smithi032 ceph-mon[147414]: pgmap v1102: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 35 MiB/s rd, 11 op/s 2024-04-06T14:56:47.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:47 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:resolve 2024-04-06T14:56:47.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:47 smithi032 ceph-mon[147414]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:resolve,1=cephfs.smithi032.mqoaof=up:resolve} 2 up:standby 2024-04-06T14:56:47.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:47 smithi086 ceph-mon[110975]: pgmap v1102: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 35 MiB/s rd, 11 op/s 2024-04-06T14:56:47.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:47 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:resolve 2024-04-06T14:56:47.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:47 smithi086 ceph-mon[110975]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:resolve,1=cephfs.smithi032.mqoaof=up:resolve} 2 up:standby 2024-04-06T14:56:48.514 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:48 smithi032 ceph-mon[147414]: reconnect by client.24357 192.168.0.1:0/3886073944 after 0.000999979 2024-04-06T14:56:48.514 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:48 smithi032 ceph-mon[147414]: reconnect by client.24357 192.168.0.1:0/3886073944 after 0.000999979 2024-04-06T14:56:48.514 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:48 smithi032 ceph-mon[147414]: reconnect by client.14516 192.168.0.1:0/714996764 after 0.000999979 2024-04-06T14:56:48.514 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:48 smithi032 ceph-mon[147414]: reconnect by client.14516 192.168.0.1:0/714996764 after 0.000999979 2024-04-06T14:56:48.515 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:48 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:reconnect 2024-04-06T14:56:48.515 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:48 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:reconnect 2024-04-06T14:56:48.515 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:48 smithi032 ceph-mon[147414]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:reconnect,1=cephfs.smithi032.mqoaof=up:reconnect} 2 up:standby 2024-04-06T14:56:48.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:48 smithi086 ceph-mon[110975]: reconnect by client.24357 192.168.0.1:0/3886073944 after 0.000999979 2024-04-06T14:56:48.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:48 smithi086 ceph-mon[110975]: reconnect by client.24357 192.168.0.1:0/3886073944 after 0.000999979 2024-04-06T14:56:48.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:48 smithi086 ceph-mon[110975]: reconnect by client.14516 192.168.0.1:0/714996764 after 0.000999979 2024-04-06T14:56:48.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:48 smithi086 ceph-mon[110975]: reconnect by client.14516 192.168.0.1:0/714996764 after 0.000999979 2024-04-06T14:56:48.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:48 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:reconnect 2024-04-06T14:56:48.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:48 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:reconnect 2024-04-06T14:56:48.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:48 smithi086 ceph-mon[110975]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:reconnect,1=cephfs.smithi032.mqoaof=up:reconnect} 2 up:standby 2024-04-06T14:56:49.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:49 smithi032 ceph-mon[147414]: pgmap v1103: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 29 MiB/s rd, 9 op/s 2024-04-06T14:56:49.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:49 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:rejoin 2024-04-06T14:56:49.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:49 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:rejoin 2024-04-06T14:56:49.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:49 smithi032 ceph-mon[147414]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:rejoin,1=cephfs.smithi032.mqoaof=up:rejoin} 2 up:standby 2024-04-06T14:56:49.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:49 smithi032 ceph-mon[147414]: daemon mds.cephfs.smithi032.mqoaof is now active in filesystem cephfs as rank 1 2024-04-06T14:56:49.717 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:49 smithi032 ceph-mon[147414]: daemon mds.cephfs.smithi032.gcjdjo is now active in filesystem cephfs as rank 0 2024-04-06T14:56:49.790 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:49 smithi086 ceph-mon[110975]: pgmap v1103: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 29 MiB/s rd, 9 op/s 2024-04-06T14:56:49.790 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:49 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:rejoin 2024-04-06T14:56:49.790 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:49 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:rejoin 2024-04-06T14:56:49.790 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:49 smithi086 ceph-mon[110975]: fsmap cephfs:2/2 {0=cephfs.smithi032.gcjdjo=up:rejoin,1=cephfs.smithi032.mqoaof=up:rejoin} 2 up:standby 2024-04-06T14:56:49.790 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:49 smithi086 ceph-mon[110975]: daemon mds.cephfs.smithi032.mqoaof is now active in filesystem cephfs as rank 1 2024-04-06T14:56:49.790 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:49 smithi086 ceph-mon[110975]: daemon mds.cephfs.smithi032.gcjdjo is now active in filesystem cephfs as rank 0 2024-04-06T14:56:50.662 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:50 smithi032 ceph-mon[147414]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T14:56:50.663 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:50 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:active 2024-04-06T14:56:50.663 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:50 smithi032 ceph-mon[147414]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:active 2024-04-06T14:56:50.663 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:50 smithi032 ceph-mon[147414]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi032.mqoaof=up:active} 2 up:standby 2024-04-06T14:56:50.663 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:50.663 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:50 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:50.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:50 smithi086 ceph-mon[110975]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T14:56:50.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:50 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] up:active 2024-04-06T14:56:50.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:50 smithi086 ceph-mon[110975]: mds.? [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] up:active 2024-04-06T14:56:50.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:50 smithi086 ceph-mon[110975]: fsmap cephfs:2 {0=cephfs.smithi032.gcjdjo=up:active,1=cephfs.smithi032.mqoaof=up:active} 2 up:standby 2024-04-06T14:56:50.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:50.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:50 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:51.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: pgmap v1104: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 4.5 KiB/s wr, 13 op/s 2024-04-06T14:56:51.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:51.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:51.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:56:51.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:56:51.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: pgmap v1105: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-04-06T14:56:51.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:51.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:56:51.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:56:51.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:56:51.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:51.865 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:51 smithi086 ceph-mon[110975]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-06T14:56:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: pgmap v1104: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 4.5 KiB/s wr, 13 op/s 2024-04-06T14:56:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:56:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:56:52.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: pgmap v1105: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-04-06T14:56:52.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:52.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:56:52.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:56:52.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:56:52.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:52.217 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:51 smithi032 ceph-mon[147414]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-06T14:56:53.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:53.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:56:53.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:53 smithi032 ceph-mon[147414]: pgmap v1106: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-04-06T14:56:54.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:56:54.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:56:54.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:53 smithi086 ceph-mon[110975]: pgmap v1106: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-04-06T14:56:55.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:55 smithi032 ceph-mon[147414]: pgmap v1107: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 8.5 MiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-06T14:56:55.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:55 smithi086 ceph-mon[110975]: pgmap v1107: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 8.5 MiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-06T14:56:56.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.280+0000 7f1360816700 1 -- 172.21.15.32:0/30580337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fe810 msgr2=0x7f135c106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.280+0000 7f1360816700 1 --2- 172.21.15.32:0/30580337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fe810 0x7f135c106e50 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f1350004940 tx=0x7f1350009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.280+0000 7f1360816700 1 -- 172.21.15.32:0/30580337 shutdown_connections 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.280+0000 7f1360816700 1 --2- 172.21.15.32:0/30580337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fe810 0x7f135c106e50 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.280+0000 7f1360816700 1 --2- 172.21.15.32:0/30580337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f135c0fde70 0x7f135c0fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.280+0000 7f1360816700 1 -- 172.21.15.32:0/30580337 >> 172.21.15.32:0/30580337 conn(0x7f135c0f7880 msgr2=0x7f135c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.280+0000 7f1360816700 1 -- 172.21.15.32:0/30580337 shutdown_connections 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.280+0000 7f1360816700 1 -- 172.21.15.32:0/30580337 wait complete. 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f1360816700 1 Processor -- start 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f1360816700 1 -- start start 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f1360816700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fde70 0x7f135c0fbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f1360816700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f135c0fe810 0x7f135c0fc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f1360816700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0fc770 con 0x7f135c0fde70 2024-04-06T14:56:56.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f1360816700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0fc8b0 con 0x7f135c0fe810 2024-04-06T14:56:56.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f135a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fde70 0x7f135c0fbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:56.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f135a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fde70 0x7f135c0fbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58246/0 (socket says 172.21.15.32:58246) 2024-04-06T14:56:56.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f135a59c700 1 -- 172.21.15.32:0/3417225879 learned_addr learned my addr 172.21.15.32:0/3417225879 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:56:56.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f1359d9b700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f135c0fe810 0x7f135c0fc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f135a59c700 1 -- 172.21.15.32:0/3417225879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f135c0fe810 msgr2=0x7f135c0fc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.281+0000 7f135a59c700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f135c0fe810 0x7f135c0fc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.282+0000 7f135a59c700 1 -- 172.21.15.32:0/3417225879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1350005040 con 0x7f135c0fde70 2024-04-06T14:56:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.282+0000 7f135a59c700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fde70 0x7f135c0fbca0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f134c00eb70 tx=0x7f134c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:56.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.282+0000 7f134affd700 1 -- 172.21.15.32:0/3417225879 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f134c007cc0 con 0x7f135c0fde70 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.282+0000 7f134affd700 1 -- 172.21.15.32:0/3417225879 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f134c00f040 con 0x7f135c0fde70 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.282+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c0fcb00 con 0x7f135c0fde70 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.282+0000 7f134affd700 1 -- 172.21.15.32:0/3417225879 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f134c0095f0 con 0x7f135c0fde70 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.282+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c11a460 con 0x7f135c0fde70 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.283+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c066710 con 0x7f135c0fde70 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.283+0000 7f134affd700 1 -- 172.21.15.32:0/3417225879 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f134c009750 con 0x7f135c0fde70 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.284+0000 7f134affd700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f134406cc30 0x7f134406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.284+0000 7f134affd700 1 -- 172.21.15.32:0/3417225879 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f134c08bcd0 con 0x7f135c0fde70 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.284+0000 7f1359d9b700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f134406cc30 0x7f134406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:56.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.284+0000 7f1359d9b700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f134406cc30 0x7f134406f0e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f135000b0d0 tx=0x7f1350009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:56.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.289+0000 7f134affd700 1 -- 172.21.15.32:0/3417225879 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f134c055c90 con 0x7f135c0fde70 2024-04-06T14:56:56.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.512+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f135c04bf40 con 0x7f134406cc30 2024-04-06T14:56:56.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.513+0000 7f134affd700 1 -- 172.21.15.32:0/3417225879 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f135c04bf40 con 0x7f134406cc30 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f134406cc30 msgr2=0x7f134406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f134406cc30 0x7f134406f0e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f135000b0d0 tx=0x7f1350009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fde70 msgr2=0x7f135c0fbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fde70 0x7f135c0fbca0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f134c00eb70 tx=0x7f134c00b040 comp rx=0 tx=0).stop 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 shutdown_connections 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f134406cc30 0x7f134406f0e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f135c0fde70 0x7f135c0fbca0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 --2- 172.21.15.32:0/3417225879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f135c0fe810 0x7f135c0fc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 >> 172.21.15.32:0/3417225879 conn(0x7f135c0f7880 msgr2=0x7f135c0f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 shutdown_connections 2024-04-06T14:56:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:56.515+0000 7f1360816700 1 -- 172.21.15.32:0/3417225879 wait complete. 2024-04-06T14:56:57.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:56 smithi086 ceph-mon[110975]: from='client.35990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:57.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:56 smithi086 ceph-mon[110975]: pgmap v1108: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 7.3 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-04-06T14:56:57.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:56 smithi032 ceph-mon[147414]: from='client.35990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:56:57.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:56 smithi032 ceph-mon[147414]: pgmap v1108: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 7.3 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-04-06T14:56:57.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph orch ps' 2024-04-06T14:56:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:56:59.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:56:58 smithi086 ceph-mon[110975]: pgmap v1109: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 7.3 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-04-06T14:56:59.115 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:56:58 smithi032 ceph-mon[147414]: pgmap v1109: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 7.3 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-04-06T14:56:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.381+0000 7f6b73fbe700 1 -- 172.21.15.32:0/3312870964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 msgr2=0x7f6b6c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.381+0000 7f6b73fbe700 1 --2- 172.21.15.32:0/3312870964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 0x7f6b6c102140 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f6b60004940 tx=0x7f6b60009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.381+0000 7f6b73fbe700 1 -- 172.21.15.32:0/3312870964 shutdown_connections 2024-04-06T14:56:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.381+0000 7f6b73fbe700 1 --2- 172.21.15.32:0/3312870964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6c102710 0x7f6b6c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.381+0000 7f6b73fbe700 1 --2- 172.21.15.32:0/3312870964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 0x7f6b6c102140 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.381+0000 7f6b73fbe700 1 -- 172.21.15.32:0/3312870964 >> 172.21.15.32:0/3312870964 conn(0x7f6b6c072bc0 msgr2=0x7f6b6c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.381+0000 7f6b73fbe700 1 -- 172.21.15.32:0/3312870964 shutdown_connections 2024-04-06T14:56:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.381+0000 7f6b73fbe700 1 -- 172.21.15.32:0/3312870964 wait complete. 2024-04-06T14:56:59.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b73fbe700 1 Processor -- start 2024-04-06T14:56:59.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b73fbe700 1 -- start start 2024-04-06T14:56:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b73fbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 0x7f6b6c10eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b73fbe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6c102710 0x7f6b6c10f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b73fbe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c10f620 con 0x7f6b6c101d70 2024-04-06T14:56:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b73fbe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c10f760 con 0x7f6b6c102710 2024-04-06T14:56:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b71d5a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 0x7f6b6c10eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b71d5a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 0x7f6b6c10eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58276/0 (socket says 172.21.15.32:58276) 2024-04-06T14:56:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b71d5a700 1 -- 172.21.15.32:0/135439 learned_addr learned my addr 172.21.15.32:0/135439 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b71559700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6c102710 0x7f6b6c10f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b71d5a700 1 -- 172.21.15.32:0/135439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6c102710 msgr2=0x7f6b6c10f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b71d5a700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6c102710 0x7f6b6c10f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b71d5a700 1 -- 172.21.15.32:0/135439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b60005040 con 0x7f6b6c101d70 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b71559700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6c102710 0x7f6b6c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.382+0000 7f6b71d5a700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 0x7f6b6c10eba0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009360 tx=0x7f6b60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.383+0000 7f6b5e7fc700 1 -- 172.21.15.32:0/135439 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b6000e040 con 0x7f6b6c101d70 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.383+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6c117a50 con 0x7f6b6c101d70 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.383+0000 7f6b5e7fc700 1 -- 172.21.15.32:0/135439 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b60013420 con 0x7f6b6c101d70 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.383+0000 7f6b5e7fc700 1 -- 172.21.15.32:0/135439 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b6001c5f0 con 0x7f6b6c101d70 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.383+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c117fb0 con 0x7f6b6c101d70 2024-04-06T14:56:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.384+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6c0ff080 con 0x7f6b6c101d70 2024-04-06T14:56:59.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.384+0000 7f6b5e7fc700 1 -- 172.21.15.32:0/135439 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6b60013590 con 0x7f6b6c101d70 2024-04-06T14:56:59.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.385+0000 7f6b5e7fc700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b5806cc30 0x7f6b5806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:56:59.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.385+0000 7f6b71559700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b5806cc30 0x7f6b5806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:56:59.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.385+0000 7f6b5e7fc700 1 -- 172.21.15.32:0/135439 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6b6008d2d0 con 0x7f6b6c101d70 2024-04-06T14:56:59.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.385+0000 7f6b71559700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b5806cc30 0x7f6b5806f0e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009fd0 tx=0x7f6b6800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:56:59.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.389+0000 7f6b5e7fc700 1 -- 172.21.15.32:0/135439 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6b60057310 con 0x7f6b6c101d70 2024-04-06T14:56:59.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.602+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b6c04bed0 con 0x7f6b5806cc30 2024-04-06T14:56:59.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.612+0000 7f6b5e7fc700 1 -- 172.21.15.32:0/135439 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6b6c04bed0 con 0x7f6b5806cc30 2024-04-06T14:56:59.599 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:56:59.599 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (37m) 16s ago 48m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:56:59.599 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (34m) 16s ago 48m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:56:59.599 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (34m) 19s ago 47m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:56:59.599 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (37m) 16s ago 48m 89.7M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:56:59.599 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (31m) 16s ago 42m 103M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:56:59.599 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (30m) 16s ago 42m 116M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (30m) 19s ago 42m 22.1M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (15m) 19s ago 42m 18.0M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (39m) 16s ago 50m 516M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (38m) 19s ago 47m 458M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (35m) 16s ago 50m 86.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (35m) 19s ago 47m 60.1M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (38m) 16s ago 48m 17.4M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (37m) 19s ago 47m 18.2M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (34m) 16s ago 46m 89.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (33m) 16s ago 46m 91.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (33m) 16s ago 45m 91.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (32m) 19s ago 45m 103M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (31m) 19s ago 44m 102M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:56:59.600 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (31m) 19s ago 44m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:56:59.601 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (37m) 16s ago 48m 82.0M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:56:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b5806cc30 msgr2=0x7f6b5806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b5806cc30 0x7f6b5806f0e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009fd0 tx=0x7f6b6800b040 comp rx=0 tx=0).stop 2024-04-06T14:56:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 msgr2=0x7f6b6c10eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:56:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 0x7f6b6c10eba0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009360 tx=0x7f6b60009e30 comp rx=0 tx=0).stop 2024-04-06T14:56:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 shutdown_connections 2024-04-06T14:56:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6b5806cc30 0x7f6b5806f0e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b6c101d70 0x7f6b6c10eba0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 --2- 172.21.15.32:0/135439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6c102710 0x7f6b6c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:56:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 >> 172.21.15.32:0/135439 conn(0x7f6b6c072bc0 msgr2=0x7f6b6c0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:56:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 shutdown_connections 2024-04-06T14:56:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:56:59.614+0000 7f6b73fbe700 1 -- 172.21.15.32:0/135439 wait complete. 2024-04-06T14:57:00.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph orch upgrade status' 2024-04-06T14:57:00.466 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:00 smithi032 ceph-mon[147414]: from='client.35994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:57:00.494 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:57:00.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:00 smithi086 ceph-mon[110975]: from='client.35994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:57:01.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:01 smithi032 ceph-mon[147414]: pgmap v1110: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 5.1 KiB/s wr, 6 op/s 2024-04-06T14:57:01.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:01 smithi086 ceph-mon[110975]: pgmap v1110: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 5.1 KiB/s wr, 6 op/s 2024-04-06T14:57:02.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.270+0000 7f2eb78bc700 1 -- 172.21.15.32:0/244162610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 msgr2=0x7f2eb00fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:02.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.270+0000 7f2eb78bc700 1 --2- 172.21.15.32:0/244162610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 0x7f2eb00fce20 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0004940 tx=0x7f2ea0009e30 comp rx=0 tx=0).stop 2024-04-06T14:57:02.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.270+0000 7f2eb78bc700 1 -- 172.21.15.32:0/244162610 shutdown_connections 2024-04-06T14:57:02.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.270+0000 7f2eb78bc700 1 --2- 172.21.15.32:0/244162610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 0x7f2eb00fce20 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:02.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.270+0000 7f2eb78bc700 1 --2- 172.21.15.32:0/244162610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2eb0102e10 0x7f2eb01031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:02.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.270+0000 7f2eb78bc700 1 -- 172.21.15.32:0/244162610 >> 172.21.15.32:0/244162610 conn(0x7f2eb00f89a0 msgr2=0x7f2eb00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:57:02.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.270+0000 7f2eb78bc700 1 -- 172.21.15.32:0/244162610 shutdown_connections 2024-04-06T14:57:02.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.270+0000 7f2eb78bc700 1 -- 172.21.15.32:0/244162610 wait complete. 2024-04-06T14:57:02.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb78bc700 1 Processor -- start 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb78bc700 1 -- start start 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb78bc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2eb0102e10 0x7f2eb0113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb78bc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 0x7f2eb0114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb78bc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb0114660 con 0x7f2eb01037b0 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb78bc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb010da50 con 0x7f2eb0102e10 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb4e57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 0x7f2eb0114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb4e57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 0x7f2eb0114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58308/0 (socket says 172.21.15.32:58308) 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb4e57700 1 -- 172.21.15.32:0/1758796906 learned_addr learned my addr 172.21.15.32:0/1758796906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb5658700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2eb0102e10 0x7f2eb0113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb4e57700 1 -- 172.21.15.32:0/1758796906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2eb0102e10 msgr2=0x7f2eb0113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb4e57700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2eb0102e10 0x7f2eb0113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.271+0000 7f2eb4e57700 1 -- 172.21.15.32:0/1758796906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea0005040 con 0x7f2eb01037b0 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.272+0000 7f2eb4e57700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 0x7f2eb0114010 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009380 tx=0x7f2ea0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:57:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.272+0000 7f2ea5ffb700 1 -- 172.21.15.32:0/1758796906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ea000e040 con 0x7f2eb01037b0 2024-04-06T14:57:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.272+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb010dcd0 con 0x7f2eb01037b0 2024-04-06T14:57:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.272+0000 7f2ea5ffb700 1 -- 172.21.15.32:0/1758796906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ea0013420 con 0x7f2eb01037b0 2024-04-06T14:57:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.272+0000 7f2ea5ffb700 1 -- 172.21.15.32:0/1758796906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ea001c630 con 0x7f2eb01037b0 2024-04-06T14:57:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.272+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb010e290 con 0x7f2eb01037b0 2024-04-06T14:57:02.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.273+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb00fe560 con 0x7f2eb01037b0 2024-04-06T14:57:02.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.274+0000 7f2ea5ffb700 1 -- 172.21.15.32:0/1758796906 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2ea00139b0 con 0x7f2eb01037b0 2024-04-06T14:57:02.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.274+0000 7f2ea5ffb700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e9c06cc20 0x7f2e9c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:57:02.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.274+0000 7f2ea5ffb700 1 -- 172.21.15.32:0/1758796906 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2ea008d1b0 con 0x7f2eb01037b0 2024-04-06T14:57:02.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.274+0000 7f2eb5658700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e9c06cc20 0x7f2e9c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:57:02.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.274+0000 7f2eb5658700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e9c06cc20 0x7f2e9c06f0d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2eac004f40 tx=0x7f2eac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:57:02.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.279+0000 7f2ea5ffb700 1 -- 172.21.15.32:0/1758796906 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2ea005ac80 con 0x7f2eb01037b0 2024-04-06T14:57:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.498+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2eb0109db0 con 0x7f2e9c06cc20 2024-04-06T14:57:02.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.499+0000 7f2ea5ffb700 1 -- 172.21.15.32:0/1758796906 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f2eb0109db0 con 0x7f2e9c06cc20 2024-04-06T14:57:02.486 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:57:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": null, 2024-04-06T14:57:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": false, 2024-04-06T14:57:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "", 2024-04-06T14:57:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-06T14:57:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": null, 2024-04-06T14:57:02.487 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-04-06T14:57:02.487 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-06T14:57:02.487 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:57:02.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e9c06cc20 msgr2=0x7f2e9c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:02.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e9c06cc20 0x7f2e9c06f0d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2eac004f40 tx=0x7f2eac00b040 comp rx=0 tx=0).stop 2024-04-06T14:57:02.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 msgr2=0x7f2eb0114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:02.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 0x7f2eb0114010 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009380 tx=0x7f2ea0009e30 comp rx=0 tx=0).stop 2024-04-06T14:57:02.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 shutdown_connections 2024-04-06T14:57:02.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2e9c06cc20 0x7f2e9c06f0d0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:02.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2eb0102e10 0x7f2eb0113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:02.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 --2- 172.21.15.32:0/1758796906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2eb01037b0 0x7f2eb0114010 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:02.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 >> 172.21.15.32:0/1758796906 conn(0x7f2eb00f89a0 msgr2=0x7f2eb00f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:57:02.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 shutdown_connections 2024-04-06T14:57:02.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:02.501+0000 7f2eb78bc700 1 -- 172.21.15.32:0/1758796906 wait complete. 2024-04-06T14:57:03.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:02 smithi086 ceph-mon[110975]: from='client.35998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:57:03.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:02 smithi086 ceph-mon[110975]: pgmap v1111: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 4.4 KiB/s wr, 5 op/s 2024-04-06T14:57:03.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:02 smithi032 ceph-mon[147414]: from='client.35998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:57:03.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:02 smithi032 ceph-mon[147414]: pgmap v1111: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 4.4 KiB/s wr, 5 op/s 2024-04-06T14:57:03.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph health detail' 2024-04-06T14:57:03.565 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:57:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.283+0000 7f5567b23700 1 -- 172.21.15.32:0/186337120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 msgr2=0x7f5560103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.283+0000 7f5567b23700 1 --2- 172.21.15.32:0/186337120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 0x7f5560103240 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f555c004940 tx=0x7f555c009e30 comp rx=0 tx=0).stop 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.283+0000 7f5567b23700 1 -- 172.21.15.32:0/186337120 shutdown_connections 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.283+0000 7f5567b23700 1 --2- 172.21.15.32:0/186337120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55600fb4a0 0x7f55600fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.283+0000 7f5567b23700 1 --2- 172.21.15.32:0/186337120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 0x7f5560103240 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.283+0000 7f5567b23700 1 -- 172.21.15.32:0/186337120 >> 172.21.15.32:0/186337120 conn(0x7f55600714c0 msgr2=0x7f55600718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.283+0000 7f5567b23700 1 -- 172.21.15.32:0/186337120 shutdown_connections 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.283+0000 7f5567b23700 1 -- 172.21.15.32:0/186337120 wait complete. 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f5567b23700 1 Processor -- start 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f5567b23700 1 -- start start 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f5567b23700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55600fb4a0 0x7f55600fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f5567b23700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 0x7f55600fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f5567b23700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600fd150 con 0x7f5560102e70 2024-04-06T14:57:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f5567b23700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600fd290 con 0x7f55600fb4a0 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f55650be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 0x7f55600fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f55650be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 0x7f55600fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57716/0 (socket says 172.21.15.32:57716) 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f55658bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55600fb4a0 0x7f55600fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.284+0000 7f55650be700 1 -- 172.21.15.32:0/3900835240 learned_addr learned my addr 172.21.15.32:0/3900835240 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.285+0000 7f55650be700 1 -- 172.21.15.32:0/3900835240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55600fb4a0 msgr2=0x7f55600fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.285+0000 7f55650be700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55600fb4a0 0x7f55600fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.285+0000 7f55650be700 1 -- 172.21.15.32:0/3900835240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c005040 con 0x7f5560102e70 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.285+0000 7f55650be700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 0x7f55600fcbc0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f5554002a50 tx=0x7f55540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.285+0000 7f55527fc700 1 -- 172.21.15.32:0/3900835240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5554007cc0 con 0x7f5560102e70 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.285+0000 7f55527fc700 1 -- 172.21.15.32:0/3900835240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f555400f040 con 0x7f5560102e70 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.285+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5560119e30 con 0x7f5560102e70 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.285+0000 7f55527fc700 1 -- 172.21.15.32:0/3900835240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f555400d9c0 con 0x7f5560102e70 2024-04-06T14:57:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.285+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556011a230 con 0x7f5560102e70 2024-04-06T14:57:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.286+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5560100110 con 0x7f5560102e70 2024-04-06T14:57:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.286+0000 7f55527fc700 1 -- 172.21.15.32:0/3900835240 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f555400db60 con 0x7f5560102e70 2024-04-06T14:57:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.287+0000 7f55527fc700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f554c06cc20 0x7f554c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:57:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.287+0000 7f55527fc700 1 -- 172.21.15.32:0/3900835240 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f555408b700 con 0x7f5560102e70 2024-04-06T14:57:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.287+0000 7f55658bf700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f554c06cc20 0x7f554c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:57:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.287+0000 7f55658bf700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f554c06cc20 0x7f554c06f0d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f555c004f90 tx=0x7f555c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:57:05.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.291+0000 7f55527fc700 1 -- 172.21.15.32:0/3900835240 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f55540556c0 con 0x7f5560102e70 2024-04-06T14:57:05.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.577+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5560065380 con 0x7f5560102e70 2024-04-06T14:57:05.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.577+0000 7f55527fc700 1 -- 172.21.15.32:0/3900835240 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f5554058ce0 con 0x7f5560102e70 2024-04-06T14:57:05.564 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T14:57:05.564 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T14:57:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T14:57:05.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f554c06cc20 msgr2=0x7f554c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:05.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f554c06cc20 0x7f554c06f0d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f555c004f90 tx=0x7f555c009e30 comp rx=0 tx=0).stop 2024-04-06T14:57:05.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 msgr2=0x7f55600fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 0x7f55600fcbc0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f5554002a50 tx=0x7f55540046b0 comp rx=0 tx=0).stop 2024-04-06T14:57:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 shutdown_connections 2024-04-06T14:57:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f554c06cc20 0x7f554c06f0d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55600fb4a0 0x7f55600fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 --2- 172.21.15.32:0/3900835240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5560102e70 0x7f55600fcbc0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 >> 172.21.15.32:0/3900835240 conn(0x7f55600714c0 msgr2=0x7f55600fa8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:57:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 shutdown_connections 2024-04-06T14:57:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:05.579+0000 7f5567b23700 1 -- 172.21.15.32:0/3900835240 wait complete. 2024-04-06T14:57:05.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:05 smithi032 ceph-mon[147414]: pgmap v1112: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-06T14:57:05.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:05 smithi086 ceph-mon[110975]: pgmap v1112: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-06T14:57:06.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph versions' 2024-04-06T14:57:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:57:06.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:06 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3900835240' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:57:06.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:06 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3900835240' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T14:57:07.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:07 smithi032 ceph-mon[147414]: pgmap v1113: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T14:57:07.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:07 smithi086 ceph-mon[110975]: pgmap v1113: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T14:57:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 -- 172.21.15.32:0/3584055675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe664102e00 msgr2=0x7fe6641031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 --2- 172.21.15.32:0/3584055675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe664102e00 0x7fe6641031d0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe654004940 tx=0x7fe654009e30 comp rx=0 tx=0).stop 2024-04-06T14:57:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 -- 172.21.15.32:0/3584055675 shutdown_connections 2024-04-06T14:57:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 --2- 172.21.15.32:0/3584055675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6641037a0 0x7fe6640fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 --2- 172.21.15.32:0/3584055675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe664102e00 0x7fe6641031d0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 -- 172.21.15.32:0/3584055675 >> 172.21.15.32:0/3584055675 conn(0x7fe6640f89b0 msgr2=0x7fe6640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:57:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 -- 172.21.15.32:0/3584055675 shutdown_connections 2024-04-06T14:57:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 -- 172.21.15.32:0/3584055675 wait complete. 2024-04-06T14:57:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 Processor -- start 2024-04-06T14:57:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.262+0000 7fe669be7700 1 -- start start 2024-04-06T14:57:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe669be7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe664102e00 0x7fe6640702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:57:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe669be7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6641037a0 0x7fe664070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:57:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe669be7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe664072210 con 0x7fe6641037a0 2024-04-06T14:57:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe669be7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe664070d40 con 0x7fe664102e00 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe662ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6641037a0 0x7fe664070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe662ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6641037a0 0x7fe664070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57740/0 (socket says 172.21.15.32:57740) 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe662ffd700 1 -- 172.21.15.32:0/1683788332 learned_addr learned my addr 172.21.15.32:0/1683788332 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe6637fe700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe664102e00 0x7fe6640702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe662ffd700 1 -- 172.21.15.32:0/1683788332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe664102e00 msgr2=0x7fe6640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe662ffd700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe664102e00 0x7fe6640702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe662ffd700 1 -- 172.21.15.32:0/1683788332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe654005040 con 0x7fe6641037a0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe662ffd700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6641037a0 0x7fe664070800 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe658002a50 tx=0x7fe658004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.263+0000 7fe64bfff700 1 -- 172.21.15.32:0/1683788332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe658007cc0 con 0x7fe6641037a0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.264+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe664071020 con 0x7fe6641037a0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.264+0000 7fe64bfff700 1 -- 172.21.15.32:0/1683788332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe65800f040 con 0x7fe6641037a0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.264+0000 7fe64bfff700 1 -- 172.21.15.32:0/1683788332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe65800d920 con 0x7fe6641037a0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.264+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66411a060 con 0x7fe6641037a0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.265+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6640fe550 con 0x7fe6641037a0 2024-04-06T14:57:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.265+0000 7fe64bfff700 1 -- 172.21.15.32:0/1683788332 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe658009680 con 0x7fe6641037a0 2024-04-06T14:57:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.266+0000 7fe64bfff700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe64c06cc20 0x7fe64c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:57:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.266+0000 7fe64bfff700 1 -- 172.21.15.32:0/1683788332 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe65808be80 con 0x7fe6641037a0 2024-04-06T14:57:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.266+0000 7fe6637fe700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe64c06cc20 0x7fe64c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:57:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.266+0000 7fe6637fe700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe64c06cc20 0x7fe64c06f0d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe654004f90 tx=0x7fe654009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:57:08.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.270+0000 7fe64bfff700 1 -- 172.21.15.32:0/1683788332 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe658055dc0 con 0x7fe6641037a0 2024-04-06T14:57:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.559+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe66411a250 con 0x7fe6641037a0 2024-04-06T14:57:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.560+0000 7fe64bfff700 1 -- 172.21.15.32:0/1683788332 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fe6580593e0 con 0x7fe6641037a0 2024-04-06T14:57:08.546 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:57:08.546 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:57:08.546 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:57:08.547 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:57:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.561+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe64c06cc20 msgr2=0x7fe64c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.561+0000 7fe669be7700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe64c06cc20 0x7fe64c06f0d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe654004f90 tx=0x7fe654009e30 comp rx=0 tx=0).stop 2024-04-06T14:57:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.562+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6641037a0 msgr2=0x7fe664070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:57:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.562+0000 7fe669be7700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6641037a0 0x7fe664070800 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe658002a50 tx=0x7fe658004600 comp rx=0 tx=0).stop 2024-04-06T14:57:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.562+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 shutdown_connections 2024-04-06T14:57:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.562+0000 7fe669be7700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe64c06cc20 0x7fe64c06f0d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.562+0000 7fe669be7700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe664102e00 0x7fe6640702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.562+0000 7fe669be7700 1 --2- 172.21.15.32:0/1683788332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6641037a0 0x7fe664070800 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:57:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.562+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 >> 172.21.15.32:0/1683788332 conn(0x7fe6640f89b0 msgr2=0x7fe6640f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:57:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.562+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 shutdown_connections 2024-04-06T14:57:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:57:08.562+0000 7fe669be7700 1 -- 172.21.15.32:0/1683788332 wait complete. 2024-04-06T14:57:08.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:57:08.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:57:09.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:09 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1683788332' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:57:09.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:09 smithi032 ceph-mon[147414]: pgmap v1114: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:09.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:09 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1683788332' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:57:09.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:09 smithi086 ceph-mon[110975]: pgmap v1114: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:11.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:10 smithi086 ceph-mon[110975]: pgmap v1115: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:11.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:10 smithi032 ceph-mon[147414]: pgmap v1115: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:12.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-06T14:57:13.082 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:57:13.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:12 smithi086 ceph-mon[110975]: pgmap v1116: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:13.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:12 smithi032 ceph-mon[147414]: pgmap v1116: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:14.705 INFO:teuthology.orchestra.run.smithi032.stdout:wait for servicemap items w/ changing names to refresh 2024-04-06T14:57:15.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'sleep 60' 2024-04-06T14:57:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:57:15.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:15 smithi032 ceph-mon[147414]: pgmap v1117: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:15.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:15 smithi086 ceph-mon[110975]: pgmap v1117: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:17.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:16 smithi086 ceph-mon[110975]: pgmap v1118: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:17.218 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:16 smithi032 ceph-mon[147414]: pgmap v1118: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:19.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:18 smithi086 ceph-mon[110975]: pgmap v1119: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:19.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:18 smithi032 ceph-mon[147414]: pgmap v1119: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:21.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:20 smithi086 ceph-mon[110975]: pgmap v1120: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:21.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:20 smithi032 ceph-mon[147414]: pgmap v1120: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:22.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:57:23.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:57:23.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:23 smithi032 ceph-mon[147414]: pgmap v1121: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:24.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:23 smithi086 ceph-mon[110975]: pgmap v1121: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:25.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:25 smithi032 ceph-mon[147414]: pgmap v1122: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:25.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:25 smithi086 ceph-mon[110975]: pgmap v1122: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:27.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:26 smithi086 ceph-mon[110975]: pgmap v1123: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:27.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:26 smithi032 ceph-mon[147414]: pgmap v1123: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:29.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:28 smithi086 ceph-mon[110975]: pgmap v1124: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:29.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:28 smithi032 ceph-mon[147414]: pgmap v1124: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:31.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:30 smithi086 ceph-mon[110975]: pgmap v1125: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:31.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:30 smithi032 ceph-mon[147414]: pgmap v1125: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:33.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:32 smithi086 ceph-mon[110975]: pgmap v1126: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:33.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:32 smithi032 ceph-mon[147414]: pgmap v1126: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:35.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:35 smithi032 ceph-mon[147414]: pgmap v1127: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:35.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:35 smithi086 ceph-mon[110975]: pgmap v1127: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:37.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:36 smithi086 ceph-mon[110975]: pgmap v1128: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:37.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:36 smithi032 ceph-mon[147414]: pgmap v1128: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:38.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:57:38.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:57:39.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:38 smithi086 ceph-mon[110975]: pgmap v1129: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:39.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:38 smithi032 ceph-mon[147414]: pgmap v1129: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:41.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:40 smithi086 ceph-mon[110975]: pgmap v1130: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:41.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:40 smithi032 ceph-mon[147414]: pgmap v1130: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:43.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:42 smithi086 ceph-mon[110975]: pgmap v1131: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:43.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:42 smithi032 ceph-mon[147414]: pgmap v1131: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:45.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:45 smithi032 ceph-mon[147414]: pgmap v1132: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:45.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:45 smithi086 ceph-mon[110975]: pgmap v1132: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:47.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:46 smithi086 ceph-mon[110975]: pgmap v1133: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:47.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:46 smithi032 ceph-mon[147414]: pgmap v1133: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:49.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:48 smithi086 ceph-mon[110975]: pgmap v1134: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:49.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:48 smithi032 ceph-mon[147414]: pgmap v1134: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:51.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:50 smithi086 ceph-mon[110975]: pgmap v1135: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:51.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:50 smithi032 ceph-mon[147414]: pgmap v1135: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:52.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:57:52.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:57:52.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:57:52.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:57:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:57:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:57:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:57:52.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:57:53.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:57:53.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:52 smithi086 ceph-mon[110975]: pgmap v1136: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:53.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:57:53.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:52 smithi032 ceph-mon[147414]: pgmap v1136: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:55.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:55 smithi032 ceph-mon[147414]: pgmap v1137: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:55.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:55 smithi086 ceph-mon[110975]: pgmap v1137: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:57.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:56 smithi086 ceph-mon[110975]: pgmap v1138: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:57.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:56 smithi032 ceph-mon[147414]: pgmap v1138: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:59.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:57:58 smithi086 ceph-mon[110975]: pgmap v1139: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:57:59.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:57:58 smithi032 ceph-mon[147414]: pgmap v1139: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:01.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:00 smithi086 ceph-mon[110975]: pgmap v1140: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:01.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:00 smithi032 ceph-mon[147414]: pgmap v1140: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:03.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:02 smithi086 ceph-mon[110975]: pgmap v1141: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:03.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:02 smithi032 ceph-mon[147414]: pgmap v1141: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:05.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:05 smithi032 ceph-mon[147414]: pgmap v1142: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:05.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:05 smithi086 ceph-mon[110975]: pgmap v1142: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:07.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:06 smithi086 ceph-mon[110975]: pgmap v1143: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:07.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:06 smithi032 ceph-mon[147414]: pgmap v1143: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:08.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:58:08.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:58:09.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:08 smithi086 ceph-mon[110975]: pgmap v1144: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:09.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:08 smithi032 ceph-mon[147414]: pgmap v1144: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:11.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:10 smithi086 ceph-mon[110975]: pgmap v1145: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:11.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:10 smithi032 ceph-mon[147414]: pgmap v1145: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:13.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:12 smithi086 ceph-mon[110975]: pgmap v1146: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:13.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:12 smithi032 ceph-mon[147414]: pgmap v1146: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:15.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:15 smithi032 ceph-mon[147414]: pgmap v1147: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:15.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:15 smithi086 ceph-mon[110975]: pgmap v1147: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:17.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:16 smithi086 ceph-mon[110975]: pgmap v1148: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:17.201 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:16 smithi032 ceph-mon[147414]: pgmap v1148: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:17.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph orch ps' 2024-04-06T14:58:18.165 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:58:19.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:18 smithi086 ceph-mon[110975]: pgmap v1149: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:19.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:18 smithi032 ceph-mon[147414]: pgmap v1149: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:19.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.729+0000 7fc096187700 1 -- 172.21.15.32:0/1816724471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0901037b0 msgr2=0x7fc0900fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:19.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.729+0000 7fc096187700 1 --2- 172.21.15.32:0/1816724471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0901037b0 0x7fc0900fce20 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc084004940 tx=0x7fc084009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:19.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.729+0000 7fc096187700 1 -- 172.21.15.32:0/1816724471 shutdown_connections 2024-04-06T14:58:19.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.729+0000 7fc096187700 1 --2- 172.21.15.32:0/1816724471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0901037b0 0x7fc0900fce20 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:19.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.729+0000 7fc096187700 1 --2- 172.21.15.32:0/1816724471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc090102e10 0x7fc0901031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:19.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.729+0000 7fc096187700 1 -- 172.21.15.32:0/1816724471 >> 172.21.15.32:0/1816724471 conn(0x7fc0900f89a0 msgr2=0x7fc0900fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:19.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.729+0000 7fc096187700 1 -- 172.21.15.32:0/1816724471 shutdown_connections 2024-04-06T14:58:19.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.729+0000 7fc096187700 1 -- 172.21.15.32:0/1816724471 wait complete. 2024-04-06T14:58:19.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc096187700 1 Processor -- start 2024-04-06T14:58:19.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc096187700 1 -- start start 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc096187700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc090102e10 0x7fc090113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc096187700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0901037b0 0x7fc090114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc096187700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090114660 con 0x7fc090102e10 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc096187700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09010da50 con 0x7fc0901037b0 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc08f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc090102e10 0x7fc090113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc08f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc090102e10 0x7fc090113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38554/0 (socket says 172.21.15.32:38554) 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc08f7fe700 1 -- 172.21.15.32:0/3588620824 learned_addr learned my addr 172.21.15.32:0/3588620824 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc08effd700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0901037b0 0x7fc090114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc08f7fe700 1 -- 172.21.15.32:0/3588620824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0901037b0 msgr2=0x7fc090114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc08f7fe700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0901037b0 0x7fc090114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.730+0000 7fc08f7fe700 1 -- 172.21.15.32:0/3588620824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084005040 con 0x7fc090102e10 2024-04-06T14:58:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.731+0000 7fc08f7fe700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc090102e10 0x7fc090113ad0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fc08000c830 tx=0x7fc080004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.731+0000 7fc077fff700 1 -- 172.21.15.32:0/3588620824 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0800106f0 con 0x7fc090102e10 2024-04-06T14:58:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.731+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09010dd30 con 0x7fc090102e10 2024-04-06T14:58:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.731+0000 7fc077fff700 1 -- 172.21.15.32:0/3588620824 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc080010d00 con 0x7fc090102e10 2024-04-06T14:58:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.731+0000 7fc077fff700 1 -- 172.21.15.32:0/3588620824 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc080012e90 con 0x7fc090102e10 2024-04-06T14:58:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.731+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09010e2c0 con 0x7fc090102e10 2024-04-06T14:58:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.732+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0900fe560 con 0x7fc090102e10 2024-04-06T14:58:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.732+0000 7fc077fff700 1 -- 172.21.15.32:0/3588620824 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc080010850 con 0x7fc090102e10 2024-04-06T14:58:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.733+0000 7fc077fff700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07806cc20 0x7fc07806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.733+0000 7fc077fff700 1 -- 172.21.15.32:0/3588620824 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc08008c010 con 0x7fc090102e10 2024-04-06T14:58:19.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.733+0000 7fc08effd700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07806cc20 0x7fc07806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:19.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.733+0000 7fc08effd700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07806cc20 0x7fc07806f0d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc09010ed30 tx=0x7fc084009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:19.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.738+0000 7fc077fff700 1 -- 172.21.15.32:0/3588620824 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc080059ae0 con 0x7fc090102e10 2024-04-06T14:58:19.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.953+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 --> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc090109e00 con 0x7fc07806cc20 2024-04-06T14:58:19.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.963+0000 7fc077fff700 1 -- 172.21.15.32:0/3588620824 <== mgr.34104 v2:172.21.15.32:6800/3171072994 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc090109e00 con 0x7fc07806cc20 2024-04-06T14:58:19.950 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (38m) 96s ago 49m 28.0M - 0.25.0 c8568f914cd2 4cac26cc7189 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (36m) 96s ago 50m 6656k - 18.2.2-1086-g052adcad 87261d5ece5a 684be509c505 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi086 smithi086 running (35m) 99s ago 49m 6660k - 18.2.2-1086-g052adcad 87261d5ece5a cdab98f80209 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (38m) 96s ago 49m 89.7M - 9.4.7 954c08fa6188 75209e7a96f5 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.gcjdjo smithi032 running (32m) 96s ago 44m 103M - 18.2.2-1086-g052adcad 87261d5ece5a c715fbf58b12 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.mqoaof smithi032 running (32m) 96s ago 44m 116M - 18.2.2-1086-g052adcad 87261d5ece5a 7543016d4a6a 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.nikfjj smithi086 running (31m) 99s ago 44m 22.1M - 18.2.2-1086-g052adcad 87261d5ece5a 6ed6ede45f38 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi086.ojmchr smithi086 running (16m) 99s ago 44m 18.0M - 18.2.2-1086-g052adcad 87261d5ece5a d3ce484c8735 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.peohxu smithi032 *:8443,9283,8765 running (40m) 96s ago 51m 516M - 18.2.2-1086-g052adcad 87261d5ece5a cbc34d1e695a 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi086.fyyhxm smithi086 *:8443,9283,8765 running (39m) 99s ago 49m 458M - 18.2.2-1086-g052adcad 87261d5ece5a 34c0d75ecd83 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (37m) 96s ago 51m 86.8M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 2c109e84844d 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi086 smithi086 running (36m) 99s ago 48m 60.1M 2048M 18.2.2-1086-g052adcad 87261d5ece5a 7224a981bc29 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (39m) 96s ago 50m 17.4M - 1.5.0 0da6a335fe13 ee028a4af73d 2024-04-06T14:58:19.951 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi086 smithi086 *:9100 running (39m) 99s ago 49m 18.2M - 1.5.0 0da6a335fe13 cfbb27bff687 2024-04-06T14:58:19.952 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (35m) 96s ago 48m 89.0M 1570M 18.2.2-1086-g052adcad 87261d5ece5a 5c8657a4a93c 2024-04-06T14:58:19.952 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (35m) 96s ago 47m 91.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a afc7d0d625a8 2024-04-06T14:58:19.952 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (34m) 96s ago 47m 91.4M 1570M 18.2.2-1086-g052adcad 87261d5ece5a beb745b5f249 2024-04-06T14:58:19.952 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi086 running (33m) 99s ago 46m 103M 2592M 18.2.2-1086-g052adcad 87261d5ece5a bcdfa7f8b79b 2024-04-06T14:58:19.952 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi086 running (33m) 99s ago 46m 102M 2592M 18.2.2-1086-g052adcad 87261d5ece5a d0008d3fcf95 2024-04-06T14:58:19.952 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi086 running (32m) 99s ago 45m 91.4M 2592M 18.2.2-1086-g052adcad 87261d5ece5a c6fe4d60741d 2024-04-06T14:58:19.952 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (39m) 96s ago 49m 82.0M - 2.43.0 a07b618ecd1d f3fb45d5e858 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07806cc20 msgr2=0x7fc07806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07806cc20 0x7fc07806f0d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc09010ed30 tx=0x7fc084009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc090102e10 msgr2=0x7fc090113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc090102e10 0x7fc090113ad0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fc08000c830 tx=0x7fc080004470 comp rx=0 tx=0).stop 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 shutdown_connections 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc07806cc20 0x7fc07806f0d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc090102e10 0x7fc090113ad0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 --2- 172.21.15.32:0/3588620824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0901037b0 0x7fc090114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 >> 172.21.15.32:0/3588620824 conn(0x7fc0900f89a0 msgr2=0x7fc0900f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 shutdown_connections 2024-04-06T14:58:19.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:19.965+0000 7fc096187700 1 -- 172.21.15.32:0/3588620824 wait complete. 2024-04-06T14:58:20.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph versions' 2024-04-06T14:58:20.841 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:58:21.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:20 smithi086 ceph-mon[110975]: from='client.36010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:58:21.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:20 smithi086 ceph-mon[110975]: pgmap v1150: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:21.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:20 smithi032 ceph-mon[147414]: from='client.36010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T14:58:21.119 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:20 smithi032 ceph-mon[147414]: pgmap v1150: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.545+0000 7fb747db7700 1 -- 172.21.15.32:0/2271926534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740102710 msgr2=0x7fb7400fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.545+0000 7fb747db7700 1 --2- 172.21.15.32:0/2271926534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740102710 0x7fb7400fbd80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fb734004940 tx=0x7fb734009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.545+0000 7fb747db7700 1 -- 172.21.15.32:0/2271926534 shutdown_connections 2024-04-06T14:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.545+0000 7fb747db7700 1 --2- 172.21.15.32:0/2271926534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740102710 0x7fb7400fbd80 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.545+0000 7fb747db7700 1 --2- 172.21.15.32:0/2271926534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740101d70 0x7fb740102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.545+0000 7fb747db7700 1 -- 172.21.15.32:0/2271926534 >> 172.21.15.32:0/2271926534 conn(0x7fb740072bc0 msgr2=0x7fb740072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.545+0000 7fb747db7700 1 -- 172.21.15.32:0/2271926534 shutdown_connections 2024-04-06T14:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.545+0000 7fb747db7700 1 -- 172.21.15.32:0/2271926534 wait complete. 2024-04-06T14:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb747db7700 1 Processor -- start 2024-04-06T14:58:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb747db7700 1 -- start start 2024-04-06T14:58:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb747db7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb740113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb747db7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 0x7fb740113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb747db7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7401146d0 con 0x7fb740101d70 2024-04-06T14:58:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb747db7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74010da30 con 0x7fb740102710 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb745b53700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb740113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb745b53700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb740113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38580/0 (socket says 172.21.15.32:38580) 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb745b53700 1 -- 172.21.15.32:0/987196403 learned_addr learned my addr 172.21.15.32:0/987196403 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb745352700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 0x7fb740113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb745b53700 1 -- 172.21.15.32:0/987196403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 msgr2=0x7fb740113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb745b53700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 0x7fb740113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.546+0000 7fb745b53700 1 -- 172.21.15.32:0/987196403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734005040 con 0x7fb740101d70 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.547+0000 7fb745b53700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb740113ab0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00b790 tx=0x7fb73c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.547+0000 7fb7327fc700 1 -- 172.21.15.32:0/987196403 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb73c00f770 con 0x7fb740101d70 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.547+0000 7fb7327fc700 1 -- 172.21.15.32:0/987196403 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb73c00fd80 con 0x7fb740101d70 2024-04-06T14:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.547+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74010dd10 con 0x7fb740101d70 2024-04-06T14:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.547+0000 7fb7327fc700 1 -- 172.21.15.32:0/987196403 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb73c00d310 con 0x7fb740101d70 2024-04-06T14:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.547+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74010e2d0 con 0x7fb740101d70 2024-04-06T14:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.548+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7400fd4c0 con 0x7fb740101d70 2024-04-06T14:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.548+0000 7fb7327fc700 1 -- 172.21.15.32:0/987196403 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb73c010420 con 0x7fb740101d70 2024-04-06T14:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.549+0000 7fb7327fc700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb72c06cc20 0x7fb72c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.549+0000 7fb7327fc700 1 -- 172.21.15.32:0/987196403 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb73c08bf30 con 0x7fb740101d70 2024-04-06T14:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.549+0000 7fb745352700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb72c06cc20 0x7fb72c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.549+0000 7fb745352700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb72c06cc20 0x7fb72c06f0d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb74010ebf0 tx=0x7fb734009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.553+0000 7fb7327fc700 1 -- 172.21.15.32:0/987196403 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb73c0bfab0 con 0x7fb740101d70 2024-04-06T14:58:22.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.844+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb74004bed0 con 0x7fb740101d70 2024-04-06T14:58:22.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.845+0000 7fb7327fc700 1 -- 172.21.15.32:0/987196403 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fb73c059ac0 con 0x7fb740101d70 2024-04-06T14:58:22.830 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-06T14:58:22.830 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-06T14:58:22.830 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:58:22.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:58:22.830 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 2 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 6 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 4 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)": 14 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-06T14:58:22.831 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-06T14:58:22.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb72c06cc20 msgr2=0x7fb72c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:22.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb72c06cc20 0x7fb72c06f0d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb74010ebf0 tx=0x7fb734009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:22.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 msgr2=0x7fb740113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:22.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb740113ab0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00b790 tx=0x7fb73c009250 comp rx=0 tx=0).stop 2024-04-06T14:58:22.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 shutdown_connections 2024-04-06T14:58:22.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb72c06cc20 0x7fb72c06f0d0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:22.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb740101d70 0x7fb740113ab0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:22.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 --2- 172.21.15.32:0/987196403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb740102710 0x7fb740113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:22.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 >> 172.21.15.32:0/987196403 conn(0x7fb740072bc0 msgr2=0x7fb7400659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:22.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 shutdown_connections 2024-04-06T14:58:22.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:22.847+0000 7fb747db7700 1 -- 172.21.15.32:0/987196403 wait complete. 2024-04-06T14:58:22.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:58:23.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:58:23.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-06T14:58:23.805 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:58:23.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:23 smithi032 ceph-mon[147414]: pgmap v1151: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:23.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:23 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/987196403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:58:24.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:23 smithi086 ceph-mon[110975]: pgmap v1151: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:24.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:23 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/987196403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:58:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.440+0000 7f6bca241700 1 -- 172.21.15.32:0/3191254175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 msgr2=0x7f6bc41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.440+0000 7f6bca241700 1 --2- 172.21.15.32:0/3191254175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 0x7f6bc41031d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8004940 tx=0x7f6bb8009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.440+0000 7f6bca241700 1 -- 172.21.15.32:0/3191254175 shutdown_connections 2024-04-06T14:58:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.440+0000 7f6bca241700 1 --2- 172.21.15.32:0/3191254175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc41037a0 0x7f6bc40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.440+0000 7f6bca241700 1 --2- 172.21.15.32:0/3191254175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 0x7f6bc41031d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.440+0000 7f6bca241700 1 -- 172.21.15.32:0/3191254175 >> 172.21.15.32:0/3191254175 conn(0x7f6bc40f89b0 msgr2=0x7f6bc40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.440+0000 7f6bca241700 1 -- 172.21.15.32:0/3191254175 shutdown_connections 2024-04-06T14:58:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.440+0000 7f6bca241700 1 -- 172.21.15.32:0/3191254175 wait complete. 2024-04-06T14:58:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bca241700 1 Processor -- start 2024-04-06T14:58:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bca241700 1 -- start start 2024-04-06T14:58:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bca241700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 0x7f6bc4113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bca241700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc41037a0 0x7f6bc4114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bca241700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc41146b0 con 0x7f6bc4102e00 2024-04-06T14:58:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bca241700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc410daa0 con 0x7f6bc41037a0 2024-04-06T14:58:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bc37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 0x7f6bc4113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bc37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 0x7f6bc4113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60836/0 (socket says 172.21.15.32:60836) 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bc37fe700 1 -- 172.21.15.32:0/3952022714 learned_addr learned my addr 172.21.15.32:0/3952022714 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.441+0000 7f6bc2ffd700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc41037a0 0x7f6bc4114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.442+0000 7f6bc37fe700 1 -- 172.21.15.32:0/3952022714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc41037a0 msgr2=0x7f6bc4114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.442+0000 7f6bc37fe700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc41037a0 0x7f6bc4114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.442+0000 7f6bc37fe700 1 -- 172.21.15.32:0/3952022714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb8005040 con 0x7f6bc4102e00 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.442+0000 7f6bc37fe700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 0x7f6bc4113b20 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8000c00 tx=0x7f6bb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.442+0000 7f6babfff700 1 -- 172.21.15.32:0/3952022714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bb800e040 con 0x7f6bc4102e00 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.442+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc410dd20 con 0x7f6bc4102e00 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.442+0000 7f6babfff700 1 -- 172.21.15.32:0/3952022714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bb8013420 con 0x7f6bc4102e00 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.442+0000 7f6babfff700 1 -- 172.21.15.32:0/3952022714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bb801c630 con 0x7f6bc4102e00 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.442+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc410e2e0 con 0x7f6bc4102e00 2024-04-06T14:58:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.443+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc40fe550 con 0x7f6bc4102e00 2024-04-06T14:58:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.443+0000 7f6babfff700 1 -- 172.21.15.32:0/3952022714 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6bb800ba80 con 0x7f6bc4102e00 2024-04-06T14:58:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.444+0000 7f6babfff700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6bac06cc20 0x7f6bac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.444+0000 7f6babfff700 1 -- 172.21.15.32:0/3952022714 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6bb808d180 con 0x7f6bc4102e00 2024-04-06T14:58:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.444+0000 7f6bc2ffd700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6bac06cc20 0x7f6bac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.444+0000 7f6bc2ffd700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6bac06cc20 0x7f6bac06f0d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6bc410ec60 tx=0x7f6bb400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:25.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.449+0000 7f6babfff700 1 -- 172.21.15.32:0/3952022714 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6bb805ac50 con 0x7f6bc4102e00 2024-04-06T14:58:25.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:25 smithi032 ceph-mon[147414]: pgmap v1152: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:25.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.738+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6bc404bed0 con 0x7f6bc4102e00 2024-04-06T14:58:25.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.738+0000 7f6babfff700 1 -- 172.21.15.32:0/3952022714 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6bb805a7e0 con 0x7f6bc4102e00 2024-04-06T14:58:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6bac06cc20 msgr2=0x7f6bac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6bac06cc20 0x7f6bac06f0d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6bc410ec60 tx=0x7f6bb400b040 comp rx=0 tx=0).stop 2024-04-06T14:58:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 msgr2=0x7f6bc4113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 0x7f6bc4113b20 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8000c00 tx=0x7f6bb8009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 shutdown_connections 2024-04-06T14:58:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f6bac06cc20 0x7f6bac06f0d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc4102e00 0x7f6bc4113b20 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 --2- 172.21.15.32:0/3952022714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc41037a0 0x7f6bc4114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 >> 172.21.15.32:0/3952022714 conn(0x7f6bc40f89b0 msgr2=0x7f6bc40f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 shutdown_connections 2024-04-06T14:58:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:25.741+0000 7f6bca241700 1 -- 172.21.15.32:0/3952022714 wait complete. 2024-04-06T14:58:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-06T14:58:25.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:25 smithi086 ceph-mon[110975]: pgmap v1152: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:26.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -e sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-06T14:58:26.708 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:58:26.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:26 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3952022714' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:58:26.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:26 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3952022714' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:58:27.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:27 smithi032 ceph-mon[147414]: pgmap v1153: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:27.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:27 smithi086 ceph-mon[110975]: pgmap v1153: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.442+0000 7f1a07b8d700 1 -- 172.21.15.32:0/2833295715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 msgr2=0x7f1a000fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.442+0000 7f1a07b8d700 1 --2- 172.21.15.32:0/2833295715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 0x7f1a000fce20 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f19f4004940 tx=0x7f19f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.442+0000 7f1a07b8d700 1 -- 172.21.15.32:0/2833295715 shutdown_connections 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.442+0000 7f1a07b8d700 1 --2- 172.21.15.32:0/2833295715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 0x7f1a000fce20 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.442+0000 7f1a07b8d700 1 --2- 172.21.15.32:0/2833295715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a00102e10 0x7f1a001031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.442+0000 7f1a07b8d700 1 -- 172.21.15.32:0/2833295715 >> 172.21.15.32:0/2833295715 conn(0x7f1a000f89a0 msgr2=0x7f1a000fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.442+0000 7f1a07b8d700 1 -- 172.21.15.32:0/2833295715 shutdown_connections 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.442+0000 7f1a07b8d700 1 -- 172.21.15.32:0/2833295715 wait complete. 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.443+0000 7f1a07b8d700 1 Processor -- start 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.443+0000 7f1a07b8d700 1 -- start start 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.443+0000 7f1a07b8d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a00102e10 0x7f1a000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.443+0000 7f1a07b8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 0x7f1a00070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.443+0000 7f1a07b8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a00070d90 con 0x7f1a001037b0 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.443+0000 7f1a07b8d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a00070ed0 con 0x7f1a00102e10 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.443+0000 7f1a05128700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 0x7f1a00070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.443+0000 7f1a05128700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 0x7f1a00070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60852/0 (socket says 172.21.15.32:60852) 2024-04-06T14:58:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.443+0000 7f1a05128700 1 -- 172.21.15.32:0/59930958 learned_addr learned my addr 172.21.15.32:0/59930958 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f1a05929700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a00102e10 0x7f1a000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f1a05128700 1 -- 172.21.15.32:0/59930958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a00102e10 msgr2=0x7f1a000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f1a05128700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a00102e10 0x7f1a000702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f1a05128700 1 -- 172.21.15.32:0/59930958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f4005040 con 0x7f1a001037b0 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f1a05128700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 0x7f1a00070800 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009380 tx=0x7f19f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f19f27fc700 1 -- 172.21.15.32:0/59930958 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19f400e040 con 0x7f1a001037b0 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f19f27fc700 1 -- 172.21.15.32:0/59930958 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19f4013420 con 0x7f1a001037b0 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f19f27fc700 1 -- 172.21.15.32:0/59930958 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19f401c630 con 0x7f1a001037b0 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a00071120 con 0x7f1a001037b0 2024-04-06T14:58:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.444+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0011a200 con 0x7f1a001037b0 2024-04-06T14:58:28.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.445+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a000fe560 con 0x7f1a001037b0 2024-04-06T14:58:28.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.445+0000 7f19f27fc700 1 -- 172.21.15.32:0/59930958 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f19f40139e0 con 0x7f1a001037b0 2024-04-06T14:58:28.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.446+0000 7f19f27fc700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19ec06cc20 0x7f19ec06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:28.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.446+0000 7f19f27fc700 1 -- 172.21.15.32:0/59930958 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f19f408d0f0 con 0x7f1a001037b0 2024-04-06T14:58:28.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.446+0000 7f1a05929700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19ec06cc20 0x7f19ec06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:28.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.446+0000 7f1a05929700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19ec06cc20 0x7f19ec06f0d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f19fc004c40 tx=0x7f19fc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:28.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.450+0000 7f19f27fc700 1 -- 172.21.15.32:0/59930958 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f19f40571f0 con 0x7f1a001037b0 2024-04-06T14:58:28.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.740+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1a0004bed0 con 0x7f1a001037b0 2024-04-06T14:58:28.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.741+0000 7f19f27fc700 1 -- 172.21.15.32:0/59930958 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f19f405a810 con 0x7f1a001037b0 2024-04-06T14:58:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19ec06cc20 msgr2=0x7f19ec06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19ec06cc20 0x7f19ec06f0d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f19fc004c40 tx=0x7f19fc009040 comp rx=0 tx=0).stop 2024-04-06T14:58:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 msgr2=0x7f1a00070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 0x7f1a00070800 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009380 tx=0x7f19f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 shutdown_connections 2024-04-06T14:58:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f19ec06cc20 0x7f19ec06f0d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a00102e10 0x7f1a000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 --2- 172.21.15.32:0/59930958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a001037b0 0x7f1a00070800 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 >> 172.21.15.32:0/59930958 conn(0x7f1a000f89a0 msgr2=0x7f1a000f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 shutdown_connections 2024-04-06T14:58:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:28.743+0000 7f1a07b8d700 1 -- 172.21.15.32:0/59930958 wait complete. 2024-04-06T14:58:28.739 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)" 2024-04-06T14:58:29.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:28 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/59930958' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:58:29.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:28 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/59930958' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T14:58:29.381 DEBUG:teuthology.parallel:result is None 2024-04-06T14:58:29.381 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T14:58:29.394 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T14:58:29.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-06T14:58:29.676 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/mon.smithi032/config 2024-04-06T14:58:30.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:29 smithi086 ceph-mon[110975]: pgmap v1154: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:30.181 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:29 smithi032 ceph-mon[147414]: pgmap v1154: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:31.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.411+0000 7f6757220700 1 -- 172.21.15.32:0/1835127045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750103720 msgr2=0x7f67500fcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:31.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.411+0000 7f6757220700 1 --2- 172.21.15.32:0/1835127045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750103720 0x7f67500fcd90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f673c004940 tx=0x7f673c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:31.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.411+0000 7f6757220700 1 -- 172.21.15.32:0/1835127045 shutdown_connections 2024-04-06T14:58:31.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.411+0000 7f6757220700 1 --2- 172.21.15.32:0/1835127045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750103720 0x7f67500fcd90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.411+0000 7f6757220700 1 --2- 172.21.15.32:0/1835127045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6750102d80 0x7f6750103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.411+0000 7f6757220700 1 -- 172.21.15.32:0/1835127045 >> 172.21.15.32:0/1835127045 conn(0x7f67500f8930 msgr2=0x7f67500fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.411+0000 7f6757220700 1 -- 172.21.15.32:0/1835127045 shutdown_connections 2024-04-06T14:58:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.411+0000 7f6757220700 1 -- 172.21.15.32:0/1835127045 wait complete. 2024-04-06T14:58:31.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.412+0000 7f6757220700 1 Processor -- start 2024-04-06T14:58:31.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.412+0000 7f6757220700 1 -- start start 2024-04-06T14:58:31.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.412+0000 7f6757220700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750102d80 0x7f67500702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:31.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.412+0000 7f6757220700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6750103720 0x7f6750070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:31.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.412+0000 7f6757220700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6750070d90 con 0x7f6750103720 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.412+0000 7f6757220700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6750070ed0 con 0x7f6750102d80 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.412+0000 7f674ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6750103720 0x7f6750070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.412+0000 7f674ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6750103720 0x7f6750070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60868/0 (socket says 172.21.15.32:60868) 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.412+0000 7f674ffff700 1 -- 172.21.15.32:0/1277395429 learned_addr learned my addr 172.21.15.32:0/1277395429 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f6754fbc700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750102d80 0x7f67500702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f674ffff700 1 -- 172.21.15.32:0/1277395429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750102d80 msgr2=0x7f67500702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f674ffff700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750102d80 0x7f67500702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f674ffff700 1 -- 172.21.15.32:0/1277395429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6744005040 con 0x7f6750103720 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f6754fbc700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750102d80 0x7f67500702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f674ffff700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6750103720 0x7f6750070800 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f673c000c00 tx=0x7f673c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f674d7fa700 1 -- 172.21.15.32:0/1277395429 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673c00e070 con 0x7f6750103720 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c005040 con 0x7f6750103720 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f674d7fa700 1 -- 172.21.15.32:0/1277395429 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f673c013de0 con 0x7f6750103720 2024-04-06T14:58:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f674d7fa700 1 -- 172.21.15.32:0/1277395429 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673c01cde0 con 0x7f6750103720 2024-04-06T14:58:31.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.413+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6750119ee0 con 0x7f6750103720 2024-04-06T14:58:31.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.414+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67500fe4d0 con 0x7f6750103720 2024-04-06T14:58:31.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.415+0000 7f674d7fa700 1 -- 172.21.15.32:0/1277395429 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f673c01c600 con 0x7f6750103720 2024-04-06T14:58:31.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.415+0000 7f674d7fa700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f674006cc20 0x7f674006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:31.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.415+0000 7f674d7fa700 1 -- 172.21.15.32:0/1277395429 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f673c08d5c0 con 0x7f6750103720 2024-04-06T14:58:31.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.415+0000 7f6754fbc700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f674006cc20 0x7f674006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:31.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.416+0000 7f6754fbc700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f674006cc20 0x7f674006f0d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6744007fd0 tx=0x7f6744009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:31.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.420+0000 7f674d7fa700 1 -- 172.21.15.32:0/1277395429 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f673c057600 con 0x7f6750103720 2024-04-06T14:58:31.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.669+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6750109d20 con 0x7f6750103720 2024-04-06T14:58:31.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.669+0000 7f674d7fa700 1 -- 172.21.15.32:0/1277395429 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 39 v39) v1 ==== 76+0+1857 (secure 0 0 0) 0x7f673c05ac20 con 0x7f6750103720 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:e39 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 39 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-06T14:14:02.044394+0000 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-06T14:56:49.438197+0000 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-06T14:58:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=34382,1=34414} 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-06T14:58:31.657 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data enabled 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.gcjdjo{0:34382} state up:active seq 467 join_fscid=1 addr [v2:172.21.15.32:6826/3498377443,v1:172.21.15.32:6827/3498377443] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.mqoaof{1:34414} state up:active seq 462 join_fscid=1 addr [v2:172.21.15.32:6828/3282445524,v1:172.21.15.32:6829/3282445524] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.nikfjj{-1:44313} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3277500545,v1:172.21.15.86:6825/3277500545] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:58:31.658 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi086.ojmchr{-1:44803} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/2158771540,v1:172.21.15.86:6827/2158771540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T14:58:31.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f674006cc20 msgr2=0x7f674006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:31.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f674006cc20 0x7f674006f0d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6744007fd0 tx=0x7f6744009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:31.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6750103720 msgr2=0x7f6750070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:31.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6750103720 0x7f6750070800 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f673c000c00 tx=0x7f673c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:31.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 shutdown_connections 2024-04-06T14:58:31.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f674006cc20 0x7f674006f0d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:31.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750102d80 0x7f67500702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:31.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 --2- 172.21.15.32:0/1277395429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6750103720 0x7f6750070800 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:31.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 >> 172.21.15.32:0/1277395429 conn(0x7f67500f8930 msgr2=0x7f67500f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:31.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 shutdown_connections 2024-04-06T14:58:31.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:31.672+0000 7f6757220700 1 -- 172.21.15.32:0/1277395429 wait complete. 2024-04-06T14:58:31.660 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 39 2024-04-06T14:58:31.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:31 smithi032 ceph-mon[147414]: pgmap v1155: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:31.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:31 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1277395429' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:58:32.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:31 smithi086 ceph-mon[110975]: pgmap v1155: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:32.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:31 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1277395429' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T14:58:32.360 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-06T14:58:32.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T14:58:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.515+0000 7f23a7b73700 1 -- 172.21.15.32:0/1927300023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0102610 msgr2=0x7f23a00fbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.515+0000 7f23a7b73700 1 --2- 172.21.15.32:0/1927300023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0102610 0x7f23a00fbc80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2394004940 tx=0x7f2394009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.515+0000 7f23a7b73700 1 -- 172.21.15.32:0/1927300023 shutdown_connections 2024-04-06T14:58:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.515+0000 7f23a7b73700 1 --2- 172.21.15.32:0/1927300023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0102610 0x7f23a00fbc80 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.515+0000 7f23a7b73700 1 --2- 172.21.15.32:0/1927300023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23a0101c70 0x7f23a0102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.515+0000 7f23a7b73700 1 -- 172.21.15.32:0/1927300023 >> 172.21.15.32:0/1927300023 conn(0x7f23a0071c80 msgr2=0x7f23a0072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.515+0000 7f23a7b73700 1 -- 172.21.15.32:0/1927300023 shutdown_connections 2024-04-06T14:58:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.515+0000 7f23a7b73700 1 -- 172.21.15.32:0/1927300023 wait complete. 2024-04-06T14:58:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a7b73700 1 Processor -- start 2024-04-06T14:58:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a7b73700 1 -- start start 2024-04-06T14:58:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a7b73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0101c70 0x7f23a010e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a7b73700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23a0102610 0x7f23a010ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a7b73700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a010f3d0 con 0x7f23a0101c70 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a7b73700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a010f510 con 0x7f23a0102610 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a590f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0101c70 0x7f23a010e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a590f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0101c70 0x7f23a010e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60890/0 (socket says 172.21.15.32:60890) 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a590f700 1 -- 172.21.15.32:0/3293706065 learned_addr learned my addr 172.21.15.32:0/3293706065 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a510e700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23a0102610 0x7f23a010ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a590f700 1 -- 172.21.15.32:0/3293706065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23a0102610 msgr2=0x7f23a010ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a590f700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23a0102610 0x7f23a010ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.516+0000 7f23a590f700 1 -- 172.21.15.32:0/3293706065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2394005040 con 0x7f23a0101c70 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.517+0000 7f23a590f700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0101c70 0x7f23a010e950 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f239c00c830 tx=0x7f239c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.517+0000 7f23927fc700 1 -- 172.21.15.32:0/3293706065 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f239c00a760 con 0x7f23a0101c70 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.517+0000 7f23927fc700 1 -- 172.21.15.32:0/3293706065 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f239c00ad70 con 0x7f23a0101c70 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.517+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a01179c0 con 0x7f23a0101c70 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.517+0000 7f23927fc700 1 -- 172.21.15.32:0/3293706065 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f239c0113a0 con 0x7f23a0101c70 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.517+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a0117d90 con 0x7f23a0101c70 2024-04-06T14:58:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.518+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a00fef80 con 0x7f23a0101c70 2024-04-06T14:58:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.518+0000 7f23927fc700 1 -- 172.21.15.32:0/3293706065 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f239c012400 con 0x7f23a0101c70 2024-04-06T14:58:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.519+0000 7f23927fc700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f238c06cc20 0x7f238c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.519+0000 7f23927fc700 1 -- 172.21.15.32:0/3293706065 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f239c08c0b0 con 0x7f23a0101c70 2024-04-06T14:58:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.519+0000 7f23a510e700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f238c06cc20 0x7f238c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.519+0000 7f23a510e700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f238c06cc20 0x7f238c06f0d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f239400b3b0 tx=0x7f2394009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.523+0000 7f23927fc700 1 -- 172.21.15.32:0/3293706065 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f239c0560f0 con 0x7f23a0101c70 2024-04-06T14:58:32.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.730+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23a00fd3c0 con 0x7f23a0101c70 2024-04-06T14:58:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.732+0000 7f23927fc700 1 -- 172.21.15.32:0/3293706065 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16504 (secure 0 0 0) 0x7f239c059710 con 0x7f23a0101c70 2024-04-06T14:58:32.718 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:32.718 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","created":"2024-04-06T14:06:48.330591+0000","modified":"2024-04-06T14:41:33.675047+0000","last_up_change":"2024-04-06T14:25:47.031924+0000","last_in_change":"2024-04-06T14:12:17.748510+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T14:11:20.110423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-06T14:14:00.974689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-06T14:14:01.457260+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6803","nonce":969207178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6805","nonce":969207178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6809","nonce":969207178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6807","nonce":969207178}]},"public_addr":"172.21.15.32:6803/969207178","cluster_addr":"172.21.15.32:6805/969207178","heartbeat_back_addr":"172.21.15.32:6809/969207178","heartbeat_front_addr":"172.21.15.32:6807/969207178","state":["exists","up"]},{"osd":1,"uuid":"9185a389-ef69-41bc-9e10-46c6a2e02a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6811","nonce":197940852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6813","nonce":197940852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6817","nonce":197940852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6815","nonce":197940852}]},"public_addr":"172.21.15.32:6811/197940852","cluster_addr":"172.21.15.32:6813/197940852","heartbeat_back_addr":"172.21.15.32:6817/197940852","heartbeat_front_addr":"172.21.15.32:6815/197940852","state":["exists","up"]},{"osd":2,"uuid":"af4793e1-b5bb-4919-838e-67703c83296a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6819","nonce":1957975204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6821","nonce":1957975204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6825","nonce":1957975204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6823","nonce":1957975204}]},"public_addr":"172.21.15.32:6819/1957975204","cluster_addr":"172.21.15.32:6821/1957975204","heartbeat_back_addr":"172.21.15.32:6825/1957975204","heartbeat_front_addr":"172.21.15.32:6823/1957975204","state":["exists","up"]},{"osd":3,"uuid":"70c8b884-5214-44f2-9a31-55556a11c20a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6801","nonce":3817339654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6803","nonce":3817339654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6807","nonce":3817339654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6805","nonce":3817339654}]},"public_addr":"172.21.15.86:6801/3817339654","cluster_addr":"172.21.15.86:6803/3817339654","heartbeat_back_addr":"172.21.15.86:6807/3817339654","heartbeat_front_addr":"172.21.15.86:6805/3817339654","state":["exists","up"]},{"osd":4,"uuid":"91f548ac-4f42-4083-a027-295b829646a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6809","nonce":1198497064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6811","nonce":1198497064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6815","nonce":1198497064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6813","nonce":1198497064}]},"public_addr":"172.21.15.86:6809/1198497064","cluster_addr":"172.21.15.86:6811/1198497064","heartbeat_back_addr":"172.21.15.86:6815/1198497064","heartbeat_front_addr":"172.21.15.86:6813/1198497064","state":["exists","up"]},{"osd":5,"uuid":"7037f613-ffd9-4b0e-84a1-0c04d4181ff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6817","nonce":3374872376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6819","nonce":3374872376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6823","nonce":3374872376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6821","nonce":3374872376}]},"public_addr":"172.21.15.86:6817/3374872376","cluster_addr":"172.21.15.86:6819/3374872376","heartbeat_back_addr":"172.21.15.86:6823/3374872376","heartbeat_front_addr":"172.21.15.86:6821/3374872376","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:13.684938+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:44.549488+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:15.893369+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:41.588409+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:08.612078+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:35.380707+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6828/1704327767":"2024-04-07T14:25:48.051314+0000","172.21.15.86:6825/3246813202":"2024-04-07T14:25:48.051344+0000","172.21.15.32:6801/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.86:0/2189072732":"2024-04-07T14:18:05.813583+0000","172.21.15.32:0/3548398153":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/935909372":"2024-04-07T14:07:51.523157+0000","172.21.15.32:6800/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6800/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6829/1704327767":"2024-04-07T14:25:48.051314+0000","172.21.15.32:6827/4235185528":"2024-04-07T14:14:16.952740+0000","172.21.15.32:6801/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/3250004174":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/711512148":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/68274703":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/3330131734":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/1187866165":"2024-04-07T14:08:57.599030+0000","172.21.15.86:0/2427526405":"2024-04-07T14:18:05.813583+0000","172.21.15.86:0/2137415724":"2024-04-07T14:18:05.813583+0000","172.21.15.32:0/2605964794":"2024-04-07T14:21:22.517015+0000","172.21.15.32:0/3796020247":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6800/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/2227455350":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/2524933837":"2024-04-07T14:21:22.517015+0000","172.21.15.32:0/2574238932":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6800/3067649142":"2024-04-07T14:16:30.572936+0000","172.21.15.32:0/1664607864":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6801/3067649142":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6826/4235185528":"2024-04-07T14:14:16.952740+0000","172.21.15.32:0/1877817963":"2024-04-07T14:07:17.560285+0000","172.21.15.86:6828/1279185277":"2024-04-07T14:18:05.813583+0000","172.21.15.86:6829/1279185277":"2024-04-07T14:18:05.813583+0000","172.21.15.32:6801/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.86:0/2712193968":"2024-04-07T14:18:05.813583+0000","172.21.15.32:0/3226572738":"2024-04-07T14:16:30.572936+0000","172.21.15.32:0/2092569962":"2024-04-07T14:21:22.517015+0000","172.21.15.32:6800/2780016783":"2024-04-07T14:21:22.517015+0000","172.21.15.32:6801/2780016783":"2024-04-07T14:21:22.517015+0000","172.21.15.32:0/69668297":"2024-04-07T14:21:22.517015+0000","172.21.15.86:6824/3246813202":"2024-04-07T14:25:48.051344+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T14:58:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f238c06cc20 msgr2=0x7f238c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f238c06cc20 0x7f238c06f0d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f239400b3b0 tx=0x7f2394009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0101c70 msgr2=0x7f23a010e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0101c70 0x7f23a010e950 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f239c00c830 tx=0x7f239c004470 comp rx=0 tx=0).stop 2024-04-06T14:58:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 shutdown_connections 2024-04-06T14:58:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f238c06cc20 0x7f238c06f0d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23a0101c70 0x7f23a010e950 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 --2- 172.21.15.32:0/3293706065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23a0102610 0x7f23a010ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 >> 172.21.15.32:0/3293706065 conn(0x7f23a0071c80 msgr2=0x7f23a00fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 shutdown_connections 2024-04-06T14:58:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.734+0000 7f23a7b73700 1 -- 172.21.15.32:0/3293706065 wait complete. 2024-04-06T14:58:32.733 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T14:11:20.110423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-06T14:14:00.974689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-06T14:14:01.457260+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T14:58:32.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T14:58:32.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 -- 172.21.15.32:0/3334996235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 msgr2=0x7fcfbc103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:32.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 --2- 172.21.15.32:0/3334996235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 0x7fcfbc103150 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcfac004940 tx=0x7fcfac009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:32.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 -- 172.21.15.32:0/3334996235 shutdown_connections 2024-04-06T14:58:32.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 --2- 172.21.15.32:0/3334996235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfbc103720 0x7fcfbc0fb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 --2- 172.21.15.32:0/3334996235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 0x7fcfbc103150 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 -- 172.21.15.32:0/3334996235 >> 172.21.15.32:0/3334996235 conn(0x7fcfbc072230 msgr2=0x7fcfbc072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 -- 172.21.15.32:0/3334996235 shutdown_connections 2024-04-06T14:58:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 -- 172.21.15.32:0/3334996235 wait complete. 2024-04-06T14:58:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 Processor -- start 2024-04-06T14:58:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.875+0000 7fcfc1a64700 1 -- start start 2024-04-06T14:58:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfc1a64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 0x7fcfbc113740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfc1a64700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfbc103720 0x7fcfbc113c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfc1a64700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc1142d0 con 0x7fcfbc102d80 2024-04-06T14:58:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfc1a64700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc10d8d0 con 0x7fcfbc103720 2024-04-06T14:58:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfbb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 0x7fcfbc113740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfbb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 0x7fcfbc113740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60914/0 (socket says 172.21.15.32:60914) 2024-04-06T14:58:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfbb7fe700 1 -- 172.21.15.32:0/188813729 learned_addr learned my addr 172.21.15.32:0/188813729 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfbaffd700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfbc103720 0x7fcfbc113c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfbb7fe700 1 -- 172.21.15.32:0/188813729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfbc103720 msgr2=0x7fcfbc113c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfbb7fe700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfbc103720 0x7fcfbc113c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfbb7fe700 1 -- 172.21.15.32:0/188813729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac005040 con 0x7fcfbc102d80 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfbb7fe700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 0x7fcfbc113740 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00b3b0 tx=0x7fcfac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfa3fff700 1 -- 172.21.15.32:0/188813729 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfac00e040 con 0x7fcfbc102d80 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfbc10db50 con 0x7fcfbc102d80 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfa3fff700 1 -- 172.21.15.32:0/188813729 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfac013420 con 0x7fcfbc102d80 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfa3fff700 1 -- 172.21.15.32:0/188813729 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfac01c630 con 0x7fcfbc102d80 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.876+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfbc10e110 con 0x7fcfbc102d80 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.877+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfbc1000e0 con 0x7fcfbc102d80 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.878+0000 7fcfa3fff700 1 -- 172.21.15.32:0/188813729 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcfac013970 con 0x7fcfbc102d80 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.878+0000 7fcfa3fff700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfa406cc20 0x7fcfa406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.878+0000 7fcfa3fff700 1 -- 172.21.15.32:0/188813729 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcfac08d290 con 0x7fcfbc102d80 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.878+0000 7fcfbaffd700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfa406cc20 0x7fcfa406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.879+0000 7fcfbaffd700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfa406cc20 0x7fcfa406f0d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc10ea90 tx=0x7fcfb000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:32.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:32.883+0000 7fcfa3fff700 1 -- 172.21.15.32:0/188813729 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcfac05ad60 con 0x7fcfbc102d80 2024-04-06T14:58:33.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.087+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfbc10ed90 con 0x7fcfbc102d80 2024-04-06T14:58:33.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.088+0000 7fcfa3fff700 1 -- 172.21.15.32:0/188813729 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcfac05a8f0 con 0x7fcfbc102d80 2024-04-06T14:58:33.074 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-06T14:58:33.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfa406cc20 msgr2=0x7fcfa406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfa406cc20 0x7fcfa406f0d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc10ea90 tx=0x7fcfb000b040 comp rx=0 tx=0).stop 2024-04-06T14:58:33.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 msgr2=0x7fcfbc113740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 0x7fcfbc113740 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00b3b0 tx=0x7fcfac009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:33.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 shutdown_connections 2024-04-06T14:58:33.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fcfa406cc20 0x7fcfa406f0d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfbc102d80 0x7fcfbc113740 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 --2- 172.21.15.32:0/188813729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfbc103720 0x7fcfbc113c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 >> 172.21.15.32:0/188813729 conn(0x7fcfbc072230 msgr2=0x7fcfbc0fac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 shutdown_connections 2024-04-06T14:58:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.090+0000 7fcfc1a64700 1 -- 172.21.15.32:0/188813729 wait complete. 2024-04-06T14:58:33.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:32 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3293706065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:58:33.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-06T14:58:33.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:32 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3293706065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:58:33.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.227+0000 7f3fcafe6700 1 -- 172.21.15.32:0/2714626548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4102640 msgr2=0x7f3fc40fc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.227+0000 7f3fcafe6700 1 --2- 172.21.15.32:0/2714626548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4102640 0x7f3fc40fc450 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0004940 tx=0x7f3fb0009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:33.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.227+0000 7f3fcafe6700 1 -- 172.21.15.32:0/2714626548 shutdown_connections 2024-04-06T14:58:33.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.227+0000 7f3fcafe6700 1 --2- 172.21.15.32:0/2714626548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4102640 0x7f3fc40fc450 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.227+0000 7f3fcafe6700 1 --2- 172.21.15.32:0/2714626548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc4101ca0 0x7f3fc4102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.227+0000 7f3fcafe6700 1 -- 172.21.15.32:0/2714626548 >> 172.21.15.32:0/2714626548 conn(0x7f3fc4072230 msgr2=0x7f3fc4072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:33.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.227+0000 7f3fcafe6700 1 -- 172.21.15.32:0/2714626548 shutdown_connections 2024-04-06T14:58:33.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.227+0000 7f3fcafe6700 1 -- 172.21.15.32:0/2714626548 wait complete. 2024-04-06T14:58:33.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fcafe6700 1 Processor -- start 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fcafe6700 1 -- start start 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fcafe6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4101ca0 0x7f3fc41137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fcafe6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc4102640 0x7f3fc4113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fcafe6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc4114350 con 0x7f3fc4101ca0 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fcafe6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc410d950 con 0x7f3fc4102640 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fc8d82700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4101ca0 0x7f3fc41137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fc8d82700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4101ca0 0x7f3fc41137c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60946/0 (socket says 172.21.15.32:60946) 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fc8d82700 1 -- 172.21.15.32:0/1947980059 learned_addr learned my addr 172.21.15.32:0/1947980059 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fc8d82700 1 -- 172.21.15.32:0/1947980059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc4102640 msgr2=0x7f3fc4113d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fc3fff700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc4102640 0x7f3fc4113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fc8d82700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc4102640 0x7f3fc4113d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fc8d82700 1 -- 172.21.15.32:0/1947980059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb0005040 con 0x7f3fc4101ca0 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.228+0000 7f3fc8d82700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4101ca0 0x7f3fc41137c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c8c0 tx=0x7f3fb80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.229+0000 7f3fc17fa700 1 -- 172.21.15.32:0/1947980059 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb800a980 con 0x7f3fc4101ca0 2024-04-06T14:58:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.229+0000 7f3fc17fa700 1 -- 172.21.15.32:0/1947980059 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fb800ce60 con 0x7f3fc4101ca0 2024-04-06T14:58:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.229+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc410dc30 con 0x7f3fc4101ca0 2024-04-06T14:58:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.229+0000 7f3fc17fa700 1 -- 172.21.15.32:0/1947980059 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb80113a0 con 0x7f3fc4101ca0 2024-04-06T14:58:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.229+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc410e170 con 0x7f3fc4101ca0 2024-04-06T14:58:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.230+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc40fd460 con 0x7f3fc4101ca0 2024-04-06T14:58:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.230+0000 7f3fc17fa700 1 -- 172.21.15.32:0/1947980059 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3fb8012400 con 0x7f3fc4101ca0 2024-04-06T14:58:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.231+0000 7f3fc17fa700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fb406cc20 0x7f3fb406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.231+0000 7f3fc3fff700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fb406cc20 0x7f3fb406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.231+0000 7f3fc17fa700 1 -- 172.21.15.32:0/1947980059 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3fb808c160 con 0x7f3fc4101ca0 2024-04-06T14:58:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.231+0000 7f3fc3fff700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fb406cc20 0x7f3fb406f0d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009fd0 tx=0x7f3fb0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:33.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.235+0000 7f3fc17fa700 1 -- 172.21.15.32:0/1947980059 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3fb8059c30 con 0x7f3fc4101ca0 2024-04-06T14:58:33.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.438+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f3fc4108c40 con 0x7f3fc4101ca0 2024-04-06T14:58:33.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.439+0000 7f3fc17fa700 1 -- 172.21.15.32:0/1947980059 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3fb80597c0 con 0x7f3fc4101ca0 2024-04-06T14:58:33.425 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fb406cc20 msgr2=0x7f3fb406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fb406cc20 0x7f3fb406f0d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009fd0 tx=0x7f3fb0009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4101ca0 msgr2=0x7f3fc41137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4101ca0 0x7f3fc41137c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c8c0 tx=0x7f3fb80044c0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 shutdown_connections 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3fb406cc20 0x7f3fb406f0d0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc4101ca0 0x7f3fc41137c0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 --2- 172.21.15.32:0/1947980059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc4102640 0x7f3fc4113d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 >> 172.21.15.32:0/1947980059 conn(0x7f3fc4072230 msgr2=0x7f3fc4106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 shutdown_connections 2024-04-06T14:58:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.441+0000 7f3fcafe6700 1 -- 172.21.15.32:0/1947980059 wait complete. 2024-04-06T14:58:33.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-06T14:58:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.578+0000 7f3f661e1700 1 -- 172.21.15.32:0/2512694334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60101be0 msgr2=0x7f3f60101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.578+0000 7f3f661e1700 1 --2- 172.21.15.32:0/2512694334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60101be0 0x7f3f60101fb0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f3f48004940 tx=0x7f3f48009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.578+0000 7f3f661e1700 1 -- 172.21.15.32:0/2512694334 shutdown_connections 2024-04-06T14:58:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.578+0000 7f3f661e1700 1 --2- 172.21.15.32:0/2512694334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60102580 0x7f3f600fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.578+0000 7f3f661e1700 1 --2- 172.21.15.32:0/2512694334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60101be0 0x7f3f60101fb0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.578+0000 7f3f661e1700 1 -- 172.21.15.32:0/2512694334 >> 172.21.15.32:0/2512694334 conn(0x7f3f60072ad0 msgr2=0x7f3f60072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.578+0000 7f3f661e1700 1 -- 172.21.15.32:0/2512694334 shutdown_connections 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.578+0000 7f3f661e1700 1 -- 172.21.15.32:0/2512694334 wait complete. 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.578+0000 7f3f661e1700 1 Processor -- start 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f661e1700 1 -- start start 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f661e1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60101be0 0x7f3f6010e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f661e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60102580 0x7f3f6010eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f661e1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6010f3f0 con 0x7f3f60102580 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f661e1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6010f530 con 0x7f3f60101be0 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f5effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60102580 0x7f3f6010eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f5effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60102580 0x7f3f6010eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60966/0 (socket says 172.21.15.32:60966) 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f5effd700 1 -- 172.21.15.32:0/2809969027 learned_addr learned my addr 172.21.15.32:0/2809969027 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f5f7fe700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60101be0 0x7f3f6010e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f5effd700 1 -- 172.21.15.32:0/2809969027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60101be0 msgr2=0x7f3f6010e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f5effd700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60101be0 0x7f3f6010e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.579+0000 7f3f5effd700 1 -- 172.21.15.32:0/2809969027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f48005040 con 0x7f3f60102580 2024-04-06T14:58:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.580+0000 7f3f5effd700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60102580 0x7f3f6010eeb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f3f50002a50 tx=0x7f3f50004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.580+0000 7f3f57fff700 1 -- 172.21.15.32:0/2809969027 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f50007cc0 con 0x7f3f60102580 2024-04-06T14:58:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.580+0000 7f3f57fff700 1 -- 172.21.15.32:0/2809969027 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f5000f040 con 0x7f3f60102580 2024-04-06T14:58:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.580+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f601179e0 con 0x7f3f60102580 2024-04-06T14:58:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.580+0000 7f3f57fff700 1 -- 172.21.15.32:0/2809969027 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f5000d920 con 0x7f3f60102580 2024-04-06T14:58:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.580+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f60117db0 con 0x7f3f60102580 2024-04-06T14:58:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.581+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f600fd330 con 0x7f3f60102580 2024-04-06T14:58:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.581+0000 7f3f57fff700 1 -- 172.21.15.32:0/2809969027 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3f50009680 con 0x7f3f60102580 2024-04-06T14:58:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.582+0000 7f3f57fff700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f4c06cc20 0x7f3f4c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.582+0000 7f3f57fff700 1 -- 172.21.15.32:0/2809969027 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3f5008bde0 con 0x7f3f60102580 2024-04-06T14:58:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.582+0000 7f3f5f7fe700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f4c06cc20 0x7f3f4c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.582+0000 7f3f5f7fe700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f4c06cc20 0x7f3f4c06f0d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800b840 tx=0x7f3f48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.586+0000 7f3f57fff700 1 -- 172.21.15.32:0/2809969027 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3f50055da0 con 0x7f3f60102580 2024-04-06T14:58:33.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.790+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f3f60064970 con 0x7f3f60102580 2024-04-06T14:58:33.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.790+0000 7f3f57fff700 1 -- 172.21.15.32:0/2809969027 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3f500593c0 con 0x7f3f60102580 2024-04-06T14:58:33.776 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.792+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f4c06cc20 msgr2=0x7f3f4c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.792+0000 7f3f661e1700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f4c06cc20 0x7f3f4c06f0d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800b840 tx=0x7f3f48009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.792+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60102580 msgr2=0x7f3f6010eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.792+0000 7f3f661e1700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60102580 0x7f3f6010eeb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f3f50002a50 tx=0x7f3f50004600 comp rx=0 tx=0).stop 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.792+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 shutdown_connections 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.792+0000 7f3f661e1700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f3f4c06cc20 0x7f3f4c06f0d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.792+0000 7f3f661e1700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60101be0 0x7f3f6010e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.792+0000 7f3f661e1700 1 --2- 172.21.15.32:0/2809969027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f60102580 0x7f3f6010eeb0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.792+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 >> 172.21.15.32:0/2809969027 conn(0x7f3f60072ad0 msgr2=0x7f3f60104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.793+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 shutdown_connections 2024-04-06T14:58:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.793+0000 7f3f661e1700 1 -- 172.21.15.32:0/2809969027 wait complete. 2024-04-06T14:58:33.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T14:58:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.931+0000 7fe10bb10700 1 -- 172.21.15.32:0/3660338339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 msgr2=0x7fe1040fbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.931+0000 7fe10bb10700 1 --2- 172.21.15.32:0/3660338339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 0x7fe1040fbc80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4004940 tx=0x7fe0f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.931+0000 7fe10bb10700 1 -- 172.21.15.32:0/3660338339 shutdown_connections 2024-04-06T14:58:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.931+0000 7fe10bb10700 1 --2- 172.21.15.32:0/3660338339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 0x7fe1040fbc80 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.931+0000 7fe10bb10700 1 --2- 172.21.15.32:0/3660338339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe104101c70 0x7fe104102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.931+0000 7fe10bb10700 1 -- 172.21.15.32:0/3660338339 >> 172.21.15.32:0/3660338339 conn(0x7fe104071c80 msgr2=0x7fe104072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.931+0000 7fe10bb10700 1 -- 172.21.15.32:0/3660338339 shutdown_connections 2024-04-06T14:58:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.931+0000 7fe10bb10700 1 -- 172.21.15.32:0/3660338339 wait complete. 2024-04-06T14:58:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe10bb10700 1 Processor -- start 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe10bb10700 1 -- start start 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe10bb10700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe104101c70 0x7fe1041137a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe10bb10700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 0x7fe104113ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe10bb10700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe104114330 con 0x7fe104102610 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe10bb10700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10410d930 con 0x7fe104101c70 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe1090ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 0x7fe104113ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe1090ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 0x7fe104113ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60990/0 (socket says 172.21.15.32:60990) 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe1090ab700 1 -- 172.21.15.32:0/3064189744 learned_addr learned my addr 172.21.15.32:0/3064189744 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe1098ac700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe104101c70 0x7fe1041137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe1090ab700 1 -- 172.21.15.32:0/3064189744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe104101c70 msgr2=0x7fe1041137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe1090ab700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe104101c70 0x7fe1041137a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe1090ab700 1 -- 172.21.15.32:0/3064189744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f4005040 con 0x7fe104102610 2024-04-06T14:58:34.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.932+0000 7fe1090ab700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 0x7fe104113ce0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009390 tx=0x7fe0f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:34.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.933+0000 7fe0fa7fc700 1 -- 172.21.15.32:0/3064189744 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f400e040 con 0x7fe104102610 2024-04-06T14:58:34.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.933+0000 7fe0fa7fc700 1 -- 172.21.15.32:0/3064189744 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0f4013420 con 0x7fe104102610 2024-04-06T14:58:34.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.933+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10410dbb0 con 0x7fe104102610 2024-04-06T14:58:34.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.933+0000 7fe0fa7fc700 1 -- 172.21.15.32:0/3064189744 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f401c630 con 0x7fe104102610 2024-04-06T14:58:34.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.933+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10410e170 con 0x7fe104102610 2024-04-06T14:58:34.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.933+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1040fd3c0 con 0x7fe104102610 2024-04-06T14:58:34.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.934+0000 7fe0fa7fc700 1 -- 172.21.15.32:0/3064189744 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe0f40139e0 con 0x7fe104102610 2024-04-06T14:58:34.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.935+0000 7fe0fa7fc700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f006cc20 0x7fe0f006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:34.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.935+0000 7fe0fa7fc700 1 -- 172.21.15.32:0/3064189744 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe0f408d2f0 con 0x7fe104102610 2024-04-06T14:58:34.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.935+0000 7fe1098ac700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f006cc20 0x7fe0f006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:34.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.935+0000 7fe1098ac700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f006cc20 0x7fe0f006f0d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe100004990 tx=0x7fe100009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:34.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:33.939+0000 7fe0fa7fc700 1 -- 172.21.15.32:0/3064189744 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe0f405adc0 con 0x7fe104102610 2024-04-06T14:58:34.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:33 smithi086 ceph-mon[110975]: pgmap v1156: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:34.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:33 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/188813729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:58:34.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:33 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1947980059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T14:58:34.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:33 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2809969027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T14:58:34.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.187+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe104064970 con 0x7fe104102610 2024-04-06T14:58:34.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.187+0000 7fe0fa7fc700 1 -- 172.21.15.32:0/3064189744 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4845 (secure 0 0 0) 0x7fe0f405a950 con 0x7fe104102610 2024-04-06T14:58:34.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:34.174 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44803,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6827/2158771540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2158771540},{"type":"v1","addr":"172.21.15.86:6827","nonce":2158771540}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:56:49.438197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34382,"mds_1":34414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34382":{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":0,"incarnation":34,"state":"up:active","state_seq":467,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34414":{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":1,"incarnation":34,"state":"up:active","state_seq":462,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:34.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f006cc20 msgr2=0x7fe0f006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f006cc20 0x7fe0f006f0d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe100004990 tx=0x7fe100009ee0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 msgr2=0x7fe104113ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 0x7fe104113ce0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009390 tx=0x7fe0f4009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 shutdown_connections 2024-04-06T14:58:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe0f006cc20 0x7fe0f006f0d0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe104101c70 0x7fe1041137a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 --2- 172.21.15.32:0/3064189744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe104102610 0x7fe104113ce0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 >> 172.21.15.32:0/3064189744 conn(0x7fe104071c80 msgr2=0x7fe1040651d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 shutdown_connections 2024-04-06T14:58:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.189+0000 7fe10bb10700 1 -- 172.21.15.32:0/3064189744 wait complete. 2024-04-06T14:58:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 39 2024-04-06T14:58:34.186 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:33 smithi032 ceph-mon[147414]: pgmap v1156: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:34.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:33 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/188813729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:58:34.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:33 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1947980059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T14:58:34.187 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:33 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2809969027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T14:58:34.188 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 18} 2024-04-06T14:58:34.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-06T14:58:34.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.326+0000 7fd5db65c700 1 -- 172.21.15.32:0/944011603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 msgr2=0x7fd5d40fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.326+0000 7fd5db65c700 1 --2- 172.21.15.32:0/944011603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 0x7fd5d40fccf0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0004940 tx=0x7fd5d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.326+0000 7fd5db65c700 1 -- 172.21.15.32:0/944011603 shutdown_connections 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.326+0000 7fd5db65c700 1 --2- 172.21.15.32:0/944011603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 0x7fd5d40fccf0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.326+0000 7fd5db65c700 1 --2- 172.21.15.32:0/944011603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5d4102ce0 0x7fd5d41030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.326+0000 7fd5db65c700 1 -- 172.21.15.32:0/944011603 >> 172.21.15.32:0/944011603 conn(0x7fd5d40f8890 msgr2=0x7fd5d40faca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.326+0000 7fd5db65c700 1 -- 172.21.15.32:0/944011603 shutdown_connections 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.326+0000 7fd5db65c700 1 -- 172.21.15.32:0/944011603 wait complete. 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5db65c700 1 Processor -- start 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5db65c700 1 -- start start 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5db65c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5d4102ce0 0x7fd5d4113740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:34.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5db65c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 0x7fd5d4113c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5db65c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d41142d0 con 0x7fd5d4103680 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5db65c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d410d8d0 con 0x7fd5d4102ce0 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5d8bf7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 0x7fd5d4113c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5d8bf7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 0x7fd5d4113c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32770/0 (socket says 172.21.15.32:32770) 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5d8bf7700 1 -- 172.21.15.32:0/1471154031 learned_addr learned my addr 172.21.15.32:0/1471154031 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5d93f8700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5d4102ce0 0x7fd5d4113740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5d8bf7700 1 -- 172.21.15.32:0/1471154031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5d4102ce0 msgr2=0x7fd5d4113740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5d8bf7700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5d4102ce0 0x7fd5d4113740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.327+0000 7fd5d8bf7700 1 -- 172.21.15.32:0/1471154031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d0005040 con 0x7fd5d4103680 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.328+0000 7fd5d8bf7700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 0x7fd5d4113c80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009380 tx=0x7fd5d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.328+0000 7fd5c9ffb700 1 -- 172.21.15.32:0/1471154031 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5d000e040 con 0x7fd5d4103680 2024-04-06T14:58:34.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.328+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d410db50 con 0x7fd5d4103680 2024-04-06T14:58:34.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.328+0000 7fd5c9ffb700 1 -- 172.21.15.32:0/1471154031 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5d0013420 con 0x7fd5d4103680 2024-04-06T14:58:34.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.328+0000 7fd5c9ffb700 1 -- 172.21.15.32:0/1471154031 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5d001c630 con 0x7fd5d4103680 2024-04-06T14:58:34.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.328+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d410e110 con 0x7fd5d4103680 2024-04-06T14:58:34.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.329+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d40fe430 con 0x7fd5d4103680 2024-04-06T14:58:34.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.329+0000 7fd5c9ffb700 1 -- 172.21.15.32:0/1471154031 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd5d00139b0 con 0x7fd5d4103680 2024-04-06T14:58:34.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.330+0000 7fd5c9ffb700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5c006cc20 0x7fd5c006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:34.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.330+0000 7fd5c9ffb700 1 -- 172.21.15.32:0/1471154031 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd5d008d1b0 con 0x7fd5d4103680 2024-04-06T14:58:34.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.330+0000 7fd5d93f8700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5c006cc20 0x7fd5c006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:34.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.330+0000 7fd5d93f8700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5c006cc20 0x7fd5c006f0d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4004f40 tx=0x7fd5c400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:34.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.334+0000 7fd5c9ffb700 1 -- 172.21.15.32:0/1471154031 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd5d005ac80 con 0x7fd5d4103680 2024-04-06T14:58:34.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.589+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7fd5d4109c80 con 0x7fd5d4103680 2024-04-06T14:58:34.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.590+0000 7fd5c9ffb700 1 -- 172.21.15.32:0/1471154031 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v39) v1 ==== 107+0+4838 (secure 0 0 0) 0x7fd5d005a810 con 0x7fd5d4103680 2024-04-06T14:58:34.576 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:34.576 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14472,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/179354627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":179354627},{"type":"v1","addr":"172.21.15.32:6827","nonce":179354627}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:16:07.889980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14460,"mds_1":24301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14460":{"gid":14460,"name":"cephfs.smithi032.mqoaof","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/1704327767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1704327767},{"type":"v1","addr":"172.21.15.32:6829","nonce":1704327767}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24301":{"gid":24301,"name":"cephfs.smithi086.nikfjj","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6825/3246813202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3246813202},{"type":"v1","addr":"172.21.15.86:6825","nonce":3246813202}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:34.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5c006cc20 msgr2=0x7fd5c006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5c006cc20 0x7fd5c006f0d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4004f40 tx=0x7fd5c400b040 comp rx=0 tx=0).stop 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 msgr2=0x7fd5d4113c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 0x7fd5d4113c80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009380 tx=0x7fd5d0009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 shutdown_connections 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd5c006cc20 0x7fd5c006f0d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5d4102ce0 0x7fd5d4113740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 --2- 172.21.15.32:0/1471154031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d4103680 0x7fd5d4113c80 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 >> 172.21.15.32:0/1471154031 conn(0x7fd5d40f8890 msgr2=0x7fd5d40f9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 shutdown_connections 2024-04-06T14:58:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.592+0000 7fd5db65c700 1 -- 172.21.15.32:0/1471154031 wait complete. 2024-04-06T14:58:34.580 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 19 2024-04-06T14:58:34.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.727+0000 7f7d29712700 1 -- 172.21.15.32:0/2928846815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 msgr2=0x7f7d24106cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.727+0000 7f7d29712700 1 --2- 172.21.15.32:0/2928846815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 0x7f7d24106cd0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f7d14004940 tx=0x7f7d14009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.728+0000 7f7d29712700 1 -- 172.21.15.32:0/2928846815 shutdown_connections 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.728+0000 7f7d29712700 1 --2- 172.21.15.32:0/2928846815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 0x7f7d24106cd0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.728+0000 7f7d29712700 1 --2- 172.21.15.32:0/2928846815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240fdcf0 0x7f7d240fe0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.728+0000 7f7d29712700 1 -- 172.21.15.32:0/2928846815 >> 172.21.15.32:0/2928846815 conn(0x7f7d240f77a0 msgr2=0x7f7d240f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.728+0000 7f7d29712700 1 -- 172.21.15.32:0/2928846815 shutdown_connections 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.728+0000 7f7d29712700 1 -- 172.21.15.32:0/2928846815 wait complete. 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.728+0000 7f7d29712700 1 Processor -- start 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.728+0000 7f7d29712700 1 -- start start 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d29712700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240fdcf0 0x7f7d240fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d29712700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 0x7f7d240fc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d29712700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d240fda70 con 0x7f7d240fe690 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d29712700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d240fc5a0 con 0x7f7d240fdcf0 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d227fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 0x7f7d240fc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d227fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 0x7f7d240fc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50748/0 (socket says 172.21.15.32:50748) 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d227fc700 1 -- 172.21.15.32:0/2235865297 learned_addr learned my addr 172.21.15.32:0/2235865297 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d22ffd700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240fdcf0 0x7f7d240fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d227fc700 1 -- 172.21.15.32:0/2235865297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240fdcf0 msgr2=0x7f7d240fbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d227fc700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240fdcf0 0x7f7d240fbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d227fc700 1 -- 172.21.15.32:0/2235865297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d14005040 con 0x7f7d240fe690 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d227fc700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 0x7f7d240fc060 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f7d14004890 tx=0x7f7d14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d0b7fe700 1 -- 172.21.15.32:0/2235865297 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d1400e040 con 0x7f7d240fe690 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d240fc820 con 0x7f7d240fe690 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d0b7fe700 1 -- 172.21.15.32:0/2235865297 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d14007e10 con 0x7f7d240fe690 2024-04-06T14:58:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d0b7fe700 1 -- 172.21.15.32:0/2235865297 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d14013660 con 0x7f7d240fe690 2024-04-06T14:58:34.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.729+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d24119bf0 con 0x7f7d240fe690 2024-04-06T14:58:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.730+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d24066a70 con 0x7f7d240fe690 2024-04-06T14:58:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.731+0000 7f7d0b7fe700 1 -- 172.21.15.32:0/2235865297 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7d1401c9e0 con 0x7f7d240fe690 2024-04-06T14:58:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.731+0000 7f7d0b7fe700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d0c06cc20 0x7f7d0c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.731+0000 7f7d0b7fe700 1 -- 172.21.15.32:0/2235865297 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7d1408ce00 con 0x7f7d240fe690 2024-04-06T14:58:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.731+0000 7f7d22ffd700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d0c06cc20 0x7f7d0c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:34.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.732+0000 7f7d22ffd700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d0c06cc20 0x7f7d0c06f0d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7d18007fd0 tx=0x7f7d18009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:34.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.736+0000 7f7d0b7fe700 1 -- 172.21.15.32:0/2235865297 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7d14056ef0 con 0x7f7d240fe690 2024-04-06T14:58:34.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:34 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3064189744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T14:58:34.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:34 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1471154031' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-06T14:58:34.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.986+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f7d24109c50 con 0x7f7d240fe690 2024-04-06T14:58:34.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.987+0000 7f7d0b7fe700 1 -- 172.21.15.32:0/2235865297 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v39) v1 ==== 107+0+4839 (secure 0 0 0) 0x7f7d1405a510 con 0x7f7d240fe690 2024-04-06T14:58:34.973 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:34.973 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14472,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/179354627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":179354627},{"type":"v1","addr":"172.21.15.32:6827","nonce":179354627}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:17:08.851867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14460,"mds_1":24301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14460":{"gid":14460,"name":"cephfs.smithi032.mqoaof","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/1704327767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1704327767},{"type":"v1","addr":"172.21.15.32:6829","nonce":1704327767}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24301":{"gid":24301,"name":"cephfs.smithi086.nikfjj","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6825/3246813202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3246813202},{"type":"v1","addr":"172.21.15.86:6825","nonce":3246813202}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d0c06cc20 msgr2=0x7f7d0c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d0c06cc20 0x7f7d0c06f0d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7d18007fd0 tx=0x7f7d18009040 comp rx=0 tx=0).stop 2024-04-06T14:58:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 msgr2=0x7f7d240fc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 0x7f7d240fc060 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f7d14004890 tx=0x7f7d14009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 shutdown_connections 2024-04-06T14:58:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7d0c06cc20 0x7f7d0c06f0d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240fdcf0 0x7f7d240fbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 --2- 172.21.15.32:0/2235865297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240fe690 0x7f7d240fc060 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 >> 172.21.15.32:0/2235865297 conn(0x7f7d240f77a0 msgr2=0x7f7d240f8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:34.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 shutdown_connections 2024-04-06T14:58:34.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:34.989+0000 7f7d29712700 1 -- 172.21.15.32:0/2235865297 wait complete. 2024-04-06T14:58:34.976 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 20 2024-04-06T14:58:34.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-06T14:58:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.125+0000 7fe28620c700 1 -- 172.21.15.32:0/4221461100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280103680 msgr2=0x7fe280064e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.125+0000 7fe28620c700 1 --2- 172.21.15.32:0/4221461100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280103680 0x7fe280064e40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe270004940 tx=0x7fe270009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.125+0000 7fe28620c700 1 -- 172.21.15.32:0/4221461100 shutdown_connections 2024-04-06T14:58:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.125+0000 7fe28620c700 1 --2- 172.21.15.32:0/4221461100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280103680 0x7fe280064e40 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.125+0000 7fe28620c700 1 --2- 172.21.15.32:0/4221461100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe280102ce0 0x7fe2801030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.125+0000 7fe28620c700 1 -- 172.21.15.32:0/4221461100 >> 172.21.15.32:0/4221461100 conn(0x7fe280072ad0 msgr2=0x7fe280072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.125+0000 7fe28620c700 1 -- 172.21.15.32:0/4221461100 shutdown_connections 2024-04-06T14:58:35.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.125+0000 7fe28620c700 1 -- 172.21.15.32:0/4221461100 wait complete. 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.126+0000 7fe28620c700 1 Processor -- start 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.126+0000 7fe28620c700 1 -- start start 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.126+0000 7fe28620c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280102ce0 0x7fe280113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.126+0000 7fe28620c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe280103680 0x7fe280113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.126+0000 7fe28620c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe280114320 con 0x7fe280102ce0 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.126+0000 7fe28620c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28010d920 con 0x7fe280103680 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.126+0000 7fe27f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280102ce0 0x7fe280113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe27f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280102ce0 0x7fe280113790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50778/0 (socket says 172.21.15.32:50778) 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe27f7fe700 1 -- 172.21.15.32:0/211373540 learned_addr learned my addr 172.21.15.32:0/211373540 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe27effd700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe280103680 0x7fe280113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:35.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe27f7fe700 1 -- 172.21.15.32:0/211373540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe280103680 msgr2=0x7fe280113cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe27f7fe700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe280103680 0x7fe280113cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe27f7fe700 1 -- 172.21.15.32:0/211373540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe270005040 con 0x7fe280102ce0 2024-04-06T14:58:35.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe27f7fe700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280102ce0 0x7fe280113790 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe27400c830 tx=0x7fe274004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe267fff700 1 -- 172.21.15.32:0/211373540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe27400a760 con 0x7fe280102ce0 2024-04-06T14:58:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28010dc00 con 0x7fe280102ce0 2024-04-06T14:58:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe267fff700 1 -- 172.21.15.32:0/211373540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe27400ad70 con 0x7fe280102ce0 2024-04-06T14:58:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe267fff700 1 -- 172.21.15.32:0/211373540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2740113a0 con 0x7fe280102ce0 2024-04-06T14:58:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.127+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28010e1c0 con 0x7fe280102ce0 2024-04-06T14:58:35.118 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:34 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3064189744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T14:58:35.118 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:34 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1471154031' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-06T14:58:35.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.128+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2800fe430 con 0x7fe280102ce0 2024-04-06T14:58:35.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.129+0000 7fe267fff700 1 -- 172.21.15.32:0/211373540 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe27400a8c0 con 0x7fe280102ce0 2024-04-06T14:58:35.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.129+0000 7fe267fff700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe26806cc20 0x7fe26806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:35.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.129+0000 7fe267fff700 1 -- 172.21.15.32:0/211373540 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe27408c0b0 con 0x7fe280102ce0 2024-04-06T14:58:35.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.129+0000 7fe27effd700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe26806cc20 0x7fe26806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:35.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.130+0000 7fe27effd700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe26806cc20 0x7fe26806f0d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe28010eae0 tx=0x7fe270009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:35.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.133+0000 7fe267fff700 1 -- 172.21.15.32:0/211373540 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe274059b80 con 0x7fe280102ce0 2024-04-06T14:58:35.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.386+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7fe280109c80 con 0x7fe280102ce0 2024-04-06T14:58:35.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.387+0000 7fe267fff700 1 -- 172.21.15.32:0/211373540 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v39) v1 ==== 107+0+4838 (secure 0 0 0) 0x7fe274059710 con 0x7fe280102ce0 2024-04-06T14:58:35.373 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:35.373 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14472,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/179354627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":179354627},{"type":"v1","addr":"172.21.15.32:6827","nonce":179354627}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:21:17.332598+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14460,"mds_1":24301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14460":{"gid":14460,"name":"cephfs.smithi032.mqoaof","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/1704327767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1704327767},{"type":"v1","addr":"172.21.15.32:6829","nonce":1704327767}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24301":{"gid":24301,"name":"cephfs.smithi086.nikfjj","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6825/3246813202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3246813202},{"type":"v1","addr":"172.21.15.86:6825","nonce":3246813202}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:35.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe26806cc20 msgr2=0x7fe26806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe26806cc20 0x7fe26806f0d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe28010eae0 tx=0x7fe270009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280102ce0 msgr2=0x7fe280113790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280102ce0 0x7fe280113790 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe27400c830 tx=0x7fe274004470 comp rx=0 tx=0).stop 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 shutdown_connections 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe26806cc20 0x7fe26806f0d0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe280102ce0 0x7fe280113790 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 --2- 172.21.15.32:0/211373540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe280103680 0x7fe280113cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 >> 172.21.15.32:0/211373540 conn(0x7fe280072ad0 msgr2=0x7fe280105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 shutdown_connections 2024-04-06T14:58:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.389+0000 7fe28620c700 1 -- 172.21.15.32:0/211373540 wait complete. 2024-04-06T14:58:35.377 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2024-04-06T14:58:35.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-06T14:58:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.525+0000 7fbc53173700 1 -- 172.21.15.32:0/1435852942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 msgr2=0x7fbc4c101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.525+0000 7fbc53173700 1 --2- 172.21.15.32:0/1435852942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 0x7fbc4c101fb0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fbc40004940 tx=0x7fbc40009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.525+0000 7fbc53173700 1 -- 172.21.15.32:0/1435852942 shutdown_connections 2024-04-06T14:58:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.525+0000 7fbc53173700 1 --2- 172.21.15.32:0/1435852942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc4c102580 0x7fbc4c0fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.525+0000 7fbc53173700 1 --2- 172.21.15.32:0/1435852942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 0x7fbc4c101fb0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.525+0000 7fbc53173700 1 -- 172.21.15.32:0/1435852942 >> 172.21.15.32:0/1435852942 conn(0x7fbc4c072ad0 msgr2=0x7fbc4c072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.525+0000 7fbc53173700 1 -- 172.21.15.32:0/1435852942 shutdown_connections 2024-04-06T14:58:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.525+0000 7fbc53173700 1 -- 172.21.15.32:0/1435852942 wait complete. 2024-04-06T14:58:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.526+0000 7fbc53173700 1 Processor -- start 2024-04-06T14:58:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.526+0000 7fbc53173700 1 -- start start 2024-04-06T14:58:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.526+0000 7fbc53173700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 0x7fbc4c10e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.526+0000 7fbc53173700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc4c102580 0x7fbc4c10ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.526+0000 7fbc53173700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c10f3d0 con 0x7fbc4c101be0 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.526+0000 7fbc53173700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c10f510 con 0x7fbc4c102580 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc50f0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 0x7fbc4c10e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc50f0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 0x7fbc4c10e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50796/0 (socket says 172.21.15.32:50796) 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc50f0f700 1 -- 172.21.15.32:0/1187424917 learned_addr learned my addr 172.21.15.32:0/1187424917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc4bfff700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc4c102580 0x7fbc4c10ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc50f0f700 1 -- 172.21.15.32:0/1187424917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc4c102580 msgr2=0x7fbc4c10ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc50f0f700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc4c102580 0x7fbc4c10ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc50f0f700 1 -- 172.21.15.32:0/1187424917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc40005040 con 0x7fbc4c101be0 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc50f0f700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 0x7fbc4c10e950 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000b3b0 tx=0x7fbc40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc497fa700 1 -- 172.21.15.32:0/1187424917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc4000e040 con 0x7fbc4c101be0 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc497fa700 1 -- 172.21.15.32:0/1187424917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc40013420 con 0x7fbc4c101be0 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc497fa700 1 -- 172.21.15.32:0/1187424917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc4001c610 con 0x7fbc4c101be0 2024-04-06T14:58:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c10f790 con 0x7fbc4c101be0 2024-04-06T14:58:35.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.527+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c117cc0 con 0x7fbc4c101be0 2024-04-06T14:58:35.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.528+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4c0fd330 con 0x7fbc4c101be0 2024-04-06T14:58:35.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.529+0000 7fbc497fa700 1 -- 172.21.15.32:0/1187424917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbc40013a00 con 0x7fbc4c101be0 2024-04-06T14:58:35.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.529+0000 7fbc497fa700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc3c06cc20 0x7fbc3c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:35.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.529+0000 7fbc497fa700 1 -- 172.21.15.32:0/1187424917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbc4008d190 con 0x7fbc4c101be0 2024-04-06T14:58:35.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.529+0000 7fbc4bfff700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc3c06cc20 0x7fbc3c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:35.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.530+0000 7fbc4bfff700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc3c06cc20 0x7fbc3c06f0d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0655b0 tx=0x7fbc3800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:35.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.534+0000 7fbc497fa700 1 -- 172.21.15.32:0/1187424917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbc400571d0 con 0x7fbc4c101be0 2024-04-06T14:58:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.795+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7fbc4c065390 con 0x7fbc4c101be0 2024-04-06T14:58:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.795+0000 7fbc497fa700 1 -- 172.21.15.32:0/1187424917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v39) v1 ==== 107+0+4837 (secure 0 0 0) 0x7fbc4005a7f0 con 0x7fbc4c101be0 2024-04-06T14:58:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:35.782 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14472,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/179354627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":179354627},{"type":"v1","addr":"172.21.15.32:6827","nonce":179354627}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:21:28.458361+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14460,"mds_1":24301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14460":{"gid":14460,"name":"cephfs.smithi032.mqoaof","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/1704327767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1704327767},{"type":"v1","addr":"172.21.15.32:6829","nonce":1704327767}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24301":{"gid":24301,"name":"cephfs.smithi086.nikfjj","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6825/3246813202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3246813202},{"type":"v1","addr":"172.21.15.86:6825","nonce":3246813202}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc3c06cc20 msgr2=0x7fbc3c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc3c06cc20 0x7fbc3c06f0d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0655b0 tx=0x7fbc3800b040 comp rx=0 tx=0).stop 2024-04-06T14:58:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 msgr2=0x7fbc4c10e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 0x7fbc4c10e950 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000b3b0 tx=0x7fbc40009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 shutdown_connections 2024-04-06T14:58:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbc3c06cc20 0x7fbc3c06f0d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4c101be0 0x7fbc4c10e950 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 --2- 172.21.15.32:0/1187424917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc4c102580 0x7fbc4c10ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 >> 172.21.15.32:0/1187424917 conn(0x7fbc4c072ad0 msgr2=0x7fbc4c104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 shutdown_connections 2024-04-06T14:58:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.798+0000 7fbc53173700 1 -- 172.21.15.32:0/1187424917 wait complete. 2024-04-06T14:58:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-06T14:58:35.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.935+0000 7ff4654b0700 1 -- 172.21.15.32:0/697931748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460101be0 msgr2=0x7ff460101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.935+0000 7ff4654b0700 1 --2- 172.21.15.32:0/697931748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460101be0 0x7ff460101fb0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff450004940 tx=0x7ff450009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.935+0000 7ff4654b0700 1 -- 172.21.15.32:0/697931748 shutdown_connections 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.935+0000 7ff4654b0700 1 --2- 172.21.15.32:0/697931748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff460102580 0x7ff4600fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.935+0000 7ff4654b0700 1 --2- 172.21.15.32:0/697931748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460101be0 0x7ff460101fb0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.935+0000 7ff4654b0700 1 -- 172.21.15.32:0/697931748 >> 172.21.15.32:0/697931748 conn(0x7ff460072ad0 msgr2=0x7ff460072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.935+0000 7ff4654b0700 1 -- 172.21.15.32:0/697931748 shutdown_connections 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.935+0000 7ff4654b0700 1 -- 172.21.15.32:0/697931748 wait complete. 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.936+0000 7ff4654b0700 1 Processor -- start 2024-04-06T14:58:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.936+0000 7ff4654b0700 1 -- start start 2024-04-06T14:58:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.936+0000 7ff4654b0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff460101be0 0x7ff46010e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.936+0000 7ff4654b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460102580 0x7ff46010edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.936+0000 7ff4654b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46010f310 con 0x7ff460102580 2024-04-06T14:58:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.936+0000 7ff4654b0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46010f450 con 0x7ff460101be0 2024-04-06T14:58:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.936+0000 7ff45e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460102580 0x7ff46010edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.936+0000 7ff45e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460102580 0x7ff46010edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50816/0 (socket says 172.21.15.32:50816) 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.936+0000 7ff45e7fc700 1 -- 172.21.15.32:0/2112271159 learned_addr learned my addr 172.21.15.32:0/2112271159 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff45effd700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff460101be0 0x7ff46010e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff45e7fc700 1 -- 172.21.15.32:0/2112271159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff460101be0 msgr2=0x7ff46010e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff45e7fc700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff460101be0 0x7ff46010e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff45e7fc700 1 -- 172.21.15.32:0/2112271159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450005040 con 0x7ff460102580 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff45e7fc700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460102580 0x7ff46010edd0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff44800eb40 tx=0x7ff44800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff4577fe700 1 -- 172.21.15.32:0/2112271159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff448007cc0 con 0x7ff460102580 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff4577fe700 1 -- 172.21.15.32:0/2112271159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff44800f040 con 0x7ff460102580 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46010f6d0 con 0x7ff460102580 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff4577fe700 1 -- 172.21.15.32:0/2112271159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff448009570 con 0x7ff460102580 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.937+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff460117c50 con 0x7ff460102580 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.938+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4600fd330 con 0x7ff460102580 2024-04-06T14:58:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.939+0000 7ff4577fe700 1 -- 172.21.15.32:0/2112271159 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff4480096d0 con 0x7ff460102580 2024-04-06T14:58:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.939+0000 7ff4577fe700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff44c06cc20 0x7ff44c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.939+0000 7ff4577fe700 1 -- 172.21.15.32:0/2112271159 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff44808bb50 con 0x7ff460102580 2024-04-06T14:58:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.939+0000 7ff45effd700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff44c06cc20 0x7ff44c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.939+0000 7ff45effd700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff44c06cc20 0x7ff44c06f0d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff45000b0d0 tx=0x7ff450009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:35.943+0000 7ff4577fe700 1 -- 172.21.15.32:0/2112271159 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff448055bc0 con 0x7ff460102580 2024-04-06T14:58:36.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:35 smithi086 ceph-mon[110975]: pgmap v1157: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:36.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:35 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2235865297' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-06T14:58:36.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:35 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/211373540' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-06T14:58:36.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:35 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1187424917' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-06T14:58:36.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.197+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7ff460108bc0 con 0x7ff460102580 2024-04-06T14:58:36.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.197+0000 7ff4577fe700 1 -- 172.21.15.32:0/2112271159 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v39) v1 ==== 107+0+3258 (secure 0 0 0) 0x7ff448019cd0 con 0x7ff460102580 2024-04-06T14:58:36.183 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:36.183 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14472,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/179354627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":179354627},{"type":"v1","addr":"172.21.15.32:6827","nonce":179354627}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:36.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff44c06cc20 msgr2=0x7ff44c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff44c06cc20 0x7ff44c06f0d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff45000b0d0 tx=0x7ff450009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460102580 msgr2=0x7ff46010edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460102580 0x7ff46010edd0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff44800eb40 tx=0x7ff44800b040 comp rx=0 tx=0).stop 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 shutdown_connections 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ff44c06cc20 0x7ff44c06f0d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff460101be0 0x7ff46010e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 --2- 172.21.15.32:0/2112271159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff460102580 0x7ff46010edd0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 >> 172.21.15.32:0/2112271159 conn(0x7ff460072ad0 msgr2=0x7ff460104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 shutdown_connections 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.199+0000 7ff4654b0700 1 -- 172.21.15.32:0/2112271159 wait complete. 2024-04-06T14:58:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-04-06T14:58:36.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:35 smithi032 ceph-mon[147414]: pgmap v1157: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:36.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:35 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2235865297' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-06T14:58:36.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:35 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/211373540' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-06T14:58:36.197 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:35 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1187424917' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-06T14:58:36.198 DEBUG:tasks.fs:max_mds not reduced in epoch 23 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:36.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-06T14:58:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.334+0000 7fb6046ea700 1 -- 172.21.15.32:0/2175880328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc102cd0 msgr2=0x7fb5fc1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.334+0000 7fb6046ea700 1 --2- 172.21.15.32:0/2175880328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc102cd0 0x7fb5fc1030a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8004940 tx=0x7fb5f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.334+0000 7fb6046ea700 1 -- 172.21.15.32:0/2175880328 shutdown_connections 2024-04-06T14:58:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.334+0000 7fb6046ea700 1 --2- 172.21.15.32:0/2175880328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5fc103670 0x7fb5fc0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.334+0000 7fb6046ea700 1 --2- 172.21.15.32:0/2175880328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc102cd0 0x7fb5fc1030a0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.334+0000 7fb6046ea700 1 -- 172.21.15.32:0/2175880328 >> 172.21.15.32:0/2175880328 conn(0x7fb5fc0f8820 msgr2=0x7fb5fc0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.334+0000 7fb6046ea700 1 -- 172.21.15.32:0/2175880328 shutdown_connections 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.334+0000 7fb6046ea700 1 -- 172.21.15.32:0/2175880328 wait complete. 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb6046ea700 1 Processor -- start 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb6046ea700 1 -- start start 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb6046ea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5fc102cd0 0x7fb5fc1137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb6046ea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc103670 0x7fb5fc113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb6046ea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc114350 con 0x7fb5fc103670 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb6046ea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc10d950 con 0x7fb5fc102cd0 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb601c85700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc103670 0x7fb5fc113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb601c85700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc103670 0x7fb5fc113d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50830/0 (socket says 172.21.15.32:50830) 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb601c85700 1 -- 172.21.15.32:0/1637762623 learned_addr learned my addr 172.21.15.32:0/1637762623 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb602486700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5fc102cd0 0x7fb5fc1137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb601c85700 1 -- 172.21.15.32:0/1637762623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5fc102cd0 msgr2=0x7fb5fc1137c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb601c85700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5fc102cd0 0x7fb5fc1137c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.335+0000 7fb601c85700 1 -- 172.21.15.32:0/1637762623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f8005040 con 0x7fb5fc103670 2024-04-06T14:58:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.336+0000 7fb601c85700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc103670 0x7fb5fc113d00 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00c800 tx=0x7fb5ec004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.336+0000 7fb5f2ffd700 1 -- 172.21.15.32:0/1637762623 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5ec00a760 con 0x7fb5fc103670 2024-04-06T14:58:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.336+0000 7fb5f2ffd700 1 -- 172.21.15.32:0/1637762623 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5ec00ad70 con 0x7fb5fc103670 2024-04-06T14:58:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.336+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc10dc30 con 0x7fb5fc103670 2024-04-06T14:58:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.336+0000 7fb5f2ffd700 1 -- 172.21.15.32:0/1637762623 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5ec0113a0 con 0x7fb5fc103670 2024-04-06T14:58:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.336+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc10e1f0 con 0x7fb5fc103670 2024-04-06T14:58:36.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.337+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5fc0fe420 con 0x7fb5fc103670 2024-04-06T14:58:36.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.337+0000 7fb5f2ffd700 1 -- 172.21.15.32:0/1637762623 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb5ec012400 con 0x7fb5fc103670 2024-04-06T14:58:36.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.338+0000 7fb5f2ffd700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb5e806cc20 0x7fb5e806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:36.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.338+0000 7fb5f2ffd700 1 -- 172.21.15.32:0/1637762623 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb5ec08bf60 con 0x7fb5fc103670 2024-04-06T14:58:36.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.338+0000 7fb602486700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb5e806cc20 0x7fb5e806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:36.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.338+0000 7fb602486700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb5e806cc20 0x7fb5e806f0d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8004f90 tx=0x7fb5f8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:36.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.342+0000 7fb5f2ffd700 1 -- 172.21.15.32:0/1637762623 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb5ec0bfab0 con 0x7fb5fc103670 2024-04-06T14:58:36.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.595+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7fb5fc109c90 con 0x7fb5fc103670 2024-04-06T14:58:36.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.596+0000 7fb5f2ffd700 1 -- 172.21.15.32:0/1637762623 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v39) v1 ==== 107+0+4813 (secure 0 0 0) 0x7fb5ec059af0 con 0x7fb5fc103670 2024-04-06T14:58:36.582 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:36.582 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14472,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/179354627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":179354627},{"type":"v1","addr":"172.21.15.32:6827","nonce":179354627}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34378,"name":"cephfs.smithi032.mqoaof","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/629297924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":629297924},{"type":"v1","addr":"172.21.15.32:6829","nonce":629297924}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44283,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/4145187217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4145187217},{"type":"v1","addr":"172.21.15.86:6825","nonce":4145187217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb5e806cc20 msgr2=0x7fb5e806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb5e806cc20 0x7fb5e806f0d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8004f90 tx=0x7fb5f8009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc103670 msgr2=0x7fb5fc113d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc103670 0x7fb5fc113d00 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00c800 tx=0x7fb5ec004470 comp rx=0 tx=0).stop 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 shutdown_connections 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb5e806cc20 0x7fb5e806f0d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5fc102cd0 0x7fb5fc1137c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 --2- 172.21.15.32:0/1637762623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5fc103670 0x7fb5fc113d00 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 >> 172.21.15.32:0/1637762623 conn(0x7fb5fc0f8820 msgr2=0x7fb5fc0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.598+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 shutdown_connections 2024-04-06T14:58:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.599+0000 7fb6046ea700 1 -- 172.21.15.32:0/1637762623 wait complete. 2024-04-06T14:58:36.586 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-04-06T14:58:36.597 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:36.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-06T14:58:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.733+0000 7ffb117b4700 1 -- 172.21.15.32:0/431496575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c102cd0 msgr2=0x7ffb0c1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.733+0000 7ffb117b4700 1 --2- 172.21.15.32:0/431496575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c102cd0 0x7ffb0c1030a0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ffafc004940 tx=0x7ffafc009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.733+0000 7ffb117b4700 1 -- 172.21.15.32:0/431496575 shutdown_connections 2024-04-06T14:58:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.733+0000 7ffb117b4700 1 --2- 172.21.15.32:0/431496575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb0c103670 0x7ffb0c0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.733+0000 7ffb117b4700 1 --2- 172.21.15.32:0/431496575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c102cd0 0x7ffb0c1030a0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.733+0000 7ffb117b4700 1 -- 172.21.15.32:0/431496575 >> 172.21.15.32:0/431496575 conn(0x7ffb0c0f8820 msgr2=0x7ffb0c0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.733+0000 7ffb117b4700 1 -- 172.21.15.32:0/431496575 shutdown_connections 2024-04-06T14:58:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.733+0000 7ffb117b4700 1 -- 172.21.15.32:0/431496575 wait complete. 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb117b4700 1 Processor -- start 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb117b4700 1 -- start start 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb117b4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb0c102cd0 0x7ffb0c1137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb117b4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c103670 0x7ffb0c113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb117b4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c114350 con 0x7ffb0c103670 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb117b4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c10d950 con 0x7ffb0c102cd0 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb0a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c103670 0x7ffb0c113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb0a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c103670 0x7ffb0c113d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50854/0 (socket says 172.21.15.32:50854) 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb0a7fc700 1 -- 172.21.15.32:0/1184824066 learned_addr learned my addr 172.21.15.32:0/1184824066 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb0affd700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb0c102cd0 0x7ffb0c1137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb0a7fc700 1 -- 172.21.15.32:0/1184824066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb0c102cd0 msgr2=0x7ffb0c1137c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb0a7fc700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb0c102cd0 0x7ffb0c1137c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.734+0000 7ffb0a7fc700 1 -- 172.21.15.32:0/1184824066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc005040 con 0x7ffb0c103670 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.735+0000 7ffb0a7fc700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c103670 0x7ffb0c113d00 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000c890 tx=0x7ffb000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.735+0000 7ffaf37fe700 1 -- 172.21.15.32:0/1184824066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb0000a7d0 con 0x7ffb0c103670 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.735+0000 7ffaf37fe700 1 -- 172.21.15.32:0/1184824066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb0000ade0 con 0x7ffb0c103670 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.735+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0c10dc30 con 0x7ffb0c103670 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.735+0000 7ffaf37fe700 1 -- 172.21.15.32:0/1184824066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb000113a0 con 0x7ffb0c103670 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.735+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c10e1f0 con 0x7ffb0c103670 2024-04-06T14:58:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.736+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c0fe420 con 0x7ffb0c103670 2024-04-06T14:58:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.736+0000 7ffaf37fe700 1 -- 172.21.15.32:0/1184824066 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ffb00012400 con 0x7ffb0c103670 2024-04-06T14:58:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.737+0000 7ffaf37fe700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffaf406cc20 0x7ffaf406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.737+0000 7ffaf37fe700 1 -- 172.21.15.32:0/1184824066 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffb0008c180 con 0x7ffb0c103670 2024-04-06T14:58:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.737+0000 7ffb0affd700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffaf406cc20 0x7ffaf406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.737+0000 7ffb0affd700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffaf406cc20 0x7ffaf406f0d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00b020 tx=0x7ffafc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:36.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.742+0000 7ffaf37fe700 1 -- 172.21.15.32:0/1184824066 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ffb00059c50 con 0x7ffb0c103670 2024-04-06T14:58:36.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:36 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2112271159' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-06T14:58:36.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:36 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1637762623' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-06T14:58:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.994+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7ffb0c109c90 con 0x7ffb0c103670 2024-04-06T14:58:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.994+0000 7ffaf37fe700 1 -- 172.21.15.32:0/1184824066 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v39) v1 ==== 107+0+4037 (secure 0 0 0) 0x7ffb000597e0 con 0x7ffb0c103670 2024-04-06T14:58:36.981 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:36.981 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34378,"name":"cephfs.smithi032.mqoaof","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/629297924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":629297924},{"type":"v1","addr":"172.21.15.32:6829","nonce":629297924}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44283,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/4145187217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4145187217},{"type":"v1","addr":"172.21.15.86:6825","nonce":4145187217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffaf406cc20 msgr2=0x7ffaf406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffaf406cc20 0x7ffaf406f0d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00b020 tx=0x7ffafc009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c103670 msgr2=0x7ffb0c113d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c103670 0x7ffb0c113d00 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000c890 tx=0x7ffb000044c0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 shutdown_connections 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7ffaf406cc20 0x7ffaf406f0d0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb0c102cd0 0x7ffb0c1137c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 --2- 172.21.15.32:0/1184824066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb0c103670 0x7ffb0c113d00 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 >> 172.21.15.32:0/1184824066 conn(0x7ffb0c0f8820 msgr2=0x7ffb0c0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.996+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 shutdown_connections 2024-04-06T14:58:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:36.997+0000 7ffb117b4700 1 -- 172.21.15.32:0/1184824066 wait complete. 2024-04-06T14:58:36.984 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-04-06T14:58:36.995 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:36.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-06T14:58:37.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:36 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2112271159' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-06T14:58:37.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:36 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1637762623' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.132+0000 7fd2222eb700 1 -- 172.21.15.32:0/448957881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 msgr2=0x7fd21c0fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.132+0000 7fd2222eb700 1 --2- 172.21.15.32:0/448957881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 0x7fd21c0fbbf0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fd210004940 tx=0x7fd210009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.132+0000 7fd2222eb700 1 -- 172.21.15.32:0/448957881 shutdown_connections 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.132+0000 7fd2222eb700 1 --2- 172.21.15.32:0/448957881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 0x7fd21c0fbbf0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.132+0000 7fd2222eb700 1 --2- 172.21.15.32:0/448957881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd21c101be0 0x7fd21c101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.132+0000 7fd2222eb700 1 -- 172.21.15.32:0/448957881 >> 172.21.15.32:0/448957881 conn(0x7fd21c072ad0 msgr2=0x7fd21c072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.132+0000 7fd2222eb700 1 -- 172.21.15.32:0/448957881 shutdown_connections 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.132+0000 7fd2222eb700 1 -- 172.21.15.32:0/448957881 wait complete. 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd2222eb700 1 Processor -- start 2024-04-06T14:58:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd2222eb700 1 -- start start 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd2222eb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd21c101be0 0x7fd21c102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd2222eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 0x7fd21c102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd2222eb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c104990 con 0x7fd21c102580 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd2222eb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c1034c0 con 0x7fd21c101be0 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd21b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 0x7fd21c102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd21b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 0x7fd21c102f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50860/0 (socket says 172.21.15.32:50860) 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd21b7fe700 1 -- 172.21.15.32:0/386302859 learned_addr learned my addr 172.21.15.32:0/386302859 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd21bfff700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd21c101be0 0x7fd21c102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd21b7fe700 1 -- 172.21.15.32:0/386302859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd21c101be0 msgr2=0x7fd21c102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd21b7fe700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd21c101be0 0x7fd21c102a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.133+0000 7fd21b7fe700 1 -- 172.21.15.32:0/386302859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd210005040 con 0x7fd21c102580 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.134+0000 7fd21b7fe700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 0x7fd21c102f80 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd210008d50 tx=0x7fd210009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.134+0000 7fd218ff9700 1 -- 172.21.15.32:0/386302859 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd21000e030 con 0x7fd21c102580 2024-04-06T14:58:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.134+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21c103740 con 0x7fd21c102580 2024-04-06T14:58:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.134+0000 7fd218ff9700 1 -- 172.21.15.32:0/386302859 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd210013de0 con 0x7fd21c102580 2024-04-06T14:58:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.134+0000 7fd218ff9700 1 -- 172.21.15.32:0/386302859 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd21001cd90 con 0x7fd21c102580 2024-04-06T14:58:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.134+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd21c119e30 con 0x7fd21c102580 2024-04-06T14:58:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.135+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21c0fd330 con 0x7fd21c102580 2024-04-06T14:58:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.135+0000 7fd218ff9700 1 -- 172.21.15.32:0/386302859 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd210013930 con 0x7fd21c102580 2024-04-06T14:58:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.135+0000 7fd218ff9700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd20406cc20 0x7fd20406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.136+0000 7fd218ff9700 1 -- 172.21.15.32:0/386302859 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd21008d8e0 con 0x7fd21c102580 2024-04-06T14:58:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.136+0000 7fd21bfff700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd20406cc20 0x7fd20406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.136+0000 7fd21bfff700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd20406cc20 0x7fd20406f0d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd20c004600 tx=0x7fd20c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.140+0000 7fd218ff9700 1 -- 172.21.15.32:0/386302859 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd210057920 con 0x7fd21c102580 2024-04-06T14:58:37.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.394+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7fd21c064970 con 0x7fd21c102580 2024-04-06T14:58:37.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.394+0000 7fd218ff9700 1 -- 172.21.15.32:0/386302859 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v39) v1 ==== 107+0+4816 (secure 0 0 0) 0x7fd21005af40 con 0x7fd21c102580 2024-04-06T14:58:37.380 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:37.380 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34378,"name":"cephfs.smithi032.mqoaof","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/629297924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":629297924},{"type":"v1","addr":"172.21.15.32:6829","nonce":629297924}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44283,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/4145187217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4145187217},{"type":"v1","addr":"172.21.15.86:6825","nonce":4145187217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd20406cc20 msgr2=0x7fd20406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd20406cc20 0x7fd20406f0d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd20c004600 tx=0x7fd20c00b040 comp rx=0 tx=0).stop 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 msgr2=0x7fd21c102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 0x7fd21c102f80 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd210008d50 tx=0x7fd210009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 shutdown_connections 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fd20406cc20 0x7fd20406f0d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd21c101be0 0x7fd21c102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 --2- 172.21.15.32:0/386302859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd21c102580 0x7fd21c102f80 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 >> 172.21.15.32:0/386302859 conn(0x7fd21c072ad0 msgr2=0x7fd21c064dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 shutdown_connections 2024-04-06T14:58:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.397+0000 7fd2222eb700 1 -- 172.21.15.32:0/386302859 wait complete. 2024-04-06T14:58:37.384 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-04-06T14:58:37.395 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:37.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-06T14:58:37.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.532+0000 7fed38526700 1 -- 172.21.15.32:0/901860477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 msgr2=0x7fed30101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.532+0000 7fed38526700 1 --2- 172.21.15.32:0/901860477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 0x7fed30101fb0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fed20004940 tx=0x7fed20009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:37.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.532+0000 7fed38526700 1 -- 172.21.15.32:0/901860477 shutdown_connections 2024-04-06T14:58:37.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.532+0000 7fed38526700 1 --2- 172.21.15.32:0/901860477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed30102580 0x7fed300fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.532+0000 7fed38526700 1 --2- 172.21.15.32:0/901860477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 0x7fed30101fb0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.532+0000 7fed38526700 1 -- 172.21.15.32:0/901860477 >> 172.21.15.32:0/901860477 conn(0x7fed300f7710 msgr2=0x7fed300f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:37.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.532+0000 7fed38526700 1 -- 172.21.15.32:0/901860477 shutdown_connections 2024-04-06T14:58:37.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.532+0000 7fed38526700 1 -- 172.21.15.32:0/901860477 wait complete. 2024-04-06T14:58:37.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed38526700 1 Processor -- start 2024-04-06T14:58:37.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed38526700 1 -- start start 2024-04-06T14:58:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed38526700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 0x7fed30112600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed38526700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed30102580 0x7fed30112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed38526700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed301131d0 con 0x7fed30101be0 2024-04-06T14:58:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed38526700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3010c790 con 0x7fed30102580 2024-04-06T14:58:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed362c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 0x7fed30112600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed362c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 0x7fed30112600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50882/0 (socket says 172.21.15.32:50882) 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed362c2700 1 -- 172.21.15.32:0/68151233 learned_addr learned my addr 172.21.15.32:0/68151233 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.533+0000 7fed35ac1700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed30102580 0x7fed30112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.534+0000 7fed362c2700 1 -- 172.21.15.32:0/68151233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed30102580 msgr2=0x7fed30112b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.534+0000 7fed362c2700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed30102580 0x7fed30112b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.534+0000 7fed362c2700 1 -- 172.21.15.32:0/68151233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20005040 con 0x7fed30101be0 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.534+0000 7fed362c2700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 0x7fed30112600 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fed20009380 tx=0x7fed20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.534+0000 7fed26ffd700 1 -- 172.21.15.32:0/68151233 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed2000e040 con 0x7fed30101be0 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.534+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3010c9c0 con 0x7fed30101be0 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.534+0000 7fed26ffd700 1 -- 172.21.15.32:0/68151233 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed20004290 con 0x7fed30101be0 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.534+0000 7fed26ffd700 1 -- 172.21.15.32:0/68151233 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed20013660 con 0x7fed30101be0 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.534+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3010cea0 con 0x7fed30101be0 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.535+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed300fd330 con 0x7fed30101be0 2024-04-06T14:58:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.535+0000 7fed26ffd700 1 -- 172.21.15.32:0/68151233 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fed20013920 con 0x7fed30101be0 2024-04-06T14:58:37.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.536+0000 7fed26ffd700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1c06cc20 0x7fed1c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:37.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.536+0000 7fed26ffd700 1 -- 172.21.15.32:0/68151233 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fed2008fe30 con 0x7fed30101be0 2024-04-06T14:58:37.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.536+0000 7fed35ac1700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1c06cc20 0x7fed1c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:37.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.536+0000 7fed35ac1700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1c06cc20 0x7fed1c06f0d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fed3010dea0 tx=0x7fed2c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.540+0000 7fed26ffd700 1 -- 172.21.15.32:0/68151233 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fed2005d880 con 0x7fed30101be0 2024-04-06T14:58:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.790+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7fed3010dcf0 con 0x7fed30101be0 2024-04-06T14:58:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.791+0000 7fed26ffd700 1 -- 172.21.15.32:0/68151233 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v39) v1 ==== 107+0+4040 (secure 0 0 0) 0x7fed2005d410 con 0x7fed30101be0 2024-04-06T14:58:37.777 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44283,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/4145187217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4145187217},{"type":"v1","addr":"172.21.15.86:6825","nonce":4145187217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1c06cc20 msgr2=0x7fed1c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1c06cc20 0x7fed1c06f0d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fed3010dea0 tx=0x7fed2c00b040 comp rx=0 tx=0).stop 2024-04-06T14:58:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 msgr2=0x7fed30112600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 0x7fed30112600 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fed20009380 tx=0x7fed20009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 shutdown_connections 2024-04-06T14:58:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fed1c06cc20 0x7fed1c06f0d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101be0 0x7fed30112600 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 --2- 172.21.15.32:0/68151233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed30102580 0x7fed30112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 >> 172.21.15.32:0/68151233 conn(0x7fed300f7710 msgr2=0x7fed300f8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 shutdown_connections 2024-04-06T14:58:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.793+0000 7fed38526700 1 -- 172.21.15.32:0/68151233 wait complete. 2024-04-06T14:58:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-06T14:58:37.791 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:37.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-06T14:58:37.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.928+0000 7f241e7a9700 1 -- 172.21.15.32:0/2354536456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418102580 msgr2=0x7f24180fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.928+0000 7f241e7a9700 1 --2- 172.21.15.32:0/2354536456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418102580 0x7f24180fbbf0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f240c004940 tx=0x7f240c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:37.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.928+0000 7f241e7a9700 1 -- 172.21.15.32:0/2354536456 shutdown_connections 2024-04-06T14:58:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.928+0000 7f241e7a9700 1 --2- 172.21.15.32:0/2354536456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418102580 0x7f24180fbbf0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.928+0000 7f241e7a9700 1 --2- 172.21.15.32:0/2354536456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2418101be0 0x7f2418101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.928+0000 7f241e7a9700 1 -- 172.21.15.32:0/2354536456 >> 172.21.15.32:0/2354536456 conn(0x7f2418072ad0 msgr2=0x7f2418072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.928+0000 7f241e7a9700 1 -- 172.21.15.32:0/2354536456 shutdown_connections 2024-04-06T14:58:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.928+0000 7f241e7a9700 1 -- 172.21.15.32:0/2354536456 wait complete. 2024-04-06T14:58:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.928+0000 7f241e7a9700 1 Processor -- start 2024-04-06T14:58:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f241e7a9700 1 -- start start 2024-04-06T14:58:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f241e7a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418101be0 0x7f24181126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f241e7a9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2418102580 0x7f2418112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f241e7a9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2418113230 con 0x7f2418101be0 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f241e7a9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241810c7f0 con 0x7f2418102580 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f2417fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418101be0 0x7f24181126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f2417fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418101be0 0x7f24181126a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50904/0 (socket says 172.21.15.32:50904) 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f2417fff700 1 -- 172.21.15.32:0/834443025 learned_addr learned my addr 172.21.15.32:0/834443025 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f24177fe700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2418102580 0x7f2418112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f2417fff700 1 -- 172.21.15.32:0/834443025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2418102580 msgr2=0x7f2418112be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f2417fff700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2418102580 0x7f2418112be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f2417fff700 1 -- 172.21.15.32:0/834443025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c005040 con 0x7f2418101be0 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f24177fe700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2418102580 0x7f2418112be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:58:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.929+0000 7f2417fff700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418101be0 0x7f24181126a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f2408002a50 tx=0x7f24080046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:37.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.930+0000 7f2414ff9700 1 -- 172.21.15.32:0/834443025 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2408007cc0 con 0x7f2418101be0 2024-04-06T14:58:37.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.930+0000 7f2414ff9700 1 -- 172.21.15.32:0/834443025 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f240800f040 con 0x7f2418101be0 2024-04-06T14:58:37.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.930+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241810cad0 con 0x7f2418101be0 2024-04-06T14:58:37.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.930+0000 7f2414ff9700 1 -- 172.21.15.32:0/834443025 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2408009540 con 0x7f2418101be0 2024-04-06T14:58:37.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.930+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241810d010 con 0x7f2418101be0 2024-04-06T14:58:37.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.930+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2418112350 con 0x7f2418101be0 2024-04-06T14:58:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.931+0000 7f2414ff9700 1 -- 172.21.15.32:0/834443025 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f24080096a0 con 0x7f2418101be0 2024-04-06T14:58:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.932+0000 7f2414ff9700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f240006cc20 0x7f240006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.932+0000 7f24177fe700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f240006cc20 0x7f240006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.932+0000 7f2414ff9700 1 -- 172.21.15.32:0/834443025 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f240808be70 con 0x7f2418101be0 2024-04-06T14:58:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.932+0000 7f24177fe700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f240006cc20 0x7f240006f0d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f241810df00 tx=0x7f240c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:37.936+0000 7f2414ff9700 1 -- 172.21.15.32:0/834443025 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f24080598c0 con 0x7f2418101be0 2024-04-06T14:58:38.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:37 smithi086 ceph-mon[110975]: pgmap v1158: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:38.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:37 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1184824066' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-06T14:58:38.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:37 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/386302859' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-06T14:58:38.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:58:38.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:37 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/68151233' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-06T14:58:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.189+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f241810dcb0 con 0x7f2418101be0 2024-04-06T14:58:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.190+0000 7f2414ff9700 1 -- 172.21.15.32:0/834443025 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v39) v1 ==== 107+0+4819 (secure 0 0 0) 0x7f2408059450 con 0x7f2418101be0 2024-04-06T14:58:38.176 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:38.176 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44283,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/4145187217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4145187217},{"type":"v1","addr":"172.21.15.86:6825","nonce":4145187217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.192+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f240006cc20 msgr2=0x7f240006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.192+0000 7f241e7a9700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f240006cc20 0x7f240006f0d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f241810df00 tx=0x7f240c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.192+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418101be0 msgr2=0x7f24181126a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.192+0000 7f241e7a9700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418101be0 0x7f24181126a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f2408002a50 tx=0x7f24080046b0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.192+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 shutdown_connections 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.192+0000 7f241e7a9700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f240006cc20 0x7f240006f0d0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.192+0000 7f241e7a9700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2418101be0 0x7f24181126a0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.192+0000 7f241e7a9700 1 --2- 172.21.15.32:0/834443025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2418102580 0x7f2418112be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.192+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 >> 172.21.15.32:0/834443025 conn(0x7f2418072ad0 msgr2=0x7f2418104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.193+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 shutdown_connections 2024-04-06T14:58:38.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.193+0000 7f241e7a9700 1 -- 172.21.15.32:0/834443025 wait complete. 2024-04-06T14:58:38.180 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-04-06T14:58:38.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:37 smithi032 ceph-mon[147414]: pgmap v1158: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:38.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:37 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1184824066' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-06T14:58:38.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:37 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/386302859' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-06T14:58:38.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:58:38.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:37 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/68151233' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-06T14:58:38.192 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:38.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-06T14:58:38.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.328+0000 7f2d5a787700 1 -- 172.21.15.32:0/3995719889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54103660 msgr2=0x7f2d540fccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.328+0000 7f2d5a787700 1 --2- 172.21.15.32:0/3995719889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54103660 0x7f2d540fccd0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f2d48004940 tx=0x7f2d48009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:38.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.328+0000 7f2d5a787700 1 -- 172.21.15.32:0/3995719889 shutdown_connections 2024-04-06T14:58:38.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.328+0000 7f2d5a787700 1 --2- 172.21.15.32:0/3995719889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54103660 0x7f2d540fccd0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.328+0000 7f2d5a787700 1 --2- 172.21.15.32:0/3995719889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d54102cc0 0x7f2d54103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.328+0000 7f2d5a787700 1 -- 172.21.15.32:0/3995719889 >> 172.21.15.32:0/3995719889 conn(0x7f2d540f8830 msgr2=0x7f2d540fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.328+0000 7f2d5a787700 1 -- 172.21.15.32:0/3995719889 shutdown_connections 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.328+0000 7f2d5a787700 1 -- 172.21.15.32:0/3995719889 wait complete. 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.329+0000 7f2d5a787700 1 Processor -- start 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.329+0000 7f2d5a787700 1 -- start start 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.329+0000 7f2d5a787700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54102cc0 0x7f2d54113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.329+0000 7f2d5a787700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d54103660 0x7f2d54113ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.329+0000 7f2d5a787700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d541142f0 con 0x7f2d54102cc0 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.329+0000 7f2d5a787700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5410d8f0 con 0x7f2d54103660 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.329+0000 7f2d53fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54102cc0 0x7f2d54113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.329+0000 7f2d53fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54102cc0 0x7f2d54113760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50930/0 (socket says 172.21.15.32:50930) 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.329+0000 7f2d53fff700 1 -- 172.21.15.32:0/1851413163 learned_addr learned my addr 172.21.15.32:0/1851413163 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d537fe700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d54103660 0x7f2d54113ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:38.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d53fff700 1 -- 172.21.15.32:0/1851413163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d54103660 msgr2=0x7f2d54113ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d53fff700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d54103660 0x7f2d54113ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d53fff700 1 -- 172.21.15.32:0/1851413163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d48005040 con 0x7f2d54102cc0 2024-04-06T14:58:38.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d537fe700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d54103660 0x7f2d54113ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:58:38.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d53fff700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54102cc0 0x7f2d54113760 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400c8c0 tx=0x7f2d440044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:38.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d50ff9700 1 -- 172.21.15.32:0/1851413163 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4400a980 con 0x7f2d54102cc0 2024-04-06T14:58:38.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d50ff9700 1 -- 172.21.15.32:0/1851413163 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d4400ce60 con 0x7f2d54102cc0 2024-04-06T14:58:38.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5410dbd0 con 0x7f2d54102cc0 2024-04-06T14:58:38.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d50ff9700 1 -- 172.21.15.32:0/1851413163 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d440113a0 con 0x7f2d54102cc0 2024-04-06T14:58:38.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.330+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5410e190 con 0x7f2d54102cc0 2024-04-06T14:58:38.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.331+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d540fe410 con 0x7f2d54102cc0 2024-04-06T14:58:38.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.331+0000 7f2d50ff9700 1 -- 172.21.15.32:0/1851413163 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2d44012400 con 0x7f2d54102cc0 2024-04-06T14:58:38.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.332+0000 7f2d50ff9700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d3c06cc20 0x7f2d3c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:38.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.332+0000 7f2d50ff9700 1 -- 172.21.15.32:0/1851413163 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2d4408c040 con 0x7f2d54102cc0 2024-04-06T14:58:38.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.332+0000 7f2d537fe700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d3c06cc20 0x7f2d3c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:38.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.332+0000 7f2d537fe700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d3c06cc20 0x7f2d3c06f0d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009fd0 tx=0x7f2d48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:38.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.336+0000 7f2d50ff9700 1 -- 172.21.15.32:0/1851413163 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2d44059bd0 con 0x7f2d54102cc0 2024-04-06T14:58:38.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.590+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f2d540640e0 con 0x7f2d54102cc0 2024-04-06T14:58:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.590+0000 7f2d50ff9700 1 -- 172.21.15.32:0/1851413163 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v39) v1 ==== 107+0+4040 (secure 0 0 0) 0x7f2d44059760 con 0x7f2d54102cc0 2024-04-06T14:58:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:38.576 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.592+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d3c06cc20 msgr2=0x7f2d3c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.592+0000 7f2d5a787700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d3c06cc20 0x7f2d3c06f0d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009fd0 tx=0x7f2d48009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.592+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54102cc0 msgr2=0x7f2d54113760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.592+0000 7f2d5a787700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54102cc0 0x7f2d54113760 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400c8c0 tx=0x7f2d440044c0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.593+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 shutdown_connections 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.593+0000 7f2d5a787700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2d3c06cc20 0x7f2d3c06f0d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.593+0000 7f2d5a787700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d54102cc0 0x7f2d54113760 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.593+0000 7f2d5a787700 1 --2- 172.21.15.32:0/1851413163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d54103660 0x7f2d54113ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.593+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 >> 172.21.15.32:0/1851413163 conn(0x7f2d540f8830 msgr2=0x7f2d540f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.593+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 shutdown_connections 2024-04-06T14:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.593+0000 7f2d5a787700 1 -- 172.21.15.32:0/1851413163 wait complete. 2024-04-06T14:58:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-04-06T14:58:38.591 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:38.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-06T14:58:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.728+0000 7fceb7b91700 1 -- 172.21.15.32:0/728159249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0101be0 msgr2=0x7fceb0101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.728+0000 7fceb7b91700 1 --2- 172.21.15.32:0/728159249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0101be0 0x7fceb0101fb0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fceac004940 tx=0x7fceac009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.728+0000 7fceb7b91700 1 -- 172.21.15.32:0/728159249 shutdown_connections 2024-04-06T14:58:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.728+0000 7fceb7b91700 1 --2- 172.21.15.32:0/728159249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceb0102580 0x7fceb00fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.728+0000 7fceb7b91700 1 --2- 172.21.15.32:0/728159249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0101be0 0x7fceb0101fb0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.728+0000 7fceb7b91700 1 -- 172.21.15.32:0/728159249 >> 172.21.15.32:0/728159249 conn(0x7fceb0072ad0 msgr2=0x7fceb0072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.728+0000 7fceb7b91700 1 -- 172.21.15.32:0/728159249 shutdown_connections 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.728+0000 7fceb7b91700 1 -- 172.21.15.32:0/728159249 wait complete. 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb7b91700 1 Processor -- start 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb7b91700 1 -- start start 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb7b91700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceb0101be0 0x7fceb0102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb7b91700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0102580 0x7fceb0102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb7b91700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01049e0 con 0x7fceb0102580 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb7b91700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01034c0 con 0x7fceb0101be0 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb512c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0102580 0x7fceb0102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb512c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0102580 0x7fceb0102f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50958/0 (socket says 172.21.15.32:50958) 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb512c700 1 -- 172.21.15.32:0/2993345693 learned_addr learned my addr 172.21.15.32:0/2993345693 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb592d700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceb0101be0 0x7fceb0102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb512c700 1 -- 172.21.15.32:0/2993345693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceb0101be0 msgr2=0x7fceb0102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb512c700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceb0101be0 0x7fceb0102a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.729+0000 7fceb512c700 1 -- 172.21.15.32:0/2993345693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac005040 con 0x7fceb0102580 2024-04-06T14:58:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.730+0000 7fceb512c700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0102580 0x7fceb0102f80 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fcea4002a50 tx=0x7fcea4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.730+0000 7fcea27fc700 1 -- 172.21.15.32:0/2993345693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcea4007cc0 con 0x7fceb0102580 2024-04-06T14:58:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.730+0000 7fcea27fc700 1 -- 172.21.15.32:0/2993345693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcea400f040 con 0x7fceb0102580 2024-04-06T14:58:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.730+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb0103770 con 0x7fceb0102580 2024-04-06T14:58:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.730+0000 7fcea27fc700 1 -- 172.21.15.32:0/2993345693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcea400d990 con 0x7fceb0102580 2024-04-06T14:58:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.730+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb0119ee0 con 0x7fceb0102580 2024-04-06T14:58:38.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.731+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb00fd330 con 0x7fceb0102580 2024-04-06T14:58:38.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.731+0000 7fcea27fc700 1 -- 172.21.15.32:0/2993345693 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcea4009710 con 0x7fceb0102580 2024-04-06T14:58:38.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.732+0000 7fcea27fc700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9c06cc20 0x7fce9c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:38.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.732+0000 7fcea27fc700 1 -- 172.21.15.32:0/2993345693 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcea408be60 con 0x7fceb0102580 2024-04-06T14:58:38.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.732+0000 7fceb592d700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9c06cc20 0x7fce9c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:38.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.732+0000 7fceb592d700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9c06cc20 0x7fce9c06f0d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fceac00b3b0 tx=0x7fceac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:38.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.736+0000 7fcea27fc700 1 -- 172.21.15.32:0/2993345693 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcea40598b0 con 0x7fceb0102580 2024-04-06T14:58:38.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:38 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/834443025' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-06T14:58:38.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:38 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1851413163' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-06T14:58:38.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.991+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7fceb0064970 con 0x7fceb0102580 2024-04-06T14:58:38.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.992+0000 7fcea27fc700 1 -- 172.21.15.32:0/2993345693 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v39) v1 ==== 107+0+4819 (secure 0 0 0) 0x7fcea4059440 con 0x7fceb0102580 2024-04-06T14:58:38.978 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:38.978 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24313,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/2372611428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2372611428},{"type":"v1","addr":"172.21.15.86:6827","nonce":2372611428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9c06cc20 msgr2=0x7fce9c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9c06cc20 0x7fce9c06f0d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fceac00b3b0 tx=0x7fceac009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0102580 msgr2=0x7fceb0102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0102580 0x7fceb0102f80 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fcea4002a50 tx=0x7fcea4004600 comp rx=0 tx=0).stop 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 shutdown_connections 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fce9c06cc20 0x7fce9c06f0d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceb0101be0 0x7fceb0102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 --2- 172.21.15.32:0/2993345693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fceb0102580 0x7fceb0102f80 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 >> 172.21.15.32:0/2993345693 conn(0x7fceb0072ad0 msgr2=0x7fceb00fa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 shutdown_connections 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:38.994+0000 7fceb7b91700 1 -- 172.21.15.32:0/2993345693 wait complete. 2024-04-06T14:58:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-06T14:58:38.993 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:38.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-06T14:58:39.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:38 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/834443025' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-06T14:58:39.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:38 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1851413163' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.130+0000 7f154bc45700 1 -- 172.21.15.32:0/3856511794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544101be0 msgr2=0x7f1544101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.130+0000 7f154bc45700 1 --2- 172.21.15.32:0/3856511794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544101be0 0x7f1544101fb0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1540004940 tx=0x7f1540009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.130+0000 7f154bc45700 1 -- 172.21.15.32:0/3856511794 shutdown_connections 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.130+0000 7f154bc45700 1 --2- 172.21.15.32:0/3856511794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1544102580 0x7f15440fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.130+0000 7f154bc45700 1 --2- 172.21.15.32:0/3856511794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544101be0 0x7f1544101fb0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.130+0000 7f154bc45700 1 -- 172.21.15.32:0/3856511794 >> 172.21.15.32:0/3856511794 conn(0x7f1544072ad0 msgr2=0x7f1544072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.130+0000 7f154bc45700 1 -- 172.21.15.32:0/3856511794 shutdown_connections 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.130+0000 7f154bc45700 1 -- 172.21.15.32:0/3856511794 wait complete. 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f154bc45700 1 Processor -- start 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f154bc45700 1 -- start start 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f154bc45700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1544101be0 0x7f15441126c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f154bc45700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544102580 0x7f1544112c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f154bc45700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1544113290 con 0x7f1544102580 2024-04-06T14:58:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f154bc45700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154410c850 con 0x7f1544101be0 2024-04-06T14:58:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f15491e0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544102580 0x7f1544112c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f15491e0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544102580 0x7f1544112c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50962/0 (socket says 172.21.15.32:50962) 2024-04-06T14:58:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f15491e0700 1 -- 172.21.15.32:0/2737377523 learned_addr learned my addr 172.21.15.32:0/2737377523 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f15499e1700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1544101be0 0x7f15441126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f15491e0700 1 -- 172.21.15.32:0/2737377523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1544101be0 msgr2=0x7f15441126c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f15491e0700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1544101be0 0x7f15441126c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.131+0000 7f15491e0700 1 -- 172.21.15.32:0/2737377523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1540005040 con 0x7f1544102580 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.132+0000 7f15491e0700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544102580 0x7f1544112c40 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f153800c800 tx=0x7f1538004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.132+0000 7f15367fc700 1 -- 172.21.15.32:0/2737377523 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f153800a760 con 0x7f1544102580 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.132+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154410cae0 con 0x7f1544102580 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.132+0000 7f15367fc700 1 -- 172.21.15.32:0/2737377523 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f153800ad70 con 0x7f1544102580 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.132+0000 7f15367fc700 1 -- 172.21.15.32:0/2737377523 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15380113a0 con 0x7f1544102580 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.132+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154410d020 con 0x7f1544102580 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.133+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15440fd330 con 0x7f1544102580 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.133+0000 7f15367fc700 1 -- 172.21.15.32:0/2737377523 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1538012400 con 0x7f1544102580 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.134+0000 7f15367fc700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f153006cc20 0x7f153006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.134+0000 7f15499e1700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f153006cc20 0x7f153006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.134+0000 7f15367fc700 1 -- 172.21.15.32:0/2737377523 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f153808c0e0 con 0x7f1544102580 2024-04-06T14:58:39.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.134+0000 7f15499e1700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f153006cc20 0x7f153006f0d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f15440730c0 tx=0x7f1540009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:39.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.139+0000 7f15367fc700 1 -- 172.21.15.32:0/2737377523 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1538059bb0 con 0x7f1544102580 2024-04-06T14:58:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.389+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f1544064970 con 0x7f1544102580 2024-04-06T14:58:39.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.390+0000 7f15367fc700 1 -- 172.21.15.32:0/2737377523 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v39) v1 ==== 107+0+4040 (secure 0 0 0) 0x7f1538059740 con 0x7f1544102580 2024-04-06T14:58:39.376 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:39.377 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:39.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f153006cc20 msgr2=0x7f153006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f153006cc20 0x7f153006f0d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f15440730c0 tx=0x7f1540009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:39.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544102580 msgr2=0x7f1544112c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544102580 0x7f1544112c40 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f153800c800 tx=0x7f1538004470 comp rx=0 tx=0).stop 2024-04-06T14:58:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 shutdown_connections 2024-04-06T14:58:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f153006cc20 0x7f153006f0d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1544101be0 0x7f15441126c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 --2- 172.21.15.32:0/2737377523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1544102580 0x7f1544112c40 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 >> 172.21.15.32:0/2737377523 conn(0x7f1544072ad0 msgr2=0x7f1544104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 shutdown_connections 2024-04-06T14:58:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.392+0000 7f154bc45700 1 -- 172.21.15.32:0/2737377523 wait complete. 2024-04-06T14:58:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 31 2024-04-06T14:58:39.390 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:39.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 -- 172.21.15.32:0/1393804632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4103670 msgr2=0x7f7ed40fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 --2- 172.21.15.32:0/1393804632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4103670 0x7f7ed40fcce0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0004940 tx=0x7f7ed0009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 -- 172.21.15.32:0/1393804632 shutdown_connections 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 --2- 172.21.15.32:0/1393804632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4103670 0x7f7ed40fcce0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 --2- 172.21.15.32:0/1393804632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed4102cd0 0x7f7ed41030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 -- 172.21.15.32:0/1393804632 >> 172.21.15.32:0/1393804632 conn(0x7f7ed40f8820 msgr2=0x7f7ed40fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 -- 172.21.15.32:0/1393804632 shutdown_connections 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 -- 172.21.15.32:0/1393804632 wait complete. 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 Processor -- start 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.530+0000 7f7edc7bd700 1 -- start start 2024-04-06T14:58:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7edc7bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4102cd0 0x7f7ed4115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7edc7bd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed4103670 0x7f7ed4115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7edc7bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed41165a0 con 0x7f7ed4102cd0 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7edc7bd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed410fba0 con 0x7f7ed4103670 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7eda559700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4102cd0 0x7f7ed4115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7eda559700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4102cd0 0x7f7ed4115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50970/0 (socket says 172.21.15.32:50970) 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7eda559700 1 -- 172.21.15.32:0/185374514 learned_addr learned my addr 172.21.15.32:0/185374514 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7eda559700 1 -- 172.21.15.32:0/185374514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed4103670 msgr2=0x7f7ed4115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7ed9d58700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed4103670 0x7f7ed4115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7eda559700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed4103670 0x7f7ed4115f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7eda559700 1 -- 172.21.15.32:0/185374514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed0005040 con 0x7f7ed4102cd0 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7eda559700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4102cd0 0x7f7ed4115a10 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c830 tx=0x7f7ec4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7ecaffd700 1 -- 172.21.15.32:0/185374514 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ec400a760 con 0x7f7ed4102cd0 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7ecaffd700 1 -- 172.21.15.32:0/185374514 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ec400ad70 con 0x7f7ed4102cd0 2024-04-06T14:58:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed410fe80 con 0x7f7ed4102cd0 2024-04-06T14:58:39.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7ecaffd700 1 -- 172.21.15.32:0/185374514 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ec40113a0 con 0x7f7ed4102cd0 2024-04-06T14:58:39.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.531+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed41103c0 con 0x7f7ed4102cd0 2024-04-06T14:58:39.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.532+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed40fe420 con 0x7f7ed4102cd0 2024-04-06T14:58:39.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.533+0000 7f7ecaffd700 1 -- 172.21.15.32:0/185374514 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7ec4012400 con 0x7f7ed4102cd0 2024-04-06T14:58:39.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.533+0000 7f7ecaffd700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7ec006cc20 0x7f7ec006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:39.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.533+0000 7f7ed9d58700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7ec006cc20 0x7f7ec006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:39.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.533+0000 7f7ecaffd700 1 -- 172.21.15.32:0/185374514 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7ec408c0b0 con 0x7f7ed4102cd0 2024-04-06T14:58:39.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.534+0000 7f7ed9d58700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7ec006cc20 0x7f7ec006f0d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009fd0 tx=0x7f7ed0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:39.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.538+0000 7f7ecaffd700 1 -- 172.21.15.32:0/185374514 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7ec4059b80 con 0x7f7ed4102cd0 2024-04-06T14:58:39.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.792+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f7ed4110ff0 con 0x7f7ed4102cd0 2024-04-06T14:58:39.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.792+0000 7f7ecaffd700 1 -- 172.21.15.32:0/185374514 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v39) v1 ==== 107+0+4819 (secure 0 0 0) 0x7f7ec4059710 con 0x7f7ed4102cd0 2024-04-06T14:58:39.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:39.779 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44803,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6827/2158771540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2158771540},{"type":"v1","addr":"172.21.15.86:6827","nonce":2158771540}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:25:48.052931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7ec006cc20 msgr2=0x7f7ec006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7ec006cc20 0x7f7ec006f0d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009fd0 tx=0x7f7ed0009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4102cd0 msgr2=0x7f7ed4115a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4102cd0 0x7f7ed4115a10 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c830 tx=0x7f7ec4004470 comp rx=0 tx=0).stop 2024-04-06T14:58:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 shutdown_connections 2024-04-06T14:58:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f7ec006cc20 0x7f7ec006f0d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ed4102cd0 0x7f7ed4115a10 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 --2- 172.21.15.32:0/185374514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed4103670 0x7f7ed4115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 >> 172.21.15.32:0/185374514 conn(0x7f7ed40f8820 msgr2=0x7f7ed40f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:39.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 shutdown_connections 2024-04-06T14:58:39.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.795+0000 7f7edc7bd700 1 -- 172.21.15.32:0/185374514 wait complete. 2024-04-06T14:58:39.782 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-06T14:58:39.793 DEBUG:tasks.fs:max_mds not reduced in epoch 32 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-06T14:58:39.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-06T14:58:39.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.930+0000 7f281ebb7700 1 -- 172.21.15.32:0/3660357824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 msgr2=0x7f2818103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.930+0000 7f281ebb7700 1 --2- 172.21.15.32:0/3660357824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 0x7f2818103090 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f280c004940 tx=0x7f280c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.930+0000 7f281ebb7700 1 -- 172.21.15.32:0/3660357824 shutdown_connections 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.930+0000 7f281ebb7700 1 --2- 172.21.15.32:0/3660357824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2818103660 0x7f28180fccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.930+0000 7f281ebb7700 1 --2- 172.21.15.32:0/3660357824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 0x7f2818103090 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.930+0000 7f281ebb7700 1 -- 172.21.15.32:0/3660357824 >> 172.21.15.32:0/3660357824 conn(0x7f28180f8830 msgr2=0x7f28180fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.930+0000 7f281ebb7700 1 -- 172.21.15.32:0/3660357824 shutdown_connections 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.930+0000 7f281ebb7700 1 -- 172.21.15.32:0/3660357824 wait complete. 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.931+0000 7f281ebb7700 1 Processor -- start 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.931+0000 7f281ebb7700 1 -- start start 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.931+0000 7f281ebb7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 0x7f28181137a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.931+0000 7f281ebb7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2818103660 0x7f2818113ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.931+0000 7f281ebb7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2818114330 con 0x7f2818102cc0 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.931+0000 7f281ebb7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281810d930 con 0x7f2818103660 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.931+0000 7f281c953700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 0x7f28181137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:39.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.931+0000 7f281c953700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 0x7f28181137a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51002/0 (socket says 172.21.15.32:51002) 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.931+0000 7f281c953700 1 -- 172.21.15.32:0/3475811580 learned_addr learned my addr 172.21.15.32:0/3475811580 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f2817fff700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2818103660 0x7f2818113ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f281c953700 1 -- 172.21.15.32:0/3475811580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2818103660 msgr2=0x7f2818113ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f281c953700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2818103660 0x7f2818113ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f281c953700 1 -- 172.21.15.32:0/3475811580 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280c005040 con 0x7f2818102cc0 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f281c953700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 0x7f28181137a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f280c00b3b0 tx=0x7f280c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f28157fa700 1 -- 172.21.15.32:0/3475811580 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f280c00e040 con 0x7f2818102cc0 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f28157fa700 1 -- 172.21.15.32:0/3475811580 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f280c013420 con 0x7f2818102cc0 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f28157fa700 1 -- 172.21.15.32:0/3475811580 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f280c01c630 con 0x7f2818102cc0 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281810dbb0 con 0x7f2818102cc0 2024-04-06T14:58:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.932+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281810e170 con 0x7f2818102cc0 2024-04-06T14:58:39.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.933+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28180fe410 con 0x7f2818102cc0 2024-04-06T14:58:39.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.934+0000 7f28157fa700 1 -- 172.21.15.32:0/3475811580 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f280c013970 con 0x7f2818102cc0 2024-04-06T14:58:39.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.934+0000 7f28157fa700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f280806cc20 0x7f280806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:39.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.934+0000 7f28157fa700 1 -- 172.21.15.32:0/3475811580 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f280c08d2b0 con 0x7f2818102cc0 2024-04-06T14:58:39.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.934+0000 7f2817fff700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f280806cc20 0x7f280806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:39.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.935+0000 7f2817fff700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f280806cc20 0x7f280806f0d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2804007fd0 tx=0x7f2804009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:39.939+0000 7f28157fa700 1 -- 172.21.15.32:0/3475811580 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f280c05ad80 con 0x7f2818102cc0 2024-04-06T14:58:40.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:39 smithi086 ceph-mon[110975]: pgmap v1159: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:40.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:39 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2993345693' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-06T14:58:40.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:39 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2737377523' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-06T14:58:40.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:39 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/185374514' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-06T14:58:40.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.189+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f2818109c70 con 0x7f2818102cc0 2024-04-06T14:58:40.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.190+0000 7f28157fa700 1 -- 172.21.15.32:0/3475811580 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v39) v1 ==== 107+0+4818 (secure 0 0 0) 0x7f280c05a910 con 0x7f2818102cc0 2024-04-06T14:58:40.176 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:40.176 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44803,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6827/2158771540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2158771540},{"type":"v1","addr":"172.21.15.86:6827","nonce":2158771540}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:56:37.574572+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:40.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f280806cc20 msgr2=0x7f280806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f280806cc20 0x7f280806f0d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2804007fd0 tx=0x7f2804009040 comp rx=0 tx=0).stop 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 msgr2=0x7f28181137a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 0x7f28181137a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f280c00b3b0 tx=0x7f280c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 shutdown_connections 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f280806cc20 0x7f280806f0d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2818102cc0 0x7f28181137a0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 --2- 172.21.15.32:0/3475811580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2818103660 0x7f2818113ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 >> 172.21.15.32:0/3475811580 conn(0x7f28180f8830 msgr2=0x7f28180f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 shutdown_connections 2024-04-06T14:58:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.192+0000 7f281ebb7700 1 -- 172.21.15.32:0/3475811580 wait complete. 2024-04-06T14:58:40.180 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-06T14:58:40.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:39 smithi032 ceph-mon[147414]: pgmap v1159: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:40.189 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:39 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2993345693' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-06T14:58:40.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:39 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2737377523' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-06T14:58:40.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:39 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/185374514' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-06T14:58:40.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-06T14:58:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.327+0000 7fbd21538700 1 -- 172.21.15.32:0/54397274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c102ce0 msgr2=0x7fbd1c1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.327+0000 7fbd21538700 1 --2- 172.21.15.32:0/54397274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c102ce0 0x7fbd1c1030b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c004940 tx=0x7fbd0c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.327+0000 7fbd21538700 1 -- 172.21.15.32:0/54397274 shutdown_connections 2024-04-06T14:58:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.327+0000 7fbd21538700 1 --2- 172.21.15.32:0/54397274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd1c103680 0x7fbd1c064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.327+0000 7fbd21538700 1 --2- 172.21.15.32:0/54397274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c102ce0 0x7fbd1c1030b0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.327+0000 7fbd21538700 1 -- 172.21.15.32:0/54397274 >> 172.21.15.32:0/54397274 conn(0x7fbd1c072ad0 msgr2=0x7fbd1c072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.327+0000 7fbd21538700 1 -- 172.21.15.32:0/54397274 shutdown_connections 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.327+0000 7fbd21538700 1 -- 172.21.15.32:0/54397274 wait complete. 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd21538700 1 Processor -- start 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd21538700 1 -- start start 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd21538700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd1c102ce0 0x7fbd1c113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd21538700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c103680 0x7fbd1c113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd21538700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c1143a0 con 0x7fbd1c103680 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd21538700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c10d910 con 0x7fbd1c102ce0 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd1a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c103680 0x7fbd1c113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd1a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c103680 0x7fbd1c113cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51028/0 (socket says 172.21.15.32:51028) 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd1a7fc700 1 -- 172.21.15.32:0/410702571 learned_addr learned my addr 172.21.15.32:0/410702571 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd1affd700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd1c102ce0 0x7fbd1c113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd1a7fc700 1 -- 172.21.15.32:0/410702571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd1c102ce0 msgr2=0x7fbd1c113780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.328+0000 7fbd1a7fc700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd1c102ce0 0x7fbd1c113780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.329+0000 7fbd1a7fc700 1 -- 172.21.15.32:0/410702571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c005040 con 0x7fbd1c103680 2024-04-06T14:58:40.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.329+0000 7fbd1a7fc700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c103680 0x7fbd1c113cc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000b790 tx=0x7fbd10009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:40.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.329+0000 7fbd037fe700 1 -- 172.21.15.32:0/410702571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1000f6f0 con 0x7fbd1c103680 2024-04-06T14:58:40.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.329+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1c10dbf0 con 0x7fbd1c103680 2024-04-06T14:58:40.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.329+0000 7fbd037fe700 1 -- 172.21.15.32:0/410702571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd1000fd00 con 0x7fbd1c103680 2024-04-06T14:58:40.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.329+0000 7fbd037fe700 1 -- 172.21.15.32:0/410702571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd100103a0 con 0x7fbd1c103680 2024-04-06T14:58:40.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.329+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1c10e1b0 con 0x7fbd1c103680 2024-04-06T14:58:40.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.330+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd1c0fe430 con 0x7fbd1c103680 2024-04-06T14:58:40.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.330+0000 7fbd037fe700 1 -- 172.21.15.32:0/410702571 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbd1000f850 con 0x7fbd1c103680 2024-04-06T14:58:40.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.331+0000 7fbd037fe700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc20 0x7fbd0406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:40.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.331+0000 7fbd037fe700 1 -- 172.21.15.32:0/410702571 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbd1008bef0 con 0x7fbd1c103680 2024-04-06T14:58:40.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.331+0000 7fbd1affd700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc20 0x7fbd0406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:40.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.331+0000 7fbd1affd700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc20 0x7fbd0406f0d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c004f90 tx=0x7fbd0c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:40.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.335+0000 7fbd037fe700 1 -- 172.21.15.32:0/410702571 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbd10059940 con 0x7fbd1c103680 2024-04-06T14:58:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.590+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7fbd1c064970 con 0x7fbd1c103680 2024-04-06T14:58:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.590+0000 7fbd037fe700 1 -- 172.21.15.32:0/410702571 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v39) v1 ==== 107+0+4841 (secure 0 0 0) 0x7fbd100594d0 con 0x7fbd1c103680 2024-04-06T14:58:40.576 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:40.576 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44803,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6827/2158771540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2158771540},{"type":"v1","addr":"172.21.15.86:6827","nonce":2158771540}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:56:37.578365+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34382,"mds_1":34414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34382":{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":0,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34414":{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":1,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.592+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc20 msgr2=0x7fbd0406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.592+0000 7fbd21538700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc20 0x7fbd0406f0d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c004f90 tx=0x7fbd0c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.592+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c103680 msgr2=0x7fbd1c113cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.592+0000 7fbd21538700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c103680 0x7fbd1c113cc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000b790 tx=0x7fbd10009220 comp rx=0 tx=0).stop 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.593+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 shutdown_connections 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.593+0000 7fbd21538700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fbd0406cc20 0x7fbd0406f0d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.593+0000 7fbd21538700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd1c102ce0 0x7fbd1c113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.593+0000 7fbd21538700 1 --2- 172.21.15.32:0/410702571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd1c103680 0x7fbd1c113cc0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.593+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 >> 172.21.15.32:0/410702571 conn(0x7fbd1c072ad0 msgr2=0x7fbd1c105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.593+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 shutdown_connections 2024-04-06T14:58:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.593+0000 7fbd21538700 1 -- 172.21.15.32:0/410702571 wait complete. 2024-04-06T14:58:40.580 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-04-06T14:58:40.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-06T14:58:40.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.728+0000 7f49301ee700 1 -- 172.21.15.32:0/1654909680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928102580 msgr2=0x7f49280fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.728+0000 7f49301ee700 1 --2- 172.21.15.32:0/1654909680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928102580 0x7f49280fbbf0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4924004940 tx=0x7f4924009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:40.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.728+0000 7f49301ee700 1 -- 172.21.15.32:0/1654909680 shutdown_connections 2024-04-06T14:58:40.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.728+0000 7f49301ee700 1 --2- 172.21.15.32:0/1654909680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928102580 0x7f49280fbbf0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.728+0000 7f49301ee700 1 --2- 172.21.15.32:0/1654909680 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4928101be0 0x7f4928101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.728+0000 7f49301ee700 1 -- 172.21.15.32:0/1654909680 >> 172.21.15.32:0/1654909680 conn(0x7f4928072ad0 msgr2=0x7f4928072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:40.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.728+0000 7f49301ee700 1 -- 172.21.15.32:0/1654909680 shutdown_connections 2024-04-06T14:58:40.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.728+0000 7f49301ee700 1 -- 172.21.15.32:0/1654909680 wait complete. 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.728+0000 7f49301ee700 1 Processor -- start 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f49301ee700 1 -- start start 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f49301ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928101be0 0x7f4928102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f49301ee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4928102580 0x7f4928102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f49301ee700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49281049e0 con 0x7f4928101be0 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f49301ee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49281034c0 con 0x7f4928102580 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f492df8a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928101be0 0x7f4928102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f492df8a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928101be0 0x7f4928102a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51046/0 (socket says 172.21.15.32:51046) 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f492df8a700 1 -- 172.21.15.32:0/3015123122 learned_addr learned my addr 172.21.15.32:0/3015123122 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f492d789700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4928102580 0x7f4928102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f492df8a700 1 -- 172.21.15.32:0/3015123122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4928102580 msgr2=0x7f4928102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f492df8a700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4928102580 0x7f4928102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f492df8a700 1 -- 172.21.15.32:0/3015123122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4924005040 con 0x7f4928101be0 2024-04-06T14:58:40.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f492d789700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4928102580 0x7f4928102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:58:40.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f492df8a700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928101be0 0x7f4928102a40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f491800eb40 tx=0x7f491800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:40.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.729+0000 7f491e7fc700 1 -- 172.21.15.32:0/3015123122 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4918007cc0 con 0x7f4928101be0 2024-04-06T14:58:40.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.730+0000 7f491e7fc700 1 -- 172.21.15.32:0/3015123122 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f491800f040 con 0x7f4928101be0 2024-04-06T14:58:40.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.730+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4928103710 con 0x7f4928101be0 2024-04-06T14:58:40.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.730+0000 7f491e7fc700 1 -- 172.21.15.32:0/3015123122 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4918009570 con 0x7f4928101be0 2024-04-06T14:58:40.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.730+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4928119df0 con 0x7f4928101be0 2024-04-06T14:58:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.730+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49280fd330 con 0x7f4928101be0 2024-04-06T14:58:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.731+0000 7f491e7fc700 1 -- 172.21.15.32:0/3015123122 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f49180096d0 con 0x7f4928101be0 2024-04-06T14:58:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.731+0000 7f491e7fc700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f491406cc20 0x7f491406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.731+0000 7f491e7fc700 1 -- 172.21.15.32:0/3015123122 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f491808bc40 con 0x7f4928101be0 2024-04-06T14:58:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.731+0000 7f492d789700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f491406cc20 0x7f491406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.732+0000 7f492d789700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f491406cc20 0x7f491406f0d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4928104170 tx=0x7f4924009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:40.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.736+0000 7f491e7fc700 1 -- 172.21.15.32:0/3015123122 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4918091020 con 0x7f4928101be0 2024-04-06T14:58:40.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:40 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3475811580' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-06T14:58:40.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:40 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/410702571' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-06T14:58:40.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:40 smithi032 ceph-mon[147414]: pgmap v1160: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:40.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.989+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f4928064970 con 0x7f4928101be0 2024-04-06T14:58:40.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.990+0000 7f491e7fc700 1 -- 172.21.15.32:0/3015123122 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v39) v1 ==== 107+0+4844 (secure 0 0 0) 0x7f4918055d30 con 0x7f4928101be0 2024-04-06T14:58:40.976 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:40.976 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44803,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6827/2158771540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2158771540},{"type":"v1","addr":"172.21.15.86:6827","nonce":2158771540}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:56:45.421092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34382,"mds_1":34414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34382":{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":0,"incarnation":34,"state":"up:resolve","state_seq":464,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34414":{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":1,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:40.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f491406cc20 msgr2=0x7f491406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f491406cc20 0x7f491406f0d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4928104170 tx=0x7f4924009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:40.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928101be0 msgr2=0x7f4928102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:40.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928101be0 0x7f4928102a40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f491800eb40 tx=0x7f491800b040 comp rx=0 tx=0).stop 2024-04-06T14:58:40.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 shutdown_connections 2024-04-06T14:58:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f491406cc20 0x7f491406f0d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4928101be0 0x7f4928102a40 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 --2- 172.21.15.32:0/3015123122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4928102580 0x7f4928102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 >> 172.21.15.32:0/3015123122 conn(0x7f4928072ad0 msgr2=0x7f49280fa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 shutdown_connections 2024-04-06T14:58:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:40.992+0000 7f49301ee700 1 -- 172.21.15.32:0/3015123122 wait complete. 2024-04-06T14:58:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 35 2024-04-06T14:58:40.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-04-06T14:58:41.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:40 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3475811580' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-06T14:58:41.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:40 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/410702571' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-06T14:58:41.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:40 smithi086 ceph-mon[110975]: pgmap v1160: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:41.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.128+0000 7fb98e72a700 1 -- 172.21.15.32:0/2191199524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 msgr2=0x7fb9881091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.128+0000 7fb98e72a700 1 --2- 172.21.15.32:0/2191199524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 0x7fb9881091e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb970004940 tx=0x7fb970009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:41.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.128+0000 7fb98e72a700 1 -- 172.21.15.32:0/2191199524 shutdown_connections 2024-04-06T14:58:41.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.128+0000 7fb98e72a700 1 --2- 172.21.15.32:0/2191199524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 0x7fb9881091e0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.128+0000 7fb98e72a700 1 --2- 172.21.15.32:0/2191199524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb988064e40 0x7fb988065210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.128+0000 7fb98e72a700 1 -- 172.21.15.32:0/2191199524 >> 172.21.15.32:0/2191199524 conn(0x7fb988072ad0 msgr2=0x7fb988072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.128+0000 7fb98e72a700 1 -- 172.21.15.32:0/2191199524 shutdown_connections 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.128+0000 7fb98e72a700 1 -- 172.21.15.32:0/2191199524 wait complete. 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.128+0000 7fb98e72a700 1 Processor -- start 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb98e72a700 1 -- start start 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb98e72a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb988064e40 0x7fb988115960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb98e72a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 0x7fb988115ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb98e72a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb988116580 con 0x7fb988065750 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb98e72a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98810faf0 con 0x7fb988064e40 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb9877fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 0x7fb988115ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb9877fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 0x7fb988115ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51068/0 (socket says 172.21.15.32:51068) 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb9877fe700 1 -- 172.21.15.32:0/3896421820 learned_addr learned my addr 172.21.15.32:0/3896421820 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb987fff700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb988064e40 0x7fb988115960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb9877fe700 1 -- 172.21.15.32:0/3896421820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb988064e40 msgr2=0x7fb988115960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb9877fe700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb988064e40 0x7fb988115960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb9877fe700 1 -- 172.21.15.32:0/3896421820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb970005040 con 0x7fb988065750 2024-04-06T14:58:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb9877fe700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 0x7fb988115ea0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb97000b3b0 tx=0x7fb970009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb984ff9700 1 -- 172.21.15.32:0/3896421820 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb97000e040 con 0x7fb988065750 2024-04-06T14:58:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98810fd70 con 0x7fb988065750 2024-04-06T14:58:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.129+0000 7fb984ff9700 1 -- 172.21.15.32:0/3896421820 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb970013420 con 0x7fb988065750 2024-04-06T14:58:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.130+0000 7fb984ff9700 1 -- 172.21.15.32:0/3896421820 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb97001c630 con 0x7fb988065750 2024-04-06T14:58:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.130+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb988110330 con 0x7fb988065750 2024-04-06T14:58:41.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.130+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9880fc220 con 0x7fb988065750 2024-04-06T14:58:41.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.131+0000 7fb984ff9700 1 -- 172.21.15.32:0/3896421820 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb970013970 con 0x7fb988065750 2024-04-06T14:58:41.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.131+0000 7fb984ff9700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb97406cc20 0x7fb97406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:41.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.131+0000 7fb984ff9700 1 -- 172.21.15.32:0/3896421820 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb97008d2b0 con 0x7fb988065750 2024-04-06T14:58:41.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.132+0000 7fb987fff700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb97406cc20 0x7fb97406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:41.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.132+0000 7fb987fff700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb97406cc20 0x7fb97406f0d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb978006fd0 tx=0x7fb978008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:41.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.136+0000 7fb984ff9700 1 -- 172.21.15.32:0/3896421820 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb97005ad80 con 0x7fb988065750 2024-04-06T14:58:41.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.388+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7fb98810bf00 con 0x7fb988065750 2024-04-06T14:58:41.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.389+0000 7fb984ff9700 1 -- 172.21.15.32:0/3896421820 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v39) v1 ==== 107+0+4847 (secure 0 0 0) 0x7fb97005a910 con 0x7fb988065750 2024-04-06T14:58:41.375 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:41.375 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44803,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6827/2158771540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2158771540},{"type":"v1","addr":"172.21.15.86:6827","nonce":2158771540}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:56:46.422696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34382,"mds_1":34414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34382":{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":0,"incarnation":34,"state":"up:resolve","state_seq":464,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34414":{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":1,"incarnation":34,"state":"up:resolve","state_seq":459,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb97406cc20 msgr2=0x7fb97406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb97406cc20 0x7fb97406f0d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb978006fd0 tx=0x7fb978008040 comp rx=0 tx=0).stop 2024-04-06T14:58:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 msgr2=0x7fb988115ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 0x7fb988115ea0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb97000b3b0 tx=0x7fb970009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 shutdown_connections 2024-04-06T14:58:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb97406cc20 0x7fb97406f0d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb988064e40 0x7fb988115960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 --2- 172.21.15.32:0/3896421820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb988065750 0x7fb988115ea0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 >> 172.21.15.32:0/3896421820 conn(0x7fb988072ad0 msgr2=0x7fb9880f95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 shutdown_connections 2024-04-06T14:58:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.391+0000 7fb98e72a700 1 -- 172.21.15.32:0/3896421820 wait complete. 2024-04-06T14:58:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 36 2024-04-06T14:58:41.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 -- 172.21.15.32:0/4075385491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 msgr2=0x7f36801030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 --2- 172.21.15.32:0/4075385491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 0x7f36801030b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f3670004940 tx=0x7f3670009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 -- 172.21.15.32:0/4075385491 shutdown_connections 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 --2- 172.21.15.32:0/4075385491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3680103680 0x7f3680064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 --2- 172.21.15.32:0/4075385491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 0x7f36801030b0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 -- 172.21.15.32:0/4075385491 >> 172.21.15.32:0/4075385491 conn(0x7f3680072ad0 msgr2=0x7f3680072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 -- 172.21.15.32:0/4075385491 shutdown_connections 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 -- 172.21.15.32:0/4075385491 wait complete. 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 Processor -- start 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.527+0000 7f3684e0c700 1 -- start start 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f3684e0c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 0x7f36800fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f3684e0c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3680103680 0x7f36800fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f3684e0c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36800fdb90 con 0x7f3680102ce0 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f3684e0c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36800fc6c0 con 0x7f3680103680 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f367e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 0x7f36800fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f367e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 0x7f36800fbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51086/0 (socket says 172.21.15.32:51086) 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f367e59c700 1 -- 172.21.15.32:0/4093820092 learned_addr learned my addr 172.21.15.32:0/4093820092 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f367e59c700 1 -- 172.21.15.32:0/4093820092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3680103680 msgr2=0x7f36800fc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f367e59c700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3680103680 0x7f36800fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f367e59c700 1 -- 172.21.15.32:0/4093820092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670005040 con 0x7f3680102ce0 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f367e59c700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 0x7f36800fbc40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f367000b3b0 tx=0x7f3670009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f366effd700 1 -- 172.21.15.32:0/4093820092 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f367000e040 con 0x7f3680102ce0 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f366effd700 1 -- 172.21.15.32:0/4093820092 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3670013420 con 0x7f3680102ce0 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36800fc940 con 0x7f3680102ce0 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f366effd700 1 -- 172.21.15.32:0/4093820092 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f367001c630 con 0x7f3680102ce0 2024-04-06T14:58:41.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.528+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3680119f00 con 0x7f3680102ce0 2024-04-06T14:58:41.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.529+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36800fe430 con 0x7f3680102ce0 2024-04-06T14:58:41.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.530+0000 7f366effd700 1 -- 172.21.15.32:0/4093820092 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3670013970 con 0x7f3680102ce0 2024-04-06T14:58:41.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.530+0000 7f366effd700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f366806cc20 0x7f366806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:41.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.530+0000 7f366effd700 1 -- 172.21.15.32:0/4093820092 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f367008d160 con 0x7f3680102ce0 2024-04-06T14:58:41.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.530+0000 7f367dd9b700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f366806cc20 0x7f366806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:41.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.531+0000 7f367dd9b700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f366806cc20 0x7f366806f0d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f36800fd370 tx=0x7f3674009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.535+0000 7f366effd700 1 -- 172.21.15.32:0/4093820092 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3670057260 con 0x7f3680102ce0 2024-04-06T14:58:41.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.790+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f3680064970 con 0x7f3680102ce0 2024-04-06T14:58:41.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.791+0000 7f366effd700 1 -- 172.21.15.32:0/4093820092 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v39) v1 ==== 107+0+4851 (secure 0 0 0) 0x7f367005a880 con 0x7f3680102ce0 2024-04-06T14:58:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:41.777 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44803,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6827/2158771540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2158771540},{"type":"v1","addr":"172.21.15.86:6827","nonce":2158771540}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:56:47.427781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34382,"mds_1":34414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34382":{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":465,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34414":{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":1,"incarnation":34,"state":"up:reconnect","state_seq":460,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.793+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f366806cc20 msgr2=0x7f366806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.793+0000 7f3684e0c700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f366806cc20 0x7f366806f0d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f36800fd370 tx=0x7f3674009ce0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.793+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 msgr2=0x7f36800fbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.793+0000 7f3684e0c700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 0x7f36800fbc40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f367000b3b0 tx=0x7f3670009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.793+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 shutdown_connections 2024-04-06T14:58:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.793+0000 7f3684e0c700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f366806cc20 0x7f366806f0d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.793+0000 7f3684e0c700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3680102ce0 0x7f36800fbc40 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.793+0000 7f3684e0c700 1 --2- 172.21.15.32:0/4093820092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3680103680 0x7f36800fc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.793+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 >> 172.21.15.32:0/4093820092 conn(0x7f3680072ad0 msgr2=0x7f3680105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.794+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 shutdown_connections 2024-04-06T14:58:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.794+0000 7f3684e0c700 1 -- 172.21.15.32:0/4093820092 wait complete. 2024-04-06T14:58:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 37 2024-04-06T14:58:41.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-04-06T14:58:41.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.928+0000 7f25591bd700 1 -- 172.21.15.32:0/1891219722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 msgr2=0x7f2554101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.928+0000 7f25591bd700 1 --2- 172.21.15.32:0/1891219722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 0x7f2554101fb0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2544004940 tx=0x7f2544009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.928+0000 7f25591bd700 1 -- 172.21.15.32:0/1891219722 shutdown_connections 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.928+0000 7f25591bd700 1 --2- 172.21.15.32:0/1891219722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2554102580 0x7f25540fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.928+0000 7f25591bd700 1 --2- 172.21.15.32:0/1891219722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 0x7f2554101fb0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.928+0000 7f25591bd700 1 -- 172.21.15.32:0/1891219722 >> 172.21.15.32:0/1891219722 conn(0x7f2554072ad0 msgr2=0x7f2554072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.928+0000 7f25591bd700 1 -- 172.21.15.32:0/1891219722 shutdown_connections 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.928+0000 7f25591bd700 1 -- 172.21.15.32:0/1891219722 wait complete. 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.929+0000 7f25591bd700 1 Processor -- start 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.929+0000 7f25591bd700 1 -- start start 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.929+0000 7f25591bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 0x7f255410e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.929+0000 7f25591bd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2554102580 0x7f255410eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.929+0000 7f25591bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255410f3f0 con 0x7f2554101be0 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.929+0000 7f25591bd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255410f530 con 0x7f2554102580 2024-04-06T14:58:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.929+0000 7f2552d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 0x7f255410e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.929+0000 7f2552d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 0x7f255410e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51112/0 (socket says 172.21.15.32:51112) 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.929+0000 7f2552d9d700 1 -- 172.21.15.32:0/1932692362 learned_addr learned my addr 172.21.15.32:0/1932692362 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f255259c700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2554102580 0x7f255410eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f2552d9d700 1 -- 172.21.15.32:0/1932692362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2554102580 msgr2=0x7f255410eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f2552d9d700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2554102580 0x7f255410eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f2552d9d700 1 -- 172.21.15.32:0/1932692362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2544005040 con 0x7f2554101be0 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f2552d9d700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 0x7f255410e970 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f254400b3b0 tx=0x7f2544009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f253b7fe700 1 -- 172.21.15.32:0/1932692362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f254400e040 con 0x7f2554101be0 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f253b7fe700 1 -- 172.21.15.32:0/1932692362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2544013420 con 0x7f2554101be0 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f253b7fe700 1 -- 172.21.15.32:0/1932692362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f254401c610 con 0x7f2554101be0 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255410f7b0 con 0x7f2554101be0 2024-04-06T14:58:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.930+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2554117ce0 con 0x7f2554101be0 2024-04-06T14:58:41.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.931+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25540fd330 con 0x7f2554101be0 2024-04-06T14:58:41.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.932+0000 7f253b7fe700 1 -- 172.21.15.32:0/1932692362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2544013a00 con 0x7f2554101be0 2024-04-06T14:58:41.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.932+0000 7f253b7fe700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f253c06cc20 0x7f253c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:41.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.932+0000 7f253b7fe700 1 -- 172.21.15.32:0/1932692362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f254408d170 con 0x7f2554101be0 2024-04-06T14:58:41.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.932+0000 7f255259c700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f253c06cc20 0x7f253c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:41.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.933+0000 7f255259c700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f253c06cc20 0x7f253c06f0d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f25540655b0 tx=0x7f254800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:41.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:41.937+0000 7f253b7fe700 1 -- 172.21.15.32:0/1932692362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f25440571b0 con 0x7f2554101be0 2024-04-06T14:58:42.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:41 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3015123122' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-06T14:58:42.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:41 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3896421820' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-06T14:58:42.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:41 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4093820092' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-06T14:58:42.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.192+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7f2554065360 con 0x7f2554101be0 2024-04-06T14:58:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.192+0000 7f253b7fe700 1 -- 172.21.15.32:0/1932692362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v39) v1 ==== 107+0+4845 (secure 0 0 0) 0x7f254405a7d0 con 0x7f2554101be0 2024-04-06T14:58:42.178 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:42.178 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44313,"name":"cephfs.smithi086.nikfjj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6825/3277500545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3277500545},{"type":"v1","addr":"172.21.15.86:6825","nonce":3277500545}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44803,"name":"cephfs.smithi086.ojmchr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6827/2158771540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2158771540},{"type":"v1","addr":"172.21.15.86:6827","nonce":2158771540}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T14:14:02.044394+0000","modified":"2024-04-06T14:56:48.433401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34382,"mds_1":34414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34382":{"gid":34382,"name":"cephfs.smithi032.gcjdjo","rank":0,"incarnation":34,"state":"up:rejoin","state_seq":466,"addr":"172.21.15.32:6827/3498377443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3498377443},{"type":"v1","addr":"172.21.15.32:6827","nonce":3498377443}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34414":{"gid":34414,"name":"cephfs.smithi032.mqoaof","rank":1,"incarnation":34,"state":"up:rejoin","state_seq":461,"addr":"172.21.15.32:6829/3282445524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3282445524},{"type":"v1","addr":"172.21.15.32:6829","nonce":3282445524}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T14:58:42.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.194+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f253c06cc20 msgr2=0x7f253c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.194+0000 7f25591bd700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f253c06cc20 0x7f253c06f0d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f25540655b0 tx=0x7f254800b040 comp rx=0 tx=0).stop 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.194+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 msgr2=0x7f255410e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.194+0000 7f25591bd700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 0x7f255410e970 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f254400b3b0 tx=0x7f2544009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.195+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 shutdown_connections 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.195+0000 7f25591bd700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f253c06cc20 0x7f253c06f0d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.195+0000 7f25591bd700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2554101be0 0x7f255410e970 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.195+0000 7f25591bd700 1 --2- 172.21.15.32:0/1932692362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2554102580 0x7f255410eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.195+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 >> 172.21.15.32:0/1932692362 conn(0x7f2554072ad0 msgr2=0x7f2554104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.195+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 shutdown_connections 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.195+0000 7f25591bd700 1 -- 172.21.15.32:0/1932692362 wait complete. 2024-04-06T14:58:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 38 2024-04-06T14:58:42.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:41 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3015123122' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-06T14:58:42.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:41 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3896421820' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-06T14:58:42.192 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:41 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4093820092' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-06T14:58:42.193 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-06T14:58:42.204 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-06T14:58:42.204 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:58:42.204 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T14:58:42.224 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:58:42.224 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T14:58:42.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 -- 172.21.15.32:0/3172350149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c1025a0 msgr2=0x7f188c0fbc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 --2- 172.21.15.32:0/3172350149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c1025a0 0x7f188c0fbc10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1874004940 tx=0x7f1874009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 -- 172.21.15.32:0/3172350149 shutdown_connections 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 --2- 172.21.15.32:0/3172350149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c1025a0 0x7f188c0fbc10 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 --2- 172.21.15.32:0/3172350149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c101c00 0x7f188c101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 -- 172.21.15.32:0/3172350149 >> 172.21.15.32:0/3172350149 conn(0x7f188c0f7750 msgr2=0x7f188c0f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 -- 172.21.15.32:0/3172350149 shutdown_connections 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 -- 172.21.15.32:0/3172350149 wait complete. 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 Processor -- start 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.463+0000 7f1892596700 1 -- start start 2024-04-06T14:58:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f1892596700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c101c00 0x7f188c1137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f1892596700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c1025a0 0x7f188c113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f1892596700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c114340 con 0x7f188c101c00 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f1892596700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c10d940 con 0x7f188c1025a0 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f188bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c101c00 0x7f188c1137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f188bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c101c00 0x7f188c1137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51122/0 (socket says 172.21.15.32:51122) 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f188bfff700 1 -- 172.21.15.32:0/346538170 learned_addr learned my addr 172.21.15.32:0/346538170 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f188b7fe700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c1025a0 0x7f188c113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f188bfff700 1 -- 172.21.15.32:0/346538170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c1025a0 msgr2=0x7f188c113cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f188bfff700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c1025a0 0x7f188c113cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f188bfff700 1 -- 172.21.15.32:0/346538170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874005040 con 0x7f188c101c00 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f188b7fe700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c1025a0 0x7f188c113cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f188bfff700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c101c00 0x7f188c1137b0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f187c00c8c0 tx=0x7f187c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f1888ff9700 1 -- 172.21.15.32:0/346538170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187c00a980 con 0x7f188c101c00 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188c10dc20 con 0x7f188c101c00 2024-04-06T14:58:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f1888ff9700 1 -- 172.21.15.32:0/346538170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f187c00ce60 con 0x7f188c101c00 2024-04-06T14:58:42.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f1888ff9700 1 -- 172.21.15.32:0/346538170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187c0113a0 con 0x7f188c101c00 2024-04-06T14:58:42.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.464+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188c10e1e0 con 0x7f188c101c00 2024-04-06T14:58:42.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.466+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f188c0fd350 con 0x7f188c101c00 2024-04-06T14:58:42.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.466+0000 7f1888ff9700 1 -- 172.21.15.32:0/346538170 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f187c012400 con 0x7f188c101c00 2024-04-06T14:58:42.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.466+0000 7f1888ff9700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f187806cc20 0x7f187806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:42.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.467+0000 7f1888ff9700 1 -- 172.21.15.32:0/346538170 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f187c08c140 con 0x7f188c101c00 2024-04-06T14:58:42.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.467+0000 7f188b7fe700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f187806cc20 0x7f187806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:42.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.467+0000 7f188b7fe700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f187806cc20 0x7f187806f0d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f187400b3b0 tx=0x7f1874009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:42.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.471+0000 7f1888ff9700 1 -- 172.21.15.32:0/346538170 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f187c059c10 con 0x7f188c101c00 2024-04-06T14:58:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.676+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f188c108b90 con 0x7f188c101c00 2024-04-06T14:58:42.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.677+0000 7f1888ff9700 1 -- 172.21.15.32:0/346538170 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16504 (secure 0 0 0) 0x7f187c0597a0 con 0x7f188c101c00 2024-04-06T14:58:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:42.664 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","created":"2024-04-06T14:06:48.330591+0000","modified":"2024-04-06T14:41:33.675047+0000","last_up_change":"2024-04-06T14:25:47.031924+0000","last_in_change":"2024-04-06T14:12:17.748510+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T14:11:20.110423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-06T14:14:00.974689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-06T14:14:01.457260+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6803","nonce":969207178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6805","nonce":969207178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6809","nonce":969207178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6807","nonce":969207178}]},"public_addr":"172.21.15.32:6803/969207178","cluster_addr":"172.21.15.32:6805/969207178","heartbeat_back_addr":"172.21.15.32:6809/969207178","heartbeat_front_addr":"172.21.15.32:6807/969207178","state":["exists","up"]},{"osd":1,"uuid":"9185a389-ef69-41bc-9e10-46c6a2e02a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6811","nonce":197940852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6813","nonce":197940852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6817","nonce":197940852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6815","nonce":197940852}]},"public_addr":"172.21.15.32:6811/197940852","cluster_addr":"172.21.15.32:6813/197940852","heartbeat_back_addr":"172.21.15.32:6817/197940852","heartbeat_front_addr":"172.21.15.32:6815/197940852","state":["exists","up"]},{"osd":2,"uuid":"af4793e1-b5bb-4919-838e-67703c83296a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6819","nonce":1957975204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6821","nonce":1957975204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6825","nonce":1957975204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6823","nonce":1957975204}]},"public_addr":"172.21.15.32:6819/1957975204","cluster_addr":"172.21.15.32:6821/1957975204","heartbeat_back_addr":"172.21.15.32:6825/1957975204","heartbeat_front_addr":"172.21.15.32:6823/1957975204","state":["exists","up"]},{"osd":3,"uuid":"70c8b884-5214-44f2-9a31-55556a11c20a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6801","nonce":3817339654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6803","nonce":3817339654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6807","nonce":3817339654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6805","nonce":3817339654}]},"public_addr":"172.21.15.86:6801/3817339654","cluster_addr":"172.21.15.86:6803/3817339654","heartbeat_back_addr":"172.21.15.86:6807/3817339654","heartbeat_front_addr":"172.21.15.86:6805/3817339654","state":["exists","up"]},{"osd":4,"uuid":"91f548ac-4f42-4083-a027-295b829646a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6809","nonce":1198497064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6811","nonce":1198497064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6815","nonce":1198497064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6813","nonce":1198497064}]},"public_addr":"172.21.15.86:6809/1198497064","cluster_addr":"172.21.15.86:6811/1198497064","heartbeat_back_addr":"172.21.15.86:6815/1198497064","heartbeat_front_addr":"172.21.15.86:6813/1198497064","state":["exists","up"]},{"osd":5,"uuid":"7037f613-ffd9-4b0e-84a1-0c04d4181ff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6817","nonce":3374872376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6819","nonce":3374872376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6823","nonce":3374872376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6821","nonce":3374872376}]},"public_addr":"172.21.15.86:6817/3374872376","cluster_addr":"172.21.15.86:6819/3374872376","heartbeat_back_addr":"172.21.15.86:6823/3374872376","heartbeat_front_addr":"172.21.15.86:6821/3374872376","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:13.684938+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:44.549488+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:15.893369+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:41.588409+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:08.612078+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:35.380707+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6828/1704327767":"2024-04-07T14:25:48.051314+0000","172.21.15.86:6825/3246813202":"2024-04-07T14:25:48.051344+0000","172.21.15.32:6801/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.86:0/2189072732":"2024-04-07T14:18:05.813583+0000","172.21.15.32:0/3548398153":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/935909372":"2024-04-07T14:07:51.523157+0000","172.21.15.32:6800/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6800/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6829/1704327767":"2024-04-07T14:25:48.051314+0000","172.21.15.32:6827/4235185528":"2024-04-07T14:14:16.952740+0000","172.21.15.32:6801/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/3250004174":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/711512148":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/68274703":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/3330131734":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/1187866165":"2024-04-07T14:08:57.599030+0000","172.21.15.86:0/2427526405":"2024-04-07T14:18:05.813583+0000","172.21.15.86:0/2137415724":"2024-04-07T14:18:05.813583+0000","172.21.15.32:0/2605964794":"2024-04-07T14:21:22.517015+0000","172.21.15.32:0/3796020247":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6800/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/2227455350":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/2524933837":"2024-04-07T14:21:22.517015+0000","172.21.15.32:0/2574238932":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6800/3067649142":"2024-04-07T14:16:30.572936+0000","172.21.15.32:0/1664607864":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6801/3067649142":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6826/4235185528":"2024-04-07T14:14:16.952740+0000","172.21.15.32:0/1877817963":"2024-04-07T14:07:17.560285+0000","172.21.15.86:6828/1279185277":"2024-04-07T14:18:05.813583+0000","172.21.15.86:6829/1279185277":"2024-04-07T14:18:05.813583+0000","172.21.15.32:6801/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.86:0/2712193968":"2024-04-07T14:18:05.813583+0000","172.21.15.32:0/3226572738":"2024-04-07T14:16:30.572936+0000","172.21.15.32:0/2092569962":"2024-04-07T14:21:22.517015+0000","172.21.15.32:6800/2780016783":"2024-04-07T14:21:22.517015+0000","172.21.15.32:6801/2780016783":"2024-04-07T14:21:22.517015+0000","172.21.15.32:0/69668297":"2024-04-07T14:21:22.517015+0000","172.21.15.86:6824/3246813202":"2024-04-07T14:25:48.051344+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T14:58:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.679+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f187806cc20 msgr2=0x7f187806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.679+0000 7f1892596700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f187806cc20 0x7f187806f0d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f187400b3b0 tx=0x7f1874009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.679+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c101c00 msgr2=0x7f188c1137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.679+0000 7f1892596700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c101c00 0x7f188c1137b0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f187c00c8c0 tx=0x7f187c0044c0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.679+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 shutdown_connections 2024-04-06T14:58:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.679+0000 7f1892596700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f187806cc20 0x7f187806f0d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.679+0000 7f1892596700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c101c00 0x7f188c1137b0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.679+0000 7f1892596700 1 --2- 172.21.15.32:0/346538170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c1025a0 0x7f188c113cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.679+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 >> 172.21.15.32:0/346538170 conn(0x7f188c0f7750 msgr2=0x7f188c0f8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.680+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 shutdown_connections 2024-04-06T14:58:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.680+0000 7f1892596700 1 -- 172.21.15.32:0/346538170 wait complete. 2024-04-06T14:58:42.678 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T14:11:20.110423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-06T14:14:00.974689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-06T14:14:01.457260+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T14:58:42.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.818+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2715772347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 msgr2=0x7fb6ec0fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.818+0000 7fb6f42b8700 1 --2- 172.21.15.32:0/2715772347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 0x7fb6ec0fbc20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8004940 tx=0x7fb6e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.818+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2715772347 shutdown_connections 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.818+0000 7fb6f42b8700 1 --2- 172.21.15.32:0/2715772347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 0x7fb6ec0fbc20 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.818+0000 7fb6f42b8700 1 --2- 172.21.15.32:0/2715772347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec101c10 0x7fb6ec101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.818+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2715772347 >> 172.21.15.32:0/2715772347 conn(0x7fb6ec0f7720 msgr2=0x7fb6ec0f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.818+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2715772347 shutdown_connections 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.818+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2715772347 wait complete. 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.818+0000 7fb6f42b8700 1 Processor -- start 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f42b8700 1 -- start start 2024-04-06T14:58:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f42b8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec101c10 0x7fb6ec1126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f42b8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 0x7fb6ec112c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f42b8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec113270 con 0x7fb6ec1025b0 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f42b8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec10c830 con 0x7fb6ec101c10 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f1853700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 0x7fb6ec112c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f1853700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 0x7fb6ec112c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51142/0 (socket says 172.21.15.32:51142) 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f1853700 1 -- 172.21.15.32:0/2043627120 learned_addr learned my addr 172.21.15.32:0/2043627120 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f1853700 1 -- 172.21.15.32:0/2043627120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec101c10 msgr2=0x7fb6ec1126a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f1853700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec101c10 0x7fb6ec1126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f1853700 1 -- 172.21.15.32:0/2043627120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e8005040 con 0x7fb6ec1025b0 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f1853700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 0x7fb6ec112c20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fb6e80093b0 tx=0x7fb6e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6e2ffd700 1 -- 172.21.15.32:0/2043627120 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6e800e040 con 0x7fb6ec1025b0 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6e2ffd700 1 -- 172.21.15.32:0/2043627120 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6e8013420 con 0x7fb6ec1025b0 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec10ca60 con 0x7fb6ec1025b0 2024-04-06T14:58:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.819+0000 7fb6e2ffd700 1 -- 172.21.15.32:0/2043627120 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6e801c630 con 0x7fb6ec1025b0 2024-04-06T14:58:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.820+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ec10cfa0 con 0x7fb6ec1025b0 2024-04-06T14:58:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.820+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ec0fd360 con 0x7fb6ec1025b0 2024-04-06T14:58:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.821+0000 7fb6e2ffd700 1 -- 172.21.15.32:0/2043627120 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb6e80139e0 con 0x7fb6ec1025b0 2024-04-06T14:58:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.821+0000 7fb6e2ffd700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.821+0000 7fb6e2ffd700 1 -- 172.21.15.32:0/2043627120 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb6e808d290 con 0x7fb6ec1025b0 2024-04-06T14:58:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.822+0000 7fb6f2054700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.822+0000 7fb6f2054700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec102d70 tx=0x7fb6dc007d40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:42.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:42.826+0000 7fb6e2ffd700 1 -- 172.21.15.32:0/2043627120 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb6e805ae10 con 0x7fb6ec1025b0 2024-04-06T14:58:42.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:42 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/1932692362' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-06T14:58:42.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:42 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/346538170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:58:42.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:42 smithi032 ceph-mon[147414]: pgmap v1161: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:43.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.030+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6ec10d220 con 0x7fb6ec1025b0 2024-04-06T14:58:43.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.031+0000 7fb6e2ffd700 1 -- 172.21.15.32:0/2043627120 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6e805a9a0 con 0x7fb6ec1025b0 2024-04-06T14:58:43.017 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-06T14:58:43.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 msgr2=0x7fb6d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec102d70 tx=0x7fb6dc007d40 comp rx=0 tx=0).stop 2024-04-06T14:58:43.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 msgr2=0x7fb6ec112c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 0x7fb6ec112c20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fb6e80093b0 tx=0x7fb6e8009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:43.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 shutdown_connections 2024-04-06T14:58:43.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fb6d806cc20 0x7fb6d806f0d0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec101c10 0x7fb6ec1126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 --2- 172.21.15.32:0/2043627120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1025b0 0x7fb6ec112c20 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 >> 172.21.15.32:0/2043627120 conn(0x7fb6ec0f7720 msgr2=0x7fb6ec0f8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:43.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 shutdown_connections 2024-04-06T14:58:43.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.033+0000 7fb6f42b8700 1 -- 172.21.15.32:0/2043627120 wait complete. 2024-04-06T14:58:43.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-06T14:58:43.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:42 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/1932692362' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-06T14:58:43.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:42 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/346538170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:58:43.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:42 smithi086 ceph-mon[110975]: pgmap v1161: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.170+0000 7f44f28ad700 1 -- 172.21.15.32:0/4132827023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec102580 msgr2=0x7f44ec0fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.170+0000 7f44f28ad700 1 --2- 172.21.15.32:0/4132827023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec102580 0x7f44ec0fbbf0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f44d4004940 tx=0x7f44d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.170+0000 7f44f28ad700 1 -- 172.21.15.32:0/4132827023 shutdown_connections 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.170+0000 7f44f28ad700 1 --2- 172.21.15.32:0/4132827023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec102580 0x7f44ec0fbbf0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.170+0000 7f44f28ad700 1 --2- 172.21.15.32:0/4132827023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44ec101be0 0x7f44ec101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.170+0000 7f44f28ad700 1 -- 172.21.15.32:0/4132827023 >> 172.21.15.32:0/4132827023 conn(0x7f44ec072ad0 msgr2=0x7f44ec072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.170+0000 7f44f28ad700 1 -- 172.21.15.32:0/4132827023 shutdown_connections 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.170+0000 7f44f28ad700 1 -- 172.21.15.32:0/4132827023 wait complete. 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.170+0000 7f44f28ad700 1 Processor -- start 2024-04-06T14:58:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44f28ad700 1 -- start start 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44f28ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec101be0 0x7f44ec1126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44f28ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44ec102580 0x7f44ec112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44f28ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec113230 con 0x7f44ec101be0 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44f28ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec10c7f0 con 0x7f44ec102580 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec101be0 0x7f44ec1126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec101be0 0x7f44ec1126a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51170/0 (socket says 172.21.15.32:51170) 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44ebfff700 1 -- 172.21.15.32:0/2674502822 learned_addr learned my addr 172.21.15.32:0/2674502822 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44eb7fe700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44ec102580 0x7f44ec112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44ebfff700 1 -- 172.21.15.32:0/2674502822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44ec102580 msgr2=0x7f44ec112be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44ebfff700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44ec102580 0x7f44ec112be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.171+0000 7f44ebfff700 1 -- 172.21.15.32:0/2674502822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d4005040 con 0x7f44ec101be0 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.172+0000 7f44ebfff700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec101be0 0x7f44ec1126a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f44dc002a50 tx=0x7f44dc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.172+0000 7f44e8ff9700 1 -- 172.21.15.32:0/2674502822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44dc007cc0 con 0x7f44ec101be0 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.172+0000 7f44e8ff9700 1 -- 172.21.15.32:0/2674502822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44dc00f040 con 0x7f44ec101be0 2024-04-06T14:58:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.172+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ec10cad0 con 0x7f44ec101be0 2024-04-06T14:58:43.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.172+0000 7f44e8ff9700 1 -- 172.21.15.32:0/2674502822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44dc00d9c0 con 0x7f44ec101be0 2024-04-06T14:58:43.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.172+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ec10cfe0 con 0x7f44ec101be0 2024-04-06T14:58:43.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.173+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ec112350 con 0x7f44ec101be0 2024-04-06T14:58:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.173+0000 7f44e8ff9700 1 -- 172.21.15.32:0/2674502822 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f44dc00db20 con 0x7f44ec101be0 2024-04-06T14:58:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.174+0000 7f44e8ff9700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44d806cc20 0x7f44d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.174+0000 7f44e8ff9700 1 -- 172.21.15.32:0/2674502822 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f44dc08bd70 con 0x7f44ec101be0 2024-04-06T14:58:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.174+0000 7f44eb7fe700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44d806cc20 0x7f44d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.174+0000 7f44eb7fe700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44d806cc20 0x7f44d806f0d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f44d4004890 tx=0x7f44d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:43.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.178+0000 7f44e8ff9700 1 -- 172.21.15.32:0/2674502822 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f44dc0597c0 con 0x7f44ec101be0 2024-04-06T14:58:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.381+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f44ec0fd330 con 0x7f44ec101be0 2024-04-06T14:58:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.381+0000 7f44e8ff9700 1 -- 172.21.15.32:0/2674502822 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f44dc059350 con 0x7f44ec101be0 2024-04-06T14:58:43.367 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-06T14:58:43.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44d806cc20 msgr2=0x7f44d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44d806cc20 0x7f44d806f0d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f44d4004890 tx=0x7f44d4009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec101be0 msgr2=0x7f44ec1126a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec101be0 0x7f44ec1126a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f44dc002a50 tx=0x7f44dc0046b0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 shutdown_connections 2024-04-06T14:58:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f44d806cc20 0x7f44d806f0d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44ec101be0 0x7f44ec1126a0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 --2- 172.21.15.32:0/2674502822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44ec102580 0x7f44ec112be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 >> 172.21.15.32:0/2674502822 conn(0x7f44ec072ad0 msgr2=0x7f44ec104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 shutdown_connections 2024-04-06T14:58:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.383+0000 7f44f28ad700 1 -- 172.21.15.32:0/2674502822 wait complete. 2024-04-06T14:58:43.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-06T14:58:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.522+0000 7f2dd9265700 1 -- 172.21.15.32:0/894674270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd4101c50 msgr2=0x7f2dd4102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.522+0000 7f2dd9265700 1 --2- 172.21.15.32:0/894674270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd4101c50 0x7f2dd4102020 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4004940 tx=0x7f2dc4009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.522+0000 7f2dd9265700 1 -- 172.21.15.32:0/894674270 shutdown_connections 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.522+0000 7f2dd9265700 1 --2- 172.21.15.32:0/894674270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dd41025f0 0x7f2dd40fbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.522+0000 7f2dd9265700 1 --2- 172.21.15.32:0/894674270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd4101c50 0x7f2dd4102020 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.522+0000 7f2dd9265700 1 -- 172.21.15.32:0/894674270 >> 172.21.15.32:0/894674270 conn(0x7f2dd40f77a0 msgr2=0x7f2dd40f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.523+0000 7f2dd9265700 1 -- 172.21.15.32:0/894674270 shutdown_connections 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.523+0000 7f2dd9265700 1 -- 172.21.15.32:0/894674270 wait complete. 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.523+0000 7f2dd9265700 1 Processor -- start 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.523+0000 7f2dd9265700 1 -- start start 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.523+0000 7f2dd9265700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dd4101c50 0x7f2dd410e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.523+0000 7f2dd9265700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd41025f0 0x7f2dd410ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.523+0000 7f2dd9265700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd410f450 con 0x7f2dd41025f0 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.523+0000 7f2dd9265700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd410f590 con 0x7f2dd4101c50 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd41025f0 0x7f2dd410ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:43.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd41025f0 0x7f2dd410ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51192/0 (socket says 172.21.15.32:51192) 2024-04-06T14:58:43.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd27fc700 1 -- 172.21.15.32:0/4144034205 learned_addr learned my addr 172.21.15.32:0/4144034205 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:43.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd2ffd700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dd4101c50 0x7f2dd410e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:43.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd27fc700 1 -- 172.21.15.32:0/4144034205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dd4101c50 msgr2=0x7f2dd410e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd27fc700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dd4101c50 0x7f2dd410e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd27fc700 1 -- 172.21.15.32:0/4144034205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4005040 con 0x7f2dd41025f0 2024-04-06T14:58:43.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd2ffd700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dd4101c50 0x7f2dd410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:58:43.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd27fc700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd41025f0 0x7f2dd410ef10 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8002a50 tx=0x7f2dc8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:43.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dbb7fe700 1 -- 172.21.15.32:0/4144034205 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dc8007cc0 con 0x7f2dd41025f0 2024-04-06T14:58:43.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dbb7fe700 1 -- 172.21.15.32:0/4144034205 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2dc800f040 con 0x7f2dd41025f0 2024-04-06T14:58:43.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd4117a40 con 0x7f2dd41025f0 2024-04-06T14:58:43.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dbb7fe700 1 -- 172.21.15.32:0/4144034205 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dc800d9a0 con 0x7f2dd41025f0 2024-04-06T14:58:43.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.524+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd4117e10 con 0x7f2dd41025f0 2024-04-06T14:58:43.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.525+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd40fef60 con 0x7f2dd41025f0 2024-04-06T14:58:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.526+0000 7f2dbb7fe700 1 -- 172.21.15.32:0/4144034205 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2dc800dc60 con 0x7f2dd41025f0 2024-04-06T14:58:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.526+0000 7f2dbb7fe700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2dbc06cc20 0x7f2dbc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.526+0000 7f2dbb7fe700 1 -- 172.21.15.32:0/4144034205 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2dc808be80 con 0x7f2dd41025f0 2024-04-06T14:58:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.526+0000 7f2dd2ffd700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2dbc06cc20 0x7f2dbc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:43.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.527+0000 7f2dd2ffd700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2dbc06cc20 0x7f2dbc06f0d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4004560 tx=0x7f2dc4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:43.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.531+0000 7f2dbb7fe700 1 -- 172.21.15.32:0/4144034205 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2dc8055e40 con 0x7f2dd41025f0 2024-04-06T14:58:43.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.734+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f2dd40fd3a0 con 0x7f2dd41025f0 2024-04-06T14:58:43.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.735+0000 7f2dbb7fe700 1 -- 172.21.15.32:0/4144034205 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2dc8059460 con 0x7f2dd41025f0 2024-04-06T14:58:43.721 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-06T14:58:43.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2dbc06cc20 msgr2=0x7f2dbc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2dbc06cc20 0x7f2dbc06f0d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4004560 tx=0x7f2dc4009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:43.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd41025f0 msgr2=0x7f2dd410ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd41025f0 0x7f2dd410ef10 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8002a50 tx=0x7f2dc8004600 comp rx=0 tx=0).stop 2024-04-06T14:58:43.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 shutdown_connections 2024-04-06T14:58:43.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f2dbc06cc20 0x7f2dbc06f0d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dd4101c50 0x7f2dd410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 --2- 172.21.15.32:0/4144034205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dd41025f0 0x7f2dd410ef10 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 >> 172.21.15.32:0/4144034205 conn(0x7f2dd40f77a0 msgr2=0x7f2dd4106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 shutdown_connections 2024-04-06T14:58:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.737+0000 7f2dd9265700 1 -- 172.21.15.32:0/4144034205 wait complete. 2024-04-06T14:58:43.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T14:58:43.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.875+0000 7f4c75800700 1 -- 172.21.15.32:0/1710860154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 msgr2=0x7f4c700fc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.875+0000 7f4c75800700 1 --2- 172.21.15.32:0/1710860154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 0x7f4c700fc850 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4c60004940 tx=0x7f4c60009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:43.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.875+0000 7f4c75800700 1 -- 172.21.15.32:0/1710860154 shutdown_connections 2024-04-06T14:58:43.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.875+0000 7f4c75800700 1 --2- 172.21.15.32:0/1710860154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 0x7f4c700fc850 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.875+0000 7f4c75800700 1 --2- 172.21.15.32:0/1710860154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c70101c80 0x7f4c70102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.875+0000 7f4c75800700 1 -- 172.21.15.32:0/1710860154 >> 172.21.15.32:0/1710860154 conn(0x7f4c70072290 msgr2=0x7f4c70072690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:43.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.875+0000 7f4c75800700 1 -- 172.21.15.32:0/1710860154 shutdown_connections 2024-04-06T14:58:43.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.875+0000 7f4c75800700 1 -- 172.21.15.32:0/1710860154 wait complete. 2024-04-06T14:58:43.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.876+0000 7f4c75800700 1 Processor -- start 2024-04-06T14:58:43.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.876+0000 7f4c75800700 1 -- start start 2024-04-06T14:58:43.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.876+0000 7f4c75800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 0x7f4c70112670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:43.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.876+0000 7f4c75800700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c70101c80 0x7f4c70112bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:43.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.876+0000 7f4c75800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70113240 con 0x7f4c700fc3e0 2024-04-06T14:58:43.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.876+0000 7f4c75800700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7010c850 con 0x7f4c70101c80 2024-04-06T14:58:43.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.876+0000 7f4c6effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 0x7f4c70112670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c6effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 0x7f4c70112670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51216/0 (socket says 172.21.15.32:51216) 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c6effd700 1 -- 172.21.15.32:0/3493952032 learned_addr learned my addr 172.21.15.32:0/3493952032 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c6e7fc700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c70101c80 0x7f4c70112bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c6effd700 1 -- 172.21.15.32:0/3493952032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c70101c80 msgr2=0x7f4c70112bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c6effd700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c70101c80 0x7f4c70112bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c6effd700 1 -- 172.21.15.32:0/3493952032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60005040 con 0x7f4c700fc3e0 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c6e7fc700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c70101c80 0x7f4c70112bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c6effd700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 0x7f4c70112670 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f4c58002a80 tx=0x7f4c58004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c677fe700 1 -- 172.21.15.32:0/3493952032 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c58007cc0 con 0x7f4c700fc3e0 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7010cae0 con 0x7f4c700fc3e0 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c677fe700 1 -- 172.21.15.32:0/3493952032 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c5800f040 con 0x7f4c700fc3e0 2024-04-06T14:58:43.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c677fe700 1 -- 172.21.15.32:0/3493952032 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5800da20 con 0x7f4c700fc3e0 2024-04-06T14:58:43.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.877+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7010d020 con 0x7f4c700fc3e0 2024-04-06T14:58:43.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.878+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c700fd3f0 con 0x7f4c700fc3e0 2024-04-06T14:58:43.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.879+0000 7f4c677fe700 1 -- 172.21.15.32:0/3493952032 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4c5800db80 con 0x7f4c700fc3e0 2024-04-06T14:58:43.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.879+0000 7f4c677fe700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4c5c06cc20 0x7f4c5c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:43.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.879+0000 7f4c677fe700 1 -- 172.21.15.32:0/3493952032 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4c5808be40 con 0x7f4c700fc3e0 2024-04-06T14:58:43.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.879+0000 7f4c6e7fc700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4c5c06cc20 0x7f4c5c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:43.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.880+0000 7f4c6e7fc700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4c5c06cc20 0x7f4c5c06f0d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4c60004f90 tx=0x7f4c60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:43.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:43.884+0000 7f4c677fe700 1 -- 172.21.15.32:0/3493952032 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4c58059890 con 0x7f4c700fc3e0 2024-04-06T14:58:44.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.101+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4c70108c10 con 0x7f4c700fc3e0 2024-04-06T14:58:44.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.101+0000 7f4c677fe700 1 -- 172.21.15.32:0/3493952032 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v85) v1 ==== 81+0+2308 (secure 0 0 0) 0x7f4c58059420 con 0x7f4c700fc3e0 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6828/1704327767 2024-04-07T14:25:48.051314+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:6825/3246813202 2024-04-07T14:25:48.051344+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/818700653 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:0/2189072732 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3548398153 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/935909372 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/818700653 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/3947155315 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6829/1704327767 2024-04-07T14:25:48.051314+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6827/4235185528 2024-04-07T14:14:16.952740+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/1833723736 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:44.088 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3250004174 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/711512148 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/68274703 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3330131734 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1187866165 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:0/2427526405 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:0/2137415724 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2605964794 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3796020247 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/1833723736 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2227455350 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2524933837 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2574238932 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/3067649142 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1664607864 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:44.089 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/3067649142 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6826/4235185528 2024-04-07T14:14:16.952740+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1877817963 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:6828/1279185277 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:6829/1279185277 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/3947155315 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:0/2712193968 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3226572738 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2092569962 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/2780016783 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/2780016783 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/69668297 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:44.090 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:6824/3246813202 2024-04-07T14:25:48.051344+0000 2024-04-06T14:58:44.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4c5c06cc20 msgr2=0x7f4c5c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4c5c06cc20 0x7f4c5c06f0d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4c60004f90 tx=0x7f4c60009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 msgr2=0x7f4c70112670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 0x7f4c70112670 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f4c58002a80 tx=0x7f4c58004600 comp rx=0 tx=0).stop 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 shutdown_connections 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f4c5c06cc20 0x7f4c5c06f0d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c700fc3e0 0x7f4c70112670 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 --2- 172.21.15.32:0/3493952032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c70101c80 0x7f4c70112bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 >> 172.21.15.32:0/3493952032 conn(0x7f4c70072290 msgr2=0x7f4c700ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 shutdown_connections 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.104+0000 7f4c75800700 1 -- 172.21.15.32:0/3493952032 wait complete. 2024-04-06T14:58:44.093 INFO:teuthology.orchestra.run.smithi032.stderr:listed 39 entries 2024-04-06T14:58:44.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:43 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2043627120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:58:44.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:43 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2674502822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T14:58:44.101 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:43 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/4144034205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T14:58:44.102 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-06T14:58:44.102 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T14:58:44.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:43 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2043627120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:58:44.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:43 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2674502822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T14:58:44.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:43 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/4144034205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T14:58:44.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T14:58:44.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 -- 172.21.15.32:0/253653298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 msgr2=0x7f8958103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 --2- 172.21.15.32:0/253653298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 0x7f8958103150 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f8948004940 tx=0x7f8948009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 -- 172.21.15.32:0/253653298 shutdown_connections 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 --2- 172.21.15.32:0/253653298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8958064e40 0x7f89580652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 --2- 172.21.15.32:0/253653298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 0x7f8958103150 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 -- 172.21.15.32:0/253653298 >> 172.21.15.32:0/253653298 conn(0x7f8958071c80 msgr2=0x7f8958072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 -- 172.21.15.32:0/253653298 shutdown_connections 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 -- 172.21.15.32:0/253653298 wait complete. 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 Processor -- start 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.299+0000 7f895de3b700 1 -- start start 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f895de3b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8958064e40 0x7f89580fbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f895de3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 0x7f89580fc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f895de3b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89580fdc40 con 0x7f8958102d80 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f895de3b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89580fc770 con 0x7f8958064e40 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f894effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 0x7f89580fc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f894effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 0x7f89580fc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51238/0 (socket says 172.21.15.32:51238) 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f894effd700 1 -- 172.21.15.32:0/869880255 learned_addr learned my addr 172.21.15.32:0/869880255 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f89577fe700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8958064e40 0x7f89580fbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f894effd700 1 -- 172.21.15.32:0/869880255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8958064e40 msgr2=0x7f89580fbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f894effd700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8958064e40 0x7f89580fbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f894effd700 1 -- 172.21.15.32:0/869880255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8948005040 con 0x7f8958102d80 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f894effd700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 0x7f89580fc230 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f894000eb40 tx=0x7f894000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f8954ff9700 1 -- 172.21.15.32:0/869880255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8940007cc0 con 0x7f8958102d80 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f8954ff9700 1 -- 172.21.15.32:0/869880255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f894000f040 con 0x7f8958102d80 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89580fc9f0 con 0x7f8958102d80 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f8954ff9700 1 -- 172.21.15.32:0/869880255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f894000c990 con 0x7f8958102d80 2024-04-06T14:58:44.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.300+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8958119e60 con 0x7f8958102d80 2024-04-06T14:58:44.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.301+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89581000a0 con 0x7f8958102d80 2024-04-06T14:58:44.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.302+0000 7f8954ff9700 1 -- 172.21.15.32:0/869880255 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f894000cc00 con 0x7f8958102d80 2024-04-06T14:58:44.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.302+0000 7f8954ff9700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f894406cc20 0x7f894406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:44.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.302+0000 7f8954ff9700 1 -- 172.21.15.32:0/869880255 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f894008bc40 con 0x7f8958102d80 2024-04-06T14:58:44.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.302+0000 7f89577fe700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f894406cc20 0x7f894406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:44.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.303+0000 7f89577fe700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f894406cc20 0x7f894406f0d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f894800b0d0 tx=0x7f8948009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:44.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.307+0000 7f8954ff9700 1 -- 172.21.15.32:0/869880255 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8940091020 con 0x7f8958102d80 2024-04-06T14:58:44.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.515+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89580fe4e0 con 0x7f8958102d80 2024-04-06T14:58:44.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.516+0000 7f8954ff9700 1 -- 172.21.15.32:0/869880255 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16504 (secure 0 0 0) 0x7f8940055cb0 con 0x7f8958102d80 2024-04-06T14:58:44.502 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-06T14:58:44.502 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"cabdc720-f41e-11ee-b649-cb9ed24678a4","created":"2024-04-06T14:06:48.330591+0000","modified":"2024-04-06T14:41:33.675047+0000","last_up_change":"2024-04-06T14:25:47.031924+0000","last_in_change":"2024-04-06T14:12:17.748510+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T14:11:20.110423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-06T14:14:00.974689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-06T14:14:01.457260+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b7d1600e-f3c2-48aa-a0e7-69a6ae8c3d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6803","nonce":969207178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6805","nonce":969207178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6809","nonce":969207178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":969207178},{"type":"v1","addr":"172.21.15.32:6807","nonce":969207178}]},"public_addr":"172.21.15.32:6803/969207178","cluster_addr":"172.21.15.32:6805/969207178","heartbeat_back_addr":"172.21.15.32:6809/969207178","heartbeat_front_addr":"172.21.15.32:6807/969207178","state":["exists","up"]},{"osd":1,"uuid":"9185a389-ef69-41bc-9e10-46c6a2e02a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6811","nonce":197940852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6813","nonce":197940852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6817","nonce":197940852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":197940852},{"type":"v1","addr":"172.21.15.32:6815","nonce":197940852}]},"public_addr":"172.21.15.32:6811/197940852","cluster_addr":"172.21.15.32:6813/197940852","heartbeat_back_addr":"172.21.15.32:6817/197940852","heartbeat_front_addr":"172.21.15.32:6815/197940852","state":["exists","up"]},{"osd":2,"uuid":"af4793e1-b5bb-4919-838e-67703c83296a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6819","nonce":1957975204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6821","nonce":1957975204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6825","nonce":1957975204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1957975204},{"type":"v1","addr":"172.21.15.32:6823","nonce":1957975204}]},"public_addr":"172.21.15.32:6819/1957975204","cluster_addr":"172.21.15.32:6821/1957975204","heartbeat_back_addr":"172.21.15.32:6825/1957975204","heartbeat_front_addr":"172.21.15.32:6823/1957975204","state":["exists","up"]},{"osd":3,"uuid":"70c8b884-5214-44f2-9a31-55556a11c20a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6801","nonce":3817339654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6803","nonce":3817339654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6807","nonce":3817339654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3817339654},{"type":"v1","addr":"172.21.15.86:6805","nonce":3817339654}]},"public_addr":"172.21.15.86:6801/3817339654","cluster_addr":"172.21.15.86:6803/3817339654","heartbeat_back_addr":"172.21.15.86:6807/3817339654","heartbeat_front_addr":"172.21.15.86:6805/3817339654","state":["exists","up"]},{"osd":4,"uuid":"91f548ac-4f42-4083-a027-295b829646a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6809","nonce":1198497064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6811","nonce":1198497064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6815","nonce":1198497064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1198497064},{"type":"v1","addr":"172.21.15.86:6813","nonce":1198497064}]},"public_addr":"172.21.15.86:6809/1198497064","cluster_addr":"172.21.15.86:6811/1198497064","heartbeat_back_addr":"172.21.15.86:6815/1198497064","heartbeat_front_addr":"172.21.15.86:6813/1198497064","state":["exists","up"]},{"osd":5,"uuid":"7037f613-ffd9-4b0e-84a1-0c04d4181ff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6817","nonce":3374872376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6819","nonce":3374872376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6823","nonce":3374872376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3374872376},{"type":"v1","addr":"172.21.15.86:6821","nonce":3374872376}]},"public_addr":"172.21.15.86:6817/3374872376","cluster_addr":"172.21.15.86:6819/3374872376","heartbeat_back_addr":"172.21.15.86:6823/3374872376","heartbeat_front_addr":"172.21.15.86:6821/3374872376","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:13.684938+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:10:44.549488+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:15.893369+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:11:41.588409+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:08.612078+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T14:12:35.380707+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6828/1704327767":"2024-04-07T14:25:48.051314+0000","172.21.15.86:6825/3246813202":"2024-04-07T14:25:48.051344+0000","172.21.15.32:6801/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.86:0/2189072732":"2024-04-07T14:18:05.813583+0000","172.21.15.32:0/3548398153":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/935909372":"2024-04-07T14:07:51.523157+0000","172.21.15.32:6800/818700653":"2024-04-07T14:07:17.560285+0000","172.21.15.32:6800/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.32:6829/1704327767":"2024-04-07T14:25:48.051314+0000","172.21.15.32:6827/4235185528":"2024-04-07T14:14:16.952740+0000","172.21.15.32:6801/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/3250004174":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/711512148":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/68274703":"2024-04-07T14:07:17.560285+0000","172.21.15.32:0/3330131734":"2024-04-07T14:08:57.599030+0000","172.21.15.32:0/1187866165":"2024-04-07T14:08:57.599030+0000","172.21.15.86:0/2427526405":"2024-04-07T14:18:05.813583+0000","172.21.15.86:0/2137415724":"2024-04-07T14:18:05.813583+0000","172.21.15.32:0/2605964794":"2024-04-07T14:21:22.517015+0000","172.21.15.32:0/3796020247":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6800/1833723736":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/2227455350":"2024-04-07T14:07:51.523157+0000","172.21.15.32:0/2524933837":"2024-04-07T14:21:22.517015+0000","172.21.15.32:0/2574238932":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6800/3067649142":"2024-04-07T14:16:30.572936+0000","172.21.15.32:0/1664607864":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6801/3067649142":"2024-04-07T14:16:30.572936+0000","172.21.15.32:6826/4235185528":"2024-04-07T14:14:16.952740+0000","172.21.15.32:0/1877817963":"2024-04-07T14:07:17.560285+0000","172.21.15.86:6828/1279185277":"2024-04-07T14:18:05.813583+0000","172.21.15.86:6829/1279185277":"2024-04-07T14:18:05.813583+0000","172.21.15.32:6801/3947155315":"2024-04-07T14:08:57.599030+0000","172.21.15.86:0/2712193968":"2024-04-07T14:18:05.813583+0000","172.21.15.32:0/3226572738":"2024-04-07T14:16:30.572936+0000","172.21.15.32:0/2092569962":"2024-04-07T14:21:22.517015+0000","172.21.15.32:6800/2780016783":"2024-04-07T14:21:22.517015+0000","172.21.15.32:6801/2780016783":"2024-04-07T14:21:22.517015+0000","172.21.15.32:0/69668297":"2024-04-07T14:21:22.517015+0000","172.21.15.86:6824/3246813202":"2024-04-07T14:25:48.051344+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T14:58:44.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f894406cc20 msgr2=0x7f894406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f894406cc20 0x7f894406f0d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f894800b0d0 tx=0x7f8948009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:44.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 msgr2=0x7f89580fc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 0x7f89580fc230 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f894000eb40 tx=0x7f894000b040 comp rx=0 tx=0).stop 2024-04-06T14:58:44.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 shutdown_connections 2024-04-06T14:58:44.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f894406cc20 0x7f894406f0d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8958064e40 0x7f89580fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 --2- 172.21.15.32:0/869880255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8958102d80 0x7f89580fc230 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 >> 172.21.15.32:0/869880255 conn(0x7f8958071c80 msgr2=0x7f89580faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:44.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 shutdown_connections 2024-04-06T14:58:44.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.518+0000 7f895de3b700 1 -- 172.21.15.32:0/869880255 wait complete. 2024-04-06T14:58:44.516 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T14:11:20.110423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-06T14:14:00.974689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-06T14:14:01.457260+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T14:58:44.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T14:58:44.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.659+0000 7fc43a9bb700 1 -- 172.21.15.32:0/966947674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434102580 msgr2=0x7fc4340fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.659+0000 7fc43a9bb700 1 --2- 172.21.15.32:0/966947674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434102580 0x7fc4340fbbf0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc428004940 tx=0x7fc428009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.659+0000 7fc43a9bb700 1 -- 172.21.15.32:0/966947674 shutdown_connections 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.659+0000 7fc43a9bb700 1 --2- 172.21.15.32:0/966947674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434102580 0x7fc4340fbbf0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.659+0000 7fc43a9bb700 1 --2- 172.21.15.32:0/966947674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc434101be0 0x7fc434101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.659+0000 7fc43a9bb700 1 -- 172.21.15.32:0/966947674 >> 172.21.15.32:0/966947674 conn(0x7fc434072ad0 msgr2=0x7fc434072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.659+0000 7fc43a9bb700 1 -- 172.21.15.32:0/966947674 shutdown_connections 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.659+0000 7fc43a9bb700 1 -- 172.21.15.32:0/966947674 wait complete. 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.659+0000 7fc43a9bb700 1 Processor -- start 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc43a9bb700 1 -- start start 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc43a9bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434101be0 0x7fc4341136d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc43a9bb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc434102580 0x7fc434113c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc43a9bb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc434114260 con 0x7fc434101be0 2024-04-06T14:58:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc43a9bb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43410d8d0 con 0x7fc434102580 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc433fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434101be0 0x7fc4341136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc433fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434101be0 0x7fc4341136d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51124/0 (socket says 172.21.15.32:51124) 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc433fff700 1 -- 172.21.15.32:0/3155234805 learned_addr learned my addr 172.21.15.32:0/3155234805 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc4337fe700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc434102580 0x7fc434113c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc433fff700 1 -- 172.21.15.32:0/3155234805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc434102580 msgr2=0x7fc434113c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc433fff700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc434102580 0x7fc434113c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc433fff700 1 -- 172.21.15.32:0/3155234805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc428005040 con 0x7fc434101be0 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc433fff700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434101be0 0x7fc4341136d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc424002a80 tx=0x7fc4240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc430ff9700 1 -- 172.21.15.32:0/3155234805 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc424007cc0 con 0x7fc434101be0 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43410dbb0 con 0x7fc434101be0 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc430ff9700 1 -- 172.21.15.32:0/3155234805 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc42400f040 con 0x7fc434101be0 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc430ff9700 1 -- 172.21.15.32:0/3155234805 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc424009640 con 0x7fc434101be0 2024-04-06T14:58:44.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.660+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43410e170 con 0x7fc434101be0 2024-04-06T14:58:44.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.661+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4340fd330 con 0x7fc434101be0 2024-04-06T14:58:44.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.662+0000 7fc430ff9700 1 -- 172.21.15.32:0/3155234805 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc4240097a0 con 0x7fc434101be0 2024-04-06T14:58:44.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.662+0000 7fc430ff9700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc41c06cc20 0x7fc41c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:44.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.663+0000 7fc430ff9700 1 -- 172.21.15.32:0/3155234805 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc42408b880 con 0x7fc434101be0 2024-04-06T14:58:44.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.663+0000 7fc4337fe700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc41c06cc20 0x7fc41c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:44.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.663+0000 7fc4337fe700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc41c06cc20 0x7fc41c06f0d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc428000c00 tx=0x7fc428009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:44.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.667+0000 7fc430ff9700 1 -- 172.21.15.32:0/3155234805 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc4240592d0 con 0x7fc434101be0 2024-04-06T14:58:44.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.874+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc434064970 con 0x7fc434101be0 2024-04-06T14:58:44.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.874+0000 7fc430ff9700 1 -- 172.21.15.32:0/3155234805 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc424058e60 con 0x7fc434101be0 2024-04-06T14:58:44.860 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.876+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc41c06cc20 msgr2=0x7fc41c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.876+0000 7fc43a9bb700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc41c06cc20 0x7fc41c06f0d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc428000c00 tx=0x7fc428009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.876+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434101be0 msgr2=0x7fc4341136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.876+0000 7fc43a9bb700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434101be0 0x7fc4341136d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc424002a80 tx=0x7fc4240046b0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.876+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 shutdown_connections 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.876+0000 7fc43a9bb700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc41c06cc20 0x7fc41c06f0d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.876+0000 7fc43a9bb700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434101be0 0x7fc4341136d0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.876+0000 7fc43a9bb700 1 --2- 172.21.15.32:0/3155234805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc434102580 0x7fc434113c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.876+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 >> 172.21.15.32:0/3155234805 conn(0x7fc434072ad0 msgr2=0x7fc434064dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.877+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 shutdown_connections 2024-04-06T14:58:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:44.877+0000 7fc43a9bb700 1 -- 172.21.15.32:0/3155234805 wait complete. 2024-04-06T14:58:44.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.014+0000 7fc33f647700 1 -- 172.21.15.32:0/1648958511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 msgr2=0x7fc3380fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.014+0000 7fc33f647700 1 --2- 172.21.15.32:0/1648958511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 0x7fc3380fbc20 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc334004940 tx=0x7fc334009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.014+0000 7fc33f647700 1 -- 172.21.15.32:0/1648958511 shutdown_connections 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.014+0000 7fc33f647700 1 --2- 172.21.15.32:0/1648958511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 0x7fc3380fbc20 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.014+0000 7fc33f647700 1 --2- 172.21.15.32:0/1648958511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc338101c10 0x7fc338101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.014+0000 7fc33f647700 1 -- 172.21.15.32:0/1648958511 >> 172.21.15.32:0/1648958511 conn(0x7fc3380f7720 msgr2=0x7fc3380f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.014+0000 7fc33f647700 1 -- 172.21.15.32:0/1648958511 shutdown_connections 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.014+0000 7fc33f647700 1 -- 172.21.15.32:0/1648958511 wait complete. 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33f647700 1 Processor -- start 2024-04-06T14:58:45.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33f647700 1 -- start start 2024-04-06T14:58:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33f647700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc338101c10 0x7fc3381136e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33f647700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 0x7fc338113c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33f647700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc338114270 con 0x7fc3381025b0 2024-04-06T14:58:45.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33f647700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33810d890 con 0x7fc338101c10 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33cbe2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 0x7fc338113c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33cbe2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 0x7fc338113c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51144/0 (socket says 172.21.15.32:51144) 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33cbe2700 1 -- 172.21.15.32:0/3556914553 learned_addr learned my addr 172.21.15.32:0/3556914553 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.015+0000 7fc33d3e3700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc338101c10 0x7fc3381136e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.016+0000 7fc33cbe2700 1 -- 172.21.15.32:0/3556914553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc338101c10 msgr2=0x7fc3381136e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.016+0000 7fc33cbe2700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc338101c10 0x7fc3381136e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.016+0000 7fc33cbe2700 1 -- 172.21.15.32:0/3556914553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334005040 con 0x7fc3381025b0 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.016+0000 7fc33cbe2700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 0x7fc338113c20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc334000c00 tx=0x7fc334009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.016+0000 7fc32dffb700 1 -- 172.21.15.32:0/3556914553 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc33400e070 con 0x7fc3381025b0 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.016+0000 7fc32dffb700 1 -- 172.21.15.32:0/3556914553 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3340093b0 con 0x7fc3381025b0 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.016+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33810db10 con 0x7fc3381025b0 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.016+0000 7fc32dffb700 1 -- 172.21.15.32:0/3556914553 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc33401ccb0 con 0x7fc3381025b0 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.016+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33810e070 con 0x7fc3381025b0 2024-04-06T14:58:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.017+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3380fd360 con 0x7fc3381025b0 2024-04-06T14:58:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.018+0000 7fc32dffb700 1 -- 172.21.15.32:0/3556914553 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc33401c490 con 0x7fc3381025b0 2024-04-06T14:58:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.018+0000 7fc32dffb700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc32406cc20 0x7fc32406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.018+0000 7fc32dffb700 1 -- 172.21.15.32:0/3556914553 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc33408d430 con 0x7fc3381025b0 2024-04-06T14:58:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.018+0000 7fc33d3e3700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc32406cc20 0x7fc32406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.019+0000 7fc33d3e3700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc32406cc20 0x7fc32406f0d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc328009b80 tx=0x7fc32800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:45.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.023+0000 7fc32dffb700 1 -- 172.21.15.32:0/3556914553 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc33405af00 con 0x7fc3381025b0 2024-04-06T14:58:45.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:44 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3493952032' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T14:58:45.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:44 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/869880255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:58:45.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:44 smithi086 ceph-mon[110975]: pgmap v1162: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:45.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.225+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fc3380640e0 con 0x7fc3381025b0 2024-04-06T14:58:45.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.226+0000 7fc32dffb700 1 -- 172.21.15.32:0/3556914553 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc33405aa90 con 0x7fc3381025b0 2024-04-06T14:58:45.212 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-06T14:58:45.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc32406cc20 msgr2=0x7fc32406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc32406cc20 0x7fc32406f0d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc328009b80 tx=0x7fc32800b040 comp rx=0 tx=0).stop 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 msgr2=0x7fc338113c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 0x7fc338113c20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc334000c00 tx=0x7fc334009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 shutdown_connections 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fc32406cc20 0x7fc32406f0d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc338101c10 0x7fc3381136e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 --2- 172.21.15.32:0/3556914553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3381025b0 0x7fc338113c20 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 >> 172.21.15.32:0/3556914553 conn(0x7fc3380f7720 msgr2=0x7fc3380f8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 shutdown_connections 2024-04-06T14:58:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.228+0000 7fc33f647700 1 -- 172.21.15.32:0/3556914553 wait complete. 2024-04-06T14:58:45.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:44 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3493952032' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T14:58:45.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:44 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/869880255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T14:58:45.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:44 smithi032 ceph-mon[147414]: pgmap v1162: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:45.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-06T14:58:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.365+0000 7fe969e2f700 1 -- 172.21.15.32:0/471210767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964102cd0 msgr2=0x7fe9641030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.365+0000 7fe969e2f700 1 --2- 172.21.15.32:0/471210767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964102cd0 0x7fe9641030a0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fe954004940 tx=0x7fe954009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.365+0000 7fe969e2f700 1 -- 172.21.15.32:0/471210767 shutdown_connections 2024-04-06T14:58:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.365+0000 7fe969e2f700 1 --2- 172.21.15.32:0/471210767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe964103670 0x7fe9640fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.365+0000 7fe969e2f700 1 --2- 172.21.15.32:0/471210767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964102cd0 0x7fe9641030a0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.365+0000 7fe969e2f700 1 -- 172.21.15.32:0/471210767 >> 172.21.15.32:0/471210767 conn(0x7fe9640f8840 msgr2=0x7fe9640fac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.365+0000 7fe969e2f700 1 -- 172.21.15.32:0/471210767 shutdown_connections 2024-04-06T14:58:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.365+0000 7fe969e2f700 1 -- 172.21.15.32:0/471210767 wait complete. 2024-04-06T14:58:45.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.366+0000 7fe969e2f700 1 Processor -- start 2024-04-06T14:58:45.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.366+0000 7fe969e2f700 1 -- start start 2024-04-06T14:58:45.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.366+0000 7fe969e2f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe964102cd0 0x7fe964113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:45.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.366+0000 7fe969e2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964103670 0x7fe964113ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:45.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.366+0000 7fe969e2f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9641142f0 con 0x7fe964103670 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.366+0000 7fe969e2f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96410d8f0 con 0x7fe964102cd0 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe962ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964103670 0x7fe964113ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe962ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964103670 0x7fe964113ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51166/0 (socket says 172.21.15.32:51166) 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe962ffd700 1 -- 172.21.15.32:0/2381871154 learned_addr learned my addr 172.21.15.32:0/2381871154 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe9637fe700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe964102cd0 0x7fe964113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe962ffd700 1 -- 172.21.15.32:0/2381871154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe964102cd0 msgr2=0x7fe964113760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe962ffd700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe964102cd0 0x7fe964113760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe962ffd700 1 -- 172.21.15.32:0/2381871154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe954005040 con 0x7fe964103670 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe962ffd700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964103670 0x7fe964113ca0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fe958002a50 tx=0x7fe958004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe94bfff700 1 -- 172.21.15.32:0/2381871154 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe958007cc0 con 0x7fe964103670 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96410dbd0 con 0x7fe964103670 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe94bfff700 1 -- 172.21.15.32:0/2381871154 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe95800f040 con 0x7fe964103670 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe94bfff700 1 -- 172.21.15.32:0/2381871154 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe95800d990 con 0x7fe964103670 2024-04-06T14:58:45.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.367+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96410e190 con 0x7fe964103670 2024-04-06T14:58:45.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.368+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9640fe420 con 0x7fe964103670 2024-04-06T14:58:45.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.369+0000 7fe94bfff700 1 -- 172.21.15.32:0/2381871154 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe958009710 con 0x7fe964103670 2024-04-06T14:58:45.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.369+0000 7fe94bfff700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe94c06cc20 0x7fe94c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:45.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.369+0000 7fe94bfff700 1 -- 172.21.15.32:0/2381871154 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe95808be60 con 0x7fe964103670 2024-04-06T14:58:45.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.369+0000 7fe9637fe700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe94c06cc20 0x7fe94c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:45.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.370+0000 7fe9637fe700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe94c06cc20 0x7fe94c06f0d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe954004f90 tx=0x7fe954009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:45.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.374+0000 7fe94bfff700 1 -- 172.21.15.32:0/2381871154 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe958059620 con 0x7fe964103670 2024-04-06T14:58:45.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.577+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fe964109cc0 con 0x7fe964103670 2024-04-06T14:58:45.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.577+0000 7fe94bfff700 1 -- 172.21.15.32:0/2381871154 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe958059440 con 0x7fe964103670 2024-04-06T14:58:45.564 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-06T14:58:45.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.579+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe94c06cc20 msgr2=0x7fe94c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.579+0000 7fe969e2f700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe94c06cc20 0x7fe94c06f0d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe954004f90 tx=0x7fe954009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.579+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964103670 msgr2=0x7fe964113ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.579+0000 7fe969e2f700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964103670 0x7fe964113ca0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fe958002a50 tx=0x7fe958004600 comp rx=0 tx=0).stop 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.579+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 shutdown_connections 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.579+0000 7fe969e2f700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7fe94c06cc20 0x7fe94c06f0d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.579+0000 7fe969e2f700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe964102cd0 0x7fe964113760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.579+0000 7fe969e2f700 1 --2- 172.21.15.32:0/2381871154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe964103670 0x7fe964113ca0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.579+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 >> 172.21.15.32:0/2381871154 conn(0x7fe9640f8840 msgr2=0x7fe9640f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.580+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 shutdown_connections 2024-04-06T14:58:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.580+0000 7fe969e2f700 1 -- 172.21.15.32:0/2381871154 wait complete. 2024-04-06T14:58:45.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T14:58:45.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.715+0000 7f699338a700 1 -- 172.21.15.32:0/1402972583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c101be0 msgr2=0x7f698c101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.715+0000 7f699338a700 1 --2- 172.21.15.32:0/1402972583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c101be0 0x7f698c101fb0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f697c004940 tx=0x7f697c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:45.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.715+0000 7f699338a700 1 -- 172.21.15.32:0/1402972583 shutdown_connections 2024-04-06T14:58:45.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.715+0000 7f699338a700 1 --2- 172.21.15.32:0/1402972583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f698c102580 0x7f698c0fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.715+0000 7f699338a700 1 --2- 172.21.15.32:0/1402972583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c101be0 0x7f698c101fb0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.715+0000 7f699338a700 1 -- 172.21.15.32:0/1402972583 >> 172.21.15.32:0/1402972583 conn(0x7f698c072ad0 msgr2=0x7f698c072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.715+0000 7f699338a700 1 -- 172.21.15.32:0/1402972583 shutdown_connections 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.715+0000 7f699338a700 1 -- 172.21.15.32:0/1402972583 wait complete. 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f699338a700 1 Processor -- start 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f699338a700 1 -- start start 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f699338a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f698c101be0 0x7f698c113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f699338a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c102580 0x7f698c113ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f699338a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c1142f0 con 0x7f698c102580 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f699338a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c10d8f0 con 0x7f698c101be0 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f6990925700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c102580 0x7f698c113ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f6990925700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c102580 0x7f698c113ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51194/0 (socket says 172.21.15.32:51194) 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f6990925700 1 -- 172.21.15.32:0/64507167 learned_addr learned my addr 172.21.15.32:0/64507167 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f6991126700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f698c101be0 0x7f698c113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f6990925700 1 -- 172.21.15.32:0/64507167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f698c101be0 msgr2=0x7f698c113760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f6990925700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f698c101be0 0x7f698c113760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f6990925700 1 -- 172.21.15.32:0/64507167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c005040 con 0x7f698c102580 2024-04-06T14:58:45.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f6991126700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f698c101be0 0x7f698c113760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T14:58:45.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.716+0000 7f6990925700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c102580 0x7f698c113ca0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f6988002a50 tx=0x7f6988004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:45.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.717+0000 7f6981ffb700 1 -- 172.21.15.32:0/64507167 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6988007cc0 con 0x7f698c102580 2024-04-06T14:58:45.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.717+0000 7f6981ffb700 1 -- 172.21.15.32:0/64507167 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f698800f040 con 0x7f698c102580 2024-04-06T14:58:45.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.717+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698c10dbd0 con 0x7f698c102580 2024-04-06T14:58:45.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.717+0000 7f6981ffb700 1 -- 172.21.15.32:0/64507167 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f698800d9a0 con 0x7f698c102580 2024-04-06T14:58:45.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.717+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698c10e190 con 0x7f698c102580 2024-04-06T14:58:45.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.718+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698c0fd330 con 0x7f698c102580 2024-04-06T14:58:45.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.718+0000 7f6981ffb700 1 -- 172.21.15.32:0/64507167 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f698800dc10 con 0x7f698c102580 2024-04-06T14:58:45.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.719+0000 7f6981ffb700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f697806cc20 0x7f697806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T14:58:45.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.719+0000 7f6991126700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f697806cc20 0x7f697806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T14:58:45.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.719+0000 7f6981ffb700 1 -- 172.21.15.32:0/64507167 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f698808bdc0 con 0x7f698c102580 2024-04-06T14:58:45.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.719+0000 7f6991126700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f697806cc20 0x7f697806f0d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f697c004560 tx=0x7f697c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T14:58:45.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.723+0000 7f6981ffb700 1 -- 172.21.15.32:0/64507167 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6988059890 con 0x7f698c102580 2024-04-06T14:58:45.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.947+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f698c064970 con 0x7f698c102580 2024-04-06T14:58:45.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.947+0000 7f6981ffb700 1 -- 172.21.15.32:0/64507167 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v85) v1 ==== 81+0+2308 (secure 0 0 0) 0x7f6988059420 con 0x7f698c102580 2024-04-06T14:58:45.933 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6828/1704327767 2024-04-07T14:25:48.051314+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:6825/3246813202 2024-04-07T14:25:48.051344+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/818700653 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:0/2189072732 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3548398153 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/935909372 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/818700653 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/3947155315 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6829/1704327767 2024-04-07T14:25:48.051314+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6827/4235185528 2024-04-07T14:14:16.952740+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/1833723736 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3250004174 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/711512148 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/68274703 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3330131734 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1187866165 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:0/2427526405 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:0/2137415724 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2605964794 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3796020247 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/1833723736 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2227455350 2024-04-07T14:07:51.523157+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2524933837 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2574238932 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/3067649142 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1664607864 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/3067649142 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6826/4235185528 2024-04-07T14:14:16.952740+0000 2024-04-06T14:58:45.935 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1877817963 2024-04-07T14:07:17.560285+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:6828/1279185277 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:6829/1279185277 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/3947155315 2024-04-07T14:08:57.599030+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:0/2712193968 2024-04-07T14:18:05.813583+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3226572738 2024-04-07T14:16:30.572936+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2092569962 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/2780016783 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/2780016783 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/69668297 2024-04-07T14:21:22.517015+0000 2024-04-06T14:58:45.936 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.86:6824/3246813202 2024-04-07T14:25:48.051344+0000 2024-04-06T14:58:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f697806cc20 msgr2=0x7f697806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f697806cc20 0x7f697806f0d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f697c004560 tx=0x7f697c009e30 comp rx=0 tx=0).stop 2024-04-06T14:58:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c102580 msgr2=0x7f698c113ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T14:58:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c102580 0x7f698c113ca0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f6988002a50 tx=0x7f6988004600 comp rx=0 tx=0).stop 2024-04-06T14:58:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 shutdown_connections 2024-04-06T14:58:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:6800/3171072994,v1:172.21.15.32:6801/3171072994] conn(0x7f697806cc20 0x7f697806f0d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f698c101be0 0x7f698c113760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 --2- 172.21.15.32:0/64507167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f698c102580 0x7f698c113ca0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T14:58:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 >> 172.21.15.32:0/64507167 conn(0x7f698c072ad0 msgr2=0x7f698c0fa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T14:58:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 shutdown_connections 2024-04-06T14:58:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-06T14:58:45.950+0000 7f699338a700 1 -- 172.21.15.32:0/64507167 wait complete. 2024-04-06T14:58:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:listed 39 entries 2024-04-06T14:58:45.947 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:45 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3155234805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:58:45.947 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:45 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/3556914553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T14:58:45.947 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:45 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/2381871154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T14:58:45.948 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2024-04-06T14:58:45.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T14:58:45.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-06T14:58:45.991 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T14:58:46.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:45 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3155234805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T14:58:46.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:45 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/3556914553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T14:58:46.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:45 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/2381871154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T14:58:47.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:46 smithi086 ceph-mon[110975]: from='client.? 172.21.15.32:0/64507167' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T14:58:47.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:46 smithi086 ceph-mon[110975]: pgmap v1163: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:47.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:46 smithi032 ceph-mon[147414]: from='client.? 172.21.15.32:0/64507167' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T14:58:47.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:46 smithi032 ceph-mon[147414]: pgmap v1163: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:49.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:48 smithi086 ceph-mon[110975]: pgmap v1164: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:49.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:48 smithi032 ceph-mon[147414]: pgmap v1164: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:51.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:50 smithi086 ceph-mon[110975]: pgmap v1165: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:51.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:50 smithi032 ceph-mon[147414]: pgmap v1165: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:52.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:51 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:58:52.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:51 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:58:53.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:58:53.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:58:53.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:58:53.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:58:53.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:52 smithi086 ceph-mon[110975]: pgmap v1166: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:53.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:58:53.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:58:53.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:58:53.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:58:53.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:52 smithi032 ceph-mon[147414]: pgmap v1166: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:55.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:55 smithi032 ceph-mon[147414]: pgmap v1167: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:55.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:55 smithi086 ceph-mon[110975]: pgmap v1167: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:57.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:56 smithi086 ceph-mon[110975]: pgmap v1168: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:57.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:56 smithi032 ceph-mon[147414]: pgmap v1168: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:59.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:58:58 smithi086 ceph-mon[110975]: pgmap v1169: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:58:59.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:58:58 smithi032 ceph-mon[147414]: pgmap v1169: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:01.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:00 smithi086 ceph-mon[110975]: pgmap v1170: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:01.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:00 smithi032 ceph-mon[147414]: pgmap v1170: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:03.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:02 smithi086 ceph-mon[110975]: pgmap v1171: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:03.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:02 smithi032 ceph-mon[147414]: pgmap v1171: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:05.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:05 smithi032 ceph-mon[147414]: pgmap v1172: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:05.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:05 smithi086 ceph-mon[110975]: pgmap v1172: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:07.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:06 smithi086 ceph-mon[110975]: pgmap v1173: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:07.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:06 smithi032 ceph-mon[147414]: pgmap v1173: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:08.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:59:08.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:59:09.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:08 smithi032 ceph-mon[147414]: pgmap v1174: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:09.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:08 smithi086 ceph-mon[110975]: pgmap v1174: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:11.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:10 smithi086 ceph-mon[110975]: pgmap v1175: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:11.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:10 smithi032 ceph-mon[147414]: pgmap v1175: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:13.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:12 smithi086 ceph-mon[110975]: pgmap v1176: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:13.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:12 smithi032 ceph-mon[147414]: pgmap v1176: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:15.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:15 smithi032 ceph-mon[147414]: pgmap v1177: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:15.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:15 smithi086 ceph-mon[110975]: pgmap v1177: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:17.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:16 smithi086 ceph-mon[110975]: pgmap v1178: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:17.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:16 smithi032 ceph-mon[147414]: pgmap v1178: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:19.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:18 smithi086 ceph-mon[110975]: pgmap v1179: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:19.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:18 smithi032 ceph-mon[147414]: pgmap v1179: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:21.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:20 smithi086 ceph-mon[110975]: pgmap v1180: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:21.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:20 smithi032 ceph-mon[147414]: pgmap v1180: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:22.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:59:23.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:59:23.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:23 smithi032 ceph-mon[147414]: pgmap v1181: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:24.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:23 smithi086 ceph-mon[110975]: pgmap v1181: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:25.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:25 smithi032 ceph-mon[147414]: pgmap v1182: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:25.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:25 smithi086 ceph-mon[110975]: pgmap v1182: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:27.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:26 smithi086 ceph-mon[110975]: pgmap v1183: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:27.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:26 smithi032 ceph-mon[147414]: pgmap v1183: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:29.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:28 smithi086 ceph-mon[110975]: pgmap v1184: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:29.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:28 smithi032 ceph-mon[147414]: pgmap v1184: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:31.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:30 smithi086 ceph-mon[110975]: pgmap v1185: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:31.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:30 smithi032 ceph-mon[147414]: pgmap v1185: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:33.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:32 smithi086 ceph-mon[110975]: pgmap v1186: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:33.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:32 smithi032 ceph-mon[147414]: pgmap v1186: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:35.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:35 smithi032 ceph-mon[147414]: pgmap v1187: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:35.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:35 smithi086 ceph-mon[110975]: pgmap v1187: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:37.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:36 smithi086 ceph-mon[110975]: pgmap v1188: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:37.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:36 smithi032 ceph-mon[147414]: pgmap v1188: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:38.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:59:38.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:59:39.216 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:38 smithi032 ceph-mon[147414]: pgmap v1189: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:38 smithi086 ceph-mon[110975]: pgmap v1189: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:41.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:40 smithi086 ceph-mon[110975]: pgmap v1190: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:41.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:40 smithi032 ceph-mon[147414]: pgmap v1190: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:43.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:42 smithi086 ceph-mon[110975]: pgmap v1191: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:43.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:42 smithi032 ceph-mon[147414]: pgmap v1191: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:45.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:45 smithi032 ceph-mon[147414]: pgmap v1192: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:45.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:45 smithi086 ceph-mon[110975]: pgmap v1192: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:47.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:46 smithi086 ceph-mon[110975]: pgmap v1193: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:47.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:46 smithi032 ceph-mon[147414]: pgmap v1193: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:49.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:48 smithi086 ceph-mon[110975]: pgmap v1194: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:49.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:48 smithi032 ceph-mon[147414]: pgmap v1194: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:51.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:50 smithi032 ceph-mon[147414]: pgmap v1195: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:51.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:50 smithi086 ceph-mon[110975]: pgmap v1195: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:52.613 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:59:52.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T14:59:53.613 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:59:53.613 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:59:53.613 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:59:53.613 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:59:53.613 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:53 smithi086 ceph-mon[110975]: pgmap v1196: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:53.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T14:59:53.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T14:59:53.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T14:59:53.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T14:59:53.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:53 smithi032 ceph-mon[147414]: pgmap v1196: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:55.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:55 smithi032 ceph-mon[147414]: pgmap v1197: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:55.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:55 smithi086 ceph-mon[110975]: pgmap v1197: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:57.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:56 smithi032 ceph-mon[147414]: pgmap v1198: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:57.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:56 smithi086 ceph-mon[110975]: pgmap v1198: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:59.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 14:59:58 smithi032 ceph-mon[147414]: pgmap v1199: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T14:59:59.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 14:59:58 smithi086 ceph-mon[110975]: pgmap v1199: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:00.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:00 smithi032 ceph-mon[147414]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T15:00:00.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:00 smithi032 ceph-mon[147414]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T15:00:00.716 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:00 smithi032 ceph-mon[147414]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T15:00:00.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:00 smithi086 ceph-mon[110975]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T15:00:00.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:00 smithi086 ceph-mon[110975]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T15:00:00.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:00 smithi086 ceph-mon[110975]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T15:00:01.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:01 smithi032 ceph-mon[147414]: pgmap v1200: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:01.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:01 smithi086 ceph-mon[110975]: pgmap v1200: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:03.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:02 smithi032 ceph-mon[147414]: pgmap v1201: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:03.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:02 smithi086 ceph-mon[110975]: pgmap v1201: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:05.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:05 smithi032 ceph-mon[147414]: pgmap v1202: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:05.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:05 smithi086 ceph-mon[110975]: pgmap v1202: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:07.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:06 smithi032 ceph-mon[147414]: pgmap v1203: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:07.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:06 smithi086 ceph-mon[110975]: pgmap v1203: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:08.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:00:08.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:00:09.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:08 smithi032 ceph-mon[147414]: pgmap v1204: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:09.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:08 smithi086 ceph-mon[110975]: pgmap v1204: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:11.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:10 smithi032 ceph-mon[147414]: pgmap v1205: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:11.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:10 smithi086 ceph-mon[110975]: pgmap v1205: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:13.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:12 smithi032 ceph-mon[147414]: pgmap v1206: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:13.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:12 smithi086 ceph-mon[110975]: pgmap v1206: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:15.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:15 smithi032 ceph-mon[147414]: pgmap v1207: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:15.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:15 smithi086 ceph-mon[110975]: pgmap v1207: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:17.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:16 smithi032 ceph-mon[147414]: pgmap v1208: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:17.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:16 smithi086 ceph-mon[110975]: pgmap v1208: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:19.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:18 smithi032 ceph-mon[147414]: pgmap v1209: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:19.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:18 smithi086 ceph-mon[110975]: pgmap v1209: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:21.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:20 smithi032 ceph-mon[147414]: pgmap v1210: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:21.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:20 smithi086 ceph-mon[110975]: pgmap v1210: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:22.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:00:23.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:00:23.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:23 smithi032 ceph-mon[147414]: pgmap v1211: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:24.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:23 smithi086 ceph-mon[110975]: pgmap v1211: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:25.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:25 smithi032 ceph-mon[147414]: pgmap v1212: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:25.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:25 smithi086 ceph-mon[110975]: pgmap v1212: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:27.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:26 smithi032 ceph-mon[147414]: pgmap v1213: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:27.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:26 smithi086 ceph-mon[110975]: pgmap v1213: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:29.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:28 smithi032 ceph-mon[147414]: pgmap v1214: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:29.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:28 smithi086 ceph-mon[110975]: pgmap v1214: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:31.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:30 smithi032 ceph-mon[147414]: pgmap v1215: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:31.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:30 smithi086 ceph-mon[110975]: pgmap v1215: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:33.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:32 smithi032 ceph-mon[147414]: pgmap v1216: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:33.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:32 smithi086 ceph-mon[110975]: pgmap v1216: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:35.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:35 smithi032 ceph-mon[147414]: pgmap v1217: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:35.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:35 smithi086 ceph-mon[110975]: pgmap v1217: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:37.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:36 smithi032 ceph-mon[147414]: pgmap v1218: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:37.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:36 smithi086 ceph-mon[110975]: pgmap v1218: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:38.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:00:38.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:00:39.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:38 smithi032 ceph-mon[147414]: pgmap v1219: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:38 smithi086 ceph-mon[110975]: pgmap v1219: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:41.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:40 smithi032 ceph-mon[147414]: pgmap v1220: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:41.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:40 smithi086 ceph-mon[110975]: pgmap v1220: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:43.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:42 smithi032 ceph-mon[147414]: pgmap v1221: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:43.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:42 smithi086 ceph-mon[110975]: pgmap v1221: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:45.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:45 smithi086 ceph-mon[110975]: pgmap v1222: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:45.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:45 smithi032 ceph-mon[147414]: pgmap v1222: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:47.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:46 smithi032 ceph-mon[147414]: pgmap v1223: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:47.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:46 smithi086 ceph-mon[110975]: pgmap v1223: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:49.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:48 smithi032 ceph-mon[147414]: pgmap v1224: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:49.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:48 smithi086 ceph-mon[110975]: pgmap v1224: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:51.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:50 smithi032 ceph-mon[147414]: pgmap v1225: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:51.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:50 smithi086 ceph-mon[110975]: pgmap v1225: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:52.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:00:53.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:00:53.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:53 smithi032 ceph-mon[147414]: pgmap v1226: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:53.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T15:00:53.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T15:00:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T15:00:53.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T15:00:54.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:53 smithi086 ceph-mon[110975]: pgmap v1226: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:54.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T15:00:54.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T15:00:54.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T15:00:54.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T15:00:55.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:55 smithi086 ceph-mon[110975]: pgmap v1227: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:55.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:55 smithi032 ceph-mon[147414]: pgmap v1227: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:57.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:56 smithi032 ceph-mon[147414]: pgmap v1228: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:57.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:56 smithi086 ceph-mon[110975]: pgmap v1228: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:59.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:00:58 smithi032 ceph-mon[147414]: pgmap v1229: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:00:59.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:00:58 smithi086 ceph-mon[110975]: pgmap v1229: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:01.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:00 smithi032 ceph-mon[147414]: pgmap v1230: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:01.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:00 smithi086 ceph-mon[110975]: pgmap v1230: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:03.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:02 smithi032 ceph-mon[147414]: pgmap v1231: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:03.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:02 smithi086 ceph-mon[110975]: pgmap v1231: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:05.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:05 smithi086 ceph-mon[110975]: pgmap v1232: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:05.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:05 smithi032 ceph-mon[147414]: pgmap v1232: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:07.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:06 smithi032 ceph-mon[147414]: pgmap v1233: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:07.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:06 smithi086 ceph-mon[110975]: pgmap v1233: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:08.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:01:08.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:01:09.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:08 smithi032 ceph-mon[147414]: pgmap v1234: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:09.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:08 smithi086 ceph-mon[110975]: pgmap v1234: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:11.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:10 smithi032 ceph-mon[147414]: pgmap v1235: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:11.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:10 smithi086 ceph-mon[110975]: pgmap v1235: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:13.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:12 smithi032 ceph-mon[147414]: pgmap v1236: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:13.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:12 smithi086 ceph-mon[110975]: pgmap v1236: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:15.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:15 smithi086 ceph-mon[110975]: pgmap v1237: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:15.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:15 smithi032 ceph-mon[147414]: pgmap v1237: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:17.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:16 smithi032 ceph-mon[147414]: pgmap v1238: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:17.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:16 smithi086 ceph-mon[110975]: pgmap v1238: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:19.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:18 smithi032 ceph-mon[147414]: pgmap v1239: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:19.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:18 smithi086 ceph-mon[110975]: pgmap v1239: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:21.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:20 smithi032 ceph-mon[147414]: pgmap v1240: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:21.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:20 smithi086 ceph-mon[110975]: pgmap v1240: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:22.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:01:23.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:01:23.975 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:23 smithi032 ceph-mon[147414]: pgmap v1241: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:24.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:23 smithi086 ceph-mon[110975]: pgmap v1241: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:25.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:25 smithi086 ceph-mon[110975]: pgmap v1242: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:25.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:25 smithi032 ceph-mon[147414]: pgmap v1242: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:27.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:26 smithi032 ceph-mon[147414]: pgmap v1243: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:27.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:26 smithi086 ceph-mon[110975]: pgmap v1243: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:29.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:28 smithi032 ceph-mon[147414]: pgmap v1244: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:29.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:28 smithi086 ceph-mon[110975]: pgmap v1244: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:31.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:30 smithi032 ceph-mon[147414]: pgmap v1245: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:31.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:30 smithi086 ceph-mon[110975]: pgmap v1245: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:33.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:32 smithi032 ceph-mon[147414]: pgmap v1246: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:33.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:32 smithi086 ceph-mon[110975]: pgmap v1246: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:35.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:35 smithi086 ceph-mon[110975]: pgmap v1247: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:35.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:35 smithi032 ceph-mon[147414]: pgmap v1247: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:37.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:36 smithi032 ceph-mon[147414]: pgmap v1248: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:37.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:36 smithi086 ceph-mon[110975]: pgmap v1248: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:38.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:01:38.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:01:39.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:38 smithi032 ceph-mon[147414]: pgmap v1249: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:39.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:38 smithi086 ceph-mon[110975]: pgmap v1249: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:41.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:40 smithi032 ceph-mon[147414]: pgmap v1250: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:41.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:40 smithi086 ceph-mon[110975]: pgmap v1250: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:43.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:42 smithi032 ceph-mon[147414]: pgmap v1251: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:43.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:42 smithi086 ceph-mon[110975]: pgmap v1251: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:45.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:45 smithi086 ceph-mon[110975]: pgmap v1252: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:45.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:45 smithi032 ceph-mon[147414]: pgmap v1252: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:47.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:46 smithi032 ceph-mon[147414]: pgmap v1253: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:47.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:46 smithi086 ceph-mon[110975]: pgmap v1253: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:49.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:48 smithi032 ceph-mon[147414]: pgmap v1254: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:49.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:48 smithi086 ceph-mon[110975]: pgmap v1254: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:51.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:50 smithi032 ceph-mon[147414]: pgmap v1255: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:51.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:50 smithi086 ceph-mon[110975]: pgmap v1255: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:52.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:52 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:01:53.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:52 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:01:53.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:53 smithi032 ceph-mon[147414]: pgmap v1256: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:53.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T15:01:54.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:53 smithi086 ceph-mon[110975]: pgmap v1256: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:54.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T15:01:54.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T15:01:54.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T15:01:54.966 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T15:01:55.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T15:01:55.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T15:01:55.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T15:01:55.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:55 smithi032 ceph-mon[147414]: pgmap v1257: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:56.114 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:55 smithi086 ceph-mon[110975]: pgmap v1257: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:57.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:56 smithi032 ceph-mon[147414]: pgmap v1258: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:57.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:56 smithi086 ceph-mon[110975]: pgmap v1258: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:59.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:01:58 smithi032 ceph-mon[147414]: pgmap v1259: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:01:59.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:01:58 smithi086 ceph-mon[110975]: pgmap v1259: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:01.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:00 smithi032 ceph-mon[147414]: pgmap v1260: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:01.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:00 smithi086 ceph-mon[110975]: pgmap v1260: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:03.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:02 smithi032 ceph-mon[147414]: pgmap v1261: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:03.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:02 smithi086 ceph-mon[110975]: pgmap v1261: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:05.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:05 smithi086 ceph-mon[110975]: pgmap v1262: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:05.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:05 smithi032 ceph-mon[147414]: pgmap v1262: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:07.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:06 smithi032 ceph-mon[147414]: pgmap v1263: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:07.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:06 smithi086 ceph-mon[110975]: pgmap v1263: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:08.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:07 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:02:08.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:07 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:02:09.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:08 smithi032 ceph-mon[147414]: pgmap v1264: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:09.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:08 smithi086 ceph-mon[110975]: pgmap v1264: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:11.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:10 smithi032 ceph-mon[147414]: pgmap v1265: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:11.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:10 smithi086 ceph-mon[110975]: pgmap v1265: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:13.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:12 smithi032 ceph-mon[147414]: pgmap v1266: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:13.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:12 smithi086 ceph-mon[110975]: pgmap v1266: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:15.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:15 smithi086 ceph-mon[110975]: pgmap v1267: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:15.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:15 smithi032 ceph-mon[147414]: pgmap v1267: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:17.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:16 smithi032 ceph-mon[147414]: pgmap v1268: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:17.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:16 smithi086 ceph-mon[110975]: pgmap v1268: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:19.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:18 smithi032 ceph-mon[147414]: pgmap v1269: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:19.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:18 smithi086 ceph-mon[110975]: pgmap v1269: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:21.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:20 smithi032 ceph-mon[147414]: pgmap v1270: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:21.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:20 smithi086 ceph-mon[110975]: pgmap v1270: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:22.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:22 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:02:23.113 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:22 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:02:23.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:23 smithi032 ceph-mon[147414]: pgmap v1271: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:24.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:23 smithi086 ceph-mon[110975]: pgmap v1271: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:25.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:25 smithi086 ceph-mon[110975]: pgmap v1272: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:25.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:25 smithi032 ceph-mon[147414]: pgmap v1272: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:27.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:26 smithi032 ceph-mon[147414]: pgmap v1273: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:27.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:26 smithi086 ceph-mon[110975]: pgmap v1273: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:29.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:28 smithi032 ceph-mon[147414]: pgmap v1274: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:29.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:28 smithi086 ceph-mon[110975]: pgmap v1274: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:31.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:30 smithi032 ceph-mon[147414]: pgmap v1275: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:31.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:30 smithi086 ceph-mon[110975]: pgmap v1275: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:33.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:32 smithi032 ceph-mon[147414]: pgmap v1276: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:33.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:32 smithi086 ceph-mon[110975]: pgmap v1276: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:35.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:35 smithi086 ceph-mon[110975]: pgmap v1277: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:35.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:35 smithi032 ceph-mon[147414]: pgmap v1277: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:37.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:36 smithi032 ceph-mon[147414]: pgmap v1278: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:37.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:36 smithi086 ceph-mon[110975]: pgmap v1278: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:38.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:02:38.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:02:39.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:38 smithi032 ceph-mon[147414]: pgmap v1279: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:39.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:38 smithi086 ceph-mon[110975]: pgmap v1279: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:41.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:40 smithi032 ceph-mon[147414]: pgmap v1280: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:41.363 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:40 smithi086 ceph-mon[110975]: pgmap v1280: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:43.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:42 smithi032 ceph-mon[147414]: pgmap v1281: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:43.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:42 smithi086 ceph-mon[110975]: pgmap v1281: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:45.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:45 smithi086 ceph-mon[110975]: pgmap v1282: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:45.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:45 smithi032 ceph-mon[147414]: pgmap v1282: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:48.215 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:47 smithi032 ceph-mon[147414]: pgmap v1283: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:48.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:47 smithi086 ceph-mon[110975]: pgmap v1283: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:50.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:50 smithi086 ceph-mon[110975]: pgmap v1284: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:50.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:50 smithi032 ceph-mon[147414]: pgmap v1284: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:52.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:52 smithi086 ceph-mon[110975]: pgmap v1285: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:52.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:52 smithi032 ceph-mon[147414]: pgmap v1285: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:53.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:53 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:02:53.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:53 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:02:54.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:54 smithi086 ceph-mon[110975]: pgmap v1286: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:54.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:54 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T15:02:54.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:54 smithi032 ceph-mon[147414]: pgmap v1286: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:54.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:54 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T15:02:55.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T15:02:55.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T15:02:55.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:55 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T15:02:55.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T15:02:55.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T15:02:55.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:55 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' 2024-04-06T15:02:56.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:56 smithi086 ceph-mon[110975]: pgmap v1287: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:56.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:56 smithi032 ceph-mon[147414]: pgmap v1287: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:58.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:02:58 smithi086 ceph-mon[110975]: pgmap v1288: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:02:58.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:02:58 smithi032 ceph-mon[147414]: pgmap v1288: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:00.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:00 smithi086 ceph-mon[110975]: pgmap v1289: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:00.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:00 smithi032 ceph-mon[147414]: pgmap v1289: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:02.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:02 smithi086 ceph-mon[110975]: pgmap v1290: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:02.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:02 smithi032 ceph-mon[147414]: pgmap v1290: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:04.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:04 smithi086 ceph-mon[110975]: pgmap v1291: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:04.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:04 smithi032 ceph-mon[147414]: pgmap v1291: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:06.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:06 smithi086 ceph-mon[110975]: pgmap v1292: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:06.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:06 smithi032 ceph-mon[147414]: pgmap v1292: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:08.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:08 smithi086 ceph-mon[110975]: pgmap v1293: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:08.863 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:08 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:03:08.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:08 smithi032 ceph-mon[147414]: pgmap v1293: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:08.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:08 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:03:10.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:10 smithi086 ceph-mon[110975]: pgmap v1294: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:10.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:10 smithi032 ceph-mon[147414]: pgmap v1294: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:12.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:12 smithi086 ceph-mon[110975]: pgmap v1295: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:12.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:12 smithi032 ceph-mon[147414]: pgmap v1295: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:14.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:14 smithi086 ceph-mon[110975]: pgmap v1296: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:14.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:14 smithi032 ceph-mon[147414]: pgmap v1296: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:16.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:16 smithi032 ceph-mon[147414]: pgmap v1297: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:17.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:16 smithi086 ceph-mon[110975]: pgmap v1297: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:18.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:18 smithi032 ceph-mon[147414]: pgmap v1298: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:19.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:18 smithi086 ceph-mon[110975]: pgmap v1298: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:20.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:20 smithi032 ceph-mon[147414]: pgmap v1299: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:21.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:20 smithi086 ceph-mon[110975]: pgmap v1299: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:22.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:22 smithi032 ceph-mon[147414]: pgmap v1300: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:23.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:22 smithi086 ceph-mon[110975]: pgmap v1300: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:23.715 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:23 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:03:24.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:23 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:03:24.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:24 smithi032 ceph-mon[147414]: pgmap v1301: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:25.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:24 smithi086 ceph-mon[110975]: pgmap v1301: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:26.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:26 smithi032 ceph-mon[147414]: pgmap v1302: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:27.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:26 smithi086 ceph-mon[110975]: pgmap v1302: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:28.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:28 smithi032 ceph-mon[147414]: pgmap v1303: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:29.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:28 smithi086 ceph-mon[110975]: pgmap v1303: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:30.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:30 smithi032 ceph-mon[147414]: pgmap v1304: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:31.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:30 smithi086 ceph-mon[110975]: pgmap v1304: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:32.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:32 smithi032 ceph-mon[147414]: pgmap v1305: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:33.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:32 smithi086 ceph-mon[110975]: pgmap v1305: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:34.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:34 smithi032 ceph-mon[147414]: pgmap v1306: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:35.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:34 smithi086 ceph-mon[110975]: pgmap v1306: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:36.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:36 smithi032 ceph-mon[147414]: pgmap v1307: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:37.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:36 smithi086 ceph-mon[110975]: pgmap v1307: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:37.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:37 smithi032 ceph-mon[147414]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:03:38.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:37 smithi086 ceph-mon[110975]: from='mgr.34104 172.21.15.32:0/235257643' entity='mgr.smithi032.peohxu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T15:03:38.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:38 smithi032 ceph-mon[147414]: pgmap v1308: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:39.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:38 smithi086 ceph-mon[110975]: pgmap v1308: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:40.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:40 smithi032 ceph-mon[147414]: pgmap v1309: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:41.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:40 smithi086 ceph-mon[110975]: pgmap v1309: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:42.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:42 smithi032 ceph-mon[147414]: pgmap v1310: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:43.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:42 smithi086 ceph-mon[110975]: pgmap v1310: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:44.964 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:44 smithi032 ceph-mon[147414]: pgmap v1311: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:45.062 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-06T15:03:45.063 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-06T15:03:45.064 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-06T15:03:45.076 INFO:tasks.cephadm:Teardown begin 2024-04-06T15:03:45.076 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephadm.py", line 1750, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-06T15:03:45.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T15:03:45.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:44 smithi086 ceph-mon[110975]: pgmap v1311: 65 pgs: 65 active+clean; 351 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-06T15:03:45.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T15:03:45.153 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-06T15:03:45.153 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-06T15:03:45.178 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-06T15:03:45.213 INFO:tasks.cephadm:Stopping all daemons... 2024-04-06T15:03:45.214 INFO:tasks.cephadm.mon.smithi032:Stopping mon.smithi032... 2024-04-06T15:03:45.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi032 2024-04-06T15:03:45.466 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:45 smithi032 systemd[1]: Stopping Ceph mon.smithi032 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T15:03:45.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:45 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T15:03:45.520+0000 7fa226c8e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi032 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T15:03:45.965 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:45 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032[147391]: 2024-04-06T15:03:45.520+0000 7fa226c8e700 -1 mon.smithi032@0(leader) e3 *** Got Signal Terminated *** 2024-04-06T15:03:46.713 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 06 15:03:46 smithi032 bash[214564]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi032 2024-04-06T15:03:46.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi032.service' 2024-04-06T15:03:46.771 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T15:03:46.772 INFO:tasks.cephadm.mon.smithi032:Stopped mon.smithi032 2024-04-06T15:03:46.772 INFO:tasks.cephadm.mon.smithi086:Stopping mon.smithi086... 2024-04-06T15:03:46.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi086 2024-04-06T15:03:47.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:46 smithi086 systemd[1]: Stopping Ceph mon.smithi086 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T15:03:47.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:47 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi086[110953]: 2024-04-06T15:03:47.069+0000 7f120fef5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi086 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T15:03:47.362 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:47 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi086[110953]: 2024-04-06T15:03:47.069+0000 7f120fef5700 -1 mon.smithi086@1(peon) e3 *** Got Signal Terminated *** 2024-04-06T15:03:47.862 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Apr 06 15:03:47 smithi086 bash[158181]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-mon-smithi086 2024-04-06T15:03:47.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@mon.smithi086.service' 2024-04-06T15:03:48.006 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T15:03:48.006 INFO:tasks.cephadm.mon.smithi086:Stopped mon.smithi086 2024-04-06T15:03:48.006 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-06T15:03:48.007 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.0 2024-04-06T15:03:48.399 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 15:03:48 smithi032 systemd[1]: Stopping Ceph osd.0 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T15:03:48.714 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 15:03:48 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[161326]: 2024-04-06T15:03:48.413+0000 7fce22576700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T15:03:48.715 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 15:03:48 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[161326]: 2024-04-06T15:03:48.413+0000 7fce22576700 -1 osd.0 85 *** Got signal Terminated *** 2024-04-06T15:03:48.715 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 15:03:48 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0[161326]: 2024-04-06T15:03:48.413+0000 7fce22576700 -1 osd.0 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T15:03:54.464 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 06 15:03:54 smithi032 bash[214731]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-0 2024-04-06T15:03:55.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.0.service' 2024-04-06T15:03:55.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T15:03:55.893 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-06T15:03:55.893 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-06T15:03:55.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.1 2024-04-06T15:03:56.215 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 15:03:55 smithi032 systemd[1]: Stopping Ceph osd.1 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T15:03:56.714 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 15:03:56 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[168700]: 2024-04-06T15:03:56.244+0000 7fa4385f4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T15:03:56.715 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 15:03:56 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[168700]: 2024-04-06T15:03:56.244+0000 7fa4385f4700 -1 osd.1 85 *** Got signal Terminated *** 2024-04-06T15:03:56.715 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 15:03:56 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1[168700]: 2024-04-06T15:03:56.244+0000 7fa4385f4700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T15:04:02.214 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 06 15:04:01 smithi032 bash[214999]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-1 2024-04-06T15:04:03.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.1.service' 2024-04-06T15:04:03.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T15:04:03.752 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-06T15:04:03.753 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-06T15:04:03.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.2 2024-04-06T15:04:03.964 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 15:04:03 smithi032 systemd[1]: Stopping Ceph osd.2 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T15:04:04.464 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 15:04:04 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[176055]: 2024-04-06T15:04:04.085+0000 7fab68783700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T15:04:04.465 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 15:04:04 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[176055]: 2024-04-06T15:04:04.085+0000 7fab68783700 -1 osd.2 85 *** Got signal Terminated *** 2024-04-06T15:04:04.465 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 15:04:04 smithi032 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2[176055]: 2024-04-06T15:04:04.085+0000 7fab68783700 -1 osd.2 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T15:04:09.965 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 06 15:04:09 smithi032 bash[215265]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-2 2024-04-06T15:04:11.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.2.service' 2024-04-06T15:04:11.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T15:04:11.555 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-06T15:04:11.555 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-06T15:04:11.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.3 2024-04-06T15:04:11.849 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 15:04:11 smithi086 systemd[1]: Stopping Ceph osd.3 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T15:04:12.112 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 15:04:11 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[123175]: 2024-04-06T15:04:11.864+0000 7fc126b49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T15:04:12.112 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 15:04:11 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[123175]: 2024-04-06T15:04:11.864+0000 7fc126b49700 -1 osd.3 85 *** Got signal Terminated *** 2024-04-06T15:04:12.112 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 15:04:11 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3[123175]: 2024-04-06T15:04:11.864+0000 7fc126b49700 -1 osd.3 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T15:04:14.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:14 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:14.326+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:15.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:15 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:15.340+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:16.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:16 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:16.345+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:17.612 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:17 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:17.354+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:52.631495+0000 front 2024-04-06T15:03:52.631512+0000 (oldest deadline 2024-04-06T15:04:17.331425+0000) 2024-04-06T15:04:17.613 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:17 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:17.331+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:18.112 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 15:04:17 smithi086 bash[158328]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-3 2024-04-06T15:04:18.567 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:18 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:18.324+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:18.567 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:18 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:18.365+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:52.631495+0000 front 2024-04-06T15:03:52.631512+0000 (oldest deadline 2024-04-06T15:04:17.331425+0000) 2024-04-06T15:04:19.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.3.service' 2024-04-06T15:04:19.571 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:19 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:19.314+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:52.631495+0000 front 2024-04-06T15:03:52.631512+0000 (oldest deadline 2024-04-06T15:04:17.331425+0000) 2024-04-06T15:04:19.572 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:19 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:19.366+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:19.572 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 15:04:19 smithi086 systemd[1]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.3.service: Succeeded. 2024-04-06T15:04:19.572 INFO:journalctl@ceph.osd.3.smithi086.stdout:Apr 06 15:04:19 smithi086 systemd[1]: Stopped Ceph osd.3 for cabdc720-f41e-11ee-b649-cb9ed24678a4. 2024-04-06T15:04:19.585 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T15:04:19.585 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-06T15:04:19.586 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-06T15:04:19.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.4 2024-04-06T15:04:19.862 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:19 smithi086 systemd[1]: Stopping Ceph osd.4 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T15:04:20.312 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:19 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:19.922+0000 7f2ac2a76700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T15:04:20.313 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:19 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:19.922+0000 7f2ac2a76700 -1 osd.4 85 *** Got signal Terminated *** 2024-04-06T15:04:20.313 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:19 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:19.922+0000 7f2ac2a76700 -1 osd.4 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T15:04:20.313 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:20 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:20.327+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:52.631495+0000 front 2024-04-06T15:03:52.631512+0000 (oldest deadline 2024-04-06T15:04:17.331425+0000) 2024-04-06T15:04:20.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:20 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:20.384+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:21.612 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:21 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:21.352+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:52.631495+0000 front 2024-04-06T15:03:52.631512+0000 (oldest deadline 2024-04-06T15:04:17.331425+0000) 2024-04-06T15:04:21.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:21 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:21.354+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:22.612 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:22 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:22.313+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:52.631495+0000 front 2024-04-06T15:03:52.631512+0000 (oldest deadline 2024-04-06T15:04:17.331425+0000) 2024-04-06T15:04:22.613 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:22 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:22.402+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:22.613 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:22 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:22.402+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:23.612 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:23 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:23.322+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:52.631495+0000 front 2024-04-06T15:03:52.631512+0000 (oldest deadline 2024-04-06T15:04:17.331425+0000) 2024-04-06T15:04:23.612 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:23 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:23.322+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:03:57.331953+0000 front 2024-04-06T15:03:57.331961+0000 (oldest deadline 2024-04-06T15:04:22.631817+0000) 2024-04-06T15:04:23.613 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:23 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:23.364+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:23.613 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:23 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:23.364+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:24.612 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:24 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:24.294+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:52.631495+0000 front 2024-04-06T15:03:52.631512+0000 (oldest deadline 2024-04-06T15:04:17.331425+0000) 2024-04-06T15:04:24.612 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:24 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4[130062]: 2024-04-06T15:04:24.294+0000 7f2abb865700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:03:57.331953+0000 front 2024-04-06T15:03:57.331961+0000 (oldest deadline 2024-04-06T15:04:22.631817+0000) 2024-04-06T15:04:24.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:24 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:24.374+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:24.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:24 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:24.374+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:25.637 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:25 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:25.380+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:25.637 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:25 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:25.381+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:26.112 INFO:journalctl@ceph.osd.4.smithi086.stdout:Apr 06 15:04:25 smithi086 bash[158597]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-4 2024-04-06T15:04:26.862 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:26 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:26.392+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:26.862 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:26 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:26.392+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:27.596 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:27 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:27.403+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:27.596 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:27 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:27.403+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:27.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.4.service' 2024-04-06T15:04:27.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T15:04:27.648 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-06T15:04:27.648 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-06T15:04:27.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.5 2024-04-06T15:04:27.862 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:27 smithi086 systemd[1]: Stopping Ceph osd.5 for cabdc720-f41e-11ee-b649-cb9ed24678a4... 2024-04-06T15:04:28.362 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:27 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:27.982+0000 7f52abf53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T15:04:28.362 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:27 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:27.982+0000 7f52abf53700 -1 osd.5 85 *** Got signal Terminated *** 2024-04-06T15:04:28.362 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:27 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:27.982+0000 7f52abf53700 -1 osd.5 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T15:04:28.862 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:28 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:28.399+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:28.862 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:28 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:28.399+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:29.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:29 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:29.359+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:29.613 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:29 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:29.359+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:30.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:30 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:30.318+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:30.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:30 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:30.318+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:31.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:31 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:31.295+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:31.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:31 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:31.295+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:31.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:31 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:31.295+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-04-06T15:04:08.299480+0000 front 2024-04-06T15:04:08.299528+0000 (oldest deadline 2024-04-06T15:04:30.599348+0000) 2024-04-06T15:04:32.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:32 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:32.256+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-06T15:03:48.997495+0000 front 2024-04-06T15:03:48.997623+0000 (oldest deadline 2024-04-06T15:04:14.297576+0000) 2024-04-06T15:04:32.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:32 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:32.256+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-06T15:04:00.698662+0000 front 2024-04-06T15:04:00.698528+0000 (oldest deadline 2024-04-06T15:04:22.398542+0000) 2024-04-06T15:04:32.612 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:32 smithi086 ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5[136198]: 2024-04-06T15:04:32.256+0000 7f52a4d42700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-04-06T15:04:08.299480+0000 front 2024-04-06T15:04:08.299528+0000 (oldest deadline 2024-04-06T15:04:30.599348+0000) 2024-04-06T15:04:34.112 INFO:journalctl@ceph.osd.5.smithi086.stdout:Apr 06 15:04:33 smithi086 bash[158860]: ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4-osd-5 2024-04-06T15:04:35.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cabdc720-f41e-11ee-b649-cb9ed24678a4@osd.5.service' 2024-04-06T15:04:35.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T15:04:35.504 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-06T15:04:35.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-06T15:04:38.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[100693]: fuse finished with error 0 and tester_r 0 2024-04-06T15:04:54.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-06T15:05:05.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T15:05:05.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T15:05:05.583 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-06T15:05:05.585 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067/remote/smithi032/crash 2024-04-06T15:05:05.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash -- . 2024-04-06T15:05:05.623 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-06T15:05:05.623 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2024-04-06T15:05:05.624 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067/remote/smithi086/crash 2024-04-06T15:05:05.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash -- . 2024-04-06T15:05:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-06T15:05:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-04-06T15:05:05.652 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-06T15:05:05.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-04-06T15:05:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:grep: /var/log/ceph/cabdc720-f41e-11ee-b649-cb9ed24678a4/ceph.log: No such file or directory 2024-04-06T15:05:05.701 INFO:tasks.cephadm:Compressing logs... 2024-04-06T15:05:05.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T15:05:05.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T15:05:05.770 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-06T15:05:05.772 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-06T15:05:22.900 INFO:teuthology.orchestra.run.smithi086.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-06T15:05:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:gzip: /var/log/ceph/ceph-client.0.log: file size changed while zipping 2024-04-06T15:06:46.550 INFO:tasks.cephadm:Archiving logs... 2024-04-06T15:06:46.551 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067/remote/smithi032/log 2024-04-06T15:06:46.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T15:06:57.868 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067/remote/smithi086/log 2024-04-06T15:06:57.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T15:07:08.877 INFO:tasks.cephadm:Removing cluster... 2024-04-06T15:07:08.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 --force 2024-04-06T15:07:09.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cabdc720-f41e-11ee-b649-cb9ed24678a4 --force 2024-04-06T15:07:09.942 INFO:tasks.cephadm:Removing cephadm ... 2024-04-06T15:07:09.942 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-06T15:07:09.963 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-06T15:07:09.981 INFO:tasks.cephadm:Teardown complete 2024-04-06T15:07:09.982 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-06T15:07:09.996 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-06T15:07:09.997 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-06T15:07:09.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T15:07:10.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T15:07:10.102 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-06T15:07:10.114 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-06T15:07:10.114 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T15:07:10.117 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T15:07:10.133 INFO:teuthology.orchestra.run.smithi032.stderr:bash: ntpq: command not found 2024-04-06T15:07:10.134 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2024-04-06T15:07:10.137 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T15:07:10.138 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-06T15:07:10.138 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 98 -701us[ -694us] +/- 78ms 2024-04-06T15:07:10.138 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv02.front.sepia.ceph.com 3 7 377 35 +637us[ +644us] +/- 54ms 2024-04-06T15:07:10.138 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 100 -105us[ -98us] +/- 66ms 2024-04-06T15:07:10.138 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T15:07:10.139 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T15:07:10.140 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-04-06T15:07:10.140 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 -707us[ -707us] +/- 78ms 2024-04-06T15:07:10.140 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 7 377 37 +610us[ +611us] +/- 54ms 2024-04-06T15:07:10.140 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -154us[ -154us] +/- 67ms 2024-04-06T15:07:10.140 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T15:07:10.140 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-06T15:07:10.154 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-06T15:07:10.155 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-06T15:07:10.187 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-06T15:07:10.265 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-06T15:07:10.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T15:07:10.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T15:07:10.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T15:07:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T15:07:10.399 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-06T15:07:10.413 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-06T15:07:10.445 INFO:teuthology.task.internal:Duration was 4280.340496 seconds 2024-04-06T15:07:10.446 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-06T15:07:10.495 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-06T15:07:10.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T15:07:10.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T15:07:10.544 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T15:07:10.545 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T15:07:10.726 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-06T15:07:10.726 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-04-06T15:07:10.726 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T15:07:10.749 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-04-06T15:07:10.750 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T15:07:10.772 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-06T15:07:10.772 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T15:07:10.792 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T15:07:10.869 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-06T15:07:10.869 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T15:07:10.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T15:07:11.015 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-06T15:07:11.028 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-06T15:07:11.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T15:07:11.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T15:07:11.083 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-06T15:07:11.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T15:07:11.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T15:07:11.127 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-04-06T15:07:11.150 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-04-06T15:07:11.172 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T15:07:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T15:07:11.213 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T15:07:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T15:07:11.231 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-06T15:07:11.245 INFO:teuthology.task.internal:Transferring archived files... 2024-04-06T15:07:11.247 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067/remote/smithi032 2024-04-06T15:07:11.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T15:07:11.301 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067/remote/smithi086 2024-04-06T15:07:11.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T15:07:11.349 INFO:teuthology.task.internal:Removing archive directory... 2024-04-06T15:07:11.349 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T15:07:11.354 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T15:07:11.398 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-06T15:07:11.412 INFO:teuthology.task.internal:Not uploading archives. 2024-04-06T15:07:11.412 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-06T15:07:11.471 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-06T15:07:11.472 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T15:07:11.474 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T15:07:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: 525357 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 6 15:07 /home/ubuntu/cephtest 2024-04-06T15:07:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: 527607 4 d--------- 2 ubuntu ubuntu 4096 Apr 6 14:14 /home/ubuntu/cephtest/mnt.0 2024-04-06T15:07:11.490 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-06T15:07:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T15:07:11.491 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-06T15:07:11.492 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-06T15:07:11.492 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-06T15:07:11.507 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-06T15:07:11.538 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-06T15:07:11.538 INFO:teuthology.nuke:Checking targets against current locks 2024-04-06T15:07:11.562 DEBUG:teuthology.nuke:shortname: smithi032 2024-04-06T15:07:11.563 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T15:07:11.583 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 13:47:45.784755', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBUzMqv0R5sLiOVPnkaYLbbQGtnOuclmW8fIea0arHXtA/fEe7V0zvFUdlGemM9Ey00z8THJzr3NxzggwIskAR0='} 2024-04-06T15:07:11.587 DEBUG:teuthology.nuke:shortname: smithi086 2024-04-06T15:07:11.587 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T15:07:11.609 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7639067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 13:47:45.786645', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLm54Q5oAVlTy7szXP2arVi+qjf+j/XFM0EQrOPWyeaO39iyb3srDW+IlRv2FfOhae+FDQ1wsR0DIB2kPciUHQU='} 2024-04-06T15:07:11.633 INFO:teuthology.orchestra.console.smithi032:Power off 2024-04-06T15:07:11.633 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T15:07:11.661 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-04-06T15:07:11.661 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T15:07:11.679 INFO:teuthology.orchestra.console.smithi086:Power off 2024-04-06T15:07:11.679 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T15:07:11.690 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-06T15:07:11.704 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-04-06T15:07:11.704 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T15:07:11.733 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-04-06T15:07:15.691 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T15:07:15.734 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T15:07:15.801 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-06T15:07:15.844 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-04-06T15:07:19.806 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T15:07:19.845 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T15:07:19.917 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-04-06T15:07:19.957 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-04-06T15:07:20.019 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-04-06T15:07:23.958 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T15:07:24.069 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-04-06T15:07:24.169 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-04-06T15:07:24.274 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4280.340496063232 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-04-06T15:07:24.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T15:07:24.386 INFO:teuthology.run:FAIL